Module 5 — Data: The Fuel of AI

Every AI system depends on data. The quality, quantity, and relevance of data fundamentally determine what an AI can learn and how well it performs. The principle of "garbage in, garbage out" is the single most important concept in applied AI.

Structured data is organised in tables with rows and columns — databases, spreadsheets, CSV files. It's easy to analyse but represents only about 20% of the world's data. Examples: sales records, customer databases, financial transactions.

Unstructured data has no predefined format — text documents, images, audio, video, social media posts. It represents about 80% of all data and is where much of the richest information lies, but it's harder to process and analyse. AI techniques like NLP and computer vision have made this data increasingly accessible.

Semi-structured data falls in between — it has some organisation but doesn't fit neatly into tables. Examples: JSON, XML, HTML, email (has fields like sender, date, subject, but the body is unstructured), and log files.

Data preparation typically consumes 60-80% of a data scientist's time. Key steps include:

Data cleaning: Handling missing values (remove, impute, or flag), correcting errors, removing duplicates, standardising formats (dates, currencies, units).

Feature engineering: Creating new variables from existing data that help the model learn. For example, combining "date of birth" into "age," extracting "day of week" from a timestamp, or calculating "average purchase frequency."

Data transformation: Normalising or standardising numerical features so they're on comparable scales, encoding categorical variables (converting "Red/Blue/Green" into numerical format).

Train-test split is fundamental: you divide your data into a training set (typically 70-80%, used to train the model), a validation set (10-15%, used to tune the model during development), and a test set (10-15%, held back and used only for final evaluation). This prevents overfitting — when a model memorises the training data rather than learning generalisable patterns.

Data governance ensures data is managed as a strategic asset: who owns the data? Who can access it? How is quality maintained? How long is it retained? Is it compliant with GDPR and other regulations? Strong governance is essential for trustworthy AI.