Raw data often contains inconsistencies, gaps, or errors that can distort analysis and lead to misleading results. Data cleaning is the process of preparing raw data to ensure accuracy, consistency, and reliability. Below are the key techniques:

1. Handling Missing Values

Missing values are one of the most common data quality issues. If not properly addressed, they can bias results or reduce the accuracy of models. Approaches include:

Deletion:

Entire rows (records) or columns (variables) with excessive missing values are removed.
Works well when the dataset is large, and the proportion of missing data is small.
Risk: Potential loss of valuable information.

Imputation:

Missing values are replaced with estimates.
Simple methods: Use statistical measures such as mean, median, or mode.
Advanced methods: Use predictive models (e.g., regression, k-nearest neighbors) to estimate missing values based on other variables.
Benefit: Preserves dataset size.

Flagging:

Instead of filling or deleting, missing values are marked with a special code (e.g., -999, “NA”, or a Boolean flag).
Useful in cases where the fact that data is missing carries information (e.g., income not disclosed).

2. Removing Duplicates

Duplicate records artificially inflate data and may lead to skewed analysis.

Identification:

Look for repeated records (e.g., the same customer entered twice with slightly different details).
Use unique identifiers (like IDs) or similarity checks across multiple columns.

Resolution:

Keep only the first occurrence.
Or keep the most complete/accurate entry if duplicates differ.
In some cases, aggregate duplicates (e.g., summing transactions by the same customer).

3. Managing Outliers

Outliers are values that significantly deviate from the rest of the dataset. They may indicate errors, unusual cases, or meaningful rare events.

Detection:

Use statistical methods (e.g., values beyond 3 standard deviations from the mean, IQR rule).
Visualization tools like boxplots or scatter plots can highlight extreme points.

Decision Making:

Remove if the value is clearly erroneous (e.g., age = 200 for a human).
Correct if the outlier is due to data entry mistakes (e.g., salary entered as 1000000 instead of 100000).
Keep if the outlier is valid and relevant (e.g., very high sales from a major client).

M3:S3 - Data Cleaning Basics

PREVIOUS STEP

Data Quality

NEXT STEP