Data Cleaning with Python pandas: A Practical Tutorial

Data scientists often say most of their time goes into cleaning data, not modelling. Dirty data gives wrong dashboards and poor models. This tutorial cleans a typical messy leads file with pandas.

The messy data

leads.csv:

Lead ID, Name ,City,Course,Fee,Joined On
101,anitha r,Pune,python,25000,2026-01-05
102,RAVI,  bengaluru ,Python ,25000,05/01/2026
103,Sameer,,Data Science,,2026-01-09
103,Sameer,,Data Science,,2026-01-09
104,Priya,Chennai,power bi,2500000,2026-01-12

Problems: spaces in headers, inconsistent case, a missing city and fee, a duplicate row, mixed date formats and an impossible fee.

Step 1: Load and inspect

import pandas as pd

df = pd.read_csv("leads.csv")
print(df.shape)        # rows, columns
print(df.head())
df.info()              # column types and non-null counts
print(df.isna().sum()) # missing values per column

Always inspect before changing anything.

Step 2: Clean column names

df.columns = (
    df.columns.str.strip()
              .str.lower()
              .str.replace(" ", "_")
)
# ['lead_id', 'name', 'city', 'course', 'fee', 'joined_on']

Step 3: Remove duplicates

print(df.duplicated().sum())        # count exact duplicate rows
df = df.drop_duplicates()
# Or keep the latest record per ID:
df = df.drop_duplicates(subset="lead_id", keep="last")

Step 4: Standardise text

df["name"] = df["name"].str.strip().str.title()
df["city"] = df["city"].str.strip().str.title()
df["course"] = df["course"].str.strip().str.title()

" bengaluru " becomes "Bengaluru" and "python" becomes "Python", so grouping and counts are correct.

Step 5: Fix data types

df["fee"] = pd.to_numeric(df["fee"], errors="coerce")
df["joined_on"] = pd.to_datetime(df["joined_on"], format="mixed", dayfirst=True, errors="coerce")

errors="coerce" turns values that cannot be parsed into NaN/NaT instead of crashing. Then you can find and review them. (format="mixed" needs pandas 2.0 or newer.)

Step 6: Handle missing values

There is no single right answer; choose per column and document it.

df["city"] = df["city"].fillna("Unknown")
df["fee"] = df["fee"].fillna(df.groupby("course")["fee"].transform("median"))

Step 7: Detect outliers

A fee of Rs 25,00,000 for a Power BI course is almost certainly a typo. The IQR rule flags unusual values:

q1, q3 = df["fee"].quantile([0.25, 0.75])
iqr = q3 - q1
outliers = df[(df["fee"] < q1 - 1.5 * iqr) | (df["fee"] > q3 + 1.5 * iqr)]
print(outliers)

Investigate outliers before removing them. Sometimes they are errors; sometimes they are the most important rows.

Step 8: Validate and save

assert df["lead_id"].is_unique
assert df["fee"].between(1000, 500000).all(), "Fee out of expected range"
df.to_csv("leads_clean.csv", index=False)

Simple assertions catch problems when new data arrives next month.

Data cleaning checklist

1. Inspect shape, types, missing values and samples.
2. Standardise column names.
3. Remove duplicates (decide on the key).
4. Trim and standardise text.
5. Convert types (numbers, dates, categories).
6. Handle missing values per column, and record why.
7. Review outliers.
8. Validate with assertions and save a clean copy, never overwriting the raw file.

Interview questions

• `dropna` vs `fillna`? dropna removes rows or columns with missing values; fillna replaces them with a value or strategy.
• `loc` vs `iloc`? loc selects by label; iloc selects by integer position.
• How do you handle missing values in a model feature? Impute (mean, median, mode or a model), add an "is_missing" indicator, or use algorithms that handle NaN natively, depending on why data is missing.

Next steps

Download a public dataset and apply this checklist end to end. Continue with exploratory analysis and machine learning in the Data Science + AI course.