Python data cleaning pipeline project for your portfolio

Most Python portfolio projects are notebooks that ran once, in order, on data that was already clean. They demonstrate that you can call pandas functions.

This brief produces something considerably rarer and more persuasive — a pipeline that someone else can clone, run, and trust.

The brief

Take a genuinely messy public dataset and build a repeatable pipeline that turns it into an analysis-ready table, with tests, logging, and defined behaviour when the input is wrong.

The deliverable

A repository containing:

• A command-line entry point that takes an input path and an output path
• Cleaning logic in importable modules, not in a script body
• A validation step that rejects or quarantines rows failing defined rules
• Logging that records what was cleaned, what was dropped, and why
• A pytest suite covering the edge cases you actually found
• A README stating the data source, the assumptions made, and the known limitations
• A data dictionary for the output columns

It must run end to end from a clean checkout with pip install -r requirements.txt and a single command.

Constraints that make it worth doing

Nothing may be cleaned by hand. If you find yourself opening the CSV to fix something, that fix belongs in code.

Dropping a row must be logged with a reason. Silent data loss is the most common and least noticed defect in data work. Your log should let someone reconstruct exactly what was discarded.

The pipeline must be idempotent. Running it twice on the same input produces the same output. This sounds obvious and is easy to violate with append logic or timestamps.

Bad input must fail loudly. A missing column should raise a clear error naming the column, not a KeyError from three functions deep.

The messy cases to handle deliberately

This is the part that makes the project interesting. Real data contains all of these, and each is a talking point in an interview.

• Mixed date formats in one column — 01/02/2024, 2024-02-01, 1 Feb 24
• Numbers stored as text, sometimes with thousands separators or currency symbols
• Trailing and non-breaking whitespace that makes two identical-looking values unequal
• Inconsistent categorical spellings — Delhi, delhi, New Delhi, NCR
• Encoding problems producing mojibake in name fields
• Duplicate rows that are genuinely duplicated versus legitimately repeated transactions
• Blank versus zero versus null, which mean three different things and are frequently conflated

Document how you decided each one. The decision matters more than the code.

Step by step

1. Profile before cleaning. Write a profiling script that reports row counts, null rates, distinct values and inferred types per column. Commit its output. This becomes evidence of what the raw data looked like.

2. Define the target schema. Decide the output columns and types before writing transformations. Cleaning without a target is how pipelines sprawl.

3. Write validation rules. Which columns are mandatory, what ranges are plausible, which combinations are impossible.

4. Build transformations as small functions. One concern each, each independently testable.

5. Add logging. Counts in, counts out, counts dropped per rule.

6. Write the tests. For every messy case you found, a test with a tiny fixture proving your handling works.

7. Write the README. Source, assumptions, limitations, how to run it.

How to review your own work

• Does it run from a clean checkout on a machine that is not yours?
• If a mandatory column is missing, is the error message useful?
• Can you tell from the logs exactly how many rows were dropped and why?
• Do the tests fail if you deliberately break a transformation?
• Would someone reading the README understand the limitations without asking you?

The interview questions this project invites

“How did you decide what to drop?” The real question. Have a considered answer about validation rules and why you chose quarantine over deletion, or the reverse.

“What would break if the source format changed?” Shows whether you thought about maintenance. Good answer names the specific assumption and where it lives.

“Why a script rather than a notebook?” Reproducibility, testability, and the fact that notebooks let you execute cells out of order and fool yourself.

“What did you get wrong the first time?” Almost everyone hits one case they handled badly and had to redo — usually dates or duplicates. Saying so demonstrates you actually built it.

Why this project is worth more than another analysis

Analysis notebooks are abundant. A tested pipeline with logging and defined failure behaviour signals that you understand data work as engineering rather than as a one-off exercise — and that is the distinction between someone who produces a chart and someone who can be trusted with a recurring process.