MEPX
Chapter 4 of 13All chapters
  1. 1What analysis is for
  2. 2Types of data
  3. 3Collecting and sampling
  4. 4Cleaning data
  5. 5Describing the middle
  6. 6Describing the spread
  7. 7Choosing a chart
  8. 8Correlation
  9. 9Getting to causation
  10. 10Hypothesis testing
  11. 11Traps and paradoxes
  12. 12Tools of the trade
  13. 13Communicating results

Chapter 4 of 13

Cleaning data

Most of the work, and it shapes the result.

The usual jobs

Missing values, duplicates, inconsistent formats, mixed units and impossible entries all have to be dealt with before any number means anything.

  • Decide and record how missing values are handled: dropped, filled or flagged.
  • A duplicate row silently doubles the weight of one observation.

Keep a record

Every cleaning decision is an analytical decision. Doing it in a script rather than by hand means it can be reviewed, repeated and corrected.