Simpson’s paradox
Treatment A wins in every group, and loses overall.
The idea
A trend can appear in every subgroup of the data and reverse when the groups are combined. It happens when a hidden variable is unevenly spread across the groups.
A well-known case comes from UC Berkeley’s 1973 admissions data: overall, men appeared to be admitted at a higher rate, but within most departments the difference disappeared or favored women. Women had applied more often to the most competitive departments.
What the lesson will build
- A small dataset where the overall trend and the group trends disagree
- A groupby that reveals the reversal
- A chart of both views side by side
Key ideas
- Confounding variables
- Aggregation
- Why to segment before concluding
- Weighted averages
The video
- Long form: One trend line pointing up, then the data splitting into colored groups whose lines all point down.
- Short: Every group says one thing. The total says the opposite.
When it’s published, the code will live in exploratory-data-analysis/ and this page will link to it.