Recommended
More Related Content
Similar to Frontiers of Computational Journalism week 4 - Statistical Inference
Similar to Frontiers of Computational Journalism week 4 - Statistical Inference (18)
More from Jonathan Stray
More from Jonathan Stray (12)
Recently uploaded
Recently uploaded (20)
Frontiers of Computational Journalism week 4 - Statistical Inference
- 1. Frontiers of Computational Journalism Columbia Journalism School Week 4: Quantification and Statistical Inference October 3, 2018
- 2. This class âą Quantification âą Data Quality âą Risk ratios âą Regression âą Causation âą Interpretation
- 5. Different types of counting âą Numeric o Continuous or discrete o Units of measurement? o Non-linear scales? âą Categorical o finite, e.g. {true, false} o infinite e.g. {red, yellow, blue, ... chartreuseâŠ} o ordered?
- 6. Choices about what to count
- 11. GDP = C + I + G + (X - M)
- 12. 1940 U.S. census enumerator instructions
- 13. 2010 U.S. census race and ethnicity questions
- 14. Some things that are tricky to quantify, but usefully quantified anyway âą Intelligence âą Academic performance âą Race, ethnicity, nationality, gender âą Number of incidents of some type âą Income âą Political Ideology
- 15. Data Quality
- 16. Intentional or unintentional problems
- 17. It looks like Lucknow and Kanpur have few traffic accidents, but deaths data suggests that accidents are not being counted. Lies and Statistics: How Indiaâs Most-Populous State Fudges Crime Data, IndiaSpend
- 18. Evaluating Data Quality Internal validity: check the data against itself âą row counts (e.g. all 50 states?) âą related data âą histograms âą do the numbers add up? External validity: compare the data to something else. âą alternate data sources âą expert knowledge âą previous versions âą common sense!
- 19. Interview the Data âą Who created this data? âą What is this data supposed to count? âą How was this data actually collected? âą Does it really count what itâs suppose to? âą For what purpose was this data collected? âą How do we know it is complete? âą If the data was collected from people, who was asked and how?
- 20. âą Who is going to look bad or lose money because of this data? âą Is the data consistent with other sources? âą Is the data consistent from day to day, or when collected by different people? âą Who has already analyzed it? âą Are there multiple versions? âą Does this data have known problems? Interview the Data
- 21. Risk ratios
- 22. Deadly Force in Black and White, ProPublica 10/10/2014
- 23. AP Clinton Foundation Story WASHINGTON (AP) â More than half the people outside the government who met with Hillary Clinton while she was secretary of state gave money â either personally or through companies or groups â to the Clinton Foundation. Itâs an extraordinary proportion indicating her possible ethics challenges if elected president. At least 85 of 154 people from private interests who met or had phone conversations scheduled with Clinton while she led the State Department donated to her family charity or pledged commitments to its international programs, according to a review of State Department calendars released so far to The Associated Press. Combined, the 85 donors contributed as much as $156 million. At least 40 donated more than $100,000 each, and 20 gave more than $1 million. - Many donors to Clinton Foundation met with her at State, AP, 8/24/2016
- 25. Relative risk (risk ratio)
- 26. AP Clinton Foundation Story âAt least 85 of 154 people from private interests who met or had phone conversations scheduled with Clinton while she led the State Department donated to her family charity or pledged commitments to its international programs, according to a review of State Department calendars.â odds
- 27. AP Clinton Foundation Story odds Not enough information to compute the odds ratio... which you can tell immediately because four values are required.
- 28. Regression
- 29. Speed Trap: Who gets a ticket, who gets a break? Boston Globe, 2004
- 30. Speed Trap: Who gets a ticket, who gets a break? Boston Globe, 2004
- 31. Speed Trap: Who gets a ticket, who gets a break? Boston Globe, 2004
- 32. Nike Says Its $250 Running Shoes Will Make You Run Much Faster, New York Times
- 33. Surgeon Scorecard, ProPublica 2015
- 34. ACR = adjusted complication rate (reported in story) Surgeon Scorecard methodology paper, ProPublica 2015
- 35. Causal Models
- 36. Does chocolate make you smarter?
- 37. Occupational Group Smoking Mortality Farmers, foresters, and fisherman 77 84 Miners and quarrymen 137 116 Gas, coke and chemical makers 117 123 Glass and ceramics makers 94 128 Furnace, forge, foundry, and rolling mill 116 155 Electrical and electronics workers 102 101 Engineering and allied trades 111 118 Woodworkers 93 113 Leather workers 88 104 Textile workers 102 88 Clothing workers 91 104 Food, drink, and tobacco workers 104 129 Paper and printing workers 107 86 Makers of other products 112 96
- 40. Does marriage make women safer?
- 43. How correlation happens YX X causes Y YX Y causes X YX random chance! YX hidden variable causes X and Y YX Z causes X and Y Z
- 44. Guns and firearm homicides? YX if you have a gun, you're going to use it YX if it's a dangerous neighborhood, you'll buy a gun YX the correlation is due to chance
- 46. Beauty and responses YX telling a woman she's beautiful makes her respond less YX if a woman is beautiful, 1) she'll respond less 2) people will tell her that Z Beauty is a "confounding variable." The correlation is real, but you've misunderstood the causal structure.
- 47. A causal network. From Statistical Modeling: A Fresh Approach
- 48. What an experiment is: intervene in a network of causes
- 49. Does Facebook news feed cause people to share links?
- 51. Same data, different meaning
- 52. More than one true story
- 53. More than one true story
- 54. Crime in context, The Marshall Project