Margins of Error: public understanding of statistics in an era of big data
Upcoming SlideShare
Loading in...5
×
 

Margins of Error: public understanding of statistics in an era of big data

on

  • 1,669 views

Understanding statistics helps people make better decisions in their daily lives and improves public is course around government policies and their implications. Equally, misunderstanding of ...

Understanding statistics helps people make better decisions in their daily lives and improves public is course around government policies and their implications. Equally, misunderstanding of statistics can lead to the wrong conclusions and poor choices.

However, statistical literacy and trust in statistics remain relatively low for large proportions of the population. What are the implications of this for individuals and policy, and how can we improve public understanding and trust?

These slides were presented by Bobby Duffy (Ipsos MORI); John Pullinger (Royal Statistical Society), Andrew Dilnot CBE (UK Statistics Authority) and Professor Denise Lievesley (King's College London)

Statistics

Views

Total Views
1,669
Views on SlideShare
1,106
Embed Views
563

Actions

Likes
0
Downloads
14
Comments
0

3 Embeds 563

http://www.ipsos-mori.com 561
http://translate.googleusercontent.com 1
http://www.ipsos-rsl.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Margins of Error: public understanding of statistics in an era of big data Margins of Error: public understanding of statistics in an era of big data Presentation Transcript

    • MarginsMarginsf Eof ErrorJohn PullingerJohn Pullinger,President of the Royal Statistical Society© Ipsos MORI / King’s College London
    • Public trust andPublic trust andunderstandinggBobby DuffyBobby DuffyDirector, Ipsos MORI Social Research Institute,Visiting Senior Fellow, King’s College London© Ipsos MORI / King’s College London
    • Focus onunderstandingd l b tand value – butfirstly on trustfirstly on trust…© Ipsos MORI / King’s College London
    • Scientists and academics win...How much trust do you have in information provided by the following types ofpeople?28 46 3A great deal A fair amount None at allScientists18134542610AcademicsAccountants12837311215StatisticiansEconomists 893128158EconomistsActuaries212172354PollstersPoliticians© Ipsos MORI / King’s College LondonBase: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • Trust in scientists vs trust in clergy– a new age of reason?...would you generally trust them to tell the truth, or not?90Clergymen/Priests Scientists758085Clergymen/Priests Scientists% Yes657075505560404550303598 99 00 01 02 03 04 05 06 07 08 09 10 11 12* 13**© Ipsos MORI / King’s College LondonBase: c.1,000-2,000Source: Ipsos MORI most years face-to-face in-hom, *2012 ICM telephone ** 2013IM telephone
    • Trust in civil servants vs politicians – views havediverged......would you generally trust them to tell the truth, or not?Civil Servants Government Ministers Politicians Generally Journalists6070% Yes4050304010200© Ipsos MORI / King’s College LondonBase: c.1,000-2,000Source: Ipsos MORI most years face-to-face in-hom, *2012 ICM telephone ** 2013IM telephone
    • But government less trusted with our datathan online retailers?5A greatCompanies such assupermarkets and online5382A greatdealA fairsupermarkets and onlineretailers collect a lot of dataon their customers (forexample through loyalty384030A fairamountNot verycards). To what extent, if atall, do you trust companiesto use the data they collectabout you appropriately401241Not verymuchabout you appropriatelyThe government collects alot of data on citizens (forl th h t12620Not at all example through taxreturns). To what extent, if atall do you trust thegovernment to use the data66Dont knowgthey collect about youappropriately?© Ipsos MORI / King’s College LondonBase: c. 500 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • Big, technicalg,issues forpeople to cometo view on…© Ipsos MORI / King’s College London
    • ...not least, debt vs deficit...As you may know there is currently a lot of discussion about our national “debt” and “deficit”.Can you tell me what these words mean when talking about government finances?The difference between3Debt means Deficit meansThe difference betweenwhat government spendsand the income itreceives each year378478receives each yearThe total amount of moneythat the government owes4Both mean the sameDon’t know82 8282 82© Ipsos MORI / King’s College LondonBase: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • ...it is a tricky one...© Ipsos MORI / King’s College London
    • ...but public also not so clear when “use itin a sentence”...And can you tell me whether the following statement is true or false?“The national debt will always go down if the deficit is decreasing”20Those who got definitions rightThe national debt will always go down if the deficit is decreasing2820TRUEThose who got definitions rightno more likely to get this rightFALSEDont knowPublic think 40% of planned cutsalready been made52already been made...© Ipsos MORI / King’s College LondonBase: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • Basicunderstandingunderstandingof numbers isof numbers iskey to statisticalyliteracy – and iti i dis mixed…© Ipsos MORI / King’s College London
    • Most get very simple questions correct...What is 50 expressed as a percentage of 200?2010 89% t9210%25%2010: 89% correct92325%50%175%1OtherDont3Don tknow© Ipsos MORI / King’s College LondonBase: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • ...and slightly trickier...What is the average of the following three numbers – 5, 10 and 15?2010: 71% correct16705102010: 71% correct70110125153Other5Dont know© Ipsos MORI / King’s College LondonBase: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • ...but real difficulties with probabilities...If you spin a coin twice what is the probability of getting two heads?1 % 2010 30% t12615%25%2010: 30% correct240%58150%75% 1275%OtherStrong relationship with education (A-level+),10Dont knowStrong relationship with education (A level ),but also big differences by age, younger groupsmore likely to get right...© Ipsos MORI / King’s College LondonBase: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • There are alsoThere are alsoknown biases inhow we considert ti tistatistics…© Ipsos MORI / King’s College London
    • A personal optimism bias...What do you think the chance or probability is of the following being injured or killedin a road accident this year (whether as a road user or a pedestrian)?S i G t B it i Y231About 1 in 2About 1 in 5Someone in Great Britain You8621About 1 in 5About 1 in 10Ab t 1 i 20Mean probability:Someone = 4.1%Y 1 6%67723About 1 in 20About 1 in 50You = 1.6%Actual probability = c1.2%?72024519About 1 in 100About 1 in 100024234027About 1 in 10,000Dont know© Ipsos MORI / King’s College LondonBase: c. 500 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • ...but focus on negative informationImagine you have a life-threatening illness and your doctor has told you that you needan operation to treat it. How likely, if at all, are you to have this operation if yourdoctor tells you that...y90% of people who have the operation are alive for at least 5 years following the operation10% of people who have the operation die within 5 years of the operation563339Very likely333386Quite likelyNot very likely162y yNot at all likely Avoid targets on “negatives”,even if hit them? Waiting716Dont knoweven if hit them? Waitingtimes, immigration...© Ipsos MORI / King’s College LondonBase: c. 500 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • But does itmatter? Dol idpeople considerevidence – orevidence orthink their leadersdo?© Ipsos MORI / King’s College London
    • Principle-based policy-making...Politicians will take decisions partly based on what they think is right, and partly on evidence ofwhat works. Do you think they base their decisions more on what they think is right than onevidence, more on evidence than on what they think is right, or do you think they consider themb th i l ?both in equal measure?More on what they think is rightthan on evidence18than on evidenceMore on evidence than whatthey think is right16y gOn evidence and what theythink is right about the samet5216 amountDont know13© Ipsos MORI / King’s College LondonBase: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • ...but mirrors people’s own use ofevidencePeople have different attitudes towards statistics. Which of the following do you agreewith most?My own experiences or those of myfamily and friends are moreimportant than statistics in helping26Statistics are more important thanme keep track of how thegovernment is doing46my own experiences or those of myfamily and friends in helping mekeep track of how the government isdoing18Both equallyNeither/Don’t know9Neither/Don t know© Ipsos MORI / King’s College LondonBase: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013
    • More broadlyMore broadly,understandinggnumbers isundervalued?© Ipsos MORI / King’s College London
    • We’re not embarrassed about lack ofunderstanding of numbers...Which of the following things would you feel most embarrassed aboutadmitting to friends and family?6Im not very good with numbers15Im not very good at reading and writing75y g g gNeither 75Neither5Dont know© Ipsos MORI / King’s College LondonBase: 516 British adults aged 16-75, interviews conducted online 9th-15th April 2013 Source: RSS/Ipsos MORI 2013
    • ...and there’s little pride in doing it wellThinking about your child/if you had a child, which of the following wouldmake you most proud?13If they were very good with numbers55If they were very good at reading anditi5516writingN ith 16Neither15Dont know© Ipsos MORI / King’s College LondonBase: 516 British adults aged 16-75, interviews conducted online 9th-15th April 2013 Source: RSS/Ipsos MORI 2013
    • We’ve got a long way to go...I keep saying that the sexy job in the next 10 years willbe statisticians. And I’m not kidding.Hal Varian, chief economist at GoogleStatistical thinking will one day be as necessary forefficient citizenship as the ability to read or writeHG WellsValue of statisticsNumber of people reachedQuantity of statistical infoMedia affectRelevanceTrustNumeracyVAS = N * [(QSA * MF) * RS * TS * NL]Enrico Giovannini, Former Chief Statistician, OECD© Ipsos MORI / King’s College London
    • Thank youybobby duffy@ipsos combobby.duffy@ipsos.com@BobbyIpsosMORI© Ipsos MORI / King’s College London
    • Understanding andUnderstanding andTrust in StatisticsTrust in StatisticsAndrew Dilnot CBE,Andrew Dilnot CBE,Chair of the UK Statistics Authority© Ipsos MORI / King’s College London
    • GDP 1948-2012 (Index 2009=100)120100)6080009=100)4060ndex(2020I0© Ipsos MORI / King’s College London
    • GDP 2000-20131201006080009=100)4060Index(20200© Ipsos MORI / King’s College London
    • GDP 2000-20131101001059095009=100)8590Index(20758070© Ipsos MORI / King’s College London
    • GDP RevisionsAvailable data( )Preliminary Estimate Second Estimate Quarterly National Accounts(output measure)25 days approx 55 days approx 3 months© Ipsos MORI / King’s College London
    • Former minister slamsnational catastrophe ofnational catastrophe ofteenage mothersaddicted to benefits UK has highest teen pregnancy rate in p g yEuropeTEENAGE PREGNANCYSOARSNO SET OF VALUESSOARSFOR GYM-SLIPMUMS© Ipsos MORI / King’s College LondonMUMS
    • Under 18 conception rate forEngland and Wales© Ipsos MORI / King’s College London
    • © Ipsos MORI / King’s College LondonNorovirus
    • Norovirus lab reports© Ipsos MORI / King’s College LondonSource: Health Protection Agency
    • Norovirus confidence intervals• 1:1500 (1 lab case = 1500 in community).• 2000 lab cases = 3million in community• But maybe 1:140But maybe 1:140• (=280,000 cases)• Or maybe 1:17,000• (=34 million cases)• Community study lab cases…• =1© Ipsos MORI / King’s College London
    • © Ipsos MORI / King’s College London
    • The 2011 Census and uncertainty1 4Relative Confidence Interval width1.21.4t)0 81(percent0.60.8alwidth(0 20.4Interva00.2England South East Kent Canterbury© Ipsos MORI / King’s College LondonEngland South East Kent Canterbury
    • Trends in police recorded crimeand CSEW© Ipsos MORI / King’s College London
    • Lies, damn liesCrimeand crimestatisticsstatisticswerestatisticsPOLICE FAIL weredistorted byPOLICE FAILTO RECORD ypoliticsTO RECORDCRIMEPROPERLY© Ipsos MORI / King’s College London
    • Crime falls to new low despite recession and unemploymentrecession and unemployment...The 6% fall in crime reported in the latest quarterly p q yfigures by both the Crime Survey for England and Wales and the separate police recorded crime figures means that crime has now dropped by more than 50 % since it peaked in the mid‐1990s...© Ipsos MORI / King’s College LondonThe Guardian, 19 October 2013
    • Public Understanding oft ti ti i f bistatistics in an era of bigdatadataDenise Lievesley,Denise Lievesley,Head of School of Social Science and Public Policy,King’s College London© Ipsos MORI / King’s College London
    • Challenges facing statisticiansH ilit C fidR lHumilityA tConfidencevsRelevanceT tAutonomyS i ivsTrust ScepticismvsMeasurement QualityvsPragmatism Purismvs© Ipsos MORI / King’s College London
    • Humility vs. ConfidenceBeing a statistician meansnever having to sayyou’re certain© Ipsos MORI / King’s College London
    • Humility – being aware of our limitations“Good science should not turn a blind eye to known imperfections –nor should these be concealed from users”Sir Roger Jowell 2007“The absence of excellent evidence does not make evidence-baseddecision making impossible: what is required is the best evidenceavailable not the best evidence possible”Si M i G 199Sir Muir Gray 1997© Ipsos MORI / King’s College London
    • ISI declaration on professional ethics 1985• One of the most important but difficult responsibilities of thet ti ti i i th t f l ti t ti l f th i d t t thstatistician is that of alerting potential users of their data to thelimits of their reliability and applicability. The twin dangers ofeither overstating or understating the validity or generalisabilityeither overstating or understating the validity or generalisabilityof data are nearly always present.• Confidence in statistical findings depends critically on theirfaithful representation. Attempts by statisticians to cover upi i i i l b derrors, or to invite over- interpretation, may not only reboundon the statisticians concerned but also on the reputation ofstatistics in generalstatistics in general.© Ipsos MORI / King’s College London
    • Confidence –using the data to make a difference•We need to provide information of high quality,We need to provide information of high quality,integrity and robustness which can be relied on.•We should be confident about our findings andprepared to account for them.p p© Ipsos MORI / King’s College London
    • CommunicationWe need to improve our communication skillsand think about impact.We should learn how to tell a story with datayand remember that communication is not whatis delivered but what is receivedis delivered but what is received.e.g.• Bill Gates has a personal fortune greater than the combinedwealth of the 106 million poorest Americans.• The cost of putting all children into school is less than is spenton icecream in Europe each year© Ipsos MORI / King’s College London
    • Sir Gus O’Donnell(former UK Cabinet Secretary)“I want [the ONS] to be boring, to put out the plain facts, andnothing but the facts and on clear predictable deadlines ” henothing but the facts, and on clear, predictable deadlines, hesaid. It would then be for politicians and government pressofficers to interpret the figures, he added.p g ,© Ipsos MORI / King’s College London
    • Response of the Royal Statistical Society• it is clearly the task of statisticians to interpret the figuresin a statistical context, to facilitate understanding andavoid misunderstanding.• The Code of Practice of the UK Statistics Authorityexplicitly states that Official statistics accompanied byexplicitly states that Official statistics, accompanied byfull and frank commentary, should be readily accessibleto all users and that all UK bodies that are responsibleto all users and that all UK bodies that are responsiblefor official statistics should prepare and disseminatecommentary and analysis that aid interpretation andcommentary and analysis that aid interpretation, andprovide factual information about the policy oroperational context of official statistics© Ipsos MORI / King’s College Londonoperational context of official statistics.
    • Relevance vs. AutonomyUN Fundamental Principles of Official StatisticsPrinciple 1“Offi i l t ti ti id i di bl l t i th“Official statistics provide an indispensable element in theinformation system of a democratic society, serving theGovernment the economy and the public To this end officialGovernment, the economy and the public ... To this end, officialstatistics that meet the test of practical utility are to be compiledand made available on an impartial basis by official statisticalagencies..”© Ipsos MORI / King’s College London
    • Impartiality• The role of statisticians: to inform political debate and decisionswithout taking partt out ta g pa t• Fear that enhancing statistical utility will compromise impartiality• There must be no political interference with the data and noperception that there isperception that there isBut does this mean we are too cautious?Are statisticians so afraid of being accused of political motivesgthat they dare not make reports useful for the public debate?© Ipsos MORI / King’s College London
    • The value of statistics to society must notjust be asserted; it must be demonstrated“Were a balance sheet for official statistics to be prepared the“Were a balance sheet for official statistics to be prepared, thecosts would be clear enough. The benefit, or value, wouldhowever be found to be much more diffuse and harder to treat inhowever be found to be much more diffuse and harder to treat intraditional accounting terms. Given this, it is possible that thevital asset that official statistics represent is undervalued inpublic sector planning processes. And we observe that littlesystematic consideration is given to how the public value couldbe maximised”be maximised .(UK Statistics Commission, The Use Made of Official Statistics,© Ipsos MORI / King’s College London(UK Statistics Commission, The Use Made of Official Statistics,2007)
    • Trust vs. Scepticism• Pre-requisite for evidence based policy and formanaging for results is that the data must be trustworthy© Ipsos MORI / King’s College London
    • But it is not enough that the data aretrustworthy they must also be trusted• Otherwise they won’t be used• There will be fights about the data rather than about theissues• Data need to be the currency of public debates© Ipsos MORI / King’s College London
    • Evidence sometimes resisted...“There is nothing a governmentThere is nothing a governmenthates more than to be well-informed: for it makes the processof arriving at decisions much moreof arriving at decisions much morecomplicated and difficult.”pJohn Maynard Keynes© Ipsos MORI / King’s College London
    • Inconvenient truths• Governments prefer good news stories• Bad news stories may be delayed or buried• They are often too focussed on populism• They are often too focussed on populism• The government’s horizons can be shorter than those ofi i i !statisticians!• They can prefer their own spin to that of the statisticiany p p© Ipsos MORI / King’s College London
    • Important aspects of building trust• Autonomy of statisticiansSt ti ti l l i l ti• Statistical legislation• Existence of an independent statistical boardD l t f d f d t• Development of codes of conduct• Breaches of the code identified, investigated and publicised• Appointment of senior statisticians removed from the politicalprocessU h ld b i l d i tti th d ( ki th• Users should be involved in setting the agenda (asking theawkward questions)• External audits of the statistical processes should be employed• External audits of the statistical processes should be employed• Audit body should report to Parliament© Ipsos MORI / King’s College London
    • Measurement vs. Quality• Statisticians need to guard against “what can’t be measured isn’treal”• The danger with a measurement culture is that excessive attentioni i t h t b il d t th f h t iis given to what can be easily measured, at the expense of what isdifficult or impossible to measure quantitatively even though thismay be fundamentalmay be fundamental.© Ipsos MORI / King’s College London
    • Challenges to integrity –the rise of performance monitoring• Performance data can be used in establishing whatorks among polic initiati es to identif ell performingworks among policy initiatives; to identify well-performingor under-performing institutions and public servants; and,equally important to hold Ministers to account for theirequally important, to hold Ministers to account for theirstewardship of the public servicesH t i b th it i th bli i• Hence, government is both monitoring the public services,and being monitored, by performance indicators.• Because of governments dual role, performancemonitoring must be done with integrity and shielded fromundue political influence© Ipsos MORI / King’s College London
    • Hitting the target but missing the pointhtt // k/PDF/P f M it i df© Ipsos MORI / King’s College Londonhttp://www.rss.org.uk/PDF/PerformanceMonitoring.pdf
    • Audit Commission report“What makes a target ‘good’ is not just the way a target isWhat makes a target good is not just the way a target isexpressed—it’s about the way it was derived, the extentto which service users were involved in its developmentto which service users were involved in its development,the extent to which it helps to achieve policy objectives,the extent to which it has the support of the staff whosethe extent to which it has the support of the staff whoseefforts will achieve it, the quality of the data used tomeasure its achievement and the clarity andmeasure its achievement, and the clarity andtransparency of its definition”© Ipsos MORI / King’s College London
    • Pragmatism vs. Purism• To what extent should we exploit data from a widerTo what extent should we exploit data from a widerrange of sources?• May allow us to produce more timely data at lower cost• Opportunities provided by BIG data© Ipsos MORI / King’s College London
    • Fundamental changes to data sources might needto involve review as to the nature of evidence• Use of ‘free form’ data raises questions about how toi h li d i i d i h hcommunicate the quality and uncertainty associated with theevidence• In the context of some moves towards greater formalisation ofevidence (such as randomised control trials)It does not remo e the need for SCIENCE• It does not remove the need for SCIENCE© Ipsos MORI / King’s College London
    • The use of big data brings challenges?• Need programmes of work on the technical and analyticchallenges especially relating to data qualitychallenges especially relating to data quality• But also on• Communication and dissemination of statistics• Culture of statistical agencies• Culture of statistical agencies• Trust of the public• Changing relationships with users and providers• The responsibilities of official statisticians• The meaning of privacy in this new world• etc© Ipsos MORI / King’s College London• etc.
    • Develop statisticians for the future• Foster adaptability• Transferable skills• Build research and innovation skills• Create a cadre of people who challenge pre-Create a cadre of people who challenge pre-conceptions• Not to mould them in our own image• Nor to create homogeneous communities• Nor to create homogeneous communities• Education is about opening minds not closing them© Ipsos MORI / King’s College London
    • © Ipsos MORI / King’s College London