The document discusses issues with some opinion polls and survey methods. It notes that the American Association of Public Opinion Research has established a task force to reassess current survey methods and provide guidance on improving reliability and transparency. Additionally, SurveyMonkey has assembled an advisory committee to review survey methods. However, the document expresses skepticism about whether these groups will truly address issues or find their own methods lacking. It provides several examples where past opinion polls have proven unreliable or inaccurate in predicting election outcomes.

1. It’s reallya matter of opinion
Russell Grenning
You may be surprisedtolearnthatthe AmericanAssociationof PublicOpinionResearch (AAPOR)
has establishedaTaskForce to reassesscurrentsurveymethodsandprovideguidance onthe types
of informationsurveypractitionersandendusersneed.
But whyshouldyoube surprised?Well,itis possible thatthisTaskForce will come toconclusions
that at leastsome of the Association’smembers’surveyingtechniquesare ratherlessthan
transparent, ethical and– mostlyimportantly –reliableandhonest. Talkaboutbitingthe handthat
feedsyouandall of that. This will shockyouIknow but some so-called“opinionpolls”are nothing
but self-servingbitsof propaganda.
Once upona time,we hadadvertisementssuchas “Nineout of ten doctorspreferCamels” which
allegedly claimedthe overwhelmingmajorityof medical practitionerslikedthisbrandof cigarette
and not,as it mightappearto youngerreaders, anindicationof theirfavouredmeansof transport
across the Sahara. Nowadays,we have similardubiousadsclaimingthat afterusingsome miracle
product,“Nine outof ten women reported thatthey had smoother,softerskin afterusing
WrinkleErase”.
Both typesof ad have a commonancestryand both use the same oldtechnique of having a
reassuringpicture orfilmof a distinguishedlookingmiddle-agedmanwithawide,brightand
reassuringsmile, asilverymane of hair,a twinkle inthe eye andwearingacrispwhite coatcomplete
withstethoscope aroundhisneck.A more recentbastardchildof thisgenre featuresattractive
youngwomeninsimilargarbwithseriousspectaclesandsensible shoestoprove thattheyare not
bimboes spruikingthe latestscientificbreakthroughinthe beautybusiness.Oftentheycarrya
clipboard,orlookup froma test tube or smile approvinglyatawomanwho isallegedly47 butlooks
19 - “ You won’tbelieve thatJenny hasfourchildren and two grandchildren!” Well,noIdon’t.
I understandthatindependentscientific surveyshave proventhatindependentscientificsurveyscan
prove anything.
Call me a nervyoldthingbutI am not convincedbyadsthat claima miracle kitchen/bathroom
productwill “kill 99.9% of all bacteria” as I have a naggingfearthat the other 0.1% will strike me
down.
However,backto AAPOR.
Theyrecentlyissuedarebuke to The New York Times forputtingitsname to internetpollsrunbyits
partners CBS News andYouGov for,it alleged,the abandonmentof “essentialstandards.”
Simultaneously,SurveyMonkey,whichmodestlydescribesitself as “theworld’sleading on-line
survey platform”,announceditwasestablishingaTask Force “to reassessthecurrent stateof survey
methods(bothnewand traditional)” andthatan “amazing teamof people” hadbeenassembledfor
itsSurvey Advisory Committee.

2. Lucky themandI feel confidentthatthis “amazing teamof people”will decide thatSurveyMonkeyis
pre-eminentinitsfield,embodyingall of the finestprinciplesof decency,transparency,
professionalism,honestyandeverythingelse uptoandincludingMomand apple pie. Incertain
critical instances,opinionpollshave shownthatyoushouldn’tholdopinionpollsif youdon’talready
knowthe outcome.
It wouldn’tsurprise me if the otherswere foundtohave seriousdeficiencies,fatal flawsand
compromisedstandardstoname butthree weaknesses.
I feel sure thatif an opinionpoll wastakenof opinionpollsters,itwouldconclude thatopinionpolls
for clientswithcommercialproductstoflogshouldnotbe conductedexceptunderwhatcanbe
euphemisticallydescribedas “controlled conditions.” However,thatisjustmyopinion.
I don’tknowwhothe Editor of The Harrisburg Pennsylvanian wasin1824 but the man hasa lotto
answerfor– it washe who commissionedthe firstrecordedpublicopinionpoll whichshowed
AndrewJacksonleadingJohnQuincyAdamsby335 votesto 169 in the race for the US Presidency.
The poll was spot-onanda whole newindustrywasspawned.
There have beensome spectacularfailuresinthe political publicopinionpoll businessoverthe
years.The prestigiousandhighlyrespectedAmericanjournal Literary Digestmade a name foritself
for conductingopinionpollswhichaccuratelypredictedthe electionof PresidentsWilson(1916),
Harding(1920), Coolidge (1924),Hoover(1928) and Roosevelt(1932).
In 1936, usingthe same pollingtechnique,the magazinepredictedthe RepublicanGovernorof
KansasAlf LandonwouldcrushPresidentRooseveltby57% to 43%. Alf who?
Well,the magazine mailedoutsome tenmillioncardsinvitingrecipientstowrite backwiththeir
nominee forPresidentandsome 2.4milliondid.Itwas- andremains - the largesteverpublicopinion
poll sample. Howdidtheygetitso wrong?Theytookaddressesfromtelephone directories,club
memberships,magazinesubscribers andcarownersand othersourceswhichhada built-inbiasin
favourof the middle andupperclassesata time of still-rampantunemploymentandwhenthe car,
the telephone andall of the above were luxuries.
The magazine’scredibilityneversurvivedanditwentbankrupt.
In 1948, pollsterspredictedthatRepublicanThomasDewywoulddefeatPresidentHarryS Truman.
Thomaswho?On electionnightasthe numberswere comingin,the Chicago Daily Tribuneran a
headline, “Dewey DefeatsTruman”andthe famouspicture of the re-electedTrumanholdingthe
paperhighand witha grin a sledgehammercouldn’tstopshouldbe onthe wall of everypolitical
pollsterinthe world.
Otherpaperswere justas guilty – The New York Times ran “ThomasE Dewey’sElection As President
Is A ForgoneConclusion”while Lifemagazine rana picture of Deweywiththe caption, “TheNext
PresidentOf The United States”. Ouch!

3. In Australia,there have beensome notablepublicopinionfailures–earlyinPMHawke’sreign,his
approval ratingof 75% was inthe stratosphere sohe wentto an earlyelectionin1984 and onlygot
back witha reducedmajority.In1980, opinionpollspredictedthe defeatof the FraserGovernment
yethe wentonto winhisthirdterm,
In the QueenslandState electionin1986, the pollssaidthatthe National PartyBjelke-Petersen
Governmentwouldbe defeated –theyweren’t.Inanecho of Truman’striumphin1948, Bjelke-
Petersenenteredthe tallyroomholdinganearlyeditionof the nextday’s Sunday Sun withits
headline, “Joh in Peril”.
To be fair,political opinionpollsinAustraliahave generallyproventobe more accurate than many
overseaspollshave beenandthere appearstobe fundamentallyone reasonfor this–compulsory
voting.
In 1992, the majoropinionpollsonthe dayof the electionpredicteddefeatforthe Conservative
Governmentyetitwonand thenfollowedatwo-yearinvestigationbythe MarketResearch Society
of Great Britain whichconcludedthatTory voterswere “morereluctantto be interviewed or to say
howthey would votethan Labourvoters”,that the sampleschosenwere “too skewed to traditional
Labourvoters”,thatmany Labour votersfailedtovote andthatthere had beena late swingtothe
Conservatives.
All toooften,political opinionpollsprovetobe theirownworstenemy – theycan produce varying
reactionsfromthe “bandwagon”effect(“let’sall getwith the winners!”), the “underdog”effect
(”The otherside shouldn’thavetoo big a majority”) andthe “backlash”effect(“Thatsettlesit, I’m
notvoting forthem”).
Howeverscientifictheirapproachwhenpollsterstrytodetermine the mostrepresentativesample,
theyrun unavoidablerisks.Respondentstoapoll mayfeel flatteredby the attention,annoyedby
the interruptionorthreatenedbythe challenge whichcanall influencethe answerstheygive orthey
mightjustbe in a lousymood.
Some respondents,forexample,mayfeara negative reactionfromthe pollsterif theyhonestly
acknowledge thattheyintendtovote forsome minorextremistparty.Some maysaytheyare voting
for a particularparty onlytogive themfalse hope whilefullyintendingtovote the otherwayon
electionday.
Andit all dependsonhowyouask the question.A poll by TheNew York Times in 1992 askeda
representative sample if theybelievedmore publicmoneyshouldbe spenton “welfare”andonly
23% saidyes.Whenaskedif more publicmoneyshouldbe spenton “assistanceforthepoor”,two-
thirdsagreed.
Whena newspaperisshortof real news,itcan alwayscommissionsomepoll seekingananswertoa
questionlike, “Should child rapistsbeexecuted?” and thendrumup a campaignunderthe headline,
“Shockverdict – the people say kill this scum”. Everypublicity-hungrypopulistwill be gratefulfor
free coverage.
I cannot butagree withOscar Wilde whoobserved,“By giving ustheopinionsof the uneducated,
journalismkeepsusin touch with the ignoranceof thecommunity.”