The Gory Details Of Datajournalism: what went wrong and who was responsible by Henk van Ess

Data Reality Check Henk van Ess henk@vaness.nl chairman WWW.VVOJ.NLSunday, March 25, 12

This year: a new book & site about data journalism Age of Data by Geoff McGhee Paul Bradshaw Digital Diggers Project Data Driven Stories Book http://www.vvoj.nl Andrew Lehren, NYTSunday, March 25, 12

Sunday, March 25, 12

Let’s cut the crapSunday, March 25, 12

Let’s cut the crap Who was in charge and what went wrong?Sunday, March 25, 12

Who was in charge and what went wrong?Sunday, March 25, 12

Who was in charge and what went wrong? The Coder The Reporter The DesignerSunday, March 25, 12

Who was in charge and what went wrong? The Coder The Reporter The Designer Showing what is possible, not what is sensibleSunday, March 25, 12

Who was in charge and what went wrong? The Coder The Reporter The Designer Assembly line syndrom: no place for new workﬂowsSunday, March 25, 12

Do you use tools like:Sunday, March 25, 12

Do you use tools like: beforeSunday, March 25, 12

Do you use tools like: before List of dangerous chemical companiesSunday, March 25, 12

Do you use tools like: before after List of dangerous chemical companiesSunday, March 25, 12

Do you use tools like: before after List of dangerous chemical companies Adobe CaptivateSunday, March 25, 12

There are many new dedicated toolsSunday, March 25, 12

There are many new dedicated tools Timeline the bad waySunday, March 25, 12

There are many new dedicated tools Timeline the bad way Timeline with Timeline 3DSunday, March 25, 12

There are many new dedicated tools This was done in 30 minutes Timeline the bad way Timeline with Timeline 3DSunday, March 25, 12

Who was in charge and what went wrong? The Coder The Reporter The Designer Mind boggling presentations.. leaving your mind boggledSunday, March 25, 12

Who was in charge and what went wrong? The Coder The Reporter The Designer Showing problems, not explaining themSunday, March 25, 12

when no one is in charge (anymore)Sunday, March 25, 12

when no one is in charge (anymore) empty pagesSunday, March 25, 12

when no one is in charge (anymore) broken databases empty pagesSunday, March 25, 12

Datajournalism isSunday, March 25, 12

Datajournalism is Finding informationSunday, March 25, 12

Datajournalism is Finding information Processing informationSunday, March 25, 12

Datajournalism is Finding information Processing information Presenting informationSunday, March 25, 12

Datajournalism is Finding information Processing information Presenting information Put like that, it seems the job description for any reporterSunday, March 25, 12

To be more speciﬁc, you need:Sunday, March 25, 12

To be more speciﬁc, you need: A ﬁne researcherSunday, March 25, 12

To be more speciﬁc, you need: A ﬁne researcher A smooth talkerSunday, March 25, 12

To be more speciﬁc, you need: A ﬁne researcher A smooth talker A good number cruncherSunday, March 25, 12

To be more speciﬁc, you need: A ﬁne researcher A smooth talker A good number cruncher A great story tellerSunday, March 25, 12

To be more speciﬁc, you need: A ﬁne researcher A smooth talker A good number cruncher A great story teller A brilliant designer/ GUI specialistSunday, March 25, 12

To be more speciﬁc, you need: A ﬁne researcher A smooth talker A good number cruncher A great story teller A brilliant designer/ GUI specialist Fantastic communicatorSunday, March 25, 12

To be more specific, you need: A fine researcher A smooth talker A good number cruncher A great story teller A brilliant designer/ GUI specialist Fantastic communicator “overlapping set of competencies drawn from disparate fields” http://bit.ly/nLlbYnSunday, March 25, 12

To be more specific, you need: A fine researcher A smooth talker A good number cruncher A great story teller A brilliant designer/ GUI specialist Fantastic communicator You offer: “overlapping set of competencies drawn from disparate fields” http://bit.ly/nLlbYnSunday, March 25, 12

Your problem Data journalism is a “baggy monster of a ﬁeld” 5 tips for getting started in data journalism http://www.poynter.org/how-tos/digital-strategies/147734/5-tips-for-getting-started-in-data-journalism/Sunday, March 25, 12

We can learn from Nokia, for once : Go for teamwork!Sunday, March 25, 12

The gory detailsSunday, March 25, 12

The problemSunday, March 25, 12

The problem Reporters bored with WikileaksSunday, March 25, 12

WHY? Most explosive documents were already publishedSunday, March 25, 12

WHY? Finding new leads in new documents was hardSunday, March 25, 12

Colleagues told me: ‘It isn’t worth our time’Sunday, March 25, 12

Me, being shockedSunday, March 25, 12

I said: Me, being shockedSunday, March 25, 12

I said: If you don’t check this, who does????? Me, being shockedSunday, March 25, 12

How we started:Sunday, March 25, 12

How we started: No inhouse codersSunday, March 25, 12

How we started: No inhouse coders No designersSunday, March 25, 12

How we started: No inhouse coders No designers No fundingSunday, March 25, 12

How we started: No inhouse coders No designers No funding No technical infrastructureSunday, March 25, 12

How it ended: http://cablesearch.org/?page_id=46Sunday, March 25, 12

So what happened here?Sunday, March 25, 12

Text www.cablesearch.orgSunday, March 25, 12

How it started http://screencast.com/t/1scMTaBgZytFSunday, March 25, 12

Why did this succeed?Sunday, March 25, 12

Why did this succeed? It solved a problemSunday, March 25, 12

Why did this succeed? It solved a problem For news beatsSunday, March 25, 12

Why did this succeed? It solved a problem For news beats By tags, for specialist & background storiesSunday, March 25, 12

Why did this succeed? It solved a problem Sharing with public gave us great new ideasSunday, March 25, 12

Why did this succeed? It solved a real problem Crowdsource! The Business plan paid for infrastructureSunday, March 25, 12

Even more gory detailsSunday, March 25, 12

Volkskrant:Sunday, March 25, 12

Volkskrant: • Newspaper demanded tons of data about the state of elderly homes in The NetherlandsSunday, March 25, 12

The responseSunday, March 25, 12

VolkskrantSunday, March 25, 12

Volkskrant • Pushed over and over again to get the dataSunday, March 25, 12

Volkskrant • Invoked the freedom of information actSunday, March 25, 12

Volkskrant • They kept pushing 39Sunday, March 25, 12

After one yearSunday, March 25, 12

After one year Finally. Succes!Sunday, March 25, 12

A cd full of dataSunday, March 25, 12

• 270.000 questions • 320.000 answers • Devided in 9 categories • More then 12 subcategoriesSunday, March 25, 12

Overwhelmed by data: • 270.000 questions • 320.000 answers • Devided in 9 categories • More then 12 subcategoriesSunday, March 25, 12

Minor detail: all 4500 forms are PDFSunday, March 25, 12

So what now?Sunday, March 25, 12

It’s 2012. Don’t despairSunday, March 25, 12

Anatomy of a project I posted the pitch for the project: “Need coder for converting 4500 PDF’s”Sunday, March 25, 12

One minute laterSunday, March 25, 12

The Whole World Is BiddingSunday, March 25, 12

And BiddingSunday, March 25, 12

The result was:Sunday, March 25, 12

The result was: • Over 60 programmers wanted to convert the data into Excel and Web (MySql) for a price of $100 -$3000 • Choose the ones that are in the top 300 (out of 200.000 coders) to avoid surprisesSunday, March 25, 12

Topcoder for $500Sunday, March 25, 12

Out of 9 projectsSunday, March 25, 12

Out of 9 projects • 8 were a success • 1 was a miserable failure • Total number of publications: 7, 1 pendingSunday, March 25, 12

Don’t forget Always 2x coders for the same job Perfect Pitch is Paramount Dataminded reporter should do the pitchSunday, March 25, 12

Project lessons Hire coders, don’t become one Do get the concepts behind data mining so you can instruct the coder Outsource smaller projects can be doneSunday, March 25, 12

Back to the VolkskrantSunday, March 25, 12

The first job was done in 5 daysSunday, March 25, 12

The first job was done in 5 days • PDF converted to normal textSunday, March 25, 12

Other steps:Sunday, March 25, 12

Other steps: • Text converted to Excel • Excel converted to MySql • Search engine for reporters enabled them to ‘Google’ data of the homes - later used as public searchSunday, March 25, 12

FINALLY SOME ANSWERSSunday, March 25, 12

FINALLY SOME ANSWERS • How many homes have financial troubles? • How many aggressive people caused trouble? • How many actions of non-medical employees were not backed up by a doctor? • Which elderly home is the best/worst?Sunday, March 25, 12

Unexpected questions: • Why are homes with a high mortality rate? • Why are homes with a sickness absence of 15-20% ?Sunday, March 25, 12

Results • Two lead news stories on the front page • A special section about the best/worst homes • ‘Top 300’ on the web for consumers: where do I put daddy away • A lot of goodwillSunday, March 25, 12

The lessons:Sunday, March 25, 12

The lessons: • 1.Datamining is the start of the project, not the end • 2 Reporters write the story • 3 Public comments on story - new storiesSunday, March 25, 12

School data The 2x 500 euro SolutionSunday, March 25, 12

What if You want to report on a regular basis on the same subject?Sunday, March 25, 12

Spidering or custom programming ? Roughly 7500 records in two spreadsheetsSunday, March 25, 12

How it was done http://functiemix.minocw.nl/resultaten.aspx?criteria=1&type=1&brin=00AH Brin codes identify schoolSunday, March 25, 12

How it was done: securing school codes Req. Get Page Page List of BRIN codes Build Screen Build URL Scraper ReportSunday, March 25, 12

How it was done: Building URL (from list) Req. Get Page Page List of BRIN codes Build Screen Build URL Scraper ReportSunday, March 25, 12

How it was done: Request web page Req. Get Page Page List of BRIN codes Build Screen Build URL Scraper ReportSunday, March 25, 12

How it was done: Get & parse web page Req. Get Page Page List of BRIN codes Build Screen Build URL Scraper ReportSunday, March 25, 12

How it was done: Build report Req. Get Page Page List of BRIN codes Build Screen Build URL Scraper ReportSunday, March 25, 12

How it was done: REPEAT until done Req. Get Page Page List of BRIN codes Build Screen Build URL Scraper ReportSunday, March 25, 12

Things to considerSunday, March 25, 12

Things to consider • Creates competitive advantageSunday, March 25, 12

Things to consider • Creates competitive advantage • - Saves timeSunday, March 25, 12

Things to consider • Creates competitive advantage • - Saves time • - More accurateSunday, March 25, 12

Things to consider • Creates competitive advantage • - Saves time • - More accurate • - Not tedious, no insult to intelligenceSunday, March 25, 12

Things to consider • Creates competitive advantage • - Saves time • - More accurate • - Not tedious, no insult to intelligence • Legal ramificationsSunday, March 25, 12

Things to consider • Creates competitive advantage • - Saves time • - More accurate • - Not tedious, no insult to intelligence • Legal ramifications – Always be respectful of your sourceSunday, March 25, 12

Things to consider • Creates competitive advantage • - Saves time • - More accurate • - Not tedious, no insult to intelligence • Legal ramifications – Always be respectful of your source – You can’t copyright factsSunday, March 25, 12

Things to consider • Creates competitive advantage • - Saves time • - More accurate • - Not tedious, no insult to intelligence • Legal ramifications – Always be respectful of your source – You can’t copyright facts – You can copyright how facts are presentedSunday, March 25, 12

Things to consider • Creates competitive advantage • - Saves time • - More accurate • - Not tedious, no insult to intelligence • Legal ramifications – Always be respectful of your source – You can’t copyright facts – You can copyright how facts are presented – Don’t make yourself conspicuousSunday, March 25, 12

Lessons Build once, use often Generic & important: zip code / geodata / feedbackSunday, March 25, 12

The 300 euro projectSunday, March 25, 12

digital digging: converting data Makelaars manipuleren woningaanbod ROTTERDAM, 3 JAN. NVM-makelaars en huizenverkopers manipuleren met te koop staande huizen, door ze een paar dagen van de markt te halen en daarna als nieuw aan te prijzen. De afgelopen drie maanden is dat over heel Nederland verspreid meer dan 2.700 keer gebeurd Source: Funda Technique: ‘datascraping’ 11/20/08 12:51 Henk van EssSunday, March 25, 12

• Script was written to download the data of more then 300.000 houses during three months • The results were put in an Excel file. • Reporters browsed manually for anomaliesSunday, March 25, 12

ResultsSunday, March 25, 12

Results • Two frontpage stories • 1 Appendix • Higher circulationSunday, March 25, 12

Nice, but.. Can’t you do it cheaper?Sunday, March 25, 12

The 0 euro-projectsSunday, March 25, 12

european databasesSunday, March 25, 12

Creative thinkingSunday, March 25, 12

Creative thinking • What databases are there online in Europe? • Hmm, there is a register of all people who attended meetings, a presence register.Sunday, March 25, 12

So what?? • What can you do with such a register? • Study which members are there and which not • Study which countries are more absent than others • How big is the effect if certain members of countries aren’t voting at all? • In reality some countries have more influence then you would expect from the actual number of seats 11/20/08 12:51 Henk van EssSunday, March 25, 12

CAR: do try this at home! The Netherlands has more power Source: EU: database, available in 20 languages 11/20/08 12:51 Henk van EssSunday, March 25, 12

Elaborate work..Sunday, March 25, 12

Healthcare regulatorsSunday, March 25, 12

11/20/08 12:51 Henk van Ess henk@vaness.nl 11/20/08 12:51 Henk van EssSunday, March 25, 12

Tuchtcollege voor de Gezondheidszorg 11/20/08 12:51 Henk van Ess henk@vaness.nl 11/20/08 12:51 Henk van EssSunday, March 25, 12

Verdicts against doctors • Uitspraken van het CTG • <http://www.tuchtcollege-gezondheidszorg.nl/recent_files/recent.htm>De meest recente uitspraken<http://www.tuchtcollege-gezondheidszorg.nl/recent_files/ recent.htm> • <http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ huisarts.htm>Uitspraken over huisartsen en basisartsen<http://www.tuchtcollege- gezondheidszorg.nl/huisarts_files/huisarts.htm> • <http://www.tuchtcollege-gezondheidszorg.nl/specialisten_files/ SPECIALISTEN.HTM>Uitspraken over medische specialisten<http:// www.tuchtcollege-gezondheidszorg.nl/specialisten_files/SPECIALISTEN.HTM> • Uitspraken over psychiaters<http://www.tuchtcollege-gezondheidszorg.nl/ psychiater.htm> • <http://www.tuchtcollege-gezondheidszorg.nl/tandarts_files/ Tandartsen.htm>Uitspraken over tandartsen<http://www.tuchtcollege- gezondheidszorg.nl/tandarts_files/Tandartsen.htm> • <http://www.tuchtcollege-gezondheidszorg.nl/verpleegkundigen.htm>Uitspraken over verpleegkundigen<http://www.tuchtcollege-gezondheidszorg.nl/ verpleegkundigen.htm> • <http://www.tuchtcollege-gezondheidszorg.nl/verloskundigen.htm>Uitspraken over verloskundigen<http://www.tuchtcollege-gezondheidszorg.nl/verloskundigen.htm> • <http://www.tuchtcollege-gezondheidszorg.nl/apotheker.htm>Uitspraken over henk@vaness.nl 11/20/08 12:51 Henk van Ess apothekers<http://www.tuchtcollege-gezondheidszorg.nl/apotheker.htm> 11/20/08 12:51 Henk van EssSunday, March 25, 12

Total chaos • 2005.009 en 046<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/2005.009%2520en %2520046.htm>2005.012<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.012.htm>2005.013<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.013.htm>2005.014<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.014.htm>2005.015<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.015.htm>2005.016<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.016.htm>2005.017<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.017.htm>2005.018<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.018.htm>2005.019<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.019.htm>2005.020<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.020.htm>2005.032<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.032.htm>2005.033<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.033.htm>2005.034<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.034.htm>2005.055<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.055.htm>2005.109<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.109.htm>2005.110<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.110.htm>2005.151<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.151.htm>2005.152<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.152.htm>2005.155<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.155.htm>2005.178<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.178.htm>2005.211<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.211.htm>2005.223<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.223.htm>2005.224<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2005.224.htm> 2004.005<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2004.005.htm>2004.012<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2004.012.htm>2004.014<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2004.014.htm>2004.021<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2004.021.htm>2004.022<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2004.022.htm>2004.023<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2004.023.htm>2004.024<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2004.024.htm>2004.025<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 2004.025.htm>2004.026<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 11/20/08 12:51 Henk van Ess henk@vaness.nl 2004.026.htm>2004.027<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/ 11/20/08 12:51 Henk van Ess 2004.027.htm>2004.028<http://www.tuchtcollege-gezondheidszorg.nl/huisarts_files/Sunday, March 25, 12

Step 1: download the site 11/20/08 12:51 Henk van Ess henk@vaness.nl 11/20/08 12:51 Henk van EssSunday, March 25, 12

Step 2 11/20/08 12:51 Henk van Ess henk@vaness.nl 11/20/08 12:51 Henk van EssSunday, March 25, 12

Step 2 • Download a desktop search tool 11/20/08 12:51 Henk van Ess henk@vaness.nl 11/20/08 12:51 Henk van EssSunday, March 25, 12

Step 3 11/20/08 12:51 Henk van Ess henk@vaness.nl 11/20/08 12:51 Henk van EssSunday, March 25, 12

Step 3 Start typing questions as you do in Google 11/20/08 12:51 Henk van Ess henk@vaness.nl 11/20/08 12:51 Henk van EssSunday, March 25, 12

Like this: 11/20/08 12:51 Henk van Ess henk@vaness.nl 11/20/08 12:51 Henk van EssSunday, March 25, 12

DIFFERENCES BETWEEN COURT RULINGSSunday, March 25, 12

Coverstory DIFFERENCES BETWEEN COURT RULINGSSunday, March 25, 12

This presentation: HENK VAN ESS www.searchbistro.com henk@vaness.nl @henkvaness http://www.slideshare.net/searchbistroSunday, March 25, 12

http://www.scoop.it	138
https://twitter.com	33
http://postpost.com	2
https://de.twitter.com	2
http://www.linkedin.com	2
http://us-w1.rockmelt.com	1
http://kred.com	1
https://si0.twimg.com	1

The Gory Details Of Datajournalism: what went wrong and who was responsible by Henk van Ess

by Henk van Ess on Mar 24, 2012

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 180

Statistics

The Gory Details Of Datajournalism: what went wrong and who was responsible by Henk van Ess Presentation Transcript