This document provides tips for finding and extracting data from online sources for journalistic purposes. It discusses using search tools and operators to find relevant data, identifies common sources of public data and how to setup alerts, describes how to make Freedom of Information requests, and explains how to scrape structured data from websites through identifying patterns in URLs, tables, and other elements. The overall aim is to help journalists and investigators efficiently obtain useful data for stories and research.

1. @PaulBradshaw, Online Journalism Blog
Birmingham City University and City University London
BBC, January 2015
Data Mining
Search, scraping, FOI and feeds
Image by Evan Long

2. 1. Search tips and tools
2. Sources and feeds
3. Data requests
4. Scraping

3. 1. Search tips and tools
2. Sources and feeds
3. Data requests
4. Scraping

4. Don’t ask for what you want:
describe what you expect to
find
Search operators

5. What text will it contain?
Where will that text be?
What text will it not contain?
Imagine the data: text

6. Specific references, not
general:
Specify a constituency…
…a school
…an institution code
…an invoice number
…a piece of jargon

12. “”

13. -

14. *

15. ..

16. “disclosure log”
“between * and 2014”
“hate crime” -religion
-"publication scheme"
Number ranges: 2000..2014

17. ‘life expectancy Birmingham’

18. "life expectancy"
"perry barr"

19. inurl:

20. inurl:foi
inurl:ccg
inurl:intranet
inurl:search.asp
inurl:search.php

21. intitle:
allintitle:

22. intitle:foi
allintitle:disclosure log
intitle:“bank fines”

23. intext:
allintext:

24. intext:“miserable failure”
allintext:miserable failure

25. "life expectancy"
"perry barr"

26. "life expectancy"
"perry barr"
filetype:xls

28. "life expectancy"
"perry barr"
filetype:xls
site:ons.gov.uk

29. "life expectancy"
"perry barr"
filetype:xls
site:ons.gov.uk
2009..2014

30. "life expectancy"
"perry barr"
filetype:xls
site:ons.gov.uk
2009..2014
-winter

32. Where is it likely to be
What format?
When was it not published?
Imagine the data: meta data

33. site:

34. site:gov.uk
site:nhs.uk
site:police.uk
site:ac.uk
site:org.uk
site:org
site:birmingham.gov.uk
site:met.police.uk/foi/
disclosure

35. filetype:

36. filetype:xls
filetype:xlsx
filetype:pdf
filetype:csv
filetype:ppt
filetype:doc
filetype:docx
filetype:xml

37. search tools

39. “disclosure log” site:gov.uk
allintitle:hate crime report
filetype:pdf site:police.uk
art inurl:search.asp -library
Combine operators:

40. research.google.com

41. zanran.com

42. Do it now:
Search for a disclosure log
for a CCG
Search for spreadsheets
mentioning Andrew Mitchell
MP

43. 1. Search tips and tools
2. Sources and feeds
3. Data requests
4. Scraping

44. Audits and transparency data
Parliamentary questions
Reports, research, sources
FOI requests, disclosure logs
Press offices
Public data and databases -
scraping

45. Open data initiatives &
activism (TWFY)
Hackdays e.g. Rewired State
Public data and databases -
scraping
Crowdsourcing or surveys
Social networks

46. NOMIS, ONS, Data.gov.uk
HES, NHSIC indicator portal
Data.Police.uk
HEFCE, HESA, Ofsted, UCAS
fullfact.org/finder
Key sources

53. Do it now:
Set up Change Detection for
the CCG disclosure log
Set up email alerts for
publications on Data.gov.uk

54. 1. Search tips and tools
2. Sources and feeds
3. Data requests
4. Scraping

60. http://www.panopticonblog.com/2014/08/01/section-11-foia-and-the-form-of-a-request/
http://www.bailii.org/ew/cases/EWCA/Civ/2014/1086.html
As per the judgement in Innes
v Information Commissioner
[2014] EWCA Civ 1086 I would
like to request the data in
spreadsheet format…

61. Do it now:
Draft an FOI request for a
local body’s data dictionary
Use WhatDoTheyKnow (so
others googling codes can
find you)

62. 1. Search tips and tools
2. Sources and feeds
3. Data requests
4. Scraping

63. Automating the repetitive
gathering of data, e.g.
Multiple tables in one page
Webpage tables
Multiple spreadsheets
Multiple PDFs
What is scraping?

64. https://www.youtube.com/watch?v=Efr-VEkwWoM

66. http://blogs.ft.com/ftdata/2014/06/11/interactive-explore-the-statistical-identity-of-every-team-at-the-world-cup/?

67. http://www.mirror.co.uk/news/uk-news/singer-best-vocal-range-uk-4323076

68. *
http://www.w4mpjobs.org/SearchJobs.aspx?search=alljobs
Tip: empty search

69. Basic tables: WYSIWYG tools
Google Sheets functions
Programming: Scraperwiki
How to scrape?

70. Paul Bradshaw
Leanpub.com/scrapingforjournalists*
<plug>

71. *
Function (Arguments)
(aka parameters)

72. *
Query (XPath)

73. *
Tip: search for structure
around data
http://www.w4mpjobs.org/SearchJobs.aspx?search=alljobs

74. *
http://
www.w4mpjobs.org/
SearchJobs.aspx?
http://www.w4mpjobs.org/SearchJobs.aspx?search=alljobs

75. *

76. *
"//div[@class=
'leftcolumn']"

77. *
//div[starts-with(@
class, ‘jobWrap’)]

78. *
A crib sheet:

79. Paul Bradshaw
Leanpub.com/scrapingforjournalists*
Scraping
tools

80. *
Chrome extension:

81. *

82. *

83. OutWit Hub

86. Do it now:
Identify a website which
has multiple pages or
documents containing data
you could combine
Where’s the structure?
Table? URL? Links?

87. 1. Search: describe the data
2. Feeds: get regular
updates
3. FOI: request detail, in CSV
format
4. Scraping: look for
structure and repetition

88. Thank you.
Image by Evan Long
@PaulBradshaw, Online Journalism Blog, HelpMeInvestigate
Birmingham City University and City University London
BBC Future Day, September 2014