This document discusses open source tools for data wrangling and conversations with data. It addresses common barriers users face including data format, shape, cleanliness, enrichment, and joining. Tools mentioned that help overcome these barriers include Google Fusion Tables, Google Spreadsheets, Google Charts, Google Visualization API, R, Shiny apps, and RCharts. The document emphasizes helping users fashion their own tools to support data conversations.

1. Data wrangling with open
source tools
Tony Hirst
Dept of Communication & Systems
The Open University, UK

2. Premises

3. “I take data
from wherever I
can get it”
1

4. “Appropriate
everything”
2

5. Conversations
with data
3

6. Visual
Conversations
with data3

7. (Accession Plot)
@mediaczar

8. If a picture’s worth a
thousand words,
maybe it should take
as long to read?

14. Most learning
analytics won’t be
performed by
learning analytics
researchers

15. How can we help
people fashion
their own tools to
support data
conversations?

16. Recipes

17. site:open.ac.uk

18. Have a
conversation
with the data…

24. Ask the right
questions…

25. xkcd.com/1138

29. Sometimes a question
makes most sense in
the context of
questions previously
asked and answers
previously received

30. DATA
USERS
Educators
Learners
Planners
Marketers
Policymakers
Researchers
Press
NGOs
“
D
E
V
E
L
O
P
E
R
S
”

31. Have
dashboard,
so what?

32. A tools and
issues
based view

33. DATA
TOOLS
USERS
PROBLEMS

34. Example – Google Fusion Tables
Fusion Table
https://www.google.com/fusiontables/DataSource?docid=1VKG7iCbFlsEYJzTuQppf4xoIqq1ABxWTdW6O_7o#rows:id=1
http://is.gd/qhuaoA
Walkthrough
http://blog.ouseful.info/2012/11/16/a-quick-look-at-gcsealevel-certificate-awards-market-share-by-examination-board/
http://is.gd/f9YAbG

35. DATA
TOOLS
USERS
PROBLEMS

36. Access/obtain data
Make sense of data
Ask specific questions of data
Communicate in a data-centric way

37. Load data
Clean data
Merge/enrich data

38. DATA
Issues
TOOLS

39. DATA
Other
TOOLS
Issues
TOOLS

40. “Tool based
programming”

41. A barrier to access
(for the tool user) is
data format

44. JSON XMLCSVXLS
TSV
.db
HTML
PDF DOCTXT

45. GLUE LOGIC(Glue code)

46. =importHTML(URL, “table”, N)
HTML
QUERYABLE
DATA

47. Try it…
Example Page
http://en.wikipedia.org/wiki/List_of_colleges_and_universities_in_the_United_States_by_endowment
http://is.gd/7Vbg6n

48. Google Spreadsheets as a database
Explorer
https://views.scraperwiki.com/run/google_spreadsheet_query/
http://is.gd/jiMJoh
Walkthrough
http://schoolofdata.org/2013/05/24/asking-questions-of-data-garment-factories-data-expedition/
http://is.gd/qJHihu

49. =importCSV(URL, N)
HTML
INTERACTIVE
DASHBOARD
Google Charts

50. Google Chart
Visualization API
https://code.google.com/apis/ajax/playground/
http://is.gd/TTHIUh

51. Google
Visualisation
API

55. googleVis
(R)

64. https://developers.facebook.com/
docs/reference/api/examples/
http://is.gd/7cRnvS

65. A barrier to access
(for the tool user) is
data shape

74. A barrier to access
(for the tool user) is
data cleanliness

86. Questions of
identity

87. The Open University
Open University
OU
Open Uni
Open University, UK
NORMALISATION/RECONCILIATION

89. Reconciliation to
a canonical name
and/orto a
unique identifier

98. A stumbling block
(for the data user)
is data enrichment

106. A stumbling block
(for the data user)
is joining datasets

107. A stumbling block
(for the data user)
is joining partially
matched data

108. Rolling your own
interactive data
exploration tools

109. R Shiny
Apps

112. ui.R server.R

119. RCharts

123. Many chart tools
do the work for
you if the data is
in the right shape

124. DATA
TOOLS
USERS PROBLEMS

125. Justask…
ask.SchoolOfData.org

126. blog.ouseful.info
@psychemedia