Presentation by @oscarmarinmiro at Itnig, November 8th 2013, about non-programming resources for getting, analyzing and visualizing data

1. Data Tools Ecosystem
for non-programmers
Óscar Marín Miró
@outliers_es
www.outliers.es
1

2. Contents at a glance
The Data Process
‣Why mess with Data?
‣A Data Pipeline
Data Acquisition Tools
‣Data Sources
‣Google Forms & Google Docs
‣Scraping tools
‣Network acquisition tools
‣PDF tools
‣Open Refine
Data Analysis
‣Excel capabilities
‣Beyond excel
Data Visualization Tools
‣Timeline.js
‣Wordle
‣Plot.ly
‣Google Fusion Tables
‣Gephi
‣CartoDB
‣MapBox
‣Tilemill
Developers needed
‣Size
‣Real-Time
‣Analysis beyond Basics
‣Interactivity*
Resources
2

3. The Data Process
http://en.wikipedia.org/wiki/Data-driven_journalism
3

4. Why mess with data?
Cognitive Complexity
Wisdom
•Non transferable
•Ethic knowledge application
Knowledge
•Personal Absortion
•Action: Information application
•‘Insights’
Information
•Structured Data
•Semantics
•Narrative
•Spatialized Data
Data
•Register observations
4

5. A Data Pipeline
Data
Data
Acquisition
•API’s
•Scraping
•Files
•DB’s
•Providers
Information
Analysis
•Text Mining
•Machine Learning
•Network Metrics
•Stats
•Filtering
Presentation
•Visualization
•Story
5

6. Data Acquisition Tools
6

7. Data Formats
• JSON (Javascript Object Notation) - Dev
• CSV, XLS - Tabular
• XML - Old, but still used
• Web data
7

8. Data Acq Techniques
• Ask the user: Forms
• Download from Data Repositories
• Web data: Scrape
• Ask the provider: API access
8

9. Google Forms
9

10. Google Forms
10

11. Google Forms
11

12. Google Forms
http://www.encuesta15m.cc/subject.html
12

13. Google Forms
http://www.encuesta15m.cc/subject.html
13

14. Google Forms
http://data.elperiodico.com/
14

15. Google Spreadsheet
http://mashe.hawksey.info/2012/10/feeding-google-spreadsheets-exercises-in-import/
15

16. Google Spreadsheet
16

17. 17

18. Scrape w/ Chrome
Scraper
https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd
18

19. Scrape w/ Chrome
Scraper
19

20. Scrape w/ Chrome
Scraper
20

21. Scrape w/ Chrome
Scraper
http://dataist.wordpress.com/2012/10/12/get-started-with-screenscraping-using-google-chromes-scraper-extension/
21

22. Scrape w/ import.io
http://import.io/
22

23. PDF Extraction: Tabula
http://tabula.nerdpower.org/
23

24. PDF Extraction: Tabula
http://tabula.nerdpower.org/
24

25. PDF Extraction: Tabula
http://tabula.nerdpower.org/
25

26. Network Data: Flocker
http://flocker.outliers.es/
26

27. Network Data: Flocker
http://flocker.outliers.es/
27

28. Open Refine: Cleaning
Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
28

29. Open Refine: Cleaning
Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
29

30. Open Refine: Cleaning
Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
30

31. Open Refine: Cleaning
Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
31

32. Open Refine: Cleaning
Data
https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
32

33. Open Refine: Cleaning
Data
33

34. Open Refine: Cleaning
Data
https://github.com/OpenRefine/OpenRefine/wiki/GREL-Functions
https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions
34

35. Data Analysis Tools
35

36. Data Analysis Tools
• Numeric analysis
• Network analysis
• Language analysis
36

37. Data Analysis w/ MS
Excel
•
•
•
•
•
•
•
Sort
Basic Stats: Mean, Median, Sum...
Filter
Charts
Pivot tables
What-if
Data Analysis Tool
37

38. Data Analysis:MS Excel
‘Pivot Tables’
38

39. Data Analysis:MS Excel
‘What-If’
39

40. Data Analysis: MS Excel
Correlations, Covariance,
Statistic Summary, Regression,
Histogram, Factor Analysis,
Fourier Analysis....
http://ire.org/media/uploads/car2013_tipsheets/excel_stats_nicar2013.pdf
40

41. Data Analysis: R
http://www.r-project.org/
41

42. Data Analysis: NLP??
42

43. Data Analysis: NLP??
43

44. Data Visualization Tools
http://well-formed.eigenfactor.org/
44

45. Data Visualization Tools
• Timelines
• Words
• Chart tools
• Networks
• Maps
45

46. Data Visualization Tools
Timeline JS
http://timeline.knightlab.com/
46

47. Data Visualization Tools
Timeliner (OKFN)
http://timeliner.okfnlabs.org
47

48. Data Visualization Tools
Wordle
http://www.wordle.net/create
48

49. Data Visualization Tools
Wordle
http://www.wordle.net/create
49

50. Data Visualization Tools
Plot.ly
https://plot.ly/
50

51. Data Visualization Tools
Plot.ly
https://plot.ly/
51

52. Data Visualization Tools
Plot.ly
https://plot.ly/
52

53. Data Visualization Tools
Plot.ly
https://plot.ly/
53

54. Data Visualization Tools
Google Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
54

55. Data Visualization Tools
Google Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
55

56. Data Visualization Tools
Google Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
56

57. Data Visualization Tools
Google Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
57

58. Data Visualization Tools
Google Fusion Tables
https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1
https://www.google.com/fusiontables/
58

59. Data Visualization Tools
http://www.tableausoftware.com/public/
59

60. Data Visualization Tools
Gephi
http://gephi.org/
60

61. Data Visualization Tools
Gephi
http://gephi.org/
61

62. Data Visualization Tools
Gephi
http://gephi.org/
62

63. Data Visualization Tools
CartoDB
http://developers.cartodb.com/tutorials/electoral_map.html
http://cartodb.com/
63

64. Data Visualization Tools
CartoDB
“Shapefile”
http://developers.cartodb.com/tutorials/electoral_map.html
http://cartodb.com/
64

65. Data Visualization Tools
CartoDB
http://cartodb.com/
65

66. Data Visualization Tools
MapBox
http://www.mapbox.com//
66

67. Data Visualization Tools
MapBox
http://www.mapbox.com//
67

68. Data Visualization Tools
MapBox
http://www.mapbox.com//
68

69. Data Visualization Tools
MapBox
http://www.mapbox.com//
69

70. Data Visualization Tools
MapBox
http://www.mapbox.com//
70

71. Data Visualization Tools
MapBox
http://www.mapbox.com//
71

72. Data Visualization Tools
MapBox
http://www.mapbox.com//
72

73. Developers needed
73

74. When you’ll need a
developer
‣Size
‣Real-Time
‣Analysis beyond Basics
‣Interactivity
‣API access
74

75. References
75

76. OKFN
http://okfn.org/
http://schoolofdata.org/
http://ckan.org/
http://2014.okfestival.org/
76

77. Books
77

78. Books
78

79. Books
79

80. Books
80

81. Other
http://itnig.net/en/events/revealing-thepower-of-data-for-non-developers
http://www.cccb.org/ca/marcperiodisme_de_dades-44600
http://www.visualisingdata.com/index.php/
resources/
81

82. Data Sources
http://ckan.org/
http://www.theguardian.com/data
http://www.google.com/publicdata/
http://data.worldbank.org/
http://datahub.io/
http://www.quora.com/Data/Wherecan-I-find-large-datasets-open-to-thepublic
82

83. Any questions?
Thanks for your attention!
83