Data Tools Ecosystem
for non-programmers

Óscar Marín Miró
@outliers_es
www.outliers.es

1
Contents at a glance
The Data Process
‣Why mess with Data?
‣A Data Pipeline

Data Acquisition Tools
‣Data Sources
‣Google ...
The Data Process

http://en.wikipedia.org/wiki/Data-driven_journalism
3
Why mess with data?

Cognitive Complexity

Wisdom

•Non transferable
•Ethic knowledge application

Knowledge

•Personal Ab...
A Data Pipeline
Data

Data

Acquisition
•API’s
•Scraping
•Files
•DB’s
•Providers

Information

Analysis

•Text Mining
•Mac...
Data Acquisition Tools

6
Data Formats
• JSON (Javascript Object Notation) - Dev
• CSV, XLS - Tabular
• XML - Old, but still used
• Web data

7
Data Acq Techniques
• Ask the user: Forms
• Download from Data Repositories
• Web data: Scrape
• Ask the provider: API acc...
Google Forms

9
Google Forms

10
Google Forms

11
Google Forms

http://www.encuesta15m.cc/subject.html

12
Google Forms

http://www.encuesta15m.cc/subject.html

13
Google Forms

http://data.elperiodico.com/

14
Google Spreadsheet

http://mashe.hawksey.info/2012/10/feeding-google-spreadsheets-exercises-in-import/
15
Google Spreadsheet

16
17
Scrape w/ Chrome
Scraper

https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd
18
Scrape w/ Chrome
Scraper

19
Scrape w/ Chrome
Scraper

20
Scrape w/ Chrome
Scraper

http://dataist.wordpress.com/2012/10/12/get-started-with-screenscraping-using-google-chromes-scr...
Scrape w/ import.io

http://import.io/

22
PDF Extraction: Tabula

http://tabula.nerdpower.org/

23
PDF Extraction: Tabula

http://tabula.nerdpower.org/

24
PDF Extraction: Tabula

http://tabula.nerdpower.org/

25
Network Data: Flocker

http://flocker.outliers.es/
26
Network Data: Flocker

http://flocker.outliers.es/
27
Open Refine: Cleaning
Data

https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
28
Open Refine: Cleaning
Data

https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
29
Open Refine: Cleaning
Data

https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
30
Open Refine: Cleaning
Data

https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
31
Open Refine: Cleaning
Data

https://github.com/OpenRefine
https://code.google.com/p/google-refine/downloads/list
32
Open Refine: Cleaning
Data

33
Open Refine: Cleaning
Data

https://github.com/OpenRefine/OpenRefine/wiki/GREL-Functions
https://github.com/OpenRefine/OpenRefi...
Data Analysis Tools

35
Data Analysis Tools
• Numeric analysis
• Network analysis
• Language analysis

36
Data Analysis w/ MS
Excel
•
•
•
•
•
•
•

Sort
Basic Stats: Mean, Median, Sum...
Filter
Charts
Pivot tables
What-if
Data An...
Data Analysis:MS Excel
‘Pivot Tables’

38
Data Analysis:MS Excel
‘What-If’

39
Data Analysis: MS Excel

Correlations, Covariance,
Statistic Summary, Regression,
Histogram, Factor Analysis,
Fourier Anal...
Data Analysis: R

http://www.r-project.org/

41
Data Analysis: NLP??

42
Data Analysis: NLP??

43
Data Visualization Tools

http://well-formed.eigenfactor.org/
44
Data Visualization Tools
• Timelines
• Words
• Chart tools
• Networks
• Maps
45
Data Visualization Tools
Timeline JS

http://timeline.knightlab.com/
46
Data Visualization Tools
Timeliner (OKFN)

http://timeliner.okfnlabs.org

47
Data Visualization Tools
Wordle

http://www.wordle.net/create

48
Data Visualization Tools
Wordle

http://www.wordle.net/create

49
Data Visualization Tools
Plot.ly

https://plot.ly/
50
Data Visualization Tools
Plot.ly

https://plot.ly/
51
Data Visualization Tools
Plot.ly

https://plot.ly/
52
Data Visualization Tools
Plot.ly

https://plot.ly/
53
Data Visualization Tools
Google Fusion Tables

https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbS...
Data Visualization Tools
Google Fusion Tables

https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbS...
Data Visualization Tools
Google Fusion Tables

https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbS...
Data Visualization Tools
Google Fusion Tables

https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbS...
Data Visualization Tools
Google Fusion Tables

https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbS...
Data Visualization Tools

http://www.tableausoftware.com/public/

59
Data Visualization Tools
Gephi

http://gephi.org/
60
Data Visualization Tools
Gephi

http://gephi.org/
61
Data Visualization Tools
Gephi

http://gephi.org/
62
Data Visualization Tools
CartoDB

http://developers.cartodb.com/tutorials/electoral_map.html
http://cartodb.com/
63
Data Visualization Tools
CartoDB
“Shapefile”

http://developers.cartodb.com/tutorials/electoral_map.html
http://cartodb.com...
Data Visualization Tools
CartoDB

http://cartodb.com/
65
Data Visualization Tools
MapBox

http://www.mapbox.com//
66
Data Visualization Tools
MapBox

http://www.mapbox.com//
67
Data Visualization Tools
MapBox

http://www.mapbox.com//
68
Data Visualization Tools
MapBox

http://www.mapbox.com//
69
Data Visualization Tools
MapBox

http://www.mapbox.com//
70
Data Visualization Tools
MapBox

http://www.mapbox.com//
71
Data Visualization Tools
MapBox

http://www.mapbox.com//
72
Developers needed

73
When you’ll need a
developer
‣Size
‣Real-Time
‣Analysis beyond Basics
‣Interactivity
‣API access

74
References

75
OKFN
http://okfn.org/
http://schoolofdata.org/
http://ckan.org/
http://2014.okfestival.org/

76
Books

77
Books

78
Books

79
Books

80
Other
http://itnig.net/en/events/revealing-thepower-of-data-for-non-developers
http://www.cccb.org/ca/marcperiodisme_de_da...
Data Sources
http://ckan.org/
http://www.theguardian.com/data
http://www.google.com/publicdata/
http://data.worldbank.org/...
Any questions?
Thanks for your attention!

83
Upcoming SlideShare
Loading in …5
×

Data tools ecosystem for non-programmers

2,944 views

Published on

Presentation by @oscarmarinmiro at Itnig, November 8th 2013, about non-programming resources for getting, analyzing and visualizing data

Published in: Technology, News & Politics

Data tools ecosystem for non-programmers

  1. 1. Data Tools Ecosystem for non-programmers Óscar Marín Miró @outliers_es www.outliers.es 1
  2. 2. Contents at a glance The Data Process ‣Why mess with Data? ‣A Data Pipeline Data Acquisition Tools ‣Data Sources ‣Google Forms & Google Docs ‣Scraping tools ‣Network acquisition tools ‣PDF tools ‣Open Refine Data Analysis ‣Excel capabilities ‣Beyond excel Data Visualization Tools ‣Timeline.js ‣Wordle ‣Plot.ly ‣Google Fusion Tables ‣Gephi ‣CartoDB ‣MapBox ‣Tilemill Developers needed ‣Size ‣Real-Time ‣Analysis beyond Basics ‣Interactivity* Resources 2
  3. 3. The Data Process http://en.wikipedia.org/wiki/Data-driven_journalism 3
  4. 4. Why mess with data? Cognitive Complexity Wisdom •Non transferable •Ethic knowledge application Knowledge •Personal Absortion •Action: Information application •‘Insights’ Information •Structured Data •Semantics •Narrative •Spatialized Data Data •Register observations 4
  5. 5. A Data Pipeline Data Data Acquisition •API’s •Scraping •Files •DB’s •Providers Information Analysis •Text Mining •Machine Learning •Network Metrics •Stats •Filtering Presentation •Visualization •Story 5
  6. 6. Data Acquisition Tools 6
  7. 7. Data Formats • JSON (Javascript Object Notation) - Dev • CSV, XLS - Tabular • XML - Old, but still used • Web data 7
  8. 8. Data Acq Techniques • Ask the user: Forms • Download from Data Repositories • Web data: Scrape • Ask the provider: API access 8
  9. 9. Google Forms 9
  10. 10. Google Forms 10
  11. 11. Google Forms 11
  12. 12. Google Forms http://www.encuesta15m.cc/subject.html 12
  13. 13. Google Forms http://www.encuesta15m.cc/subject.html 13
  14. 14. Google Forms http://data.elperiodico.com/ 14
  15. 15. Google Spreadsheet http://mashe.hawksey.info/2012/10/feeding-google-spreadsheets-exercises-in-import/ 15
  16. 16. Google Spreadsheet 16
  17. 17. 17
  18. 18. Scrape w/ Chrome Scraper https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd 18
  19. 19. Scrape w/ Chrome Scraper 19
  20. 20. Scrape w/ Chrome Scraper 20
  21. 21. Scrape w/ Chrome Scraper http://dataist.wordpress.com/2012/10/12/get-started-with-screenscraping-using-google-chromes-scraper-extension/ 21
  22. 22. Scrape w/ import.io http://import.io/ 22
  23. 23. PDF Extraction: Tabula http://tabula.nerdpower.org/ 23
  24. 24. PDF Extraction: Tabula http://tabula.nerdpower.org/ 24
  25. 25. PDF Extraction: Tabula http://tabula.nerdpower.org/ 25
  26. 26. Network Data: Flocker http://flocker.outliers.es/ 26
  27. 27. Network Data: Flocker http://flocker.outliers.es/ 27
  28. 28. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 28
  29. 29. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 29
  30. 30. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 30
  31. 31. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 31
  32. 32. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 32
  33. 33. Open Refine: Cleaning Data 33
  34. 34. Open Refine: Cleaning Data https://github.com/OpenRefine/OpenRefine/wiki/GREL-Functions https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions 34
  35. 35. Data Analysis Tools 35
  36. 36. Data Analysis Tools • Numeric analysis • Network analysis • Language analysis 36
  37. 37. Data Analysis w/ MS Excel • • • • • • • Sort Basic Stats: Mean, Median, Sum... Filter Charts Pivot tables What-if Data Analysis Tool 37
  38. 38. Data Analysis:MS Excel ‘Pivot Tables’ 38
  39. 39. Data Analysis:MS Excel ‘What-If’ 39
  40. 40. Data Analysis: MS Excel Correlations, Covariance, Statistic Summary, Regression, Histogram, Factor Analysis, Fourier Analysis.... http://ire.org/media/uploads/car2013_tipsheets/excel_stats_nicar2013.pdf 40
  41. 41. Data Analysis: R http://www.r-project.org/ 41
  42. 42. Data Analysis: NLP?? 42
  43. 43. Data Analysis: NLP?? 43
  44. 44. Data Visualization Tools http://well-formed.eigenfactor.org/ 44
  45. 45. Data Visualization Tools • Timelines • Words • Chart tools • Networks • Maps 45
  46. 46. Data Visualization Tools Timeline JS http://timeline.knightlab.com/ 46
  47. 47. Data Visualization Tools Timeliner (OKFN) http://timeliner.okfnlabs.org 47
  48. 48. Data Visualization Tools Wordle http://www.wordle.net/create 48
  49. 49. Data Visualization Tools Wordle http://www.wordle.net/create 49
  50. 50. Data Visualization Tools Plot.ly https://plot.ly/ 50
  51. 51. Data Visualization Tools Plot.ly https://plot.ly/ 51
  52. 52. Data Visualization Tools Plot.ly https://plot.ly/ 52
  53. 53. Data Visualization Tools Plot.ly https://plot.ly/ 53
  54. 54. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 54
  55. 55. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 55
  56. 56. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 56
  57. 57. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 57
  58. 58. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 58
  59. 59. Data Visualization Tools http://www.tableausoftware.com/public/ 59
  60. 60. Data Visualization Tools Gephi http://gephi.org/ 60
  61. 61. Data Visualization Tools Gephi http://gephi.org/ 61
  62. 62. Data Visualization Tools Gephi http://gephi.org/ 62
  63. 63. Data Visualization Tools CartoDB http://developers.cartodb.com/tutorials/electoral_map.html http://cartodb.com/ 63
  64. 64. Data Visualization Tools CartoDB “Shapefile” http://developers.cartodb.com/tutorials/electoral_map.html http://cartodb.com/ 64
  65. 65. Data Visualization Tools CartoDB http://cartodb.com/ 65
  66. 66. Data Visualization Tools MapBox http://www.mapbox.com// 66
  67. 67. Data Visualization Tools MapBox http://www.mapbox.com// 67
  68. 68. Data Visualization Tools MapBox http://www.mapbox.com// 68
  69. 69. Data Visualization Tools MapBox http://www.mapbox.com// 69
  70. 70. Data Visualization Tools MapBox http://www.mapbox.com// 70
  71. 71. Data Visualization Tools MapBox http://www.mapbox.com// 71
  72. 72. Data Visualization Tools MapBox http://www.mapbox.com// 72
  73. 73. Developers needed 73
  74. 74. When you’ll need a developer ‣Size ‣Real-Time ‣Analysis beyond Basics ‣Interactivity ‣API access 74
  75. 75. References 75
  76. 76. OKFN http://okfn.org/ http://schoolofdata.org/ http://ckan.org/ http://2014.okfestival.org/ 76
  77. 77. Books 77
  78. 78. Books 78
  79. 79. Books 79
  80. 80. Books 80
  81. 81. Other http://itnig.net/en/events/revealing-thepower-of-data-for-non-developers http://www.cccb.org/ca/marcperiodisme_de_dades-44600 http://www.visualisingdata.com/index.php/ resources/ 81
  82. 82. Data Sources http://ckan.org/ http://www.theguardian.com/data http://www.google.com/publicdata/ http://data.worldbank.org/ http://datahub.io/ http://www.quora.com/Data/Wherecan-I-find-large-datasets-open-to-thepublic 82
  83. 83. Any questions? Thanks for your attention! 83

×