Your SlideShare is downloading. ×
Data tools ecosystem for non-programmers
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Data tools ecosystem for non-programmers

2,044
views

Published on

Presentation by @oscarmarinmiro at Itnig, November 8th 2013, about non-programming resources for getting, analyzing and visualizing data

Presentation by @oscarmarinmiro at Itnig, November 8th 2013, about non-programming resources for getting, analyzing and visualizing data

Published in: Technology, News & Politics

0 Comments
15 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,044
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
57
Comments
0
Likes
15
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Data Tools Ecosystem for non-programmers Óscar Marín Miró @outliers_es www.outliers.es 1
  • 2. Contents at a glance The Data Process ‣Why mess with Data? ‣A Data Pipeline Data Acquisition Tools ‣Data Sources ‣Google Forms & Google Docs ‣Scraping tools ‣Network acquisition tools ‣PDF tools ‣Open Refine Data Analysis ‣Excel capabilities ‣Beyond excel Data Visualization Tools ‣Timeline.js ‣Wordle ‣Plot.ly ‣Google Fusion Tables ‣Gephi ‣CartoDB ‣MapBox ‣Tilemill Developers needed ‣Size ‣Real-Time ‣Analysis beyond Basics ‣Interactivity* Resources 2
  • 3. The Data Process http://en.wikipedia.org/wiki/Data-driven_journalism 3
  • 4. Why mess with data? Cognitive Complexity Wisdom •Non transferable •Ethic knowledge application Knowledge •Personal Absortion •Action: Information application •‘Insights’ Information •Structured Data •Semantics •Narrative •Spatialized Data Data •Register observations 4
  • 5. A Data Pipeline Data Data Acquisition •API’s •Scraping •Files •DB’s •Providers Information Analysis •Text Mining •Machine Learning •Network Metrics •Stats •Filtering Presentation •Visualization •Story 5
  • 6. Data Acquisition Tools 6
  • 7. Data Formats • JSON (Javascript Object Notation) - Dev • CSV, XLS - Tabular • XML - Old, but still used • Web data 7
  • 8. Data Acq Techniques • Ask the user: Forms • Download from Data Repositories • Web data: Scrape • Ask the provider: API access 8
  • 9. Google Forms 9
  • 10. Google Forms 10
  • 11. Google Forms 11
  • 12. Google Forms http://www.encuesta15m.cc/subject.html 12
  • 13. Google Forms http://www.encuesta15m.cc/subject.html 13
  • 14. Google Forms http://data.elperiodico.com/ 14
  • 15. Google Spreadsheet http://mashe.hawksey.info/2012/10/feeding-google-spreadsheets-exercises-in-import/ 15
  • 16. Google Spreadsheet 16
  • 17. 17
  • 18. Scrape w/ Chrome Scraper https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd 18
  • 19. Scrape w/ Chrome Scraper 19
  • 20. Scrape w/ Chrome Scraper 20
  • 21. Scrape w/ Chrome Scraper http://dataist.wordpress.com/2012/10/12/get-started-with-screenscraping-using-google-chromes-scraper-extension/ 21
  • 22. Scrape w/ import.io http://import.io/ 22
  • 23. PDF Extraction: Tabula http://tabula.nerdpower.org/ 23
  • 24. PDF Extraction: Tabula http://tabula.nerdpower.org/ 24
  • 25. PDF Extraction: Tabula http://tabula.nerdpower.org/ 25
  • 26. Network Data: Flocker http://flocker.outliers.es/ 26
  • 27. Network Data: Flocker http://flocker.outliers.es/ 27
  • 28. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 28
  • 29. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 29
  • 30. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 30
  • 31. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 31
  • 32. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 32
  • 33. Open Refine: Cleaning Data 33
  • 34. Open Refine: Cleaning Data https://github.com/OpenRefine/OpenRefine/wiki/GREL-Functions https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions 34
  • 35. Data Analysis Tools 35
  • 36. Data Analysis Tools • Numeric analysis • Network analysis • Language analysis 36
  • 37. Data Analysis w/ MS Excel • • • • • • • Sort Basic Stats: Mean, Median, Sum... Filter Charts Pivot tables What-if Data Analysis Tool 37
  • 38. Data Analysis:MS Excel ‘Pivot Tables’ 38
  • 39. Data Analysis:MS Excel ‘What-If’ 39
  • 40. Data Analysis: MS Excel Correlations, Covariance, Statistic Summary, Regression, Histogram, Factor Analysis, Fourier Analysis.... http://ire.org/media/uploads/car2013_tipsheets/excel_stats_nicar2013.pdf 40
  • 41. Data Analysis: R http://www.r-project.org/ 41
  • 42. Data Analysis: NLP?? 42
  • 43. Data Analysis: NLP?? 43
  • 44. Data Visualization Tools http://well-formed.eigenfactor.org/ 44
  • 45. Data Visualization Tools • Timelines • Words • Chart tools • Networks • Maps 45
  • 46. Data Visualization Tools Timeline JS http://timeline.knightlab.com/ 46
  • 47. Data Visualization Tools Timeliner (OKFN) http://timeliner.okfnlabs.org 47
  • 48. Data Visualization Tools Wordle http://www.wordle.net/create 48
  • 49. Data Visualization Tools Wordle http://www.wordle.net/create 49
  • 50. Data Visualization Tools Plot.ly https://plot.ly/ 50
  • 51. Data Visualization Tools Plot.ly https://plot.ly/ 51
  • 52. Data Visualization Tools Plot.ly https://plot.ly/ 52
  • 53. Data Visualization Tools Plot.ly https://plot.ly/ 53
  • 54. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 54
  • 55. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 55
  • 56. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 56
  • 57. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 57
  • 58. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 58
  • 59. Data Visualization Tools http://www.tableausoftware.com/public/ 59
  • 60. Data Visualization Tools Gephi http://gephi.org/ 60
  • 61. Data Visualization Tools Gephi http://gephi.org/ 61
  • 62. Data Visualization Tools Gephi http://gephi.org/ 62
  • 63. Data Visualization Tools CartoDB http://developers.cartodb.com/tutorials/electoral_map.html http://cartodb.com/ 63
  • 64. Data Visualization Tools CartoDB “Shapefile” http://developers.cartodb.com/tutorials/electoral_map.html http://cartodb.com/ 64
  • 65. Data Visualization Tools CartoDB http://cartodb.com/ 65
  • 66. Data Visualization Tools MapBox http://www.mapbox.com// 66
  • 67. Data Visualization Tools MapBox http://www.mapbox.com// 67
  • 68. Data Visualization Tools MapBox http://www.mapbox.com// 68
  • 69. Data Visualization Tools MapBox http://www.mapbox.com// 69
  • 70. Data Visualization Tools MapBox http://www.mapbox.com// 70
  • 71. Data Visualization Tools MapBox http://www.mapbox.com// 71
  • 72. Data Visualization Tools MapBox http://www.mapbox.com// 72
  • 73. Developers needed 73
  • 74. When you’ll need a developer ‣Size ‣Real-Time ‣Analysis beyond Basics ‣Interactivity ‣API access 74
  • 75. References 75
  • 76. OKFN http://okfn.org/ http://schoolofdata.org/ http://ckan.org/ http://2014.okfestival.org/ 76
  • 77. Books 77
  • 78. Books 78
  • 79. Books 79
  • 80. Books 80
  • 81. Other http://itnig.net/en/events/revealing-thepower-of-data-for-non-developers http://www.cccb.org/ca/marcperiodisme_de_dades-44600 http://www.visualisingdata.com/index.php/ resources/ 81
  • 82. Data Sources http://ckan.org/ http://www.theguardian.com/data http://www.google.com/publicdata/ http://data.worldbank.org/ http://datahub.io/ http://www.quora.com/Data/Wherecan-I-find-large-datasets-open-to-thepublic 82
  • 83. Any questions? Thanks for your attention! 83