Data tools ecosystem for non-programmers

2,567 views
2,410 views

Published on

Presentation by @oscarmarinmiro at Itnig, November 8th 2013, about non-programming resources for getting, analyzing and visualizing data

Published in: Technology, News & Politics

Data tools ecosystem for non-programmers

  1. 1. Data Tools Ecosystem for non-programmers Óscar Marín Miró @outliers_es www.outliers.es 1
  2. 2. Contents at a glance The Data Process ‣Why mess with Data? ‣A Data Pipeline Data Acquisition Tools ‣Data Sources ‣Google Forms & Google Docs ‣Scraping tools ‣Network acquisition tools ‣PDF tools ‣Open Refine Data Analysis ‣Excel capabilities ‣Beyond excel Data Visualization Tools ‣Timeline.js ‣Wordle ‣Plot.ly ‣Google Fusion Tables ‣Gephi ‣CartoDB ‣MapBox ‣Tilemill Developers needed ‣Size ‣Real-Time ‣Analysis beyond Basics ‣Interactivity* Resources 2
  3. 3. The Data Process http://en.wikipedia.org/wiki/Data-driven_journalism 3
  4. 4. Why mess with data? Cognitive Complexity Wisdom •Non transferable •Ethic knowledge application Knowledge •Personal Absortion •Action: Information application •‘Insights’ Information •Structured Data •Semantics •Narrative •Spatialized Data Data •Register observations 4
  5. 5. A Data Pipeline Data Data Acquisition •API’s •Scraping •Files •DB’s •Providers Information Analysis •Text Mining •Machine Learning •Network Metrics •Stats •Filtering Presentation •Visualization •Story 5
  6. 6. Data Acquisition Tools 6
  7. 7. Data Formats • JSON (Javascript Object Notation) - Dev • CSV, XLS - Tabular • XML - Old, but still used • Web data 7
  8. 8. Data Acq Techniques • Ask the user: Forms • Download from Data Repositories • Web data: Scrape • Ask the provider: API access 8
  9. 9. Google Forms 9
  10. 10. Google Forms 10
  11. 11. Google Forms 11
  12. 12. Google Forms http://www.encuesta15m.cc/subject.html 12
  13. 13. Google Forms http://www.encuesta15m.cc/subject.html 13
  14. 14. Google Forms http://data.elperiodico.com/ 14
  15. 15. Google Spreadsheet http://mashe.hawksey.info/2012/10/feeding-google-spreadsheets-exercises-in-import/ 15
  16. 16. Google Spreadsheet 16
  17. 17. 17
  18. 18. Scrape w/ Chrome Scraper https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd 18
  19. 19. Scrape w/ Chrome Scraper 19
  20. 20. Scrape w/ Chrome Scraper 20
  21. 21. Scrape w/ Chrome Scraper http://dataist.wordpress.com/2012/10/12/get-started-with-screenscraping-using-google-chromes-scraper-extension/ 21
  22. 22. Scrape w/ import.io http://import.io/ 22
  23. 23. PDF Extraction: Tabula http://tabula.nerdpower.org/ 23
  24. 24. PDF Extraction: Tabula http://tabula.nerdpower.org/ 24
  25. 25. PDF Extraction: Tabula http://tabula.nerdpower.org/ 25
  26. 26. Network Data: Flocker http://flocker.outliers.es/ 26
  27. 27. Network Data: Flocker http://flocker.outliers.es/ 27
  28. 28. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 28
  29. 29. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 29
  30. 30. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 30
  31. 31. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 31
  32. 32. Open Refine: Cleaning Data https://github.com/OpenRefine https://code.google.com/p/google-refine/downloads/list 32
  33. 33. Open Refine: Cleaning Data 33
  34. 34. Open Refine: Cleaning Data https://github.com/OpenRefine/OpenRefine/wiki/GREL-Functions https://github.com/OpenRefine/OpenRefine/wiki/GREL-String-Functions 34
  35. 35. Data Analysis Tools 35
  36. 36. Data Analysis Tools • Numeric analysis • Network analysis • Language analysis 36
  37. 37. Data Analysis w/ MS Excel • • • • • • • Sort Basic Stats: Mean, Median, Sum... Filter Charts Pivot tables What-if Data Analysis Tool 37
  38. 38. Data Analysis:MS Excel ‘Pivot Tables’ 38
  39. 39. Data Analysis:MS Excel ‘What-If’ 39
  40. 40. Data Analysis: MS Excel Correlations, Covariance, Statistic Summary, Regression, Histogram, Factor Analysis, Fourier Analysis.... http://ire.org/media/uploads/car2013_tipsheets/excel_stats_nicar2013.pdf 40
  41. 41. Data Analysis: R http://www.r-project.org/ 41
  42. 42. Data Analysis: NLP?? 42
  43. 43. Data Analysis: NLP?? 43
  44. 44. Data Visualization Tools http://well-formed.eigenfactor.org/ 44
  45. 45. Data Visualization Tools • Timelines • Words • Chart tools • Networks • Maps 45
  46. 46. Data Visualization Tools Timeline JS http://timeline.knightlab.com/ 46
  47. 47. Data Visualization Tools Timeliner (OKFN) http://timeliner.okfnlabs.org 47
  48. 48. Data Visualization Tools Wordle http://www.wordle.net/create 48
  49. 49. Data Visualization Tools Wordle http://www.wordle.net/create 49
  50. 50. Data Visualization Tools Plot.ly https://plot.ly/ 50
  51. 51. Data Visualization Tools Plot.ly https://plot.ly/ 51
  52. 52. Data Visualization Tools Plot.ly https://plot.ly/ 52
  53. 53. Data Visualization Tools Plot.ly https://plot.ly/ 53
  54. 54. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 54
  55. 55. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 55
  56. 56. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 56
  57. 57. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 57
  58. 58. Data Visualization Tools Google Fusion Tables https://www.google.com/fusiontables/data?docid=1-941Px73b_XWWn3pmPHKp6WhbbSVNiEmKMadwe0#rows:id=1 https://www.google.com/fusiontables/ 58
  59. 59. Data Visualization Tools http://www.tableausoftware.com/public/ 59
  60. 60. Data Visualization Tools Gephi http://gephi.org/ 60
  61. 61. Data Visualization Tools Gephi http://gephi.org/ 61
  62. 62. Data Visualization Tools Gephi http://gephi.org/ 62
  63. 63. Data Visualization Tools CartoDB http://developers.cartodb.com/tutorials/electoral_map.html http://cartodb.com/ 63
  64. 64. Data Visualization Tools CartoDB “Shapefile” http://developers.cartodb.com/tutorials/electoral_map.html http://cartodb.com/ 64
  65. 65. Data Visualization Tools CartoDB http://cartodb.com/ 65
  66. 66. Data Visualization Tools MapBox http://www.mapbox.com// 66
  67. 67. Data Visualization Tools MapBox http://www.mapbox.com// 67
  68. 68. Data Visualization Tools MapBox http://www.mapbox.com// 68
  69. 69. Data Visualization Tools MapBox http://www.mapbox.com// 69
  70. 70. Data Visualization Tools MapBox http://www.mapbox.com// 70
  71. 71. Data Visualization Tools MapBox http://www.mapbox.com// 71
  72. 72. Data Visualization Tools MapBox http://www.mapbox.com// 72
  73. 73. Developers needed 73
  74. 74. When you’ll need a developer ‣Size ‣Real-Time ‣Analysis beyond Basics ‣Interactivity ‣API access 74
  75. 75. References 75
  76. 76. OKFN http://okfn.org/ http://schoolofdata.org/ http://ckan.org/ http://2014.okfestival.org/ 76
  77. 77. Books 77
  78. 78. Books 78
  79. 79. Books 79
  80. 80. Books 80
  81. 81. Other http://itnig.net/en/events/revealing-thepower-of-data-for-non-developers http://www.cccb.org/ca/marcperiodisme_de_dades-44600 http://www.visualisingdata.com/index.php/ resources/ 81
  82. 82. Data Sources http://ckan.org/ http://www.theguardian.com/data http://www.google.com/publicdata/ http://data.worldbank.org/ http://datahub.io/ http://www.quora.com/Data/Wherecan-I-find-large-datasets-open-to-thepublic 82
  83. 83. Any questions? Thanks for your attention! 83

×