dataviz on d3.js + elasticsearch

5,497 views

Published on

dataviz on d3.js + elasticsearch

  1. 1. dataviz on bordeaux open data + elasticsearch + d3js + angular.js + google refinemardi 19 mars 13
  2. 2. goal • full stack dataviz • front js centric with angular.js and sass • d3.js -> best lib for dataviz (a bit complex) • elasticsearch scalable search engine access form js ajax client • grab open data and reformat with google refinemardi 19 mars 13
  3. 3. philosophy • smarter and smarter browsers will grab a huge part of the stack • scalable NoSQL solution talk REST: super easy access • you always can enrich, refine, store, model data from anysource: yes you can !mardi 19 mars 13
  4. 4. go open data ! • go to http://opendata.bordeaux.fr/ • we focus ou capc contemporary museum of bordeaux • http://opendata.bordeaux.fr/content/ collections-du-capc-musee-dart- contemporain • curl https:// bdxconfigogdi.blob.core.windows.net/ converteddata/capc.csv -o capc.csvmardi 19 mars 13
  5. 5. google refine • data many times isn’t perfect: humans input, unsuitable model... • with google refine you can tidy data, enrich with web services call and so on... • i think you should never think you don’t have the good data for the job • be smart and be creativ: you have everything you need, all the time !mardi 19 mars 13
  6. 6. import data in elasticsearch • bulk api import for es is handy • http://www.elasticsearch.org/guide/ reference/api/bulk.html • we use templating in google refine to export data to the correct format • { "index" : { "_index" : "musees", "_type" : "capc"} } [snipp]mardi 19 mars 13
  7. 7. elasticsearch • scalable search engine • adding more power == adding more nodes • sharding • replication • fault tolerantmardi 19 mars 13
  8. 8. elasticsearch • store unstructured document (json) in indexes (NoSQL way) • talk REST (api) • advanced query langage • multiple analyzers (tokens, languages, etc...) • blazing fast ! • no alternativ solutions (to my advice) (and kimshy advice too ;) )mardi 19 mars 13
  9. 9. yeoman • perfect tool for the new browser focus stack coming soon • yeoman init angular • yeoman init angular:route capc • yeoman install d3 • yeoman install jquery • yeoman server -> yeah !!!mardi 19 mars 13
  10. 10. angular.js • mvc framework for browser (js) • by google • rely better on html doc • cleaner controller handling than backbone (to my advice)mardi 19 mars 13
  11. 11. d3.js • best javascript lib for dataviz (to my advice) • slow learning curve • based on svg browser markup • data and dom oriented • generic (go low level with svg easily)mardi 19 mars 13
  12. 12. dataviz • make appears informations from pure data • you should explore, analyse and be creativ to grab the most value possible from the data: go as far as you can • tables (excell way) -> forgive • visualization will reveal your data: data is made to be know to take decision, manage, understandmardi 19 mars 13
  13. 13. open data • today, wide range of datas come to the public domain • again, without computing and vizualisation, data has a poor value • open data without data scientists and data visualization has no futurmardi 19 mars 13
  14. 14. futur • with low bandwith, storage and computing cost, you can grab open data from a lot of sources • you can put them in big data store and make analysis, relations between, with open source technologies • you can then share this to the word with data visualization on your website, blogs...mardi 19 mars 13 • amazing isn’t it ?? !!!!!
  15. 15. questions ? infos ? • https://github.com/mathieue • http://www.mathieu-elie.net/ • https://twitter.com/mathieuel • https://www.facebook.com/mathieuelietechmardi 19 mars 13

×