NYC DataWeb                A platform for Integrating Public Data into NYC.gov                                     Joel Na...
About Me•   TCG Software    •   Software Services arm of “The Chatterjee Group”    •   Several Portfolio companies in Life...
Background
Main Goals•   stimulate development of apps    that improve access to info    and govt transparency,    and;•   encourage ...
CROWDSOURCING
CROWDSOURCING • Wisdom of the Crowd • Self-selecting, motivated developers • Bang for the Buck • Ignites Entrepreneurship
CROWDSOURCING•   Challenge:    Improve Recommendation Algorithm    by 10%• Dataset:                                       ...
CROWDSOURCING•   Challenge:    Improve Recommendation Algorithm    by 10%• Dataset:                                       ...
CROWDSOURCING
• Washington DC CTO - Vivek Kundra
•   First Federal CIO - Vivek Kundra
•   First Federal CIO - Vivek Kundra•   Open Government Initiative    •   Recovery.gov    •   Data.gov    •   USAspending....
•   First Federal CIO - Vivek Kundra•   Open Government Initiative    •   Recovery.gov    •   Data.gov    •   USAspending....
•   First Federal CIO - Vivek Kundra•   Open Government Initiative    •   Recovery.gov    •   Data.gov    •   USAspending....
•   First Federal CIO - Vivek Kundra•   Open Government Initiative    •   Recovery.gov    •   Data.gov    •   USAspending....
•   First Federal CIO - Vivek Kundra•   Open Government Initiative    •   Recovery.gov                          }    •   D...
•   First Federal CIO - Vivek Kundra           •   Open Government Initiative               •                  sh   ed    ...
Open Data in NYCCouncil Member Gale Brewer
$ 500 m i l l i o n ! ! !
Wh y $ 500m i l l i o n? ! ? !
Wh y $ 500m i l l i o n? ! ? !
“Integrated”Inter-Agency System
Data Integration Alphabet Soup       JMS         SOA              XS                                      LTM OM         E...
Data Integration Alphabet Soup        JMS       SOA                             XS                               LT   M   ...
and              Principles              b io ni                                                ch•   Cost Effective (NOT ...
The Next Web of Open Linked Data         February 2009
Useable Data Now•   “Beautiful” Website•   Useable by Developers/Publishers/Citizens•   based on Open Standards•   Low Ado...
What	  NYCBigApps	  Developers	                                      were	  Doing                                         ...
There must be a  Better Way
How it Started•   Oct 12, 2010 - NYCBigApps 2.0 announced•   Nov 9, 2010 - NYCBigApps 2.0 kickoff meeting•   late Nov 2010...
What	  We	  Did                            Domain                            Ontology                                     ...
“Beautiful” Website       Three dashboards were built• NYC Agile Analytics (Spry)• NYCreation (SMW+)  - visualized SPARQL ...
What’s Next?
Semantic Gap
DevelopersSemantic Gap
?!?Semantic Gap
3.0
3.0 Developers
3.0JumpStart Semantics
3.0
The Computer for the           rest of us.
Semantics for the       rest of us.
Semantics for the    REST of us.
Phase 2         Aug 2011 (Powered by NYCDataWeb)•   Hide Complexity               •   Open-source    (Simplicity = Adoptio...
Phase 2         Aug 2011 (Powered by NYCDataWeb)•   Hide Complexity               •   Open-source    (Simplicity = Adoptio...
Phase 3            Nov 2011 (NYCBigApps 2011)•   DataWeb Deployment Framework SMW bundle•   More Data Sources (Federator -...
The	  Broader	  Vision                                    Domain                                    Ontology              ...
Phase 4                Post NYC BigApps 2011•   Multiple solutions powered by NYCDataWeb•   <Your city/community/company h...
SemanticWeb
Hans Rosling shows the best stats       youve ever seen           February 2006
PUBLIC
PUBLIC
We need your help & feedback  A Platform for Integrating Public Data into NYC.gov                 Find out more at  http:/...
CREDITS•   Lego Faceparty picture by RichardAM (http://www.richard-am.net/)•   Lego Inauguration Pictures from various Fli...
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC
Upcoming SlideShare
Loading in …5
×

NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC

1,007 views
933 views

Published on

An Open Public Data Exchange for New York City, submitted to the NYCBigApps 2010 Challenge.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,007
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

NYC Data Web (static version) - A Semantic, Open Public Data Exchange for NYC

  1. 1. NYC DataWeb A platform for Integrating Public Data into NYC.gov Joel NatividadClick here for narrated version TCG Thursday, June 9, 2011 SemTech 2011
  2. 2. About Me• TCG Software • Software Services arm of “The Chatterjee Group” • Several Portfolio companies in Lifesciences, Telecom, Aviation, Energy, Real Estate, & Info Technology• Headquartered in NYC• Delivery Centers in Bangalore, Kolkata & Mumbai• Look after Knowledge Engineering Practice of TCG
  3. 3. Background
  4. 4. Main Goals• stimulate development of apps that improve access to info and govt transparency, and;• encourage innovation & the creation of new IP with commercial potential
  5. 5. CROWDSOURCING
  6. 6. CROWDSOURCING • Wisdom of the Crowd • Self-selecting, motivated developers • Bang for the Buck • Ignites Entrepreneurship
  7. 7. CROWDSOURCING• Challenge: Improve Recommendation Algorithm by 10%• Dataset: STATISTICS • 100 million ratings (training set) • just 6 days into contest, • Half a million Users Cinematch bested by 1% • 18 thousand movies • 20,000 Teams, 150 countries • Entrants:• Prize: • Bell Labs One million US Dollars • Opera Solutions • Well-renowned universities
  8. 8. CROWDSOURCING• Challenge: Improve Recommendation Algorithm by 10%• Dataset: STATISTICS • 100 million ratings (training set) • just 6 days into contest, • Half a million Users Cinematch bested by 1% • 18 thousand movies • 20,000 Teams, 150 countries • Entrants:• Prize: • Bell Labs One million US Dollars • Opera Solutions • Well-renowned universities
  9. 9. CROWDSOURCING
  10. 10. • Washington DC CTO - Vivek Kundra
  11. 11. • First Federal CIO - Vivek Kundra
  12. 12. • First Federal CIO - Vivek Kundra• Open Government Initiative • Recovery.gov • Data.gov • USAspending.gov • IT Dashboard • Performance.gov • Fedspace • Citizen Services Dashboard
  13. 13. • First Federal CIO - Vivek Kundra• Open Government Initiative • Recovery.gov • Data.gov • USAspending.gov • IT Dashboard • Performance.gov • Fedspace • Citizen Services Dashboard
  14. 14. • First Federal CIO - Vivek Kundra• Open Government Initiative • Recovery.gov • Data.gov • USAspending.gov • IT Dashboard • Performance.gov • Fedspace • Citizen Services Dashboard
  15. 15. • First Federal CIO - Vivek Kundra• Open Government Initiative • Recovery.gov • Data.gov • USAspending.gov • IT Dashboard • Performance.gov • Fedspace • Citizen Services Dashboard
  16. 16. • First Federal CIO - Vivek Kundra• Open Government Initiative • Recovery.gov } • Data.gov Li fe S u pp o r t • USAspending.gov • IT Dashboard • Performance.gov • Fedspace • Citizen Services Dashboard
  17. 17. • First Federal CIO - Vivek Kundra • Open Government Initiative • sh ed Recovery.gov } e t• sla o u Li fe t S pp B u dg i lli on Data.gov • m ort $ 34 o n USAspending.govfr o m •m i l l i $8 IT Dashboard • Performance.gov • Fedspace • Citizen Services Dashboard
  18. 18. Open Data in NYCCouncil Member Gale Brewer
  19. 19. $ 500 m i l l i o n ! ! !
  20. 20. Wh y $ 500m i l l i o n? ! ? !
  21. 21. Wh y $ 500m i l l i o n? ! ? !
  22. 22. “Integrated”Inter-Agency System
  23. 23. Data Integration Alphabet Soup JMS SOA XS LTM OM EAI B OR EJB SOAP D A XML M RPC BPM PO JO BPEL
  24. 24. Data Integration Alphabet Soup JMS SOA XS LT M EAIMO ORBEJ XM L B SO AP BPM MDA BPEL RPC PO JO
  25. 25. and Principles b io ni ch• Cost Effective (NOT $500 million dollars)• Easy to Use (Developers/Publishers/Citizens)• based on Open Standards• Low Adoption Curve• Help Accelerate Open Data Innovation• Useable Data Now!
  26. 26. The Next Web of Open Linked Data February 2009
  27. 27. Useable Data Now• “Beautiful” Website• Useable by Developers/Publishers/Citizens• based on Open Standards• Low Adoption Curve• Help Accelerate Open Data Innovation• Useable Data Now!
  28. 28. What  NYCBigApps  Developers   were  Doing Download & Decipher ETL Text ProcessesSiloed Data • Spend inordinate amount of time interpreting data • Massaged Data was then staged locally • Developers kept reinventing the wheel • Limited Data mashups • Applications disconnected from NYCDatamine 46
  29. 29. There must be a Better Way
  30. 30. How it Started• Oct 12, 2010 - NYCBigApps 2.0 announced• Nov 9, 2010 - NYCBigApps 2.0 kickoff meeting• late Nov 2010 - spoke with Revelytix/Spry about collaborating• early Dec 2010 - started work on NYCDataWeb• Jan 26, 2011 ~4:30p - submitted entry
  31. 31. What  We  Did Domain Ontology Query & Results Cache Optimizer Definitions Re-Writer PlannerSiloed Data Indexes Rules Re-Writer Optimizer Mapping Ontology Indexes Planner Rules Metadata Ontology 51
  32. 32. “Beautiful” Website Three dashboards were built• NYC Agile Analytics (Spry)• NYCreation (SMW+) - visualized SPARQL query results• NYCmantics (SMW+) - NYC datamine explorer
  33. 33. What’s Next?
  34. 34. Semantic Gap
  35. 35. DevelopersSemantic Gap
  36. 36. ?!?Semantic Gap
  37. 37. 3.0
  38. 38. 3.0 Developers
  39. 39. 3.0JumpStart Semantics
  40. 40. 3.0
  41. 41. The Computer for the  rest of us.
  42. 42. Semantics for the  rest of us.
  43. 43. Semantics for the  REST of us.
  44. 44. Phase 2 Aug 2011 (Powered by NYCDataWeb)• Hide Complexity • Open-source (Simplicity = Adoption) collaboration with vendors & other• Incorporate the whole institutions NYC datamine • Incorporate the best of• Make it easier for Socrata and data.gov Publishers • Improved Visualizations• Make it easier for Developers• Make it easier for Citizens
  45. 45. Phase 2 Aug 2011 (Powered by NYCDataWeb)• Hide Complexity • Open-source (Simplicity = Adoption) collaboration with vendors & other• Incorporate the whole institutions NYC datamine • Incorporate the best of• Make it easier for Socrata and data.gov Publishers • Improved Visualizations• Make it easier for Developers • Position NYCDataWeb as the accelerated data• Make it easier for Citizens mashup platform
  46. 46. Phase 3 Nov 2011 (NYCBigApps 2011)• DataWeb Deployment Framework SMW bundle• More Data Sources (Federator - Spinner)• Linked Open Data• Make it easier STILL for Publishers, Developers and Citizens• Enable Widespread adoption of NYCDataWeb (NYCDataWeb bootcamp)
  47. 47. The  Broader  Vision Domain Ontology Query & Results RDF Ontology NYC Information Web Partners RDF RDF RDF RDF RDF Web Pages OtherAgency  Data   Sensorss Triplestores 85
  48. 48. Phase 4 Post NYC BigApps 2011• Multiple solutions powered by NYCDataWeb• <Your city/community/company here> DataWeb• Help foster a viable ecosystem of Linked Data• ... keep standing on the shoulders of giants
  49. 49. SemanticWeb
  50. 50. Hans Rosling shows the best stats youve ever seen February 2006
  51. 51. PUBLIC
  52. 52. PUBLIC
  53. 53. We need your help & feedback A Platform for Integrating Public Data into NYC.gov Find out more at http://knoodl.com/ui/groups/NYC_Homepage
  54. 54. CREDITS• Lego Faceparty picture by RichardAM (http://www.richard-am.net/)• Lego Inauguration Pictures from various Flickr Users (sluggobear, Atwater, Dan Hontz)• Lego Luke looses his Hand by Flickr user wwwayazdotcom• Tim Berners-Lee highlight from TED (http://www.ted.com/talks/ tim_berners_lee_on_the_next_web.html)• Hans Rosling highlight from TED (http://www.ted.com/talks/ hans_rosling_shows_the_best_stats_you_ve_ever_seen.html)• FlowerPowerpont2.pptx provided by Anna Rosling Rönnlund of gapminder• “Star Wars Gangsta Rap” highlight, SizzlechestXXX (http://www.youtube.com/watch?v=Ij4w7ChpuaM)• Various screenshots provided by Revelytix, Spry Inc. and TCG Software Services

×