Demolitions and Dali : Web Dev and Data in a Graph Database
•
0 likes•1,036 views
This document discusses using a graph database to model art and artist data from the Museum of Modern Art (MoMA). It describes creating nodes for artists and artworks and connecting them with relationship types. It also discusses using a graph database to model OpenStreetMap data by creating nodes for roads and connecting ways that share nodes. Finally, it discusses using a graph network of streets to improve predictions of which houses may be demolished by including information on related streets.
Recommended
More Related Content
Viewers also liked
Viewers also liked (6)
Similar to Demolitions and Dali : Web Dev and Data in a Graph Database
Similar to Demolitions and Dali : Web Dev and Data in a Graph Database (20)
More from Nicholas Doiron
More from Nicholas Doiron (20)
Recently uploaded
Recently uploaded (20)
Demolitions and Dali : Web Dev and Data in a Graph Database
- 2. • — TO —>
- 3. Also
- 4. OK, graph databases • Instead of tables and SQL • Nodes and relationships • Specialized queries • Not everything is a graph (and this is not sponsored)
- 5. Install / Update Neo4j • Neo4j • http://localhost:7474 Community Edition 3.0.3 • Python, PIP, and Py2Neo • py2neo.__version__ = ‘3b1’
- 6. Step 0 - installing • Install Neo4j - neo4j.com/install • brew on Mac • DigitalOcean has Linux instructions • change default password • Trouble installing locally? • heroku addons:add graphene
- 7. Who uses graphs? • Panama Papers • IMDB / Six Degrees of Kevin Bacon • Especially: • social networks, research data, maps • anywhere number of joins is large, indefinite, or unlimited
- 9. Cypher
- 10. MoMA.org • PostgreSQL sync to “The Museum System” CMS outside our control
- 11. Who uses MoMA.org? • Tourists • Researchers • Distant art fans • Members
- 12. The trouble with tables • Many joins to get people, titles, photos, additional relationship info • Speed of query • Difficult to write new queries
- 13. Art Graph DB • did Picasso collaborate with other artists in his lifetime? • are any artists credited as painter, director, sculptor, etc? (maybe an art EGOT)
- 14. Let’s build that graph • Artists and artworks • Basic bio data, MoMA ID -> Artist node • Future DB: all people connected • Title, date, MoMA ID -> Artwork node • ARTIST_OF relationship (include order)
- 15. Let’s build that graph • git clone https://github.com/mapmeld/graph ! • Building a scraper for MoMA
- 16. Demolitions and Dalí in a Graph Database Nick Doiron - @mapmeld
- 17. Cypher
- 18. Cypher
- 20. On to OSM
- 21. If you’re interested • Google: MapZen Extracts • download a city • for this script, download the OSM XML file • if you like PostGIS, there is a download (no import script)
- 22. Benefits of OSM • Open to use / full data • Open to edit / choose tags • HOT community • Civil e-mail lists (Crimea)
- 23. Benefits of OSM
- 24. Google on OSM • "Our maps represent what you or I need to do on a day-to-day basis in the developed part of the world” • — Google Maps Geospatial Technologist (quoted in FastCompany)
- 25. In Haiti and worldwide
- 26. In Haiti and worldwide
- 27. XML data
- 28. XML data • Nodes, ways, and relations • Ways made up of multiple nodes • Relations contain nodes and ways • Practically: • Multiple ways connect / combine • Tags are a community construct
- 29. Smart Renderer • When is a <way> a line (cul-de-sac) or a polygon (river, lake, parking lot)? • Has to support world’s fonts • Tag for real life, not for the renderer
- 30. Building graph data • Script adds all roads to Neo4j • Includes an array of node ids (can mix content types, similar to a document database) • If two ways share a node with the same ID, link them both ways <—>
- 31. Cypher + OSM * you can put an index on schema fields now
- 32. Problem
- 33. Google Prediction API • Prediction based on a CSV • Categorization or numerical • Google generates a model and estimates accuracy • Not allowed in Myanmar
- 34. Predicting Houses • Format 60,000+ rows of database export • Choose categories to predict 2-3 years • Competing models determine how important each column is • Can it parse dates? Find patterns • Edging up to ~74 percent accuracy
- 35. Network effect • Adding network of streets • Now tokens include not just my street and neighbors, but shared streets
- 36. Network effect • Most demolitions have one house on their street demolished (it’s them)
- 37. Network effect
- 38. Network effect • Google Prediction API reported 81% accuracy • But is it good? • Early optimization studies moved fire stations and left neighborhoods vulnerable • City can’t maintain it… hasn’t continued to open their data
- 39. Looking forward • Ideas for graph databases? Ways to release large graph data - as an API? As JSON files? As Neo4j dump? • Ideas for statisticians / future research?
- 40. Demolitions and Dalí in a Graph Database Nick Doiron - @mapmeld