Seattle Scalability Meetup

860
-1

Published on

Presentation from the Seattle Scalability meet up on Cloudant Geospatial

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
860
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Seattle Scalability Meetup

  1. 1. CloudantGeo:Scaling geospatial for the massesMike MillerCoFounder, Chief Scientist
  2. 2. {Problems: [‘Mobile’, ‘Data’]} Mobile Big Data These break our existing models for computing Mike Miller, 2013/02/27 2
  3. 3. {Introductions: ‘Cloudant’} Database that ships with a mobile strategy Mike Miller, 2013/02/27 3
  4. 4. {Introductions: ‘Cloudant’}• Distributed Database as a Service (DBaaS)• For developers of high-velocity web and mobile apps• Venture Funded, YCombinator• Ends drudgery of SQL and scale-it- yourself NoSQL• 13,000+ users• Founded by big data scientists• Speaks CouchDB API Mike Miller, 2013/02/27 4
  5. 5. {Introductions: ‘Cloudant’} Mike Miller, 2013/02/27 5
  6. 6. {Schemas: ‘Optional’} Schemas & protocols can be restrictive and inhibit data integration JSON over HTTP Mike Miller, 2013/02/27 6
  7. 7. {Install: ‘Cloudant’} You do this: We give you: That’s It Mike Miller, 2013/02/27 7
  8. 8. {API: ‘REST’} Write a doc...from the browser No client install necessary JSON, REST, HTTP Mike Miller, 2013/02/27 8
  9. 9. {API: ‘Search’} Create Secondary IndexesQuery Those indexes Rinse and repeat.... billions of times per day Mike Miller, 2013/02/27 9
  10. 10. {Challenge: ‘Dogma’} “Scaling would be a good problem to have. We’ll deal with that later” False: Scaling Mandatory Mike Miller, 2013/02/27 10
  11. 11. {Have: ‘Success’} Mike Miller, 2013/02/27 11
  12. 12. {Minimize: ‘Latency’} Public or Private Cloud Mike Miller, 2013/02/27 12
  13. 13. {Write: ‘Local’, Sync: ‘Later’} Embedded, Edge, Sattelites Desktop, Browser Mike Miller, 2013/02/27 Cloud 13
  14. 14. Mike Miller, 2013/02/27 14
  15. 15. {Example: ‘HotheadGames’} Mike Miller, 2013/02/27 15
  16. 16. {Example: ‘HotheadGames’} ~100x Scaling in last 9 months This Database is bigger than many (most?) Hadoop clusters Mike Miller, 2013/02/27 16
  17. 17. How do you scale a database from MB to PB? How do you add new features that scale? Mike Miller, 2013/02/27 17
  18. 18. What if my application cares about geospatial location?What if I need to combine FTI/search, SELECT, and geo? Cloudant: MapReduce + Lucene + Geospatial Mike Miller, 2013/02/27 18
  19. 19. {Sharding: ‘Automatic’} • Clustering in a ring (a la Dynamo) PUT http://rnewson.cloudant.com/dbname/blah?w=2 • Any node can handle a request • O(1) lookup Load Balancer • Quorum system (N, R, W) • Views distributed like documents Node 1 24 No • Distributed Erlang de A B C D de No B 2 Z A C • Masterless Y D X hash(blah) = E E C N od e D 3 E N=3 F W=2 R=2 D No de E 4 F G Mike Miller, 2013/02/27 19
  20. 20. {Manual_Sharding: ‘Sucks’}http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/spanner-osdi2012.pdf Mike Miller, 2013/02/27 20
  21. 21. {Format: ‘geojson’} http://geojson.org/geojson-spec.html Mike Miller, 2013/02/27 21
  22. 22. {Format: ‘topojson’} https://github.com/mbostock/topojson/wiki Mike Miller, 2013/02/27 22
  23. 23. {Index: ‘R-Tree’} Wikpedia Mike Miller, 2013/02/27 23
  24. 24. {Index: ‘R*-Tree’} Wikpedia Mike Miller, 2013/02/27 24
  25. 25. {Index: ‘TPR*-Tree’}• Predictive spatio-temporal query retrieves the set of moving objects that will intersect a query window during a future time interval http://www.cs.ust.hk/~dimitris/PAPERS/VLDB03-TPR.pdf Mike Miller, 2013/02/27 25
  26. 26. {API: ‘geo’} Create Secondary IndexesQuery Those indexes Mike Miller, 2013/02/27 26
  27. 27. {Operators: ‘more’ }• Disjoint • Crosses• Equals • Within• DWithin • Contains• Beyond • Overlaps• Intersect • BBX• Touches Mike Miller, 2013/02/27 27
  28. 28. {Want: ‘Beta Testers’ } For a demo or a beta invite, pm me @mlmilleratmit Mike Miller, 2013/02/27 28
  29. 29. {Status: [‘Now’, ‘Future’] }• Big and Getting Bigger • Product Roadmap •Databases -- 100M+ •Security enhancements •Daily Transactions -- 10s of •Advanced geospatial Billions •Graph engine •Indexed Data -- 100s TBs •Map Reduces/day -- Billions Mike Miller, 2013/02/27 29

×