Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scalability 09262012

506 views

Published on

Here's a pdf of the slides from my talk at http://www.meetup.com/Seattle-Hadoop-HBase-NoSQL-Meetup/events/62000472/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Scalability 09262012

  1. 1. Mike Miller, CoFounder, Chief Scientist @mlmilleratmit mike@cloudant.com 1
  2. 2. My BackgroundCloudant CoFounder, Chief ScientistAssistant Professor, Particle Physics(U. Washington, Affiliate)Background: machine learning,analysis, big data, globally distributedsystems Cloudant, 9-26-2012 2
  3. 3. The face of big data http://abstract.cs.washington.edu/~shwetak/ Cloudant, 9-26-2012 3
  4. 4. The face of big data Cloudant, 9-26-2012 4
  5. 5. The face of big data “The future is stranger and sooner than you think” Reid Hoffman, LinkedIn/Greylock Cloudant, 9-26-2012 5
  6. 6. Perfect Storm Parallel Processing Big Data HTML5/JS Mobile 9M Trained Developers Cloudant, 9-26-2012 6
  7. 7. Focus on your Application not data operations Cloudant, 9-26-2012 7
  8. 8. If your data is stuck in the warehouse... ... you’re losing Cloudant, 9-26-2012 8
  9. 9. Data Layer for the WebFounded (2009) by leading MIT datascientistsFunded by Y Combinator & AvalonGlobal network of 20+ data centers-- Application Data Network (ADN)Built on leading NoSQL standard:most durable data store on planet10,000 users and growing. Cloudant: Akamai of dynamic content Cloudant, 9-26-2012 9
  10. 10. Cloudant Product Line• Application State Hyper-Scalable Document Store (JSON+HTTP) MVCC Secondary indexes for flexible query• Application Data Security Accounts/API keys, data sharing, permission roles• Application Analytics Fully Integrated (Incremental) MapReduce engine• Application Search Fully Integrated (Incremental) Lucene + Geospatial API Compatible• Application Object Storage images, audio, video...• Application State Distribution cloud <==> tablet <==> PC <==> mobile Cloudant, 9-26-2012 10
  11. 11. Cloudant Install You do this: We give you: That’s It Cloudant, 9-26-2012 11
  12. 12. API Examples Write a doc...from the browser No client install necessary Cloudant, 9-26-2012 12
  13. 13. API Examples Create Secondary IndexesQuery Those indexes Cloudant, 9-26-2012 13
  14. 14. http://examples.cloudant.com/lobby-search/_design/lookup/index.html Cloudant, 9-26-2012 14
  15. 15. Global Data Network Cloudant scales within & between data centers Availability, low-latency Cloudant, 9-26-2012 15
  16. 16. Anatomy of the Data Layer PUT {document} Secondary Data Centers (for DR & distributed access) US-EAST “Node” AP-JP Filtered Replication & Sync EU-NL Single-tenant cluster Multi-tenant cluster Disconnected Horizontally Scalable DB Devices • Fault tolerant • Always consistent Edge Database • Schemaless (NoSQL) Cluster • Automatic sharding • Distributed, parallel analytics • Incremental, chainable MapReduce • Full-text search Single-Tenant or Multi-Tenant 16
  17. 17. https://cloudant.com/blog/cloudant-labs-on-google-spanner/ Cloudant, 9-26-2012 17
  18. 18. Why It Matters Cloudant, 9-26-2012 18
  19. 19. >1. Visualization Wins http://sosolimited.com/blog/2012/07/from-tweets-to-lightshow/ Cloudant, 9-26-2012 19
  20. 20. >2. Prepare For Success Three #1 apps, from 6 to 90 servers in weeks Cloudant, 9-26-2012 20
  21. 21. >3. Scale Invariance Cloudant, 9-26-2012 21
  22. 22. >3. Scale Invariance mobile/tablet desktop Goal: Megabytes to Petabytes Cloud Cloudant, 9-26-2012 22
  23. 23. >3. Scale Invariance ‘Carry Small, Live Large’single user experience at vastly different scales Cloudant, 9-26-2012 23
  24. 24. >4. No Preferred Frame So why do you have a global ‘write master’? Cloudant, 9-26-2012 24
  25. 25. >4. No Preferred FrameThis simple document......establishes Continuous Pipe from Europe to US Cloudant, 9-26-2012 25
  26. 26. >4. No Preferred FrameAnd you can do the reverse... ...at the same time Cloudant, 9-26-2012 26
  27. 27. >4. No Preferred Frame Write local, live globalWhat could you do with relaxed constraints? Cloudant, 9-26-2012 27
  28. 28. >4. No Preferred Frame Data Import 18 18 Size [GB] Doc Count [Million] Actual Customer Data France to Amsterdam 16 Data Size [GB] 16 Disk Size [GB] 14 14 Documents [M] 12 12 10 10 8 8 6 6 4 4 2 2 0 0 0 2000 4000 6000 8000 10000 12000 14000 Time [sec] One click (continuous) Import Cloudant, 9-26-2012 28
  29. 29. Big and Getting Bigger Cloudant, 9-26-2012 29
  30. 30. Big and Getting Bigger• And of course, we are hiring Languages erlang, scala, c, javascript, python, clojure, html5, iOS, Android, ruby/chef Sample problems in the Seattle office Create file format optimized for (huge) structured time-series data Integrate Cubism into two-tier application stack Profile creation of 100M databases (real customer) PIG / HIVE integration Prototype read-in-place Hadoop connector Cloudant, 9-26-2012 30

×