Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Google Cloud and Couchbase Server: Zero to Millions of Operations in No Time: Couchbase Connect 2015

2,611 views

Published on

What happens when you combine the industry-leading performance and scalability of Couchbase with the horsepower of Google’s Compute Cloud? In this joint talk between Google and Couchbase we’ll provide expert insight into how Google’s Compute Cloud is constructed. We’ll then walk you through how we provisioned and tuned Couchbase for this platform before revealing how little it costs to perform 1 million writes to a dataset of 3 billion documents!

Published in: Technology
  • Be the first to comment

Google Cloud and Couchbase Server: Zero to Millions of Operations in No Time: Couchbase Connect 2015

  1. 1. Couchbase on Google Cloud Platform Ivan Santa Maria Filho, Performance Engineer/Technical Lead @ Google David Haikney, Technical Support Manager @ Couchbase
  2. 2. Decreasing cost enables virtually limitless storage in the cloud. $600 can buy enough storage for the world’s music. (Source: McKinsey Global Institute May 2011) Computing as a utility is now available for easy purchase, provided from massively efficient data centers. The internet allows for a model of real-time access to new innovation, information, and applications from a wide range of devices. Affordable capacity On-demand computing Instant access Trends
  3. 3. “People tend to overestimate what can be done in one year and to underestimate what can be done in five or ten years.” Amara's Law
  4. 4. 75 years 1957 2003 2013 500 25 years 10 years (average age of a company joining the S&P 500)
  5. 5. A Global Software Based Network For the past 15 years, Google has been building out the world’s fastest, most powerful, highest quality cloud infrastructure on the planet. Google has been running some of the world’s largest distributed systems with unique and stringent requirements.
  6. 6. SpannerDremelMapReduce Bigtable Colossus 2012 20132002 2004 2006 2008 2010 GFS Compute Engine Innovating Software & Driving Technology Forward
  7. 7. Management Mobile Developer Tools Compute Networking Big Data Services Storage
  8. 8. Best Price/Performance, Efficient, Highest Throughput - Lowest Latency, Predictable, Scalable, Easy to Analyze Deliver the Best Performing Cloud Platform Provide expert benchmarking Support sales on competitive deals Capture a competitive view of our products to support business decisions Drive end-to-end performance improvements with internal teams Develop a vibrant community of benchmarkers and partners for cloud performance Guide our customers to better performance through talks, papers and tools
  9. 9. ManagementNetworkingCompute Big Data Storage Mobile Developer Tools Compute Engine Container Engine App Engine Compute
  10. 10. Manage your infrastructure Flexibility Agility Google Compute Engine Your Code Replica Pools Provisioning and health checking Managed VMs OS management, deployments, logging and monitoring Your Code Your Code App Engine Managed Runtimes Manage your serving stack Your Code Compute as a Spectrum
  11. 11. • Flexible and Familiar Infrastructure • High Data Security • Connect with the Google Network • Fast and Easy Provisioning • Flexible Billing • Large and Powerful Disks • Green Computing • Partner Powered Compute Engine Value
  12. 12. • Sub-hour Billing & Sustained-Use Discounts • Up to 10TB Persistent Disk • Over 44 Instance Types • Advanced Networking • Instance Metadata and Startup Scripts • Load Balancing • Monitoring • Snapshotting Compute Engine Features
  13. 13. Persistent Disks Storage System Storage Node Cache Storage ... Storage Engine Storage Log Storage Node Cache Storage Storage Node Cache Storage Virtualization Layer OS IO Subsystem VMComputer OS IO Subsystem Network • Exposed as block devices • “Sectors” are spread over several servers • Read/write on one VM • Read-only on up to 500 VMs • Point in time snapshot to Google Cloud Storage • Differential snapshots • Global snapshot replication, restore anywhere in the world
  14. 14. Standard PD Boot Volumes Small Fileservers and DBs Streaming IO Consistent Throughput Burst Capability Any size from 1GB to 10TB Yes Yes Yes Yes Yes SSD PD Heavy Use Databases Assured Throughput Any size from 1GB to 10TB Yes Yes Yes Yes Yes Local SSD Internet Scale DBs High Performance Scratch Hadoop* Assured Throughput Sub-ms Latency Up to 4 x 400GB Yes Yes No No No Uses Performance Capacity Encryption Live migration Redundancy Snapshots Checksums 20080 Local SSD IOPS/GB (log scale) 0 Standard PD 302 156 SSD PD Priced Per GB. No IO charges.
  15. 15. Couchbase on Google Compute Engine The challenge Several published claims of: “1 Million writes per second with x nodes” How does Couchbase measure up?.....
  16. 16. Making the smallprint bigger 3 Billion Documents 100% of documents 200 Bytes Document SizeDataset Size HardwareLatencyDurability Working Set Measured by client after replication 50 x n1-standard-16 500GB PD-SSD replicated to additional server See next slide Transparency matters: github.com/couchbaselabs/google_compute_benchmark
  17. 17. When has your write been written? • Couchbase acknowledges writes at the managed cache • Application can request callbacks: • When item is persisted to disk • When item has been replicated to another node • Typically reserved for significant writes • This benchmark uses replicated latency • Guards against catastrophic loss of node + disk Latency for each op calculated by client as follows: (i) Clock starts ticking when client fires op (ii) Clock stops only when client confirms second node has a copy
  18. 18. The Benchmark Environment • 32 Clients (n1-highcpu-8) • 8 CPUs • Running libcouchbase pillowfight • 50 Servers (n1-standard-16) • 16 CPUs • 60 GB RAM • 500GB PD-SSD • Couchbase 3.0.2 • debian-wheezy backports
  19. 19. Tuning OS • Separate data volume • Journaling disabled Clients • Number of clients (32) • Number of threads (4) • 100% sets • Buffered not batched Server • Increased compaction threshold • Fewer writer threads (2) 90% achieved straight out of the box
  20. 20. Results - 1 Million Writes per Second Results Data Set 3 Billion 100 Million Working Set 100% 100% Sustained writes / s 1.1 Million 1 Million Median Latency 14 ms 18 ms 95th% latency 27 ms 36 ms 99th% latency 38 ms 48 ms Instances n1-standard-16 n1-standard 8 Servers 50 40 Storage 500GB SSD-PD 500GB PD Price / Performance $56.30 $45.90/hr $21.28 $17.12/hr
  21. 21. Working With Google Compute Engine Built for Big Data ● Billions of documents ● Millions of operations per second ● Consistent low latency even under high load ● Storage throughput ● Excellent price / performance ○ just $17.12/hr! Before sustained use discounts Fantastic Ease-of use ● CLI tools make authentication and scripting simple ● Docs both simple and comprehensive ○ copy and paste examples ● Great provisioning times ● Performant out of the box
  22. 22. The Last Word Couchbase + Google Compute Engine Performance. Scalability. Availability
  23. 23. cloud.google.com Try our open source benchmarking: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker

×