Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Role of NoSQL in a Hadoop World – Couchbase Connect 2015

2,888 views

Published on

This session kicks off the Couchbase Connect Big Data track by answering some fundamental questions about the relationship between NoSQL and Hadoop. Both of these address the “big data” challenge but they target different parts of it. We’ll get concrete about where it makes sense to deploy NoSQL versus where it makes sense to deploy Hadoop. More importantly, we’ll discuss how NoSQL and Hadoop compliment each other and why they’re stronger together.

Published in: Technology
  • Be the first to comment

The Role of NoSQL in a Hadoop World – Couchbase Connect 2015

  1. 1. NoSQL IN A HADOOP WORLD: COUCHBASE, HADOOP, SPARK, KAFKA AND MORE WILL GARDELLA, PRODUCT MANAGER, COUCHBASE
  2. 2. ©2015 Couchbase Inc. 2 Title and Bullets  Intro – NoSQL, Couchbase, and what’s new in 4.0  Analytics & Data Integration  The Big Data Big Picture  Hadoop, Spark, Kafka, and Storm
  3. 3. ©2015 Couchbase Inc. 3 Where does “big” data come from?
  4. 4. ©2015 Couchbase Inc. 4 Where does “big” data come from? MobileWeb/Cloud Internet ofThings
  5. 5. ©2015 Couchbase Inc. 5 This is where Couchbase comes in… High availability cache Key-value store Document database Embedded database Sync management NoSQL Data Management for a broad range of apps and use cases Couchbase Server Couchbase Lite Couchbase Sync Gateway
  6. 6. ©2015 Couchbase Inc. 6 Couchbase meets today’s & tomorrow’s requirements Flexible data model Consistent performance at scale High availability Easy, affordable scalability 24x365
  7. 7. ©2015 Couchbase Inc. 7 Oh, and Couchbase is efficient w/ Hardware… http://googlecloudplatform.blogspot.com/2015/05/Couchbase-Server-Hits-One-Million-Writes-Per-Second-with-Just-50-Nodes-of-Google-Compute-Engine.html 1.1M writes/sec 1/6 the hardware of NoSQL competitor 3 Billion records
  8. 8. ©2015 Couchbase Inc. 8 Couchbase Server 4.0 – Beta 8 Coming soon! Download now: www.couchbase.com/beta Multi Dimensional Scaling Option to separate, isolate, and scale querying, indexing, and data as independent services N1QL SQL for JSON Powerful query language based on SQL and global secondary indexes with support for JOINs and more ForestDB High performance storage engine engineered for multi- core processors and solid state drives
  9. 9. Analytics & Data Integration Powered by N1QL
  10. 10. ©2015 Couchbase Inc. 10 N1QL – EnterpriseTool / Application Ecosystem ODBC / JDBC App CB Node ODBC / JDBC ETL ODBC / JDBC BI ODBC / JDBC Visualization CB Node CB Node Standards-based drivers Integrations, partnerships
  11. 11. ©2015 Couchbase Inc. 11 N1QL – EnterpriseTool / Application EcosystemTodayTomorrow June 4 1:45 PM - 2:30 PM Open Connectivity: BI, Integration and Applications onCouchbase UsingODBC and JDBC Kyle Porter, SimbaTechnologies June 4 3:45 PM - 4:30 PM MetanautixQuest and Couchbase: ScalableAnalyticsAcross NoSQL, RDBMS, and Hadoop JimAdler, Metanautix June 3 1:00 PM - 1:45 PM OperationalAnalytics with Looker and N1QL Keenan Rice, Looker June 3 3:45 PM - 4:30 PM Enterprise Data Integration: Using Informatica ETL with Couchbase Nicolas Brisoux, Informatica June 3 4:30 PM - 5:15 PM VisualAnalytics withTableau & Couchbase Jeff Feng & Clara Siegel,Tableau Software
  12. 12. The Big Data Big Picture What’s Hadoop got to do with it?
  13. 13. ©2015 Couchbase Inc. 13 NoSQL versus Hadoop NoSQL Hadoop NoSQL Hadoop Overlap Compliment NoSQL or Hadoop? NoSQL and Hadoop.
  14. 14. ©2015 Couchbase Inc. 14 Big Data at a Glance Couchbase Spark Hadoop (Hive) Use cases • Operational • Web / Mobile • Analytics • Machine Learning • Analytics • Machine Learning Processing mode • Online • Ad Hoc (New!) • Streaming • Ad Hoc • Batch • Batch • Ad Hoc Low latency = < 1ms ops Seconds Minutes Users are typically Millions of customers 100’s of analysts 100’s of analysts Big data = 10s ofTerabytes Petabytes(?) Petabytes ANALYTICALOPERATIONAL
  15. 15. ©2015 Couchbase Inc. 15 Couchbase: Full range of Connectors
  16. 16. ©2015 Couchbase Inc. 16 Database Change Protocol (DCP)  Innovative protocol for data sync in Couchbase Server  Increases data sync efficiency with massive data footprints  Removes slower disk-IO from the data sync path  Improves latencies to replication for data durability  Powers data replication & XDCR for HA / DR, maintains indexes, and performs other critical functions  In future, will provide a programmable data sync protocol for external stores outside Couchbase Server 16
  17. 17. ©2015 Couchbase Inc. 17 Lambda Architecture 1 4 5 DATA SERVE QUER Y New Data Stream Analysis All Data Precompute Views (Map Reduce) Process Stream Incremental Views Batch Recompute Real-Time Increment Batch Layer Serving Layer Speed Layer 2 BATCH 3 SPEED
  18. 18. ©2015 Couchbase Inc. 18 Lambda Architecture New Data Stream MergedView All Data Precompute Views (Map Reduce) Process Stream Incremental Views Partial Aggregate Partial Aggregate Partial Aggregate Real-Time Data Batch Recompute BatchViews Real-TimeViews Real-Time Increment Merge Batch Layer Serving Layer Speed Layer
  19. 19. ©2015 Couchbase Inc. 19 Lambda + Couchbase New Data Stream MergedView All Data Precompute Views (Map Reduce) Process Stream Incremental Views Partial Aggregate Partial Aggregate Partial Aggregate Real-Time Data Batch Recompute BatchViews Real-TimeViews Real-Time Increment Merge Batch Layer Serving Layer Speed Layer
  20. 20. ©2015 Couchbase Inc. 20 Lamba + Hadoop + Spark + Storm New Data Stream MergedView All Data Precompute Views (Map Reduce) Process Stream Incremental Views Partial Aggregate Partial Aggregate Partial Aggregate Real-Time Data Batch Recompute BatchViews Real-TimeViews Real-Time Increment Merge Batch Layer Serving Layer Speed Layer Couchbase Hadoop Connector (Sqoop)
  21. 21. ©2015 Couchbase Inc. 21 Hadoop, Spark, and StormTodayTomorrow June 4 5:15 PM - 6:00 PM Stream Processing with Spark and Storm Shane Johnson, Couchbase June 4 2:30 PM - 3:15 PM Spark and Couchbase JohnTripitier, Databricks Michael Nitschinger, Couchbase June 3 11:30 AM - 12:15 PM Introduction to the Hadoop Ecosystem Shivaji Dutta, Hortonworks June 3 5:15 PM - 6:00 PM (DeveloperTrack) Spark with Couchbase to ElectrifyYour Data Processing Michael Nitschinger, Couchbase
  22. 22. ©2015 Couchbase Inc. 22 New: Couchbase Spark Connector Available Now: Dev Preview 2  Spark Core  Create RDDs from Documents, Views and N1QL Queries.  Writing RDDs and DStreams into Couchbase.  FullyTransparent cluster and bucket management, including direct access if needed  Spark SQL – based on N1QL 22 Planned forGA  Spark Streaming  Experimental support: DCP to feed Spark streams  Planned for GA github.com/couchbaselabs/couchbase-spark-connector
  23. 23. ©2015 Couchbase Inc. 23 Kafka  Data broker w/ publish / subscribe system  Massively scalable, well decoupled  Messages queued until the recipient can retrieve them TodayTomorrow Tomorrow 10:30 AM - 11:15 AM PayPal: Creating a Central Data Backbone:Couchbase to Couchbase to Kafka to Hadoop and Back Shibi Sudhakaran, Paypal Justin Michaels, Couchbase Today 2:30 PM - 3:15 PM RealTime Streams at Scale with Kafka Ewen Cheslack-Postava, Confluent David Maier, Couchbase
  24. 24. ©2015 Couchbase Inc. 24 Lamba + Hadoop + Spark + Storm + Kafka New Data Stream MergedView All Data Precompute Views (Map Reduce) Process Stream Incremental Views Partial Aggregate Partial Aggregate Partial Aggregate Real-Time Data Batch Recompute BatchViews Real-TimeViews Real-Time Increment Merge Batch Layer Serving Layer Speed Layer
  25. 25. ©2015 Couchbase Inc. 25 Couchbase NoSQL – Success Stories at ScaleTodayTomorrow June 3 1:45 PM - 2:30 PM Couchbase at LinkedIn 2015 Michael Kehoe & Brian Sherwin, LinkedIn Justin Michaels, Couchbase Tomorrow 4:30 PM - 5:15 PM Big Data Query Landscape: N1QL and More Yingyi Bu, Couchbase June 4 1:00 PM - 1:45 PM Big Data is Complex and Chaotic: Couchbase Makes it Manageable Chad Chatfield & Andy Kruth, Avalon Consulting, June 3 2:30 PM - 3:15 PM (Operations) Couchbase Server Scalability and Performance at LinkedIn Benjamin (Jerry) Franz, LinkedIn June 4 11:15 AM - 12:00 PM Finding NoSQL's Home in the Enterprise Data Strategy Roadmap Gabe Arnett, Moody's Analytics
  26. 26. BuildYour AppTODAY! Download Couchbase Server 4.0 Beta at www.couchbase.com/beta
  27. 27. Thank you. will.gardella@couchbase.com Twitter: @WillGardella
  28. 28. Get Started withCouchbase Server 4.0: www.couchbase.com/beta GetTrained on Couchbase: training.couchbase.com

×