Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Elasticsearch and Couchbase: Configuration to Capabilities to Real Time BI: Couchbase Connect 2015

2,382 views

Published on

Couchbase and Elasticsearch are both optimized for JSON. Where do the two technologies come together and what are the best practices in using them? This session from David Maier of Couchbase and Bala Venkatrao from Elastic will answer exactly those questions. After a quick overview and discussion of use cases, we will describe how the Elasticsearch plugin is set up and then configured to index data from Couchbase. In addition, we will also showcase the use of Kibana for data visualization on top of Elasticsearch. You will come away armed with the knowledge needed to successfully deploy these technologies together in your apps.

Published in: Technology
  • Be the first to comment

Elasticsearch and Couchbase: Configuration to Capabilities to Real Time BI: Couchbase Connect 2015

  1. 1. ELASTIC AND COUCHBASE: CONFIGURATION TO CAPABILITIES TO REAL-TIME BI Bala Venkatrao Senior Director Product Management @ Elastic David Maier Principal Solutions Engineer
  2. 2. ©2015 Couchbase Inc. 2 Agenda  Welcome  Introduction to Elastic  Demo  Behind the scenes  What’s new in the 2.0 Plug-in
  3. 3. ©2015 Couchbase Inc. 3 Welcome! BalaVenkatrao Senior Director, Product Management
  4. 4. Introduction to Elastic Unlocking the value of your data through search, analytics, and more . . . www.elastic.co
  5. 5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Problem: Complex Data + Diverse Requirements Social Locatio n User- Activity Machine Documents 5 COMPLEX & DIVERSE DATA MISSION CRITICAL BUSINESS REQUIREMENTS Developer requirements Many users / use cases Fast data processing Large data volumes Data quality & integrity How do you build scalable, Search/Analy tics platform? Cross-source insights
  6. 6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 6 New Systems to Address New Requirements Schemaless Horizontal Scale Real Time Data Availability Fast Search & Query Execution Structured and unstructured Flexibility Sophisticated Query Language
  7. 7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 7 Elasticsearch: Search, Store, and Analyze Distributed, scalable, and resilient Designed for scale-out; high availability Developer friendly API-first; schemaless, native JSON, client libraries for any language Real-time Search & Analytics Real-time aggregations, geospatial, full-text search; query structured and unstructured data
  8. 8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 8 Introducing the Elastic Product Portfolio Kibana Visualize and explore data Elasticsearch Store, search, analyze Logstash Hadoop Connector Collect, parse and enrich data Marvel Monitor and manage Shield Secure and protect Found Elasticsearch as a Service Open Source Products Commercial Products Training Professional Services Support Subscriptions Watcher Notify and Alert Powering diverse Use Cases from Search, Analytics, Log Analysis and more…
  9. 9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 9 • Analytics/Search Engine based on Apache Lucene • Schema-free JSON document store • Every field is indexed/searchable • Scalable sharded distributed architecture • Simple REST API • (Near) Real-time Search • Aggregations/Analytics support • Geospatial support • Extensibility via plugins Why use Elasticsearch and Couchbase together? Elasticsearch • Key-Value document store • Supports JSON, Binary, XML … • N1QL Query Language => SQL + JSON • Scalable, distributed architecture • Active memory management (built-in cache) • SDK for language integration in Java, .NET, Node.js, Python… • Uses binary wire protocol for data access • Extends to iOS, Android and Xamarin with Couchbase Mobile Couchbase
  10. 10. Couchbase & Elasticsearch A brief Introduction
  11. 11. ©2015 Couchbase Inc. 11 Couchbase Server & Elasticsearch Key Reference tim #2 bob #3 jim #1 Secondary Index Key Reference #1 { name : ‘jim’ } #2 { name : ‘tim’ } #3 { name : ‘bob’ } Key-Value Access Term Count Reference tim 1 #1 is 2 #1, #2 sitting 2 #1, #2 next 2 #1, #2 to 2 #1, #2 bob 2 #1, #2 jim 1 #2 Inverted Index
  12. 12. ©2015 Couchbase Inc. 12 What is it all about? +
  13. 13. Demo https://github.com/dmaier-couchbase/cb-twitter-demo
  14. 14. ©2015 Couchbase Inc. 14 Demo Overview
  15. 15. ©2015 Couchbase Inc. 15 Twitter Dashboard
  16. 16. Behind the Scenes
  17. 17. ©2015 Couchbase Inc. 17 Reference Architecture
  18. 18. ©2015 Couchbase Inc. 18 XDCR Bucket A Bucket B Bucket C Cluster 1 Bucket A Bucket B Bucket C Cluster 2  Continuous Replication  Topology Aware  Recovery from Network Failure
  19. 19. ©2015 Couchbase Inc. 19 The CouchbaseTransport Plug-in  Needs to be installed on side of Elasticsearch  REST Service  Per default on port 9091  XDCR protocol v1 bin/plugin -install transport-couchbase -url http://packages.couchbase.com.s3.amazonaws.com/rel eases/elastic-search-adapter/2.0.0/elasticsearch- transport-couchbase-2.0.0.zip I speak ‘Couchbase’!
  20. 20. ©2015 Couchbase Inc. 20 Connecting Couchbase Server to Elasticsearch
  21. 21. ©2015 Couchbase Inc. 21 Couchbase specificTypes
  22. 22. ©2015 Couchbase Inc. 22 Default Mapping  Applies to every index  couchbaseCheckpoint  The _source field contains the content of the document not the metadata  Don’t store or index  Don’t make it accessible via the _all field  couchbaseDocument  Handled by the overridden _default_ mapping  The _source field contains the content of the metadata (document id, ..)  Don’t include the metadata in the _all field  Index the content by using Elasticsearch’s _default_ mapping  Can be overridden by using Elasticsearch’s REST interface
  23. 23. ©2015 Couchbase Inc. 23 Plug-in Settings - DocumentType Selector  couchbase.typeSelector  Fully qualified class name of a type selector class  DefaultTypeSelector maps to ‘couchbaseDocument’  e.g. RegExp on the document id which matches ‘review-123’ couchbase.typeSelector: org.elasticsearch.transport.couchbase.capi.RegexTypeSelector couchbase.documentTypes.review: ^review-(d)+$
  24. 24. ©2015 Couchbase Inc. 24 Plug-in Settings - Advanced  couchbase.port  couchbase.username  couchbase.password  couchbase.num_vbuckets  couchbase.defaultDocumentType  couchbase.checkpointDocumentType  couchbase.ignoreDeletes  couchbase.wrapCounters  …
  25. 25. What’s new in the 2.0 Plug-in
  26. 26. ©2015 Couchbase Inc. 26 What’s new in the 2.0 Plug-in  Type Selectors  Support of more than one document type  Support for filtering documents  Possibility to  Ignore deletes  Ignore Elasticsearch errors  To automatically wrap Couchbase counter documents with JSON for indexing purposes  Bug fixes Did you know … ?
  27. 27. Thank you! Q&A

×