Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
Milliarden von Messages in
Echtzeit: Warum Paypal &
LinkedIn auf eine
Engagement-Datenbank
vertrauen
09.10.2018
Bruno Šimić – Solutions Engineer
ATTRIBUTES OF AN
ENGAGEMENT DATABASE
Always on,
always fast
Secure, secure,
secure
Built-in
smarts
Seamlessly
mobile
Hello cloud,
hello world
Built for change
- at scale
Why
Customers
Choose
Couchbase?
Memory-first
Architecture
Full SQL Query
Language
Active-Active
Global Data
Replication
Multi-dimensional
scaling
Mobile
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
Core Principles of Couchbase
Platform Evolution
Started with Core Principles
– True auto sharding
– JSON-based flexible data model
– Memory-first Architecture
– Asynchronous approach to everything
– Scale workloads independently
Managed
Cache
Key-Value
Store
Document
Database Mobile
N1QL
Query
Full Text
Search Analytics
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 6Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
Couchbase
Hybrid / Multi-Cloud Data
Platform
• Service-Centric Clustered Data System
- Multi-process Architecture
- Dynamic Distribution of Facilities
- Cluster Map Distribution
- Automatic Failover
• Offline Mobile Data Integration
• N1QL - SQL-like Query Engine for JSON
• Global Secondary Indexes
• Lowest Latency Key-Value API
• Active-Active Inter-DC Replication
• Full-Text Search
• Eventing
• Operational Analytics
On-premPrivate Cloud
AWSAzure
GCP
Eventing
Mobile Query
Key
Value
Analytics
Indexing
Full Text
Search
Couchbase designed for containerized applications
Introducing Couchbase Autonomous Operator
Couchbase Autonomous Operator is an application-specific controller that extends the
Kubernetes API to create, configure and manage instances of complex stateful applications
on behalf of a Kubernetes user.
It builds upon the basic Kubernetes resource and controller concepts, but also includes
domain or application-specific knowledge to automate common tasks better managed by
computers.
Couchbase K8 operator Architecture
Next Generation Data Management Architecture
Data
Sources
Enterprise
Applications
Social Media
Web &
Ecommerce
Mobile AppsSensor Data MainframeExternal DataSystem Logs
Data
Integration
Speed Layer Batch Layer
Data
Management
Data Lake Data Warehouse
Business Intelligence Dashboards Query ToolsData
Access
In Memory Cache
Data Serving Layer
Next Generation Data Management Architecture
Enterprise
Applications
Social Media
Web &
Ecommerce
Mobile AppsSensor Data MainframeExternal DataSystem Logs
Speed Layer Batch Layer
Data Lake
Business Intelligence Dashboards Query Tools
In Memory Cache
Data Warehouse
N1QL Query Workbench
Data
Sources
Data
Integration
Data
Management
Data
Access
Data Serving Layer
Real Time
Data Ingestion
Ingest, Process, Load and Serve Data at global scale
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
Kafka Connect Couchbase
Connector 3.4
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
• Use Couchbase as either a consumer or producer with Kafka
message queues
• Continuous streaming, filter, and transformation of events to and
from Couchbase with Source and Sink connectors.
• Fast, reliable and fault tolerant: Based on DCP (Couchbase
replication protocol).
• Efficient: Only load new or modified documents.
• Real-time: Every mutation to Couchbase generates an event
which is published to a Kafka topic.
• Compression and IPv6 support
• Support for rollback mitigation
Couchbase cluster
…
Kafka cluster
Kafka Connect
(Connectors to Extract and Load data)
Confluent and Couchbase - Synergies
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
• Distributed and fault tolerant
• Horizontally scalable
• Geographically replicated
• Low latency
• Open source
Customer Spotlight
1.1 trillion hits a day
75% of flight bookings
worldwide are made
through Amadeus
50M Unique
monthly visitors
2.5B monthly page views
Replaced MongoDB
2821 nodes, 100+ clusters
16M entries every 5 min
2.5 millon ops/sec. on a
single cluster
1 billion+ documents
10TB+ data
Sub-2oo ms
response time
E-Commerce Travel CommunicationsGaming
Financial Services Health Industrial IoTDigital Media
17
Objectives & Challenges
Provide business users with real
time reports and visualizations of
user interaction data
• Collect web and mobile clickstream in
real time
• Integrate with other big data
technologies (Hadoop and Storm)
• Provide views of data across
multiple dimensions (e.g., time,
location, browser and device types)
Solution
Deploy Couchbase Server to capture,
store, and process real time web data
• Ingests data (via Storm) from
multiple inputs, including mobile,
web, and other services, storing
data as JSON documents
• Integrates with Hadoop to pass data
for additional offline analytics
• 130m+ active
accounts, in 190+
countries, 25
currencies
• 10TB data
• 1B documents
The Couchbase
Advantage
Real time performance,
easy integration with Kafka, Storm
and Hadoop
Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2018. All rights reserved.
User Activity Tracking
and real-time analytics
@ Paypal
Results
• Consistent low latency
(sub 10-msec response)
• High availability enabled by
distributed caching and XDCR
• Views for business users are
generated in under 1 minute,
based on 10-minute data
collection intervals
17
Objectives & Challenges
Read scaling and TCO becomes
important
• Constant growth of user profiles,
groups, jobs and publications
• Difficult to move data across data
centers
• Very complex administration of the
system
• Risk of security breach by using
many different components
Solution
Deploy Couchbase Server to cache,
store and replicate data in real time
with low latency
• One singe solution for in memory
caching, ephemeral counter store,
temporary de-duping store and SoT
for internal tooling
• Center of excellence for
Couchbase within Linkedin created
• LinkedIn build ist own NoSQL
databas (Voldemort), Couchbase is
still used as the main database for
this use cases
• 560M users, 26M
companies, 15M
active jobs listings
• 1.5B documents in
cluster
• 10M queries/sec
• 30 Offices in 24
countries
• 24 languages
Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2018. All rights reserved.
Read scaling & data
replication across DCs
worldwide
Results
• Consistent low latency
• High availability enabled by
distributed caching and
XDCR
• Higher acceptance at users
• Scalable and performant
system with zero downtime
• Lower TCO
Additional information
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
Couchbase Mobile: The Full-Stack
Mobile Data Platform
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
Lightweight embedded NoSQL database
with full CRUD and query functionality.
Secure web gateway with synchronization, data access,
and data integration APIs for accessing, integrating, and
synchronizing data over the web.
Highly scalable, highly available, high
performance NoSQL database server.
Built-in enterprise level security throughout the entire stack includes user authentication, user and role based
data access control (RBAC), secure transport (TLS), and 256-bit AES full database encryption.
Couchbase Lite Sync Gateway Couchbase Cluster
SECURITY
EMBEDDED DATABASE SYNCHRONIZATION DATABASE SERVERCLIENT WEB TIER DATABASEinternet intranet
Developing
with
Couchbase
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
Couchbase supports a wide
range of frameworks,
languages, platforms and
infrastructure choices.
FRAMEWORKS LANGUAGES PLATFORMS INFRASTRUCTURE
MOBILEMOBILE
Services & Training geared to customer
success
2
3
Professional Services
• Center of Excellence with product experts
• Packaged Services
• Workshops
• Custom Consulting
• Comprehensive NoSQL & Couchbase training
• Hands-on, intensive labs
• Best practices based on 100s of real use cases
• Developer and Admin courses
• Global Couchbase facilities, or at customer site
Learning Services
Email inquiries: Services@couchbase.com
Visit us: http://www.couchbase.com/couchbase-services
Email inquiries: Training@couchbase.com
Visit us: http://training.couchbase.com
Additional Resources
• Kafka connector:
https://docs.couchbase.com/kafka-
connector/3.4/index.html
• Blog:
http://blog.couchbase.com
• Forum:
http://forums.couchbase.com
• General Docs:
http://docs.couchbase.com
• Developer Portal:
http://developer.couchbase.com
• Couchbase Labs:
https://github.com/couchbaselabs
• Query Portal:
http://query.couchbase.com
Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
Thank you
22

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement Database

  • 1.
    Confidential and Proprietary.Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. Milliarden von Messages in Echtzeit: Warum Paypal & LinkedIn auf eine Engagement-Datenbank vertrauen 09.10.2018 Bruno Šimić – Solutions Engineer
  • 2.
    ATTRIBUTES OF AN ENGAGEMENTDATABASE Always on, always fast Secure, secure, secure Built-in smarts Seamlessly mobile Hello cloud, hello world Built for change - at scale
  • 3.
    Why Customers Choose Couchbase? Memory-first Architecture Full SQL Query Language Active-Active GlobalData Replication Multi-dimensional scaling Mobile Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
  • 4.
    Core Principles ofCouchbase Platform Evolution Started with Core Principles – True auto sharding – JSON-based flexible data model – Memory-first Architecture – Asynchronous approach to everything – Scale workloads independently Managed Cache Key-Value Store Document Database Mobile N1QL Query Full Text Search Analytics Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  • 5.
    Confidential and Proprietary.Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 6Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. Couchbase Hybrid / Multi-Cloud Data Platform • Service-Centric Clustered Data System - Multi-process Architecture - Dynamic Distribution of Facilities - Cluster Map Distribution - Automatic Failover • Offline Mobile Data Integration • N1QL - SQL-like Query Engine for JSON • Global Secondary Indexes • Lowest Latency Key-Value API • Active-Active Inter-DC Replication • Full-Text Search • Eventing • Operational Analytics On-premPrivate Cloud AWSAzure GCP Eventing Mobile Query Key Value Analytics Indexing Full Text Search
  • 6.
    Couchbase designed forcontainerized applications
  • 7.
    Introducing Couchbase AutonomousOperator Couchbase Autonomous Operator is an application-specific controller that extends the Kubernetes API to create, configure and manage instances of complex stateful applications on behalf of a Kubernetes user. It builds upon the basic Kubernetes resource and controller concepts, but also includes domain or application-specific knowledge to automate common tasks better managed by computers.
  • 8.
  • 9.
    Next Generation DataManagement Architecture Data Sources Enterprise Applications Social Media Web & Ecommerce Mobile AppsSensor Data MainframeExternal DataSystem Logs Data Integration Speed Layer Batch Layer Data Management Data Lake Data Warehouse Business Intelligence Dashboards Query ToolsData Access In Memory Cache Data Serving Layer
  • 10.
    Next Generation DataManagement Architecture Enterprise Applications Social Media Web & Ecommerce Mobile AppsSensor Data MainframeExternal DataSystem Logs Speed Layer Batch Layer Data Lake Business Intelligence Dashboards Query Tools In Memory Cache Data Warehouse N1QL Query Workbench Data Sources Data Integration Data Management Data Access Data Serving Layer Real Time Data Ingestion
  • 11.
    Ingest, Process, Loadand Serve Data at global scale Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
  • 12.
    Kafka Connect Couchbase Connector3.4 Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. • Use Couchbase as either a consumer or producer with Kafka message queues • Continuous streaming, filter, and transformation of events to and from Couchbase with Source and Sink connectors. • Fast, reliable and fault tolerant: Based on DCP (Couchbase replication protocol). • Efficient: Only load new or modified documents. • Real-time: Every mutation to Couchbase generates an event which is published to a Kafka topic. • Compression and IPv6 support • Support for rollback mitigation Couchbase cluster … Kafka cluster Kafka Connect (Connectors to Extract and Load data)
  • 13.
    Confluent and Couchbase- Synergies Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. • Distributed and fault tolerant • Horizontally scalable • Geographically replicated • Low latency • Open source
  • 14.
    Customer Spotlight 1.1 trillionhits a day 75% of flight bookings worldwide are made through Amadeus 50M Unique monthly visitors 2.5B monthly page views Replaced MongoDB 2821 nodes, 100+ clusters 16M entries every 5 min 2.5 millon ops/sec. on a single cluster 1 billion+ documents 10TB+ data Sub-2oo ms response time E-Commerce Travel CommunicationsGaming Financial Services Health Industrial IoTDigital Media
  • 15.
    17 Objectives & Challenges Providebusiness users with real time reports and visualizations of user interaction data • Collect web and mobile clickstream in real time • Integrate with other big data technologies (Hadoop and Storm) • Provide views of data across multiple dimensions (e.g., time, location, browser and device types) Solution Deploy Couchbase Server to capture, store, and process real time web data • Ingests data (via Storm) from multiple inputs, including mobile, web, and other services, storing data as JSON documents • Integrates with Hadoop to pass data for additional offline analytics • 130m+ active accounts, in 190+ countries, 25 currencies • 10TB data • 1B documents The Couchbase Advantage Real time performance, easy integration with Kafka, Storm and Hadoop Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. User Activity Tracking and real-time analytics @ Paypal Results • Consistent low latency (sub 10-msec response) • High availability enabled by distributed caching and XDCR • Views for business users are generated in under 1 minute, based on 10-minute data collection intervals
  • 16.
    17 Objectives & Challenges Readscaling and TCO becomes important • Constant growth of user profiles, groups, jobs and publications • Difficult to move data across data centers • Very complex administration of the system • Risk of security breach by using many different components Solution Deploy Couchbase Server to cache, store and replicate data in real time with low latency • One singe solution for in memory caching, ephemeral counter store, temporary de-duping store and SoT for internal tooling • Center of excellence for Couchbase within Linkedin created • LinkedIn build ist own NoSQL databas (Voldemort), Couchbase is still used as the main database for this use cases • 560M users, 26M companies, 15M active jobs listings • 1.5B documents in cluster • 10M queries/sec • 30 Offices in 24 countries • 24 languages Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. Read scaling & data replication across DCs worldwide Results • Consistent low latency • High availability enabled by distributed caching and XDCR • Higher acceptance at users • Scalable and performant system with zero downtime • Lower TCO
  • 17.
    Additional information Confidential andProprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
  • 18.
    Couchbase Mobile: TheFull-Stack Mobile Data Platform Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. Lightweight embedded NoSQL database with full CRUD and query functionality. Secure web gateway with synchronization, data access, and data integration APIs for accessing, integrating, and synchronizing data over the web. Highly scalable, highly available, high performance NoSQL database server. Built-in enterprise level security throughout the entire stack includes user authentication, user and role based data access control (RBAC), secure transport (TLS), and 256-bit AES full database encryption. Couchbase Lite Sync Gateway Couchbase Cluster SECURITY EMBEDDED DATABASE SYNCHRONIZATION DATABASE SERVERCLIENT WEB TIER DATABASEinternet intranet
  • 19.
    Developing with Couchbase Confidential and Proprietary.Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. Couchbase supports a wide range of frameworks, languages, platforms and infrastructure choices. FRAMEWORKS LANGUAGES PLATFORMS INFRASTRUCTURE MOBILEMOBILE
  • 20.
    Services & Traininggeared to customer success 2 3 Professional Services • Center of Excellence with product experts • Packaged Services • Workshops • Custom Consulting • Comprehensive NoSQL & Couchbase training • Hands-on, intensive labs • Best practices based on 100s of real use cases • Developer and Admin courses • Global Couchbase facilities, or at customer site Learning Services Email inquiries: Services@couchbase.com Visit us: http://www.couchbase.com/couchbase-services Email inquiries: Training@couchbase.com Visit us: http://training.couchbase.com
  • 21.
    Additional Resources • Kafkaconnector: https://docs.couchbase.com/kafka- connector/3.4/index.html • Blog: http://blog.couchbase.com • Forum: http://forums.couchbase.com • General Docs: http://docs.couchbase.com • Developer Portal: http://developer.couchbase.com • Couchbase Labs: https://github.com/couchbaselabs • Query Portal: http://query.couchbase.com Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
  • 22.