Stream me to the Cloud (and
back) with Confluent & MongoDB
Felix
Reichenbach
MongoDB
Gianluca
Natali
Confluent
Agenda
Current Challenges
A Cloud Story
Solution Presentation & Architecture
Demo
Joint use cases & Customer references
Current Challenges
Business Challenges
Provider needs:
- Tap new revenue streams to stay
competitive
- Scale with the demand and keep
TCO low
- Continuously evolve and improve
user/customer experience
Customer wants:
- Consume services anywhere and
anytime
- Omnichannel with seamless
transition (Mobile <-> Browser)
- Fast response times and Real Time
Interactions (Notifications, Status
Updates, Confirmations…)
Technical Challenges
Made worse by this crisis:
- Scaling for peak hours requires
heavy overprovisioning
- Purchasing Hardware may not be
possible
- Mobility of personnel is limited
Evergreen problems:
- Making use of data is very difficult
- Data is spread across many
applications / domains
- Low latency access is almost
impossible
- Agile development methodologies
cannot be applied
Common Initiatives
● Application Modernization
● Microservices Architectures
● Single View of XYZ
● Real-time Analytics
● Content Management
● IoT
● ...
Across Verticals:
Healthcare
Financial Services
Telco
Technology
Retail
Manufacturing / Automotive
a retail story...
AllGoodThings Inc.
Our Meetings
are changing...
By now probably everyone
of us had experienced a
crowded business video
call from home.
Let me introduce you to
the Conference call
Bingo!
😀
CEO CTO
CFO
● Our online store had 2 main outages last
week
● Surge in Customers complaints and
escalation are a concern
● We are unable to add/change products
as fast as we’d like
● 600% increase in overall traffic to website
● A tweet with a link to an hand sanitizer
product in our store went viral
● Adding a product requires downtime that
we can’t afford with current queue setup
Customer Success Head of Eng.
CMO
● Outages due to lack of resources in DC
● Our DBs is cracking under load.
● We limited concurrent users to stay online
● Need to triple our DC to support peak load
● Health category has skyrocketed to #1
● “Home office” and “Fitness” are
trending upwards
● People can’t visit our shops, all the
business moved to our online store
● #1 Complaints: “Item was successfully
ordered and paid but then received email
that is unavailable”,
● #2 Complaint: “Can’t access the website,
I am stuck in queue”
● Our Support team is working double
shifts, we need to grow the team ASAP
● DC: We can’t triple our DC size, the ROI
doesn’t work. We need to maintain our
margins
● Support: Can we establish more efficient
processes and automate concierge/triage
via support app or Chat bot?
● I can’t approve your requests as they are
AllTheGoodThings Inc.
Meeting Conclusion / Requirements
● Fix the scalability issue while limiting the investment on our DC.
● Leverage public Cloud services to reduce TCO.
● Explore the feasibility of an app to allow automatic resolution of
simple customer queries
● Increase agility in adding products
● Provide real time stock information
a few days later...
Solution Overview
Eng.CEO CTOCS CMOCFO
● Dynamic Scalability
● High Availability
● Increased Agility
● Avoid Lock In
● ...
Event Driven Architectures
Eng.CEO CTOCS CMOCFO
ETL/Data Integration Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Highly Scalable
Durable
Persistent
Ordered
Real-time
ETL/Data Integration Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Highly Scalable
Durable
Persistent
Ordered
Real-timeHighly Scalable
Durable
Persistent
Ordered
Real-time
Event Streaming
Today
Real-Time
Inventory
Real-Time
Fraud
Detection
Real-Time
Customer 360
Machine
Learning
Models
Real-Time
Data
Transformation
...
Contextual Event-Driven Applications
Universal Event Pipeline
Data Stores Logs 3rd Party Apps Custom Apps/Microservices
TREAMSSTREAMS
CONNECT CLIENTS
With
Event
Streaming
and
Kafka
Agility and Scalability
with MongoDB
Eng.CEO CTOCS CMOCFO
Isolated data in many different
applications / domains and structures
22
Data which is accessed together, is stored together...
Customer Product Contact
Pricing Phone Phone
Objects
Tables isolated in several applications and domains
Lead
NameNameParts
ARR Address Contact Roles
SummaryCustomer Detail Specs
How to build a flexible and scalable operational data store??
23
RDBMS vs. Flexible Data Model
Relational MongoDB
{ customer_id : 3,
first_name : ”Danilo",
last_name : ”Nobrega",
city : ”Stockholm",
phones: [
{
number : “46-64-223-9828”,
dnc : true,
type : “home”
},{
number : “46-173-555-
12144”,
type : “cell”
}]
}
Customer ID First Name Last Name City
0 Jonah Rosenboom Hamburg
1 Tim Schojohann München
2 Boris Bialek Zürich
3 Danilo Nobrega Stockholm
4 Ingo Marienfeld Düsseldorf
Phone Number Type DNC Customer ID
49-69-223-9828 home T 3
49-69-143-45986 home T 0
49-173-555-12144 cell (null) 3
49-69-777-1212 home T 1
49-175-698-1213 cell (null) 1
49-162-767-444 cell F 2
24
The document model is the perfect fit because it is...
Easy:
Work with data in a
natural, intuitive
way, fully
transactional
Flexible:
Adapt and
make changes
quickly
Fast:
Get great
performance
with less code
Versatile:
Supports a
wide variety of
data models
and queries
25
Multiple data models, rich query functionality...
Rich Queries
Point | Range | Geospatial | Faceted Search | Aggregations | JOINs | Graph Traversals
JSON Documents Tabular Key-Value Text GraphGeospatial
26
Mobile App with
MongoDB Realm
Personalized
Marketing
Campaign
Analytics
Real-Time Online
Data Store
MongoDB Data Platform
Real Time
Data Sync
SPA with
MongoDB
Stitch
Freedom of Choice: Run wherever you need
Private Cloud
Deploy and self manage on premises
Deploy via Kubernetes with Kubernetes
Operators
Multi- Cloud / PaaS
Build on top of fully managed services
Leverage horizontal and vertical
elasticity
Avoid cloud provider lock-in
Hybrid Cloud
Deploy a homogeneous technology
stack across on-prem and cloud.
Reduce complexity and avoid lock-in
Joint Reference Architecture
Reference Architecture: https://www.confluent.io/resources/confluent-platform-reference-architecture-mongodb/
DEMO
Joint References
Centene, Healthcare Provider
Centene has the largest medicare and medicaid managed
care providers
Challenge
Centene’s Core Challenge is growth
caused by Mergers & Acquisitions
This caused them to reevaluate their
Enterprise Data Integration and Data
Migration Strategies…They wanted
better scalability, availability, faster ETL
Solution
Build centralized eventing
framework for enterprise use
across all Centene domains
Employing CDC for ingest and
leveraging microservices using
Confluent, Kafka, and MongoDB
Financial Services - a large global bank
Leading global financial services firm, a major provider of various investment banking,
retail banking and financial services.
Challenges
Need for centralized incident and event
management solution
Modernize legacy applications
Provide complete health dashboards
across various systems
Streamline payment processing,
customer activity tracking and real-time
data integration across the bank
Solution
The bank implemented a secure logging platform
running Kafka as a service and MongoDB databases
The environment has over 80 clusters and supports
data volumes between 200TB and 1PB per day
Data collection and normalization is at the edge and
data is replicated back to two central data centers
for global reporting and trend analysis
Modernized security information and event management
Filter, transform
aggregate
APP SIEM
Index
Search
Curated streams
Forensic
Archive
MongoDB
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
Sensor Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
Front, rear and top
view cameras
Parking assistant
Environment pointer
Ultrasonic Sensors
Parking assistant with
front and rear camera plus
environment indicator
Crash Sensors
Front protection adaptivity
Side protection
Tail impact protection
Front Camera
Audi Active lane assistant
Speed limit indicator
Adaptive light
Infrared Camera
Rearview assistance with
Pedestrian recognition
Front and Rear
Radar Sensors
ACC with stop and go function
Side assist
Real time data cluster
Real-Time Monitoring
Logs
Reference
data
(monitoring)
CDC
Syslog
MQTT
APP
Filter, transform
aggregate SIEM
Search
Aggregated event data
/ curated streams
iOT
iOT
iOT
Kafka Streams/ksqlDB
Modernize your architectures with Confluent and MongoDB
A new generation of technologies is needed to consume and exploit today's real
time, fast moving data sources. Run your business in real-time, building real-time
applications with historical context.
Together, Confluent and MongoDB enable you to
● Build sophisticated data-driven and event-driven applications
● Modernize your application architecture
● Uncover new sources of data
● Derive insights for a competitive advantage
&
Accelerate your Journey to the Cloud
and back! 😉
Driving innovation and remaining
operational in uncertain times
Koen Rousseau, Digital Lead,
Sanoma Learning Online Education
Tomorrow
10:30am CET
Tim Nutman, Manager Solutions
Architecture, MongoDB
mongodb.com/webinars
Thank You!
felix@mongodb.com gnatali@confluent.io

Stream me to the Cloud (and back) with Confluent & MongoDB

  • 1.
    Stream me tothe Cloud (and back) with Confluent & MongoDB Felix Reichenbach MongoDB Gianluca Natali Confluent
  • 2.
    Agenda Current Challenges A CloudStory Solution Presentation & Architecture Demo Joint use cases & Customer references
  • 3.
  • 4.
    Business Challenges Provider needs: -Tap new revenue streams to stay competitive - Scale with the demand and keep TCO low - Continuously evolve and improve user/customer experience Customer wants: - Consume services anywhere and anytime - Omnichannel with seamless transition (Mobile <-> Browser) - Fast response times and Real Time Interactions (Notifications, Status Updates, Confirmations…)
  • 5.
    Technical Challenges Made worseby this crisis: - Scaling for peak hours requires heavy overprovisioning - Purchasing Hardware may not be possible - Mobility of personnel is limited Evergreen problems: - Making use of data is very difficult - Data is spread across many applications / domains - Low latency access is almost impossible - Agile development methodologies cannot be applied
  • 6.
    Common Initiatives ● ApplicationModernization ● Microservices Architectures ● Single View of XYZ ● Real-time Analytics ● Content Management ● IoT ● ... Across Verticals: Healthcare Financial Services Telco Technology Retail Manufacturing / Automotive
  • 7.
  • 8.
    Our Meetings are changing... Bynow probably everyone of us had experienced a crowded business video call from home. Let me introduce you to the Conference call Bingo! 😀
  • 9.
    CEO CTO CFO ● Ouronline store had 2 main outages last week ● Surge in Customers complaints and escalation are a concern ● We are unable to add/change products as fast as we’d like ● 600% increase in overall traffic to website ● A tweet with a link to an hand sanitizer product in our store went viral ● Adding a product requires downtime that we can’t afford with current queue setup Customer Success Head of Eng. CMO ● Outages due to lack of resources in DC ● Our DBs is cracking under load. ● We limited concurrent users to stay online ● Need to triple our DC to support peak load ● Health category has skyrocketed to #1 ● “Home office” and “Fitness” are trending upwards ● People can’t visit our shops, all the business moved to our online store ● #1 Complaints: “Item was successfully ordered and paid but then received email that is unavailable”, ● #2 Complaint: “Can’t access the website, I am stuck in queue” ● Our Support team is working double shifts, we need to grow the team ASAP ● DC: We can’t triple our DC size, the ROI doesn’t work. We need to maintain our margins ● Support: Can we establish more efficient processes and automate concierge/triage via support app or Chat bot? ● I can’t approve your requests as they are AllTheGoodThings Inc.
  • 10.
    Meeting Conclusion /Requirements ● Fix the scalability issue while limiting the investment on our DC. ● Leverage public Cloud services to reduce TCO. ● Explore the feasibility of an app to allow automatic resolution of simple customer queries ● Increase agility in adding products ● Provide real time stock information
  • 11.
    a few dayslater... Solution Overview
  • 12.
    Eng.CEO CTOCS CMOCFO ●Dynamic Scalability ● High Availability ● Increased Agility ● Avoid Lock In ● ...
  • 13.
  • 14.
  • 15.
    ETL/Data Integration Messaging Batch Expensive TimeConsuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Real-time
  • 16.
    ETL/Data Integration Messaging Batch Expensive TimeConsuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Real-timeHighly Scalable Durable Persistent Ordered Real-time Event Streaming
  • 17.
  • 18.
    Real-Time Inventory Real-Time Fraud Detection Real-Time Customer 360 Machine Learning Models Real-Time Data Transformation ... Contextual Event-DrivenApplications Universal Event Pipeline Data Stores Logs 3rd Party Apps Custom Apps/Microservices TREAMSSTREAMS CONNECT CLIENTS With Event Streaming and Kafka
  • 19.
  • 20.
  • 21.
    Isolated data inmany different applications / domains and structures
  • 22.
    22 Data which isaccessed together, is stored together... Customer Product Contact Pricing Phone Phone Objects Tables isolated in several applications and domains Lead NameNameParts ARR Address Contact Roles SummaryCustomer Detail Specs How to build a flexible and scalable operational data store??
  • 23.
    23 RDBMS vs. FlexibleData Model Relational MongoDB { customer_id : 3, first_name : ”Danilo", last_name : ”Nobrega", city : ”Stockholm", phones: [ { number : “46-64-223-9828”, dnc : true, type : “home” },{ number : “46-173-555- 12144”, type : “cell” }] } Customer ID First Name Last Name City 0 Jonah Rosenboom Hamburg 1 Tim Schojohann München 2 Boris Bialek Zürich 3 Danilo Nobrega Stockholm 4 Ingo Marienfeld Düsseldorf Phone Number Type DNC Customer ID 49-69-223-9828 home T 3 49-69-143-45986 home T 0 49-173-555-12144 cell (null) 3 49-69-777-1212 home T 1 49-175-698-1213 cell (null) 1 49-162-767-444 cell F 2
  • 24.
    24 The document modelis the perfect fit because it is... Easy: Work with data in a natural, intuitive way, fully transactional Flexible: Adapt and make changes quickly Fast: Get great performance with less code Versatile: Supports a wide variety of data models and queries
  • 25.
    25 Multiple data models,rich query functionality... Rich Queries Point | Range | Geospatial | Faceted Search | Aggregations | JOINs | Graph Traversals JSON Documents Tabular Key-Value Text GraphGeospatial
  • 26.
    26 Mobile App with MongoDBRealm Personalized Marketing Campaign Analytics Real-Time Online Data Store MongoDB Data Platform Real Time Data Sync SPA with MongoDB Stitch
  • 27.
    Freedom of Choice:Run wherever you need Private Cloud Deploy and self manage on premises Deploy via Kubernetes with Kubernetes Operators Multi- Cloud / PaaS Build on top of fully managed services Leverage horizontal and vertical elasticity Avoid cloud provider lock-in Hybrid Cloud Deploy a homogeneous technology stack across on-prem and cloud. Reduce complexity and avoid lock-in
  • 28.
    Joint Reference Architecture ReferenceArchitecture: https://www.confluent.io/resources/confluent-platform-reference-architecture-mongodb/
  • 29.
  • 30.
  • 31.
    Centene, Healthcare Provider Centenehas the largest medicare and medicaid managed care providers Challenge Centene’s Core Challenge is growth caused by Mergers & Acquisitions This caused them to reevaluate their Enterprise Data Integration and Data Migration Strategies…They wanted better scalability, availability, faster ETL Solution Build centralized eventing framework for enterprise use across all Centene domains Employing CDC for ingest and leveraging microservices using Confluent, Kafka, and MongoDB
  • 32.
    Financial Services -a large global bank Leading global financial services firm, a major provider of various investment banking, retail banking and financial services. Challenges Need for centralized incident and event management solution Modernize legacy applications Provide complete health dashboards across various systems Streamline payment processing, customer activity tracking and real-time data integration across the bank Solution The bank implemented a secure logging platform running Kafka as a service and MongoDB databases The environment has over 80 clusters and supports data volumes between 200TB and 1PB per day Data collection and normalization is at the edge and data is replicated back to two central data centers for global reporting and trend analysis
  • 33.
    Modernized security informationand event management Filter, transform aggregate APP SIEM Index Search Curated streams Forensic Archive MongoDB CDC Syslog Network traffic Firewall logs RDBMS Application logs Sensor Data HTTP proxy logs QRadar Arcsight Splunk Elastic
  • 34.
    Front, rear andtop view cameras Parking assistant Environment pointer Ultrasonic Sensors Parking assistant with front and rear camera plus environment indicator Crash Sensors Front protection adaptivity Side protection Tail impact protection Front Camera Audi Active lane assistant Speed limit indicator Adaptive light Infrared Camera Rearview assistance with Pedestrian recognition Front and Rear Radar Sensors ACC with stop and go function Side assist Real time data cluster
  • 35.
    Real-Time Monitoring Logs Reference data (monitoring) CDC Syslog MQTT APP Filter, transform aggregateSIEM Search Aggregated event data / curated streams iOT iOT iOT Kafka Streams/ksqlDB
  • 36.
    Modernize your architectureswith Confluent and MongoDB A new generation of technologies is needed to consume and exploit today's real time, fast moving data sources. Run your business in real-time, building real-time applications with historical context. Together, Confluent and MongoDB enable you to ● Build sophisticated data-driven and event-driven applications ● Modernize your application architecture ● Uncover new sources of data ● Derive insights for a competitive advantage & Accelerate your Journey to the Cloud and back! 😉
  • 37.
    Driving innovation andremaining operational in uncertain times Koen Rousseau, Digital Lead, Sanoma Learning Online Education Tomorrow 10:30am CET Tim Nutman, Manager Solutions Architecture, MongoDB mongodb.com/webinars
  • 38.