SlideShare a Scribd company logo
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Beyond Messaging: Enterprise-Scale, Multi-Cloud Intelligent
Routing
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
About Me
2
Copyright 2018 Adobe Inc. All Rights Reserved.
Anand Phatak
Software Architect, Adobe Experience Cloud Pipeline
aphatak@adobe.com
/anandphatak
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Platform Pipeline
3
200
Kafka Brokers
Multiple
Data Centers
Multiple
Public Clouds
40 Billion
Messages per day
25,000
Topics
200,000
Partitions
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Cloud
Collection of best-in-class solutions for marketing, analytics, advertising, and
commerce
4
Copyright 2018 Adobe Inc. All Rights Reserved.
Analytics Content ManagementAudience Profile
Campaign Execution Personalization Advertising
Commerce
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Platform
5
§ Enterprise Data Pipeline
§ Unified Customer Profile
§ Data Science and Analysis
§ Open, Extensible Architecture
§ Standardization and Privacy
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Your Products and Services
6
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Everything is connected
7
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9
Kafka
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Global Footprint
10
KafkaKafka
Ireland
Kafka
Amsterdam Singapore
Kafka
US West
Kafka
US East
Kafka
London
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Platform Pipeline (P6E)
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Inside Pipeline
12
Pipeline Console
Smart Edge
Produce Consume
Kafka
Kafka MirrorMaker
Monitoring/Alerting
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Pipeline Multi-Cloud Transport Networking
13
Smart Edge
Produce Consume
Smart Edge
Produce Consume
Kafka
Kafka
Kafka
Kafka
MirrorMaker
MirrorMaker
Data Center 1 Data Center 2
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Enterprise Scale, Multi-Cloud Intelligent Routing.
14
KafkaKafka
Ireland
Kafka
Amsterdam Singapore
Kafka
US West
Kafka
US East
Kafka
London
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
15
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Pipeline Smart Edge Service
§ REST APIs to Produce
§ REST APIs to Consume
§ SYNC Markers for Commits
§ Authorization and ACLs
§ Message Routing
§ Message Filtering
§ Client SDKs
16
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
REST API to Produce
§ Individual or batched messages
§ Few hundred bytes to 1 MB
§ JSON and Google Protocol Buffers (protobuf)
§ Schema awareness using Adobe’s XDM
Registry
17
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
REST API to Consume
§ Firehose API – messages are pushed near real
time
§ Consumers and consumer groups
§ Server side message filtering
§ Offset management through SYNC packets
§ SDKs provide easier SYNC handling
18
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
19
Success Ingredients
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Experience Data Model
20
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Experience Data Model
§ Adobe Experience Data Model (XDM)
§ Open specification for digital experiences
§ https://github.com/adobe/xdm
§ Interoperability, expressiveness
§ JSON schema
21
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 22
Standardized Message Header
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Standardized Message Header
§ Tenant ID – (customer’s unique
identifier)
§ Service ID – source service
§ Name/value pair metadata
§ XDM schema reference
23
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
24
Message Routing
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
MirrorMaker
MirrorMaker
Routing Scheme
25
SmartEdge
topicA
Location 2
topicA
topicA_loc2
Location 1
SmartEdge
Producer
Consumer
topicA_loc1
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Routing Rules
§ Topic level routing – static destination setting
§ Tenant Based Routing – dynamic destination lookup
§ Service-based Routing – service specific destination
§ Pluggable routing logic
26
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 27
Smart Edge
Produce Consume
Produce Consume
Smart EdgeSmart Edge
Produce Consume
Kafka
MirrorMaker
Produce Consume
Smart Edge
Direct Connectivity
Hop
Routing Hub Cluster
Hop
HopHop
Mirroring
Kafka
Kafka
Kafka
Kafka
Kafka
Kafka
Kafka
Kafka
MirrorMaker
MirrorMaker MirrorMaker
MirrorMaker
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
28
Self Service
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Pipeline Console
§ Self-service for topic creation
§ Routing rules
§ Usage reports
§ Monitoring dashboard
29
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
30
Lessons
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Lessons
31
§ Standardize deployments across clouds
§ Inconsistent configurations
§ Long Deployment cycles
§ Monitoring everything centrally
§ API latencies
§ Consumer lags
§ Partition fragmentation
§ Network glitches
§ Mirroring lags
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Lessons
32
§ Throttle misbehaving clients
§ Usage beyond granted quota
§ Excessive API calls and disconnects/reconnects
§ Self-serve
§ Provide adequate graphs and charts
§ Monitoring and alerting tools
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Future
33
§ Ephemeral fronting produce-only clusters
§ Persistent consume-only clusters
§ Self-service on-demand clusters
§ Make intelligent routing without topic duplication
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 34
Thank You!
Copyright 2018 Adobe Inc. All Rights Reserved.
Anand Phatak
aphatak@adobe.com
/anandphatak

More Related Content

More from confluent

Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
confluent
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
confluent
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
confluent
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
confluent
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
confluent
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
confluent
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
confluent
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
confluent
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
confluent
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
confluent
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
confluent
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
confluent
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
confluent
 
The Journey to Data Mesh with Confluent
The Journey to Data Mesh with ConfluentThe Journey to Data Mesh with Confluent
The Journey to Data Mesh with Confluent
confluent
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
confluent
 
Confluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with ReplyConfluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with Reply
confluent
 
Citi Tech Talk Disaster Recovery Solutions Deep Dive
Citi Tech Talk  Disaster Recovery Solutions Deep DiveCiti Tech Talk  Disaster Recovery Solutions Deep Dive
Citi Tech Talk Disaster Recovery Solutions Deep Dive
confluent
 
Citi Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid CloudCiti Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid Cloud
confluent
 
Partner Tech Talk Q3: Q&A with PS - Migration and Upgrade
Partner Tech Talk Q3: Q&A with PS - Migration and UpgradePartner Tech Talk Q3: Q&A with PS - Migration and Upgrade
Partner Tech Talk Q3: Q&A with PS - Migration and Upgrade
confluent
 

More from confluent (20)

Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
 
The Journey to Data Mesh with Confluent
The Journey to Data Mesh with ConfluentThe Journey to Data Mesh with Confluent
The Journey to Data Mesh with Confluent
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
 
Confluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with ReplyConfluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with Reply
 
Citi Tech Talk Disaster Recovery Solutions Deep Dive
Citi Tech Talk  Disaster Recovery Solutions Deep DiveCiti Tech Talk  Disaster Recovery Solutions Deep Dive
Citi Tech Talk Disaster Recovery Solutions Deep Dive
 
Citi Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid CloudCiti Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid Cloud
 
Partner Tech Talk Q3: Q&A with PS - Migration and Upgrade
Partner Tech Talk Q3: Q&A with PS - Migration and UpgradePartner Tech Talk Q3: Q&A with PS - Migration and Upgrade
Partner Tech Talk Q3: Q&A with PS - Migration and Upgrade
 

Recently uploaded

June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
Daiki Mogmet Ito
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
IndexBug
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
Things to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUUThings to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUU
FODUU
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 

Recently uploaded (20)

June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
Things to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUUThings to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUU
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 

Beyond Messaging: Enterprise-Scale, Multi-Cloud Intelligent Routing

  • 1. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Beyond Messaging: Enterprise-Scale, Multi-Cloud Intelligent Routing
  • 2. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. About Me 2 Copyright 2018 Adobe Inc. All Rights Reserved. Anand Phatak Software Architect, Adobe Experience Cloud Pipeline aphatak@adobe.com /anandphatak
  • 3. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Experience Platform Pipeline 3 200 Kafka Brokers Multiple Data Centers Multiple Public Clouds 40 Billion Messages per day 25,000 Topics 200,000 Partitions Copyright 2018 Adobe Inc. All Rights Reserved.
  • 4. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Experience Cloud Collection of best-in-class solutions for marketing, analytics, advertising, and commerce 4 Copyright 2018 Adobe Inc. All Rights Reserved. Analytics Content ManagementAudience Profile Campaign Execution Personalization Advertising Commerce
  • 5. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Experience Platform 5 § Enterprise Data Pipeline § Unified Customer Profile § Data Science and Analysis § Open, Extensible Architecture § Standardization and Privacy Copyright 2018 Adobe Inc. All Rights Reserved.
  • 6. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Your Products and Services 6 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 7. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Everything is connected 7 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 8. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 9. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9 Kafka Copyright 2018 Adobe Inc. All Rights Reserved.
  • 10. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Global Footprint 10 KafkaKafka Ireland Kafka Amsterdam Singapore Kafka US West Kafka US East Kafka London Copyright 2018 Adobe Inc. All Rights Reserved.
  • 11. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Experience Platform Pipeline (P6E)
  • 12. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Inside Pipeline 12 Pipeline Console Smart Edge Produce Consume Kafka Kafka MirrorMaker Monitoring/Alerting Copyright 2018 Adobe Inc. All Rights Reserved.
  • 13. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Pipeline Multi-Cloud Transport Networking 13 Smart Edge Produce Consume Smart Edge Produce Consume Kafka Kafka Kafka Kafka MirrorMaker MirrorMaker Data Center 1 Data Center 2 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 14. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Enterprise Scale, Multi-Cloud Intelligent Routing. 14 KafkaKafka Ireland Kafka Amsterdam Singapore Kafka US West Kafka US East Kafka London Copyright 2018 Adobe Inc. All Rights Reserved.
  • 15. © 2018 Adobe Systems Incorporated. All Rights Reserved. 15 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 16. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Pipeline Smart Edge Service § REST APIs to Produce § REST APIs to Consume § SYNC Markers for Commits § Authorization and ACLs § Message Routing § Message Filtering § Client SDKs 16 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 17. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. REST API to Produce § Individual or batched messages § Few hundred bytes to 1 MB § JSON and Google Protocol Buffers (protobuf) § Schema awareness using Adobe’s XDM Registry 17 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 18. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. REST API to Consume § Firehose API – messages are pushed near real time § Consumers and consumer groups § Server side message filtering § Offset management through SYNC packets § SDKs provide easier SYNC handling 18 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 19. © 2018 Adobe Systems Incorporated. All Rights Reserved. 19 Success Ingredients Copyright 2018 Adobe Inc. All Rights Reserved.
  • 20. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Experience Data Model 20 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 21. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Experience Data Model § Adobe Experience Data Model (XDM) § Open specification for digital experiences § https://github.com/adobe/xdm § Interoperability, expressiveness § JSON schema 21 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 22. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 22 Standardized Message Header Copyright 2018 Adobe Inc. All Rights Reserved.
  • 23. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Standardized Message Header § Tenant ID – (customer’s unique identifier) § Service ID – source service § Name/value pair metadata § XDM schema reference 23 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 24. © 2018 Adobe Systems Incorporated. All Rights Reserved. 24 Message Routing Copyright 2018 Adobe Inc. All Rights Reserved.
  • 25. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. MirrorMaker MirrorMaker Routing Scheme 25 SmartEdge topicA Location 2 topicA topicA_loc2 Location 1 SmartEdge Producer Consumer topicA_loc1 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 26. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Routing Rules § Topic level routing – static destination setting § Tenant Based Routing – dynamic destination lookup § Service-based Routing – service specific destination § Pluggable routing logic 26 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 27. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 27 Smart Edge Produce Consume Produce Consume Smart EdgeSmart Edge Produce Consume Kafka MirrorMaker Produce Consume Smart Edge Direct Connectivity Hop Routing Hub Cluster Hop HopHop Mirroring Kafka Kafka Kafka Kafka Kafka Kafka Kafka Kafka MirrorMaker MirrorMaker MirrorMaker MirrorMaker Copyright 2018 Adobe Inc. All Rights Reserved.
  • 28. © 2018 Adobe Systems Incorporated. All Rights Reserved. 28 Self Service Copyright 2018 Adobe Inc. All Rights Reserved.
  • 29. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Pipeline Console § Self-service for topic creation § Routing rules § Usage reports § Monitoring dashboard 29 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 30. © 2018 Adobe Systems Incorporated. All Rights Reserved. 30 Lessons Copyright 2018 Adobe Inc. All Rights Reserved.
  • 31. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Lessons 31 § Standardize deployments across clouds § Inconsistent configurations § Long Deployment cycles § Monitoring everything centrally § API latencies § Consumer lags § Partition fragmentation § Network glitches § Mirroring lags Copyright 2018 Adobe Inc. All Rights Reserved.
  • 32. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Lessons 32 § Throttle misbehaving clients § Usage beyond granted quota § Excessive API calls and disconnects/reconnects § Self-serve § Provide adequate graphs and charts § Monitoring and alerting tools Copyright 2018 Adobe Inc. All Rights Reserved.
  • 33. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Future 33 § Ephemeral fronting produce-only clusters § Persistent consume-only clusters § Self-service on-demand clusters § Make intelligent routing without topic duplication Copyright 2018 Adobe Inc. All Rights Reserved.
  • 34. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 34 Thank You! Copyright 2018 Adobe Inc. All Rights Reserved. Anand Phatak aphatak@adobe.com /anandphatak