SlideShare a Scribd company logo
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Beyond Messaging: Enterprise-Scale, Multi-Cloud Intelligent
Routing
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
About Me
2
Copyright 2018 Adobe Inc. All Rights Reserved.
Anand Phatak
Software Architect, Adobe Experience Cloud Pipeline
aphatak@adobe.com
/anandphatak
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Platform Pipeline
3
200
Kafka Brokers
Multiple
Data Centers
Multiple
Public Clouds
40 Billion
Messages per day
25,000
Topics
200,000
Partitions
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Cloud
Collection of best-in-class solutions for marketing, analytics, advertising, and
commerce
4
Copyright 2018 Adobe Inc. All Rights Reserved.
Analytics Content ManagementAudience Profile
Campaign Execution Personalization Advertising
Commerce
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Platform
5
§ Enterprise Data Pipeline
§ Unified Customer Profile
§ Data Science and Analysis
§ Open, Extensible Architecture
§ Standardization and Privacy
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Your Products and Services
6
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Everything is connected
7
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9
Kafka
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Global Footprint
10
KafkaKafka
Ireland
Kafka
Amsterdam Singapore
Kafka
US West
Kafka
US East
Kafka
London
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Platform Pipeline (P6E)
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Inside Pipeline
12
Pipeline Console
Smart Edge
Produce Consume
Kafka
Kafka MirrorMaker
Monitoring/Alerting
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Pipeline Multi-Cloud Transport Networking
13
Smart Edge
Produce Consume
Smart Edge
Produce Consume
Kafka
Kafka
Kafka
Kafka
MirrorMaker
MirrorMaker
Data Center 1 Data Center 2
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Enterprise Scale, Multi-Cloud Intelligent Routing.
14
KafkaKafka
Ireland
Kafka
Amsterdam Singapore
Kafka
US West
Kafka
US East
Kafka
London
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
15
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Pipeline Smart Edge Service
§ REST APIs to Produce
§ REST APIs to Consume
§ SYNC Markers for Commits
§ Authorization and ACLs
§ Message Routing
§ Message Filtering
§ Client SDKs
16
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
REST API to Produce
§ Individual or batched messages
§ Few hundred bytes to 1 MB
§ JSON and Google Protocol Buffers (protobuf)
§ Schema awareness using Adobe’s XDM
Registry
17
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
REST API to Consume
§ Firehose API – messages are pushed near real
time
§ Consumers and consumer groups
§ Server side message filtering
§ Offset management through SYNC packets
§ SDKs provide easier SYNC handling
18
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
19
Success Ingredients
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Experience Data Model
20
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Experience Data Model
§ Adobe Experience Data Model (XDM)
§ Open specification for digital experiences
§ https://github.com/adobe/xdm
§ Interoperability, expressiveness
§ JSON schema
21
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 22
Standardized Message Header
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Standardized Message Header
§ Tenant ID – (customer’s unique
identifier)
§ Service ID – source service
§ Name/value pair metadata
§ XDM schema reference
23
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
24
Message Routing
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
MirrorMaker
MirrorMaker
Routing Scheme
25
SmartEdge
topicA
Location 2
topicA
topicA_loc2
Location 1
SmartEdge
Producer
Consumer
topicA_loc1
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Routing Rules
§ Topic level routing – static destination setting
§ Tenant Based Routing – dynamic destination lookup
§ Service-based Routing – service specific destination
§ Pluggable routing logic
26
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 27
Smart Edge
Produce Consume
Produce Consume
Smart EdgeSmart Edge
Produce Consume
Kafka
MirrorMaker
Produce Consume
Smart Edge
Direct Connectivity
Hop
Routing Hub Cluster
Hop
HopHop
Mirroring
Kafka
Kafka
Kafka
Kafka
Kafka
Kafka
Kafka
Kafka
MirrorMaker
MirrorMaker MirrorMaker
MirrorMaker
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
28
Self Service
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Pipeline Console
§ Self-service for topic creation
§ Routing rules
§ Usage reports
§ Monitoring dashboard
29
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved.
30
Lessons
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Lessons
31
§ Standardize deployments across clouds
§ Inconsistent configurations
§ Long Deployment cycles
§ Monitoring everything centrally
§ API latencies
§ Consumer lags
§ Partition fragmentation
§ Network glitches
§ Mirroring lags
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Lessons
32
§ Throttle misbehaving clients
§ Usage beyond granted quota
§ Excessive API calls and disconnects/reconnects
§ Self-serve
§ Provide adequate graphs and charts
§ Monitoring and alerting tools
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Future
33
§ Ephemeral fronting produce-only clusters
§ Persistent consume-only clusters
§ Self-service on-demand clusters
§ Make intelligent routing without topic duplication
Copyright 2018 Adobe Inc. All Rights Reserved.
© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 34
Thank You!
Copyright 2018 Adobe Inc. All Rights Reserved.
Anand Phatak
aphatak@adobe.com
/anandphatak

More Related Content

More from confluent

Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
confluent
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
confluent
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
confluent
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
confluent
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
confluent
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
confluent
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
confluent
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
confluent
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
confluent
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
confluent
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
confluent
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
confluent
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
confluent
 
The Journey to Data Mesh with Confluent
The Journey to Data Mesh with ConfluentThe Journey to Data Mesh with Confluent
The Journey to Data Mesh with Confluent
confluent
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
confluent
 
Confluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with ReplyConfluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with Reply
confluent
 
Citi Tech Talk Disaster Recovery Solutions Deep Dive
Citi Tech Talk  Disaster Recovery Solutions Deep DiveCiti Tech Talk  Disaster Recovery Solutions Deep Dive
Citi Tech Talk Disaster Recovery Solutions Deep Dive
confluent
 
Citi Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid CloudCiti Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid Cloud
confluent
 
Partner Tech Talk Q3: Q&A with PS - Migration and Upgrade
Partner Tech Talk Q3: Q&A with PS - Migration and UpgradePartner Tech Talk Q3: Q&A with PS - Migration and Upgrade
Partner Tech Talk Q3: Q&A with PS - Migration and Upgrade
confluent
 

More from confluent (20)

Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
 
The Journey to Data Mesh with Confluent
The Journey to Data Mesh with ConfluentThe Journey to Data Mesh with Confluent
The Journey to Data Mesh with Confluent
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
 
Confluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with ReplyConfluent Partner Tech Talk with Reply
Confluent Partner Tech Talk with Reply
 
Citi Tech Talk Disaster Recovery Solutions Deep Dive
Citi Tech Talk  Disaster Recovery Solutions Deep DiveCiti Tech Talk  Disaster Recovery Solutions Deep Dive
Citi Tech Talk Disaster Recovery Solutions Deep Dive
 
Citi Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid CloudCiti Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid Cloud
 
Partner Tech Talk Q3: Q&A with PS - Migration and Upgrade
Partner Tech Talk Q3: Q&A with PS - Migration and UpgradePartner Tech Talk Q3: Q&A with PS - Migration and Upgrade
Partner Tech Talk Q3: Q&A with PS - Migration and Upgrade
 

Recently uploaded

Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
Zilliz
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdfAI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
Techgropse Pvt.Ltd.
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
David Brossard
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
Mariano Tinti
 

Recently uploaded (20)

Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdfAI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
 

Beyond Messaging: Enterprise-Scale, Multi-Cloud Intelligent Routing

  • 1. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Beyond Messaging: Enterprise-Scale, Multi-Cloud Intelligent Routing
  • 2. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. About Me 2 Copyright 2018 Adobe Inc. All Rights Reserved. Anand Phatak Software Architect, Adobe Experience Cloud Pipeline aphatak@adobe.com /anandphatak
  • 3. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Experience Platform Pipeline 3 200 Kafka Brokers Multiple Data Centers Multiple Public Clouds 40 Billion Messages per day 25,000 Topics 200,000 Partitions Copyright 2018 Adobe Inc. All Rights Reserved.
  • 4. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Experience Cloud Collection of best-in-class solutions for marketing, analytics, advertising, and commerce 4 Copyright 2018 Adobe Inc. All Rights Reserved. Analytics Content ManagementAudience Profile Campaign Execution Personalization Advertising Commerce
  • 5. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Experience Platform 5 § Enterprise Data Pipeline § Unified Customer Profile § Data Science and Analysis § Open, Extensible Architecture § Standardization and Privacy Copyright 2018 Adobe Inc. All Rights Reserved.
  • 6. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Your Products and Services 6 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 7. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Everything is connected 7 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 8. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 9. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9 Kafka Copyright 2018 Adobe Inc. All Rights Reserved.
  • 10. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Global Footprint 10 KafkaKafka Ireland Kafka Amsterdam Singapore Kafka US West Kafka US East Kafka London Copyright 2018 Adobe Inc. All Rights Reserved.
  • 11. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Experience Platform Pipeline (P6E)
  • 12. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Inside Pipeline 12 Pipeline Console Smart Edge Produce Consume Kafka Kafka MirrorMaker Monitoring/Alerting Copyright 2018 Adobe Inc. All Rights Reserved.
  • 13. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Pipeline Multi-Cloud Transport Networking 13 Smart Edge Produce Consume Smart Edge Produce Consume Kafka Kafka Kafka Kafka MirrorMaker MirrorMaker Data Center 1 Data Center 2 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 14. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Enterprise Scale, Multi-Cloud Intelligent Routing. 14 KafkaKafka Ireland Kafka Amsterdam Singapore Kafka US West Kafka US East Kafka London Copyright 2018 Adobe Inc. All Rights Reserved.
  • 15. © 2018 Adobe Systems Incorporated. All Rights Reserved. 15 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 16. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Pipeline Smart Edge Service § REST APIs to Produce § REST APIs to Consume § SYNC Markers for Commits § Authorization and ACLs § Message Routing § Message Filtering § Client SDKs 16 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 17. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. REST API to Produce § Individual or batched messages § Few hundred bytes to 1 MB § JSON and Google Protocol Buffers (protobuf) § Schema awareness using Adobe’s XDM Registry 17 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 18. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. REST API to Consume § Firehose API – messages are pushed near real time § Consumers and consumer groups § Server side message filtering § Offset management through SYNC packets § SDKs provide easier SYNC handling 18 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 19. © 2018 Adobe Systems Incorporated. All Rights Reserved. 19 Success Ingredients Copyright 2018 Adobe Inc. All Rights Reserved.
  • 20. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Experience Data Model 20 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 21. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Experience Data Model § Adobe Experience Data Model (XDM) § Open specification for digital experiences § https://github.com/adobe/xdm § Interoperability, expressiveness § JSON schema 21 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 22. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 22 Standardized Message Header Copyright 2018 Adobe Inc. All Rights Reserved.
  • 23. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Standardized Message Header § Tenant ID – (customer’s unique identifier) § Service ID – source service § Name/value pair metadata § XDM schema reference 23 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 24. © 2018 Adobe Systems Incorporated. All Rights Reserved. 24 Message Routing Copyright 2018 Adobe Inc. All Rights Reserved.
  • 25. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. MirrorMaker MirrorMaker Routing Scheme 25 SmartEdge topicA Location 2 topicA topicA_loc2 Location 1 SmartEdge Producer Consumer topicA_loc1 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 26. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Routing Rules § Topic level routing – static destination setting § Tenant Based Routing – dynamic destination lookup § Service-based Routing – service specific destination § Pluggable routing logic 26 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 27. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 27 Smart Edge Produce Consume Produce Consume Smart EdgeSmart Edge Produce Consume Kafka MirrorMaker Produce Consume Smart Edge Direct Connectivity Hop Routing Hub Cluster Hop HopHop Mirroring Kafka Kafka Kafka Kafka Kafka Kafka Kafka Kafka MirrorMaker MirrorMaker MirrorMaker MirrorMaker Copyright 2018 Adobe Inc. All Rights Reserved.
  • 28. © 2018 Adobe Systems Incorporated. All Rights Reserved. 28 Self Service Copyright 2018 Adobe Inc. All Rights Reserved.
  • 29. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Pipeline Console § Self-service for topic creation § Routing rules § Usage reports § Monitoring dashboard 29 Copyright 2018 Adobe Inc. All Rights Reserved.
  • 30. © 2018 Adobe Systems Incorporated. All Rights Reserved. 30 Lessons Copyright 2018 Adobe Inc. All Rights Reserved.
  • 31. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Lessons 31 § Standardize deployments across clouds § Inconsistent configurations § Long Deployment cycles § Monitoring everything centrally § API latencies § Consumer lags § Partition fragmentation § Network glitches § Mirroring lags Copyright 2018 Adobe Inc. All Rights Reserved.
  • 32. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Lessons 32 § Throttle misbehaving clients § Usage beyond granted quota § Excessive API calls and disconnects/reconnects § Self-serve § Provide adequate graphs and charts § Monitoring and alerting tools Copyright 2018 Adobe Inc. All Rights Reserved.
  • 33. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Future 33 § Ephemeral fronting produce-only clusters § Persistent consume-only clusters § Self-service on-demand clusters § Make intelligent routing without topic duplication Copyright 2018 Adobe Inc. All Rights Reserved.
  • 34. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 34 Thank You! Copyright 2018 Adobe Inc. All Rights Reserved. Anand Phatak aphatak@adobe.com /anandphatak