(Anand Phatak, Adobe) Kafka Summit SF 2018 You have data centers all over the world. You have set them up close to your customers and services. You have decentralized data handling with a network of data processing systems. The trouble is now how do you make sure your messages reach the appropriate data centers without getting into a complicated mess of dedicated topic-to-topic replication? What if your messaging bus was smart enough to figure out where to deliver each message? With Adobe’s Experience Cloud Pipeline, we have tackled this very issue. Adobe’s Experience Cloud gives access to an integrated set of Adobe products and solutions that help Adobe’s customers to build campaigns, manage to advertise, gain profound insights and deliver personalized experiences all in near real time. To achieve this Adobe solutions must exchange data close to real time. Often data processed by the solutions need to travel across globally distributed data centers hosted on both public and private clouds. Based on Apache Kafka, Adobe’s Experience Cloud Pipeline is a globally distributed, mission-critical messaging bus for asynchronous communication across Adobe solutions that process tens of billion messages each day. The Experience Cloud Pipeline also serves an entry point for the streaming data flowing from external systems into the Adobe Experience Cloud. Multi-data center deployment is commonplace to most large enterprises. Having a smart messaging bus that is capable of routing, filtering and enriching messages in near real time, in proximity to your solutions and customers is of paramount importance. Adobe’s Experience Cloud Pipeline demonstrates a successful large-scale deployment of such use case.

1. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Beyond Messaging: Enterprise-Scale, Multi-Cloud Intelligent
Routing

2. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
About Me
2
Copyright 2018 Adobe Inc. All Rights Reserved.
Anand Phatak
Software Architect, Adobe Experience Cloud Pipeline
aphatak@adobe.com
/anandphatak

3. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Platform Pipeline
3
200
Kafka Brokers
Multiple
Data Centers
Multiple
Public Clouds
40 Billion
Messages per day
25,000
Topics
200,000
Partitions
Copyright 2018 Adobe Inc. All Rights Reserved.

4. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Cloud
Collection of best-in-class solutions for marketing, analytics, advertising, and
commerce
4
Copyright 2018 Adobe Inc. All Rights Reserved.
Analytics Content ManagementAudience Profile
Campaign Execution Personalization Advertising
Commerce

5. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Platform
5
§ Enterprise Data Pipeline
§ Unified Customer Profile
§ Data Science and Analysis
§ Open, Extensible Architecture
§ Standardization and Privacy
Copyright 2018 Adobe Inc. All Rights Reserved.

6. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Your Products and Services
6
Copyright 2018 Adobe Inc. All Rights Reserved.

7. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Everything is connected
7
Copyright 2018 Adobe Inc. All Rights Reserved.

8. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 8
Copyright 2018 Adobe Inc. All Rights Reserved.

9. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9
Kafka
Copyright 2018 Adobe Inc. All Rights Reserved.

10. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Global Footprint
10
KafkaKafka
Ireland
Kafka
Amsterdam Singapore
Kafka
US West
Kafka
US East
Kafka
London
Copyright 2018 Adobe Inc. All Rights Reserved.

11. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Experience Platform Pipeline (P6E)

12. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Inside Pipeline
12
Pipeline Console
Smart Edge
Produce Consume
Kafka
Kafka MirrorMaker
Monitoring/Alerting
Copyright 2018 Adobe Inc. All Rights Reserved.

13. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Pipeline Multi-Cloud Transport Networking
13
Smart Edge
Produce Consume
Smart Edge
Produce Consume
Kafka
Kafka
Kafka
Kafka
MirrorMaker
MirrorMaker
Data Center 1 Data Center 2
Copyright 2018 Adobe Inc. All Rights Reserved.

14. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Enterprise Scale, Multi-Cloud Intelligent Routing.
14
KafkaKafka
Ireland
Kafka
Amsterdam Singapore
Kafka
US West
Kafka
US East
Kafka
London
Copyright 2018 Adobe Inc. All Rights Reserved.

15. © 2018 Adobe Systems Incorporated. All Rights Reserved.
15
Copyright 2018 Adobe Inc. All Rights Reserved.

16. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Pipeline Smart Edge Service
§ REST APIs to Produce
§ REST APIs to Consume
§ SYNC Markers for Commits
§ Authorization and ACLs
§ Message Routing
§ Message Filtering
§ Client SDKs
16
Copyright 2018 Adobe Inc. All Rights Reserved.

17. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
REST API to Produce
§ Individual or batched messages
§ Few hundred bytes to 1 MB
§ JSON and Google Protocol Buffers (protobuf)
§ Schema awareness using Adobe’s XDM
Registry
17
Copyright 2018 Adobe Inc. All Rights Reserved.

18. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
REST API to Consume
§ Firehose API – messages are pushed near real
time
§ Consumers and consumer groups
§ Server side message filtering
§ Offset management through SYNC packets
§ SDKs provide easier SYNC handling
18
Copyright 2018 Adobe Inc. All Rights Reserved.

19. © 2018 Adobe Systems Incorporated. All Rights Reserved.
19
Success Ingredients
Copyright 2018 Adobe Inc. All Rights Reserved.

20. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Experience Data Model
20
Copyright 2018 Adobe Inc. All Rights Reserved.

21. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Experience Data Model
§ Adobe Experience Data Model (XDM)
§ Open specification for digital experiences
§ https://github.com/adobe/xdm
§ Interoperability, expressiveness
§ JSON schema
21
Copyright 2018 Adobe Inc. All Rights Reserved.

22. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 22
Standardized Message Header
Copyright 2018 Adobe Inc. All Rights Reserved.

23. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Standardized Message Header
§ Tenant ID – (customer’s unique
identifier)
§ Service ID – source service
§ Name/value pair metadata
§ XDM schema reference
23
Copyright 2018 Adobe Inc. All Rights Reserved.

24. © 2018 Adobe Systems Incorporated. All Rights Reserved.
24
Message Routing
Copyright 2018 Adobe Inc. All Rights Reserved.

25. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
MirrorMaker
MirrorMaker
Routing Scheme
25
SmartEdge
topicA
Location 2
topicA
topicA_loc2
Location 1
SmartEdge
Producer
Consumer
topicA_loc1
Copyright 2018 Adobe Inc. All Rights Reserved.

26. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Routing Rules
§ Topic level routing – static destination setting
§ Tenant Based Routing – dynamic destination lookup
§ Service-based Routing – service specific destination
§ Pluggable routing logic
26
Copyright 2018 Adobe Inc. All Rights Reserved.

27. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 27
Smart Edge
Produce Consume
Produce Consume
Smart EdgeSmart Edge
Produce Consume
Kafka
MirrorMaker
Produce Consume
Smart Edge
Direct Connectivity
Hop
Routing Hub Cluster
Hop
HopHop
Mirroring
Kafka
Kafka
Kafka
Kafka
Kafka
Kafka
Kafka
Kafka
MirrorMaker
MirrorMaker MirrorMaker
MirrorMaker
Copyright 2018 Adobe Inc. All Rights Reserved.

28. © 2018 Adobe Systems Incorporated. All Rights Reserved.
28
Self Service
Copyright 2018 Adobe Inc. All Rights Reserved.

29. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Pipeline Console
§ Self-service for topic creation
§ Routing rules
§ Usage reports
§ Monitoring dashboard
29
Copyright 2018 Adobe Inc. All Rights Reserved.

30. © 2018 Adobe Systems Incorporated. All Rights Reserved.
30
Lessons
Copyright 2018 Adobe Inc. All Rights Reserved.

31. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Lessons
31
§ Standardize deployments across clouds
§ Inconsistent configurations
§ Long Deployment cycles
§ Monitoring everything centrally
§ API latencies
§ Consumer lags
§ Partition fragmentation
§ Network glitches
§ Mirroring lags
Copyright 2018 Adobe Inc. All Rights Reserved.

32. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Lessons
32
§ Throttle misbehaving clients
§ Usage beyond granted quota
§ Excessive API calls and disconnects/reconnects
§ Self-serve
§ Provide adequate graphs and charts
§ Monitoring and alerting tools
Copyright 2018 Adobe Inc. All Rights Reserved.

33. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Future
33
§ Ephemeral fronting produce-only clusters
§ Persistent consume-only clusters
§ Self-service on-demand clusters
§ Make intelligent routing without topic duplication
Copyright 2018 Adobe Inc. All Rights Reserved.

34. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 34
Thank You!
Copyright 2018 Adobe Inc. All Rights Reserved.
Anand Phatak
aphatak@adobe.com
/anandphatak