Für die Automobilindustrie ist die digitale Transformation wie für jede andere Branche zugleich eine digitale Revolution: Neue Marktspieler, neue Technologien und die in immer größeren Mengen anfallenden Daten schaffen neue Chancen, aber auch neue Herausforderungen – und erfordern neben neuen IT-Architekturen auch völlig neue Denkansätze. 60% der Fortune500-Unternehmen setzen zur Umsetzung ihrer Daten-Streaming-Projekte auf die umfassende verteilte Streaming-Plattform Apache Kafka®, darunter auch die AUDI AG. Erfahren Sie in diesem Webinar: Wie Kafka als Grundlage sowohl für Daten-Pipelines als auch für Anwendungen dient, die Echtzeit-Datenströme konsumieren und verarbeiten. Wie Kafka Connect und Kafka Streams geschäftskritische Anwendungen unterstützt Wie Audi mithilfe von Kafka und Confluent eine Fast Data IoT-Plattform umgesetzt hat, die den Bereich „Connected Car“ revolutioniert Sprecher: David Schmitz, Principal Architect, Audi Electronics Venture GmbH Kai Waehner, Technology Evangelist, Confluent

1. 1
Introduction to Apache Kafka as
Event-Driven Open Source Streaming Platform
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

2. 2
Business Digitalization Trends are Driving the Need to Process
Events at a whole new Scale, Speed and Efficiency
The World has Changed
Mobile Cloud Microservices Internet of Things Machine Learning

3. Events
What is an event?

4. Events

5. 5
Events
A Sale An Invoice A Trade A Customer
Experience

6. 6
Where are they?
Events haven’t had a
proper home in
infrastructure or in code.
They are implicit.
Here!

7. A Streaming Platform is the Underpinning of an
Event-driven Architecture
Ubiquitous connectivity
Globally scalable platform for all
event producers and consumers
Immediate data access
Data accessible to all
consumers in real time
Single system of record
Persistent storage to enable
reprocessing of past events
Continuous queries
Stream processing capabilities
for in-line data transformation
Microservices
DBs
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservices
events
SaaS
data
Customer
experience
s
Streams of real time events
Stream processing appsStream processing apps Stream processing apps

9. The beginning of a new Era
https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
The first use case. This is why Kafka was created!

10. 10
● Global-scale
● Real-time
● Persistent Storage
● Stream Processing
Apache Kafka: The De-facto Standard for Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache Kafka

11. Apache Kafka at Scale at Tech Giants
> 4.5 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data

12. Confluents Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)

13. A Modern, Distributed Platform for
Data Streams.
Messaging + Storage + Processing!

14. Apache Kafka is made up of
distributed, immutable, append-only
commit logs

15. P
Producing to Kafka
Time

16. P
Producing to Kafka
Time
C2 C3C1

17. Partition Leadership and Replication
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

18. Confluent Platform
Operations and Security
Development & Stream Processing
Support,Services,Training&Partners
Apache Kafka
Security plugins | Role-Based Access Control
Control Center | Replicator | Auto Data Balancer | Operator
Connectors
Clients | REST Proxy
MQTT Proxy | Schema Registry
KSQL
Connect Continuous Commit Log Streams
Complete Event
Streaming
Platform
Mission-critical
Reliability
Freedom of
ChoiceDatacenter Public Cloud Confluent Cloud
Self-Managed Software Fully-Managed Service
Confluent Delivers a Mission-Critical Event Streaming Platform

19. KSQL – A Streaming SQL Engine for Apache Kafka

20. 2020
Confluent Control Center (C3)
Monitors all pipelines end-to-end
• Lost Messages?
• Duplicates?
• Latency Issues?
• What is the problem?
• Where is the problem?
• Etc.

21. 2121
Best-of-breed Platforms, Partners and Services for Multi-cloud Streams
Private Cloud
Deploy on bare-metal, VMs,
containers or Kubernetes in your
datacenter with Confluent Platform
and Confluent Operator
Public Cloud
Implement self-managed in the public
cloud or adopt a fully managed service
with Confluent Cloud
Hybrid Cloud
Build a persistent bridge between
datacenter and cloud with
Confluent Replicator
Confluent
Replicator
VM
SELF MANAGED FULLY MANAGED

22. 22
Confluent’s Streaming Maturity Model - where are you?
Value
Maturity (Investment & time)
2
Enterprise
Streaming Pilot /
Early Production
Pub + Sub Store Process
5
Central Nervous
System
1
Developer
Interest
Pre-Streaming
4
Global
Streaming
3
SLA Ready,
Integrated
Streaming
Projects
Platform

23. 23Highly Scalable Microservices with Apache Kafka + Mesos
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
LinkedIn
Questions? Feedback?
Please contact me!

24. Internal
Audi Electronics Venture
David Schmitz Principal Architect Next Cloud Platform
Fast Cars in a streaming World
Audi Electronics Venture

25. 2 Audi Electronics Venture GmbH July 2018
Internal
Audi Electronics Venture GmbH
Subsidiary of AUDI AG
Audi Electronics Venture... INNOVATIONS
& TECHNOLOGIES
DISCOVER SECURE/PROTECT USE
“… Audi Electronics Venture
ensures access to key
technologies in the field of
electronics by development of
new technologies and
realization of collaborations,
joint ventures and investments.“
AEV

26. Internal
Data Driven World
Audi Data Collector
ACDC Platform
Vision
Agenda

27. Internal
Data
Driven
World

28. Internal
Amazon uses instant
personalisation to
increase revenue by
30%
annually.

29. Internal
Netflix analyses
600 Billion
data objects a day to
determine that House
of Cards will be a hit,
before it even airs.

30. Internal
Autonomous vehicles
like ‘Jack’ stream
roughly
4 terabytes
of data a day.

31. Internal
Data is not only the ‘new oil’.
It is the new DNA.

32. Internal
Data engineering is
poised to deliver
the next wave of
“Vorsprung durch
Technik” for Audi.

33. Internal
Audi
Data
Collector

34. Internal
CAMPAIGN
MANAGER
MBB
ADMINISTRATE
AUTHORIZE
ODC
COLLECT
ACDC
ANALYSE
INSIGHT
APPLY
MODULARER BACKEND
BAUKASTEN
ONBOARD DATA
COLLECTOR
AUTOMOTIVE CLOUD
DATA COLLECTOR
Audi Data Collector

35. Internal
010101011
00111111
1010100110
101010100101
10101010110
101010101010
01010101101010
0101010
10101001010
10101010101001
01010010101010
1011011
10101010101001
101001
010101001001
010101010
1101011
11011
1001
Backend
REQUEST
“Where do I find vacant parking lots in my city?“
CONFIGURATION
Vacant parking lots in his/her city
EVALUATION
Displaying vacant parking lots in my
Audi connect service
DATA
Available parking lots in his/her city
Audi Data Collector
Working Example: Realtime event

36. Internal
Life does not happen in “batch mode“

37. Internal
Audi Data Collector
First Use within the brand-new Audi

38. Internal
ACDC
Platform

39. Internal
Requirements of the
ACDC platform
› High compute &
network performance
› Scalablility &
Resilience
› Regional availability
with data sovereignty
› Scriptable
troubleshooting
Fast Data
IoT Platform

40. Internal
› Processing data
streams in real-time
to identify black-ice
and sending this
information via ACDC
to all 3rd party
subscribed queues
› Scoring of relevant
information and
instantly distributing
fleet wide
Real-time
Data Analysis

41. Internal
› Generating
comprehensive,
always-current
information about
the traffic
› Examples: green
waves in city traffic,
changing speed
limits and free
parking spots
Swarm
Intelligence

42. Internal
Reconciling the best
route to preserve
battery power due to
car sensor data,
location of charging
stations and situational
data like weather
combined with
predictive AI.
Predictive AI

43. Internal
Campaign
Manager
API
3rd Party
Service API
Streaming
Analytics
BI AnalyticsCar Authorisation myAudi
3rd
Parties
AUDI
BACK
CHANNELS
TO VEHICLES
+ SERVICES
DATA
INGESTION
ACDC Cloud
Kafka Queue
Kafka Queue
Kafka Queue
WORKER
CLOUD DATA STORAGE
Integration
Message
Transport
Audi Backend

44. Internal
Data Supply Chain: SMACK Stack Technologies

45. Internal
Apache Spark is a fast and general-purpose
cluster computing system. It provides high-
level APIs in Scala and has an optimised
engine that supports general execution
graphs.
Features:
› Data integration and ETL
› Interactive analytics or business
intelligence
› High performance batch computation
› Machine learning and advanced
analytics
› Real-time stream processing
Spark - The engine

46. Internal
Apache Mesos is the open-source distributed
systems kernel that abstracts the entire
datacentre into a single pool of computing
resources, simplifying running distributed
systems at scale.
Features:
› Scale up (Concurrency)
› Scale out (Remoting)
› Production proven at massive scale
› Supports Containers & Big Data
› Extensible, State of the Art
Architecture
Mesos: - The freighter

47. Internal
Akka provides scalable real-time
transaction processing and is an unified
runtime and programming model.
Features:
› Scale up (Concurrency)
› Scale out (Remoting)
› Fault tolerance
Akka - The model

48. Internal
Cassandra has linear scalability and proven
fault-tolerance on commodity hardware or
cloud infrastructure making it the perfect
platform for mission-critical data.
Features:
› Distributed datastore with a built-in
coordinator
› Very fast and especially shining with
writing heavy workflows
› Scaling linearly
› Embracing eventual consistency
› Masterless replication across data
centers
Cassan - The storage

49. Internal
Kafka is needed for building real-time
streaming applications that transform or
react to the streams of data.
Features:
› Publishing and subscribing to
streams of records
› Scoring streams of records in a fault-
tolerant way
› Processing streams of records as they
occur
Kafka: - The message broker

50. Internal
Vision

51. Internal
Vision: Creating a
entirely new
customer experience
My autonomous Audi will enable
the 25th hour of the day:
› Quality time: time for family
› Productive time: for work
› Down time: time for
entertainment

52. Internal
Thank
You!
David Schmitz