This talk provides an introduction and key ideas for how to design streaming architecture, how streaming can support microservices, and what capabilities are needed in message transport (event streams) such as Apache Kafka and MapR Streams (that uses Kafka API).

1. © 2017 Ellen Friedman 11
Streaming Architecture
Ellen Friedman, PhD
14 March 2017
San Jose #Strataconf

2. © 2017 Ellen Friedman 22
Contact Information
Ellen Friedman, PhD
O’Reilly Author & Solutions Consultant for MapR Technologies
Committer Apache Drill & Apache Mahout projects
Email ellenf@apache.org efriedman@maprtech.com
Twitter @Ellen_Friedman #StrataConf

3. © 2017 Ellen Friedman 33
Why stream?

4. © 2017 Ellen Friedman 44
Life doesn’t happen in batches…

5. © 2017 Ellen Friedman 55
Time Value of Data: WAZE Navigation
“Outsmarting traffic, together”
-WAZE website https://www.waze.com/
• Real-time streaming traffic &
road information shared by 65
million drivers/ month
• Intended to save fuel and
time during commute
• Time-value of data matters

6. © 2017 Ellen Friedman 66
Image © E Friedman
Self-driving cars:
Huge volume of
sensor data
IoT Data: Connected Vehicles
Self-driving car from Waymo, seen in Mountain View CA
Image © Ellen Friedman 2017

7. © 2017 Ellen Friedman 77
Images © Friedman & Dunning from O’Reilly book A New Look at Anomaly Detection, used with permis
Time Series Data & the IoT
Sensors in planes
• send data to the
ERD (black box)
• report back to
manufacturers
of “smart parts”
.

8. © 2017 Ellen Friedman 88
Factory 2050 / AMRC at University of Sheffield
• Respond quickly to new requirements
• IoT enabled “smart tools” on
manufacture floor
• Reconfigurable factory
• Remove barriers to communication:
engineering, design, analysis,
manufacture in one space
Image credit Bond Bryan Architecture, used with permission

9. © 2017 Ellen Friedman 99
Factory 2050 / AMRC at University of Sheffield
• Respond quickly to new requirements
• IoT enabled “smart tools” on
manufacture floor
• Reconfigurable factory
• Remove barriers to communication:
engineering, design, analysis,
manufacture in one space
Image credit Bond Bryan Architecture, used with permission

10. © 2017 Ellen Friedman 1010
Factory 2050 / AMRC at University of Sheffield
• Respond quickly to new requirements
• IoT enabled “smart tools” on
manufacture floor
• Reconfigurable factory
• Remove barriers to communication:
engineering, design, analysis,
manufacture in one space
Image credit Bond Bryan Architecture, used with permission

11. © 2017 Ellen Friedman 1111
Predictive Maintenance
Streaming sensor data + long term maintenance histories !
• Machine learning model detects anomalous pattern
• Signals need for maintenance before damage occurs
Image courtesy Mtell
in Real World Hadoop by Dunning & Friedman © 2015

12. © 2017 Ellen Friedman 1212
Streaming data has value beyond
real-time insights

13. © 2017 Ellen Friedman 1313
Stream-1st Architecture
Real-time
analytics
EMR
Patient Facilities
management
Insurance
audit
A
B
Medical tests
C
Medical test
results
People often start with event
data streamed to a real-time
application (A)

14. © 2017 Ellen Friedman 1414
At the Heart: Message Transport
Real-time
analytics
EMR
Patient Facilities
management
Insurance
audit
A
B
Medical tests
C
Medical test
results
With the right messaging
tool at the heart of
stream-1st architecture you
support other classes of use
cases (B & C)

15. © 2017 Ellen Friedman 1515
Some context:
What happened to Hadoop?

16. © 2017 Ellen Friedman 1616
Hadoop is just one of many workloads on
a modern big data platform

17. © 2017 Ellen Friedman 1717
Mobile Phone was a BIG Innovation
You could talk to people from
anywhere (almost)…
Image © Ellen Friedman 2017

18. © 2017 Ellen Friedman 1818
Mobile Phone Now
Voice is just one – and not the major –
reason you use a smart phone
We’ve come a long way…
Image © Ellen Friedman 2017

19. © 2017 Ellen Friedman 1919
MapR Converged Data Platform
Open Source Engines & Tools Commercial Engines & Applications
Utility-Grade Platform Services
DataProcessing
Enterprise Storage
MapR-FS MapR-DB MapR Streams
Database Event Streaming
Global
Namespace
High Availability Data Protection Self-healing Unified Security Real-time Multi-tenancy
Search &
Others
Cloud &
Managed
Services
Custom
Apps
UnifiedManagementand
Monitoring

20. © 2017 Ellen Friedman 2020
Example
Files
Table
Streams
Directories
Cluster
Volume mount point

21. © 2017 Ellen Friedman 2121
Key capabilities
Message Transport Technology:
Apache Kafka & MapR Streams
• Highly scalable
• High throughput, low latency
• Multiple producers &
consumers: decoupled
• Durable messages
• Geo-distributed replication
preserves offsets (unique to
MapR Streams)
Consumer
group
Messages
Producer
Consumer
group
Consumer
group
Producer

22. © 2017 Ellen Friedman 2222
Stream
transport
supports
micro
services

23. © 2017 Ellen Friedman 2323
Stream-first Architecture: Basis for Micro-Services
Stream instead of database as the shared “truth”
POS
1..n
Fraud
detector
Last card
use
Updater
Card
analytics
Other
card activity

24. © 2017 Ellen Friedman 2424
MapR Streams: Data Topics & Partitions
• Data is assigned to topics (as it is in Kafka)
• Topic can be partitioned for load balancing/ performance
• Topic partition is distributed across the MapR cluster (not
restricted to one node as in Kafka)
– Makes long-term auditable history practical
Producer
2
Producer
1 Topic 1
Consumer 2
Consumer 1
Consumer 3
Consumer group

25. © 2017 Ellen Friedman 2525
MapR Stream as Long-Term History
• Set time-to-live to weeks, months, years
– Feasible because with MapR Streams, topic partition is
distributed across MapR cluster (not limited to one node as
generally true with Kafka cluster)
• Event-by-event history: who knew what when?
• Useful in predictive maintenance
• Could serve as a transaction record

26. © 2017 Ellen Friedman 2626
Unique to MapR: Manage Topics at Stream Level
• Many more topics on MapR cluster
• Topics are grouped together in Stream (different from Kafka)
• Policies are set at the Stream level such as time-to-live, ACEs (controlled
access at this level is different than Kafka)
• Geo-distributed stream replication (different from Kafka)
Stream
Topic 1
Topic 3
Topic 2

27. © 2017 Ellen Friedman 2727
With MapR, Geo-Distributed Data Appears Local
stream
Data
source
Consumer

28. © 2017 Ellen Friedman 2828
With MapR, Geo-Distributed Data Appears Local
stream
stream
Data
source
Consumer

29. © 2017 Ellen Friedman 2929
MapR Streams: Replication Across Data Centers
• Replication across data centers with
preserved offsets (unlike Kafka)
• Opens new use cases
Example: Shared inventory in
ad-tech industry
Inventory
model
Global
analytics
Database
Local
state
Inventory
model
Local
state
Data center 1 Data center 2
Central data center

30. © 2017 Ellen Friedman 3030
New: MapR Edge – What Is It? What Does It Do?
• Small footprint cluster for remote processing
– Analysis, filtering, aggregation at the edge
– Decision making at the edge
– Unified end-to-end security policy
– Reliable replication to cloud and data center, even with occasional
connections
• MapR Edge drives intelligent processing and analytics close to
the source for greater speed and impact

31. © 2017 Ellen Friedman 3131
Why MapR Edge is Useful
Data
source
Data
source
Data
source
Report
Data
source

32. © 2017 Ellen Friedman 3232
Who needs MapR Edge?
• Connected car industry
• Telecommunications industry
• Hospitals and medical testing facilities
• Anyone who benefits from global learning but needs local action
© Ellen Friedman 2017
© Ellen Friedman 2017
© XXX
©WesAbrams with permission

33. © 2017 Ellen Friedman 3333
Example Use Case: Telecommunications
Callers
Towers
cdr data

34. © 2017 Ellen Friedman 3434
Streaming in Telecom
• Data collection & handling happens at different levels
– tower, local data center, central data center)
• Batch: Can take 30 minutes per level
• Streaming: Latency drops to seconds or sub-seconds per level
• Ability to respond as events occur
• MapR Streams enables stream replication with offsets across
data centers

35. © 2017 Ellen Friedman 3535
Telecom: Real-time Antenna Tuning
• Sudden localized change in cell usage can occur (such as a
sporting event with large audience)
• Surge in usage could overwhelm a tower
• Ability to tune antenna of 2nd tower to take part of the surge
improves customer experience

36. © 2017 Ellen Friedman 3636
Act
locally,
learn
globally

37. © 2017 Ellen Friedman 3737
Geo-Distribution of Big Data and Analytics
Download free pdf courtesy of MapR:
http://bit.ly/mapr-geo-distribution-ebook-pdf
New O’Reilly data report by Ted Dunning & Ellen Friedman © March 2017

38. © 2017 Ellen Friedman 3838
Apache Flink Stream Processing
Ellen Friedman
& Kostas Tzoumas
Introduction
toApacheFlink
Stream Processing for
Real Time and Beyond
New ebook by
Ellen Friedman and
Kostas Tzoumas
In this book you’ll learn:
· What Apache Flink can do
· How it maintains consistency and provides flexibility
· How people are using it, including in production
· Best practices for streaming architectures
Download your copy:
mapr.com/flink-book
O’Reilly book by Ellen Friedman &
Kostas Tzoumas © Sept 2016
Flink talk at Strata by Jamie Grier
(dataArtisans) Thur 1:50pm
Visit Kostas at dataArtisans booth or
see me at MapR booth

39. © 2017 Ellen Friedman 3939
COMPLIMENTARY BOOKS & AUTHOR SIGNING

40. © 2017 Ellen Friedman 4040
Please support women in tech – help build
girls’ dreams of what they can accomplish
© Ellen Friedman 2015#womenintech #datawomen

41. © 2017 Ellen Friedman 4141
Thank you !

42. © 2017 Ellen Friedman 4242
Contact Information
Ellen Friedman, PhD
O’Reilly Author & Solutions Consultant for MapR Technologies
Committer Apache Drill & Apache Mahout projects
Email ellenf@apache.org efriedman@maprtech.com
Twitter @Ellen_Friedman #StrataConf