More Related Content Similar to Ted Dunning-Faster and Furiouser- Flink Drift (20) More from Flink Forward (20) Ted Dunning-Faster and Furiouser- Flink Drift2. © 2014 MapR Technologies 2
Me, Us
• Ted Dunning, MapR Chief Application Architect, Apache Member
– Committer PMC member Zookeeper, Drill, others
– Mentor for Flink, Beam (nee Dataflow), Drill, Storm, Zeppelin
– VP Incubator
– Bought the beer at the first HUG
• MapR
– Produces first converged platform for big and fast data
– Includes data platform (files, streams, tables) + open source
– Adds major technology for performance, HA, industry standard API’s
• Contact
@ted_dunning, ted.dunning@gmail.com, tdunning@mapr.com
3. © 2014 MapR Technologies 3
New book on Apache Flink
Download free pdf
courtesy of MapR Technologies
mapr.com/flink-book
4. © 2014 MapR Technologies 4
Agenda
• Why streaming first architecture
• What does fast mean?
• How do I make something fast?
• Minor pause for reality check
• First steps … heavy bottlenecks
• Real results
• Deeper insights
5. © 2014 MapR Technologies 5
Is this really a
revolutionary moment?
6. © 2014 MapR Technologies 6
Scenario:
Profile Database
7. © 2014 MapR Technologies 7
The task
?
POS 1
location, t, card #
yes/no?
POS 2
location, t, card #
yes/no?
8. © 2014 MapR Technologies 8
Traditional Solution
POS
1..n
Fraud
detector
Last card
use
9. © 2014 MapR Technologies 9
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector
10. © 2014 MapR Technologies 10
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector
11. © 2014 MapR Technologies 11
How to Get Service Isolation
POS
1..n
Fraud
detector
Last card
use
Updater
card activity
12. © 2014 MapR Technologies 12
New Uses of Data
POS
1..n
Fraud
detector
Last card
use
Updater
Card
location
history
Other
card activity
13. © 2014 MapR Technologies 13
Scaling Through Isolation
POS
1..n
Last card
use
Updater
POS
1..n
Last card
use
Updater
card activity
Fraud
detector
Fraud
detector
14. © 2014 MapR Technologies 14
For this to work (socially),
streaming has to be faster
than almost any requirement
15. © 2014 MapR Technologies 15
So how do we make something
go really fast?
16. © 2014 MapR Technologies 16
Make some
data
Process itmove it
17. © 2014 MapR Technologies 17
Make some
data
Process itmove it
World
domination
move it
18. © 2014 MapR Technologies 18
Well, perhaps not quite so
simple?
19. © 2014 MapR Technologies 19
Interactive recommendation
query
db
Off-line analysis
Real-time event
source
Recent
history
Item
linkage
Search Recommendations
Cooccurrence
analysis
Long-
term
history
queue
queue
Recommendations
20. © 2014 MapR Technologies 20
mySQL
mySQL
files
Web-site
Auth
service
Upload
service
Image
extractor
Transcoder
User
profiles
Search
User action
logging
Recommendation
analysis
mySQL
mySQL
mySQL
Oracle
Solr
Elastic
User Generated Content
21. © 2014 MapR Technologies 21
Yahoo Streaming Benchmark
Ad server Filter
Group by
campaign
impressions
Campaign
info
Count
impressions Results
Project
Augment
Window by event time
22. © 2014 MapR Technologies 22
Ad server Filter
Group by
campaign
impressions
Campaign
info
Count
impressions Results
Project
Augment
Window by event time
Client lock Partitions Threads/machineShuffle
23. © 2014 MapR Technologies 23
Ad server Filter
Group by
campaign
impressions
Campaign
info
Count
impressions Results
Project
Augment
Window by event time
Client lock Partitions Threads/machineShuffle
Threads/machineShuffle
24. © 2014 MapR Technologies 24
Ad server Filter
Group by
campaign
impressions
Campaign
info
Count
impressions Results
Project
Augment
Window by event time
Ad server impressions Filter Project
Augment
Group by
campaign
Count
impressions
Client lock Partitions Threads/machineShuffle
Threads/machineShuffle
26. © 2014 MapR Technologies 26
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Over decades of progress,
Unix-based systems have set the
standard for compatibility and
functionality
27. © 2014 MapR Technologies 27
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Hadoop achieves much higher
scalability by trading away
essentially all of this compatibility
Evolution of Data Storage
28. © 2014 MapR Technologies 28
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
MapR enhanced Apache Hadoop by
restoring the compatibility while
increasing scalability and performance
Functionality
Compatibility
Scalability
POSIX
29. © 2014 MapR Technologies 29
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Evolution of Data Storage
Adding converged tables and streams
enhances the functionality of the base
file system
30. © 2014 MapR Technologies 30
http://bit.ly/fastest-big-data
31. © 2014 MapR Technologies 31
Key Ideas
• Convergence of files, tables, streams into single platform
– All forms of persistence share common implementation base
• Very high abstraction from hardware … no need to provision
clusters for tables and files
– Common disaster recovery, security, availability models for files,
directories, tables and streams
• Very high performance levels
32. © 2014 MapR Technologies 32
Key Issues
• MapR itself is heavily threaded internally (as many as 50k
threads/core)
• MapR client can have multiple internal threads
• Ordering boundaries require serialization, locks or memory
contention
– At client level and also within single stream/topic/partition
• Replication, splitting, data location completely automated by
default, explicit control available
• MapR Streams and Flink are in same cluster, but some shuffles
still required
33. © 2014 MapR Technologies 33
Initial Configuration
• 10 nodes in cluster
• 1 Flink task manager / node
• 72 partitions in impressions stream
• Each task manager spawns 72
generator threads
Ad server impressions
Ad server impressions
10x72 threads
72 partitions
• At full speed, partition insert points wander around cluster to
avoid hot-spotting
• MapR client connection shared by all threads in task
manager. Having more client connections could help
34. © 2014 MapR Technologies 34
Tuning #1
• Large number of threads and single client connection per node
caused massive contention at serialization point inside client
• Switched to 3 Flink task managers per node
• 2 task managers each run 1 producer thread
– More data pushed by 1 thread than previously sent by 72
35. © 2014 MapR Technologies 35
Tuning #2
• Effective cluster-wide parallelism limited by 72 partitions in
stream
• Increasing to 300 partitions substantially improved performance
36. © 2014 MapR Technologies 36
The consumer
• Initial tuning had 72 consumer threads per
node
• Final tuning used single consumer thread
per Flink task manager
Filter
Campaign
info
Project
Augment
Filter Project
Augment
37. © 2014 MapR Technologies 37
The Shuffle / Group-by
• Shuffles were also run by the
single consumer task
manager
• Even with shuffle, consumer
processes balanced
producer processes
Group by
campaign
Count
impressions Results
Window by event time
Group by
campaign
Count
impressions
38. © 2014 MapR Technologies 38
Tuning #3
• In separate experiments, number of campaigns was increased to
1e6 from original 100
• This caused bottle neck to shift massively to data export step
• Serving results directly from Flink memory avoids this step
39. © 2014 MapR Technologies 39
Final Comparisons
Flink on MapR
no tuning
Transactions / second
(millions)
0 5 10 15
Flink on MapR
tuned
Final result for tuning was
250% improvement
No serious optimization was
required, however
40. © 2014 MapR Technologies 40
The Moral
• Default of 10 partitions per topic is fine for large-scale multi-
tenancy, but special purpose applications may need tuning to
higher levels (we ended up with 30 partitions per node)
• Asynchronous client gives effective threading with small number
of producer threads, large number of producer threads was
counter-productive
• Net speedup of 250% with tuning, so far
• Gut feel is that there is ~4x more performance still to come
41. © 2014 MapR Technologies 41
Me, Us
• Ted Dunning, MapR Chief Application Architect, Apache Member
– Committer PMC member Zookeeper, Drill, others
– Mentor for Flink, Beam (nee Dataflow), Drill, Storm, Zeppelin
– VP Incubator
– Bought the beer at the first HUG
• MapR (www.mapr.com)
– Produces first converged platform for big and fast data
– Includes data platform (files, streams, tables) + open source
– Adds major technology for performance, HA, industry standard API’s
• Contact
@ted_dunning, ted.dunning@gmail.com, tdunning@mapr.com
42. © 2014 MapR Technologies 42
New book on Apache Flink
Download free pdf
courtesy of MapR Technologies
mapr.com/flink-book
43. © 2014 MapR Technologies 43
Streaming Architecture
by Ted Dunning and Ellen Friedman © 2016 (published by O’Reilly)
Free signed hard copies at
MapR booth at Flink
Forward
http://bit.ly/mapr-ebook-streams
44. © 2014 MapR Technologies 44
Short Books by Ted Dunning & Ellen Friedman
• Published by O’Reilly in 2014 - 2016
• For sale from Amazon or O’Reilly
• Free e-books currently available courtesy of MapR
Download pdfs: mapr.com/ebooks-pdf
46. © 2014 MapR Technologies 46
Q&A
@mapr maprtech
tdunning@maprtech.com
Engage with us!
MapR
maprtech
mapr-technologies