Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ted Dunning-Faster and Furiouser- Flink Drift

319 views

Published on

http://flink-forward.org/kb_sessions/faster-and-furiouser-flink-drift/

Not long ago, we had the opportunity to test Apache Flink to see just how fast it would go on a moderately realistic task with fast hardware and with a good streaming transport layer underneath. Our goal was not so much careful comparison with other software, but flat-out speed, Flink against Flink. In the process, we learned a lot about what it takes to go fast. Some of the lessons were ones that we had “learned” a number of times before: – the bottleneck isn’t where you thought it was – copying data is expensive – context switches are expensive – measure twice, cut once But there were some real surprises along the way. The really important knobs weren’t quite what people say you should turn. One of the biggest surprises was the degree to which high performance libraries have threading built into them which makes the actual concurrrency much higher than the apparent concurrency. The result was that at least one cluster parameter needed to be adjusted by 30x to get real

Published in: Data & Analytics
  • Dear Sir, I'm unable to download the free book kindly let me know an alternate method to download the book or kindly mail me the PDF version of the book to aabidhzubair@gmail.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Ted Dunning-Faster and Furiouser- Flink Drift

  1. 1. © 2014 MapR Technologies 1© 2014 MapR Technologies
  2. 2. © 2014 MapR Technologies 2 Me, Us • Ted Dunning, MapR Chief Application Architect, Apache Member – Committer PMC member Zookeeper, Drill, others – Mentor for Flink, Beam (nee Dataflow), Drill, Storm, Zeppelin – VP Incubator – Bought the beer at the first HUG • MapR – Produces first converged platform for big and fast data – Includes data platform (files, streams, tables) + open source – Adds major technology for performance, HA, industry standard API’s • Contact @ted_dunning, ted.dunning@gmail.com, tdunning@mapr.com
  3. 3. © 2014 MapR Technologies 3 New book on Apache Flink Download free pdf courtesy of MapR Technologies mapr.com/flink-book
  4. 4. © 2014 MapR Technologies 4 Agenda • Why streaming first architecture • What does fast mean? • How do I make something fast? • Minor pause for reality check • First steps … heavy bottlenecks • Real results • Deeper insights
  5. 5. © 2014 MapR Technologies 5 Is this really a revolutionary moment?
  6. 6. © 2014 MapR Technologies 6 Scenario: Profile Database
  7. 7. © 2014 MapR Technologies 7 The task ? POS 1 location, t, card # yes/no? POS 2 location, t, card # yes/no?
  8. 8. © 2014 MapR Technologies 8 Traditional Solution POS 1..n Fraud detector Last card use
  9. 9. © 2014 MapR Technologies 9 What Happens Next? POS 1..n Fraud detector Last card use POS 1..n Fraud detector POS 1..n Fraud detector
  10. 10. © 2014 MapR Technologies 10 What Happens Next? POS 1..n Fraud detector Last card use POS 1..n Fraud detector POS 1..n Fraud detector
  11. 11. © 2014 MapR Technologies 11 How to Get Service Isolation POS 1..n Fraud detector Last card use Updater card activity
  12. 12. © 2014 MapR Technologies 12 New Uses of Data POS 1..n Fraud detector Last card use Updater Card location history Other card activity
  13. 13. © 2014 MapR Technologies 13 Scaling Through Isolation POS 1..n Last card use Updater POS 1..n Last card use Updater card activity Fraud detector Fraud detector
  14. 14. © 2014 MapR Technologies 14 For this to work (socially), streaming has to be faster than almost any requirement
  15. 15. © 2014 MapR Technologies 15 So how do we make something go really fast?
  16. 16. © 2014 MapR Technologies 16 Make some data Process itmove it
  17. 17. © 2014 MapR Technologies 17 Make some data Process itmove it World domination move it
  18. 18. © 2014 MapR Technologies 18 Well, perhaps not quite so simple?
  19. 19. © 2014 MapR Technologies 19 Interactive recommendation query db Off-line analysis Real-time event source Recent history Item linkage Search Recommendations Cooccurrence analysis Long- term history queue queue Recommendations
  20. 20. © 2014 MapR Technologies 20 mySQL mySQL files Web-site Auth service Upload service Image extractor Transcoder User profiles Search User action logging Recommendation analysis mySQL mySQL mySQL Oracle Solr Elastic User Generated Content
  21. 21. © 2014 MapR Technologies 21 Yahoo Streaming Benchmark Ad server Filter Group by campaign impressions Campaign info Count impressions Results Project Augment Window by event time
  22. 22. © 2014 MapR Technologies 22 Ad server Filter Group by campaign impressions Campaign info Count impressions Results Project Augment Window by event time Client lock Partitions Threads/machineShuffle
  23. 23. © 2014 MapR Technologies 23 Ad server Filter Group by campaign impressions Campaign info Count impressions Results Project Augment Window by event time Client lock Partitions Threads/machineShuffle Threads/machineShuffle
  24. 24. © 2014 MapR Technologies 24 Ad server Filter Group by campaign impressions Campaign info Count impressions Results Project Augment Window by event time Ad server impressions Filter Project Augment Group by campaign Count impressions Client lock Partitions Threads/machineShuffle Threads/machineShuffle
  25. 25. © 2014 MapR Technologies 25 What we do at MapR
  26. 26. © 2014 MapR Technologies 26 Evolution of Data Storage Functionality Compatibility Scalability Linux POSIX Over decades of progress, Unix-based systems have set the standard for compatibility and functionality
  27. 27. © 2014 MapR Technologies 27 Functionality Compatibility Scalability Linux POSIX Hadoop Hadoop achieves much higher scalability by trading away essentially all of this compatibility Evolution of Data Storage
  28. 28. © 2014 MapR Technologies 28 Evolution of Data Storage Functionality Compatibility Scalability Linux POSIX Hadoop MapR enhanced Apache Hadoop by restoring the compatibility while increasing scalability and performance Functionality Compatibility Scalability POSIX
  29. 29. © 2014 MapR Technologies 29 Functionality Compatibility Scalability Linux POSIX Hadoop Evolution of Data Storage Adding converged tables and streams enhances the functionality of the base file system
  30. 30. © 2014 MapR Technologies 30 http://bit.ly/fastest-big-data
  31. 31. © 2014 MapR Technologies 31 Key Ideas • Convergence of files, tables, streams into single platform – All forms of persistence share common implementation base • Very high abstraction from hardware … no need to provision clusters for tables and files – Common disaster recovery, security, availability models for files, directories, tables and streams • Very high performance levels
  32. 32. © 2014 MapR Technologies 32 Key Issues • MapR itself is heavily threaded internally (as many as 50k threads/core) • MapR client can have multiple internal threads • Ordering boundaries require serialization, locks or memory contention – At client level and also within single stream/topic/partition • Replication, splitting, data location completely automated by default, explicit control available • MapR Streams and Flink are in same cluster, but some shuffles still required
  33. 33. © 2014 MapR Technologies 33 Initial Configuration • 10 nodes in cluster • 1 Flink task manager / node • 72 partitions in impressions stream • Each task manager spawns 72 generator threads Ad server impressions Ad server impressions 10x72 threads 72 partitions • At full speed, partition insert points wander around cluster to avoid hot-spotting • MapR client connection shared by all threads in task manager. Having more client connections could help
  34. 34. © 2014 MapR Technologies 34 Tuning #1 • Large number of threads and single client connection per node caused massive contention at serialization point inside client • Switched to 3 Flink task managers per node • 2 task managers each run 1 producer thread – More data pushed by 1 thread than previously sent by 72
  35. 35. © 2014 MapR Technologies 35 Tuning #2 • Effective cluster-wide parallelism limited by 72 partitions in stream • Increasing to 300 partitions substantially improved performance
  36. 36. © 2014 MapR Technologies 36 The consumer • Initial tuning had 72 consumer threads per node • Final tuning used single consumer thread per Flink task manager Filter Campaign info Project Augment Filter Project Augment
  37. 37. © 2014 MapR Technologies 37 The Shuffle / Group-by • Shuffles were also run by the single consumer task manager • Even with shuffle, consumer processes balanced producer processes Group by campaign Count impressions Results Window by event time Group by campaign Count impressions
  38. 38. © 2014 MapR Technologies 38 Tuning #3 • In separate experiments, number of campaigns was increased to 1e6 from original 100 • This caused bottle neck to shift massively to data export step • Serving results directly from Flink memory avoids this step
  39. 39. © 2014 MapR Technologies 39 Final Comparisons Flink on MapR no tuning Transactions / second (millions) 0 5 10 15 Flink on MapR tuned Final result for tuning was 250% improvement No serious optimization was required, however
  40. 40. © 2014 MapR Technologies 40 The Moral • Default of 10 partitions per topic is fine for large-scale multi- tenancy, but special purpose applications may need tuning to higher levels (we ended up with 30 partitions per node) • Asynchronous client gives effective threading with small number of producer threads, large number of producer threads was counter-productive • Net speedup of 250% with tuning, so far • Gut feel is that there is ~4x more performance still to come
  41. 41. © 2014 MapR Technologies 41 Me, Us • Ted Dunning, MapR Chief Application Architect, Apache Member – Committer PMC member Zookeeper, Drill, others – Mentor for Flink, Beam (nee Dataflow), Drill, Storm, Zeppelin – VP Incubator – Bought the beer at the first HUG • MapR (www.mapr.com) – Produces first converged platform for big and fast data – Includes data platform (files, streams, tables) + open source – Adds major technology for performance, HA, industry standard API’s • Contact @ted_dunning, ted.dunning@gmail.com, tdunning@mapr.com
  42. 42. © 2014 MapR Technologies 42 New book on Apache Flink Download free pdf courtesy of MapR Technologies mapr.com/flink-book
  43. 43. © 2014 MapR Technologies 43 Streaming Architecture by Ted Dunning and Ellen Friedman © 2016 (published by O’Reilly) Free signed hard copies at MapR booth at Flink Forward http://bit.ly/mapr-ebook-streams
  44. 44. © 2014 MapR Technologies 44 Short Books by Ted Dunning & Ellen Friedman • Published by O’Reilly in 2014 - 2016 • For sale from Amazon or O’Reilly • Free e-books currently available courtesy of MapR Download pdfs: mapr.com/ebooks-pdf
  45. 45. © 2014 MapR Technologies 45 Thank You!
  46. 46. © 2014 MapR Technologies 46 Q&A @mapr maprtech tdunning@maprtech.com Engage with us! MapR maprtech mapr-technologies

×