Big Data, Fast Data - MapReduce in Hazelcast

1,833 views

Published on

Quick introduction into distributed computing based on map reduce in Hazelcast

Published in: Engineering, Technology
2 Comments
6 Likes
Statistics
Notes
No Downloads
Views
Total views
1,833
On SlideShare
0
From Embeds
0
Number of Embeds
26
Actions
Shares
0
Downloads
52
Comments
2
Likes
6
Embeds 0
No embeds

No notes for slide

Big Data, Fast Data - MapReduce in Hazelcast

  1. 1. BIG DATA - FAST DATA USING MAPREDUCE IN HAZELCAST Source:http://www.newscientist.com/gallery/dn17805-computer-museums-of-the-world/11 www.hazelcast.com
  2. 2. WHO AM I Christoph Engelbert(@noctarius2k) 8+ years of JavaWeirdoness Performance, GC, traffic topics Apache DirectMemoryPMC Previous companies incl. Ubisoftand HRS CastMapRMapReduce for Hazelcast3 www.hazelcast.com
  3. 3. TOPICS Hazelcast Distributed Computing Map &Reduce Demonstration Questions www.hazelcast.com
  4. 4. HAZELCAST A SHORT SPACE TRIP www.hazelcast.com
  5. 5. WHAT IS HAZELCAST? In-MemoryData-Grid DataPartioning(Sharding) JavaCollections Implementation Distributed ComputingPlatform www.hazelcast.com
  6. 6. WHY HAZELCAST? www.hazelcast.com
  7. 7. WHY IN-MEMORY COMPUTING? www.hazelcast.com
  8. 8. TREND OF PRICES DataSource:http://www.jcmit.com/memoryprice.htm www.hazelcast.com
  9. 9. SPEED DIFFERENCE DataSource:http://i.imgur.com/ykOjTVw.png www.hazelcast.com
  10. 10. DISTRIBUTED COMPUTING OR MULTICORE CPU ON STEROIDS www.hazelcast.com
  11. 11. THE IDEA OF DISTRIBUTED COMPUTING Source:https://www.flickr.com/photos/stefan_ledwina/1853508040 www.hazelcast.com
  12. 12. THE BEGINNING Source:http://en.wikipedia.org/wiki/File:KL_Advanced_Micro_Devices_AM9080.jpg www.hazelcast.com
  13. 13. MULTICORE IS NOT NEW Source:http://en.wikipedia.org/wiki/File:80386with387.JPG www.hazelcast.com
  14. 14. CLUSTER IT Source:http://rarecpus.com/images2/cpu_cluster.jpg www.hazelcast.com
  15. 15. SUPER COMPUTER Source:http://www.dkrz.de/about/aufgaben/dkrz-geschichte/rechnerhistorie-1 www.hazelcast.com
  16. 16. CLOUD COMPUTING Source:https://farm6.staticflickr.com/5523/11407118963_e0e0870846_b_d.jpg www.hazelcast.com
  17. 17. MAP & REDUCE THE BLACK MAGIC FROM PLANET GOOGLE www.hazelcast.com
  18. 18. USE CASES LogAnalysis DataQuerying Aggregation and summing Distributed Sort ETL (ExtractTransform Load) and more... www.hazelcast.com
  19. 19. SIMPLE STEPS Read Map /Transform Reduce www.hazelcast.com
  20. 20. FULL STEPS Read Map /Transform Combining Grouping/Shuffling Reduce Collating www.hazelcast.com
  21. 21. MAPREDUCE WORKFLOW www.hazelcast.com
  22. 22. Dataare mapped /transformed in asetof key-value pairs SOME PSEUDO CODE (1/3) MAPPING map( key:String, document:String ):Void -> for each w:word in document: emit( w, 1 ) www.hazelcast.com
  23. 23. Multiple values are combined to an intermediate resultto preserve traffic SOME PSEUDO CODE (2/3) COMBINING combine( word:String, counts:List[Int] ):Void -> emit( word, sum( counts ) ) www.hazelcast.com
  24. 24. Values are reduced /aggregated to the requested result SOME PSEUDO CODE (3/3) REDUCING reduce( word:String, counts:List[Int] ):Int -> return sum( counts ) www.hazelcast.com
  25. 25. FOR MATHEMATICIANS Process: (K x V)*→ (L x W)* ⇒ [(l1, w1), …, (lm, wm)] Mapping: (K x V) → (L x W)* ⇒ (k, v) → [(l1, w1), …, (ln, wn)] Reducing: L x W*→ X* ⇒ (l, [w1, …, wn]) → [x1, …,xn] www.hazelcast.com
  26. 26. MAPREDUCE PROGRAMS IN GOOGLE SOURCE TREE Source:http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0005.html www.hazelcast.com
  27. 27. DEMONSTRATION www.hazelcast.com
  28. 28. @noctarius2k @hazelcast http://www.sourceprojects.com http://github.com/noctarius THANK YOU! ANY QUESTIONS? Images:AllimagesarelicensedunderCreativeCommons www.hazelcast.com

×