Quick introduction into distributed computing based on map reduce in Hazelcast

1. BIG DATA - FAST DATA
USING MAPREDUCE IN HAZELCAST
Source:http://www.newscientist.com/gallery/dn17805-computer-museums-of-the-world/11
www.hazelcast.com

2. WHO AM I
Christoph Engelbert(@noctarius2k)
8+ years of JavaWeirdoness
Performance, GC, traffic topics
Apache DirectMemoryPMC
Previous companies incl. Ubisoftand HRS
CastMapRMapReduce for Hazelcast3
www.hazelcast.com

3. TOPICS
Hazelcast
Distributed Computing
Map &Reduce
Demonstration
Questions
www.hazelcast.com

4. HAZELCAST
A SHORT SPACE TRIP
www.hazelcast.com

5. WHAT IS HAZELCAST?
In-MemoryData-Grid
DataPartioning(Sharding)
JavaCollections Implementation
Distributed ComputingPlatform
www.hazelcast.com

6. WHY HAZELCAST?
www.hazelcast.com

7. WHY IN-MEMORY
COMPUTING?
www.hazelcast.com

8. TREND OF PRICES
DataSource:http://www.jcmit.com/memoryprice.htm
www.hazelcast.com

9. SPEED DIFFERENCE
DataSource:http://i.imgur.com/ykOjTVw.png
www.hazelcast.com

10. DISTRIBUTED
COMPUTING
OR
MULTICORE CPU ON STEROIDS
www.hazelcast.com

11. THE IDEA OF DISTRIBUTED COMPUTING
Source:https://www.flickr.com/photos/stefan_ledwina/1853508040
www.hazelcast.com

12. THE BEGINNING
Source:http://en.wikipedia.org/wiki/File:KL_Advanced_Micro_Devices_AM9080.jpg
www.hazelcast.com

13. MULTICORE IS NOT NEW
Source:http://en.wikipedia.org/wiki/File:80386with387.JPG
www.hazelcast.com

14. CLUSTER IT
Source:http://rarecpus.com/images2/cpu_cluster.jpg
www.hazelcast.com

15. SUPER COMPUTER
Source:http://www.dkrz.de/about/aufgaben/dkrz-geschichte/rechnerhistorie-1
www.hazelcast.com

16. CLOUD COMPUTING
Source:https://farm6.staticflickr.com/5523/11407118963_e0e0870846_b_d.jpg
www.hazelcast.com

17. MAP & REDUCE
THE BLACK MAGIC FROM PLANET GOOGLE
www.hazelcast.com

18. USE CASES
LogAnalysis
DataQuerying
Aggregation and summing
Distributed Sort
ETL (ExtractTransform Load)
and more...
www.hazelcast.com

19. SIMPLE STEPS
Read
Map /Transform
Reduce
www.hazelcast.com

20. FULL STEPS
Read
Map /Transform
Combining
Grouping/Shuffling
Reduce
Collating
www.hazelcast.com

21. MAPREDUCE WORKFLOW
www.hazelcast.com

22. Dataare mapped /transformed in asetof key-value pairs
SOME PSEUDO CODE (1/3)
MAPPING
map( key:String, document:String ):Void ->
for each w:word in document:
emit( w, 1 )
www.hazelcast.com

23. Multiple values are combined to an
intermediate resultto preserve traffic
SOME PSEUDO CODE (2/3)
COMBINING
combine( word:String, counts:List[Int] ):Void ->
emit( word, sum( counts ) )
www.hazelcast.com

24. Values are reduced /aggregated to the requested result
SOME PSEUDO CODE (3/3)
REDUCING
reduce( word:String, counts:List[Int] ):Int ->
return sum( counts )
www.hazelcast.com

25. FOR MATHEMATICIANS
Process: (K x V)*→ (L x W)* ⇒ [(l1, w1), …, (lm, wm)]
Mapping: (K x V) → (L x W)* ⇒ (k, v) → [(l1, w1), …, (ln, wn)]
Reducing: L x W*→ X* ⇒ (l, [w1, …, wn]) → [x1, …,xn]
www.hazelcast.com

26. MAPREDUCE PROGRAMS IN
GOOGLE SOURCE TREE
Source:http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0005.html
www.hazelcast.com

27. DEMONSTRATION
www.hazelcast.com

28. @noctarius2k
@hazelcast
http://www.sourceprojects.com
http://github.com/noctarius
THANK YOU!
ANY QUESTIONS?
Images:AllimagesarelicensedunderCreativeCommons
www.hazelcast.com