Distributed Computing in Hazelcast - Geekout 2014 Edition
•
2 likes•2,087 views
Today’s amounts of collected data are showing a nearly exponential growth. More than 75% of all the data have been collected in the past 5 years. To store this data and process it in an appropriate time you need to partition the data and parallelize the processing of reports and analytics. This talk will demonstrate how to parallelize data processing using Hazelcast and it’s underlying distributed data structures. With a quick introduction into the different terms and some short live coding examples we will make the journey into the distributed computing. Sourcecode of the demonstrations are available here: 1. https://github.com/noctarius/hazelcast-mapreduce-presentation 2. https://github.com/noctarius/hazelcast-distributed-computing
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (16)
Similar to Distributed Computing in Hazelcast - Geekout 2014 Edition
Similar to Distributed Computing in Hazelcast - Geekout 2014 Edition (20)
More from Christoph Engelbert
More from Christoph Engelbert (20)
Recently uploaded
Recently uploaded (20)
Distributed Computing in Hazelcast - Geekout 2014 Edition
- 2. THAT'S ME Christoph Engelbert Twitter: @noctarius2k 8+ years of JavaWeirdoness Performance, GC, traffic topics Apache Committer Gaming, TravelManagement, ... www.hazelcast.com
- 3. OUR SPACE TRIP ROADMAP Hazelcast Distributed Computing Distributed ExecutorService EntryProcessor Map &Reduce Questions www.hazelcast.com
- 5. WHAT IS HAZELCAST? In-MemoryData-Grid DataPartioning(Sharding) JavaCollections Implementation Distributed ComputingPlatform www.hazelcast.com
- 6. WHY HAZELCAST? Automatic Partitioning FaultTolerance Sync /Async Backups FullyDistributed In-Memoryfor HighestSpeed www.hazelcast.com
- 12. DISTRIBUTED COMPUTING OR MULTICORE CPU ON STEROIDS www.hazelcast.com
- 13. THE IDEA OF DISTRIBUTED COMPUTING Source:https://www.flickr.com/photos/stefan_ledwina/1853508040 www.hazelcast.com
- 15. MULTICORE IS NOT NEW Source:http://en.wikipedia.org/wiki/File:80386with387.JPG www.hazelcast.com
- 19. DISTRIBUTED EXECUTORSERVICE THE WHOLE CLUSTER IN YOUR HANDS www.hazelcast.com
- 20. WHY A DISTRIBUTED EXECUTORSERVICE? j.l.Runnable /j.u.c.Callable Onlyneeds to be serializable Same Task all/multiple Nodes Should notwork on Data www.hazelcast.com
- 21. Printnode name on allnodes QUICK EXAMPLE Runnablerunnable=()->println("RunningonNode:"+member.node); IExecutorServiceexecutorService=hazelcastInstance.getExecutorService("default"); executorService.executeOnAllMembers(runnable); www.hazelcast.com
- 24. WHY ENTRYPROCESSOR? Prevents externalLocking Guarantees Atomicity Kinda"Cluster-wide Thread-Safe" www.hazelcast.com
- 27. MAP & REDUCE THE BLACK MAGIC FROM PLANET GOOGLE www.hazelcast.com
- 28. USE CASES LogAnalysis DataQuerying Aggregation and summing Distributed Sort ETL (ExtractTransform Load) and more... www.hazelcast.com
- 32. Dataare mapped /transformed in asetof key-value pairs SOME PSEUDO CODE (1/3) MAPPING map(key:String,document:String):Void-> foreachw:Wordindocument: emit(w,1) www.hazelcast.com
- 33. Multiple values are combined to an intermediate resultto preserve traffic SOME PSEUDO CODE (2/3) COMBINING combine(word:Word,counts:List[Int]):Void-> emit(word,sum(counts)) www.hazelcast.com
- 34. Values are reduced /aggregated to the requested result SOME PSEUDO CODE (3/3) REDUCING reduce(word:String,counts:List[Int]):Int-> returnsum(counts) www.hazelcast.com
- 35. FOR MATHEMATICIANS Process: (KxV)*→(LxW)* ⇒ [(l1,w1),…,(lm,wm)] Mapping: (KxV) →(LxW)* ⇒ (k,v) →[(l1,w1),…,(ln,wn)] Reducing: LxW*→X* ⇒ (l,[w1,…,wn]) →[x1,…,xn] www.hazelcast.com
- 36. MAPREDUCE PROGRAMS IN GOOGLE SOURCE TREE Source:http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0005.html www.hazelcast.com