Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building scalable applications with hazelcast

892 views

Published on

This is the slides from SVCC conference

Published in: Software
  • Be the first to comment

Building scalable applications with hazelcast

  1. 1. © 2015 Hazelcast Inc. Confidential & Proprietary 1 FUAD MALIKOV CO-FOUNDER Building Scalable Applications with Hazelcast
  2. 2. © 2015 Hazelcast Inc. Confidential & Proprietary 2 What Is Hazelcast? Hazelcast is a distributed, highly available and scalable Open Source In-Memory Data Grid
  3. 3. © 2015 Hazelcast Inc. Confidential & Proprietary 3 In Memory Data Grid 01001 10101 01010 In Memory Data Computing In Memory Data Messaging ++In Memory Data Storage
  4. 4. © 2015 Hazelcast Inc. Confidential & Proprietary 4 java.util.Map import java.util.HashMap; import java.util.Map; public static void main(String[] args) { Map<Integer, String> map = new HashMap<>(); map.put(1, "Paris"); map.put(2, "London"); map.put(3, "San Francisco"); String oldValue = map.remove(2); }
  5. 5. © 2015 Hazelcast Inc. Confidential & Proprietary 5 java.util.concurrent.ConcurrentMap import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; public static void main(String[] args) { ConcurrentMap<Integer, String> map = new ConcurrentHashMap<>(); map.put(1, "Paris"); map.put(2, "London"); map.put(3, "San Francisco"); String oldValue = map.remove(2); }
  6. 6. © 2015 Hazelcast Inc. Confidential & Proprietary 6 Distributed Map import java.util.concurrent.ConcurrentMap; import com.hazelcast.core.Hazelcast; import com.hazelcast.core.HazelcastInstance; public static void main(String[] args) { HazelcastInstance h = Hazelcast.newHazelcastInstance(); ConcurrentMap<Integer, String> map = h.getMap("myMap"); map.put(1, "Paris"); map.put(2, "London"); map.put(3, "San Francisco"); String oldValue = map.remove(2); }
  7. 7. © 2015 Hazelcast Inc. Confidential & Proprietary 7 Ecosystem Traction Dozens of Commercial and Open Source Projects Embed Hazelcast
  8. 8. © 2015 Hazelcast Inc. Confidential & Proprietary 8 Demo
  9. 9. © 2015 Hazelcast Inc. Confidential & Proprietary 9 Why Hazelcast? Scale-out Computing enables cluster capacity to be increased or decreased on-demand Resilience with automatic recovery from member failures without losing data while minimizing performance impact on running applications Programming Model provides a way for developers to easily program a cluster application as if it is a single process Fast Application Performance enables very large data sets to be held in main memory for real-time performance
  10. 10. © 2015 Hazelcast Inc. Confidential & Proprietary 10 Rebalance Data on New Node 10  
  11. 11. © 2015 Hazelcast Inc. Confidential & Proprietary 11 Distributed Maps Fixed number of partitions (default 271) Each key falls into a partition partitionId = hash(keyData)%PARTITION_COUNT Partition ownerships are reassigned upon membership change A B C
  12. 12. © 2015 Hazelcast Inc. Confidential & Proprietary 12 New Node Added DA B C
  13. 13. © 2015 Hazelcast Inc. Confidential & Proprietary 13 Migration DA B C
  14. 14. © 2015 Hazelcast Inc. Confidential & Proprietary 14 Migration DA B C
  15. 15. © 2015 Hazelcast Inc. Confidential & Proprietary 15 Migration DA B C
  16. 16. © 2015 Hazelcast Inc. Confidential & Proprietary 16 Migration DA B C
  17. 17. © 2015 Hazelcast Inc. Confidential & Proprietary 17 Migration DA B C
  18. 18. © 2015 Hazelcast Inc. Confidential & Proprietary 18 Migration DA B C
  19. 19. © 2015 Hazelcast Inc. Confidential & Proprietary 19 Migration Complete DA B C
  20. 20. © 2015 Hazelcast Inc. Confidential & Proprietary 20 Data Safety when Node Dies 20  
  21. 21. © 2015 Hazelcast Inc. Confidential & Proprietary 21 Node Crashes DA B C Crash
  22. 22. © 2015 Hazelcast Inc. Confidential & Proprietary 22 Backups Are Restored DA B C Crash
  23. 23. © 2015 Hazelcast Inc. Confidential & Proprietary 23 Backups Are Restored DA B C Crash
  24. 24. © 2015 Hazelcast Inc. Confidential & Proprietary 24 Backups Are Restored DA B C Crash
  25. 25. © 2015 Hazelcast Inc. Confidential & Proprietary 25 Backups Are Restored DA B C Crash
  26. 26. © 2015 Hazelcast Inc. Confidential & Proprietary 26 Backups Are Restored DA B C Crash
  27. 27. © 2015 Hazelcast Inc. Confidential & Proprietary 27 Backups Are Restored DA B C Crash
  28. 28. © 2015 Hazelcast Inc. Confidential & Proprietary 28 Backups Are Restored DA B C Crash
  29. 29. © 2015 Hazelcast Inc. Confidential & Proprietary 29 Backups Are Restored DA B C Crash
  30. 30. © 2015 Hazelcast Inc. Confidential & Proprietary 30 Recovery Is Complete DA C
  31. 31. © 2015 Hazelcast Inc. Confidential & Proprietary 31 Deployment Strategies
  32. 32. © 2015 Hazelcast Inc. Confidential & Proprietary 32 Deployment Options Great for early stages of rapid application development and iteration Necessary for scale up or scale out deployments – decouples upgrading of clients and cluster for long term TCO Embedded Hazelcast Hazelcast Node 1 Applications Java API Client-Server Mode Hazelcast Node 3 Java API Applications Java API Applications Java API Applications Hazelcast Node 2 Hazelcast Node 1 Hazelcast Node 2 Applications Java API Hazelcast Node 3 Applications Java API
  33. 33. © 2015 Hazelcast Inc. Confidential & Proprietary 33 Easy API // Creating a new Hazelcast node HazelcastInstance hz = Hazelcast.newHazelcastInstance(); // Getting a Map, Queue, Topic, ... Map map = hz.getMap("my-map"); Queue queue = hz.getQueue("my-queue"); ITopic topic = hz.getTopic("my-topic"); //Creating a Hazelcast Client HazelcastInstance client = Hazelcast.newHazelcastClient(); // Shutting down the node hz.shutdown();
  34. 34. © 2015 Hazelcast Inc. Confidential & Proprietary 34 Feature Overview
  35. 35. © 2015 Hazelcast Inc. Confidential & Proprietary 35 Easy to Unit Test public class SomeTestCase { private HazelcastInstance[] instances; @Before public void before() throws Exception { // Multiple instances on the same JVM instances = new HazelcastInstance[2]; instances[0] = Hazelcast.newHazelcastInstance(); instances[1] = Hazelcast.newHazelcastInstance(); } @After public void after() throws Exception { Hazelcast.shutdownAll(); } }
  36. 36. © 2015 Hazelcast Inc. Confidential & Proprietary 36 IM Data Store (Caching) Use Case Database Caching Use-Case Business Systems A B C RDBMS Mainframe MongoDB NoSQL REST Scale Hazelcast HD Cache Dist. Compute Dist. Message
  37. 37. © 2015 Hazelcast Inc. Confidential & Proprietary 37 Java Collection API: Map, List, Set, Queue JCache High Density Memory Store Hibernate 2nd Level Cache Web Session Replication: Tomcat, Jetty Predicate API: Indexes, SQL Query Persistence: Map/Queue Store & Loader. Write Behind/Through Eviction Near Cache Transactions: Local & XA WAN & DR Replication Memcached Interface IM Data Store (Caching) Features HD Cache Dist. Compute Dist. Message
  38. 38. © 2015 Hazelcast Inc. Confidential & Proprietary 38 Map API interface com.hazelcast.core.IMap<K, V> extends java.util.Map, java.util.ConcurrentMap HazelcastInstance hz = getHazelcastInstance(); //java.util.concurrent.ConcurrentMap implementation IMap<String, User> hzMap = hz.getMap("users"); hzMap.put("Peter", new User("Peter", "Veentjer")); hzMap.putIfAbsent("Peter", new User("Peter", "Veentjer")); //Distributed Lock hzMap.lock("Peter"); User peter = map.get("Peter");
  39. 39. © 2015 Hazelcast Inc. Confidential & Proprietary 39 Persistence API public class MapStorage implements MapStore<String, User>, MapLoader<String, User> { // Some methods missing ... @Override public User load(String key) { return loadValueDB(key); } @Override public Set<String> loadAllKeys() { return loadKeysDB(); } @Override public void delete(String key) { deleteDB(key); } @Override public void store(String key, User value) { storeToDatabase(key, value); } } <map name="users"> <map-store enabled="true"> <class-name>com.hazelcast.example.MapStorage</class-name> <write-delay-seconds>0</write-delay-seconds> </map-store> </map>
  40. 40. © 2015 Hazelcast Inc. Confidential & Proprietary 40 JCache API // Retrieve the CachingProvider which is automatically baced by // the chosen Hazelcast server or client provider CachingProvider cachingProvider = Caching.getCachingProvider(); // Create a CacheManager CacheManager cacheManager = cachingProvider.getCacheManager(); // Cache<String, String> cache = cacheManager // .getCache( name, String.class, String.class ); // Create a simple but typesafe configuration for the cache CompleteConfiguration<String, String> config = new MutableConfiguration<String, String>() .setTypes( String.class, String.class );
  41. 41. © 2015 Hazelcast Inc. Confidential & Proprietary 41 JCache API // Create and get the cache Cache<String, String> cache = cacheManager .createCache( "example", config ); // Alternatively to request an already existing cache Cache<String, String> cache = cacheManager .getCache( name, String.class, String.class ); // Put a value into the cache cache.put( "world", "Hello World" ); // Retrieve the value again from the cache String value = cache.get( "world" ); System.out.println( value );
  42. 42. © 2015 Hazelcast Inc. Confidential & Proprietary 42 High Density Caching On-Heap Memory Store (Objects Stored as Objects) High-Density Memory Store (Objects Serialized and Stored as Bytes) On-Heap SLAB Allocator* On-Heap SLAB Allocator* 2-4GB (Limited by Garbage Collection) 0-1TB (Limited by Machine RAM) * coming in 3.6 Memory Stores • Member • Client (Near Cache) RAM in JVM Process APIs JCache (ICache) Map (IMap) HD Cache Dist. Compute Dist. Message
  43. 43. © 2015 Hazelcast Inc. Confidential & Proprietary 43 On Heap Vs. High-Density Memory Management On Heap Memory HD Memory v2 0 MB Native 3.3 GB 3.9 GB Heap Storage 0.6 GB 9 (4900 ms) Major GC 0 (0 ms) 31 (4200 ms) Minor GC 356 (349 ms) Node Used Heap Total Heap Max. Heap Heap Usage Percentage Used Heap: 0.2 GB 192.168.1.10:5701 57 MB 229 MB 910 MB 6.28% Memory Utilization Home Offheap-test Node Used Heap: 3.9 GB 192.168.1.10:5701 3933 MB 4658MB 4653MB 84.45% Memory Utilization Home Used Heap Total Heap Max. Heap Heap Usage Percentage HD Cache Dist. Compute Dist. Message Example: On Heap Memory Example: HD Memory v2
  44. 44. © 2015 Hazelcast Inc. Confidential & Proprietary 44 Hazelcast Servers Hazelcast Server JVM [Memory] IM Distributed Computing Use Case A B C Business Logic Data Data Data CE = Compute Engine Result Business / Processing Logic Result TCP / IP Client Client HD Cache Dist. Compute Dist. Message
  45. 45. © 2015 Hazelcast Inc. Confidential & Proprietary 45 IM Distributed Computing Feature HD Cache Dist. Compute Dist. Message Java Concurrency API (Lock, Semaphore, AtomicLong, AtomicReference, Executor Service, Blocking Queue) Entry and Item Listeners Entry Processor Aggregators Map/Reduce Data Affinity Continues Query Map Interceptors Delta Update
  46. 46. © 2015 Hazelcast Inc. Confidential & Proprietary 46 Lock API HazelcastInstance hz = getHazelcastInstance(); // Distributed Reentrant L​ock l​ock = hz.getLock("myLock"); l​ock.lock(); try { // Do something } finally { l​ock.unlock(); }
  47. 47. © 2015 Hazelcast Inc. Confidential & Proprietary 47 Executor Service API public interface com.hazelcast.core.IExecutorService extends java.util.concurrent.ExecutorService HazelcastInstance hz = getHazelcastInstance(); //java.util.concurrent.ExecutorService implementation IExecutorService es = hz.getExecutorService("name"); es.executeOnAllMembers(buildRunnable()); es.executeOnKeyOwner(buildRunnable(), "Peter"); es.execute(buildRunnable()); Map<..> futures = es.submitToAllMembers(buildCallable()); Future<..> future = es.submitToKeyOwner(buildCallable(), "Peter"); es.submitToAllMembers(buildCallable(), buildCallback()); es.submitToKeyOwner(buildCallable(), "Peter", buildCallback());
  48. 48. © 2015 Hazelcast Inc. Confidential & Proprietary 48 Map/Reduce API HazelcastInstance hz = getHazelcastInstance(); Map users = hz.getMap("users"); JobTracker tracker = hz.getJobTracker("default"); KeyValueSource source = KeyValueSource.fromMap(users); Job job = tracker.newJob(source); ICompleteFuture future = job.mapper(new MyMapper()) .reducer(new MyReducer()) .submit(); Map result = future.get();
  49. 49. © 2015 Hazelcast Inc. Confidential & Proprietary 49 Aggregations API HazelcastInstance hz = getHazelcastInstance(); Map users = hz.getMap("users"); int sum = users.aggregate( Supplier.all((user) -> user.getSalary()), Aggregations.longSum() );
  50. 50. © 2015 Hazelcast Inc. Confidential & Proprietary 50 IM Distributed Messaging Use Case Hazelcast Distributed Topic Bus Hazelcast Topic Hazelcast Node 1 Hazelcast Node 2 Hazelcast Node 3 MSG Subscribes Delivers Subscribes Delivers HD Cache Dist. Compute Dist. Message
  51. 51. © 2015 Hazelcast Inc. Confidential & Proprietary 51 Queue Topic (Pub/Sub) Event Listeners Ring Buffers IM Distributed Messaging Features HD Cache Dist. Compute Dist. Message
  52. 52. © 2015 Hazelcast Inc. Confidential & Proprietary 52 Queue API interface com.hazelcast.core.IQueue<E> extends java.util.concurrent.BlockingQueue HazelcastInstance hz = getHazelcastInstance(); //java.util.concurrent.BlockingQueue implementation IQueue<Task> queue = hz.getQueue("tasks"); queue.offer(newTask()); queue.offer(newTask(), 500, TimeUnit.MILLISECONDS); Task task = queue.poll(); Task task = queue.poll(100, TimeUnit.MILLISECONDS); Task task = queue.take();
  53. 53. © 2015 Hazelcast Inc. Confidential & Proprietary 53 Topic API public class Example implements MessageListener<String> { public void sendMessage { HazelcastInstance hz = getHazelcastInstance(); ITopic<String> topic = hz.getTopic("topic"); topic.addMessageListener(this); topic.publish("Hello World"); } @Override public void onMessage(Message<String> message) { System.out.println("Got message: " + message.getMessageObject()); } }
  54. 54. Thank you @fuadm, @hazelcast hazelcast@googlegroups.com http://www.hazelcast.com http://github.com/hazelcast/hazelcast

×