Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Devoxx 2013 - Hazelcast

793 views

Published on

www.hazelcast.com

Published in: Software, Technology
  • Be the first to comment

Devoxx 2013 - Hazelcast

  1. 1. @PeterVeentjer#DV13-{session hashtag} Distributed Systems Using Hazelcast Peter Veentjer
  2. 2. @PeterVeentjer#DV13-{session hashtag} • Working for Hazelcast • Senior developer • Solution architect • Author of ‘Pragmatic Guide to Hazelcast 3' • 13 years Java experience • Big love • Concurrency control • Distributed computing Whoami
  3. 3. @PeterVeentjer#DV13-{session hashtag}
  4. 4. @PeterVeentjer#DV13-{session hashtag} What is Hazelcast?
  5. 5. @PeterVeentjer#DV13-{session hashtag} What is Hazelcast? • Leading Open Source Java In Memory Data/Compute Grid • Main goal is to simplify development of: • Scalable systems • Highly available systems
  6. 6. @PeterVeentjer#DV13-{session hashtag} Why Hazelcast? • 2.5 MByte JAR • no other dependencies! • no need to install software! • Its a library, not an application framework • Scale at the application level • Apache 2 License • Free to use • no limitations
  7. 7. @PeterVeentjer#DV13-{session hashtag} Distributed Data-structures • IAtomicLong • IdGenerator • Lock/Condition • CountDownLatch • Semaphore • Queue • Map • MultiMap • Set • List • Topic • Executor • TransactionalMap/Q/S… • Write your own!
  8. 8. @PeterVeentjer#DV13-{session hashtag} So what can I do with it?
  9. 9. @PeterVeentjer#DV13-{session hashtag} Scale up
  10. 10. @PeterVeentjer#DV13-{session hashtag} Scale Out
  11. 11. @PeterVeentjer#DV13-{session hashtag} High Availability
  12. 12. @PeterVeentjer#DV13-{session hashtag} Raspberry Pi Cluster
  13. 13. @PeterVeentjer#DV13-{session hashtag} When? • Messaging Solution • Event processing • Clustered Scheduling • Job Processing • Cluster management • HTTP session clustering • Caching
  14. 14. @PeterVeentjer#DV13-{session hashtag} Which companies • Financials • JP Morgan, Morgan Stanley, HSBC, Deutsche Bank • Telco’s • AT&T, Ericsson • Gaming • Ubisoft/Blue Byte • E-Commerce • Apple, eBay
  15. 15. @PeterVeentjer#DV13-{session hashtag} Which Open Source Projects • WSO2 Carbon • Mule ESB • Vert.x • Apache Camel • Apache Shiro • OrientDB • Alfresco • Karaf
  16. 16. @PeterVeentjer#DV13-{session hashtag} NO SQL FAMILY Hazelcast Coherence Infinispan Gigaspaces Datagrids KeyValue Stores Document DB’s Column DB’s Mongo MarkLogic CassandraRedis CouchDB HBaseRiak Voldimort Graph DB’s Neo4J
  17. 17. @PeterVeentjer#DV13-{session hashtag} How Often • In October 2013 • 3M startups • 48K unique
  18. 18. @PeterVeentjer#DV13-{session hashtag} Where is the code!
  19. 19. @PeterVeentjer#DV13-{session hashtag} HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance(); Creating Hazelcast Cluster
  20. 20. @PeterVeentjer#DV13-{session hashtag} <hazelcast>          <network>…</network>          <map  name=“m”>…</map>          <queue  name=“q”>…</queue>          <…   </hazelcast> XML Configuration
  21. 21. @PeterVeentjer#DV13-{session hashtag} Config  config  =  new  Config();   …  make  config  modifications   HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance(config); Programmatic Configuration
  22. 22. Demo @PeterVeentjer#DV13-{session hashtag}
  23. 23. @PeterVeentjer#DV13-{session hashtag} Map<String,String>  products  =  new  HashMap();   map.put("1","IPhone"); Map
  24. 24. @PeterVeentjer#DV13-{session hashtag} Map<String,String>  products  =  new  ConcurrentHashMap();   map.put("1","IPhone"); Map
  25. 25. @PeterVeentjer#DV13-{session hashtag} HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance();   Map<String,String>  products  =  hz.getMap("products");   cities.put("1","IPhone");   Map
  26. 26. Demo @PeterVeentjer#DV13-{session hashtag}
  27. 27. @PeterVeentjer#DV13-{session hashtag} ! <map  name="products">        <time-­‐to-­‐live-­‐seconds>4</time-­‐to-­‐live-­‐seconds>        <indexes>                <index>name</index>        </indexes>        <..>   </map> Map Configuration
  28. 28. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability 271 Partitions partitionid = hash(key) % partitioncount …
  29. 29. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1
  30. 30. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1 Member 2 Member 3
  31. 31. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1 Member 2 Member 3
  32. 32. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1 Member 2 Member 3
  33. 33. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1 Member 2
  34. 34. Demo @PeterVeentjer#DV13-{session hashtag}
  35. 35. @PeterVeentjer#DV13-{session hashtag} Map Persistence • Data Connection • MapLoader • MapStore ! • Write through • Write behind
  36. 36. @PeterVeentjer#DV13-{session hashtag} Multimap • Collection as value • Serialization Overhead • Lost update
  37. 37. Demo @PeterVeentjer#DV13-{session hashtag}
  38. 38. @PeterVeentjer#DV13-{session hashtag} BlockingQueue  queue  =  new  LinkedBlockingQueue();   queue.offer("1");   Object  item  =  queue.take();   Queue
  39. 39. @PeterVeentjer#DV13-{session hashtag} ! HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance();   BlockingQueue  queue  =  hz.getQueue("queue");   queue.offer("1");   Object  item  =  queue.take(); Queue
  40. 40. Demo @PeterVeentjer#DV13-{session hashtag}
  41. 41. @PeterVeentjer#DV13-{session hashtag} ITopic  topic  =  hz.getTopic("topic");           //publishing   topic.publish(msg);       ! //subscribing         topic.addMessageListener(new  TopicSubscriber());   ! public  class  TopicSubscriber  implements  MessageListener<String>   {          public  void  onMessage(Message<String>  m)  {                  System.out.println(m.getMessageObject());          }   } Topic
  42. 42. @PeterVeentjer#DV13-{session hashtag} Client • Simplified • Encryption • Load Balance Policy • C++ and C# version
  43. 43. @PeterVeentjer#DV13-{session hashtag} HazelcastInstance  client  =   HazelcastClient.newHazelcastClient();   Client
  44. 44. @PeterVeentjer#DV13-{session hashtag} Network Communication • Cluster Discovery • Multicast • TCP/IP Cluster • AWS Cluster • Normal Network Communication • TCP/IP
  45. 45. @PeterVeentjer#DV13-{session hashtag} Advanced Hazelcast
  46. 46. @PeterVeentjer#DV13-{session hashtag} IMap  map1  =  hz.getMap(“map1”);   IMap  map2  =  hz.getMap(“map2”);
 
 <map  name=“map1”>     <time-­‐to-­‐live-­‐seconds>10</time-­‐to-­‐live-­‐seconds>   </map>   ! <map  name=“map2”>          <time-­‐to-­‐live-­‐seconds>10</time-­‐to-­‐live-­‐seconds>   </map> Wildcard configuration
  47. 47. @PeterVeentjer#DV13-{session hashtag} IMap  map1  =  hz.getMap(“map1”);   IMap  map2  =  hz.getMap(“map2”);   ! <map  name=“map*”>            <time-­‐to-­‐live-­‐seconds>10</time-­‐to-­‐live-­‐seconds>   </map>
 Wildcard configuration
  48. 48. @PeterVeentjer#DV13-{session hashtag} <map  name=“map”>          <time-­‐to-­‐live-­‐seconds>${ttl}</time-­‐to-­‐live-­‐seconds>   </map> XML Variables
  49. 49. @PeterVeentjer#DV13-{session hashtag} Map: In Memory Format • 2 Options • BINARY • OBJECT • Predicates • EntryProcessor
  50. 50. @PeterVeentjer#DV13-{session hashtag} ! ILock  lock  =  hz.getLock(“someLock”);   IAtomicLong  counter  =  getAtomicLong(“someCounter”);   IMap  map  =  hz.getMap(“someMap”);   map.put(“somePerson”,new  Person());   map.get(“somePerson”); Data Locality: Problem
  51. 51. @PeterVeentjer#DV13-{session hashtag} Member 1 Member 2 Member 3 SomeLock SomeCounter SomePerson
  52. 52. @PeterVeentjer#DV13-{session hashtag} name@partitionkey name
  53. 53. @PeterVeentjer#DV13-{session hashtag} ILock  lock  =  hz.getLock(“someLock@foo”);
 IAtomicLong  counter  =  hz.getAtomicLong(“someCounter@foo”);   IMap  map  =  hz.getMap(“someMap”);   map.put(“somePerson@foo”,  new  Person());   Person  p  =  map.get(“somePerson@foo”); Data Locality: Fixed
  54. 54. @PeterVeentjer#DV13-{session hashtag} Member 1 Member 2 Member 3 SomeLock SomeCounter SomePerson Foo Partition
  55. 55. @PeterVeentjer#DV13-{session hashtag} IAtomicLong  c1  =  …   IAtomicLong  c2  =  hz.getAtomicLong(“c2@“+c1.getPartitionKey()); Adding object in same partition
  56. 56. @PeterVeentjer#DV13-{session hashtag} Executor • Execute task anywhere • Execute task on key owner • Execute task • one/all/subset members • Synchronisation • ExecutionCallback • Futures
  57. 57. Demo @PeterVeentjer#DV13-{session hashtag}
  58. 58. @PeterVeentjer#DV13-{session hashtag} Concurrency Control • Locks • TransactionalMap.getForUpdate • Map.Lock
  59. 59. Demo @PeterVeentjer#DV13-{session hashtag}
  60. 60. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          Account  account  =  accounts.get(accountId);          account.balance+=amount;          accounts.put(accountId,  account);   }   Race problem
  61. 61. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          accounts.lock(accountId);          try{                  Account  account  =  accounts.get(accountId);                  account.balance+=amount;                  accounts.put(accountId,  account);          }finally{                  accounts.unlock(accountId);          }   }   Pessimistic Increment
  62. 62. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          accounts.lock(accountId);          try{                  Account  account  =  accounts.get(accountId);                  account.balance+=amount;                  accounts.put(accountId,  account);          }finally{                  accounts.unlock(accountId);          }   }   Pessimistic Increment
  63. 63. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          for(;;){                  Account  oldAccount  =  accounts.get(accountId);                  Account  newAccount  =  new  Account(oldAccount);                  newAccount.balance+=amount;                  if(accounts.replace(accountId,  oldAccount,newAccount)){                          return;                  }          }   }   Optimistic Increment
  64. 64. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          for(;;){                  Account  oldAccount  =  accounts.get(accountId);                  Account  newAccount  =  new  Account(oldAccount);                  newAccount.balance+=amount;                  if(accounts.replace(accountId,  oldAccount,newAccount)){                          return;                  }          }   }   Optimistic Increment
  65. 65. @PeterVeentjer#DV13-{session hashtag} DATA Bad: Send Data to Function DATA Good: Send Function to Data Read Data Write Data Write Function
  66. 66. @PeterVeentjer#DV13-{session hashtag} private  class  BalanceTask  implements  Runnable,Serializable{          private  int  accountId,  amount;   !        public  void  run()  {                  for(;;){                          Account  oldAccount  =  accounts.get(accountId);                          Account  newAccount  =  new  Account(oldAccount);                          newAccount.balance+=amount;                          if(accounts.replace(accountId,   oldAccount,newAccount)){                                  return;                          }                  }          }   }         Increment with runnable
  67. 67. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,  int  amount){          BalanceTask  task  =  new  BalanceTask(accountId,amount);          executorService.executeOnKeyOwner(task,accountId);   } Increment with runnable
  68. 68. @PeterVeentjer#DV13-{session hashtag} class  BalanceProcessor                  extends  AbstractEntryProcessor<Integer,Account>{          int  amount;          BalanceProcessor(int  amount)  {                  this.amount  =  amount;          }   !        @Override          public  Object  process(Map.Entry<Integer,  Account>  entry)  {                  entry.getValue().balance+=amount;                  return  null;          }   } Increment with EntryProcessor
  69. 69. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,  int  amount){          BalanceProcessor  processor  =  new  BalanceProcessor(amount);          accounts.executeOnKey(accountId,  processor);   }   Using entry processor
  70. 70. @PeterVeentjer#DV13-{session hashtag} SPI • You can write your own distributed data-structures • Examples • Distributed Actors implementation • Remote interfaces
  71. 71. @PeterVeentjer#DV13-{session hashtag} Serialization API • Serializable/Externalizable • Portable • ByteArraySerializable • StreamSerializer • Kryo • Jackson Smile • Protobuf
  72. 72. @PeterVeentjer#DV13-{session hashtag} ! boolean  compress  =  …;   ! SerializerConfig  serialiserCfg  =  new  SerializerConfig()            .setTypeClass(Product.class)            .setImplementation(new  ProductSerializer(compress));   ! Config  config  =  new  Config();   config.getSerializationConfig().addSerializerConfig(serializerC fg);   HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance(config); Pluggable Serialization
  73. 73. @PeterVeentjer#DV13-{session hashtag} Kryo Serialization SizeinBytes 0 5,000 10,000 15,000 20,000 Serialization Kryo Kryp + Compr
  74. 74. @PeterVeentjer#DV13-{session hashtag} IExecutorService  executor  =  hz.getExecutor(“executor”);   executor.execute(()-­‐>System.out.println”hello”));   ! //configuration   SerializerConfig  sc  =  new  SerializerConfig().                                  setImplementation(new  KryoSerializer(false)).                                  setTypeClass(Runnable.class);   Config  config  =  new  Config();   config.getSerializationConfig().addSerializerConfig(sc);   Kryo: Lambda expression
  75. 75. @PeterVeentjer#DV13-{session hashtag} IMap<Integer,Account>  accounts  =  hz.getMap(“accounts”);   Map<Integer,Integer>  result  =  accounts            .map(new  AccountSumMapper())            .reduce(new  AccountSumReducer())            .submit() Hazelcast 3.2? Map Reduce
  76. 76. @PeterVeentjer#DV13-{session hashtag} Enterprise vs Community edition • Support contracts • Elastic Memory • Security • Management Center
  77. 77. @PeterVeentjer#DV13-{session hashtag} The Future • Topologies • Hosted Management Center • Dynamic Creation • Map of Maps • Tiered storage
  78. 78. @PeterVeentjer#DV13-{session hashtag} We are hiring!
  79. 79. @PeterVeentjer#DV13-{session hashtag} Questions? You can find us at the Hazelcast booth! Good Question? 
 Get a Hazelcast book!

×