Devoxx 2013 - Hazelcast
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Devoxx 2013 - Hazelcast

on

  • 145 views

www.hazelcast.com

www.hazelcast.com

Statistics

Views

Total Views
145
Views on SlideShare
139
Embed Views
6

Actions

Likes
1
Downloads
1
Comments
0

1 Embed 6

http://www.slideee.com 6

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Devoxx 2013 - Hazelcast Presentation Transcript

  • 1. @PeterVeentjer#DV13-{session hashtag} Distributed Systems Using Hazelcast Peter Veentjer
  • 2. @PeterVeentjer#DV13-{session hashtag} • Working for Hazelcast • Senior developer • Solution architect • Author of ‘Pragmatic Guide to Hazelcast 3' • 13 years Java experience • Big love • Concurrency control • Distributed computing Whoami
  • 3. @PeterVeentjer#DV13-{session hashtag}
  • 4. @PeterVeentjer#DV13-{session hashtag} What is Hazelcast?
  • 5. @PeterVeentjer#DV13-{session hashtag} What is Hazelcast? • Leading Open Source Java In Memory Data/Compute Grid • Main goal is to simplify development of: • Scalable systems • Highly available systems
  • 6. @PeterVeentjer#DV13-{session hashtag} Why Hazelcast? • 2.5 MByte JAR • no other dependencies! • no need to install software! • Its a library, not an application framework • Scale at the application level • Apache 2 License • Free to use • no limitations
  • 7. @PeterVeentjer#DV13-{session hashtag} Distributed Data-structures • IAtomicLong • IdGenerator • Lock/Condition • CountDownLatch • Semaphore • Queue • Map • MultiMap • Set • List • Topic • Executor • TransactionalMap/Q/S… • Write your own!
  • 8. @PeterVeentjer#DV13-{session hashtag} So what can I do with it?
  • 9. @PeterVeentjer#DV13-{session hashtag} Scale up
  • 10. @PeterVeentjer#DV13-{session hashtag} Scale Out
  • 11. @PeterVeentjer#DV13-{session hashtag} High Availability
  • 12. @PeterVeentjer#DV13-{session hashtag} Raspberry Pi Cluster
  • 13. @PeterVeentjer#DV13-{session hashtag} When? • Messaging Solution • Event processing • Clustered Scheduling • Job Processing • Cluster management • HTTP session clustering • Caching
  • 14. @PeterVeentjer#DV13-{session hashtag} Which companies • Financials • JP Morgan, Morgan Stanley, HSBC, Deutsche Bank • Telco’s • AT&T, Ericsson • Gaming • Ubisoft/Blue Byte • E-Commerce • Apple, eBay
  • 15. @PeterVeentjer#DV13-{session hashtag} Which Open Source Projects • WSO2 Carbon • Mule ESB • Vert.x • Apache Camel • Apache Shiro • OrientDB • Alfresco • Karaf
  • 16. @PeterVeentjer#DV13-{session hashtag} NO SQL FAMILY Hazelcast Coherence Infinispan Gigaspaces Datagrids KeyValue Stores Document DB’s Column DB’s Mongo MarkLogic CassandraRedis CouchDB HBaseRiak Voldimort Graph DB’s Neo4J
  • 17. @PeterVeentjer#DV13-{session hashtag} How Often • In October 2013 • 3M startups • 48K unique
  • 18. @PeterVeentjer#DV13-{session hashtag} Where is the code!
  • 19. @PeterVeentjer#DV13-{session hashtag} HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance(); Creating Hazelcast Cluster
  • 20. @PeterVeentjer#DV13-{session hashtag} <hazelcast>          <network>…</network>          <map  name=“m”>…</map>          <queue  name=“q”>…</queue>          <…   </hazelcast> XML Configuration
  • 21. @PeterVeentjer#DV13-{session hashtag} Config  config  =  new  Config();   …  make  config  modifications   HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance(config); Programmatic Configuration
  • 22. Demo @PeterVeentjer#DV13-{session hashtag}
  • 23. @PeterVeentjer#DV13-{session hashtag} Map<String,String>  products  =  new  HashMap();   map.put("1","IPhone"); Map
  • 24. @PeterVeentjer#DV13-{session hashtag} Map<String,String>  products  =  new  ConcurrentHashMap();   map.put("1","IPhone"); Map
  • 25. @PeterVeentjer#DV13-{session hashtag} HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance();   Map<String,String>  products  =  hz.getMap("products");   cities.put("1","IPhone");   Map
  • 26. Demo @PeterVeentjer#DV13-{session hashtag}
  • 27. @PeterVeentjer#DV13-{session hashtag} ! <map  name="products">        <time-­‐to-­‐live-­‐seconds>4</time-­‐to-­‐live-­‐seconds>        <indexes>                <index>name</index>        </indexes>        <..>   </map> Map Configuration
  • 28. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability 271 Partitions partitionid = hash(key) % partitioncount …
  • 29. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1
  • 30. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1 Member 2 Member 3
  • 31. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1 Member 2 Member 3
  • 32. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1 Member 2 Member 3
  • 33. @PeterVeentjer#DV13-{session hashtag} Map Scalability and High Availability Member 1 Member 2
  • 34. Demo @PeterVeentjer#DV13-{session hashtag}
  • 35. @PeterVeentjer#DV13-{session hashtag} Map Persistence • Data Connection • MapLoader • MapStore ! • Write through • Write behind
  • 36. @PeterVeentjer#DV13-{session hashtag} Multimap • Collection as value • Serialization Overhead • Lost update
  • 37. Demo @PeterVeentjer#DV13-{session hashtag}
  • 38. @PeterVeentjer#DV13-{session hashtag} BlockingQueue  queue  =  new  LinkedBlockingQueue();   queue.offer("1");   Object  item  =  queue.take();   Queue
  • 39. @PeterVeentjer#DV13-{session hashtag} ! HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance();   BlockingQueue  queue  =  hz.getQueue("queue");   queue.offer("1");   Object  item  =  queue.take(); Queue
  • 40. Demo @PeterVeentjer#DV13-{session hashtag}
  • 41. @PeterVeentjer#DV13-{session hashtag} ITopic  topic  =  hz.getTopic("topic");           //publishing   topic.publish(msg);       ! //subscribing         topic.addMessageListener(new  TopicSubscriber());   ! public  class  TopicSubscriber  implements  MessageListener<String>   {          public  void  onMessage(Message<String>  m)  {                  System.out.println(m.getMessageObject());          }   } Topic
  • 42. @PeterVeentjer#DV13-{session hashtag} Client • Simplified • Encryption • Load Balance Policy • C++ and C# version
  • 43. @PeterVeentjer#DV13-{session hashtag} HazelcastInstance  client  =   HazelcastClient.newHazelcastClient();   Client
  • 44. @PeterVeentjer#DV13-{session hashtag} Network Communication • Cluster Discovery • Multicast • TCP/IP Cluster • AWS Cluster • Normal Network Communication • TCP/IP
  • 45. @PeterVeentjer#DV13-{session hashtag} Advanced Hazelcast
  • 46. @PeterVeentjer#DV13-{session hashtag} IMap  map1  =  hz.getMap(“map1”);   IMap  map2  =  hz.getMap(“map2”);
 
 <map  name=“map1”>     <time-­‐to-­‐live-­‐seconds>10</time-­‐to-­‐live-­‐seconds>   </map>   ! <map  name=“map2”>          <time-­‐to-­‐live-­‐seconds>10</time-­‐to-­‐live-­‐seconds>   </map> Wildcard configuration
  • 47. @PeterVeentjer#DV13-{session hashtag} IMap  map1  =  hz.getMap(“map1”);   IMap  map2  =  hz.getMap(“map2”);   ! <map  name=“map*”>            <time-­‐to-­‐live-­‐seconds>10</time-­‐to-­‐live-­‐seconds>   </map>
 Wildcard configuration
  • 48. @PeterVeentjer#DV13-{session hashtag} <map  name=“map”>          <time-­‐to-­‐live-­‐seconds>${ttl}</time-­‐to-­‐live-­‐seconds>   </map> XML Variables
  • 49. @PeterVeentjer#DV13-{session hashtag} Map: In Memory Format • 2 Options • BINARY • OBJECT • Predicates • EntryProcessor
  • 50. @PeterVeentjer#DV13-{session hashtag} ! ILock  lock  =  hz.getLock(“someLock”);   IAtomicLong  counter  =  getAtomicLong(“someCounter”);   IMap  map  =  hz.getMap(“someMap”);   map.put(“somePerson”,new  Person());   map.get(“somePerson”); Data Locality: Problem
  • 51. @PeterVeentjer#DV13-{session hashtag} Member 1 Member 2 Member 3 SomeLock SomeCounter SomePerson
  • 52. @PeterVeentjer#DV13-{session hashtag} name@partitionkey name
  • 53. @PeterVeentjer#DV13-{session hashtag} ILock  lock  =  hz.getLock(“someLock@foo”);
 IAtomicLong  counter  =  hz.getAtomicLong(“someCounter@foo”);   IMap  map  =  hz.getMap(“someMap”);   map.put(“somePerson@foo”,  new  Person());   Person  p  =  map.get(“somePerson@foo”); Data Locality: Fixed
  • 54. @PeterVeentjer#DV13-{session hashtag} Member 1 Member 2 Member 3 SomeLock SomeCounter SomePerson Foo Partition
  • 55. @PeterVeentjer#DV13-{session hashtag} IAtomicLong  c1  =  …   IAtomicLong  c2  =  hz.getAtomicLong(“c2@“+c1.getPartitionKey()); Adding object in same partition
  • 56. @PeterVeentjer#DV13-{session hashtag} Executor • Execute task anywhere • Execute task on key owner • Execute task • one/all/subset members • Synchronisation • ExecutionCallback • Futures
  • 57. Demo @PeterVeentjer#DV13-{session hashtag}
  • 58. @PeterVeentjer#DV13-{session hashtag} Concurrency Control • Locks • TransactionalMap.getForUpdate • Map.Lock
  • 59. Demo @PeterVeentjer#DV13-{session hashtag}
  • 60. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          Account  account  =  accounts.get(accountId);          account.balance+=amount;          accounts.put(accountId,  account);   }   Race problem
  • 61. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          accounts.lock(accountId);          try{                  Account  account  =  accounts.get(accountId);                  account.balance+=amount;                  accounts.put(accountId,  account);          }finally{                  accounts.unlock(accountId);          }   }   Pessimistic Increment
  • 62. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          accounts.lock(accountId);          try{                  Account  account  =  accounts.get(accountId);                  account.balance+=amount;                  accounts.put(accountId,  account);          }finally{                  accounts.unlock(accountId);          }   }   Pessimistic Increment
  • 63. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          for(;;){                  Account  oldAccount  =  accounts.get(accountId);                  Account  newAccount  =  new  Account(oldAccount);                  newAccount.balance+=amount;                  if(accounts.replace(accountId,  oldAccount,newAccount)){                          return;                  }          }   }   Optimistic Increment
  • 64. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,int  amount){          for(;;){                  Account  oldAccount  =  accounts.get(accountId);                  Account  newAccount  =  new  Account(oldAccount);                  newAccount.balance+=amount;                  if(accounts.replace(accountId,  oldAccount,newAccount)){                          return;                  }          }   }   Optimistic Increment
  • 65. @PeterVeentjer#DV13-{session hashtag} DATA Bad: Send Data to Function DATA Good: Send Function to Data Read Data Write Data Write Function
  • 66. @PeterVeentjer#DV13-{session hashtag} private  class  BalanceTask  implements  Runnable,Serializable{          private  int  accountId,  amount;   !        public  void  run()  {                  for(;;){                          Account  oldAccount  =  accounts.get(accountId);                          Account  newAccount  =  new  Account(oldAccount);                          newAccount.balance+=amount;                          if(accounts.replace(accountId,   oldAccount,newAccount)){                                  return;                          }                  }          }   }         Increment with runnable
  • 67. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,  int  amount){          BalanceTask  task  =  new  BalanceTask(accountId,amount);          executorService.executeOnKeyOwner(task,accountId);   } Increment with runnable
  • 68. @PeterVeentjer#DV13-{session hashtag} class  BalanceProcessor                  extends  AbstractEntryProcessor<Integer,Account>{          int  amount;          BalanceProcessor(int  amount)  {                  this.amount  =  amount;          }   !        @Override          public  Object  process(Map.Entry<Integer,  Account>  entry)  {                  entry.getValue().balance+=amount;                  return  null;          }   } Increment with EntryProcessor
  • 69. @PeterVeentjer#DV13-{session hashtag} public  void  increment(int  accountId,  int  amount){          BalanceProcessor  processor  =  new  BalanceProcessor(amount);          accounts.executeOnKey(accountId,  processor);   }   Using entry processor
  • 70. @PeterVeentjer#DV13-{session hashtag} SPI • You can write your own distributed data-structures • Examples • Distributed Actors implementation • Remote interfaces
  • 71. @PeterVeentjer#DV13-{session hashtag} Serialization API • Serializable/Externalizable • Portable • ByteArraySerializable • StreamSerializer • Kryo • Jackson Smile • Protobuf
  • 72. @PeterVeentjer#DV13-{session hashtag} ! boolean  compress  =  …;   ! SerializerConfig  serialiserCfg  =  new  SerializerConfig()            .setTypeClass(Product.class)            .setImplementation(new  ProductSerializer(compress));   ! Config  config  =  new  Config();   config.getSerializationConfig().addSerializerConfig(serializerC fg);   HazelcastInstance  hz  =  Hazelcast.newHazelcastInstance(config); Pluggable Serialization
  • 73. @PeterVeentjer#DV13-{session hashtag} Kryo Serialization SizeinBytes 0 5,000 10,000 15,000 20,000 Serialization Kryo Kryp + Compr
  • 74. @PeterVeentjer#DV13-{session hashtag} IExecutorService  executor  =  hz.getExecutor(“executor”);   executor.execute(()-­‐>System.out.println”hello”));   ! //configuration   SerializerConfig  sc  =  new  SerializerConfig().                                  setImplementation(new  KryoSerializer(false)).                                  setTypeClass(Runnable.class);   Config  config  =  new  Config();   config.getSerializationConfig().addSerializerConfig(sc);   Kryo: Lambda expression
  • 75. @PeterVeentjer#DV13-{session hashtag} IMap<Integer,Account>  accounts  =  hz.getMap(“accounts”);   Map<Integer,Integer>  result  =  accounts            .map(new  AccountSumMapper())            .reduce(new  AccountSumReducer())            .submit() Hazelcast 3.2? Map Reduce
  • 76. @PeterVeentjer#DV13-{session hashtag} Enterprise vs Community edition • Support contracts • Elastic Memory • Security • Management Center
  • 77. @PeterVeentjer#DV13-{session hashtag} The Future • Topologies • Hosted Management Center • Dynamic Creation • Map of Maps • Tiered storage
  • 78. @PeterVeentjer#DV13-{session hashtag} We are hiring!
  • 79. @PeterVeentjer#DV13-{session hashtag} Questions? You can find us at the Hazelcast booth! Good Question? 
 Get a Hazelcast book!