Hacking Infinispan                         Open Source Data Grids meets NoSQL                                         Mani...
#infinispan                         Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 Ma...
“Could data storage be the one thing that hampers true cloud         scalability and elasticity?”                         ...
Who is Manik?       • R&D Engineer, Red Hat Inc.             •       Founder and project lead, Infinispan             •   ...
• What is Data-as-a-Service?      • Introducing Infinispan      • Implementing Data-as-a-Service with             Infinisp...
Traditional 3-tier App                                             Manik Surtani http://twitter.com/maniksurtani http://bl...
Typical IaaS App                                            Manik Surtani http://twitter.com/maniksurtani http://blog.infi...
Typical PaaS App                                            Manik Surtani http://twitter.com/maniksurtani http://blog.infi...
Where’s your data stored??                         Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.or...
Clouds are ephemeral!                                Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan....
Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
Virtualizing Data         • Some public services do exist             •      Amazon RDS and SimpleDB             •      Fa...
What about private clouds?      • Not all cloud deployments         are public!      • Private cloud is very        import...
Characteristics of DaaS  •     Elastic data  •     Needs to scale with other tiers  •     Response times should be linear ...
SQL and NoSQL DB                         • Lack of distribution hampers                              elasticity and HA    ...
SQL and NoSQL DB                         • Lack of distribution hampers                              elasticity and HA    ...
Distributed Data Grid    •     Far better suited to elastic data    •     Distributed by nature    •     Highly available ...
API is king       • Apps should use their native data storage              APIs           • E.g., JPA (Java EE), ActiveRec...
Introducing                                       Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org...
• What is Infinispan?                  •      Open source (LGPL) in-memory Data Grid                  •      Written in Ja...
Embedded Mode                           Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, ...
P2P Embedded Architecture                                   Manik Surtani http://twitter.com/maniksurtani http://blog.infi...
• API                         • Map-like key/value store                         • Upcoming JPA-like layer                ...
• Consistent hash based distribution                    •    Self healing                    •    No single point of failu...
• Persistence                   •     Not just in memory!                   •     Write through and write behind          ...
• XA Transactions                    •    2-phase commit based                    •    Deadlock detection algorithms      ...
• Map/Reduce                    •    In a pre-release state right now                    •    Please try out 5.0.0.ALPHA2 ...
Client/Server Mode                             Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSat...
Client/Server Architecture                                                                       Supported Protocols      ...
WTF is Hot Rod?  • Wire protocol for client server        communications  •     Open  •     Language independent  •     Bu...
Server Endpoint Comparison                                      Client                              Smart             Load...
So is Infinispan a     data grid?• In-memory• P2P, distributed• Low-latency, fast key/value  store                        ...
So is Infinispan a     data grid?• In-memory• P2P, distributed             ... or is it a             NoSQL• Low-latency, ...
So is Infinispan a     data grid?• In-memory• P2P, distributed             ... or is it a             NoSQL• Low-latency, ...
So why is Infinispan sexy?                              Manik Surtani http://twitter.com/maniksurtani http://blog.infinisp...
So why is Infinispan sexy?              •          Transparent horizontal scalability                    •      Elastic in...
DaaS with Infinispan                              Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org...
Architecture                                                                                                       Manage ...
Where are we headed?                            Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSa...
• 4.0.0 Starobrno           •      Map-like API           •      Consistent Hash based distribution           •      Write...
• 4.1.0 Radegast              •      Deadlock detection              •      Client/Server protocols                    •  ...
• 5.0.0 Pagoa               •         JPA-like API               •         Fine-grained replication               •       ...
What have we learned?              •          Elastic data is hard              •          Public data services not always...
Questions?             •       http://www.infinispan.org             •       http://blog.infinispan.org             •     ...
Upcoming SlideShare
Loading in...5
×

Hacking Infinispan: the new open source data grid meets NoSQL

2,462

Published on

La presentazione di Manik Surtani
in occasione del Codemotion, Roma 5 marzo 2011 http://www.codemotion.it

Published in: Technology

Transcript of "Hacking Infinispan: the new open source data grid meets NoSQL"

  1. 1. Hacking Infinispan Open Source Data Grids meets NoSQL Manik Surtani Founder and Project Lead, Infinispan JBoss, by Red Hat Inc. Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  2. 2. #infinispan Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  3. 3. “Could data storage be the one thing that hampers true cloud scalability and elasticity?” Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  4. 4. Who is Manik? • R&D Engineer, Red Hat Inc. • Founder and project lead, Infinispan • Project lead, JBoss Cache • Frequent speaker on cloud computing and cloud data storage • http://twitter.com/maniksurtani • http://blog.infinispan.org Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  5. 5. • What is Data-as-a-Service? • Introducing Infinispan • Implementing Data-as-a-Service with Infinispan Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  6. 6. Traditional 3-tier App Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  7. 7. Typical IaaS App Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  8. 8. Typical PaaS App Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  9. 9. Where’s your data stored?? Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  10. 10. Clouds are ephemeral! Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  11. 11. Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  12. 12. Virtualizing Data • Some public services do exist • Amazon RDS and SimpleDB • FathomDB • Cloundant • MongoHQ • etc. Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  13. 13. What about private clouds? • Not all cloud deployments are public! • Private cloud is very important • How can you build DaaS yourself? Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  14. 14. Characteristics of DaaS • Elastic data • Needs to scale with other tiers • Response times should be linear • Needs to be highly available • Nodes will die! The service shouldn’t. Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  15. 15. SQL and NoSQL DB • Lack of distribution hampers elasticity and HA • These limitations can be worked around • ... but this isn’t trivial • ... or cheap Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  16. 16. SQL and NoSQL DB • Lack of distribution hampers elasticity and HA • These limitations can be worked around • ... but this isn’t trivial • ... or cheap Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  17. 17. Distributed Data Grid • Far better suited to elastic data • Distributed by nature • Highly available by nature • A good building block for your data service Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  18. 18. API is king • Apps should use their native data storage APIs • E.g., JPA (Java EE), ActiveRecord (Ruby), etc. • Key/value stores too low level • Akin to direct JDBC calls! Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  19. 19. Introducing Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  20. 20. • What is Infinispan? • Open source (LGPL) in-memory Data Grid • Written in Java and Scala • Some concepts from Amazon Dynamo • 2 usage modes • Embedded • Client-server • memcached, REST and Hot Rod Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  21. 21. Embedded Mode Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  22. 22. P2P Embedded Architecture Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  23. 23. • API • Map-like key/value store • Upcoming JPA-like layer • Other high-level APIs being discussed in the community e.g., ActiveRecord Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  24. 24. • Consistent hash based distribution • Self healing • No single point of failure • Highly concurrent • MVCC locking Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  25. 25. • Persistence • Not just in memory! • Write through and write behind • Pluggable “drivers” • Eviction and expiry • Efficient, adaptive algorithms • Addresses shortcomings of LRU & FIFO Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  26. 26. • XA Transactions • 2-phase commit based • Deadlock detection algorithms • Coming soon: Atomic Broadcast Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  27. 27. • Map/Reduce • In a pre-release state right now • Please try out 5.0.0.ALPHA2 with these APIs! • Querying • Using Lucene and Hibernate Search to index Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  28. 28. Client/Server Mode Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  29. 29. Client/Server Architecture Supported Protocols •REST •Memcached •Hot Rod Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  30. 30. WTF is Hot Rod? • Wire protocol for client server communications • Open • Language independent • Built-in failover and load balancing • Smart routing Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  31. 31. Server Endpoint Comparison Client Smart Load Balancing/ Protocol Clustered? Libraries Routing Failover Any HTTP load REST Text N/A Yes No balancer Only with predefined Memcached Text Plenty Yes No server list Currently Hot Rod Binary Yes Yes Dynamic only Java Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  32. 32. So is Infinispan a data grid?• In-memory• P2P, distributed• Low-latency, fast key/value store Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  33. 33. So is Infinispan a data grid?• In-memory• P2P, distributed ... or is it a NoSQL• Low-latency, fast key/value database? store • Persistence • Map/Reduce Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  34. 34. So is Infinispan a data grid?• In-memory• P2P, distributed ... or is it a NoSQL• Low-latency, fast key/value database? store • Persistence • Map/Reduce ... or something else? • Querying support • Transactional Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  35. 35. So why is Infinispan sexy? Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  36. 36. So why is Infinispan sexy? • Transparent horizontal scalability • Elastic in both directions • Fast, low latency data access • Ability to address a very large heap • Cloud-ready datastore • Not just for Java • Open Source Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  37. 37. DaaS with Infinispan Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  38. 38. Architecture Manage and Monitor Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  39. 39. Where are we headed? Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  40. 40. • 4.0.0 Starobrno • Map-like API • Consistent Hash based distribution • Write-through, write-behind • Eviction, expiration • Management tooling • REST API • Hibernate 2nd Level Cache • Released Feb 2010 Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  41. 41. • 4.1.0 Radegast • Deadlock detection • Client/Server protocols • Memcached • Hot Rod • Smart clients using Hot Rod • Lucene Directory implementation • LIRS: adaptive, recency-based eviction policies • Released August 2010 Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  42. 42. • 5.0.0 Pagoa • JPA-like API • Fine-grained replication • Distributed code execution • Map/reduce • Virtual nodes for more even distribution • In active development Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  43. 43. What have we learned? • Elastic data is hard • Public data services not always suitable • Data grids make elastic storage easy • Infinispan server endpoints help build elastic data tiers Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011
  44. 44. Questions? • http://www.infinispan.org • http://blog.infinispan.org • http://github.com/infinispan • http://twitter.com/infinispan • #infinispan on FreeNode • http://bit.ly/rate_me_codemotion Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.orgSaturday, 5 March 2011

×