Uploaded byCodemotion
Hacking Infinispan: the new open source data grid meets NoSQL
The document discusses Infinispan, an open-source in-memory data grid developed by Red Hat that facilitates elastic data storage and scalability in cloud computing environments. It covers its architecture, features such as cloud readiness, low-latency access, and the importance of distributed data grids, as well as comparisons with traditional SQL and NoSQL databases. The author, Manik Surtani, emphasizes the significance of Data-as-a-Service (DaaS) and presents the advantages of using Infinispan for building adaptable and highly available data services.
More Related Content
Similar to Hacking Infinispan: the new open source data grid meets NoSQL
Hacking Infinispan: the new open source data grid meets NoSQL
- 1. Hacking Infinispan Open Source Data Grids meets NoSQL Manik Surtani Founder and Project Lead, Infinispan JBoss, by Red Hat Inc. Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 2. #infinispan Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 3. “Could data storagebe the one thing that hampers true cloud scalability and elasticity?” Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 4. Who is Manik? • R&D Engineer, Red Hat Inc. • Founder and project lead, Infinispan • Project lead, JBoss Cache • Frequent speaker on cloud computing and cloud data storage • http://twitter.com/maniksurtani • http://blog.infinispan.org Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 5. • What isData-as-a-Service? • Introducing Infinispan • Implementing Data-as-a-Service with Infinispan Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 6. Traditional 3-tier App Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 7. Typical IaaS App Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 8. Typical PaaS App Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 9. Where’s your datastored?? Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 10. Clouds are ephemeral! Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 11. Manik Surtani http://twitter.com/maniksurtanihttp://blog.infinispan.org Saturday, 5 March 2011
- 12. Virtualizing Data • Some public services do exist • Amazon RDS and SimpleDB • FathomDB • Cloundant • MongoHQ • etc. Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 13. What about privateclouds? • Not all cloud deployments are public! • Private cloud is very important • How can you build DaaS yourself? Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 14. Characteristics of DaaS • Elastic data • Needs to scale with other tiers • Response times should be linear • Needs to be highly available • Nodes will die! The service shouldn’t. Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 15. SQL and NoSQLDB • Lack of distribution hampers elasticity and HA • These limitations can be worked around • ... but this isn’t trivial • ... or cheap Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 16. SQL and NoSQLDB • Lack of distribution hampers elasticity and HA • These limitations can be worked around • ... but this isn’t trivial • ... or cheap Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 17. Distributed Data Grid • Far better suited to elastic data • Distributed by nature • Highly available by nature • A good building block for your data service Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 18. API is king • Apps should use their native data storage APIs • E.g., JPA (Java EE), ActiveRecord (Ruby), etc. • Key/value stores too low level • Akin to direct JDBC calls! Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 19. Introducing Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 20. • What isInfinispan? • Open source (LGPL) in-memory Data Grid • Written in Java and Scala • Some concepts from Amazon Dynamo • 2 usage modes • Embedded • Client-server • memcached, REST and Hot Rod Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 21. Embedded Mode Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 22. P2P Embedded Architecture Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 23. • API • Map-like key/value store • Upcoming JPA-like layer • Other high-level APIs being discussed in the community e.g., ActiveRecord Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 24. • Consistent hashbased distribution • Self healing • No single point of failure • Highly concurrent • MVCC locking Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 25. • Persistence • Not just in memory! • Write through and write behind • Pluggable “drivers” • Eviction and expiry • Efficient, adaptive algorithms • Addresses shortcomings of LRU & FIFO Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 26. • XA Transactions • 2-phase commit based • Deadlock detection algorithms • Coming soon: Atomic Broadcast Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 27. • Map/Reduce • In a pre-release state right now • Please try out 5.0.0.ALPHA2 with these APIs! • Querying • Using Lucene and Hibernate Search to index Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 28. Client/Server Mode Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 29. Client/Server Architecture Supported Protocols •REST •Memcached •Hot Rod Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 30. WTF is HotRod? • Wire protocol for client server communications • Open • Language independent • Built-in failover and load balancing • Smart routing Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 31. Server Endpoint Comparison Client Smart Load Balancing/ Protocol Clustered? Libraries Routing Failover Any HTTP load REST Text N/A Yes No balancer Only with predefined Memcached Text Plenty Yes No server list Currently Hot Rod Binary Yes Yes Dynamic only Java Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 32. So is Infinispana data grid? • In-memory • P2P, distributed • Low-latency, fast key/value store Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 33. So is Infinispana data grid? • In-memory • P2P, distributed ... or is it a NoSQL • Low-latency, fast key/value database? store • Persistence • Map/Reduce Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 34. So is Infinispana data grid? • In-memory • P2P, distributed ... or is it a NoSQL • Low-latency, fast key/value database? store • Persistence • Map/Reduce ... or something else? • Querying support • Transactional Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 35. So why isInfinispan sexy? Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 36. So why isInfinispan sexy? • Transparent horizontal scalability • Elastic in both directions • Fast, low latency data access • Ability to address a very large heap • Cloud-ready datastore • Not just for Java • Open Source Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 37. DaaS with Infinispan Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 38. Architecture Manage and Monitor Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 39. Where are weheaded? Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 40. • 4.0.0 Starobrno • Map-like API • Consistent Hash based distribution • Write-through, write-behind • Eviction, expiration • Management tooling • REST API • Hibernate 2nd Level Cache • Released Feb 2010 Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 41. • 4.1.0 Radegast • Deadlock detection • Client/Server protocols • Memcached • Hot Rod • Smart clients using Hot Rod • Lucene Directory implementation • LIRS: adaptive, recency-based eviction policies • Released August 2010 Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 42. • 5.0.0 Pagoa • JPA-like API • Fine-grained replication • Distributed code execution • Map/reduce • Virtual nodes for more even distribution • In active development Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 43. What have welearned? • Elastic data is hard • Public data services not always suitable • Data grids make elastic storage easy • Infinispan server endpoints help build elastic data tiers Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011
- 44. Questions? • http://www.infinispan.org • http://blog.infinispan.org • http://github.com/infinispan • http://twitter.com/infinispan • #infinispan on FreeNode • http://bit.ly/rate_me_codemotion Manik Surtani http://twitter.com/maniksurtani http://blog.infinispan.org Saturday, 5 March 2011