More Related Content
Similar to What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)
Similar to What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012) (20)
Recently uploaded
Recently uploaded (20)
What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)
- 1. 4 June 2012 Sanne Grinovero, Red Hat What you get by replicating Lucene indexes on the Infinispan Data Grid
- 2. Who is that guy? • Sanne Grinovero • From this planet • Team Hibernate • Hibernate Search • Hibernate OGM • Team Infinispan • Infinispan Core • Infinispan Query • Apache Lucene, Netty, HotSpot, ANTLR, JGroups, Byteman, The Jokre
- 3. What are we talking about? • Apache Lucene • Infinispan • Integrations with Lucene ● Infinispan Lucene Directory
- 5. • An in-memory datagrid • Memory of multiple nodes • Cluster modes • CacheLoaders • Integrations with Lucene • Lucene Directory
- 6. Infinispan API? • Map-like key/value store • JSR 107 javax.cache.Cache interface • JSR 347 ?? • Asynchronous API
- 7. In practice: cache.put( “user-34”, userInstance ); cache.get( “user-34” ); cache.remove( “user-34” ); cache.putIfAbsent( “user-38”, other );
- 9. Connected via JGroups A Toolkit for Reliable Multicast Communication http://jgroups.org
- 10. Or remote clients via: • Memcached • REST • Hot Rod (Ruby, Python, C, C#, ...) • Netty
- 12. Transactions!
- 13. JBoss AS7 core component • Cluster nodes autodiscovery • Session replication / failover • Hibernate second level cache • mod_cluster integration
- 14. In-memory volatile? Cache Stores: durability, warm caches, more capacity... • Cassandra • HBase • JDBC • Clouds (S3, ...) • Plain Old Files • Many more + custom
- 15. Back on Lucene: Single Writer lock
- 16. Queue-based clustering (filesystem index)
- 19. Index stored in Infinispan
- 20. Example architecture : JIRA / Scarlet
- 21. Hints • Some tuning options might have different effects than what you're used • Network is orders of magnitude faster than disk (YMMV) • But data locality helps • Balance resources • Get mergers to avoid segment chunking, or readlocks will engage
- 22. “benchmarks”, stats and more lies Write ops/sec Queries/sec RAMDirectory RAMDirectory Infinispan 0 Infinispan 0 Infinispan D4 Infinispan D4 queries per second Infinispan D40 Infinispan D40 FSDirectory FSDirectory Infinispan Local Infinispan Local 0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000
- 23. It's not about the figures Write ops/sec Queries/sec RAMDirectory RAMDirectory Infinispan 0 Infinispan 0 Infinispan D4 Infinispan D4 queries per second Infinispan D40 Infinispan D40 FSDirectory FSDirectory Infinispan Local Infinispan Local 0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000
- 24. What's next? • Infinispan (core) 5.2 and 6 • Lucene 4.x • Dynamic chunk sizes • Ad-hoc “Lucene native” CacheStore • NIO byte buffers?
- 25. Conclusions • Quick index replication • Transactions • Not a replacements for shards • Cloud-friendly • Delegates to any storage
- 26. Q&A http://infinispan.org @Infinispan http://in.relation.to @Hibernate http://jboss.org @SanneGrinovero