Your SlideShare is downloading. ×
What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)

2,902
views

Published on

At the core Infinispan is an in-memory data grid with options for multi node replication or distribution. It is also able to index and search stored data using Apache Lucene, or inverting the roles to …

At the core Infinispan is an in-memory data grid with options for multi node replication or distribution. It is also able to index and search stored data using Apache Lucene, or inverting the roles to have Lucene store any index in the grid. This talk will illustrate how to get started and what benefits (or drawbacks!) you get by storing your Lucene indexes in Infinispan.

http://berlinbuzzwords.de/sessions/what-you-get-replicating-lucene-indexes-infinispan-data-grid

Published in: Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,902
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
41
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 4 June 2012 Sanne Grinovero, Red HatWhat you get by replicating Lucene indexes on the Infinispan Data Grid
  • 2. Who is that guy?• Sanne Grinovero • From this planet• Team Hibernate • Hibernate Search • Hibernate OGM• Team Infinispan • Infinispan Core • Infinispan Query• Apache Lucene, Netty, HotSpot, ANTLR, JGroups, Byteman, The Jokre
  • 3. What are we talking about?• Apache Lucene• Infinispan • Integrations with Lucene ● Infinispan Lucene Directory
  • 4. Apache Lucene ?
  • 5. • An in-memory datagrid • Memory of multiple nodes • Cluster modes • CacheLoaders • Integrations with Lucene • Lucene Directory
  • 6. Infinispan API?• Map-like key/value store • JSR 107 javax.cache.Cache interface • JSR 347 ??• Asynchronous API
  • 7. In practice:cache.put( “user-34”, userInstance );cache.get( “user-34” );cache.remove( “user-34” );cache.putIfAbsent( “user-38”, other );
  • 8. Distributed Data
  • 9. Connected via JGroupsA Toolkit for Reliable Multicast Communication http://jgroups.org
  • 10. Or remote clients via:• Memcached• REST• Hot Rod (Ruby, Python, C, C#, ...) • Netty
  • 11. Consistent Hashing: DIST
  • 12. Transactions!
  • 13. JBoss AS7 core component• Cluster nodes autodiscovery• Session replication / failover• Hibernate second level cache• mod_cluster integration
  • 14. In-memory volatile?Cache Stores: durability, warm caches,more capacity...• Cassandra• HBase• JDBC• Clouds (S3, ...)• Plain Old Files• Many more + custom
  • 15. Back on Lucene:Single Writer lock
  • 16. Queue-based clustering (filesystem index)
  • 17. Lucene index storage
  • 18. Index stored in Infinispan
  • 19. Example architecture : JIRA / Scarlet
  • 20. Hints• Some tuning options might have different effects than what youre used• Network is orders of magnitude faster than disk (YMMV) • But data locality helps • Balance resources• Get mergers to avoid segment chunking, or readlocks will engage
  • 21. “benchmarks”, stats and more lies Write ops/sec Queries/sec RAMDirectory RAMDirectory Infinispan 0 Infinispan 0 Infinispan D4 Infinispan D4 queries per second Infinispan D40 Infinispan D40 FSDirectory FSDirectoryInfinispan Local Infinispan Local 0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000
  • 22. Its not about the figures Write ops/sec Queries/sec RAMDirectory RAMDirectory Infinispan 0 Infinispan 0 Infinispan D4 Infinispan D4 queries per second Infinispan D40 Infinispan D40 FSDirectory FSDirectoryInfinispan Local Infinispan Local 0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000
  • 23. Whats next?• Infinispan (core) 5.2 and 6• Lucene 4.x• Dynamic chunk sizes• Ad-hoc “Lucene native” CacheStore • NIO byte buffers?
  • 24. Conclusions• Quick index replication• Transactions• Not a replacements for shards• Cloud-friendly • Delegates to any storage
  • 25. Q&Ahttp://infinispan.org @Infinispanhttp://in.relation.to @Hibernatehttp://jboss.org @SanneGrinovero

×