What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)
Upcoming SlideShare
Loading in...5
×
 

What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012)

on

  • 3,304 views

At the core Infinispan is an in-memory data grid with options for multi node replication or distribution. It is also able to index and search stored data using Apache Lucene, or inverting the roles to ...

At the core Infinispan is an in-memory data grid with options for multi node replication or distribution. It is also able to index and search stored data using Apache Lucene, or inverting the roles to have Lucene store any index in the grid. This talk will illustrate how to get started and what benefits (or drawbacks!) you get by storing your Lucene indexes in Infinispan.

http://berlinbuzzwords.de/sessions/what-you-get-replicating-lucene-indexes-infinispan-data-grid

Statistics

Views

Total Views
3,304
Views on SlideShare
3,303
Embed Views
1

Actions

Likes
1
Downloads
31
Comments
0

1 Embed 1

http://us-w1.rockmelt.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012) What you get by replicating Lucene indexes on the Infinispan Data Grid (Berlin Buzzwords 2012) Presentation Transcript

    • 4 June 2012 Sanne Grinovero, Red HatWhat you get by replicating Lucene indexes on the Infinispan Data Grid
    • Who is that guy?• Sanne Grinovero • From this planet• Team Hibernate • Hibernate Search • Hibernate OGM• Team Infinispan • Infinispan Core • Infinispan Query• Apache Lucene, Netty, HotSpot, ANTLR, JGroups, Byteman, The Jokre
    • What are we talking about?• Apache Lucene• Infinispan • Integrations with Lucene ● Infinispan Lucene Directory
    • Apache Lucene ?
    • • An in-memory datagrid • Memory of multiple nodes • Cluster modes • CacheLoaders • Integrations with Lucene • Lucene Directory
    • Infinispan API?• Map-like key/value store • JSR 107 javax.cache.Cache interface • JSR 347 ??• Asynchronous API
    • In practice:cache.put( “user-34”, userInstance );cache.get( “user-34” );cache.remove( “user-34” );cache.putIfAbsent( “user-38”, other );
    • Distributed Data
    • Connected via JGroupsA Toolkit for Reliable Multicast Communication http://jgroups.org
    • Or remote clients via:• Memcached• REST• Hot Rod (Ruby, Python, C, C#, ...) • Netty
    • Consistent Hashing: DIST
    • Transactions!
    • JBoss AS7 core component• Cluster nodes autodiscovery• Session replication / failover• Hibernate second level cache• mod_cluster integration
    • In-memory volatile?Cache Stores: durability, warm caches,more capacity...• Cassandra• HBase• JDBC• Clouds (S3, ...)• Plain Old Files• Many more + custom
    • Back on Lucene:Single Writer lock
    • Queue-based clustering (filesystem index)
    • Lucene index storage
    • Index stored in Infinispan
    • Example architecture : JIRA / Scarlet
    • Hints• Some tuning options might have different effects than what youre used• Network is orders of magnitude faster than disk (YMMV) • But data locality helps • Balance resources• Get mergers to avoid segment chunking, or readlocks will engage
    • “benchmarks”, stats and more lies Write ops/sec Queries/sec RAMDirectory RAMDirectory Infinispan 0 Infinispan 0 Infinispan D4 Infinispan D4 queries per second Infinispan D40 Infinispan D40 FSDirectory FSDirectoryInfinispan Local Infinispan Local 0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000
    • Its not about the figures Write ops/sec Queries/sec RAMDirectory RAMDirectory Infinispan 0 Infinispan 0 Infinispan D4 Infinispan D4 queries per second Infinispan D40 Infinispan D40 FSDirectory FSDirectoryInfinispan Local Infinispan Local 0 50 100 150 200 250 300 350 400 0 5000 10000 15000 20000 25000
    • Whats next?• Infinispan (core) 5.2 and 6• Lucene 4.x• Dynamic chunk sizes• Ad-hoc “Lucene native” CacheStore • NIO byte buffers?
    • Conclusions• Quick index replication• Transactions• Not a replacements for shards• Cloud-friendly • Delegates to any storage
    • Q&Ahttp://infinispan.org @Infinispanhttp://in.relation.to @Hibernatehttp://jboss.org @SanneGrinovero