Enterprise Search Summit - Speeding Up Search

729 views

Published on

In this presentation, Gil Tene, CTO of Azul Systems, discusses the benefits of using in-memory, in-process, and in-heap index representations with heap sizes that can now make full use of current commodity server scale. He also compares throughput and latency characteristics of the various architectural alternatives, using some of the configurable choices available in Apache Lucene™ 4.0 for specific examples.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
729
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
10
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Enterprise Search Summit - Speeding Up Search

  1. 1. Speeding up Enterprise Search with In-Memory Indexing Gil Tene, CTO & co-Founder, Azul Systems ©2011 Azul Systems, Inc.
  2. 2. This Talk’s Purpose / Goals Describe how in-memory indexing can help enterprise search Explain why in-memory and large memory use has been delayed Show how in-memory indexing is now (finally) practical for interactive search applications ©2011 Azul Systems, Inc.
  3. 3. About me: Gil Tene co-founder, CTO @Azul Systems Have been working on a “think different” GC approaches since 2002 Created Pauseless & C4 core GC algorithms (Tene, Wolf) A Long history building Virtual & Physical Machines, Operating Systems, Enterprise apps, etc... ©2011 Azul Systems, Inc. * working on real-world trash compaction issues, circa 2004
  4. 4. About Azul Vega We make scalable Virtual Machines Have built “whatever it takes to get job done” since 2002 3 generations of custom SMP Multi-core HW (Vega) Now Pure software for commodity x86 (Zing) C4 “Industry firsts” in Garbage collection, elastic memory, Java virtualization, memory scale ©2011 Azul Systems, Inc.
  5. 5. About Azul Supporting mission-critical deployments around ©2012 Azul Systems, Inc.
  6. 6. High level agenda How can in-memory help things? Why isn’t everyone just using it? The “Application Memory Wall” problem Some Garbage Collection banter What an actual solution looks like... ©2011 Azul Systems, Inc.
  7. 7. Memory use How many of you use heap sizes of: F more than ½ GB? F more than 1 GB? F more than 2 GB? F more than 4 GB? F more than 10 GB? F more than 20 GB? F more than 50 GB? ©2011 Azul Systems, Inc.
  8. 8. Why is In-Memory indexing interesting? ©2011 Azul Systems, Inc.
  9. 9. Interesting things happen when an entire problem fits in a single heap No more out-of-process access for “hot” stuff ~1 microsecond instead of ~1 millisecond access 10s of GB/sec instead of 100s of MB/sec rates Work elimination No more reading/parsing/deserializing per access No network or I/O load These can amount to dramatic speed benefits ©2011 Azul Systems, Inc.
  10. 10. Many business problems can now comfortably fit in server memory Product Catalogs are often in the 2-10GB size range Overnight reports are often done on 10-20GB data sets All of English Language Wikipedia text is ~29GB Indexing “bloats” things, but not by that much... A full index of English Language Wikipedia (including stored fields and term vectors fits for highlighting) fits in ~78GB. You’ll need even more than that in heap size... ©2011 Azul Systems, Inc.
  11. 11. CPU% Heap size vs. GC CPU % 100% Live set Heap size
  12. 12. Case in point: The impact of in-memory index in Lucene Lucene provides multiple index representation options SimpleFSDirectory, NIOFSDirectory (On-disk file-based) MMapDirectory (mapped files, better leverage of OS cache memory) RAMDirectory (In-process, in-heap) Comparing throughput (e.g. on English Lang. Wikipedia) RAMDirectory ~2-3x throughput vs. MMapDirectory Similar impact on average response. Even higher impact with promising things in Lucene 4.0 But average is not all that matters There is a good reason people aren’t doing this (yet...) ©2011 Azul Systems, Inc.
  13. 13. Why isn’t all search done with in-memory indexes already? ©2011 Azul Systems, Inc.
  14. 14. The “Application Memory Wall” ©2011 Azul Systems, Inc.
  15. 15. Memory use How many of you use heap sizes of: F more than ½ GB? F more than 1 GB? F more than 2 GB? F more than 4 GB? F more than 10 GB? F more than 20 GB? F more than 50 GB? ©2011 Azul Systems, Inc.
  16. 16. Reality check: servers in 2012 Retail prices, major web server store (US $, Oct 2012) 24 vCore, 128GB server ≈ $5K 24 vCore, 256GB server ≈ $8K 32 vCore, 384GB server ≈ $14K 48 vCore, 512GB server ≈ $19K 64 vCore, 1TB server ≈ $36K Cheap (< $1/GB/Month), and roughly linear to ~1TB 10s to 100s of GB/sec of memory bandwidth
  17. 17. The Application Memory Wall A simple observation: Application instances appear to be unable to make effective use of modern server memory capacities The size of application instances as a % of a server’s capacity is rapidly dropping ©2011 Azul Systems, Inc.
  18. 18. How much memory do applications need? “640KB ought to be enough for anybody” WRONG! So what’s the right number? 6,400K? 64,000K? 640,000K? 6,400,000K? 64,000,000K? There is no right number Target moves at 50x-100x per decade ©2011 Azul Systems, Inc. “I've said some stupid things and some wrong things, but not that. No one involved in computers would ever say that a certain amount of memory is enough for all time …” - Bill Gates, 1996
  19. 19. “Tiny” application history Assuming Moore’s Law means: 2010 “transistor counts grow at ≈2x every ≈18 months” It also means memory size grows ≈100x every 10 years 1990 1980 ??? GB apps on 256 GB Application Memory Wall 2000 1GB apps on a 2 – 4 GB server 10MB apps on a 32 – 64 MB server 100KB apps on a ¼ to ½ MB Server “Tiny”: would be “silly” to distribute ©2011 Azul Systems, Inc.
  20. 20. What is causing the Application Memory Wall? Garbage Collection is a clear and dominant cause There seem to be practical heap size limits for applications with responsiveness requirements [Virtually] All current commercial JVMs will exhibit a multi-second pause on a normally utilized 2-6GB heap. It’s a question of “When” and “How often”, not “If”. GC tuning only moves the “when” and the “how often” around Root cause: The link between scale and responsiveness ©2011 Azul Systems, Inc.
  21. 21. What quality of GC is responsible for the Application Memory Wall? It is NOT about overhead or efficiency: CPU utilization, bottlenecks, memory consumption and utilization It is NOT about speed Average speeds, 90%, even 95% speeds, are all perfectly fine It is NOT about minor GC events (right now) GC events in the 10s of msec are usually tolerable for most apps It is NOT about the frequency of very large pauses It is ALL about the worst observable pause behavior People avoid building/deploying visibly broken systems
  22. 22. GC Problems ©2011 Azul Systems, Inc.
  23. 23. Framing the discussion: Garbage Collection at modern server scales Modern Servers have 100s of GB of memory Each modern x86 core (when actually used) produces garbage at a rate of ¼ - ½ GB/sec + We want to make use of that memory and throughput in responsive applications Monolithic stop-the-world operations are the cause of the current Application Memory Wall Even if they are done “only a few times a day” ©2011 Azul Systems, Inc.
  24. 24. One way to deal with Monolithic-STW GC
  25. 25. Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" 99.99%" Max" Hiccup&Dura*on&(msec)& 18000" 16000" 14000" 12000" 10000" 8000" 6000" 4000" 2000" 0" 0" 2000" 4000" 6000" 8000" 10000" &Elapsed&Time&(sec)& Hiccups&by&Percen*le&Distribu*on& Hiccup&Dura*on&(msec)& 18000" 16000" Max=16023.552& 14000" 12000" 10000" 8000" 6000" 4000" 2000" 0" 0%" 90%" & 99%" 99.9%" 99.99%" & 99.999%" Percen*le& 99.9999%"
  26. 26. Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" 99.99%" Max" Hiccup&Dura*on&(msec)& 18000" 16000" 14000" 12000" 10000" 8000" 6000" 4000" 2000" 0" 0" 2000" 4000" 6000" 8000" 10000" &Elapsed&Time&(sec)& Hiccups&by&Percen*le&Distribu*on& Hiccup&Dura*on&(msec)& 18000" 16000" Max=16023.552& 14000" 12000" 10000" 8000" 6000" 4000" 2000" 0" 0%" 90%" & 99%" 99.9%" 99.99%" & 99.999%" Percen*le& 99.9999%"
  27. 27. Another way to cope: “Creative Language” “Guarantee a worst case of X msec, 99% of the time” “Mostly” Concurrent, “Mostly” Incremental Translation: “Will at times exhibit long monolithic stopthe-world pauses” “Fairly Consistent” Translation: “Will sometimes show results well outside this range” “Typical pauses in the tens of milliseconds” Translation: “Some pauses are much longer than tens of milliseconds” ©2012 Azul Systems, Inc.
  28. 28. Actually measuring things ©2012 Azul Systems, Inc.
  29. 29. Incontinuities in Java platform execution Hiccups"by"Time"Interval" Max"per"Interval" 99%" 99.90%" 99.99%" Max" Hiccup&Dura*on&(msec)& 1800" 1600" 1400" 1200" 1000" 800" 600" 400" 200" 0" 0" 200" 400" 600" 800" 1000" 1200" 1400" 1600" 1800" &Elapsed&Time&(sec)& Hiccups"by"Percen@le"Distribu@on" 1800" Hiccup&Dura*on&(msec)& 1600" Max=1665.024& 1400" 1200" 1000" 800" 600" 400" 200" 0" ©2012 Azul Systems, Inc. 0%" 90%" 99%" & 99.9%" & Percen*le& 99.99%" 99.999%"
  30. 30. But reality is simple People deal with GC problems primarily by keeping heap sizes artificially small. Keeping heaps small has translated to avoiding the use of in-process, in-memory contents That needs to change... ©2011 Azul Systems, Inc.
  31. 31. Solving the problems behind the Application Memory Wall ©2011 Azul Systems, Inc.
  32. 32. We needed to solve the right problems Focus on the causes of the Application Memory Wall Scale is artificially limited by responsiveness Responsiveness must be unlinked from scale: Data size, Throughput, Queries per second, Concurrent Users, ... Heap size, Live Set size, Allocation rate, Mutation rate, ... Responsiveness must be continually sustainable Can’t ignore “rare” events Eliminate all Stop-The-World Fallbacks At modern server scales, any Stop-The-World fallback is a failure ©2011 Azul Systems, Inc.
  33. 33. The GC problems that needed solving Robust Concurrent Marking In the presence of high throughput (mutation and allocation rates) Compaction that is not stop-the-world E.g. stay responsive while compacting 100+GB heaps Must be robust: not just a tactic to delay STW compaction Young-Gen that is not stop-the-world Stay responsive while promoting multi-GB data spikes Surprisingly little work done in this specific area ©2011 Azul Systems, Inc.
  34. 34. Azul’s “C4” Collector Continuously Concurrent Compacting Collector Concurrent guaranteed-single-pass marker Oblivious to mutation rate Concurrent ref (weak, soft, final) processing Concurrent Compactor Objects moved without stopping mutator References remapped without stopping mutator Can relocate entire generation (New, Old) in every GC cycle Concurrent, compacting old generation Concurrent, compacting new generation No stop-the-world fallback Always compacts, and always does so concurrently ©2011 Azul Systems, Inc.
  35. 35. Benefits ©2011 Azul Systems, Inc.
  36. 36. Sample responsiveness behavior ๏ SpecJBB + Slow churning 2GB LRU Cache ๏ Live set is ~2.5GB across all measurements ๏ Allocation rate is ~1.2GB/sec across all measurements ©2011 Azul Systems, Inc.
  37. 37. A JVM for Linux/x86 servers ELIMINATES Garbage Collection as a concern for enterprise applications Decouples scale metrics from response time concerns Transaction rate, data set size, concurrent users, heap size, allocation rate, mutation rate, etc. Leverages elastic memory for resilient operation ©2012 Azul Systems, Inc.
  38. 38. Sustainable Throughput: The throughput achieved while safely maintaining service levels ©2011 Azul Systems, Inc. Unsustainable Throughout
  39. 39. Instance capacity test: “Fat Portal” HotSpot CMS: Peaks at ~ 3GB / 45 concurrent users * LifeRay portal on JBoss @ 99.9% SLA of 5 second response times ©2011 Azul Systems, Inc.
  40. 40. Instance capacity test: “Fat Portal” C4: still smooth @ 800 concurrent users ©2012 Azul Systems, Inc.
  41. 41. Lucene: English Language Wikipedia RAMDirectory based index ©2011 Azul Systems, Inc.
  42. 42. Fun with jHiccup ©2012 Azul Systems, Inc.
  43. 43. Oracle HotSpot CMS, 1GB in an 8GB heap Zing 5, 1GB in an 8GB heap Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" Hiccups&by&Time&Interval& 99.99%" Max" Max"per"Interval" 12000" 10000" 8000" 6000" 4000" 2000" 0" 2000" 2500" Max" 20" 15" 10" 5" 500" 1000" 1500" 2000" 2500" 3000" 3500" 0" 500" 1000" &Elapsed&Time&(sec)& 1500" 3000" 3500" &Elapsed&Time&(sec)& Hiccups&by&Percen*le&Distribu*on& Hiccups&by&Percen*le&Distribu*on& 14000" 25" Max=13156.352& 12000" Hiccup&Dura*on&(msec)& Hiccup&Dura*on&(msec)& 99.99%" 0" 0" 10000" 8000" 6000" 4000" 2000" 0" 99.90%" 25" Hiccup&Dura*on&(msec)& Hiccup&Dura*on&(msec)& 14000" 99%" 0%" ©2012 Azul Systems, Inc. 90%" 99%" &99.9%" & Percen*le& 99.99%" 99.999%" 20" Max=20.384& 15" 10" 5" 0" 0%" 90%" 99%" & 99.9%" & Percen*le& 99.99%" 99.999%" 99.9999%"
  44. 44. Oracle HotSpot CMS, 1GB in an 8GB heap Zing 5, 1GB in an 8GB heap Hiccups&by&Time&Interval& Max"per"Interval" 99%" 99.90%" Hiccups&by&Time&Interval& 99.99%" Max" Max"per"Interval" 12000" 99.99%" Max" 12000" 10000" 8000" 6000" 4000" 2000" 0" 10000" 8000" 6000" 4000" 2000" 0" 0" 500" 1000" 1500" 2000" 2500" 3000" 3500" 0" 500" 1000" &Elapsed&Time&(sec)& 2000" 2500" 3000" 3500" Hiccups&by&Percen*le&Distribu*on& 14000" 14000" Hiccup&Dura*on&(msec)& Max=13156.352& 12000" 10000" 8000" 6000" 4000" 2000" 0" 1500" &Elapsed&Time&(sec)& Hiccups&by&Percen*le&Distribu*on& Hiccup&Dura*on&(msec)& 99.90%" 14000" Hiccup&Dura*on&(msec)& Hiccup&Dura*on&(msec)& 14000" 99%" 0%" 90%" 99%" &99.9%" & Percen*le& 99.99%" 99.999%" 12000" 10000" 8000" 6000" 4000" 2000" 0" 0%" 90%" Max=20.384& Drawn to scale ©2012 Azul Systems, Inc. 99%" & 99.9%" 99.99%" & Percen*le& 99.999%" 99.9999%"
  45. 45. Summary: In-memory indexing and search Interesting content can now fit in memory In-memory indexing makes things go fast In-memory indexing makes new things possible Change Batch to Interactive. Change business processes. It is finally practical to use in-memory indexing for interactive search applications Zing makes this work... ©2011 Azul Systems, Inc.
  46. 46. Q&A Lucene In-RAM English Lang. Wikipedia test: http:/ /blog.mikemccandless.com/2012/07/lucene-index-in-ram-withazuls-zing-jvm.html jHiccup: http:/ /www.azulsystems.com/dev_resources/jhiccup GC : G. Tene, B. Iyengar and M. Wolf C4: The Continuously Concurrent Compacting Collector In Proceedings of ISMM’11, ACM, pages 79-88 ©2011 Azul Systems, Inc.

×