Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
OCTOBER 11-14, 2016 • BOSTON, MA
Why is my Solr slow?
Mike Drob
Software Engineer, Cloudera
Who Am I?
●
Apache HTrace (Incubating) Committer
●
Software Engineer @ Cloudera
●
Middle School LEGO Robotics Mentor
Outline
●
The need for tracing
●
HTrace introduction
●
Examples with Solr
●
Finding the slow server!
Scenario
●
User complains that queries are “slow”
●
Caches?
●
Or faceting?
●
Or sorting?
●
Or...?
The State of the Art
Run a query with debug=timing:
"timing":{
"time":178.0,
"prepare":{"time":7.0,
"query":{"time":4.0}}
...
What is Apache HTrace
●
Distributed Tracing Library
●
Primitive Type is the “Span”
●
Unique ID
●
Source + Description
●
St...
Reference Architecture
Application
Trace
Gen
Application
Trace
Gen
Application
Trace
Gen
Application
Trace
Gen
Collector
Example Architecture
Solr
trace
Solr
trace
Solr
trace
Solr
trace
htraced
How Do I Trace?
●
Can Trace Any “Unit Of Work”
●
Method Calls
●
Threads
●
RPCs doQuery();
try (TraceScope scope = tracer.n...
What Does It Look Like?
● Trace of q=*:* and resulting sub-queries
● Data modeled on debug=timing
Aside: Benefits over debug=timing
●
Separate time-lines for each server
●
Ability to mark annotations
●
Data stored for la...
Something More Complex
techproducts/select?q=*:*&facet.field=popularity
And Other Commands
And Other Commands
Configuration
●
New section in solr.xml
<trace>
<str name="span.receiver.classes">org.apache.htrace.impl.HTracedSpanReceiv...
Performance
●
Each trace add ~25ns of overhead
●
Search handler can generate ~20 traces
●
Traces are buffered in memory
●
...
Future Work
●
Add tracing to clients (SolrJ)
●
Leverage MDC logging
●
Async Requests
Demo
Demo (Backup)
"responseHeader":{
"zkConnected":true,
"status":0,
"QTime":5251,
"params":{
"q":"*:*",
"indent":"on",
"wt":"...
Demo (Backup)
Demo (Backup)
Demo (Backup)
Demo (Backup)
Thank you
mdrob@apache.org
@mikhaildrob
Upcoming SlideShare
Loading in …5
×

Why Is My Solr Slow?: Presented by Mike Drob, Cloudera

737 views

Published on

Presented at Lucene/Solr Revolution 2016

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Why Is My Solr Slow?: Presented by Mike Drob, Cloudera

  1. 1. OCTOBER 11-14, 2016 • BOSTON, MA
  2. 2. Why is my Solr slow? Mike Drob Software Engineer, Cloudera
  3. 3. Who Am I? ● Apache HTrace (Incubating) Committer ● Software Engineer @ Cloudera ● Middle School LEGO Robotics Mentor
  4. 4. Outline ● The need for tracing ● HTrace introduction ● Examples with Solr ● Finding the slow server!
  5. 5. Scenario ● User complains that queries are “slow” ● Caches? ● Or faceting? ● Or sorting? ● Or...?
  6. 6. The State of the Art Run a query with debug=timing: "timing":{ "time":178.0, "prepare":{"time":7.0, "query":{"time":4.0}} "process":{"time":160.0, "query":{"time":118.0}, "facet":{"time":33.0}}
  7. 7. What is Apache HTrace ● Distributed Tracing Library ● Primitive Type is the “Span” ● Unique ID ● Source + Description ● Start/Stop Time ● Additional Metadata ● See Also: Dapper, Zipkin, Wingtips
  8. 8. Reference Architecture Application Trace Gen Application Trace Gen Application Trace Gen Application Trace Gen Collector
  9. 9. Example Architecture Solr trace Solr trace Solr trace Solr trace htraced
  10. 10. How Do I Trace? ● Can Trace Any “Unit Of Work” ● Method Calls ● Threads ● RPCs doQuery(); try (TraceScope scope = tracer.newScope(“work”)) { doQuery(); }
  11. 11. What Does It Look Like? ● Trace of q=*:* and resulting sub-queries ● Data modeled on debug=timing
  12. 12. Aside: Benefits over debug=timing ● Separate time-lines for each server ● Ability to mark annotations ● Data stored for later analysis ● Trace non-query requests ● Trace calls to other systems
  13. 13. Something More Complex techproducts/select?q=*:*&facet.field=popularity
  14. 14. And Other Commands
  15. 15. And Other Commands
  16. 16. Configuration ● New section in solr.xml <trace> <str name="span.receiver.classes">org.apache.htrace.impl.HTracedSpanReceiver</str> <str name="htraced.receiver.address">trace-server:9075</str> <str name="sampler.classes">org.apache.htrace.core.ProbabilitySampler</str> </trace> ● Sampler Options ● Advanced: Force Trace … traceid=<64-bit> … ● Advanced: Buffer Options
  17. 17. Performance ● Each trace add ~25ns of overhead ● Search handler can generate ~20 traces ● Traces are buffered in memory ● May consider multiple recievers
  18. 18. Future Work ● Add tracing to clients (SolrJ) ● Leverage MDC logging ● Async Requests
  19. 19. Demo
  20. 20. Demo (Backup) "responseHeader":{ "zkConnected":true, "status":0, "QTime":5251, "params":{ "q":"*:*", "indent":"on", "wt":"json", "_":"1476334957442"}},
  21. 21. Demo (Backup)
  22. 22. Demo (Backup)
  23. 23. Demo (Backup)
  24. 24. Demo (Backup)
  25. 25. Thank you mdrob@apache.org @mikhaildrob

×