Query log analytics
28.11.2013
Niels Henrik Hagen

OSLO STOCKHOLM

LONDON

BOSTON
Who am I?
Niels Henrik Hagen
@nhhagen
Search enthusiast
Analyzing query logs
brings insight into how a
search application is used,
use that insight to
improve the search
application
What is a query log?
{"message":{"time":3016,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystri...
What is a query log made up of ?
•
•
•
•
•
•

Timestamp
Query text
Activated facets
Pagination
Number of hits
…
How to understand
the content of a log
file?
Visualization!
Where to find the log data?
How to make text into a graph?
Logstash
log crawling, parsing and processing
ElasticSearch
distributed restful search and analytics
Kibana
real-time analysis of streaming data
Simplified architecture
The result
What could be next?
Pieces missing from the puzzle
Track clicks in the result set
– What was clicked and what was not clicked
– Try to calculate how relevant returned result...
Track multiple queries across a “session”
– Data could be used to expand queries
Thank you
Niels Henrik Hagen
@nhhagen
Query log analytics - using logstash, elasticsearch and kibana 28.11.2013
Query log analytics - using logstash, elasticsearch and kibana 28.11.2013
Query log analytics - using logstash, elasticsearch and kibana 28.11.2013
Query log analytics - using logstash, elasticsearch and kibana 28.11.2013
Query log analytics - using logstash, elasticsearch and kibana 28.11.2013
Query log analytics - using logstash, elasticsearch and kibana 28.11.2013
Upcoming SlideShare
Loading in …5
×

Query log analytics - using logstash, elasticsearch and kibana 28.11.2013

2,696 views

Published on

As a search consultant I need to understand how a search application is used with the end goal of providing a better search experience for the end user. That story can come from many places and part of that story can be found in the query logs.

Blog post about the same topic: http://nhhagen.wordpress.com/2013/11/28/query-log-analysis-using-logstash-elasticsearch-and-kibana/

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,696
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
58
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • This example is preformatted in JSON, but often the data is in plain text.
  • And more
  • GraphsQuery load over time Average query latency over timeTop Queries Top Queries with 0 hits Top search modes top refiners/facets usedChanges in query load pct. last hour, last day, last 30dRefiner/facet usage over time
  • GraphsQuery latency distribution count and pct Average query latency over timeQuery latency distribution count and pct over timeTable: queries over 200ms latency (really slow queries)
  • A “typical” search applicationThe user sends a query to the search applicationThat query finds its way to the search application APIThe logic in the application is set up to do 3 parallel queries against the search engines for different types of data
  • Query logs can be taken from the search engine, but are not the query log for the user, they are technical query logsThose logs can be analyzed to figure out when you need new servers, more RAM, CPU etc
  • When you create the query log in the search application API, just before returning to the user, you can put the context of the query and “sum” into the query log
  • For now only used to read the JSON log files and transport the log events to elasticsearch
  • Indexes all the log events in per day indices
  • Queries elasticsearch and builds graphs using facets. Provides an interactive application to generate graphs on the fly
  • Very simplified architecture
  • GraphsQuery load over time Average query latency over timeTop Queries Top Queries with 0 hits Top search modes top refiners/facets usedChanges in query load pct. last hour, last day, last 30dRefiner/facet usage over time
  • GraphsQuery latency distribution count and pct Average query latency over timeQuery latency distribution count and pct over timeTable: queries over 200ms latency (really slow queries)
  • Query log analytics - using logstash, elasticsearch and kibana 28.11.2013

    1. 1. Query log analytics 28.11.2013 Niels Henrik Hagen OSLO STOCKHOLM LONDON BOSTON
    2. 2. Who am I? Niels Henrik Hagen @nhhagen Search enthusiast
    3. 3. Analyzing query logs brings insight into how a search application is used,
    4. 4. use that insight to improve the search application
    5. 5. What is a query log?
    6. 6. {"message":{"time":3016,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:07:02.011+00:00","@version":1,"source_host":"search3.kaw. {"message":{"time":492,"hitCounts":{"directmail":0,"refiners":0,"products":5},"searchQuery":{"attributes":{"querystring":"fisk","sortby":"kawdefault","sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:07:19.714+00:00","@version":1,"source_host":"search3.kaw.no {"message":{"time":459,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:11:37.852+00:00","@version":1,"source_host":"sea {"message":{"time":157,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112530","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:11:43.858+00:00","@version":1,"source_ho {"message":{"time":155,"hitCounts":{"products":2524},"searchQuery":{"attributes":{"querystring":"skjenk hyller og kommoder ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"112530","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:11:44.810+00:00","@version":1,"s {"message":{"time":252,"hitCounts":{"directmail":0,"refiners":0,"products":5},"searchQuery":{"attributes":{"querystring":"fisk","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:12:03.340+00:00","@version":1,"source_host":"sear {"message":{"time":460,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:12:47.508+00:00","@version":1,"source_host":"sea {"message":{"time":303,"hitCounts":{"directmail":5,"refiners":0,"products":122},"searchQuery":{"attributes":{"querystring":"leker","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:12:49.918+00:00","@version":1,"source_host":"s {"message":{"time":386,"hitCounts":{"directmail":1,"refiners":0,"products":91},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":"E {"message":{"time":22,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112173","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:13:19.196+00:00","@version":1,"source_hos {"message":{"time":152,"hitCounts":{"products":58},"searchQuery":{"attributes":{"querystring":"peisovn jøtul ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"112173","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:13:21.289+00:00","@version":1,"source_host":"s {"message":{"time":46,"hitCounts":{"directmail":0},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1201","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:13:54.993+00:00","@version":1,"source_host":"sea {"message":{"time":292,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:14:08.393+00:00","@version":1,"source_host":"sea {"message":{"time":335,"hitCounts":{"directmail":0,"refiners":0,"products":8},"searchQuery":{"attributes":{"querystring":"dyner","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:14:20.384+00:00","@version":1,"source_host":"sea {"message":{"time":326,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:14:36.072+00:00","@version":1,"source_host":"sea {"message":{"time":39,"hitCounts":{"directmail":0},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1201","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:14:42.194+00:00","@version":1,"source_host":"sea {"message":{"time":289,"hitCounts":{"directmail":2,"refiners":0,"products":222},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":" {"message":{"time":333,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:15:22.236+00:00","@version":1,"source_host":"sea {"message":{"time":316,"hitCounts":{"directmail":2,"refiners":0,"products":47},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"dmcategory","displayName":"Kategorier","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{ {"message":{"time":24,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1447","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:15:55.959+00:00","@version":1,"source_host":"sea {"message":{"time":294,"hitCounts":{"directmail":0,"refiners":0,"products":12},"searchQuery":{"attributes":{"querystring":"dusjkabinett","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:16:13.633+00:00","@version":1,"source_ho {"message":{"time":22,"hitCounts":{"directmail":0},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1023","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:16:15.842+00:00","@version":1,"source_host":"sea {"message":{"time":260,"hitCounts":{"directmail":0,"refiners":0,"products":0,"expiredproducts":0},"searchQuery":{"attributes":{"querystring":"cd591700627no","sortby":"kawdefault","hits":30,"sortorder":"","offset":0}," metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:16:30.752+00:00","@v {"message":{"time":20,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1447","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:16:31.534+00:00","@version":1,"source_host":"sea {"message":{"time":26,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112597","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:16:34.619+00:00","@version":1,"source_hos {"message":{"time":140,"hitCounts":{"products":132},"searchQuery":{"attributes":{"querystring":"dusjkabinett baderomsinnredning ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"112597","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:16:37.566+00:00","@version" {"message":{"time":298,"hitCounts":{"directmail":0,"refiners":0,"products":30},"searchQuery":{"attributes":{"querystring":"kjøkkenutstyr","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:16:42.008+00:00","@version":1,"source_h {"message":{"time":20,"hitCounts":{"directmail":0},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1062","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:16:46.663+00:00","@version":1,"source_host":"sea {"message":{"time":22,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1447","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:16:52.785+00:00","@version":1,"source_host":"sea {"message":{"time":20,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"111811","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:17:02.328+00:00","@version":1,"source_hos {"message":{"time":146,"hitCounts":{"products":135},"searchQuery":{"attributes":{"querystring":"dusjkabinett element baderomsinnredning ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"111811","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:17:04.688+00:00","@ {"message":{"time":22,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1447","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:17:04.729+00:00","@version":1,"source_host":"sea {"message":{"time":22,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1447","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:17:10.554+00:00","@version":1,"source_host":"sea {"message":{"time":209,"hitCounts":{"products":135},"searchQuery":{"attributes":{"querystring":"dusjkabinett element baderomsinnredning ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"111811","offset":9},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:17:26.784+00:00","@ {"message":{"time":324,"hitCounts":{"directmail":2,"refiners":0,"products":47},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"dmcategory","displayName":"Kategorier","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{ {"message":{"time":21,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112601","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:18:08.067+00:00","@version":1,"source_hos {"message":{"time":145,"hitCounts":{"products":132},"searchQuery":{"attributes":{"querystring":"dusjkabinett baderomsinnredning ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"112601","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:18:10.400+00:00","@version" {"message":{"time":21,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"111675","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:18:29.017+00:00","@version":1,"source_hos {"message":{"time":349,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:18:31.404+00:00","@version":1,"source_host":"sea {"message":{"time":338,"hitCounts":{"directmail":1,"refiners":0,"products":59},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":" {"message":{"time":22,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"106264","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:19:26.681+00:00","@version":1,"source_hos {"message":{"time":561,"hitCounts":{"directmail":0,"refiners":0,"products":174},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"dmcategory","displayName":"Kategorier","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[ {"message":{"time":371,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:20:58.128+00:00","@version":1,"source_host":"sea {"message":{"time":415,"hitCounts":{"directmail":0,"refiners":0,"products":11,"expiredproducts":11},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"dmcategory","displayName":"Kategorier","interval":fals e,"upperValue":null,"lowerValue {"message":{"time":161,"hitCounts":{"directmail":0,"refiners":0,"products":7},"searchQuery":{"attributes":{"querystring":"snøfreser","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:22:29.636+00:00","@version":1,"source_host": {"message":{"time":21,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"82298","offset":0,"filters ":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:22:31.727+00:00","@version":1,"source_host {"message":{"time":318,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:22:44.942+00:00","@version":1,"source_host":"sea {"message":{"time":431,"hitCounts":{"directmail":8,"refiners":0,"products":474,"expiredproducts":474},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null," {"message":{"time":584,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:23:24.424+00:00","@version":1,"source_host":"sea {"message":{"time":18,"hitCounts":{"directmail":0},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1201","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:23:41.415+00:00","@version":1,"source_host":"sea {"message":{"time":19,"hitCounts":{"directmail":0},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1201","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:23:41.696+00:00","@version":1,"source_host":"sea {"message":{"time":375,"hitCounts":{"directmail":1,"refiners":0,"products":113,"expiredproducts":113},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null," {"message":{"time":298,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:24:05.682+00:00","@version":1,"source_host":"sea {"message":{"time":265,"hitCounts":{"directmail":0,"refiners":0,"products":3},"searchQuery":{"attributes":{"querystring":"nettbrett samsung","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:24:05.936+00:00","@version":1,"sourc {"message":{"time":331,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:24:08.566+00:00","@version":1,"source_host":"sea {"message":{"time":333,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:25:18.267+00:00","@version":1,"source_host":"sea {"message":{"time":245,"hitCounts":{"directmail":2,"refiners":0,"products":222},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":" {"message":{"time":286,"hitCounts":{"directmail":0,"refiners":0,"products":100},"searchQuery":{"attributes":{"querystring":"sofa","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:26:36.636+00:00","@version":1,"source_host":"se {"message":{"time":303,"hitCounts":{"directmail":2,"refiners":0,"products":222},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":" {"message":{"time":305,"hitCounts":{"directmail":0,"refiners":0,"products":100},"searchQuery":{"attributes":{"querystring":"sofa","sortby":"kawdefault","hits":30,"sortorder":"","offset":30},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:26:57.286+00:00","@version":1,"source_host":"s {"message":{"time":30,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112781","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:07.063+00:00","@version":1,"source_hos {"message":{"time":22,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1423","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:10.700+00:00","@version":1,"source_host":"sea {"message":{"time":24,"hitCounts":{"directmail":0},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"100610","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:10.901+00:00","@version":1,"source_hos {"message":{"time":23,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"108910","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:11.605+00:00","@version":1,"source_hos {"message":{"time":554,"hitCounts":{"directmail":3,"refiners":0,"products":180,"expiredproducts":180},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null," {"message":{"time":25,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"113365","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:15.965+00:00","@version":1,"source_hos {"message":{"time":220,"hitCounts":{"products":99},"searchQuery":{"attributes":{"querystring":"buesofa sofa ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"113365","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:17.322+00:00","@version":1,"source_host":"se {"message":{"time":21,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"113325","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:33.139+00:00","@version":1,"source_hos {"message":{"time":51,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"113419","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:39.825+00:00","@version":1,"source_hos {"message":{"time":20,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"113425","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:46.738+00:00","@version":1,"source_hos {"message":{"time":22,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"113319","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:27:53.654+00:00","@version":1,"source_hos {"message":{"time":41,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112136","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:28:00.294+00:00","@version":1,"source_hos {"message":{"time":20,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112473","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:28:07.574+00:00","@version":1,"source_hos {"message":{"time":23,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112434","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:28:14.495+00:00","@version":1,"source_hos {"message":{"time":125,"hitCounts":{"directmail":0,"refiners":0,"products":7},"searchQuery":{"attributes":{"querystring":"snøfreser","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:28:47.808+00:00","@version":1,"source_host": {"message":{"time":22,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"113135","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:28:56.820+00:00","@version":1,"source_hos {"message":{"time":224,"hitCounts":{"directmail":2,"refiners":0},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":""Jula Norge AS {"message":{"time":27,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"109528","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:30:29.552+00:00","@version":1,"source_hos {"message":{"time":181,"hitCounts":{"products":59},"searchQuery":{"attributes":{"querystring":"panelovn adax husholdningsapparater ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"109528","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:30:31.191+00:00","@versi {"message":{"time":31,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1397","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:30:37.338+00:00","@version":1,"source_host":"sea {"message":{"time":79,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112411","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:30:46.252+00:00","@version":1,"source_hos {"message":{"time":159,"hitCounts":{"products":2524},"searchQuery":{"attributes":{"querystring":"panelovn adax oppvarming og ventilasjon ovn ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"112411","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:30:47.651+00:0 {"message":{"time":25,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1452","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:31:00.325+00:00","@version":1,"source_host":"sea {"message":{"time":25,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1452","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:31:05.232+00:00","@version":1,"source_host":"sea {"message":{"time":34,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1452","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:31:08.092+00:00","@version":1,"source_host":"sea {"message":{"time":28,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1452","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:31:14.208+00:00","@version":1,"source_host":"sea {"message":{"time":24,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1452","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:31:18.324+00:00","@version":1,"source_host":"sea {"message":{"time":25,"hitCounts":{"directmail":0},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"99828","offset":0,"filters ":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:31:43.469+00:00","@version":1,"source_host {"message":{"time":316,"hitCounts":{"directmail":0,"refiners":0,"products":89},"searchQuery":{"attributes":{"querystring":"verktøy","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:31:45.228+00:00","@version":1,"source_host":" {"message":{"time":28,"hitCounts":{"directmail":0},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"84401","offset":0,"filters ":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:31:46.311+00:00","@version":1,"source_host {"message":{"time":248,"hitCounts":{"directmail":2,"refiners":0,"products":222},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":" {"message":{"time":305,"hitCounts":{"directmail":0,"refiners":0,"products":89},"searchQuery":{"attributes":{"querystring":"verktøy","sortby":"kawdefault","hits":30,"sortorder":"","offset":30},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:32:05.858+00:00","@version":1,"source_host": {"message":{"time":228,"hitCounts":{"directmail":2,"refiners":0,"products":511},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":" {"message":{"time":279,"hitCounts":{"directmail":2,"refiners":0,"products":222},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":" {"message":{"time":252,"hitCounts":{"directmail":0,"refiners":0,"products":35},"searchQuery":{"attributes":{"querystring":"data","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:32:38.465+00:00","@version":1,"source_host":"sea {"message":{"time":280,"hitCounts":{"directmail":2,"refiners":0,"products":511},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":" {"message":{"time":20,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112424","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:33:13.364+00:00","@version":1,"source_hos {"message":{"time":333,"hitCounts":{"products":2524},"searchQuery":{"attributes":{"querystring":"takpanel s�dra gulv, tak og vegg ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"112424","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:33:20.508+00:00","@versi {"message":{"time":26,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"82112","offset":0,"filters ":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:33:28.893+00:00","@version":1,"source_host {"message":{"time":229,"hitCounts":{"directmail":0,"refiners":0,"products":100},"searchQuery":{"attributes":{"querystring":"sofa","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:33:48.443+00:00","@version":1,"source_host":"se {"message":{"time":211,"hitCounts":{"directmail":0,"refiners":0,"products":7},"searchQuery":{"attributes":{"querystring":"snøfreser","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:33:51.851+00:00","@version":1,"source_host": {"message":{"time":418,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:34:08.625+00:00","@version":1,"source_host":"sea {"message":{"time":25,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"82183","offset":0,"filters ":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:34:16.094+00:00","@version":1,"source_host {"message":{"time":314,"hitCounts":{"directmail":27,"refiners":0,"products":2529},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:34:20.807+00:00","@version":1,"source_host":"sea {"message":{"time":21,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"112432","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:34:35.963+00:00","@version":1,"source_hos {"message":{"time":157,"hitCounts":{"products":64},"searchQuery":{"attributes":{"querystring":"snøfreser peisovn ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"112432","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:34:37.153+00:00","@version":1,"source_host {"message":{"time":365,"hitCounts":{"directmail":1,"refiners":0,"products":99},"searchQuery":{"attributes":{"querystring":"ǂ","sortby":"kawdefault","hits":30,"sortorder":"","navigators":[{"key":"shop","displayName":"Butikk","interval":false,"upperValue":null,"lowerValue":null,"modifiers":[{"value":" {"message":{"time":19,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","offset":0,"dmid":"1452","filters":[]}, "metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:35:12.913+00:00","@version":1,"source_host":"sea {"message":{"time":89,"hitCounts":{"directmail":1},"searchQuery":{"attributes":{"hits":1,"querystring":"","sortby":"","navigators":[],"sortorder":"","productid":"109114","offset":0,"filter s":[]},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:35:21.318+00:00","@version":1,"source_hos {"message":{"time":179,"hitCounts":{"products":59},"searchQuery":{"attributes":{"querystring":"panelovn nobø husholdningsapparater ","sortby":"kawdefault","hits":9,"sortorder":"","similarto":"109114","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:35:23.062+00:00","@versi {"message":{"time":139,"hitCounts":{"directmail":0,"refiners":0,"products":7},"searchQuery":{"attributes":{"querystring":"snøfreser","sortby":"kawdefault","hits":30,"sortorder":"","offset":0},"metaAttributes":{"queryModel":{}}}},"@timestamp":"2013-10-22T11:35:28.482+00:00","@version":1,"source_host":
    7. 7. What is a query log made up of ? • • • • • • Timestamp Query text Activated facets Pagination Number of hits …
    8. 8. How to understand the content of a log file?
    9. 9. Visualization!
    10. 10. Where to find the log data?
    11. 11. How to make text into a graph?
    12. 12. Logstash log crawling, parsing and processing
    13. 13. ElasticSearch distributed restful search and analytics
    14. 14. Kibana real-time analysis of streaming data
    15. 15. Simplified architecture
    16. 16. The result
    17. 17. What could be next? Pieces missing from the puzzle
    18. 18. Track clicks in the result set – What was clicked and what was not clicked – Try to calculate how relevant returned results are
    19. 19. Track multiple queries across a “session” – Data could be used to expand queries
    20. 20. Thank you Niels Henrik Hagen @nhhagen

    ×