Realtime Analytics With Elasticsearch [New Media Inspiration 2013]

Real time analyticsof big data with ElasticsearchKarel Minařík

cets Fa ly tics SON AnaJ http://www.youtube.com/watch?v=-GftBySG99Q

http://karmi.czhttp://elasticsearch.com Realtime Analytics With ElasticSearch

Using a search engine for analytics?wat? Realtime Analytics With ElasticSearch

HOW DOES SEARCH WORK?A collection of documents file_1.txt The ruby is a pink to blood-‐red colored gemstone ... file_2.txt Ruby is a dynamic, reflective, general-‐purpose object-‐oriented programming language ... file_3.txt "Ruby" is a song by English rock band Kaiser Chiefs ...

HOW DOES SEARCH WORK?How do you search documents?File.read(file_1.txt).include?(ruby)File.read(file_2.txt).include?(ruby)...

HOW DOES SEARCH WORK?The inverted indexTOKENS POSTINGS ruby file_1.txt file_2.txt file_3.txt pink file_1.txt gemstone file_1.txt dynamic file_2.txt reflective file_2.txt programming file_2.txt song file_3.txt english file_3.txt rock file_3.txthttp://en.wikipedia.org/wiki/Index_(search_engine)#Inverted_indices

HOW DOES SEARCH WORK?The inverted indexsearch "ruby" ruby file_1.txt file_2.txt file_3.txt pink file_1.txt gemstone file_1.txt dynamic file_2.txt reflective file_2.txt programming file_2.txt song file_3.txt english file_3.txt rock file_3.txthttp://en.wikipedia.org/wiki/Index_(search_engine)#Inverted_indices

HOW DOES SEARCH WORK?The inverted indexsearch "song" ruby file_1.txt file_2.txt file_3.txt pink file_1.txt gemstone file_1.txt dynamic file_2.txt reflective file_2.txt programming file_2.txt song file_3.txt english file_3.txt rock file_3.txthttp://en.wikipedia.org/wiki/Index_(search_engine)#Inverted_indices

HOW DOES SEARCH WORK?The inverted indexsearch "ruby AND song" ruby file_1.txt file_2.txt file_3.txt pink file_1.txt gemstone file_1.txt dynamic file_2.txt reflective file_2.txt programming file_2.txt song file_3.txt english file_3.txt rock file_3.txthttp://en.wikipedia.org/wiki/Index_(search_engine)#Inverted_indices

HOW DOES SEARCH WORK?The inverted indexTOKENS POSTINGS Statistics! ruby 3 file_1.txt file_2.txt file_3.txt pink 1 file_1.txt gemstone file_1.txt dynamic file_2.txt reflective file_2.txt programming file_2.txt song file_3.txt english file_3.txt rock file_3.txthttp://en.wikipedia.org/wiki/Index_(search_engine)#Inverted_indices

http://elasticsearch.org

ElasticSearch is an open source, scalable,distributed, cloud-ready, highly-available full-text search engine and database with powerfulaggregation features, communicating by JSONover RESTful HTTP, based on ApacheLucene. Realtime Analytics With ElasticSearch

FACETS Faceted NavigationQueryFacets http://blog.linkedin.com/2009/12/14/linkedin-faceted-search/

FACETSFaceted Navigation with Elasticsearchcurl "http://localhost:9200/people/_search?pretty=true" -‐d { "query" : { "match" : { "name" : "John"} User query }, "filter" : { "terms" : { "employer" : ["IBM"] } “Checkboxes” }, "facets" : { "employer" : { "terms" : { Facets "field" : "employer", "size" : 3 } "facets" : { } "employer" : { } "missing" : 0,} "total" : 10, "other" : 3, "terms" : [ { "term" : "ibm", Response "count" : 3 }, { "term" : "twitter", "count" : 2 }, { "term" : "apple", "count" : 2 } ] } }http://www.elasticsearch.org/guide/reference/api/search/facets/index.html

FACETSVisualizing the Facets "facets" : { "employer" : { "missing" : 0, "total" : 10, "other" : 3, "terms" : [ { "term" : "ibm", "count" : 3 }, { "term" : "twitter", "count" : 2 }, { "term" : "apple", "count" : 2 } ] DEMO: http://bl.ocks.org/4571766 } } d3.js ~ A Bar Chart, Part 1 http://mbostock.github.com/d3/tutorial/bar-1.html

FACETSVisualizing the Facets

FACETSVisualizing the Facetshttp://demo.kibana.org

Important Concepts‣ No batch orientation‣ No stats precomputation and caching‣ No predefined metrics or schemas‣ Combination of free text search, structured search, and facets‣ Scripting for performing ad–hoc analytics‣ Extendable: write your own facet types Realtime Analytics With ElasticSearch

FACETSScriptingExtract and aggregate most popular domains from article URLscurl -X DELETE localhost:9200/demo-articlescurl -X POST localhost:9200/demo-articles -d {"mappings": { "a": { "properties": {"url": {type: "string", "index": "not_analyzed"}} } } }curl -X PUT localhost:9200/demo-articles/a/1 -d {"title":"...","url":"http://some.blogger.com/2012/09/01/index.html"}curl -X PUT localhost:9200/demo-articles/a/2 -d {"title":"...","url":"http://some.blogger.com/2012/09/11/index.html"}curl -X PUT localhost:9200/demo-articles/a/3 -d {"title":"...","url":"http://some.blogger.com/about.html"}curl -X PUT localhost:9200/demo-articles/a/5 -d {"title":"...","url":"https://github.com/user/A"}curl -X PUT localhost:9200/demo-articles/a/5 -d {"title":"...","url":"http://github.com/user/B"}curl -X POST localhost:9200/demo-articles/_refreshcurl -X GET localhost:9200/demo-articles/_search/?search_type=count&pretty -d { "facets": { "popular-domains": { "terms": { "field" : "url", "script" : "term.replace(new RegExp("https?://"), "").split("/")[0]", "lang" : "javascript" } } } "facets" : {} "popular-‐domains" : { // ... "terms" : [ { Response "term" : "some.blogger.com", "count" : 3 }, { "term" : "github.com", "count" : 1 } ] } }

FACETSDemonstrationsExtract and aggregate most popular domains from article URLscurl -X DELETE localhost:9200/demo-articlescurl -X POST localhost:9200/demo-articles -d {"mappings": { "a": { "properties": {"url": {type: "string", "index": "not_analyzed"}} } } }curl -X PUT localhost:9200/demo-articles/a/1 -d {"title":"...","url":"http://some.blogger.com/2012/09/01/index.html"}curl -X PUT localhost:9200/demo-articles/a/2 -d {"title":"...","url":"http://some.blogger.com/2012/09/11/index.html"}curl -X PUT localhost:9200/demo-articles/a/3 -d {"title":"...","url":"http://some.blogger.com/about.html"}curl -X PUT localhost:9200/demo-articles/a/5 -d {"title":"...","url":"https://github.com/user/A"}curl -X PUT localhost:9200/demo-articles/a/5 -d {"title":"...","url":"http://github.com/user/B"}curl -X POST localhost:9200/demo-articles/_refreshcurl -X GET localhost:9200/demo-articles/_search/?search_type=count&pretty -d { "facets": { "popular-domains": { "terms": { "field" : "url", "script" : "term.replace(new RegExp("https?://"), "").split("/")[0]", "lang" : "javascript" }} } } Demo "facets" : { "popular-‐domains" : { // ... "terms" : [ { Response "term" : "some.blogger.com", "count" : 3 }, { "term" : "github.com", "count" : 1 } ] } }

Thanks! d

Realtime Analytics With Elasticsearch [New Media Inspiration 2013]

by Karel Minarik on Jan 19, 2013

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 161

Statistics

Realtime Analytics With Elasticsearch [New Media Inspiration 2013] Presentation Transcript

https://twitter.com	160
http://twylah.local	1