Craig Brown speaks on ElasticSearch
Upcoming SlideShare
Loading in...5

Craig Brown speaks on ElasticSearch






Total Views
Views on SlideShare
Embed Views



6 Embeds 292 285 2 2 1 1 1


Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Craig Brown speaks on ElasticSearch Craig Brown speaks on ElasticSearch Presentation Transcript

    • 3 DBAs walk into a NOSQL bar. A little while later they walk out ..... because they couldn't find a table.
    • 3 nosql guys walk into a SQL bar... a little while later they leave because they couldn't find a relationship
    • Objective – Understanding of ElasticSeach as a search engine and nosql datastore.
    • About Me Search Architect Big Data/Hadoop Engineer NoSql Advocate @nosqltips on twitter
    • ElasticSearch You know, for search Shay Banon - compass Built on Lucene core (4.3 as of 0.90.3) JSON over HTTP (REST)
    • ElasticSearch Very easy scalability – multicast, unicast, AWS Open Source – apache 2 license  Transports – HTTP, memcached, thrift Scripting – mvel, javascript, java, python, groovy  custom scoring, document updates
    • Schema Free & Document Oriented $ curl -XPUT http://localhost:9200/twitter/user/kimchy -d '{ "name" : "Shay Banon" }' $ curl -XPUT http://localhost:9200/twitter/tweet/1 -d '{ "user": "kimchy", "post_date": "2009-11-15T13:12:00", "message": "Trying out elasticsearch, so far so good?" }' $ curl -XPUT http://localhost:9200/twitter/tweet/2 -d '{ "user": "kimchy", "post_date": "2009-11-15T14:12:12", "message": "You know, for Search" }'
    • Search $ curl -XGET http://localhost:9200/twitter/tweet/_search?q=user:kimchy $ curl -XGET http://localhost:9200/twitter/tweet/_search -d '{ "query" : { "term" : { "user": "kimchy" } } }' $ curl -XGET http://localhost:9200/twitter/_search?pretty=true -d '{ "query" : { "range" : { "post_date" : { "from" : "2009-11-15T13:00:00", "to" : "2009-11-15T14:30:00" } } } }'
    • GETting Some Data $ curl -XPUT http://localhost:9200/twitter/tweet/2 -d '{ "user": "kimchy", "post_date": "2009-11-15T14:12:12", "message": "You know, for Search" }' $ curl -XGET http://localhost:9200/twitter/tweet/2
    • Schema Mapping $ curl -XPUT http://localhost:9200/twitter $ curl -XPUT http://localhost:9200/twitter/user/_mapping -d '{ "properties" : { "name" : { "type" : "string" } } }'
    • Multi Tenancy $ curl -XPUT http://localhost:9200/kimchy $ curl -XPUT http://localhost:9200/elasticsearch $ curl -XPUT http://localhost:9200/elasticsearch/tweet/1 -d '{ "post_date": "2009-11-15T14:12:12", "message": "Zug Zug", "tag": "warcraft" }' $ curl -XPUT http://localhost:9200/kimchy/tweet/1 -d '{ "post_date": "2009-11-15T14:12:12", "message": "Whatyouwant?", "tag": "warcraft" }' $ curl -XGET http://localhost:9200/kimchy,elasticsearch/tweet/_search?q=tag:warcraft $ curl -XGET http://localhost:9200/_all/tweet/_search?q=tag:warcraft
    • Settings $ curl -XPUT http://localhost:9200/elasticsearch/ -d '{ "settings" : { "number_of_shards" : 2, "number_of_replicas" : 3 } }'
    • Distributed Shards – write scale Replicas – read scale, durability Segments Routing – index and search Discovery – multicast, unicast, AWS
    • Consistency Always read consistent with RT GET  View always consistent after write Document searchable after short delay(1s) Write tunable – one, quorum, all
    • Gateway Local/NFS Amazon S3 Hadoop
    • River Twitter CouchDB MongoDB RabbitMQ Wikipedia Logstash
    • SOLR over ElasticSearch Release synchronized with Lucene Larger community Larger tool set Feature set a bit better XML configuration
    • ElasticSearch over SOLR Natively distributed JSON based Dynamic, template, and defined schema Returns source document by default Avoids overhead of index commit after write Mock SOLR interface Rivers
    • Who uses ElasticSearch StumbleUpon Mozilla Foundation Sony Computer Entertainment Infochimps Foursquare Github Ataxo Social Insider Sonian Inc.
    • Demo
    • Hadoop Integration Hadoop as gateway storage LoadFunc/StorFunc – Pig/Map Reduce Hadoop streaming interface Manual export and import of data
    • ES in Big Data Endpoint for processed data Aggregator for BI or dashboard (facets) Used to query reduced data sets for machine learning algorithms Data storage engine in it's own right plus full search capabilities
    • ElasticStore Make ES look and function more like a document store while exposing advanced ES features Influenced by Mongo API Expose a simpler, more programmer centric API Expose A QueryBuilder style API (HQL) Expose annotations for easier schema definition, properties, analyzers, etc. Allow both strong and weak object mapping
    • Resources