Craig Brown speaks on ElasticSearch

1,988 views

Published on

Published in: Technology, News & Politics
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,988
On SlideShare
0
From Embeds
0
Number of Embeds
349
Actions
Shares
0
Downloads
26
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Craig Brown speaks on ElasticSearch

  1. 1. 3 DBAs walk into a NOSQL bar. A little while later they walk out ..... because they couldn't find a table.
  2. 2. 3 nosql guys walk into a SQL bar... a little while later they leave because they couldn't find a relationship
  3. 3. Objective – Understanding of ElasticSeach as a search engine and nosql datastore.
  4. 4. About Me Search Architect Big Data/Hadoop Engineer NoSql Advocate blog.nosqltips.com @nosqltips on twitter
  5. 5. ElasticSearch You know, for search Shay Banon - compass Elasticsearch.org Built on Lucene core (4.3 as of 0.90.3) JSON over HTTP (REST)
  6. 6. ElasticSearch Very easy scalability – multicast, unicast, AWS Open Source – apache 2 license  https://github.com/elasticsearch Transports – HTTP, memcached, thrift Scripting – mvel, javascript, java, python, groovy  custom scoring, document updates
  7. 7. Schema Free & Document Oriented $ curl -XPUT http://localhost:9200/twitter/user/kimchy -d '{ "name" : "Shay Banon" }' $ curl -XPUT http://localhost:9200/twitter/tweet/1 -d '{ "user": "kimchy", "post_date": "2009-11-15T13:12:00", "message": "Trying out elasticsearch, so far so good?" }' $ curl -XPUT http://localhost:9200/twitter/tweet/2 -d '{ "user": "kimchy", "post_date": "2009-11-15T14:12:12", "message": "You know, for Search" }'
  8. 8. Search $ curl -XGET http://localhost:9200/twitter/tweet/_search?q=user:kimchy $ curl -XGET http://localhost:9200/twitter/tweet/_search -d '{ "query" : { "term" : { "user": "kimchy" } } }' $ curl -XGET http://localhost:9200/twitter/_search?pretty=true -d '{ "query" : { "range" : { "post_date" : { "from" : "2009-11-15T13:00:00", "to" : "2009-11-15T14:30:00" } } } }'
  9. 9. GETting Some Data $ curl -XPUT http://localhost:9200/twitter/tweet/2 -d '{ "user": "kimchy", "post_date": "2009-11-15T14:12:12", "message": "You know, for Search" }' $ curl -XGET http://localhost:9200/twitter/tweet/2
  10. 10. Schema Mapping $ curl -XPUT http://localhost:9200/twitter $ curl -XPUT http://localhost:9200/twitter/user/_mapping -d '{ "properties" : { "name" : { "type" : "string" } } }'
  11. 11. Multi Tenancy $ curl -XPUT http://localhost:9200/kimchy $ curl -XPUT http://localhost:9200/elasticsearch $ curl -XPUT http://localhost:9200/elasticsearch/tweet/1 -d '{ "post_date": "2009-11-15T14:12:12", "message": "Zug Zug", "tag": "warcraft" }' $ curl -XPUT http://localhost:9200/kimchy/tweet/1 -d '{ "post_date": "2009-11-15T14:12:12", "message": "Whatyouwant?", "tag": "warcraft" }' $ curl -XGET http://localhost:9200/kimchy,elasticsearch/tweet/_search?q=tag:warcraft $ curl -XGET http://localhost:9200/_all/tweet/_search?q=tag:warcraft
  12. 12. Settings $ curl -XPUT http://localhost:9200/elasticsearch/ -d '{ "settings" : { "number_of_shards" : 2, "number_of_replicas" : 3 } }'
  13. 13. Distributed Shards – write scale Replicas – read scale, durability Segments Routing – index and search Discovery – multicast, unicast, AWS http://www.youtube.com/watch?v=l4ReamjCxHo
  14. 14. Consistency Always read consistent with RT GET  View always consistent after write Document searchable after short delay(1s) Write tunable – one, quorum, all
  15. 15. Gateway Local/NFS Amazon S3 Hadoop
  16. 16. River Twitter CouchDB MongoDB RabbitMQ Wikipedia Logstash
  17. 17. SOLR over ElasticSearch Release synchronized with Lucene Larger community Larger tool set Feature set a bit better XML configuration
  18. 18. ElasticSearch over SOLR Natively distributed JSON based Dynamic, template, and defined schema Returns source document by default Avoids overhead of index commit after write Mock SOLR interface Rivers
  19. 19. Who uses ElasticSearch StumbleUpon Mozilla Foundation Sony Computer Entertainment Infochimps Foursquare Github Ataxo Social Insider Sonian Inc.
  20. 20. Demo
  21. 21. Hadoop Integration Hadoop as gateway storage LoadFunc/StorFunc – Pig/Map Reduce Hadoop streaming interface Manual export and import of data
  22. 22. ES in Big Data Endpoint for processed data Aggregator for BI or dashboard (facets) Used to query reduced data sets for machine learning algorithms Data storage engine in it's own right plus full search capabilities
  23. 23. ElasticStore Make ES look and function more like a document store while exposing advanced ES features Influenced by Mongo API Expose a simpler, more programmer centric API Expose A QueryBuilder style API (HQL) Expose annotations for easier schema definition, properties, analyzers, etc. Allow both strong and weak object mapping https://github.com/nosqltips/elasticstore
  24. 24. Resources www.elasticsearch.org www.elasticsearch.com https://github.com/elasticsearch http://lucene.apache.org/core

×