SlideShare a Scribd company logo
1 of 156
Elasticsearch : search engine
designed for cloud
               by David Pilato
         @dadoonet and @elasticsearchfr




                                          1
{ “speaker” : “David Pilato” }
$ curl http://localhost:9200/devoxx/speaker/dpilato
{
    "name" : "David Pilato",
    "jobs" : [
     { "company" : "SRA Europe (SSII)", "mission" : "bon à tout faire", "duration" : 3 },
     { "company" : "SFR", "mission" : "touche à tout", "duration" : 3 },
     { "company" : "e-Brands / Vivendi", "mission" : "chef de projets", "duration" : 4 },
     { "company" : "DGDDI (customs)", "mission" : "mouton à 5 pattes", "duration" : 7 } ],
    "passions" : [ "family", "job", "deejay" ],
    "blog" : "http://dev.david.pilato.fr/",
    "twitter" : [ "@dadoonet", "@elasticsearchfr" ],
    "email" : "david@pilato.fr"
}




                                                                                             2
Abstract
• The need for a search engine ?
• Elasticsearch : a complete, simple and performant solution
• What about indexing Twitter ?

             Make some noise on @DevoxxFR
             with the #elasticsearch hashtag !



                                                               3
A search engine ? What for ?
DO WE NEED A SEARCH ENGINE ?




                               4
Usual use case with « SQL old school »
           Having a document persisted in database :
          • date attribute : 19/04/2012
          • coded attribute country : FR
          • Association table code/label
           • Code : FR
           • Label : France
          • comment attribute : "There is a type error in the comment for this
            product. We should call David."


Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                 5
Usual use case with « SQL old school »
           Having a document persisted in database :                   doc       country
          • date attribute : 19/04/2012                                date       code
          • coded attribute country : FR                             country      label
          • Association table code/label                            comment

           • Code : FR
           • Label : France
          • comment attribute : "There is a type error in the comment for this
            product. We should call David."


Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                           5
Usual need with « SQL old school »

            • Find a document from december 2011 about france containing
              error and david
             • SQL :




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           6
Usual need with « SQL old school »

            • Find a document from december 2011 about france containing
              error and david
             • SQL :
                   SELECT
                     doc.*, pays.*
                   FROM
                     doc, pays
                   WHERE
                     doc.pays_code = pays.code AND
                     doc.date_doc > to_date('2011-12', 'yyyy-mm') AND
                     doc.date_doc < to_date('2012-01', 'yyyy-mm') AND
                     lower(pays.libelle) = 'france' AND
                     lower(doc.commentaire) LIKE ‘%error%' AND
                     lower(doc.commentaire) LIKE ‘%david%';




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           6
Performance impact of like ‘%’




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           7
Performance impact of like ‘%’




                                 See also : http://www.cestpasdur.com/2012/04/01/elasticsearch-vs-mysql-recherche




Engine   Elasticsearch Rivers   Facets    Demo       Architecture        Community
                                                                                                                    7
What is a search engine ?




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           8
What is a search engine ?
          • A search engine is :
           • an index engine for documents
           • a search engine on indexes




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           8
What is a search engine ?
          • A search engine is :
           • an index engine for documents
           • a search engine on indexes
          • A search engine is more powerful to do searches :




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           8
What is a search engine ?
          • A search engine is :
           • an index engine for documents
           • a search engine on indexes
          • A search engine is more powerful to do searches :
                                                it’s designed for it !



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           8
ELASTICSEARCH




                9
Your Data, your Search !
ELASTICSEARCH




                           9
Elasticsearch




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               10
Elasticsearch
          • Search engine for the NoSQL generation




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               10
Elasticsearch
          • Search engine for the NoSQL generation
          • Based on the standard Apache Lucene library




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               10
Elasticsearch
          • Search engine for the NoSQL generation
          • Based on the standard Apache Lucene library
          • Hide the Java / Lucene complexity with standard HTTP / RESTful /
              JSON services




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               10
Elasticsearch
          • Search engine for the NoSQL generation
          • Based on the standard Apache Lucene library
          • Hide the Java / Lucene complexity with standard HTTP / RESTful /
            JSON services
          • You can use it from whatever language or platform




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               10
Elasticsearch
          • Search engine for the NoSQL generation
          • Based on the standard Apache Lucene library
          • Hide the Java / Lucene complexity with standard HTTP / RESTful /
            JSON services
          • You can use it from whatever language or platform
          • Add the cloud layer that Lucene miss




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               10
Elasticsearch
          • Search engine for the NoSQL generation
          • Based on the standard Apache Lucene library
          • Hide the Java / Lucene complexity with standard HTTP / RESTful /
            JSON services
          • You can use it from whatever language or platform
          • Add the cloud layer that Lucene miss
          • It’s an engine, not a graphical user interface !




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               10
Key points




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                            11
Key points
          • Easy ! In some minutes (Zero Conf), you will get a full search engine
              ready to get your documents and perform your searches.




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                    11
Key points
          • Easy ! In some minutes (Zero Conf), you will get a full search engine
            ready to get your documents and perform your searches.
          • Efficient ! Just start new Elasticsearch nodes to scale horizontally
            with replication and load balancing.




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                    11
Key points
          • Easy ! In some minutes (Zero Conf), you will get a full search engine
            ready to get your documents and perform your searches.
          • Efficient ! Just start new Elasticsearch nodes to scale horizontally
            with replication and load balancing.
          • Powerful ! Lucene based product, with parallel processing to get
            acceptable response time (mainly less than 100ms).




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                    11
Key points
          • Easy ! In some minutes (Zero Conf), you will get a full search engine
            ready to get your documents and perform your searches.
          • Efficient ! Just start new Elasticsearch nodes to scale horizontally
            with replication and load balancing.
          • Powerful ! Lucene based product, with parallel processing to get
            acceptable response time (mainly less than 100ms).
          • Complete ! Many features : analysis and facets, percolation, rivers,
            plugins, …


Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                    11
Storing your data




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           12
Storing your data
          • Document : A full object containing all your data (NoSQL meaning).
              To think "search", you have to forget RDBMS and think "Documents"




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                  12
Storing your data
                • Document : A full object containing all your data (NoSQL meaning).
                    To think "search", you have to forget RDBMS and think "Documents"
           {
             "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
             "created_at": "2012-04-06T20:45:36.000Z",
             "source": "Twitter for iPad",
             "truncated": false,
 A tweet




             "retweet_count": 0,
             "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
                          { "text": "devoxxfr", "start": 47, "end": 55 } ],
             "user": { "id": 51172224, "name": "David Pilato",
                       "screen_name": "dadoonet", "location": "France",
                       "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL
           world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
           }




Engine         Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                                             12
Storing your data
                • Document : A full object containing all your data (NoSQL meaning).
                    To think "search", you have to forget RDBMS and think "Documents"
           {
             "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
             "created_at": "2012-04-06T20:45:36.000Z",
             "source": "Twitter for iPad",
             "truncated": false,
 A tweet




             "retweet_count": 0,
             "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
                          { "text": "devoxxfr", "start": 47, "end": 55 } ],
             "user": { "id": 51172224, "name": "David Pilato",
                       "screen_name": "dadoonet", "location": "France",
                       "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL
           world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
           }


               • Type : Includes all documents of the same type


Engine         Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                                             12
Storing your data
                • Document : A full object containing all your data (NoSQL meaning).
                    To think "search", you have to forget RDBMS and think "Documents"
           {
             "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
             "created_at": "2012-04-06T20:45:36.000Z",
             "source": "Twitter for iPad",
             "truncated": false,
 A tweet




             "retweet_count": 0,
             "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
                          { "text": "devoxxfr", "start": 47, "end": 55 } ],
             "user": { "id": 51172224, "name": "David Pilato",
                       "screen_name": "dadoonet", "location": "France",
                       "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL
           world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
           }


               • Type : Includes all documents of the same type
               • Index : Logical storage of related document types

Engine         Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                                             12
Playing with Elasticsearch
             REST API : http://host:port/[index]/[type]/[_action/id]
             HTTP Methods : GET, POST, PUT, DELETE




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           13
Playing with Elasticsearch
             REST API : http://host:port/[index]/[type]/[_action/id]
             HTTP Methods : GET, POST, PUT, DELETE
             Documents
                 • curl -XPUT http://localhost:9200/twitter/tweet/1




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           13
Playing with Elasticsearch
             REST API : http://host:port/[index]/[type]/[_action/id]
             HTTP Methods : GET, POST, PUT, DELETE
             Documents
                 • curl -XPUT http://localhost:9200/twitter/tweet/1
                 • curl -XGET http://localhost:9200/twitter/tweet/1




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           13
Playing with Elasticsearch
             REST API : http://host:port/[index]/[type]/[_action/id]
             HTTP Methods : GET, POST, PUT, DELETE
             Documents
                 • curl -XPUT http://localhost:9200/twitter/tweet/1
                 • curl -XGET http://localhost:9200/twitter/tweet/1
                 • curl -XDELETE http://localhost:9200/twitter/tweet/1




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           13
Playing with Elasticsearch
             REST API : http://host:port/[index]/[type]/[_action/id]
             HTTP Methods : GET, POST, PUT, DELETE
             Documents
                 • curl -XPUT http://localhost:9200/twitter/tweet/1
                 • curl -XGET http://localhost:9200/twitter/tweet/1
                 • curl -XDELETE http://localhost:9200/twitter/tweet/1
             Search
                 • curl -XGET http://localhost:9200/twitter/tweet/_search




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                            13
Playing with Elasticsearch
             REST API : http://host:port/[index]/[type]/[_action/id]
             HTTP Methods : GET, POST, PUT, DELETE
             Documents
                 • curl -XPUT http://localhost:9200/twitter/tweet/1
                 • curl -XGET http://localhost:9200/twitter/tweet/1
                 • curl -XDELETE http://localhost:9200/twitter/tweet/1
             Search
                 • curl -XGET http://localhost:9200/twitter/tweet/_search
                 • curl -XGET http://localhost:9200/twitter/_search



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                            13
Playing with Elasticsearch
             REST API : http://host:port/[index]/[type]/[_action/id]
             HTTP Methods : GET, POST, PUT, DELETE
             Documents
                 • curl -XPUT http://localhost:9200/twitter/tweet/1
                 • curl -XGET http://localhost:9200/twitter/tweet/1
                 • curl -XDELETE http://localhost:9200/twitter/tweet/1
             Search
                 • curl -XGET http://localhost:9200/twitter/tweet/_search
                 • curl -XGET http://localhost:9200/twitter/_search
                 • curl -XGET http://localhost:9200/_search


Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                            13
Playing with Elasticsearch
             REST API : http://host:port/[index]/[type]/[_action/id]
             HTTP Methods : GET, POST, PUT, DELETE
             Documents
                 • curl -XPUT http://localhost:9200/twitter/tweet/1
                 • curl -XGET http://localhost:9200/twitter/tweet/1
                 • curl -XDELETE http://localhost:9200/twitter/tweet/1
             Search
                 • curl -XGET http://localhost:9200/twitter/tweet/_search
                 • curl -XGET http://localhost:9200/twitter/_search
                 • curl -XGET http://localhost:9200/_search
             Elasticsearch Meta Data
                 • curl -XGET http://localhost:9200/twitter/_status
Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                            13
Let’s index a document
          $ curl -XPUT localhost:9200/twitter/tweet/1 -d '
          {
             "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
             "created_at": "2012-04-06T20:45:36.000Z",
             "source": "Twitter for iPad",
             "truncated": false,
             "retweet_count": 0,
             "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
                          { "text": "devoxxfr", "start": 47, "end": 55 } ],
             "user": { "id": 51172224, "name": "David Pilato",
                       "screen_name": "dadoonet", "location": "France",
                       "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying
          NoSQL world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
          }'




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                                       14
Let’s index a document
          $ curl -XPUT localhost:9200/twitter/tweet/1 -d '
          {
             "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
             "created_at": "2012-04-06T20:45:36.000Z",
             "source": "Twitter for iPad",
             "truncated": false,
             "retweet_count": 0,
             "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 },
                          { "text": "devoxxfr", "start": 47, "end": 55 } ],
             "user": { "id": 51172224, "name": "David Pilato",
                       "screen_name": "dadoonet", "location": "France",
                       "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying
          NoSQL world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" }
          }'


          {
              "ok":true,
              "_index":"twitter",
              "_type":"tweet",
              "_id":"1"
          }



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                                       14
Let’s search for documents
           $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           15
Let’s search for documents
           $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch

           {
               "took" : 24,
               "timed_out" : false,
               "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
               "hits" : {
                 "total" : 1,
                 "max_score" : 0.227,
                 "hits" : [ {
                   "_index" : "twitter",
                   "_type" : "tweet",
                   "_id" : "1",
                   "_score" : 0.227, "_source" : {
                     "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
                     "created_at": "2012-04-06T20:45:36.000Z",
                     "source": "Twitter for iPad",
                     […]
                   }
                 } ]
               }
           }



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                          15
Let’s search for documents
           $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch

           {
               "took" : 24,
               "timed_out" : false,
                                                  Total number of
                                                    documents
               "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
               "hits" : {
                 "total" : 1,
                 "max_score" : 0.227,
                 "hits" : [ {
                   "_index" : "twitter",
                   "_type" : "tweet",
                   "_id" : "1",
                   "_score" : 0.227, "_source" : {
                     "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
                     "created_at": "2012-04-06T20:45:36.000Z",
                     "source": "Twitter for iPad",
                     […]
                   }
                 } ]
               }
           }



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                          15
Let’s search for documents
           $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch

           {
               "took" : 24,
               "timed_out" : false,
               "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
               "hits" : {
                 "total" : 1,
                 "max_score" : 0.227,
                 "hits" : [ {                                       location
                   "_index" : "twitter",
                   "_type" : "tweet",
                   "_id" : "1",
                   "_score" : 0.227, "_source" : {
                     "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
                     "created_at": "2012-04-06T20:45:36.000Z",
                     "source": "Twitter for iPad",
                     […]
                   }
                 } ]
               }
           }



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                          15
Let’s search for documents
           $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch

           {
               "took" : 24,
               "timed_out" : false,
               "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
               "hits" : {
                 "total" : 1,
                 "max_score" : 0.227,
                 "hits" : [ {
                   "_index" : "twitter",
                   "_type" : "tweet",
                   "_id" : "1",
                   "_score" : 0.227, "_source" : {
                     "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
                     "created_at": "2012-04-06T20:45:36.000Z",
                     "source": "Twitter for iPad",
                     […]

                                                               Relevance
                   }
                 } ]
               }
           }



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                          15
Let’s search for documents
           $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch

           {
               "took" : 24,
               "timed_out" : false,

                                                                Document
               "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
               "hits" : {

                                                                 source
                 "total" : 1,
                 "max_score" : 0.227,
                 "hits" : [ {
                   "_index" : "twitter",
                   "_type" : "tweet",
                   "_id" : "1",
                   "_score" : 0.227, "_source" : {
                     "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
                     "created_at": "2012-04-06T20:45:36.000Z",
                     "source": "Twitter for iPad",
                     […]
                   }
                 } ]
               }
           }



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                          15
Search results




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                            16
Search results
          • Elasticsearch gives you the 10 first results (even on many millions) :
            pagination
          • You can move in the resultset
           $ curl "localhost:9200/twitter/tweet/_search?q=elasticsearch&from=10&size=10"




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                           16
Search results
          • Elasticsearch gives you the 10 first results (even on many millions) :
            pagination
          • You can move in the resultset
           $ curl "localhost:9200/twitter/tweet/_search?q=elasticsearch&from=10&size=10"




          • Scoring is computed with term frequency in a document relative to the
              term frequency in the index
           $ curl "localhost:9200/twitter/tweet/_search?q=elasticsearch&explain=true"




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                           16
Searches
         QueryDSL for advanced searches
                   Type                                                               Description
                                    Search for everything (useful combined with filters)
                                    Search with term analysis, wildcards (Lucene syntax* +, -, FROM, TO, ^)
                                    Search for individual term without analysis
                                    Search for a text with analysis (OR is applied between tokens by default)
                                    Wildcard search (*, ?)
                                    Combine many criteria (MUST, MUST NOT, SHOULD)
                                    Range search (>, >=, <, <=)
                                    Useful for autocomplete requirements
                                    Filtering queries
                                    Useful to find documents that are “like” provided text
                                    Useful to find documents that are “like” provided text with a minimal constraint on found terms



Engine   Elasticsearch Rivers   Facets   Demo    Architecture   Community
                                                                                                                                      17
Searches
         QueryDSL for advanced searches
                       Type                                                                         Description
                    Match All               Search for everything (useful combined with filters)
                  QueryString               Search with term analysis, wildcards (Lucene syntax* +, -, FROM, TO, ^)
                      Term                  Search for individual term without analysis
                       Text                 Search for a text with analysis (OR is applied between tokens by default)
                    Wildcard                Wildcard search (*, ?)
                       Bool                 Combine many criteria (MUST, MUST NOT, SHOULD)
                     Range                  Range search (>, >=, <, <=)
                      Prefix                Useful for autocomplete requirements
                     Filtered               Filtering queries
                Fuzzy like this             Useful to find documents that are “like” provided text
                 More like this             Useful to find documents that are “like” provided text with a minimal constraint on found terms
         * http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html

Engine   Elasticsearch Rivers          Facets      Demo      Architecture        Community
                                                                                                                                              17
AUTOMATIC PULLING DATA




                         18
Or "life is a long quiet river !"
AUTOMATIC PULLING DATA




                                    18
Pulling documents




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           19
Pulling documents


                                                                           Database




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                      19
Pulling documents

              Doc

                                                                           Database




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                      19
Pulling documents


                                                                           Database
                                                                            Engine
                                                                           Doc




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                      20
Pulling documents

              Doc

                                                                           Database
                                                                            Engine
                                                                           Doc




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                      20
Pulling documents


                                                                           Database
                                                                           Doc   Doc




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                       21
Pulling documents


                                                                           Database
                                                                           Doc   Doc




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                       21
Pulling documents


                                                                           Database
                                                                           Doc   Doc




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                       21
Pulling documents


                                                                           Database


                                                                             Doc



                                                                             Doc




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                      22
Pulling documents

                Doc


                                                                           Database


                                                                             Doc



                                                                             Doc




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                      22
Pulling documents


                                                                           Database

                                                                             Doc




                                                                             Doc




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                      23
Pulling documents


                                                                           Database




                                                                             Doc




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                      24
Rivers




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               25
Rivers
          •   CouchDB River




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               25
Rivers
          • CouchDB River
          • MongoDB River




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               25
Rivers
          • CouchDB River
          • MongoDB River
          • Wikipedia River




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               25
Rivers
          • CouchDB River
          • MongoDB River
          • Wikipedia River
          • Twitter River




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               25
Rivers
          •   CouchDB River
          •   MongoDB River
          •   Wikipedia River
          •   Twitter River
          •   RabbitMQ River




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               25
Rivers
          •   CouchDB River
          •   MongoDB River
          •   Wikipedia River
          •   Twitter River
          •   RabbitMQ River
          •   RSS River




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               25
Rivers
          •   CouchDB River
          •   MongoDB River
          •   Wikipedia River
          •   Twitter River
          •   RabbitMQ River
          •   RSS River
          •   Dick Rivers




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               25
Looking at your data from different points of views
RESULT ANALYSIS (IN NEAR REAL TIME)




                                                      26
Facets
                       ID                   Username                                 Date       Hashtags
                       1                     dadoonet                              2012-04-18      1
                       2                     devoxxfr                              2012-04-18      5
   Some tweets




                       3                 elasticsearchfr                           2012-04-18      2
                       4                     dadoonet                              2012-04-18      2
                       5                     devoxxfr                              2012-04-18      6
                       6                 elasticsearchfr                           2012-04-19      3
                       7                     dadoonet                              2012-04-19      3
                       8                     devoxxfr                              2012-04-19      7
                       9                 elasticsearchfr                           2012-04-20      4


Engine           Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                           27
Term Facet

               Username                         Date                 Hashtags
                dadoonet                 2012-04-18                         1
                devoxxfr                 2012-04-18                         5
            elasticsearchfr              2012-04-18                         2
                dadoonet                 2012-04-18                         2
                devoxxfr                 2012-04-18                         6
            elasticsearchfr              2012-04-19                         3
                dadoonet                 2012-04-19                         3
                devoxxfr                 2012-04-19                         7
            elasticsearchfr              2012-04-20                         4

Engine   Elasticsearch Rivers   Facets   Demo    Architecture   Community
                                                                                28
Term Facet

               Username                         Date                 Hashtags
                dadoonet                 2012-04-18                         1
                devoxxfr                 2012-04-18                         5
            elasticsearchfr              2012-04-18             Username
                                                                       2        Count
                dadoonet                 2012-04-18             dadoonet
                                                                       2          3
                devoxxfr                 2012-04-18             devoxxfr6         3
            elasticsearchfr              2012-04-19 elasticsearchfr
                                                                3                 3
                dadoonet                 2012-04-19                         3
                devoxxfr                 2012-04-19                         7
            elasticsearchfr              2012-04-20                         4

Engine   Elasticsearch Rivers   Facets   Demo    Architecture   Community
                                                                                        28
Term Facet
                                               "facets" : {
                                                 "users" : { "terms" : {"field" : "username"} }
                                           }
  ID                Username                   Date                 Hashtags
   1                 dadoonet               2012-04-18                 1
   2                 devoxxfr               2012-04-18                 5
   3              elasticsearchfr           2012-04-18                 2
   4                 dadoonet               2012-04-18                 2
   5                 devoxxfr               2012-04-18                 6
   6              elasticsearchfr           2012-04-19                 3
   7                 dadoonet               2012-04-19                 3
   8                 devoxxfr               2012-04-19                 7
   9              elasticsearchfr           2012-04-20                 4


Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                  29
Term Facet
                                               "facets" : {
                                                 "users" : { "terms" : {"field" : "username"} }
                                           }
  ID                Username                  Date
                                          "facets" : {    Hashtags
   1                 dadoonet              2012-04-18 : {
                                            "users"          1
   2                 devoxxfr              2012-04-18 : "terms",
                                              "_type"        5
                                              "missing" : 0,
   3              elasticsearchfr          2012-04-18        2
                                              "total": 9,
   4                 dadoonet              2012-04-18
                                              "other": 0,    2
   5                 devoxxfr              2012-04-18 : [
                                              "terms"        6
   6              elasticsearchfr               { "term" : "dadoonet", "count" : 3 },
                                           2012-04-19        3
                                                { "term" : "devoxxfr", "count" : 3 },
   7                 dadoonet              2012-04-19        3
                                                { "term" : "elasticsearchfr", "count" : 3 }
   8                 devoxxfr              2012-04-19
                                              ]              7
   9              elasticsearchfr           }
                                           2012-04-20        4


Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                  29
Date Histogram Facet

ame                   Date                  Hashtags
onet             2012-04-18                       1
xxfr             2012-04-18                       5
archfr           2012-04-18                       2
onet             2012-04-18                       2
xxfr             2012-04-18                       6
archfr           2012-04-19                       3
onet             2012-04-19                       3
xxfr             2012-04-19                       7
archfr           2012-04-20                       4

  Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                             30
Date Histogram Facet

ame                   Date                  Hashtags
onet             2012-04-18                       1
                                                                         Per month
                                                                   Date                Count
xxfr             2012-04-18                       5
                                                                  2012-04                9
archfr           2012-04-18                       2
onet             2012-04-18                       2                          Per day
xxfr             2012-04-18                       6                 Date               Count
archfr           2012-04-19                       3              2012-04-18              5
onet             2012-04-19                       3              2012-04-19              3
xxfr             2012-04-19                       7              2012-04-20              1
archfr           2012-04-20                       4

  Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                               30
Date Histogram Facet
                                            "facets" : {
                                              "perday" : {
                                                 "date_histogram" : {
                                                   "field" : "date",
ame                   Date                         "interval" : "day"
                                            Hashtags
                                                 }
onet             2012-04-18                   }1
xxfr             2012-04-18                 } 5
archfr           2012-04-18                       2
onet             2012-04-18                       2
xxfr             2012-04-18                       6
archfr           2012-04-19                       3
onet             2012-04-19                       3
xxfr             2012-04-19                       7
archfr           2012-04-20                       4

  Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                             31
Date Histogram Facet
                                            "facets" : {
                                              "perday" : {
                                                 "date_histogram" : {
                                                   "field" : "date",
ame                   Date                         "interval" : "day"
                                            Hashtags
                                                 }
onet             2012-04-18                   }1
xxfr             2012-04-18                 } 5
archfr           2012-04-18                     2
                                           "facets" : {
onet             2012-04-18                     2
                                             "perday" : {
xxfr             2012-04-18                    "_type" : "date_histogram",
                                                6
                                               "entries": [
archfr           2012-04-19                     3
                                                  { "time": 1334700000000, "count": 5 },
onet             2012-04-19                     3 { "time": 1334786400000, "count": 3 },
xxfr             2012-04-19                     7 { "time": 1334872800000, "count": 1 }
                                               ]
archfr           2012-04-20                  } 4
                                           }
  Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                           31
Ranges Facet

      Hashtags
8            1
8            5
8            2
8            2
8            6
9            3
9            3
9            7
0            4

    Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                               32
Ranges Facet

      Hashtags
8            1
8            5
                                             Ranges         Count          Min   Max   Mean    Total
8            2
                                              x<3               3            1    2    1.667    5
8            2
                                         3 <= x < 5             3            3    4    3.333    10
8            6
                                             x >= 5             3            5    7     6       18
9            3
9            3
9            7
0            4

    Engine   Elasticsearch Rivers   Facets    Demo    Architecture   Community
                                                                                                       32
Ranges Facet
                            "facets" : { "hashtags" : {
                              "range" : { "field" : "hashtags",
                              "ranges" : [
                                 { "to" : 3 }, { "from" : 3, "to" : 5 }, { "from" : 5 }
      Hashtags                ] } } }
8            1
8            5
8            2
8            2
8            6
9            3
9            3
9            7
0            4

    Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                          33
Ranges Facet
                            "facets" : { "hashtags" : {
                              "range" : { "field" : "hashtags",
                              "ranges" : [
                                 { "to" : 3 }, { "from" : 3, "to" : 5 }, { "from" : 5 }
      Hashtags                ] } } }
8            1          "facets" : {
8            5            "hashtags" : {
                            "_type" : "range",
8            2              "ranges" : [
8            2                { "to": 3,
8            6                   "count": 3, "min": 1, "max": 2, "total": 5, "mean": 1.667
                              },
9            3                { "from":3, "to" : 5,
9            3                   "count": 3, "min": 3, "max": 4, "total": 10, "mean": 3.333
                              },
9            7                { "from":5,
0            4                   "count": 3, "min": 5, "max": 7, "total": 18, "mean": 6
                              } ] } }
    Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                              33
Commerce site usage




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           34
Commerce site usage




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           34
Commerce site usage




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           34
Commerce site usage
                                                                           Ranges




                                                                           Term



                                                                           Term



                                                                           Ranges
Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                    34
Faceted navigation




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           35
Faceted navigation

    Fixed Criteria




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           35
Faceted navigation

    Fixed Criteria




     Results



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           35
Faceted navigation

    Fixed Criteria
                                                                           Term




     Results



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                  35
Faceted navigation

    Fixed Criteria
                                                                            Term




                                                                           Date histogram
     Results



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                            35
Faceted navigation

    Fixed Criteria
                                                                            Term




                         Ranges                                            Date histogram
     Results



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                            35
Faceted navigation




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           36
Faceted navigation




         Criteria




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           36
Near Real Time Data Visualization

          • Perform a matchAll search on all data
          • Update screen every x seconds
          • While indexing new documents


                                                                           Date histogram



                                                                           Term



Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                            37
Did we make noise ?
DEMO APPLICATION




                      38
Demo architecture




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           39
Demo architecture




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                           39
Demo architecture

                                                                            Twitter
                                                                           Streaming
                                                                              API




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                       39
Demo architecture

                                                                            Twitter
                                                                           Streaming
                                                                              API




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                       39
Demo architecture

                                                                            Twitter
                                                                           Streaming
                                                                              API




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                       39
Demo architecture

                                                                                            Twitter
                                                                 Twitter                   Streaming
                                                                  River                       API


                                   $ curl -XPUT localhost:9200/_river/twitter/_meta -d '
                                   {
                                      "type" : "twitter",
                                      "twitter" : {
                                        "user" : "twitter_user",
                                        "password" : "twitter_passowrd",
                                        "filter" : { "tracks" : ["devoxxfr"] }
                                      }
                                   }'


Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                       39
Demo architecture

           Chrome
                                                                                            Twitter
                                                                 Twitter                   Streaming
                                                                  River                       API


                                   $ curl -XPUT localhost:9200/_river/twitter/_meta -d '
                                   {
                                      "type" : "twitter",
                                      "twitter" : {
                                        "user" : "twitter_user",
                                        "password" : "twitter_passowrd",
                                        "filter" : { "tracks" : ["devoxxfr"] }
                                      }
                                   }'


Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                       39
Let’s go further : sharding / replica / scalabilty
ARCHITECTURE




                                                     40
Glossary




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                             41
Glossary

            • Node : An Elasticsearch instance (~ server ?)




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                             41
Glossary

            • Node : An Elasticsearch instance (~ server ?)
            • Cluster : A set of nodes




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                             41
Glossary

            • Node : An Elasticsearch instance (~ server ?)
            • Cluster : A set of nodes
            • Shard : an index shard where you distribute documents




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                             41
Glossary

            •   Node : An Elasticsearch instance (~ server ?)
            •   Cluster : A set of nodes
            •   Shard : an index shard where you distribute documents
            •   Replica : One or more shard copy in the cluster




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                             41
Glossary

            •   Node : An Elasticsearch instance (~ server ?)
            •   Cluster : A set of nodes
            •   Shard : an index shard where you distribute documents
            •   Replica : One or more shard copy in the cluster
            •   Primary shard : shard elected as primary in the cluster. Lucene
                index documents there.




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                  41
Glossary

            • Node : An Elasticsearch instance (~ server ?)
            • Cluster : A set of nodes
            • Shard : an index shard where you distribute documents
            • Replica : One or more shard copy in the cluster
            • Primary shard : shard elected as primary in the cluster. Lucene
              index documents there.
            • Secondary shard : store replicas of primary shards




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                41
Let’s create an index
                                                                                    Cluster

                                                                           Nœud 1




                                                                                     Client
                                                                                     CURL




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                              42
Let’s create an index
           $ curl -XPUT localhost:9200/twitter -d '{                                                         Cluster
              "index" : {
                "number_of_shards" : 2,
                                                                                                   Nœud 1
                "number_of_replicas" : 1                                                           Shard 0
              }
           }'                                                                                      Shard 1




                                                               replication rule is not satisfied
                                                                                                              Client
                                                                                                              CURL




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                                       42
Let’s create an index
           $ curl -XPUT localhost:9200/twitter -d '{                                                     Cluster
              "index" : {
                "number_of_shards" : 2,                                                        Node 1              Node 2
                "number_of_replicas" : 1                                                       Shard 0             Shard 0
              }
           }'                                                                                  Shard 1             Shard 1




                                                               replication rule is satisfied
                                                                                                          Client
                                                                                                          CURL




Engine   Elasticsearch Rivers   Facets   Demo   Architecture   Community
                                                                                                                             42
Dynamic reallocation
                                                                            Cluster

                                         Node 1                   Node 2
                                         Shard 0                  Shard 0


                                         Shard 1                  Shard 1




Engine   Elasticsearch Rivers   Facets    Demo     Architecture    Community
                                                                                      43
Dynamic reallocation
                                                                            Cluster

                                         Node 1                   Node 2              Node 3
                                         Shard 0                  Shard 0


                                         Shard 1                  Shard 1




Engine   Elasticsearch Rivers   Facets    Demo     Architecture    Community
                                                                                               43
Dynamic reallocation
                                                                            Cluster

                                         Node 1                   Node 2              Node 3
                                         Shard 0                  Shard 0             Shard 0


                                         Shard 1                  Shard 1




Engine   Elasticsearch Rivers   Facets    Demo     Architecture    Community
                                                                                                44
Dynamic reallocation
                                                                            Cluster

                                         Node 1                   Node 2              Node 3
                                         Shard 0                                      Shard 0


                                         Shard 1                  Shard 1




Engine   Elasticsearch Rivers   Facets    Demo     Architecture    Community
                                                                                                44
Dynamic reallocation
                                                                            Cluster

                                         Node 1                   Node 2              Node 3    Node 4
                                         Shard 0                                      Shard 0


                                         Shard 1                  Shard 1




Engine   Elasticsearch Rivers   Facets    Demo     Architecture    Community
                                                                                                         44
Dynamic reallocation
                                                                            Cluster

                                         Node 1                   Node 2              Node 3    Node 4
                                         Shard 0                                      Shard 0


                                         Shard 1                  Shard 1                       Shard 1




Engine   Elasticsearch Rivers   Facets    Demo     Architecture    Community
                                                                                                          45
Dynamic reallocation
                                                                            Cluster

                                         Node 1                   Node 2              Node 3    Node 4
                                         Shard 0                                      Shard 0


                                                                  Shard 1                       Shard 1




                                                    Tuning is finding the best numbers for
                                                      nodes, shards and replicas !



Engine   Elasticsearch Rivers   Facets    Demo     Architecture    Community
                                                                                                          45
Let’s index a document
                                                                               Cluster

                                         Node 1                 Node 2                      Node 3               Node 4
                                          Shard 0                                            Shard 0


                                                                 Shard 1                                          Shard 1




                                                    Doc
                                                     1 Client          $ curl -XPUT localhost:9200/twitter/tweet/1 -d '
                                                       CURL            {
                                                                            "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
                                                                            "created_at": "2012-04-06T20:45:36.000Z",
                                                                            "source": "Twitter for iPad",
                                                                            ...
                                                                       }'




Engine   Elasticsearch Rivers   Facets   Demo   Architecture    Community
                                                                                                                                                 46
Let’s index a document
                                                                                  Cluster

                                         Node 1                    Node 2                      Node 3               Node 4
                                                    Doc
                                          Shard 0    1                                          Shard 0


                                                                    Shard 1                                          Shard 1




                                                          Client
                                                                          $ curl -XPUT localhost:9200/twitter/tweet/1 -d '
                                                          CURL            {
                                                                               "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
                                                                               "created_at": "2012-04-06T20:45:36.000Z",
                                                                               "source": "Twitter for iPad",
                                                                               ...
                                                                          }'




Engine   Elasticsearch Rivers   Facets   Demo   Architecture       Community
                                                                                                                                                    47
Let’s index a document
                                                                                  Cluster

                                         Node 1                    Node 2                      Node 3               Node 4
                                                    Doc
                                          Shard 0    1                                          Shard 0


                                                                    Shard 1                                          Shard 1




                                                          Client
                                                                          $ curl -XPUT localhost:9200/twitter/tweet/1 -d '
                                                          CURL            {
                                                                               "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
                                                                               "created_at": "2012-04-06T20:45:36.000Z",
                                                                               "source": "Twitter for iPad",
                                                                               ...
                                                                          }'




Engine   Elasticsearch Rivers   Facets   Demo   Architecture       Community
                                                                                                                                                    47
Let’s index a document
                                                                                  Cluster

                                         Node 1                    Node 2                      Node 3               Node 4
                                                    Doc                                                   Doc
                                          Shard 0    1                                          Shard 0    1


                                                                    Shard 1                                          Shard 1




                                                          Client
                                                                          $ curl -XPUT localhost:9200/twitter/tweet/1 -d '
                                                          CURL            {
                                                                               "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr",
                                                                               "created_at": "2012-04-06T20:45:36.000Z",
                                                                               "source": "Twitter for iPad",
                                                                               ...
                                                                          }'




Engine   Elasticsearch Rivers   Facets   Demo   Architecture       Community
                                                                                                                                                    48
Let’s index another document
                                                                                  Cluster

                                         Node 1                    Node 2                      Node 3               Node 4
                                                    Doc                                                   Doc
                                          Shard 0    1                                          Shard 0    1


                                                                    Shard 1                                          Shard 1




                                                    Doc
                                                     2
                                                          Client
                                                                          $ curl -XPUT localhost:9200/twitter/tweet/2 -d '
                                                          CURL            {
                                                                               "text": "Je fais du bruit pour #elasticsearch à #devoxxfr",
                                                                               "created_at": "2012-04-06T21:12:52.000Z",
                                                                               "source": "Twitter for iPad",
                                                                               ...
                                                                          }'




Engine   Elasticsearch Rivers   Facets   Demo   Architecture       Community
                                                                                                                                             49
Let’s index another document
                                                                                  Cluster

                                         Node 1                    Node 2                      Node 3               Node 4
                                                    Doc                                                   Doc
                                          Shard 0    1                                          Shard 0    1


                                                                    Shard 1                                          Shard 1
                                            Doc
                                             2




                                                          Client
                                                                          $ curl -XPUT localhost:9200/twitter/tweet/2 -d '
                                                          CURL            {
                                                                               "text": "Je fais du bruit pour #elasticsearch à #devoxxfr",
                                                                               "created_at": "2012-04-06T21:12:52.000Z",
                                                                               "source": "Twitter for iPad",
                                                                               ...
                                                                          }'




Engine   Elasticsearch Rivers   Facets   Demo     Architecture     Community
                                                                                                                                             50
Let’s index another document
                                                                                  Cluster

                                         Node 1                    Node 2                      Node 3               Node 4
                                                    Doc                                                   Doc
                                          Shard 0    1                                          Shard 0    1


                                                                    Shard 1     Doc                                  Shard 1
                                                                                 2




                                                          Client
                                                                          $ curl -XPUT localhost:9200/twitter/tweet/2 -d '
                                                          CURL            {
                                                                               "text": "Je fais du bruit pour #elasticsearch à #devoxxfr",
                                                                               "created_at": "2012-04-06T21:12:52.000Z",
                                                                               "source": "Twitter for iPad",
                                                                               ...
                                                                          }'




Engine   Elasticsearch Rivers   Facets   Demo   Architecture       Community
                                                                                                                                             51
Let’s index another document
                                                                                  Cluster

                                         Node 1                    Node 2                      Node 3               Node 4
                                                    Doc                                                   Doc
                                          Shard 0    1                                          Shard 0    1


                                                                    Shard 1     Doc                                  Shard 1
                                                                                 2




                                                          Client
                                                                          $ curl -XPUT localhost:9200/twitter/tweet/2 -d '
                                                          CURL            {
                                                                               "text": "Je fais du bruit pour #elasticsearch à #devoxxfr",
                                                                               "created_at": "2012-04-06T21:12:52.000Z",
                                                                               "source": "Twitter for iPad",
                                                                               ...
                                                                          }'




Engine   Elasticsearch Rivers   Facets   Demo   Architecture       Community
                                                                                                                                             51
Let’s index another document
                                                                                  Cluster

                                         Node 1                    Node 2                      Node 3               Node 4
                                                    Doc                                                   Doc
                                          Shard 0    1                                          Shard 0    1


                                                                                Doc                                            Doc
                                                                    Shard 1                                          Shard 1    2
                                                                                 2




                                                          Client
                                                                          $ curl -XPUT localhost:9200/twitter/tweet/2 -d '
                                                          CURL            {
                                                                               "text": "Je fais du bruit pour #elasticsearch à #devoxxfr",
                                                                               "created_at": "2012-04-06T21:12:52.000Z",
                                                                               "source": "Twitter for iPad",
                                                                               ...
                                                                          }'




Engine   Elasticsearch Rivers   Facets   Demo   Architecture       Community
                                                                                                                                             52
Let’s search for documents
                                                                                Cluster

                                         Node 1                    Node 2                 Node 3          Node 4
                                                    Doc                                             Doc
                                          Shard 0    1                                    Shard 0    1


                                                                              Doc                                   Doc
                                                                    Shard 1                               Shard 1    2
                                                                               2




                                                          Client   $ curl localhost:9200/twitter/_search?q=elasticsearch
                                                          CURL




Engine   Elasticsearch Rivers   Facets   Demo   Architecture       Community
                                                                                                                           53
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version
Elasticsearch - Devoxx France 2012 - English version

More Related Content

What's hot

Your Data, Your Search, ElasticSearch (EURUKO 2011)
Your Data, Your Search, ElasticSearch (EURUKO 2011)Your Data, Your Search, ElasticSearch (EURUKO 2011)
Your Data, Your Search, ElasticSearch (EURUKO 2011)Karel Minarik
 
Introduction to Elasticsearch
Introduction to ElasticsearchIntroduction to Elasticsearch
Introduction to ElasticsearchRuslan Zavacky
 
Introduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneIntroduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneRahul Jain
 
Battle of the Giants round 2
Battle of the Giants round 2Battle of the Giants round 2
Battle of the Giants round 2Rafał Kuć
 
From zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and ElasticsearchFrom zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and ElasticsearchRafał Kuć
 
ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014Roy Russo
 
Introduction to Elasticsearch
Introduction to ElasticsearchIntroduction to Elasticsearch
Introduction to ElasticsearchJason Austin
 
Elasticsearch quick Intro (English)
Elasticsearch quick Intro (English)Elasticsearch quick Intro (English)
Elasticsearch quick Intro (English)Federico Panini
 
ElasticSearch in Production: lessons learned
ElasticSearch in Production: lessons learnedElasticSearch in Production: lessons learned
ElasticSearch in Production: lessons learnedBeyondTrees
 
Elastic search overview
Elastic search overviewElastic search overview
Elastic search overviewABC Talks
 
Cool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearchCool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearchclintongormley
 
An Introduction to Elastic Search.
An Introduction to Elastic Search.An Introduction to Elastic Search.
An Introduction to Elastic Search.Jurriaan Persyn
 
Building a CRM on top of ElasticSearch
Building a CRM on top of ElasticSearchBuilding a CRM on top of ElasticSearch
Building a CRM on top of ElasticSearchMark Greene
 
Intro to elasticsearch
Intro to elasticsearchIntro to elasticsearch
Intro to elasticsearchJoey Wen
 
Elasticsearch - DevNexus 2015
Elasticsearch - DevNexus 2015Elasticsearch - DevNexus 2015
Elasticsearch - DevNexus 2015Roy Russo
 
Query DSL In Elasticsearch
Query DSL In ElasticsearchQuery DSL In Elasticsearch
Query DSL In ElasticsearchKnoldus Inc.
 
The ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch pluginsThe ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch pluginsItamar
 

What's hot (20)

Your Data, Your Search, ElasticSearch (EURUKO 2011)
Your Data, Your Search, ElasticSearch (EURUKO 2011)Your Data, Your Search, ElasticSearch (EURUKO 2011)
Your Data, Your Search, ElasticSearch (EURUKO 2011)
 
Solr vs ElasticSearch
Solr vs ElasticSearchSolr vs ElasticSearch
Solr vs ElasticSearch
 
Introduction to Elasticsearch
Introduction to ElasticsearchIntroduction to Elasticsearch
Introduction to Elasticsearch
 
Elasticsearch Introduction at BigData meetup
Elasticsearch Introduction at BigData meetupElasticsearch Introduction at BigData meetup
Elasticsearch Introduction at BigData meetup
 
Introduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneIntroduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of Lucene
 
Battle of the Giants round 2
Battle of the Giants round 2Battle of the Giants round 2
Battle of the Giants round 2
 
From zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and ElasticsearchFrom zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and Elasticsearch
 
ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014
 
Introduction to Elasticsearch
Introduction to ElasticsearchIntroduction to Elasticsearch
Introduction to Elasticsearch
 
Elasticsearch quick Intro (English)
Elasticsearch quick Intro (English)Elasticsearch quick Intro (English)
Elasticsearch quick Intro (English)
 
ElasticSearch in Production: lessons learned
ElasticSearch in Production: lessons learnedElasticSearch in Production: lessons learned
ElasticSearch in Production: lessons learned
 
Elastic search overview
Elastic search overviewElastic search overview
Elastic search overview
 
Cool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearchCool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearch
 
An Introduction to Elastic Search.
An Introduction to Elastic Search.An Introduction to Elastic Search.
An Introduction to Elastic Search.
 
Building a CRM on top of ElasticSearch
Building a CRM on top of ElasticSearchBuilding a CRM on top of ElasticSearch
Building a CRM on top of ElasticSearch
 
Intro to elasticsearch
Intro to elasticsearchIntro to elasticsearch
Intro to elasticsearch
 
Elasticsearch - DevNexus 2015
Elasticsearch - DevNexus 2015Elasticsearch - DevNexus 2015
Elasticsearch - DevNexus 2015
 
Query DSL In Elasticsearch
Query DSL In ElasticsearchQuery DSL In Elasticsearch
Query DSL In Elasticsearch
 
Elasticsearch Introduction
Elasticsearch IntroductionElasticsearch Introduction
Elasticsearch Introduction
 
The ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch pluginsThe ultimate guide for Elasticsearch plugins
The ultimate guide for Elasticsearch plugins
 

Viewers also liked

Data modeling for Elasticsearch
Data modeling for ElasticsearchData modeling for Elasticsearch
Data modeling for ElasticsearchFlorian Hopf
 
Searching Relational Data with Elasticsearch
Searching Relational Data with ElasticsearchSearching Relational Data with Elasticsearch
Searching Relational Data with Elasticsearchsirensolutions
 
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search Engine
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search EngineElasticSearch: Distributed Multitenant NoSQL Datastore and Search Engine
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search EngineDaniel N
 
Elasticsearch in Zalando
Elasticsearch in ZalandoElasticsearch in Zalando
Elasticsearch in ZalandoAlaa Elhadba
 
Elasticsearch as a search alternative to a relational database
Elasticsearch as a search alternative to a relational databaseElasticsearch as a search alternative to a relational database
Elasticsearch as a search alternative to a relational databaseKristijan Duvnjak
 
Elasticsearch Introduction to Data model, Search & Aggregations
Elasticsearch Introduction to Data model, Search & AggregationsElasticsearch Introduction to Data model, Search & Aggregations
Elasticsearch Introduction to Data model, Search & AggregationsAlaa Elhadba
 
Elastic Search (엘라스틱서치) 입문
Elastic Search (엘라스틱서치) 입문Elastic Search (엘라스틱서치) 입문
Elastic Search (엘라스틱서치) 입문SeungHyun Eom
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaAmazee Labs
 

Viewers also liked (8)

Data modeling for Elasticsearch
Data modeling for ElasticsearchData modeling for Elasticsearch
Data modeling for Elasticsearch
 
Searching Relational Data with Elasticsearch
Searching Relational Data with ElasticsearchSearching Relational Data with Elasticsearch
Searching Relational Data with Elasticsearch
 
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search Engine
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search EngineElasticSearch: Distributed Multitenant NoSQL Datastore and Search Engine
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search Engine
 
Elasticsearch in Zalando
Elasticsearch in ZalandoElasticsearch in Zalando
Elasticsearch in Zalando
 
Elasticsearch as a search alternative to a relational database
Elasticsearch as a search alternative to a relational databaseElasticsearch as a search alternative to a relational database
Elasticsearch as a search alternative to a relational database
 
Elasticsearch Introduction to Data model, Search & Aggregations
Elasticsearch Introduction to Data model, Search & AggregationsElasticsearch Introduction to Data model, Search & Aggregations
Elasticsearch Introduction to Data model, Search & Aggregations
 
Elastic Search (엘라스틱서치) 입문
Elastic Search (엘라스틱서치) 입문Elastic Search (엘라스틱서치) 입문
Elastic Search (엘라스틱서치) 입문
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
 

Similar to Elasticsearch - Devoxx France 2012 - English version

Qui Quaerit, Reperit. AWS Elasticsearch in Action
Qui Quaerit, Reperit. AWS Elasticsearch in ActionQui Quaerit, Reperit. AWS Elasticsearch in Action
Qui Quaerit, Reperit. AWS Elasticsearch in ActionGlobalLogic Ukraine
 
06 integrate elasticsearch
06 integrate elasticsearch06 integrate elasticsearch
06 integrate elasticsearchErhwen Kuo
 
Elastic Stack Introduction
Elastic Stack IntroductionElastic Stack Introduction
Elastic Stack IntroductionVikram Shinde
 
Solr and ElasticSearch demo and speaker feb 2014
Solr  and ElasticSearch demo and speaker feb 2014Solr  and ElasticSearch demo and speaker feb 2014
Solr and ElasticSearch demo and speaker feb 2014nkabra
 
APEX Alpe Adria Mike Hichwa Keynote April 11th 2019- Zagreb
APEX Alpe Adria Mike Hichwa Keynote April 11th 2019- ZagrebAPEX Alpe Adria Mike Hichwa Keynote April 11th 2019- Zagreb
APEX Alpe Adria Mike Hichwa Keynote April 11th 2019- ZagrebMichael Hichwa
 
NoSQL Now! Webinar Series: Innovations in NoSQL Query Languages
NoSQL Now! Webinar Series: Innovations in NoSQL Query Languages  NoSQL Now! Webinar Series: Innovations in NoSQL Query Languages
NoSQL Now! Webinar Series: Innovations in NoSQL Query Languages DATAVERSITY
 
Java EE microservices architecture - evolving the monolith
Java EE microservices architecture - evolving the monolithJava EE microservices architecture - evolving the monolith
Java EE microservices architecture - evolving the monolithMarkus Eisele
 
Using Data Lakes: Data Analytics Week SF
Using Data Lakes: Data Analytics Week SFUsing Data Lakes: Data Analytics Week SF
Using Data Lakes: Data Analytics Week SFAmazon Web Services
 
Building APIs in an easy way using API Platform
Building APIs in an easy way using API PlatformBuilding APIs in an easy way using API Platform
Building APIs in an easy way using API PlatformAntonio Peric-Mazar
 
ElasticSearch - index server used as a document database
ElasticSearch - index server used as a document databaseElasticSearch - index server used as a document database
ElasticSearch - index server used as a document databaseRobert Lujo
 
NoSQL, Apache SOLR and Apache Hadoop
NoSQL, Apache SOLR and Apache HadoopNoSQL, Apache SOLR and Apache Hadoop
NoSQL, Apache SOLR and Apache HadoopDmitry Kan
 
Drupal and Apache Stanbol
Drupal and Apache StanbolDrupal and Apache Stanbol
Drupal and Apache StanbolAlkuvoima
 
Restful Best Practices
Restful Best PracticesRestful Best Practices
Restful Best PracticesBelighted
 
Design for scale
Design for scaleDesign for scale
Design for scaleDoug Lampe
 
Building an intelligent big data application in 30 minutes
Building an intelligent big data application in 30 minutesBuilding an intelligent big data application in 30 minutes
Building an intelligent big data application in 30 minutesClaudiu Barbura
 
Alex Wade, Digital Library Interoperability
Alex Wade, Digital Library InteroperabilityAlex Wade, Digital Library Interoperability
Alex Wade, Digital Library Interoperabilityparker01
 
Frameworks Galore: A Pragmatic Review
Frameworks Galore: A Pragmatic ReviewFrameworks Galore: A Pragmatic Review
Frameworks Galore: A Pragmatic Reviewnetc2012
 

Similar to Elasticsearch - Devoxx France 2012 - English version (20)

Qui Quaerit, Reperit. AWS Elasticsearch in Action
Qui Quaerit, Reperit. AWS Elasticsearch in ActionQui Quaerit, Reperit. AWS Elasticsearch in Action
Qui Quaerit, Reperit. AWS Elasticsearch in Action
 
06 integrate elasticsearch
06 integrate elasticsearch06 integrate elasticsearch
06 integrate elasticsearch
 
Elastic Stack Introduction
Elastic Stack IntroductionElastic Stack Introduction
Elastic Stack Introduction
 
Solr and ElasticSearch demo and speaker feb 2014
Solr  and ElasticSearch demo and speaker feb 2014Solr  and ElasticSearch demo and speaker feb 2014
Solr and ElasticSearch demo and speaker feb 2014
 
APEX Alpe Adria Mike Hichwa Keynote April 11th 2019- Zagreb
APEX Alpe Adria Mike Hichwa Keynote April 11th 2019- ZagrebAPEX Alpe Adria Mike Hichwa Keynote April 11th 2019- Zagreb
APEX Alpe Adria Mike Hichwa Keynote April 11th 2019- Zagreb
 
Elastic pivorak
Elastic pivorakElastic pivorak
Elastic pivorak
 
NoSQL Now! Webinar Series: Innovations in NoSQL Query Languages
NoSQL Now! Webinar Series: Innovations in NoSQL Query Languages  NoSQL Now! Webinar Series: Innovations in NoSQL Query Languages
NoSQL Now! Webinar Series: Innovations in NoSQL Query Languages
 
Java EE microservices architecture - evolving the monolith
Java EE microservices architecture - evolving the monolithJava EE microservices architecture - evolving the monolith
Java EE microservices architecture - evolving the monolith
 
Semtech2006
Semtech2006Semtech2006
Semtech2006
 
Using Data Lakes: Data Analytics Week SF
Using Data Lakes: Data Analytics Week SFUsing Data Lakes: Data Analytics Week SF
Using Data Lakes: Data Analytics Week SF
 
Building APIs in an easy way using API Platform
Building APIs in an easy way using API PlatformBuilding APIs in an easy way using API Platform
Building APIs in an easy way using API Platform
 
Using Data Lakes
Using Data Lakes Using Data Lakes
Using Data Lakes
 
ElasticSearch - index server used as a document database
ElasticSearch - index server used as a document databaseElasticSearch - index server used as a document database
ElasticSearch - index server used as a document database
 
NoSQL, Apache SOLR and Apache Hadoop
NoSQL, Apache SOLR and Apache HadoopNoSQL, Apache SOLR and Apache Hadoop
NoSQL, Apache SOLR and Apache Hadoop
 
Drupal and Apache Stanbol
Drupal and Apache StanbolDrupal and Apache Stanbol
Drupal and Apache Stanbol
 
Restful Best Practices
Restful Best PracticesRestful Best Practices
Restful Best Practices
 
Design for scale
Design for scaleDesign for scale
Design for scale
 
Building an intelligent big data application in 30 minutes
Building an intelligent big data application in 30 minutesBuilding an intelligent big data application in 30 minutes
Building an intelligent big data application in 30 minutes
 
Alex Wade, Digital Library Interoperability
Alex Wade, Digital Library InteroperabilityAlex Wade, Digital Library Interoperability
Alex Wade, Digital Library Interoperability
 
Frameworks Galore: A Pragmatic Review
Frameworks Galore: A Pragmatic ReviewFrameworks Galore: A Pragmatic Review
Frameworks Galore: A Pragmatic Review
 

More from David Pilato

2018-10-02 - Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouv...
2018-10-02 - Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouv...2018-10-02 - Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouv...
2018-10-02 - Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouv...David Pilato
 
Managing your black friday logs Voxxed Luxembourg
Managing your black friday logs Voxxed LuxembourgManaging your black friday logs Voxxed Luxembourg
Managing your black friday logs Voxxed LuxembourgDavid Pilato
 
Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouver...
Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouver...Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouver...
Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouver...David Pilato
 
Managing your Black Friday Logs NDC Oslo
Managing your  Black Friday Logs NDC OsloManaging your  Black Friday Logs NDC Oslo
Managing your Black Friday Logs NDC OsloDavid Pilato
 
Managing your black friday logs - Code Europe
Managing your black friday logs - Code EuropeManaging your black friday logs - Code Europe
Managing your black friday logs - Code EuropeDavid Pilato
 
Managing your black Friday logs - CloudConf.IT
Managing your black Friday logs - CloudConf.ITManaging your black Friday logs - CloudConf.IT
Managing your black Friday logs - CloudConf.ITDavid Pilato
 
Elasticsearch in 15 minutes
Elasticsearch in 15 minutesElasticsearch in 15 minutes
Elasticsearch in 15 minutesDavid Pilato
 
Elastify you application: from SQL to NoSQL in less than one hour!
Elastify you application: from SQL to NoSQL in less than one hour!Elastify you application: from SQL to NoSQL in less than one hour!
Elastify you application: from SQL to NoSQL in less than one hour!David Pilato
 
Elasticsearch - Esme sudria
Elasticsearch - Esme sudriaElasticsearch - Esme sudria
Elasticsearch - Esme sudriaDavid Pilato
 
Lausanne JUG - Elasticsearch
Lausanne JUG - ElasticsearchLausanne JUG - Elasticsearch
Lausanne JUG - ElasticsearchDavid Pilato
 
Normandy JUG - Elasticsearch
Normandy JUG - ElasticsearchNormandy JUG - Elasticsearch
Normandy JUG - ElasticsearchDavid Pilato
 
Paris data geek - Elasticsearch
Paris data geek - ElasticsearchParis data geek - Elasticsearch
Paris data geek - ElasticsearchDavid Pilato
 
Nantes JUG - Elasticsearch
Nantes JUG - ElasticsearchNantes JUG - Elasticsearch
Nantes JUG - ElasticsearchDavid Pilato
 
Finist JUG - Elasticsearch
Finist JUG - ElasticsearchFinist JUG - Elasticsearch
Finist JUG - ElasticsearchDavid Pilato
 
Poitou charentes JUG - Elasticsearch
Poitou charentes JUG - ElasticsearchPoitou charentes JUG - Elasticsearch
Poitou charentes JUG - ElasticsearchDavid Pilato
 
Elasticsearch - Montpellier JUG
Elasticsearch - Montpellier JUGElasticsearch - Montpellier JUG
Elasticsearch - Montpellier JUGDavid Pilato
 
Lyon JUG - Elasticsearch
Lyon JUG - ElasticsearchLyon JUG - Elasticsearch
Lyon JUG - ElasticsearchDavid Pilato
 
Elasticsearch - OSDC France 2012
Elasticsearch - OSDC France 2012Elasticsearch - OSDC France 2012
Elasticsearch - OSDC France 2012David Pilato
 
Hands on lab Elasticsearch
Hands on lab ElasticsearchHands on lab Elasticsearch
Hands on lab ElasticsearchDavid Pilato
 
Elasticsearch - Devoxx France 2012
Elasticsearch - Devoxx France 2012Elasticsearch - Devoxx France 2012
Elasticsearch - Devoxx France 2012David Pilato
 

More from David Pilato (20)

2018-10-02 - Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouv...
2018-10-02 - Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouv...2018-10-02 - Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouv...
2018-10-02 - Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouv...
 
Managing your black friday logs Voxxed Luxembourg
Managing your black friday logs Voxxed LuxembourgManaging your black friday logs Voxxed Luxembourg
Managing your black friday logs Voxxed Luxembourg
 
Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouver...
Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouver...Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouver...
Un moteur de recherche NoSQL pour chercher^H^H^H^H^H^H^H^H trouver...
 
Managing your Black Friday Logs NDC Oslo
Managing your  Black Friday Logs NDC OsloManaging your  Black Friday Logs NDC Oslo
Managing your Black Friday Logs NDC Oslo
 
Managing your black friday logs - Code Europe
Managing your black friday logs - Code EuropeManaging your black friday logs - Code Europe
Managing your black friday logs - Code Europe
 
Managing your black Friday logs - CloudConf.IT
Managing your black Friday logs - CloudConf.ITManaging your black Friday logs - CloudConf.IT
Managing your black Friday logs - CloudConf.IT
 
Elasticsearch in 15 minutes
Elasticsearch in 15 minutesElasticsearch in 15 minutes
Elasticsearch in 15 minutes
 
Elastify you application: from SQL to NoSQL in less than one hour!
Elastify you application: from SQL to NoSQL in less than one hour!Elastify you application: from SQL to NoSQL in less than one hour!
Elastify you application: from SQL to NoSQL in less than one hour!
 
Elasticsearch - Esme sudria
Elasticsearch - Esme sudriaElasticsearch - Esme sudria
Elasticsearch - Esme sudria
 
Lausanne JUG - Elasticsearch
Lausanne JUG - ElasticsearchLausanne JUG - Elasticsearch
Lausanne JUG - Elasticsearch
 
Normandy JUG - Elasticsearch
Normandy JUG - ElasticsearchNormandy JUG - Elasticsearch
Normandy JUG - Elasticsearch
 
Paris data geek - Elasticsearch
Paris data geek - ElasticsearchParis data geek - Elasticsearch
Paris data geek - Elasticsearch
 
Nantes JUG - Elasticsearch
Nantes JUG - ElasticsearchNantes JUG - Elasticsearch
Nantes JUG - Elasticsearch
 
Finist JUG - Elasticsearch
Finist JUG - ElasticsearchFinist JUG - Elasticsearch
Finist JUG - Elasticsearch
 
Poitou charentes JUG - Elasticsearch
Poitou charentes JUG - ElasticsearchPoitou charentes JUG - Elasticsearch
Poitou charentes JUG - Elasticsearch
 
Elasticsearch - Montpellier JUG
Elasticsearch - Montpellier JUGElasticsearch - Montpellier JUG
Elasticsearch - Montpellier JUG
 
Lyon JUG - Elasticsearch
Lyon JUG - ElasticsearchLyon JUG - Elasticsearch
Lyon JUG - Elasticsearch
 
Elasticsearch - OSDC France 2012
Elasticsearch - OSDC France 2012Elasticsearch - OSDC France 2012
Elasticsearch - OSDC France 2012
 
Hands on lab Elasticsearch
Hands on lab ElasticsearchHands on lab Elasticsearch
Hands on lab Elasticsearch
 
Elasticsearch - Devoxx France 2012
Elasticsearch - Devoxx France 2012Elasticsearch - Devoxx France 2012
Elasticsearch - Devoxx France 2012
 

Recently uploaded

AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentationyogeshlabana357357
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024Lorenzo Miniero
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxFIDO Alliance
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfFIDO Alliance
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jNeo4j
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch TuesdayIvanti
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideStefan Dietze
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...FIDO Alliance
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?Paolo Missier
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...ScyllaDB
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxFIDO Alliance
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...FIDO Alliance
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxFIDO Alliance
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!Memoori
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfSrushith Repakula
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGDSC PJATK
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctBrainSell Technologies
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceSamy Fodil
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...ScyllaDB
 

Recently uploaded (20)

AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 

Elasticsearch - Devoxx France 2012 - English version

  • 1. Elasticsearch : search engine designed for cloud by David Pilato @dadoonet and @elasticsearchfr 1
  • 2. { “speaker” : “David Pilato” } $ curl http://localhost:9200/devoxx/speaker/dpilato { "name" : "David Pilato", "jobs" : [ { "company" : "SRA Europe (SSII)", "mission" : "bon à tout faire", "duration" : 3 }, { "company" : "SFR", "mission" : "touche à tout", "duration" : 3 }, { "company" : "e-Brands / Vivendi", "mission" : "chef de projets", "duration" : 4 }, { "company" : "DGDDI (customs)", "mission" : "mouton à 5 pattes", "duration" : 7 } ], "passions" : [ "family", "job", "deejay" ], "blog" : "http://dev.david.pilato.fr/", "twitter" : [ "@dadoonet", "@elasticsearchfr" ], "email" : "david@pilato.fr" } 2
  • 3. Abstract • The need for a search engine ? • Elasticsearch : a complete, simple and performant solution • What about indexing Twitter ? Make some noise on @DevoxxFR with the #elasticsearch hashtag ! 3
  • 4. A search engine ? What for ? DO WE NEED A SEARCH ENGINE ? 4
  • 5. Usual use case with « SQL old school » Having a document persisted in database : • date attribute : 19/04/2012 • coded attribute country : FR • Association table code/label • Code : FR • Label : France • comment attribute : "There is a type error in the comment for this product. We should call David." Engine Elasticsearch Rivers Facets Demo Architecture Community 5
  • 6. Usual use case with « SQL old school » Having a document persisted in database : doc country • date attribute : 19/04/2012 date code • coded attribute country : FR country label • Association table code/label comment • Code : FR • Label : France • comment attribute : "There is a type error in the comment for this product. We should call David." Engine Elasticsearch Rivers Facets Demo Architecture Community 5
  • 7. Usual need with « SQL old school » • Find a document from december 2011 about france containing error and david • SQL : Engine Elasticsearch Rivers Facets Demo Architecture Community 6
  • 8. Usual need with « SQL old school » • Find a document from december 2011 about france containing error and david • SQL : SELECT doc.*, pays.* FROM doc, pays WHERE doc.pays_code = pays.code AND doc.date_doc > to_date('2011-12', 'yyyy-mm') AND doc.date_doc < to_date('2012-01', 'yyyy-mm') AND lower(pays.libelle) = 'france' AND lower(doc.commentaire) LIKE ‘%error%' AND lower(doc.commentaire) LIKE ‘%david%'; Engine Elasticsearch Rivers Facets Demo Architecture Community 6
  • 9. Performance impact of like ‘%’ Engine Elasticsearch Rivers Facets Demo Architecture Community 7
  • 10. Performance impact of like ‘%’ See also : http://www.cestpasdur.com/2012/04/01/elasticsearch-vs-mysql-recherche Engine Elasticsearch Rivers Facets Demo Architecture Community 7
  • 11. What is a search engine ? Engine Elasticsearch Rivers Facets Demo Architecture Community 8
  • 12. What is a search engine ? • A search engine is : • an index engine for documents • a search engine on indexes Engine Elasticsearch Rivers Facets Demo Architecture Community 8
  • 13. What is a search engine ? • A search engine is : • an index engine for documents • a search engine on indexes • A search engine is more powerful to do searches : Engine Elasticsearch Rivers Facets Demo Architecture Community 8
  • 14. What is a search engine ? • A search engine is : • an index engine for documents • a search engine on indexes • A search engine is more powerful to do searches : it’s designed for it ! Engine Elasticsearch Rivers Facets Demo Architecture Community 8
  • 16. Your Data, your Search ! ELASTICSEARCH 9
  • 17. Elasticsearch Engine Elasticsearch Rivers Facets Demo Architecture Community 10
  • 18. Elasticsearch • Search engine for the NoSQL generation Engine Elasticsearch Rivers Facets Demo Architecture Community 10
  • 19. Elasticsearch • Search engine for the NoSQL generation • Based on the standard Apache Lucene library Engine Elasticsearch Rivers Facets Demo Architecture Community 10
  • 20. Elasticsearch • Search engine for the NoSQL generation • Based on the standard Apache Lucene library • Hide the Java / Lucene complexity with standard HTTP / RESTful / JSON services Engine Elasticsearch Rivers Facets Demo Architecture Community 10
  • 21. Elasticsearch • Search engine for the NoSQL generation • Based on the standard Apache Lucene library • Hide the Java / Lucene complexity with standard HTTP / RESTful / JSON services • You can use it from whatever language or platform Engine Elasticsearch Rivers Facets Demo Architecture Community 10
  • 22. Elasticsearch • Search engine for the NoSQL generation • Based on the standard Apache Lucene library • Hide the Java / Lucene complexity with standard HTTP / RESTful / JSON services • You can use it from whatever language or platform • Add the cloud layer that Lucene miss Engine Elasticsearch Rivers Facets Demo Architecture Community 10
  • 23. Elasticsearch • Search engine for the NoSQL generation • Based on the standard Apache Lucene library • Hide the Java / Lucene complexity with standard HTTP / RESTful / JSON services • You can use it from whatever language or platform • Add the cloud layer that Lucene miss • It’s an engine, not a graphical user interface ! Engine Elasticsearch Rivers Facets Demo Architecture Community 10
  • 24. Key points Engine Elasticsearch Rivers Facets Demo Architecture Community 11
  • 25. Key points • Easy ! In some minutes (Zero Conf), you will get a full search engine ready to get your documents and perform your searches. Engine Elasticsearch Rivers Facets Demo Architecture Community 11
  • 26. Key points • Easy ! In some minutes (Zero Conf), you will get a full search engine ready to get your documents and perform your searches. • Efficient ! Just start new Elasticsearch nodes to scale horizontally with replication and load balancing. Engine Elasticsearch Rivers Facets Demo Architecture Community 11
  • 27. Key points • Easy ! In some minutes (Zero Conf), you will get a full search engine ready to get your documents and perform your searches. • Efficient ! Just start new Elasticsearch nodes to scale horizontally with replication and load balancing. • Powerful ! Lucene based product, with parallel processing to get acceptable response time (mainly less than 100ms). Engine Elasticsearch Rivers Facets Demo Architecture Community 11
  • 28. Key points • Easy ! In some minutes (Zero Conf), you will get a full search engine ready to get your documents and perform your searches. • Efficient ! Just start new Elasticsearch nodes to scale horizontally with replication and load balancing. • Powerful ! Lucene based product, with parallel processing to get acceptable response time (mainly less than 100ms). • Complete ! Many features : analysis and facets, percolation, rivers, plugins, … Engine Elasticsearch Rivers Facets Demo Architecture Community 11
  • 29. Storing your data Engine Elasticsearch Rivers Facets Demo Architecture Community 12
  • 30. Storing your data • Document : A full object containing all your data (NoSQL meaning). To think "search", you have to forget RDBMS and think "Documents" Engine Elasticsearch Rivers Facets Demo Architecture Community 12
  • 31. Storing your data • Document : A full object containing all your data (NoSQL meaning). To think "search", you have to forget RDBMS and think "Documents" { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", "truncated": false, A tweet "retweet_count": 0, "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 }, { "text": "devoxxfr", "start": 47, "end": 55 } ], "user": { "id": 51172224, "name": "David Pilato", "screen_name": "dadoonet", "location": "France", "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" } } Engine Elasticsearch Rivers Facets Demo Architecture Community 12
  • 32. Storing your data • Document : A full object containing all your data (NoSQL meaning). To think "search", you have to forget RDBMS and think "Documents" { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", "truncated": false, A tweet "retweet_count": 0, "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 }, { "text": "devoxxfr", "start": 47, "end": 55 } ], "user": { "id": 51172224, "name": "David Pilato", "screen_name": "dadoonet", "location": "France", "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" } } • Type : Includes all documents of the same type Engine Elasticsearch Rivers Facets Demo Architecture Community 12
  • 33. Storing your data • Document : A full object containing all your data (NoSQL meaning). To think "search", you have to forget RDBMS and think "Documents" { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", "truncated": false, A tweet "retweet_count": 0, "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 }, { "text": "devoxxfr", "start": 47, "end": 55 } ], "user": { "id": 51172224, "name": "David Pilato", "screen_name": "dadoonet", "location": "France", "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" } } • Type : Includes all documents of the same type • Index : Logical storage of related document types Engine Elasticsearch Rivers Facets Demo Architecture Community 12
  • 34. Playing with Elasticsearch REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Engine Elasticsearch Rivers Facets Demo Architecture Community 13
  • 35. Playing with Elasticsearch REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Documents • curl -XPUT http://localhost:9200/twitter/tweet/1 Engine Elasticsearch Rivers Facets Demo Architecture Community 13
  • 36. Playing with Elasticsearch REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Documents • curl -XPUT http://localhost:9200/twitter/tweet/1 • curl -XGET http://localhost:9200/twitter/tweet/1 Engine Elasticsearch Rivers Facets Demo Architecture Community 13
  • 37. Playing with Elasticsearch REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Documents • curl -XPUT http://localhost:9200/twitter/tweet/1 • curl -XGET http://localhost:9200/twitter/tweet/1 • curl -XDELETE http://localhost:9200/twitter/tweet/1 Engine Elasticsearch Rivers Facets Demo Architecture Community 13
  • 38. Playing with Elasticsearch REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Documents • curl -XPUT http://localhost:9200/twitter/tweet/1 • curl -XGET http://localhost:9200/twitter/tweet/1 • curl -XDELETE http://localhost:9200/twitter/tweet/1 Search • curl -XGET http://localhost:9200/twitter/tweet/_search Engine Elasticsearch Rivers Facets Demo Architecture Community 13
  • 39. Playing with Elasticsearch REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Documents • curl -XPUT http://localhost:9200/twitter/tweet/1 • curl -XGET http://localhost:9200/twitter/tweet/1 • curl -XDELETE http://localhost:9200/twitter/tweet/1 Search • curl -XGET http://localhost:9200/twitter/tweet/_search • curl -XGET http://localhost:9200/twitter/_search Engine Elasticsearch Rivers Facets Demo Architecture Community 13
  • 40. Playing with Elasticsearch REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Documents • curl -XPUT http://localhost:9200/twitter/tweet/1 • curl -XGET http://localhost:9200/twitter/tweet/1 • curl -XDELETE http://localhost:9200/twitter/tweet/1 Search • curl -XGET http://localhost:9200/twitter/tweet/_search • curl -XGET http://localhost:9200/twitter/_search • curl -XGET http://localhost:9200/_search Engine Elasticsearch Rivers Facets Demo Architecture Community 13
  • 41. Playing with Elasticsearch REST API : http://host:port/[index]/[type]/[_action/id] HTTP Methods : GET, POST, PUT, DELETE Documents • curl -XPUT http://localhost:9200/twitter/tweet/1 • curl -XGET http://localhost:9200/twitter/tweet/1 • curl -XDELETE http://localhost:9200/twitter/tweet/1 Search • curl -XGET http://localhost:9200/twitter/tweet/_search • curl -XGET http://localhost:9200/twitter/_search • curl -XGET http://localhost:9200/_search Elasticsearch Meta Data • curl -XGET http://localhost:9200/twitter/_status Engine Elasticsearch Rivers Facets Demo Architecture Community 13
  • 42. Let’s index a document $ curl -XPUT localhost:9200/twitter/tweet/1 -d ' { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", "truncated": false, "retweet_count": 0, "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 }, { "text": "devoxxfr", "start": 47, "end": 55 } ], "user": { "id": 51172224, "name": "David Pilato", "screen_name": "dadoonet", "location": "France", "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" } }' Engine Elasticsearch Rivers Facets Demo Architecture Community 14
  • 43. Let’s index a document $ curl -XPUT localhost:9200/twitter/tweet/1 -d ' { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", "truncated": false, "retweet_count": 0, "hashtag": [ { "text": "elasticsearch", "start": 27, "end": 40 }, { "text": "devoxxfr", "start": 47, "end": 55 } ], "user": { "id": 51172224, "name": "David Pilato", "screen_name": "dadoonet", "location": "France", "description": "Soft Architect, Project Manager, Senior Developper.rnAt this time, enjoying NoSQL world : CouchDB, ElasticSearch.rnDeeJay 4 times a year, just for fun !" } }' { "ok":true, "_index":"twitter", "_type":"tweet", "_id":"1" } Engine Elasticsearch Rivers Facets Demo Architecture Community 14
  • 44. Let’s search for documents $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch Engine Elasticsearch Rivers Facets Demo Architecture Community 15
  • 45. Let’s search for documents $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch { "took" : 24, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.227, "hits" : [ { "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_score" : 0.227, "_source" : { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", […] } } ] } } Engine Elasticsearch Rivers Facets Demo Architecture Community 15
  • 46. Let’s search for documents $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch { "took" : 24, "timed_out" : false, Total number of documents "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.227, "hits" : [ { "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_score" : 0.227, "_source" : { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", […] } } ] } } Engine Elasticsearch Rivers Facets Demo Architecture Community 15
  • 47. Let’s search for documents $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch { "took" : 24, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.227, "hits" : [ { location "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_score" : 0.227, "_source" : { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", […] } } ] } } Engine Elasticsearch Rivers Facets Demo Architecture Community 15
  • 48. Let’s search for documents $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch { "took" : 24, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.227, "hits" : [ { "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_score" : 0.227, "_source" : { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", […] Relevance } } ] } } Engine Elasticsearch Rivers Facets Demo Architecture Community 15
  • 49. Let’s search for documents $ curl localhost:9200/twitter/tweet/_search?q=elasticsearch { "took" : 24, "timed_out" : false, Document "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { source "total" : 1, "max_score" : 0.227, "hits" : [ { "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_score" : 0.227, "_source" : { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", […] } } ] } } Engine Elasticsearch Rivers Facets Demo Architecture Community 15
  • 50. Search results Engine Elasticsearch Rivers Facets Demo Architecture Community 16
  • 51. Search results • Elasticsearch gives you the 10 first results (even on many millions) : pagination • You can move in the resultset $ curl "localhost:9200/twitter/tweet/_search?q=elasticsearch&from=10&size=10" Engine Elasticsearch Rivers Facets Demo Architecture Community 16
  • 52. Search results • Elasticsearch gives you the 10 first results (even on many millions) : pagination • You can move in the resultset $ curl "localhost:9200/twitter/tweet/_search?q=elasticsearch&from=10&size=10" • Scoring is computed with term frequency in a document relative to the term frequency in the index $ curl "localhost:9200/twitter/tweet/_search?q=elasticsearch&explain=true" Engine Elasticsearch Rivers Facets Demo Architecture Community 16
  • 53. Searches QueryDSL for advanced searches Type Description Search for everything (useful combined with filters) Search with term analysis, wildcards (Lucene syntax* +, -, FROM, TO, ^) Search for individual term without analysis Search for a text with analysis (OR is applied between tokens by default) Wildcard search (*, ?) Combine many criteria (MUST, MUST NOT, SHOULD) Range search (>, >=, <, <=) Useful for autocomplete requirements Filtering queries Useful to find documents that are “like” provided text Useful to find documents that are “like” provided text with a minimal constraint on found terms Engine Elasticsearch Rivers Facets Demo Architecture Community 17
  • 54. Searches QueryDSL for advanced searches Type Description Match All Search for everything (useful combined with filters) QueryString Search with term analysis, wildcards (Lucene syntax* +, -, FROM, TO, ^) Term Search for individual term without analysis Text Search for a text with analysis (OR is applied between tokens by default) Wildcard Wildcard search (*, ?) Bool Combine many criteria (MUST, MUST NOT, SHOULD) Range Range search (>, >=, <, <=) Prefix Useful for autocomplete requirements Filtered Filtering queries Fuzzy like this Useful to find documents that are “like” provided text More like this Useful to find documents that are “like” provided text with a minimal constraint on found terms * http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html Engine Elasticsearch Rivers Facets Demo Architecture Community 17
  • 56. Or "life is a long quiet river !" AUTOMATIC PULLING DATA 18
  • 57. Pulling documents Engine Elasticsearch Rivers Facets Demo Architecture Community 19
  • 58. Pulling documents Database Engine Elasticsearch Rivers Facets Demo Architecture Community 19
  • 59. Pulling documents Doc Database Engine Elasticsearch Rivers Facets Demo Architecture Community 19
  • 60. Pulling documents Database Engine Doc Engine Elasticsearch Rivers Facets Demo Architecture Community 20
  • 61. Pulling documents Doc Database Engine Doc Engine Elasticsearch Rivers Facets Demo Architecture Community 20
  • 62. Pulling documents Database Doc Doc Engine Elasticsearch Rivers Facets Demo Architecture Community 21
  • 63. Pulling documents Database Doc Doc Engine Elasticsearch Rivers Facets Demo Architecture Community 21
  • 64. Pulling documents Database Doc Doc Engine Elasticsearch Rivers Facets Demo Architecture Community 21
  • 65. Pulling documents Database Doc Doc Engine Elasticsearch Rivers Facets Demo Architecture Community 22
  • 66. Pulling documents Doc Database Doc Doc Engine Elasticsearch Rivers Facets Demo Architecture Community 22
  • 67. Pulling documents Database Doc Doc Engine Elasticsearch Rivers Facets Demo Architecture Community 23
  • 68. Pulling documents Database Doc Engine Elasticsearch Rivers Facets Demo Architecture Community 24
  • 69. Rivers Engine Elasticsearch Rivers Facets Demo Architecture Community 25
  • 70. Rivers • CouchDB River Engine Elasticsearch Rivers Facets Demo Architecture Community 25
  • 71. Rivers • CouchDB River • MongoDB River Engine Elasticsearch Rivers Facets Demo Architecture Community 25
  • 72. Rivers • CouchDB River • MongoDB River • Wikipedia River Engine Elasticsearch Rivers Facets Demo Architecture Community 25
  • 73. Rivers • CouchDB River • MongoDB River • Wikipedia River • Twitter River Engine Elasticsearch Rivers Facets Demo Architecture Community 25
  • 74. Rivers • CouchDB River • MongoDB River • Wikipedia River • Twitter River • RabbitMQ River Engine Elasticsearch Rivers Facets Demo Architecture Community 25
  • 75. Rivers • CouchDB River • MongoDB River • Wikipedia River • Twitter River • RabbitMQ River • RSS River Engine Elasticsearch Rivers Facets Demo Architecture Community 25
  • 76. Rivers • CouchDB River • MongoDB River • Wikipedia River • Twitter River • RabbitMQ River • RSS River • Dick Rivers Engine Elasticsearch Rivers Facets Demo Architecture Community 25
  • 77. Looking at your data from different points of views RESULT ANALYSIS (IN NEAR REAL TIME) 26
  • 78. Facets ID Username Date Hashtags 1 dadoonet 2012-04-18 1 2 devoxxfr 2012-04-18 5 Some tweets 3 elasticsearchfr 2012-04-18 2 4 dadoonet 2012-04-18 2 5 devoxxfr 2012-04-18 6 6 elasticsearchfr 2012-04-19 3 7 dadoonet 2012-04-19 3 8 devoxxfr 2012-04-19 7 9 elasticsearchfr 2012-04-20 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 27
  • 79. Term Facet Username Date Hashtags dadoonet 2012-04-18 1 devoxxfr 2012-04-18 5 elasticsearchfr 2012-04-18 2 dadoonet 2012-04-18 2 devoxxfr 2012-04-18 6 elasticsearchfr 2012-04-19 3 dadoonet 2012-04-19 3 devoxxfr 2012-04-19 7 elasticsearchfr 2012-04-20 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 28
  • 80. Term Facet Username Date Hashtags dadoonet 2012-04-18 1 devoxxfr 2012-04-18 5 elasticsearchfr 2012-04-18 Username 2 Count dadoonet 2012-04-18 dadoonet 2 3 devoxxfr 2012-04-18 devoxxfr6 3 elasticsearchfr 2012-04-19 elasticsearchfr 3 3 dadoonet 2012-04-19 3 devoxxfr 2012-04-19 7 elasticsearchfr 2012-04-20 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 28
  • 81. Term Facet "facets" : { "users" : { "terms" : {"field" : "username"} } } ID Username Date Hashtags 1 dadoonet 2012-04-18 1 2 devoxxfr 2012-04-18 5 3 elasticsearchfr 2012-04-18 2 4 dadoonet 2012-04-18 2 5 devoxxfr 2012-04-18 6 6 elasticsearchfr 2012-04-19 3 7 dadoonet 2012-04-19 3 8 devoxxfr 2012-04-19 7 9 elasticsearchfr 2012-04-20 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 29
  • 82. Term Facet "facets" : { "users" : { "terms" : {"field" : "username"} } } ID Username Date "facets" : { Hashtags 1 dadoonet 2012-04-18 : { "users" 1 2 devoxxfr 2012-04-18 : "terms", "_type" 5 "missing" : 0, 3 elasticsearchfr 2012-04-18 2 "total": 9, 4 dadoonet 2012-04-18 "other": 0, 2 5 devoxxfr 2012-04-18 : [ "terms" 6 6 elasticsearchfr { "term" : "dadoonet", "count" : 3 }, 2012-04-19 3 { "term" : "devoxxfr", "count" : 3 }, 7 dadoonet 2012-04-19 3 { "term" : "elasticsearchfr", "count" : 3 } 8 devoxxfr 2012-04-19 ] 7 9 elasticsearchfr } 2012-04-20 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 29
  • 83. Date Histogram Facet ame Date Hashtags onet 2012-04-18 1 xxfr 2012-04-18 5 archfr 2012-04-18 2 onet 2012-04-18 2 xxfr 2012-04-18 6 archfr 2012-04-19 3 onet 2012-04-19 3 xxfr 2012-04-19 7 archfr 2012-04-20 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 30
  • 84. Date Histogram Facet ame Date Hashtags onet 2012-04-18 1 Per month Date Count xxfr 2012-04-18 5 2012-04 9 archfr 2012-04-18 2 onet 2012-04-18 2 Per day xxfr 2012-04-18 6 Date Count archfr 2012-04-19 3 2012-04-18 5 onet 2012-04-19 3 2012-04-19 3 xxfr 2012-04-19 7 2012-04-20 1 archfr 2012-04-20 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 30
  • 85. Date Histogram Facet "facets" : { "perday" : { "date_histogram" : { "field" : "date", ame Date "interval" : "day" Hashtags } onet 2012-04-18 }1 xxfr 2012-04-18 } 5 archfr 2012-04-18 2 onet 2012-04-18 2 xxfr 2012-04-18 6 archfr 2012-04-19 3 onet 2012-04-19 3 xxfr 2012-04-19 7 archfr 2012-04-20 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 31
  • 86. Date Histogram Facet "facets" : { "perday" : { "date_histogram" : { "field" : "date", ame Date "interval" : "day" Hashtags } onet 2012-04-18 }1 xxfr 2012-04-18 } 5 archfr 2012-04-18 2 "facets" : { onet 2012-04-18 2 "perday" : { xxfr 2012-04-18 "_type" : "date_histogram", 6 "entries": [ archfr 2012-04-19 3 { "time": 1334700000000, "count": 5 }, onet 2012-04-19 3 { "time": 1334786400000, "count": 3 }, xxfr 2012-04-19 7 { "time": 1334872800000, "count": 1 } ] archfr 2012-04-20 } 4 } Engine Elasticsearch Rivers Facets Demo Architecture Community 31
  • 87. Ranges Facet Hashtags 8 1 8 5 8 2 8 2 8 6 9 3 9 3 9 7 0 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 32
  • 88. Ranges Facet Hashtags 8 1 8 5 Ranges Count Min Max Mean Total 8 2 x<3 3 1 2 1.667 5 8 2 3 <= x < 5 3 3 4 3.333 10 8 6 x >= 5 3 5 7 6 18 9 3 9 3 9 7 0 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 32
  • 89. Ranges Facet "facets" : { "hashtags" : { "range" : { "field" : "hashtags", "ranges" : [ { "to" : 3 }, { "from" : 3, "to" : 5 }, { "from" : 5 } Hashtags ] } } } 8 1 8 5 8 2 8 2 8 6 9 3 9 3 9 7 0 4 Engine Elasticsearch Rivers Facets Demo Architecture Community 33
  • 90. Ranges Facet "facets" : { "hashtags" : { "range" : { "field" : "hashtags", "ranges" : [ { "to" : 3 }, { "from" : 3, "to" : 5 }, { "from" : 5 } Hashtags ] } } } 8 1 "facets" : { 8 5 "hashtags" : { "_type" : "range", 8 2 "ranges" : [ 8 2 { "to": 3, 8 6 "count": 3, "min": 1, "max": 2, "total": 5, "mean": 1.667 }, 9 3 { "from":3, "to" : 5, 9 3 "count": 3, "min": 3, "max": 4, "total": 10, "mean": 3.333 }, 9 7 { "from":5, 0 4 "count": 3, "min": 5, "max": 7, "total": 18, "mean": 6 } ] } } Engine Elasticsearch Rivers Facets Demo Architecture Community 33
  • 91. Commerce site usage Engine Elasticsearch Rivers Facets Demo Architecture Community 34
  • 92. Commerce site usage Engine Elasticsearch Rivers Facets Demo Architecture Community 34
  • 93. Commerce site usage Engine Elasticsearch Rivers Facets Demo Architecture Community 34
  • 94. Commerce site usage Ranges Term Term Ranges Engine Elasticsearch Rivers Facets Demo Architecture Community 34
  • 95. Faceted navigation Engine Elasticsearch Rivers Facets Demo Architecture Community 35
  • 96. Faceted navigation Fixed Criteria Engine Elasticsearch Rivers Facets Demo Architecture Community 35
  • 97. Faceted navigation Fixed Criteria Results Engine Elasticsearch Rivers Facets Demo Architecture Community 35
  • 98. Faceted navigation Fixed Criteria Term Results Engine Elasticsearch Rivers Facets Demo Architecture Community 35
  • 99. Faceted navigation Fixed Criteria Term Date histogram Results Engine Elasticsearch Rivers Facets Demo Architecture Community 35
  • 100. Faceted navigation Fixed Criteria Term Ranges Date histogram Results Engine Elasticsearch Rivers Facets Demo Architecture Community 35
  • 101. Faceted navigation Engine Elasticsearch Rivers Facets Demo Architecture Community 36
  • 102. Faceted navigation Criteria Engine Elasticsearch Rivers Facets Demo Architecture Community 36
  • 103. Near Real Time Data Visualization • Perform a matchAll search on all data • Update screen every x seconds • While indexing new documents Date histogram Term Engine Elasticsearch Rivers Facets Demo Architecture Community 37
  • 104. Did we make noise ? DEMO APPLICATION 38
  • 105. Demo architecture Engine Elasticsearch Rivers Facets Demo Architecture Community 39
  • 106. Demo architecture Engine Elasticsearch Rivers Facets Demo Architecture Community 39
  • 107. Demo architecture Twitter Streaming API Engine Elasticsearch Rivers Facets Demo Architecture Community 39
  • 108. Demo architecture Twitter Streaming API Engine Elasticsearch Rivers Facets Demo Architecture Community 39
  • 109. Demo architecture Twitter Streaming API Engine Elasticsearch Rivers Facets Demo Architecture Community 39
  • 110. Demo architecture Twitter Twitter Streaming River API $ curl -XPUT localhost:9200/_river/twitter/_meta -d ' { "type" : "twitter", "twitter" : { "user" : "twitter_user", "password" : "twitter_passowrd", "filter" : { "tracks" : ["devoxxfr"] } } }' Engine Elasticsearch Rivers Facets Demo Architecture Community 39
  • 111. Demo architecture Chrome Twitter Twitter Streaming River API $ curl -XPUT localhost:9200/_river/twitter/_meta -d ' { "type" : "twitter", "twitter" : { "user" : "twitter_user", "password" : "twitter_passowrd", "filter" : { "tracks" : ["devoxxfr"] } } }' Engine Elasticsearch Rivers Facets Demo Architecture Community 39
  • 112. Let’s go further : sharding / replica / scalabilty ARCHITECTURE 40
  • 113. Glossary Engine Elasticsearch Rivers Facets Demo Architecture Community 41
  • 114. Glossary • Node : An Elasticsearch instance (~ server ?) Engine Elasticsearch Rivers Facets Demo Architecture Community 41
  • 115. Glossary • Node : An Elasticsearch instance (~ server ?) • Cluster : A set of nodes Engine Elasticsearch Rivers Facets Demo Architecture Community 41
  • 116. Glossary • Node : An Elasticsearch instance (~ server ?) • Cluster : A set of nodes • Shard : an index shard where you distribute documents Engine Elasticsearch Rivers Facets Demo Architecture Community 41
  • 117. Glossary • Node : An Elasticsearch instance (~ server ?) • Cluster : A set of nodes • Shard : an index shard where you distribute documents • Replica : One or more shard copy in the cluster Engine Elasticsearch Rivers Facets Demo Architecture Community 41
  • 118. Glossary • Node : An Elasticsearch instance (~ server ?) • Cluster : A set of nodes • Shard : an index shard where you distribute documents • Replica : One or more shard copy in the cluster • Primary shard : shard elected as primary in the cluster. Lucene index documents there. Engine Elasticsearch Rivers Facets Demo Architecture Community 41
  • 119. Glossary • Node : An Elasticsearch instance (~ server ?) • Cluster : A set of nodes • Shard : an index shard where you distribute documents • Replica : One or more shard copy in the cluster • Primary shard : shard elected as primary in the cluster. Lucene index documents there. • Secondary shard : store replicas of primary shards Engine Elasticsearch Rivers Facets Demo Architecture Community 41
  • 120. Let’s create an index Cluster Nœud 1 Client CURL Engine Elasticsearch Rivers Facets Demo Architecture Community 42
  • 121. Let’s create an index $ curl -XPUT localhost:9200/twitter -d '{ Cluster "index" : { "number_of_shards" : 2, Nœud 1 "number_of_replicas" : 1 Shard 0 } }' Shard 1 replication rule is not satisfied Client CURL Engine Elasticsearch Rivers Facets Demo Architecture Community 42
  • 122. Let’s create an index $ curl -XPUT localhost:9200/twitter -d '{ Cluster "index" : { "number_of_shards" : 2, Node 1 Node 2 "number_of_replicas" : 1 Shard 0 Shard 0 } }' Shard 1 Shard 1 replication rule is satisfied Client CURL Engine Elasticsearch Rivers Facets Demo Architecture Community 42
  • 123. Dynamic reallocation Cluster Node 1 Node 2 Shard 0 Shard 0 Shard 1 Shard 1 Engine Elasticsearch Rivers Facets Demo Architecture Community 43
  • 124. Dynamic reallocation Cluster Node 1 Node 2 Node 3 Shard 0 Shard 0 Shard 1 Shard 1 Engine Elasticsearch Rivers Facets Demo Architecture Community 43
  • 125. Dynamic reallocation Cluster Node 1 Node 2 Node 3 Shard 0 Shard 0 Shard 0 Shard 1 Shard 1 Engine Elasticsearch Rivers Facets Demo Architecture Community 44
  • 126. Dynamic reallocation Cluster Node 1 Node 2 Node 3 Shard 0 Shard 0 Shard 1 Shard 1 Engine Elasticsearch Rivers Facets Demo Architecture Community 44
  • 127. Dynamic reallocation Cluster Node 1 Node 2 Node 3 Node 4 Shard 0 Shard 0 Shard 1 Shard 1 Engine Elasticsearch Rivers Facets Demo Architecture Community 44
  • 128. Dynamic reallocation Cluster Node 1 Node 2 Node 3 Node 4 Shard 0 Shard 0 Shard 1 Shard 1 Shard 1 Engine Elasticsearch Rivers Facets Demo Architecture Community 45
  • 129. Dynamic reallocation Cluster Node 1 Node 2 Node 3 Node 4 Shard 0 Shard 0 Shard 1 Shard 1 Tuning is finding the best numbers for nodes, shards and replicas ! Engine Elasticsearch Rivers Facets Demo Architecture Community 45
  • 130. Let’s index a document Cluster Node 1 Node 2 Node 3 Node 4 Shard 0 Shard 0 Shard 1 Shard 1 Doc 1 Client $ curl -XPUT localhost:9200/twitter/tweet/1 -d ' CURL { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", ... }' Engine Elasticsearch Rivers Facets Demo Architecture Community 46
  • 131. Let’s index a document Cluster Node 1 Node 2 Node 3 Node 4 Doc Shard 0 1 Shard 0 Shard 1 Shard 1 Client $ curl -XPUT localhost:9200/twitter/tweet/1 -d ' CURL { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", ... }' Engine Elasticsearch Rivers Facets Demo Architecture Community 47
  • 132. Let’s index a document Cluster Node 1 Node 2 Node 3 Node 4 Doc Shard 0 1 Shard 0 Shard 1 Shard 1 Client $ curl -XPUT localhost:9200/twitter/tweet/1 -d ' CURL { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", ... }' Engine Elasticsearch Rivers Facets Demo Architecture Community 47
  • 133. Let’s index a document Cluster Node 1 Node 2 Node 3 Node 4 Doc Doc Shard 0 1 Shard 0 1 Shard 1 Shard 1 Client $ curl -XPUT localhost:9200/twitter/tweet/1 -d ' CURL { "text": "Bienvenue à la conférence #elasticsearch pour #devoxxfr", "created_at": "2012-04-06T20:45:36.000Z", "source": "Twitter for iPad", ... }' Engine Elasticsearch Rivers Facets Demo Architecture Community 48
  • 134. Let’s index another document Cluster Node 1 Node 2 Node 3 Node 4 Doc Doc Shard 0 1 Shard 0 1 Shard 1 Shard 1 Doc 2 Client $ curl -XPUT localhost:9200/twitter/tweet/2 -d ' CURL { "text": "Je fais du bruit pour #elasticsearch à #devoxxfr", "created_at": "2012-04-06T21:12:52.000Z", "source": "Twitter for iPad", ... }' Engine Elasticsearch Rivers Facets Demo Architecture Community 49
  • 135. Let’s index another document Cluster Node 1 Node 2 Node 3 Node 4 Doc Doc Shard 0 1 Shard 0 1 Shard 1 Shard 1 Doc 2 Client $ curl -XPUT localhost:9200/twitter/tweet/2 -d ' CURL { "text": "Je fais du bruit pour #elasticsearch à #devoxxfr", "created_at": "2012-04-06T21:12:52.000Z", "source": "Twitter for iPad", ... }' Engine Elasticsearch Rivers Facets Demo Architecture Community 50
  • 136. Let’s index another document Cluster Node 1 Node 2 Node 3 Node 4 Doc Doc Shard 0 1 Shard 0 1 Shard 1 Doc Shard 1 2 Client $ curl -XPUT localhost:9200/twitter/tweet/2 -d ' CURL { "text": "Je fais du bruit pour #elasticsearch à #devoxxfr", "created_at": "2012-04-06T21:12:52.000Z", "source": "Twitter for iPad", ... }' Engine Elasticsearch Rivers Facets Demo Architecture Community 51
  • 137. Let’s index another document Cluster Node 1 Node 2 Node 3 Node 4 Doc Doc Shard 0 1 Shard 0 1 Shard 1 Doc Shard 1 2 Client $ curl -XPUT localhost:9200/twitter/tweet/2 -d ' CURL { "text": "Je fais du bruit pour #elasticsearch à #devoxxfr", "created_at": "2012-04-06T21:12:52.000Z", "source": "Twitter for iPad", ... }' Engine Elasticsearch Rivers Facets Demo Architecture Community 51
  • 138. Let’s index another document Cluster Node 1 Node 2 Node 3 Node 4 Doc Doc Shard 0 1 Shard 0 1 Doc Doc Shard 1 Shard 1 2 2 Client $ curl -XPUT localhost:9200/twitter/tweet/2 -d ' CURL { "text": "Je fais du bruit pour #elasticsearch à #devoxxfr", "created_at": "2012-04-06T21:12:52.000Z", "source": "Twitter for iPad", ... }' Engine Elasticsearch Rivers Facets Demo Architecture Community 52
  • 139. Let’s search for documents Cluster Node 1 Node 2 Node 3 Node 4 Doc Doc Shard 0 1 Shard 0 1 Doc Doc Shard 1 Shard 1 2 2 Client $ curl localhost:9200/twitter/_search?q=elasticsearch CURL Engine Elasticsearch Rivers Facets Demo Architecture Community 53

Editor's Notes

  1. \n
  2. \n
  3. Points abord&amp;#xE9;s :\nA quels besoins essaye t on de r&amp;#xE9;pondre ? A quoi servirait un moteur de recherche dans mon SI ?\nComment Elasticsearch r&amp;#xE9;pond &amp;#xE0; ces besoins et &amp;#xE0; bien d&apos;autres encore\nD&amp;#xE9;mo Live : indexation de messages Twitter ! Faites du bruit en twittant sur @devoxxfr et #elasticsearch\n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n
  34. \n
  35. \n
  36. \n
  37. \n
  38. \n
  39. \n
  40. \n
  41. \n
  42. \n
  43. \n
  44. \n
  45. \n
  46. \n
  47. \n
  48. \n
  49. \n
  50. \n
  51. \n
  52. \n
  53. \n
  54. \n
  55. \n
  56. \n
  57. \n
  58. \n
  59. \n
  60. \n
  61. \n
  62. \n
  63. \n
  64. \n
  65. \n
  66. \n
  67. \n
  68. \n
  69. \n
  70. \n
  71. \n
  72. \n
  73. \n
  74. \n
  75. \n
  76. \n
  77. \n
  78. \n
  79. \n
  80. \n
  81. \n
  82. \n
  83. \n
  84. \n
  85. \n
  86. \n
  87. \n
  88. \n
  89. \n
  90. \n
  91. \n
  92. \n
  93. \n
  94. \n
  95. \n
  96. \n
  97. \n
  98. \n
  99. \n
  100. \n
  101. \n
  102. \n
  103. \n
  104. \n
  105. \n
  106. \n
  107. \n
  108. \n
  109. \n
  110. \n
  111. \n
  112. \n
  113. \n
  114. \n
  115. \n
  116. \n
  117. \n
  118. \n
  119. \n
  120. \n
  121. \n
  122. \n
  123. \n
  124. \n
  125. \n
  126. \n
  127. \n
  128. \n
  129. \n
  130. \n
  131. \n
  132. \n
  133. \n
  134. \n
  135. \n
  136. \n
  137. \n
  138. \n
  139. \n
  140. \n
  141. \n
  142. \n
  143. \n
  144. \n
  145. \n
  146. \n
  147. \n
  148. \n
  149. \n
  150. \n
  151. \n
  152. \n
  153. \n
  154. \n
  155. \n
  156. \n
  157. \n
  158. \n
  159. \n
  160. \n
  161. \n
  162. \n
  163. \n
  164. \n
  165. \n
  166. \n
  167. \n
  168. \n
  169. \n
  170. \n
  171. \n
  172. \n
  173. \n
  174. \n
  175. \n
  176. \n
  177. \n
  178. \n
  179. \n
  180. \n
  181. \n
  182. \n
  183. \n
  184. \n
  185. \n
  186. \n
  187. \n
  188. \n
  189. \n
  190. \n
  191. \n
  192. \n
  193. \n
  194. \n
  195. \n
  196. \n
  197. \n
  198. \n
  199. \n
  200. \n
  201. \n
  202. \n
  203. \n
  204. \n
  205. \n
  206. \n
  207. \n
  208. \n
  209. \n
  210. \n
  211. \n
  212. \n
  213. \n
  214. \n
  215. \n
  216. \n
  217. \n