This document discusses big data and how to make sense of it using Elasticsearch. It begins by noting the massive scale of data being generated. It then demonstrates how to install and use Elasticsearch to index, search, and analyze sample user data. Aggregations are shown to compute statistics like counts by date. The document concludes by suggesting Elasticsearch can be used to gain insights from various data types like logs, social media, code repositories, and marketing information.

1. ‹#›
make sense of your (BIG) data!
David Pilato
Developer | Evangelist
@dadoonet

2. ‹#›

3. 3

4. 4

5. 5

6. 6

7. 6

8. 7

9. 7

10. 8

11. 8

12. 9

13. 9

14. Big data?
10

15. 11

16. 12 Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode

17. 12 Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode
35.000.000.000.000.000 mb

18. 13 Source: http://www.domo.com/learn/data-never-sleeps-2

20. Some data
CREATE TABLE user
(
name VARCHAR(100),
comments VARCHAR(1000)
);
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');
INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at
french customs service');
INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');
INSERT INTO user VALUES ('David David', 'Who is that guy?');
15

21. Search on term
SELECT * FROM user WHERE name="David";
Empty set (0,00 sec)
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');
INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at
french customs service');
INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');
INSERT INTO user VALUES ('David David', 'Who is that guy?');
16

22. Search like
SELECT * FROM user WHERE name LIKE "%David%";
+--------------+----------------------+
| name | comments |
+--------------+----------------------+
| David Pilato | Developer at elastic |
| David Gageot | Engineer at Docker |
| David David | Who is that guy? |
+--------------+----------------------+
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');
INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at
french customs service');
INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');
INSERT INTO user VALUES ('David David', 'Who is that guy?');
17

23. Search like
SELECT * FROM user WHERE name LIKE "%David%Pilato%";
+--------------+----------------------+
| name | comments |
+--------------+----------------------+
| David Pilato | Developer at elastic |
+--------------+----------------------+
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');
INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at
french customs service');
INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');
INSERT INTO user VALUES ('David David', 'Who is that guy?');
18

24. Search like with inverted terms
SELECT * FROM user WHERE name LIKE "%Pilato%David%";
Empty set (0,00 sec)
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');
INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at
french customs service');
INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');
INSERT INTO user VALUES ('David David', 'Who is that guy?');
19

25. Search in two fields
SELECT * FROM user WHERE name LIKE "%David%" OR
comments LIKE "%David%";
+--------------+---------------------------------------------+
| name | comments |
+--------------+---------------------------------------------+
| David Pilato | Developer at elastic |
| Malloum Laya | Worked with David at french customs service |
| David Gageot | Engineer at Docker |
| David David | Who is that guy? |
+--------------+---------------------------------------------+
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');
INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at
french customs service');
INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');
INSERT INTO user VALUES ('David David', 'Who is that guy?');
20

26. 21

27. 22

28. 23

29. search engine?
24

30. search engine?
24

31. 25

32. 25
Lucene

33. 25
REST/JSON Lucene

34. 25
REST/JSON
scalable
Lucene

35. 25
plug & play
REST/JSON
scalable
Lucene

36. 25
plug & play
REST/JSON
scalable
Apache 2 license
Lucene

37. start…
26
$ wget https://artifacts.elastic.co/downloads/elasticsearch/
elasticsearch-5.0.0-beta1.tar.gz
$ tar -xf elasticsearch-5.0.0-beta1.tar.gz
$ ./elasticsearch-5.0.0-beta1/bin/elasticsearch
[INFO ][node ][Ghost Maker] version[5.0.0-beta1], pid[72965], …
[INFO ][transport][Ghost Maker] publish_address {[/127.0.0.1:9300]}
[INFO ][http ][Ghost Maker] publish_address {[/127.0.0.1:9200]}
[INFO ][node ][Ghost Maker] started

38. … and play!
27
$ curl -XPUT localhost:9200/sessions/session/1 -d '{
"title" : "Elasticsearch",
"subtitle" : "Make sense of your (BIG) data !",
"date" : "2016-10-07T16:30:00",
"tags" : [ "realtime", "bigdata" ],
"speakers" : [{
"first_name" : "David",
"last_name" : "Pilato"
}]
}'

39. search!
28
$ curl http://localhost:9200/sessions/session/_search -d'
{
"query": {
"multi_match": {
"query": "elasticsearch bigdata david",
"fields": [ "title^3", "tags^2", "speakers.first_name" ]
}
},
"post_filter": {
"range": {
"date": {
"from": "2016-10-01",
"to": "2016-11-01"
}
}
}
}'

40. compute?
29

41. compute!
$ curl http://localhost:9200/sessions/session/_search -d'
{
"query": { ... },
"aggs": {
"by_date": {
"date_histogram": {
"field": "date",
"interval": "day",
"format" : "dd/MM/yyyy"
}
}
}
}'
30

42. compute!
$ curl http://localhost:9200/sessions/session/_search -d'
{
"query": { ... },
"aggs": {
"by_date": {
"date_histogram": {
"field": "date",
"interval": "day",
"format" : "dd/MM/yyyy"
}
}
}
}'
30
"by_date": [
{ "key_as_string": "03/10/2016", "doc_count": 1 },
{ "key_as_string": "07/10/2016", "doc_count": 2 },
{ "key_as_string": "10/10/2016", "doc_count": 3 }
]

43. compute!
$ curl http://localhost:9200/sessions/session/_search -d'
{
"query": { ... },
"aggs": {
"by_date": {
"date_histogram": {
"field": "date",
"interval": "day",
"format" : "dd/MM/yyyy"
}
}
}
}'
30
"by_date": [
{ "key_as_string": "03/10/2016", "doc_count": 1 },
{ "key_as_string": "07/10/2016", "doc_count": 2 },
{ "key_as_string": "10/10/2016", "doc_count": 3 }
]

45. Let’s make sense of …
• logs
• twitter
• github
• marketing data
• ...
• your data
• your big data
32

46. Let’s make sense of …
• logs
• twitter
• github
• marketing data
• ...
• your data
• your big data
33
{
"name":"Pilato David",
"dateOfBirth":"1971-12-26",
"gender":"male",
"children":3,
"marketing":{
"fashion":334,
"music":3363,
"hifi":2351
},
"address":{
"country":"France",
"city":"Paris",
"location": [2.332395, 48.861871]
}
}

47. Let's inject 1 000 000
marketing documents
34
Demo

48. ‹#›
Demo
35

49. 36
infom
ercial

50. 37
The only Elasticsearch as a Service offering
powered by the creators of the Elastic Stack
• Always runs on the latest software
• One-click to scale/upgrade with no downtime
• Free Kibana and backups every 30 minutes
• Dedicated, SLA-based support
• Easily add X-Pack features: security (Shield),
alerting (Watcher), and monitoring (Marvel)
• Pricing starts at $45 a month
infom
ercial

52. 39

53. ‹#›
https://www.elastic.co/subscriptions
Thank you!
David Pilato
Developer | Evangelist
@dadoonet