This document discusses Elasticsearch and how it can be used to search, analyze, and make sense of large amounts of data. It provides examples of how Elasticsearch is being used by large companies to handle petabytes of data and gain insights. Implementations in France are highlighted. The document concludes by demonstrating how easily Elasticsearch can be deployed and used to ingest and search sample data.

2. Agenda & Intervenants

3. Introduction

4. Alter Way in 2 slides

5. Alter Way in 2 slides

6. Elasticsearch in 1 slide
• More than 11 million downloads
• 650,000 New Downloads per Month
• 1000s of Mission Critical Implementations
• Top Investors: Benchmark Capital, Index
Ventures
• Seasoned Executive Team
– Founded by Creator of Elasticsearch
– Seasoned Executives from SpringSource

7. Les enjeux de la recherche à
l’ère du BigData

8. Big Data in Todayʼs Business and Technology
Environment : some significant figures
• 2.7 Zetabytes of data exist in the digital universe today. (=1 billion Terabytes)
• 235 Terabytes of data has been collected by the U.S. Library of Congress in
April 2011.
• Facebook stores, accesses, and analyzes 30+ Petabytes of user generated
data.
• Akamai analyzes 75 million events per day to better target advertisements.
• Walmart handles more than 1 million customer transactions every hour,
which is imported into databases estimated to contain more than 2.5 petabytes
of data.
• The largest AT&T database boasts titles including the largest volume of data in
one unique database (312 terabytes) and the second largest number of rows in
a unique database (1.9 trillion), which comprises AT&Tʼ’s extensive calling
records.
• Hadoop :
– 94% of Hadoop users perform analytics on large volumes of data not
possible before
– 88% analyze data in greater detail;
– while 82% can now retain more of their data.

9. The Rapid Growth of Unstructured Data
• YouTube users upload 48 hours of new video every minute of the
day.
• 500+ new websites are created every minute of the day.
• Brands and organizations on Facebook receive 34,722 Likes every
minute of the day.
• 100 terabytes of data uploaded daily to Facebook.
• According to Twitterʼ’s own research in early 2012, it sees roughly
175 million tweets every day, and has more than 465 million
accounts.
• 30 Billion pieces of content shared on Facebook every month.
Data production will be 44 times greater in 2020 than it was in 2009.

10. Big Data & Real Business Issues
• 25+ % of decision‐makers surveyed predict that data volumes in their
companies will rise by more than 60% by the end of 2014, with the
average of all respondents anticipating a growth of no less than 42 %.
• 40% projected growth in global data generated per year vs. 5% growth in
global IT spending.
• According to estimates, the volume of business data worldwide, across all
companies, doubles every 1.2 years.
– Poor data can cost businesses 20%–35% of their operating revenue.
– Bad data or poor data quality costs US businesses $600 billion annually.
• 75+ % of decision-makers surveyed anticipate significant impacts in the
domain of storage systems as a result of the “Big Data” phenomenon.
• We anticipate a new challenge : to be able to Search and Analyse all
those datas … in real time !

11. Elasticsearch
A solution already in production
with significant french
implementations
Revolutionizing Data Search and
Analytics
Richard Maurer– SEMEA Territory Manager

12. Purpose of Elasticsearch
• Organize data and make it easily accessible
– Through powerful search and analytics
– Easily consumable (even for non-data scientists)
– Elegantly handles extremely large data volumes
– Delivers results in real time
• Technology stack agnostic
• Used across all market verticals

13. Features of Elasticsearch
• Structured & unstructured search
• Advanced analytics capabilities
• Unmatched performance
• Real-time results
• Highly scalable
• User friendly installation and maintenance

14. Elasticsearch 1.4: a solution
production ready
• Real time data Indexation
• Distributed
• High Availability
• Schema Free
• Real Time Data Analytics
• Multi Tenancy
• Much more….

15. Unprecedented Uptake
Elasticsearch has more than11 Million downloads
… and 650,000 more each month
Cumulative

16. French Users

17. French Use Cases
Bouygues Telecom:
Uses Elasticsearch in their Big Data Platform. Cut their web resolution time by 10X
Daily Motion:
Indexing their 20 million Videos on Elasticsearch. On production for over 2 years
Voyages SNCF
They have recently announced ES has being live on their “Usine Logicielle”
Fotolia:
Search Engine made on Elasticsearch, to access 24 Million Images, move over to ES
Orange:
With over 1.2 billion docs, looking at better solution and cost reduction

18. Product Offerings:
Support Throughout Your Project
1. Core Elasticsearch Training (2 days)
2. ELK Workshop (1 day)
3. Development and Production Support
4. Marvel, Monitoring of your ES clusters

19. 2: Support

20. Resources
• www.elasticsearch.com
• www.elasticsearch.org
• User Groups:
http://www.elasticsearch.org/community/forum/
• Contact:
Richard Maurer
Territory Manager
Richard.maurer@elasticsearch.com

21. MAKE SENSE OF YOUR (BIG) DATA!
David Pilato
Technical advocate!
!
elasticsearch.
@dadoonet

23. StartUp
data ?

24. StartUp

25. StartUp

26. StartUp

27. StartUp

28. StartUp

29. StartUp
BIG data ?

30. StartUp
BIG data ?

31. 35.000.000.000.000.000 mb
Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode
StartUp

32. StartUp
Source: http://www.domo.com/learn/data-never-sleeps-2

34. StartUp
search = like % ?
SELECT !
doc.*, country.* !
FROM !
doc, country!
WHERE !
doc.country_code = country.code AND!
doc.date_doc > to_date('2011-12', 'yyyy-mm') AND !
doc.date_doc < to_date('2012-01', 'yyyy-mm') AND !
lower(country.name) = 'france' AND !
lower(doc.comment) LIKE ‘%product%' AND
lower(doc.comment) LIKE ‘%david%';

35. StartUp
Search engine ?

36. elasticsearch ?
StartUp
plug & play
REST/JSON
scalable
Apache 2 license
Lucene
elasticsearch

37. Start…
$ wget https://download.elasticsearch.org/elasticsearch/
elasticsearch/elasticsearch-1.1.1.tar.gz!
$ tar -xf elasticsearch-1.1.1.tar.gz!
$ ./elasticsearch-1.1.1/bin/elasticsearch!
[INFO ][node ][Ghost Maker] {1.1.1}[5645]: initializing

38. … and play!
$ curl -XPUT localhost:9200/sessions/session/1 -d '{!
"title" : "Elasticsearch",!
"subtitle" : "Make sense of your (BIG) data !",!
"date" : "2014-05-20T10:30:00",!
"tags" : [ "elasticsearch", "alterway", "bigdata" ],!
"speakers" : [{!
"first_name" : "David", !
"last_name" : "Pilato" !
}]!
}'

39. Search!
$ curl http://localhost:9200/sessions/session/_search -d'
{
"query": {
"multi_match": {
"query": "elasticsearch alterway david",
"fields": [ "title^3", "tags^2", "speakers.first_name" ]
}
},
"post_filter": {
"range": {
"date": {
"from": "2014-05-01",
"to": "2014-06"
}
}
}
}'

40. StartUp
Compute?

41. Compute!
$ curl http://localhost:9200/sessions/session/_search -d'
{
"query": { ... },
"aggs": {
"by_date": {
"date_histogram": {
"field": "date",
"interval": "day",
"format" : "dd/MM/yyyy"
}
}
}
}'
"by_date": [
{ "key_as_string": "03/04/2014", "doc_count": 1 },
{ "key_as_string": "12/04/2014", "doc_count": 2 },
{ "key_as_string": "16/04/2014", "doc_count": 3 }
]

43. Let’s make sense of …
• logs!
• twitter!
• github!
• marketing data!
• ...!
• your data!
• your big data
#mstechdays #elasticsearch StartUp

44. Let’s make sense of …
• logs!
• twitter!
• github!
• marketing data!
• ...!
• your data!
• your big data
{
"name":"Pilato David",
"dateOfBirth":"1971-12-26",
"gender":"male",
"children":3,
"marketing":{
"fashion":334,
"music":3363,
"hifi":2351
},
"address":{
"country":"France",
"city":"Paris",
"location": [2.332395, 48.861871]
}
}
#mstechdays #elasticsearch StartUp

45. démo
MAKE SENSE OF YOUR (BIG) DATA!
let’s inject some marketing documents…
#mstechdays #elasticsearch StartUp

46. elasticsearch.
elasticsearch
kibana
logstash
Marvel

47. thanks
@dadoonet

48. Comment insérer ElasticSearch
dans votre Système d’Information
et en tirer le meilleur parti

49. ElasticSearch to do What ?

50. STORE

51. SEARCH

52. ANALYZE

53. Are you ready to use
ElasticSearch in your IT?

54. What you need to run it
• Java 8 update 20 or later, or Java 7 update 55 or
later
• Only Oracle’s Java and the OpenJDK are
supported.

55. Github projects
• Many projects
• Big activity
• Many languages
6 mois !

56. Clients

57. Scripting Plugins Language

58. Why it ‘s easy

59. • One to many
• ~ Zero conf
• Cloud oriented
• Scalability DNA
• Replication
• Sharding
• Distributed
• Resilience
• Snapshot
• Restore
Start Small Grow Big

60. • One to many
• ~ Zero conf
• Cloud oriented
• Scalability DNA
• Replication
• Sharding
• Distributed
• Resilience
• Snapshot
• Restore
Start Small Grow Big

61. Where / How can you use
ElasticSearch?

62. Centralized Log Storage 1/2
VIA

63. Centralized Log Storage 2/2

64. …
CMS Search Engine

65. Ecommerce Enhanced Search
Engine
• Faceting
• Fuzzy Search
• Speed
• Auto Completion
• Geo Search
• Log Analysis

66. Combining Hadoop & ElasticSearch
• REST based
• Memory and I/O efficient
• Adaptive I/O
• Map/Reduce API support
• Pig support
• Hive support
 elasticsearch-hadoop

67. What Else ?

68. It’s up to you to decide what to build with ES

69. Analysis / Dasboards
Some Examples

70. Kibana examples : IRC Activity

71. Kibana examples : Pfsense Monitoring

72. Kibana examples : Windows Events

73. Kibana examples : Inventory

74. Kibana examples : Syslog

75. Kibana examples : Web Activity

76. ES = No Limits

78. Conclusion

79. Conclusion
• Il est temps de révolutionner la façon dont vous valorisez
vos données : offrez Elasticsearch à vos applicatifs !
• La stack ELK (Elasticsearch, Logstash, Kibana) est déjà
massivement utilisée en production !
• Faites vous accompagner pour bénéficier des bonnes
pratiques et du support à tous les stades de votre projet :
conception, développement, production

80. Questions / Réponses