SlideShare a Scribd company logo
Finding Cars and Hunting Down Logs:
Elasticsearch @ AutoScout24
AutoScout24
24 Nov 2016
Philipp Garbe Lead developer (philipp.garbe@scout24.com)
Juri Smarschevski Team lead (juri.smarschevski@scout24.com)
Search
AutoScout24 search journey in nutshell
2
Who we are ?
Unique Monthly Visitors in Europa
3
…10 more
Some numbers
Search index contains ~2.6M classifieds
4
Unique visitors (monthly): ~10M
Search requests per day: ~36M
Index update rate per day ~400.000 classifieds
Status quo. March 2013.
Endeca used as a search engine
5
Use case: providing search results and facets for the entire AS24
platform
Problems:
• New product requirements, performance of Endeca becomes slower
• Time to market of our required features is not sufficient
• Maintenance is complex / expensive
Possible candidates
Solr ?
• <feeling> too complex installation / configuration </feeling>
6
Sphinx ?
• Support situation is unclear
Elasticsearch ?
• Fresh buzzword
• From beginning on built for distributed systems (rumors)
• Easy installation / configuration (fact)
POC
Goals
• Performance should be comparable with Endeca
• The solution should be scalable
7
8
Rollout plan. 03.2013 - 11.2013
07.2013 11.201302.2013 03.2013 05.2013
POC
Implementation & migration
Training
Go live phase
#real_project_picture_squeezed
9
Endeca Elasticsearch
(0.9.x)
Amount of machines 60 20
[Re]index time ~180 min ~45 min
Deploy to Live up to 2 days < 3 hours
Effort for testing an
issue on local machine
4 h 1 h
Performance = =
Product / dev guys
satisfaction
:( :)
300%
400%
1000%
400%
% ?
Results after 8 months of working.
No problems after migration ?
Cluster split brain
Has in fact nothing to do with Elasticsearch, is more related to learn phase at AS24
10
Deep pagination
Elasticsearch 5.x release notes: “Deep pagination of search results is now possible with the search_after
feature, which efficiently skips over previously returned results to return just the next page.“
11
Status quo. November 2014.
Project “Tatsu” has started
.NET => JVM
C# => Scala
IIS / Windows => Play / Linux
Local data center => AWS
Monolith => Micro services
Windows workstations => Mac notebooks
... => ...
12
Status quo. November 2014.
Project “Tatsu” has started
.NET => JVM
C# => Scala
IIS / Windows => Play / Linux
Local data center => AWS
Monolith => Micro services
Windows workstations => Mac notebooks
? => ?
=> 2015
13
Elasticsearch clusters “lift & shift” to AWS ?
AWS Elasticsearch Service ?
Elasticsearch as a service (SaaS) ?
Own hosting in AWS ?
16
Rolling update in details (possible scenario).
Time
1
Initial state
17
Rolling update in details (possible scenario).
Node has
been replaced
Time
1 2
Initial state
~ 60 sec
18
Rolling update in details (possible scenario).
Master has
been killed
Node has
been replaced
Time
1 2 3
Initial state
19
Rolling update in details (possible scenario).
Master has
been killed
Node has
been replaced
Master election
Time
1 2 3 4
Initial state
20
Rolling update in details (possible scenario).
Master has
been killed
Node has
been replaced
Master election
Time
1 2 3 4 5
Initial state Last node has
been replaced
21
Rolling update findings
Master has
been killed
?Outage=
22
Rolling update findings
Logging
Continuously deployed, immutable and stateful
23
7.4 billion
documents
Some numbers
36 TB
EBS
18 nodes á
m4.4xlarge
(64GB / 53.5
cpu units)
Unified Logs
25
Challenge: Deployment time
Rolling updates
27
Challenge: Costs
First setup
● 18x m4.4xlarge
● 18x 2TB gp2
● 3TB/day cross-zone traffic
Cost/Usage Optimized Setup
● 15x m4.x2large
● 15x 384GB gp2
● 6x SpotFleet
● 6x 4TB st1
● 9TB/day cross-zone traffic
Savings: ~40%
Future. What next ?
Percolator (saved search)
36
Elastic Graph (recommendations)
Freetext search
37
Conclusion
Here is a simple question - if we had the
possibility to go back in the time and start the
same journey with Elasticsearch,
would we do it the same way ?
Q & A
38

More Related Content

What's hot

Find your data
Find your dataFind your data
Find your data
Oliver Busse
 
Elk meetup
Elk meetupElk meetup
Elk meetup
Asaf Yigal
 
DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...
DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...
DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...
Frank van der Linden
 
FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...
FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...
FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...
New Relic
 
Datadog- Monitoring In Motion
Datadog- Monitoring In Motion Datadog- Monitoring In Motion
Datadog- Monitoring In Motion
Cloud Native Apps SF
 
Kibana overview
Kibana overviewKibana overview
Kibana overview
Rinat Tainov
 
Kibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stackKibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stack
Sylvain Wallez
 
Intro stream processing.be meetup #1
Intro stream processing.be meetup #1Intro stream processing.be meetup #1
Intro stream processing.be meetup #1
Peter Vandenabeele
 
OSOM Operations in the Cloud
OSOM Operations in the CloudOSOM Operations in the Cloud
OSOM Operations in the Cloud
mstuparu
 
OSOM - Operations in the Cloud
OSOM - Operations in the CloudOSOM - Operations in the Cloud
OSOM - Operations in the Cloud
Marcela Oniga
 
Matt Chung (Independent) - Serverless application with AWS Lambda
Matt Chung (Independent) - Serverless application with AWS Lambda Matt Chung (Independent) - Serverless application with AWS Lambda
Matt Chung (Independent) - Serverless application with AWS Lambda
Outlyer
 
Recommendations at Scale
Recommendations at ScaleRecommendations at Scale
Recommendations at Scale
Evention
 
2016 SUTOL: React.js – High-Performance Client for Domino
2016 SUTOL: React.js – High-Performance Client for Domino2016 SUTOL: React.js – High-Performance Client for Domino
2016 SUTOL: React.js – High-Performance Client for Domino
Knut Herrmann
 
Operationnal challenges behind Serverless architectures by Laurent Bernaille
Operationnal challenges behind Serverless architectures by Laurent BernailleOperationnal challenges behind Serverless architectures by Laurent Bernaille
Operationnal challenges behind Serverless architectures by Laurent Bernaille
The Incredible Automation Day
 
GraphDb in XPages
GraphDb in XPagesGraphDb in XPages
GraphDb in XPages
Oliver Busse
 
From logging to monitoring to reactive insights - C Schneider
From logging to monitoring to reactive insights - C SchneiderFrom logging to monitoring to reactive insights - C Schneider
From logging to monitoring to reactive insights - C Schneider
mfrancis
 
Wcs-1785 How Watson, Bluemix, Cloudant and XPages can work together in a real...
Wcs-1785 How Watson, Bluemix, Cloudant and XPages can work together in a real...Wcs-1785 How Watson, Bluemix, Cloudant and XPages can work together in a real...
Wcs-1785 How Watson, Bluemix, Cloudant and XPages can work together in a real...
Frank van der Linden
 
Introducing Tupilak, Snowplow's unified log fabric
Introducing Tupilak, Snowplow's unified log fabricIntroducing Tupilak, Snowplow's unified log fabric
Introducing Tupilak, Snowplow's unified log fabric
Alexander Dean
 
Modern Monitoring and processing logs
Modern Monitoring and processing logsModern Monitoring and processing logs
Modern Monitoring and processing logs
Dmitry Lavrinenko
 
Icinga Camp San Diego 2016 - Enter the Metrics
Icinga Camp San Diego 2016 - Enter the MetricsIcinga Camp San Diego 2016 - Enter the Metrics
Icinga Camp San Diego 2016 - Enter the Metrics
Icinga
 

What's hot (20)

Find your data
Find your dataFind your data
Find your data
 
Elk meetup
Elk meetupElk meetup
Elk meetup
 
DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...
DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...
DEV-1129 How Watson, Bluemix, Cloudant, and XPages Can Work Together In A Rea...
 
FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...
FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...
FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engin...
 
Datadog- Monitoring In Motion
Datadog- Monitoring In Motion Datadog- Monitoring In Motion
Datadog- Monitoring In Motion
 
Kibana overview
Kibana overviewKibana overview
Kibana overview
 
Kibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stackKibana + timelion: time series with the elastic stack
Kibana + timelion: time series with the elastic stack
 
Intro stream processing.be meetup #1
Intro stream processing.be meetup #1Intro stream processing.be meetup #1
Intro stream processing.be meetup #1
 
OSOM Operations in the Cloud
OSOM Operations in the CloudOSOM Operations in the Cloud
OSOM Operations in the Cloud
 
OSOM - Operations in the Cloud
OSOM - Operations in the CloudOSOM - Operations in the Cloud
OSOM - Operations in the Cloud
 
Matt Chung (Independent) - Serverless application with AWS Lambda
Matt Chung (Independent) - Serverless application with AWS Lambda Matt Chung (Independent) - Serverless application with AWS Lambda
Matt Chung (Independent) - Serverless application with AWS Lambda
 
Recommendations at Scale
Recommendations at ScaleRecommendations at Scale
Recommendations at Scale
 
2016 SUTOL: React.js – High-Performance Client for Domino
2016 SUTOL: React.js – High-Performance Client for Domino2016 SUTOL: React.js – High-Performance Client for Domino
2016 SUTOL: React.js – High-Performance Client for Domino
 
Operationnal challenges behind Serverless architectures by Laurent Bernaille
Operationnal challenges behind Serverless architectures by Laurent BernailleOperationnal challenges behind Serverless architectures by Laurent Bernaille
Operationnal challenges behind Serverless architectures by Laurent Bernaille
 
GraphDb in XPages
GraphDb in XPagesGraphDb in XPages
GraphDb in XPages
 
From logging to monitoring to reactive insights - C Schneider
From logging to monitoring to reactive insights - C SchneiderFrom logging to monitoring to reactive insights - C Schneider
From logging to monitoring to reactive insights - C Schneider
 
Wcs-1785 How Watson, Bluemix, Cloudant and XPages can work together in a real...
Wcs-1785 How Watson, Bluemix, Cloudant and XPages can work together in a real...Wcs-1785 How Watson, Bluemix, Cloudant and XPages can work together in a real...
Wcs-1785 How Watson, Bluemix, Cloudant and XPages can work together in a real...
 
Introducing Tupilak, Snowplow's unified log fabric
Introducing Tupilak, Snowplow's unified log fabricIntroducing Tupilak, Snowplow's unified log fabric
Introducing Tupilak, Snowplow's unified log fabric
 
Modern Monitoring and processing logs
Modern Monitoring and processing logsModern Monitoring and processing logs
Modern Monitoring and processing logs
 
Icinga Camp San Diego 2016 - Enter the Metrics
Icinga Camp San Diego 2016 - Enter the MetricsIcinga Camp San Diego 2016 - Enter the Metrics
Icinga Camp San Diego 2016 - Enter the Metrics
 

Viewers also liked

Conheça-me
Conheça-meConheça-me
Conheça-me
Stefano Manzolli
 
Unidad educativa experimental universal misionera st
Unidad educativa experimental universal misionera stUnidad educativa experimental universal misionera st
Unidad educativa experimental universal misionera st
Carlos Stewar
 
11th Docker Switzerland User Group Meetup
11th Docker Switzerland User Group Meetup11th Docker Switzerland User Group Meetup
11th Docker Switzerland User Group Meetup
Philipp Grossenbacher
 
Film noir – conventions
Film noir – conventionsFilm noir – conventions
Film noir – conventions
mhutchinson4
 
11thDockerMeetupSwitzerland
11thDockerMeetupSwitzerland11thDockerMeetupSwitzerland
11thDockerMeetupSwitzerland
Michael Mueller
 
Gt4.pibid4
Gt4.pibid4Gt4.pibid4
Gt4.pibid4
pibidmat
 
EXPOSURE - San Diego Benefits
EXPOSURE - San Diego BenefitsEXPOSURE - San Diego Benefits
EXPOSURE - San Diego Benefits
exposureskate
 
Presentación concepto de negocios Mingreso Inteligente - mii 2016
Presentación concepto de negocios Mingreso Inteligente  - mii 2016Presentación concepto de negocios Mingreso Inteligente  - mii 2016
Presentación concepto de negocios Mingreso Inteligente - mii 2016
Mi Ingreso Inteligente
 
Presentación1
Presentación1Presentación1
Presentación1
martacons2001
 
Organizational Change and Innovation
Organizational Change and Innovation Organizational Change and Innovation
Organizational Change and Innovation
majhapa
 
Progressive Web Apps (español - spanish)
Progressive Web Apps (español - spanish)Progressive Web Apps (español - spanish)
Progressive Web Apps (español - spanish)
Maximiliano Firtman
 
Progressive Web Apps
Progressive Web AppsProgressive Web Apps
Progressive Web Apps
Toninho Sousa
 
Chapter 2 sales and marketing strategy of sm es
Chapter 2  sales and marketing strategy of sm esChapter 2  sales and marketing strategy of sm es
Chapter 2 sales and marketing strategy of sm es
Prof.(Dr.) Hong K. D.Litt, D.Sc., PhD.ក្រោយបណ្ឌិត
 
Nuevas Terapias para la Obesidad Como controlar el Apetito y la Saciedad La v...
Nuevas Terapias para la Obesidad Como controlar el Apetito y la Saciedad La v...Nuevas Terapias para la Obesidad Como controlar el Apetito y la Saciedad La v...
Nuevas Terapias para la Obesidad Como controlar el Apetito y la Saciedad La v...
Conferencia Sindrome Metabolico
 
Chapter 1 introduction definition problem &amp; solving factor effect to sme
Chapter 1  introduction  definition problem &amp; solving factor effect to smeChapter 1  introduction  definition problem &amp; solving factor effect to sme
Chapter 1 introduction definition problem &amp; solving factor effect to sme
Prof.(Dr.) Hong K. D.Litt, D.Sc., PhD.ក្រោយបណ្ឌិត
 
Bluetooth Low Energy - A Case Study
Bluetooth Low Energy - A Case StudyBluetooth Low Energy - A Case Study
Bluetooth Low Energy - A Case Study
FReeze FRancis
 
Agile transformation_keynote
Agile transformation_keynoteAgile transformation_keynote
Agile transformation_keynote
Provectus
 
Designing Business Models
Designing Business ModelsDesigning Business Models
Designing Business Models
Yves Pigneur
 
Turku &lt;3 Frontend - Progressive Web Apps, Web and Apps
Turku &lt;3 Frontend - Progressive Web Apps, Web and AppsTurku &lt;3 Frontend - Progressive Web Apps, Web and Apps
Turku &lt;3 Frontend - Progressive Web Apps, Web and Apps
Joni Juup
 

Viewers also liked (19)

Conheça-me
Conheça-meConheça-me
Conheça-me
 
Unidad educativa experimental universal misionera st
Unidad educativa experimental universal misionera stUnidad educativa experimental universal misionera st
Unidad educativa experimental universal misionera st
 
11th Docker Switzerland User Group Meetup
11th Docker Switzerland User Group Meetup11th Docker Switzerland User Group Meetup
11th Docker Switzerland User Group Meetup
 
Film noir – conventions
Film noir – conventionsFilm noir – conventions
Film noir – conventions
 
11thDockerMeetupSwitzerland
11thDockerMeetupSwitzerland11thDockerMeetupSwitzerland
11thDockerMeetupSwitzerland
 
Gt4.pibid4
Gt4.pibid4Gt4.pibid4
Gt4.pibid4
 
EXPOSURE - San Diego Benefits
EXPOSURE - San Diego BenefitsEXPOSURE - San Diego Benefits
EXPOSURE - San Diego Benefits
 
Presentación concepto de negocios Mingreso Inteligente - mii 2016
Presentación concepto de negocios Mingreso Inteligente  - mii 2016Presentación concepto de negocios Mingreso Inteligente  - mii 2016
Presentación concepto de negocios Mingreso Inteligente - mii 2016
 
Presentación1
Presentación1Presentación1
Presentación1
 
Organizational Change and Innovation
Organizational Change and Innovation Organizational Change and Innovation
Organizational Change and Innovation
 
Progressive Web Apps (español - spanish)
Progressive Web Apps (español - spanish)Progressive Web Apps (español - spanish)
Progressive Web Apps (español - spanish)
 
Progressive Web Apps
Progressive Web AppsProgressive Web Apps
Progressive Web Apps
 
Chapter 2 sales and marketing strategy of sm es
Chapter 2  sales and marketing strategy of sm esChapter 2  sales and marketing strategy of sm es
Chapter 2 sales and marketing strategy of sm es
 
Nuevas Terapias para la Obesidad Como controlar el Apetito y la Saciedad La v...
Nuevas Terapias para la Obesidad Como controlar el Apetito y la Saciedad La v...Nuevas Terapias para la Obesidad Como controlar el Apetito y la Saciedad La v...
Nuevas Terapias para la Obesidad Como controlar el Apetito y la Saciedad La v...
 
Chapter 1 introduction definition problem &amp; solving factor effect to sme
Chapter 1  introduction  definition problem &amp; solving factor effect to smeChapter 1  introduction  definition problem &amp; solving factor effect to sme
Chapter 1 introduction definition problem &amp; solving factor effect to sme
 
Bluetooth Low Energy - A Case Study
Bluetooth Low Energy - A Case StudyBluetooth Low Energy - A Case Study
Bluetooth Low Energy - A Case Study
 
Agile transformation_keynote
Agile transformation_keynoteAgile transformation_keynote
Agile transformation_keynote
 
Designing Business Models
Designing Business ModelsDesigning Business Models
Designing Business Models
 
Turku &lt;3 Frontend - Progressive Web Apps, Web and Apps
Turku &lt;3 Frontend - Progressive Web Apps, Web and AppsTurku &lt;3 Frontend - Progressive Web Apps, Web and Apps
Turku &lt;3 Frontend - Progressive Web Apps, Web and Apps
 

Similar to Finding Cars and Hunting Down Logs - ElasticSearch @AutoScout24

Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCP
Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCPSimpler, faster, cheaper Enterprise Apps using only Spring Boot on GCP
Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCP
Daniel Zivkovic
 
Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...
Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...
Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...
Landon Robinson
 
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy MonitoringApache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Databricks
 
Serverless in production, an experience report
Serverless in production, an experience reportServerless in production, an experience report
Serverless in production, an experience report
Yan Cui
 
Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)
Yan Cui
 
Serverless in Production, an experience report (AWS UG South Wales)
Serverless in Production, an experience report (AWS UG South Wales)Serverless in Production, an experience report (AWS UG South Wales)
Serverless in Production, an experience report (AWS UG South Wales)
Yan Cui
 
Launching Your First Big Data Project on AWS
Launching Your First Big Data Project on AWSLaunching Your First Big Data Project on AWS
Launching Your First Big Data Project on AWS
Amazon Web Services
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Demi Ben-Ari
 
agile microservices @scaibo
agile microservices @scaiboagile microservices @scaibo
agile microservices @scaibo
Ciro Donato Caiazzo
 
Life of a startup - Sjoerd Mulder - Codemotion Amsterdam 2017
Life of a startup - Sjoerd Mulder - Codemotion Amsterdam 2017Life of a startup - Sjoerd Mulder - Codemotion Amsterdam 2017
Life of a startup - Sjoerd Mulder - Codemotion Amsterdam 2017
Codemotion
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Demi Ben-Ari
 
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Codemotion
 
Discussion for Anomaly & Prediction Engine
Discussion for Anomaly & Prediction EngineDiscussion for Anomaly & Prediction Engine
Discussion for Anomaly & Prediction Engine
HisashiOsanai
 
Empowering the AWS DynamoDB™ application developer with Alternator
Empowering the AWS DynamoDB™ application developer with AlternatorEmpowering the AWS DynamoDB™ application developer with Alternator
Empowering the AWS DynamoDB™ application developer with Alternator
ScyllaDB
 
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Codemotion
 
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
Demi Ben-Ari
 
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
Nane Kratzke
 
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and KafkaStream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
Databricks
 
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)
Maarten Balliauw
 
Data Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEAData Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEA
Andrew Morgan
 

Similar to Finding Cars and Hunting Down Logs - ElasticSearch @AutoScout24 (20)

Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCP
Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCPSimpler, faster, cheaper Enterprise Apps using only Spring Boot on GCP
Simpler, faster, cheaper Enterprise Apps using only Spring Boot on GCP
 
Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...
Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...
Spark + AI Summit 2019: Apache Spark Listeners: A Crash Course in Fast, Easy ...
 
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy MonitoringApache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
 
Serverless in production, an experience report
Serverless in production, an experience reportServerless in production, an experience report
Serverless in production, an experience report
 
Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)Serverless in production, an experience report (FullStack 2018)
Serverless in production, an experience report (FullStack 2018)
 
Serverless in Production, an experience report (AWS UG South Wales)
Serverless in Production, an experience report (AWS UG South Wales)Serverless in Production, an experience report (AWS UG South Wales)
Serverless in Production, an experience report (AWS UG South Wales)
 
Launching Your First Big Data Project on AWS
Launching Your First Big Data Project on AWSLaunching Your First Big Data Project on AWS
Launching Your First Big Data Project on AWS
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
 
agile microservices @scaibo
agile microservices @scaiboagile microservices @scaibo
agile microservices @scaibo
 
Life of a startup - Sjoerd Mulder - Codemotion Amsterdam 2017
Life of a startup - Sjoerd Mulder - Codemotion Amsterdam 2017Life of a startup - Sjoerd Mulder - Codemotion Amsterdam 2017
Life of a startup - Sjoerd Mulder - Codemotion Amsterdam 2017
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
 
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
 
Discussion for Anomaly & Prediction Engine
Discussion for Anomaly & Prediction EngineDiscussion for Anomaly & Prediction Engine
Discussion for Anomaly & Prediction Engine
 
Empowering the AWS DynamoDB™ application developer with Alternator
Empowering the AWS DynamoDB™ application developer with AlternatorEmpowering the AWS DynamoDB™ application developer with Alternator
Empowering the AWS DynamoDB™ application developer with Alternator
 
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
 
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
 
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
Smuggling Multi-Cloud Support into Cloud-native Applications using Elastic Co...
 
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and KafkaStream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
Stream, Stream, Stream: Different Streaming Methods with Apache Spark and Kafka
 
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)
Get more than a cache back! The Microsoft Azure Redis Cache (NDC Oslo)
 
Data Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEAData Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEA
 

More from Philipp Garbe

Run Jenkins as Managed Product on ECS - AWS Meetup
Run Jenkins as Managed Product on ECS - AWS MeetupRun Jenkins as Managed Product on ECS - AWS Meetup
Run Jenkins as Managed Product on ECS - AWS Meetup
Philipp Garbe
 
Is Platform Engineering the new Ops?
Is Platform Engineering the new Ops?Is Platform Engineering the new Ops?
Is Platform Engineering the new Ops?
Philipp Garbe
 
Managed Container Orchestration with Amazon ECS
Managed Container Orchestration with Amazon ECSManaged Container Orchestration with Amazon ECS
Managed Container Orchestration with Amazon ECS
Philipp Garbe
 
Deliver Docker Containers Continuously on AWS - QCon 2017
Deliver Docker Containers Continuously on AWS - QCon 2017Deliver Docker Containers Continuously on AWS - QCon 2017
Deliver Docker Containers Continuously on AWS - QCon 2017
Philipp Garbe
 
Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016
Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016
Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016
Philipp Garbe
 
Docker Container automatisiert nach AWS deployen - Continuous Lifecycle 2016
Docker Container automatisiert nach AWS deployen  - Continuous Lifecycle 2016Docker Container automatisiert nach AWS deployen  - Continuous Lifecycle 2016
Docker Container automatisiert nach AWS deployen - Continuous Lifecycle 2016
Philipp Garbe
 
Deliver docker containers continuously on aws
Deliver docker containers continuously on awsDeliver docker containers continuously on aws
Deliver docker containers continuously on aws
Philipp Garbe
 
ElasticSearch on AWS
ElasticSearch on AWSElasticSearch on AWS
ElasticSearch on AWS
Philipp Garbe
 
DockerCon 2016 Seattle Recap
DockerCon 2016 Seattle RecapDockerCon 2016 Seattle Recap
DockerCon 2016 Seattle Recap
Philipp Garbe
 

More from Philipp Garbe (9)

Run Jenkins as Managed Product on ECS - AWS Meetup
Run Jenkins as Managed Product on ECS - AWS MeetupRun Jenkins as Managed Product on ECS - AWS Meetup
Run Jenkins as Managed Product on ECS - AWS Meetup
 
Is Platform Engineering the new Ops?
Is Platform Engineering the new Ops?Is Platform Engineering the new Ops?
Is Platform Engineering the new Ops?
 
Managed Container Orchestration with Amazon ECS
Managed Container Orchestration with Amazon ECSManaged Container Orchestration with Amazon ECS
Managed Container Orchestration with Amazon ECS
 
Deliver Docker Containers Continuously on AWS - QCon 2017
Deliver Docker Containers Continuously on AWS - QCon 2017Deliver Docker Containers Continuously on AWS - QCon 2017
Deliver Docker Containers Continuously on AWS - QCon 2017
 
Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016
Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016
Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016
 
Docker Container automatisiert nach AWS deployen - Continuous Lifecycle 2016
Docker Container automatisiert nach AWS deployen  - Continuous Lifecycle 2016Docker Container automatisiert nach AWS deployen  - Continuous Lifecycle 2016
Docker Container automatisiert nach AWS deployen - Continuous Lifecycle 2016
 
Deliver docker containers continuously on aws
Deliver docker containers continuously on awsDeliver docker containers continuously on aws
Deliver docker containers continuously on aws
 
ElasticSearch on AWS
ElasticSearch on AWSElasticSearch on AWS
ElasticSearch on AWS
 
DockerCon 2016 Seattle Recap
DockerCon 2016 Seattle RecapDockerCon 2016 Seattle Recap
DockerCon 2016 Seattle Recap
 

Recently uploaded

manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaalmanuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
wolfsoftcompanyco
 
制作毕业证书(ANU毕业证)莫纳什大学毕业证成绩单官方原版办理
制作毕业证书(ANU毕业证)莫纳什大学毕业证成绩单官方原版办理制作毕业证书(ANU毕业证)莫纳什大学毕业证成绩单官方原版办理
制作毕业证书(ANU毕业证)莫纳什大学毕业证成绩单官方原版办理
cuobya
 
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
uehowe
 
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
ysasp1
 
Search Result Showing My Post is Now Buried
Search Result Showing My Post is Now BuriedSearch Result Showing My Post is Now Buried
Search Result Showing My Post is Now Buried
Trish Parr
 
Gen Z and the marketplaces - let's translate their needs
Gen Z and the marketplaces - let's translate their needsGen Z and the marketplaces - let's translate their needs
Gen Z and the marketplaces - let's translate their needs
Laura Szabó
 
Design Thinking NETFLIX using all techniques.pptx
Design Thinking NETFLIX using all techniques.pptxDesign Thinking NETFLIX using all techniques.pptx
Design Thinking NETFLIX using all techniques.pptx
saathvikreddy2003
 
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
ukwwuq
 
Discover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to IndiaDiscover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to India
davidjhones387
 
Azure EA Sponsorship - Customer Guide.pdf
Azure EA Sponsorship - Customer Guide.pdfAzure EA Sponsorship - Customer Guide.pdf
Azure EA Sponsorship - Customer Guide.pdf
AanSulistiyo
 
重新申请毕业证书(RMIT毕业证)皇家墨尔本理工大学毕业证成绩单精仿办理
重新申请毕业证书(RMIT毕业证)皇家墨尔本理工大学毕业证成绩单精仿办理重新申请毕业证书(RMIT毕业证)皇家墨尔本理工大学毕业证成绩单精仿办理
重新申请毕业证书(RMIT毕业证)皇家墨尔本理工大学毕业证成绩单精仿办理
vmemo1
 
国外证书(Lincoln毕业证)新西兰林肯大学毕业证成绩单不能毕业办理
国外证书(Lincoln毕业证)新西兰林肯大学毕业证成绩单不能毕业办理国外证书(Lincoln毕业证)新西兰林肯大学毕业证成绩单不能毕业办理
国外证书(Lincoln毕业证)新西兰林肯大学毕业证成绩单不能毕业办理
zoowe
 
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
uehowe
 
学位认证网(DU毕业证)迪肯大学毕业证成绩单一比一原版制作
学位认证网(DU毕业证)迪肯大学毕业证成绩单一比一原版制作学位认证网(DU毕业证)迪肯大学毕业证成绩单一比一原版制作
学位认证网(DU毕业证)迪肯大学毕业证成绩单一比一原版制作
zyfovom
 
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdfMeet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Florence Consulting
 
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
cuobya
 
7 Best Cloud Hosting Services to Try Out in 2024
7 Best Cloud Hosting Services to Try Out in 20247 Best Cloud Hosting Services to Try Out in 2024
7 Best Cloud Hosting Services to Try Out in 2024
Danica Gill
 
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
xjq03c34
 
Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?
Paul Walk
 
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
cuobya
 

Recently uploaded (20)

manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaalmanuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
 
制作毕业证书(ANU毕业证)莫纳什大学毕业证成绩单官方原版办理
制作毕业证书(ANU毕业证)莫纳什大学毕业证成绩单官方原版办理制作毕业证书(ANU毕业证)莫纳什大学毕业证成绩单官方原版办理
制作毕业证书(ANU毕业证)莫纳什大学毕业证成绩单官方原版办理
 
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
 
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
 
Search Result Showing My Post is Now Buried
Search Result Showing My Post is Now BuriedSearch Result Showing My Post is Now Buried
Search Result Showing My Post is Now Buried
 
Gen Z and the marketplaces - let's translate their needs
Gen Z and the marketplaces - let's translate their needsGen Z and the marketplaces - let's translate their needs
Gen Z and the marketplaces - let's translate their needs
 
Design Thinking NETFLIX using all techniques.pptx
Design Thinking NETFLIX using all techniques.pptxDesign Thinking NETFLIX using all techniques.pptx
Design Thinking NETFLIX using all techniques.pptx
 
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
 
Discover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to IndiaDiscover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to India
 
Azure EA Sponsorship - Customer Guide.pdf
Azure EA Sponsorship - Customer Guide.pdfAzure EA Sponsorship - Customer Guide.pdf
Azure EA Sponsorship - Customer Guide.pdf
 
重新申请毕业证书(RMIT毕业证)皇家墨尔本理工大学毕业证成绩单精仿办理
重新申请毕业证书(RMIT毕业证)皇家墨尔本理工大学毕业证成绩单精仿办理重新申请毕业证书(RMIT毕业证)皇家墨尔本理工大学毕业证成绩单精仿办理
重新申请毕业证书(RMIT毕业证)皇家墨尔本理工大学毕业证成绩单精仿办理
 
国外证书(Lincoln毕业证)新西兰林肯大学毕业证成绩单不能毕业办理
国外证书(Lincoln毕业证)新西兰林肯大学毕业证成绩单不能毕业办理国外证书(Lincoln毕业证)新西兰林肯大学毕业证成绩单不能毕业办理
国外证书(Lincoln毕业证)新西兰林肯大学毕业证成绩单不能毕业办理
 
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
 
学位认证网(DU毕业证)迪肯大学毕业证成绩单一比一原版制作
学位认证网(DU毕业证)迪肯大学毕业证成绩单一比一原版制作学位认证网(DU毕业证)迪肯大学毕业证成绩单一比一原版制作
学位认证网(DU毕业证)迪肯大学毕业证成绩单一比一原版制作
 
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdfMeet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdf
 
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
 
7 Best Cloud Hosting Services to Try Out in 2024
7 Best Cloud Hosting Services to Try Out in 20247 Best Cloud Hosting Services to Try Out in 2024
7 Best Cloud Hosting Services to Try Out in 2024
 
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
 
Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?
 
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
 

Finding Cars and Hunting Down Logs - ElasticSearch @AutoScout24

  • 1. Finding Cars and Hunting Down Logs: Elasticsearch @ AutoScout24 AutoScout24 24 Nov 2016 Philipp Garbe Lead developer (philipp.garbe@scout24.com) Juri Smarschevski Team lead (juri.smarschevski@scout24.com)
  • 3. Who we are ? Unique Monthly Visitors in Europa 3 …10 more
  • 4. Some numbers Search index contains ~2.6M classifieds 4 Unique visitors (monthly): ~10M Search requests per day: ~36M Index update rate per day ~400.000 classifieds
  • 5. Status quo. March 2013. Endeca used as a search engine 5 Use case: providing search results and facets for the entire AS24 platform Problems: • New product requirements, performance of Endeca becomes slower • Time to market of our required features is not sufficient • Maintenance is complex / expensive
  • 6. Possible candidates Solr ? • <feeling> too complex installation / configuration </feeling> 6 Sphinx ? • Support situation is unclear Elasticsearch ? • Fresh buzzword • From beginning on built for distributed systems (rumors) • Easy installation / configuration (fact)
  • 7. POC Goals • Performance should be comparable with Endeca • The solution should be scalable 7
  • 8. 8 Rollout plan. 03.2013 - 11.2013 07.2013 11.201302.2013 03.2013 05.2013 POC Implementation & migration Training Go live phase #real_project_picture_squeezed
  • 9. 9 Endeca Elasticsearch (0.9.x) Amount of machines 60 20 [Re]index time ~180 min ~45 min Deploy to Live up to 2 days < 3 hours Effort for testing an issue on local machine 4 h 1 h Performance = = Product / dev guys satisfaction :( :) 300% 400% 1000% 400% % ? Results after 8 months of working.
  • 10. No problems after migration ? Cluster split brain Has in fact nothing to do with Elasticsearch, is more related to learn phase at AS24 10 Deep pagination Elasticsearch 5.x release notes: “Deep pagination of search results is now possible with the search_after feature, which efficiently skips over previously returned results to return just the next page.“
  • 11. 11 Status quo. November 2014. Project “Tatsu” has started .NET => JVM C# => Scala IIS / Windows => Play / Linux Local data center => AWS Monolith => Micro services Windows workstations => Mac notebooks ... => ...
  • 12. 12 Status quo. November 2014. Project “Tatsu” has started .NET => JVM C# => Scala IIS / Windows => Play / Linux Local data center => AWS Monolith => Micro services Windows workstations => Mac notebooks ? => ? => 2015
  • 13. 13 Elasticsearch clusters “lift & shift” to AWS ? AWS Elasticsearch Service ? Elasticsearch as a service (SaaS) ? Own hosting in AWS ?
  • 14. 16 Rolling update in details (possible scenario). Time 1 Initial state
  • 15. 17 Rolling update in details (possible scenario). Node has been replaced Time 1 2 Initial state ~ 60 sec
  • 16. 18 Rolling update in details (possible scenario). Master has been killed Node has been replaced Time 1 2 3 Initial state
  • 17. 19 Rolling update in details (possible scenario). Master has been killed Node has been replaced Master election Time 1 2 3 4 Initial state
  • 18. 20 Rolling update in details (possible scenario). Master has been killed Node has been replaced Master election Time 1 2 3 4 5 Initial state Last node has been replaced
  • 19. 21 Rolling update findings Master has been killed ?Outage=
  • 22. 7.4 billion documents Some numbers 36 TB EBS 18 nodes á m4.4xlarge (64GB / 53.5 cpu units)
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 32. First setup ● 18x m4.4xlarge ● 18x 2TB gp2 ● 3TB/day cross-zone traffic
  • 33. Cost/Usage Optimized Setup ● 15x m4.x2large ● 15x 384GB gp2 ● 6x SpotFleet ● 6x 4TB st1 ● 9TB/day cross-zone traffic Savings: ~40%
  • 34. Future. What next ? Percolator (saved search) 36 Elastic Graph (recommendations) Freetext search
  • 35. 37 Conclusion Here is a simple question - if we had the possibility to go back in the time and start the same journey with Elasticsearch, would we do it the same way ?