SlideShare a Scribd company logo
Using Elasticsearch for Analytics
How we use Elasticsearch for Analytics at Wingify?
Vaidik Kapoor
github.com/vaidik
twitter.com/vaidikkapoor
Problem Statement
VWO collects number of visitors and conversions per goal per variation for every campaign
created. These numbers are used by our customers to make optimization decisions - very
useful but limiting as these numbers are overall numbers and drilling down was not possible.
There is a need to develop an analytics engine:
● capable of storing millions of daily data points, essentially JSON docs.
● should expose flexible and powerful query interface for segmenting visitors and
conversions data. This is extremely useful for our customers to derive insights.
● querying should not be extremely slow - response times of 2-5 seconds are acceptable.
● not too difficult to maintain in production - operations should be easy for a lean team.
● should be easy to extend to provide new features.
● A distributed near real-time search engine, also considered as an analytics
engine since a lot of people use it that way - proven solution.
● Highly available, fault tolerant, distributed - built from the ground up to work
in the cloud.
● Elasticsearch is distributed - cluster management takes care of node
downtimes which makes operations rather easy instead of being a headache.
Application development remains the same no matter how you deploy
Elasticsearch i.e. a cluster or single node.
● Capable of performing all the major types of searches, matches and
aggregations. Also supports limited Regular Expressions.
● Easy index and replica creation on live cluster.
● Easy management of cluster and indices through REST API.
to the rescue
1. Store a document for every unique
visitor per campaign in Elasticsearch.
Document contains:
a. Visitor related segment
properties like geo data,
platform information, referral,
etc.
b. Information related to
conversion of goals
2. Use Nested Types for creating
hierarchy between every unique
visitor’s visit and conversions.
3. Use Aggregations/Facets framework
for generating datewise count of
visitors and conversions and basic stats
like average and total revenue, sum of
squares of revenue, etc.
4. Never use script facets/aggs to get
counts of a combination of values from
the same document. Scripts are slow.
Instead index result of script at index
time.
Visitor documents in Elasticsearch:
{
"account": 196,
"experiment": 77,
"combination": "5",
"hit_time": "2014-07-09T23:21:15",
"ip": "71.12.234.0"
"os": "Android",
"os_version": "4.1.2",
"device": "Huawei Y301A2",
"device_type": "Mobile",
"touch_capable": true,
"browser": "Android",
"browser_version": "4.1.2",
"document_encoding": "UTF-8",
"user_language": "en-us",
"city": "Mandeville",
"country": "United States",
"region": "Louisiana",
"url": "https://vwo.com/free-
trial",
"query_params": [],
"direct_traffic": true,
"search_traffic": false,
"email_traffic": false,
"returning_visitor": false,
"converted_goals": [...],
...
}
How we use Elasticsearch
"converted_goals": [
{
"id": 2,
"facet_term": "5_2",
"conversion_time":
"2014-07-09T23:32:41"
},
{
"id": 6,
"facet_term": "5_6",
"conversion_time":
"2014-07-09T23:37:04"
}
]
Alongside Elasticsearch as our primary data store, we use a bunch of other
things:
● RabbitMQ - our central queue which receives all the analytics data
and pushes to all the consumers which write to different data stores
including Elasticsearch and MySQL.
● MySQL for storing overall counters of visitors and conversions per
goal per variations of every campaign. This serves as a cache in front
of Elasticsearch - prevents us from calculating total counts by
iterating over all the documents and makes loading of reports faster.
● Consumers - written in Python, responsible for sanitizing and storing
data in Elasticsearch and MySQL. New visitors are inserted as a
document in Elasticsearch. Conversions of existing visitors are
recorded in the document previously inserted for the visitor that
converted using Elasticsearch’s Update API (Script Updates).
● Analytics API Server - written in Python using Flask, Gevent and
Celery
○ Exposes APIs for querying segmented data and for other
tasks such as start tracking campaign, flushing campaign
data, flushing account data, etc.
○ Provides a custom JSON based Query DSL which makes the
Query API easy to consumer. The API server translates this
Query DSL to Elasticsearch’s DSL. Example:
{
“and”: [
{ “or”: [ { “city”: “New Delhi” },
{ “city”: “Gurgaon” } ] },
{ “not”: { “device_type”: “Mobile” } }
]
}
Current Architecture
USA West AsiaEuropeUSA East
Data Acquisition Servers
Central Queue
1 2 3 4
Consumers / Workers
Front-end
Application
Analytics API
Server
U
pdate
counters
Sync visitors and
conversions
Elasticsearch scales, only when planned for. Consider the following:
● Make your data shardable - cannot emphasize enough on this. If you cannot shard your data, then
scaling out will always be a problem, especially with time-series data as it always grows. There are
options like user and time based indices. You may shard according to something else. Find what works
for you.
● Use routing to scale reads. Without routing, queries will hit all the shards to find lesser number of
documents out of total documents per shard (difficult to find needle in a larger haystack). If you have
a lot of shards, then ES will not return unless response from all the shards have arrived and
aggregated at the node that received the request.
● Avoid hotspots because of routing. Sometimes some shards can have a lot more data as compared to
rest of the shards.
● Use Bulk API for the right things - updating or deleting large number of documents on adhoc basis,
bulk indexing from another source, etc.
● Increase the number of shards per index for data distribution but keep it sane if you are creating too
many indices (like per day) as shards are resource hungry.
● Increase replica count to get higher search throughput.
Plan for Scaling
● Elasticsearch does not have ACL - important if you are dealing with user data.
○ There are existing 3rd party plugins for ACL.
○ In our opinion, run Elasticsearch behind Nginx (or Apache) and let Nginx take care of
ACL. This can be easily achieved using Nginx + Lua. You may use something equivalent.
● Have dedicated Master nodes - these will ensure that Elasticsearch’s cluster management
does not stop (important for HA). Master-only nodes can run on relatively small machines as
compared to Data nodes.
● Disable deleting of indices using wildcards or _all to avoid the most obvious disaster.
● Spend some time with the JVM. Monitor resource consumption, especially memory and see
which Garbage Collector is working the best for you. For us, G1GC worked better than CMS
due to high indexing rate requirement.
● Consider using Doc Values - major advantage is that it takes off memory management out of
JVM and let the kernel do the memory management for disk cache.
● Use the Snapshot API and prepare to use Restore API, hoping you never really have to.
● Consider rolling restarts with Optimizing indices before restart.
Ops - What We Learned

More Related Content

What's hot

Introduction to elasticsearch
Introduction to elasticsearchIntroduction to elasticsearch
Introduction to elasticsearch
hypto
 
What's new in Elasticsearch v5
What's new in Elasticsearch v5What's new in Elasticsearch v5
What's new in Elasticsearch v5
Idan Tohami
 
Log analysis using Logstash,ElasticSearch and Kibana
Log analysis using Logstash,ElasticSearch and KibanaLog analysis using Logstash,ElasticSearch and Kibana
Log analysis using Logstash,ElasticSearch and Kibana
Avinash Ramineni
 
ECS위에 Log Server 구축하기
ECS위에 Log Server 구축하기ECS위에 Log Server 구축하기
ECS위에 Log Server 구축하기
AWSKRUG - AWS한국사용자모임
 
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming
Building Realtime Data Pipelines with Kafka Connect and Spark StreamingBuilding Realtime Data Pipelines with Kafka Connect and Spark Streaming
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming
Jen Aman
 
Elastic stack Presentation
Elastic stack PresentationElastic stack Presentation
Elastic stack Presentation
Amr Alaa Yassen
 
Data Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageData Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby Usage
SATOSHI TAGOMORI
 
Open Source DataViz with Apache Superset
Open Source DataViz with Apache SupersetOpen Source DataViz with Apache Superset
Open Source DataViz with Apache Superset
Carl W. Handlin
 
To Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToTo Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT To
SATOSHI TAGOMORI
 
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Cohesive Networks
 
Webinar: Event Processing & Data Analytics with Lucidworks Fusion
Webinar: Event Processing & Data Analytics with Lucidworks FusionWebinar: Event Processing & Data Analytics with Lucidworks Fusion
Webinar: Event Processing & Data Analytics with Lucidworks Fusion
Lucidworks
 
tdtechtalk20160330johan
tdtechtalk20160330johantdtechtalk20160330johan
tdtechtalk20160330johan
Johan Gustavsson
 
Scala and jvm_languages_praveen_technologist
Scala and jvm_languages_praveen_technologistScala and jvm_languages_praveen_technologist
Scala and jvm_languages_praveen_technologist
pmanvi
 
WSO2Con ASIA 2016: An Introduction to the WSO2 Analytics Platform
WSO2Con ASIA 2016: An Introduction to the WSO2 Analytics PlatformWSO2Con ASIA 2016: An Introduction to the WSO2 Analytics Platform
WSO2Con ASIA 2016: An Introduction to the WSO2 Analytics Platform
WSO2
 
GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...
GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...
GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...
Yann Cluchey
 
Ahsay Backup Software v7 - Datasheet
Ahsay Backup Software v7 - DatasheetAhsay Backup Software v7 - Datasheet
Ahsay Backup Software v7 - Datasheet
Ronnie Chan
 
ElasticSearch Basic Introduction
ElasticSearch Basic IntroductionElasticSearch Basic Introduction
ElasticSearch Basic Introduction
Mayur Rathod
 
Scaling horizontally on AWS
Scaling horizontally on AWSScaling horizontally on AWS
Scaling horizontally on AWS
Bozhidar Bozhanov
 
Building a unified data pipeline in Apache Spark
Building a unified data pipeline in Apache SparkBuilding a unified data pipeline in Apache Spark
Building a unified data pipeline in Apache SparkDataWorks Summit
 

What's hot (20)

Introduction to elasticsearch
Introduction to elasticsearchIntroduction to elasticsearch
Introduction to elasticsearch
 
What's new in Elasticsearch v5
What's new in Elasticsearch v5What's new in Elasticsearch v5
What's new in Elasticsearch v5
 
Log analysis using Logstash,ElasticSearch and Kibana
Log analysis using Logstash,ElasticSearch and KibanaLog analysis using Logstash,ElasticSearch and Kibana
Log analysis using Logstash,ElasticSearch and Kibana
 
ECS위에 Log Server 구축하기
ECS위에 Log Server 구축하기ECS위에 Log Server 구축하기
ECS위에 Log Server 구축하기
 
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming
Building Realtime Data Pipelines with Kafka Connect and Spark StreamingBuilding Realtime Data Pipelines with Kafka Connect and Spark Streaming
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming
 
Elastic stack Presentation
Elastic stack PresentationElastic stack Presentation
Elastic stack Presentation
 
Data Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageData Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby Usage
 
Open Source DataViz with Apache Superset
Open Source DataViz with Apache SupersetOpen Source DataViz with Apache Superset
Open Source DataViz with Apache Superset
 
To Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToTo Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT To
 
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
 
Webinar: Event Processing & Data Analytics with Lucidworks Fusion
Webinar: Event Processing & Data Analytics with Lucidworks FusionWebinar: Event Processing & Data Analytics with Lucidworks Fusion
Webinar: Event Processing & Data Analytics with Lucidworks Fusion
 
tdtechtalk20160330johan
tdtechtalk20160330johantdtechtalk20160330johan
tdtechtalk20160330johan
 
Scala and jvm_languages_praveen_technologist
Scala and jvm_languages_praveen_technologistScala and jvm_languages_praveen_technologist
Scala and jvm_languages_praveen_technologist
 
963
963963
963
 
WSO2Con ASIA 2016: An Introduction to the WSO2 Analytics Platform
WSO2Con ASIA 2016: An Introduction to the WSO2 Analytics PlatformWSO2Con ASIA 2016: An Introduction to the WSO2 Analytics Platform
WSO2Con ASIA 2016: An Introduction to the WSO2 Analytics Platform
 
GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...
GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...
GOTO Aarhus 2014: Making Enterprise Data Available in Real Time with elastics...
 
Ahsay Backup Software v7 - Datasheet
Ahsay Backup Software v7 - DatasheetAhsay Backup Software v7 - Datasheet
Ahsay Backup Software v7 - Datasheet
 
ElasticSearch Basic Introduction
ElasticSearch Basic IntroductionElasticSearch Basic Introduction
ElasticSearch Basic Introduction
 
Scaling horizontally on AWS
Scaling horizontally on AWSScaling horizontally on AWS
Scaling horizontally on AWS
 
Building a unified data pipeline in Apache Spark
Building a unified data pipeline in Apache SparkBuilding a unified data pipeline in Apache Spark
Building a unified data pipeline in Apache Spark
 

Similar to Using Elasticsearch for Analytics

Azure Data Explorer deep dive - review 04.2020
Azure Data Explorer deep dive - review 04.2020Azure Data Explorer deep dive - review 04.2020
Azure Data Explorer deep dive - review 04.2020
Riccardo Zamana
 
Headless approach for offloading heavy tasks in Magento
Headless approach for offloading heavy tasks in MagentoHeadless approach for offloading heavy tasks in Magento
Headless approach for offloading heavy tasks in Magento
Sander Mangel
 
Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cl...
Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cl...Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cl...
Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cl...
Certus Solutions
 
Lessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatternsLessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatternsClaudiu Barbura
 
Bigdata.sunil_6+yearsExp
Bigdata.sunil_6+yearsExpBigdata.sunil_6+yearsExp
Bigdata.sunil_6+yearsExpbigdata sunil
 
ADV Slides: Comparing the Enterprise Analytic Solutions
ADV Slides: Comparing the Enterprise Analytic SolutionsADV Slides: Comparing the Enterprise Analytic Solutions
ADV Slides: Comparing the Enterprise Analytic Solutions
DATAVERSITY
 
1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for release1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for release
Jen Stirrup
 
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Piyush Kumar
 
Elastic Stack: Using data for insight and action
Elastic Stack: Using data for insight and actionElastic Stack: Using data for insight and action
Elastic Stack: Using data for insight and action
Elasticsearch
 
WSO2 Product Release Webinar: WSO2 Data Analytics Server 3.0
WSO2 Product Release Webinar: WSO2 Data Analytics Server 3.0WSO2 Product Release Webinar: WSO2 Data Analytics Server 3.0
WSO2 Product Release Webinar: WSO2 Data Analytics Server 3.0
WSO2
 
Azure Monitoring Overview
Azure Monitoring OverviewAzure Monitoring Overview
Azure Monitoring Overview
gjuljo
 
WSO2 Data Analytics Server - Product Overview
WSO2 Data Analytics Server - Product OverviewWSO2 Data Analytics Server - Product Overview
WSO2 Data Analytics Server - Product Overview
WSO2
 
James Turner (Caplin) - Enterprise HTML5 Patterns
James Turner (Caplin) - Enterprise HTML5 PatternsJames Turner (Caplin) - Enterprise HTML5 Patterns
James Turner (Caplin) - Enterprise HTML5 Patterns
akqaanoraks
 
SplunkLive! Milano 2016 - customer presentation - Unicredit
SplunkLive! Milano 2016 -  customer presentation - UnicreditSplunkLive! Milano 2016 -  customer presentation - Unicredit
SplunkLive! Milano 2016 - customer presentation - Unicredit
Splunk
 
AWS Big Data in everyday use at Yle
AWS Big Data in everyday use at YleAWS Big Data in everyday use at Yle
AWS Big Data in everyday use at Yle
Rolf Koski
 
Real time analytics on deep learning @ strata data 2019
Real time analytics on deep learning @ strata data 2019Real time analytics on deep learning @ strata data 2019
Real time analytics on deep learning @ strata data 2019
Zhenxiao Luo
 
Serverless_with_MongoDB
Serverless_with_MongoDBServerless_with_MongoDB
Serverless_with_MongoDB
Amazon Web Services
 
Streamlio and IoT analytics with Apache Pulsar
Streamlio and IoT analytics with Apache PulsarStreamlio and IoT analytics with Apache Pulsar
Streamlio and IoT analytics with Apache Pulsar
Streamlio
 
Log Analysis Engine with Integration of Hadoop and Spark
Log Analysis Engine with Integration of Hadoop and SparkLog Analysis Engine with Integration of Hadoop and Spark
Log Analysis Engine with Integration of Hadoop and Spark
IRJET Journal
 

Similar to Using Elasticsearch for Analytics (20)

Azure Data Explorer deep dive - review 04.2020
Azure Data Explorer deep dive - review 04.2020Azure Data Explorer deep dive - review 04.2020
Azure Data Explorer deep dive - review 04.2020
 
Headless approach for offloading heavy tasks in Magento
Headless approach for offloading heavy tasks in MagentoHeadless approach for offloading heavy tasks in Magento
Headless approach for offloading heavy tasks in Magento
 
Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cl...
Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cl...Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cl...
Melbourne: Certus Data 2.0 Vault Meetup with Snowflake - Data Vault In The Cl...
 
Lessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatternsLessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatterns
 
Bigdata.sunil_6+yearsExp
Bigdata.sunil_6+yearsExpBigdata.sunil_6+yearsExp
Bigdata.sunil_6+yearsExp
 
ADV Slides: Comparing the Enterprise Analytic Solutions
ADV Slides: Comparing the Enterprise Analytic SolutionsADV Slides: Comparing the Enterprise Analytic Solutions
ADV Slides: Comparing the Enterprise Analytic Solutions
 
1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for release1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for release
 
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
Importance of ‘Centralized Event collection’ and BigData platform for Analysis !
 
Elastic Stack: Using data for insight and action
Elastic Stack: Using data for insight and actionElastic Stack: Using data for insight and action
Elastic Stack: Using data for insight and action
 
WSO2 Product Release Webinar: WSO2 Data Analytics Server 3.0
WSO2 Product Release Webinar: WSO2 Data Analytics Server 3.0WSO2 Product Release Webinar: WSO2 Data Analytics Server 3.0
WSO2 Product Release Webinar: WSO2 Data Analytics Server 3.0
 
Azure Monitoring Overview
Azure Monitoring OverviewAzure Monitoring Overview
Azure Monitoring Overview
 
WSO2 Data Analytics Server - Product Overview
WSO2 Data Analytics Server - Product OverviewWSO2 Data Analytics Server - Product Overview
WSO2 Data Analytics Server - Product Overview
 
James Turner (Caplin) - Enterprise HTML5 Patterns
James Turner (Caplin) - Enterprise HTML5 PatternsJames Turner (Caplin) - Enterprise HTML5 Patterns
James Turner (Caplin) - Enterprise HTML5 Patterns
 
SplunkLive! Milano 2016 - customer presentation - Unicredit
SplunkLive! Milano 2016 -  customer presentation - UnicreditSplunkLive! Milano 2016 -  customer presentation - Unicredit
SplunkLive! Milano 2016 - customer presentation - Unicredit
 
AWS Big Data in everyday use at Yle
AWS Big Data in everyday use at YleAWS Big Data in everyday use at Yle
AWS Big Data in everyday use at Yle
 
Real time analytics on deep learning @ strata data 2019
Real time analytics on deep learning @ strata data 2019Real time analytics on deep learning @ strata data 2019
Real time analytics on deep learning @ strata data 2019
 
Cassandra in xPatterns
Cassandra in xPatternsCassandra in xPatterns
Cassandra in xPatterns
 
Serverless_with_MongoDB
Serverless_with_MongoDBServerless_with_MongoDB
Serverless_with_MongoDB
 
Streamlio and IoT analytics with Apache Pulsar
Streamlio and IoT analytics with Apache PulsarStreamlio and IoT analytics with Apache Pulsar
Streamlio and IoT analytics with Apache Pulsar
 
Log Analysis Engine with Integration of Hadoop and Spark
Log Analysis Engine with Integration of Hadoop and SparkLog Analysis Engine with Integration of Hadoop and Spark
Log Analysis Engine with Integration of Hadoop and Spark
 

More from Vaidik Kapoor

Understanding Non Blocking I/O with Python
Understanding Non Blocking I/O with PythonUnderstanding Non Blocking I/O with Python
Understanding Non Blocking I/O with Python
Vaidik Kapoor
 
Vagrant for Effective DevOps Culture
Vagrant for Effective DevOps CultureVagrant for Effective DevOps Culture
Vagrant for Effective DevOps Culture
Vaidik Kapoor
 
Building an event/conference website like FUDCon.in
Building an event/conference website like FUDCon.inBuilding an event/conference website like FUDCon.in
Building an event/conference website like FUDCon.in
Vaidik Kapoor
 
Queue Everything and Please Everyone
Queue Everything and Please EveryoneQueue Everything and Please Everyone
Queue Everything and Please Everyone
Vaidik Kapoor
 
Firefox Extension Development | By JIIT OSDC
Firefox Extension Development | By JIIT OSDCFirefox Extension Development | By JIIT OSDC
Firefox Extension Development | By JIIT OSDCVaidik Kapoor
 

More from Vaidik Kapoor (6)

Understanding Non Blocking I/O with Python
Understanding Non Blocking I/O with PythonUnderstanding Non Blocking I/O with Python
Understanding Non Blocking I/O with Python
 
Vagrant for Effective DevOps Culture
Vagrant for Effective DevOps CultureVagrant for Effective DevOps Culture
Vagrant for Effective DevOps Culture
 
Building an event/conference website like FUDCon.in
Building an event/conference website like FUDCon.inBuilding an event/conference website like FUDCon.in
Building an event/conference website like FUDCon.in
 
Queue Everything and Please Everyone
Queue Everything and Please EveryoneQueue Everything and Please Everyone
Queue Everything and Please Everyone
 
Version Controlling
Version ControllingVersion Controlling
Version Controlling
 
Firefox Extension Development | By JIIT OSDC
Firefox Extension Development | By JIIT OSDCFirefox Extension Development | By JIIT OSDC
Firefox Extension Development | By JIIT OSDC
 

Recently uploaded

一比一原版(UofS毕业证书)萨省大学毕业证如何办理
一比一原版(UofS毕业证书)萨省大学毕业证如何办理一比一原版(UofS毕业证书)萨省大学毕业证如何办理
一比一原版(UofS毕业证书)萨省大学毕业证如何办理
v3tuleee
 
【社内勉強会資料_Octo: An Open-Source Generalist Robot Policy】
【社内勉強会資料_Octo: An Open-Source Generalist Robot Policy】【社内勉強会資料_Octo: An Open-Source Generalist Robot Policy】
【社内勉強会資料_Octo: An Open-Source Generalist Robot Policy】
NABLAS株式会社
 
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
axoqas
 
Quantitative Data AnalysisReliability Analysis (Cronbach Alpha) Common Method...
Quantitative Data AnalysisReliability Analysis (Cronbach Alpha) Common Method...Quantitative Data AnalysisReliability Analysis (Cronbach Alpha) Common Method...
Quantitative Data AnalysisReliability Analysis (Cronbach Alpha) Common Method...
2023240532
 
The affect of service quality and online reviews on customer loyalty in the E...
The affect of service quality and online reviews on customer loyalty in the E...The affect of service quality and online reviews on customer loyalty in the E...
The affect of service quality and online reviews on customer loyalty in the E...
jerlynmaetalle
 
一比一原版(UPenn毕业证)宾夕法尼亚大学毕业证成绩单
一比一原版(UPenn毕业证)宾夕法尼亚大学毕业证成绩单一比一原版(UPenn毕业证)宾夕法尼亚大学毕业证成绩单
一比一原版(UPenn毕业证)宾夕法尼亚大学毕业证成绩单
ewymefz
 
Q1’2024 Update: MYCI’s Leap Year Rebound
Q1’2024 Update: MYCI’s Leap Year ReboundQ1’2024 Update: MYCI’s Leap Year Rebound
Q1’2024 Update: MYCI’s Leap Year Rebound
Oppotus
 
一比一原版(UniSA毕业证书)南澳大学毕业证如何办理
一比一原版(UniSA毕业证书)南澳大学毕业证如何办理一比一原版(UniSA毕业证书)南澳大学毕业证如何办理
一比一原版(UniSA毕业证书)南澳大学毕业证如何办理
slg6lamcq
 
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
vcaxypu
 
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
ahzuo
 
Machine learning and optimization techniques for electrical drives.pptx
Machine learning and optimization techniques for electrical drives.pptxMachine learning and optimization techniques for electrical drives.pptx
Machine learning and optimization techniques for electrical drives.pptx
balafet
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
slg6lamcq
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
74nqk8xf
 
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
nscud
 
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
nscud
 
一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单
enxupq
 
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
mbawufebxi
 
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdfCriminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP
 
一比一原版(QU毕业证)皇后大学毕业证成绩单
一比一原版(QU毕业证)皇后大学毕业证成绩单一比一原版(QU毕业证)皇后大学毕业证成绩单
一比一原版(QU毕业证)皇后大学毕业证成绩单
enxupq
 

Recently uploaded (20)

一比一原版(UofS毕业证书)萨省大学毕业证如何办理
一比一原版(UofS毕业证书)萨省大学毕业证如何办理一比一原版(UofS毕业证书)萨省大学毕业证如何办理
一比一原版(UofS毕业证书)萨省大学毕业证如何办理
 
【社内勉強会資料_Octo: An Open-Source Generalist Robot Policy】
【社内勉強会資料_Octo: An Open-Source Generalist Robot Policy】【社内勉強会資料_Octo: An Open-Source Generalist Robot Policy】
【社内勉強会資料_Octo: An Open-Source Generalist Robot Policy】
 
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
哪里卖(usq毕业证书)南昆士兰大学毕业证研究生文凭证书托福证书原版一模一样
 
Quantitative Data AnalysisReliability Analysis (Cronbach Alpha) Common Method...
Quantitative Data AnalysisReliability Analysis (Cronbach Alpha) Common Method...Quantitative Data AnalysisReliability Analysis (Cronbach Alpha) Common Method...
Quantitative Data AnalysisReliability Analysis (Cronbach Alpha) Common Method...
 
The affect of service quality and online reviews on customer loyalty in the E...
The affect of service quality and online reviews on customer loyalty in the E...The affect of service quality and online reviews on customer loyalty in the E...
The affect of service quality and online reviews on customer loyalty in the E...
 
一比一原版(UPenn毕业证)宾夕法尼亚大学毕业证成绩单
一比一原版(UPenn毕业证)宾夕法尼亚大学毕业证成绩单一比一原版(UPenn毕业证)宾夕法尼亚大学毕业证成绩单
一比一原版(UPenn毕业证)宾夕法尼亚大学毕业证成绩单
 
Q1’2024 Update: MYCI’s Leap Year Rebound
Q1’2024 Update: MYCI’s Leap Year ReboundQ1’2024 Update: MYCI’s Leap Year Rebound
Q1’2024 Update: MYCI’s Leap Year Rebound
 
一比一原版(UniSA毕业证书)南澳大学毕业证如何办理
一比一原版(UniSA毕业证书)南澳大学毕业证如何办理一比一原版(UniSA毕业证书)南澳大学毕业证如何办理
一比一原版(UniSA毕业证书)南澳大学毕业证如何办理
 
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
 
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
 
Machine learning and optimization techniques for electrical drives.pptx
Machine learning and optimization techniques for electrical drives.pptxMachine learning and optimization techniques for electrical drives.pptx
Machine learning and optimization techniques for electrical drives.pptx
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
 
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
 
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
 
一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单
 
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
一比一原版(Bradford毕业证书)布拉德福德大学毕业证如何办理
 
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdfCriminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdf
 
一比一原版(QU毕业证)皇后大学毕业证成绩单
一比一原版(QU毕业证)皇后大学毕业证成绩单一比一原版(QU毕业证)皇后大学毕业证成绩单
一比一原版(QU毕业证)皇后大学毕业证成绩单
 

Using Elasticsearch for Analytics

  • 1. Using Elasticsearch for Analytics How we use Elasticsearch for Analytics at Wingify? Vaidik Kapoor github.com/vaidik twitter.com/vaidikkapoor
  • 2. Problem Statement VWO collects number of visitors and conversions per goal per variation for every campaign created. These numbers are used by our customers to make optimization decisions - very useful but limiting as these numbers are overall numbers and drilling down was not possible. There is a need to develop an analytics engine: ● capable of storing millions of daily data points, essentially JSON docs. ● should expose flexible and powerful query interface for segmenting visitors and conversions data. This is extremely useful for our customers to derive insights. ● querying should not be extremely slow - response times of 2-5 seconds are acceptable. ● not too difficult to maintain in production - operations should be easy for a lean team. ● should be easy to extend to provide new features.
  • 3. ● A distributed near real-time search engine, also considered as an analytics engine since a lot of people use it that way - proven solution. ● Highly available, fault tolerant, distributed - built from the ground up to work in the cloud. ● Elasticsearch is distributed - cluster management takes care of node downtimes which makes operations rather easy instead of being a headache. Application development remains the same no matter how you deploy Elasticsearch i.e. a cluster or single node. ● Capable of performing all the major types of searches, matches and aggregations. Also supports limited Regular Expressions. ● Easy index and replica creation on live cluster. ● Easy management of cluster and indices through REST API. to the rescue
  • 4. 1. Store a document for every unique visitor per campaign in Elasticsearch. Document contains: a. Visitor related segment properties like geo data, platform information, referral, etc. b. Information related to conversion of goals 2. Use Nested Types for creating hierarchy between every unique visitor’s visit and conversions. 3. Use Aggregations/Facets framework for generating datewise count of visitors and conversions and basic stats like average and total revenue, sum of squares of revenue, etc. 4. Never use script facets/aggs to get counts of a combination of values from the same document. Scripts are slow. Instead index result of script at index time. Visitor documents in Elasticsearch: { "account": 196, "experiment": 77, "combination": "5", "hit_time": "2014-07-09T23:21:15", "ip": "71.12.234.0" "os": "Android", "os_version": "4.1.2", "device": "Huawei Y301A2", "device_type": "Mobile", "touch_capable": true, "browser": "Android", "browser_version": "4.1.2", "document_encoding": "UTF-8", "user_language": "en-us", "city": "Mandeville", "country": "United States", "region": "Louisiana", "url": "https://vwo.com/free- trial", "query_params": [], "direct_traffic": true, "search_traffic": false, "email_traffic": false, "returning_visitor": false, "converted_goals": [...], ... } How we use Elasticsearch "converted_goals": [ { "id": 2, "facet_term": "5_2", "conversion_time": "2014-07-09T23:32:41" }, { "id": 6, "facet_term": "5_6", "conversion_time": "2014-07-09T23:37:04" } ]
  • 5. Alongside Elasticsearch as our primary data store, we use a bunch of other things: ● RabbitMQ - our central queue which receives all the analytics data and pushes to all the consumers which write to different data stores including Elasticsearch and MySQL. ● MySQL for storing overall counters of visitors and conversions per goal per variations of every campaign. This serves as a cache in front of Elasticsearch - prevents us from calculating total counts by iterating over all the documents and makes loading of reports faster. ● Consumers - written in Python, responsible for sanitizing and storing data in Elasticsearch and MySQL. New visitors are inserted as a document in Elasticsearch. Conversions of existing visitors are recorded in the document previously inserted for the visitor that converted using Elasticsearch’s Update API (Script Updates). ● Analytics API Server - written in Python using Flask, Gevent and Celery ○ Exposes APIs for querying segmented data and for other tasks such as start tracking campaign, flushing campaign data, flushing account data, etc. ○ Provides a custom JSON based Query DSL which makes the Query API easy to consumer. The API server translates this Query DSL to Elasticsearch’s DSL. Example: { “and”: [ { “or”: [ { “city”: “New Delhi” }, { “city”: “Gurgaon” } ] }, { “not”: { “device_type”: “Mobile” } } ] } Current Architecture USA West AsiaEuropeUSA East Data Acquisition Servers Central Queue 1 2 3 4 Consumers / Workers Front-end Application Analytics API Server U pdate counters Sync visitors and conversions
  • 6. Elasticsearch scales, only when planned for. Consider the following: ● Make your data shardable - cannot emphasize enough on this. If you cannot shard your data, then scaling out will always be a problem, especially with time-series data as it always grows. There are options like user and time based indices. You may shard according to something else. Find what works for you. ● Use routing to scale reads. Without routing, queries will hit all the shards to find lesser number of documents out of total documents per shard (difficult to find needle in a larger haystack). If you have a lot of shards, then ES will not return unless response from all the shards have arrived and aggregated at the node that received the request. ● Avoid hotspots because of routing. Sometimes some shards can have a lot more data as compared to rest of the shards. ● Use Bulk API for the right things - updating or deleting large number of documents on adhoc basis, bulk indexing from another source, etc. ● Increase the number of shards per index for data distribution but keep it sane if you are creating too many indices (like per day) as shards are resource hungry. ● Increase replica count to get higher search throughput. Plan for Scaling
  • 7. ● Elasticsearch does not have ACL - important if you are dealing with user data. ○ There are existing 3rd party plugins for ACL. ○ In our opinion, run Elasticsearch behind Nginx (or Apache) and let Nginx take care of ACL. This can be easily achieved using Nginx + Lua. You may use something equivalent. ● Have dedicated Master nodes - these will ensure that Elasticsearch’s cluster management does not stop (important for HA). Master-only nodes can run on relatively small machines as compared to Data nodes. ● Disable deleting of indices using wildcards or _all to avoid the most obvious disaster. ● Spend some time with the JVM. Monitor resource consumption, especially memory and see which Garbage Collector is working the best for you. For us, G1GC worked better than CMS due to high indexing rate requirement. ● Consider using Doc Values - major advantage is that it takes off memory management out of JVM and let the kernel do the memory management for disk cache. ● Use the Snapshot API and prepare to use Restore API, hoping you never really have to. ● Consider rolling restarts with Optimizing indices before restart. Ops - What We Learned