SlideShare a Scribd company logo
Scaling UP
Challenges Encountered Scaling Up
Recommendation Services @Gravity R&D
Bottyán Németh
Who we are and what we do
Gravity R&D is a recommender system vendor company.
We provide recommendation as a service since 2009 for
our customers all around the globe.
2
How we imagine growth?
3
?
How we imagine growth?
4
How it actually happens?
5
?
How it actually happens?
6
# of requests
7
Vatera.hu largest online marketplace in Hungary
served by one “server”
Alexa TOP100 video chat webpage
(~40M recommendation requests / day):
 Served by 5 application servers and 1 DB
 Too many events to store in MySQL  using
Cassandra (v0.6)
 Training time for IALS too long  speedup by IALS1
 Max. 5 sec latency in “product” availability
Using new/beta technologies
8
Cassandra (v0.6)
Nginx (v0.5) (22% of top 1M sites)
Kafka (v0.8)
MySQL auto. failover
Reaching the limits
9
Even if the technology is widely used if you reach it’s
limits the optimization is very costly / time consuming.
Java GC – service collapsed because increased minor GC
times due to a JVM bug (26th of January 2013)
Maintaining MySQL with lots of data (optimize table,
slave replication lag, faster storage device)
Complexity increases
10
There is always a business request or an algorithmic
development which requires more resources.
Optimizations
11
Infrastructure
12
Currently 200+ hosts and 3500+ services monitored
0
50
100
150
200
250
2008 2009 2010 2011 2012 2013 2014 2015 2016
Number of servers
# of items
13
How to store item model / metadata in memory to serve
requests fast?
# of items
14
How to store item model / metadata in memory to serve
requests fast?
VS.
Auto increment IDs for the items?
231 not enough
Preconceptions
15
More data better results.
If the CTR of a new algorithm is low than the old
algorithm is better.
Daily retrain is enough.
Training frequency
16
CTR decreased in the morning
100+ Algorithms
17
0
10
20
30
40
50
60
0 20 40 60 80 100 120
Number of times an algorithm is used
Now
18
• Performance: Gravity’s performance
oriented architecture enables real-time
response to the always changing
environment and user behavior
• Algorithms: more than 100 different
recommendation algorithm enables true
personalization and to reach the highest
KPIs in different domains
• Infrastructure: fast response times all
around the globe and data security thanks
to the private cloud infrastructure located
in 4 different data centers
• Flexibility: the advanced business rule
engine with intuitive user interface allows
to satisfy various business requirements
Performance
140M requests
served daily
Algorithms
30 man-years
invested
Infrastructure
4 data centers
globally
Flexibility
100s of logics
configurable
Cross the river when you come to it
19
Thank you!
20

More Related Content

What's hot

GluonCV
GluonCVGluonCV
GluonCV
Soji Adeshina
 
The challenges of live events scalability
The challenges of live events scalabilityThe challenges of live events scalability
The challenges of live events scalability
Guy Tomer
 
ASTQB washington-sept-2015
ASTQB washington-sept-2015ASTQB washington-sept-2015
ASTQB washington-sept-2015
Dan Boutin
 
Microsoft AI Platform - AETHER Introduction
Microsoft AI Platform - AETHER IntroductionMicrosoft AI Platform - AETHER Introduction
Microsoft AI Platform - AETHER Introduction
Karthik Murugesan
 
Sri Rajan - Driving cloud adoption through DevOps / Unlocked: the Hybrid Clou...
Sri Rajan - Driving cloud adoption through DevOps / Unlocked: the Hybrid Clou...Sri Rajan - Driving cloud adoption through DevOps / Unlocked: the Hybrid Clou...
Sri Rajan - Driving cloud adoption through DevOps / Unlocked: the Hybrid Clou...
Rackspace Academy
 
What is changed in products/service licensing with Cloud?
What is changed in products/service licensing with Cloud?What is changed in products/service licensing with Cloud?
What is changed in products/service licensing with Cloud?
Tomislav Lulic
 
Industrial Data Science
Industrial Data ScienceIndustrial Data Science
Industrial Data Science
Niko Vuokko
 
ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...
ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...
ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...
Johann Schleier-Smith
 
Rail Performance in the Cloud - Opening
Rail Performance in the Cloud - OpeningRail Performance in the Cloud - Opening
Rail Performance in the Cloud - Opening
Engine Yard
 
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBASolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
Razorleaf Corporation
 
EIA2017Italy - Danny Lange - Artificial Intelligence - A Game Changer in App ...
EIA2017Italy - Danny Lange - Artificial Intelligence - A Game Changer in App ...EIA2017Italy - Danny Lange - Artificial Intelligence - A Game Changer in App ...
EIA2017Italy - Danny Lange - Artificial Intelligence - A Game Changer in App ...
European Innovation Academy
 
AWS Webcast - Tibco Jaspersoft
AWS Webcast - Tibco JaspersoftAWS Webcast - Tibco Jaspersoft
AWS Webcast - Tibco Jaspersoft
Amazon Web Services
 
12 Ways to Manage Cloud Costs and Optimize Cloud Spend
12 Ways to Manage Cloud Costs and Optimize Cloud Spend12 Ways to Manage Cloud Costs and Optimize Cloud Spend
12 Ways to Manage Cloud Costs and Optimize Cloud Spend
RightScale
 
Real time machine learning
Real time machine learningReal time machine learning
Real time machine learning
Vinoth Kannan
 
Creating a Culture of Cost Management in Your Organization – J.R. Storment, C...
Creating a Culture of Cost Management in Your Organization – J.R. Storment, C...Creating a Culture of Cost Management in Your Organization – J.R. Storment, C...
Creating a Culture of Cost Management in Your Organization – J.R. Storment, C...
Amazon Web Services
 
Big Data in Production: Lessons from Running in the Cloud
Big Data in Production: Lessons from Running in the CloudBig Data in Production: Lessons from Running in the Cloud
Big Data in Production: Lessons from Running in the Cloud
Jen Aman
 
SnapLogic Overview: Are You Feeling SMACT?
SnapLogic Overview: Are You Feeling SMACT?SnapLogic Overview: Are You Feeling SMACT?
SnapLogic Overview: Are You Feeling SMACT?
SnapLogic
 
How Companies are Using Cloud-Based Data Visualization & Analytics to Transfo...
How Companies are Using Cloud-Based Data Visualization & Analytics to Transfo...How Companies are Using Cloud-Based Data Visualization & Analytics to Transfo...
How Companies are Using Cloud-Based Data Visualization & Analytics to Transfo...
Amazon Web Services
 
AWS Webcast - Journey through the Cloud - Cost Optimization
AWS Webcast - Journey through the Cloud - Cost OptimizationAWS Webcast - Journey through the Cloud - Cost Optimization
AWS Webcast - Journey through the Cloud - Cost Optimization
Amazon Web Services
 
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
Data Con LA
 

What's hot (20)

GluonCV
GluonCVGluonCV
GluonCV
 
The challenges of live events scalability
The challenges of live events scalabilityThe challenges of live events scalability
The challenges of live events scalability
 
ASTQB washington-sept-2015
ASTQB washington-sept-2015ASTQB washington-sept-2015
ASTQB washington-sept-2015
 
Microsoft AI Platform - AETHER Introduction
Microsoft AI Platform - AETHER IntroductionMicrosoft AI Platform - AETHER Introduction
Microsoft AI Platform - AETHER Introduction
 
Sri Rajan - Driving cloud adoption through DevOps / Unlocked: the Hybrid Clou...
Sri Rajan - Driving cloud adoption through DevOps / Unlocked: the Hybrid Clou...Sri Rajan - Driving cloud adoption through DevOps / Unlocked: the Hybrid Clou...
Sri Rajan - Driving cloud adoption through DevOps / Unlocked: the Hybrid Clou...
 
What is changed in products/service licensing with Cloud?
What is changed in products/service licensing with Cloud?What is changed in products/service licensing with Cloud?
What is changed in products/service licensing with Cloud?
 
Industrial Data Science
Industrial Data ScienceIndustrial Data Science
Industrial Data Science
 
ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...
ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...
ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent P...
 
Rail Performance in the Cloud - Opening
Rail Performance in the Cloud - OpeningRail Performance in the Cloud - Opening
Rail Performance in the Cloud - Opening
 
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBASolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
SolidWorks Design Automation Using the SolidWorks API, Microsoft Excel and VBA
 
EIA2017Italy - Danny Lange - Artificial Intelligence - A Game Changer in App ...
EIA2017Italy - Danny Lange - Artificial Intelligence - A Game Changer in App ...EIA2017Italy - Danny Lange - Artificial Intelligence - A Game Changer in App ...
EIA2017Italy - Danny Lange - Artificial Intelligence - A Game Changer in App ...
 
AWS Webcast - Tibco Jaspersoft
AWS Webcast - Tibco JaspersoftAWS Webcast - Tibco Jaspersoft
AWS Webcast - Tibco Jaspersoft
 
12 Ways to Manage Cloud Costs and Optimize Cloud Spend
12 Ways to Manage Cloud Costs and Optimize Cloud Spend12 Ways to Manage Cloud Costs and Optimize Cloud Spend
12 Ways to Manage Cloud Costs and Optimize Cloud Spend
 
Real time machine learning
Real time machine learningReal time machine learning
Real time machine learning
 
Creating a Culture of Cost Management in Your Organization – J.R. Storment, C...
Creating a Culture of Cost Management in Your Organization – J.R. Storment, C...Creating a Culture of Cost Management in Your Organization – J.R. Storment, C...
Creating a Culture of Cost Management in Your Organization – J.R. Storment, C...
 
Big Data in Production: Lessons from Running in the Cloud
Big Data in Production: Lessons from Running in the CloudBig Data in Production: Lessons from Running in the Cloud
Big Data in Production: Lessons from Running in the Cloud
 
SnapLogic Overview: Are You Feeling SMACT?
SnapLogic Overview: Are You Feeling SMACT?SnapLogic Overview: Are You Feeling SMACT?
SnapLogic Overview: Are You Feeling SMACT?
 
How Companies are Using Cloud-Based Data Visualization & Analytics to Transfo...
How Companies are Using Cloud-Based Data Visualization & Analytics to Transfo...How Companies are Using Cloud-Based Data Visualization & Analytics to Transfo...
How Companies are Using Cloud-Based Data Visualization & Analytics to Transfo...
 
AWS Webcast - Journey through the Cloud - Cost Optimization
AWS Webcast - Journey through the Cloud - Cost OptimizationAWS Webcast - Journey through the Cloud - Cost Optimization
AWS Webcast - Journey through the Cloud - Cost Optimization
 
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
Big Data Day LA 2015 - Building a Big Data Culture in the Entertainment Indus...
 

Viewers also liked

Recommenders on video sharing portals - business and algorithmic aspects
Recommenders on video sharing portals - business and algorithmic aspectsRecommenders on video sharing portals - business and algorithmic aspects
Recommenders on video sharing portals - business and algorithmic aspects
Domonkos Tikk
 
Gravity rd corporate introduction - nlp matiné 2014
Gravity rd corporate introduction  - nlp matiné 2014Gravity rd corporate introduction  - nlp matiné 2014
Gravity rd corporate introduction - nlp matiné 2014
Zoltan Varju
 
Xây dựng mạng lưới tài năng trẻ trong sáng tạo – khởi nghiệp
Xây dựng mạng lưới tài năng trẻ trong sáng tạo – khởi nghiệpXây dựng mạng lưới tài năng trẻ trong sáng tạo – khởi nghiệp
Xây dựng mạng lưới tài năng trẻ trong sáng tạo – khởi nghiệp
Tri Dung, Tran
 
Gravity personalizaton intro
Gravity personalizaton introGravity personalizaton intro
Gravity personalizaton intro
Eszter Nagy
 
Entrepreneurship & Innovation: Dual-core Engine
Entrepreneurship & Innovation: Dual-core EngineEntrepreneurship & Innovation: Dual-core Engine
Entrepreneurship & Innovation: Dual-core Engine
Tri Dung, Tran
 
The rise of Recommendation Engines
The rise of Recommendation EnginesThe rise of Recommendation Engines
The rise of Recommendation Engines
lamnk
 
Lessons learnt at building recommendation services at industry scale
Lessons learnt at building recommendation services at industry scaleLessons learnt at building recommendation services at industry scale
Lessons learnt at building recommendation services at industry scale
Domonkos Tikk
 
RecSys 2015: Large-scale real-time product recommendation at Criteo
RecSys 2015: Large-scale real-time product recommendation at CriteoRecSys 2015: Large-scale real-time product recommendation at Criteo
RecSys 2015: Large-scale real-time product recommendation at Criteo
Romain Lerallut
 
Understanding How CQL3 Maps to Cassandra's Internal Data Structure
Understanding How CQL3 Maps to Cassandra's Internal Data StructureUnderstanding How CQL3 Maps to Cassandra's Internal Data Structure
Understanding How CQL3 Maps to Cassandra's Internal Data Structure
DataStax
 
Dynamically Allocate Cluster Resources to your Spark Application
Dynamically Allocate Cluster Resources to your Spark ApplicationDynamically Allocate Cluster Resources to your Spark Application
Dynamically Allocate Cluster Resources to your Spark Application
DataWorks Summit
 
Creating an end-to-end Recommender System with Apache Spark and Elasticsearch...
Creating an end-to-end Recommender System with Apache Spark and Elasticsearch...Creating an end-to-end Recommender System with Apache Spark and Elasticsearch...
Creating an end-to-end Recommender System with Apache Spark and Elasticsearch...
sparktc
 
Organizational-culture
Organizational-cultureOrganizational-culture
Organizational-culture
Udayraj Vichare
 
Centralization and Decentralization
Centralization and DecentralizationCentralization and Decentralization
Centralization and Decentralization
Dr. Vickram Aadityaa
 
Using Docker for GPU Accelerated Applications
Using Docker for GPU Accelerated ApplicationsUsing Docker for GPU Accelerated Applications
Using Docker for GPU Accelerated Applications
NVIDIA
 
LDA Beginner's Tutorial
LDA Beginner's TutorialLDA Beginner's Tutorial
LDA Beginner's Tutorial
Wayne Lee
 
10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems
Xavier Amatriain
 

Viewers also liked (16)

Recommenders on video sharing portals - business and algorithmic aspects
Recommenders on video sharing portals - business and algorithmic aspectsRecommenders on video sharing portals - business and algorithmic aspects
Recommenders on video sharing portals - business and algorithmic aspects
 
Gravity rd corporate introduction - nlp matiné 2014
Gravity rd corporate introduction  - nlp matiné 2014Gravity rd corporate introduction  - nlp matiné 2014
Gravity rd corporate introduction - nlp matiné 2014
 
Xây dựng mạng lưới tài năng trẻ trong sáng tạo – khởi nghiệp
Xây dựng mạng lưới tài năng trẻ trong sáng tạo – khởi nghiệpXây dựng mạng lưới tài năng trẻ trong sáng tạo – khởi nghiệp
Xây dựng mạng lưới tài năng trẻ trong sáng tạo – khởi nghiệp
 
Gravity personalizaton intro
Gravity personalizaton introGravity personalizaton intro
Gravity personalizaton intro
 
Entrepreneurship & Innovation: Dual-core Engine
Entrepreneurship & Innovation: Dual-core EngineEntrepreneurship & Innovation: Dual-core Engine
Entrepreneurship & Innovation: Dual-core Engine
 
The rise of Recommendation Engines
The rise of Recommendation EnginesThe rise of Recommendation Engines
The rise of Recommendation Engines
 
Lessons learnt at building recommendation services at industry scale
Lessons learnt at building recommendation services at industry scaleLessons learnt at building recommendation services at industry scale
Lessons learnt at building recommendation services at industry scale
 
RecSys 2015: Large-scale real-time product recommendation at Criteo
RecSys 2015: Large-scale real-time product recommendation at CriteoRecSys 2015: Large-scale real-time product recommendation at Criteo
RecSys 2015: Large-scale real-time product recommendation at Criteo
 
Understanding How CQL3 Maps to Cassandra's Internal Data Structure
Understanding How CQL3 Maps to Cassandra's Internal Data StructureUnderstanding How CQL3 Maps to Cassandra's Internal Data Structure
Understanding How CQL3 Maps to Cassandra's Internal Data Structure
 
Dynamically Allocate Cluster Resources to your Spark Application
Dynamically Allocate Cluster Resources to your Spark ApplicationDynamically Allocate Cluster Resources to your Spark Application
Dynamically Allocate Cluster Resources to your Spark Application
 
Creating an end-to-end Recommender System with Apache Spark and Elasticsearch...
Creating an end-to-end Recommender System with Apache Spark and Elasticsearch...Creating an end-to-end Recommender System with Apache Spark and Elasticsearch...
Creating an end-to-end Recommender System with Apache Spark and Elasticsearch...
 
Organizational-culture
Organizational-cultureOrganizational-culture
Organizational-culture
 
Centralization and Decentralization
Centralization and DecentralizationCentralization and Decentralization
Centralization and Decentralization
 
Using Docker for GPU Accelerated Applications
Using Docker for GPU Accelerated ApplicationsUsing Docker for GPU Accelerated Applications
Using Docker for GPU Accelerated Applications
 
LDA Beginner's Tutorial
LDA Beginner's TutorialLDA Beginner's Tutorial
LDA Beginner's Tutorial
 
10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems
 

Similar to Challenges Encountered by Scaling Up Recommendation Services at Gravity R&D

There are 250 Database products, are you running the right one?
There are 250 Database products, are you running the right one?There are 250 Database products, are you running the right one?
There are 250 Database products, are you running the right one?
Aerospike, Inc.
 
How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)
How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)
How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)
Ontico
 
The Cloud - What's different
The Cloud - What's differentThe Cloud - What's different
The Cloud - What's different
Chen-Tien Tsai
 
Serverless Computing: Driving Innovation and Business Value
Serverless Computing: Driving Innovation and Business ValueServerless Computing: Driving Innovation and Business Value
Serverless Computing: Driving Innovation and Business Value
Alibaba Cloud
 
Intro to AWS: Amazon EC2 and Compute Services
Intro to AWS: Amazon EC2 and Compute ServicesIntro to AWS: Amazon EC2 and Compute Services
Intro to AWS: Amazon EC2 and Compute Services
Amazon Web Services
 
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Continuent
 
Intro to AWS: Amazon EC2 and Compute Services
Intro to AWS: Amazon EC2 and Compute ServicesIntro to AWS: Amazon EC2 and Compute Services
Intro to AWS: Amazon EC2 and Compute Services
Amazon Web Services
 
Scale Your Load Balancer from 0 to 1 million TPS on Azure
Scale Your Load Balancer from 0 to 1 million TPS on AzureScale Your Load Balancer from 0 to 1 million TPS on Azure
Scale Your Load Balancer from 0 to 1 million TPS on Azure
Avi Networks
 
AWS Summit Kuala Lumpur Keynote with Stephen Orban - Head of Enterprise Strategy
AWS Summit Kuala Lumpur Keynote with Stephen Orban - Head of Enterprise StrategyAWS Summit Kuala Lumpur Keynote with Stephen Orban - Head of Enterprise Strategy
AWS Summit Kuala Lumpur Keynote with Stephen Orban - Head of Enterprise Strategy
Amazon Web Services
 
Proact SYNC 2013 Breakout session - NetApp Clustered DataONTAP, dé storage hy...
Proact SYNC 2013 Breakout session - NetApp Clustered DataONTAP, dé storage hy...Proact SYNC 2013 Breakout session - NetApp Clustered DataONTAP, dé storage hy...
Proact SYNC 2013 Breakout session - NetApp Clustered DataONTAP, dé storage hy...
Proact Netherlands B.V.
 
Intro to AWS: EC2 & Compute Services
Intro to AWS: EC2 & Compute ServicesIntro to AWS: EC2 & Compute Services
Intro to AWS: EC2 & Compute Services
Amazon Web Services
 
Data Culture Series - Keynote - 3rd Dec
Data Culture Series - Keynote - 3rd DecData Culture Series - Keynote - 3rd Dec
Data Culture Series - Keynote - 3rd Dec
Jonathan Woodward
 
Leveraging Big Data with Hadoop, NoSQL and RDBMS
Leveraging Big Data with Hadoop, NoSQL and RDBMSLeveraging Big Data with Hadoop, NoSQL and RDBMS
Leveraging Big Data with Hadoop, NoSQL and RDBMS
Aerospike, Inc.
 
Using real time big data analytics for competitive advantage
 Using real time big data analytics for competitive advantage Using real time big data analytics for competitive advantage
Using real time big data analytics for competitive advantage
Amazon Web Services
 
Five Early Challenges Of Building Streaming Fast Data Applications
Five Early Challenges Of Building Streaming Fast Data ApplicationsFive Early Challenges Of Building Streaming Fast Data Applications
Five Early Challenges Of Building Streaming Fast Data Applications
Lightbend
 
Data Virtualization Journey: How to Grow from Single Project and to Enterpris...
Data Virtualization Journey: How to Grow from Single Project and to Enterpris...Data Virtualization Journey: How to Grow from Single Project and to Enterpris...
Data Virtualization Journey: How to Grow from Single Project and to Enterpris...
Denodo
 
Introducing Amazon Kinesis: Real-time Processing of Streaming Big Data (BDT10...
Introducing Amazon Kinesis: Real-time Processing of Streaming Big Data (BDT10...Introducing Amazon Kinesis: Real-time Processing of Streaming Big Data (BDT10...
Introducing Amazon Kinesis: Real-time Processing of Streaming Big Data (BDT10...
Amazon Web Services
 
Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...
Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...
Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...
confluent
 
Vertica Analytics Database general overview
Vertica Analytics Database general overviewVertica Analytics Database general overview
Vertica Analytics Database general overview
Stratebi
 
Migrating from Oracle to Postgres
Migrating from Oracle to PostgresMigrating from Oracle to Postgres
Migrating from Oracle to Postgres
EDB
 

Similar to Challenges Encountered by Scaling Up Recommendation Services at Gravity R&D (20)

There are 250 Database products, are you running the right one?
There are 250 Database products, are you running the right one?There are 250 Database products, are you running the right one?
There are 250 Database products, are you running the right one?
 
How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)
How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)
How to run Real Time processing on Big Data / Ron Zavner (GigaSpaces)
 
The Cloud - What's different
The Cloud - What's differentThe Cloud - What's different
The Cloud - What's different
 
Serverless Computing: Driving Innovation and Business Value
Serverless Computing: Driving Innovation and Business ValueServerless Computing: Driving Innovation and Business Value
Serverless Computing: Driving Innovation and Business Value
 
Intro to AWS: Amazon EC2 and Compute Services
Intro to AWS: Amazon EC2 and Compute ServicesIntro to AWS: Amazon EC2 and Compute Services
Intro to AWS: Amazon EC2 and Compute Services
 
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
Webinar Slides: High Volume MySQL HA: SaaS Continuous Operations with Terabyt...
 
Intro to AWS: Amazon EC2 and Compute Services
Intro to AWS: Amazon EC2 and Compute ServicesIntro to AWS: Amazon EC2 and Compute Services
Intro to AWS: Amazon EC2 and Compute Services
 
Scale Your Load Balancer from 0 to 1 million TPS on Azure
Scale Your Load Balancer from 0 to 1 million TPS on AzureScale Your Load Balancer from 0 to 1 million TPS on Azure
Scale Your Load Balancer from 0 to 1 million TPS on Azure
 
AWS Summit Kuala Lumpur Keynote with Stephen Orban - Head of Enterprise Strategy
AWS Summit Kuala Lumpur Keynote with Stephen Orban - Head of Enterprise StrategyAWS Summit Kuala Lumpur Keynote with Stephen Orban - Head of Enterprise Strategy
AWS Summit Kuala Lumpur Keynote with Stephen Orban - Head of Enterprise Strategy
 
Proact SYNC 2013 Breakout session - NetApp Clustered DataONTAP, dé storage hy...
Proact SYNC 2013 Breakout session - NetApp Clustered DataONTAP, dé storage hy...Proact SYNC 2013 Breakout session - NetApp Clustered DataONTAP, dé storage hy...
Proact SYNC 2013 Breakout session - NetApp Clustered DataONTAP, dé storage hy...
 
Intro to AWS: EC2 & Compute Services
Intro to AWS: EC2 & Compute ServicesIntro to AWS: EC2 & Compute Services
Intro to AWS: EC2 & Compute Services
 
Data Culture Series - Keynote - 3rd Dec
Data Culture Series - Keynote - 3rd DecData Culture Series - Keynote - 3rd Dec
Data Culture Series - Keynote - 3rd Dec
 
Leveraging Big Data with Hadoop, NoSQL and RDBMS
Leveraging Big Data with Hadoop, NoSQL and RDBMSLeveraging Big Data with Hadoop, NoSQL and RDBMS
Leveraging Big Data with Hadoop, NoSQL and RDBMS
 
Using real time big data analytics for competitive advantage
 Using real time big data analytics for competitive advantage Using real time big data analytics for competitive advantage
Using real time big data analytics for competitive advantage
 
Five Early Challenges Of Building Streaming Fast Data Applications
Five Early Challenges Of Building Streaming Fast Data ApplicationsFive Early Challenges Of Building Streaming Fast Data Applications
Five Early Challenges Of Building Streaming Fast Data Applications
 
Data Virtualization Journey: How to Grow from Single Project and to Enterpris...
Data Virtualization Journey: How to Grow from Single Project and to Enterpris...Data Virtualization Journey: How to Grow from Single Project and to Enterpris...
Data Virtualization Journey: How to Grow from Single Project and to Enterpris...
 
Introducing Amazon Kinesis: Real-time Processing of Streaming Big Data (BDT10...
Introducing Amazon Kinesis: Real-time Processing of Streaming Big Data (BDT10...Introducing Amazon Kinesis: Real-time Processing of Streaming Big Data (BDT10...
Introducing Amazon Kinesis: Real-time Processing of Streaming Big Data (BDT10...
 
Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...
Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...
Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...
 
Vertica Analytics Database general overview
Vertica Analytics Database general overviewVertica Analytics Database general overview
Vertica Analytics Database general overview
 
Migrating from Oracle to Postgres
Migrating from Oracle to PostgresMigrating from Oracle to Postgres
Migrating from Oracle to Postgres
 

More from Domonkos Tikk

Neighbor methods vs matrix factorization - case studies of real-life recommen...
Neighbor methods vs matrix factorization - case studies of real-life recommen...Neighbor methods vs matrix factorization - case studies of real-life recommen...
Neighbor methods vs matrix factorization - case studies of real-life recommen...
Domonkos Tikk
 
General factorization framework for context-aware recommendations
General factorization framework for context-aware recommendationsGeneral factorization framework for context-aware recommendations
General factorization framework for context-aware recommendations
Domonkos Tikk
 
Tartalomgazdagítás (content enrichment)
Tartalomgazdagítás (content enrichment) Tartalomgazdagítás (content enrichment)
Tartalomgazdagítás (content enrichment)
Domonkos Tikk
 
Idomaar crowd rec_reference_fw
Idomaar crowd rec_reference_fwIdomaar crowd rec_reference_fw
Idomaar crowd rec_reference_fw
Domonkos Tikk
 
Big Data in Online Classifieds
Big Data in Online ClassifiedsBig Data in Online Classifieds
Big Data in Online Classifieds
Domonkos Tikk
 
Context-aware similarities within the factorization framework - presented at ...
Context-aware similarities within the factorization framework - presented at ...Context-aware similarities within the factorization framework - presented at ...
Context-aware similarities within the factorization framework - presented at ...
Domonkos Tikk
 
Slides from CARR 2012 WS - Enhancing Matrix Factorization Through Initializat...
Slides from CARR 2012 WS - Enhancing Matrix Factorization Through Initializat...Slides from CARR 2012 WS - Enhancing Matrix Factorization Through Initializat...
Slides from CARR 2012 WS - Enhancing Matrix Factorization Through Initializat...
Domonkos Tikk
 
Fast ALS-Based Tensor Factorization for Context-Aware Recommendation from Imp...
Fast ALS-Based Tensor Factorization for Context-Aware Recommendation from Imp...Fast ALS-Based Tensor Factorization for Context-Aware Recommendation from Imp...
Fast ALS-Based Tensor Factorization for Context-Aware Recommendation from Imp...
Domonkos Tikk
 
Recommender Systems Evaluation: A 3D Benchmark - presented at RUE 2012 worksh...
Recommender Systems Evaluation: A 3D Benchmark - presented at RUE 2012 worksh...Recommender Systems Evaluation: A 3D Benchmark - presented at RUE 2012 worksh...
Recommender Systems Evaluation: A 3D Benchmark - presented at RUE 2012 worksh...
Domonkos Tikk
 

More from Domonkos Tikk (9)

Neighbor methods vs matrix factorization - case studies of real-life recommen...
Neighbor methods vs matrix factorization - case studies of real-life recommen...Neighbor methods vs matrix factorization - case studies of real-life recommen...
Neighbor methods vs matrix factorization - case studies of real-life recommen...
 
General factorization framework for context-aware recommendations
General factorization framework for context-aware recommendationsGeneral factorization framework for context-aware recommendations
General factorization framework for context-aware recommendations
 
Tartalomgazdagítás (content enrichment)
Tartalomgazdagítás (content enrichment) Tartalomgazdagítás (content enrichment)
Tartalomgazdagítás (content enrichment)
 
Idomaar crowd rec_reference_fw
Idomaar crowd rec_reference_fwIdomaar crowd rec_reference_fw
Idomaar crowd rec_reference_fw
 
Big Data in Online Classifieds
Big Data in Online ClassifiedsBig Data in Online Classifieds
Big Data in Online Classifieds
 
Context-aware similarities within the factorization framework - presented at ...
Context-aware similarities within the factorization framework - presented at ...Context-aware similarities within the factorization framework - presented at ...
Context-aware similarities within the factorization framework - presented at ...
 
Slides from CARR 2012 WS - Enhancing Matrix Factorization Through Initializat...
Slides from CARR 2012 WS - Enhancing Matrix Factorization Through Initializat...Slides from CARR 2012 WS - Enhancing Matrix Factorization Through Initializat...
Slides from CARR 2012 WS - Enhancing Matrix Factorization Through Initializat...
 
Fast ALS-Based Tensor Factorization for Context-Aware Recommendation from Imp...
Fast ALS-Based Tensor Factorization for Context-Aware Recommendation from Imp...Fast ALS-Based Tensor Factorization for Context-Aware Recommendation from Imp...
Fast ALS-Based Tensor Factorization for Context-Aware Recommendation from Imp...
 
Recommender Systems Evaluation: A 3D Benchmark - presented at RUE 2012 worksh...
Recommender Systems Evaluation: A 3D Benchmark - presented at RUE 2012 worksh...Recommender Systems Evaluation: A 3D Benchmark - presented at RUE 2012 worksh...
Recommender Systems Evaluation: A 3D Benchmark - presented at RUE 2012 worksh...
 

Recently uploaded

办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
xjq03c34
 
Search Result Showing My Post is Now Buried
Search Result Showing My Post is Now BuriedSearch Result Showing My Post is Now Buried
Search Result Showing My Post is Now Buried
Trish Parr
 
不能毕业如何获得(USYD毕业证)悉尼大学毕业证成绩单一比一原版制作
不能毕业如何获得(USYD毕业证)悉尼大学毕业证成绩单一比一原版制作不能毕业如何获得(USYD毕业证)悉尼大学毕业证成绩单一比一原版制作
不能毕业如何获得(USYD毕业证)悉尼大学毕业证成绩单一比一原版制作
bseovas
 
[HUN][hackersuli] Red Teaming alapok 2024
[HUN][hackersuli] Red Teaming alapok 2024[HUN][hackersuli] Red Teaming alapok 2024
[HUN][hackersuli] Red Teaming alapok 2024
hackersuli
 
留学学历(UoA毕业证)奥克兰大学毕业证成绩单官方原版办理
留学学历(UoA毕业证)奥克兰大学毕业证成绩单官方原版办理留学学历(UoA毕业证)奥克兰大学毕业证成绩单官方原版办理
留学学历(UoA毕业证)奥克兰大学毕业证成绩单官方原版办理
bseovas
 
Design Thinking NETFLIX using all techniques.pptx
Design Thinking NETFLIX using all techniques.pptxDesign Thinking NETFLIX using all techniques.pptx
Design Thinking NETFLIX using all techniques.pptx
saathvikreddy2003
 
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
uehowe
 
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
ysasp1
 
Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?
Paul Walk
 
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
cuobya
 
7 Best Cloud Hosting Services to Try Out in 2024
7 Best Cloud Hosting Services to Try Out in 20247 Best Cloud Hosting Services to Try Out in 2024
7 Best Cloud Hosting Services to Try Out in 2024
Danica Gill
 
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
cuobya
 
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
uehowe
 
Discover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to IndiaDiscover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to India
davidjhones387
 
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
ukwwuq
 
一比一原版(USYD毕业证)悉尼大学毕业证如何办理
一比一原版(USYD毕业证)悉尼大学毕业证如何办理一比一原版(USYD毕业证)悉尼大学毕业证如何办理
一比一原版(USYD毕业证)悉尼大学毕业证如何办理
k4ncd0z
 
Gen Z and the marketplaces - let's translate their needs
Gen Z and the marketplaces - let's translate their needsGen Z and the marketplaces - let's translate their needs
Gen Z and the marketplaces - let's translate their needs
Laura Szabó
 
Ready to Unlock the Power of Blockchain!
Ready to Unlock the Power of Blockchain!Ready to Unlock the Power of Blockchain!
Ready to Unlock the Power of Blockchain!
Toptal Tech
 
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaalmanuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
wolfsoftcompanyco
 
HijackLoader Evolution: Interactive Process Hollowing
HijackLoader Evolution: Interactive Process HollowingHijackLoader Evolution: Interactive Process Hollowing
HijackLoader Evolution: Interactive Process Hollowing
Donato Onofri
 

Recently uploaded (20)

办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
 
Search Result Showing My Post is Now Buried
Search Result Showing My Post is Now BuriedSearch Result Showing My Post is Now Buried
Search Result Showing My Post is Now Buried
 
不能毕业如何获得(USYD毕业证)悉尼大学毕业证成绩单一比一原版制作
不能毕业如何获得(USYD毕业证)悉尼大学毕业证成绩单一比一原版制作不能毕业如何获得(USYD毕业证)悉尼大学毕业证成绩单一比一原版制作
不能毕业如何获得(USYD毕业证)悉尼大学毕业证成绩单一比一原版制作
 
[HUN][hackersuli] Red Teaming alapok 2024
[HUN][hackersuli] Red Teaming alapok 2024[HUN][hackersuli] Red Teaming alapok 2024
[HUN][hackersuli] Red Teaming alapok 2024
 
留学学历(UoA毕业证)奥克兰大学毕业证成绩单官方原版办理
留学学历(UoA毕业证)奥克兰大学毕业证成绩单官方原版办理留学学历(UoA毕业证)奥克兰大学毕业证成绩单官方原版办理
留学学历(UoA毕业证)奥克兰大学毕业证成绩单官方原版办理
 
Design Thinking NETFLIX using all techniques.pptx
Design Thinking NETFLIX using all techniques.pptxDesign Thinking NETFLIX using all techniques.pptx
Design Thinking NETFLIX using all techniques.pptx
 
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
办理毕业证(UPenn毕业证)宾夕法尼亚大学毕业证成绩单快速办理
 
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
成绩单ps(UST毕业证)圣托马斯大学毕业证成绩单快速办理
 
Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?
 
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
假文凭国外(Adelaide毕业证)澳大利亚国立大学毕业证成绩单办理
 
7 Best Cloud Hosting Services to Try Out in 2024
7 Best Cloud Hosting Services to Try Out in 20247 Best Cloud Hosting Services to Try Out in 2024
7 Best Cloud Hosting Services to Try Out in 2024
 
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
可查真实(Monash毕业证)西澳大学毕业证成绩单退学买
 
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
留学挂科(UofM毕业证)明尼苏达大学毕业证成绩单复刻办理
 
Discover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to IndiaDiscover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to India
 
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
制作原版1:1(Monash毕业证)莫纳什大学毕业证成绩单办理假
 
一比一原版(USYD毕业证)悉尼大学毕业证如何办理
一比一原版(USYD毕业证)悉尼大学毕业证如何办理一比一原版(USYD毕业证)悉尼大学毕业证如何办理
一比一原版(USYD毕业证)悉尼大学毕业证如何办理
 
Gen Z and the marketplaces - let's translate their needs
Gen Z and the marketplaces - let's translate their needsGen Z and the marketplaces - let's translate their needs
Gen Z and the marketplaces - let's translate their needs
 
Ready to Unlock the Power of Blockchain!
Ready to Unlock the Power of Blockchain!Ready to Unlock the Power of Blockchain!
Ready to Unlock the Power of Blockchain!
 
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaalmanuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
manuaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaal
 
HijackLoader Evolution: Interactive Process Hollowing
HijackLoader Evolution: Interactive Process HollowingHijackLoader Evolution: Interactive Process Hollowing
HijackLoader Evolution: Interactive Process Hollowing
 

Challenges Encountered by Scaling Up Recommendation Services at Gravity R&D

  • 1. Scaling UP Challenges Encountered Scaling Up Recommendation Services @Gravity R&D Bottyán Németh
  • 2. Who we are and what we do Gravity R&D is a recommender system vendor company. We provide recommendation as a service since 2009 for our customers all around the globe. 2
  • 3. How we imagine growth? 3 ?
  • 4. How we imagine growth? 4
  • 5. How it actually happens? 5 ?
  • 6. How it actually happens? 6
  • 7. # of requests 7 Vatera.hu largest online marketplace in Hungary served by one “server” Alexa TOP100 video chat webpage (~40M recommendation requests / day):  Served by 5 application servers and 1 DB  Too many events to store in MySQL  using Cassandra (v0.6)  Training time for IALS too long  speedup by IALS1  Max. 5 sec latency in “product” availability
  • 8. Using new/beta technologies 8 Cassandra (v0.6) Nginx (v0.5) (22% of top 1M sites) Kafka (v0.8) MySQL auto. failover
  • 9. Reaching the limits 9 Even if the technology is widely used if you reach it’s limits the optimization is very costly / time consuming. Java GC – service collapsed because increased minor GC times due to a JVM bug (26th of January 2013) Maintaining MySQL with lots of data (optimize table, slave replication lag, faster storage device)
  • 10. Complexity increases 10 There is always a business request or an algorithmic development which requires more resources.
  • 12. Infrastructure 12 Currently 200+ hosts and 3500+ services monitored 0 50 100 150 200 250 2008 2009 2010 2011 2012 2013 2014 2015 2016 Number of servers
  • 13. # of items 13 How to store item model / metadata in memory to serve requests fast?
  • 14. # of items 14 How to store item model / metadata in memory to serve requests fast? VS. Auto increment IDs for the items? 231 not enough
  • 15. Preconceptions 15 More data better results. If the CTR of a new algorithm is low than the old algorithm is better. Daily retrain is enough.
  • 17. 100+ Algorithms 17 0 10 20 30 40 50 60 0 20 40 60 80 100 120 Number of times an algorithm is used
  • 18. Now 18 • Performance: Gravity’s performance oriented architecture enables real-time response to the always changing environment and user behavior • Algorithms: more than 100 different recommendation algorithm enables true personalization and to reach the highest KPIs in different domains • Infrastructure: fast response times all around the globe and data security thanks to the private cloud infrastructure located in 4 different data centers • Flexibility: the advanced business rule engine with intuitive user interface allows to satisfy various business requirements Performance 140M requests served daily Algorithms 30 man-years invested Infrastructure 4 data centers globally Flexibility 100s of logics configurable
  • 19. Cross the river when you come to it 19