Discover how Shopback is building its own Product Catalog and how Elastic plays an important role on it.

1. ELASTIC @ SHOPBACK
Alberto Resco Perez
Engineering Manager
Elastic Meetup Singapore – Dec 15, 2016

2. Agenda
1. Introduction to ShopBack
2. Product Catalog in ShopBack
3. Logs/Metrics Collection
4. Future development
5. QA

3. Our Founding Journey

4. Who we are
• We are the smarter way to shop
• Over 1.5M shoppers in 6 countries have adapted the
shop-and-save lifestyle
• Southeast Asia’s No. 1 Online Cashback portal
• Bringing happiness to users via:
• Cashback Rewards
• New Store Discoveries
• Enhanced Shopping Experiences

5. What we do

6. Where we are
SG: SEP 2014
MY: FEB 2015
PH: JUN 2015
IN: JAN 2016
ID: MAR 2016
TW: SEP 2016

7. Product Catalog
Challenges

8. Thousands of categories & stores, Millions of Products
Fast, Reliable & Concise information
Multi country – Multi Language
Presentation in small screens
Product Catalog
Challenges

9. 9
Gateway
API Product Service
MySQLMongoDB
Web IOS Android Extension
Product Catalog
Moving towards micro-services

10. • Moved to our Elasticsearch 2.4 cluster -> Already in 5.0!
• Latest version with new features available
• Schema for multi-country, multi-language
• Indices
• “Product” micro-service
• Performance is great
Product Catalog
Moving towards micro-services

11. Crawler
Product Catalog
Product Updates
Crawler
Synchronizer
MapperImage Uploader
Indexer

12. No Cluster Metrics
Slow browsing
Limited by language
Fast browsing based on ES
Large amount of SKUs
Language is not a issue
Cluster Metrics collected
Product Catalog
Post-elastic implementation

13. Initial
Setup
Cluster
v1
Cluster
v2
• Spin up cluster
• Basic setup
• Single index, Single type
by country:
• Products x 5
• Stores x 5
• Better configuration of
analyzers (different
languages)
• Removed Boolean
queries
• Single index, 2 types:
• Products
• Stores
• Types complete
different, need different
analyzers…
• Multiple Boolean queries
• Country, domain
Product Catalog
Our journey to current status
Cluster
v3
• Single index, Single type
by country:
• Products x 6
• Stores x 6
• Categories x 6
• Campaigns x 6
• Auto-suggestion

14. Before Now
Documents Indexed
• 10M products
• 2k stores
• 15k categories
• 100+ custom pages
• Updated when we find a change
Performance
• Product search: 1ms
• Store listing: 4s
• Auto-suggestion: Live. 100ms
Product Catalog
Some data
Documents Indexed
• 60k products
• Updated every week
Performance
• Product search: 2s
• Store listing: 30s
• Auto-suggestion: 30Kb preload

15. • Papertrail
‒ Cloud provided  $$$
‒ 7 days Retention
‒ No dashboards
• AWS Cloud Watch
‒ 2 days Retention
‒ 5 Minutes interval or 1 Minute interval  $$$
‒ Not all metrics
• New Relic
‒ Cloud provided -> $$$$$$$$
‒ 3 months Retention
‒ 5 Minutes interval or 1 Minute interval -> $$$
‒ Not all metrics
Logs / Metrics Collection
Former/Legacy System

16. Logs / Metrics Collection
Former/Legacy System

17. Logs / Metrics Collection
ELK based

18. • Implement country multi-language -> Using different indexes
• Implement fuzzy matching
• Recommendation engine
• Machine Learning  Fraud detection
• Collect all services metrics with Metricbeat  Mongodb, MySQL, Memcached
• Move Logstash to Filebeat  lightweight shipper
• Setup Alerting -> X-Pack Alerting
Future Implementations

19. • Previously:
‒ Papertrail
‒ New Relic
‒ AWS CloudWatch Metrics
• Moving towards Beats + Kibana
‒ All covered except APM (NewRelic)
‒ First Micro-Service (Product Catalog) migrated
Log management and metrics collection