Recommended
More Related Content
What's hot
What's hot (20)
Similar to Apache kylin meetup berlin olx v1.0
Similar to Apache kylin meetup berlin olx v1.0 (20)
Recently uploaded
Recently uploaded (20)
Apache kylin meetup berlin olx v1.0
- 1. Apache Kylin Meetup Berlin, October 2019
- 2. 2 19:15 - Leveraging analytics at OLX Group with Kylin Victor Fujihara – Data Engineering Manager Mateusz Jerzyk – Senior Data Engineer Rafael Correa – Senior System Engineer 20:00 - Apache Kylin: Boost your SQLs on an extremely large dataset George Ni – Apache Kylin committer Agenda
- 3. 3 Table of contents ● Introduction to OLX Group ● What problem we were trying to solve ? ● Data Engineering with Kylin ● Handling Kylin infrastructure
- 4. 4 THE OLX GROUP IS PART OF NASPERS A collection of leading companies and exciting businesses! US$107Bn Founded 1915 South Africa Market Cap: $100B
- 5. 5 THE OLX GROUP – NASPERS CLASSIFIEDS We improve lives by bringing people together for win-win exchanges
- 6. 6 OLX GROUP TODAY: THE WORLD'S #1 CLASSIFIEDS BUSINESS HORIZONTALS REAL ESTATE VERTICALS OTHER VERTICALS CAR VERTICALS global global app-only Russia UAE Africa and Philippines Russia Portugal Poland Romania, Egypt Furniture, Europe Heavy machinery, global Services, Poland Poland South Africa Romania Portugal CONVENIENT TRANSACTIONS Global UAE Latin America South Africa* *Pending approval Jobs, India Jobs, Poland
- 7. 7 7 OLX GROUP - WHO WE ARE We are a global product and tech group. ★ +30 countries ★ +35 offices ★ +5000 people ★ +1000 in Product & Tech ★ +350 MAU ★ +4B events per day
- 8. 8 WE LOVE C2C EXCHANGES 1/4 of Russian population uses Avito every month >80% of secondhand car trade in India are through OLX 500,000 items are listed everyday at OLX Brazil letgo: fastest growing app to trade to buy and sell locally in US OLX is the most visited website in Romania Every second are listed on OLX: 2 houses 2 cars 3 fashion items 2.5 mobile phones Sources: Avito (Mediacorp); Romania (Audience and Traffic Measurement (SATI); Brazil, India and OLX (BI). People spend 2x more times in OLX apps versus competitors
- 9. 9 Table of contents ● Introduction to OLX Group ● What problem we were trying to solve ? ● Data Engineering with Kylin ● Handling Kylin infrastructure
- 10. 10 Challenges we faced in Global BI Performance Data democracyDesign • Daily night job • Slow response time for detailed data • It’s difficult for users to get numbers • Needs access to database ( SQL knowledge ) • Can’t access the database. It will compete with our daily job • Hard to add more metrics to dashboards • Very complex architecture • Low flexibility solution • Hard to maintain and to add new metrics
- 13. 13 Technical Requirements Has very good response time Is easy to manage and maintain Scales horizontally Has good integration with Tableau
- 14. 14 Table of contents ● Introduction to OLX Group ● What problem we were trying to solve ? ● Data Engineering with Kylin ● Handling Kylin infrastructure
- 15. 15 Data pipeline CREATE PARTITIONS TRANSFORM THE DATA CREATE EXTERNAL TABLES Native ODBC LISTINGS REPLIES ... UNLOAD ORC Stay tuned! Rafael will talk more about this part
- 16. 16 One of the challenges Dimension Fact Table Dimension Dimension Dimension Dimension ➔ Apache Kylin supports only star or snowflake schemas which means we need to use only a single fact table ➔ We can’t use separate cubes to aggregate measures from multiple facts in a single query
- 17. 17 One of the challenges Dimension Fact Table Dimension Dimension Dimension Dimension Fact Table Dimension Dimension Fact Table
- 18. 18 One of the challenges Fact Table Fact Table Fact Table NULLS NULLS NULLS NULLS NULLSNULLS Dimension Dimension Dimension Dimension Dimension Dimension Dimension One big Fact tablePros: - we have star schema - supposed to be easy to build Cons: - the size of the table is big - difficult to change just part of the data - cube creation process takes a lot of time
- 19. 19 One of the challenges Fact Table Fact Table Fact Table Dimension Dimension Bridge Table Dimension Dimension Dimension Dimension Dimension Simplified Bus Architecture Factless fact table Common dimensions
- 20. 20 One of the challenges common dimensions facts fact specific dimensions facts
- 23. 23 Statistics
- 24. 24 Statistics
- 25. 25 Kylin vs SSAS vs Redshift Apache Kylin Amazon RedshiftMicrosoft SSAS Cluster size: 2 x r4.2xlarge; 3 x r4.xlarge; 10 x r4.2xlarge ~ 2TB vvvvvvvvvvvvvvvvvvv Data size: source table size 12GB (ungzipped); cube size 1 GB Processing time: ~ 17 min vvvvvvvvvvvv Price: ~ 450 € / month (including whole infrastructure costs) vvvvvvvvvvvv Response time in Tableau: 3.7 sec Cluster size: s1 instance memory 24 GB Data size: 2.6 GB. vvvvvvv vvvvvvvvvvv Processing time: ~ 15 min vvvvvvvvvvv Price: ~ 1232 € / month + additional infrastructure costs vvvvvvvvvvvvvvvv Response time in Tableau: 6.8 sec Cluster size: 13 nodes X dc2.8xlarge (memory 244 GB, storage 2.56TB SSD) Data size: 51 GB vvvvvvvvvvvvvvvvvv Processing time: ~ 2 min* vvvvvvvvvvvvv Price: ~ 2000 € / month** vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv vv Response time in Tableau: 14.1 sec Input data: 100,000,000 rows, 1 big table Model: 7 distinct counts, 12 sums, 8 small dimensions
- 26. 26 Table of contents ● Introduction to OLX Group ● What problem we were trying to solve ? ● Data Engineering with Kylin ● Handling Kylin infrastructure
- 27. 27 Handling kylin infrastructure - requirements (extended) Needs to: ● Be easily accessible by analysts and engineers. ○ Implies: user management integrated with existing company-wide solution. ● Scale for multiple teams and projects, storage wise and performance wise. ○ Implies: user authorisation per project ○ Also implies: diminish "noisy neighbours" effect as much as possible, for both serving cube queries and building cubes. ● Be highly available, to serve every tableau dashboard we have in the company. ○ Implies: components need to be resilient (self healing, fault tolerant) ● Be maintainable by the SRE team. ○ Implies: observability through existing monitoring, alerting and logging tools ○ Also implies: leverage existing operational knowledge and simplify it as much as possible. ● Be cost effective.
- 28. 28 AAAAA
- 29. 29
- 30. 30 Key technical decisions ● Hbase on HDFS in EMR (build hbase on EC2 would be great, but would take too long) ○ Limitation: Can't read HDFS in S3 (EMRFS) from outside EMR (found out the hard way). ○ Single-AZ failure covered by backup/restore tool (internally developed). ○ Single-AZ deployment to minimize cross-AZ data traffic costs and increase speed. ● Spark on k8s for kylin and hive (running in client mode) ○ No spark master management required. ○ We have our own custom spark build (prevents jar conflicts, also required by hive on spark). ○ More cost-effective and scalable than running in EMR. ○ Leveraged existing k8s knowledge to operate and scale. ○ Limitation: no support for tolerations/pod anti-affinity (yet ).
- 31. 31 Key technical decisions ● SAML federation w/ OKTA for user management ○ openldap + custom-developer okta sync script which allows creating system users as well. ○ Limitation: We noticed some issues with non-admin users in the UI as of version 2.6.3 (couldn't select projects from the select box on top). ● Kylin in cluster mode ○ Memcached cluster for tomcat session sharing (enables rolling deployments with zero-downtime). ○ using zookeeper from hbase for job locking. ○ job running won't impact kylin query serving performance due to spark on k8s.
- 32. 32 Spark on k8s
- 33. 33 Spark on k8s When bootstrapping (client mode):
- 34. 34 HBase backup/restore In-house developed tool for taking a live backup from an Hbase cluster. Currently kylin-specific. Components: ● backup agent ○ written in python ○ runs in k8s as a single-replica deployment (to auto- restart in case of failures) ○ performs daily backups from HDFS to S3 using distcp ● restore-agent ○ restores the latest finished backup from an S3 bucket path. ○ written in jruby (easier to interact w/ hbase native libraries for restoring a distcp snapshot in a reliable way). ○ runs in hbase, current as an EMR step
- 35. 35 References Kylin in cluster mode http://kylin.apache.org/docs/install/kylin_cluster.html SAML federation for user management http://kylin.apache.org/docs/howto/howto_ldap_and_sso.html Memcached driver for tomcat https://github.com/magro/memcached-session-manager Spark on K8s https://spark.apache.org/docs/latest/running-on-kubernetes.html https://github.com/GoogleCloudPlatform/spark-on-k8s-operator Hive on Spark https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
- 36. 36 Thank you ! Questions ? Victor Fujihara Data Engineering Manager victor.Fujihara@olx.com Mateusz Jerzyk Senior Data Engineer mateusz.jerzyk@olx.com Rafael Correa Senior System Engineer rafael.correa@olx.com OLX Group www.olx.com We are hiring! www.joinolx.com Roles: • Data engineers • Data scientists • Java / Android / iOS developers Locations: Berlin, Lisbon, Buenos Aires, Dubai, Barcelona, Moscow, Delhi
- 37. 37