Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Traveloka's journey to no ops streaming analytics

23 views

Published on

Traveloka's journey to no ops streaming analytics

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Traveloka's journey to no ops streaming analytics

  1. 1. Traveloka’s Journey to No Ops Streaming Analytics Rendy Bambang Jr., Data Eng Lead - Traveloka Gaurav Anand, Solutions Engineer - Google
  2. 2. ● Business Intelligence ● Analytics ● Personalization ● Fraud Detection ● Ads optimization ● Cross selling ● AB Test ● etc. How we use the data
  3. 3. 6 offices Incl. Singapore 1,000+ Global employees 400+ Engineers Our technology core has enabled us to scale Traveloka into 6 countries across ASEAN rapidly in less than 2 years.
  4. 4. #EnablingMobility
  5. 5. In the beginning...
  6. 6. Consumer of Data Initial Data Architecture Streaming Batch Traveloka App Kafka ETL In Memory Real Time DW Data Warehouse S3 Data Lake Batch Ingest Android, iOS NoSQL Realtime DB Traveloka Services Hive, Presto Query DOMO Analytics UI
  7. 7. Key Numbers ● Volume kafka: billions of messages/day ● In-Memory DB: hundreds of GB in-memory data ● NoSQL DB: 50+ nodes, 20+ TB storage, 50+ use cases ● S3: hundreds of TB ● Spark: 20+ nodes, 200+ core ● Redshift DW: 20+ Nodes, tens of TB ● Team: 8 Developers + 3 SysOps/DevOps
  8. 8. Consumer of Data Problems with Initial Data Architecture Streaming Batch Traveloka App Kafka ETL In Memory Real Time DW Data Warehouse S3 Data Lake Batch Ingest Android, iOS NoSQL Realtime DB Traveloka Services Hive, Presto Query DOMO Analytics UI
  9. 9. Problems with Initial Data Architecture Debugging Kafka Issues - Dedicated On-call Data Warehouse throughput issues for high frequency load, coupling storage & compute Team well being, paged on holiday, even honeymoon for infra issue! Scaling Issues with NoSQL DB and In-Memory DB Scaling Issues with Custom-built Java Consumers
  10. 10. How do we..?
  11. 11. Ideal Solution Fully-managed infrastructure to free engineers to solve business problems Autoscaling of Storage and Compute Low end-to-end latency with guaranteed SLA Resilience, end-to-end system availability
  12. 12. Solution Components ● Google Cloud PubSub (Events Data Ingestion) ● Google Cloud Dataflow (Stream Processing) ● Google Bigquery (Analytics) ● Cross-Cloud Environment (AWS-GCP) ● AWS DynamoDB (Operational datastore) Note: Although Cloud Datastore was our prefered operational DB, but its non availability in SG region necessitated use of Dynamodb.
  13. 13. How did we..?
  14. 14. Analytics Architecture: Reimagined Consumer of Data Streaming Batch Traveloka App Kafka ETL Data Warehouse S3 Data Lake Batch Ingest Android, iOS DOMO Analytics UI NoSQL DB Traveloka Services Ingest Cloud Pub/Sub Storage Cloud Storage Pipelines Cloud Dataflow Analytics BigQuery Monitoring Logging Hive, Presto Query
  15. 15. Developed two Common Dataflow Engine ● Self-Service Streaming analytics to BigQuery
  16. 16. Developed two Common Dataflow Engine ● Stream processing to DynamoDB, common features for dev: ○ Combine by key ○ Optimistic Concurrency ○ Local-file based integration test
  17. 17. Key Facts/Numbers ● End to End Pipeline Latency: seconds ● Volume: hundreds of GB/day ● Team: 2 Developers, 0 Ops ● Agility: POC + Pilot in 1 month ● Migrate 50+ different stream processing use case in 1 month ● Bigquery Integration with BI tools: thousands of dashboard, hundreds of users
  18. 18. Awesome Autoscale Pubsub & Dataflow could absorb spiky load just fine! Our case: promo PubSub Publish Count DataFlow vcpus Count
  19. 19. Why Cloud Dataflow (Beam): Tuning Pipeline vs. Managing Servers
  20. 20. The Lambda Architecture
  21. 21. Unified Model with Apache Beam Batch or Streaming
  22. 22. http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
  23. 23. Unified Model with Apache Beam
  24. 24. Optimize Schedule
  25. 25. Why Google Cloud..?
  26. 26. Traveloka Data Team Philosophy ● Managed Service ● NoOps ● Self-Service Focus more on solving complex business problems rather than focusing on infrastructure
  27. 27. What required us to change? ● Ever increasing scale ● Ever increasing operations burden ● New business needs: Streaming Analytics
  28. 28. What’s Next..?
  29. 29. Next Generation Architecture Cloud Pub/Sub Cloud Dataflow BigQuery Cloud Storage Kubernetes Cluster Collector Managed services Simplify! BI & Analytics UI
  30. 30. Conclusion Our engineering team of 2 produces and maintains like a team of 8 because of products like PubSub, Dataflow & Bigquery “ ”
  31. 31. Q&A
  32. 32. Thank You.

×