Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DIY : Data Is Yours

33 views

Published on

Real Time data aggregation in simple steps

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

DIY : Data Is Yours

  1. 1. DIY Data is Yours
  2. 2. • What to expect ?
  3. 3. Daggers : Data Aggregator • Real time data aggregation by selecting few drop downs & writing a SQL like query • A DIY platform for Product Managers, analysts or Engineers to analyse data
  4. 4. Impact • Enabled GO-JEK to be data driven in real time • 100+ Daggers created by analysts over period of 3 months • Dashboards for tracking different metrics used by City Managers & Product Managers • 500+ GB of data gets processed in real time everyday
  5. 5. Sample Query Query to get all booking grouped by various parameters
  6. 6. Creating dagger • Kafka is used as the data source. Uses Protocol Buffers (Protobuf) as schema • Kafka technicalities are abstracted from user with the help of Protobuf to Kafka topic mapping. • Single form for dagger creation, allows analysts and Product Managers to be the power users
  7. 7. Data Sink - Timeseries • Influxdb (timeseries database) • Preview mode • Default data sink • Integrated with grafana - used for monitoring & alerting
  8. 8. Monitoring & Alerting Using grafana, user configure there dashboards for monitoring & alerting.
  9. 9. Data Sink - Apache Kafka • Output can also be published to one of the Kafka topics • Firehose, in-house DIY tool. Allows consuming data from Kafka. • Firehose sink could be one of the following : • Services - HTTP OR GRPC • Db - Relational OR Timeseries • Clever-tap • Log - for debugging
  10. 10. Case Studies • Allocation metrics • More than 25 metrics are being captured • Dynamic Surge Pricing • Helps us improve riders experience & drivers income • System uptime • Kong log is analysed at real time, more than 5B messages day • Growth Campaigns • Using customer signal & intent create user segmentation.
  11. 11. Summary • Daggers - DIY tool for real time data aggregation • Powered developers & non-developers to analyse data • One form to create a dagger • Influx & Kafka as the data sink • Integration with grafana and Firehose allowed users to consume data for Monitoring OR Application
  12. 12. THANK YOU

×