Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Druid for real time monitoring & analytics at eBay

949 views

Published on

Slides posted with permission from eBay's Monitoring Platform team. The original talk was given at the Druid meetup on 2018-03-13, and the abstract was:

"At eBay, we are using Druid for application monitoring and analytics in near real time. We process data coming from thousands of apps which generates more than 100’s of Billion events per day. Druid helped us to monitor critical events and health of our apps in near real time and scale for our needs. We will be presenting how we built and scale Druid at eBay for our use-case in detail. We will be sharing challenges, learnings, and our journey building on this platform at eBay."

Published in: Data & Analytics
  • Be the first to comment

Druid for real time monitoring & analytics at eBay

  1. 1. Motivation ➢ Existing system is legacy product. ➢ Scaling challenges with architecture evolution. ➢ Cardinality explosion with Microservices. ➢ Missing features like percentiles, topN.
  2. 2. Data received from Apps Messaging Layer Druid Deployment Apps Event Publisher Upstream Processor Kafka Architecture
  3. 3. ➢ Built on Kubernetes ➢ Indexed at 1 min, 15 min, 1 hour. Reindex for 1 day ➢ Median Query response times < 1 second ➢ Smart granularity selection for different range queries ➢ HDFS for deep storage Druid Deep Dive
  4. 4. Resiliency Architecture ➢ Dual-cluster setup ➢ Smart query routing ➢ Druid monitoring through events ➢ Resilient infra ➢ Two replicas of data per DC
  5. 5. Scale 5000+ Apps 4M+ Events/Sec 20M+ Unique Dimension Values 250+ Nodes in Druid Cluster 100s B Events Processed/Day
  6. 6. ➢ Used by SREs to monitor application health ➢ Used by developer community to monitor their apps ➢ Anomaly detection ➢ Alerting Customers & Usage
  7. 7. ➢ Percentiles ➢ Self-serve Alerting Next steps

×