Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (14)
Similar to Engineers guide to data analysis
Similar to Engineers guide to data analysis (20)
Recently uploaded
Recently uploaded (20)
Engineers guide to data analysis
- 1. Engineer’s guide to Data Analysis Avishai Ish-Shalom github.com/avishai-ish-shalom@nukembergnukemberg@wix.com
- 2. Wix in numbers ~ 400 Engineers ~ 1400 employees ~ 100M Sites ~ 250 micro services
- 3. IaaS (Insult as a Service) ▪ Thin API, written in Flask (python) ▪ CouchDB ▪ Apache proxy ▪ StatsD, Graphite, ELK
- 5. Graphite ▪ Metrics collector, storage and UI ▪ Math functions ▪ Common ▪ De-facto standard
- 6. Oops, I think something is broken
- 7. What is this “metric” you speak of?
- 8. A metric is ▪ Numeric data ▪ Often with timestamp (time series) ▪ A “measurement” of something ▪ Discrete
- 9. Where do metrics come from? ▪ Events with numeric data ▪ Counting/aggregating ▪ Sampling
- 10. Sampling
- 11. Sampling
- 12. Events ▪ Data about something that happened ▪ timestamp (time series data) ▪ Has properties - numeric and non- numeric { “timestamp”: “2016-11- 15T18:43:39+00:00”, “host”: “test01.example.net”, “status”: “ok”, “latency”: 14.31 }
- 13. 10000 events/sec x 0.5kb/event = How much data? 400GB a day
- 14. Telemetry is a big data problem
- 15. Aggregates are lossy compression We must decide in advance how we’ll use the metric
- 16. Aggregates ▪ Max, Min, Sum, Average, etc ▪ Last, random point ▪ Percentiles (quantiles) ▪ Historgrams, reverse quantiles ▪ Each is suitable for a particular use case
- 17. Averages are mean to me
- 18. Percentiles p99 - The sampled value that is larger than other 99% of samples ▪ O(n) memory complexity ▪ O(n*log n) computation complexity ▪ Some shortcuts for p50 (median), p100 (max), p0 (min) Use when clients experience individual values
- 19. Percentiles ▪ Precentiles are not additive ▪ You cannot average percentiles Example: s1 (100 points) = [0, 0, ....., 100, 100] => p99 = 100 s2 (100 points) = [0, 0, …., 50, 50] => p99 = 50 p99(s1 : s2) = 50, avg(p99(s1), p99(s2)) = 75
- 20. Histograms Distribution visualization of sample ▪ Count of events in each bin ▪ Beans are usually evenly spaced ▪ Use logarithmically spaced bins for long tails ▪ Additive
- 21. Histograms :-( So why aren’t we all using this? ▪ Storage ▪ Have to decide on bins schema ▪ Not many tools support this
- 22. Choosing the right aggregate ▪ Percentiles/histograms for latency ▪ Max/min for latency and sizes ▪ Histogram analysis for sizes and latency ▪ Sums/averages for capacity and money ▪ Aggregate per domain ▪ Look for deviations
- 23. Resolution ▪ Humans need ~5 data points to see a trend ▪ Hides faster changes ▪ Rollups/downscaling is hard ▪ Multi tier FTW!
- 24. It ain't what you don’t know that gets you into trouble. It's what you know for sure that just ain’t so. “ “
- 25. Peak Erasure/Spike erosion ■ When lowering resolution, data points are aggregated ■ Default aggregation is average ■ Peaks are erased ■ This can happen in storage or visualization
- 26. Peak Erasure/Spike erosion ■ Storages down-sample to save space ■ Aggregation function may be configurable ■ Metric collectors aggregate too ○ carbon-cache uses last value ○ StatsD - gauges, timers, counters
- 27. Counters vs Gauges Behaviour in low res time window ■ Low res sampling erases fast changes ■ “Round numbers” syndrom ■ Counters smear changes, but don’t erase them TLDR: use counters when possible
- 28. Mixed modes Aggregating multiple modes reduces usability of aggregates ■ Different transaction types differ in latencies/sizes ■ Errors, successes have very different latencies/sizes ■ Makes your graphs weird TLDR: use separate metrics for different things
- 30. Visualization ■ Timeframe ■ No more than 3 series ■ Be weary of multiple Y scales, but scale if needed ■ Only related series on the same graph ■ Never mix X scales ■ Visual references: bounds, Y min/max values, legend
- 31. Metric design ■ Choose your aggregates wisely ■ Decide on a proper resolution, sampling rate, aggregation time windows ■ Explore the distribution ■ Separate known modes to independent metrics
- 32. Separate signal from noise ■ Use low-pass filters to smooth ■ Trend changes ■ Timeshifts ■ Filter out outliers
- 33. Working with clusters ■ Most-deviant/outliers ■ Max/Min ■ Sum (capacity) ■ Pre-aggregate percentiles