Debug production server by counter
•
2 likes•336 views
當線上運作環境發生問題時，如何在最短時間找出問題核心？我們使用 counter 這個工具來解決。Counter 是在程式裡的一行程式碼，用來記錄感興趣的事件。本演講包含以下內容：Golang counter 程式碼範例，後台系統架構，線上 dashboard，以及如何使用 counter 來偵錯，甚至可以一路追至某一行程式碼。另外，本演講會提及幾個有趣的應用：counter 如何協助定期伺服器更新，如何使用 counter 協助 autoscaling，以及未來的應用。 When production alert triggers, how to identify the root cause within the shortest amount of time? We solve the problem by counter, a line of code inserted by developer to count interesting events. In this talk, we'll cover the following topics: 1) how counter looks like in Golang production code, 2) our counter pipeline, 3) service dashboard with counters, 4) how to use counter to find production issues all the way to certain line of code. We'll also cover a few interesting counter use-cases, including: 1) How counter helps our weekly server upgrade, 2) Use counter for autoscaling, and 3) case-studies to demonstrate what counter can do when outages happen.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Debug production server by counter
Similar to Debug production server by counter (20)
Recently uploaded
Recently uploaded (20)
Debug production server by counter
- 1. Debug production server by counter 羅仲成 Roy Lou 17 Media 2016 July
- 2. About me - 17 Media Architect - Past - HTC: cloud backend - Google: Google Fiber, embedded system - NVIDIA: vlsi hardware - roylou@gmail.com
- 3. About HTC CSI Project - Cloud service infrastructure for mobile apps (similar to Parse.com) - Backed 5+ apps and 3M+ users - 50 < # of VMs < 200 (Autoscaled) - ~15 microservices - Team of 15 engineers One Gallery Umadeit (Fun Fit)
- 6. Problems to Solve Need utility to monitor, alert, debug production cluster issues: - Infrastructure outage - Application outage
- 7. What choices do I have Infrastructure monitoring Application monitoring (for weak typing languages)
- 8. Counter
- 9. func (s *Store) Get(key string) ([]byte, error) { defer ctr.Time("get.proc_time", time.Now()) if val, err := s.Cache.Get(key); err == nil { ctr.Event("get.cache_hit", 1) return val, nil } val, err := s.DB.Get(key) if err != nil { ctr.Event("get.db.err", 1) return nil, err } return msgs, nil } Counter Example - Read Cache Client Cache DB
- 10. func (t *RoundTripper) RoundTrip(req *http.Request) (*http.Response, error) { ctr.Event("qps", 1) ctr.Event("send.bytes", uint64(req.ContentLength)) defer ctr.Time("latency", time.Now()) res, err := t.rt.RoundTrip(req) if err == nil { ctr.Event(fmt.Sprintf("status.%d", res.StatusCode), 1) } else { ctr.Err("internal.err", 1) } return res, err } Counter Roundtripper Client Server Roundtrip Counter Example - Http Roundtrip
- 13. App Container Fluentd Agent VM How frequent should I send counter?
- 14. How Frequent Should I Send Counter? Option 1: Forward every counter to Elasticsearch Option 2: Aggregate locally before forwarding 1000 counters / container * 100 counts / second = 100k qps For us, aggregate and send every 30 sec
- 15. App Container Fluentd Agent VM How long can I store counters?
- 16. How Long Can I Store Counters? - 50,000 counters - 1 record every 30 seconds To save counter for 1 year: 50,000 * 4 (bytes) * 2 (counters/minute) * 525,600 (mins/year) = 210,240,000,000 Bytes = 210.24 TB Need to aggregate for long term storage
- 17. App Container Fluentd Agent VM Counter Aggregator Counter Granularity: - Past 10 days: 30 sec - Past 1 month: 5 min - Past 3 month: 30 min - Past year: 1 hr
- 19. Topology View
- 21. Deploy with Counters Docker Registry docker push code Review CI git push deploy - Mon night: Code freeze - Tue morning: Deploy to staging - If okay, deploy to production 30% => 50% => 100%
- 22. Rolling to X% - Health check - Manually inspect counters - Minimal e2e test - Compare counter with last deploy
- 23. Monitor / Alert with Counters
- 26. eQstr = 'host:"prod-cg-docvcs-group" AND pkg:docvcs_worker AND name:overall.err' rQstr = 'host:"prod-cg-docvcs-group" AND pkg:docvcs_worker AND name:overall.request' errors = esq_scalar('sum', 'total', eQstr, 'now-5m', 'now') requests = esq_scalar('sum', 'total', rQstr, 'now-5m', 'now') error_rate = errors * 100 / requests -- Fail rate should be less than 10/s alert_p2('docvcs fail_rate', error_rate, '>', 10, '15m') alert_p0('docvcs fail_rate', error_rate, '>', 10, '45m') Alarm when high error rate
- 30. Debug with Counters - GDB - Bisect with log - Bisect with counters
- 31. counter
- 35. qstr = 'name: docvcs.jobs.min.outstanding' outstanding = esq_scalar(qstr, 'now-10m', 'now') workload = outstanding / 200 autoscale(workload, 'docvcs', 6, 30, 6, 'diff', 0.65, 0.2, 2/3)
- 36. autoscale(workload, 'docvcs', 6, 30, 6, 'diff', 0.65, 0.2, 2/3) minimum # of instances maximum # of instances maximum # of VMs to be scaled target workload safeguard workload ▵Instance 0.65 0.85 0.45 6 safeguard
- 38. Business Logic with Counters
- 39. Business Logic with Counters
- 40. What else can counter do?
- 41. What can’t counter do? Counter solves problem on 90% users. Counter can’t solve problem on 1 user.
- 42. If so, need logs
- 43. Summary of Counter A line of code. Can be used for: - Rolling update - Monitor / alert - Debug cluster - Autoscale cluster - Simple business logics - And many others (use your imagination)