Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

If you give a mouse a clickhouse, by Alex Hofsteede, Sentry

852 views

Published on

Presented at ClickHouse Meetup in Cloudflare, February 19, 2019
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry

Published in: Technology
  • Be the first to comment

  • Be the first to like this

If you give a mouse a clickhouse, by Alex Hofsteede, Sentry

  1. 1. If you give a mouse a ClickHouse ClickHouse at Sentry by Alex Hofsteede
  2. 2. He’s going to ask for a Search backend Time-series database Log search and aggregation tool Offline BI / analytics tool Transactional RDBMS (?)
  3. 3. Here’s what sentry does
  4. 4. And here’s how that works { “event_id”: “7fa801de”, “type”: “UnhandledRejection”, “value”: “//sentry/io/...”, “tags”: { “browser”: “Chrome”, “release”: “abc123”, “version”: “1.2”, }, “stacktrace”: [ ... ] }
  5. 5. We take the events { “event_id”: “7fa801de”, “type”: “UnhandledRejection”, “value”: “//sentry/io/...”, “tags”: { “browser”: “Chrome”, “release”: “abc123”, “version”: “1.2”, }, “stacktrace”: [ ... ] } ● Validate ● Scrub data (PII) ● Apply filter rules ● Apply sampling ● Apply rate limits ● Un-minify (sourcemaps) ● Symbolicate (debug symbols) ● Similarity analysis / grouping ● Check Alerting thresholds ● Send Alerts
  6. 6. We store ‘em { “event_id”: “7fa801de”, “group_id”: “11111”, “type”: “UnhandledRejection”, “value”: “//sentry/io/...”, “tags”: { “browser”: “Chrome”, “release”: “abc123”, “version”: “1.2”, }, “stacktrace”: [ ... ] } / event_id type 7fa801de UnhandledRejection issue_id type 11111 UnhandledRejection
  7. 7. We count ‘em tag_val count Chrome 20 tag_key count release 120 tag_key count browser 100 { “event_id”: “7fa801de”, “tags”: { “browser”: “Chrome”, “release”: “abc123”, “version”: “1.2”, }, } tag_val count abc123 1 tag_key count version 95 tag_val count 1.2 34 day-2019-01-01 hour-2019-01-01T12:00 min-2019-01-01T12:34 135 10 2
  8. 8. It’s a lot of work {…} /Event Body / Stack Trace Event / Group / Tag Keys / Tag Values Updates Time Series Counters
  9. 9. We needed something else
  10. 10. Project ● Make writes fast and cheap ● Filter, search, and aggregate on any field, including stack traces. ● Group by any field ● Time-series counts ● Advanced filters (IN, NOT, LIKE) ● Open Source ● Scalable ● Fast (interactive) queries
  11. 11. First we put it all in Kafka { “event_id”: “7fa801de”, “group_id”: “11111”, “type”: “UnhandledRejection”, “value”: “//sentry/io/...”, “tags”: { “browser”: “Chrome”, “release”: “abc123”, “version”: “1.2”, }, “stacktrace”: [ ... ] }
  12. 12. We needed a database DIY
  13. 13. We wrote an API for it SELECT ... INSERT ... API Events Queries Comparison results
  14. 14. The results came in
  15. 15. We had to tune it API
  16. 16. How to make CH fast
  17. 17. Sharding / Distribution ABC 3 ZeroDiv prod 123 1 TypeErr test DEF 3 TypeErr prod A1B 2 ValueErr test Event ID Proj Error Env Host 1 Host 2 1006 1001 1002 1005 Time 123 1 TypeErr test1001 ABC 3 ZeroDiv prod1006 A1B 2 ValueErr test1005 DEF 3 TypeErr prod1002 Our Choice: cityHash64(event_id)
  18. 18. Partitioning Host 1 111 2 TypeErr test2033 XYZ 2 ValueErr test2045 234 1 ZeroDiv prod3004 BBB 2 ZeroDiv prod3006 123 1 TypeErr test1001 ABC 3 TypeErr prod1006 Partition 1 Partition 2 Partition 3 Event ID Proj Error EnvTime Our Choice: toMonday(Time)
  19. 19. Sorting Host 1 FFF 2 ValueErr test1002 DEF 2 TypeErr test1004 999 1 ValueErr prod1005 000 1 TypeErr prod1010 123 1 TypeErr test1001 ABC 3 TypeErr prod1003 Partition 1 Event ID Proj Error EnvTime Mark Mark Our Choice: (project, Time)
  20. 20. Columns Host 1 Partition 1 FFF DEF 999 000 123 ABC Event ID 2 2 1 1 1 3 Proj Error test test prod prod test prod Env 1002 1004 1005 1010 1001 1003 Time ValueErr TypeErr ValueErr TypeErr TypeErr TypeErr
  21. 21. Query Settings
  22. 22. Query Settings
  23. 23. Query Settings
  24. 24. Then it was fast enough API Queries
  25. 25. Then everything went quiet
  26. 26. So, about that mouse... Now that we gave him a Clickhouse, he wants ● Event deletions ● GDPR compliance (scrubbing data) ● Mutable Issue workflow data (resolved, assigned, etc.) All in Clickhouse
  27. 27. Then he’ll want

×