This document summarizes Martin Törnwall's experience developing Metyr, a metrics storage and processing system built on Erlang, Riak, and HTTP. Some key points: 1) Metyr was designed to store distributed metrics in a way that leverages familiar tools like Erlang and Riak without relying on unfamiliar technologies. 2) Initial attempts involved storing each metric sample as an individual object in Riak, but this proved slow with large overhead. 3) A better approach grouped samples into chunks by time, storing chunks as list values with keys denoting the time period. This allowed for faster range queries but writing became slower. 4) To address write performance, M

1. Metrics with Riak
A retrospective
Martin
Törnwall

2. Metrics?
Many definitions, but here's ours...

3. Recording things
that change over time
So we can visualize it and search for
patterns

4. OS
CPU, network, memory and disk usage, ...

5. Application
Number of requests, errors, events, ...

6. External events
Text messages or emails sent, customer
service calls, ...

7. What is a Metric?
● A named variable: "sys.mem.free"
● With tags: "host=sl075", "code=403", ...
avg("sys.mem.free") from 1 hour ago
where host="sl075"

8. Going Technical

9. We have distributed
services
Why not have distributed metrics?

10. Reinventing the wheel?
Solutions exist, but rely on technology stacks
we had no experience of (e.g., HBASE)

11. I mean, really...
Just how hard can it be?

12. I mean, really...
Just how hard can it be?

13. Introducing Metyr
Our weekend hack glorious metrics
storage and processing software

14. Design Decisions
● Use familiar tools: Erlang, Riak, HTTP
● Not a critical service but ...
● ... Avoid SPOF
● Write performance >> read performance
● Centralized reference clock
● Integer only
● Avoid 2i if possible
● When in doubt, leave it to Riak

15. In Theory...
Client Client Client
Metyr Metyr Metyr
Riak cluster

16. Storing metrics in Riak
No SQL, no schemas, no indices (?), no
aggregate operations

17. Attempt 1
The naïve way just never works...

18. Make each sample an
object
A bucket per metric; index by Epoch time

19. The Good™
Atomicity, write-once, fast range queries

20. The Bad
Slow, large overhead, requires 2i

21. Attempt 2
Combine samples into chunks by time

22. Key Points
● One bucket per metric as before
● Split into hour-sized chunks
(configurable)
● Chunk key: Epoch time
● Chunk value: List of samples
● To read: Fetch chunks within interval
● To write: Fetch chunk, add sample, write
back

23. Chunk Anatomy
One sample
Time0 Value0 Tags0... ... TimeN ValueN TagsN...
64 bits 64 bits

24. Writing just got harder
Slower since we must fetch a chunk first;
potential race conditions, ...

25. (Arbitrary) Goal:
Write 1K samples/sec
Tests showed that the solution described
so far was inadequate

26. Buffer them writes
Keep per-metric write buffers, flushed
every 10 seconds or so

27. Some Remaining Issues
● Race condition on write
● Storage requirements
● Downsampling of old data

28. Thank you!