Loki is a prototype OpenTracing implementation written in Go thats takes the Prometheus service-discovery and pull based approach to distributed tracing.

1. Loki: a Zipkin/Prometheus Mashup
@tom_wilkie, CNCFCon Berlin April 2017
+ =

3. Why did I write my
own tracer?

4. Debugging a latency
performance issue
with Cortex…
Distributor
Ingester Ingester…

5. Well, thats my
rationalisation…

6. In reality, this is attempt #2
• Prototype “Weave Tracer” circa
2015
• Concept didn’t require application
instrumentation
• Used ptrace to intercept syscalls
and infer application behaviour
• Kinda worked, for a very limited
definition of “worked”

7. Prometheus = Greek god.
Loki = Norse equivalent?

8. So what makes Loki
different?

9. Prometheus is to Graphite
as
Loki is to Zipkin

10. Push vs Pull
https://prometheus.io/docs/introduction/faq/#why-do-you-pull-rather-than-push?
https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or-does-it/

11. • Jobs must know where monitoring is
• Can overwhelm graphite with too many
samples
Graphite Prometheus
scraping
your jobs
Prometheus
pushing
your jobs
Graphite
• Tell Prometheus where jobs are (via service
discovery)
• Prometheus can back off when overwhelmed
• Prometheus knows the identity of each job

12. Zipkin Loki
scraping
your jobs
Loki
pushing
your jobs
Zipkin

13. http://job/traces
your job
Loki client library
spans
scraping
Loki
• Client library keep pending
spans in an in-memory ring
buffer.
• /traces HTTP handler
grabs all the in-memory
spans and serialises them
using Thrift.
• Spans will be dropped if not
collected frequently enough.

14. • Retrieval library ‘knows’ identity of scraped
endpoints, adds that to received spans
• … jobs don’t need to know their own
identity
• … can be consistent with identity used in
Prometheus
• Naive in memory storage implementation
• … makes queries slow, as its just a loop.
• Zipkin-compatible API endpoints
• UI _is_ the Zipkin UI
Loki
Prometheus retrieval library
In memory
storage
Zipkin API
Zipkin UI

15. Its all open source:
https://github.com/weaveworks-experiments/loki
…and it’s written in go

16. This all sounds great!
Where’s the catch?

17. ❌ Client library doesn’t actually support multiple
scrapers (yet)
❌ Loki query performance sucks (for now)
❌ Loki single-process architecture limits scalability
❌ Can dropped spans, gets worse through jitter

18. … that Cortex
performance issue

19. Debugging a latency
performance issue
with Cortex…
Distributor
Ingester Ingester…

21. It was garbage collection…

22. 100ms ➡ 25ms

23. Demo

24. Client Library
• Make is support multiple scrapers
• Move away from thrift to protos
• More languages
• Useful HTML /traces
Loki Server
• Local storage with BoltDB
• Make queries faster
• Make it distributed, use cloud
storage
TODO

25. Why did I write my
own tracer?

26. Because with OpenTracing, I can.

27. Thank you!
Questions?

29. We’re hiring!
London BerlinSan Francisco
jobs@weave.works