In Data Engineer’s Lunch #23, Rahul Singh will be covering the topics of Thanos and Cortex. Accompanying Blog: https://blog.anant.us/data-engineers-lunch-23-thanos-and-cortex/ Accompanying YouTube: https://youtu.be/uCD7cetQ8z4 Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday: https://www.meetup.com/Data-Wranglers-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/

1. Version 1.0
Prometheus at Scale : Thanos / Cortex / etc.
An Anant Corporation Story.
How Prometheus scales in global business platforms

2. Prometheus (recap)
● Multidimensional data model over time via metric
name, and key/value pairs
● PromQL, now standard query language
● Time series collection via pull or push (via gateway)
● Dynamic service discovery or via static configuration
● Separation of concerns in graphing / dashboarding

3. Prometheus (on Kubernetes)

4. Prometheus at Scale
● Cortex
● Thanos
● M3DB (from Uber)
● Victoria Metrics
● Vulcan (from Digital Ocean)
https://sysdig.com/blog/challenges-scale-prometheus/

5. Prometheus at Scale Needs
● Global View - Queries over multiple promethei
● Multi-Replica / High Availability - No downtime, no data loss
● Long Term Storage - Store data in cold storage for future
● Global Scale - Millions of containers / pods / vms
● Community Support - Many people using it
● Community Knowledge Online - Many people documenting

6. Cortex
● Global View - Centralized data
● Multi-Replica / High Availability - Dedupe at write
● Long Term Storage - NoSQL Index + Chunks
○ Index (Cassandra / DynamoDB/ BigTable)
○ Chunk (Cassandra / DynamoDB/ BigTable/S3 /
GCS/Azure)

7. Cortex
●

8. Cortex
●

9. Cortex
●

10. Thanos
● Global View - Federated Data / Fan out queries
● Multi-Replica / High Availability - Query time dedupe
● Long Term Storage - TSDB blocks in object store
○ GCS
○ S3 Compatible (Ceph/ Minio
○ Azure Blob Storage
○ ….

11. Thanos

12. Thanos - Basic Architecture
●

13. Thanos Architecture

14. Thanos
●

15. Thanos
●

16. Thanos / Cortex Together

17. Resources
● Thanos - Scalable Prometheus
(https://www.infoq.com/news/2018/06/thanos-scalable-prometheus )
● Cortex Architecture (https://cortexmetrics.io/docs/architecture/)
● Thanos (https://thanos.io/)
● Challenges of Prometheus at Scale (https://sysdig.com/blog/challenges-scale-prometheus/)
● Tutorial : Prometheus at Scale
(https://epsagon.com/tools/thanos-tutorial-prometheus-at-scale/)
● Github / Cortex (https://github.com/cortexproject/cortex)

18. Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037