Monitorer l'inconnu, 1000 * 100 series par jour - talk avec @clementd à #devoxxFR 2018
•Download as PPTX, PDF•
0 likes•404 views
Slide créé sur google slides https://docs.google.com/presentation/d/1pZvS5BEFfXceS3xXIePKkeAx-aZpxhloNInaIHD5eTw/edit?usp=sharing Comment monitorer ce qu’on ne connait pas? Un des défis technique chez Clever CLoud, à part la scalabilité, c’est de monitorer automatiquement toutes les stacks techniques de nos clients, sans que l’on sache quoi que ce soit. Notre premier but quand nous avons reconstruit notre plateforme de monitoring était de supporter notre pattern Imutable Infrastructure qui génère quantité de hosts éphémères chaque minute. L’approche traditionel est de se concentrer sur les VMs et les Hosts, pas les applications? Il fallait changer de paradigme pour avoir une approche de découverte automatique des métriques à monitorer, permettre à du code tiers de publier ses propres métriques. Ce talk décrit le chemin qui nous a ammené à construire Clever Cloud Metrics, basé sur Warp10 ( basé sur Kafka/Hadoop/Storm) pour améliorer les conditions de travail de nos utilisateurs et la stabilité de nos applications.
Recommended
Recommended
More Related Content
Similar to Monitorer l'inconnu, 1000 * 100 series par jour - talk avec @clementd à #devoxxFR 2018
Similar to Monitorer l'inconnu, 1000 * 100 series par jour - talk avec @clementd à #devoxxFR 2018 (20)
More from Quentin Adam
More from Quentin Adam (20)
Recently uploaded
Recently uploaded (20)
Monitorer l'inconnu, 1000 * 100 series par jour - talk avec @clementd à #devoxxFR 2018
- 2. @clementd & @waxzce Monitoring the Unknown 1000*100 series a day Quentin Adam / @waxzce Clément Delafargue / @clementd
- 3. @clementd & @waxzce Who are we? Quentin 🐧 ADAM CEO @ Clever Cloud @waxzce on twitter Clément DELAFARGUE🐇 CTO @ Clever Cloud @clementd on Twitter
- 4. @clementd & @waxzce What is Clever Cloud?
- 5. @clementd & @waxzce You need metrics
- 6. @clementd & @waxzce Metrics are not just for ops
- 7. @clementd & @waxzce … but most tools stay ops-oriented
- 8. @clementd & @waxzce Developers also deserve metrics
- 9. @clementd & @waxzce Not just for forensics
- 10. @clementd & @waxzce We need metrics for devs and ops
- 11. @clementd & @waxzce We need metrics for DevOps
- 12. @clementd & @waxzce We still live in a serverful world
- 13. @clementd & @waxzce Cross-compare application-level and system-level metrics
- 14. @clementd & @waxzce A brave new world
- 15. @clementd & @waxzce An immutable new world Understand immutable infrastructure by @waxzce https://www.youtube.com/watch?v=WrZCbgQsPVU
- 16. @clementd & @waxzce Pet versus Cattle
- 17. @clementd & @waxzce 1 service, many servers
- 18. @clementd & @waxzce More is more
- 19. @clementd & @waxzce More volume
- 20. @clementd & @waxzce More series
- 21. @clementd & @waxzce How we do it at
- 22. @clementd & @waxzce We don’t know the apps we have to monitor
- 23. @clementd & @waxzce Metrics collection
- 24. @clementd & @waxzce Collection agent
- 25. @clementd & @waxzce We chose telegraf
- 26. @clementd & @waxzce JVM agent to grab JMX data
- 27. @clementd & @waxzce Extensibility with statsd
- 28. @clementd & @waxzce Extensibility with prometheus
- 29. @clementd & @waxzce Push / Pull metrics
- 30. @clementd & @waxzce How do we store all that?
- 31. @clementd & @waxzce Our needs
- 32. @clementd & @waxzce Humongous volume
- 33. @clementd & @waxzce Various access patterns
- 34. @clementd & @waxzce Time Series DataBase
- 35. @clementd & @waxzce What are time series?
- 36. @clementd & @waxzce What you need to pay attention to
- 37. @clementd & @waxzce Real time raw points access
- 38. @clementd & @waxzce Analysis capabilities
- 44. @clementd & @waxzce Home-made solution based on Cassandra
- 45. @clementd & @waxzcehttps://github.com/cityzendata/warp10-platform / http://www.warp10.io/ Warp10 Plain old java Standalone version fat jar using levelDB GeoTime Series
- 46. @clementd & @waxzce Awesome Distributed system
- 47. @clementd & @waxzce Token-based access
- 49. @clementd & @waxzce Raw data access
- 51. @clementd & @waxzce "<TOKEN>" 'cpu.usage_user' { 'host' '<instance>' 'app_id' '=<app_id>'} NOW -140 FETCH <open in Quantum>
- 52. @clementd & @waxzce [ $POINTS bucketizer.mean // bucketizer 0 // lastbucket 0 // bucketspan 5 // bucketcount ] BUCKETIZE <open in quantum>
- 53. @clementd & @waxzce [ $POINTS mapper.rate // mapper 1 // before 0 // after 0 // points ] MAP <open in quantum>
- 54. @clementd & @waxzce [ $POINTS [ ‘app_id’ ] reducer.mean.exclude-nulls ] REDUCE <open in quantum>
- 55. @clementd & @waxzce Full reference http://www.warp10.io/reference/reference/
- 56. @clementd & @waxzce Triggers / μBatching
- 57. @clementd & @waxzce Build a new series on a series, recompute the data, become smarter and smarter
- 58. @clementd & @waxzce Poke, good place to get sample https://poke.digital https://docs.google.com/presentation/d/1RfpX-KdfAa5ZxsnuRYi34JbidbVVUcz7bYy5x- k6BBE/edit?usp=sharing https://www.pscp.tv/waxzce/1OwGWEEvapkxQ?t=4m51s
- 59. @clementd & @waxzce We’ve got a lot of data points. Good. Now what?
- 61. @clementd & @waxzce Graphs are good.
- 62. @clementd & @waxzce Graphs without units… not so good
- 64. @clementd & @waxzce Data good. Meaning better.
- 65. @clementd & @waxzce Metrics-spec (collaboration with the OVH metrics team) https://github.com/CleverCloud/metrics-spec
- 67. @clementd & @waxzce What you should look for when you build your own metrics pipeline.
- 68. @clementd & @waxzce List what you’ll need,
- 69. @clementd & @waxzce Collect more
- 70. @clementd & @waxzce Mind the ACLs (or lack thereof)
- 71. @clementd & @waxzce Visualize, then analyze
- 72. @clementd & @waxzce Thanks @waxzce & @clementd Try 💡☁, here is a voucher: metrics4devoxx
Editor's Notes
- Clever Cloud: IT automation, on gère la prod, et on l’automatise Public Cloud, Entreprise, On Prem
- Interesting things happen when you’re not looking. For errors, you have logs, sure, but for subtler things, you need a finer view (CPU use, network usage, latency, …)
- Usually metrics are an ops thing: system-level metrics like CPU, RAM, network. Application-level metrics are important: JVM GC status, number of active sessions, number of validated carts…
- Zabbix, Centreon stuff like that are made for ops. Metrics are gathered host by host (ie machine by machine), and access control is complicated. Normally only ops have access to this platform, so it’s complex for devs to have access
- Application-level metrics are super important, and help to produce better applications.
- We tend to use metrics to gather information about outages or issues, but also to see how people use the applications, or to help with perf optimization efforts
- There are metrics system for application-level metrics, but both views are important.
- Servers still happen, so application-level and system-level metrics are both equally important, they give interesting context with each other.
- You need an unified metrics gathering pipeline, both for system and application-level metrics
- Systems like zabbix and centreon are not suited for new architectures
- Traditional monitoring / metrics in an immutable infrastructure does not make sense
- VMs are short-lived and disposable. In zabbix, metrics are lost when a VM is shut down.
- in an immutable infrastructure, one service is handled by many servers along its lifetime. you need to aggregate metrics by service, not see it server by server.
- Instead of a few servers, there are now hundreds or thousands of ephemeral virtual machines. Traditional metrics pipeline have a hard time with that.
- Clever Cloud: 20GB / hour
- ~100 series per VM, ~1000 new VMs everyday. Need to aggregate series with metadata, not feasible to consume data series by series.
- Immutable infrastructure, Thousands of applications, lots of tech stacks.
- Collection happens on the VMs, both for system & application level metrics
- We can’t just send metrics naively to the platform. Buffer: gather metrics and send batches periodically Retry: re-send metrics if needed Jitter: make sure all agents don’t send metrics at the same time
- Written in go, easy to deploy, lots of input plugins
- Understands the statsd protocol. Does local aggregation
- Polls a prometheus endpoint, does gathering
- statsd is push-based: metrics are sent directly. Less work on the app side prometheus is pull-based: app is queried by the agent. not so convenient, but with telegraf it’s not so bad.
- lots of data (1 point every minute * 100, + 1 point every 10 seconds for a few series, like CPU & RAM)
- we have specific needs: immutable infra, lots of applications, lots of tech stacks
- several thousand instances running at the same time
- We need both real time access for the dashboard, and analysis over longer periods.
- DBs made to store Time Series
- TimeSeries: a Series of points over time. Eg used RAM over time, or % Idle CPU time. Each point is the same thing, at a different time.
- TSDBs have different needs than regular DBs
- What is cardinality, why is it an important thing?
- Allow access delegation, everybody talks directly to the DB. Secure by default, simpler network config
- Different way to use the stored data.
- Just list points
- Aggregate and filter things
- Aggregate and filter things https://quantum.services.clever-cloud.com/#/warpscript/IlFCUGY0WkREYk5DNXE3TEwyVkUzVFRSZEdWZVk2V0Nqbm5JLnUxZ2hBRm9fMmNrSjNkZXMyQXlxcmsyWGd5T3RRZ3RKeFQ5d3dLblFaTi53WEdVVnlsNnZLbUN3Zzc4clIyWHYudE5fWHRXZ3JlUXZiQjhsVnVGQTVjS1NSd05tUkYzZUpReVFNOUYwdmR6NGdGRFJZVTNFNkRDOFZxUjhkMEhZZkwuTldxRnJxMk1OLmdRUGxoSXJBYUp6X29MZTdvZUpydFNXOFJ3IgondG9rZW4nIFNUT1JFCgokdG9rZW4KJ2Zhc3RfY3B1LnVzYWdlX2lkbGUnCnsgJ2hvc3QnICd%2BKDI2NjhiNDQ1LThiMmQtNDk1MS05OGRlLTgyYjcxNGVkOGU3YXw5ZTk0ZDU5NC0xOTU5LTQ2MjUtOTAwNC1iYmMzY2FlOGU2MTApJyAnYXBwX2lkJyAnPWFwcF9iODg2MTdhOC02MDRmLTQ2N2YtYTk2Zi02MDZjYWJhYjNjODYnICdjcHUnICdjcHUtdG90YWwnIH0KTk9XIC0xNDAKRkVUQ0g%3D/eyJ1cmwiOiJodHRwczovL2MxLXdhcnAxMC1jbGV2ZXJjbG91ZC1jdXN0b21lcnMuc2VydmljZXMuY2xldmVyLWNsb3VkLmNvbS9hcGkvdjAiLCJmZXRjaEVuZHBvaW50IjoiL2ZldGNoIiwiaGVhZGVyTmFtZSI6IlgtV2FycDEwIn0%3D
- https://quantum.services.clever-cloud.com/#/warpscript/IlFCUGY0WkREYk5DNXE3TEwyVkUzVFRSZEdWZVk2V0Nqbm5JLnUxZ2hBRm9fMmNrSjNkZXMyQXlxcmsyWGd5T3RRZ3RKeFQ5d3dLblFaTi53WEdVVnlsNnZLbUN3Zzc4clIyWHYudE5fWHRXZ3JlUXZiQjhsVnVGQTVjS1NSd05tUkYzZUpReVFNOUYwdmR6NGdGRFJZVTNFNkRDOFZxUjhkMEhZZkwuTldxRnJxMk1OLmdRUGxoSXJBYUp6X29MZTdvZUpydFNXOFJ3IgondG9rZW4nIFNUT1JFCgokdG9rZW4KJ2Zhc3RfY3B1LnVzYWdlX2lkbGUnCnsgJ2hvc3QnICd%2BKDI2NjhiNDQ1LThiMmQtNDk1MS05OGRlLTgyYjcxNGVkOGU3YXw5ZTk0ZDU5NC0xOTU5LTQ2MjUtOTAwNC1iYmMzY2FlOGU2MTApJyAnYXBwX2lkJyAnPWFwcF9iODg2MTdhOC02MDRmLTQ2N2YtYTk2Zi02MDZjYWJhYjNjODYnICdjcHUnICdjcHUtdG90YWwnIH0KTk9XIC0xNDAKRkVUQ0gKJ3BvaW50cycgU1RPUkUKClsKICAkcG9pbnRzCiAgYnVja2V0aXplci5tZWFuCiAgMAogIDAKICA1Cl0gQlVDS0VUSVpF/eyJ1cmwiOiJodHRwczovL2MxLXdhcnAxMC1jbGV2ZXJjbG91ZC1jdXN0b21lcnMuc2VydmljZXMuY2xldmVyLWNsb3VkLmNvbS9hcGkvdjAiLCJmZXRjaEVuZHBvaW50IjoiL2ZldGNoIiwiaGVhZGVyTmFtZSI6IlgtV2FycDEwIn0%3D
- https://quantum.services.clever-cloud.com/#/warpscript/IlFCUGY0WkREYk5DNXE3TEwyVkUzVFRSZEdWZVk2V0Nqbm5JLnUxZ2hBRm9fMmNrSjNkZXMyQXlxcmsyWGd5T3RRZ3RKeFQ5d3dLblFaTi53WEdVVnlsNnZLbUN3Zzc4clIyWHYudE5fWHRXZ3JlUXZiQjhsVnVGQTVjS1NSd05tUkYzZUpReVFNOUYwdmR6NGdGRFJZVTNFNkRDOFZxUjhkMEhZZkwuTldxRnJxMk1OLmdRUGxoSXJBYUp6X29MZTdvZUpydFNXOFJ3IgondG9rZW4nIFNUT1JFCgokdG9rZW4KJ2Zhc3RfY3B1LnVzYWdlX2lkbGUnCnsgJ2hvc3QnICd%2BKDI2NjhiNDQ1LThiMmQtNDk1MS05OGRlLTgyYjcxNGVkOGU3YXw5ZTk0ZDU5NC0xOTU5LTQ2MjUtOTAwNC1iYmMzY2FlOGU2MTApJyAnYXBwX2lkJyAnPWFwcF9iODg2MTdhOC02MDRmLTQ2N2YtYTk2Zi02MDZjYWJhYjNjODYnICdjcHUnICdjcHUtdG90YWwnIH0KTk9XIC0xNDAKRkVUQ0gKJ3BvaW50cycgU1RPUkUKClsKICAkcG9pbnRzCiAgbWFwcGVyLnJhdGUKICAxCiAgMAogIDAKXSBNQVA%3D/eyJ1cmwiOiJodHRwczovL2MxLXdhcnAxMC1jbGV2ZXJjbG91ZC1jdXN0b21lcnMuc2VydmljZXMuY2xldmVyLWNsb3VkLmNvbS9hcGkvdjAiLCJmZXRjaEVuZHBvaW50IjoiL2ZldGNoIiwiaGVhZGVyTmFtZSI6IlgtV2FycDEwIn0%3D
- https://quantum.services.clever-cloud.com/#/warpscript/IlFCUGY0WkREYk5DNXE3TEwyVkUzVFRSZEdWZVk2V0Nqbm5JLnUxZ2hBRm9fMmNrSjNkZXMyQXlxcmsyWGd5T3RRZ3RKeFQ5d3dLblFaTi53WEdVVnlsNnZLbUN3Zzc4clIyWHYudE5fWHRXZ3JlUXZiQjhsVnVGQTVjS1NSd05tUkYzZUpReVFNOUYwdmR6NGdGRFJZVTNFNkRDOFZxUjhkMEhZZkwuTldxRnJxMk1OLmdRUGxoSXJBYUp6X29MZTdvZUpydFNXOFJ3IgondG9rZW4nIFNUT1JFCgokdG9rZW4KJ2Zhc3RfY3B1LnVzYWdlX2lkbGUnCnsgJ2hvc3QnICd%2BKDI2NjhiNDQ1LThiMmQtNDk1MS05OGRlLTgyYjcxNGVkOGU3YXw5ZTk0ZDU5NC0xOTU5LTQ2MjUtOTAwNC1iYmMzY2FlOGU2MTApJyAnYXBwX2lkJyAnPWFwcF9iODg2MTdhOC02MDRmLTQ2N2YtYTk2Zi02MDZjYWJhYjNjODYnICdjcHUnICdjcHUtdG90YWwnIH0KTk9XIC0xNDAKRkVUQ0gKJ3BvaW50cycgU1RPUkUKClsKICAkcG9pbnRzCiAgWyAnYXBwX2lkJyBdCiAgcmVkdWNlci5tZWFuLmV4Y2x1ZGUtbnVsbHMKXSBSRURVQ0U%3D/eyJ1cmwiOiJodHRwczovL2MxLXdhcnAxMC1jbGV2ZXJjbG91ZC1jdXN0b21lcnMuc2VydmljZXMuY2xldmVyLWNsb3VkLmNvbS9hcGkvdjAiLCJmZXRjaEVuZHBvaW50IjoiL2ZldGNoIiwiaGVhZGVyTmFtZSI6IlgtV2FycDEwIn0%3D
- Before automation, visual exploration is super important.
- Great way to see what’s going on
- Metrics are stored without units. Are those bits, bytes, megabytes, number of packets?
- We have data, but that’s not enough. We’re not interested in raw data points, we want meaning.
- We have data, but that’s not enough. We’re not interested in raw data points, we want meaning.