The document discusses monitoring strategies and technologies for IT infrastructure, particularly focusing on data collection and analysis through various tools like InfluxDB and Kafka. It emphasizes building a solution for ephemeral instances and highlights the integration of the monitoring service with systems using agents like Telegraf. The author, Quentin Adam from Clever Cloud, advocates for community contribution and offers insights into improving developer productivity through enhanced access to metrics and logs.

1. Monitoring the unknown,
1000*100 series a day
Quentin ADAM - @waxzce
2017

2. Who am I ?
Quentin ADAM from the Clever Cloud
@waxzce on twitter – github- soundcloud – instagram ….

3. My day to day work :
, the IT automation company

4. Keep your apps online. made with node.js,
scala, java, ruby, php, python, go…
Cloud & on premise

5. And learn a lot of things
about your code, apps,
and good/bad design…

6. Give back to the
community
NEVER GONNA
LET YOU DOWN
clever-cloud.com

7. Automate infrastructure

8. What are we talking about?

9. Immutable infrastructure
FR : https://www.youtube.com/watch?v=WrZCbgQsPVU
EN : https://www.infoq.com/presentations/server-cloud

10. Ephemeral instances

11. Pet
!=
Cattle

12. Monitoring service
!=
Monitoring host

13. Monitoring the unknow
Part 1

14. Which tech stack
peoples will use?

15. Monitoring the unknow
Part 2

16. There is no ready solution…

17. Let’s build something

18. Data needs

19. Store measure in time

20. Analyze it
Most of the time using a micro batching approach

21. Traditional needs not required
• No ACID
• No Transaction
• Just a few full text
• Eventual consistency if enough

22. SCALE

23. Time Series DB

24. Influx DB

25. Prometeus

26. Open TSDB

27. Cassandra + Akka persistence/stream

28. Warp10

29. Standalone
Java fat jar

30. Distributed
• Consensus using Zookeepers
• Store and analysis on Hadoop
• Event bus with Kafka
• Several roles in a single JAR, all stateless

31. So, We can now store data

32. Agent: Telegraf

33. Agent goals

34. Collect data
• System
• Many connector to specific softs
• Open protocols…

35. JMX

36. Statsd

37. Prometeus

38. Buffer to send batched data
~100 metric point / minutes

39. Token based security

40. More tokens: JWT and Macaroon
EN : https://www.youtube.com/watch?v=SO5G0Gsm5-w
FR : https://www.youtube.com/watch?v=mvKeCsxGZhE

41. ✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
❔ Analyze data
❔ Trigger data
❔ Display data

42. WarpScript, distributed on hadoop

43. Warp10 roles
• Ingress
• Directory
• Store
• Egress
• Webcall
• Plasma
• Runner
• Fetcher

44. µBatching WarpScriptµBatching WarpScriptµBatching WarpScript

45. Build a new series

46. Listen on Kafka or Plasma for triggers

47. Immutable DataBase

48. ✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
✅ Analyze data
✅ Trigger data
❔ Display data

49. Quantum
Tool
+
PolymerJS lib

50. Quantum
Tool
+
PolymerJS lib

51. Which data expose?

52. Templating the data

53. Templating the data

54. Templating the data

57. Give access
to the API

58. Enhance developer work, happiness and
productivity

59. Give access to metrics and logs

60. Thank you
find me on twitter
@waxzce
Gift coupon for
clever-cloud.com:
devoxxMA2017