KOWAZA for mackerel

2,399 views

Published on

Mackerel Meetup #7 LT

Published in: Engineering
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,399
On SlideShare
0
From Embeds
0
Number of Embeds
1,789
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

KOWAZA for mackerel

  1. 1. KOWAZA for Mackerel id:karupanerura Mackerel Meetup #7 LT
  2. 2. KOWAZA means small trick in Japanese
  3. 3. about me • id:karupanerura (twitter/github) • Perl/Go/Swift/Kotlin/Java/JS/etc.. • Senior Engineer at Mobile Factoy, Inc. • Join to Infrastructure Team since 2016/02 • Director at Japan Perl Association (JPA)
  4. 4. mackerel with me • Mackerel contributer • mackerel-agent-plugins (gearman) • mackerel-check—plugins (masterha, ssh) • Mackerel UG Organizer ← NEW
  5. 5. monitor processes
  6. 6. check-procs -p mysqld
  7. 7. Wait…
  8. 8. (In my environment) MySQL is stopped to backup at 5am-6am.
  9. 9. 🤔
  10. 10. true && echo “true” || echo “false”
  11. 11. 😙
  12. 12. [ 5 -le “$hour” -and “$hour” -le 6 ] && echo “OK check-procs: In shutdown” || check-procs -p mysqld
  13. 13. 😎
  14. 14. monitor logs
  15. 15. grep foo /path/to/log | grep bar | grep -v baz | wc -l
  16. 16. check-log plugin
  17. 17. check-log -f /path/to/log -p foo -p bar -E baz -c 1
  18. 18. Wait…
  19. 19. check-log -f /path/to/log -p foo -p bar ←Cannot do it :( -E baz -c 1
  20. 20. 🤔
  21. 21. check-log -f <(grep foo /path/to/log) -p bar -E baz -c 1
  22. 22. 😙
  23. 23. grep foo /path/to/log | check-log -f /dev/stdin -p bar -E baz -c 1
  24. 24. 😎
  25. 25. monitor mouts
  26. 26. Should mount a HDD as readable & writable
  27. 27. (If mount with errors=remout-ro) When the HDD is breaking, re-mount the HDD readonly
  28. 28. monitor with Mackerel
  29. 29. # XXX: exit code 2 is CRITICAL touch /tmp/write-test.watch || exit 2
  30. 30. But, hackish…
  31. 31. no kind message outputs…
  32. 32. 🤔
  33. 33. check-log -f /proc/mounts -p ' ro(,|$)’ --no-state
  34. 34. 😙
  35. 35. Let’s hack Mackerel !!
  36. 36. 😎

×