Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
@omerlh
The Dark Side of Monitoring
Omer Levi Hevroni
May 2019
@omerlh
Observability
https://www.navantis.com/managed-monitoring/
@omerlh
Can Hackers Use It Too?
@omerlh
I’m a builder
@omerlh
@omerlh
DevSecOps @
@omerlh
Threat Modeling?
@omerlh
Threat Modeling?
● What are we building?
● What can go wrong?
● What are we doing about it?
● Did we do a good job?
@omerlh
What is Monitoring?
@omerlh
@omerlh
@omerlh
Black Box
Monitoring
White Box
Monitoring
https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-...
@omerlh
Let’s Dive In!
@omerlh
White Box
Monitoring
@omerlh
Threat Modeling
● What are we building?
● What can go wrong?
● What are we doing about it?
● Did we do a good job?
@omerlh
White-box Monitoring
Monitoring based on metrics exposed by the internals of the system, including
logs, interface...
@omerlh
White-box Monitoring for Kubernetes
● Probes (liveness/readiness)
● Prometheus
@omerlh
Probes
@omerlh
What are we building?
@omerlh
Probes
@omerlh
Why?
●Check dependencies
○ Databases
○ Other micro-services
●Kill services if it misfunction before users experien...
@omerlh
Code Example
https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks
@omerlh
What Can go wrong?
@omerlh
Exploiting liveness - information disclosure
@omerlh
Exploiting liveness – Denial of Service
@omerlh
@omerlh
What are we doing about it?
@omerlh
Block access
https://www.edureka.
co/community/19277/
access-some-
specific-paths-while-
using-kubernetes-
ingress...
@omerlh
Caching
Source code - Kamus
@omerlh
Prometheus
@omerlh
What are we building?
@omerlh
Prometheus
● Monitoring system
● Time-Series database
● Alerting system
● Auto-discovery
https://prometheus.io/
@omerlh
Prometheus Scraping Model
@omerlh
What Can go wrong?
@omerlh
Exploiting Prometheus - information disclosure
@omerlh
@omerlh
Some interesting metrics...
@omerlh
What about this metric?
@omerlh
Here is the code behind it…
@omerlh
What Can We Do About It?
@omerlh
Block access
https://www.edureka.
co/community/19277/
access-some-
specific-paths-while-
using-kubernetes-
ingress...
@omerlh
Prometheus Metrics Limit
https://www.omerlh.info/2019/03/04/keeping-prometheus-in-shape/
@omerlh
Did we do a good job?
@omerlh
Threat Modeling
● What are we building?
● What can go wrong?
● What are we doing about it?
● Did we do a good job?
@omerlh
Black Box
Monitoring
White Box
Monitoring
https://landing.google.com/sre/sre-
book/chapters/monitoring-distributed...
@omerlh
Movie Streaming
@omerlh
What are we building?
@omerlh
Black-box Monitoring
Testing externally visible behavior as a user would see it.
@omerlh
Our Monitoring System
@omerlh
What Can go wrong?
@omerlh
STRIDE
S – Spoofing
T – Tampering
R – Repudiation
I – Information Disclosure
D – Denial of Service
E – Elevation o...
@omerlh
Spoofing
@omerlh
Information Disclosure
@omerlh
Repudiation
@omerlh
Denial of Service
@omerlh
What are we doing about it?
@omerlh
Potential Mitigations
● Least Privilege
● Block access
● Tracing
● Limit to test data only
@omerlh
Did we do a good job?
@omerlh
http://www.applestory.biz/hermione-hand-raise-gif.html
Questions?
@omerlh
Can Hackers Use It Too?
@omerlh
Key Take Away
● Monitoring is just code
● Careful when exposed to the internet
● Use cache when possible
● Conduct...
@omerlh
Feedback appreciated
@omerlh
Observability
https://www.navantis.com/managed-monitoring/
@omerlh
Thank You!
Omer Levi Hevroni
April 2019
Upcoming SlideShare
Loading in …5
×

The Dark Side of Monitoring

49 views

Published on

We all use monitoring - after all, we all want to ensure that our applications are working as expected. But can hackers use it to exploit our application? Join me to explore different ways to exploit application monitoring tools, and what mitigations we can use.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The Dark Side of Monitoring

  1. 1. @omerlh The Dark Side of Monitoring Omer Levi Hevroni May 2019
  2. 2. @omerlh Observability https://www.navantis.com/managed-monitoring/
  3. 3. @omerlh Can Hackers Use It Too?
  4. 4. @omerlh I’m a builder @omerlh
  5. 5. @omerlh DevSecOps @
  6. 6. @omerlh Threat Modeling?
  7. 7. @omerlh Threat Modeling? ● What are we building? ● What can go wrong? ● What are we doing about it? ● Did we do a good job?
  8. 8. @omerlh What is Monitoring?
  9. 9. @omerlh
  10. 10. @omerlh
  11. 11. @omerlh Black Box Monitoring White Box Monitoring https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/
  12. 12. @omerlh Let’s Dive In!
  13. 13. @omerlh White Box Monitoring
  14. 14. @omerlh Threat Modeling ● What are we building? ● What can go wrong? ● What are we doing about it? ● Did we do a good job?
  15. 15. @omerlh White-box Monitoring Monitoring based on metrics exposed by the internals of the system, including logs, interfaces like the Java Virtual Machine Profiling Interface, or an HTTP handler that emits internal statistics.
  16. 16. @omerlh White-box Monitoring for Kubernetes ● Probes (liveness/readiness) ● Prometheus
  17. 17. @omerlh Probes
  18. 18. @omerlh What are we building?
  19. 19. @omerlh Probes
  20. 20. @omerlh Why? ●Check dependencies ○ Databases ○ Other micro-services ●Kill services if it misfunction before users experience it
  21. 21. @omerlh Code Example https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks
  22. 22. @omerlh What Can go wrong?
  23. 23. @omerlh Exploiting liveness - information disclosure
  24. 24. @omerlh Exploiting liveness – Denial of Service
  25. 25. @omerlh
  26. 26. @omerlh What are we doing about it?
  27. 27. @omerlh Block access https://www.edureka. co/community/19277/ access-some- specific-paths-while- using-kubernetes- ingress?show=19278 #a19278
  28. 28. @omerlh Caching Source code - Kamus
  29. 29. @omerlh Prometheus
  30. 30. @omerlh What are we building?
  31. 31. @omerlh Prometheus ● Monitoring system ● Time-Series database ● Alerting system ● Auto-discovery https://prometheus.io/
  32. 32. @omerlh Prometheus Scraping Model
  33. 33. @omerlh What Can go wrong?
  34. 34. @omerlh Exploiting Prometheus - information disclosure
  35. 35. @omerlh
  36. 36. @omerlh Some interesting metrics...
  37. 37. @omerlh What about this metric?
  38. 38. @omerlh Here is the code behind it…
  39. 39. @omerlh What Can We Do About It?
  40. 40. @omerlh Block access https://www.edureka. co/community/19277/ access-some- specific-paths-while- using-kubernetes- ingress?show=19278 #a19278
  41. 41. @omerlh Prometheus Metrics Limit https://www.omerlh.info/2019/03/04/keeping-prometheus-in-shape/
  42. 42. @omerlh Did we do a good job?
  43. 43. @omerlh Threat Modeling ● What are we building? ● What can go wrong? ● What are we doing about it? ● Did we do a good job?
  44. 44. @omerlh Black Box Monitoring White Box Monitoring https://landing.google.com/sre/sre- book/chapters/monitoring-distributed-systems/
  45. 45. @omerlh Movie Streaming
  46. 46. @omerlh What are we building?
  47. 47. @omerlh Black-box Monitoring Testing externally visible behavior as a user would see it.
  48. 48. @omerlh Our Monitoring System
  49. 49. @omerlh What Can go wrong?
  50. 50. @omerlh STRIDE S – Spoofing T – Tampering R – Repudiation I – Information Disclosure D – Denial of Service E – Elevation of Privileges
  51. 51. @omerlh Spoofing
  52. 52. @omerlh Information Disclosure
  53. 53. @omerlh Repudiation
  54. 54. @omerlh Denial of Service
  55. 55. @omerlh What are we doing about it?
  56. 56. @omerlh Potential Mitigations ● Least Privilege ● Block access ● Tracing ● Limit to test data only
  57. 57. @omerlh Did we do a good job?
  58. 58. @omerlh http://www.applestory.biz/hermione-hand-raise-gif.html Questions?
  59. 59. @omerlh Can Hackers Use It Too?
  60. 60. @omerlh Key Take Away ● Monitoring is just code ● Careful when exposed to the internet ● Use cache when possible ● Conduct threat model for black-box monitoring
  61. 61. @omerlh Feedback appreciated
  62. 62. @omerlh Observability https://www.navantis.com/managed-monitoring/
  63. 63. @omerlh Thank You! Omer Levi Hevroni April 2019

×