Anomaly Detection by Mean and Standard Deviation (LT at AQ)

2,510 views

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,510
On SlideShare
0
From Embeds
0
Number of Embeds
53
Actions
Shares
0
Downloads
27
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Anomaly Detection by Mean and Standard Deviation (LT at AQ)

  1. 1. Anomaly Detection iwanaga
  2. 2. Who am I @quake_alert @quake_alert_en @quake_alert_fr @quake_alert_kr Yoshihiro Iwanaga
  3. 3. Motivation for detecting anomaly Traditional system monitoring • process existence • ping, http, tcp response • disk usage → “fixed” rule / threshold
  4. 4. Motivation for detecting anomaly Notice something out of ordinary • network traffic is heavier than usual • number of login try is obviously larger • a colleague is strangely gracious today → Unusual behaviors; Indications of fault. Such info helps preventing service degrading in advance!! but rule/threshold vary with service, host, client, time…
  5. 5. key to detect anomaly usual unusual Watch differences b/w
  6. 6. e.g. Network Traffic Mon Tue Wed Thu Fri traffic time
  7. 7. Superimpose 24 hour plot Traffic at 15:00 on workday is about 1.2 Gbps traffic time Periodicity!!
  8. 8. mean mean - 3σ mean + 3σ amount of dispersion from mean Acceptable “range” → e.g. Acceptable range of traffic at 15:00 on workday is 1.01 to 1.38 Gbps
  9. 9. Case examples
  10. 10. DDoS partial hardware failure Traffic
  11. 11. number of mail passed spam filterspam rate e-mail Applied a wrong spam rule
  12. 12. However Reality is not that simple… 人生楽ありゃ苦もあるさ 涙の後には虹も出る 歩いてゆくんだしっかりと 自分の道をふみしめて 山上路夫
  13. 13. downloading large files mass e-mail sending “Traffic spike” happens so frequently Frequent false-positive alerting will be “cry-wolf” system…
  14. 14. heuristic filtering In usual, traffic gets cool down within 15 minutes notify engineers if anomaly continues more than 15 minutes Engineers’ knowledge is gold mine for better algorithm  → one practical example:

×