Anomaly Detection by Mean and Standard Deviation (LT at AQ)
Upcoming SlideShare
Loading in...5
×
 

Anomaly Detection by Mean and Standard Deviation (LT at AQ)

on

  • 668 views

 

Statistics

Views

Total Views
668
Views on SlideShare
618
Embed Views
50

Actions

Likes
1
Downloads
4
Comments
0

1 Embed 50

http://www.scoop.it 50

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Anomaly Detection by Mean and Standard Deviation (LT at AQ) Anomaly Detection by Mean and Standard Deviation (LT at AQ) Presentation Transcript

  • Anomaly Detection iwanaga
  • Who am I @quake_alert @quake_alert_en @quake_alert_fr @quake_alert_kr Yoshihiro Iwanaga
  • Motivation for detecting anomaly Traditional system monitoring • process existence • ping, http, tcp response • disk usage → “fixed” rule / threshold
  • Motivation for detecting anomaly Notice something out of ordinary • network traffic is heavier than usual • number of login try is obviously larger • a colleague is strangely gracious today → Unusual behaviors; Indications of fault. Such info helps preventing service degrading in advance!! but rule/threshold vary with service, host, client, time…
  • key to detect anomaly usual unusual Watch differences b/w
  • e.g. Network Traffic Mon Tue Wed Thu Fri traffic time
  • Superimpose 24 hour plot Traffic at 15:00 on workday is about 1.2 Gbps traffic time Periodicity!!
  • mean mean - 3σ mean + 3σ amount of dispersion from mean Acceptable “range” → e.g. Acceptable range of traffic at 15:00 on workday is 1.01 to 1.38 Gbps
  • Case examples
  • DDoS partial hardware failure Traffic
  • number of mail passed spam filterspam rate e-mail Applied a wrong spam rule
  • However Reality is not that simple… 人生楽ありゃ苦もあるさ 涙の後には虹も出る 歩いてゆくんだしっかりと 自分の道をふみしめて 山上路夫
  • downloading large files mass e-mail sending “Traffic spike” happens so frequently Frequent false-positive alerting will be “cry-wolf” system…
  • heuristic filtering In usual, traffic gets cool down within 15 minutes notify engineers if anomaly continues more than 15 minutes Engineers’ knowledge is gold mine for better algorithm  → one practical example: