1. Anomaly Detection
iwanaga

2. Who am I
@quake_alert
@quake_alert_en
@quake_alert_fr
@quake_alert_kr
Yoshihiro Iwanaga

3. Motivation for detecting
anomaly
Traditional system monitoring
• process existence
• ping, http, tcp response
• disk usage
→ “fixed” rule / threshold

4. Motivation for detecting
anomaly
Notice something out of ordinary
• network traffic is heavier than usual
• number of login try is obviously larger
• a colleague is strangely gracious today
→ Unusual behaviors; Indications of fault.
Such info helps
preventing service degrading in advance!!
but rule/threshold vary with service, host, client, time…

5. key to detect anomaly
usual unusual
Watch differences b/w

6. e.g. Network Traffic
Mon Tue Wed Thu Fri
traffic
time

7. Superimpose 24 hour plot
Traffic at 15:00 on workday
is about 1.2 Gbps
traffic
time
Periodicity!!

8. mean
mean － 3σ
mean + 3σ
amount of dispersion from mean
Acceptable “range”
→ e.g. Acceptable range of traffic at 15:00 on workday is
1.01 to 1.38 Gbps

9. Case examples

10. DDoS
partial
hardware failure
Traffic

11. number of mail passed spam filterspam rate
e-mail
Applied a wrong spam rule

12. However
Reality is not that simple…
人生楽ありゃ苦もあるさ
涙の後には虹も出る
歩いてゆくんだしっかりと
自分の道をふみしめて
山上路夫

13. downloading large files
mass e-mail sending
“Traffic spike” happens so frequently
Frequent false-positive alerting will be
“cry-wolf” system…

14. heuristic filtering
In usual, traffic gets cool down
within 15 minutes
notify engineers
if anomaly continues more than 15 minutes
Engineers’ knowledge is gold mine
for better algorithm 
→ one practical example: