Link to our Github: https://github.com/zivdar001matin/web-crawler-detection Authors: Ahmad Etezadi, Matin Zivdar We build a web crawler detection model to predict anomaly requests for server log requests in this project. Technologies Used: Machine Learning, MLflow, Scikit-learn, Tensorflow

1. By Ahmad Etezadi &
Matin Zivdar
Log Anomaly
Detection

2. ● Anomaly detection
● Server Logs Dataset
● Pre Processing
● Models
● Evaluation
● API
Summary

3. Anomaly detection
Anomaly detection is one of the most popular machine learning techniques.
In this project, we are asked to identify abnormal behaviors in a system, which relies on the
analysis of logs collected in real-time from the log aggregation systems of an enterprise.

4. Sample Log
IP [TIME] [Method Path] StatusCode ResponseLength [[UserAgent]] ResponseTime

5. Pre Processing
Feature
Extraction
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis sit
amet odio vel purus bibendum luctus.
Generation of features from data that are in a
format that is difficult to analyse directly.
Lorem ipsum dolor sit amet, consectetur
adipiscing elit. Duis sit amet odio vel purus
bibendum luctus.
Feature
Transformation
Transformation of Data:
Encoding, Normalization, etc.

6. Extraction
tehran_traffic_statistics
is_spider
Pre Processing
url_depth is_phone

7. Extraction
URL
Depth
Iran Network Traffic Statistics
OS Family Device Brand
http://members.tehran-ix.ir/statistics/ixp/pkts

8. Transformation
One-hot
encoding
Normalization
time_weight
status_code
Pre Processing
method
Bucketing
response_length requested_file_type

9. Transformation
Bucketing Normalization One hot encoding

10. ● Learning Algorithm
Unsupervised Learning
02
● K-means
● PCA
● Isolation Forest
● AutoEncoder
Supervised Learning
01
● k-NN
● Local Outlier Factor (LOF)
● SVM
● Neural Networks Based
Anomaly
Detection
Algorithm

11. PCA is an unsupervised machine learning algorithm
that attempts to reduce the dimensionality.
Using PCA, you can reduce the dimensionality data
and reconstruct the it.
Since anomaly show the largest reconstruction error,
abnormalities can be found based on the error
between the original data and the reconstructed data.
PCA

12. Anomaly Samples

13. Conclusion

14. Isolation forest works on the principle of the
decision tree algorithm.
Due to the fact that anomalies require fewer
random partitions than normal normal data
points in the data set. So anomalies are points
that have a shorter path in the tree.
Isolation Forest

15. Anomaly Samples

16. Conclusion

17. Comparison
Isolation
Forest
vs.
PCA
Number of Isolation Forest outlier: 124,091
Number of PCA outlier: 123,695
common data in PCA & Isolation Forest outlier: 20,512

18. Let's Dig Deeper

19. An autoencoder is a special type of neural
network that copies the input values to the
output values.
It does not require the target variable like the
conventional Y, thus it is categorized as
unsupervised learning.
Autoencoder

20. Sample of error reconstruction

21. Histogram of error by Autoencoder

22. Evaluation Models
Autoencoder PCA Isolation Forest
Follow the Rules!
- User agents
- Malicious IPs dataset

23. Model ROC Curve

26. API Example

27. We also tried:
- Extracting time series features
- Labeling all other requests for an
anomaly “IP” and “User Agent” as an
anomaly request

28. Thank You!