This document proposes an adaptive and auto-scaling stream processor called Wisdom to enable real-time intrusion detection in network traffic. Wisdom can dynamically optimize complex event processing (CEP) rules using hybrid optimization algorithms like particle swarm optimization and bisection. Tests show Wisdom can detect attacks like HTTP slow header denial of service and port scans with over 99.95% accuracy. Wisdom also allows functionally auto-scaling deployments of CEP rules to optimize resource usage.

1. Real-time Intrusion Detection in Network Traffic Using
Adaptive and Auto-scaling Stream Processor
Gobinath Loganathan∗
, Jagath Samarabandu†
and Xianbin Wang‡
Department of Electrical and Computer Engineering
The University of Western Ontario
London, Ontario, N6A 5B9, Canada
∗
lgobinat@uwo.ca, †
jagath@uwo.ca, ‡
xianbin.wang@uwo.ca

2. Introduction
● Hacking is no more a (black) magic
○ Data and software migrated to servers
○ Availability of Free and Open Source (FOSS) attack simulators
■ Easy to invent new attacks
● Eg: GoldenEye [1] from HTTP Unbearable Load King (HULK) DoS [2]
● Need advanced tools to detect evading attacks
● Intrusion Detection System (IDS)
○ Anomaly-based detection
○ Signature-based detection

3. ● Domain experts define rules to detect attacks
● Highly accurate than anomaly-based detection [3]
○ Hard to derive threshold values
○ Static rules - Soon become obsolete
■ New attacks are being developed by evading known rules
● Complex Event Processing (CEP) has been used for intrusion detection [4, 5]
○ Reactive programming paradigm used in responding to real-time events
○ Stream Processor - A tool to process stream of events
■ Modern Stream Processors have built-in Complex Event Processors
○ Multiple sources and sinks - Use network packets, system logs, user input
○ Suffer from static rules
Signature-based Detection

4. Dynamic CEP and Automatic Rule Mining
● Dynamic CEP - Ability to deploy CEP rules without restarting the system
● Automatic CEP Rule Mining [6,7, 8]
○ Technique - Frequently occuring patterns
○ Intention - Replace domain experts by machines
■ Humans are better than machines in defining logical rules but we struggle with numbers
○ Limitation - Unrealistic assumptions
■ Raw events not being complex
■ A single CEP rule template being able to represent all complex events

5. HTTP Slow Header DoS attack
● Several incomplete connections to keep an HTTP server busy for a long time
@app(name='SlowHeader', version='1.0.0')
def stream PacketStream;
def stream AttackStream;
from PacketStream
filter 'HTTP' == app_protocol and dst_port == 80 and 'rnrn' in
data
and 'Keep-Alive: d+' matches data
partition by dst_ip
window.externalTimeBatch('timestamp', time.millis(1189))
aggregate count() as no_of_packets
filter no_of_packets >= 3
select src_ip, dst_ip, no_of_packets, timestamp
insert into AttackStream;
Wisdom rule to detect HTTP Slow Header DoS Attack

6. Contribution
● An Open Source Stream Processor: “Wisdom1
”
○ Dynamic without compromising the performance
○ Adaptive at the run-time
○ Distributed and functionally auto-scaling
■ Consumes fewer system resources
● A hybrid algorithm to optimize CEP rules
● Tested the application of self-tuning rules in intrusion detection
● Rule: Humans are better at logical reasoning
1. https://slgobinath.github.io/wisdom

7. CEP Rule Optimization Problem
● A CEP rule may have more than one variables
○ Integers
○ Floating points
○ Constants → Can be treated as integers
● Some variables may have bounds some may not
○ E.g: minimum_no_of_packets has a lower bound 0 but not an upper bound
● Variables are highly correlated - Cannot take all values in the space
● A CEP rule may depend on external conditions - E.g: Database
○ Discontinuous black box function

8. CEP Rule Optimization Problem
● Map a CEP rule to a continuous profit/loss function f where x1..xn are
threshold variables
○ Optimizing f should optimize the CEP rule
● Optimizing f is an NP-Hard problem
○ Mixed Integer Linear Programming (MILP) problem if f is linear [9]
○ Mixed Integer Non-Linear Programming (MINLP) if f is non-linear [10]

9. Optimization Algorithm Selection
● Widely used optimization algorithms:
○ Bayesian Optimization Algorithm (BOA) [11]
○ Particle Swarm Optimization (PSO) [12]
○ Genetic Algorithm (GA) [13] - Inefficient for applications with highly correlated parameters [13]
Made up profit function and comparison of BOA with PSO
Stream Processor BOA PSO
Initial points/Swarm size 100 100
Maximum iterations 10 10
Avg. execution time (sec) 255.788 = 4.3 min 0.029 = 29 millis
Avg. optimal points (x, y) 8.742, 409.921 20.681, 199.919
Avg. Profit 0 0.998

10. Wisdom Optimizer
Input: function , constraints , steps
Output: optimal values
optimal values, loss ← PSO(function, constraints)
for all val ∈ optimal values do
val ← Bisection(function, val, constraints[val], step[val])
end for
return optimal values
1: Read pcap files
2: Convert them into Events

11. Wisdom Stream Processor
● Commercial stream processors are tuned for performance
● Dynamic stream processors are used only in research environment
● Wisdom - An adaptive and self-boosting stream processor
Stream Processor Throughput Latency
Apache Flink [14] 6,711,544 events/sec 100 nanoseconds
WSO2 Siddhi [15] 3,811,876 events/sec 216 nanoseconds
Wisdom 2,543,299 events/sec 332 nanoseconds
Esper [16] 2,247,807 events/sec 334 nanoseconds
iCEP[6] Thousands of events in a a few minutes ~ 17 events/sec

12. Wisdom - Functionally Auto-scaling
● Existing stream processors focus on distributing the system [17]
● Not all stateful CEP operators are distributable [17]
● Some CEP rules may require more system resources than other rules
○ E.g: DoS attack detector vs SQL Injection detector
● “Wisdom Manager” controls other Wisdom instances
○ Start/Stop Wisdom applications

13. Test 1 - Self-tuning Rules
● Developed rules:
○ HTTP Slow Header DoS
○ FTP brute force
○ “nmap -sS” Port scan
● Dataset: CICIDS 2017 pcaps containing selected attacks [18]
○ Training data: Randomly selected anomalous traffic within a 10 minutes interval
● Turchin et al. obtained a maximum precision of 80% and a maximum recall of 90%
○ Probability-based CEP rule optimized using Kalman Filter and trained using complete dataset [19]
CEP Rule Avg. Precision Avg. Recall Avg. Training Time
FTP Brute Force 100% 99.61% 47 sec
Slow Header DoS 100% 96.85% 54 sec
Port Scan 99.95% 83.80% 46 sec

14. Test 2 - Auto-scaling Deployment

15. Test 2 - Auto-scaling Deployment
Memory consumption in (a) manual deployment and (b) auto-scaling deployment
(a) (b)

16. Conclusion & Future Work
● Conclusion
○ Wisdom stream processor - Can be used for any adaptive stream processing
■ Intrusion detection
■ Fraud detection
○ Detected selected attacks with a minimum of 99.95% accuracy
● Future Work
○ A hybrid IDS using both anomaly based and signature based detections
○ Use additional sources like application log to write CEP rules
○ Write rules to detect advanced attacks and test them using honeynet

17. Acknowledgement
● We gratefully acknowledge financial supporters
○ Western Engineering
○ Natural Sciences and Engineering Research Council (NSERC), Canada

18. References
1. Jan Seidl. Goldeneye layer 7 (keepalive+nocache) dos test tool, may 2018.
2. Barry Shteiman. Hulk - http unbearable load king packet storm, may 2018.
3. Stefan Axelsson. Intrusion detection systems: A survey and taxonomy. Technical report, Technical report, 2000.
4. Antonio González, Pastana Lobato, Martin Andreoni Lopez, and Otto Carlos Muniz Bandeira Duarte. An accurate threat
detection system through real-time stream processing. 2016.
5. M. Ficco and L. Romano. A generic intrusion detection and diagnoser system based on complex event processing. In 2011
First International Conference on Data Compression, Communications and Processing, pages 275–284, June 2011.
6. Alessandro Margara, Gianpaolo Cugola, and Giordano Tamburrelli. Learning from the past: Automated rule generation for
complex event processing. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems,
DEBS ’14, pages 47–58, New York, NY, USA, 2014. ACM.
7. O-Joun Lee and Jai E. Jung. Sequence clustering-based automated rule generation for adaptive complex event processing.
Future Generation Computer Systems, 66:100 – 109, 2017.
8. N. Mehdiyev, J. Krumeich, D. Werth, and P. Loos. Determination of event patterns for complex event processing using fuzzy
unordered rule induction algorithm with multi-objective evolutionary feature subset selection. In 2016 49th Hawaii
International Conference on System Sciences (HICSS), pages 1719–1728, Jan 2016.

19. References
9. Pierre Bonami, Mustafa Kilinç, and Jeff Linderoth. Algorithms and software for convex mixed integer nonlinear programs. In
Jon Lee and Sven Leyffer, editors, Mixed Integer Nonlinear Programming, pages 1–39, New York, NY, 2012. Springer New
York.
10. Raymond Hemmecke, Matthias Köppe, Jon Lee, and Robert Weismantel. Nonlinear integer programming. 2010.
11. Martin Pelikan, David E. Goldberg, and Erick Cantú-Paz. Boa: The bayesian optimization algorithm. In Proceedings of the 1st
Annual Conference on Genetic and Evolutionary Computation - Volume 1, GECCO’99, pages 525–532, San Francisco, CA, USA,
1999. Morgan Kaufmann Publishers Inc.
12. Zwe-Lee Gaing. A particle swarm optimization approach for optimum design of pid controller in avr system. IEEE
Transactions on Energy Conversion, 19(2):384–391, June 2004.
13. F. H. F. Leung, H. K. Lam, S. H. Ling, and P. K. S. Tam. Tuning of the structure and parameters of a neural network using an
improved genetic algorithm. IEEE Transactions on Neural Networks, 14(1):79–88, Jan 2003.
14. The Apache Software Foundation. Apache flink: Scalable stream and batch data processing, may 2018.
15. Sriskandarajah Suhothayan, Kasun Gajasinghe, Isuru Loku Narangoda, Subash Chaturanga, Srinath Perera, and Vishaka
Nanayakkara. Siddhi: A second look at complex event processing architectures. In Proceedings of the 2011 ACM Workshop
on Gateway Computing Environments, GCE ’11, pages 43–50, New York, NY, USA, 2011. ACM.
16. EsperTech Inc. Esper - espertech, apr 2018.

20. References
17. Sachini Jayasekara, Sameera Kannangara, Tishan Dahanayakage, Isuru Ranawaka, Srinath Perera, and Vishaka
Nanayakkara. Wihidum: Distributed complex event processing. Journal of Parallel and Distributed Computing, 79-80:42 – 51,
2015. Special Issue on Scalable Systems for Big Data Management and Analytics.
18. Iman Sharafaldin, Arash Habibi Lashkari, and Ali Ghorbani. Toward generating a new intrusion detection dataset and
intrusion traffic characterization, 01 2018.
19. Yulia Turchin, Avigdor Gal, and Segev Wasserkrug. Tuning complex event processing rules using the prediction-correction
paradigm. In Proceedings of the Third ACM International Conference on Distributed Event-Based Systems, DEBS ’09, pages
10:1–10:12, New York, NY, USA, 2009. ACM.

21. Q&A
Thank You!

22. Appendix

23. Wisdom - Adaptive Stream Processor
@app(name='SlowHeader', version='1.0.0')
def stream PacketStream;
def stream AttackStream;
@config(trainable=true, minimum=100, maximum=60000, step=-1)
def variable time_threshold = 1500;
@config(trainable=true, minimum=3, maximum=1000, step=1)
def variable count_threshold = 10;
from PacketStream
filter 'HTTP' == app_protocol and dst_port == 80 and 'rnrn' in
data and 'Keep-Alive: d+' matches data
partition by dst_ip
window.externalTimeBatch('timestamp', $time_threshold)
aggregate count() as no_of_packets
filter no_of_packets >= $count_threshold
select src_ip, dst_ip, no_of_packets, timestamp
insert into AttackStream;
Optimizable Wisdom rule to detect HTTP Slow Header DoS Attack

24. @app(name='packet_filter', version='1.0.0', priority=10,
stats='StatisticsStream', stats_freq=time.sec(5), stats_vars=['port'])
@source(type='kafka', bootstrap='localhost:9092', topic='PacketStream')
def stream PacketStream;
@config(stats=true)
@sink(type='kafka', bootstrap='localhost:9092', topic='PossibleDosStream')
def stream PossibleDosStream;
@sink(type='kafka', bootstrap='localhost:9092', topic='_Statistics')
def stream StatisticsStream;
@query(name='FilterDosAttacks')
from PacketStream
filter 'http' == app_protocol and destPort == 80 and 'rnrn' in data
and 'Keep-Alive: d+' in data
insert into PossibleDosStream;
Functionally Auto-scaling Deployment
Filter Query which is running all the time

25. Functionally Auto-scaling Deployment
@app(name='dos_detector', version='1.0.0', priority=5,
requires=['PossibleDosStream'])
@source(type='kafka', bootstrap='localhost:9092',
topic='PossibleDosStream')
def stream PossibleDosStream;
@sink(type='file.text', path='/home/gobinath/dos.txt')
def stream DosAttackStream;
from PossibleDosStream
partition by destIp
window.externalTimeBatch('timestamp', time.millis(1189))
aggregate count() as no_of_packets
filter no_of_packets >= 3
select srcIp, destIp, no_of_packets, timestamp
insert into DosAttackStream;
Dos Attack Detector which will run only if there are events in PossibleDosStream