Can we use AI/ML to reliably detect cybersecurity incidents in safety critical systems

1. Can we use AI/ML to reliably detect
cybersecurity incidents in safety
critical systems?
Moojan Pordelkhaki
Vitor Jesus
Afshin Hariry
Shereen Fouad
IET SSCS Conference 2018

2. Who We Are
Moojan Pordelkhaki : Cyber Security MSc, Researcher at Centre of CyberSecurity,
School of Computing and Digital Technology, Birmingham City University
Dr Vitor Jesus : Senior Lecturer, Centre of CyberSecurity, School of Computing and
Digital Technology, Birmingham City University
Afshin Hariry : Electronic Engineer, Industrial Control system specialist
Dr Shereen Fouad : Lecturer in Computer Science, School of Computing and Digital
Technology, Birmingham City University

3. What is AI/ML
Science of Pattern Discovery
Making Prediction Data
Create a System Learn from
Experience

4. AI/ML Techniques
Unsupervised
Supervised
Semi-Supervised
Reinforcement Learning

5. Can We Use AI/ML in Cybersecurity for Real-Time Safety Systems?!

6. Previous Works
Hongbiao Li and Sujuan Qin Simulat Siemens SIMANTIC S7-200
 2 Simulated Modbus Client on Separate Virtual
Machines
 DOS Attack, AR Attack and UA Attack
Simulated
Malicious Traffic Identification
Identification of Attacks

7. Previous Works
Imtiaz Ullah and Qusay H.Mahmoud
 Study was developed at Mississippi State
University using the gas pipeline system as a
testbed.
 J48 Classifier Trained and used
Attack Classification
Binary Classification Result
Multi Class Classification Result

8. Previous Works
Wei Gao and His Team Mississippi State University SCADA Security
Laboratory
 MITM Response Injection Attack
 DOS Based Response Injection Attack
MITM Response Injection
DOS Attack

9. Previous Works
Ken Yau and His Team Simulate Siemens S7-1212C PLC
 Traffic light control program
 Monitored the PLC memory addresses over
the network and recorded the values along
with their timestamps(libnodave)
 Create anomalous PLC operations(Snap7)

10. AI/ML in Critical System Anomaly Detection
Many academic research efforts has been done on
SUCCESSFUL APPLICATION of AI/ML in Anomaly
Detection (IT & OT)
Unfortunately the success of such systems in
operational environment has been VERY LIMITED.
Why ?!

11. Anomaly Detection = Classification Task
AI/ML is good at finding similarities (New Attack ??? )
Define normal samples and assume the rest are benign
Accurate Model for Normal Operation ? Necessarily
lacking
context !
Datasets should include large data of all classes
Challenges of AI/ML Anomaly Detection

12. Not adaptive to different sites
Diversity of Process/ Critical System Application
 FP should be analysed (normal or not)
 FN cause serious damages
Errors
Challenges of AI/ML Anomaly Detection

13. The task of finding attacks is fundamentally different from other applications of AI/ML, making it
significantly harder for the intrusion detection – Sommer, Paxon, “Outside the Closed World: On
Using Machine Learning For Network Intrusion Detection”, IEEE S&P 2010
Challenges of AI/ML Anomaly Detection
In other words, AI/ML:
 Is good at classification not finding outliers
 It basically reports what was seen before: needs abundance of both “normal” (we have)
and “anomalous” (we do not have, by nature of the problem)
 An early error, such as false-positive, at training stage, dearly propagates
 Is good with homogeneneity, not diversity  this could work for ICS/Safety
 it is overly dependent on the training data  arguably, given the rarity of cyberattacks, one can
never capture it because we can only train the ML with known ones when we want the unknown

14. Can We Use AI/ML in Cybersecurity for Real-Time Safety Systems?!
NOT
IN
PRACTICE

15. Research Methodology
Simulate a Simple Control Loop in Real
Condition
https://Automationforum.co/basics-of-pressure-transmitter
Simulate an Attack Command to the Control
Valve
Preparing Datasets (Training, Test)
Train and Test a Supervised Classification
Learner
Train and Test a Semi-supervised
Classification Learner

16. Supervised: Normal Condition (Theoretically)
Normal Condition Added Attack

17. Disturbance and Noise
Normal Operation with Small
Amount of Disturbance
Compromised System with Small
Amount of Disturbance

18. Normal Operation with Noise
Compromised System with Noise
Disturbance and Noise

19. Normal Operation with Periodic Noise
Compromised System with Periodic Noise
Disturbance and Noise

20. Supervised: Results

21. Supervised: New Attack
New Attack on the Control Command
Would Be Predicted ?!

22. Semi-Supervised: Results

23. Summary
Application of AI/ML for detecting cybersecurity incidence
in safety critical systems requires further studies
 Supervised methods  More accurate in detecting
known attacks  Not reliable for detecting new
attacks
 Semi-Supervised methods  More practical for
detecting attacks  Anomaly Detection  Yet low
performance  High Cost Errors  Accurate model
for normal condition is required

24. Summary
 Focusing on network traffic data  Network anomaly
detection  Not a reliable approach
 Process modelling  Semi-Supervised methods 
Detecting cyber physical anomality More practical 
Lack of practical study
AI/ML anomaly detection application for detecting
cybersecurity incidence in safety critical systems requires
further considerations:

25. Thank you!
questions?