EpiSys Science develops cognitive technologies and systems using surprise-based learning (SBL). SBL monitors deep learning models at runtime to detect out-of-distribution data and provide early warnings. This allows models to identify new situations and request operator assistance to avoid failures. EpiSys has applied SBL to tasks like object recognition and autonomous vehicle steering to catch anomalies. SBL is scalable and can work with any AI/ML model to provide explanations and ensure safe interactions between humans and autonomous systems.

1. Developing Next Generation Cognitive Technologies
and Systems for C4ISR Applications
Bo Ryu – President, EpiSys Science
March 21, 2018

2. Who We Are
• Founded in 2012
• Received over $7M
R&D funding to date
from DoD and NASA
 SBIR: 80%
 Non-SBIR: 20%
• Strong IP Portfolio
 2 patents issued
(US 9,537,954)
(US 9,894,158)
 2 patents pending
“Intelligent Autonomous Systems”

3. SwarmSense 데모 비디오
https://youtu.be/X8Mx_tCmGio
https://youtu.be/5H6kv9ukrL0

4. Founding Members
4 | March 2, 2017
Bo Ryu, Founder &
President
• PhD in Electrical
Engineering from
Columbia University
• Track record of wining
over $30M in research
funding as PI and PM
• Expertise in Self-
Organizing Wireless
Networking Systems
Tamal Bose, Co-Founder
& Vice President
• PhD in Electrical
Engineering Southern
Illinois University
• Department head of
electrical and computer
engineering at the
University of Arizona
• Expertise in Cognitive
Radio Systems
Wei-Min Shen, Co-
Founder & Vice President
• PhD in Computer Science
from Carnegie-Mellon
University
• Director of Polymorphic
Robotics Lab in USC/ISI
• Co-inventor of Surprise
Based Learning and
SwarmSense
• Expertise in AI Systems
and Robotics
Nadeesha Ranasinghe,
Co-Founder & Director
• PhD in Computer Science
from University of Southern
California
• PhD thesis on Surprise
Based Learning
• Co-inventor of Surprise
Based Learning and
SwarmSense
• Expertise in AI Systems and
Robotics
Founders with combined experiences of 100+ years in high-risk, high-payoff R&D from DoD and NASA

5. surprise
noun sur·prise sə(r)-ˈprīz
“An unexpected or astonishing event”

6. surprise
noun sur·prise sə(r)-ˈprīz
Input data which will likely lead to
prediction or action errors because of
unexpected features

7. surprise
noun sur·prise sə(r)-ˈprīz
Includes more than just anomalies
i.e. really bad surprises

8. surprise
noun sur·prise sə(r)-ˈprīz
Norm
Range and
Strength of Surprise
Anomaly
Rate of Surprise
(Occurrence)
time
Includes more than just anomalies
i.e. really bad surprises

9. Surprise Based Learning
SBL imitates how a child
learns by learning a model
(abstract relationship
between inputs and
outputs) which forecasts
the expected observation
of executing an action,
and improves this model
when contradictions
(surprises) occur.

10. SBL: How It Came About
 Learns a model which forecasts the expected observation of
executing an action, and improves this model when contradictions
(surprises) occur
 Superior to Reinforcement Learning
• Learns the structure (relationships) in data, not just a policy
o Discovers the number of states with predictable capability
o Does not over-fits the data in structured environments
o Produces a good model even with small training data sets
o Learns generic models that can solve specific goals
• Runtime detection and adaptation to unforeseen changes
o Adapts to individual/simultaneous changes in actions, sensors, environment & goals
o Repairs model faster than rebuilding it from scratch
o Reasons with noise and gaps in data
Herbert Simon
(Nobel Laureate and one of
Founding Fathers of A.I.)
Wei-Min Shen
(Inventor of SBL and
Co-Founder of EpiSci)
Nadeesha Ranasinghe
(Co-inventor of SBL and
Co-Founder of EpiSci)
PhD Adviser PhD Adviser

11. SBL Background
 SL was designed for Learning to Detect and Adapt to Uninformed
Changes
• Permanent changes to an autonomous agent’s norm
o Addition, Deletion, Definition-Change (Rotation, Translation, Scale) in Sensors,
Actions (Making a Decision), Goals and the Environment
• Short term changes including noise & missing data or gaps
 SL learns a predictive model which forecasts the expected
observation of executing an action, and improves this model when
contradictions (surprises) occur
• The prediction model is comprised of complementary prediction rules
o Rule = Conditions → Action+ → Predictions
• A surprise occurs when Observation ≠ Prediction
o http://www-scf.usc.edu/~nadeeshr/dissertationNOR.pdf
Rotated cameraGoal change Gap

12. Deep Learning vs Surprise Learning
Deep Learning (DL) Surprise Learning (SL)
• Excellent in reduction of dimensionality
• Unable to adapt to “surprises” (new data not
learned) without complete/partial
relearning
• Unable to support run-time self-monitoring
and adaptation for new inputs and outputs
(dynamic reconfiguration)
• Excellent at runtime adaptation
• Life-long learning does not require extensive offline
training
• Well suited for temporal and logical reasoning
• Not a black box: human readable learned models
are easy to debug
• Does not scale well with high-dimensional data sets
as the convergence time increases exponentially
Prediction
Action
Observation
Detection of
surprise
Analysis of
surprise
Adaptation
No
Yes

13. DL Recap: Pros and Cons
 Strengths
• Works well with large datasets and raw sensor data
• Supports supervised and unsupervised learning of features
• Facilitates fusion of neural network and other machine learning techniques
• High accuracy in image, text, audio and video recognition tasks
 Weaknesses
• Requires lot of data for high accuracy
• Training is computationally very expensive
• Hard to determine ideal architecture, parameters and tuning
• Cannot adapt to unforeseen changes in sensors, actions, environment and
goals
• Exhibits a new sort of instability caused by virtually indistinguishable adversarial
examples
• Lacks methods to build sophisticated background knowledge or reasoning
capabilities
AddresswithSBL

14. SBL: Complements DL!
 Strengths
• Does not require a lot of training data to build good predictive models
• Discretizes raw data and learns its structure without human supervision
• Human readable models support debugging
• Detect surprises, identifies possible causes, supports a human-in-the-loop
• Adapts to unforeseen changes in sensors, actions, environment and goals
 Weaknesses
• Does not scale well with the number of sensors due to exponential growth
in surprise analysis
• Does not explicitly tolerate interference during model learning
• Assumes that all states are observable provided sufficient sensor
redundancy
• Planning with prediction models adds overhead
AddresswithDL

15. Adapting to Changes in the Norm
 Adaptation requires learning and forgetting
• Forget obsolete data and phantom features to scale-up
 Rule Forgetting
• Mark inappropriate rules as “rejected”
o Not used in future predictions or maintained further
o Kept in memory as a “reminder” to avoid recreation
• Reject contradictory prediction rules
o Immediately after rule splitting if neither of the complementary rules
predicted the surprised result
o Rule was created with a wrong c0 or split with an incorrect prediction
px
• Reject rules with consecutive and consistent surprises
o Probability of a rule’s success = times successful / times predicted
o Select a cutoff probability, e.g. p(R|O) < 0.5
SBL++

16. SBL Adapting to New Norm

17. Learning to Identify Anomalies
A Prediction Model describes how
data from relevant sensors should
change when a behavior executes
under normal circumstances
SBL observes raw and
processed sensor data to
learn a prediction model
A Structured Model is formed by grouping
prediction rules that fire together into states
and recording their transition probabilities
Surprises are not necessarily changes in the norm
Anomalies = Invalid States and Transitions are indicative of changes in the norm

18. Anomaly Detection on a UAV Swarm

19.  An action may be represented by multiple models
• E.g. verbs executed by people in a video stream such as approach,
arrive, bounce, carry, catch, chase, collide, drop, exchange, exit, flee,
go, haul, leave, lift, move, pass, pick up, push, put down, raise, replace,
throw, walk
 An action has occurred, if a positive model is true and all false
positive models are false
 Adapting the norm can be achieved by introducing new positive or
false positive models at runtime
Multi-Model Detectors
Positive
Examples
Negative
Examples
Positive
Models
False Positive
Models
False Positive
Examples
Action
-
+Training
Videos
share the same
variable bindings
variable
bindings can
be different

20. Action Recognition Example
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500
approach
carrry
catch
collide
haul
leave
move
pass
pickup
push
run
stop
throw
walk
SBL
Ground truth
inaccurate model
tracking errors
sample rate affects accuracy
track gapped, SBL filled
Contributions++++

21. Results on DARPA Mind’s Eye Videos
 Visint.org Year 1 Dataset, USC tracks, 1200 videos
 F1 = (2*precision*recall)/(precision+recall)
 Outperformed expert hand-coded Structured Models by more
than 10%
 Reasoning with detector accuracies below 20%, much higher
scores can be achieved with better object detectors feeding SBL
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Approach Carry Catch Collide Haul Move Pickup Push Run Stop Throw Walk
F1-score
Overlap exact AMT Novel exact AMT Overlap majority AMT Novel majority AMT
Contributions++++

22. SafeguardAI
22
Episys Science, Inc.
Protect your AI solutions from unintended behavior
Bo Ryu, Ph.D.
President & Founder

23. LabelPrediction
33
with trained MNIST model
In-distribution test case Out-of-Distribution test case
SafeguardAI
Correct
LabelPrediction
97Wrong
LabelPrediction
a7
LabelPrediction
-9

24. LabelPrediction
33
with trained MNIST model
In-distribution test case Out-of-Distribution test case
SafeguardAI
Surprise Indicator: 90% > 50%
Surprise Indicator: 77% > 50%
Surprise Indicator: 4% < 50%
Surprise Indicator: 21% < 50%
Correct
LabelPrediction
97Wrong
LabelPrediction
a7
LabelPrediction
-9

25. • SafeguardAI focuses on catching out-of-distribution data by
observing intermediate neurons and prediction result.
• SafeguardAI learns to distinguish out-of-distribution data
from test phase of existing deep learning application.
• Deployed alongside of DL models, it provides additional
information indicating likelihood that the model has
encountered an abnormal input.
SafeguardAIFeatures

26. How it works
Trained Neural Network
Training Deploying
Ranasinghe, N., & Shen, W.-M. (2008). Surprised-based learning for developmental
robotics. Proceedings of the ECSIS Symposium on Learning and Adaptive Behaviors for
Robotic Systems (pp. 65- 70). Edinburgh, Scotland: IEEE Press.
“Surprise Based Learning”
Trained Neural Network
Out-of-Distribution Information
Correct
Predictions
“3” “7”
80%

27. Statistics
MNIST
SurpriseAI testset
Out-of-Distribution
caught / Total (500)
Out-of-Distribution
caught / Total (1000)
notMNIST 470 / 500 (94%) 936 / 1000 (93.6%)
MNIST 63 / 500 (87.4%) 119 / 1000 (88.1%)
• Sample used in training SafeguardAI : 2000 (200 * 10 classes)
• Binary decision (out-of-distribution, in-distribution) by set threshold
as 50%

28. Statistics
Belgian Traffic Sign Dataset (1/2)
• Dataset: Belgian Traffic Sign Dataset
(http://btsd.ethz.ch/shareddata/)
• Number of classes: 62
• Image size: various shape
=> standardize by (32x32x3)
• Number of dataset: 7,000
(110 images per class)

29. Statistics
Belgian Traffic Sign Dataset (2/2)
Class Boundary
(In-distr / Out-distr)
Precision
(ΣTP / ΣPCP)
Recall
(ΣTP / ΣCP)
F1 Score
2/((1/Pre)+(1/Rec))
class (0-9) / (10-61) 0.941 0.899 0.919
class (0-19) / (20-61) 0.929 0.891 0.909
class (0-29) / (30-61) 0.952 0.866 0.906
Control group:
70 training images for each in-distribution class,
70 test images for each class
By set out-of-distribution threshold as 50%:

30. DL-based Self-Driving Problem
Difficult to anticipate and train DL models for every situation
Cannot prove that the model will work for future
unexpected data or cases despite careful training
Time consuming to create “good” training/testing datasets
for near-zero-error models with sufficient generalization
High operational costs due to lack of autonomous self-
monitoring capabilities at runtime

31. Self Driving Car Application
SafeguardAI knows when to disengage the AI and request operator
assistance or force autonomous re-training depending on the strength
of surprise (SOS) and rate of surprise (ROS)

32. Car steering angle prediction
Problem specification
• Dataset: 50,000 (wheel angle, image) pairs
• Model: NVIDIA self-driving network architecture
(66x200x3 input, 5 Conv layers, 5 FC layers)
• Avg. prediction angle error ≈ 20 (from -270 to 270)
• Train SafeguardAI with images that has error < 30
as In-distribution
• Test SafeguardAI to see if it catches anomaly behaviors

33. Car steering angle prediction
Demo video

34. SafeguardAI Vision
To become every AI model’s companion:
• Complementary to customer’s AI model development
• Easing the burden of building near-zero-error models
• “Explainable” on how much your data differs from the model
• Model agnostic
To offer intuitive, natural Human-AI Interaction:
• Humans most effective when AI is not sure of what to do
• Giving humans sufficient time (via early warning) and information
(via SOS and ROS) on whether to engage or not

35. We Claim that…
DL models are bound to fail in new, ambiguous, and complex environments that
are different from its training environment
SBL/SafeguardAI can identify WHAT and WHEN it is observing something new and
different from what it’s seen before, and alert the human supervisor / developer
in runtime or during model engineering
SBL/SafeguardAI can quantitatively explain how big/small the surprise is with
respect to DL models
Scalable implementation: once we build the code for one system (e.g. camera,
LIDAR), it is easily transferrable to the same system of another company
Applicable to virtually any AI/ML model type  Tremendous Market Opportunity!

36. Thank you
Contact: boryu@episyscience.com