1. BOTIUM
Building an End-to-End Test Automation Pipeline for Conversational AI
February 2021

2. Botium.
THE
PLAYBOOK
1
2
3
4 ADD TESTS TO CI/CD PIPELINE
WRITE TESTS <no></code>{codeless}
DISCUSS THE VALUE OF TESTING CHATBOTS
DEFINE A SOLID BOT TESTING STRATEGY

3. Plan Design Develop Train Test Deploy
PLAN DEVELOP TRAIN TEST DEPLOY
DESIGN
analyzing
user needs
channel
decision
NLP
selection
deliver
domain data
provider
benchmark
tests
conversation
flow design
conversation
model import
developing
functionality
crawler for unit
testing
speech
support
natural
language
processing
NLP score
testing
AI model
import
machine
learning
tuning
conversation
flow testing
end-to-end
testing
performance
testing
security testing
voice
based
testing
GDPR
testing
functional and non-functional regression testing
full end-to-end acceptance testing
continuous integration
quick adoption
short TTM
production
monitoring
chatbot
analytics
Phases High
Importance Low
Bot
Development
Lifecycle
Testing
value
created
DevOps
Botium.
VALUE OF TESTING

4. Botium.
SOLID TESTING STRATEGY
Functional
C
o
n
v
e
r
s
a
t
i
o
n
F
l
o
w
T
e
s
t
i
n
g
I
d
e
n
t
i
f
y
fl
a
w
s
i
n
y
o
u
r
c
h
a
t
b
o
t
d
i
a
l
o
g
s
P
e
r
f
o
r
m
a
n
c
e
&
S
e
c
u
r
i
t
y
T
e
s
t
i
n
g
K
e
e
p
y
o
u
r
b
o
t
s
e
c
u
r
e
a
n
d
r
e
s
p
o
n
s
i
v
e
u
n
d
e
r
h
i
g
h
l
o
a
d
M
o
n
i
t
o
r
i
n
g
G
e
t
n
o
t
i
fi
e
d
w
h
e
n
p
r
o
b
l
e
m
s
a
r
i
s
e
Functional
Smoke,
Regression
Regression
Regression,
Acceptance
Non-Functional
Non-Functional
Measure
E
n
d
-
t
o
-
E
n
d
T
e
s
t
i
n
g
V
e
r
i
f
y
t
h
e
e
n
d
-
u
s
e
r
e
x
p
e
r
i
e
n
c
e
o
f
y
o
u
r
c
h
a
t
b
o
t
Integrate
Automate
DevOps
N
L
P
S
c
o
r
e
T
e
s
t
i
n
g
I
m
p
r
o
v
e
y
o
u
r
c
h
a
t
b
o
t
u
n
d
e
r
s
t
a
n
d
i
n
g