Automated testing is an essential part of assuring the quality and functional state of your software. One of the biggest challenges in automating your testing can be the simulating user interaction with your UI. For Alexa custom skills, users generally interact via multi-step conversations, which result in a request-response flow to your endpoint. The right set of tools and APIs can help you tackle the complexity of simulating such interactions in an automated fashion. This session demonstrates how to create conversational scripts that can be executed against an Alexa skill endpoint to validate the skill’s output against predefined assertions. You'll leave this talk armed with practical knowledge of how to establish powerful test automation suites for your Alexa skills, helping you ensure that your new skills behave exactly as you expect, and that you don’t accidentally break things while working on new versions. Bring your own laptop and AWS account.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Test Automation for Alexa Skills
K a y L e r c h / / A m a z o n A l e x a
A L X 3 1 5
N o v e m b e r 2 8 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
About
... this workshop. It will:
• Tell you why you should test your skills
• Introduce you to the tools you can use
• Teach you how to actually use these tools
• Close with the benefits of automating tests for Alexa skills
… the speaker.
• Leading the team of solution architects for the Alexa Skills Kit in Germany
• Started at Amazon 26 days before Alexa was launched in Germany and Austria in October 2016
• Loves to code, build Alexa skills, tinker with IoT, and try new things
• Open-sourced numerous tools and Alexa skills for developers
At the end of this workshop, you will have set up and run automated tests that simulate multiturn
conversations with an Alexa skill to validate its functional state. You will leave this room with all the
knowledge and tools needed to start testing your own skills.

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
1. Basics
2. Test execution
3. Test validation
4. Test automation
5. Wrap up
I n t r o

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What you will get…

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Basics
1. Why should you test your Alexa skill?
2. What makes testing Alexa skills different?
3. What tools can I use to test my Alexa skill?

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
B a s i c s
Why should you test your Alexa
skill?

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why should you test your Alexa skills?

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
B a s i c s
What makes testing Alexa skills
different?

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What makes testing Alexa skills different?
Automated speech recognition (ASR)
1. Voice is a non-linear input  give up 100% test coverage—there are 7 billion voices out there

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What makes testing Alexa skills different?
ASR
Natural language understanding (NLU)
2. You build on top of an always-learning cloud service that keeps changing its behavior  how do you
simulate that?

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What makes testing Alexa skills different?
ASR
NLU
Skill Lambda
Request handlers
3. Your skill deals with non-deterministic user interactions  how do you make sure tests cover the most
common situations?

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What makes testing Alexa skills different?
4. There are no barriers for users to interact with your skill  your tests should cover the robustness of your
skill using unexpected user input
User: “Alexa, open my skill”
Alexa: “Welcome to my skill. Please tell me where you live”
User: “Banana”

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
B a s i c s
What tools can I use to test my
Alexa skill?

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alexa-enabled devices for manual tests
ASR
NLU
Skill Lambda
Request handlers
Echo device

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EchoSim.io for testing without a device

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EchoSim.io for testing without a device
ASR
NLU
Skill Lambda
Request handlers
Echo device
Alexa Voice Service (AVS) APIEchoSim.io

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service simulator in the developer console

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service simulator in the developer console
ASR
NLU
Skill Lambda
Request handlers
Echo device
AVS API
Skill Simulation API
EchoSim.io
Service simulator

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASK CLI to access testing APIs via SMAPI
ASR
NLU
Skill Lambda
Request handlers
Echo device
AVS API
ASK Skill Management API
(SMAPI)
Skill Simulation API
Skill Invocation API
EchoSim.io
Alexa Skills Kit Command-
line Interface (ASK CLI)
Service simulator

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Custom extensions and OS tools
ASR
NLU
Skill Lambda
Request handlers
Echo device
Unit testing SDK
AVS API
ASK SMAPI
Skill Simulation API
Skill Invocation API
EchoSim.io
ASK CLI
Service simulator
Custom service simulator
Custom device simulator

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Test execution
1. Skill Management API (SMAPI) and ASK CLI
2. Service simulator
3. Skill Invocation API
4. Skill Simulation API

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T e s t e x e c u t i o n
Skill Management API (SMAPI) and
ASK CLI

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Skill Management API (SMAPI) / ASK CLI
This is a set of API operations that allows you to programmatically manage and test Alexa skills and related
resources, such as interaction models. The Alexa Skills Kit Command-line Interface (ASK CLI) is a command-
line application that lets users create, update, test, and submit Alexa skills for publishing by calling SMAPI
under the hood.
Alexa skill
Skill Lambda function
SMAPIASK CLI
Create, update,
test, submit
Upload,
download,
deploy
Calls

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab 1
https://github.com/alexa/skill-sample-nodejs-test-automation
Goal: Work with SMAPI and ASK CLI to set up a custom Alexa skill
Duration: 15 minutes
At the end of this lab, you will have created a custom Alexa skill
from scratch over the command-line interface

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T e s t e x e c u t i o n
Service simulator

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service simulator
The developer portal provides tools for testing without a device.
The voice simulator lets you enter a phrase in plain text or SSML and hear how Alexa says the text.
The service simulator lets you type in utterances, then see the JSON request sent to your service and the
JSON response returned by your service. You can also play back the response to hear how Alexa says it.
NLU
Skill Lambda function
Skill
Simulation
API
Service
simulator
In: Utterance
Out: Intent
In: JSON request
Out: JSON response
Calls

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab 2
Goal: Work with service simulator to test an Alexa skill
Duration: 10 minutes
At the end of this lab, you will have a quick conversation with the skill by giving it
some utterances. You will have had a look at the request and response payloads
and developed an understanding of how this tool can help manually test a skill
without an Alexa-enabled device.

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T e s t e x e c u t i o n
Skill Invocation API

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Skill Invocation API
…invokes the AWS Lambda or third-party HTTPS endpoint for a specified skill. A successful response will
contain information related to what endpoint was called, as well as the payload sent to and received from
the endpoint.
You can use this API to invoke skills for which you are the developer. The skill being invoked must be
enabled on your account and must be in the development stage. A call to the skill endpoint will time out
after 10 seconds.
Skill Lambda function
Skill
Invocation
API
ASK CLI
In: JSON request
Out: JSON response
In: JSON request
Out: JSON response

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T e s t E x e c u t i o n
Skill Simulation API

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Skill Simulation API
… is an asynchronous API that simulates a skill execution. A successful response will contain a header with
the location of the simulation resource.
You can use this API to simulate those skills for which you are the developer. The skill being simulated must
be enabled on your account and must be in the development stage. This API does not support concurrent
requests per user.
NLU
Skill Lambda function
Skill
Simulation
API
ASK CLI
In: Utterance
Out: Intent
In: JSON request
Out: JSON response
In: Utterance
Out: JSON response

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab 3
Goal: Use Skill Invocation API and Skill Simulation API to test an Alexa skill
Duration: 15 minutes
At the end of this lab, you will have called the Skill Invocation API and Skill
Simulation API via ASK CLI to pass a request payload and utterances to
your skill and get its JSON response as an output

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Test validation
1. Test client
2. Assertions
3. Multiturn conversations
4. Multipath conversations

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T e s t v a l i d a t i o n
Test client

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Test client
A typical conversation with a skill consists of more than one step. It is a choreography of skill invocations.
The Alexa cloud service keeps track of session attributes that are being returned from the skill and passes
them to the next skill request. We need an instance that mimics this sort of session-handling and service-
call orchestration.
Skill
Skill
Invocation
API
Test client
Session
state
Conversation step 1
Conversation step 2
Conversation step 3
Conversation step 1
Conversation step 2
Conversation step 3
Forwards
session
state

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Test client
ASR
NLU
Skill Lambda
Request handlers
Echo device
Unit testing SDK
AVS API
ASK SMAPI
Skill Simulation API
Skill Invocation API
EchoSim.io
ASK CLI
Service simulator
Test client
Custom device simulator

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab 4
Goal: Set up the test SDK and instantiate a test client
Duration: 15 minutes
At the end of this lab, you will have set the stage for future labs
that involve executing multiturn conversations with a skill

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T e s t v a l i d a t i o n
Assertions

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assertions
Assertions are used in software tests to define an expected result of a test execution (such as skill
invocation) that will be compared to the actual result of a test execution. If assertions do not meet
expectations, a test case fails. The test client SDK comes with lots of predefined assertions on skill
response assets, such as the output-speech or a returned card.
Skill
Skill
Invocation
API
Test
client
Conversation step 1
Conversation step 2
Conversation step 3
Conversation step 1
Conversation step 2
Conversation step 3
Assert that
output
speech
contains
“welcome”
Assert that
returned card
contains an
image
Assert that
session ends

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab 5
Goal: Set up a single-turn conversation with the skill and make a few
assertions on the expected response
Duration: 10 minutes
At the end of this lab, you will have run a single-turn conversation with
the test client and made a few assertions on the expected outcome

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T e s t v a l i d a t i o n
Multiturn conversations

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiturn conversations
Skill sessions usually consist of more than one interaction with a user. We call all interactions of a user within
a single session a conversation. To simulate a conversation, the test client needs to define each individual
step and arrange them in a sequential flow (test path). Assertions can be made individually per turn and skill
response.
Skill
Skill
Invocation
API
Test
client
Conversation step 1
Conversation step 2
Conversation step 3
Conversation step 1
Conversation step 2
Conversation step 3
Test path

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab 6
Goal: Set up a multiturn conversation with the skill and make a few
assertions on the expected response
Duration: 10 minutes
At the end of this lab, you will have run a multiturn conversation
with the test client and made a few assertions on each of the
expected results

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T e s t v a l i d a t i o n
Multipath conversations

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multipath conversations
There are lots of ways for a user to go through your skill, and they vary based on one of the following:
Either the skill responds differently in particular situations (date is Monday; user is first-timer; personalized
or randomized content, etc.) or the user making one of many decisions does. The test client can take these
factors into consideration and execute non-deterministic test paths.
Skill
Skill
Invocation
API
Test
client
Conversation step 1
Conversation step 2
Conversation step 3
Conversation step 1
Conversation step 2
Conversation step 3
Test path

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab 7
Goal: Set up a multiturn conversation with the skill and make a few
assertions on the expected response. In addition to the previous lab, add
an alternative path to the test that is entered based on certain conditions.
Duration: 15 minutes
At the end of this lab, you will have run a multiturn conversation with the
test client and made a few assertions on each of the expected results. By
creating an alternative test path, you will have simulated a user decision
that is made based on a dynamic skill response.

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T e s t a u t o m a t i o n
Run conversation scripts periodically

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lab 8
Goal: Schedule execution of the test client to run regression tests
Duration: 15 minutes
At the end of this lab, you will have an automated regression test that periodically simulates a conversation
with an Alexa skill and sends out an alarm in case your assertions do not meet the skill’s actual response

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
W r a p u p
What are the benefits?

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Design compliance
Conversation scripts are the technical specification that cover all features a skill provides. They accompany
the skill along its lifecycle. The scripts ensure compliance with expected and actual functionality of an Alexa
skill so it “works as designed” at any time. Conversation scripts are artifacts that derive from VUI-diagrams.
Given an adequate conversation path coverage, a script takes care of the skill implementation to stay
compliant with the underlying VUI design. It also paves the way for test-driven development—a best-
practice of building high-quality software.
Scope Design Code Test
Use
cases
Deploy
VUI
diagram
Conver-
sation
scripts
Implem-
entation
Runtime
CI/CD
pipeline
(Local)
test
client
Live-test
client
Deploys
Periodic
executions
Triggered
executions
Goes into
Monitor
Metrics,
alarms
Notifie
s
Executes
Continuous improvement

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Story-Boards
User Story 1:
Description
Alexa says: “….”
User says: “....”
Alexa says: “....”
User says: “....”
Alexa says: “....”
User says: “....”
Alexa says: “....”
VUI-DiagramsVUI-DiagramsVUI-Diagrams
Test-driven development
Scope Design CodeSpecify Test and run
Storyboards
User story 1:
Description
Alexa says: “…”
User says: “...”
Alexa says: “...”
User says: “...”
Alexa says: “...”
User says: “...”
Alexa says: “...”
VUI diagrams Test scripts Skill code
Request
handler
Request
handler
Request
handler
Request
handler
Request
handler
Test results

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why should you test your Alexa skills?

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!