IoT Hessen Meetup January 2017, Mainz: Talk by Moritz Kammerer (@phxql, Senior Software Engineer at QAware). Abstract: The Amazon Echo is a digital assistant developed by Amazon. Her functionality can be extended with so-called skills. This hands-on talk shows with an example in the programming language Java how a natural user interface to a warehouse management system can be created using the Amazon Echo. We talk about the Alexa Skill Kit, which enables the development of such skill. Besides the programming we also talk about the system architecture. Speaker: Moritz Kammerer (moritz.kammerer@qaware.de) works as a software engineer at QAware GmbH. He already successfully carried out projects with the Amazon Echo and supervised a bachelor thesis on natural user interfaces. Besides that he is author of open-source software.

1. Developing skills for Amazon Echo
Mainz, 26.01.2017
Moritz Kammerer

2. About the author
2
● 2016 Supervised bachelor thesis about NUIs
● 2016 Developed customer project with Amazon Echo
● GitHub: @phxql / Twitter: @phxql
● Blog: https://www.mkammerer.de/blog
● Sourcecode: https://github.com/qaware/iot-hessen-amazon-echo

3. What is a Amazon Echo?
3
● Digital assistent from Amazon
● Provides an audio interface
● https://www.youtube.com/watch?v=KkOCeAtKHIc

4. What’s the cool thing?
4
● Alexa has built-in features: weather, facts, news, music, …
● And it is also extensible via so called skills
● In this talk: We develop a skill for warehouse management

5. Demo
5
● Utterances:
○ “Wie viele Schrauben haben wir noch?”
○ “Bestelle mir neue Schrauben”
○ “In welchem Regal befinden sich die Schrauben?”

6. How does that work?
6
● Register the skill in the Amazon Skill Store
● Develop an application with a HTTP interface
● Enable the skill on your Echo

7. Okay, but how does that really work?
7
Echo
Amazon Cloud Your service
Natural Language
HTTPS / JSON

8. Develop a skill, step by step. Step 1.
8
● Create a new skill: Amazon Developer Console / Alexa / Alexa Skill Kit [7]

9. Skill types
9
● SmartHome Skill API: Switch on/off lamps, control heating
● Flash Briefing Skill API: News as audio or text feed
● Custom Interaction Model: Define own utterances, most flexible
○ That’s what we will use

10. Develop a skill, step by step. Step 2.
10

11. What are intents?
11
● “an intent represents an action that fulfills a user’s spoken request”
● Intent schema is a JSON formatted list of intents
{
"intent": "QueryInventory",
"slots": [
{
"name": "ware",
"type": "LIST_OF_WARES"
}
]
}
Slots (parameter) of the intent
Slot type

12. Intents of our skill
12
● QueryInventory (ware): Determine how many of a ware is in the warehouse
● OrderWare (ware): Orders a new ware
● LocateWare (ware): Find a ware in the warehouse
● Quit (): Aborts the conversation

13. Intent slot types
13
Attention: Alexa tries to match the spoken word to this list, but other values can still be sent to the skill!
Hint: There are predefined slots, e.g. for dates, durations, time, etc.: [1]

14. Utterances - Combine intents with audio commands
14
Intent name
Command from the user
Slot name
Hint: Best practices and a handbook for the utterances design: [2], [3]

15. Skill configuration
15
HTTPS endpoint
of our skill
You can also use AWS Lambda for your service
OAuth2 link to an
non-Amazon account

16. SSL configuration
16

17. Skill configured, time for some code!
17
● We create a Spring Boot application for our service
● Open Spring Initializr [6], dependencies: only web
● Add the Alexa Skills Kit for Java [4] to the Maven POM:
<dependency>
<groupId>com.amazon.alexa</groupId>
<artifactId>alexa-skills-kit</artifactId>
<version>1.2</version>
</dependency>

18. Implement the Speechlet
18
com.amazon.speech.speechlet.SpeechletV2
● void onSessionStarted(...)
○ Gets called when a session is started
● SpeechletResponse onLaunch(...)
○ Gets called when the user starts a conversation
● SpeechletResponse onIntent(...)
○ Gets called when the user invokes an intent
● void onSessionEnded(...)
○ Gets called when the session is ended

19. Our speechlet implementation
19
● onSessionStarted: not needed
● onLaunch:
○ When called, user wants a conversation. Set a session flag:
requestEnvelope.getSession().setAttribute("conversation", "true");
○ When not called, user wants a one-shot intent
● onSessionEnded: not needed

20. The skill logic resides in onIntent
20
● onIntent: read the intent name and handle the intent
Intent intent = requestEnvelope.getRequest().getIntent();
switch (intent.getName()) {
case "QueryInventory":
return handleQueryInventory(requestEnvelope);
…
}

21. QueryInventory intent handling
21
● Read slot “ware”:
Slot wareSlot = intent.getSlot("ware");
String ware = wareSlot == null ? null : wareSlot.getValue();

22. QueryInventory intent handling
22
● If ware is missing from the intent (ware == null), tell the user.
● If in conversation mode, let the user retry:
○ return SpeechletResponse.newAskResponse(new PlainTextOutputSpeech("Ich
habe die Ware nicht verstanden. Was möchten Sie tun?"),
new Reprompt(...));
● If not in conversation mode (one-shot intent)
○ return SpeechletResponse.newTellResponse(new PlainTextOutputSpeech("Ich
habe die Ware nicht verstanden."));
SSML [5] is also supported
If the user doesn’t answer the quesion, this
text is spoken.

23. QueryInventory intent handling
23
● Now find the ware amount:
int amount = warehouseService.getAmount(ware.get());
● And create a response to the user:
return SpeechletResponse.newTellResponse(new PlainTextOutputSpeech(
String.format("Es sind noch %d %s im Lager.", amount, ware)
));

24. One-shot flow
24

25. Conversation flow
25

26. Wire the speechlet into Spring Boot
26
@Bean
public ServletRegistrationBean alexaServlet(WarehouseSpeechlet speechlet) {
SpeechletServlet speechServlet = new SpeechletServlet();
speechServlet.setSpeechlet(speechlet);
ServletRegistrationBean servlet = new ServletRegistrationBean(speechServlet, "/alexa");
servlet.setName("alexa");
return servlet;
}
Speechlet will answer on /alexa

27. Set speechlet properties
27
// Disable signature checks for development
System.setProperty(Sdk.DISABLE_REQUEST_SIGNATURE_CHECK_SYSTEM_PROPERTY, "true");
// Allow all application ids for development
System.setProperty(Sdk.SUPPORTED_APPLICATION_IDS_SYSTEM_PROPERTY, "");
// Disable timestamp verification for development
System.setProperty(Sdk.TIMESTAMP_TOLERANCE_SYSTEM_PROPERTY, "");
For production you’ll find this ID in the “Skill Information” section

28. Skill implemented, how to test it?
28
This request gets
sent to the skill
service
Enter text here
Hint: Copy the JSON and POST it with your preferred tool (curl, Postman, etc.) to /alexa

29. And now with voice...
29
● Amazon forces you to use SSL
● So... configure Spring Boot to use SSL
○ Create a new keystore
○ Create a new keypair, the CN must be set to your servers DNS name
○ Enable SSL in Spring Boot:
server.port: 443
server.ssl.key-store: classpath:keystore.jks
server.ssl.key-store-password: ""
server.ssl.key-store-type: jks
server.ssl.key-alias: ec2-35-157-19-115.eu-central-1.compute.amazonaws.com

30. And now with voice...
30
● Export the SSL certificate as X.509 (starts with -----BEGIN CERTIFICATE-----)
● Paste the X.509 certificate in the skill configuration under “SSL Certificate”
● Build the Spring boot application
● Upload the application to a publicly accessible server (e.g. EC2)
● Start the application
● Add your Alexa to the account which created the skill
● Enable the skill in the Alexa App
● Now you can invoke the skill

31. What I wish I knew
31
● TLS: Must be port 443, the CN in the certificate must match the DNS
● Slot types are not enums, but only recommendations for the speech recognition
● Slots can be null! (e.g. “Bestelle mir neue “)
● The user id is changing when the user removes and re-adds the skill
● Alexa Skill Kit for Java: Exclude log4j and slf4j-log4j12
● When testing the skill, use the Alexa Mobile App: The cards contain useful debug information
● Implement local and test with voice: Use SSH remote port forwarding:
ssh -R 443:localhost:8443 root@server

32. What have we learned?
32
● Develop an interaction model:
○ Intents, slots and slot types
○ Utterances
● To use the Alexa Skills Kit and implement the SpeechletV2 interface
○ When onSessionStarted(), onLaunch(), onIntent() and onSessionEnded() are called
○ What the difference between newAskResponse() and newTellResponse() is
● How to beat TLS

33. Conclusion
33
● Pro:
○ Echo can be extended with custom skills
○ Skills are very flexible
○ The Alexa Skills Kit abstracts all the request and response handling
○ Amazon handles the speech-to-text and text-to-speech
● Contra:
○ No semantic analysis, just matching the utterances exactly
○ Have to invoke the skill by saying, “Alexa, tell [skill] …”
○ TLS is very annoying (and badly documented) when developing a skill

34. 34
https://github.com/qaware/iot-hessen-amazon-echo
Source Code:

35. References
35
● [1] https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/built-in-intent-ref/slot-type-reference
● [2] https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/alexa-skills-kit-voice-design-best-practices
● [3] https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/alexa-skills-kit-voice-design-handbook
● [4] https://github.com/amzn/alexa-skills-kit-java
● [5] https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/speech-synthesis-markup-language-ssml-reference
● [6] http://start.spring.io/
● [7] https://developer.amazon.com/edw/home.html#/skills/list

36. Appendix
36

37. Timings
37