My EuroSTAR 2018 tutorial (13 November 2018 9:00 - 13:00 hrs) was about testing of intelligent machines. I showed information about quality characteristics for artificial intelligence and robotics. Also I introduced the six angles of quality for intelligent machines. The delegates got exercises to test a chatbot (various examples). After the break they had exercises to expoloratory tests robots, they had a choice of Lego Mindstorm Robots, a robot vacuum cleaner, an image recognition machine and more. My book "Testing in the digital age; AI makes the difference" was an important source and also a prize in the quizzes.

1. Testing
Intelligent Machines
Rik Marselis
EuroSTAR Tutorial K
@rikmarselis 13 September 2018
Management consultant digital assurance & testing
Sogeti Nederland

2. During this tutorial we’ll be using Kahoot
You can already download the app
Or use it in a browser
WiFi network:
<<network>>
WiFi password:
<<password>>

3. Testing in the digital age
AI makes the difference
This tutorial is based on our new book:
You can win a copy in the quizzes !!

4. 4© 2018 Sogeti. All rights reserved.
Rik Marselis
Management consultant Digital Assurance & Testing
2007 2012 20122008
Workgroup-
member
20142009
20181980

5. Agenda of this tutorial
9:00 Welcome
9:10 Setting the scene(s) – Quiz & discussion
9:35 Quality characteristics and chatbot testing
10:30 ~ Coffee-break
11:00 Quality angles and robot testing
12:20 Fun exercise how an AI-chatbot learns
12:35 Final Quiz
12:50 Closing

6. Setting the scene(s)

7. Who of you has a robot?
How do you know if it is good enough quality?

8. Testing WITH intelligent machines

9. So that was an example of testing WITH
intelligent machines.
Today’s workshop is about
testing OF intelligent machines

10. Who of you already uses AI?

11. What do you already know about
AI, ML, Robotics, testing, etc.???
KAHOOT quiz
Open the app on your mobile device
OR
Go to https://kahoot.it
Enter game pin: <<game pin>>
Numbers 1, 2 and 3 win a copy of my book!!

12. Why is testing of intelligent machines different?
Group-discussion
Groups of 4 – 6 people
Write keywords on a flipchart
7 minutes 7 6 5 4 3 2 1 Finished
Question

13. Debriefing / feedback
• Tell the top-keyword of your group
(if it was already mentioned take the next of your list of keywords)
• I will list them on a flipchart
Why is testing of intelligent machines different?

14. Quality characteristics &
chatbot testing

15. How can we measure quality?
What quality characteristics do you know?

16. ISO25010 Quality in use

17. ISO25010 Product Quality

18. Quality characteristics extended
for intelligent machines

19. The new quality
characteristics for
intelligent machines
handout

20. Who of you has a chatbot operational
in your business process?
Who is thinking about adding a chatbot to your
business process?

21. Example chatbot conversations
https://www.facebook.com/messages/t/KLMhttps://www.facebook.com/messages/t/kayak

22. Human – chatbot communication:
utterance intent
action
response

23. Exercise:
Test the chatbot
(you’ll get an exploratory testing charter)

24. Exploratory Testing has many
flavours, this is our flavour:
• Focus on confidence (risk-based)
• Structured (charter, log, debriefing)
• Session-based & Timeboxed (not too short, not too long)
• Tandem-approach (two people, mob-approach also possible)
• Combine experience-based and coverage-based testing
• Simultaneous test design, test execution and learning
• Flexible (fit for Agile and DevOps)
• Prepared (test ideas, testing tours)
• Tools (heuristics, checklists, test design tooling)
• Serious Fun !!

25. Exploratory Testing charter
Test ideas Debriefing
information
Logging of results
(we’ll use separate
log forms)

26. Chatbot testing exercise
• You will get a charter work with 2 (or 3) people on one computer
• First add your specific test ideas to the charter
use the quality characteristics to generate test ideas
• Then execute the charter
• You can choose from the following chatbots:
• Kayak travel booking chatbot (english) (on facebook messenger)
• KLM bluebot (english) (on facebook messenger)
• Schade melden bij Centraal Beheer (nederlands) website
https://www.centraalbeheer.nl/mijn-cb/schade-melden/Paginas/schade-melden.aspx
• Time: 20 minutes

27. Debriefing Chatbot testing
• Discuss the results of your chatbot testing
with the team next to you
• Give specific attention to the quality characteristics you used
• I will walk around and collect observations for central feedback

28. Agenda of this tutorial
9:00 Welcome
9:10 Setting the scene(s) – Quiz & discussion
9:35 Quality characteristics and chatbot testing
10:30 ~ Coffee-break
11:00 Quality angles and robot testing
12:20 Fun exercise how an AI-chatbot learns
12:35 Final Quiz
12:50 Closing

30. Part 2
Quality angles &
Robot testing

31. Vision on future of quality assurance

32. Vision on future of quality assurance
Reactive Active Proactive

33. To reach this forecasting goal…

34. Quality assurance
for intelligent
machines…
Traditional angles of quality Digital angles of quality
Six angles of
Quality

35. Machine intelligence: Tesla example
The car starts braking before the accident of the 2 cars in front happens, that is really predictive!

36. How would you solve the issue of children attacking your robot?
What may happen if you introduce
a robot in your business environment?

37. What may happen if you introduce
a robot in your business environment?
The solution doesn’t have to be technical…
The robot moves
towards adults so
the kids don’t dare
to attack the robot

38. Social impact example
Video of a milking robot at a farm

39. Social impact example
Since the introduction of the milking robot, the farmer and his family can
now have dinner together at 18:00 o’clock (as usual in the Netherlands)
De aardappeleters. Vincent van Gogh

40. Robotesting

41. The IT crowd: It’s a robot …

42. Definition of a Robot
A robot is a machine that gathers information about its environment by
input of sensors and, based on this input, changes its behavior.
Combined with machine learning and machine intelligence the robot’s
reactions over time become more adequate.
The use of Internet of Things (IoT), Big Data Analytics
and Cloud technology make a robot versatile.
Robots come in many different shapes and forms.
It's not just the metallic man. Robots may equally
be a smart algorithm on social media, an autonomous
vacuum cleaner or a self-driving car.

43. Testing intelligent machines is
different from “normal” testing…
For intelligent machines predicting outcome is difficult;
Todays correct result may differ from tomorrows correct result.
To cope with this you can apply one or more of these four
approaches:
1.Define tolerances
2.Control input
3.Test the machine in its intended environment
4.Use new quality characteristics

44. 3. Testing in the intended environment
The Wepod, an autonomous
bus in Wageningen.
On the public road since
October 2016 !!
(and tested by colleagues of Sogeti)

45. 4. Use the new Quality characteristics
for intelligent machines

46. © Sogeti 201846
Super intelligence & super stupidity
Photo from the film “Ex Machina” 2015

47. Testing human friendliness
The robot must sense
the human and avoid
hurting the human.
Picture source: Article “A methodology to develop collaborative robotic cyber physical systems for production environments”, Azfar Khalid et al. © Sogeti 2018

48. Energizer…
Stand up & cross your arms
If change seems hard, try it multiple times!!

49. Agenda of this tutorial
9:00 Welcome
9:10 Setting the scene(s) – Quiz & discussion
9:35 Quality characteristics and chatbot testing
10:30 ~ Coffee-break
11:00 Quality angles and robot testing (2 sprints)
12:20 Fun exercise how an AI-chatbot learns
12:35 Final Quiz
12:50 Closing

50. Exercise for groups of + 6 people:
Test a robot
2 sprints of 30 minutes

51. Choose an exercise for your group
• Test a Lego mindstorms robot
(some of you may have seen this exercise in my workshop at EuroSTAR last year)
• Test a robot-vacuum-cleaner
• Test the image recognition machine
• Draw your favorite household robot
• Search youtube for Boston Dynamics robots and evaluate them)
For all choices: USE THE Quality Characteristics and Quality Angles
Time: XX minutes. Debriefing with the team next to you.

52. Fun exercise:
How does a visual AI learn

53. Open your browser on
- Smartphone
- Tablet
- Laptop
Go to
Do the doodling
Then check how the AI evaluated
your doodles and how the AI learned.
https://quickdraw.withgoogle.com/

54. Wrap-up
The final quizzzz ☺

55. KAHOOT quiz
Open the app on your mobile device
OR
Go to https://kahoot.it
Enter game pin: <<game pin>>
Numbers 1, 2 and 3 win a copy of my book!!
What have we learned about
testing intelligent machines?

56. Thanks for your participation
Good luck,
with testing
intelligent machines!!
@rikmarselis rik.marselis@sogeti.com