How to test when robots become part of your process? In this workshop you'll experience what differences there will be when robots, chatbots and other smart machines become part of your business process. This workshop is presented by Rik Marselis at the Agile Testing Days 2017 in Potsdam. This workshop is based on the Exploratory Testing flavor of TMap as documented on www.TMap.net

1. Robotesting
Workshop
Agile Testing Days
15 November 2017

2. How to test when
robots become part
of your process?
Agile Testing Days 15 November 2017

3. |
Agenda
1. Welcome & introductions
10 minutes
2. Intro: What is “robotics”, “machine intelligence”, “cognitive computing”
etc.?
10 minutes
3. Workshop: 2 sprints
10 minutes intro and 2 x 20 minutes execution
4. Wrap-up: debriefing of your results and what you learned on robotesting
20 minutes; 1 minute per team and 10 minutes overall
© Sogeti 2017 4

4. |
Welcome and introductions
Workshop Robotesting
Rik Marselis, Sogeti Nederland
5© Sogeti 2017

5. |
Who has a robot?
© Sogeti 2017 6
Who has a robot? In 10 years all of you will !!

6. |
This is R0B3 (pronounced as: ar-null-bee-three)
How can you stop the robot, without touching it?
© Sogeti 2017 7

7. 1980
Rik Marselis@rikmarselis
2007 2012 20122008
Workgroup
-member
20142009

8. |
2. What is “robotics”,
“machine intelligence”,
“cognitive computing” etc.?
Workshop Robotesting
Rik Marselis, Sogeti Nederland
9© Sogeti 2017

9. |
Terminology
© Sogeti 2017 10
Natural
language
processing

10. |
What is a “Robot”?
It’s a machine that gathers information about its environment by
input of sensors and based on this input changes its behavior.
Combined with machine learning and machine intelligence the
robot’s reactions over time get more and more adequate.
The use of Internet of Things, Big Data Analytics and Cloud
technology make a robot versatile.
A Robot can come in many different shapes and forms. It’s not just
the metallic man. It may just as well be a smart algorithm on social
media, an autonomous vacuum cleaner or a self-driving car.
This definition was assembled by Rik Marselis from various sources. © Sogeti 2017 11

11. |
What is a “Cobot”?
Autonomous, intelligent, and self-learning:
A new generation of robots is taking over factories around the world.
They work with their human colleagues more closely and safely than ever
before. Sensors and image recognition technology make it possible.
A collaborative robot  Cobot !!
© Sogeti 2017 12

12. |
What is a “bot” or “chatbot”?
The following definition appears workable:
‘A bot (short for robot) is an algorithm acting on
behalf of an individual, business or program,
that can mimic human conversation’.
Also known as “digital assistant”.
Source: Paper “The Bot Effect: ‘Friending Your Brand’ on Machine Intelligence”by SogetiLabs 2016 © Sogeti 2017 13

13. |
Machine learning (in the film “Short circuit” 1986)
© Sogeti 2017 14
<<< vervangen door filmclip >>>

14. |© Sogeti 2017 15

15. |
Testing machine learning: it’s about the input!!
© Sogeti 2017 16
The tester as “psychologist” of the intelligent machine

16. |
Cognitive
The use of cognitive computing to assist
quality assurance & testing.
Cognitive =
“knowing and perceiving”
Source of picture: Film “Wargames” 1983 © Sogeti 2017 17

17. |
Image recognition (how hard can that be?)
© Sogeti 2017 18

18. |
Robot automatically performs regression tests
© Sogeti 19

19. |
Robot automatically performs regression tests
Uses a € 300,- robot-arm © Sogeti 2017 20
Low-cost
robot-arm
tests mobile
devices.
Connected
to test
manage-
ment tool.
Test cases
check if
the quality
goals are
met and
risks are
covered.

20. |
Setting the stage
Testing OF robots
Testing WITH robots
© Sogeti 2017 21
Todays focus
The next level in
test automation

21. |
For today we use “Robotics” as overarching term
© Sogeti 2017 22

22. |
How to limit social impact of robots?
The laws of Isaac Asimov:
0. A robot may not harm humanity, or, by inaction, allow
humanity to come to harm.
1. A robot may not injure a human being or, through inaction,
allow a human being to come to harm.
2. A robot must obey the orders given it by human beings
except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such
protection does not conflict with the First or Second Laws.
Others created some more laws:
4. A robot must establish its identity as a robot in all cases.
5. A robot must know it is a robot.
6. A robot must reproduce. As long as such reproduction does
not interfere with the First or Second or Third Law.
source: Wikipedia © Sogeti 2017 23

23. |
Different angles of quality for robotics
© Sogeti 2017 24
Mechanical
Electrical
Information
Processing
Machine
Intelligence
Business
Impact
Social
Impact
new
new
new

24. |
Quality attributes
© ISO 25010 2011 25
ISO25010
product quality
Functionality
Completeness
Correctness
Appropriate-
ness
Performance
Time behavior
Resource
utilisation
Capacity
Compatibility
Co-existence
Interoperability
Usability
Operability
Learnability
User error
protection
User interface
aesthetics
Accessibility
Reliability
Maturity
Availibility
Fault tolerance
Recoverability
Security
Confidentiality
Integrity
Non-
repudiation
Accountability
Authenticity
Maintainability
Modularity
Reusability
Analysability
Modifiability
Testability
Portability
Adaptability
Installability
Replaceability
ISO 25010 consists of 2 parts:
- Product Quality (this slide)
- Quality in use (next slide)

25. |
Quality attributes
© ISO 25010 2011 26
ISO25010
Quality in use
Effectiveness
Effectiveness
Efficiency
Efficiency
Satisfaction
Usefulness
Trust
Pleasure
Comfort
Freedom from
risk
Economic risk
mitigation
Health and
Safety risk
mitigation
Environmental
risk mitigation
Context
coverage
Context
completeness
Flexibility
So ISO25010 is extensive, but is it enough for the new
AI & Robotics world?
What about:
• Ethics
• Embodiment
• Predictability
• Natural interaction
• Working in a team
• Transparency
• Generalizability
• and more (work in progress) © Sogeti 2017
• Privacy
• Empathy
• Engagement
• Human friendliness
(Asimov’s laws)
• Charisma (wow-factor)
• Happiness

26. |
Let’s get started!
We’ll use Exploratory Testing
(the proper way!!)
Workshop Robotesting
Rik Marselis, Sogeti Nederland
27© Sogeti 2017

27. |
E.T. has many flavours, this is our flavour:
Source: www.tmap.net Source for term “mob-testing”: Maaret Pyhäjärvi © Sogeti 2017 28
• Focus on confidence (risk-based)
• Structured (charter, log, debriefing)
• Session-based & Timeboxed (not too short, not too long)
• Tandem-approach (today we’ll do “mob-testing”)
• Combine experience-based and coverage-based testing
• Simultaneous test design, test execution and learning
• Flexible (fit for Agile and DevOps)
• Prepared (test ideas, testing tours)
• Tools (heuristics, checklists, test design tooling)
• Serious Fun !!

28. |
Our pragmatic charter and log template
Exploratory Testing © Sogeti 2017 29
Charter Log
De-
briefing
Test ideas: e.g.
testing tours,
test data, etc.

29. |
Even with E.T.: always think of expected output
Exploratory Testing steps:
• Before start of the session think of “Test ideas” and write on the charter
(useful thoughts that help in determining what next test to do)
• During the test session iterate the next steps:
• Think of what you want to learn
• Think of what test you can do to learn this
• Think of the result you expect from the test
• Log your test and expected result
• Execute the test (“test case” if you like)
• Log the actual result
• What did you learn?
• What is the next test you now want to do?
© Sogeti 2017 30

30. |
Let’s get going!
2 challenges for every team
Workshop Robotesting
Rik Marselis, Sogeti Nederland
31© Sogeti 2017

31. |
Challenge 1: Test the chatbot
We’ll use the chatbot “Kayak” on facebook messenger.
One person of your group logs in to facebook messenger
Contact KAYAK
Find a return flight from your airport to an airport on another continent,
next find a hotel in that city and two things to do.
© Sogeti 2017 32
You’ll get a charter

32. |
Workshop item 1: chatbot
https://www.facebook.com/messages/t/kayak
© Sogeti 2017 33
Short demo

33. |
Challenge 2: Test the robot
We’ll use the Lego Mindstorms robots.
The robots have a color-scanning capability that makes them react to
certain combinations of colors.
Determine whether the robot works as designed and is fit for purpose.
© Sogeti 2017 34
You’ll get a charter

34. |
User-stories for color-recognizing robot
© Sogeti 2017 35
• As a robot I scan 3 colors during the session
so that I can show how that makes me feel
• As a robot I will wink when I see white
so that everyone knows I’m not that serious
• As a robot I will show loving eyes when I see black
so that everyone knows I’m into gothic
• As a robot I will cheer when I see red two times
so that everyone can see I’m happy
• As a robot I will shout “boo” when I see blue twice
so that everyone knows I don’t like blue
• As a robot I stop moving when I hit something
so that I don’t hurt any human (as per Asimov’s law)
• As a robot I’ll drive backwards when I’m ready scanning AND the remote control is operated
so that I will return to my starting position and be ready for the next try.
Short demo

35. |
Let’s get started!
We’ll make 8 teams (of equal size).
Workshop Robotesting
Rik Marselis, Sogeti Nederland
36© Sogeti 2017

36. |
The challenge
We’ll make 8 teams.
Every team gets:
• Exploratory testing charter (with room for debriefing info on the back)
• Log form to register your tests (and, very important, expected results!!)
© Sogeti 2017 37

37. |
Workshop:
Round 1
20 minutes
38© Sogeti 2017

38. |
 Team Asimov
 Team Johnny 5
 Team Robot
 Team R2D2
Test the robot
 Team Grassbot
 Team Vacubot
 Team Cobot
 Team C3P0
Test the chatbot
Who does what challenge in round 1?
© Sogeti 2017 39

39. |
Workshop:
Round 2
20 minutes
40© Sogeti 2017

40. |
 Team Grassbot
 Team Vacubot
 Team Cobot
 Team C3P0
Test the robot
 Team Asimov
 Team Johnny 5
 Team Robot
 Team R2D2
Test the chatbot
Who does what challenge in round 2?
© Sogeti 2017 41

41. |
Debriefing:
your results and what you
learned on Robotesting
Workshop Robotesting
Rik Marselis, Sogeti Nederland
42© Sogeti 2017

42. |
Your key-learnings in 1 minute per team
© Sogeti 2017 43
What
did
you
learn
about
testing
of
Robots
and
Chatbots?

43. |
What did you learn about quality attributes?
© Sogeti 2017 44

44. |
Digital happiness
Exponential growth of technology, Ever-increasing speed of digitalization,
Security-breaches, Fake-news, Cyberwar, and Privacy-violations…
People wonder: ‘Does all this technology make me happier?’
The conclusion for now is that:
• Happiness is becoming humanity’s explicit goal (instead of only GDP for instance)
• Technology will help measuring happiness (and maybe increase our
happiness obsession?)
• Companies will be reviewed through this happiness perspective:
“Does your business make me happier?”
© Sogeti 2017 45

46. Robotesting
The end Thank you!
Agile Testing Days
15 November 2017