Talk given at Develop:Brighton 2019 where I shared some of the findings of our research in how machine learning can be applied in games and how this can heighten positive aspects in the player-computer interaction. The research and technology presented in this session explores how using interactive machine learning techniques in the design of sensor-based control schemes of digital games can improve the player experience and how simple it can be to use. IML can be used to quickly prototype controls by directly showing human actions and computer responses without writing code! Players can design enjoyable control schemes for themselves and create unexpected playful interactions!

1. Carlos González Díaz
carlosglesdiaz@gmail.com
Interactive Machine Learning
for More Expressive Game
Interactions

2. • I’m a massive nerd about games,
tech and science
• PhD in Intelligent Games and
Games Intelligence
Carlos González Díaz @Carlotes247
Who am I?

3. • PhD in Intelligent Games and Games
Intelligence
• Joint programme between 4 leading
UK universities
• Strong ties to industry (let´s have a
chat! :D )
• My thesis: How game control
customisation in VR affects players
Carlos González Díaz @Carlotes247

6. Carlos González Díaz @Carlotes247
Carlos Gonzalez Diaz Phoenix Perry Dr. Rebecca Fiebrink
@carlotes247 @phoenixperry @rebeccafiebrink

7. •Can you think of at least 3 examples of
devices/controller that use sensors?
Carlos González Díaz @Carlotes247

8. Carlos González Díaz @Carlotes247

9. Carlos González Díaz @Carlotes247
• Hard to implement accurate sensor analysis when sensors
are high dimensional or noisy
• Players might want to customise sensor-based game
interfaces (like gamepads)
• Disabled players might need a highly customisable
unconventional interface
Problems with Sensors

10. Carlos González Díaz @Carlotes247
Controls Remapping
• What about
sensor
interactions?

11. Carlos González Díaz @Carlotes247
• But what is IML?
IML can be a solution

12. Interactive Machine Learning (IML)

13. Carlos González Díaz @Carlotes247
• Can you think of an example of something that might already
use IML?
Quick question…

14. Carlos González Díaz @Carlotes247
“IML is Machine Learning with a
human in the learning loop,
observing the result of learning
and providing input meant to
improve the learning outcome.” -
Dr. Brad Knox (MIT, 2013)
Interactive Machine Learning (IML)

15. Carlos González Díaz @Carlotes247
Interactive Machine Learning (IML)

16. Carlos González Díaz @Carlotes247
• Training data: set of
examples our IML system
learns from. I.e.:
• Mouse position
• Body position in
webcam
• Hand gesture
What is training data?

17. Carlos González Díaz @Carlotes247
● Model: function that
performs Classification or
Regression on live input
data.
• Training data outputs
to a Model.
What is a model?

18. Our Models are Supervised Learning Models

19. Carlos González Díaz @Carlotes247
• Classification refers to the
problem of assigning a category
or a label to any incoming input.
• Categories are mutually
exclusive.
• Classifier always output one of
the known categories.
Classification

20. Carlos González Díaz @Carlotes247
Example of Classification (Dr Rebecca Fiebrink)

21. Carlos González Díaz @Carlotes247
• Regression computes a numerical
value from inputs
• Outputs changes smoothly as the
inputs change.
• Control continuous parameters
• Volume
• Position
Regression

22. Carlos González Díaz @Carlotes247
Example of Regression (Dr Rebecca Fiebrink)

23. Carlos González Díaz @Carlotes247
Existing IML Tools: Wekinator
• wekinator.org
• Java service (send and receive
data through OSC)

24. Carlos González Díaz @Carlotes247
• Can you name limitations of Wekinator?
• Too many apps open sending and receiving data
• Java
• Portability
Limitations of Wekinator

25. Carlos González Díaz @Carlotes247
• Unity node-based plugin
• Made for Unity
• No code required!
Our solution: InteractML

26. Carlos González Díaz @Carlotes247
• We provide a range of feature
extractors for your game
• You can easily make your own
• Shown at the GDC AI track
InteractML

27. Carlos González Díaz @Carlotes247
• InteractML.com
• github.com/Interactml/
iml-unity
• Alpha release available!
Get InteractML

28. Carlos González Díaz @Carlotes247
• Still early novel software
• Many interaction designs not clear to us as the paradigm is
very new to games
• Please try it and send us feedback <3
InteractML is still under development
github.com/Interactml/iml-unity

29. Carlos González Díaz @Carlotes247
InteractML DEMO

30. Carlos González Díaz @Carlotes247
• Let’s imagine ideas for future interactive machine learning
games…
• Can you name 3 ideas?
Let’s talk about the future

31. Carlos González Díaz @Carlotes247
• Interaction with NPCs using gestures.
• Gestural interactions where there is a lot of variation
My ideas for the future

32. Carlos González Díaz @Carlotes247
Thank you!!

33. Interactive Machine Learning for
More Expressive Game Interactions
@carlotes247 @phoenixperry @rebeccafiebrink
github.com/interactml/iml-unity

34. Carlos González Díaz @Carlotes247
Quick Example of IML (Dr Rebecca Fiebrink)

35. Carlos González Díaz @Carlotes247
Example of DTW (Dr Rebecca Fiebrink)