Skynet? Really? How close are we to self aware, self replicating machines? In this fun session learn some of what computers can do and what they can’t. You think you know. You may be surprised. The emerging focus on Cognitive computing, general AI, Computer Vision, Internet of Things, etc. signpost the way to new opportunities and new challenges for computers and humans alike. We decided to see how far we could get in building our own version of an all powerful controlling entity. In this session we’ll cover how we did it, what we learned and answer those important questions like: “Can we build a Skynet yet?”, “Can my computer be my best friend?”, ”Will I ever able to program without a keyboard?”, ”Can a computer read my mind?” and the all important “will drones be able to deliver beer at the right temperature?”

1. THE RISE OF THE MACHINE - IS
SKYNET CLOSER THAN EVER?

2. JVM Developer
DevOps practitioner
Developer Advocate
Robot Builder
IoT speculator
AI explorer @spoole167
Work at IBM’s UK Research
and Development Laboratory
Means
I get to play
with cool stuff
Steve Poole : IBM

3. How close are we to
building Skynet?
And can anyone do it?

4. “Skynet is a fictional neural net-based conscious group mind
and artificial general intelligence (see also superintelligence)
system that features centrally in the Terminator franchise
and serves as the franchise's main antagonist.”
https://en.wikipedia.org/wiki/Skynet_(Terminator)

5. We live in interesting times

6. Autonomous Robots

7. Meet ‘Handle’
Another Robot from Boston Dynamics
http://www.bostondynamics.com/
Handle is 6.5 feet tall, can jump 4 feet and travels at speeds of up to 9 mph.
and can travel up to 15 miles between charges

8. Autonomous Cars

9. Hyundai - The Empty Car Convoy

10. Autonomous Drones

11. The test of the world's largest micro-
drone swarm in California in October
2016 included 103 Perdix micro-drones
measuring around six inches launched
from three F/A-18 Super Hornet fighter
jets,

12. Large organisations of devices

13. Bot Nets
500,000+ hijacked internet-connected things
like cameras, lightbulbs, and thermostats
launched the largest DDoS attack ever
against a top security blogger
IOT
An integrated end-to-end solution that enables
your apps to communicate with, control, analyze,
and update connected electronics

14. 50 Billion devices connected to
the internet by 2020
Wikipedia: “Skynet gained self-awareness after it had spread into millions of
computer servers all across the world”

15. AI’s that can beat humans at
Chess, Go and even Jeopardy
Jan 2016
Feb1996
Feb 2011

16. Computers we can talk to

17. Apple’s Siri Google’s AssistantAmazon’s Alexa

18. Computers that can understand
the world

19. https://github.com/karpathy/neuraltalk2
Dynamic Image Captioning

20. Computers that can recognise you

22. Q: How long do we have left?
In the movie (set in 1984) the first Terminator came from 2024

23. A: AI is not scary in the way you might
imagine

24. The Nature of AI: a worked
example

25. Can I build a terminator?
My wife says “no”

26. Can I build a terminator?
Too late!

27. Can I build a robot that does something else?
Naturally my colleagues suggest we should build a beer delivery system

28. Terminator4J RobotDelivery4J

29. My Robot
• Needs to be able to deliver a can to an individual within a specific
amount of time
ie before the drink gets too warm
• Needs to take verbal instructions
no keyboards thank you
• Needs to be able to communicate with the target individual
ie ”here is your drink”
• Plus all the simple stuff like finding its way around a room

30. My Robot needs “senses”
Hear -> microphone
Speak -> loudspeaker
Locate -> position & orientation?
Touch -> ultra-sonic range sensor?
Vision -> camera
Temperature -> temperature sensor?

31. My toolkit
• Java
• OpenCV
• Cuda4J
• Apache Spark
• DeepLearning4j
• Neuroph
• Processing
• Gazebo
• Raspberry PI
• Robot HAT
• Robot chassis with wheels
• Ti SensorTag
• Ultra sonic range finder
• Webcam
• Loudspeaker
• Microphone
• Batteries x many, many

32. My Robot
Distance
Sensor
Wheels
Ti Tag
Camera
Raspberry PI
Robot HAT
Battery

33. My Robot
Distance
Sensor
Wheels
Ti Tag
Camera
Drink

34. Ti SensorTag
• Bluetooth enabled
• 9 axis motion sensor
• IR thermopile temperature sensor
• Digital microphone
• Magnet sensor
• Humidity sensor
• Pressure sensor
• Light sensor

35. How hard can it be?

36. Something simple – direction control

37. Learning with Neural Nets
• Simple example
• If target to the left - turn left
• If target to the right – turn right
• If target straight ahead – move forwards
• If target within 5cm- stop
Target
Distance
Sensor
Wheels
0o
180o

38. Left Motor = back
Right Motor = forward
Left Motor = forward
Right Motor = back
Left Motor = forward
Right Motor = forward
Left Motor = stop
Right Motor stop

39. Psudocode
if distance to target <=5 cm // close
left motor = stop
right motor = stop
else if angle to target > 355o or < 5o // straight ahead
left motor = forward
right motor = forward
else if angle < 180o // on the right
left motor = forward
right motor = backward
Else // on the left
left motor = backward
right motor = forward

40. Motor direction plot (distance ignored)
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right
+0.5
Forward
-0.5
Backwards
Angle of target

41. 0.5 0.7
0.52
Values from
Nodes
Are weighted
0.10.9
summed
=(0.5*0.9)+(0.7*0.1)
= 0.45+0.07
=0.52
And a signmoid transfer function
applied
1.0 ./ ( 1.0 + exp(-0.52) )
0.6271477663131956

42. 0.5 0.7
0.52
0.1 - 0.010.9+0.02
0.6271477663131956
We take
expected
answer 0.7
And adjust the
weights
Depending on their
contribution
=(0.5*0.92)+(0.7*0.09)
= 0.46+0.063
=0.523
0.6278489986434628And repeat a few
times

43. My first neural net
Input node
“direction”
Hidden layer
Output mode
“right motor”
Output mode
“left motor”

44. My first neural net
• All input and output is between 0 & 1
• Uses a “sigmoid function” as the trigger
• Uses a back propagation method
• Uses supervised training.

45. Motor direction 50 sample data
-0.07
-0.035
0.
0.035
0.07
0.105
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right
+0.5 Forward
-0.5 Backwards

46. Motor direction 500 sample data
-0.75
-0.5
-0.25
0.
0.25
0.5
0.75
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right
+0.5 Forwards
-0.5 Backwards

47. Motor direction 5000 sample data
-0.75
-0.5
-0.25
0.
0.25
0.5
0.75
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right
-0.5 Backwards
+0.5 Forwards

48. Motor direction 50000 sample data
-0.75
-0.5
-0.25
0.
0.25
0.5
0.75
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right
-0.5 Backwards
+0.5 Forwards

49. Motor direction 500000 sample data
-0.75
-0.5
-0.25
0.
0.25
0.5
0.75
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right
-0.5 Backwards
+0.5 Forwards

50. Motor direction 5000000 sample data
-0.8
-0.5
-0.3
0.
0.3
0.5
0.8
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right
-0.5 Backwards
+0.5 Forwards

51. But..
Errors in your training can cause errors
in the AI

52. Motor direction plot (distance ignored)
-0.8
-0.5
-0.3
0.
0.3
0.5
0.8
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right
I had a bug
in the training
algorithm

53. Motor direction 500000 sample data
-0.75
-0.5
-0.25
0.
0.25
0.5
0.75
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right
Which the NN
copied

54. What about other neural net structures
1x2x2 ?
1x12x2 ?

55. Motor direction 500000 sample data 1x2x2
-0.75
-0.5
-0.25
0.
0.25
0.5
0.75
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right

56. Motor direction 500000 sample data 1x12x2
-0.75
-0.5
-0.25
0.
0.25
0.5
0.75
0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
Left Right

57. Training costs
• More nodes -> more time (no guarantee of increased accuracy)
• More accuracy -> more training -> (no guarantee of increased
accuracy)
• Increased complexity of problem -> more nodes—> more time -> (no
guarantee of increased accuracy)

58. We’ve seen a simple NN learning the output of an
algorithm
• The algorithm is a sort of mathematical
function
• It already ‘knows’ how to map the input
value to some output values
• The NN is trained to find the same
function mapping in a 3 dimensional
space.
• Somewhere in the space of (angle, right
motor, left motor) there is the right
answer.

59. Now we’ll add in distance
• we’re creating a new algorithm that maps a value in a 4 dimensional
space
• Angle , distance , left motor, right motor
Distance

60. My second neural net
Input node
“distance”
Hidden layer
Output mode
“right motor”
Output mode
“left motor”
Input node
“direction”

61. AI Visualiser
Screenshot

62. Poorly trained
Loop 5000 times
pick random angle
get expected answer
get answer from neural net
apply positive & negative feedback
Loop throw angles 0..360
get expected answer
get answer from neural net
apply positive & negative feedback
Properly trained

63. Next steps.
• Add in the other sensors
• Temperature, proximity sensor, acceleration, light, compass
direction etc
Distance

64. I’m going to need a bigger neural net
left motor right motor I’m stuck No drink drink too warmdrink delivered
?

65. How do I decide on the structure of the network?
How do I train this thing!!!
How do I visualise how the AI ‘thinks’?

66. Training =Simulation + Neuroevolution
• Gazebo for robot simulation
• Use genetic algorithms to represent Neural Network.
• Define fitness selections and add incrementally
• Fastest to deliver
• Nearest to target
• able to spot being stuck …
• Evolve.
• Will let you know when this works.
• Unsupervised training - survival of the fittest

67. Terminator4J RobotDelivery4J
• Training is stupidly hard
• I hadn’t appreciated how hard.
• You need something like Gazebo
• But it still takes significant time to train and AI
• Plus visualising the AI decision process is difficult if not impossible
http://gazebosim.org/

68. Computer Vision : face recognition
• How do you turn a picture of a face into a ‘key’ ?
• Especially when the face is at a different distance or relative angle?

69. “face landmark estimation"
Convert the image into HOG format
Identify key points around the face.
Train an AI to find those points in an image
Use the located points to morph the face
image into a standard form
Invented by Vahid Kazemi and Josephine
Sullivan.
http://www.csc.kth.se/~vahidk/papers/KazemiCVPR14.pdf
http://sharky93.github.io/docs/gallery/auto_examples/plot_hog.html

70. Once you can identify
the key parts of a face
you can morph the image
in other interesting ways
face landmark estimation

71. ‘Encode” the face
• With a standardised face image
• Identify the ‘important’ parts of the face so you can do pattern matching
• The challenge
• We don’t know what the ‘important’ parts are
• The answer
• Use a Neural Network to work it out.
• http://www.cv-foundation.org/openaccess/content_cvpr_2015/app/1A_089.pdf

72. Voice to Text
• Humans are very good at understanding spoken words :-)
• Computers have a harder time

74. http://cmusphinx.sourceforge.net/
Sphinx4
A pure Java speech
recognition library
Watson Speech to Text
Java API to online speech
recognition solution
(free tier)
https://speech-to-text-demo.mybluemix.net/

75. Results
“love they do is”
“Open the pod bay doors hal.”
Speaker 0: Open the pod bay
doors hal.
Speaker 1: Sorry. Great I can.
“Open the pod bay doors hal.
I’m sorry Dave. I can’t do that”
Sphinx4 Watson

76. Speech to Words
• Accurate translation of speech to words can be very good
• It’s almost as accurate as humans
• It will soon be much better.
• Without good microphones results can be mixed!
• Behind the scenes there is still a significant amount of compute
power being used.

77. Practically
• How would I get my robot to understand commands like
• “take this drink to Rob in the corner”
• “take this drink to the lady talking to my wife”
• “give this drink to the man wearing the Hawaiian shirt and black
shorts”
• How do I extract meaning?

78. Chat Bots
User: "Alexa, open History Buff"
Alexa: "History Buff. What day do you want events for?"
User: "August thirtieth."
Alexa: "For August thirtieth, in 2003, [...] . Wanna go deeper in history
User: "Yes."
Alexa: "In 1995, Bosnian war [...] . Wanna go deeper in history?"
User: "No."
Alexa: "Good bye!"
https://github.com/amzn/alexa-skills-kit-java

79. Chat Bots: a little bit of a cheat
GetFirstEventIntent get events for {day}
GetFirstEventIntent give me events for {day}
GetFirstEventIntent what happened on {day}
GetFirstEventIntent what happened
GetFirstEventIntent {day}
GetNextEventIntent yes
GetNextEventIntent yup
GetNextEventIntent sure
GetNextEventIntent yes please
AMAZON.StopIntent no
AMAZON.StopIntent nope
AMAZON.StopIntent no thanks
AMAZON.StopIntent no thank you

80. Chat bots
• Map intent to actions
• Work out your intent using various levels of Natural Language Parsing
• Hard coded string matching
• Templates (like Alexa)
• Trained sentence mapping (like Watson Conversation Service)
• Respond to you appropriately
• Don’t have any intrinsic understanding of your intent (yet)

81. Artificial Intelligence
• Relies on ‘good’ data representations
• Relies on appropriate internal data structures
• Relies on the ‘right’ sort of training
• Embeds knowledge inside its data structures
• Very quickly becomes opaque to humans
• Needs significant processing power
This is
‘art’
not
science

82. GPUs to the rescue?
• GPU’s can massively reduce neural net training and execution times.
• They are ideally suited to parallel processing the calculations of a Neural Net.
• You can use your own or buy time in the cloud
• For instance Amazon offers an 16x NVIDIA K80 GPU instance in ECS at $14/hour
• You get, nominally, x10 performance improvements
• Multiple bindings for GPUs for most languages
• IBM JVM has intrinsic support for GPU’s. JIT can utilise GPUs under the covers for
appropriate operations.

83. How long do we have left?
In the movie (set in 1984) the first Terminator came from 2024

84. Checkpoint
• Building you own cognitive robot or service is hard
• It takes time.
• It takes data.
• It takes patience.
• What AI techniques are giving us is the ability to mine data effectively and be able to teach systems to ‘understand’
that data
• It’s clear that companies are creating AI’s that will be significant assets.
• It’s also clear that differentiator will be the quality and quantity of data used to train the AI
• No sign of sentience yet.
• However…

85. • We design the AI
• We train the AI
• Our desires / agendas / biases can easily get encoded into the AI
• How do we ‘trust’ AI’s?
The Skynet idea still has an achilles heel …
Don’t worry about Skynet yet.
Worry about how we learn to understand and visualise the AIs we’ve created.

86. The Achilles heel of Skynet

87. My Robot uses 6 AA batteries
• Gives it a few hours of running

88. How many AA batteries would you need to run a
Terminator?
• Here’s the Human equivalent
• Number of AA batteries to play the piano for an hour?
• Number of AA batteries to walk for an hour?
• Number of AA batteries to stand up and paint a wall for an hour?
91
127
152

89. Yucky question
How much energy (in number of AA batteries) is stored in 1kg of human
fat?
1869

90. So actually human beings are pretty efficient
batteries?
Maybe we’re in the wrong movie?

91. Thank you
any questions?