a preview of the platform TouchNet: A touch simulator and dataset of touchable objects to teach AIs how to interact with their environments via motor-sensory systems and touch

1. TouchNet preview
by Jason Toy

2. TouchNet
• A touch simulator and dataset
of touchable objects to teach
AIs how to interact with their
environments via motor-
sensory neurons

3. history
• idea around 2015
• started coding September 2017
• initial release 2017 Q4

4. why TouchNet?

5. why TouchNet?
• On Intelligence
• future predicting machines
• neuroscience based principles

6. why TouchNet?
• Helen Keller
• intelligence through fingers
• consciousness “started” from I

7. Helen Keller
• “Every object is associated in my mind with tactual
qualities which, combined in countless ways, give me
a sense of power, of beauty.”
• “..with the fingers I see only a very small portion of a
surface, and that I must pass my hand continually over
it before my touch grasps the whole. It is still more
important, however, to remember that my imagination
is not tethered to certain points, locations, and
distances. It puts all the parts together simultaneously
as if it saw or knew instead of feeling them.”

8. where is AI now?
• images, CNNs, LSTMs, GANs
• most business focus on static models; some
work in RL
• where is time?
• motor sensory neurons?
• where is the agent compared to the world?

9. - John Locke, philosophy of mind
“An unknown, but potentially large, fraction of
animal and human intelligence is a direct
consequence of the perceptual and physical
richness of our environment, and is unlikely to
arise without it.”

10. TouchNet consists of 4 parts
• dataset of 3D objects to touch and interact with
• an evolving simulation
• a simple extendable API
• benchmarks to measure and track progress

11. Dataset

12. Dataset
• procedurally generated 3D objects
• common objects: cups, plates, bowls, etc
• starting with tool objects
• cifar100 …. imagenet

13. Simulator

14. Simulators and physics
• need a real physics engine
• needs to be fully programmable & modifiable
• tested all of them: MuJoCo, DeepMind Lab,
Roboschool, project Malmo, VizDoom,Gazebo,
bullet , Processing

15. Simulators and physics
• bullet and pybullet are a secret weapon!
• robust physics engine used for many applications
• super easy to install
• movies: Shrek, Hancock,Sherlock Holmes….
• games: Toy Story 3, Grand Theft Auto 4,Hot
Wheels…

16. pybullet
• rigid and soft body simulations
• collision detection
• works with unity, blender,CUDA,OpenCL,…..
• developed by Erwin Coumans & community since
~2003
• now at Google and used for RL & Robotics
• http://pybullet.org

17. What we want to simulate

18. What we have

19. simulation demo

20. coevolving simulator & agent
• index finger with merkel touch sensors
• all sensor types in finger
• all fingers
• full arm
• full body?

21. API
• simple to get started and use
• easy to install
• modifiable and customizable

22. reinforcement learning api
• based off OpenAI’s gym and universe API
• can plug in almost any code for reinforcement learning

23. OpenAI’s Universe API

24. Same API

25. Benchmarks

26. “Blind” objection
classification
• goal: classify the object
• CNN and actor critic DNN models trained
concurrently
• pytorch and tensorflow versions
• 4% accuracy, please beat mine!

27. “blind” classification
code walk through

28. Benchmarks
• HTM version?
• more benchmarks in the future e.g:
• moving objects, differentiating textures,…..

29. Help!
• very ambitious project, too big for me
• 100% open source
• I do a little of everything ( suck at everything? )
• looking for contributors and experts in
physics,math, neuroscience, ml

30. Jason Toy
jtoy@jtoy.net
twitter.com/jtoy
Thank you!
Questions and Feedback
http://github.com/jtoy/touchnet