Exploring the Brain Computer Interface

3,055 views

Published on

Computer-brain interface is a mainstay of science fiction, and devices are available today to use our brainwaves as a computer input. But is it practical? How far away is it? Will "Big Brother" read our thoughts and hack our brains?

In this class, we will dive into the future of thought as input for wearable devices with real-world examples and code. Demonstrations will be shown using the Emotiv EPOC headset, a revolutionary high resolution, neuro-signal acquisition and processing wireless neuroheadset that uses a set of sensors to tune into electric signals produced by the brain to detect thoughts, feelings and expressions.

You will see the EEG neuroheadset and computer interface with examples of interfacing with desktop, mobile and wearable apps. We will dive into the roots of the technology, showing code and examples along with big pictures of the technology. You will walk away with an understanding of how this still evolving and largely unknown technology really works, how it can be used, as well as longer-term implications.

Published in: Technology, Health & Medicine

Exploring the Brain Computer Interface

  1. 1. Android and the Brain-Computer Interface JIM MCKEETH EMBARCADERO TECHNOLOGIES JIM.MCKEETH@EMBARCADERO.COM @JIMMCKEETH
  2. 2. ‹#› About Jim McKeeth • Lead World Wide Developer Evangelist • Host of Podcast at Delphi.org • Longtime developer ◦ Object Pascal, Java, C#, JavaScript, Objective-C, etc. • Invented and patented swipe to unlock in 2000 ◦ US Patent # 8352745 & 6766456, etc. • Improvisational performer • I am not a neuroscientist, but I find this all fascinating! 2
  3. 3. ‹#3› Embarcadero Technologies • Founded 1993 • Provides Architect, Database and Developer Tools • 3.2 Million Customers including 97% of Fortune 2000 • 35+ Product Awards – Consistent Innovation • 500+ Employees in 29 Countries •www.embarcadero.com
  4. 4. ‹#› Agenda • The Science of Reading the Brain • The Science of Direct Brain Input • Consumer Brain-Computer Interfaces • Demonstration - Volunteer flying the Quadricopter • Summary & MindTunes Images from Wikimedia Commons or Product websites. 4
  5. 5. -Claus Torp Jensen (@ClausTorpJensen) Senior Technical Staff Member and Chief Architect API Economy at IBM (@ibmapimgt) ‹#› “In five years, science fiction will be real!” 5 What will the world be like in five years?
  6. 6. ‹#› Brain-Computer Interface ST:TNG Episode 93 (4x19): The Nth Degree (1991) 6
  7. 7. ‹#› Brain Reading and Writing SQUIDS from Strange Days (1995) X
  8. 8. The Science of Reading the Brain
  9. 9. ‹#› Vocabulary and Roots • Encephalon = Latin. The brain. • -graphy (sufix) = Greek. One who draws or writes. • Encephalography = Drawing (or picture) of the brain • Poly- (prefix) = Greek. Many • Tomos = Greek. Cut into parts or sections. • Neuro- (prefix) = Greek. Relating to nerves or nervous system • Hemo- (prefix) = Greek. Pertaining to blood. • Dynamics = Greek. Active; in motion X
  10. 10. ‹#› Understanding the Brain X Delta 4 Hz Hemodynamics (Blood flow) Theta 4-­‐7 Hz Alpha 7-­‐14 Hz Mu 8-­‐13 Hz Beta 15-­‐30 Hz Gamma 30-­‐100 Hz Neural Oscillation
  11. 11. ‹#› Polygraph (aka Lie Detector) • Measures physical manifestations of stress that come with lying • Blood pressure, pulse, respiration, skin conductivity, etc. • Looks for changes. • Requires specialized training. • Invented in 1921 8
  12. 12. https://developer.bluetooth.org/TechnologyOverview/Pages/HRP.aspx ‹#› Heart Rate Monitors on Android • Bluetooth GATT Profiles supports all Bluetooth Smart Heart Rate Monitors. • Your app doesn’t need to support individual heart rate monitors. • Assigned Number: 0x180D 9 https://developer.bluetooth.org/gatt/profiles/Pages/ProfilesHome.aspx
  13. 13. ‹#› Magnetic Resonance Imaging (MRI) • Uses oscillating magnetic fields at different resonant frequencies. • Functional MRI (fMRI) uses software to analyze images. • Shows activity (blood flow) in different parts of the brain. • Combined with brain mapping can reveal thought information. 10
  14. 14. CT / CAT Scan (Computed Tomography X-ray) • Faster & cheaper than an MRI • Uses ionizing X-ray radiation • Lower detail images • Sometimes combined with Positron ‹#› Emission Tomography (PET) scans which use gamma radiation. X
  15. 15. ‹#› Near-Infrared Spectroscopy (NIRS) • Spectroscopic use of near-infrared (700-900nm) light through the skull and detecting degree remerging light is attenuated. • Detects blood oxygenation levels in different areas of the brain. • More portable than fMRI but only scans cortical tissue. X
  16. 16. ‹#› Electroencephalography (EEG) •Measures electrical activity along the scalp. • Detects voltage fluctuations resulting from ionic current flows within neurons of brain. •Millisecond-range resolution (faster than CT or MRI) • Typical clinical usage has 19+ input electrodes as well as ground and reference. • Invented between 1875 and 1924 11
  17. 17. ‹#› Electroencephalography (continued) • Series of electrodes placed next to scalp. • Difference in voltage is measured between each electrode and the reference. X
  18. 18. ‹#› Magnetoencephalography (MEG) 12 Uses Arrays of SQUIDs (Superconducting QUantum Interference Devices)
  19. 19. The Science of Direct Brain Input
  20. 20. ‹#› Change Perception of Reality The “headjack” from The Matrix movies -­‐ 1999 14
  21. 21. Transcranial Direct-Current Stimulation (tDCS) • Constant, low current delivered via electrodes. • Results: ◦ Increase cognitive performance ◦ Enhance language & math ability ◦ Improve attention span ◦ Aid problem solving, memory & coordination • Recent DIY “Brain Hacking” Craze ◦ http://www.reddit.com/r/tDCS 15 ‹#› Accelerated Learning tdcsplacements.com http://thebrainstimulator.net/
  22. 22. ‹#› tDCS with Mobile • foc.us just launched new tDCS headsets with Bluetooth API • Developer devices are available for preorder • Targeting athletes and gamers • http://www.foc.us/index.php/v2-tdcs-developer- edition • http://www.foc.us/bluetooth-api 16
  23. 23. ‹#› Transcranial Magnetic Stimulation (TMS) • Similar strength to MRI •Magnetic induction of electrical currents • Used to treat neurological conditions • Produces: ◦ Flashes of lights in vision (phosphene) ◦Muscular activity 17
  24. 24. Ultrasonic Transcranial Stimulation Device • 2000 patent from Sony (US6584357 B1) ◦ Finer targeting than TMS ◦ Theoretically invoking sensory experiences ◦ Continued research and patents (US7350522 B2) 18 ‹#› http://www.newscientist.com/article/mg18624944.600 Speculative work, “There were not any experiments done.”
  25. 25. ‹#› Ultrasonic Neuromodulation for Spatial Resolution • Virginia Tech Carilion Research Institute study published in Jan 2014 Nature Neuroscience • Used “targeted ultrasound” to boost “sensory perception” • Targeted an area of the brain as small as the size of an “M&M” http://www.eurekalert.org/pub_releases/2014-­‐01/vt-­‐udt011014.php 19
  26. 26. ‹#X› "In neuroscience, it's easy to disrupt things. We can distract you, make you feel numb, trick you with optical illusions. It's easy to make things worse, but it's hard to make them better. These findings make us believe we're on the right path." -- William "Jamie " Tyler, Assistant Professor Virginia Tech Carilion Research Institute
  27. 27. 2‹#0› Using More than 10% of Our Brain? We all use 100% of our Brain Capacity LUCY 2014
  28. 28. Science Fiction Becoming Reality Interesting Milestones
  29. 29. ‹#› Brain-to-Brain Input The “Hat” from Brainstorm -­‐ 1983 22
  30. 30. ‹#› Brain-to-Brain Input at University of Washington in 2013 • Input from Computer • EEG from brain • Transmitted over campus internet • Replayed through TMS to another brain • Output to computer More information http://bit.ly/EEG2TMS & http://bit.ly/bci2cbi 23
  31. 31. ‹#› Cybernetic Limbs Captain America: The Winter Soldier -­‐ 2014 X
  32. 32. ‹#› DARPA’s Brain Controlled Prosthetic Arm • Part of DARPA’s Innovation Pathway. • Uses microchip implanted in brain. • 27 different fine grained controls to arm, hand and fingers. • Expected on market by 2016 X http://www.fastcompany.com/1725799/darpas-­‐mind-­‐controlled-­‐ prosthetic-­‐arm-­‐could-­‐be-­‐market-­‐four-­‐years
  33. 33. ‹#› See What You See - Hear What You Hear The “Monitor” from Ender’s Game – 2013 24
  34. 34. ‹#› See What You See • In 2010 another team lead by Shinji Nishimoto partially reconstructed YouTube videos based on brain recordings. • Used fMRI • Overlaid similar videos to create composite video. • http://youtu.be/nsjDnYxJ0bo & http://youtu.be/EdGjB6XCXOA http://bit.ly/ReconstructingVisual 25
  35. 35. ‹#› See What You See X • In 2008 IBM used a feedback loop based on the fusiform face area of the observer’s brain for facial recognition. • Patent # US20100049076 A1
  36. 36. ‹#› Hear What You Hear • 2012 - UC Berkeley & Brian Pasley decoded internal neural processing of auditory information. • Reconstructed sound on computer from electrical signals in observer’s brains. • Resulting sound was recognizable when compared to original. http://bit.ly/ReconstructingSpeech 26
  37. 37. ‹#› Identifying Thoughts • 2008 & 2009 Using fMRI recognized thoughts within the human brain. • Discovered similar thoughts appear similar is different subject’s brains. • Was able to discern which of 10 objects subject was thinking about. • Also predicted subject intentions up to 10 seconds before subject expressed them. http://bit.ly/IdentifyThoughts X
  38. 38. ‹#› Data Entry via BCI ST:TNG Episode 93 (4x19): The Nth Degree (1991) 27
  39. 39. ‹#› Typing via BCI • Brain researcher Jonathan Wolpaw at Wadsworth Center in Albany, NY •Wear EEG while looking at a flashing matrix of characters • Each character flashes with a different pattern • ~15 seconds to type a character • The Verge http://embt.co/typebci 28 . . . . . . . . . A B C D E F . . G H I J K L . . M N O P Q R . . S T U V W X . . Y Z 1 2 3 4 . . 5 6 7 8 9 0 . . . . . . . . .
  40. 40. ‹#› Uploading Your Consciousness Transcendence -­‐ 2014 29
  41. 41. ‹#› Human Connectome Project •Map of neural connection the brain • The neurological version of the genome • Unique to each person, changes over time • Representation of memories, thoughts, etc. •www.ted.com/talks/sebastian_seung • humanconnectomeproject.org • humanconnectome.org Image by Xavier Gigandet et. al. 30
  42. 42. ‹#› OpenWorm Project •Mapped the 300+ neurons and 7000+ connections of a worm (C. elegans) • Created a software model of connectome • Connected to LEGO robotic representation • Behaved like worm (without training or programming) • http://bit.ly/openwormlego & http://www.openworm.org/ 31
  43. 43. As of July 2014 ‹#› Wearable User Interfaces expectations Speech-to-Speech Translation Hype Cycle Brain-Computer Interface Internet of Things Autonomous Vehicles Disillusionment time Plateau of Productivity Slope of Enlightenment Trough of Peak of Inflated Expectations Innovation Trigger Plateau will be reached in: less than 2 years 2 to 5 years 5 to 10 years more than 10 years obsolete before plateau http://www.gartner.com/newsroom/id/2819918 32 Gartner's 2014 Hype Cycle for Emerging Technologies August 11, 2014 BCI
  44. 44. Read-Only Consumer Brain-Computer Interfaces
  45. 45. ‹#› Consumer BCI Devices • Emotiv EPOC, Insight & EPOC+ www.emotiv.com • NeuroSky’s MindWave neurosky.com • OpenBCI www.openbci.com • Neuroelectrics Enobio www.neuroelectrics.com • InteraXon’s Muse choosemuse.com X
  46. 46. NeuroSky’s MindWave (also in XWave headsets) 3‹#4› • EEG with 1 dry electrode • Measures “attention,” “meditation” and eye-blinks • Bluetooth communication • SDK for iOS, Android, PC & Mac • $129.99 to $79.95 retail neurosky.com • Hacking: ◦ http://makezine.com/2011/05/10/behind-the-brain-blinker/ ◦ http://hackaday.com/2012/12/20/modifying-an-eeg-headset-for-lucid- dreaming/
  47. 47. ‹#› Emotiv EPOC • 16 wet electrodes ◦ 14 EEG electrodes – read brain waves + 2 reference electrodes • Gyroscope to read head movements • 4 mental states, 13 conscious thoughts & facial expressions – 4 Processing suites • Wireless (encrypted) connection to Windows, Linux or Mac • $399 desktop or $499 for Bluetooth Smart (coming soon) • www.emotiv.com 35
  48. 48. ‹#› InteraXon’s Muse • 7 dry EEG sensors via Bluetooth • Focuses on “brain fitness” • 4 hour battery • Basic Mac & Windows SDK now, full SDK coming soon (incl. iOS & Android) • Works with Windows, OS X, iOS, Android, etc. • $299 available now • choosemuse.com 36
  49. 49. ‹#› OpenBCI • Open source hardware & software • Bluetooth, Arduino, EEG • 8 wet Electrodes per board (can daisy chain boards) • Full open EEG & BCI platform • $449 - $799 (now shipping) • www.openbci.com • No iOS or Android yet 37
  50. 50. ‹#› Emotiv Insight • Expected 2015 • Starting at $299 • 5+2 dry sensors • Bluetooth 4.0 LE (Smart) • Battery Life: 4+ hours • SDK: Android, iOS, Mac, Linux and Windows Platforms www.emotiv.com -­‐ Comparison http://bit.ly/1cfliTn 38
  51. 51. ‹#› No More Woof (“work in progress”) • EEG for Dogs • Recognize brain patterns and match with mental states: • Hungry, Playful, Curious, etc. • Translates to human • By Nordic Society for Invention and Discovery (NSID) • Raised $22K on IndiGoGo http://www.nordicinvention.com/nmw.html 39
  52. 52. More Details on the Emotiv EPOC
  53. 53. ‹#› Further Specs • Channels: ◦ AF3, AF4, F3, F4, F7, F8, FC5, FC6, P3 (CMS), P4 (DRL), P7, P8, T7, T8, O1, O2 • Battery: ◦ 12 hours operation • Sampling rate: ◦ ~128Hz (2048Hz internal) • Resolution: ◦ 16 bits (14 bits effective) 1 LSB = 0.51μV • Bandwidth: ◦ 0.2 - 45Hz, digital notch filters at 50Hz and 60Hz • More information: ◦ http://emotiv.wikia.com/wiki/Emotiv_EPOC X Int. 10-­‐20 locations
  54. 54. ‹#› Head Rotation Suite • Built in solid state gyroscope • Tracks angular velocity of head movement ◦ Yaw & Pitch ◦ No roll detection • Separate from EEG X
  55. 55. ‹#› Expressiv Suite • Detect facial expressions ◦ Eyelid & eyebrow positions – Wink, blink, furrow ◦ Horizontal eye movement ◦ Smile, laugh, clenching & smirking • EEG sensors picking up signals to muscles (not brain waves) • Very fast (10ms) 41
  56. 56. ‹#› Uses for Facial Detection • Realistic avatar expressions • Anything requiring fast input • Facial gesture detection • Basic eye tracking • Fast input & responses 42
  57. 57. ‹#› Affectiv Suite • Detect emotions / mental states ◦ Excitement ◦ Engagement or Boredom ◦ Meditation ◦ Frustration • Passive detection • Variable levels of each • Short & long term tracking 43
  58. 58. ‹#› Uses for Emotion Tracking • Realtime media evaluation • Stress analysis and management •Mood monitoring • For more accuracy combine with other data sources • Eye tracking, heart rate, etc. 44
  59. 59. ‹#› Cognitiv Suite • Detect conscious thoughts • Requires training • Detect 13 thought patterns with intensity level • Left, right, up, down, forward, pull • CW, CCW, left, right, sway-back, sway-forward, Disappear • Not as fast as Expresiv 45
  60. 60. ‹#› Uses for Cognitive Thought Identification •Whole new input method • Variable intensity for fine control • Observe subconscious responses •More training improves accuracy • Improves the model • Trains your brain to focus 46
  61. 61. ‹#› The Pieces 47 Emotiv EPOC Brain-Computer Interface Google Glass Wearable Computer Parrot.AR Drone 2 Simi-Autonomous
  62. 62. ‹#› About Appmethod • Integrated Development Environment • Multi-platform visual designer • Native compiler for Win32, Win64, OS X, iOS & Android • Framework for rapid prototyping and accelerated development with C++ or Object Pascal • Architecture allowing shared code & UI across platforms • Advanced language features like ARC, Generics, etc. 48 www.appmethod.com Starts at $25/mo per platform or C++ Android Phone for Free
  63. 63. ‹#› Laptop EDK My App How it Works 49 Brain Electrical Signals Wireless Signal AT*REF=1,290718208 Feedback Provides WiFi Hotspot
  64. 64. Demonstration Emotiv EPOC SDK + Parrot AR.Drone + Google Glass Tw eet @ A ppm e t h o d and #IoT or G+, LinkedIn, Facebook, etc.
  65. 65. ‹#› Emotiv EPOC SDK • Platforms ◦ Windows ◦Mac OS X ◦ Ubuntu ◦ Fedora ◦ iOS & Android coming soon 51 • Languages ◦ C / C++ / Objective-C ◦ C# (.NET) ◦ Python ◦ Object Pascal / Delphi ◦ Java coming soon
  66. 66. ‹#› MindTunes • Three disabled musicians • Collaborated with DJ Fresh and Julien Castet (tech) • Created dance track • http://youtu.be/HyF4ZxGhPHw • Full documentary • http://youtu.be/PgfxKZiSCDQ 52
  67. 67. More Information • Email: jim.mckeeth@embarcadero.com • Emotiv EPOC: www.emotiv.com • Parrot AR.Drone: projects.ardrone.org • Appmethod: www.appmethod.com • My blog: delphi.org • Full Slide Deck: http://www.slideshare.net/jimmckeeth/jim-mc-keeth- ‹#› wearable-thought-input • Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC 53

×