Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a Thought Controlled Drone

5,002 views

Published on

Overview of building a Thought controlled drone using a Brain Computer Interface. The drone is the Parrot AR.Drone 2.0 and the BCI is the Emotiv EPOC.

  • Have u ever tried external professional writing services like ⇒ www.HelpWriting.net ⇐ ? I did and I am more than satisfied.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • How Do Social Media Jobs Pay $35 Per Hour? ➤➤ http://t.cn/AieX6y8B
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • WAIT!!! Do this simple 60-second habit to reverse type 2 diabetes(& lose 1 pound of belly fat every 72 hours). ➤➤ https://bit.ly/2n5cFHd
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I love you Ben you’ve literally saved my life! Lets just say I was in a lot of trouble financially until I found you.Now I’m in complete AWE every time I open my betting account it’s like the numbers aren’t real, that’s why I’m constantly withdrawing the cash lol.God bless you Ben! and thank you so much for allowing me access to this amazing service. ■■■ http://t.cn/A6vAMkWt
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • What if you had a printing press that could spit out hundred dollar bills on demand? Do you think that would change your life? ■■■ http://t.cn/AisJWzdm
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Building a Thought Controlled Drone

  1. 1. Building a 
 Thought-Controlled Drone Jim McKeeth, Embarcadero Technologies jim.mckeeth@embarcadero.com
 @JimMcKeeth Presented at InterDrone Conference Sept 9th, 2015
  2. 2. About Jim McKeeth • Lead World Wide Developer Evangelist • Longtime developer • Object Pascal, Java, C#, JavaScript, Objective-C, etc. • Invented and patented swipe to unlock in 2000 • US Patent # 8352745 & 6766456, etc. • Improvisational performer with ComedySportz Boise • I am not a neuroscientist, but I find this all fascinating!
  3. 3. • Founded 1993 • Provides Architect, Database and Developer Tools • 3.2 Million Customers including 97% of Fortune 2000 • 35+ Product Awards – Consistent Innovation • 500+ Employees in 29 Countries • www.embarcadero.com
  4. 4. Agenda • Drone technical details • Headset technical details • Drone API • Headset SDK • Concerns, Challenges and the Future
  5. 5. More Information • My email: jim.mckeeth@embarcadero.com • My blog: delphi.org • This Slide Deck
 http://www.slideshare.net/jimmckeeth/build-brain-controlled-drone • Overview Slide Deck 
 http://www.slideshare.net/jimmckeeth/jim-mc-keeth-wearable-thought-input • The Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC
  6. 6. “In five years, science fiction will be real!” — Claus Torp Jensen (@ClausTorpJensen) Senior Technical Staff Member and Chief Architect API Economy at IBM (@ibmapimgt) What will the world be like 
 in five years?
  7. 7. The Pieces Emotiv EPOC Brain-Computer Interface Parrot.AR Drone 2 Simi-Autonomous Connected App Platform
  8. 8. Drone Technical Details
  9. 9. The Drone: Parrot AR.Drone 2.0 • Price: $299.99 • Runs Linux internally, but not user accessible • Contains it’s own WiFi access point • Connect via UDP & TCP • Send control strings • Maintain transmission rate to keep alive • Can also receive status and video back
  10. 10. Completely Rebuildable HD Battery $60 Gears+Shafts $15 Tools $20 Cross bars $30 Screws $7 Propellers $12 4x Engines $200 Bottom structure $60 Main board $110 Nav board $70 Charger $40 Outdoor haul $25 ———————— Total $650
  11. 11. New contender: Parrot Bebop • Price: $499.99 • Lots of upgrades, especially camera • Also contains own WiFi access point • Whole new SDK • iOS, Android & Linux • Published Nov 2014 • Incompatible with AR.Drone 2.0
  12. 12. SDK vs. API • Parrot AR.Drone 2.0 provides both an API and SDK (Bebop has just an SDK at this point.) • An SDK is a software library that does additional work locally - makes it easier to handle video streaming, etc. • An API is a specification of how to connect and control. • SDK works with specific tools and platforms. • Typically SDK uses API. • My examples use the API so it works across platforms.
  13. 13. Brain Computer Interface (BCI) Headset Options
  14. 14. Electroencephalography (EEG) • Measures electrical activity along the scalp. • Detects voltage fluctuations resulting from ionic current flows within neurons of brain. Image source: https://commons.wikimedia.org/wiki/File:EEG_cap.jpg
  15. 15. Neural Oscillation / Brain WavesBeta 15-30Hz Waking Consciousness and Reasoning Alpha 7-14Hz Deep Relaxation Delta 4Hz Deep Sleep Theta 4-7Hz Light Meditation And Sleeping Mu 8-13Hz Voluntary Movement Gamma 30-100Hz Conscious Perception Waveforms by Hugo Gamboa
  16. 16. NeuroSky’s MindWave (also in XWave headsets) • EEG with 1 dry electrode • Measures “attention,” “meditation” and eye-blinks • Bluetooth communication • SDK for iOS, Android, PC & Mac • $129.99 to $79.95 retail neurosky.com • Hacking: http://makezine.com/2011/05/10/behind- the-brain-blinker/
  17. 17. InteraXon’s Muse • 7 dry EEG sensors via Bluetooth • Focuses on “brain fitness” • 4 hour battery • Android, Mac & Windows SDK now, 
 iOS coming soon • Works with Windows, OS X, iOS, Android, etc. • $299 at choosemuse.com
  18. 18. OpenBCI • Open source hardware & software • Bluetooth, Arduino, EEG • 8 wet Electrodes per board (daisy chainable) • Full open EEG & BCI platform • SDK provides EEG data • www.openbci.com starting at $449 • No iOS or Android yet
  19. 19. Emotiv EPOC • 16 wet electrodes • 14 EEG electrodes + 2 reference electrodes • Gyroscope to read head movements • 4 mental states, 13 conscious thoughts & facial expressions – 4 Processing suites • Wireless (encrypted) for Windows, Linux or Mac • $399 for Desktop • $499 for Bluetooth Smart (EPOC+) • www.emotiv.com
  20. 20. Emotiv Insight • Expected eventually • Starting at $299 (estimate) • 5 dry sensors + 2 reference • Bluetooth 4.0 LE (Smart) • Battery Life: 4+ hours • SDK: Android, iOS, Mac, Linux and Windows Platforms • www.emotiv.com • Comparison http://bit.ly/1cfliTn
  21. 21. Emotiv EPOC SDK Overview
  22. 22. Expressiv Suite • Detect facial expressions • Eyelid & eyebrow positions • Wink, blink, furrow • Horizontal eye movement • Smile, laugh, clenching & smirking • EEG sensors picking up signals to muscles (not brain waves) • Very fast (10ms) Image Source: https://commons.wikimedia.org/wiki/File:Duchenne-FacialExpressions.jpg
  23. 23. Uses for Facial Detection • Anything requiring fast input • Facial gesture detection • Basic eye tracking • Fast input & responses Image Source: https://commons.wikimedia.org/wiki/File:Duchenne-FacialExpressions.jpg
  24. 24. Affectiv Suite • Detect emotions / mental states • Excitement • Engagement or Boredom • Meditation • Frustration • Passive detection • Variable levels of each • Short & long term tracking Image Source: https://commons.wikimedia.org/wiki/File:Drama-icon.svg
  25. 25. Uses for Emotion Tracking • Realtime media evaluation • Stress analysis and management • Mood monitoring • For more accuracy combine with other data sources • Eye tracking, heart rate, etc. Image Source: https://commons.wikimedia.org/wiki/File:Drama-icon.svg
  26. 26. Cognitiv Suite • Detect conscious thoughts • Requires training • Detect 13 thought patterns with intensity level (4 at a time) • Left, right, up, down, forward, pull, CW, CCW, left, right, sway-back, sway-forward, Disappear • Not as fast as Expresiv Image Source: https://commons.wikimedia.org/wiki/File:ThinkingMan_Rodin.jpg
  27. 27. Uses for Cognitive Thought Identification • Whole new input method • Variable intensity for fine control • Observe subconscious responses • More training improves accuracy • Improves the model • Trains your brain to focus Image Source: https://commons.wikimedia.org/wiki/File:ThinkingMan_Rodin.jpg
  28. 28. Brain-Computer Interface Internet of Things Autonomous Vehicles expectations Speech-to-Speech Translation time Plateau of Productivity Slope of EnlightenmentTrough of Disillusionment Peak of Inflated Expectations Innovation Trigger Plateau will be reached in: more than 10 years5 to 10 years2 to 5 yearsless than 2 years obsolete before plateau Gartner's 2014 
 Hype Cycle for Emerging Technologies August 11, 2014 Smart Advisors Data Science Prescriptive Analytics Neurobusiness Biochips Affective Computing Smart Robots 3D Bioprinting Systems Volumetric and Holographic Displays Software-Defined Anything Quantum Computing Human Augmentation Connected Home Quantified Self Smart WorkspaceVirtual Personal Assistants Digital Security Bioacoustic Sensing Natural-Language Question Answering Consumer 3D Printing Cryptocurrencies Complex-Event Processing Big Data In-Memory Database Management Content Analytics Hybrid Cloud Computing Gamification Augmented Reality Machine-to-Machine Communication Services Mobile Health Monitoring Cloud Computing NFC Virtual Reality Gesture Control In-Memory Analytics Activity Streams Enterprise 3D Printing 3D Scanners Consumer Telematics Speech Recognition Wearable User Interfaces
  29. 29. Using the AR.Drone API
  30. 30. Command Strings • Connect via WiFi access point to 192.168.1.1 UDP port 5556 • Command string format: AT*[cmd]=[seq],[arg array] • Sequence number must not repeat • Argument array is comma separated typically • Floating point numbers must be encoded as decimals (more later) • Send command every 30 ms (>2 seconds = session dropped)
  31. 31. Command String Examples • FlatTrim: AT*FTROM=[seq] • Movement: AT*PCMD=[seq],1,Roll, Pitch, Vert, Yaw • Values of 1.0 to -1.0 • Use 0 in all values to hover • Takeoff: AT*REF=[seq],290718208 • Land: AT*REF=[seq],290717696 • Emergency: AT*REF=[seq],290717952
  32. 32. Float as Integer • Single precision floating point number is 32 bits • Integer is 32 bits • Map IEEE-754 bit value from float to integer • -0.8 = 0xBF4CCCCD = 10000001011001100110011001 = 1085485875
  33. 33. Drone Concerns • Manual override - keyboard override BCI, just in case • Send emergency command to clear error state • Start with FlatTrims to level drone • Use state machine to send command 30 times a second 1. Keyboard override 2. BCI Input 3. Hover
  34. 34. ToDo with Drone • Process NavData (sent back on UDP port 5554) • Get battery data to know remaining flight time, errors, etc. • Process Video stream (sent back on UDP port 5555) • Display on Google Glass for HUD • Find use for drone “Animations” like flips • Maybe use LED animations when receiving BCI input
  35. 35. Using Emotiv EDK
  36. 36. SDK Format • On Windows a Visual C++ DLL provides the interface • Examples in VC++, C#.NET, Java, Python & Matlab • All my code is in Object Pascal • On OS X a dylib provides the interface • Examples in Qt & Xcode
  37. 37. Emotiv EDK Workflow • Call EE_EngineConnect to connect • Create event handler EE_EmoEngineEventCreate • Create state structure EE_EmoStateCreate • Define actions to listen for and then train those actions (more later) • Every 10 ms call EE_EngineGetNextEvent (more later)
  38. 38. Handle Cognitive Actions • If state is EDK_OK then call EE_EmoEngineEventGetType • If EventType is not EE_EmulatorError then call EE_EmoEngineEventGetUserId • If EventType is EE_CognitivEvent then it was a training event (next slide) • If EventType is EE_EmoStateUpdated then get EE_EmoEngineEventGetEmoState • Read ES_CognitivGetCurrentAction and ES_CognitivGetCurrentActionPower
  39. 39. Listen & Train • Listening: • Call EE_CognitivSetActiveActions with mask of Actions • Call EE_CognitivSetTrainingAction with the ID of the action to train • Call EE_CognitivSetTrainingControl with COG_START • On EE_CognitivEventGetType of EE_CognitivTrainingSucceeded call EE_CognitivSetTrainingControl with COG_ACCEPT
  40. 40. New Emotiv Xavier SDK
  41. 41. New Emotiv Xavier SDK • EPOC+ and Insight are Bluetooth Smart • Connect to mobile devices - iOS & Android • Allows to additional user definable cognitive actions • Still only listens for 4 cognitive actions at a time
  42. 42. Thought Controlled Drone
  43. 43. Thoughts to Movement • Moves while thought active, otherwise hover in position • Speed of movement = intensity of thought • How to allow more than 4 directions of movement? • Use Expressiv facial gesture to switch between sets • Future consideration: Use strobing grid of controls
  44. 44. Typing via BCI Jonathan Wolpaw
 Wadsworth Center
 Albany, NY ~15 seconds / key . . . . . . . . . A B C D E F . . G H I J K L . . M N O P Q R . . S T U V W X . . Y Z 1 2 3 4 . . 5 6 7 8 9 0 . . . . . . . . .
  45. 45. BCI Control Panel Example
  46. 46. Purpose of Feedback • Using a visual feedback gives the wearer something to focus on • Otherwise they often change their thought pattern when the drone is moving • Also gives them immediate awareness of what is picked up
  47. 47. Moverio BT-200 Product Overview Integrated Sensors Accelerometer Compass Gyro Head Set Camera Earbuds with Microphone +x +y +z
  48. 48. Epson Moverio BT-200 Specs Epson BT-200 Retail Price $699 Display Type Transparent Resolution 960×540 QHD Android v4.0.4 API Level 15 - Ice Cream Sandwich Camera VGA, 640x480 WLAN 802.11b/g/n 2.4GHz Bluetooth 3.0 micro USB v2.0 (On-The-Go) GPS P-GPS CPU TI OMAP 4460 (ARM Cortex A9 Dual Core 1.2GHz) RAM 1GB Flash 8GB microSD Max 32GB
  49. 49. Laptop EDK My   App Data Flow Electrical   Signals Wireless  Signal AT*REF=1,290718208 Provides   WiFi     Hotspot Feedback
  50. 50. Q & A
  51. 51. More Information • My email: jim.mckeeth@embarcadero.com • My blog: delphi.org • This Slide Deck
 http://www.slideshare.net/jimmckeeth/build-brain-controlled-drone • Overview Slide Deck 
 http://www.slideshare.net/jimmckeeth/jim-mc-keeth-wearable-thought-input • The Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC

×