The Mind and Consciousness As an Interface

The Mind and Consciousness As an Interface Julian Bleecker & Nicolas Nova SXSW12 | Austin, TXAfter Julian presented the cultural backdrop for mind and consciousness interfaces, I want to focus on the hereand now, explain you the technologies and discuss the interaction design opportunities and limits

Adam Wilson’s EEG-generated tweet

Cortical control of a prosthetic arm for self-feeding by Meel Velliste, Sagi Perel, M. ChanceSpalding, Andrew S. Whitford & Andrew B. Schwartz http://www.youtube.com/watch?v=gnWSah4RD2E

Neurowear - necomini: http://neurowear.com/“Necomimi is the new communication toolthat augments the human bodies and abilities.This cat’s earshaped machine utilizes brainwavesand expresses your emotional state before you start talking”

Are these interfaces real?Yes, they all are

BCI: Brain-Computer Interfaces “create a direct communication pathway between a human brain and any external devices like computers” “BCI are system that provide direct translation of the activity of the human brain into command which can control device in real time.”Research started with disabled people... now it expands to any kind of users ( as usual in thehistory of innovation... “augmentation” )

“direct interaction”A common trope in human-object interactions: lever / mouse / keyboard / touch / voice /physiological

Process > invasive technologyThere are two ways to do it1. Invasive technologies: sensors directly implanted in the brain:+ = high temporal and spatial deﬁnition- = require surgery, cannot be moved, cover only limited portion of the brainPicture source: Nick Ramsey (http://www.nick-ramsey.eu/pics/surgery.jpg)

Process > non-invasive > EEGThis is the other possibility, no surgery. For the signal acquisition there are many different measurementmethods such as electroencephalography (EEG), magnetoencephalography (MEG), functional magneticresonance imaging (fMRI) or near-infrared spectroscopy (NIRS).

Process > non-invasive > fMRIThis is what you will need for functional magnetic resonance imaging (fMRI)fMRI measures changes in blood flow to various areas of the brain over time. It is believed that blood flows tobrain tissue that is active. So, the reasoning goes, if we put someone in the scanner, and ask them to think about,say, playing tennis, then blood should flow to the part of the brain that is involved with thinking about tennisplaying.

Process > non-invasive > EEGEEG is the most common., electrodes placed on the scalp measure the weak electrical potentials generated bythe brain activity+: no surgery-= noise, low spatial resolution (2-3cm 7-11 inches accuracy, which corresponds to the location of areas ofspeciﬁc brain activities: moving your hands versus listening to music)Most of the research is in there: data acquisitionCredit photo: http://www.bsp.brain.riken.jp/Photos/EEG%20Acquisition%20-%20%2064-QuickCap%20NeuroScan%20+%20256-Geodesic%20Net%20EGI%20-%20LABSP,%20BSI,%20Riken.jpg?w=300&h=300

Process > non-invasive > “dry cap”

Process > making sense of this1. Detect speciﬁc mental state or activity in certain areas of the brain (you can then say the user isreading but it’s almost impossible to tell what)2. Train users to create certain brain activity and use it instead of a motor movement (moving thehand)

Process > making sense of this delta wave: slow waves, sleep theta wave: drowsiness or arousal alpha wave: relaxed, reﬂecting beta wave: alter, working, anxious thinking, concentration

Process > making sense of thisIt was to difficult for us to resist presenting you this domain AND NOT USING the graphicemployed by researchers

Typology of applicationsVarious domains: gaming, spelling applications,2D cursor control, relaxation tool, game, accessto consciousness/dreams, serious games/ braintraining programs (arithmetic exercises, numberor letter recognition, mental card games), brainto brain communication, mind-controlledwhatever, zen-like interfaces

Typology > touch-free interfaceMind-controlled parachute (Sky1 HD TV show)

Typology > touch-free interface for hostile environment Mind-controlled parachute (Sky1 HD TV show)Action at distance: http://sky1.sky.com/gadget-geeks/mind-controlled-parachuteurinal-video-game

Typology > thought pattern recognition Mental state identiﬁcation to deliver dynamically adapted content/responseAdaptive Brain-Computer Interface by Audrey Girouard (Tufts University: “Difficulty levels during video game play.Distinguishing difficulty levels could prove to be an interesting input signal, on which to adapt the interface.Theexperiment presented the user with two levels of difficulty of an arcade game (Pacman). Data from nineparticipants shows we can discriminate well between the subject playing or resting (94% accuracy,with chance at50%), as well as discriminate between two difficulty levels and rest periods (77% accuracy,with chance at 33%),which shows potential for use in an adaptive interface. I investigated the data using both statistical analysis andmachine learning classification”

Typology > modern day lie detectorNeuroscientist Daniel Langleben. This image shows average brain regions for 22 subjects during testing. Blueareas represent brain regions more active when telling the truth, red areas, when lying.

Typology > neurofeedback Practice generating the state of mind most beneﬁcial to sport/activityPLX Wave - Xwave headset: http://www.plxwave.com/By safely detecting your brains rhythm through a small sensor contact gently placed on the skin ofyour forehead, XWave will let you be able to control and ﬂoat objects in video games by simplythinking about it, or train your mind to focus and relax on command.

Interaction design repertoire Explicit versus implicit user interactions (or user-driven versus stimulus-driven interactions) > “Vocabulary of interaction”: control, detect Synchronous versus asynchronous (as usual most of the applications focus on the real-time). Detection of certain cognitive states / brain activity Stand-alone brain-computer interface or BCI+other physiological data (hearbeat, turning one’s head...)Design parameters we can play with

Interaction design repertoire Language-related cognitive states Implicit Explicit Motor-related cognitive statesDesign parameters we can play with

1. Cognitive state versus semanticFinding general cognitive processes is not mind reading! Researchers can tell if a user isprocessing language but they we cannot easily determine the semantics of the content. Thereare projects that trying doing so but it’s really far-fetched into the future (big material, lot ofnoise).

1. Cognitive state versus semantic Presented clip Clip reconstructed from brain activityAn exception though: movie clip reconstruction using fMRI: The left clip is a segment of a Hollywood movie trailerthat the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brainactivity measured using fMRI.Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies by Shinji Nishimoto, AnT. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, Jack L. Gallan (UC Berkeley): http://dx.doi.org/10.1016/j.cub.2011.08.031

but this is what you will need for this (not to mention the algorithm)

2. Training usersUsers have to learn to intentionally manipulate their brain signals, which is not trivial.Two options 1. users are given speciﬁc cognitive tasks such as motor imagery to generate measurablebrain activity, 2. Operant conditioning, provides users with continuous feedback as they try to controlthe interface.

2. Training usersMain problem = even with feedback it’s hard to see what you can do with your mind tocontrol the attention interface on the upper left-hand corner!

3. Signal versus noise: seamful design?Noise = background noise + lots of data (some from the required brain activity + some from otheractivities like moving your head/hand) + as people grow tired. But we can maybe adopt a seamfuldesign approach.

4. Taking design context into accountWhere are you going to use these BCI helmet? Most of the pictures about these projects showlab pics... and we know that design is about context and taking into account contextualparameters. (This is the “board of imagination”, a mind-controlled skateboard by ChaoticMoon Lab)http://www.chaoticmoon.com/labs/chaotic-moon-labs-board-of-imagination/

5. Context and devices Specific type of app Specific type of app Specific type of appDifferent paths... depending on context, you can find about different activity

Conclusion> The “direct translation” trope and how to gobeyond it> We need to build an interaction design perspective,ask design issues, not only address technologicalproblems> What’s the equivalent of the blue screen of deathfor BCI? What happens for social norms in the long-run?

Thanks!julian@nearfuturelaboratory.comnicolas@nearfuturelaboratory.com

http://nearfuturelaboratory.com	1458
http://lanyrd.com	36
http://www.scoop.it	29
http://feeds.feedburner.com	13
http://bo.lt	4
http://www.hanrss.com	3
http://107.22.196.6	3
http://www.alphawavesandbeyond.com	2
http://stream.olishaw.com	2
http://translate.googleusercontent.com	2
http://www.nearfuturelaboratory.com	2
http://www.newsblur.com	1
http://storify.com	1
http://127.0.0.1	1
http://liftlab.com	1
http://www.twylah.com	1

The Mind and Consciousness As an Interface

by nicolas nova

Accessibility

Categories

Upload Details

Usage Rights

16 Embeds 1,559

Statistics

The Mind and Consciousness As an Interface Document Transcript