Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Neuromechanics of a Button Press: A talk at CHI 2018, April 2018

277 views

Published on

To press a button, a finger must push down and pull up with the right force and timing. How the motor system succeeds in button-pressing, in spite of neural noise and lacking direct access to the mechanism of the button, is poorly understood. This paper investigates a unifying account based on neurome- chanics. Mechanics is used to model muscles controlling the finger that contacts the button. Neurocognitive principles are used to model how the motor system learns appropriate muscle activations over repeated strokes though relying on degraded sensory feedback. Neuromechanical simulations yield a rich set of predictions for kinematics, dynamics, and user performance and may aid in understanding and improving input devices. We present a computational implementation and evaluate predictions for common button types.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Neuromechanics of a Button Press: A talk at CHI 2018, April 2018

  1. 1. Neuromechanics Antti Oulasvirta, Sunjun Kim, and Byungjoo Lee bit.do/neuromechanics of a Button Press Related papers at CHI 2018: 1. Control-theoretic Models of Pointing Tue 9-11.30 517C 2. Impact Activation Improves Rapid Key Pressing Mon 16.30-17.50 514AB 3. Moving Target Selection: A Cue Integration Model This session
  2. 2. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics10x slowdown Long-term goal: A theory of input Design Feedback Goals Anatomy Skill Movement Efficiency Efficacy Effort
  3. 3. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Existing theories Information theory Human performance Control theory Cognitive models Characterize Intrapolate Poor transfer Change in device or task insists on data collection or manual task modeling
  4. 4. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics The brain’s point-of-view
  5. 5. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Design Anatomy Goals Feedback Kinematics Dynamics Precision Effort Human-like responses Adapt A generative approach
  6. 6. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics The physical The neural What is neuromechanics? Neural principles of motor control in biomechanical systems The physiological [Enoka 2009]
  7. 7. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Input is control of mediated sensations The brain Peripheral nervous system Sensory nervous system Action potential Physical stimulation Feedforward Feedback Limbs Sensory organs Device
  8. 8. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Probabilistic internal model Sensory signals Activation signal Cue integration Integrated percept Perceptual control task Prediction error Prediction Muscle, bone, tissue, device, sensory organs Overview of the theory
  9. 9. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Theory: Elements Biomechanical simulation Cue integration Probabilistic internal model Perceptual control [Enoka 2009] [Sung-Hee Lee 2009] [Ernst 2004, 2006] [Powers 1973, 2009] [Clark 2013] [Hohwy 2016]
  10. 10. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Too brief for closed-loop control Button mechanism not perceived Ephemeral sensations Noisy neuromuscular system 10x slowdown Precise time and force Effective skill transfer Ability to adapt and recover Button- pressing: A miracle
  11. 11. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Random movements are unsuccessful
  12. 12. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Biomechanics simulation Noisy muscle activation signal Mechanoreceptive sensor Proprioceptive sensor
  13. 13. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Silicone finger tip Pressure sensor Hill type muscle Joint angle sensor Real button Robotic implementation + Noise + Noise+ Noise
  14. 14. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics No motor noise à super-human performance
  15. 15. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Sensory signals Cue integration Perceptual center eeds a threshold value. This ls: visual and auditory (beep). o-beep delays are assumed to utation of p-Centers m is connected to four extero- ion, proprioception, audition, ity i produces a p-center pci. sfer of a neural signal evoked hanoreceptors. We are espe- noreceptors on the finger pad ming of a button press. Slowly ensitive to coarse spatial struc- flat top surface of the button), ers respond to motion. Kim in signals from the fingertip ion, and jerk from the finger t force and indentation have d force correlates highly with use maximum likelihood estimation (MLE) to obtain estimate of pco. For another implementation of cue inte see [35]. In MLE, assuming that a single-cue estim unbiased but corrupted by Gaussian noise, the optimal s for estimating pco is a weighted average [16, 17]: pco = Â i wi pci where wi = 1/s2 i Âi 1/s2 i with wi being the weight given to the ith single-cue es and s2 i being that estimate’s variance. Figure 6 sho emplary p-center calculations: signal-specific (pci) an grated p-centers (pco) from 100 simulated runs of NEU CHANIC pressing a tactile button. Note that absolute ences among pci do not affect pco, only signal varian The integrated timing estimate is robust to long delays auditory or visual feedback. This assumption is base study showing that physiological events that take place q within a few hundred milliseconds, do not tend to be over- nor underestimations of event durations [14]. Maximum likelihood estimator “When was the button activated?” Modality-specific noise variances Proprio- ceptive Tactile Integration is sensitive to how reliable the cues are
  16. 16. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics No mechanoreceptive feedback signal
  17. 17. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Perceptual control task Choose (1) a motor command and (2) expected perception that minimizes this objective: The oned ntin- prio- ion), Gaus- form error tiva- cuss esult ptual enter com- f cue (GP) and e the with signal offset µ, signal amplitude t, and duration s of the agonist (A+) muscle. We have set physiologically plausible extrema (min and max) for the activation parameters. Note that this formulation assumes that the antagonist muscle re- sists motion passively. More determinate pull-up motion can be achieved by adding similar parameters for the antagonist muscle (A-). The objective is to determine motor command (q) and as- sociated estimate of perceived button activation (pce) that minimize error: min q,pce EP(q, pce)+EA(q)+EC(q)+wFM(q) (2) where EP is perceptual error, EA is error in activating the button, and EC is error in making contact (button cap not touched). FM is muscle force expenditure computed from the Hill muscle model (see below), and w is a tuning factor. We assume that activation and contact errors are trivial to perceive. Therefore, EA and EC are binary: 1 in the case of error and 0 otherwise. Perceptual error EP is defined as distance (in time) between expected p-center pce and observed p-center pco: Objective function Perceptual error Activation failure (binary) Contact failure (binary) Muscle force User goal: Lightness vs. precision
  18. 18. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Perceptual error Minimize error between expected and perceived activation time
  19. 19. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics No button- activation term
  20. 20. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics No muscle effort term
  21. 21. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics A probabilistic internal model learns motor outputs that minimize the perceptual objective A Bayesian optimizer with a Gaussian Process prior Approximate Bayesian Computation (ABC)Approximate Bayesian Computation (ABC) Approximate Bayesian Computat ... Finds a good button press after 10-20 trials
  22. 22. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Model parameters Table 1. Model parameters. Button parameters here given for physical buttons. Task parameters (e.g., finger starting height) are given in text. f denotes function Variable Description Value, Unit Ref. fr Radius of finger cone 7.0 mm fw Length of finger 60 mm rf Density of finger 985 kg/m3 cf Damping of finger pulp 1.5 N·s/m [64] kf Stiffness of finger pulp f, N/m [65] wb Width of key cap 14 mm db Depth of key cap 10 mm rb Density of key cap 700 kg/m3 cb Damping of button 0.1 N·s/m ks Elasticity of muscle 0.8·PCSA [38] kd Elasticity of muscle 0.1·ks [38] kc Damping of muscle 6 N·s/m [38] PCSA Phys. cross-sectional area 4 cm2 L0ag, L0an Initial muscle length 300 mm sn Neuromuscular noise 5·10 2 sm Mechanoreception noise 1·10 8 sp Proprioception noise 8·10 7 sa Sound and audition noise 5·10 4 sv Display and vision noise 2·10 2 system. We have set these parameters manually in order to reproduce certain basic effects: Neuromuscular noise, which reflects the joint additive contribution of neural and muscular Physically measurable Tuned based on literature
  23. 23. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Simulation workbench (MATLAB) bit.do/neuromechanics But are the outputs realistic?
  24. 24. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Precision & success: Predictions Figure 7. Data collection on press kinematics: A single-subject study. High-fidelity optical motion tracking was used to track a marker on the finger nail. A custom-made single-button setup was created using switches and key caps from commercial keyboards. SIMULATIONS: COMPARING BUTTON DESIGNS Most precise Less precise Least precise High success High success Low success Push- button Touch Mid-air Order supported by literature
  25. 25. Data collection: A single-subject study “Press rhythmically in a manner natural for you”
  26. 26. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Mid-air button: Kinematics [Torre & Balasubramaniam 2009] Human Model Similar to our data and literature
  27. 27. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Dynamics Force-displacement curves Predicted peak forces 1.5-2.9N Force ranges similar to literature cle force–displacement behavior for a tactile with an effort-minimizing term in the objective task performance (performance in button activation). We co clude, that although much work remains to be done, the resul support the ’optimal black box’ assumption. And many mor analyses could done, such as looking at the effect of unreliab feedback, oscillation of the finger tip, such as when walkin or the effects that impairments like essential tremor have. FUTURE WORK Modeling latent neural and cognitive constructs, such as nois poses a scientific challenge for future research. Change i noise parameters has a large and poorly understood effect o dynamics downstream. However, without noise, a button ca be activated with arbitrary precision. For example, cuttin sensory noise parameters to 10 9 reduces perceptual error t the order of 1.5·10 6 s. Our noise model was tuned manuall to reproduce some standard findings on sensory modalities. T “Light touch” Force(N)
  28. 28. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Touch buttons: Kinematics Displacement-velocity curves e 8. Displacement–velocity curves for four button types from single-subject recordings (top) and simulations (bottom Shape similar except in release Human Model Similar result for push-buttons
  29. 29. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Signal-dependent noise Muscle and joint models Force perception Noise parameter identification Limitations
  30. 30. Neuromechanics of a Button Press Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics Neuro- science PhysicsPhysiology Machine learning Biomech. simulation EE and signal processing A unifying account A generative simulation Can it be made to work beyond buttons?
  31. 31. bit.do/neuromechanics Antti Oulasvirta, Sunjun Kim, Byungjoo Lee Acknowledgements: Jong-In Lee, Aleksi Pesonen, Yunfei Xiu, and Crista Kaukinen Related papers at CHI 2018: 1. Control-theoretic models of Pointing Tue 9.00 517C 2. Impact Activation Improves Rapid Key Pressing Mon 16.30 514AB 3. Moving Target Selection: A Cue Integration Model This session Figure 8. Displacement–velocity curves for four button types from single-subject recordings (top) and simulations (bottom). Data Matlab codeRobot

×