Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this document? Why not share!

Journal of Neurophysiology






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Journal of Neurophysiology Journal of Neurophysiology Document Transcript

  • Journal of Neurophysiology Copy of e-mail Notification z9k8250 Proof of your article (# J00760-6 ) from "Journal of Neurophysiology" is available for download ===== Dear Sir or Madam: Please refer to this URL address http://rapidproof.cadmus.com/RapidProof/retrieval/index.jsp Login: your e-mail address Password: ---- The file at the above URL address contains the following: • Proofreading marks • Reprint Order form • Copyedited page proof of your article The site contains 1 file. You will need Adobe Acrobat® Reader to read this file. Adobe Acrobat® Reader is a free software, available for user downloading at http://www.adobe.com/products/acrobat/readstep.html. After you print the PDF file of your paper, please read the page proof carefully and 1) clearly indicate all changes or corrections on the margin; 2) answer all queries (footnotes 1, 2, 3, etc.) listed on the last page of the PDF proof; 3) carefully proofread all/any tables and equations; 4) make sure that any special characters, such as Greek letters, especially µ (mu), have translated correctly; 5) If you have questions about figure quality, note your concerns on the margin of the relevant page. Please keep in mind that the final printed version will be of higher quality than the PDF proof and that the online version of the published article will appear identical. IMPORTANT: Do NOT make online alterations to the PDF itself. Corrections within the PDF, made with the Adobe Acrobat-provided tools will remain unseen by the Journal staff. This will result in the final article NOT reflecting your intended changes. ONLY the hand-marked PDF printout guarantees that the changes will be incorporated into the final text of the article. We encourage you to retain a copy of the proof with your corrections, should further changes and/or clarifications be required. IMPORTANT NOTES 1. To guarantee the placement of your article in the next available issue of the "Journal of Neurophysiology", please return the corrected set of PDF page proof via an overnight courier service to this address WITHIN 48 HOURS: Barbara A. Meckley DPS-Ephrata Cadmus Professional Communications 300 West Chestnut Street Ephrata, PA 17522 USA 717 445-8430 2. If you require a hard copy of your color image(s), you may request it by replying to this message. The color proof will then be provided to you at an additional fee of $75.00 per color figure, which will be added to the publication fees for your article. You must provide the appropriate figure number(s) and your mailing address in your reply if you wish to receive the hard copy of the color proof(s).
  • Journal of Neurophysiology Copy of e-mail Notification z9k8250 3. The filled out Reprint Form must be returned within 48 hours to the address below: Cadmus Reprints P.O. Box 751903 Charlotte, NC 28275-1903 For reprint inquiries, please contact Mary Leonard at 410 819-3912 (phone), 410-820-9765 (fax) or LeonardM@cadmus.com. If you have any problems or questions with your electronic proofs, please contact me. PLEASE ALWAYS INCLUDE YOUR ARTICLE NO. ( J00760-6 ) WITH ALL CORRESPONDENCE. Sincerely, Barbara A. Meckley Editorial Production Manager Journal of Neurophysiology 300 West Chestnut St. P.O. Box 497 Ephrata, PA 17522-0497 E-mail: meckleyb@cadmus.com
  • Journal of Neurophysiology 2007 Published by The American Physiological Society This is your reprint order form or pro forma invoice (Please keep a copy of this document for your records. This form is not for commercial ordering.) IMPORTANT Order form must be returned within 48 hours of receipt to avoid late charges. Orders received after 48 hours will be charged an additional fee of 25%. Orders received after 30 days will be charged an additional 50%. Reprints containing color figures are available only if ordered before the journal is printed. It is the policy of Cadmus Reprints to issue only one invoice per order. Please print clearly. Please return form whether reprints are ordered or not. Author Name _______________________________________________________________________________________________ Title of Article _______________________________________________________________________________________________ Issue of Journal______________________ Reprint # ___________ Manuscript #_____________ Publication Date___________ J00760-6 Number of Pages________________________ Color in Article? Yes / No (Please Circle) Symbol JN Please include the journal name and reprint number or manuscript number on your purchase order or other correspondence. Order and Shipping Information Reprint Costs (Please see page 2 of 2 for reprint costs/fees.) Shipping Address (cannot ship to a P.O. Box) Please Print Clearly ________ Number of reprints ordered $_________ Name _________________________________________ ________ Number of color reprints ordered $_________ Institution______________________________________ Street _________________________________________ Subtotal $_________ City ______________ State ______ Zip ___________ Add appropriate sales tax/GST to subtotal $_________ Country________________________________________ First address included, add $32 for each Quantity___________________ Fax _______________ additional shipping address $________ Phone: Day _________________ Evening ___________ E-mail Address _________________________________ Publication Fees (Please see page 2 for fees and descriptions.) Additional Shipping Address* (cannot ship to a P.O. Box) Page Charges: $70 per journal page $_________ Name _________________________________________ Color Figures: $400 per color figure $_________ Institution______________________________________ Hard copy color proof: $75 per figure $_________ Street _________________________________________ Toll-Free Link: $150 $_________ City ______________ State ______ Zip ___________ ___________ __________________________________ Country________________________________________ Member No. Member Signature Quantity__________________ Fax ________________ Total Publication Fees $__________ Phone: Day ________________ Evening ____________ E-mail Address __________________________________ TOTAL TO REMIT $__________ * Add $32 for each additional shipping address Payment and Credit Card Details (FEIN #:541274108) Invoice or Credit Card Information Invoice Address Please Print Clearly Enclosed: Personal Check ___________ Please complete Invoice address as it appears on credit card statement Institutional Purchase Order _________ Name ______________________________________ Credit Card Payment Details _________ Institution ______________________________________ Checks must be paid in U.S. dollars and drawn on a U.S. Bank. Department _____________________________________ Street ______________________________________ Credit Card: __ VISA __ Am. Exp. __ MasterCard City _________________ State _____ Zip ______ Card Number __________________________________ Expiration Date___________________________________ Country ______________________________________ Signature: _______________________________________ Phone __________________ Fax ______________ E-mail Address __________________________________ Please send your order form and purchase order or prepayment made Purchase Order No._______________________________ payable to: Cadmus Reprints Cadmus will process credit cards and Cadmus Journal P.O. Box 751903 Charlotte, NC 28275-1903 Services will appear on the credit card statement. Note: Do not send express packages to this location, PO Box. If you do not mail your order form, you may fax it to 410-820- 9765 with your credit card information. SIGNATURE REQUIRED: By signing this form the author agrees to accept responsibility for the payment of the mandatory page charges of $70 per page, reprints ordered, as well as any color charges, late payments, and split shipment charges. If the charges are billed to an institution, the author must assume the responsibility for making the necessary arrangements for the issuance of a formal institut ional purchase order. Otherwise, it is understood that the author will bear the cost of these charges. Failure to pay any of these agreed-upon charges could jeopardize future submissions. AUTHOR Signature __________________________________ Fax_______________________________________ Telephone __________________________________________ E-mail ____________________________________ ML-9-29-06 Page 1 of 2
  • Journal of Neurophysiology 2007 Published by The American Physiological Society REPRINT AND PUBLICATION CHARGES; Author rates only. Not to be used for commercial ordering Black and White Reprint Prices Color Reprint Prices Domestic (USA only) Domestic (USA only) # of Addl # of Addl 100 200 300 400 500 100’s 100 200 300 400 500 100’s Pages Pages 1-4 $216 $301 $386 $471 $556 $78 1-4 $315 $501 $685 $871 $1,055 $179 5-8 $293 $442 $594 $745 $894 $134 5-8 $393 $642 $894 $1,144 $1,394 $234 9-12 $377 $573 $774 $969 $1,167 $151 9-12 $477 $773 $1,074 $1,369 $1,668 $250 13-16 $452 $715 $978 $1,241 $1,506 $250 13-16 $553 $915 $1,279 $1,641 $2,006 $350 17-20 $526 $848 $1,167 $1,489 $1,805 $306 17-20 $626 $1,048 $1,468 $1,889 $2,305 $407 21-24 $612 $988 $1,364 $1,740 $2,116 $359 21-24 $712 $1,188 $1,664 $2,141 $2,616 $459 25-28 $686 $1,130 $1,572 $2,014 $2,456 $421 25-28 $786 $1,330 $1,872 $2,414 $2,956 $521 29-32 $774 $1,271 $1,780 $2,287 $2,795 $482 29-32 $874 $1,471 $2,081 $2,687 $3,295 $582 International (includes Canada and Mexico) International (includes Canada and Mexico) # of Addl # of Addl 100 200 300 400 500 100’s 100 200 300 400 500 100’s Pages Pages 1-4 $243 $339 $440 $538 $637 $90 1-4 $343 $539 $740 $938 $1,137 $190 5-8 $332 $508 $687 $864 $1,042 $162 5-8 $432 $708 $987 $1,264 $1,542 $262 9-12 $431 $666 $911 $1,143 $1,386 $191 9-12 $531 $866 $1,211 $1,543 $1,886 $291 13-16 $519 $836 $1,153 $1,471 $1,790 $306 13-16 $619 $1,036 $1,453 $1,871 $2,290 $406 17-20 $607 $996 $1,386 $1,773 $2,157 $374 17-20 $707 $1,196 $1,686 $2,173 $2,657 $474 21-24 $704 $1,162 $1,621 $2,079 $2,537 $440 21-24 $804 $1,362 $1,921 $2,479 $3,037 $540 25-28 $795 $1,331 $1,868 $2,408 $2,941 $517 25-28 $895 $1,531 $2,168 $2,808 $3,441 $617 29-32 $895 $1,502 $2,120 $2,736 $3,354 $592 29-32 $995 $1,702 $2,420 $3,136 $3,854 $692 Minimum order is 100 copies. For orders larger than 500 copies, Late Order Charges please consult Cadmus Reprints at 800-407-9190. Articles more than 90 days from publication date will carry an additional charge of $5.50 per page for file retrieval. Page Charges $70 per journal page for all pages in the article, whether or not TOLL-FREE LINK you buy reprints. A link can be created from a url of your choice to your article Color online so that readers accessing your article from your url can do so without a subscription. The cost is $150. This is especially Reprints containing color figures are available. If your article contains color, you must pay subsidized color charges of useful if your article contains electronic supplemental material. For more information, please click on this link: $400/fig. (reprint charge is $1000/fig for those who do not pay promptly), whether or not you buy reprints. These color charges http://www.the-aps.org/publications/sprooflink.pdf are waived for APS Members who are the first or last author of the paper. If you requested a hard copy color figure proof Ordering when you reviewed your S-proof, the charge is $75. Please fax your order form and purchase order to Shipping 410-820-9765. Prepayment of checks should be mailed to Shipping costs are included in the reprint prices. Domestic orders address below: are shipped via UPS Ground service. Foreign orders are shipped Cadmus Reprints via an expedited air service. The shipping address printed on an P.O. Box 751903 institutional purchase order always supersedes. Charlotte, NC 28275-1903 Multiple Shipments Note: Do not send express packages to this location. Orders can be shipped to more than one location. Please be aware FEIN #:541274108 Reprint Order Forms that it will cost $32 for each additional location. Please direct all inquiries to: and Purchase Orders or prepayments must State Sales Tax and Canadian GST Mary Leonard be received 48 hours Residents of Virginia, Maryland, Pennsylvania, and the District of 800-407-9190 (toll free number) after receipt of form. Columbia are required to add the appropriate sales tax to each 410-819-3912 (direct number) reprint order. For orders shipped to Canada, please add 6% 410-820-9765 (FAX number) Please return this Canadian GST unless exemption is claimed. LeonardM@cadmus.com form even if no reprints are ordered. Page 2 of 2
  • tapraid4/z9k-neurop/z9k-neurop/z9k00707/z9k8250d07a moyerr S 6 5/11/07 22:29 MS: J00760-6 Ini: Input-dlw J Neurophysiol 98: 000 – 000, 2007. First published April 11, 2007; doi:10.1152/jn.00760.2006. Decoding M1 Neurons During Multiple Finger Movements S. Ben Hamed,1 M. H. Schieber,1,2 and A. Pouget1 1 Department of Brain and Cognitive Science and the Center for Visual Science, University of Rochester, Rochester, New York; and 2 Departments of Neurology and Neurobiology, University of Rochester, School of Medicine and Dentistry, Rochester, New York Submitted 21 July 2006; accepted in final form 5 April 2007 Hamed SB, Schieber MH, Pouget A. Decoding M1 neurons during The situation becomes more complex, however, in decoding multiple finger movements. J Neurophysiol 98: 000 – 000, 2007. First the concurrent motion of multiple effectors. Finger movements published April 11, 2007; doi:10.1152/jn.00760.2006. We tested sev- provide an important example because one finger might move eral techniques for decoding the activity of primary motor cortex (M1) in isolation or multiple fingers might move together. One neurons during movements of single fingers or pairs of fingers. We report that single finger movements can be decoded with 99% solution to the problem of decoding these movements might be accuracy using as few as 30 neurons randomly selected from popu- to develop a separate estimator for each finger, but this ap- lations of task-related neurons recorded from the M1 hand represen- proach rests on the assumption that each finger is controlled tation. This number was reduced to 20 neurons or less when the independently of the others. In most natural behaviors, how- neurons were not picked randomly but selected on the basis of their ever, multiple fingers move together (for review, see Schieber information content. We extended techniques for decoding single and Santello 2004), suggesting that the pattern of neural finger movements to the problem of decoding the simultaneous activity needed to move the index finger, for example, may movement of two fingers. Movements of pairs of fingers were de- differ depending on whether the index moves in combination coded with 90.9% accuracy from 100 neurons. The techniques we with the middle finger, in combination with the ring finger, or used to obtain these results can be applied, not only to movements of in isolation. It is therefore unclear whether training multiple single fingers and pairs of fingers as reported here, but also to movements of arbitrary combinations of fingers. The remarkably decoders in parallel— one per finger— can lead to a high level small number of neurons needed to decode a relatively large repertoire of classification performance using a reasonably small number of movements involving either one or two effectors is encouraging for of M1 neurons. the development of neural prosthetics that will control hand We therefore examined several different methods of decod- movements. ing finger movements. Surprisingly, we found that 100 neurons are sufficient to decode movements of pairs of fingers with an accuracy of 90.9%. Moreover, the same estimator can be used INTRODUCTION to decode single finger movements with near-perfect ( 99%) accuracy using as few as 30 neurons. Decoding population neuronal activities has become an important topic in neuroscience. From a theoretical perspec- METHODS tive, the ability to read out neural patterns of activity can help us understand the structure of the neuronal code, how infor- This study describes analysis of data collected from two monkeys mation is distributed over several thousand neurons, and how it that have been the subject of previous reports (Georgopoulos et al. flows from one area to the next (Dayan and Abbott 2001; 1999; Schieber 1991; Schieber and Hibbard 1993). Methods used for Pouget et al. 2003). From a more clinical perspective, efficient behavioral training and data collection have been described in these reports and are summarized here as needed. All care and use of these decoding of neuronal population activity will be crucial for monkeys complied with the U.S.P.H.S. Policy on Humane Care and development of neuroprosthetics that are controlled on-line by Use of Laboratory Animals and was approved by the institutional neural activities. animal care and use committee. Several decoders have been used in previous studies, such as the population vector, the optimal linear estimator, the maxi- mum likelihood estimator, or the Bayesian estimator (Georgo- Behavioral task poulos et al. 1986; Pouget et al. 2003; Salinas and Abbott 1994; Each monkey was trained to perform visually cued individuated Seung and Sompolinsky 1993). All of these methods use a flexion and extension movements of the right hand fingers and/or population pattern of activity as input and return an estimate wrist (Schieber 1991). As the monkey sat in a primate chair, the right (or a probability distribution in the case of the Bayesian elbow was restrained in a molded cast, and the right hand was placed approach) of the encoded variable. In all cases, the encoded in a pistol-grip manipulandum that separated each finger into a variable is assumed to take only one value at any given time. different slot. At the end of each slot, the fingertip lay between two This is indeed a reasonable assumption in most cases. For microswitches. By flexing or extending the digit a few millimeters, the monkey closed the ventral or dorsal switch, respectively. The manipu- instance, in decoding a reaching movement, the hand can be landum, in turn, was mounted on an axis that permitted flexion and moving in only one direction at any given time. Likewise, extension wrist movements. Each monkey viewed a display on which when decoding the position of a rat in a maze, the rat can be in each digit (and the wrist) was represented by a row of five light- only one place at any time. emitting diodes (LEDs). When the monkey flexed or extended a digit, AQ: A Address for reprint requests and other correspondence: A. Pouget, Brain and The costs of publication of this article were defrayed in part by the payment AQ: A Cognitive Science Dept., Meliora Hall, Univ. of Rochester, Rochester, NY of page charges. The article must therefore be hereby marked “advertisement” 14627 (E-mail: alex@bcs.rochester.edu). in accordance with 18 U.S.C. Section 1734 solely to indicate this fact. www.jn.org 0022-3077/07 $8.00 Copyright © 2007 The American Physiological Society 1
  • tapraid4/z9k-neurop/z9k-neurop/z9k00707/z9k8250d07a moyerr S 6 5/11/07 22:29 MS: J00760-6 Ini: Input-dlw 2 S. B. HAMED, M. H. SCHIEBER, AND A. POUGET closing a microswitch, the central yellow LED went out and a green three-dimensional (3D) space, with all 12 directions equally spaced in LED to the left or right, respectively, came on, cueing the monkey as 3D. Then, on each trial k, the PV, pk, is computed according to to which switch(es) had been closed. Red LEDs to the far left or right N were illuminated one at a time, under microprocessor control, instruct- ing the monkey to close that one switch (or move the wrist). If the pk cirik (1) monkey closed the instructed switch within the 700 ms allowed after i 1 illumination of the red instruction LED and held it closed for a 500-ms where r k is the activity of neuron i on trial k (i.e., the spike count in i final hold period without closing any other switches, the monkey the 100-ms interval preceding the end of the movement), and ci is the received a water reward. After each rewarded trial, the movement to preferred direction for neuron i. be instructed for the next trial was rotated in a pseudorandom order. For all cells, the preferred directions {ci} were calculated using the We abbreviate each instructed movement with the number of the data from all available trials but two, and the PV was estimated using instructed digit (1 thumb through 5 little finger, W wrist), and the data from the remaining two trials. This was repeated on all the first letter of the instructed direction (f – flexion; e – extension); possible pairs out of the nine minimum trials retained for all cells, and for example, “4f” indicates instructed flexion of the ring finger. the resulting PVs were averaged to produce a mean PV estimate. Monkeys K and S each performed all 12 possible movements In the Results, we report the performance of the PV in terms of the involving flexion or extension of a single digit or of the wrist (1f, 1e, average angular difference between the vector assigned to the actual 2f, 2e, 3f, 3e, 4f, 4e, 5f, 5e, Wf, and We). In addition, each monkey movement on trial k and the decoded population vector pk. We also also performed movements involving concurrent flexion or extension report the percentage of correct responses taken as the percentage of of two digits. Concurrent flexion of the thumb and index finger, for conditions in which the PV was closest to the vector assigned to the example, which we denote 1 2f, was instructed to the monkey by actual movement. simultaneous illumination of the red LEDs used to instruct thumb flexion and index finger flexion. To obtain a reward, the monkey was OPTIMAL PV. The optimal population vector (OPV) is similar to the required to close either the thumb or index finger flexion switch within regular PV estimator in the sense that, on every trial k, we first 700 ms, to close the remaining switch within 100 ms of closing the estimate a vector pk by taking a linear combination of the neural first, and to hold them both closed for 500 ms without closing any activities other switches. Monkey K performed six multiple finger movements: 1 2f, 1 2e, 2 3f, 2 3e, 4 5f, 4 5e. Monkey S performed N two: 1 2f, 1 2e. For each monkey, these multiple finger pk wirik movements were presented intermingled with the 12 single finger i 1 movements in a pseudorandomized block design. The main difference from the PV is that for the OPV the weights, w i i 1. . .N, are not set a priori to the preferred directions, c i i 1. . .N. Data collection Instead, we seek the weights that minimize the cost function After training, aseptic surgery under general anesthesia was used to k perform a craniotomy over the left central sulcus at the level of the E pk p*k 2 hand representation to implant a rectangular Lucite recording chamber t 1 over the craniotomy and to implant head-holding posts. Once the monkey had recovered from this procedure and had become accus- where p*k is the vector corresponding to the actual movement on trial tomed to performing the finger movement task with the head held k. The optimal weights are given by stationary, single neuron activity was recorded with conventional WT 1 Crp* techniques as the monkey performed the finger movement task each day. Data were collected from 115 task-related neurons in the primary where W is the weight matrix with column w i i 1. . .N, is the motor cortex (M1) of monkey K and from 61 M1 neurons in monkey covariance matrix of the neuronal responses, and Crp* is the covari- S. In both cases, neurons were recorded from area 4 in the anterior ance matrix between the response and the vector p* corresponding to bank and lip of the central sulcus. the actual movements. Here again, for all cells, the optimal weights w i i 1. . .N were Decoding methods calculated on all available trials but two, and the PV was estimated on the remaining two trials. This was repeated on all possible pairs out of To include a neuron in this analysis, we required that the neuron the nine minimum trials retained for all cells, and the resulting PVs had been recorded for a minimum of nine trials of each type of finger were averaged to produce a mean PV estimate. movement performed by the monkey. Data sufficient for these anal- yses were available from a total of 140 neurons: 100 from monkey K LOGISTIC REGRESSION. Logistic regression (LR) is equivalent to and 40 from monkey S. For a few neurons, as many as 19 trials of training a nonlinear two layer neural network. The input layer con- each movement were available. For each trial and for each neuron, sisted of N units corresponding to N M1 neurons. N was varied as spikes were counted in the 100-ms interval directly preceding the end described in the final section of METHODS. These N input units of the movement (switch closure), a period when most M1 neurons projected onto 12 output units, 1 output unit for each possible reach peak firing rates. Using these data from M1 neurons in monkeys movement (Fig. 1A). The activity of output unit i on trial k, y k , was i F1 K and S, we examined the performance of the following five methods determined according to of decoding finger movements: 1 y ik REGULAR POPULATION VECTOR. The population vector (PV) is N typically used to estimate a periodic variable, like the direction of arm 1 exp wijrjk bi movement, from a set of neuronal responses (Georgopoulos et al. j 1 1986). Georgopoulos and colleagues have shown how to extend this approach to discrete classes, and this approach has been used previ- where w ij i 1. . .6, j 1. . .N are the weights, bi is the bias of output unit i, ously to analyze movements of single fingers (Georgopoulos et al. r k j 1. . .N are the activities of the M1 neurons on trial k, and is a j 1999). In this study, each finger movement is assigned a direction in constant ( 0.0002). The weights were optimized through gradient J Neurophysiol • VOL 98 • JULY 2007 • www.jn.org
  • tapraid4/z9k-neurop/z9k-neurop/z9k00707/z9k8250d07a moyerr S 6 5/11/07 22:29 MS: J00760-6 Ini: Input-dlw DECODING MULTIPLE FINGER MOVEMENTS 3 FIG. 1. Network representations of the logistic regression (LR) and softmax (SM) estimators. A: logistic regression. Input layer encodes the activity of N M1 neurons before a finger movement (N varies from 5 to 140, depending on simulation). Output layer contains 1 unit per movement for a total of 12 units (5 fingers plus wrist and 2 movements per finger or wrist). Output units are trained to encode probability of movement they encode given current input activities. Training involves optimizing weights W through gradient descent with cross-validation. B: SM. Input layer is as in A. Output layer is organized in 6 triplets of units (5 fingers plus wrist). Each triplets encodes probability distributions over 3 mutually exclusive states of corresponding finger (or wrist): no movement, flexion, and extension. SM normalization is used within each triplet to ensure that activity of the 3 units always sum up to 1, as should be the case for a probability distribution. Weights W are trained through gradient descent with cross-validation. descent with cross-validation as described below. With this particular I r i ;m H ri H ri m activation function, the value of each output unit is constrained to lie within the [0,1] interval. This enabled us to interpret the activity of where ri and m are two random variables corresponding to the output unit i as encoding the probability that movement i was response of neuron i and the movement performed by the monkey. produced on trial k. The probability distributions, p(ri) and p(ri m), which are needed to This approach was applied to both single finger movements and compute the entropies H(ri) and H(ri m), were estimated from the data movements of pairs of fingers, but with one major caveat: the number by counting the frequency of each response using all the data avail- of fingers moving on a given trial was known for each trial. If only one able. This method is known to overestimate information, but we used finger moved, for that trial, we picked the one finger with the highest I only to rank the neurons from most informative to least informative, probability. If two fingers moved, we picked the two fingers with the enabling us to reduce the number of neurons needed for optimal highest probability. decoding performance using the softmax estimator. This approach is labeled SM-best vote in Figs. 3 and 4. SOFTMAX. The softmax (SM) estimator (Bishop 1995) was used to address the main limitation of the LR approach, the requirement that the number of fingers moving on each trial must be known a priori. Training and cross-validation For the SM network, the output layer was organized into six sub- The parameters of all of the estimators described above were groups corresponding to the five fingers and the wrist. Each subgroup estimated with cross-validation. The dataset contained N neurons for contained three units receiving inputs from all M1 neurons. These which neuronal activity recorded in 9 –19 trials was available (the three output units were trained to encode the probabilities of no number of neurons, N, was varied). For each of the 18 movements (12 movement, of extension, and of flexion, respectively (Fig. 1B). single finger movements and 6 multiple finger movements), two trials (Again, weights were optimized through gradient descent with cross- of data from each neuron were set aside for testing the network. For validation, see METHODS). Because these three states of each finger are training the network, single remaining trials of data from each neuron mutually exclusive (e.g., a given finger cannot flex and extend at the were selected randomly to generate 10,000 patterns of population same time), the three probabilities were constrained to sum to unity activity ( 56 for each of the 18 movements). In this way, we through an SM normalization; thus the activity of unit i (with i 1.3), generated population patterns of activity for each movement even in the jth subgroup (j 1.6), for the kth trial is given by though the neurons actually were recorded one at a time on different N days. For testing the network, we generated 200 new population exp w ij lr tk activity patterns ( 1 for each of the 18 movements) using the two i 1 trials of data from each neuron during each movement set aside for k o ij this purpose. 3 N In the case of LR and SM, we used stochastic gradient descent to exp w ij lr k 1 find the weights minimizing the squared error over all movements and j 1 t 1 all trials. While optimizing the weights on the training set, we also Note that this equation ensures that the activity of the three units monitored classification performance on the testing set. Training was 3 stopped when performance started decreasing on the testing set, a within a subgroup sums up to one, i.e., o k for any subgroup i and ij procedure known as early stopping. This procedure prevents overfit- j 1 ting on the training set and provides a better estimate of generalization any example k. performance (Morgan and Bourlard 1990). SM-BEST VOTE-SELECTED SET (USING MUTUAL INFORMATION). Here we ranked the neurons from most informative to least informative. To Varying the number of neurons being decoded do so, we computed the mutual information between single neurons and finger movements, using the standard definition (Dayan and For all five decoding methods, we systematically varied the number Abbott 2001) of neurons, N, being decoded to determine the minimum number of J Neurophysiol • VOL 98 • JULY 2007 • www.jn.org
  • tapraid4/z9k-neurop/z9k-neurop/z9k00707/z9k8250d07a moyerr S 6 5/11/07 22:29 MS: J00760-6 Ini: Input-dlw 4 S. B. HAMED, M. H. SCHIEBER, AND A. POUGET it estimates the probability distribution over all possible move- ments given the vector of neural activity, instead of generating a single answer as do the PV and OPV. Figure 2 shows the performance of the LR estimator compared with PV and OPV. LR outperformed both population vector estimators, decoding single finger movements with 99.48% accuracy using as few as 40 randomly selected cells. To address the main limitation of the LR estimator, namely the need to know how many fingers are moving on each trial, we developed another estimator, which we call an SM estima- tor. This estimator returns six probability distributions: one for each finger and one for the wrist. Each probability distribution is defined over three possible states: flexion, extension, and no movement. The SM estimator can be read out on every trial without prior knowledge of the number of fingers moving: we simply look for the most probable state of each finger. A typical answer might look like the following: no movement for finger 1, flexion for finger 2, no movement for finger 3, flexion C for finger 4, no movement for finger 5, and no movement for O the wrist. This approach can be used to estimate single finger L movements, two finger movements, and more generally, any O arbitrary combination of flexion and extension over the five R fingers and wrist, regardless of the number of fingers moving. FIG. 2. Comparative performance of the regular population vector (PV), Figure 3 shows the performance of the SM estimator (SM F3 optimal population vector (OPV), and LR estimators on decoding single finger exact match) compared with LR on decoding single finger movements as a function of number of neurons being decoded. Gray bars movements. The SM estimator performed just as well as LR, represent SE. Long gray vertical line indicates number of cells for which LR reached a detection performance of 100%. even though it used no prior knowledge of the number of fingers moving on each trial. Remarkably, the SM estimator neurons needed for the best possible classification. N was set to 140, not only assigns the highest probability to the proper move- 100, 60, 40, 30, 20, 15, 10, and 5. For each value of N, the performance of the estimator was evaluated by randomly choosing N cells with replacement from the 140 cells available and optimizing and testing the estimator with those N cells. This process was repeated 10 times, and we report the average of these 10 repetitions. RESULTS Decoding of single finger movements with the regular PV and OPV estimators We first decoded the 12 single finger and wrist movements from the discharge of M1 neurons using both the regular PV and OPV estimators (see METHODS). The average error, ex- pressed in terms of the angle between the estimated direction C and true direction, was 34.07° for the PV and 14.18° for the O OPV, when using all 140 cells. These numbers are comparable L with those reported by Georgopoulos et al. (1999), although O this previous study did not use cross-validation. R F2 Figure 2 shows the results expressed in terms of percentage FIG. 3. Comparative performance of LR and SM estimators on decoding of correct classification as a function of the number of cells single finger movements as a function of dataset size. Two different classifi- being decoded. The OPV outperformed the PV for all numbers cation criteria were applied to SM. In a most restrictive decoding scheme, SM of cells. In fact, the PV never approached 99% correct, even estimator was considered to be correct only when it predicted correct move- ment for moving finger and no movement for all other fingers. In SM best vote, with 140 cells, whereas OPV reached 99.2% accuracy using as SM estimator was considered to be correct when it assigned highest probability few as 60 randomly selected cells. to correct movement for moving finger, even if it predicted no movement for all other fingers. In other words, we assume that we know that only 1 finger Decoding of single finger movements with LR and SM moves on each trial, as was done for the LR estimator. Finally, for SM best vote, selected set, we applied the same criteria as for SM best vote, but we used LR is the optimal classifier when the data follows a multi- the N cells with highest mutual information between responses and movement. variate Gaussian distribution with a fixed covariance matrix Gray vertical dashed line indicates number of cells for which SM best vote reached a detection performance of 100%. Gray vertical continuous line (Bishop 1995). Because this assumption holds to a first ap- indicates number of cells for which SM best vote, selected set reached a proximation with neural activities, LR was a natural estimator detection performance of 100%. SE is not plotted on this graph to avoid to try with our data set. Moreover, LR has the advantage that unnecessary clutter, but they are in the same range as in Fig. 2. J Neurophysiol • VOL 98 • JULY 2007 • www.jn.org
  • tapraid4/z9k-neurop/z9k-neurop/z9k00707/z9k8250d07a moyerr S 6 5/11/07 22:29 MS: J00760-6 Ini: Input-dlw DECODING MULTIPLE FINGER MOVEMENTS 5 ment for the proper finger but also assigns the highest proba- performed reasonably well on some two-finger combinations bility to no movement for all the other fingers. (e.g., 94% correct on 1 2e) but poorly on others (2% on 4 For further comparison with the LR estimator, we also tested 5e, which is below chance) for an average performance of 38% the SM estimator when assuming that we know that only one correct over all six two-finger movements. The SM estimator finger moved on each trial. This means that we read out the SM fared worse, failing to generalize from single finger movement estimator in the same way as the LR estimator, searching only to two-finger movements. The inability of the LR and SM for the movement state (flexion or extension) with the highest methods to estimate two-finger movements after training on probability, while ignoring all other answers. Therefore even if single finger movements indicates that 1) the population activ- the SM estimator predicted that more than one finger moved, ity patterns generated for different single-finger movements for this comparison, we nevertheless considered the prediction overlap (in that single cells discharge significantly during correct as long as the most likely movement was indeed the one several single-finger movements (Poliakov and Schieber 1999; that took place on that trial. As can be seen in Fig. 3 (SM best Schieber and Hibbard 1993) and 2) the patterns of neural AQ: 1 vote), this SM approach improves performance still further, activity for two-finger movements are unlikely to be simple decoding which finger moved at 99.6% accuracy using as few linear combinations of the patterns for single-finger move- as 30 randomly selected neurons. ments. Finally, we asked whether the number of cells needed to To examine this idea further, we ran the following test. For decode at 99% accuracy could be reduced below 30 if we all two-finger movements AB, where A and B refer to the selected the neurons that provided the most information about single finger movements involved in this combination, we which finger movement was performed. We therefore com- computed the angle between the population activity vector for puted the mutual information between each neuron and the AB and the vector obtained by taking the sum of the two finger movements, and we trained the SM estimator using the population activity vectors for A and B. If the activity for the N neurons with the highest information content. The number N two finger movement is simply equal, or proportional to, the was varied systematically to obtain the performance of the SM sum of the activities for the single finger movement, the angle estimator as a function of the number of neurons. Using the should be close to zero. As can be seen on Fig. 4, the angle is F4 most informative neurons, 20 selected cells were sufficient for close to 45° on average, indicating that some nonlinearity is 99.91% performance (Fig. 3, SM best vote-selected set). In involved. fact, accuracy still was very high with as few as 15 cells Given the inability of the LR and SM approaches to gener- (97.3% correct). alize from single finger movements to two-finger movements, Of the 20 most informative cells, none discharged for one we tested both methods after training on all 18 single-finger finger movement only. Instead, each of these neurons was and two-finger movements. Figure 5 shows the performance of F5 intensely active during two or more finger movements and several networks based on the activity of 100 neurons during virtually inactive during others. Such distributed coding indeed the different movements. The LR network achieved nearly is to be expected, given that a neuron that fired during only one perfect performance on all combinations (overall average of finger movement would convey relatively little information 98.98%; Fig. 4, LR). However, this performance still relied on (0.4 bits) compared, for instance, with a neuron that fired prior knowledge of whether the movement involved one finger strongly for six movements and not at all for the other six (1 or two fingers. The SM performance was nearly perfect on bit). single-finger movements (99.25%) and several two-finger Our selection procedure was not necessarily optimal, how- movements (Fig. 5, SM exact match), for an average perfor- ever. We do not know that the 20 neurons with the highest mance of 90.9% over all 18 movements. Interestingly, and mutual information constitute the best subset for classifying all unlike what we observed for single-finger movements, perfor- movements. Therefore 20 neurons is an upper bound on the mance of the SM estimator did not improve when tested with minimum number of neurons needed for near-perfect classifi- a priori knowledge of whether one or two fingers moved on cation. However, better performance might be obtained with each trial (Fig. 5, SM best vote). fewer neurons selected in a manner other than the one used here. DISCUSSION Decoding of multiple finger movements: training on single Decoding of single finger movements and implications for finger movements neural prosthetics Neither the PV nor the OPV is suitable for decoding simul- Although many neural prosthetic approaches use recordings taneous movements of multiple fingers because these estima- from large numbers of neurons, we were able to decode 12 tors each return a single answer on any given trial. We single finger movements with 99% accuracy using as few as therefore studied the LR and SM approaches for decoding trials 30 neurons randomly selected from a population of 140 task- in which two fingers moved at the same time. Here we used the related neurons. If we assume that an individual can produce population of 100 neurons from monkey K that had been three finger movements per second, our performance corre- recorded during at least nine trials of the 12 single finger and sponds to a bit rate of 10.75 bits/s [3 log2(12)] for 40 wrist movements plus the 6 movements in which the monkey neurons, which is significantly better than the current state of moved two fingers at the same time. the art, which is 6 bits/s for 100 neurons (Santhanam et al. First we explored whether an estimator trained on single 2006; Taylor et al. 2003). This performance is also particularly finger movements can generalize to two-finger movements. remarkable given that it was obtained with relatively little data, Our results show that this is not the case. The LR estimator because we had at most 19 trials per finger movement collected J Neurophysiol • VOL 98 • JULY 2007 • www.jn.org
  • tapraid4/z9k-neurop/z9k-neurop/z9k00707/z9k8250d07a moyerr S 6 5/11/07 22:29 MS: J00760-6 Ini: Input-dlw 6 S. B. HAMED, M. H. SCHIEBER, AND A. POUGET FIG. 4. Test on nonlinearity for 2 finger movements. For each 2-finger movement, we computed angle between pop- ulation activity vector for this 2-finger movement and sum of population activity for the 2 single finger movements in- volved in this pair. Black dots correspond to single trials; circles correspond to average angles for each 2-finger move- ment. For all 2-finger movements, angle is close to 45° on average, indicating that patterns of neural activity for 2-fin- ger movements are not simply sums of patterns for single- finger movements. from any given neuron during recordings that lasted at most 30 could affect decoding performance. The precise impact of this min. Decoding performance using the LR and SM estimators factor cannot be assessed with our data set but deserves to be should improve considerably with long-term recordings from investigated in future studies. implanted electrodes, when hundreds to thousands of trials per movement condition could be collected and used to train the decoding estimators. Decoding of multiple finger movements These conclusions, however, should be tempered by the fact that we used independent neurons, whereas these other studies Decoding multiple finger movements failed when the LR relied on neurons recorded simultaneously. As we discuss and SM estimators were trained exclusively on single finger below, it is possible that our performance would decrease movements. This failure may arise in part from the fact that the significantly with correlated neurons. Moreover, it is possible neural and muscular activation required to move two fingers is that the neurons we used were particularly well tuned com- not simply the linear combination of the activations required to pared with the neurons typically found with multiunit record- move each finger in isolation (Schieber 1990). Because multi- ings using grids. Indeed, in single cell recordings, the experi- ple adjacent fingers tend to move together, movement of a mentalist can select neurons with high firing rates, which are single finger in isolation requires contraction of both the likely to be more informative. With grids recordings, this is not muscles that move that finger and additional muscles that an option because the electrodes cannot be moved indepen- check unwanted motion in adjacent digits (Reilly and Schieber dently. Finally, the M1 neurons from which we recorded may 2003; Schieber 1995; Schieber and Santello 2004). Hence have responded in part to proprioceptive feedback. In a para- generalization of a network trained on single finger movements lyzed individual, such feedback would not be present, and the to decode multiple finger movements is unlikely to succeed. discharge of M1 neurons might differ significantly. This in turn When the LR and SM estimators were trained using data from FIG. 5. Comparative performance of LR and SM estimators on decoding single and multiple finger move- ments. SM exact match and SM best vote are as in Fig. 3. Downward pointing bars represent SE. J Neurophysiol • VOL 98 • JULY 2007 • www.jn.org
  • tapraid4/z9k-neurop/z9k-neurop/z9k00707/z9k8250d07a moyerr S 6 5/11/07 22:29 MS: J00760-6 Ini: Input-dlw DECODING MULTIPLE FINGER MOVEMENTS 7 all 12 single-finger and 6 two-finger movements, however, correlations on decoding multiple movements should be thor- both methods provided accurate decoding using 100 neurons. oughly studied in future experiments. Nonetheless, with 140 cells, we were able to obtain 90% GRANTS accuracy in decoding combination finger movements. We do not have enough data to know whether this performance would S. B. Hamed and A. Pouget were supported by National Institute of Mental generalize to arbitrary combinations of movements. Assuming Health Grant MH-57823, research grants from the Office of Naval Research, and the McDonnell-Pew, Sloan, and Schmitt foundations. M. H. Schieber was for purposes of discussion that a subject could produce three supported by National Institute of Neurological Disorders and Stroke Grant combinations of six possible movements per second (1 bit per NS-27686. digit or wrist), the bit rate potentially would reach 18 bits/s. In part, such a hypothetical bit rate, well above what has been REFERENCES reported by other groups, may reflect the simultaneous control Abbott LF, Dayan P. The effect of correlated variability on the accuracy of of six effectors. These encouraging numbers are likely to be a population code. Neural Comput 11: 91–101, 1999. overestimates, however, given that all the caveats we have Averbeck BB, Lee D. Neural noise and movement-related codes in the macaque supplementary motor area. J Neurosci 23: 7630 –7641, 2003. raised in decoding single movements would apply to combi- Bair W, Zohary E, Newsome WT. Correlated firing in macaque visual area nation movements as well, and in particular, the issue of MT: time scales and relationship to behavior. J Neurosci 21: 1676 –1697, correlations. 2001. Bishop CM. Neural Networks for Pattern Recognition. New York: Oxford, 1995. Impact of correlations Dayan P, Abbott LF. Theoretical Neuroscience. Cambridge, MA: MIT Press, Because each of the present population of M1 neurons was 2001. Georgopoulos AP, Pellizzer G, Poliakov AV, Schieber MH. Neural coding recorded at a different time, the activities of different neurons of finger and wrist movements. J Comput Neurosci 6: 279 –288, 1999. in our population estimates effectively were uncorrelated. Our Georgopoulos AP, Schwartz AB, Kettner RE. Neuronal population coding results might have been different had we used neural data of movement direction. Science 233: 1416 –1419, 1986. obtained from simultaneous recordings of multiple single neu- Maynard EM, Hatsopoulos NG, Ojakangas CL, Acuna BD, Sanes JN, Normann RA, Donoghue JP. Neuronal interactions improve cortical pop- rons, because the activity of simultaneously recorded cortical ulation coding of movement direction. J Neurosci 19: 8083– 8093, 1999. neurons often can be shown to be correlated (Bair et al. 2001; Morgan N, Bourlard HA. Generalization and parameter estimation in feed- Maynard et al. 1999; Zohary et al. 1994). Such correlations can forward nets: some experiments. In: Advances in Neural Information Pro- either increase or decrease the quality of decoding depending cessing Systems 2, edited by Touretzky D. San Mateo, CA: Morgan on their exact pattern (Abbott and Dayan 1999; Yoon and Kaufmann, 1990. AQ: 2 Panzeri S, Petroni F, Petersen RS, Diamond ME. Decoding neuronal Sompolinsky 1999), however, making it difficult to predict population activity in rat somatosensory cortex: role of columnar organiza- whether performance would have been better or worse using tion. Cereb Cortex 13: 45–52, 2003. simultaneously recorded neurons. Studies that have compared Pouget A, Dayan P, Zemel RS. Inference and computation with population truly simultaneous recordings to “shuffled” recordings (equiv- codes. Annu Rev Neurosci 26: 381– 410, 2003. Reilly KT, Schieber MH. Incomplete functional subdivision of the human alent to single unit recordings because shuffling data from multitendoned finger muscle flexor digitorum profundus: an electromyo- different trials effectively removes the correlations between graphic study. J Neurophysiol 90: 2560 –2570, 2003. neurons) suggest that the impact of correlations is insignificant, Salinas E, Abbott LF. Vector reconstruction from firing rates. J Comput or very small, for population of tens of neurons (Averbeck and Neurosci 1: 89 –107, 1994. Lee 2003; Panzeri et al. 2003). Moreover, Serruya et al. (2002) Santhanam G, Ryu SI, Yu BM, Afshar A, Shenoy KV. A high-performance brain-computer interface. Nature 442: 195–198, 2006. have found recently that 7–30 neurons recorded simultaneously Schieber MH. How might the motor cortex individuate movements? Trends are sufficient to control the location of a computer cursor on a Neurosci 13: 440 – 445, 1990. screen. Although this task is quite different from the present Schieber MH. Individuated finger movements of rhesus monkeys: a means of finger movement task, their findings support the notion that quantifying the independence of the digits. J Neurophysiol 65: 1381–1391, 1991. 40 neurons can be sufficient for decoding, even when the Schieber MH. Muscular production of individuated finger movements: the neurons are recorded simultaneously. Our finding that as few roles of extrinsic finger muscles. J Neurosci 15: 284 –297, 1995. as 30 – 40 M1 neurons were sufficient for decoding single Schieber MH, Hibbard LS. How somatotopic is the motor cortex hand area? finger movements therefore may have been similar even with Science 261: 489 – 492, 1993. simultaneously recorded neurons. Schieber MH, Santello M. Hand function: peripheral and central constraints on performance. J Appl Physiol 96: 2293–2300, 2004. For larger neuronal sets, such as the 140 neurons we used for Serruya MD, Hatsopoulos NG, Paninski L, Fellows MR, Donoghue JP. single finger movements, assessing the impact of correlated Instant neural control of a movement signal. Nature 416: 141–142, 2002. neuronal activity is more difficult because little is known about Seung HS, Sompolinsky H. Simple models for reading neuronal population correlations among such large neuronal populations in vivo. codes. Proc Natl Acad Sci USA 90: 10749 –10753, 1993. Taylor DM, Tillery SIH, Schwartz AB. Information conveyed through Extrapolation from pairwise recordings suggests that, if our brain-control: cursor vs. robot. IEEE Trans Neural Syst Rehabil Eng 11: neurons were all equally positively correlated (the worse case 195–199, 2003. scenario from the standpoint of decoding information), corre- Yoon H, Sompolinsky H.The effect of correlations on the Fisher information lations become a major concern only for 1,000 neurons or more of population codes. In: Advances in Neural Information Processing Sys- (Zohary et al. 1994). Therefore our results with 140 neurons tems, edited by Kearns M, Solla S, Cohn D. Cambridge, MA: MIT Press, 1999, p. 167–173. recorded one at a time are likely to be reasonably close to what Zohary E, Shadlen MN, Newsome WT. Correlated neuronal discharge rate would have been obtained with 140 simultaneously recorded and its implications for psychophysical performance. Nature 370: 140 –143, neurons. Nonetheless, there is little doubt that the impact of 1994. J Neurophysiol • VOL 98 • JULY 2007 • www.jn.org
  • JOBNAME: AUTHOR QUERIES PAGE: 1 SESS: 1 OUTPUT: Thu May 10 05:57:21 2007 /tapraid4/z9k neurop/z9k neurop/z9k00707/z9k8250d07a AUTHOR QUERIES AUTHOR PLEASE ANSWER ALL QUERIES 1 AQA— Please verify the accuracy of your e-mail address or delete it if you do not wish it included.Au: Please check all Tables and Equations carefully for presentation and accuracy. All tables and display equations (those set off from running text) are set by hand during the composition process. I have edited all of these slightly. As you check the page proofs, please indicate all necessary changes to tables and equations. AQ1— Please add Poliakov and Schieber 1999 to reference list or delete citation here. AQ2— Please provide page range.