Color Vision(e.g., spatial frequency, orientation, motion, depth) which the stimulus diﬀers perceptually from a purelywithin a local cortical region. With respect to color achromatic (i.e., white, gray, black) axis. The thirdvision per se, the primary processing involves separ- dimension is brightness or lightness. That our per-ating color and luminance information, and further ceptual space is three-dimensional reﬂects the basicseparating changes due to the illuminant from those trichromacy of vision.due to visual objects, by lateral interactions over large A normal observer can describe the hue of any lightregions. (disregarding surface characteristics) by using one or To separate luminance and color information, the more of only four color names (red, yellow, green, andoutputs of Pc cells are combined in two diﬀerent ways. blue). These so-called unique hues form two opponentWhen their outputs are summed in one way, the pairs, red–green and blue–yellow. Red and greenluminance components to their responses sum and the normally cannot be seen in the same place at the samecolor components cancel. Summed in a diﬀerent time; if unique red and unique green lights are added incombination, the color components sum and the appropriate proportions, the colors cancel and oneluminance components cancel. Consider a striate sees a neutral gray. Orange can be seen as a mixture ofcortex cell that combines inputs from one or more red and yellow, and purple as a mixture of red andjLo and jMo cells in a region. The cortical cell would blue, but there is no color seen as a red–green mixturerespond to luminance variations but not to color (or as a blue–yellow mixture). This perceptual op-variations, since the neurons providing its inputs both ponency is also reﬂected in color contrast. Red canﬁre to luminance increments in the RF center and to induce the appearance of green into neighboringdecrements in the surround, but the color organi- regions, and after staring at a red surface one sees azations of its inputs are opposite to each other (one green after-image. The yellow–blue opponent pairbeing L-M and the other M-L). Combined with input produces similar eﬀects. It was these perceptualfrom a jSo cell, this would produce a V1 cell that ﬁres characteristics of color that led Ewald Hering in theto white (light increments) and inhibits to black (light nineteenth century to propose that the various colordecrements) but does not respond to pure color systems were not independent but rather that colorvariations. This is represented in the top row of Fig. was processed in a spectrally opponent organization,1C. However, a V1 cell receiving inputs from both an idea which has since been amply veriﬁed in thejLo and kMo cells, or from both jMo and kLo cells presence, discussed above, of spectrally-opponent cells(columns in Fig. 1C), would respond to color changes in the path from receptors to the cortex.but not to luminance variations since their colorresponses would add, but their luminance RFs, which See also: Color Vision Theory; Vision, Low-levelare opposite to each other, would cancel. This organi- Theory of; Vision, Psychology of; Visual Perception,zation by itself would produce L-M color cells that Neural Basis of; Visual System in the Brainwould ﬁre to so-called warm colors (red and yellow)and inhibit to cool colors (blue and green). M-L cellswould ﬁre to cool colors and inhibit to warm colors. BibliographyAs shown in Fig. 1C, the further addition of jSo or De Valois R L, De Valois R L 1988 Spatial Vision. OxfordkSo cells can split these classes into separate red University Press, New Yorkand yellow, and separate blue and green systems, Hurvich L M 1981 Color Vision. Sinauer Press, Sunderland, MArespectively. Kaiser P K, Boynton R M 1996 Human Color Vision. Optical All of the primary visual information is passed Society of America, Washington, DCthrough V1, but subsequent visual areas are partially Neitz J, Neitz M 1998 Molecular genetics and the biologicalspecialized for the further analysis of various diﬀerent basis of color vision. In: Backhaus W G S, Kliegl R, Wernerfunctional aspects of vision. One later visual area (V4) J S (eds.) Color Vision. Walter de Gruyter, Berlin, pp. 101–19 Spillmann L, Werner J S 1990 Visual Perception: The Neuro-is crucially involved with color perception. Individuals physiological Foundations. Academic Press, New Yorkwith localized V4 lesions can still discriminate objectson the basis of their color variations, but they report K. K. De Valois and R. L. De Valoisthat the objects now appear to have no hue, as ifviewed on a black-white television screen. There is alsoa report of one case with the reverse loss: a patient whocould see colored but not black-white objects. Color Vision Theory11. Color Appearance Color vision is the ability to distinguish and identifyThe appearance of a color can be speciﬁed by values lights and objects on the basis of their spectralalong just three perceptual dimensions known as hue, properties. This entry presents several key topics thatsaturation and brightness. Hue refers to the character- underlie current theories of human color vision. Theseistic described by such color names as red, yellow, are trichromacy, color opponency, adaptation, andgreen, and blue. Saturation refers to the extent to color constancy.2256
Color Vision Theory1. Introduction primary intensities until the mixture has the same color appearance as the test light. The primaries usedInformation about color is transformed as it ﬂows in the experiment are chosen to be independent, sofrom the stimulus through the initial stages of the that no weighted mixture of any two produces a matchhuman visual system. At each image location, the to the third.color stimulus is speciﬁed by the amount of power it Because the matching light is constrained to be acontains at each wavelength. The classic color match- weighted mixture of three primaries, it will noting experiment shows that the normal human visual generally be possible for the observer to make the testsystem is trichromatic: only three dimensions of and matching lights physically identical. For manyspectral variation are coded by the visual system. The test lights, however, the observer can adjust thebiological basis of normal trichromacy is that the matching light so that it appears identical to the testretina contains three classes of cone photopigment. light even though the two diﬀer physically. For someAfter the initial encoding of light by the cones, further test lights, no choice of primary intensities will aﬀordprocessing occurs. Two aspects of this processing are a match. In these cases one or more of the primariesparticularly important. First, signals from three classes can be mixed with the test light and primary intensitiesof cones are recombined to form a luminance and two found so that the primarytest mixture matches thecolor opponent channels. Second, there is adaptive mixture of the remaining primaries. A useful descrip-signal regulation that keeps neural signals within their tive convention for the color matching experiment isoperating range and stabilizes the appearance of to assign a negative intensity to any primary that mustobjects across changes of illumination. be mixed with the test to make a match. Given this convention, any test light can be matched by a mixture of three independent primaries.2. Trichromacy The color matching experiment is an empirical system. Given a test light described by a vector b, the2.1 Color Matching experiment returns a vectorThe physical property of light relevant for color vision A Cis the spectral power distribution. A light’s spectral t "power distribution speciﬁes the amount of power it tl t (2)contains at each wavelength in the visible spectrum, # t B $ Doften taken to lie roughly between 400 and 700 nm. Inpractice, spectral power distributions are measured at whose entries are the individual primary intensities.discrete sample wavelengths. Let the measured power When the primaries are scaled by these intensities andvalues be denoted by b , …, bNλ where Nλ denotes the "number of sample wavelengths. Then the vector mixed, a match to the test light is created. The vector t speciﬁes what are called the tristimulus coordinates A C of the light b. A theory of color matching should let us b " predict t for any test light b, given the spectral power distributions of the primary lights. bl < (1) As an empirical generalization, the color matching system is a linear system (e.g., Wyszecki and Stiles 1982, Brainard 1995, Wandell 1995). That is, if we bNλ have two test lights b and b with tristimulus coordinates t and t , then any # weighted mixture " B Dprovides a compact representation of the spectral " # (a b ja b ) of the two test lights has tristimuluspower distribution. Use of a vector representation for " " # # coordinates given by the corresponding mixturespectral quantities facilitates a variety of colorimetric (a t ja t ). In these vector expressions, multiplicationcomputations (e.g., Brainard 1995). Wavelength of" a vector (e.g., b ) by a scalar (e.g., a ) consists of " ##sample spacings between 1 and 10 nm are typical. " " multiplying each entry of the vector by the scalar, Trichromacy is demonstrated by the basic color while addition of two vectors (e.g., a b and a b )matching experiment (Wandell 1995, Brainard 1995). " " # # consists of adding the corresponding entries of the twoIn this experiment, an observer views a bipartite ﬁeld. vectors.One side of the ﬁeld contains a test light. This light is The linearity of color matching makes it possible toexperimentally controlled and can have an arbitrary predict the match that will be made to any test light onspectral power distribution. On the other side of the the basis of a relatively small number of measurements.ﬁeld is the matching light. This consists of the weighted Consider the set of monochromatic lights with unitmixture of three primary lights. Each primary has a power. If Nλ wavelength samples are used in theﬁxed relative spectral power distribution, but its underlying representation, then there are Nλ suchoverall intensity in the mixture can be controlled by lights and we can denote their spectral representationsthe observer. The observer’s task is to adjust the by c , c , …, cNλ. Each of the ci has a 1 as its ith entry " # 2257
Color Vision Theoryand zeros elsewhere. Note that any light b may be this hypothesis (see Wandell 1995, Rodieck 1998).thought of as a weighted mixture of monochromatic First, the responses of individual cones depend onlylights, so that b l bici where bi is the ith entry of b. on the rate at which photopigment molecules are iLet the vectors ti specify the tristimulus coordinates isomerized by the absorption of light quanta; once theof the monochromatic lights ci. The linearity of color intensity of two lights has been adjusted so that theymatching then tells us that the tristimulus coordinates produce the same isomerization rates, the cone re-of any light b are given by t l biti. sponse does not distinguish the two lights. This idea is i A set of tristimulus values ti measured for mono- referred to as the principle of univariance. Second,chromatic lights ci is referred to as a set of color individual cones may be classiﬁed into one of threematching functions. Although these are often plotted distinct types, each with a characteristic spectralas a function of wavelength, they do not represent the sensitivity. The spectral sensitivity is proportional tospectral power distributions of lights. The color the probability that light quanta of diﬀerent wave-matching functions may be speciﬁed by a single matrix lengths will isomerize a molecule of the cone’s photo- A C pigment. The three types of cones are often referred to Tl t t t (tNλ D (3) B # $ as the long- (L), middle- (M), and short- (S) wave-whose Nλ columns consist of the individual tristimulus length-sensitive cones. If an observer has only threecoordinate vectors ti. This speciﬁcation allows com- types of cones, each of which obeys the principle ofputation of tristimulus coordinates from spectral univariance, two physically distinct lights that producepower distributions through simple matrix multipli- the same isomerization rates for all three classes ofcation: cones will be indistinguishable to the visual system. Quantitative comparison conﬁrms that color matches t l Tb. (4) set by a standard observer (deﬁned as the average of matches set by many individual observers) are wellBoth tristimulus values and color matching functions predicted by the equations of isomerization rates inare deﬁned with respect to the primaries chosen for the the L-, M-, and S-cones.underlying color matching experiment. The Com- As described above, trichromacy occurs for mostmission Internationale de l’Eclairage (CIE) has stan- observers because their retinas contain cones withdardized a system for color representation based on three classes of photopigments. Genetic consider-the ideas outlined above. The CIE system is widely ations, however, indicate that some individuals haveused to specify color stimuli and many sources describe retinas containing four classes of cone photopigmentsit in detail (e.g., Wyszecki and Stiles 1982, Brainard (Sharpe et al. 1999). Either these individuals are1995, Kaiser and Boynton 1996). tetrachromatic (mixture of four primaries required to The advantage of using tristimulus coordinates to match any light) or else their trichromacy is mediateddescribe color stimuli is that they provide a more by information lost after quantal absorption. Incompact and tractable description than a description addition, some human observers are dichromatic (onlyin terms of wavelength. Tristimulus coordinates are two primaries must be mixed to make a match to anycompact precisely because they do not preserve physi- light.) Most cases of dichromacy occur because onecal diﬀerences that are invisible to the human visual photopigment is missing (Sharpe et al. 1999, Neitz andsystem. The representational simpliﬁcation aﬀorded Neitz 2000).by tristimulus coordinates is extremely valuable for An alternative to using tristimulus coordinates tostudying processing that occurs after the initial encod- represent the spectral properties of lights is to use coneing of light. On the other hand, it is important to coordinates. These are proportional to the isomeriz-remember that the standard tristimulus represen- ation rates of the three classes of cone photopigments.tations (e.g., the CIE system) are based on matches The three dimensional vectormade by a typical observer looking directly at a small A Cstimulus at moderate to high light levels. These qLrepresentations are not necessarily appropriate forapplications involving some individual observers, non- q l qM (5)human color vision, or color cameras (e.g., Wyszecki q B S Dand Stiles 1982, Brainard 1995). speciﬁes cone coordinates where qL, qM, and qS denote the isomerization rates of the L-, M-, and S-cone2.2 Biological Basis of Color Matching photopigments respectively. It can be shown (e.g.,The color matching experiment is agnostic about the Brainard 1995) that cone coordinates and tristimulusbiological mechanisms that underlie trichromacy. It is coordinates are related by a linear transformation, sogenerally accepted, however, that trichromacy typi- thatcally arises because color vision is mediated by three q l Mtqt (6)types of cone photoreceptor. Direct physiologicalmeasurements of individual primate cones support where Mtq is an appropriately chosen 3 by 3 matrix.2258
Color Vision Theory Computation of cone coordinates from light spectra A possible approach to understanding post-absorp-requires estimates of the cone spectral sensitivities. tion processing is to keep the modeling close to theFor each cone class, these specify the isomerization underlying anatomy and physiology and to character-rates produced by monochromatic lights of unit ize what happens to signals at each synapse in thepower. The sensitivities may be speciﬁed in matrix neural chain between photoreceptors and some site inform as visual cortex. The diﬃculty is that it is not presently possible to cope with the complexity of actual neural A C sL processing. Thus many color theorists have attempted S l sM (7) to step back from the details and develop more abstract descriptions of the eﬀect of neural processing. s B S D Models of this sort are often called mechanistic models. These models generally specify a transform-where each row of the matrix is a vector whose entries ation between the quantal absorption rates q elicitedare the spectral sensitivities for one cone class at the by a stimulus and a corresponding visual represen-sample wavelengths. Given S, cone coordinates are tation u postulated to exist at some central site. Thecomputed from the spectral power distribution of a idea is to choose a transformation so that (a) the colorlight as appearance perceived at a location may be obtained q l Sb (8) directly from the central representation corresponding to that location and (b) the discriminability of twoBecause Eqns. (4), (6), and (8) hold for any light stimuli is predictable from the diﬀerence in theirspectrum b, it follows that central representations. Most mechanistic models assume that signals from S l MtqT (9) the cones are combined additively to produce signals at three postreceptoral sites. Two of these sites carryCurrent estimates of human cone spectral sensitivities opponent signals. These are often referred to as theare obtained from color matching data using Eqn. (9) red-green (RG) and blue-yellow (BY) signals. A thirdtogether with a variety of considerations that put site carries a luminance (LUM) signal, which is notconstraints on the matrix Mtq (Stockman and Sharpe thought to be opponent. If we take1999). A C uLUM3. Postabsorption Processing u l uRG (10) u B BY DColor vision does not end with the absorption of lightby cone photopigments. Rather, the signals thatoriginate with the absorption of light are transformed to be a three-dimensional vector with entries given byas they propagate through neurons in the retina and the LUM, RG, and BY signals, then the additivecortex. Two ideas dominate models of this post- relation between cone coordinates q and the visualabsorption processing. The ﬁrst is color opponency: representation u may be expressed in matrix form:signals from diﬀerent cone types are combined in anantagonistic fashion to produce the visual represen- u l Moq (11)tation at a more central site. The second idea is Many (but not all) detailed models take LUM to be aadaptation: the relation between the cone coordinates weighted sum of L- and M-cone signals, RG to beof a light and its central visual representation is not a weighted diﬀerence between the L- and M-coneﬁxed but depends instead on the context in which the signals, and BY to be a weighted diﬀerence betweenlight is viewed. Section 3.1 treats opponency, while the S-cone signal and a weighted sum of the L- and M-Sect. 3.2 treats adaptation. cone signals. In these models Mo would have the form A C m m 03.1 Opponency # Mo l m km 0 (12)Direct physiological measurements of the responses of # ##neurons in the primate retina support the general idea km km m B $ $# $$ Dof opponency (e.g., Dacey 2000). These measurementsreveal, for example, that some retinal ganglion cells where all of the mij are positive scalars representingare excited by signals from L-cones and inhibited by how strongly one cone class contributes to the signal atsignals from M-cones. One suggestion about why this one post-receptoral site.occurs is that it is an eﬀective way to code the cone Considerable eﬀort has been devoted to establishingsignals for transmission down the optic nerve (see whether the linear form for the mapping between qWandell 1995). and u is appropriate, and if so, what values should be 2259
Color Vision Theoryused for the mij. Several types of experimental evidencehave been brought to bear on the question. As an example, one line of research argues that fourcolor perceptions, those of redness, greenness, blue-ness, and yellowness, have a special psychologicalstatus, in that any color experience may be intuitivelydescribed in terms of these four basic perceptions.Thus orange may be naturally described as reddish-yellow and aqua as greenish-blue. In addition, bothintrospection and color scaling experiments suggestthat the percepts of redness and greenness are mutuallyexclusive so that both are not experienced simul- Figure 1taneously in response to the same stimulus, and A color context eﬀect. The ﬁgure illustrates the colorsimilarly for blueness and yellowness (e.g., Hurvich context eﬀect known as simultaneous contrast. The twoand Jameson 1957, Abramov and Gordon 1994). central disks are physically the same but appearGiven these observations, it is natural to associate the diﬀerent. The diﬀerence in appearance is caused by theRG signal with the amount of redness or greenness fact that each disk is seen in the context of a diﬀerentperceived in a light (redness if the signal is positive, annular surround. This ﬁgure is best viewed in color.greenness if it is negative, and neither red nor green if A color version is available in the on-line version of theit is zero) and the BY signal with the amount of Encyclopediablueness or yellowness. Judgments of the four fun-damental color perceptions, obtained either through at other locations and at preceding times. To help ﬁxdirect scaling (e.g., Abramov and Gordon 1994) or ideas, it is useful to restrict attention to the disk-through a hue cancellation procedure (e.g., Hurvich annulus conﬁguration. For this conﬁguration, theand Jameson 1957), are then used to deduce the visual representation of the disk may be written asappropriate values of the mij in the second and thirdrows of Mo. When this framework is used, the entries ud l f (qd; qa, ) (13)for the ﬁrst row of Mo, corresponding to the LUMsignal, are typically established through other means where ud is the visual response to the disk, qd and qa aresuch as ﬂicker photometry (e.g., Kaiser and Boynton the cone coordinates of the disk and annulus re-1996). spectively, and represents other contextual variables Other approaches to studying the opponent trans- such as the size of the disk and annulus and anyformation include analyzing measurements of the temporal variation in the stimulus. Clearly, f( ) mustdetection and discrimination of stimuli (e.g., Wyszecki incorporate the sort of transformation described byand Stiles 1982, Kaiser and Boynton 1996, Eskew, the matrix Mo in Sect. 3.1 above.et al. 1999, Wandell 1999), and measurements of how As was the case with the discussion of opponencythe color appearance of lights is aﬀected by the context above, there is not wide agreement about how best toin which they are viewed (e.g., Webster 1996). In part model adaptation. A reasonable point of departure isbecause of a lack of quantitative agreement in the a cone-speciﬁc aﬃne model. In this model, the visualconclusions drawn from diﬀerent paradigms, there is representation u of a light is related to its conecurrently not much consensus about the details of the coordinates q through an equation of the formtransformation between q and u. One of the majoropen issues in color theory remains how to extend the u l Mo(D qkq ) (14) simple linear model described above so that it accountsfor a wider range of results. where Mo is as in Eqn. (12) and A C A C gL 0 0 qL D l 0 gM 0 , q l qM3.2 Adaptation (15) 0 0 gS qFigure 1 illustrates a case where the same light has a B D B S Dvery diﬀerent color appearance when seen in twodiﬀerent contexts. The ﬁgure shows two disk-annulusstimulus conﬁgurations. The central disk is the same in In this formulation, the g’s on the diagonals of Deach conﬁguration, but the appearance of the two characterize multiplicative adaptation that occurs at adisks is quite diﬀerent. To explain this and other cone-speciﬁc site in visual processing, before signalscontext eﬀects, mechanistic models assume that at any from separate cone classes are combined. The entriesgiven time and image location, the relation between of the vector q characterize subtractive adaptation.the quantal absorption rates q and the visual rep- Equation (14) is written in a form that implies that theresentation u depends on the quantal absorption rates subtractive adaptation also occurs at a cone-speciﬁc2260
Color Vision Theorysite. The entries of D and q depend on the cone 4. Color Constancy coordinates qa of the annulus as well as on spatial andtemporal variables characterized by . Note that the The discussion so far has focussed on how the visualcone-speciﬁc aﬃne model is a generalization of the system represents and processes the spectrum of lightidea that the visual representation consists of a that enters the eye. This is natural, since light is thecontrast code. proximal stimulus that initiates color vision. On the Asymmetric matching may be used to test the other hand, we use color primarily to name objects.adaptation model of Eqn. (14). In an asymmetric The spectrum of the light reﬂected to the eye from anmatching experiment, an observer adjusts a match object depends both on an intrinsic property of thestimulus seen in one context so that it appears to have object, its surface reﬂectance function, and on extrinsicthe same color as a test stimulus seen in another factors, including the spectral power distribution ofcontext. More concretely, consider Fig. 1. In the the illuminant and how the object is oriented relativecontext of this ﬁgure, an asymmetric matching ex- to the observer.periment could be conducted where the observer was Given that the light reﬂected to the eye varies withasked to adjust the central disk on the right so that it the illuminant and viewing geometry, how is it thatmatched the appearance of the central test disk on the color is a useful psychological property of objects? Theleft. Suppose such data are collected for a series of answer is that the visual system processes the retinalN test disks with cone coordinates qti. Denote the image to stabilize the color appearance of objectscone coordinates of the matches by qmi. Within the across changes extrinsic to the object (e.g., changes inmechanistic framework, the corresponding visual rep- the spectrum of the illuminant). This stabilizationresentations uti and umi should be equal. If Eqn. (14) process is called color constancy.provides a good description of performance then Color constancy is closely linked to the phenom- enon of adaptation described above (Maloney 1999). Indeed, quantitative models of color constancy gen- erally incorporate the same idea that underlies mech-Mo(Dm qmikqm ) l Mo(Dt qtikqt ) anistic models of visual processing: at some central site qmi l D− (Dt qtikqt jqm ) m (16) there is a visual representation u that correlates with qmi l Dtmqtikqtm color appearance. To stabilize this representation against changes in illumination, it is supposed that the relation between the quantal absorption rates q elicitedwhere Dtm l D− Dt and qtm l D− (qt kqm ). This by the light reﬂected from an object and the visualprediction may be checked by ﬁnding the diagonal m m representation u depends on the scene in which thematrix Dtm and vector qtm that provide the best ﬁt to object is viewed. In the case of color constancy, thethe data and evaluating the quality of the ﬁt. Tests of emphasis has been on how the visual system processesthis sort indicate that the cone speciﬁc aﬃne model the retinal image so that the transformation between qaccounts for much of the variance in asymmetric and u has the eﬀect of compensating for the variationmatching data, both for the disk annulus conﬁguration in the light reﬂected to the eye caused by changes of(Wandell 1995, 1999) and for more complex stimuli illumination and viewing geometry. Psychophysical(Brainard and Wandell 1992). Nonetheless, there are data on the color appearance of objects viewed underclear instances for which Eqn. (16) does not give diﬀerent illuminants are often well-modeled by trans-a complete account of asymmetric matching (e.g., formations consistent with Eqn. (14) (e.g., BrainardDelahunt and Brainard 2000) and other color ap- and Wandell 1992).pearance data (e.g., Webster 1996, Mausfeld 1998, The central theoretical question of color constancyD’Zmura and Singer 1999). is how the visual system can start with image data and The cone-speciﬁc aﬃne model may also be tested factor it into an accurate representation of the surfacesagainst psychophysical data on the detection and and illuminants in the scene. This question has receiveddiscrimination of colored lights. Here again the model extensive treatment, at least for simple scenes. A briefprovides a reasonable point of departure but fails in introduction to this literature on computational colordetail (e.g., Eskew et al. 1999). constancy follows. To extend the cone speciﬁc aﬃne model, varioustheorists have suggested the need for adaptation at a 4.1 Computational Color Constancysecond site (after signals from separate cone classeshave been combined) and for the inclusion of non- Consider a scene consisting of diﬀusely illuminatedlinearities in the relation between q and u (see ﬂat, matte surfaces. For such scenes, the spectrum breferences cited in the previous two paragraphs). An reﬂected to the eye from each surface is given by theadditional open question concerns how the entries of wavelength-by-wavelength product of the spectralD and q are determined by the viewing context power distribution of the illuminant e and the surface (e.g., Brainard and Wandell 1992, Delahunt and reﬂectance function s. The surface reﬂectance functionBrainard 2000). speciﬁes, at each sample wavelength, the fraction of 2261
Color Vision Theoryincident light reﬂected to the eye. The information linear model also describes illuminant spectral powerabout b coded by the visual system is its cone distributions, so thatcoordinates, which may be computed as e $ B ew e (22) q l S b l S diag(e) s (17) The second is that the spatial average of the surfacewhere the function diag( ) returns a square diagonal reﬂectance functions (s- ) is the same in all scenes andmatrix with its argument placed along the diagonal. known. These additional constraints imply thatClearly e and s are not uniquely determined fromknowledge of q: without additional constraints the ` V V q l [S diag(s` )Be]we l Ls` we (23)color constancy problem is underdetermined. For-tunately the spectra of naturally occurring illuminants - where q is the spatial average of the quantal absorp-and surfaces are not arbitrary. Although the physical tion rates and Ls- is a known three-by-three matrix.processes that constrain these spectra are not well Inverting Eqn. 23 yields an estimate for the illuminantunderstood, analyses of measurements of both illumi- # e l BeweV . This estimate is then used to provide thenants and surfaces shows that their spectra are well matrix L− to be used in Eqn. (21). eVdescribed by small-dimensional linear models (see Buchsbaum’s algorithm shows how the addition ofBrainard 1995, Maloney 1999). appropriate assumptions allows solution of the Consider surface reﬂectances. It is possible to deﬁne computational color constancy problem. The diﬃcultythree ﬁxed basis functions so that naturally occurring with Buchsbaum’s algorithm is that its assumptionssurface reﬂectances are reasonably well approximated are too restrictive. In particular, it seems unlikely thatby a linear combination of these basis functions. Thus the spatial average of surface reﬂectances is constantfor any surface, we have across scenes, nor do real scenes consist of diﬀusely illuminated ﬂat, matte surfaces. Subsequent work has s$ws bs jws bs jws bs (18) focused on ways to provide reasonable estimates of # # $ $ the illuminant and surface reﬂectances under otherwhere bs , bs , and bs are the spectra of the basis sets of assumptions (e.g., Maloney 1999).functions and ws , ws$ , and ws are scalar weights # # $that provide the best approximation of s within thelinear model. Eqn. (18) may be rewritten as 4.2 Computational Color Constancy and Human Performance s$Bsws (19) How does the computational work relate to humanwhere the three columns of matrix Bs contain the basis performance? This question has not yet been resolved,functions and the vector ws contains the scalar but it seems appropriate to close with a few obser-weights. vations. First, the estimated linear model weights of When the surface reﬂectance functions lie within Eqn. (21) may be associated with the mechanisma three-dimensional linear model Eqn. (17) may responses u discussed in Sect. 2. In both types of #inverted, once an estimate e of the illuminant has theory, these quantities represent the visual responsebeen obtained (see below for discussion of illuminant computed from the quantal absorption rates, and bothestimation.) Start by rewriting Eqn. (17) as: are meant to allow direct prediction of appearance. In the mechanistic approach, one considers a series of # q l [S diag(e)Bs] ws l Le# ws (20) transformations whose form is derived from experi- ments with simple stimulus conﬁgurations. In thewhere Le# is a three-by-three matrix that depends on the computational approach, the form of the transform-illuminant estimate. This matrix may be inverted using ation is derived from consideration of the problemstandard methods to yield an estimate of ws: color vision is trying to solve. In both cases, however, the emphasis is on ﬁnding the appropriate parametric ws l L− q V eV (21) form of the transformation and on understanding how the parameters are set as a function of the image data.The estimate may then be used together with Eqn. (19) The connection between the two approaches is dis-to estimate the surface reﬂectance function. cussed in more detail by Maloney (1999). Many computational color constancy algorithms The value of the computational approach to under-assume a linear model constraint for surface reﬂec- standing human vision depends on how accurately thetance functions. This reduces the constancy problem transformations it posits may be used to predict theto ﬁnding an estimate of the illuminant to plug into appearance of stimuli measured in psychophysicalEqn. (20). For illustrative purposes, an algorithm experiments. There have been only a few empiricaldue to Buchsbaum (1980) is described here. In comparisons of this sort to date. These comparisonsBuchsbaum’s algorithm, two additional assumptions do, however, indicate that the computational ap-are added. The ﬁrst is that a three-dimensional proach shows promise for advancing our understand-2262
Combinatorial Data Analysising of human color vision (Bloj, Kersten, and Hurlbert Gegenfurtner K, Sharpe L T (eds.) Color Vision: From Genes1999, Brainard, Kraft, and Longre 2001). to Perception. Cambridge University Press, Cambridge, UK, pp. 3–51 Stockman A, Sharpe L T 1999 Cone spectral sensitivities andSee also: Color Vision; Psychophysical Theory and color matching. In: Gegenfurtner K, Sharpe L T (eds.) ColorLaws, History of; Psychophysics; Vision, Low-level Vision: From Genes to Perception. Cambridge UniversityTheory of; Vision, Psychology of; Visual Perception, Press, Cambridge, UK, pp. 53–87Neural Basis of; Visual System in the Brain Wandell B A 1995 Foundations of Vision. Sinauer, Sunderland, MA Wandell B A 1999 Computational neuroimaging: color repre- sentations and processing. In: Gazzaniga M (ed.) The NewBibliography Cogniti e Neurosciences, 2nd edn. MIT Press, Cambridge, MA, pp. 291–303Abramov I, Gordon J 1994 Color appearance: on seeing red—or Webster M A 1996 Human colour perception and its adaptation. yellow, or green, or blue. Annual Re iew of Psychology 45: Network: Computation in Neural Systems 7: 587–634 451–85 Wyszecki G, Stiles W S 1982 Color Science—Concepts andBloj M G, Kersten D, Hurlbert A C 1999 Perception of three- Methods. Quantitati e Data and Formulae, 2nd edn. John dimensional shape inﬂuences colour perception through Wiley, New York mutual illumination. Nature 402: 877–9Brainard D H 1995 Colorimetry. In: Bass M (ed.) Handbook of Optics: Volume 1. Fundamentals, Techniques, and Design. D. H. Brainard McGraw-Hill, New York, pp. 26.1–26.54Brainard D H, Brunt W A, Speigle J M 1997 Color constancy in the nearly natural image. 1. Asymmetric matches. Journal of the Optical Society of America A 14: 2091–110 Brainard D H, Kraft J M, Longere P 2001 Color constancy: developing empirical tests of computational models. In: Combinatorial Data Analysis Mausfeld R, Heyer D (eds.) Colour Perception: From Light to Object. Oxford University Press, Oxford, UK Combinatorial data analysis (CDA) refers to a class ofBrainard D H, Wandell B A 1992 Asymmetric color-matching: methods for the study of relevant data sets in which How color appearance depends on the illuminant. Journal of the Optical Society of America A 9(9): 1433–48 the arrangement of a collection of objects is theBuchsbaum G 1980 A spatial processor model for object colour absolutely central concept. Characteristically, CDA is perception. Journal of the Franklin Institute 310: 1–26 involved with either: (a) the identiﬁcation of arrange-D’Zmura M, Singer B 1999 Contrast gain control. In: ments that are optimal for a speciﬁc representation Gegenfurtner K, Sharpe L T (eds.) Color Vision: From of a given data set, and where such an exploratory Genes to Perception. Cambridge University Press, Cambridge, process is typically carried out according to some UK, pp. 369–85 speciﬁc loss or merit function that guides a combina-Dacey D M 2000 Parallel pathways for spectral coding in torial search over a domain of possible structures primate retina. Annual Re iew of Neuroscience 23: 743–75 constructed from the constraints imposed by theDelahunt P B, Brainard D H 2000 Control of chromatic adaptation: Signals from separate cone classes interact. Vision particular representation selected; or (b) a conﬁrma- Research 40: 2885–903 tory determination as to whether a speciﬁc objectEskew R T, McLellan J S, Giulianini F 1999 Chromatic de- arrangement given a priori reﬂects the observed data, tection and discrimination. In: Gegenfurtner K, Sharpe L T and where such a conﬁrmatory process is typically (eds.) Color Vision: From Genes to Perception. Cambridge operationalized by comparing the empirically observed University Press, Cambridge, UK, pp. 345–68 degree of correspondence between some given data setHurvich L M, Jameson D 1957 An opponent-process theory of and the speciﬁc structure conjectured for it, to a color vision. Psychological Re iew 64(6): 384–404 reference distribution constructed from the collectionKaiser P K, Boynton R M 1996 Human Color Vision, 2nd edn. of all possible structures of the same form that could Optical Society of America, Washington, DCMaloney L T 1999 Physics-based approaches to modeling have been conjectured. surface color perception. In: Gegenfurtner K, Sharpe L T The boundaries of what CDA might encompass are (eds.) Color Vision: From Genes to Perception. Cambridge somewhat open but generally we would exclude University Press, Cambridge, UK, pp. 387–416 methods based on the postulation of strong stochasticMausfeld R 1998 Color perception: From Grassman codes to a models and their speciﬁc unknown parametric struc- dual code for object and illumination colors. In: Backhaus tures as underlying a given data set. Although CDA W G K, Kliegl R, Werner J S (eds.) Color Vision—Pers- might use or empirically construct various weighting pecti es from Diﬀerent Disciplines. Walter de Gruyter, Berlin, functions, the weights so obtained are not to be pp. 219–50 interpreted as parameter estimates in some presumedNeitz M, Neitz J 2000 Molecular genetics of color vision and color vision defects. Archi es of Ophthalmology 118: 691–700 stochastic model viewed in turn as responsible forRodieck R W 1998 The First Steps in Seeing. Sinauer, Sunder- generating the data. Manifest data are emphasized land, MA solely, and the traditional concern for an assumedSharpe L T, Stockman A, Jagle H, Nathans J 1999 Opsin genes, relationship between the data and a restrictively cone photopigments, color vision, and color blindness. In: parameterized stochastic model is avoided. For 2263Copyright # 2001 Elsevier Science Ltd. All rights reserved.International Encyclopedia of the Social Behavioral Sciences ISBN: 0-08-043076-7
Implications VOL. 03 ISSUE 5 www.informedesign.umn.edu A Newsletter by InformeDesign. A Web site for design and human behavior research. Seeing Color and was developed by a Finnish Color is the most dominant design ele- astronomer, Aron Sigfrid Forsius and ment, and ironically, the most relative was soon followed by Newton’s color aspect of design. The perception of color wheel in 1704. The primary objectives of involves human physiological and psy- these systems are to give order to the chological responses. Object, light, eye, variables of color and to concretely rep- and brain are involved in a complex resent colors, because “words are incom- process of sensation and perception. plete expression as color” (Munsell, Color attracts our attention, helps us 1981). Munsell developed a three- make sense of our environment, and dimensional color tree. The three vari- affects our behavior. Color plays a cul- ables of color — hue, value, and chroma tural role, an informational role, and are displayed on plexiglass branches, even a survival role. It functions on a one for each hue (see Figure 1). Darker basic level of appeal and can elicit strong values of the hue are toward the bottom; feelings of like or dislike. Color is a lighter values are toward the top. source of sensual pleasure (Pentak Brighter hues are seen at the outsideFigure 1: Munsell Color Wheel perimeter; duller hues are toward the Roth, 2003). center of the tree. A color wheel made of IN THIS ISSUE hats and shoes, featured in an exhibition Seeing Color Color Order Systems in the Goldstein Museum of Design, We are familiar with the most common arranged the objects in spectral order Typography and Color type of color arrangement—a color wheel (see Figure 2). Related Research arranged in spectral order. Spectral Summaries order is especially pleasing to the human Michel Eugene Chevreul developed a perceptual system. The spectrum occurs system to explain how colors affect each in nature in the refraction of light into other. As director of the Gobelins tapes- bands of color—red, orange, yellow, try studio (France), he realized that color green, blue, and violet. One hue gradates systems did not account for perceived into the next, creating a dynamic color color and that colors tend to tinge adja- sensation. cent hues with its complementary hue. In response, he designed a color circle Theoretical Color Systems that accounted for differences of satura- Scientists, artists, and color theorists tion and value within each hue family. have developed variations of the color He also created a framework about the wheel. The first wheel appeared in 1611 effects of simultaneous contrast.
Implications www.informedesign.umn.edu 2 The Effect of Surface Quality on Color Perception Surface quality contributes to the variability of color, “one and the same color evokes innumerable read- ings” (Albers, 1963, p. 1). This variability is due to differences in the human visual system, light, and the material and surface quality of the object. When we view the color of an object, we are really seeing reflected light. Objects are typically colored with either pigment or dyes. Dyes permeate the molecular structure of the object; pigments lay in a coat of color on the surface of an object. This difference is evident in viewing fabric that has been painted versus fabric that has been dyed.Figure 2: A color wheel made from hats and shoes that are in thecollection of the Goldstein Museum of Design. Surface materiality also affects the appearance of a color. A smooth, glossy surface will reflect a hue veryPractical Color Systems differently than a rough surface, and they tend toSpecialized color systems are used in product design reflect more light than a matte or rough surface.and manufacturing. Both the Pantone color system Matte or rough surfaces reflect light in a scattered,and the Munsell system are widely used. Pantone diffuse manner that randomly mixes the wavelengthshas developed color systems and products for the and tends to soften the color, changing it.graphic, interior, textile, architectural, and industri- Transparent materials allow color and light to beal design fields. Pantone has also recently begun seen through them (see Figure 3). Reflection fromforecasting color trends in fashion and interior glossy paper can make reading a menu or a magazinedesign. The primary goal of both the Munsell and difficult just as reflection from a highly polished floorPantone systems is to communicate color in a sys- can create spatial perceptual challenges.tematic way, leaving little room for error. The CIE(Commission Internationale de l’Eclairage) chro-maticity diagram displays a color matching systembased on light, and it is shaped like a luminositycurve. The system attempts to eliminate differencesof color perception through mechanical measure-ment of the three variables of a color—luminance,hue, and saturation. While these practical color sys-tems help to ensure accurate color specification,color appearance still varies due to lighting, context,and surface quality. Figure 3: Glass designed by Dale Chihuly, Museum of Glass, Tacoma, WA. Where Research Informs Design®
Implications www.informedesign.umn.edu 3Albers (1963) discusses the interdependence of color Color Harmonywith form and placement, quantity, and quality. It is There are strategies for creating color harmony:a constant challenge to predict how a color will look using similar values or hues, using hues with com-on the designed object when seen under different plementary contrast, or using a large number of hueslight sources. While the typical color wheel repre- in careful proportions. Constrast provides a sense ofsents only two or three dimensions, a color system visual balance. Munsell recommended balancingdeveloped by Albert-Vanel attempted to include vari- light and dark hues, dull and bright hues, and coolations due to surface quality, light, and human per- and warm hues. A sense of color harmony is basedception. This system, called the Planetary color sys- partially in human perception and partially in colortem and developed in 1983, includes not only hue, trends (see Figure 4).value, and chroma, but also accounts for contrastand material. Human Perception of Color Color can have a profound effect on humans. It can affect our brain waves, heart rate, blood pressure, and respiratory rate. Color also affects us emotional- ly. Exposure to color has an effect on our biological systems. Not only does color affect our sense of well- being, but it also may play a role in medical treat- ments for depression, cancer, and bacterial infec- tions. Visual Perception Our perception of color is dependent on light, object,Figure 4: The colorful facades of Burano, Italy. and our eyes and brain. We know that colors are influenced by adjacent colors, indeed, it is rare to seeDyes and Colorants an isolated color or color in its pure state. ChevreulThe color of objects is dependent on the pigments or discussed how colors tend to tinge neighboring huesdyes used in the production of the product. Color with their complement. Including color oppositestrends often evolve out of technological develop- within close proximity in a particular space can mit-ments. In the mid-1850s, William Henry Perkins igate this phenomenon. Surgical personnel in hospi-accidentally developed effective synthetic dyes for tals wear greenish-blue scrubs to counter-balancewool and silk as he attempted to synthesize quinine the visual effect of afterimages. During surgery, allfrom aniline. He named the color mauve. Other eyes focus on the patient and typically see a varietychemists developed synthetic aniline dyes that were of pink and red hues. The red receptors in the eyesignificantly brighter and more saturated than early would become fatigued if not for the color of thenatural dyes. This discovery, along with the develop- scrubs providing the opposite hue and thus balanc-ment of organic chemistry as a discipline, fueled the ing the visual experience.development of numerous synthetic dyes. Neon dyesand pigments that were developed in the mid-1980s Color contrast is essential for our understanding ofresulted in bright fabrics, accessories, and paper form and legibility. At least a 70% contrast betweenproducts. the background and letterforms is ideal for signs and Where Research Informs Design®
Implications www.informedesign.umn.edu 4painted materials. Conversely, too much contrast in nomena also affect the popularity of colors. Fashionan environment may increase anxiety and tension. prints in the 1960s used the bright palette of colorsSharp contrasts of color on flooring may create known as psychedelic. These colors were fully satu-uncomfortable illusions for walkers as they deter- rated and were intended to mimic the sensationmine whether the floor is flat or not. Research has caused by drugs (see Figure 5). Most of the informa-shown that the most visible combinations of colors tion about color meaning is highly subjective andare yellow and black, white and black, white and based on tacit beliefs, rather than research. There isblue, and red and white. a significant need for systematic research on color and human perception.Psychological Responses to ColorWe all react differently to color. We have different Typography and Colorcolor preferences, and we all have our least favorite Typography, the set of alphabetic characters, numer-colors. Color response is highly personal. What one als, and symbols used to compose copy, can beperson believes is a restful color, another may find manipulated in any number of ways by a graphicstimulating. Frequently these color preferences are designer. Size, typeface, letterspacing, leading (thebased on our own personal experience—a fondly space between lines of type), case (upper or lower case), structure (normal, light, bold, italic, bold ital- ic, etc.), and—of course—color can all be used to improve the legibility (how easy the text is to read), readability (how inviting the text is to a reader), and the hierarchy or structure of typeset copy. While each of the previously mentioned characteris-Figure 5: Fabric samples from 1960s-era clothing. tics can be manipulated by designers setting type, color is an especially important property. We oftenremembered yellow kitchen that belonged to grand- imagine type (or copy) that is set in black on a whitemother. There are also cultural associations that background—this is perhaps the most familiar wayinfluence our reactions to color. In several cultures, to set type on a printed page. However, when weblue is seen as peaceful, protecting, and soothing think of typography in signage and the built environ-color. Red typically signifies passion and revolution. ment, a variety of colors and color combinations,There are multiple associations for each color. For come to mind. Consider the new, colorful green andexample, black may be seen as sophisticated or as yellow logo signage of BP (British Petroleum) that isdepressing. Orange can be warm or aggressive. employed in the design of gas stations. Or, think ofYellow can be upbeat or acidic. the familiar white type on a green background of road signs. Color is employed frequently in environ-Marketing research attempts to discover what colors mental signage to create a memorable identity thatinfluence human behavior and how people will act helps users navigate a space, remember the businesswhen they shop, eat, or travel. Findings by market- or company, and create a pleasant impression.ing researchers are typically short-lived, however;trends seem to come and go, and other variables in When creating environmental signage, it is critical toaddition to color affect behavior. While technology consider some of the variables associated with thecontributes to color trends, culture and social phe- application of color. Here are a few ideas and tips: Where Research Informs Design®
Implications www.informedesign.umn.edu 5 This is not an exhaustive list of issues to consider when applying color to environmental signage and typography. If possible, it is beneficial to have a graphic designer who understands the interactions between typography, color, and the built environ- ment on a design team when designing environments with signage. In addition, InformeDesign has Research Summaries about graphic design for the built environment. References —Albers, J. (1963). Interaction of Color. New Haven, CT: Yale University Press.Figure 6: An example of poor and excellent contrast between —Fehrman, K., Fehrman, C. (2004). Color: Thetypography and background. Secret Influence. Upper Saddle River, NJ: Prentice• Consider the contrast between the color of the Hall. typography and the background to ensure that the —Munsell, A. H. (1946). A Color Notation. Baltimore: type is easy to decipher and read. Type/back- Macbeth. ground color combinations can cause the text to —Pentak, S., Roth, R. (2003). Color Basics. either advance or recede (see Figure 6). Stamford, CT: Wadsworth.• Consider the impact of color on interpretation and —Sharpe, D. (1981). The Psychology of Color and understanding of the content. What does a red Design. Totowa, NJ: Littlefields, Adams Co. heading indicate versus a brown heading? Does —Stromer, K. (Ed.). (1999). Color Systems in Art and setting less important information in a brighter, Science. Edition Farbe/Regenbogen Verlag. more prominent color impact the order that infor- —Walch, M., Hope, A. (1990). The Color mation is retrieved? Compendium. New York: Van Nostrand Reinhold.• Consider the user. Be aware of the cultural context of the environment and the signage, and consider About the Authors: cultural norms for particular colors. For example, Barbara Martinson, Ph.D., in Europe and the US, red typography generally is the Buckman Professor of means warning or attention. The application of Design Education in the color to type can either play into cultural norms for Department of Design, color or can contradict them. Housing, and Apparel,• Consider the lighting levels of the environment. University of Minnesota. While a color combination may work well when She has taught founda- evaluated in your office, the combination may be tions-level color courses for inappropriate when the lighting levels are different. 20 years, as well as graphic• Consider the properties of the signage material. design, design history, and How will a surface that is reflective or flat change human factors courses. She the legibility of the content? How will lighting levels recently curated Seeing interact with the surface properties? Color, an exhibition at the Goldstein Museum of Where Research Informs Design®
Chapter 13How does visual memory work?Photo courtesy of Ann Cantelow. The multichannel neuron model ascribes numbers to channels. The channelnumbers store and communicate analog data. They can also be used, in a distinct addressing system, tosequentially query the twigs of visual memory.Addressing and retrievalFor retrieval, the model requires two types of neurons: 1) an address generating neuron, which drives 2) a datastorage neuron. To activate a memory stored as a thing in a place, a stored datapoint must be addressed atprecisely that place. In the speciﬁc case of a stored pattern of three bleached disks imported from aphotoreceptor, a trio of associated datapoints, twigs, must be addressed, one right after the other.We have a mechanism for generating sequential addresses. The principle is inherent in the multichannel neuronmodel. The address generator can be the commutator we have postulated at the axon hillock.
To stimulate the ﬁrst 9 twigs of memory, #1 through #9, each in turn, requires this sort of circuit. The outputlines of the axon driven by the addressing commutator are telodendrions, each corresponding to a channel. Inthis illustration of this model, telodendrions are numbered in order of their ﬁring. Each individual channelsynapses to a dendrite. Each dendrite will be stimulated in its turn, in accordance with the ascending circularorder of the addressing commutator.Each dendrite is a “twig memory”. It stores a channel number that stipulates which channel shall be ﬁred inresponse to the addressing signal. The effect can be tabulated:
The dendrites, which comprise the twigs of memory in this simple model, are each stimulated in turn. Thepattern of bleached disks that each twig has memorized is ﬁred back into the nervous system – preciselyreplicating the pattern originally dispatched from a single photoreceptor’s outer segment at some time and dayin the past. In the table, 9 upticks of the address counter’s commutator correspond to a trio of 3D pixels and 3frames of a ﬁlm strip. [A slicker model might use just one address tick to elicit all three datapoints,characterizing intensity, wavelength, phase -- but the point is, visual memory is sampled and read out by theticking of a sequential address counter. It is probably written in the same way.]All pixels recorded from the retina at the same time, stored in twigs on other photoreceptor antipodal treeswill have identically the same time stamp in their address. So simultaneously, synchronously, one pixel fromevery other “tree” or photoreceptor antipode in the retina of memory is being triggered.The effect is to pump out of memory a stream of past images -- each image made up of millions of 3D pixels.The system is massively parallel and, therefore, moves whole images all at once. It is lightning fast.Why dont we see these torrents of images from the past? Why arent we drowning in images? Because theseare not literal images. They are images of the Fourier plane. Fourier images are invisible to us, except perhapsin the special case of LSD users. Literal images may impinge on the consciousness as, in effect, searchproducts, but the search itself is conducted as a Fourier process and is unconscious -- offstage and out of sight.Numbered synapses -- new evidence, old ideaThe idea there might be some sort of detectable ordering or sequencing of synapses on the dendrites isattributed to Wilfrid Rall, who suggested it in 1964 in support of a wholly different and unrelated model of thenervous system. In the 24 September 2010 issue of Science there is a featured report that reinforces the notionthere exists some sort of sequentially ordered input pattern in the dendrites.
In these experiments, a programmed series of successive stimuli is made to “walk” from synapse to synapsealong the dendrite. If the stimulus series progresses toward the cell body it is more likely to trigger off actionpotentials than a programmed series of stimuli that walks the other way, away from the soma, toward the tips ofthe dendrites.The front half of this experiment consists of the selective stimulation of a row of individual dendritic spines,one after another, using a laser to precisely localize release of glutamate. The basic technology was outlinedhere. The back half of the experiment is conventional, and consists of electronic monitoring and tabulation ofthe axon’s response.In terms of the multichannel model electrophysiology is difﬁcult to interpret. However, a signiﬁcant feature ofthe model is a staircase of ﬁring thresholds. One might speculate that as the stimulus is made to approach thesoma, it is ﬁnding or ultimately directing a pointer to lower and lower ﬁring thresholds, which is to say, lowerchannel numbers. These low numbered channels would be more easily triggered than higher numberedchannels.Unfortunately there is easy no way to directly measure or guess the channel number associated with an actionpotential in passage, if indeed multiple channels exist. Again in terms of the model, a plot of channel numbersversus synapse position on dendrites (or, using different techniques, on the teledendrions) would produce afascinating picture. In any event it is interesting that even conventional electrophysiology suggests there may besome kind sequential ordering, progression, or directional structuring that underlies a map of dendritic spines.The modelIn modeling this visual memory system I think it would be best to use automated rotating or looping machinery,just as you would in many familiar recording and playback devices. The rotating machine is the commutator. Ateach addressing tree, let the loftiest addressing commutators walk forward through time automatically,incrementing higher channel by channel. Rough synchronization among trees should sufﬁce. Now, instead ofhardwiring and broadcasting addresses in detail, the retrieval system can simply be given a start date/time andtriggered off. A string of retrieval instructions will ensue. The system will, in effect, read itself out like a diskdrive.As a practical matter, the model of a retina of memory should probably be constructed in software. Each tree ofmemory can be modeled as a disk drive storing analog numbers representing 3D pixels, stacked in serial order,that is, the order or sequence in which they were originally captured from the eye. Millions of disk drives, then,each of relatively modest capacity, comprise a retina of memory. In a primitive animal one would expect to ﬁnda single retina of memory. In a sophisticated animal, many.Let’s say the memory trees pre-exist in a newborn animal and that their twigs are unwritten. Each branch is apoint in a commutator sequence, and identiﬁes time (that is, sequence) ranges.From the point of view of addressing the visual memory, reading and writing are, as in a disk drive, similarprocesses. The writing commutator walks forward through the present moments, guiding incoming 3D pixelsfrom the eye to a series of novel addresses. To elicit a visual memory a reading commutator, which could be theself-same machine, walks forward through addresses denoting a ﬁlm strip of past moments.In effect, the pointer of the base commutator on the address generator, as it ticks ahead, is the pointer of thesecond hand of a system clock. Although the images are recorded at a stately and regular rate, such as one persecond -- the recall can be made to happen as fast as the commutator is made to sweep. And it could scanbackwards as well as forwards.How is a pixel memory deployed?This is an unsolved problem in the model. We have to assume it happens but the answer isnt easy or obvious.
We have stipulated what a 3D pixel memory is: Three numbers -- integers -- that represent a pattern of lightrecorded from three disks in a single photoreceptor at a particular moment in time. The three numbers aresufﬁcient to specify the instantaneous wavelength, intensity and phase of the incoming light, as read out of astanding wave in the outer segment of the photoreceptor.We are suggesting these three numbers are conﬁgured and stored in the brain as an addressable twig ofmemory -- three dendritic launch pads for three action potentials to be ﬁred down three speciﬁc, numbered axonchannels. It is nicely set up, this memory, but how did it happen?The operation of an initial readout commutator in the addressing neuron seems clear. It simply counts up ordown. Other commutators fan out from the initial or system counter. At the upper tier of the addressing tree, thecommutators, once toggled, can tick forward “on automatic.”But what about the commutator in the memory neuron?In the most basic model of the multichannel neuron, developed in Chapter 2, the neuron is functioning as asensory transducer. The commutator pointer rotates up to a speciﬁc numbered channel in proportion to an inputvoltage or graded stimulus.But in the memory neuron, we want the pointer to go, ﬁrst, straight to a remembered channel. Then, second, toanother remembered channel. Then, third, to another remembered channel. Hop hop hop. From the addressneuron the memory neuron receives three signals in a sequence, via telodendrions 1, 2, 3. The data neuron ﬁreschannels corresponding to three remembered photoreceptor disk positions: 2, 7, 34.Instead of responding proportionately to an input voltage, as in a sensory neuron, the commutator in thememory neuron is responding discontinuously to a memorized set of three channel ﬁring instructions. So theneedle of this commutator must swing, not in response to an analog voltage input, but in response to a pixelmemory.In the multichannel model synapses connect individual channels, rather than individual neurons. It could be thatthe commutator is simply bypassed, so that the appropriate axon channels are hardwired to the dendritic twigsof memory. Synapses at the soma could suggest a short cut past or a way to overrule the inherent commutator.Maybe there is some rewiring or cross wiring at the level of the dendritic synapses. To borrow a term of artfrom the conventional playbook of memory biochemistry, maybe the synapses are subject to tagging. Maybebiochemical markers delivered into the dendrites when the memory was originally recorded are specifying insome way the channel numbers to be ﬁred.This model suggests a Y-convergence of three neurons, not just two. One delivers addresses. One stores the
data. A third neuron delivers original data from the retinal photoreceptor – data to be written in sequential orderinto the dendrites of the memory neuron.Whatever speciﬁc mechanism one might choose or invent, the model requires that pixel memory arriving from aphotoreceptor in the eye be stored in an antipodal neuron as a trio or linkage of three distinct channel numbers.ExperimentOne interesting aspect of this memory model is that it suggests an experiment. We are guessing that theindividual channels of an addressing axon are, in effect, split out and made accessible as numberedtelodendrions. If there is indeed a numerical succession – a sequential ﬁring order – of the telodendrions, thenthis should be detectable. We were taught that the telodendrions must ﬁre simultaneously. Is this always true? Ibet not.Superimposed networksNote that we have assumed there exists a double network. Above the information tree there is a second tree, areplica of the ﬁrst, used to individually address each memory twig.The principle of two superimposed networks, one for content and the other for control, is a technicalcommonplace. An early application was the superimposition of a telegraph network as a control system for therailway network. The egregious present day example is the digital computer, with its superimposed but distinctnetworks for information storage and addressing.We are long in the habit of dividing the nervous system into afferent and efferent, sensory and motor, butsurely there must be other ways to split it, e.g., into an information network and a addressing network. It istypically biological that one network should be a near replica of the other. Evolution proceeds throughreplication and modiﬁcation.Arborization and addressing capacityThe ﬁrst anatomist who isolated a big nerve, maybe the sciatic, probably thought it was an integral structure –in essence, one wire. Closer scrutiny revealed that the nerve was a bundle of individual neurons. We areproposing here yet another zoom-down in perspective, this time to the sub-microscopic level . We suspect thateach neuron within a nerve bundle is itself a bundle of individual channels.
It follows that the functional wiring of the nervous system is at the level of channels. Synapses connectchannels, not neurons. This is why one might count 10,000 synaptic boutons on a single neuron’s soma. Theboutons were not put there, absurdly, to “make better contact” nor to follow the textbook model of signalintegration. They are speciﬁc channel connectors, each with a speciﬁc channel number.The neuroanatomical feature that most interests us at this point is axon branching. This is because branching isof paramount importance in familiar digital technologies for addressing – search trees and other data structures.We have proposed a treelike addressing system for the visual memory in the brain. It is reasonable to ask --where are the nodes?Not at the branch points.Photo courtesy of Ann CantelowBranching in a nerve axon is just a teasing apart and re-routing of the underlying channels. It is not a branchingmarked by nodes or connections in the sense of an T or Y connected electrical branch, or a logical branch in abinary tree.For an axon that addresses a dendritic twig of memory, all functional branching occurs at the commutator.Any anatomical branching downstream of the commutator, such as the sprouting from the axon oftelodendrions , simply marks a diverging pathway – an unwinding or unraveling, rather than a distinct node orconnection. In other words, the tree is a circular data store. The datapoints are stored at twigs mounted on theperiphery of a circle. The twigs are accessible through a circular array of addresses. It is analogous to a diskdrive in which the disk holds still and the read-write head rotates.
Photo courtesy of Ann CantelowSummary of the technology to this pointThe tree in this photograph is a metaphor for the brain structure which corresponds to, and is antipodal to, asingle photoreceptor of the eye. It is one single photoreceptor cells remote memory warehouse -- a tree ofmemory.Each twig is a destination with an address, a neuronal process narrowed down to just two or three channels.For example channels 3, 7 and 29, only, might constitute a given twig. Each twig is a 3D pixel frozen in time.The tree will store as many unique picture elements from the photoreceptor’s past as it has twigs.As many as 125 million of these trees will constitute a retina of memory. We will look for ways to hack downthis number, but for the moment let it stand. The point is, we are talking about millions of trees.All these trees must be queried simultaneously with a particular numerical address, probably associated with atime of storage, to elicit ﬁring from all the right twigs -- just one twig per tree. Properly addressed, a forest ofthese trees will recreate, almost instantly, a whole-retina image from memory.In a primitive animal, it would be sufﬁcient to remember 300 images from the recent past. This could beaccomplished with a single addressing neuron, a single commutator. But in a modern mammal, it will benecessary to stack the commutators. A bottom commutator can point to any of 300 other commutators. Andeach of these can, in turn, point to 300 more commutators. With a simple tree of neurons, which is to say, alogical tree built with commutators, one can very quickly generate an astronomical number of unique addresses.We require one unique address for each twig of the data trees.Are there enough addresses available in this system to organize a mammalian lifetime of visual memories? Yes.Easily. Are there enough memory neurons to match the addressing capacity of the addressing neurons.Probably not. The neuronal brain that lights up our scanners is probably running its memory neurons as ascratchpad memory. It seems likely there is a deeper store.But will it work?The memory mechanism we have sketched is probably adequate as a place to start. It would work for adirectional eye in which changes in wavelength are highly signiﬁcant cues to the position and movement of atarget. It is a visual memory for retaining the just now, a ﬁlm strip comprising a few recent frames.