On colour – a visual list
Upcoming SlideShare
Loading in...5

On colour – a visual list



Be aware: this presentation is malformed, putting the slides on top of each other. However the download is OK again!

Be aware: this presentation is malformed, putting the slides on top of each other. However the download is OK again!



Total Views
Views on SlideShare
Embed Views



2 Embeds 2

http://www.slideshare.net 1
http://www.linkedin.com 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Be aware: this presentation is malformed, putting the slides on top of each other. However the download is OK again!

    Note: please visit > http://saaltinkworks.businesscatalyst.com/b.html if you want to download a number of selected titles on color FOR FREE.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

On colour – a visual list On colour – a visual list Document Transcript

  • On colourSome referencesA visual list
  • Books
  • On the web
  • Papers
  • Color Vision(e.g., spatial frequency, orientation, motion, depth) which the stimulus differs perceptually from a purelywithin a local cortical region. With respect to color achromatic (i.e., white, gray, black) axis. The thirdvision per se, the primary processing involves separ- dimension is brightness or lightness. That our per-ating color and luminance information, and further ceptual space is three-dimensional reflects the basicseparating changes due to the illuminant from those trichromacy of vision.due to visual objects, by lateral interactions over large A normal observer can describe the hue of any lightregions. (disregarding surface characteristics) by using one or To separate luminance and color information, the more of only four color names (red, yellow, green, andoutputs of Pc cells are combined in two different ways. blue). These so-called unique hues form two opponentWhen their outputs are summed in one way, the pairs, red–green and blue–yellow. Red and greenluminance components to their responses sum and the normally cannot be seen in the same place at the samecolor components cancel. Summed in a different time; if unique red and unique green lights are added incombination, the color components sum and the appropriate proportions, the colors cancel and oneluminance components cancel. Consider a striate sees a neutral gray. Orange can be seen as a mixture ofcortex cell that combines inputs from one or more red and yellow, and purple as a mixture of red andjLo and jMo cells in a region. The cortical cell would blue, but there is no color seen as a red–green mixturerespond to luminance variations but not to color (or as a blue–yellow mixture). This perceptual op-variations, since the neurons providing its inputs both ponency is also reflected in color contrast. Red canfire to luminance increments in the RF center and to induce the appearance of green into neighboringdecrements in the surround, but the color organi- regions, and after staring at a red surface one sees azations of its inputs are opposite to each other (one green after-image. The yellow–blue opponent pairbeing L-M and the other M-L). Combined with input produces similar effects. It was these perceptualfrom a jSo cell, this would produce a V1 cell that fires characteristics of color that led Ewald Hering in theto white (light increments) and inhibits to black (light nineteenth century to propose that the various colordecrements) but does not respond to pure color systems were not independent but rather that colorvariations. This is represented in the top row of Fig. was processed in a spectrally opponent organization,1C. However, a V1 cell receiving inputs from both an idea which has since been amply verified in thejLo and kMo cells, or from both jMo and kLo cells presence, discussed above, of spectrally-opponent cells(columns in Fig. 1C), would respond to color changes in the path from receptors to the cortex.but not to luminance variations since their colorresponses would add, but their luminance RFs, which See also: Color Vision Theory; Vision, Low-levelare opposite to each other, would cancel. This organi- Theory of; Vision, Psychology of; Visual Perception,zation by itself would produce L-M color cells that Neural Basis of; Visual System in the Brainwould fire to so-called warm colors (red and yellow)and inhibit to cool colors (blue and green). M-L cellswould fire to cool colors and inhibit to warm colors. BibliographyAs shown in Fig. 1C, the further addition of jSo or De Valois R L, De Valois R L 1988 Spatial Vision. OxfordkSo cells can split these classes into separate red University Press, New Yorkand yellow, and separate blue and green systems, Hurvich L M 1981 Color Vision. Sinauer Press, Sunderland, MArespectively. Kaiser P K, Boynton R M 1996 Human Color Vision. Optical All of the primary visual information is passed Society of America, Washington, DCthrough V1, but subsequent visual areas are partially Neitz J, Neitz M 1998 Molecular genetics and the biologicalspecialized for the further analysis of various different basis of color vision. In: Backhaus W G S, Kliegl R, Wernerfunctional aspects of vision. One later visual area (V4) J S (eds.) Color Vision. Walter de Gruyter, Berlin, pp. 101–19 Spillmann L, Werner J S 1990 Visual Perception: The Neuro-is crucially involved with color perception. Individuals physiological Foundations. Academic Press, New Yorkwith localized V4 lesions can still discriminate objectson the basis of their color variations, but they report K. K. De Valois and R. L. De Valoisthat the objects now appear to have no hue, as ifviewed on a black-white television screen. There is alsoa report of one case with the reverse loss: a patient whocould see colored but not black-white objects. Color Vision Theory11. Color Appearance Color vision is the ability to distinguish and identifyThe appearance of a color can be specified by values lights and objects on the basis of their spectralalong just three perceptual dimensions known as hue, properties. This entry presents several key topics thatsaturation and brightness. Hue refers to the character- underlie current theories of human color vision. Theseistic described by such color names as red, yellow, are trichromacy, color opponency, adaptation, andgreen, and blue. Saturation refers to the extent to color constancy.2256
  • Color Vision Theory1. Introduction primary intensities until the mixture has the same color appearance as the test light. The primaries usedInformation about color is transformed as it flows in the experiment are chosen to be independent, sofrom the stimulus through the initial stages of the that no weighted mixture of any two produces a matchhuman visual system. At each image location, the to the third.color stimulus is specified by the amount of power it Because the matching light is constrained to be acontains at each wavelength. The classic color match- weighted mixture of three primaries, it will noting experiment shows that the normal human visual generally be possible for the observer to make the testsystem is trichromatic: only three dimensions of and matching lights physically identical. For manyspectral variation are coded by the visual system. The test lights, however, the observer can adjust thebiological basis of normal trichromacy is that the matching light so that it appears identical to the testretina contains three classes of cone photopigment. light even though the two differ physically. For someAfter the initial encoding of light by the cones, further test lights, no choice of primary intensities will affordprocessing occurs. Two aspects of this processing are a match. In these cases one or more of the primariesparticularly important. First, signals from three classes can be mixed with the test light and primary intensitiesof cones are recombined to form a luminance and two found so that the primarytest mixture matches thecolor opponent channels. Second, there is adaptive mixture of the remaining primaries. A useful descrip-signal regulation that keeps neural signals within their tive convention for the color matching experiment isoperating range and stabilizes the appearance of to assign a negative intensity to any primary that mustobjects across changes of illumination. be mixed with the test to make a match. Given this convention, any test light can be matched by a mixture of three independent primaries.2. Trichromacy The color matching experiment is an empirical system. Given a test light described by a vector b, the2.1 Color Matching experiment returns a vectorThe physical property of light relevant for color vision A Cis the spectral power distribution. A light’s spectral t "power distribution specifies the amount of power it tl t (2)contains at each wavelength in the visible spectrum, # t B $ Doften taken to lie roughly between 400 and 700 nm. Inpractice, spectral power distributions are measured at whose entries are the individual primary intensities.discrete sample wavelengths. Let the measured power When the primaries are scaled by these intensities andvalues be denoted by b , …, bNλ where Nλ denotes the "number of sample wavelengths. Then the vector mixed, a match to the test light is created. The vector t specifies what are called the tristimulus coordinates A C of the light b. A theory of color matching should let us b " predict t for any test light b, given the spectral power distributions of the primary lights. bl < (1) As an empirical generalization, the color matching system is a linear system (e.g., Wyszecki and Stiles 1982, Brainard 1995, Wandell 1995). That is, if we bNλ have two test lights b and b with tristimulus coordinates t and t , then any # weighted mixture " B Dprovides a compact representation of the spectral " # (a b ja b ) of the two test lights has tristimuluspower distribution. Use of a vector representation for " " # # coordinates given by the corresponding mixturespectral quantities facilitates a variety of colorimetric (a t ja t ). In these vector expressions, multiplicationcomputations (e.g., Brainard 1995). Wavelength of" a vector (e.g., b ) by a scalar (e.g., a ) consists of " ##sample spacings between 1 and 10 nm are typical. " " multiplying each entry of the vector by the scalar, Trichromacy is demonstrated by the basic color while addition of two vectors (e.g., a b and a b )matching experiment (Wandell 1995, Brainard 1995). " " # # consists of adding the corresponding entries of the twoIn this experiment, an observer views a bipartite field. vectors.One side of the field contains a test light. This light is The linearity of color matching makes it possible toexperimentally controlled and can have an arbitrary predict the match that will be made to any test light onspectral power distribution. On the other side of the the basis of a relatively small number of measurements.field is the matching light. This consists of the weighted Consider the set of monochromatic lights with unitmixture of three primary lights. Each primary has a power. If Nλ wavelength samples are used in thefixed relative spectral power distribution, but its underlying representation, then there are Nλ suchoverall intensity in the mixture can be controlled by lights and we can denote their spectral representationsthe observer. The observer’s task is to adjust the by c , c , …, cNλ. Each of the ci has a 1 as its ith entry " # 2257
  • Color Vision Theoryand zeros elsewhere. Note that any light b may be this hypothesis (see Wandell 1995, Rodieck 1998).thought of as a weighted mixture of monochromatic First, the responses of individual cones depend onlylights, so that b l  bici where bi is the ith entry of b. on the rate at which photopigment molecules are iLet the vectors ti specify the tristimulus coordinates isomerized by the absorption of light quanta; once theof the monochromatic lights ci. The linearity of color intensity of two lights has been adjusted so that theymatching then tells us that the tristimulus coordinates produce the same isomerization rates, the cone re-of any light b are given by t l  biti. sponse does not distinguish the two lights. This idea is i A set of tristimulus values ti measured for mono- referred to as the principle of univariance. Second,chromatic lights ci is referred to as a set of color individual cones may be classified into one of threematching functions. Although these are often plotted distinct types, each with a characteristic spectralas a function of wavelength, they do not represent the sensitivity. The spectral sensitivity is proportional tospectral power distributions of lights. The color the probability that light quanta of different wave-matching functions may be specified by a single matrix lengths will isomerize a molecule of the cone’s photo- A C pigment. The three types of cones are often referred to Tl t t t (tNλ D (3) B" # $ as the long- (L), middle- (M), and short- (S) wave-whose Nλ columns consist of the individual tristimulus length-sensitive cones. If an observer has only threecoordinate vectors ti. This specification allows com- types of cones, each of which obeys the principle ofputation of tristimulus coordinates from spectral univariance, two physically distinct lights that producepower distributions through simple matrix multipli- the same isomerization rates for all three classes ofcation: cones will be indistinguishable to the visual system. Quantitative comparison confirms that color matches t l Tb. (4) set by a standard observer (defined as the average of matches set by many individual observers) are wellBoth tristimulus values and color matching functions predicted by the equations of isomerization rates inare defined with respect to the primaries chosen for the the L-, M-, and S-cones.underlying color matching experiment. The Com- As described above, trichromacy occurs for mostmission Internationale de l’Eclairage (CIE) has stan- observers because their retinas contain cones withdardized a system for color representation based on three classes of photopigments. Genetic consider-the ideas outlined above. The CIE system is widely ations, however, indicate that some individuals haveused to specify color stimuli and many sources describe retinas containing four classes of cone photopigmentsit in detail (e.g., Wyszecki and Stiles 1982, Brainard (Sharpe et al. 1999). Either these individuals are1995, Kaiser and Boynton 1996). tetrachromatic (mixture of four primaries required to The advantage of using tristimulus coordinates to match any light) or else their trichromacy is mediateddescribe color stimuli is that they provide a more by information lost after quantal absorption. Incompact and tractable description than a description addition, some human observers are dichromatic (onlyin terms of wavelength. Tristimulus coordinates are two primaries must be mixed to make a match to anycompact precisely because they do not preserve physi- light.) Most cases of dichromacy occur because onecal differences that are invisible to the human visual photopigment is missing (Sharpe et al. 1999, Neitz andsystem. The representational simplification afforded Neitz 2000).by tristimulus coordinates is extremely valuable for An alternative to using tristimulus coordinates tostudying processing that occurs after the initial encod- represent the spectral properties of lights is to use coneing of light. On the other hand, it is important to coordinates. These are proportional to the isomeriz-remember that the standard tristimulus represen- ation rates of the three classes of cone photopigments.tations (e.g., the CIE system) are based on matches The three dimensional vectormade by a typical observer looking directly at a small A Cstimulus at moderate to high light levels. These qLrepresentations are not necessarily appropriate forapplications involving some individual observers, non- q l qM (5)human color vision, or color cameras (e.g., Wyszecki q B S Dand Stiles 1982, Brainard 1995). specifies cone coordinates where qL, qM, and qS denote the isomerization rates of the L-, M-, and S-cone2.2 Biological Basis of Color Matching photopigments respectively. It can be shown (e.g.,The color matching experiment is agnostic about the Brainard 1995) that cone coordinates and tristimulusbiological mechanisms that underlie trichromacy. It is coordinates are related by a linear transformation, sogenerally accepted, however, that trichromacy typi- thatcally arises because color vision is mediated by three q l Mtqt (6)types of cone photoreceptor. Direct physiologicalmeasurements of individual primate cones support where Mtq is an appropriately chosen 3 by 3 matrix.2258
  • Color Vision Theory Computation of cone coordinates from light spectra A possible approach to understanding post-absorp-requires estimates of the cone spectral sensitivities. tion processing is to keep the modeling close to theFor each cone class, these specify the isomerization underlying anatomy and physiology and to character-rates produced by monochromatic lights of unit ize what happens to signals at each synapse in thepower. The sensitivities may be specified in matrix neural chain between photoreceptors and some site inform as visual cortex. The difficulty is that it is not presently possible to cope with the complexity of actual neural A C sL processing. Thus many color theorists have attempted S l sM (7) to step back from the details and develop more abstract descriptions of the effect of neural processing. s B S D Models of this sort are often called mechanistic models. These models generally specify a transform-where each row of the matrix is a vector whose entries ation between the quantal absorption rates q elicitedare the spectral sensitivities for one cone class at the by a stimulus and a corresponding visual represen-sample wavelengths. Given S, cone coordinates are tation u postulated to exist at some central site. Thecomputed from the spectral power distribution of a idea is to choose a transformation so that (a) the colorlight as appearance perceived at a location may be obtained q l Sb (8) directly from the central representation corresponding to that location and (b) the discriminability of twoBecause Eqns. (4), (6), and (8) hold for any light stimuli is predictable from the difference in theirspectrum b, it follows that central representations. Most mechanistic models assume that signals from S l MtqT (9) the cones are combined additively to produce signals at three postreceptoral sites. Two of these sites carryCurrent estimates of human cone spectral sensitivities opponent signals. These are often referred to as theare obtained from color matching data using Eqn. (9) red-green (RG) and blue-yellow (BY) signals. A thirdtogether with a variety of considerations that put site carries a luminance (LUM) signal, which is notconstraints on the matrix Mtq (Stockman and Sharpe thought to be opponent. If we take1999). A C uLUM3. Postabsorption Processing u l uRG (10) u B BY DColor vision does not end with the absorption of lightby cone photopigments. Rather, the signals thatoriginate with the absorption of light are transformed to be a three-dimensional vector with entries given byas they propagate through neurons in the retina and the LUM, RG, and BY signals, then the additivecortex. Two ideas dominate models of this post- relation between cone coordinates q and the visualabsorption processing. The first is color opponency: representation u may be expressed in matrix form:signals from different cone types are combined in anantagonistic fashion to produce the visual represen- u l Moq (11)tation at a more central site. The second idea is Many (but not all) detailed models take LUM to be aadaptation: the relation between the cone coordinates weighted sum of L- and M-cone signals, RG to beof a light and its central visual representation is not a weighted difference between the L- and M-conefixed but depends instead on the context in which the signals, and BY to be a weighted difference betweenlight is viewed. Section 3.1 treats opponency, while the S-cone signal and a weighted sum of the L- and M-Sect. 3.2 treats adaptation. cone signals. In these models Mo would have the form A C m m 03.1 Opponency "" "# Mo l m km 0 (12)Direct physiological measurements of the responses of #" ##neurons in the primate retina support the general idea km km m B $" $# $$ Dof opponency (e.g., Dacey 2000). These measurementsreveal, for example, that some retinal ganglion cells where all of the mij are positive scalars representingare excited by signals from L-cones and inhibited by how strongly one cone class contributes to the signal atsignals from M-cones. One suggestion about why this one post-receptoral site.occurs is that it is an effective way to code the cone Considerable effort has been devoted to establishingsignals for transmission down the optic nerve (see whether the linear form for the mapping between qWandell 1995). and u is appropriate, and if so, what values should be 2259
  • Color Vision Theoryused for the mij. Several types of experimental evidencehave been brought to bear on the question. As an example, one line of research argues that fourcolor perceptions, those of redness, greenness, blue-ness, and yellowness, have a special psychologicalstatus, in that any color experience may be intuitivelydescribed in terms of these four basic perceptions.Thus orange may be naturally described as reddish-yellow and aqua as greenish-blue. In addition, bothintrospection and color scaling experiments suggestthat the percepts of redness and greenness are mutuallyexclusive so that both are not experienced simul- Figure 1taneously in response to the same stimulus, and A color context effect. The figure illustrates the colorsimilarly for blueness and yellowness (e.g., Hurvich context effect known as simultaneous contrast. The twoand Jameson 1957, Abramov and Gordon 1994). central disks are physically the same but appearGiven these observations, it is natural to associate the different. The difference in appearance is caused by theRG signal with the amount of redness or greenness fact that each disk is seen in the context of a differentperceived in a light (redness if the signal is positive, annular surround. This figure is best viewed in color.greenness if it is negative, and neither red nor green if A color version is available in the on-line version of theit is zero) and the BY signal with the amount of Encyclopediablueness or yellowness. Judgments of the four fun-damental color perceptions, obtained either through at other locations and at preceding times. To help fixdirect scaling (e.g., Abramov and Gordon 1994) or ideas, it is useful to restrict attention to the disk-through a hue cancellation procedure (e.g., Hurvich annulus configuration. For this configuration, theand Jameson 1957), are then used to deduce the visual representation of the disk may be written asappropriate values of the mij in the second and thirdrows of Mo. When this framework is used, the entries ud l f (qd; qa, ) (13)for the first row of Mo, corresponding to the LUMsignal, are typically established through other means where ud is the visual response to the disk, qd and qa aresuch as flicker photometry (e.g., Kaiser and Boynton the cone coordinates of the disk and annulus re-1996). spectively, and represents other contextual variables Other approaches to studying the opponent trans- such as the size of the disk and annulus and anyformation include analyzing measurements of the temporal variation in the stimulus. Clearly, f( ) mustdetection and discrimination of stimuli (e.g., Wyszecki incorporate the sort of transformation described byand Stiles 1982, Kaiser and Boynton 1996, Eskew, the matrix Mo in Sect. 3.1 above.et al. 1999, Wandell 1999), and measurements of how As was the case with the discussion of opponencythe color appearance of lights is affected by the context above, there is not wide agreement about how best toin which they are viewed (e.g., Webster 1996). In part model adaptation. A reasonable point of departure isbecause of a lack of quantitative agreement in the a cone-specific affine model. In this model, the visualconclusions drawn from different paradigms, there is representation u of a light is related to its conecurrently not much consensus about the details of the coordinates q through an equation of the formtransformation between q and u. One of the majoropen issues in color theory remains how to extend the u l Mo(D qkq ) (14) " "simple linear model described above so that it accountsfor a wider range of results. where Mo is as in Eqn. (12) and A C A C gL 0 0 qL " " D l 0 gM 0 , q l qM3.2 Adaptation " " " " (15) 0 0 gS qFigure 1 illustrates a case where the same light has a B " D B S" Dvery different color appearance when seen in twodifferent contexts. The figure shows two disk-annulusstimulus configurations. The central disk is the same in In this formulation, the g’s on the diagonals of Deach configuration, but the appearance of the two characterize multiplicative adaptation that occurs at a"disks is quite different. To explain this and other cone-specific site in visual processing, before signalscontext effects, mechanistic models assume that at any from separate cone classes are combined. The entriesgiven time and image location, the relation between of the vector q characterize subtractive adaptation.the quantal absorption rates q and the visual rep- " Equation (14) is written in a form that implies that theresentation u depends on the quantal absorption rates subtractive adaptation also occurs at a cone-specific2260
  • Color Vision Theorysite. The entries of D and q depend on the cone 4. Color Constancy " "coordinates qa of the annulus as well as on spatial andtemporal variables characterized by . Note that the The discussion so far has focussed on how the visualcone-specific affine model is a generalization of the system represents and processes the spectrum of lightidea that the visual representation consists of a that enters the eye. This is natural, since light is thecontrast code. proximal stimulus that initiates color vision. On the Asymmetric matching may be used to test the other hand, we use color primarily to name objects.adaptation model of Eqn. (14). In an asymmetric The spectrum of the light reflected to the eye from anmatching experiment, an observer adjusts a match object depends both on an intrinsic property of thestimulus seen in one context so that it appears to have object, its surface reflectance function, and on extrinsicthe same color as a test stimulus seen in another factors, including the spectral power distribution ofcontext. More concretely, consider Fig. 1. In the the illuminant and how the object is oriented relativecontext of this figure, an asymmetric matching ex- to the observer.periment could be conducted where the observer was Given that the light reflected to the eye varies withasked to adjust the central disk on the right so that it the illuminant and viewing geometry, how is it thatmatched the appearance of the central test disk on the color is a useful psychological property of objects? Theleft. Suppose such data are collected for a series of answer is that the visual system processes the retinalN test disks with cone coordinates qti. Denote the image to stabilize the color appearance of objectscone coordinates of the matches by qmi. Within the across changes extrinsic to the object (e.g., changes inmechanistic framework, the corresponding visual rep- the spectrum of the illuminant). This stabilizationresentations uti and umi should be equal. If Eqn. (14) process is called color constancy.provides a good description of performance then Color constancy is closely linked to the phenom- enon of adaptation described above (Maloney 1999). Indeed, quantitative models of color constancy gen- erally incorporate the same idea that underlies mech-Mo(Dm qmikqm ) l Mo(Dt qtikqt ) " " " " anistic models of visual processing: at some central site  qmi l D−" (Dt qtikqt jqm ) m" (16) there is a visual representation u that correlates with " " "  qmi l Dtmqtikqtm color appearance. To stabilize this representation against changes in illumination, it is supposed that the relation between the quantal absorption rates q elicitedwhere Dtm l D−" Dt and qtm l D−" (qt kqm ). This by the light reflected from an object and the visualprediction may be checked by finding "the diagonal m" " m" " representation u depends on the scene in which thematrix Dtm and vector qtm that provide the best fit to object is viewed. In the case of color constancy, thethe data and evaluating the quality of the fit. Tests of emphasis has been on how the visual system processesthis sort indicate that the cone specific affine model the retinal image so that the transformation between qaccounts for much of the variance in asymmetric and u has the effect of compensating for the variationmatching data, both for the disk annulus configuration in the light reflected to the eye caused by changes of(Wandell 1995, 1999) and for more complex stimuli illumination and viewing geometry. Psychophysical(Brainard and Wandell 1992). Nonetheless, there are data on the color appearance of objects viewed underclear instances for which Eqn. (16) does not give different illuminants are often well-modeled by trans-a complete account of asymmetric matching (e.g., formations consistent with Eqn. (14) (e.g., BrainardDelahunt and Brainard 2000) and other color ap- and Wandell 1992).pearance data (e.g., Webster 1996, Mausfeld 1998, The central theoretical question of color constancyD’Zmura and Singer 1999). is how the visual system can start with image data and The cone-specific affine model may also be tested factor it into an accurate representation of the surfacesagainst psychophysical data on the detection and and illuminants in the scene. This question has receiveddiscrimination of colored lights. Here again the model extensive treatment, at least for simple scenes. A briefprovides a reasonable point of departure but fails in introduction to this literature on computational colordetail (e.g., Eskew et al. 1999). constancy follows. To extend the cone specific affine model, varioustheorists have suggested the need for adaptation at a 4.1 Computational Color Constancysecond site (after signals from separate cone classeshave been combined) and for the inclusion of non- Consider a scene consisting of diffusely illuminatedlinearities in the relation between q and u (see flat, matte surfaces. For such scenes, the spectrum breferences cited in the previous two paragraphs). An reflected to the eye from each surface is given by theadditional open question concerns how the entries of wavelength-by-wavelength product of the spectralD and q are determined by the viewing context power distribution of the illuminant e and the surface " "(e.g., Brainard and Wandell 1992, Delahunt and reflectance function s. The surface reflectance functionBrainard 2000). specifies, at each sample wavelength, the fraction of 2261
  • Color Vision Theoryincident light reflected to the eye. The information linear model also describes illuminant spectral powerabout b coded by the visual system is its cone distributions, so thatcoordinates, which may be computed as e $ B ew e (22) q l S b l S diag(e) s (17) The second is that the spatial average of the surfacewhere the function diag( ) returns a square diagonal reflectance functions (s- ) is the same in all scenes andmatrix with its argument placed along the diagonal. known. These additional constraints imply thatClearly e and s are not uniquely determined fromknowledge of q: without additional constraints the ` V V q l [S diag(s` )Be]we l Ls` we (23)color constancy problem is underdetermined. For-tunately the spectra of naturally occurring illuminants - where q is the spatial average of the quantal absorp-and surfaces are not arbitrary. Although the physical tion rates and Ls- is a known three-by-three matrix.processes that constrain these spectra are not well Inverting Eqn. 23 yields an estimate for the illuminantunderstood, analyses of measurements of both illumi- # e l BeweV . This estimate is then used to provide thenants and surfaces shows that their spectra are well matrix L−" to be used in Eqn. (21). eVdescribed by small-dimensional linear models (see Buchsbaum’s algorithm shows how the addition ofBrainard 1995, Maloney 1999). appropriate assumptions allows solution of the Consider surface reflectances. It is possible to define computational color constancy problem. The difficultythree fixed basis functions so that naturally occurring with Buchsbaum’s algorithm is that its assumptionssurface reflectances are reasonably well approximated are too restrictive. In particular, it seems unlikely thatby a linear combination of these basis functions. Thus the spatial average of surface reflectances is constantfor any surface, we have across scenes, nor do real scenes consist of diffusely illuminated flat, matte surfaces. Subsequent work has s$ws bs jws bs jws bs (18) focused on ways to provide reasonable estimates of " " # # $ $ the illuminant and surface reflectances under otherwhere bs , bs , and bs are the spectra of the basis sets of assumptions (e.g., Maloney 1999).functions" and ws , ws$ , and ws are scalar weights # " # $that provide the best approximation of s within thelinear model. Eqn. (18) may be rewritten as 4.2 Computational Color Constancy and Human Performance s$Bsws (19) How does the computational work relate to humanwhere the three columns of matrix Bs contain the basis performance? This question has not yet been resolved,functions and the vector ws contains the scalar but it seems appropriate to close with a few obser-weights. vations. First, the estimated linear model weights of When the surface reflectance functions lie within Eqn. (21) may be associated with the mechanisma three-dimensional linear model Eqn. (17) may responses u discussed in Sect. 2. In both types of #inverted, once an estimate e of the illuminant has theory, these quantities represent the visual responsebeen obtained (see below for discussion of illuminant computed from the quantal absorption rates, and bothestimation.) Start by rewriting Eqn. (17) as: are meant to allow direct prediction of appearance. In the mechanistic approach, one considers a series of # q l [S diag(e)Bs] ws l Le# ws (20) transformations whose form is derived from experi- ments with simple stimulus configurations. In thewhere Le# is a three-by-three matrix that depends on the computational approach, the form of the transform-illuminant estimate. This matrix may be inverted using ation is derived from consideration of the problemstandard methods to yield an estimate of ws: color vision is trying to solve. In both cases, however, the emphasis is on finding the appropriate parametric ws l L−" q V eV (21) form of the transformation and on understanding how the parameters are set as a function of the image data.The estimate may then be used together with Eqn. (19) The connection between the two approaches is dis-to estimate the surface reflectance function. cussed in more detail by Maloney (1999). Many computational color constancy algorithms The value of the computational approach to under-assume a linear model constraint for surface reflec- standing human vision depends on how accurately thetance functions. This reduces the constancy problem transformations it posits may be used to predict theto finding an estimate of the illuminant to plug into appearance of stimuli measured in psychophysicalEqn. (20). For illustrative purposes, an algorithm experiments. There have been only a few empiricaldue to Buchsbaum (1980) is described here. In comparisons of this sort to date. These comparisonsBuchsbaum’s algorithm, two additional assumptions do, however, indicate that the computational ap-are added. The first is that a three-dimensional proach shows promise for advancing our understand-2262
  • Combinatorial Data Analysising of human color vision (Bloj, Kersten, and Hurlbert Gegenfurtner K, Sharpe L T (eds.) Color Vision: From Genes1999, Brainard, Kraft, and Longre 2001). to Perception. Cambridge University Press, Cambridge, UK, pp. 3–51 Stockman A, Sharpe L T 1999 Cone spectral sensitivities andSee also: Color Vision; Psychophysical Theory and color matching. In: Gegenfurtner K, Sharpe L T (eds.) ColorLaws, History of; Psychophysics; Vision, Low-level Vision: From Genes to Perception. Cambridge UniversityTheory of; Vision, Psychology of; Visual Perception, Press, Cambridge, UK, pp. 53–87Neural Basis of; Visual System in the Brain Wandell B A 1995 Foundations of Vision. Sinauer, Sunderland, MA Wandell B A 1999 Computational neuroimaging: color repre- sentations and processing. In: Gazzaniga M (ed.) The NewBibliography Cogniti e Neurosciences, 2nd edn. MIT Press, Cambridge, MA, pp. 291–303Abramov I, Gordon J 1994 Color appearance: on seeing red—or Webster M A 1996 Human colour perception and its adaptation. yellow, or green, or blue. Annual Re iew of Psychology 45: Network: Computation in Neural Systems 7: 587–634 451–85 Wyszecki G, Stiles W S 1982 Color Science—Concepts andBloj M G, Kersten D, Hurlbert A C 1999 Perception of three- Methods. Quantitati e Data and Formulae, 2nd edn. John dimensional shape influences colour perception through Wiley, New York mutual illumination. Nature 402: 877–9Brainard D H 1995 Colorimetry. In: Bass M (ed.) Handbook of Optics: Volume 1. Fundamentals, Techniques, and Design. D. H. Brainard McGraw-Hill, New York, pp. 26.1–26.54Brainard D H, Brunt W A, Speigle J M 1997 Color constancy in the nearly natural image. 1. Asymmetric matches. Journal of the Optical Society of America A 14: 2091–110 Brainard D H, Kraft J M, Longere P 2001 Color constancy: developing empirical tests of computational models. In: Combinatorial Data Analysis Mausfeld R, Heyer D (eds.) Colour Perception: From Light to Object. Oxford University Press, Oxford, UK Combinatorial data analysis (CDA) refers to a class ofBrainard D H, Wandell B A 1992 Asymmetric color-matching: methods for the study of relevant data sets in which How color appearance depends on the illuminant. Journal of the Optical Society of America A 9(9): 1433–48 the arrangement of a collection of objects is theBuchsbaum G 1980 A spatial processor model for object colour absolutely central concept. Characteristically, CDA is perception. Journal of the Franklin Institute 310: 1–26 involved with either: (a) the identification of arrange-D’Zmura M, Singer B 1999 Contrast gain control. In: ments that are optimal for a specific representation Gegenfurtner K, Sharpe L T (eds.) Color Vision: From of a given data set, and where such an exploratory Genes to Perception. Cambridge University Press, Cambridge, process is typically carried out according to some UK, pp. 369–85 specific loss or merit function that guides a combina-Dacey D M 2000 Parallel pathways for spectral coding in torial search over a domain of possible structures primate retina. Annual Re iew of Neuroscience 23: 743–75 constructed from the constraints imposed by theDelahunt P B, Brainard D H 2000 Control of chromatic adaptation: Signals from separate cone classes interact. Vision particular representation selected; or (b) a confirma- Research 40: 2885–903 tory determination as to whether a specific objectEskew R T, McLellan J S, Giulianini F 1999 Chromatic de- arrangement given a priori reflects the observed data, tection and discrimination. In: Gegenfurtner K, Sharpe L T and where such a confirmatory process is typically (eds.) Color Vision: From Genes to Perception. Cambridge operationalized by comparing the empirically observed University Press, Cambridge, UK, pp. 345–68 degree of correspondence between some given data setHurvich L M, Jameson D 1957 An opponent-process theory of and the specific structure conjectured for it, to a color vision. Psychological Re iew 64(6): 384–404 reference distribution constructed from the collectionKaiser P K, Boynton R M 1996 Human Color Vision, 2nd edn. of all possible structures of the same form that could Optical Society of America, Washington, DCMaloney L T 1999 Physics-based approaches to modeling have been conjectured. surface color perception. In: Gegenfurtner K, Sharpe L T The boundaries of what CDA might encompass are (eds.) Color Vision: From Genes to Perception. Cambridge somewhat open but generally we would exclude University Press, Cambridge, UK, pp. 387–416 methods based on the postulation of strong stochasticMausfeld R 1998 Color perception: From Grassman codes to a models and their specific unknown parametric struc- dual code for object and illumination colors. In: Backhaus tures as underlying a given data set. Although CDA W G K, Kliegl R, Werner J S (eds.) Color Vision—Pers- might use or empirically construct various weighting pecti es from Different Disciplines. Walter de Gruyter, Berlin, functions, the weights so obtained are not to be pp. 219–50 interpreted as parameter estimates in some presumedNeitz M, Neitz J 2000 Molecular genetics of color vision and color vision defects. Archi es of Ophthalmology 118: 691–700 stochastic model viewed in turn as responsible forRodieck R W 1998 The First Steps in Seeing. Sinauer, Sunder- generating the data. Manifest data are emphasized land, MA solely, and the traditional concern for an assumedSharpe L T, Stockman A, Jagle H, Nathans J 1999 Opsin genes, relationship between the data and a restrictively cone photopigments, color vision, and color blindness. In: parameterized stochastic model is avoided. For 2263Copyright # 2001 Elsevier Science Ltd. All rights reserved.International Encyclopedia of the Social & Behavioral Sciences ISBN: 0-08-043076-7
  • Implications VOL. 03 ISSUE 5 www.informedesign.umn.edu A Newsletter by InformeDesign. A Web site for design and human behavior research. Seeing Color and was developed by a Finnish Color is the most dominant design ele- astronomer, Aron Sigfrid Forsius and ment, and ironically, the most relative was soon followed by Newton’s color aspect of design. The perception of color wheel in 1704. The primary objectives of involves human physiological and psy- these systems are to give order to the chological responses. Object, light, eye, variables of color and to concretely rep- and brain are involved in a complex resent colors, because “words are incom- process of sensation and perception. plete expression as color” (Munsell, Color attracts our attention, helps us 1981). Munsell developed a three- make sense of our environment, and dimensional color tree. The three vari- affects our behavior. Color plays a cul- ables of color — hue, value, and chroma tural role, an informational role, and are displayed on plexiglass branches, even a survival role. It functions on a one for each hue (see Figure 1). Darker basic level of appeal and can elicit strong values of the hue are toward the bottom; feelings of like or dislike. Color is a lighter values are toward the top. source of sensual pleasure (Pentak & Brighter hues are seen at the outsideFigure 1: Munsell Color Wheel perimeter; duller hues are toward the Roth, 2003). center of the tree. A color wheel made of IN THIS ISSUE hats and shoes, featured in an exhibition Seeing Color Color Order Systems in the Goldstein Museum of Design, We are familiar with the most common arranged the objects in spectral order Typography and Color type of color arrangement—a color wheel (see Figure 2). Related Research arranged in spectral order. Spectral Summaries order is especially pleasing to the human Michel Eugene Chevreul developed a perceptual system. The spectrum occurs system to explain how colors affect each in nature in the refraction of light into other. As director of the Gobelins tapes- bands of color—red, orange, yellow, try studio (France), he realized that color green, blue, and violet. One hue gradates systems did not account for perceived into the next, creating a dynamic color color and that colors tend to tinge adja- sensation. cent hues with its complementary hue. In response, he designed a color circle Theoretical Color Systems that accounted for differences of satura- Scientists, artists, and color theorists tion and value within each hue family. have developed variations of the color He also created a framework about the wheel. The first wheel appeared in 1611 effects of simultaneous contrast.
  • Implications www.informedesign.umn.edu 2 The Effect of Surface Quality on Color Perception Surface quality contributes to the variability of color, “one and the same color evokes innumerable read- ings” (Albers, 1963, p. 1). This variability is due to differences in the human visual system, light, and the material and surface quality of the object. When we view the color of an object, we are really seeing reflected light. Objects are typically colored with either pigment or dyes. Dyes permeate the molecular structure of the object; pigments lay in a coat of color on the surface of an object. This difference is evident in viewing fabric that has been painted versus fabric that has been dyed.Figure 2: A color wheel made from hats and shoes that are in thecollection of the Goldstein Museum of Design. Surface materiality also affects the appearance of a color. A smooth, glossy surface will reflect a hue veryPractical Color Systems differently than a rough surface, and they tend toSpecialized color systems are used in product design reflect more light than a matte or rough surface.and manufacturing. Both the Pantone color system Matte or rough surfaces reflect light in a scattered,and the Munsell system are widely used. Pantone diffuse manner that randomly mixes the wavelengthshas developed color systems and products for the and tends to soften the color, changing it.graphic, interior, textile, architectural, and industri- Transparent materials allow color and light to beal design fields. Pantone has also recently begun seen through them (see Figure 3). Reflection fromforecasting color trends in fashion and interior glossy paper can make reading a menu or a magazinedesign. The primary goal of both the Munsell and difficult just as reflection from a highly polished floorPantone systems is to communicate color in a sys- can create spatial perceptual challenges.tematic way, leaving little room for error. The CIE(Commission Internationale de l’Eclairage) chro-maticity diagram displays a color matching systembased on light, and it is shaped like a luminositycurve. The system attempts to eliminate differencesof color perception through mechanical measure-ment of the three variables of a color—luminance,hue, and saturation. While these practical color sys-tems help to ensure accurate color specification,color appearance still varies due to lighting, context,and surface quality. Figure 3: Glass designed by Dale Chihuly, Museum of Glass, Tacoma, WA. Where Research Informs Design®
  • Implications www.informedesign.umn.edu 3Albers (1963) discusses the interdependence of color Color Harmonywith form and placement, quantity, and quality. It is There are strategies for creating color harmony:a constant challenge to predict how a color will look using similar values or hues, using hues with com-on the designed object when seen under different plementary contrast, or using a large number of hueslight sources. While the typical color wheel repre- in careful proportions. Constrast provides a sense ofsents only two or three dimensions, a color system visual balance. Munsell recommended balancingdeveloped by Albert-Vanel attempted to include vari- light and dark hues, dull and bright hues, and coolations due to surface quality, light, and human per- and warm hues. A sense of color harmony is basedception. This system, called the Planetary color sys- partially in human perception and partially in colortem and developed in 1983, includes not only hue, trends (see Figure 4).value, and chroma, but also accounts for contrastand material. Human Perception of Color Color can have a profound effect on humans. It can affect our brain waves, heart rate, blood pressure, and respiratory rate. Color also affects us emotional- ly. Exposure to color has an effect on our biological systems. Not only does color affect our sense of well- being, but it also may play a role in medical treat- ments for depression, cancer, and bacterial infec- tions. Visual Perception Our perception of color is dependent on light, object,Figure 4: The colorful facades of Burano, Italy. and our eyes and brain. We know that colors are influenced by adjacent colors, indeed, it is rare to seeDyes and Colorants an isolated color or color in its pure state. ChevreulThe color of objects is dependent on the pigments or discussed how colors tend to tinge neighboring huesdyes used in the production of the product. Color with their complement. Including color oppositestrends often evolve out of technological develop- within close proximity in a particular space can mit-ments. In the mid-1850s, William Henry Perkins igate this phenomenon. Surgical personnel in hospi-accidentally developed effective synthetic dyes for tals wear greenish-blue scrubs to counter-balancewool and silk as he attempted to synthesize quinine the visual effect of afterimages. During surgery, allfrom aniline. He named the color mauve. Other eyes focus on the patient and typically see a varietychemists developed synthetic aniline dyes that were of pink and red hues. The red receptors in the eyesignificantly brighter and more saturated than early would become fatigued if not for the color of thenatural dyes. This discovery, along with the develop- scrubs providing the opposite hue and thus balanc-ment of organic chemistry as a discipline, fueled the ing the visual experience.development of numerous synthetic dyes. Neon dyesand pigments that were developed in the mid-1980s Color contrast is essential for our understanding ofresulted in bright fabrics, accessories, and paper form and legibility. At least a 70% contrast betweenproducts. the background and letterforms is ideal for signs and Where Research Informs Design®
  • Implications www.informedesign.umn.edu 4painted materials. Conversely, too much contrast in nomena also affect the popularity of colors. Fashionan environment may increase anxiety and tension. prints in the 1960s used the bright palette of colorsSharp contrasts of color on flooring may create known as psychedelic. These colors were fully satu-uncomfortable illusions for walkers as they deter- rated and were intended to mimic the sensationmine whether the floor is flat or not. Research has caused by drugs (see Figure 5). Most of the informa-shown that the most visible combinations of colors tion about color meaning is highly subjective andare yellow and black, white and black, white and based on tacit beliefs, rather than research. There isblue, and red and white. a significant need for systematic research on color and human perception.Psychological Responses to ColorWe all react differently to color. We have different Typography and Colorcolor preferences, and we all have our least favorite Typography, the set of alphabetic characters, numer-colors. Color response is highly personal. What one als, and symbols used to compose copy, can beperson believes is a restful color, another may find manipulated in any number of ways by a graphicstimulating. Frequently these color preferences are designer. Size, typeface, letterspacing, leading (thebased on our own personal experience—a fondly space between lines of type), case (upper or lower case), structure (normal, light, bold, italic, bold ital- ic, etc.), and—of course—color can all be used to improve the legibility (how easy the text is to read), readability (how inviting the text is to a reader), and the hierarchy or structure of typeset copy. While each of the previously mentioned characteris-Figure 5: Fabric samples from 1960s-era clothing. tics can be manipulated by designers setting type, color is an especially important property. We oftenremembered yellow kitchen that belonged to grand- imagine type (or copy) that is set in black on a whitemother. There are also cultural associations that background—this is perhaps the most familiar wayinfluence our reactions to color. In several cultures, to set type on a printed page. However, when weblue is seen as peaceful, protecting, and soothing think of typography in signage and the built environ-color. Red typically signifies passion and revolution. ment, a variety of colors and color combinations,There are multiple associations for each color. For come to mind. Consider the new, colorful green andexample, black may be seen as sophisticated or as yellow logo signage of BP (British Petroleum) that isdepressing. Orange can be warm or aggressive. employed in the design of gas stations. Or, think ofYellow can be upbeat or acidic. the familiar white type on a green background of road signs. Color is employed frequently in environ-Marketing research attempts to discover what colors mental signage to create a memorable identity thatinfluence human behavior and how people will act helps users navigate a space, remember the businesswhen they shop, eat, or travel. Findings by market- or company, and create a pleasant impression.ing researchers are typically short-lived, however;trends seem to come and go, and other variables in When creating environmental signage, it is critical toaddition to color affect behavior. While technology consider some of the variables associated with thecontributes to color trends, culture and social phe- application of color. Here are a few ideas and tips: Where Research Informs Design®
  • Implications www.informedesign.umn.edu 5 This is not an exhaustive list of issues to consider when applying color to environmental signage and typography. If possible, it is beneficial to have a graphic designer who understands the interactions between typography, color, and the built environ- ment on a design team when designing environments with signage. In addition, InformeDesign has Research Summaries about graphic design for the built environment. References —Albers, J. (1963). Interaction of Color. New Haven, CT: Yale University Press.Figure 6: An example of poor and excellent contrast between —Fehrman, K., & Fehrman, C. (2004). Color: Thetypography and background. Secret Influence. Upper Saddle River, NJ: Prentice• Consider the contrast between the color of the Hall. typography and the background to ensure that the —Munsell, A. H. (1946). A Color Notation. Baltimore: type is easy to decipher and read. Type/back- Macbeth. ground color combinations can cause the text to —Pentak, S., & Roth, R. (2003). Color Basics. either advance or recede (see Figure 6). Stamford, CT: Wadsworth.• Consider the impact of color on interpretation and —Sharpe, D. (1981). The Psychology of Color and understanding of the content. What does a red Design. Totowa, NJ: Littlefields, Adams & Co. heading indicate versus a brown heading? Does —Stromer, K. (Ed.). (1999). Color Systems in Art and setting less important information in a brighter, Science. Edition Farbe/Regenbogen Verlag. more prominent color impact the order that infor- —Walch, M., & Hope, A. (1990). The Color mation is retrieved? Compendium. New York: Van Nostrand Reinhold.• Consider the user. Be aware of the cultural context of the environment and the signage, and consider About the Authors: cultural norms for particular colors. For example, Barbara Martinson, Ph.D., in Europe and the US, red typography generally is the Buckman Professor of means warning or attention. The application of Design Education in the color to type can either play into cultural norms for Department of Design, color or can contradict them. Housing, and Apparel,• Consider the lighting levels of the environment. University of Minnesota. While a color combination may work well when She has taught founda- evaluated in your office, the combination may be tions-level color courses for inappropriate when the lighting levels are different. 20 years, as well as graphic• Consider the properties of the signage material. design, design history, and How will a surface that is reflective or flat change human factors courses. She the legibility of the content? How will lighting levels recently curated Seeing interact with the surface properties? Color, an exhibition at the Goldstein Museum of Where Research Informs Design®
  • Implications www.informedesign.umn.edu 6Design. Her research focuses on design education, Bright, Saturated Colors Attract Attentionlearning styles, and the use of digital media in teach- —Color Research and Applicationing. Her favorite color is blue. Determining Color in the Built EnvironmentKate Bukoski, author of —Color Research and Application“Typography and Color,” is aPh.D. candidate in graphic Effects of Office Color Scheme on Workersdesign and holds teaching —Color Research and Applicationand reasearch assisantshipsin the Department of Design, Color Aids Wayfinding for Young ChildrenHousing, and Apparel, —Early Childhood Education JournalUniversity of Minnesota. Her research interests focuson the history and state of the profession of graphic Space and Color Affects Cooperation Among Childrendesign. —Environment and BehaviorAdditional Resources Color Judgment is Influenced by the Aging Eyewww.digitalanarchy.com/theory/theory_main.html —Family and Consumer Sciences Research Journalwww.colorsystem.comwww.colormatters.com/colortheory.html Light Source, Color, and Visual Contrastpoynterextra.org/cp/ —Family and Consumer Sciences Research Journalwww.colorcube.com/articles/theory/theory.htmwww.tigercolor.com/ColorLab/Default.htm Color of Light Affects Psychological Processeswww.fadu.uba.ar/sicyt/color/bib.htm —Journal of Environmental Psychologyhttp://webexhibits.org/colorart/ch.htmlwww.digitalanarchy.com/theory/theory_main.html Color, Meaning, Culture, and Design —Journal of Interior DesignRelated Research SummariesInformeDesign has many Research Summaries about Photos Courtesy of:color and related, pertinent topics. This knowledge Barbara Martinson, University of Minnesota (p. 1, 2,will be valuable to you as you consider your next 4, & 5)design solution and is worth sharing with your Caren Martin, University of Minnesota (p. 3)clients and collaborators. The Mission The Mission of InformeDesign is to facilitate designers’ use of current, research-based information as a decision-making tool in the design process, thereby integrating research and practice. Created by: Sponsored by:© 2002, 2005 by the Regents of the University of Minnesota.
  • Chapter 13How does visual memory work?Photo courtesy of Ann Cantelow. The multichannel neuron model ascribes numbers to channels. The channelnumbers store and communicate analog data. They can also be used, in a distinct addressing system, tosequentially query the twigs of visual memory.Addressing and retrievalFor retrieval, the model requires two types of neurons: 1) an address generating neuron, which drives 2) a datastorage neuron. To activate a memory stored as "a thing in a place," a stored datapoint must be addressed atprecisely that place. In the specific case of a stored pattern of three bleached disks imported from aphotoreceptor, a trio of associated datapoints, twigs, must be addressed, one right after the other.We have a mechanism for generating sequential addresses. The principle is inherent in the multichannel neuronmodel. The address generator can be the commutator we have postulated at the axon hillock.
  • To stimulate the first 9 twigs of memory, #1 through #9, each in turn, requires this sort of circuit. The outputlines of the axon driven by the addressing commutator are telodendrions, each corresponding to a channel. Inthis illustration of this model, telodendrions are numbered in order of their firing. Each individual channelsynapses to a dendrite. Each dendrite will be stimulated in its turn, in accordance with the ascending circularorder of the addressing commutator.Each dendrite is a “twig memory”. It stores a channel number that stipulates which channel shall be fired inresponse to the addressing signal. The effect can be tabulated:
  • The dendrites, which comprise the twigs of memory in this simple model, are each stimulated in turn. Thepattern of bleached disks that each twig has memorized is fired back into the nervous system – preciselyreplicating the pattern originally dispatched from a single photoreceptor’s outer segment at some time and dayin the past. In the table, 9 upticks of the address counter’s commutator correspond to a trio of 3D pixels and 3frames of a film strip. [A slicker model might use just one address tick to elicit all three datapoints,characterizing intensity, wavelength, phase -- but the point is, visual memory is sampled and read out by theticking of a sequential address counter. It is probably written in the same way.]All pixels recorded from the retina at the same time, stored in twigs on other photoreceptor antipodal "trees"will have identically the same time stamp in their address. So simultaneously, synchronously, one pixel fromevery other “tree” or photoreceptor antipode in the retina of memory is being triggered.The effect is to pump out of memory a stream of past images -- each image made up of millions of 3D pixels.The system is massively parallel and, therefore, moves whole images all at once. It is lightning fast.Why dont we see these torrents of images from the past? Why arent we drowning in images? Because theseare not literal images. They are images of the Fourier plane. Fourier images are invisible to us, except perhapsin the special case of LSD users. Literal images may impinge on the consciousness as, in effect, searchproducts, but the search itself is conducted as a Fourier process and is unconscious -- offstage and out of sight.Numbered synapses -- new evidence, old ideaThe idea there might be some sort of detectable ordering or sequencing of synapses on the dendrites isattributed to Wilfrid Rall, who suggested it in 1964 in support of a wholly different and unrelated model of thenervous system. In the 24 September 2010 issue of Science there is a featured report that reinforces the notionthere exists some sort of sequentially ordered input pattern in the dendrites.
  • In these experiments, a programmed series of successive stimuli is made to “walk” from synapse to synapsealong the dendrite. If the stimulus series progresses toward the cell body it is more likely to trigger off actionpotentials than a programmed series of stimuli that walks the other way, away from the soma, toward the tips ofthe dendrites.The front half of this experiment consists of the selective stimulation of a row of individual dendritic spines,one after another, using a laser to precisely localize release of glutamate. The basic technology was outlinedhere. The back half of the experiment is conventional, and consists of electronic monitoring and tabulation ofthe axon’s response.In terms of the multichannel model electrophysiology is difficult to interpret. However, a significant feature ofthe model is a staircase of firing thresholds. One might speculate that as the stimulus is made to approach thesoma, it is finding or ultimately directing a pointer to lower and lower firing thresholds, which is to say, lowerchannel numbers. These low numbered channels would be more easily triggered than higher numberedchannels.Unfortunately there is easy no way to directly measure or guess the channel number associated with an actionpotential in passage, if indeed multiple channels exist. Again in terms of the model, a plot of channel numbersversus synapse position on dendrites (or, using different techniques, on the teledendrions) would produce afascinating picture. In any event it is interesting that even conventional electrophysiology suggests there may besome kind sequential ordering, progression, or directional structuring that underlies a map of dendritic spines.The modelIn modeling this visual memory system I think it would be best to use automated rotating or looping machinery,just as you would in many familiar recording and playback devices. The rotating machine is the commutator. Ateach addressing tree, let the loftiest addressing commutators walk forward through time automatically,incrementing higher channel by channel. Rough synchronization among trees should suffice. Now, instead ofhardwiring and broadcasting addresses in detail, the retrieval system can simply be given a start date/time andtriggered off. A string of retrieval instructions will ensue. The system will, in effect, read itself out like a diskdrive.As a practical matter, the model of a retina of memory should probably be constructed in software. Each tree ofmemory can be modeled as a disk drive storing analog numbers representing 3D pixels, stacked in serial order,that is, the order or sequence in which they were originally captured from the eye. Millions of disk drives, then,each of relatively modest capacity, comprise a retina of memory. In a primitive animal one would expect to finda single retina of memory. In a sophisticated animal, many.Let’s say the memory trees pre-exist in a newborn animal and that their twigs are unwritten. Each branch is apoint in a commutator sequence, and identifies time (that is, sequence) ranges.From the point of view of addressing the visual memory, reading and writing are, as in a disk drive, similarprocesses. The writing commutator walks forward through the present moments, guiding incoming 3D pixelsfrom the eye to a series of novel addresses. To elicit a visual memory a reading commutator, which could be theself-same machine, walks forward through addresses denoting a film strip of past moments.In effect, the pointer of the base commutator on the address generator, as it ticks ahead, is the pointer of thesecond hand of a system clock. Although the images are recorded at a stately and regular rate, such as one persecond -- the recall can be made to happen as fast as the commutator is made to sweep. And it could scanbackwards as well as forwards.How is a pixel memory deployed?This is an unsolved problem in the model. We have to assume it happens but the answer isnt easy or obvious.
  • We have stipulated what a 3D pixel memory is: Three numbers -- integers -- that represent a pattern of lightrecorded from three disks in a single photoreceptor at a particular moment in time. The three numbers aresufficient to specify the instantaneous wavelength, intensity and phase of the incoming light, as read out of astanding wave in the outer segment of the photoreceptor.We are suggesting these three numbers are configured and stored in the brain as an addressable twig ofmemory -- three dendritic launch pads for three action potentials to be fired down three specific, numbered axonchannels. It is nicely set up, this memory, but how did it happen?The operation of an initial readout commutator in the addressing neuron seems clear. It simply counts up ordown. Other commutators fan out from the initial or system counter. At the upper tier of the addressing tree, thecommutators, once toggled, can tick forward “on automatic.”But what about the commutator in the memory neuron?In the most basic model of the multichannel neuron, developed in Chapter 2, the neuron is functioning as asensory transducer. The commutator pointer rotates up to a specific numbered channel in proportion to an inputvoltage or graded stimulus.But in the memory neuron, we want the pointer to go, first, straight to a remembered channel. Then, second, toanother remembered channel. Then, third, to another remembered channel. Hop hop hop. From the addressneuron the memory neuron receives three signals in a sequence, via telodendrions 1, 2, 3. The data neuron fireschannels corresponding to three remembered photoreceptor disk positions: 2, 7, 34.Instead of responding proportionately to an input voltage, as in a sensory neuron, the commutator in thememory neuron is responding discontinuously to a memorized set of three channel firing instructions. So theneedle of this commutator must swing, not in response to an analog voltage input, but in response to a pixelmemory.In the multichannel model synapses connect individual channels, rather than individual neurons. It could be thatthe commutator is simply bypassed, so that the appropriate axon channels are hardwired to the dendritic twigsof memory. Synapses at the soma could suggest a short cut past or a way to overrule the inherent commutator.Maybe there is some rewiring or cross wiring at the level of the dendritic synapses. To borrow a term of artfrom the conventional playbook of memory biochemistry, maybe the synapses are subject to "tagging." Maybebiochemical markers delivered into the dendrites when the memory was originally recorded are specifying insome way the channel numbers to be fired.This model suggests a Y-convergence of three neurons, not just two. One delivers addresses. One stores the
  • data. A third neuron delivers original data from the retinal photoreceptor – data to be written in sequential orderinto the dendrites of the memory neuron.Whatever specific mechanism one might choose or invent, the model requires that pixel memory arriving from aphotoreceptor in the eye be stored in an antipodal neuron as a trio or linkage of three distinct channel numbers.ExperimentOne interesting aspect of this memory model is that it suggests an experiment. We are guessing that theindividual channels of an addressing axon are, in effect, split out and made accessible as numberedtelodendrions. If there is indeed a numerical succession – a sequential firing order – of the telodendrions, thenthis should be detectable. We were taught that the telodendrions must fire simultaneously. Is this always true? Ibet not.Superimposed networksNote that we have assumed there exists a double network. Above the information tree there is a second tree, areplica of the first, used to individually address each memory "twig".The principle of two superimposed networks, one for content and the other for control, is a technicalcommonplace. An early application was the superimposition of a telegraph network as a control system for therailway network. The egregious present day example is the digital computer, with its superimposed but distinctnetworks for information storage and addressing.We are long in the habit of dividing the nervous system into afferent and efferent, sensory and motor, butsurely there must be other ways to split it, e.g., into an information network and a addressing network. It istypically biological that one network should be a near replica of the other. Evolution proceeds throughreplication and modification.Arborization and addressing capacityThe first anatomist who isolated a big nerve, maybe the sciatic, probably thought it was an integral structure –in essence, one wire. Closer scrutiny revealed that the nerve was a bundle of individual neurons. We areproposing here yet another zoom-down in perspective, this time to the sub-microscopic level . We suspect thateach neuron within a nerve bundle is itself a bundle of individual channels.
  • It follows that the functional wiring of the nervous system is at the level of channels. Synapses connectchannels, not neurons. This is why one might count 10,000 synaptic boutons on a single neuron’s soma. Theboutons were not put there, absurdly, to “make better contact” nor to follow the textbook model of signalintegration. They are specific channel connectors, each with a specific channel number.The neuroanatomical feature that most interests us at this point is axon branching. This is because branching isof paramount importance in familiar digital technologies for addressing – search trees and other data structures.We have proposed a treelike addressing system for the visual memory in the brain. It is reasonable to ask --where are the nodes?Not at the branch points.Photo courtesy of Ann CantelowBranching in a nerve axon is just a teasing apart and re-routing of the underlying channels. It is not a branchingmarked by nodes or connections in the sense of an T or Y connected electrical branch, or a logical branch in abinary tree.For an axon that addresses a dendritic twig of memory, all functional branching occurs at the commutator.Any anatomical branching downstream of the commutator, such as the sprouting from the axon oftelodendrions , simply marks a diverging pathway – an unwinding or unraveling, rather than a distinct node orconnection. In other words, the tree is a circular data store. The datapoints are stored at twigs mounted on theperiphery of a circle. The twigs are accessible through a circular array of addresses. It is analogous to a diskdrive in which the disk holds still and the read-write head rotates.
  • Photo courtesy of Ann CantelowSummary of the technology to this pointThe tree in this photograph is a metaphor for the brain structure which corresponds to, and is antipodal to, asingle photoreceptor of the eye. It is one single photoreceptor cells remote memory warehouse -- a tree ofmemory.Each twig is a destination with an address, a neuronal process narrowed down to just two or three channels.For example channels 3, 7 and 29, only, might constitute a given twig. Each twig is a 3D pixel frozen in time.The tree will store as many unique picture elements from the photoreceptor’s past as it has twigs.As many as 125 million of these trees will constitute a retina of memory. We will look for ways to hack downthis number, but for the moment let it stand. The point is, we are talking about millions of trees.All these trees must be queried simultaneously with a particular numerical address, probably associated with atime of storage, to elicit firing from all the right twigs -- just one twig per tree. Properly addressed, a forest ofthese trees will recreate, almost instantly, a whole-retina image from memory.In a primitive animal, it would be sufficient to remember 300 images from the recent past. This could beaccomplished with a single addressing neuron, a single commutator. But in a modern mammal, it will benecessary to stack the commutators. A bottom commutator can point to any of 300 other commutators. Andeach of these can, in turn, point to 300 more commutators. With a simple tree of neurons, which is to say, alogical tree built with commutators, one can very quickly generate an astronomical number of unique addresses.We require one unique address for each twig of the data trees.Are there enough addresses available in this system to organize a mammalian lifetime of visual memories? Yes.Easily. Are there enough memory neurons to match the addressing capacity of the addressing neurons.Probably not. The neuronal brain that lights up our scanners is probably running its memory neurons as ascratchpad memory. It seems likely there is a deeper store.But will it work?The memory mechanism we have sketched is probably adequate as a place to start. It would work for adirectional eye in which changes in wavelength are highly significant cues to the position and movement of atarget. It is a visual memory for retaining the "just now," a film strip comprising a few recent frames.
  • For an imaging eye, or a human visual memory, this memory system is not yet practical because, in its presentform, it is a hog for time and resources.Bear in mind that this model is extremely fast in comparison with any conventional model of the brain based onsingle channel all-or-none neurons. Two reasons: 1) It is an analog memory, and 2) it is massively parallel.But a persistent difficulty with this model is serial recall. It appears this memory has to scroll back through allhistory to find relevant past images. And each image to be tested for a "hit" is composed from as many as 125million 3D pixels. This is a huge array to deploy and compare, even though the pixels pop up in parallel. This iswhy van Heerdens memory seems to be such a dream system -- comparison and recall are instantaneous.The van Heerden memory has a limitation, however, which a film strip memory does not. If a single face ispresented to the van Heerden memory system, it can respond with a class picture in which the face appeared.This supplies context -- a surround of useful information associated in the past with this particular face.However, the system does not automatically position the memory in time.In a film strip memory, in contrast, progression through time is built in. If the input is an image of a shark, thefilm strip memory will (like the van Heerden memory) turn up an image that puts the shark in a momentarycontext from the past. Maybe the shark of memory is freeze-framed in the middle of a school of fish.But in the film strip memory, the film strip can progress forward through past time. The remembered sharkturns, sees you, comes toward you, looms large, opens its mouth. In other words the film strip memoryprovides not only context and associations -- but also shows cause and effect. Here is a shark. Fast forward.Here are rows of teeth.It is reasonable to imagine that the memory we have today evolved from a simple (possibly Cambrian) filmstrip memory of the type we have described. For a simple animal in fixed surroundings, a film strip memory isfairly easy to model and easy to evolve -- to a certain point. But the film strip model soon becomes oppressivelyslow and heavy with data.How could this model be speeded up and expanded, that is, modernized? First, by taking full advantage of theFourier plane. Second, by introducing a metamemory, in effect, a hit parade. Third, by multitasking.The Fourier FlashlightLets take a moment to orient ourselves, using the central, red DC spot as a point of reference. The red spotmarks, in effect, the center of the Fourier plane. It also marks, probably, the position of the fovea.
  • We are looking at a structure inside a brain -- the retinas memory -- and the red spot marks that part of thememory antipodal to the foveal cones. The fovea is a wonderful thing but we can ignore it in this discussion. Itis the hole in the doughnut in terms of Fourier processing and filtering. In terms of natural history, it cant tellus much. The fovea is a rare and special feature, a splendid particularity of primates, birds, and a few othersmart and lucky vertebrates.But the visual memory evolved in vertebrates that had no fovea. It seems a reasonable guess that the visualmemory is grounded on Fourier processing. We have speculated in Chapters 5 and 6 that Fourier processingevolved in vertebrates as means of clarifying a blurry picture of the world obscured by glia, neurons, andvascular tissue because vertebrate photoreceptors are wired from the front.One could pump a time-series of addresses into the whole forest of memory, but it makes more sense toaddress the memory very selectively, so that only part of it responds. This approach is indicated in the photoabove with a yellow disk -- effectively, a Fourier flashlight.
  • Recall that any part of the Fourier plane can be transformed into the whole of a literal, spatial image from theworld. By selecting such a small part of the whole retinal output, illuminated by the flashlight, we have reducedthe data storage and processing problem to a tiny fraction of that associated with the original 125 million neuronsource. It is because of this holograph like effect -- the whole contained in each of its parts -- that Karl Lashleywas able to physically demolish so much of the visual cortex with lesions without producing a significant lossin the animals visual memory.Let me emphasize that the Fourier flashlight is a metaphor. It draws a convenient circle around a smallpopulation of neurons. In the model this is accomplished by addressing that small population, rather than thewhole retina of memory. As the population of neurons and, thus, the flashlight spot gets smaller, the resolutionof the literal image that can be recreated by Fourier transformation deteriorates. For rapid scanning and quickerretrieval, one would favor the smallest practical spot. Say a "hit" occurs, that is, a Fourier pattern scanned upfrom memory is found to match, more or less, a Fourier pattern at play on the retina.At this point, the spot we are calling the Fourier Flashlight could be expanded in diameter to improve theresolution of the remembered image, broaden the range of spatial frequencies to be included, and perhaps pickup some additional and finer detail.
  • Where on the retina of memory should we shine the flashlight? Spotlight addressing of the memory map givesus a means to accomplish Fourier filtering. For edge detection, we should select a circle of trees at theoutermost rim of the Fourier pattern (and retina), where the highest spatial frequencies are stored. For lowspatial frequencies, position a circle near the red DC spot.For most animals most of the time, an enhancement of high spatial frequencies has significant survival value.Here are two images from the Georgia Tech database, the first literal, the second with high spatial frequenciesenhanced.Remark the sharp definition of the edges of the mirror frame, the clowns arm, and of the edges of the makeupbrushes and pencils. In effect, an image in which edges are soft and ill defined has been turned into a cartoon,with heavy outlines emphasizing the edges of objects. This is the information an animal needs immediately. Ifthe animal were looking at a shark in the shadows, filtering for high spatial frequencies would make the sharksshape unmistakable.Edge enhancement in visual image processing and visual memory has an additional advantage, which is that itcreates a very spare, parsimonious image consisting of a few crucial outlines. This important data of highspatial frequency needs to be surfaced quickly for survival purposes.
  • This yellow band indicates an address map for the outer regions of the Fourier plane, where high spatialfrequency information crucial to edge detection is concentrated. These are neurons antipodal to rod cells at theouter periphery of the retina. It is interesting that this neglected outer frontier of the retina might have such acritical survival benefit for the animal. The animals concept of "an object" arises from edge detection. Theuncanny ability to distinguish the integrity of an object even though other objects may intervene is probablyrooted in high spatial frequency detection in this part of the retina.Chopping for speedA memory that is capable of storing a film strip of images is valuable but slow. We can speed it up bymassively cropping the images to be scanned for recall, using the Fourier flashlight technique described above.But one must still scan the images accumulated for "all time" to identify the objects currently in focus on theretina. If the object is, in fact, a shark, one doesn’t have time to scan through a lifetime of accumulatedmemories, in serial order, in order to recognize it.One solution is to chop the serial film strip into, for instance, one hundred short film strips. Or one thousand.Or ten thousand.Use these to play multiple Fourier flashlights upon the retina of memory. In this way one could make multiplesimultaneous scans and comparisons with the incoming retinal image from the eye.
  • Maybe the image to be matched is an old automobile. Figuratively, one Fourier flashlight can scan for the carsof the 60s, one for the cars of the 70s, one for cars of the 80s, and one for the cars of the 90s.This works because each flashlight is addressing a spot in the Fourier plane -- where any and every spot thatmight be addressed contains all the information needed to match or reconstruct a whole image.Memory anticipates realityBy multitasking the scans, we are breaking past the cumbersome requirement for serial, linear scanning andrecall. It is possible now to see an advantage in importing from the eye a huge retinal Fourier plane. Because ofits large area, the incoming Fourier pattern is open and accessible to thousands of simultaneous memory scans.Think of each Fourier "flashlight" as a projector running, from memory, a short, looped film strip. Looping iseasy because the addressing mechanism is a commutator.Each projected frame in this little movie is a Fourier pattern that corresponds to (and is transformable into)some remembered object.All these projectors run constantly. In this metaphor, the essential comparator, which is derived from the ideaoriginally conceived by Pieter van Heerden, is a screen. The Fourier flashlights play constantly at spots on oneside of the comparator screen. The Fourier plane imported from the eye plays on the other side of thecomparator screen. Say the comparator has sensitivity to the sum of the juxtaposed signals on either side of thethin screen. Thus, the comparator will develop a high amplitude signal -- the "Voila!" -- wherever andwhenever there is good agreement between a projected pattern from a memory flashlight and an importedpattern from the eye.There are of course no literal flashlights projecting Fourier patterns onto comparator screens in the brain. Thelights and projections are metaphors for processes that are carried out numerically in the model, using channelnumbers for addressing, for pixel memory including phase conservation, and for summation.If this model is viable then the photo above polka dotted with an array of Fourier flashlights is a significantillustration. It explains how we can glance at an object from a bygone époque and immediately identify it. It alsoexplains how that same object can be pictured in different visual contexts captured at several different past
  • moments.The visual memory is not an image retrieved by combing through a serial archive of old, static, stored images.The memory is "live," fully deployed in an enormous array, waiting in anticipation for reality to arrive from theretina of the eye.What does this suggest about the performance of the system? Neuroanatomy has identified, so far, about 30representations of the retina in the cerebral cortex. By replicating the incoming Fourier plane, one can multiplythe area available for the deployment of Fourier flashlights, increasing the number and variety of the arrayedmemories that wait in anticipation of an incoming image.So many flashlights. It has a Darwinian quality. Thousands of memories are on offer all the time. Upon thearrival of a new image on the retina, one or more memorized images shall be selected. It is never necessary todiscover an exact fit to the incoming image. The signal from a Van Heerden detector ascends with and reportssimilarity, so it naturally finds a “best fit.”The memory of memoriesThe chopped, short, looped film strip memories for images can be further edited or recombined in such a wayas to store, in a serial format, only useful images. By this we mean images that have been very frequentlymatched to incoming images from the retina. Such images are, in effect, played back again and again. Thisrequires a memory for memories. A hit parade of useful associations. We can call such a pared down memoryprojector a metamemory.The comparators criterion for a "hit" is likeness. Similarity. Over time, objects that happen to be alike wouldsteadily accumulate in a metamemory. The notion is roughly analogous to the conventional concept of priming.A metamemory based on like-kind associations will outscore the serial memories and produce the quickestrecalls. In other words, it will succeed and grow. With "likeness" as the selection criterion, one would expect tosee the whole system evolve as the animal matures -- growing steadily away from serial recall, which is simplybased on recording order, and toward recall based on association and analogy.For example, a sea animal’s immediate surroundings – a rock, a coral head, a neighbor who is a Grouper, asprinkled pattern of featherduster worms… all this could be stored as a readily accessible library of constantlyrecurring rims, edges and patterns and colors. These valuable memory strips could be allocated extra space onthe screen, for more detail and higher resolution.One purpose of this more compact and efficient strip of memory would be to help the animal notice novelty inits immediate environment – anything not familiar gliding into the quotidian scene. A second advantage is aquick read of edible prey and dangerous predators -- and visual elements associated with them -- lines, curves,colors, patterns and textures.It would be helpful to have a metamemory that excerpts the recent past, the "just now." A metamemory couldbe written to exclude all the trivial steps that intervene between cause and effect.A modern creature – a Grand Prix racing driver for example – will have developed a metamemory that stringstogether the downshift points and apexes of every successive corner and straightaway in Monaco or theNürburgring.But note that it is no longer necessary to imagine a stream of memories that arrive and are stored in a linear timesequence like a film strip of a racetrack. An anthropologist might have a metamemory for the totems and iconsof every culture she has studied. A biochemistry student will have a metamemory for amino acids, for sugars,for the ox-phos pathway, and branching, non-linear alternative pathways like the phosphogluconate shunt.
  • One can probably regress this principle, so that there come to be metamemories of metamemories, using keyimages as tabs, or points of entry. The comparators all operate in the Fourier plane, which is offstage andinvisible to the conscious intelligence.Replacing the retinal imageFinally, instead of simply scanning against the incoming image on the retina of the eye, seeking somethingsimilar from past experience -- the metamemories might scan against each other’s products, which are imagesextracted from the past.An image from a memory associated with a former boyfriend from the 1980s might include a Chineserestaurant from the 1980s. A bright red menu from that restaurant might be matched by a bright red pair ofboots that seemed fashionable in 2005.The requirement for serial recall is now exploded. We can jump from visual fragment to visual fragment, andmatch memories not just with the eye’s reality of the moment but also with past realities glimpsed and recordedin past moments. The fragments are serial recordings, but they are short, selected, and looped.ConclusionsThe past and present coexist in the visual memory of the brain.It seems that a visual memory model structured in this way could mix present and past imagery into somethingnew. In the frequency domain images can be combined and recombined and subsequently Fourier transformedinto literal images never before seen. In this way the system can do more than remember. It can invent.By arraying thousands of Fourier flashlights simultaneously and in parallel, memory recall can be madesudden.This is a brain model that can actually function using our absurdly slow-moving nerve impulses, which havetypical speeds ranging from just 60 mph to 265 mph.The model is able to do so much work in parallel and simultaneously because of a peculiar property ofrecordings made in the frequency domain that conserve spatial phase information: Each tiny part one mightisolate encodes and can be used to reproduce the whole of a spatial image.Thousands of elements of an incoming Fourier plane from the retina can be separately and simultaneouslycompared with past Fourier plane images pumped out of memory. The model is massively parallel, incremental-analog, and massively, simultaneously multitasking.It requires a multichannel neuron and the conservation of spatial phase. Its playground and operating system isthe Fourier plane of the brain’s retina of memory. This crucial Fourier plane of the brain is antipodal to, and is arepresentation of, the back focal plane of the lens of the human eye.posted by John Harris at 11:43 AM
  • Color Appearance and the Emergence and Evolution of Basic Color Lexicons Paul Kay Luisa Maffi U. of California, Berkeley Northwestern U. kay@cogsci.berkeley.edu maffi@nwu.edu March 7, 1999 Abstract Various revisions of the Berlin and Kay (1969) model of the evolution of basiccolor term systems have been produced in the last thirty years, motivated by bothempirical and theoretical considerations. On the empirical side, new facts about colornaming systems have continually come to light, which have demanded adjustments inthe descriptive model. On the theoretical side, there has been a sustained effort to findmotivation in the vision science literature regarding color appearance for thesynchronic and diachronic constraints observed to govern color terminology systems.The present paper continues the pursuit of both of these goals. A new empiricalquestion is addressed with data from the World Color Survey (WCS) and a revisedmodel is proposed, which both responds to recently raised empirical questions andprovides new motivation from the field of color vision for the observed constraints oncolor naming.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 1
  • 0 Introduction The fact that different languages provide different lexical classifications of colorhas long been known. In the nineteenth century, it was not uncommon to infer fromthis observation that languages which fail to make a lexical distinction between whatEuropeans recognize as two qualitatively distinct colors, such as green and blue, do sobecause their speakers cannot discriminate the colors in question perceptually. Forexample, William Gladstone wrote, on the basis of philological investigations ofHomeric Greek, "... that the organ or color and its impressions were but partiallydeveloped among the Greeks of the heroic age" (1858, cited by Berlin and Kay 1969: 135).Similar views were widespread among Gladstones contemporaries (see Berlin and Kay1969: 134-151). They did not, however, go entirely unchallenged. As early as 1880, theGerman opthalmologist, Hugo Magnus, recognized that a populations failure toimpose a lexical distinction between colors does not necessarily reflect a deficit amongits members in the perceptual ability to discriminate those colors (Magnus 1880: 34-35,discussed in Berlin and Kay 1969: 144ff). While the nineteenth and early twentieth century students of color vocabulariesworked mostly within the predominantly evolutionary approach to things social andcultural characteristic of the time, with the ascendance in the 1920s, 30s and 40s oflinguistic and cultural relativity, spearheaded by Edward Sapir (e.g., 1921: 219) and B.L.Whorf (e.g., 1956 [1940]: 212 ff.), color came to be singled out as the parade example of alexical domain in which the control of language over perception is patent, that is, of theview diameterically opposed to that of Gladstone and his fellows. Although neitherSapir nor Whorf ever wrote on color words, the presentation of the lexical domain ofcolor as the empirical locus classicus of linguistic relativity and language determinismwas reflected in a small number of highly influential empirical studies (Ray 1952, 1953,Conklin 1955) and in numerous survey and textbook presentations (e.g., Nida 1959: 13,Gleason 1961: 4, Bohannan 1963: 35ff, Krauss 1968). Berlin and Kay (1969) used a set of stimulus materials developed earlier byLenneberg and Roberts (1956) in a Whorfian-influenced study to assess the meanings ofthe basic color terms of twenty languages and extended their two main conclusions toanother seventy-eight languages reported in the literature. These conclusions were (1)that there are universals in the semantics of color in (probably) all languages: all of themajor color terms they found appeared to be based on one or more of eleven focalcolors, and (2) that there exists an apparent evolutionary sequence for the developmentof color lexicons according to which black and white precede red, red precedes green andyellow, green and yellow precede blue, blue precedes brown and brown precedes purple,pink, orange and gray. While psychologists, including specialists in color vision,largely welcomed these findings (Bornstein 1973a,b, Brown 1976, Collier et al. 1976,Miller and Johnson-Laird 1976, Ratliff 1976, Shepard 1992, Zollinger 1972, 1976, 1979),anthropologists expressed skepticism, principally on methodological grounds (e.g.,Hickerson 1971, Durbin 1972, Collier 1973, Conklin 1973).1 In the ensuing years, a number of empirical studies of color terminology systemsin field settings confirmed the broad outlines of the Berlin and Kay findings, whileamending many details (e.g., Heider 1972a,b, Heider and Olivier 1972, Heinrich 1972,Kuschel and Monberg 1974, Dougherty 1975, 1977, Hage and Hawkes 1975, Berlin andColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 2
  • Berlin 1975, among many others). These studies led to an early reformulation of theencoding sequence (Berlin and Berlin 1975, Kay 1975). Subsequently, Kay and McDaniel(1978) again reconceptualized the encoding sequence. This reformulation was based on(1) further empirical descriptive work, (2) earlier experiments of Chad K. McDanielworking with William Wooten (McDaniel 1972), which had established the identity ofthe green, yellow and blue Berlin and Kay semantic focal points with the correspondingpsychophysically determined unique hues, and (3) the introduction of a fuzzy setformalism2 (See now Zadeh 1996). The Kay and McDaniel model emphasized (1) thesix primary colors of opponent theory (black, white, red, yellow, green, blue)3, (2) certainfuzzy unions of these categories (notably, green or blue, red or yellow, black or green orblue, white or red or yellow), which are named only in evolutionarily early systems,and (3) the binary colors of the vision literature (e.g., purple, orange), which Kay andMcDaniel referred to as derived categories. These are based on fuzzy intersections ofprimaries and tend strongly to be named only in systems in which all (or most) of theunion- based (or composite) categories have already dissolved into their constituentprimaries. Kay and McDaniel also related the universals of color semantics in thismodel, which was based squarely on the six psychophysical primaries of opponenttheory, to the psychophysical and neurophysiological results of R. De Valois and hisassociates (De Valois et al. 1974 [psychophysics of macaque color vision], De Valois et al.1966, De Valois and Jacobs 1968 [neurophysiology of macaque color vision]). In recent years there have been two additional refinements of the model (Kay,Berlin and Merrifield 1991 [KBM], Kay, Berlin, Maffi and Merrifield 1997 [KBMM]), towhich we will return. Also there have been two major empirical surveys, whoseresults largely support the two broad hypotheses of semantic universals andevolutionary development of basic color term systems. These are the World ColorSurvey, whose results are discussed in this paper and the Mesoamerican Color Survey(MacLaury 1997, and earlier publications cited there).4 Throughout all these revisions,two of the original empirical generalizations of Berlin and Kay (1969) have beenmaintained.I There exists a small set of perceptual landmarks (that we can now identify with the Hering primary colors: black, white, red, yellow, green, blue5) which individually or in combination form the basis of the denotation of most of the major color terms of most of the languages of world.6II Languages are frequently observed to gain basic color terms in a partially fixed order. Languages are infrequently or never observed to lose basic color terms.7 The various revisions of the 1969 model have been motivated by both empiricaland theoretical considerations. On the empirical side, new facts about color namingsystems have come to light, which have demanded adjustments in the descriptivemodel. On the theoretical side, there has been a sustained effort to find motivation inthe literature on color appearance for the synchronic and diachronic constraintsobserved to govern color terminology systems. The present paper continues thepursuit of both of these goals. A new empirical question is addressed with data fromthe World Color Survey (WCS) and a revised model is proposed, which responds toColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 3
  • recently raised empirical questions and provides new motivation from the field ofcolor vision for the observed constraints on color naming.0.1 The Emergence Hypothesis A tacit assumption made by Berlin and Kay (1969) and maintained throughoutrevisions of the model to date has been the proposition that "all languages possess asmall set of words (or word senses) each of whose significatum is a color concept andwhose significata jointly partition the psychological color space" (Kay in press 1: 1).This assumption has been challenged, explicitly by Maffi (1990a) and Levinson (1997),implicitly by Lyons (1995, in press, cf. Kay in press 2), and by Lucy and the team ofSaunders and van Brakel.8 The rejection of this assumption has been christened theEmergence Hypothesis (EH). According to the EH, not all languages necessarily possessa small set of words or word senses each of whose significatum is a color concept andwhose significata jointly partition the perceptual color space. If we admit the EH as aworking hypothesis, several questions immediately arise. First, what proportion of the worlds languages are non-partition languages, thatis, fail to have lexical sets of simple, salient words whose significata partition theperceptual color space? Secondly, in the case of partition languages, to what extent and in what mannerdo they conform to generalizations I and II above? Thirdly, in the case of non-partition languages, to what extent and in whatmanner do they correspond to generalizations I and II? Regarding the first question, it appears that in the ethnographic present non-partition languages are rare. The data from most languages studied in the WCS give noindication of non-partition status. (The exceptions are discussed in section 3 below.)Also, most reports on color term systems in the literature and in personalcommunications received by the authors give no suggestion that the language beingreported fails to provide a simple lexical partition of the color space. One might objectthat such reports merely betray an unreflecting assumption, based on the reportersown language, that every language partitions the color space with a simple lexical set.Such a conjecture is neither provable nor disprovable. In any case, the apparent paucityof non-partition languages in the ethnographic present may not be representative ofhuman history. Specifically, just as there are no two-term ("Stage I" in the model to beintroduced) languages in the WCS sample and very few reported in the literature9, therelative lack of non-partition languages in the ethnographic present may reflect to anunknown degree the (putative) facts that (1) some extant partition languages were non-partition languages in the past and (2) some extinct non-partition languages may haveleft no non-partitioning descendants, or no descendants at all. Again, it is not obvioushow empirical evidence may be brought to bear on such conjectures. We hope that thepresent paper will help stimulate field linguists and linguistic ethnographers toexamine the color lexicons of the languages they encounter for evidence of non-partition status. It is unlikely at this point in world history that many more non-partition languages will be discovered, which makes the discovery and careful study ofColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 4
  • each one all the more important. Philological reconstructions of data on extinctlanguages (e.g., Lyons 1995, in press on Ancient Greek) and exegetical reanalyses ofreports that were originally aimed at different goals (e.g., Lyons in press on Hanunóo,Lucy 1996, 1997 on Hanunóo and Zuni, Wierzbicka, 1996: 306-308 on Hanunóo) areunlikely to cast more than hazy light on the matter. Rather, carefully controlled,contemporary field studies aimed directly at EH issues, like that of Levinson 1997, areneeded. (For discussion, see Kay 1997). The answer to the second question (How do color-space-partitioning languagessatisfy I and II?) will largely be provided, we hope, by a forthcoming monographreporting the results of the WCS. That monograph will assess in detail the extent towhich each of the 110 languages of the survey fits, or fails to fit, the new modelpresented here. The present paper also provides an initial attempt to answer the third question(How might non-partition languages satisfy I and II?) by examining the data of Yélîdnye(Levinson 1977) and the relevant data from the WCS. The new model maintains theapplication of generalizations I and II to partition languages embodied in the KBMMmodel, while extending their application to non-partition languages. The goal of thispaper is, therefore, to propose a general model of universals and evolution of basiccolor term systems, which (a) yields a slightly modified version of the KBMM model asthe statistically predominant special case, partition languages, (b) accounts for non-partition (EH) languages and (c) derives these results from independent observationsregarding (i) lexical structure and (ii) color appearance. Additionally, the proposedmodel provides an explanation for the hitherto recalcitrant puzzle posed by theexistence of composite categories comprising both yellow and green (KBM, MacLaury1987, 1997: 74, passim).1 Principles of the New Model The model is based on four principles. The first principle derives from linguisticobservations, the other three from observations regarding color appearance.1.1 Partition The partition principle subsumes under a broad generalization the specifictendency for languages to provide a small set of basic color terms which jointlypartition the perceptual color space. Studies of other lexical domains by ethnographicsemanticists and structuralist lexicographers have shown a tendency for languages tocontain sets of lexical items which partition certain obvious notional domains, such askin relations, locally observable living organisms, regions of human (and animal)bodies, periods of the solar day, cardinal directions, seasons of the solar year,conversational participants (e.g., as reflected in person/number/gender systems), and soon.10 Ethnographic semanticists have often emphasized the differences in the waysdistinct languages lexically partition a given notionally defined domain. Less oftenthey have called attention to cross-language similarities in the ways certain notionaldomains are lexically partitioned. All such comparisons are based on the tacitassumption that each of the languages being compared partitions the domain lexically.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 5
  • This widespread tendency for notionally salient domains to be partitioned by a set oflexemes is what we refer to as the partition principle.(0) Partition: In notional domains of universal or quasi-universal cultural salience (kin relations, living things, colors, etc.), languages tend to assign significata to lexical items in such a way as to partition the denotata of the domain. 11 The strong tendency of languages to conform to Partition accounts for the rarityof non-parition languages. The fact that Partition expresses a strong tendency, ratherthan an exceptionless rule, is consistent with the fact that non-partition languages doexist. The amount of information carried by the colors of objects may affect the salienceof the color domain. In a technologically simple society, color is a more predictableproperty of things than in a technologically complex one. Except perhaps for a few pairsof closely related species of birds or of fish, it is rare that naturally occurring objects orthe artifacts of technologically simple societies are distinguishable only by color. Intechnologically complex societies, on the other hand, artifacts are frequently to be toldapart only by color. The limiting case is perhaps color coding, as used in signal lights,electric wires and other color-based semiotic media. But almost every kind of materialthing we encounter in daily life: clothing, books, cars, houses, ... presents us with thepossibility that two tokens of the same type will be distinguishable only, or most easily,by their colors. As the colors of artifacts become increasingly subject to deliberatemanipulation, color becomes an increasingly important dimension for distinguishingthings and hence for distinguishing them in discourse. As technology develops, theincreased importance of color as a distinguishing property of objects appears to be animportant factor in causing languages to add basic color terms, i. e., to refine the lexicalpartition of the color domain (Casson 1997). The same process provides a plausible reason for the transition from non-partition to partition languages. Specifically, non-partition languages, like early-stagelanguages, may be spoken in societies where color is of relatively low culturalsalience.12 If we assume that cultural salience is promoted by increased functional loadin communication, we expect a rise in technological complexity to both push a non-partition language toward full partition status and cause a language that already has afull partition of the color space to refine that partition, that is, to move further alongthe (partially ordered) universal evolutionary trajectory. On this view, both theevolution of basic color term systems and the evolution toward basic color termsystems result in large measure from increasing technological control of color: astechnological control of color increases, its manipulation in the manufacture ofeveryday artifacts causes it to bear an increasingly greater functional load in everydaylinguistic communication and thereby to achieve greater cultural salience.13 Greatercultural salience of color induces partition of the color space where it does not alreadyexist and leads to increasingly finer partitions of the color space where a partitionalready exists. This process may still be going on (Kay and McDaniel 1978, Chapanis1965).Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 6
  • 1.2 Principles of color term universals and evolution based on color appearance The three remaining principles of the currently proposed model are color-appearance-based. All presuppose the elemental nature of (1) the four primary huesensations of opponent theory: red, yellow, green and blue and (2) the two fundamentalachromatic sensations black and white. The overwhelming majority of visionscientists interested in color appearance and categorization now accept the basic natureof these six color sensations on the basis of a wide range of psychophysical andcognitive psychological evidence.14 The model of Kay and McDaniel (1978) mistakenlyequated these six primary color sensations with the six classes of cells identified by DeValois et al. (1966) in the parvocellular layer of the macaque lateral geniculate nucleus(LGN), and called them fundamental neural response categories.15 These six cell typescannot simply constitute the neural substrate of the six primary color sensationsbecause, among other reasons, (1) they contain nothing corresponding to the shortwavelength red response and (2) the points at which the spectrally opponent cells areneither excited nor inhibited are not in the right places to produce the observed uniquehue points (Derrington et al. 1984, Abramov and Gordon 1994, Abramov 1997). Weshould note, however, that it is psychophysical experiments that have established theshort wavelength red response and the unique hue points in a variety of ways,involving diverse techniques such as hue cancellation and hue scaling (Boynton andGordon 1965, Jameson and Hurvich 1955, Hurvich and Jameson 1955, Ingling et al.1995, Sternheim and Boynton 1966, Werner and Wooten 1979, Wooten and Miller 1997.See Hardin 1988, Chapter I for general discussion.) The elemental character of black,white, red, yellow, green and blue in human color sensation, within a conceptualframework that includes the notions of chromacy/achromacy, unique hues andopponent processes, is no longer thought to be grounded in macaque LGN neurons, butthis framework is nonetheless broadly accepted by vision scientists as the best way toorganize a wide range of psychophysical, cognitive-psychological and animal-behavioral observations (Abramov 1997, Abramov and Gordon 1997, Bornstein 1997,Hardin 1988, Ingling 1997, Kaiser and Boynton 1996, Miller 1997a,b, Sandell et al. 1979,Shepard 1994, Van Laar 1997, Werner and Bieber 1997, Wooten and Miller 1997, Sivik1997.16 For dissent, from two distinct points of view, see Jameson and DAndrade 1997and Saunders and van Brakel 1997).1.2.1 Black and White The first principle governing the refinement of lexical partitions of the colorspace is given by the fact that object recognition is possible without color, e.g., in blackand white movies and photographs. In fact, it is often claimed – probably anexaggeration, according to Wooten and Miller (1997) – that the rods are only active inscotopic (low illumination, black and white) vision and contribute nothing to photopic(bright illumination, color) vision. Certainly, the cones transmit luminance as well aschromatic information (De Valois and De Valois 1975, 1993). It is clear nonetheless thatobjects can be distinguished rather well at levels of illumination too low to stimulatethe cones to give rise to hue sensations. The distinction between spectral sensitivity(spectral opponency) and spectral non-sensitivity (spectral non-opponency) is alsoreflected in the anatomical and physiological distinction between the magna layer andparvo layer cells of the lateral geniculate nucleus. "The great majority, if not all, of theColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 7
  • P-cells in a macaque... have responses that are spectrally opponent ... while M-cells aregenerally spectrally non-opponent..." (Abramov 1997: 101, citing the primary literaturefor both observations.) Macaque color vision has been shown to be in essential respectslike that of humans by De Valois et al. (1974). At a more phenomenal level we canobserve that people with no color vision (those suffering from achromatopsia) oftenhave no problem with object recognition (Davidoff 1997, Mollon 1989). In short, wehave a black-and-white vision system that gives us most of shape discrimination andobject recognition with color vision laid on top of it. Indeed, students of vision haveoccasionally been led to speculate how and why our species should have evolved colorvision at all (e.g., Mollon 1989, Hardin 1992). A person lacking color vision is not blind.A person lacking the black-and-white vision necessary to recognize objects is blind. The partitioning principle motivated by these observations is:(1) Black and White (Bk&W): Distinguish black and white.1.2.2 Warm and Cool A distinction between "warm" and "cool" colors has long been recognized bycolor specialists from both the arts (e.g., art critics and historians and teachers ofpainting) and the sciences. Red, yellow and intermediate orange are "warm"; green andblue are "cool." Hardin (1988: 129ff) provides an excellent discussion of bothexperimental and philosophical considerations of the warm/cool distinctions,beginning with Hume and concluding, in part, "These explanations [of the warm/coolhue associations and cross-modal associations] are of varying degrees of persuasiveness,but they should at least caution us not to put too much weight on any single analogicalformulation. However, they should not blind us to the striking fact that there is aremarkable clustering of oppositions which correlate with this hue division" (Hardin1988: 129). Early experiments (e.g., Newhall 1941) established red as a warm hue. Morerecent experiments (Katra and Wooten 1995), controlled for brightness and saturation,have shown that English-speaking subjects judgments of warm color peak in theorange region and cover reds and yellows, while judgments of cool color peak in theblue region and cover greens and blues. Judgments of warmth/coolness also correlatewith saturation (saturated colors are judged warm), but not significantly with lightness.These groupings of basic hue sensations into warm and cool agree with those commonin the art world. A recent study of color term acquisition in two-year-olds, besidesfinding surprising control of color terms in very young children, found no significantdifferences among colors in the age at which they were acquired but did find that "therewas some evidence that our subjects maintained the warm-cool boundary; in generalthey make more within- than across-boundary errors" (Shatz et al. 1996: 197). Bothartistic tradition and recent experimental evidence thus point to an affinity between redand yellow on the one hand and between green and blue on the other. A recent colormodel based on observed cone frequencies (De Valois and De Valois 1993, 1996) positsan intermediate stage of chromatic information processing that consists of twochannels: one red/yellow and one green/blue (See Kay and Berlin 1997 for discussion ofthe possible relevance of this model to cross-language color naming). Thepsychological color space, so-called, is notoriously lacking in a reliable long-distancemetric17. We take the facts mentioned in this paragraph to indicate, albeit indirectly,Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 8
  • that red and yellow are experienced as in some respect similar and that green and blueare experienced as similar in that same respect. The partitioning principle motivated by the warm and cool groupings of hues is:(2) Warm and Cool (Wa&C): Distinguish the warm primaries (red and yellow) from the cool primaries (green and blue).1.2.3 Red The final principle we propose for explaining the ways languages go aboutlexically partitioning the color space is the apparent salience of red among the huesensations. Despite the intuitive judgment, shared by vision specialists and lay people,that red is somehow the most salient of hues, non-anecdotal support for this idea is notoverwhelming. Humphrey (1976) writes I shall list briefly some of the particular evidence which demonstrates how, in a variety of contexts, red seems to have a very special significance for man. (1) Large fields of red light induce physiological symptoms of emotional arousal – changes in heart rate, skin resistance and the electrical activity of the brain. (2) In patients suffering from certain pathological disorders, for instance cerebellar palsy, these physiological effects become exaggerated – in cerebellar patients red light may cause intolerable distress, exacerbating the disorders of posture and movement, lowering pain thresholds and causing a general disruption of thought and skilled behaviour. (3) When the affective value of colours is measured by a technique, the semantic differential, which is far subtler than a simple preference test, men rate red as a heavy, powerful, active, hot colour. (4) When the apparent weight of colours is measured directly by asking men to find the balance point between two discs of colour, red is consistently judged to be the heaviest. (5) In the evolution of languages, red is without exception the first colour word to enter the vocabulary – in a study of ninety-six [sic, actually ninety-eight] languages Berlin and Kay (1969) found thirty [sic, actually twenty-one] in which the only colour word (apart from black and white) was red. (6) In the development of a childs language red again usually comes first, and when adults are asked simply to reel off colour words as fast as they can they show a very strong tendency to start with red. (7) When colour vision is impaired by central brain lesions, red vision is most resistant to loss and quickest to recover (Humphrey 1976: 97f). It is disquieting to note, however, that the only reference provided for thevarious claims in the passage just cited is to Berlin and Kay (1969) and that both of thenumbers reported from that work are inaccurate. Following the publication of Berlin and Kay (1969), Floyd Ratliff, a distinguishedvision scientist, attempted to provide motivation from color science for the 1969 model(Ratliff 1976). Among the elements he sought to explain was the prominence of red.Ratliff noted that the long-wavelength cones are very frequent in the fovea and aremuch more sensitive in the long wave end of the spectrum than the other two coneColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 9
  • types. This line of argument has not, to our knowledge, been found persuasive. Forexample, Wooten and Miller (1997: 86) point out that Ratliff established no linkbetween the observation of a dense population of long-wavelength sensitive cones inthe fovea and the subjective salience of red. They note further that subjective colorsensations are linked quite indirectly to cone responses, probably at cortical levelsbeyond the primary visual area. At this time, the firmest warrant we can find for the apparent prominence of redamong the hue sensations comes from research on color term acquisition. There havebeen several studies of the acquisition of color terms in English-speaking children.Some of these have noted a weak correlation of the order of acquisition of basic colorterms with the original Berlin and Kay encoding sequence and others no suchcorrelation. An observation that has not previously been made about these studies andother studies of acquisition of color terms by English-speaking children is that in everycase in which acquisition data is reported by term, red is the first of the hue termsacquired (Wolfe 1890 [data reproduced in Descoeudres 1946: 119]; Winch 1910: 475,passim; Heider 1971: 453. Table 3; Johnson 1977: 309f, Tables 1, 3 and 4). The same fact –that red is the first hue term acquired by children – is also evidenced by studies onGerman (Winch 1910: 477); Spanish (Harkness 1973: 185, Figure 4); Russian (Istomina1963: 42f, Tables 6, 7); Italian (Winch 1910: 456-457); French (Descoeudres 1946: 118f),Mam [Mayan] (Harkness 1973: 184, Figure 3 [red and green tied for first for 7-8 yearolds]); Setswana [Bantu] (Davies et al. 1994: 701-702, tables 4 and 5 [Setswana termsonly]); and West Futuna (Dougherty 1975, table 5.718). In every study we have found inwhich a difference between colors was reported in the order with which childrenacquire terms for them, the term for red was the first hue term acquired.19 The finalprinciple of color naming expresses the primacy of red among the hue sensations.(3) Red: Distinguish red.2. The WCS Data to be Accounted For The 110 basic color terminology systems of the WCS were classified by KBMM (p.33, Figure 2.4) into eleven basic types, based on the combinations of Hering primaryterms they contain. As shown in Figure 1, Stages I (two terms) and II (three terms) eachcorrespond to a single type, Stage III (four terms) comprises three types, Stage IV (fiveterms) three types, and Stage V a single type. (Two stages hypothesized by KBMM,IIIBk/Bu and IIIY/G, have been eliminated from the model because no instances of themhave been discovered in the WCS data.20) In Figure 1, columns represent evolutionarystages, every stage containing one more basic color term than the preceding stage.KBMM recognized languages in transition between types. In Figure 1, an arrowindicates the transitions from the type occurring on its left to the type toward which itpoints. For example, Stage II systems can develop into either type IIIG/Bu or typeIIIBk/G/Bu.21 Stage IIIBk/G/Bu systems can develop into systems of either Stage IVG/Bu orStage IVBk/Bu, and so on. Progression through successive stages, starting with a two-term systems andadding a term at each stage, results from the interaction of the Partition principle withthe six Hering primaries. Initially, minimal application of Partition dictates division ofColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 10
  • the color space into two categories. Of course, Partition alone doesnt tell us what thesecategories will be, that is, how the primaries will be grouped in the cells of the resultingpartition. That is the job of the three additional, color-appearance-based principles.Each of the three remaining principles is applied in order until an unequivocal result isdetermined. At each succeeding change point, this process is repeated: Partition isapplied, minimally, to dictate that the number of cells (= named basic color categories =basic color terms) be increased by one. Then principles (1), (2) and (3) are applied inorder until an uniquivocal result regarding the nature of the new partition is achieved.(Whenever application of a principle is decisive in determining the refinement of thepartition, principles of lower priority are not consulted. Eventually there remains onlyone possible refinement of the existing partition, so application of principle (0) sufficesto produce an unequivocal result and no other principles are consulted.) The order of application (1) > (2) > (3) expresses an empirical hypothesisregarding the relative importance of the principles. This order seems to correlate –impressionistically speaking – with the wieght of the evidence we have been able toamass for principles (1), (2), and (3) in sections 1.2.1, 1.2.2, and 1.2.3, respectively. Theordering of Parition (0) before the other three principles follows from the fact that whatwe are using the principles for is to refine a partition and principle (0) is the one thatsays, "Refine the partition." W R  R  W ➙ Y  GY ➘  (III  Bk/G/Bu   ➘  Bk/Bu  Bk/G/Bu)  (IVBk/Bu)  W W  R/Y  R  R  W W/R/Y  ➙ W ➚  G/Bu  ➙ Y ➙ Y  G  (III  R/Y Bk G/Bu Bk/G/Bu   Bk/G/Bu  ➙    Bu    Bk G/Bu)  (IVG/Bu)  Bk W R  R  W ➚  Y/G/Bu   Bu Y/G  ➚  (III  Bk   ➙  Bk  Y/G/Bu)  (IVY/G)  I II III IV V Figure 1. Types and Evolutionary Stages of Basic Color Term Systems (Adapted from KBMM, Figure 2.4, page 33)2.1 The Main Line of Basic Color Term Evolution The languages of the WCS indicate five possible paths ending in Stage V, whichcan be traced by following the arrows from stage to stage in Figure 1. These define fiveevolutionary trajectories, identified as A, B, C, D, E, in Table 1.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 11
  • A: I ➙ II ➙ IIIG/Bu ➙ IVG/Bu ➙ V B: I ➙ II ➙ IIIBk/G/Bu ➙ IVG/Bu ➙ V C: I ➙ II ➙ IIIBk/G/Bu ➙ IVBk/Bu ➙ V D: ? 2 2 ? IIIY/G/Bu ➙ IVG/Bu ➙ V E: ? ? IIIY/G/Bu ➙ IVY/G ➙ V Table 1. Five Evolutionary Trajectories of Basic Color Term Systems The evolutionary trajectories of Table 1 are not equally frequent in the WCS data.A single trajectory, which we call the main line of color term evolution, accounts forthe vast majority of WCS languages. Ninety-one of the 110 WCS languages (83%)belong either to one of the five stages of Trajectory A or to a transition between two ofthese stages, as shown in Figure 2, where an outlined numeral within bracketsrepresents the number of WCS languages found at the corresponding stage and anoutlined numeral between brackets represents the number of WCS languages found intransition between the stages indicated.23 W W  R/Y  R  R  W W/R/Y W Bk/G/Bu  ➙  R/Y 6   Bk/G/Bu 3➙   G/Bu  4➙ Y  11 ➙  Y 23     Bu  Bk 3 G/Bu 41 G    Bk    (IIIG/Bu) (IVG/Bu)  Bk I II III IV V Figure 2. Main Line (Trajectory A) of Evolutionary Development of Basic Color Lexicons. Total number of languages represented is 91 (83% of WCS languages)24.2.2 Accounting for the Main Line of Color Term Evolution Our internal representation of color, independent of language, appears to play animportant role in determining the evolution of color term systems. Our task in thepresent section is to explain why Stage I systems have the particular shape they do andwhy each type of basic color lexicon on the main line (Figure 2) evolves into thesucceeding type. The evolutionary sequence of the main line can be motivated byassuming, as we have above, that at each stage transition principles (0) Partition, (1)Bk & W, (2) Wa & C and (3) Red operate in that order until an uniquivocal result isreached. We assume that Partition acts minimally and incrementally. That is, webegin with the color space lexically partitioned into just two cells, that is, namedcategories, each cell (named category) representing a union of some subset of the sixfuzzy sets corresponding to the primary colors and then at each new stage, reapplicationof Partition and the other three principles adds a single new cell (i.e., term), until the sixprimaries have each received a distinct basic color term.2.2.1 Stage I Stage I is motivated as follows. Principle (1) [ Bk&W] dictates that one cell of thetwo-cell partition shall contain B and the other W. Principle (2) [Wa&C] dictates thatone cell shall contain both R and Y and the other shall contain both G and Bu. Itremains to be determined whether the warm primaries will be grouped with W andthe cool with Bk or vice versa. Yellow is an inherently light color. Perusal of theColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 12
  • systematically arranged stimuli of any standard color order system, e.g., Munsell, NCS,or OSA, shows that low lightness colors of the same dominant wavelengths as yelloware not seen as yellow, but as orange, olive, brown, or something hard to name. To saythat Y is an inherently light color it to say that Y and W have an inherent affinity. Thefact that one of the warm colors, Y, is seen as similar to W correlates with, and partiallyexplains, the apparently universal association of the warm hues with W and, therefore,of the cool hues with Bk in Stage I systems.25 Independent of the inherent lightness of Y, in discussing various cross-modalassociations to the warm/cool distinction in hues, Hardin (1988: 129) notes that amongthese are active/passive, exciting/inhibiting, up/down, and positive/negative (in anon-evaluative sense). Hardin advances – cautiously – the speculation that we mayhave sensitivity to the polarity of opponent processes, in particular that we may havesome neural level which records such facts as that R, Y and W each represent excitationof their opponent process, while G, Bu and Bk represent inhibition of thecorresponding opponent mechanisms (1988: 130). Our interest here is not to evaluateHardins speculation regarding a possible neural basis for the white/warm, dark/cooland correlative cross-modal associations but simply to note the existence of thewhite/warm and dark/cool associations. The strength of the association of warm hues with W and of cool hues with Bk isreinforced by experiments performed by James Boster (1986). In one experiment Bostergave twenty-one naive English-speaking subjects eight color chips, representing focalexamples of the categories black, white, red, orange, yellow, green, blue and purple. Theinitial instruction was to sort the chips into two groups "on the basis of which colorsyou think are most similar to each other..." (Boster 1986: 64). The overwhelmingpreference was to put white, red, orange and yellow into one group and green, blue andblack and purple into the other. Two thirds of Bosters subjects chose this exactdivision into two subsets. (There are 2,080 ways a set of eight elements can be dividedinto two non-empty subsets.) In a second experiment, the same instruction was givento a group of eighteen subjects, using as stimuli the eight color words rather than thecolored chips. Substantially the same result was obtained.2.2.2 From Stage I to Stage II As indicated above, in deriving each stage from the preceding stage, we apply tothe earlier system principles (0), (1), (2) and (3) in that order. Applying Principle (1) to aStage I system means that either W and R/Y are given separate terms or that Bk andG/Bu are given separate terms. Principle (2) is irrelevant to the decision whether R/Yor G/Bu gets a separate term, so principle (3) is consulted. Principle (3) is relevant,dictating that the division be made between W and R/Y, since this choice promotes thedistinguishing of R more than if the division were made between Bk and G/Bu. Theresult is a Stage II system, with terms for W, R/Y, and Bk/G/Bu.2.2.3 From Stage II to Stage IIIG/Bu Applying Principle (1) to a Stage II system requires the extraction of Bk fromBk/G/Bu, since W already has a separate term. The result is a Stage IIIG/Bu system, withColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 13
  • terms for W, Bk, R/Y and G/Bu. Principles (2) and (3) have no opportunity to applybecause application of (1) has been sufficient to add a term, satisfying Partition.2.2.4 From Stage IIIG/Bu to Stage IVG/Bu Principle (1) does not apply to a Stage IIIG/Bu system, since Bk and W already haveseparate terms. Principle (2) is uninformative with respect to breaking up R/Y or G/Bu.Principle (3) requires breaking up R/Y into R and Y. The result is a Stage IVG/Bu system,with terms for Bk, W, R, Y and G/Bu.2.2.5 From Stage IVG/Bu to Stage V Since a Stage IVG/Bu system contains only one composite category, G/Bu,application of Partition alone is sufficient to determine the result. To satisfy Partition,G/Bu must be divided into G and Bu, yielding a Stage V system with terms for Bk, W,R, Y, G, and Bu. Partition, Bk&W, Wa&C and Red, operating in that order, account forthe evolution of eighty-three percent of the WCS languages.2.3 Less Frequent Evolutionary Trajectories As shown in Figure 1, there are also cases of WCS languages in which thetransition from Stage II to Stage III involves separating R and Y, instead of Bk andG/Bu. The result is a Stage IIIBk/G/Bu system. Such systems are involved inevolutionary trajectories B and C in Table 1. A Stage IIIBk/G/Bu system can in turndevelop into either a Stage IVBk/Bu or a Stage IVG/Bu system, as shown in Figure 3.Figure 3 adds these types, and related transitions, to the main line of developmentshown in Figure 2. W R    1➙ W  R Y  YG 3   III Bk/G/Bu   Bk/Bu   Bk/G/Bu   IVBk/Bu  1➘ 1➘ R  W W/R/Y W  ➙  R/Y 6  1➚ Y  Bk/G/Bu   Bk/G/Bu  3➘ G 23   Bu Bk  W W  R/Y  R  11➚  G/Bu 3  4➙ Y 41    Bk G/Bu   Bk  IIIG/Bu  IVG/Bu  I II III IV V Figure 3, Evolutionary Trajectories A, B and C26 As shown in Figure 3 and note 26, an additional ten languages (10% of the WCStotal) reflect the minority choice of splitting R and Y in going from Stages II to III, ratherColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 14
  • than dividing Bk/G/Bu into Bk and G/Bu. This amounts to promoting Principle (3)[Red] over Principles (1) [Bk&W] and (2) [Wa&C]. Of these ten languages, one is intransition from a mainline type (II) to a non-mainline type (IIIBk/G/Bu), while five are intransition from a non-mainline type to a mainline type.27 Summarizing to this point, 101 of the 110 WCS languages (92%) showexceptionless operation of Partition – that is, no evidence of the EH – either in theirpresent condition or, by plausible inference, in a former state. Of these, ninety-one(90%) conform to the ordering of Partition and the three color-appearance-basedprinciples, Bk & W, Wa & C, and Red: (0) > (1) > (2) > (3). Ten of these 101 languages(10%) order Principle (3) over Principles (1) and (2) at some point in their evolutionarydevelopment. We turn our attention now to the exceptional cases, the languages inwhich Partition appears to fail at least partially, and in which the EH consequently findssupport.283 Predictions of the Model for Non-Partition (EH) Languages The only thoroughly documented non-partition language of which we are awareis not a WCS language but Yélîdnye, a Non-Austronesian language of Rossel Island(Papua New Guinea), reported in Levinson (1997). Because Levinson undertook hisinvestigation of Yélîdnye color naming with the EH specifically in mind and because hecollected, in addition to the WCS color naming tasks, a fuller range of morphosyntacticand usage information than it was possible to ask the WCS field linguists to record, hisreport of a positive finding on the EH deserves close attention. In very brief summary,Yélîdnye has basic color terms for B, W and R and a secondary but well establishedsimple term for a certain red color, specifically that of a shell used in traditional inter-island (Kula) trade. The three basic terms kpêdekpêde black, kpaapîkpaapî white and mtyemtye(or taataa) red are recognizable as reduplications of nominal roots denoting a treespecies, a pure white cockatoo and a "startling crimson" parrot, respectively. Levinsonnotes that there is a "regular", that is, partially productive, derivational pattern in thislanguage according to which reduplication of a nominal root may derive an adjectivedenoting a salient property of the denotatum of the noun. For example, mty:aamty:aasweet < mty:aa honey. Levinson points out that if one knows the white cockatooand red parrot one might well guess the meanings of the reduplicated forms of theirrespective names to mean white and red, though of course one could not be certainthat some other salient property (such as the loud screech of the parrot) was not beingpicked out. One might wish to argue on the basis of these observations that the red andwhite words of Yélîdnye fail the first criterion of basicness of Berlin and Kay: "... [the]meaning [of the color word] is not predictable from the meaning of its parts" (1969: 6).Having raised the issue, and suggesting that it may be one that arises in manylanguages of Oceania and Australia, Levinson appears convinced in the end that thewhite and red terms of Yélîdnye should be considered basic color terms, whatever anarrow application to them of the Berlin and Kay criteria might yield. But he suggeststhat observations such as these might be interpreted as casting doubt on the claim thatYélîdnye has, aside from kpêdekpêde black, any basic color terms in the sense of BerlinColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 15
  • and Kay (1969) and perhaps that some languages of Oceania or Australia have any basiccolor terms at all. On closer examination, this fear appears to be groundless. Yélîdnye kpaapîkpaapîwhite and mtyemtye (or taataa) red do not fail the Berlin and Kay (1969: 6) criterionof non-predictability of meaning. At issue is the proper understanding of(non)-predictability of meaning. Makkai (1972) makes a relevant distinction betweenencoding idioms and decoding idioms (see also Fillmore, Kay and OConnor 1988:540f). An expression that a speaker would not know how to assemble from knowledgeof everything else in a language is an encoding idiom. An expression that a hearerwould not be able to interpret from knowledge of everything else in a language is adecoding idiom. There are many encoding idioms which are not decoding idioms, thatis, there are many expressions which are interpretable on first hearing but that onewouldnt know how to form from knowledge of everything else in the grammar. Forexample, on first hearing one of the expressions light as a feather, heavy as lead orquick as a wink, any English speaker could probably figure out exactly what was meant,but one could not know in advance that these are conventional ways of saying verylight, very heavy, very quick, even knowing that English contains a pattern [A as aN] for forming expressions meaning very A. There is no way to know in advance thatone may say, for example, light as a feather, easy as pie or easy as duck soup, but not*light as an ash, *easy as cake or *easy as goose fritters, or that one may say one (two, ...)at a time, but not *one at the time [as in French], *one to a time, *one by the time, etc.,without learning each separate fact. Analogously, Yélîdnye could have reduplicated forms of the word meaning leaffor green, of turmeric or banana for yellow, and of sky for blue, but it doesnt.29Even though this particular derivational process of Yélîdnye is used frequently (and isin that sense "regular"), the speaker of Yélîdnye nonetheless has to memorizeseparately each of the cases in which it is used, so each of these cases represents aseparate encoding idiom although it is possible that none are decoding idioms. If weinterpret the non-predictability criterion for basic color terms as requiring that suchterms be encoding idioms – which seems appropriate since language users have tospeak their language as well as understand it – then kpaapîkpaapî and mtyemtye (ortaataa) meet the non-predictability criterion for basicness – as they meet all the otherB&K criteria. Insofar as similar reduplication process are reflected in the color terms ofother Oceanic and Australian languages, as Levinson suggests, the same argumentapplies to them. The Bk, W and R terms of Yélîdnye are not extended; this is not a Stage IIlanguage, in which, for example, the term that includes Bk also includes G and Bu andthe term that includes R also includes Y and orange. Interestingly, there are fixedphrasal expressions denoting each of the colors G, Y and Bu. The most highlyconventionalized and widely shared of these is for G, then Y, then Bu – the last subjectto a large number of phrasal expressions and considerable interspeaker variation. TheBk and W terms are somewhat more firmly established and subject to less interspeakervariation than the basic R terms (due perhaps to dialect synonymy in R, plus possibleinterference from the Kula-shell term). Much of the color space is simply unnamed byany expression Levinson was able to elicit.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 16
  • Yélîdnye seems clearly to be a non-partition language, i.e., one testifying to thecorrectness of the EH. On the other hand, Yélîdnye has a very Berlin and Kay (1969)feel to it: the best established terms are for Bk and W, then R, all basic, then non-basicG, Y, and Bu in that order, and after these nothing worth mentioning. Yélîdnye is not apartition language. It nevertheless exhibits the salience of Bk and W dictated byPrinciple (1 ) and the salience of R dictated by Principle (3). Principle (2) has no scope tooperate in Yélîdnye, since in this non-partition language there are no composite termsfor Principle (2) to apply to.3.1 WCS Evidence for the EH Levinson suggests strongly that for Yélîdnye we should think of Bk, W and R asreceiving basic color terms (the last with two competing synonyms, deriving fromdifferent dialect names for the eponymous parrot), and these only. There are alsoseveral languages in the WCS with well-established words for BK, W and R (notextended), with varying ways of treating lexically the rest of the colors. We mustcaution here that the WCS data were not collected specifically to test the EH and that welack for these data much information on the morpho-syntactic status of the terms andthe kind of ethnographic observation of their use in natural discourse that would bevery useful for assessing the applicability to these languages of the EH. Nevertheless,some patterns may be observed. The existence of languages with basic terms only for (non-extended) Bk, W and Ris consistent with the fact that Bk, W and R are singled out by Principles (1) and (3),while Y, G, and Bu are not distinguished per se by any principle of the model. Suchlanguages are spoken in communities in which color as such may not have achievedsufficient cultural salience, and thus functional load in communication, for Partition totake full effect in the color domain, leaving the field open, as it were, for Principles (1)and (3) to cause only the inherently most prominent color sensations to receive simplenames. So far our model has yielded an explanation for color systems with basic termsfor Bk, W and R only, and which therefor do not partition the perceptual color space. Ifa language has gone this far and no farther, we will find well established terms for Bk,W and R and widespread variability on WCS tasks in the rest of the color space, withmany competing terms and little agreement among speakers. To a significant degree,six of the seven of the WCS languages that remain to be discussed fit this descriptionand the seventh, Cree, although a partition language in the present, may be inferred tohave been non-partition in the reconstructable past..3.2 The Residue Predicted: Y/G/Bu Terms Suppose a language has developed non-extended terms for B, W, and R, ignoringPartition. If Partition now asserts itself, a composite term for Y/G/Bu appears,producing a Stage IIIY/G/Bu system. This type of system contains basic terms for Bk, Wand R and a composite term covering Y, G and Bu. The WCS sample contains two clearexample of such systems, Karajá (Brazil) and Lele (Chad).Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 17
  • Systems of this type are reported elsewhere in the literature. For example,Arrernte (Pama-Nyungan, Australia) apparently had such a system (David Wilkins, pc1998, see also Spencer and Gillin 1927, noted in Berlin and Kay 1969: 67f).30 Kinkade(1988) reconstructs a Proto-Salishan Y/G/Bu term because of clear etymologicalrelatedness of terms including or restricted to Y and terms including or restricted to Buin contemporary Salishan languages. (See also the discussion in MacLaury 1997: 74,passim) The notion that the IIIY/G/Bu systems developed historically from systems likeYélîdnye – with basic color terms for Bk, W, and R and no lexical partition of the colorspace – is of course speculative. We have no historical record or detailedreconstruction of such a development for either WCS or non-WCS languages. But thisconjecture fits the model to the available data very neatly, accounting for evolutionarytrajectories D and E of Table 1. The questions signaled by the question marks in Table 1have now been addressed.3.2.1 The Yellow/Green Mystery Resolved The development of a Y/G/Bu term as a delayed assertion of Partition provides aplausible explanation of the puzzle regarding the origin of Y/G terms.31 In the Kay andMcDaniel model, every language is assumed to start out as a Stage I (fully partitioned)system and to develop further via successive division of composites until all sixlandmark colors receive separate terms. Since Y and G belong to distinct composites atStage I, it is a mystery under this model how Y/G composites ever come into being(See KBM for further discussion.) Under the present model, which allows for the EHand therefore does not assume that all languages start from a fully partitioned Stage Isystem, a plausible scenario for the genesis of Y/G composites suggests itself. Once asystem with restricted Bk, W, and R plus a composite Y/G/Bu exists (Stage IIIY/G/Bu), itmay develop further in either to two ways. If the Y/G/Bu composite splits into Y andG/Bu the result is a mainline Stage IVG/Bu system, with terms for Bk, W, R, Y andG/Bu (Trajectory D). But if the other possible split of the Y/G/Bu category occurs, intoY/G and Bu, the result is a Stage IVY/G system, with terms for Bk, W, R, Y/G, and Bu(Trajectory E). Among WCS languages, Cree is an example (the sole example) of such asystem and it is the only WCS language with a Y/G composite. To our knowledge, allother languages reported to contain Y/G composites are also of this type, Stage IVY/G.The developmental scenario just sketched, in which Y/G/Bu categories result from thelate imposition of Partition on Bk-W-R (only) languages and in which Y/G compositesresult from the breakup of Y/G/Bu composites, eliminates from the theory the logicallypossible but unattested KBMM Stage IIIY/G type, with terms for W, R, Y/G and Bk/Bu.MacLaury (1987) has documented Y/G terms in several Salishan languages, confirmingthe earlier reports of Kinkade and others. Kinkade (1988) and MacLaury (1997: 74,passim) conclude that some G/Bu, Bu and Y/G terms observed in modern Salishanlanguages reflect a Proto-Salishan Y/G/Bu term. To summarize the Y/G story: Y/G/Bu terms arise when ascendency of the color-appearance-based priciples (1) and (3) over Partition and (2) leads to the naming of Bk,W and R, leaving the rest of the color space unnamed; then Partition exerts itself,resulting in the creation of a Y/G/Bu term to name the rest of the primary colors andpartition the space. The inherently unstable Y/G/Bu category (containing the opponentcolors Y and Bu) usually breaks down into Y and G/Bu, leaving no trace of its priorColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 18
  • existence (since the resulting mainline Stage IVG/Bu type more usually arises from thebreakup of R/Y in a mainline IIIG/Bu system). But occasionally Y/G/Bu breaks downinto Y/G and Bu, producing a Stage IVY/G system, with terms for Bk, W, R, Y/G and Bu.3.2.2 Mopping Up: Four EH Languages? Finally, the WCS files include four languages which appear to represent mixedcases of the patterns outlined above, in the sense of including terms clearly centered onBk, W, and R, with two or more conflicting patterns competing for the remaining area.The single generalization that brings these cases together is that the regions of the colorspace corresponding to Bk, W and R are well named (including either a separate nameor inclusion in a standard composite category like Bk/G/Bu), while the strategy fornaming the remaining areas is some combination of (1) extension of the Bk, W, Rterms according to the usual story of composites, (2) existence of a special Y/G/Bu (orNot-[Bk/W/R]) term, or (3) relatively strong secondary terms for Y, G, Bu, or G/Bu (orsome subset thereof, akin to the Yélîdnye pattern). These languages tend also to bethose in which there is unusual interspeaker variation in the use of shared terms and amarked degree of idiosyncrasy in the selection of terms used. Culina (Peru, Brazil) is similar to Karajá and Lele in containing terms for W, Rand an extended yellow term that covers much of G and Bu, especially in the lightershades. There is, however, no Bk term, but instead an unmistakable Bk/G/Bu term.Mundu (Sudan) represents a similar situation. There are clear terms for W, R, andBk/G/Bu, but there is also a highly salient term which includes Y, G and Bu, issomewhat focused in Y, and which seems to gloss best as everything which is notblack, white or red. Moreover, Mundu contains a secondary term largely synonymouswith the one just mentioned but much less well established. Culina and Mundu bothseem to mix the W, R, Y, Bk/G/Bu strategy (Stage IIIBk/G/Bu) with the W, R, Bk, Y/G/Bustrategy (Stage IIIY/G/Bu). The final two languages, Kuku-Yalanji and Murrinh-Patha (both Australian)illustrate most clearly the pattern of Bk, W, R plus confusion. In this respect they comethe closest in the WCS sample to the Yélîdnye pattern in which only restricted Bk, Wand R receive basic color terms. Kuku-Yalanji has well-established terms for Bk, W,and R, although the Bk term shows some extension into Bu (as well as into Br, which iscommon). The R term, ngala-ngala (< ngala blood) does not include yellow. Thelanguage contains two additional major terms, although these are less well establishedthan the first three. One, of these, kayal, is used regularly by only half of the speakersconsulted, maps as a G/Bu term for the language as a whole, is focused in G, anddenotes only G for some speakers. It also means unripe according to the WCS fieldlinguists, H. and R. Hershberger. Oates (1992: 126) gives kayal with the gloss [color]green only, indicating that the word is among those "not recognised by speakers today"[Recall that the WCS data were gathered fourteen years before the Oates dictionary wasproduced.] Oates also contains an entry kalki unripe. Only nine of twenty WCScollaborators used kayal with a well-established green or grue sense; kayal is not a basiccolor term of Kuku-Yalanji. There is also a word used by seventeen of the twentyKuku-Yalanji speakers for everything outside of Bk, W and R proper, burrkul (orburkul). However, it is clear that collaborators with well-established words for green orColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 19
  • grue do not use burrkul for those colors. The Hershbergers gloss burrkul as non-descript, dirty and anything which is not black, white or red. The last gloss seemsaimed less at the conceptual content of the word than at the way it is deployed in theWCS naming task. Oates lists burkul, not among the color words, but among"Describing Words Relating to Things", giving its gloss as not clear, not clean, murkyor dirty, said about water, windows, mirrors, photos, skin (Oates 1992: 83). burrkul isnot a basic color term of Kuku-Yalanji. Murrinh-Patha presents perhaps the most confusing array of terms in the WCS.In addition to standard Bk, W, and R terms (with the Bk term thipmam extended a bitinto Bu, as well as into Br, and the R term bukmantharr not extended into Y), there arefour other widely used terms: ngatin (used by twenty-one of the twenty-five WCScollaborators), wudanil (twenty-four speakers), tumamka/tupmanka (nineteenspeakers) and wipmanarri (fifteen speakers). ngatin appears in the pooled data to be aY/G term, but it is used by some speakers for yellow/orange/(brown) only, by someothers for G/Bu only, and by some for G only. wudanil is used by one or anotherspeaker for virtually everything outside of Bk, W, and R. Its distribution on the WCStasks lead one to infer that it might be a non-color term, like Kuku-Yalanji burrkul(non-descript, not clear, not clean,...) and could be used for any surface appearance forwhich the speaker does not have an apt descriptor. However, Michael Walsh (pc 1998)is unable to corroborate that gloss. "[wudanil] could be a verb form which as beenconventionalized to refer to colours but could also have an independent (verbal) life ofits own." tumamka/tupmanka appears to be a widely extend, low consensus G/Buterm if one considers the aggregate mapping, but there is great interspeaker variation inhow the term is used. For some speakers tumamka/tupmanka is blue, for some G/Bu,for many nothing so easy to describe. Walsh writes (pc 1998) that tumamka/tupmankaalso appears to be a verbal form. Finally, wipmanarri covers approximately the samerange of colors as Warlpiri walyawalya (< walya earth), which can denote deepbrowns, reddish browns, lighter – yellowish– browns and oranges, yellowish salmons,pinkish purples and other light purples. This is just about the range of colors earthtakes on in the central Australian desert, where Warlpiri is located (although we donthave comparable information for the area in which Murrinh-Patha is spoken).However, there is no indication in Walshs information that wipmanarri has anetymological relation to earth, possibly being related instead to the body-part word forback. The Murrinh-Patha Bk, W and R terms are much better established than the lastfour discussed (and some less frequent terms that we havent discussed here).Murrinh-Patha fits the best of any language in the WCS sample the formula Bk, W, Rplus confusion32.4 Summary This paper presents a model of color term evolution employing one language-based principle, Partition, and three color-appearance-based principles: Bk&W, Wa&Cand Red. The Emergence Hypothesis is defined as the possibility that not all languagesobey Partition perfectly in the color domain. Straightforward application of these fourprinciples, with the ranking: Partition > Bk&W > Wa&C > Red, defines the main lineof color term evolution (Trajectory A of Table 1, Figure 2), accounting for 91 (83%) ofthe languages in the WCS sample. When Red supersedes Bk&W and Wa&C at theColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 20
  • transition from Stage II to Stage III, the possibility of two additional types is created,accounting for an additional 10 WCS languages, bringing the part of the total WCSsample accounted for to 101 (92%) (Figure 3, Trajectories B and C.) Two more languagesdepart non-wildly from any of the nine types in Figure 1 but do not challenge the EH(See note 22), bringing the number of non-EH languages to 103 (94% of the WCS total).The remaining seven languages show, to varying degrees, evidence for the possibleoperation of the EH. Two of these, Karajá and Lele are IIIY/G/Bu languages, illustratingTrajectory D. One language, Cree, illustrates Stage IVY/G (Trajectory E). The remainingfour languages (Culina, Mundu, Kuku-Yalanji and Murrinh-Patha) all show Bk, W,and R prominence, with a mixture of other strategies, combined with considerableinterspeaker variability. A plausible solution to the apparent mystery of Y/G composites is provided bythe current model: EH languages may develop somewhat along the lines of Yélîdnye,assigning basic terms, according to principles (1) [B&W] and (3) [Red], only to restrictedBk, W, and R, violating Partition. Subsequently, Partition comes into play and aY/G/Bu term appears, covering the remaining primary colors.33 (There is somesuggestive evidence that Y is the most common focus for this term, but the data are sosparse that no reliable conclusion can be drawn here.) In some cases, the Y/G/Bu termmay then divide into Bu and Y/G terms. According to Kinkade (1988) and MacLaury(1997: 74, passim) this appears to have happened in some Salishan languages.34 Since the original Berlin and Kay (1969) study, there have been numerous fieldstudies by linguists and anthropologists which have added data to test and refine thetheory of universals and evolutionary development of basic color term systems. Tothis we can add the Mesoamerican Color Survey and the WCS. This line of researchhas resulted in several reformulations of the evolutionary model and will probablycontinue to do so. Recently, a striking aspect of this tradition of research has consistedin the complex of observations and speculations we have referred to globally as theEmergence Hypothesis. The reformulations of the evolutionary model have, since1978, also been guided by an effort to explain whatever universals in color semantics wecan by independent findings from the vision literature. It is encouraging that thepresent reformulation of the model (1) covers a wider range of partitioning languagesthan any model hitherto, (2) is based more more firmly on independent principlesgoverning color appearance than previous models, (3) sheds some new light on non-partitioning languages and on what the relation of these may be to the partitioninglanguages, their evolutionary sequence, and the color appearance factors that appear tounderly it and (4) goes some way toward solving the hitherto unresolved problem ofcomposite (fuzzy union) categories comprising both yellow and green.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 21
  • ReferencesAbramov, I. 1997 Physiological mechanisms of color vision. In Hardin and Maffi (1997) 89- 117. [See below]Abramov, I. and J. Gordon 1994 Color appearance: On seeing red – or yellow, or green, or blue. Annual Review of Psychology 45: 451-485.Abramov, I. and J. Gordon 1997 Constraining color categories: The problem of the baby and the bathwater. Behavioral and Brain Sciences. 20 (2): 179-180.Berlin, Brent 1992 Ethnobiological classification : principles of categorization of plants and animals in traditional societies. Princeton, N.J. : Princeton University Press.Berlin, Brent and Elois Ann Berlin 1975 Aguaruna color categories. American Ethnologist 2: 61-87.Berlin Brent and Paul Kay 1969 Basic Color Terms: Their Universality and Evolution. Berkeley and Los Angeles: University of California.Bohannon, Paul 1963 Social Anthropology. New York: Holt, Rinehart and Winston.Bornstein, Marc H. 1973a The psychophysiological component of cultural difference in color naming and illusion susceptibility. Behavioral Science Notes 1: 41-101. 1973b Color vision and color naming: A psychophysiological hypothesis of cultural difference. Psychological Bulletin 80: 257-285. 1997 Selective vision. Behavioral and Brain Sciences 20 (2): 180-181.Boster, James 1986 Can individuals recapitulate the evolutionary development of color lexicons? Ethnology 26 (1): 61-74.Boynton, R.M. and J. Gordon 1965 Bezold-Brücke shift measured by color-naming technique. Journal of the Optical Society of America 55: 78-86.Brown, Roger W. 1976 Reference. Cognition 4: 125-153.Casson, Ronald W. 1997 Color shift: evolution of English color terms from brightness to hue. In Hardin and Maffi (1997) 224-239.Chapanis, Alphonse 1965 Color names for color space. American Scientist 53: 327-346.Collier, George A. 1973 Review of Basic Color Terms. Language 49: 245-248 1976 Further evidence for universal color categories. Language 52: 884-890.Conklin, Harold C. 1955 Hanunóo color categories. Southwestern Journal of Anthropology 11: 339- 344. 1973 Color categorization: Review of Basic Color Terms, by Brent Berlin and Paul Kay. Language 75: 931-942.Davidoff, JulesColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 22
  • 1997 The neuropsychology of color. In Hardin and Maffi (1997) 118-134.Davies, Ian, Greville Corbett, Harry McGurk and David Jerrett 1994 A developmental study of the acquisition of colour terms in Setswana. Journal of Child Language 21: 693-712.Derrington, A.M, J. Krauskopf and P. Lennie (1984) Chromatic mechanisms in lateral geniculate nucleus of macaque. Journal of Physiology 357: 241-265.Descoeudres, Alice 1946 Le développement de lenfant de deux à sept ans. Neuchatel et Paris: Delachaux & Niestlé.De Valois, Russell L., I. Abramov and G.H. Jacobs 1966 Analysis of response patterns of LGN cells. Journal of the Optical Society of America 56: 966-977.De Valois, Russell L. and G.H. Jacobs 1968 Primate color vision. Science 162: 533-540.De Valois, Russel L. and Karen K. De Valois 1975 Neural coding of color. In E.C. Carterette and M.P. Friedman (eds) Handbook of Perception. Vol V. New York: Academic. 117-166. 1993 A multi-stage color model. Vision Research. 33: 1053-1065.` 1996 A three-stage color model – comment. Vision Research. 36:833-836.De Valois, Russell L., Herman C. Morgan, Martha C. Polson, William R. Mead, and Elaine M. Hull 1974 Psychophysical studies of monkey vision–I. Macaque luminosity and color vision tests. Vision Research 14: 53-67.Dougherty, Janet D. 1975 A universalist analysis of variation and change in color semantics. Berkeley Ph. D. dissertation. 1977 Color categorization in West Futunese: Variability and change. In Sociocultural Dimensions of Language Change. B.G. Blount and M. Sanches, eds., New Yor/London: Plenum. pp. 133-148.Durbin, Marshall 1972 Review of Basic Color Terms. Semiotica 6: 257-278.Fillmore, Charles J., Paul Kay and M.C. OConnor 1988 Regularity and idiomaticity in grammatical constructions: the case of let alone. Language 64 (3) 501-538.Gladstone, William E. 1858 Studies on Homer and the Homeric Age. London: Oxford University Press.Gleason, H.A. 1961 An Introduction to Descriptive Linguistics. New York: Holt, Rinehart and Winston.Hage, Per and Christen Hawkes 1975 Binumarien color categories. Ethnology 24: 287-300.Hård, A. and L. Sivik 1981 NCS-Natural Color System: a Swedish standard for color notation. Color Research and Application 6 (3): 129-138.Hardin, C.L. 1988 Color for Philosophers: Unweaving the Rainbow. Indianapolis, IN: Hackett.Hardin, C.L.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 23
  • 1992 The virtues of illusion. Philosophical Studies 67: 153-166.Hardin, C.L and Luisa Maffi (eds). 1997. Color Categories in Thought and Language. Cambridge, UK: Cambridge University Press.Harkness, Sara 1973 Universal aspects of learning color codes: A study in two cultures. Ethos 1(2): 175-200.Heider, Eleanor Rosch 1971 "Focal" color areas and the development of color names. Developmental Psychology 4 (3): 447-455. 1972a Universals in color naming and memory. Journal of Experimental Psychology 93: 1-20. 1972b Probabilities, sampling and the ethnographic method: The case of Dani colour names. Man 7: 448-466.Heider, Eleanor Rosch and Donald C. Olivier 1972 The structure of the color space for naming and memory in two languages. Cognitive Psychology 3: 337-354.Heinrich, Albert C. 1972 A non-European system of color classification. Anthropological Linguistics 14: 220-227.Hering, Ewald 1964 [1920] Outlines of a theory of the light sense. Translated by L. M. Hurvich and D. Jameson. Cambridge, Mass: Harvard.Hickerson, Nancy 1971 Review of Berlin and Kay (1969). International Journal of American Linguistics 37: 257-270.Humphrey, N.K. 1976 The colour currency of nature. In T. Porter and B. Mikelides (eds) Color for Architecture. New York: Van Nostrand.Hurvich, L.M. and D. Jameson 1955 Some qualitative effects of an opponent-colors theory: II. Brightness, saturation and hue in normal and dichromatic vision. Journal of the Optical Society of America 45: 602-616.Ingling, C.R., Jr. 1997 Constraints on the definitions of "unique hues" and "opponent channels". Behavioral and Brain Sciences. 20 (2): 194-195.Ingling, C.R., Jr., Jonathan P. Barley, and Nadeem Ghani (1995) Chromatic content of spectral lights. Vision Research 36 (16): 2537-2551.Istomina, Z. M. 1963 Perception and naming of color in early childhood. Soviet Psychology and Psychiatry 1 (2): 37-45.Jameson, D. and L.M. Hurvich 1955 Some quantitative aspects of an opponent-colors theory: I. Chromatic responses and spectral saturation. Journal of the Optical Society of America 45: 546-552.Jameson, K. and R.G. D’AndradeColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 24
  • 1977 It’s not really red, green, yellow, blue: An inquiry into perceptual color space. In Hardin and Maffi (1977) 295-319.Johnson, E. G. 1977 The development of color knowledge in preschool children. Child Development 48: 308-311.Katra, E. and B.R. Wooten 1995 Perceived lightness/darkness and warmth/coolness in chromatic experience. ms.Kaiser, Peter K. and Robert M. Boynton 1996 Human Color Vision, 2nd edition. Washington, D.C.: Optical Society of AmericaKay, Paul 1975 Synchronic variability and diachronic change in basic color terms. Language in Society 4: 257-270. In press 1 The emergence of basic color lexicons hypothesis: A comment on "The vocabulary of colour with particular reference to Ancient Greek and Classical Latin" by John Lyons. The Language of Colour in the Mediterranean, Alexander Borg (ed) Wiesbaden: Otto Harrassowitz. In press 2 Sur la méthodologie de létude comparative des termes de couleur. Anthropologie et Sociétés. [English version available at www.icsi.berkeley.edu/~kay/color.meth.ps]Kay, Paul and Brent Berlin 1997 There are non-trivial constraints on color categorization. Behavioral and Brain Sciences 20: 196-202.Kay, Paul, Brent Berlin, Luisa Maffi and William Merrifield 1997 Color naming across languages. In Color Categories in Thought and Language. C.L. Hardin and Luisa Maffi, eds., Cambridge, England: Cambridge University Press.Kay, Paul, Brent Berlin and William Merrifield 1991 Biocultural implications of systems of color naming. Journal of Linguistic Anthropology 1: 12-25.Kay, Paul and Chad K. McDaniel 1978 The linguistic significance of the meanings of basic color terms. Language 54: 610-646.Kinkade, M. Dale 1988 Proto-Salishan color. In W. Shipley (ed) In Honor of Mary Haas. Berlin: Mouton de Gruyter.Krauss, Robert M. 1968 Language as a symbolic process. American Scientist 56: 265-278.Kuschel, Rolf and Torben Monberg 1974 We dont talk much about colour here: A study of colour semantics on Bellona Island. Man 9: 213-242.Lenneberg, Erik H. and John M. Roberts 1956 The Language of Experience: A study in Methodology. Memoir 13 of International Journal of American Linguistics.Levinson, Stephen A.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 25
  • 1997. The theory of basic color terms and Yélîdnye. Working papers of the Cognitive Anthropology Group, Max Plank Institute for Psycholinguistics. Nijmegen.Lucy, John A. 1996 The scope of linguistic relativity: An analysis and review of empirical research. In J. Gumperz and S. Levinson (eds) Rethinking Linguistic Relativity. Cambridge University Press. 37-69. 1997 The linguistics of color. In Hardin and Maffi (1997) 320-346.Lyons , John A. 1995 Colour in language. In T. Lamb and J. Bourriau (eds) Colour: Art and Science. Cambridge University Press. In Press The vocabulary of colour with particular reference to Ancient Greek and Classical Latin. In The Language of Colour in the Mediterranean, Alexander Borg (ed) Wiesbaden: Otto Harrassowitz.MacLaury, Robert.E. 1987 Color-category evolution and Shuswap yellow-with-green. American Anthropologist 89: 107-124. (1997) Color and Cognition in Mesoamerica. Austin, Texas: University of Texas Press.Maffi, Luisa 1988a World Color Survey report. Department of Anthropology, U.C., Berkeley. ms. 33 pp. 1988b World Color Survey typology. Department of Anthropology, U.C., Berkeley. ms. 29 pp. 1990a Cognitive anthroplogy and human categorization research. Department of Anthropology, U.C., Berkeley. ms. 113 pp. 1990b Somali color term evolution: Grammatical and semantic evidence. Anthropological Linguistics 32: 316–334.Magnus, Hugo 1880 Die geschichtliche Entwickelung des Farbensinnes. Leipzig: Viet.Makkai, Adam 1972 Idiom Stucture in English. The Hague: Mouton.McDaniel, Chad K. 1972 Hue Perception and Hue Naming. Unpublished B.A. thesis. Harvard University.Miller, David L. 1997a Beyond the elements: investigations of hue. In Hardin and Maffi eds (1997) 151-162. 1997b Over the rainbow: The classification of unique hues. Behavioral and Brain Sciences 20 (2): 204-205.Miller, George A. and P. Johnson-Laird 1976 Language and Perception. Cambridge, Massachusetts: Harvard University Press.Mollon, J.D. 1989 "Tho she kneeld in that place where they grew ...": The use and origins of primate color vision. Journal of Experimental Biology 146: 21-38.Newhall, S.M. 1941 Warmth and coolness of colors. Psychological Record 4: 198-212.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 26
  • Nida, Eugene A. 1959 Principles of translation as exemplified by Bible translating. In On Translation. Reuben A. Brower, ed., Cambridge, Massachusetts: Harvard University Press.Oates, Lynette F. 1992 Kuku-Yalanji Dictionary. Alsbury, NSW: Graeme van Brummelen.Palmer, Stephen A. in press Color, Consciousness and the isomorphism constrain. Behavioral and Brain Sciences.Pokorny, Julius, 1948-69 Indogermanisches etymologisches Worterbuch. Bern: A. Francke .Ratliff, Floyd 1976 On the psychophysiological bases of universal color terms. Proceedings of the American Philosophical Society 120: 311-330.Ray, Verne 1952 Techniques and problems in the study of human color perception. Southwestern Journal of Anthropology 8: 251-959. 1953 Human color perception and behavioral response. Transactions of the New York Academy of Sciences (series 2) 16: 98-104.Sapir, Edward 1921 Language. New York: Harcourt, Brace.Sandell, J.H., Gross, C.G., and Bornstein, M.H. 1979 Color categories in macaques. Journal of Comparative and Physiological Psychology 93: 626-635.Saunders, B.A.C and J. van Brakel 1988 Re-evaluating basic colour terms. Cultural Dynamics 1: 359-378. 1995 Translating the World Color Survey. In Geuijen, K., D. Raven and J. de Wolf (eds) Post-Modernism and Anthropology. Assen, Netherlands: van Corcum. 1997 Are there non-trivial constraints on colour categorization? Behavioral and Brain Sciences 20: 167-228.Shatz , Marilyn, Douglas Behrend, Susan A. Gelman, and Karen S. Ebeling 1996 Colour term knowledge in two-year-olds: evidence for early competence. Journal of Child Language 23: 177-199.Sivik, Lars 1997 Color systems for cognitive research. In Hardin and Maffi (1997: 163-193).Shepard, Roger 1992 The perceptual organization of colors. In The Adapted Mind. J. Barkow, L. Cosmides and J. Tooby, eds., Oxford: Oxford University Press. 1994 Perceptual-cognitive universals as reflections of the world. Psychonomic Bulletin and Review 1 (1): 2-28.Spencer, B. and F.K. Gillin 1927 The Arunta: A Study of a Stone Age People. London: Macmillan.Sternheim, C.E. and R.M. Boynton 1966 Uniqueness of perceived hues investigated with a continuous judgmental technique. Journal of Experimenal Psychology 72: 770-776.Taylor, John R. and Robert E. MacLaury, edsColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 27
  • 1995 Language and the cognitive construal of the world. (Trends in linguistics. Studies and monographs 82). Berlin, New York: Mouton de GruyterVan Laar, D. 1997 Ekphrasis in colour categorisation: Time for research, or time for revolution? Behavioral and Brain Sciences. 20 (2): 210.Werner, J.S. and M.L. Bieber 1977 Hue opponency: A constraint on colour categorization known from experience and experiment. Behavioral and Brain Sciences. 20 (2): 210.Werner, J.S. and B.R. Wooten 1979 Opponent chromatic mechanisms: Relation to photopigments and hue naming. Journal of the Optical Society of America 69: 422-434.Wierzbicka, Anna 1996 Semantics: Primes and Universals. Oxford: Oxford University Press.Whorf, Benjamin L. 1956 [1940] Science and Linguistics. In Language, Thought and Reality: The Collected Papers of Benjamin Lee Whorf, John B. Carroll, ed., Cambridge, Massachusetts: MIT Press. Originally published in Technology Review 42: 229- 231, 247-248.Winch, W.H. 1910 Color-names of English school-children. American Journal of Psychology 21 (3): 453-482.Wolfe, H.K. 1890 On the color-vocabulary of children. Nebraska University Studies 1 (3) July 1890.Wooten, B.R. and D. Miller 1997 The psychophysics of color. In Hardin and Maffi (1997) 59-88.Zadeh, Lotfi A. 1996 Fuzzy sets, fuzzy logic, and fuzzy systems: selected papers by Lotfi A. Zadeh (Editors: George J. Klir & Bo Yuan). Singapore, River Edge, N.J.: World Scientific.Zollinger, Heinrich 1972 Human color vision: An interdisciplinary research problem. Palette 40: 1-7. 1976 A linguistic approach to the cognition of colour vision. Folia Linguistica 9: 265-293. 1979 Correlations between the neurobiology of colour vision and the psycholinguistics of colour naming. Experientia 35 :1-8.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 28
  • ACKNOWLEDGMENTSBrent Berlin provided much appreciated comments on an earlier draft. He andWilliam Merrifield have been valued colleagues throughout the World Color Survey.We also acknowledge with gratitude the advice of David Nash regarding theinterpretation of the WCS Warlpiri, Kuku-Yalanji and Murrinh-Patha data, DavidWilkins regarding the interpretation of his Arrernte data, as well as the WCS Warlpiri,Kuku-Yalanji, and Murrinh-Patha materials, and Michael Walsh regarding theinterpretation of the WCS Murrinh-Patha data. David Wilkins also allowed us toreview original data on Arrernte color naming, focal choices, and discourse use,collected by him in 1997. Our heartfelt thanks to all these colleagues. Errors are thecontribution of the authors.Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 29
  • NOTES1It is perhaps worthy of passing note that the qualms regarding experimental method were expressedalmost exclusively by non-experimentalist anthropologists, while interested psychologists, all of whomwere experimentalists, apparently accepted the rough-and-ready experimental procedures of Berlin andKay because of the robustness of their results (See, for example, Boynton 1997:135f). Collier, both ananthropologist and an experimentalist, is a special case. In (1973) he expresssed the suspicion that theBerlin and Kay results might be an artifact of their stimuli providing maximum available saturation ateach hue/lightness coordinate. Subsequently, Collier et al. (1976) reported an experiment in which thishypothesis was examined and rejected, confirming the Berlin and Kay results at a non-maximal, uniformlevel of saturation.2 Fuzzy sets allow for degrees of membership. For example a yellowish orange color can be thought of as,say, 25% red and 75% yellow, that is a member of the fuzzy set red to the degree .25 and of the fuzzy setyellow to the degree .75. The membership of an individual x in the union if two fuzzy sets, A, B, is themaximum of its membership in either. The membership of an individual in the intersection of two fuzzysets is the minimum of its membership in either. For a non-technical introduction to the basics of fuzzy settheory, see Kay and McDaniel (1978); for full technical detail see Zadeh (1996).3 Contemporary color vision theory recognizes the six primary colors, originally posited in the opponenttheory of Ewald Hering (1964 [1920]), as arranged in three opponent pairs: black/white, red/green,yellow/blue. Any color percept can be formed by combining two or more of these colors perceptually (not aspigments). Red, yellow, green and blue are the unique hues. That is, these four hues and only these can beseen as unmixed. Orange is seen as a mixture of red and yellow, chartreusse is seen as a mixture of yellowand green, but yellow, although it falls between orange and chartreuse on the hue circle, is not seen as amixture of orange and chartruese. Along with black and white, the four unique hues provide the primarylandmarks, or cardinal points, of perceptual color space, with other colors located in relation to these six.The chromatic opponent pairs are perceptually privative. That is, we cannot see red and green in the samepart of the visual field and the same for blue and yellow. (That a green pigment can be produced by mixingblue and yellow pigments is irrelevant.) Hering inferred that there must be a neural process which signalsred in one state and green in another (analogously for yellow and blue), hence the appelation "opponent"process. The achronmatic pair, black and white, are opposed, but not privative. We do see black and whitesimultaneously in various shades of gray. See Kaiser and Boynton (1996 23 f, 250-258) and, for a non-technical introduction to opponent theory, Wooten and Miller (1997).4 MacLaurys investigations of basic color term systems have led him to develop a theory of cognitivepoints of view, vantages, involving alternating attention to similarities and dissimilarities amongcognitive categories. MacLaurys (1997) interpretation of the evolution of basic color term systems isformulated largely within the vocabulary of vantage theory. Vantage theory makes broad claims in thefield of cognitive psychology (MacLaury 1997, Taylor and MacLaury 1995), which are beyond the scope ofthe present paper.5 Abbreviated below Bk, W, R, Y, G, Bu.6 Some critics of this tradition of research have misconstrued as an a priori assumption the empiricalfinding that semantic universals in color names are substantially based on the universal primary colorsensations. See, for example, Saunders and van Brakel (1988, 1995, 1997), Lucy (1996, 1997). Compare Maffi(1990a), Kay and Berlin (1997), Kay (in press). Generalization I is broader than the narrow claim of Berlinand Kay (1969) (abandoned since Kay and McDaniel 1978) that "a total universal inventory of exactlyeleven basic color categories exists from which the eleven or fewer basic color terms of any given languageare always drawn" (Berlin and Kay 1969: 2).7 We do not mean by this that basic color words are not frequently replaced by other words denoting thesame category, often borrowed words. We mean that in a given language a category once named by a basiccolor term rarely if ever becomes unnamed.8 See references in the previous note.9 Dani is the only thoroughly studied case (Heider 1972a, 1972b, Heider and Olivier 1972).10 With regard to observable living organisms, probably few languages push this tendency to the extreme ofa literally exhaustive lexical partition of the entire domain (Berlin 1992).Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 30
  • 11 In the case of color, where the categories are gradient and overlapping, in the way treated formally byKay and McDaniel (1978), by partition we intend fuzzy partition as it is there defined (Kay andMcDaniel 1978: 641ff).12 For example, Kuschel and Monberg (1974), in reporting a careful ethnographic investigation of a Stage IIcolor system, make much of their impression to this effect, going so far as to entitle their report "We donttalk much about color here; a study of colour semantics on Bellona Island."13 Development due to culture contact is doubtedless the major engine of increased technological complexityin recent – perhaps in all – times. Culture contact often provides new artifacts and manufacturingtechniques, which render color a less predictable attribute of objects. Moreover, contact with a morecomplex technology is often accompanied by contact with a language whose lexicon names more distinctcolor categories (Maffi 1990b).14 See, for example, Abramov and Gordon (1994), Hård and Sivik (1981), Wooten and Miller (1997), Hardin(1988: 29f, passim). The primacy of these six color sensations has been challenged by the post-modernistsSaunders and van Brakel (1995, 1997), who reject Kay and McDaniels (1978) "reductionist argument... [to]six basic or atomic colour categories" on the epistemological grounds, among others, that "there is noprivileged discourse in which what is true is independent of our choices, hopes and fears" (Saunders andvan Brakel 1995: 170).15 The misleading expression "fundamental neural response category" was retained in KBM.16 "Eventually someone may actually locate cells that carry out these operations" (Abramov 1997: 115).17 For example, if you wish to assess one the one hand the "distance" between a yellowish red and agreenish blue and on the other the "distance" between a yellowish green and a purplish red, there is nowell-defined, overall metric defined in color space that can tell you which of these "distances" is thegreater.18 Of the forty-seven children reported on in Dougherty (1975), eight had a term for red and lacked a termfor at least one of Y, G and Bu, while one child had terms for G and Bu but lacked a term for R (also Y).19 Not all of these differences were subjected to statistical test. A few other studies of color termacquisition were found. One reported presence and two reported absence of correlation with the full Berlinand Kay 1969 sequence, but age of acquisition for individual terms was not reported. The remainder also didnot record acquisition data for individual colors.20 In KBM two languages, Kuku-Yalanji and Murrinh-Patha, were represented as having terms for W, R,Y/G and Bk/Bu, that is, as Stage IIIY/G languages. These languages are reanalyzed in section 3, wherethey are discussed along with other languages showing strong naming for Bk, W, and R, with variablenaming elsewhere.21 The antecedents of Stage IIIY/G/Bu languages are discussed in section 3.2.22 The question marks appearing in this figure are explained in section 3.2.23 The concentration of of WCS languages on this single evolutionary path was first noted by Maffi(1988a,b).24 Since a given type may figure in more than one trajectory (e.g., type IVG/Bu appears in trajectories A, Band D), our assignment of ninety-one languages to the main line represents the maximum number of typescompatible with this trajectory, not the number of types uniquely assignable to this trajectory.25 There is independent evidence that blue is an inherently cool color (Palmer in press).26 Three languages not shown on Figure 3 are in apparent transition directly from Stage IIIBk/G/Bu toStage V.27 As may also be seen in Figure 3 (and note 26), the WCS sample does not contain any simple cases ofIIIBk/G/Bu languages, although it does contain six cases of apparent transitions either into or out of thattype.28Two of the languages in the WCS sample do not fit perfectly any of the types discussed so far, but alsoshow no evidence of the EH. Gunu (Cameroon) has terms for W, R/Y, Bk/G/Bu and Bu. It thus represents astandard Stage II system except for the presence of the blue term. The blue term is stronger than theBk/G/Bu term in the blue area, requiring that it be considered basic and therefore that Gunu be considered aviolation of the model sensu strictu. Waorani (Ecuador) is an anomalous Stage IIIG/Bu system; it containsColor Appearance and the Emergence and Evolution of Basic Color Lexicons page 31
  • terms for Bk, W/Y, R and G/Bu (rather than the standard Bk, W, R/Y and G/Bu). These two cases bring to103 (95%) the number of WCS languages which offer no support for the EH.29 And, similarly, it could have had the reduplication of a form denoting a white blossom for white or areduplication of blood for red. That is, the (hypothetical) coiner of the color term not only has to chose toform it by reduplication, but then has to chose which of several plausible bases to use. Some languageschoose blood, others fire, yet others – like Yélîdnye – choose a red bird.30 Recent unpublished data on Arrernte color terms, collected by Wilkins using the WCS stimuli, suggeststrongly that the putative Arrernte Y/G/Bu term is focused in G by all speakers and extended into both Yand Bu by a minority. One of several hypotheses consistent with the available data is that historicallythe Arrernte term now focused in green denoted a Y/G/Bu category, as reported by Spencer and Gillen(perhaps focused in green, perhaps not) and has retracted for some speakers under pressure from English.31 Maffi (1990a) raises the question whether certain Y/G/Bu (and other ) terms might not profitably beregarded as interstitial.32The word confusion here, and above, does not, of course, indicate that speakers are confused about how touse their language, but that the results of the WCS naming task are confused because the language does notappear to have a single, widely shared lexical strategy for naming certain regions of the color space. TheEH is, of course, about just such circumstances.33 Yélîdnye, however, does not show evidence of developing a Y/G/Bu term.34 As KBM point out, Latin had a G/Bu term viridis while Ancient Greek had a Y/G term khlôros. If thesewords were related, the situation would be comparable to that of the Salishan family. They are notrelated. The former probably comes from a PIE root denoting a surface appearance – perhaps shiny orbrilliant, the latter a PIE root related to growth (Pokorny 1948).Color Appearance and the Emergence and Evolution of Basic Color Lexicons page 32
  • OFFPRINTSThe Retinex Theory of Color Vision SCIENTIFIC AMERICAN DECEMBER 1977by Edwin H. Land VOL 237 NO 6 P 108-128 PUBLISHED BY W. H. FREEMAN AND COMPANY 660 MARKET STREET, SAN FRANCISCO, CALIFORNIA 94104Copyright ©1977 by Scientific American Inc. All rights reserved, Printed in the U. S. A No part of this offprint may be reproduced by any mechanical, photographic or electronic process, orin the form of a phonographic recording, nor may it be stored in a retrieval system, transmitted or otherwise copied for public or private use without written permission of the publisher.
  • The Retinex Theory of Color Vision Λ retina-and-cortex system (retinex) may treat a color as a code for a three-part report from the retina, independent of the flux of radiant energy but correlated with the reflectance of objects by Edwin H. LandT he scientific tradition of simplify­ ing the conditions of an experi­ ment has left us until recentlywithout a satisfactory explanation ofhow the eye sees color in everyday life. taken through a red filter and the picture projected in white light is taken through a green filter. It would be expected that the superposed image on the projection screen could generate only red. white retina cover the visible spectrum in three broad, overlapping curves. The pigment with a peak sensitivity at a wavelength of 440 nanometers responds in some degree to the entire lower-fre­Paradoxically the modern technology and various shades of pink. Actually quency half of the visible spectrum.of color photography has reinforced the one sees a picture remarkably similar to Each of the other two pigments re­belief that the colors discerned by New­ the full-color photograph reproduced sponds to almost two-thirds of the visi­ton in the spectrum are. with minor on the opposite page. In the red-and- ble spectrum, the two being offset atqualifications, the colors of the world white photographic projection peppers their peaks by barely 30 nanometers.around us. We know, for example, that are green, radishes and strawberries are with their peak sensitivities located atif we use daylight color film when we red. the orange is orange, the lemon and 535 and 565 nanometers [see upper illus­take a picture in the light shed by an bananas are pale yellow, the wood cut­ tration on page 4].ordinary tungsten-filament lamp, the ting board and knife handle are brown In this discussion the names of col­picture will turn out to have a strong and the design on the plate is blue. ors — " r e d " , " g r e e n " , " b l u e " and so on —reddish cast. That, we say. is because the will be reserved for the color sensation The challenge presented by our earlyrays from the tungsten filament are too we have when we look at the world red-and-white experiments led us step"red. " never asking how we ourselves around us. In short, only our eyes can by step over a 20-year period to an ex­can move constantly in and out of tung­ categorize the color of objects; spectro­ planation of how the visual system issten-lit worlds without experiencing any photometers cannot. This point is not a able to extract reliable color informa­change in the color of familiar objects: trivial one because many people view­ tion from the world around us. a worldapples, lemons, strawberries, bread, hu­ ing some of our experiments for the first in which virtually every scene is lightedman faces (the tones of which are so time will identify something as being red unevenly, in which the spectral compo­hard to get right on a television screen). or green but will then ask, as if their eyes sition of the radiation falling on a scene How, then, does the eye deal with the were being fooled. "What color is it real­ can vary enormously and in which illu­excess of " r e d " i n a tungsten-lit room? ly?" The answer is that the eye is not mination as brief as a lightning flash suf­As I hope to demonstrate in this article. being fooled. It is functioning exactly as fices for the accurate identification ofthe eye. in determining color, never per­ it must with involuntary reliability to color. If the nature of the responses ofceives the extra red because it does not see constant colors in a world illuminat­ the photoreceptors in the retina of thedepend on the flux of radiant energy ed by shifting and unpredictable fluxes eye even approximated what most of usreaching it. The eye has evolved to see of radiant energy. were taught in school, functioning pri­the world in unchanging colors, regard­ marily as intensity-level meters withless of always unpredictable, shifting Since I believe the study of color in peaks in three different parts of the spec­and uneven illumination. How the eye fully colored images is best begun by trum. we would be continually confus­achieves this remarkable feat has fasci­ examining images that are completely- ing one color with another. An objectnated me for many years. devoid of and completely uncomplicat­ that looked yellow in one part of our ed by the experience of color, let me field of view might look green or gray or In 1959 I described in these pages a describe that experience in some detail. even red when moved to a different partseries of experiments in which a scene The hypersensitive system based on the of the field. The fact remains that ob­created by the superposition of two rod cells in the retina functions at light jects retain their color identity under ablack-and-white transparencies, one pro­ levels as much as 1. 000 times weaker great variety of lighting conditions. Thisjected through a red filler and the oth­ than the systems based on the cone cells constancy is not a minor second-orderer projected without a filter (that is, in do, so that it is possible to answer the effect but is so fundamental as to call forwhite light), conveys to the eye nearly interesting question: What colors will a new description of how we sec color.the gamut of colors present in the origi­ one see if only the rod system is activat­nal scene [see "Experiments in Color Vi­ ed? One procedure is to put on a pairsion. " by Edwin H. Land: SCIENTIFIC of tightly fitting goggles equipped withAMERICAN Offprint N o . 223]. To pro­duce such "red-and-white" images the T he visual pigments are photosensi­ tive molecules that respond to a wide band of light frequencies. The neutral-density filters that reduce the in­ cident light by a factor of 30. 000. Afterpicture projected through the red filter is one has worn the goggles for about half three pigments in the cone cells of the2
  • an h o u r objects in a r o o m i l l u m i n a t e d to in a b l a c k - a n d - w h i t e p h o t o g r a p h t a k e n r e a c h i n g t h e eye. T h e i l l u m i n a t i o n cant h e typical level of 20 foot-candles will t h r o u g h a g r e e n c o l o r - s e p a r a t i o n filter. easily be a r r a n g e d so t h a t t h e r e is m o r eb e c o m e visible. T h e effective i l l u m i n a - In o t h e r w o r d s , t h e r e d s will a p p e a r very flux f r o m a region t h a t c o n t i n u e s to l o o ktion in the r o o m will t h u s be 1 / 1 , 500 d a r k , the greens lighter, the blues d a r k . v e r y d a r k t h a n t h e r e is f r o m a r e g i o nfoot-candle. As one looks around the t h e w h i t e s light a n d t h e b l a c k s v e r y t h a t c o n t i n u e s t o l o o k v e r y light, w h e t h -r o o m t h e familiar c o l o r e d objects will dark. er these r e g i o n s a r e t h r e e - d i m e n s i o n a lbe seen devoid of color, exhibiting in- I n this c o l o r l e s s w o r l d o n e f i n d s t h a t objects or artifacts contrived with astead a r a n g e of lightnesses f r o m w h i t e t h e n a t u r e of t h e i m a g e is n o t deter- m o n t a g e of d a r k a n d light pieces of p a -t o black, m u c h a s t h e y w o u l d a p p e a r m i n e d by t h e flux of r a d i a n t e n e r g y p e r . T h e p a r a d o x i m m e d i a t e l y arisesSTILL LIFE was used to make the four black-and-white images pre- were. The black-and-white images were made with film-filter com-sented below. The reproduction of the still life above was made by binations that closely duplicate the separate wavelength sensitivitiesconventional processes of color photography and photoengraving to of the four systems of photoreceptors in the retina of the eye: theshow the reader what the colors of the original objects in the scene three systems of cone cells and the hypersensitive system of rod cells.BLACK-AND-WHITE IMAGES OF STILL LIFE were taken with ture at the top right shows the same scene as it would be viewed byfour different film-filter combinations, creating what the author calls the middle-wave cone pigment The picture at the bottom left is theretinex records. The picture at the top left was taken with a film scene as it would be viewed by the short-wave cone pigment. The pic-whose spectra] response was altered so that it matched the spectral ture at bottom right corresponds to the image seen by the rods. Un-sensitivity of the long-wave cone pigments in the eye. This photo- like cone images, which cannot be viewed independently, images pro-graph enables the observer to see a colorless image that approximates duced by the rod pigment can be studied in isolation at very low lightthe image produced by the long-wave cones by themselves. The pic- levels, without interference from much less sensitive cone systems. 3
  • that each of the objects, the pieces of in any part of the collage and associated the observations about the stability ofpaper for example, whether dark or with a new arbitrary surround. When a lightness values can readily be repro-light or in between, maintains its light- small area is totally surrounded by a duced with a montage of white, blackness without significant change as it is large area, the lightness of the small area and gray papers viewed at ordinary lightmoved around the room into regions of will change somewhat depending on levels. If. for example, a square ofhigher or lower flux. Light papers will whether the large area is darker or light- matte-surface black paper or, betterbe seen as being light and dark papers er than the small one. In general, howev- still, black velvet is placed at one side ofsimultaneously as being dark, even with er. the impressive fact is that the light- such a montage and a square of whitethe same flux coming from each of them ness of a given area is not appreciably paper is placed at the opposite side sev-to the eye. Strong gradients of flux modified by the immediately surround- eral feet away, with an assortment ofacross the field will be apparent only ing areas, nor is it modified by the still light and dark papers scattered in be-weakly, if at all. larger areas surrounding them. tween. one can place a strong light Furthermore, in an intricate collage source close enough to the black squareof areas of various lightnesses sizes and Although I have been describing a col- so that it sends more radiant energy toshapes, the lightness of a given element orless world as it is seen by the hy- the eye than the white square, remotedoes not change visibly as it is relocated persensitive receptors of rod vision, all from the light: yet the black square will continue to look black and the white square white. In fact, with the montage still strongly illuminated from one side either the black square or the white one can be moved to any other part of the. montage without a significant change in its appearance. This remarkable ability of the eye to discover lightness values independent of flux, so convincingly demonstrated when only a single photoreceptor sys- tem is operating, is the rock on which a satisfactory description of color vision can be built. The first response of the visual system is for the receptors to ab- sorb the light falling on the retina. Whereas the initial signal produced in the outer segment of the receptor cell is apparently proportional to the light flux absorbed by the visual pigment, the final comprehensive response of the visual WAVELENGTH (NANOMETERS) system is "lightness. " which shows little or no relation to the light flux absorbedNORMALIZED SPECTRAL SENSITIVITIES OF FOUR VISUAL PIGMENTS (solid by the visual pigment.lines) span the visual spectrum in overlapping curves. Curve that peaks at about 500 nanome- The processing of fluxes to generateters corresponds to sensitivity of rod pigment. Other three curves represent cone pigments. Bro- lightnesses could occur in the retina, orken lines show sensitivities of the film-filter combinations that were selected to match the sen-sitivities of the four retinal pigments and used to make the black-and-white retinex records in in the cerebral cortex, or partially inthe illustration at bottom of preceding page. Cone curves arc adapted from work of Paul Brown both. Since we are uncertain of the loca-and George Wald of Harvard University. The rod curve is standard scotopic luminosity curve. tion of the mechanisms that mediate these processes. I have coined the term retinex (a combination of retina and cor- tex) to describe the ensemble of biologi- cal mechanisms that convert flux into a pattern of lightnesses. 1 shall therefore use the term throughout this article in referring to these biological mecha- nisms. I shall also reserve the term light- ness to mean the sensation produced by a biological system. Although the rods can be stimulated at light intensities be- low the cone threshold, the cones cannot be stimulated without exciting the rods. For cones we must study the lightness images produced by each individual set of receptors using retinex photography. as I shall explain below, or learn the properties of lightness images from model calculations based on spectrora¬ diometric measurements. WAVELENGTH (NANOMETERS)THRESHOLD RESPONSES OF RETINAL RECEPTORS vary by large factors. The hyper- N ow that we know thai at low light levels an isolated receptor system generates an image in terms of lightnesssensitive rod system provides vision at radiance levels about 1000 times weaker than the light that is completely free of color, might itlevels needed to activate (he cone systems. It has been shown in authors laboratory that multi- be possible to bring one of the cone sys-colored scenes exhibit nearly their normal range of colors when they are viewed at light levels tems into operation along with the hy-so adjusted that only rod system and one cone system, the long-wave system, are responding. persensitive system, so that only the
  • "COLOR MONDRIAN" E X P E R I M E N T employs two identical dis- from that area. The photometer reading is projected onto the scaleplays of sheets of colored paper mounted on boards four and a half above the two displays. In a typical experiment the illuminators canfeet square. The colored papers have a matte finish to minimize spec- be adjusted so that the white area in the Mondrian at the left and theular reflection. Each "Mondrian" is illuminated with its own set of green area (or some other area) in the Mondrian at the right are boththree projector illuminators equipped with band-pass filters and in- sending the same triplet of radiant energies to the eye. The actual ra-dependent brightness controls so that the long-wave ("red"), middle- diant-energy fluxes cannot be re-created here because of the limita-wave ("green") and short-wave ("blue") illumination can be mixed tions of color reproduction. Under actual viewing conditions whitein any desired ratio. A telescopic photometer can be pointed at any area continues to look white and green area continues to look greenarea to measure the flux, one wave band at a time, coming to the eye even though the eye is receiving the same flux triplet from both areas.LONG WAVE 5. 8 LONG WAVE 5. 8 LONG WAVE 5. 8MIDDLE WAVE 3. 2 MIDDLE WAVE 3. 2 MIDDLE WAVE 3. 2SHORT WAVE 1. 6 SHORT WAVE 1. 6 SHORT WAVE 1. 6ENERGY AT EYE ENERGY AT EYE ENERGY AT EYE(MILLIWATTS PER (MILLIWATTS PER (MILLIWATTS PERSTERADIAN PER SQUARE METER) STERADIAN PER SQUARE METER) STERADIAN PER SQUARE METER)IDENTICAL ENERGY FLUXES AT T H E EYE provide different and an area that looks green continues to look green (right), evencolor sensations in the Mondrian experiments. In this example, with though all three arc sending to the eye the same triplet of long-, mid-the illuminants from the long-wave, middle-wave and short-wave il- dle- and short-wave energies. The same triplet can be made to comeluminators adjusted as indicated, an area that looks red continues to from any other area: if the area is white, it remains white; if the arealook red (left), an area that looks blue continues to look blue (middle) is gray, it remains gray; if it is yellow, it remains yellow, and so on. 5
  • completely colorless system and one color display with a narrow wave band receptor systems, namely the rods andother were functioning? This two-recep- of light at 550 nanometers. The light the long-wave cones, were receivingtor experiment has been carried out and level was raised just above the amount enough light to function.provides a powerful confirmation of the needed to make the display visible to the The resulting image exhibited a re-ideas derived from all our binary work dark-adapted eye. thus ensuring that markable range of color, enabling anwith red-and-white images and subse- only the hypersensitive system was op- observer to assign to each area in thequent ternary studies with multicolored erating. They then added a second nar- display the same color name it woulddisplays seen under various illuminants. row-band illuminant at 656 nanometers. have if it were illuminated above theThe experiment, rapidly becoming a with its level adjusted so that it was just cone threshold. The result is reminiscentclassic, was devised by my colleagues sufficient to activate the long-wave re- of the multicolored images produced byJohn J. McCann and Jeanne L. Benton. ceptor system but not the middle-wave the red-and-white system. The demon- McCann and Benton illuminated a system. Under these conditions only two stration explicitly confirms our early proposition that the lightness informa- tion collected at two wave bands by sep- arate receptor systems is not averaged. EXPERIMENTAL ILLUMINANTS STANDARD "WHITE" point by point and area by area, but is (NANOMETERS) ILLUMINANT (NANOMETERS) kept distinct and is compared. We know that the rod system does not produce a colored image when the image is seen by itself, and we know that the long-wave light alone cannot produce an image with a variety of colors. The combina- tion. however, gives rise to a wide va- riety of colors, namely reds, yellows, browns, blue-greens, grays and blacks. What, then, accounts for the color? The emergence of variegated colors can be ascribed to a process operating some- where along the visual pathway that compares the lightnesses of the separate images on two wave bands, provided by the two independent retinex systems. The two-receptor experiment makes it plausible that when three independent images constituting the lightnesses of the short-, middle- and long-wave sets of receptors are associated to give a full- colored image, it is the comparison of the respective lightnesses, region by re- gion. that determines the color of each region. The reason the color at any point in an image is essentially independent of the ratio of the three fluxes on three wave bands is that color depends only on the lightness in each wave band and lightness is independent of flux. As we have seen, the spectral sensitivi- ties of the visual pigments overlap broadly. If we illuminated a scene with the entire range of wavelengths to which a single visual pigment is sensitive, we LEFT EYE RIGHT EYE would see a large variety of colors be- cause more than one retinex system would respond. With the help of filters and appropriate film emulsions, how- ever. we can isolate the lightnesses that would ordinarily be incorporated into the sensation of color. We call black-COLOR-MATCHING EXPERIMENT uses a simplified Mondrian of 17 color areas (left) and-white photographs made for thisand a standard color reference, The Munsell Book of Color, which contains 1150 color "chips" purpose retinex records.(right). The Mondrian is illuminated with three narrow-band light sources: one at 630 nano-meters (long-wave- light), one at S30 nanometers (middle-wave light) and one at 450 nanome- The photographic technique, makingters (short-wave light). The ratio of the three illuminants can be adjusted so that the triplet of use of silver emulsions, performs twoenergies reflected to the eye from any chosen area will exactly equal the triplet that previously functions. First, the system providesreached the eye from some other area. In this experiment five areas, gray, red, yellow, blue and spectral sensitivities that are the same asgreen, were selected in sequence to send the same triplet of energies to the eye. In each of the those of the visual pigments. Second, itfive consecutive parts of this experiment the observer selected from the Munsell book the chipsthat came closest to matching the 17 areas of the Mondrian. The Munsell book was illuminated generates black-and-white pictures for athroughout the experiment with a constant spectral mixture of three narrow-band lights ad- human observer to examine. It is the hu-justed at the outset so that the white Munsell chip appeared the "best white. " The experiment man visual system that converts thewas set up so that the observers used one eye for viewing the Mondrian and the other eye for photographic pattern deposited in sil-viewing chips. Gray paper with an opening was used to provide chips with a constant surround. ver into lightness. Ideally we should like
  • SQUARE METER) (MILLIWATTS PER STERADIAN PER ILLUMINANT (MILLIWATTS PER SQUARE METER) ENERGY AT EYE STERADIAN PER WAVELENGTHS WAVELENGTHS WAVELENGTHS WAVELENGTHS WAVELENGTHS (NANOMETERS) (NANOMETERS) (NANOMETERS) (NANOMETERS) (NANOMETERS) PROPORTIONS OF NARROW-BAND ILLUMINANTS used to let of energies: 5. 8 flux units of long-wave light, 3. 2 flux units of mid- light the simplified Mondrian in the Munsell-chip matching experi- dle-wave light and 1. 6 flux units of short-wave light. The illustration ments were adjusted as is shown by the bars at the lop of this illustra- below shows the Munsell chips that were selected in the constant illu¬ tion so that five different areas of the Mondrian (indicated by arrows) minant to match the five Mondrian areas (gray, red, yellow, blue and sent to the observers eye in successive matching trials the same trip- green) that had sent to the eye exactly the same triplet of energies.STANDARD "WHITE (MILLIWATTS PER SQUARE METER) STERADIAN PER ILLUMINANTMATCHINGMUNSELL CHIP GRAY RED YELLOW BLUE GREEN(MILLIWATTS PERSQUARE METER) ENERGY AT EYE STERADIAN PER WAVELENGTHS WAVELENGTHS WAVELENGTHS WAVELENGTHS WAVELENGTHS (NANOMETERS) (NANOMETERS) (NANOMETERS) (NANOMETERS) (NANOMETERS) MUNSELL CHIPS SELECTED BY OBSERVERS to match the five was sent to the eye by the selected Munsell chips is shown by the bars Mondrian areas that had sent identical triplets of energy to the eye at the bottom of the illustration. It is evident that the match between are reproduced. The Munsell book was illuminated with a constant the Mondrian areas and the Munsell chips is not made on the basis of spectral mixture of narrow-band illuminants (bars at top) and the the flux of radiant energy at the eye of the observer. What does cause chips were viewed within a constant gray surround. The energy that the two areas to match is described in the illustrations that follow.
  • EXPERIMENTAL ILLUMINANTS STANDARD "WHITE" ILLUMINANT LEFT EYE RIGHT EYE MONDRIAN MUNSELL BOOK OF COLOR ILLUMINANT ENERGY AT EYE ILLUMINANT ENERGY AT EYE MONDRIAN (MILLIWATTS PER (MILLIWATTS PER MUNSELL (MILLIWATTS PER (MILLIWATTS PER AREA STERADIAN PER STERADIAN PER CHIP STERADIAN PER STERADIAN PER (REFLECTANCE) SQUARE METER) SQUARE METER) (REFLECTANCE) SQUARE METER) SQUARE METER)GRAY MATCHESRED MATCHESYELLOW MATCHESBLUE MATCHES GREEN MATCHES WAVELENGTHS WAVELENGTHS WAVELENGTHS WAVELENGTHS WAVELENGTHS WAVELENGTHS (NANOMETERS) (NANOMETERS) (NANOMETERS) (NANOMETERS) (NANOMETERS) (NANOMETERS)F U R T H E R ANALYSIS OF MATCHING E X P E R I M E N T begins the energy triplet that reaches the eye (third column). The three col-to identify the basis on which the visual system makes the color match umns at the right contain comparable data for the Munsell chips se-between the Mondrian area and the Munsell chip without regard to lected as a match for the Mondrian areas. Whereas illustration at bot-the flux each member of the pair sends to the eye. The efficiency tom of the preceding page shows that the eye does not match colorswith which a given area in the Mondrian reflects light in each of the using a "meter" that measures triplets of energies at the eye, this il-three wave bands {first column) multiplied by the amount of energy lustration shows that when a match is made, it is the reflectances ofstriking that area in each of the wave bands (second column) yields two areas that correspond, as is shown in first and fourth columns.8
  • our observer to examine the black-and- jectors equipped with sharply cutting so that both Mondrians can be viewedwhite pattern with only one set of cones. band-pass filters (not retinex filters): one simultaneously. The white area on thereporting the lightnesses appropriate to at 670 nanometers embracing a band of left continues to look white and thethat set. At any point in the black-and- long waves, one at 540 nanometers em- green area on the right continues to lookwhite pattern, however, the reflectance bracing a band of middle waves and one green, yet both are sending to the eye theis essentially the same throughout the at 450 nanometers embracing a band of same triplet of energies: 65. 30 and fivevisible spectrum. Therefore with a short waves. The amount of light from in the chosen units.black-and-white photograph we stimu- each illuminating projector is controlled We turn off the illuminants for bothlate all the receptors with the same in- by a separate variable transformer. In Mondrians and select some other areaformation. that is. with the energies that addition the illuminating projectors in the left Mondrian and sequentiallywould be absorbed by a single visual have synchronized solenoid-activated adjust the energies reaching the eyepigment. If we assume that all the reti¬ shutters to control the duration of illu- from it so that they are the same as thenex systems process information in an mination. There is a telescopic photom- energies that originally gave rise to theidentical manner, we can propose that eter that can be precisely aimed at any sensation of white and also gave rise tosending this identical information to region of either Mondrian to measure the sensation of green in the rightseveral sets of receptors is the same as the amount of radiation reflected from Mondrian. When we turn on all threesending it to only one receptor, thereby any point and therefore the amount projectors illuminating the left Mondri-enabling us to see what the image would of flux reaching the eye. The output of an, we see that this time the selected arealook like if it were possible to isolate it. the photometer is projected on a scale is yellow. The triplet of energies reach- above the Mondrian. where it can be ing our eye is the same one that had On page 3 the reader will see three seen by those taking part in the demon-black-and-white pictures taken through previously produced the sensations of stration. white and green. Again, if we wish, theretinex filters that simulate the responseof the three cone pigments. The straw- yellow and green can be viewed simul- The demonstration begins with theberries and radishes, for example, are taneously. with yellow on the left and three illuminating projectors turned onlight on the long-wave record, darker on green on the right. the Mondrian on the left; the Mondrianthe middle-wave record and darkest on on the right remains dark. The variable We can continue the demonstrationthe short-wave record. Although the or- transformers are set so that the entire with other areas such as blue, gray, redange and lemon are about as dark as the array of papers in the left Mondrian are and so on. It is dramatically demonstrat-strawberries and radishes on the short- deeply colored and at the same time the ed that the sensation of color is not relat-wave record, they are nearly as light on whites are good whites. This setting is ed to the product of reflectance timesthe middle-wave record as they are on not critical. Then, using one projector at illumination, namely energy, althoughthe long-wave record. On the printed a time and hence only one wave band at that product appears to be the only in-page the distinctions are subtle. To the a time, we measure with the telescopic formation reaching the eye from theeye viewing an actual full-color scene photometer the energy reaching the eye various areas in the Mondrians.the subtle distinctions provide all the in- from some particular area, say a white In order to demonstrate that the colorformation needed to distinguish count- rectangle. The readings from the white sensations in these experiments do notless shades and tints of every color. area (in milliwatts per steradian per involve extensive chromatic adaptation square meter) are 65 units of long-wave of retinal pigments the projectors are After the three lightnesses of an area light. 30 units of middle-wave light andhave been determined by the three reti- equipped with synchronized shutters so five units of short-wave light. We have that the Mondrians can be viewed in anex systems no further information is now established the three energies asso-necessary to characterize the color of brief flash, a tenth of a second or less in ciated with that sensation of white. duration. Regardless of the brevity ofany object in the field of view. Any spe-cific color is a report on a trio of three We turn off the three projectors illu- observation the results of the demon-specific lightnesses. For each trio of minating the color Mondrian on the left. strations are not altered. Thus one canlightnesses there is a specific and unique On the right we turn on only the long- say that neither chromatic adaptationcolor. wave projector. We select a different nor eye motion is involved in producing area of unknown color and adjust the the observed colors. Finally, the very essence of the design of the color Mon-T he limitations of color photography make it impossible to show the read-er the demonstrations readily accom- long-wave light until the long-wave en- ergy coming to the eye from the selected area is the same as the long-wave energy drian is to obviate the significance of the shape and size of surrounding areas, of the familiarity of objects and of theplished in our laboratory, which dra- that a moment ago came from the white paper in the Mondrian on the left, 65 memory of color. Curiously, from timematically reveal the independence of units. We turn off the long-wave projec- to time there is a casual attempt to ad-perceived color from the flux reaching tor and separately adjust the transform- duce what is called color constancy asthe eye. What the reader would see ers controlling the middle- and short- an explanation of these demonstrations.would be two boards four and a half feet wave projectors, one after the other, so Clearly color constancy is only a com-square identically covered with about that the energies sent to the eye from the pact designation of the remarkable com-100 pieces of paper of various colors selected area are also the same as those petence that is the subject of this article.and shapes. In order to minimize therole of specular reflectance the papers that came from the white area on thehave matte surfaces and. except for left. We have not yet turned on all threeblack, have a minimum reflectance of atleast 10 percent for any part of the visi- light sources simultaneously, but we know that when we do so, the triplet of energies reaching the eye from the se- T he mystery is how we can all agree with precision on the colors we see when there is no obvious physical quan-ble spectrum. In these displays, which tity at a point that will enable us to speci-we call "color Mondrians" (after the lected area of still unknown color will be identical with the triplet that had previ- fy the color of an object. Indeed, one canDutch painter to whose work they bear a say the stimulus for the color of a pointcertain resemblance), the papers are ar- ously produced the sensation white. in an area is not the radiation from thatranged so that each one is surrounded by point. The task of psychophysics is toat least five or six others of different col- When we turn on the three illumi¬ nants, we discover that the area in the find the nature of the stimulus for thators [see top illustration on page 5 ]. color. Mondrian on the right is green. We now Each of the identical Mondrians is il- illuminate the Mondrian on the left with Here let us remember that what theluminated by its own set of three pro- its illuminants at their original settings eye does unfailingly well is to discover 9
  • lightness values independent of flux. We will occupy a different rank order from It is evident that the lightnesses exhib-saw this to be true for a single receptor lightest to darkest. Under the short- ited by a given piece of colored pa-system, the rod system, operating alone wave-band illuminant there will be yet a per under illuminants of three differentand for the three cone systems operating third rank order. Specifically, a red pa- wave bands is related to the amount ofcollectively when they viewed an array per will be seen as being light in the long- energy the paper reflects to the eye atof white, gray and black papers. Let us wave light, darker in middle-wave light different wavelengths. Let us now exam-now illuminate the colored Mondrian and very dark in short-wave light. A ine. by means of a particular experi-array with light from just one of the blue paper, on the other hand, will be ment. how such reflectances can be re-three projectors, say the projector sup- light in short-wave light and very dark in lated step by step to perceived lightness-plying long-wave light, and observe the both middle- and long-wave light. Pa- es and how. in the process, the radianteffect of increasing and decreasing the pers of other colors will exhibit different flux that reaches the eye — the ultimateflux by a large factor. We observe that triplets of lightnesses. When we conduct- source of knowledge about lightness — fi-the various areas maintain a constant ed such experiments nearly 20 years nally becomes irrelevant to the sensa-rank order of lightness. If, however, we ago. we were led inevitably to the con- tion of color.switch the illumination to a different clusion that the triplets of lightnesses.wave band, say the middle wave band. area by area, provided the set of con-the lightnesses of many of the areas willchange: many of the 100 or so areas stancies we needed to serve as the stimu- li for color, independent of flux. I n our laboratory McCann, Suzanne P. McKee and Thomas H. Taylor made a systematic study of observers re- REFLECTANCES REFLECTANCES INTEGRATED INTEGRATED INTEGRATED RADIANCES (PERCENT) RATIO OF EXPERIMENTAL ILLUMINANTS SCALED RETINEX RETINEX FILTERS RECORDSBLUE 27. 3AREA IN 27. 3 5. 8MONDRIAN 100 LONG WAVE 35. 9 35. 9 6. 5 100WHITEPAPER MIDDLE WAVE 60. 7 60. 7 8. 1 100 STANDARD "WHITE" ILLUMINANT SHORT WAVEBLUEMUNSELL 34. 6 34. 6 6. 4CHIP 100 LONG WAVE 38. 5 38. 5 6. 7 100WHITEPAPER MIDDLE WAVE 57. 1 57. 1 7. 9 100 SHORT WAVE WAVELENGTH (NANOMETERS)ROLE OF REFLECTANCE and its psychophysical correlate, light- integrated radiances yields the integrated reflectance of the Mondri-ness, in guiding the eye to match Munsell chips with Mondrian areas an area, expressed here in percent For the matching Munsell chip awas examined with the help of retinex filter-photomultiplier combi- set of ratios was similarly determined (bottom). The final step in de-nations that match the spectral sensitivity of the cone pigments. Un- riving a physical equivalent of lightness is the scaling, or spacing, ofder each combination of illuminants (top) the integrated radiance, or integrated reflectances to be consistent with the spacing of lightnessflux, in each retinex wave band of a Mondrian area was compared sensations. This transformation is explained in the illustration on thewith the integrated radiance of a sheet of white paper. The ratio of opposite page. The scaled values appear in the column at the right10
  • sponses to a simplified color Mondrianwith areas of 17 different colors. Theyasked the observers to match the 17 ar-eas one at a time under different illumi¬nants with colored squares of paper thathad been selected from a standard col-or-reference book. The Munsell Book of Color and that were viewed under a constant "white" illumination. The illuminants on the Mondrianwere adjusted in five separate matching REFLECTANCE (PERCENT)experiments so that five different areas(gray, red. yellow, blue and green) sentto the eye an identical triplet of radian-ces. The observer began by selecting amatching Munsell "chip" for each of the 17 areas in the Mondrian when the grayarea in the Mondrian sent a particulartriplet of energies to the eye. Another setof 17 matching Munsell chips was se-lected when the same triplet was latersent to the eye by a red area in theMondrian. and the same was done foryellow, blue and green areas under illu-minants that supplied the same triplet ofenergies. The illustrations on page 7 showthe details of the experiment and the fivedifferent Munsell colors the observersselected to match the five areas wheneach area sent to the eye precisely thesame triplet of energies. In spite of theconstancy of the energy reaching oneeye from the Mondrian. each observer.using the other eye. selected Munsellchips that were gray, red, yellow, blue LIGHTNESS SENSATIONand green. SENSATION OF LIGHTNESS is plotted on an equal-interval scale. Observers are shown a The constant illumination used in sheet of white paper (9) and a sheet of black paper (1) and are then asked to choose a sheet ofviewing the Munsell book was a triplet paper whose shade of gray lies halfway between the two. The selection is the gray labeled 5.of illuminants at three wavelengths that Similar selections are made to determine the locations of midpoints between 1 and 5 and be-observers judged to produce the "best" tween 5 and 9 and so on until the equal-interval scale is filled. The end values 0 and 10 arewhite. The actual triplet of wavelengths extrapolations. The curve is then plotted by measuring the reflectances of the various papers selected by the observers. The curve makes it possible to convert values of integrated reflec-reaching the eye from the whitest paper tance into values of scaled integrated reflectance, as is given in illustration on opposite page.in the Munsell book was 11. 5 units oflong-wave light. 7. 8 units of middle-wave light and 3. 3 units of short-wavelight. The illuminants supplied energy in the eye from the Mondrian and its sur- wave bands, corresponding to the spec-narrow bands with peaks at 630 nano- round remains the same regardless of tral sensitivities of the cone pigments?meters. 530 nanometers and 450 nano- the spectral composition of the light Can such a precise physical correlate ofmeters. A similar triplet of narrow-band needed to establish a constant triplet lightness be demonstrated?illuminants were mixed in various pro- from area to area. We have done this McCann, McKee and Taylor nextportions to illuminate the Mondrian. in One case by surrounding the entire measured the radiance, or energy at the At this point the reader might ask: Mondrian with brightly colored papers eye. of the various Mondrian areas andWould not a single gray area exhibit a selected in such a way that they exactly of the matching Munsell chips by usingpronounced change in color if the sur- offset the average mixture of wave a photomultiplier in conjunction with arounding papers had reflected light of bands from the Mondrian itself and. version of the retinex filters. Since thewidely differing spectral composition? more dramatically, by cutting the 17 ar- retinex-photomultiplier combination in-Could these changes in color account eas of the Mondrian apart and placing tegrates the flux of radiant energy over afor the results of the Mondrian experi- them well separated on the backgrounds broad band of wavelengths, the instru-ments? The answer to the questions is of offsetting color. Neither arrangement ment provides a value we call integratedthat no manipulation of surrounding pa- has any significant effect on the Munsell radiance. McCann and his colleaguespers in the Mondrian is capable of mak- chips chosen to match the various areas then obtained the integrated radiancesing the gray paper match the red. yel- of the Mondrian. from a large sheet of white paper placedlow. blue and green Munsell papers se- under each of the experimental illumi-lected by the observers in the Mondrian nants that had been used to light theexperiment. McCann, John A. Hall and I have ex- L et us return, then, to the search for the stimulus that guides us so accurate- ly to the correct identification of colors. Mondrian in the chip-matching experi- ments. If the integrated radiance from a Mondrian area is used as the numeratoramined the matter further by repeating If it is not a flux of radiant energy at the in a fraction and the integrated radiancethe Mondrian-Munsell experiment in eye from each point in the field of view. from the white paper is used as the de-various ways so that the average spec- what are the physical correlates of the nominator. one obtains a value for in-tral composition of the light reaching lightnesses of objects on three separate 11
  • tegrated reflectance, which can be ex- grated reflectances of 5. 8. 6. 5 and 8. 1. It is one thing to measure a triplet pressed as a percent. whereas the corresponding values for of lightness equivalents using a retinex The integrated reflectances for the the matching Munsell chip are 6. 4. 6. 7 filter coupled to a photomultiplier: it is various Munsell chips are determined in and 7. 9. If we study the five areas that quite another for the eye to determine the same manner under the constant successively sent identical triplets of en- lightnesses in the unevenly lighted world "white" illumination. This amounts to ergies to the eye and compare their without reference sheets of white paper. measuring the percentage of reflectance scaled integrated reflectances with those I described above the ability of an iso- using detectors with the same spectral of their matching Munsell chips, we find lated receptor system — the hypersensi- sensitivity as the visual pigments. The that all the values are in excellent agree- tive system of rod vision — to classify ob- results show that the Munsell chip cho- ment. In other words, in the triplets of jects correctly according to their inher- sen by the eye to match a given Mondri¬ integrated reflectances we have identi- ent reflectivity regardless of whether the an area will have approximately the fied a highly accurate physical correlate objects happened to be in a brightly or same three integrated reflectances as the of color sensations. The data fall along a dimly lighted region of visual space. area. For example, the blue area in the the 45-degree line that describes the lo- The ability of one receptor system to Mondrian has a triplet of integrated re- cus of perfect correlation [see illustration work in this way makes it plausible that flectances (long-, middle- and short- below]. the other three systems of normal day- wave) of 27. 3. 35. 9 and 60. 7 percent. time vision possess the same ability, each We have sought a physical correlate system viewing the world through a The comparable values for the matched for lightness, and we have found that the Munsell chip are 34. 6. 38. 5 and 57. 1 broad but restricted region of the spec- scaled integrated reflectances of the five trum. the regions we duplicate with reti- percent [see illustration on page 10]. areas that sent identical triplets of fluxes nex filters. Each system forms a sepa- Finally, the integrated reflectances to our eyes are the same as those of rate lightness image of the world. The are "scaled" so that their equal spacing the matching Munsell chip. This correla- images are not mixed but compared. is consistent with the equal spacing of tion enables us to use scaled integrated The comparison of lightnesses at each lightness sensations. The curve for this reflectances as a measured lightness area gives rise to the range of sensations transformation is shown in the illustra- equivalent. The problem now shifts to we know as color. tion on the preceding page. Using this one of how the eye derives the lightness curve, we see that the blue area in the that corresponds to the reflectances of How could the biological system gen- Mondrian has a triplet of scaled inte- objects in each wave band. erate a hierarchy and spacing of light- ness values given only the flux from each point in a scene and knowing noth- ing about the pattern of illumination and nothing about the reflectances of objects? The scheme I am about to de-SCALED INTEGRATED REFLECTANCE OF MATCHING MUNSELL CHIPS scribe is the most general we have found that surmounts these limitations: its physiological embodiment could take many forms. Let me begin by pointing out the sig- nificance of edges in defining objects or areas in a scene. If a sheet of white paper is lighted strongly from one side, we see no discontinuity in color from one side to the other. Let us now imagine two light detectors positioned to measure the luminance from two different places on the paper. If the illumination is non- uniform. the luminances of the two places will of course be different. As the two detectors are moved closer together the luminances approach the same val- ue and the ratio of the two outputs ap- proaches unity. If, however, the two de- tectors bridge the boundary between two areas that differ abruptly in reflec- tance. such as would be the case with even a pale gray square on the white paper, the ratio of the outputs of the two detectors will approach the ratio of the two reflectances. Thus the single proce- dure of taking the ratio between two ad- jacent points can both detect an edge and eliminate the effect of nonuniform illumination. If we process the entire SCALED INTEGRATED REFLECTANCE OF AREAS IN MONDRIAN image in terms of the ratios of lumi- nances at closely adjacent points, we can AGREEMENT IN SCALED INTEGRATED REFLECTANCES between Mondrian areas generate dimensionless numbers that and Munsell chips chosen to match them is summarized for all three wave-band systems. The are independent of the illumination. scaled integrated reflectances of five Mondrian areas and matching Munsell chips were deter- These numbers give the ratio of reflec- mined as is described in illustration on page 10. In this graph triplets of scaled integrated re- tances at the edge between adjacent ar- flectances of five Mondrian areas that sent identical fluxes to the eye are plotted against scaled integrated reflectances of Munsell chips chosen to match them. Although the dots collectively eas: the reflectances themselves arc not represent correspondence for all three retinex wave bands, any particular dot denotes the de- yet ascertained. gree of correspondence on one retinex wave band between a Mondrian area and a Munsell chip. Close correspondences show that scaled integrated reflectance is physical correlate of the In order to determine reflectances we sensation "lightness, " showing precision with which a triplet of lightnesses determines color. need to relate all these ratios of reflec- 12
  • RATIO AFTER LIGHT THRESHOLD SEQUENTIAL FLUX TEST PRODUCT EVEN ILLUMINATION (RELATIVE UNITS) THE EYES METHOD OF DISCOVERING LIGHTNESS in com- the models response for that point and the signal sent along to be plex images remains to be established. An efficient and physiological- multiplied by the next ratio. When the path crosses an edge between ly plausible scheme is depicted in this illustration and the one below. two lightnesses, there is a sharp change in the threshold-tested ratio The numbers inside the schematic Mondrian represent the long-wave and hence a similar change in the sequential product Here the path integrated radiances coming from each area of a display that is even- is started in the white area, where the flux of radiant energy is 100. ly lighted. The long-wave retinex system independently "measures" By the time the path reaches the brown area at the lower right the the long-wave integrated radiance, point by point, as if it were doing product is. 18. The retinex system has thus determined that the brown so along an arbitrary pathway {color). The flux at each successive area reflects 18 percent as much long-wave energy as the white area. closely spaced pair of points is converted into a ratio. This ratio is sub- Any other path ending in the brown area would yield the same result jected to a threshold test: any ratio to he regarded as a change must as long as it had been through the white area. By averaging the re- vary from unity by more than some email threshold amount (plus or sponses for each area, as computed by many arbitrary paths, the long- minus. 003 in the computer program). If the ratio docs not vary from wave retinex system arrives at a single reflectance value for each unity by this amount, it is regarded as being "unchanged" and is set to area, which designates perceived lightness. Middle- and short-wave equal unity. A second threshold-tested ratio along the same pathway retinex systems compute their own sets of lightness values. Compari- is multiplied by the first ratio to give a sequential product that is both son of triplet of lightnesses for each area provides sensation of color. RATIO AFTER LIGHT THRESHOLD SEQUENTIAL FLUX TEST PRODUCTGRADIENT ILLUMINATION (RELATIVE UNITS) MORE REALISTIC CASE OF G R A D E D ILLUMINATION is han- of the white area {57). The scheme hence provides a means for arriv- dled equally well by the sequential-product method to arrive at the ing at computed reflectance independent of flux and without resort same reflectance value of. 18 for the brown area at the end of the to white cards as standards. Precise values of light flux along pathway path, even though here the long-wave retinex system receives as much in this diagram were derived from a computer program that works flux from the middle of the brown area {57) as it does from the middle with 75 values between every two values printed within Mondrian. 13
  • t a n c e s in the field of view. G i v e n theratio of luminances at the edge betweena first a r e a a n d a s e c o n d o n e , we m u l t i -ply it by t h e r a t i o of l u m i n a n c e s at t h eedge between the second area and athird. T h i s p r o d u c t o f r a t i o s a p p r o a c h e sthe r a t i o of reflectances b e t w e e n the firsta n d third a r e a s , r e g a r d l e s s of t h e dis-t r i b u t i o n o f i l l u m i n a t i o n . Similarly, w ec a n o b t a i n the r a t i o of reflectances ofa n y t w o a r e a s i n a n i m a g e , h o w e v e r re-m o t e they a r e from e a c h o t h e r , b y m u l t i -plying t h e r a t i o s of all t h e b o u n d a r i e sbetween the starling a r e a a n d t h e r e m o t ea r e a . W e can a l s o establish t h e r a t i o o ft h e reflectance o f a n y a r e a o n t h e p a t hby t a p p i n g off the s e q u e n t i a l p r o d u c tr e a c h e d at t h a t a r e a [see illustrations onpreceding page]. We a r e n o w c o m i n g close to t h e an- swer t o the q u e s t i o n : H o w c a n t h ee y e a s c e r t a i n the reflectance of an a r e aw i t h o u t in effect p l a c i n g a c o m p a r i s o ns t a n d a r d next t o t h e a r e a ? T h e s e q u e n -tial p r o d u c t can be used as a s u b s t i t u t efor the p l a c e m e n t of t w o a r e a s adjacentto e a c h o t h e r , t h u s defining a p h o t o m e t -ric o p e r a t i o n feasible for t h e e y e . T h e r e m a i n i n g task is to suggest h o wthe e y e can discover t h e a r e a of high-est reflectance in t h e field of view a n dt h e n decide w h e t h e r that a r e a is a c t u a l -ly white or s o m e o t h e r c o l o r . In the m o d -el we have proposed, sequential prod-ucts a r e c o m p u t e d a l o n g m a n y a r b i t r a r yp a t h w a y s that w a n d e r t h r o u g h the t w o -d i m e n s i o n a l a r r a y of e n e r g i e s on t h em o d e l s "retina. " Since t h e p a t h w a y sCOLOR "SOLID" shows the location of allperceivable colors, including white and black,in a three-dimensional color space construct-ed according to the authors retinex theory.The position of a color in this space is deter-mined not by the triplet of energies at a pointbut by the triplet of lightnesses computed bythe eye for each area. The color photographat the top left shows the location of represent-ative colors throughout the space. The direc-tion of increasing lightness along each axis isshown by the arrows. The three black-and-white photographs of the color solid were tak-en with retinex filter-film combinations. Theyshow the lightness values of the representativecolors as they would be perceived separatelyby the eyes long-wave (top), middle-wave(middle) and short-wave (bottom) visual pig-ments. The set of 10 color pictures at the rightrepresents horizontal planes cut through thethree-dimensional color space. Each plane isthe locus of colors possible with a constantshort-wave lightness. For example, the fifthplane from the bottom shows the variety ofcolor sensations from all possible long- andmiddle-wave lightness values when those val-ues are combined with a short-wave lightnessof 5. The colored squares are samples takenfrom The Munsell Book of Color. In generalthe blank areas on each plane represent re-gions where colors could be produced only byfluorescent dyes, if they were produced at all.
  • can begin anywhere, not just in regions tem will be seen not as white but as yel- changing the flux it is possible to esti-of the highest reflectance, the first value low. A similar intercomparison of trip- mate the corresponding change in per-in any sequence is arbitrarily assumed to lets of lightnesses at the same place ceived lightness. What one finds is thatbe 100 percent. Because of this delib- within each scene provides the sensation the estimated lightness changes onlyerately adopted fiction the sequential of color, area by area, in spite of unpre- slowly with enormous changes in flux.product becomes greater than unity dictable variations in illumination. For example, decreasing the flux by awhenever the path reaches an area If one looks at black-and-white photo- very large amount will be seen as a verywhose reflectance is higher than that of graphs taken through retinex filters, one small reduction in lightness. If the spotthe starting area. sees a dramatic difference in lightness of light is composed of a narrow band of The attainment of a sequential prod- for most objects between the photo- long wavelength, say 600 nanometers.uct greater than unity indicates that the graph representing the short-wave sys- one can expect all three cone receptorssequence should be started afresh with tem and either of the photographs repre- to absorb the radiation in some degree.the new area of high reflectance taken as senting the other two systems. And yet it but significantly more radiation will bebeing 100 percent. This procedure is the is the comparatively small differences absorbed by the long-wave cones thanheart of the technique for finding the between the long-wave and the middle- by the other two kinds. When the threehighest reflectance in the path. After the wave lightnesses that are responsible for values are read on a scale of perceivedpath reaches the highest reflectance in the experience of vivid reds and greens. lightness, the three lightnesses are 9 onthe scene, each of the sequential prod- Such reliable and sensitive respon- the long-wave system. 8. 5 on the mid-ucts computed thereafter becomes a siveness to small lightness differences dle-wave system and 7. 5 on the short-fraction of the highest value. A satisfac- provides the basis for the colors seen wave system [see illustration on this page].tory computer program has been de- under anomalous conditions far re- This combination of lightnesses is seensigned to study the number of paths. moved from those the eye has evolved as a light reddish orange, a color nottheir lengths and convolutions, the to see. Two examples of interest are the commonly perceived under ordinarythreshold values for recognizing edges color of a spot of light in a total sur- conditions unless the surfaces are fluo-and. perhaps most important, how to rounding area devoid of light and the rescent. The spectrum, a strikinglyutilize all the pathways starting in all spectrum of colors produced by a prism. anomalous display, can be regarded as aareas. series of three laterally displaced contin- One can readily measure the flux at uous gradients involving both the prop- The biological counterpart of this the eye from a spot of light in a void. Byprogram is performed in undeterminedparts of the pathway between the retinaand the cortex. The process that corre-sponds to computing sequential prod-ucts does not involve the averaging ofareas or the averaging of flux. It does.however, call for an arithmetic that ex-tends over the entire visual field. Fur-thermore. since the relevant phenomenaare seen in a brief pulse of light, all thecomputations and conclusions aboutlightness must be carried out in a frac- FLUX (RELATIVE UNITS)tion of a second without dependence oneye movement. With a single pulse, eyemovement, by definition, is not neces-sary. With continuous illumination thenormal quick motions of the eye proba-bly serve to maintain the freshness ofthe process. With our computer model we can ob-tain a triplet of lightnesses for each area in the color Mondrian that correspondsclosely to the lightnesses one would measure with a combined retinex filterand photomultiplier. The color corre-sponding to any given triplet can be vi-sualized with the aid of the color "sol-id" we have built, in which the Munsellcolors are located in three dimensionsin "lightness-color space" according totheir lightness values measured in threewave bands through retinex filters [see LIGHTNESS SENSATION illustration on page 14]. SPOT OF LIGHT IN A VOID, that is, a single spot of narrow-band light viewed in an other- In normal images the sensation of wise totally dark environment, has a color that would seem to depend solely on Ms wavelength.white light will be generated by any area The color can also be explained, however, by the retinex theory in terms of lightness as per-that is placed at the top of the lightness ceived by the eyes three receptor systems. Psychophysical measurements show that when thescale by all three retinex systems. On the eye is presented with a spot of light in a void, the perceived lightness is changed only slightly byother hand, an area that stands at the top very large changes in flux, as is indicated by the straight line. For example, if the spot is com-of only two of the three lightness scales posed of a narrow-wavelength band centered, say, at 600 nanometers, the three cone pigments will absorb the flux in quite different amounts because of the shape of their absorption curves.will be seen as some other color. Hence In arbitrary units the long-wave pigment might absorb 80 units, the middle-wave pigment 20an area that is at the top of the lightness units and the short-wave pigment a few tenths of a unit at most. If these ratios are plotted onscale in the long- and middle-wave sys- the spot-in-a-void curve, the corresponding lightness values are 9 for the long waves, 8. 5 fortems but is surpassed in lightness by the middle waves and 7. 5 for the short This combination of lightnesses is perceived as a lightsome other area in the short-wave sys- reddish orange, not ordinarily seen under normal conditions unless surfaces are fluorescent 15
  • erties of spots and the properties of ar- tion in lightness-color space. rectly. In this important meeting pointeas. From these properties it is possible One can now understand the red-and- of the blue-green shadows with the col-to predict the colors of the spectrum. white images of our early work as a pro- ored images, provided by the red-and-whereas it is not possible, as we have cedure that carries the colored shadow white display, the extended taking andseen, to attribute a specific spectral com- to a richly variegated family of colors multiplication of ratios determine theposition to the radiance from a colored no longer in shadows but in images. The lightness of each small area. Finally, allarea in everyday life. colors seen in a red-and-white projec- these principles are applied in everyday tion can be readily predicted by extend- ternary vision, which creates a distinctP erhaps the first observation pointedly relevant to the mechanism of colorformation in images is not Newtons ing the analysis followed in predicting the color of von Guerickes shadow. To demonstrate this point we reproduce on lightness image for each of the three sen- sitive systems and compares them in or- der to generate color.spectrum but the. phenomenon of col- page 17 the "red" and "green" separa-ored shadows, described in 1672 by Ottovon Guericke. "This is how it happens. "he wrote, "that in the early morning twi- tion images used in making a red-and- white multicolored projection. (In this demonstration no attempt is made to re- T he train of interlocking concepts and experiments started 25 years ago with the observation that the relative en-light a clear blue shadow can be pro- produce the colors seen in the actual ergies of the red-and-white projectorsduced upon a white piece of paper [by multicolored image. ) The red-and-white can be altered without changing theholding] a linger or other object... be- projection was photographed through names of the various colors. This obser-tween a lighted candle and the paper be- long-, middle- and short-wave retinex vation negated the simplistic explana-neath. " This important experiment, we film-filter combinations. The three im- tion in terms of contrast, fatigue andnow know, depicts an elementary exam- ages are reproduced below the pair of surround and led to the fundamentalple of generating three different light- long- and middle-wave separation im- concept of independent long- and short-nesses on the three receptor systems. A ages that were superposed to make the wave image forming systems that ulti-diagram of this experiment with long- red-and-white image. The significant malely evolved to the concept of threewave ("red") light and white light ap- point is that when the eye views the red- independent retinex systems and to thepears below. Here the color of the shad- and-white images on the screen with its Mondrian demonstration. The conceptow is blue-green. The diagram shows own retinex system, it is provided with a of the percentage of available light onthat the triplet of lightnesses in the shad- triplet of lightnesses for each part of the each wave band as a determining vari-ow corresponds to the blue-green color scene that resembles the triplet it would able and the technique of measuring itone would predict for it from its posi- obtain if it viewed the original scene di- evolved to the concept that lightnesses ADSORBED FLUX (ARBITRARY UNITS) BLUE-GREEN FROM LONG-WAVE FROM TWO SHADOW FROM "WHITE" ("RED") ILLUMINATOR ILLUMINATORS ILLUMINATOR (SCATTER) COMBINED LIGHTNESS COLOR SHORT-WAVE RECEPTOR SYSTEM 100 MIDDLE-WAVE RECEPTOR SYSTEM 100 LONG-WAVE RECEPTOR SYSTEM 100"WHITE" LONGWAVEILLUMINATOR ("RED") ILLUMINATORBLUE-GREEN COLORED SHADOW is seen when a hand or some absorbed by the long-wave system, SO by the middle-wave system andother object is placed in the beam of a projector that is sending long- five by the short-wave system. (A small amount of scattered long-wovc ("red") light to a screen while the screen is illuminated by a beam wave light also appears in the shadow. ) The third column of boxesof white light. The author regards Otto von Guerickes description shows the combined amounts of flux from both sources absorbed byin 1672 of seeing colored shadows made by candlelight as the first each receptor system. The fractions represent the ratio at edges ofobservation pointedly relevant to the mechanism of image and color the flux from within the shadow divided by the flux from outside.formation. In the analysis at the right it is assumed that one projector The fourth column shows the lightness on each receptor system. Thesends white light to the screen. The other projector, equipped with a lightness of the lightest place in the scene for each receptor systemred filter, sends only long wavelengths to the screen. Assume that the will be near the top of the lightness scale, being determined by thewhite light contributes 100 arbitrary units of flux to each of the short-, flux of radiant energy in the same way that a spot has its lightnessmiddle- and long-wave receptors. The long-wave flux is absorbed determined by flux. Triplet of lightnesses within the shadow falls inby the three receptor systems in different proportions: 100 units are the region of color space that the eye perceives as being blue-green.16
  • maintain an independent rank order on ness. What was needed was a far-reach- flectance" as the external partner tolong- and short-wave bands. This mea- ing, edge-reading arithmetic: the se- which the retinex system relates the in-suring technique in turn evolved from a quential product of ratios at edges. For ternal partner: constructed lightness.projected black-and-white image to an the color Mondrian the ratio at edges Color can be arranged in the lightnessarrangement of colored papers in the was early recognized as requiring a ratio solid with long-, middle- and short-wavecolor Mondrian. The manifest stability of the integrals of the product at each axes of lightness. All visible colors re-and constancy of the lightnesses of all wavelength of the absorbance of the side in this solid independent of flux.the papers of the Mondrian when a sin- cone pigment times the reflectance of each color having a unique position giv-gle wave band illuminates it with vary- the colored paper times the illuminants. en by the three axial values of lightness.ing intensity dramatizes the concept that Separate integrals were taken over the It should be remembered that the realityevery colored paper has three reflec- wave bands of the three cone pigments. of color lies in this solid. When the col-tances on three wave bands and that In a long series of binocular com- or Mondrian is nonuniformly illuminat-these reflectances are somehow connect- parison-and-selection observations the ed. photographed and measured, reflec-ed with the biological characteristic: quantity satisfying the integral was tance in the photograph no longer corre-lightnesses. shown to be impressively well correlat- lates with the color but the lightness ed with lightness, particularly after the docs. The three sets of ratios of integrals realization that the scale, or spacing, of at edges and the product of these inte- the reflectance integral should be madeA black-and-white Mondrian taught that nonuniformity of illumination.size and shape of area and length of to correspond with the spacing of the biological quantity lightness. This led to grals within a set emerge as the physical determinants in the partnership between the biological system and areas in theedges were basically irrelevant to light- the designation "scaled integrated re- external world. LONG-WAVE ("RED") SEPARATION RECORD MIDDLE-WAVE ("GREEN") SEPARATION RECORD LONG-WAVE RETINEX RECORD MIDDLE-WAVE RETINEX RECORD SHORT-WAVE RETINEX RECORDRETINEX RECORDS OF RED-AND-WHITE projections show screen. The retinex records are reproduced in the bottom part of thethat red-and-white images produce a triplet of lightnesses for each illustration: long-wave at the left, middle-wave in the middle andpart of the scene that are consistent with the observed color sensa- short-wave at the right. The colors seen in red-and-white projectionstions. The two photographs in the top half of this illustration are re- are those one would expect from their triplets of lightnesses. The ap-productions of the long-wave {left) and middle-wave (right) separa- ple is light on the long record and darker in the middle and short rec-tion records taken of the original still life. The long-wave record was ords. The orange is lightest on the long record, intermediate on theprojected onto a screen with a long-wave (red) filter in the beam of middle record and darkest on the short. It is impressive that with hislight. The middle-wave record was projected in superposition onto own retinex systems the observer can see a blue cup, a brown strawthe same screen in the light of a tungsten-filament lamp. Three reti- basket and pale yellow bananas with lightness differences so small asnex photographs were then taken of projected images appearing on to challenge photoengraving process used to reproduce photographs. 17
  • The Author INTERACTION OF THE LONG-WAVE CONES AND THE RODS TO PRODUCE EDWIN H. L A N D is chairman of the COLOR SENSATIONS. John J. McCannboard, director of research and chief ex- and Jeanne L. Benton in Journal of theecutive officer of the Polaroid Corpora- Optical Society of America, Vol. 59,tion. Born in 1909. he attended Harvard No. 1. pages 103-107: January. 1969.College, where he developed a new type LIGHTNESS AND RETINEX THEORY. Ed-of polarizing filter in the form of an ex- win H. Land and John J. McCann intensive synthetic sheet. In 1937 he Journal of the Optical Society of Ameri-founded Polaroid for research in the ca, Vol. 6 1 . No. 1. pages 1-11; Janu-new field of applied polarization, and in ary, 1971.1944 he began his pioneering work inthe development of "instant" photogra-phy. His one-step photographic processwas first demonstrated to the Optical So- The Coverciety of America in February. 1947, and The pattern on the cover was used inwas made available to the public at the experiments testing Edwin H. Lands re-end of 1948. Land has received 14 hon- tinex theory of color vision. Because theorary degrees, has held visiting academ- pattern bears a resemblance to theic appointments at Harvard and is cur- works of the Dutch painter Piet Mondri¬rently Institute Professor (Visiting) at an. Land refers to this display and simi-the Massachusetts Institute of Technol- lar ones as Mondrians. In more elabo-ogy. From 1960 to 1973 he was consul¬ rate examples (see top illustration on pagetant-at-large to the Presidents Science 5) perhaps 100 pieces of paper of vari-Advisory Committee, and in 1967 he re- ous colors and sizes are mounted onceived the National Medal of Science. large boards and so arranged that eachThis year, on the occasion of his 500th piece of paper is surrounded by at leastU. S. patent, he was elected to the Na- five or six other pieces of different col-tional Inventors Hall of Fame. Land has ors. In a typical demonstration thepursued his lively interest in the mecha- Mondrian is illuminated by projectorsnisms of color vision for the past 25 that provide adjustable amounts of radi-years. ant energy in three wave bands: long ("red"), middle ("green") and short ("blue"). With the proper selection of the mixture of illuminants falling on theBibliography Mondrian the radiant flux reaching the eye from any selected area can be madeCOLOR VISION AND THE NATURAL IM- to match the flux that had previously AGE: PART I. Edwin H. Land in Pro- reached the eye from a totally different ceedings of the National Academy of area. In the first instance the selected Sciences. Vol. 45, No. 1. pages 115- area could have been red: in the second 129: January, 1959. instance it could have been green. WithCOLOR VISION AND THE N A T U R A L I M - the same flux of energy reaching the eye AGE: PART II. Edwin H. Land in Pro- the two areas will still be seen as red and ceedings of the National Academy of green. (Cover photograph by Julius J. Sciences, Vol. 45, No. 4. pages 6 3 6 - Scarpetti) 644: April. 1959.
  • Frequently Asked Questions about ColorCharles Poynton This FAQ is intended to clarify aspects of color that are important to color image coding, computer graphics, image processing, video, and thewww.poynton.com transfer of digital images to print.poynton@poynton.com I assume that you are familiar with intensity, luminance (CIE Y), light- ness (CIE L*), and the nonlinear relationship between CRT voltage and intensity (gamma). To learn more about these topics, please read the companion Frequently Asked Questions about Gamma before starting this. This document is available on the Internet from Toronto at <http://www.poynton.com/PDFs/ColorFAQ.pdf> I retain copyright to this note. You have permission to use it, but you may not publish it. Table of Contents 1 What is color? 3 2 What is intensity? 3 3 What is luminance? 3 4 What is lightness? 4 5 What is hue? 4 6 What is saturation? 4 7 How is color specified? 4 8 Should I use a color specification system for image data? 5 9 What weighting of red, green and blue corresponds to brightness? 5 10 Can blue be assigned fewer bits than red or green? 6 11 What is “luma”? 6 12 What are CIE XYZ components? 7 13 Does my scanner use the CIE spectral curves? 7 14 What are CIE x and y chromaticity coordinates? 7 15 What is white? 8 16 What is color temperature? 8 17 How can I characterize red, green and blue? 9 © 1997-03-02a Charles A. Poynton. All rights reserved. 1 of 24
  • 2 Frequently Asked Questions About Colour 18 How do I transform between CIE XYZ and a particular set of RGB primaries? 9 19 Is RGB always device-dependent? 10 20 How do I transform data from one set of RGB primaries to another? 10 21 Should I use RGB or XYZ for image synthesis? 11 22 What is subtractive color? 11 23 Why did my grade three teacher tell me that the primaries are red, yellow and blue? 11 24 Is CMY just one-minus-RGB? 12 25 Why does offset printing use black ink in addition to CMY? 12 26 What are color differences? 13 27 How do I obtain color difference components from tristimulus values? 14 28 How do I encode YPBPR components? 14 29 How do I encode YCBCR components from RGB in [0, +1]? 15 30 How do I encode YCBCR components from computer RGB ? 15 31 How do I encode YCBCR components from studio video? 16 32 How do I decode RGB from PhotoYCC™? 17 33 Will you tell me how to decode YUV and YIQ? 17 34 How should I test my encoders and decoders? 17 35 What is perceptual uniformity? 18 36 What are HSB and HLS? 19 37 What is true color? 19 38 What is indexed color? 20 39 I want to visualize a scalar function of two variables. Should I use RGB values corresponding to the colors of the rainbow? 21 40 What is dithering? 21 41 How does halftoning relate to color? 21 42 What’s a color management system? 22 43 How does a CMS know about particular devices? 22 44 Is a color management system useful for color specification? 22 45 I’m not a color expert. What parameters should I use to code my images? 23 46 References 23 47 Contributors 242
  • Frequently Asked Questions About Color 31 What is color? Color is the perceptual result of light in the visible region of the spectrum, having wavelengths in the region of 400 nm to 700 nm, incident upon the retina. Physical power (or radiance) is expressed in a spectral power distri- bution (SPD), often in 31 components each representing a 10 nm band. The human retina has three types of color photoreceptor cone cells, which respond to incident radiation with somewhat different spectral response curves. A fourth type of photoreceptor cell, the rod, is also present in the retina. Rods are effective only at extremely low light levels (colloquially, night vision), and although important for vision play no role in image reproduction. Because there are exactly three types of color photoreceptor, three numer- ical components are necessary and sufficient to describe a color, providing that appropriate spectral weighting functions are used. This is the concern of the science of colorimetry. In 1931, the Commission Interna- tionale de L’Éclairage (CIE) adopted standard curves for a hypothetical Standard Observer. These curves specify how an SPD can be transformed into a set of three numbers that specifies a color. The CIE system is immediately and almost universally applicable to self- luminous sources and displays. However the colors produced by reflec- tive systems such as photography, printing or paint are a function not only of the colorants but also of the SPD of the ambient illumination. If your application has a strong dependence upon the spectrum of the illu- minant, you may have to resort to spectral matching. Sir Isaac Newton said, “Indeed rays, properly expressed, are not colored.” SPDs exist in the physical world, but colour exists only in the eye and the brain. Berlin and Kay [1] state that although different languages encode in their vocabularies different numbers of basic color categories, a total universal inventory of exactly eleven basic color categories exists from which the eleven or fewer basic color terms of any given language are always drawn. The eleven basic color categories are WHITE, BLACK, RED, GREEN, YELLOW, BLUE, BROWN, PURPLE, PINK, ORANGE, and GRAY.2 What is intensity? Intensity is a measure over some interval of the electromagnetic spectrum of the flow of power that is radiated from, or incident on, a surface. Inten- sity is what I call a linear-light measure, expressed in units such as watts per square meter. The voltages presented to a CRT monitor control the intensities of the color components, but in a nonlinear manner. CRT voltages are not proportional to intensity.3 What is luminance? Brightness is defined by the CIE as the attribute of a visual sensation according to which an area appears to emit more or less light. Because bright- ness perception is very complex, the CIE defined a more tractable quan- tity luminance which is radiant power weighted by a spectral sensitivity function that is characteristic of vision. The luminous efficiency of the Stan- dard Observer is defined numerically, is everywhere positive, and peaks
  • 4 Frequently Asked Questions About Color at about 555 nm. When an SPD is integrated using this curve as a weighting function, the result is CIE luminance, denoted Y. The magnitude of luminance is proportional to physical power. In that sense it is like intensity. But the spectral composition of luminance is related to the brightness sensitivity of human vision. Strictly speaking, luminance should be expressed in a unit such as candelas per meter squared, but in practice it is often normalized to 1 or 100 units with respect to the luminance of a specified or implied white reference. For example, a studio broadcast monitor has a white reference whose luminance is about 80 cd•m -2, and Y = 1 refers to this value.4 What is lightness? Human vision has a nonlinear perceptual response to brightness: a source having a luminance only 18% of a reference luminance appears about half as bright. The perceptual response to luminance is called Lightness. It is denoted L* and is defined by the CIE as a modified cube root of lumi- nance: 1  Y 3 Y L* = 116   − 16 ; 0.008856 <  Yn  Yn Yn is the luminance of the white reference. If you normalize luminance to reference white then you need not compute the fraction. The CIE defini- tion applies a linear segment with a slope of 903.3 near black, for (Y/Yn) ≤ 0.008856. The linear segment is unimportant for practical purposes but if you don’t use it, make sure that you limit L* at zero. L* has a range of 0 to 100, and a “delta L-star” of unity is taken to be roughly the threshold of visibility. Stated differently, lightness perception is roughly logarithmic. An observer can detect an intensity difference between two patches when their intensities differ by more than one about percent. Video systems approximate the lightness response of vision using R’G’B’ signals that are each subject to a 0.45 power function. This is comparable to the 1⁄ 3 power function defined by L*.5 What is hue? According to the CIE [2], hue is the attribute of a visual sensation according to which an area appears to be similar to one of the perceived colours, red, yellow, green and bue, or a combination of two of them. Roughly speaking, if the dominant wavelength of an SPD shifts, the hue of the associated color will shift.6 What is saturation? Again from the CIE, saturation is the colourfulness of an area judged in propor- tion to its brightness. Saturation runs from neutral gray through pastel to saturated colors. Roughly speaking, the more an SPD is concentrated at one wavelength, the more saturated will be the associated color. You can desaturate a color by adding light that contains power at all wavelengths.7 How is color specified? The CIE system defines how to map an SPD to a triple of numerical components that are the mathematical coordinates of color space. Their function is analagous to coordinates on a map. Cartographers have different map projections for different functions: some map projections preserve areas, others show latitudes and longitudes as straight lines. No24
  • Frequently Asked Questions About Color 5 single map projection fills all the needs of map users. Similarly, no single Color system fills all of the needs of color users. The systems useful today for color specification include CIE XYZ, CIE xyY, CIE L *u*v* and CIE L*a*b*. Numerical values of hue and satura- tion are not very useful for color specification, for reasons to be discussed in section 36. A color specification system needs to be able to represent any color with high precision. Since few colors are handled at a time, a specification system can be computationally complex. Any system for color specifica- tion must be intimately related to the CIE specifications. You can specify a single “spot” color using a color order system such as Munsell. Systems like Munsell come with swatch books to enable visual color matches, and have documented methods of transforming between coordinates in the system and CIE values. Systems like Munsell are not useful for image data. You can specify an ink color by specifying the proportions of standard (or secret) inks that can be mixed to make the color. That’s how PANTONE™ works. Although widespread, it’s propri- etary. No translation to CIE is publicly available.8 Should I use a color A digitized color image is represented as an array of pixels, where each specification system for pixel contains numerical components that define a color. Three compo- image data? nents are necessary and sufficient for this purpose, although in printing it is convenient to use a fourth (black) component. In theory, the three numerical values for image coding could be provided by a color specification system. But a practical image coding system needs to be computationally efficient, cannot afford unlimited precision, need not be intimately related to the CIE system and generally needs to cover only a reasonably wide range of colors and not all of the colors. So image coding uses different systems than color specification. The systems useful for image coding are linear RGB, nonlinear R’G’B’, nonlinear CMY, nonlinear CMYK, and derivatives of nonlinear R’G’B’ such as Y’CBCR. Numerical values of hue and saturation are not useful in color image coding. If you manufacture cars, you have to match the color of paint on the door with the color of paint on the fender. A color specification system will be necessary. But to convey a picture of the car, you need image coding. You can afford to do quite a bit of computation in the first case because you have only two colored elements, the door and the fender. In the second case, the color coding must be quite efficient because you may have a million colored elements or more. For a highly readable short introduction to color image coding, see DeMarsh and Giorgianni [3]. For a terse, complete technical treatment, read Schreiber [4].9 What weighting of red, Direct acquisition of luminance requires use of a very specific spectral green and blue weighting. However, luminance can also be computed as a weighted sum corresponds to of red, green and blue components. brightness?
  • 6 Frequently Asked Questions About Color If three sources appear red, green and blue, and have the same radiance in the visible spectrum, then the green will appear the brightest of the three because the luminous efficiency function peaks in the green region of the spectrum. The red will appear less bright, and the blue will be the darkest of the three. As a consequence of the luminous efficiency func- tion, all saturated blue colors are quite dark and all saturated yellows are quite light. If luminance is computed from red, green and blue, the coeffi- cients will be a function of the particular red, green and blue spectral weighting functions employed, but the green coefficient will be quite large, the red will have an intermediate value, and the blue coefficient will be the smallest of the three. Contemporary CRT phosphors are standardized in Rec. 709 [9], to be described in section 17. The weights to compute true CIE luminance from linear red, green and blue (indicated without prime symbols), for the Rec. 709, are these: Y709 = 0.2125 R + 0.7154 G + 0.0721B This computation assumes that the luminance spectral weighting can be formed as a linear combination of the scanner curves, and assumes that the component signals represent linear-light. Either or both of these conditions can be relaxed to some extent depending on the application. Some computer systems have computed brightness using (R+G+B)/3. This is at odds with the properties of human vision, as will be discussed under What are HSB and HLS? in section 36. The coefficients 0.299, 0.587 and 0.114 properly computed luminance for monitors having phosphors that were contemporary at the introduction of NTSC television in 1953. They are still appropriate for computing video luma to be discussed below in section 11. However, these coeffi- cients do not accurately compute luminance for contemporary monitors.10 Can blue be assigned Blue has a small contribution to the brightness sensation. However, fewer bits than red or human vision has extraordinarily good color discrimination capability in green? blue colors. So if you give blue fewer bits than red or green, you will introduce noticeable contouring in blue areas of your pictures.11 What is “luma”? It is useful in a video system to convey a component representative of luminance and two other components representative of color. It is impor- tant to convey the component representative of luminance in such a way that noise (or quantization) introduced in transmission, processing and storage has a perceptually similar effect across the entire tone scale from black to white. The ideal way to accomplish these goals would be to form a luminance signal by matrixing RGB, then subjecting luminance to a nonlinear transfer function similar to the L* function. There are practical reasons in video to perform these operations in the opposite order. First a nonlinear transfer function – gamma correction – is applied to each of the linear R, G and B. Then a weighted sum of the nonlinear components is computed to form a signal representative of luminance. The resulting component is related to brightness but is not CIE luminance. Many video engineers call it luma and give it the symbol Y’. It is often carelessly called luminance and given the symbol Y. You24
  • Frequently Asked Questions About Color 7 must be careful to determine whether a particular author assigns a linear or nonlinear interpretation to the term luminance and the symbol Y. The coefficients that correspond to the “NTSC” red, green and blue CRT phosphors of 1953 are standardized in ITU-R Recommendation BT. 601-4 (formerly CCIR Rec. 601). I call it Rec. 601. To compute nonlinear video luma from nonlinear red, green and blue: Y601 = 0.299 R′ + 0.587G ′ + 0.114 B′ ′ The prime symbols in this equation, and in those to follow, denote nonlinear components.12 What are CIE XYZ The CIE system is based on the description of color as a luminance components? component Y, as described above, and two additional components X and Z. The spectral weighting curves of X and Z have been standardized by the CIE based on statistics from experiments involving human observers. XYZ tristimulus values can describe any color. (RGB tristimulus values will be described later.) The magnitudes of the XYZ components are proportional to physical energy, but their spectral composition corresponds to the color matching characteristics of human vision. The CIE system is defined in Publication CIE No 15.2, Colorimetry, Second Edition (1986) [5].13 Does my scanner use the Probably not. Scanners are most often used to scan images such as color CIE spectral curves? photographs and color offset prints that are already “records” of three components of color information. The usual task of a scanner is not spec- tral analysis but extraction of the values of the three components that have already been recorded. Narrowband filters are more suited to this task than filters that adhere to the principles of colorimetry. If you place on your scanner an original colored object that has “original” SPDs that are not already a record of three components, chances are your scanner will not very report accurate RGB values. This is because most scanners do not conform very closely to CIE standards.14 What are CIE x and y It is often convenient to discuss “pure” color in the absence of brightness. chromaticity coordinates? The CIE defines a normalization process to compute “little” x and y chro- maticity coordinates: X Y x= y= X +Y + Z X +Y + Z A color plots as a point in an (x, y) chromaticity diagram. When a narrow- band SPD comprising power at just one wavelength is swept across the range 400 nm to 700 nm, it traces a shark-fin shaped spectral locus in (x, y) coordinates. The sensation of purple cannot be produced by a single wavelength: to produce purple requires a mixture of shortwave and long- wave light. The line of purples on a chromaticity diagram joins extreme blue to extreme red. All colors are contained in the area in (x, y) bounded by the line of purples and the spectral locus.
  • 8 Frequently Asked Questions About Color A color can be specified by its chromaticity and luminance, in the form of an xyY triple. To recover X and Z from chromaticities and luminance, use these relations: x 1− x − y X= Y Z= Y y y The bible of color science is Wyszecki and Styles, Color Science [6]. But it’s daunting. For Wyszecki’s own condensed version, see Color in Business, Science and Industry, Third Edition [7]. It is directed to the color industry: ink, paint and the like. For an approachable introduction to the same theory, accompanied by descriptions of image reproduction, try to find a copy of R.W.G. Hunt, The Reproduction of Colour [8]. But sorry to report, as I write this, it’s out of print.15 What is white? In additive image reproduction, the white point is the chromaticity of the color reproduced by equal red, green and blue components. White point is a function of the ratio (or balance) of power among the primaries. In subtractive reproduction, white is the SPD of the illumination, multiplied by the SPD of the media. There is no unique physical or perceptual defini- tion of white, so to achieve accurate color interchange you must specify the characteristics of your white. It is often convenient for purposes of calculation to define white as a uniform SPD. This white reference is known as the equal-energy illuminant, or CIE Illuminant E. A more realistic reference that approximates daylight has been specified numerically by the CIE as Illuminant D65. You should use this unless you have a good reason to use something else. The print industry commonly uses D50 and photography commonly uses D55. These represent compro- mises between the conditions of indoor (tungsten) and daylight viewing.16 What is color Planck determined that the SPD radiated from a hot object – a black body temperature? radiator – is a function of the temperature to which the object is heated. Many sources of illumination have, at their core, a heated object, so it is often useful to characterize an illuminant by specifying the temperature (in units of kelvin, K) of a black body radiator that appears to have the same hue. Although an illuminant can be specified informally by its color tempera- ture, a more complete specification is provided by the chromaticity coor- dinates of the SPD of the source. Modern blue CRT phosphors are more efficient with respect to human vision than red or green. In a quest for brightness at the expense of color accuracy, it is common for a computer display to have excessive blue content, about twice as blue as daylight, with white at about 9300 K. Human vision adapts to white in the viewing environment. An image viewed in isolation – such as a slide projected in a dark room – creates its own white reference, and a viewer will be quite tolerant of errors in the white point. But if the same image is viewed in the presence of an external white reference or a second image, then differences in white point can be objectionable.24
  • Frequently Asked Questions About Color 9 Complete adaptation seems to be confined to the range 5000 K to 5500 K. For most people, D65 has a little hint of blue. Tungsten illumination, at about 3200 K, always appears somewhat yellow.17 How can I characterize Additive reproduction is based on physical devices that produce all-posi- red, green and blue? tive SPDs for each primary. Physically and mathematically, the spectra add. The largest range of colors will be produced with primaries that appear red, green and blue. Human color vision obeys the principle of superposition, so the color produced by any additive mixture of three primary spectra can be predicted by adding the corresponding fractions of the XYZ components of the primaries: the colors that can be mixed from a particular set of RGB primaries are completely determined by the colors of the primaries by themselves. Subtractive reproduction is much more complicated: the colors of mixtures are determined by the primaries and by the colors of their combinations. An additive RGB system is specified by the chromaticities of its primaries and its white point. The extent (gamut) of the colors that can be mixed from a given set of RGB primaries is given in the (x, y) chromaticity diagram by a triangle whose vertices are the chromaticities of the prima- ries. In computing there are no standard primaries or white point. If you have an RGB image but have no information about its chromaticities, you cannot accurately reproduce the image. The NTSC in 1953 specified a set of primaries that were representative of phosphors used in color CRTs of that era. But phosphors changed over the years, primarily in response to market pressures for brighter receivers, and by the time of the first the videotape recorder the primaries in use were quite different than those “on the books”. So although you may see the NTSC primary chromaticities documented, they are of no use today. Contemporary studio monitors have slightly different standards in North America, Europe and Japan. But international agreement has been obtained on primaries for high definition television (HDTV), and these primaries are closely representative of contemporary monitors in studio video, computing and computer graphics. The primaries and the D65 white point of Rec. 709 [9] are: R G B white x 0.640 0.300 0.150 0.3127 y 0.330 0.600 0.060 0.3290 z 0.030 0.100 0.790 0.3582 For a discussion of nonlinear RGB in computer graphics, see Lindbloom [10]. For technical details on monitor calibration, consult Cowan [11].18 How do I transform RGB values in a particular set of primaries can be transformed to and between CIE XYZ and a from CIE XYZ by a three-by-three matrix transform. These transforms particular set of RGB involve tristimulus values, that is, sets of three linear-light components primaries? that conform to the CIE color matching functions. CIE XYZ is a special case of tristimulus values. In XYZ, any color is represented by a positive set of values.
  • 10 Frequently Asked Questions About Color Details can be found in SMPTE RP 177-1993 [12]. To transform from CIE XYZ into Rec. 709 RGB (with its D65 white point), use this transform:  R709   3.240479 −1.537150 −0.498535  X        G709  = −0.969256 1.875992 0.041556 •  Y   B709   0.055648 −0.204043 1.057311  Z        This matrix has some negative coefficients: XYZ colors that are out of gamut for a particular RGB transform to RGB where one or more RGB components is negative or greater than unity. Here’s the inverse transform. Because white is normalized to unity, the middle row sums to unity:  X  0.412453 0.357580 0.180423  R709   Y  =  0.212671 0.715160 0.072169 • G       709   Z  0.019334 0.119193 0.950227  B709        To recover primary chromaticities from such a matrix, compute little x and y for each RGB column vector. To recover the white point, transform RGB=[1, 1, 1] to XYZ, then compute x and y.19 Is RGB always device- Video standards specify abstract R’G’B’ systems that are closely matched dependent? to the characteristics of real monitors. Physical devices that produce addi- tive color involve tolerances and uncertainties, but if you have a monitor that conforms to Rec. 709 within some tolerance, you can consider the monitor to be device-independent. The importance of Rec. 709 as an interchange standard in studio video, broadcast television and high definition television, and the perceptual basis of the standard, assures that its parameters will be used even by devices such as flat-panel displays that do not have the same physics as CRTs.20 How do I transform data RGB values in a system employing one set of primaries can be trans- from one set of RGB formed into another set by a three-by-three linear-light matrix transform. primaries to another? Generally these matrices are normalized for a white point luminance of unity. For details, see Television Engineering Handbook [13]. As an example, here is the transform from SMPTE 240M (or SMPTE RP 145) RGB to Rec. 709:  R709   0.939555 0.050173 0.010272  R240 M        G709  =  0.017775 0.965795 0.016430 • G240 M   B709  −0.001622 −0.004371 1.005993  B240 M        All of these terms are close to either zero or one. In a case like this, if the transform is computed in the nonlinear (gamma-corrected) R’G’B’ domain the resulting errors will be insignificant.24
  • Frequently Asked Questions About Color 11 Here’s another example. To transform EBU 3213 RGB to Rec. 709:  R709  1.044036 −0.044036 0.   REBU      • G  G709  = 0. 1. 0.   EBU   B709  0.    0.011797 0.988203  BEBU     Transforming among RGB systems may lead to an out of gamut RGB result where one or more RGB components is negative or greater than unity.21 Should I use RGB or XYZ Once light is on its way to the eye, any tristimulus-based system will for image synthesis? work. But the interaction of light and objects involves spectra, not tristim- ulus values. In synthetic computer graphics, the calculations are actually simulating sampled SPDs, even if only three components are used. Details concerning the resultant errors are found in Hall [14].22 What is subtractive color? Subtractive systems involve colored dyes or filters that absorb power from selected regions of the spectrum. The three filters are placed in tandem. A dye that appears cyan absobs longwave (red) light. By control- ling the amount of cyan dye (or ink), you modulate the amount of red in the image. In physical terms the spectral transmission curves of the colorants multiply, so this method of color reproduction should really be called “multiplicative”. Photographers and printers have for decades measured transmission in base-10 logarithmic density units, where transmission of unity corresponds to a density of 0, transmission of 0.1 corresponds to a density of 1, transmission of 0.01 corresponds to a density of 2 and so on. When a printer or photographer computes the effect of filters in tandem, he subtracts density values instead of multiplying transmission values, so he calls the system subtractive. To achieve a wide range of colors in a subtractive system requires filters that appear colored cyan, yellow and magenta (CMY). Cyan in tandem with magenta produces blue, cyan with yellow produces green, and magenta with yellow produces red. Smadar Nehab suggests this memory aid: R G B R G B Cy Mg Yl Cy Mg Yl Additive primaries are at the top, subtractive at the bottom. On the left, magenta and yellow filters combine to produce red. On the right, red and green sources add to produce yellow.23 Why did my grade three To get a wide range of colors in an additive system, the primaries must teacher tell me that the appear red, green and blue (RGB). In a subtractive system the primaries primaries are red, yellow must appear yellow, cyan and magenta (CMY). It is complicated to and blue? predict the colors produced when mixing paints, but roughly speaking, paints mix additively to the extent that they are opaque (like oil paints), and subtractively to the extent that they are transparent (like water- colors). This question also relates to color names: your grade three “red” was probably a little on the magenta side, and “blue” was probably quite cyan. For a discussion of paint mixing from a computer graphics perspec- tive, consult Haase [15].
  • 12 Frequently Asked Questions About Color24 Is CMY just one-minus- In a theoretical subtractive system, CMY filters could have spectral RGB? absorption curves with no overlap. The color reproduction of the system would correspond exactly to additive color reproduction using the red, green and blue primaries that resulted from pairs of filters in combina- tion. Practical photographic dyes and offset printing inks have spectral absorp- tion curves that overlap significantly. Most magenta dyes absorb medi- umwave (green) light as expected, but incidentally absorb about half that amount of shortwave (blue) light. If reproduction of a color, say brown, requires absorption of all shortwave light then the incidental absorption from the magenta dye is not noticed. But for other colors, the “one minus RGB” formula produces mixtures with much less blue than expected, and therefore produce pictures that have a yellow cast in the mid tones. Similar but less severe interactions are evident for the other pairs of prac- tical inks and dyes. Due to the spectral overlap among the colorants, converting CMY using the “one-minus-RGB” method works for applications such as business graphics where accurate color need not be preserved, but the method fails to produce acceptable color images. Multiplicative mixture in a CMY system is mathematically nonlinear, and the effect of the unwanted absorptions cannot be easily analyzed or compensated. The colors that can be mixed from a particular set of CMY primaries cannot be determined from the colors of the primaries them- selves, but are also a function of the colors of the sets of combinations of the primaries. Print and photographic reproduction is also complicated by nonlineari- ties in the response of the three (or four) channels. In offset printing, the physical and optical processes of dot gain introduce nonlinearity that is roughly comparable to gamma correction in video. In a typical system used for print, a black code of 128 (on a scale of 0 to 255) produces a reflectance of about 0.26, not the 0.5 that you would expect from a linear system. Computations cannot be meaningfully performed on CMY components without taking nonlinearity into account. For a detailed discussion of transferring colorimetric image data to print media, see Stone [16].25 Why does offset printing Printing black by overlaying cyan, yellow and magenta ink in offset use black ink in addition printing has three major problems. First, colored ink is expensive. to CMY? Replacing colored ink by black ink – which is primarily carbon – makes economic sense. Second, printing three ink layers causes the printed paper to become quite wet. If three inks can be replaced by one, the ink will dry more quickly, the press can be run faster, and the job will be less expensive. Third, if black is printed by combining three inks, and mechanical tolerances cause the three inks to be printed slightly out of register, then black edges will suffer colored tinges. Vision is most demanding of spatial detail in black and white areas. Printing black with a single ink minimizes the visibility of registration errors. Other printing processes may or may not be subject to similar constraints.24
  • Frequently Asked Questions About Color 1326 What are color This term is ambiguous. In its first sense, color difference refers to numer- differences? ical differences between color specifications. The perception of color differences in XYZ or RGB is highly nonuniform. The study of perceptual uniformity concerns numerical differences that correspond to color differ- ences at the threshold of perceptibility (just noticeable differences, or JNDs). In its second sense, color difference refers to color components where brightness is “removed”. Vision has poor response to spatial detail in colored areas of the same luminance, compared to its response to lumi- nance spatial detail. If data capacity is at a premium it is advantageous to transmit luminance with full detail and to form two color difference components each having no contribution from luminance. The two color components can then have spatial detail removed by filtering, and can be transmitted with substantially less information capacity than luminance. Instead of using a true luminance component to represent brightness, it is ubiquitous for practical reasons to use a luma signal that is computed nonlinearly as outlined above (What is luma? ). The easiest way to “remove” brightness information to form two color channels is to subtract it. The luma component already contains a large fraction of the green information from the image, so it is standard to form the other two components by subtracting luma from nonlinear blue (to form B’-Y’) and by subtracting luma from nonlinear red (to form R’-Y’). These are called chroma. Various scale factors are applied to (B’-Y’) and (R’-Y’) for different appli- cations. The Y ’PBPR scale factors are optimized for component analog video. The Y ’CBCR scaling is appropriate for component digital video such as studio video, JPEG and MPEG. Kodak’s PhotoYCC™ uses scale factors optimized for the gamut of film colors. Y’UV scaling is appropriate as an intermediate step in the formation of composite NTSC or PAL video signals, but is not appropriate when the components are kept separate. The Y’UV nomenclature is now used rather loosely, and it sometimes denotes any scaling of (B’-Y’) and (R’-Y’). Y ’IQ coding is obsolete. The subscripts in CBCR and PBPR are often written in lower case. I find this to compromise readability, so without introducing any ambiguity I write them in uppercase. Authors with great attention to detail some- times “prime” these quantities to indicate their nonlinear nature, but because no practical image coding system employs linear color differ- ences I consider it safe to omit the primes.
  • 14 Frequently Asked Questions About Color27 How do I obtain color Here is the block diagram for luma/color difference encoding and difference components decoding: from tristimulus values? TRISTIMULUS TRANSFER COLOR DIFF. SUBSAMPLING 3 ×3 FUNCTION ENCODE FILTER [M] 0.45 [M] TRISTIMULUS TRANSFER COLOR DIFF. INTERPOLATION YCBCR 3 ×3 FUNCTION DECODE FILTER e.g. 4:2:2 2.5 [M] [M] XYZ RGB RGB YCBCR or R1G1B1 From linear XYZ – or linear R1 G1 B1 whose chromaticity coordinates are different from the interchange standard – apply a 3×3 matrix transform to obtain linear RGB according to the interchange primaries. Apply a nonlinear transfer function (“gamma correction”) to each of the compo- nents to get nonlinear R’G’B’. Apply a 3×3 matrix to obtain color differ- ence components such as Y’PBPR , Y’CBCR or PhotoYCC. If necessary, apply a color subsampling filter to obtain subsampled color difference components. To decode, invert the above procedure: run through the block diagram right-to-left using the inverse operations. If your monitor conforms to the interchange primaries, decoding need not explicitly use a transfer function or the tristimulus 3×3. The block diagram emphasizes that 3×3 matrix transforms are used for two distinctly different tasks. When someone hands you a 3×3, you have to ask for which task it is intended.28 How do I encode YPBPR Although the following matrices could in theory be used for tristimulus components? signals, it is ubiquitous to use them with gamma-corrected signals. To encode Y’PBPR , start with the basic Y’, (B’-Y’) and (R’-Y’) relation- ships: ′  Y601   0.299 0.587 0.114  R ′        Eq 1  B′ − Y601 = −0.299 −0.587 0.886 • G ′  ′  R′ − Y601  0.701 −0.587 −0.114  B′   ′      Y’PBPR components have unity excursion, where Y’ ranges [0..+1] and each of PB and PR ranges [-0.5..+0.5]. The (B’-Y’) and (R’-Y’) rows need to24
  • Frequently Asked Questions About Color 15 0.5 0.5 be scaled by ------------ and ------------ . To encode from R’G’B’ where reference - - 0.886 0.701 black is 0 and reference white is +1: ′ Y601  0.299 0.587 0.114   R′   P  = −0.168736 −0.331264 0.5  • G ′  Eq 2  B      PR   0.5    −0.418688 −0.081312  B′     The first row comprises the luma coefficients; these sum to unity. The second and third rows each sum to zero, a necessity for color difference components. The +0.5 entries reflect the maximum excursion of PB and PR of +0.5, for the blue and red primaries [0, 0, 1] and [1, 0, 0]. The inverse, decoding matrix is this:  R ′  1 0 1.402 ′  Y601 G ′  = 1 • P  −0.344136 −0.714136  B          B′  1 1.772 0   PR    29 How do I encode YCBCR Rec. 601 specifies eight-bit coding where Y’ has an excursion of 219 and components from RGB an offset of +16. This coding places black at code 16 and white at code 235, in [0, +1]? reserving the extremes of the range for signal processing headroom and footroom. CB and CR have excursions of ±112 and offset of +128, for a range of 16 through 240 inclusive. To compute Y’CBCR from R’G’B’ in the range [0..+1], scale the rows of the matrix of Eq 2 by the factors 219, 224 and 224, corresponding to the excur- sions of each of the components: Y601  16   65.481 128.553 24.966  R′  ′  C  = 128  + −37.797 −74.203 112.  • G ′  Eq 3  B        CR  128  112.      −93.786 −18.214  B′     Summing the first row of the matrix yields 219, the luma excursion from black to white. The two entries of 112 reflect the positive CBCR extrema of the blue and red primaries. Clamp all three components to the range 1 through 254 inclusive, since Rec. 601 reserves codes 0 and 255 for synchronization signals. To recover R’G’B’ in the range [0..+1] from Y’CBCR, use the inverse of Eq 3 above:  R′  0.00456621 0. 0.00625893  Y601   16  ′ G ′  = 0.00456621 −0.00153632 −0.00318811 •   C  − 128       B     B′  0.00456621 0.00791071    0.    CR  128       This looks overwhelming, but the Y’CBCR components are integers in eight bits and the reconstructed R’G’B’ are scaled down to the range [0..+1].30 How do I encode YCBCR In computing it is conventional to use eight-bit coding with black at code components from 0 and white at 255. To encode Y’CBCR from R’G’B’ in the range [0..255], computer RGB ?
  • 16 Frequently Asked Questions About Color using eight-bit binary arithmetic, scale the Y’CBCR matrix of Eq 3 by 256⁄ 255: ′ Y601  16  ′  65.738 129.057 25.064  R255   C  = 128  + 1 −37.945 −74.494 112.439 • G ′   B   256    255   CR  128      112.439 −94.154 −18.285  B255     ′  To decode R’G’B’ in the range [0..255] from Rec. 601 Y’CBCR, using eight- bit binary arithmetic: ′  R255  298.082 0. 408.583  Y601  16  ′   1   •   C  − 128 Eq 4 G255  = 256 298.082 −100.291 −208.120   B   ′   B255   ′  298.082 516.411  0.    CR  128      The multiplications by 1⁄ 256 can be accomplished by shifting. Some of the coefficients, when scaled by 1⁄ 256, are larger than unity. These coefficients will need more than eight multiplier bits. For implementation in binary arithmetic the matrix coefficients have to be rounded. When you round, take care to preserve the row sums of [1, 0, 0]. The matrix of Eq 4 will decode standard Y’CBCR components to RGB components in the range [0..255], subject to roundoff error. You must take care to avoid overflow due to roundoff error. But you must protect against overflow in any case, because studio video signals use the extremes of the coding range to handle signal overshoot and undershoot, and these will require clipping when decoded to an RGB range that has no headroom or footroom.31 How do I encode YCBCR Studio R’G’B’ signals use the same 219 excursion as the luma component components from studio of Y’CBCR.To encode Y’CBCR from R’G’B’ in the range [0..219], using video? eight-bit binary arithmetic, scale the Y’CBCR encoding matrix of Eq 3 above by 256⁄ 219. Here is the encoding transform for studio video: ′ Y601  16   76.544 150.272 ′ 29.184  R219   C  = 128  + 1 −44.182    B  −86.740 130.922 • G219  ′  256    CR  128      130.922  −109.631 −21.291  B219    ′  To decode R’G’B’ in the range [0..219] from Y’CBCR, using eight-bit binary arithmetic: ′  R219  256. 0. 350.901  Y601  16  ′   1   •   C  − 128 G219  = 256 256. −86.132 −178.738   B   ′   B219   ′  256. 443.506  0.    CR  128      The entries of 256 in this matrix indicate that the corresponding compo- nent can simply be added; there is no need for a multiplication operation. This matrix contains entries larger than 256; the corresponding multi- pliers will need capability for nine bits. The matrices in this section conform to Rec. 601 and apply directly to conventional 525/59.94 and 625/50 video. It is not yet decided whether emerging HDTV standards will use the same matrices, or adopt a new set of matrices having different luma coefficients. In my view it would be24
  • Frequently Asked Questions About Color 17 unfortunate if different matrices were adopted, because then image coding and decoding would depend on whether the picture was small (conventional video) or large (HDTV). In digital video, Rec. 601 standardizes subsampling denoted 4:2:2, where CB and CR components are subsampled horizontally by a factor of two with respect to luma. JPEG and MPEG conventionally subsample by a factor of two in the vertical dimension as well, denoted 4:2:0. Color difference coding is standardized in Rec. 601. For details on color difference coding as used in video, consult Poynton [17].32 How do I decode RGB Kodak’s PhotoYCC uses the Rec. 709 primaries, white point and transfer from PhotoYCC™? function. Reference white codes to luma 189; this preserves film high- lights. The color difference coding is asymmetrical, to encompass film gamut. You are unlikely to encounter any raw image data in PhotoYCC form because YCC is closely associated with the PhotoCD™ system whose compression methods are proprietary. But just in case, the following equation is comparable to in that it produces R’G’B’ in the range [0..+1] from integer YCC. If you want to return R’G’B’ in a different range, or implement the equation in eight-bit integer arithmetic, use the techniques in the section above. ′  R709  0.0054980 0. 0.0051681  Y601189   0  ′ ,     •   C  − 156  G709  = 0.0054980 −0.0015446 −0.0026325   ′ 1     B709  0.0054980 0.0079533 0.  ′        C2  137     Decoded R’G’B’ components from PhotoYCC can exceed unity or go below zero. PhotoYCC extends the Rec. 709 transfer function above unity, and reflects it around zero, to accommodate wide excursions of R’G’B’. To decode to CRT primaries, clip R’G’B’ to the range zero to one.33 Will you tell me how to No, I won’t! Y’UV and Y’IQ have scale factors appropriate to composite decode YUV and YIQ? NTSC and PAL. They have no place in component digital video! You shouldn’t code into these systems, and if someone hands you an image claiming it’s Y’UV, chances are it’s actually Y’CBCR, it’s got the wrong scale factors, or it’s linear-light. Well OK, just this once. To transform Y’, (B’-Y’) and (R’-Y’) components from Eq 1 to Y’UV, scale (B’-Y’) by 0.492111 to get U and scale R’-Y’ by 0.877283 to get V. The factors are chosen to limit composite NTSC or PAL amplitude for all legal R’G’B’ values: 1 [ ] − ≤ Y ′ ± 0.492111 ( B′ − Y ′) + 0.877283 ( R′ − Y ′) ≤ 3 4 3 To transform from Y’IQ to Y’UV, perform a 33° rotation and an exchange of color difference axes: ′ Y601 1 0 0 ′  Y601  U  = 0 −0.544639 0.838671• I            V  0   0.838671 0.544639  Q 34 How should I test my To test your encoding and decoding, ensure that colorbars are handled encoders and decoders? correctly. A colorbar signal comprises a binary RGB sequence ordered for
  • 18 Frequently Asked Questions About Color decreasing luma: white, yellow, cyan, green, magenta, red, blue and black. 1 1 0 0 1 1 0 0 1 1 1 1 0 0 0 0   1 0 1 0 1 0 1 0   To ensure that your scale factors are correct and that clipping is not being invoked, test 75% bars, a colorbar sequence having 75%-amplitude bars instead of 100%.35 What is perceptual A system is perceptually uniform if a small perturbation to a component uniformity? value is approximately equally perceptible across the range of that value. The volume control on your radio is designed to be perceptually uniform: rotating the knob ten degrees produces approximately the same percep- tual increment in volume anywhere across the range of the control. If the control were physically linear, the logarithmic nature of human loudness perception would place all of the perceptual “action” of the control at the bottom of its range. The XYZ and RGB systems are far from exhibiting perceptual uniformity. Finding a transformation of XYZ into a reasonably perceptually-uniform space consumed a decade or more at the CIE and in the end no single system could be agreed. So the CIE standardized two systems, L *u*v* and L*a*b*, sometimes written CIELUV and CIELAB. (The u and v are unre- lated to video U and V.) Both L*u*v* and L*a*b* improve the 80:1 or so perceptual nonuniformity of XYZ to about 6:1. Both demand too much computation to accommodate real-time display, although both have been successfully applied to image coding for printing. Computation of CIE L*u*v* involves intermediate u’ and v’ quantities, where the prime denotes the successor to the obsolete 1960 CIE u and v system: 4X 9Y u = , v = X + 15 Y + 3 Z X + 15 Y + 3 Z First compute un and vn for your reference white Xn , Yn and Zn . Then ’ ’ compute u’ and v‘ – and L* as discussed earlier – for your colors. Finally, compute: u * = 13 L* (u ′ − un ), ′ v * = 13 L* (v ′ − vn ) ′ L*a*b* is computed as follows, for (X/Xn, Y/Yn, Z/Zn ) > 0.01:  1 1  1 1  X 3  Y 3  Y 3  Z 3 a * = 500   −    , b * = 200   −    X   Yn  Y   Zn   n     n    These equations are great for a few spot colors, but no fun for a million pixels. Although it was not specifically optimized for this purpose, the nonlinear R’G’B’ coding used in video is quite perceptually uniform, and has the advantage of being fast enough for interactive applications.24
  • Frequently Asked Questions About Color 1936 What are HSB and HLS? HSB and HLS were developed to specify numerical Hue, Saturation and Brightness (or Hue, Lightness and Saturation) in an age when users had to specify colors numerically. The usual formulations of HSB and HLS are flawed with respect to the properties of color vision. Now that users can choose colors visually, or choose colors related to other media (such as PANTONE), or use perceptually-based systems like L *u*v* and L*a*b*, HSB and HLS should be abandoned. Here are some of problems of HSB and HLS. In color selection where “lightness” runs from zero to 100, a lightness of 50 should appear to be half as bright as a lightness of 100. But the usual formulations of HSB and HLS make no reference to the linearity or nonlinearity of the underlying RGB, and make no reference to the lightness perception of human vision. The usual formulation of HSB and HLS compute so-called “lightness” or “brightness” as (R+G+B)/3. This computation conflicts badly with the properties of color vision, as it computes yellow to be about six times more intense than blue with the same “lightness” value (say L=50). HSB and HSL are not useful for image computation because of the discontinuity of hue at 360°. You cannot perform arithmetic mixtures of colors expressed in polar coordinates. Nearly all formulations of HSB and HLS involve different computations around 60° segments of the hue circle. These calculations introduce visible discontinuities in color space. Although the claim is made that HSB and HLS are “device independent”, the ubiquitous formulations are based on RGB components whose chro- maticities and white point are unspecified. Consequently, HSB and HLS are useless for conveyance of accurate color information. If you really need to specify hue and saturation by numerical values, rather than HSB and HSL you should use polar coordinate version of u* and v*: h* for hue angle and c*uv for chroma. uv37 What is true color? True color is the provision of three separate components for additive red, green and blue reproduction. A high quality true color system provides 8 bits for each of the three components; this is known as 24 bit color. A high-quality true color system interposes a lookup table between each component of the framestore and each channel of the display. This makes it possible to use a true color system with either linear or nonlinear coding. In the X Window System, true color refers to fixed lookup tables, and direct color refers to lookup tables that are under the control of appli- cation software. A hicolor system provides 16 bits for each pixel, partitioned into red, green, and blue components. Hicolor is a variant of truecolor, but with an insufficient number of bits to provide photographic quality. The 16 bits may be partitioned as 5 bits for each component (with the extra bit some- times used to convey transparency), or as 5 bits of red, 6 bits of green, and 5 bits of blue. Hicolor systems usually offer no lookup table at the output of the framebuffer, so the image data is coded like video: The RGB components are assumed to have been raised to a power of about 0.45.
  • 20 Frequently Asked Questions About Color38 What is indexed color? Indexed color (or pseudocolor), is the provision of a relatively small number of discrete colors – often 256 – in a colormap or palette. The framebuffer stores, at each pixel, the index number of a color. At the output of the framebuffer, a lookup table uses the index to retrieve red, green and blue components that are then sent to the display. The colors in the map may be fixed systematically at the design of a system. As an example, 216 index entries an eight-bit indexed color system can be partitioned systematically into a 6 × 6 × 6 colorcube to imple- ment what amounts to a direct color system where each of red, green and blue has a value that is an integer in the range zero to five. An RGB image can be converted to a predetermined colormap by choosing, for each pixel in the image, the colormap index corresponding to the “closest” RGB triple. With a systematic colormap such as a 6 × 6 × 6 colorcube this is straightforward. For an arbitrary colormap, the colormap has to be searched looking for entries that are “close” to the requested color. “Closeness” should be determined according to the perceptibility of color differences. Using color systems such as CIE L *u*v* or L*a*b* is computationally prohibitive, but in practice it is adequate to use a Euclidean distance metric in R’G’B’ components coded nonlinearly according to video practice. A direct color image can be converted to indexed color with an image- dependent colormap by a process of color quantization that searches through all of the triples used in the image, and chooses the palette for the image based on the colors that are in some sense most “important”. Again, the decisions should be made according to the perceptibility of color differences. Adobe Photoshop™ can perform this conversion. UNIX™ users can employ the pbm package. If your system accommodates arbitrary colormaps, when the map associ- ated with the image in a particular window is loaded into the hardware colormap, the maps associated with other windows may be disturbed. In window system such as the X Window System™ running on a multi- tasking operating system such as UNIX, even moving the cursor between two windows with different maps can cause annoying colormap flashing. An eight-bit indexed color system requires less data to represent a picture than a twenty-four bit truecolor system. But this data reduction comes at a high price. The truecolor system can represent each of its three compo- nents according to the principles of sampled continuous signals. This makes it possible to accomplish, with good quality, operations such as resizing the image. In indexed color these operations introduce severe artifacts because the underlying representation lacks the properties of a continuous representation, even if converted back to RGB. In graphic file formats such as GIF of TIFF, an indexed color image is accompanied by its colormap. Generally such a colormap has RGB entries that are gamma corrected: the colormap’s RGB codes are intended to be presented directly to a CRT, without further gamma correction.24
  • Frequently Asked Questions About Color 2139 I want to visualize a When you look at a rainbow you do not see a smooth gradation of colors. scalar function of two Instead, some bands appear quite narrow, and others are quite broad. variables. Should I use Perceptibility of hue variation near 540 nm is half that of either 500 nm or RGB values 600 nm. If you use the rainbow’s colors to represent data, the visibility of corresponding to the differences among your data values will depend on where they lie in the colors of the rainbow? spectrum. If you are using color to aid in the visual detection of patterns, you should use colors chosen according to the principles of perceptual uniformity. This an open research problem, but basing your system on CIE L*a*b* or L*u*v*, or on nonlinear video-like RGB, would be a good start.40 What is dithering? A display device may have only a small number of choices of greyscale values or color values at each device pixel. However if the viewer is suffi- ciently distant from the display, the value of neighboring pixels can be set so that the viewer’s eye integrates several pixels to achieve an apparent improvement in the number of levels or colors that can be reproduced. Computer displays are generally viewed from distances where the device pixels subtend a rather large angle at the viewer’s eye, relative to his visual acuity. Applying dither to a conventional computer display often introduces objectionable artifacts. However, careful application of dither can be effective. For example, human vision has poor acuity for blue spatial detail but good color discrimination capability in blue. Blue can be dithered across two-by-two pixel arrays to produce four times the number of blue levels, with no perceptible penalty at normal viewing distances.41 How does halftoning The processes of offset printing and conventional laser printing are intrin- relate to color? sically bilevel: a particular location on the page is either covered with ink or not. However, each of these devices can reproduce closely-spaced dots of variable size. An array of small dots produces the perception of light gray, and an array of large dots produces dark gray. This process is called halftoning or screening. In a sense this is dithering, but with device dots so small that acceptable pictures can be produced at reasonable viewing distances. Halftone dots are usually placed in a regular grid, although stochastic screening has recently been introduced that modulates the spacing of the dots rather than their size. In color printing it is conventional to use cyan, magenta, yellow and black grids that have exactly the same dot pitch but different carefully-chosen screen angles. The recently introduced technique of Flamenco screening uses the same screen angles for all screens, but its registration require- ments are more stringent than conventional offset printing. Agfa’s booklet [18] is an excellent introduction to practical concerns of printing. And it’s in color! The standard reference to halftoning algo- rithms is Ulichney [19], but that work does not detail the nonlinearities found in practical printing systems. For details about screening for color reproduction, consult Fink [20]. Consult Frequently Asked Questions about Gamma for an introduction to the transfer function of offset printing.
  • 22 Frequently Asked Questions About Color42 What’s a color Software and hardware for scanner, monitor and printer calibration have management system? had limited success in dealing with the inaccuracies of color handling in desktop computing. These solutions deal with specific pairs of devices but cannot address the end-to-end system. Certain application devel- opers have added color transformation capability to their applications, but the majority of application developers have insufficient expertise and insufficient resources to invest in accurate color. A color management system (CMS) is a layer of software resident on a computer that negotiates color reproduction between the application and color devices. It cooperates with the operating system and the graphics library components of the platform software. Color management systems perform the color transformations necessary to exchange accurate color between diverse devices, in various color coding systems including RGB, CMYK and CIE L*a*b*. The CMS makes available to the application a set of facilities whereby the application can determine what color devices and what color spaces are available. When the application wishes to access a particular device, it requests that the color manager perform a mathematical transform from one space to another. The color spaces involved can be device-indepen- dent abstract color spaces such as CIE XYZ, CIE L*a*b* or calibrated RGB. Alternatively a color space can be associated with a particular device. In the second case the color manager needs access to characterization data for the device, and perhaps also to calibration data that reflects the state of the particular instance of the device. Apple’s ColorSync™ provides an interface between a Mac application program and color management capabilities either built-in to ColorSync or provided by a plug-in. Kodak’s CMS is built-into the latest version of Sun’s Solaris operating system. The basic CMS services provided with desktop operating systems are likely to be adequate for office users, but are unlikely to satisfy high-end users such as in prepress. All of the announced systems have provisions for plug-in color management modules (CMMs) that can provide sophisti- cated transform machinery. Advanced color management modules are commercially available from Kodak, Agfa, and others. For an application developer’s prespective on color management, see Aldus [21].43 How does a CMS know A CMS needs access to information that characterizes the color repro- about particular devices? duction capabilities of particular devices. The set of characterization data for a device is called a device profile. Industry agreement has been reached on the format of device profiles; information is available from the Interna- tional Color Consortium (ICC). Vendors of color peripherals will soon provide industry-standard profiles with their devices, and they will have to make, buy or rent characterization services. If you have a device that has not been characterized by its manufacturer, Agfa’s FotoTune™ software – part of Agfa’s FotoFlow™ color manager – can create device profiles.44 Is a color management Not quite yet. But future color management system are likely to include system useful for color the ability to accommodate commercial proprietary color specification specification? systems such as PANTONE™ and COLORCURVE™. These vendors are likely24
  • Frequently Asked Questions About Color 23 to provide their color specification systems in shrink-wrapped form to plug into color managers. In this way, users will have guaranteed color accuracy among applications and peripherals, and application vendors will no longer need to pay to license these systems individually.45 I’m not a color expert. Use the CIE D65 white point (6504 K) if you can. What parameters should Use the Rec. 709 primary chromaticities. Your monitor is probably I use to code my images? already quite close to this. Rec. 709 has international agreement, offers excellent performance, and is the basis for HDTV development so it’s future-proof. If you need to operate in linear light, so be it. Otherwise, for best percep- tual performance and maximum ease of interchange with digital video, use the Rec. 709 transfer function, with its 0.45-power law. If you need Mac compatibility you will have to suffer a penalty in perceptual perfor- mance. Raise tristimulus values to the 1 ⁄1.4 -power before presenting them to QuickDraw. To code luma, use the Rec. 601 luma coefficients 0.299, 0.587 and 0.114. Use Rec. 601 digital video coding with black at 16 and white at 235. Use prime symbols (’) to denote all of your nonlinear components! PhotoCD uses all of the preceding measures. PhotoCD codes color differ- ences asymmetrically, according to film gamut. Unless you have a requirement for film gamut, you should code into color differences using Y‘CBCR coding with Rec. 601 studio video (16..235/128±112) excursion. Tag your image data with the primary and white chromaticity, transfer function and luma coefficients that you are using. TIFF 6.0 tags have been defined for these parameters. This will enable intelligent readers, today or in the future, to determine the parameters of your coded image and give you the best possible results.46 References [1] B. Berlin and P. Kay, Basic Color Terms (Berkeley, Calif.: U. of Calif. Press, 1969) [2] Publication CIE No 17.4, International Lighting Vocabulary (Vienna, Austria: Central Bureau of the Commission Internationale de L’Éclairage) [3] LeRoy E. DeMarsh and Edward J. Giorgianni, “Color Science for Imaging Systems,” in Physics Today, September 1989, 44-52. [4] W.F. Schreiber, Fundamentals of Electronic Imaging Systems, Second Edition (Springer-Verlag, 1991) [5] Publication CIE No 15.2, Colorimetry, Second Edition (Vienna, Austria: Central Bureau of the Commission Internationale de L’Éclairage, 1986) [6] Günter Wyszecki and W.S. Styles, Color Science: Concepts and Methods, Quantitative Data and Formulae, Second Edition (New York: John Wiley & Sons, 1982) [7] D.B. Judd and Günter Wyszecki, Color in Business, Science and Industry, Third Edition (New York: John Wiley & Sons, 1975) [8] R.W.G. Hunt, The Reproduction of Colour in Photography, Printing and Tele- vision, Fourth Edition (Tolworth, England: Fountain Press, 1987)
  • 24 Frequently Asked Questions About Color [9] ITU-R Recommendation BT.709, Basic Parameter Values for the HDTV Stan- dard for the Studio and for International Programme Exchange (1990), [formerly CCIR Rec. 709] (Geneva: ITU, 1990) [10] Bruce J. Lindbloom, “Accurate Color Reproduction for Computer Graphics Applications”, Computer Graphics, Vol. 23, No. 3 (July 1989), 117-126 (proceedings of SIGGRAPH ’89) [11] William B. Cowan, “An Inexpensive Scheme for Calibration of a Colour Monitor in terms of CIE Standard Coordinates”, in Computer Graphics, Vol. 17, No. 3 (July 1983), 315-321. [12] SMPTE RP 177-1993, Derivation of Basic Television Color Equations. [13] Television Engineering Handbook, Featuring HDTV Systems, Revised Edition by K. Blair Benson, revised by Jerry C. Whitaker (New York: McGraw- Hill, 1992). This supersedes the Second Edition. [14] Roy Hall, Illumination and Color in Computer Generated Imagery (Springer- Verlag, 1989) [15] Chet S. Haase and Gary W. Meyer, “Modelling Pigmented Materials for Realistic Image Synthesis”, in ACM Transactions on Graphics, Vol. 11, No. 4, 1992, p. 305. [16] Maureen C. Stone, William B. Cowan and John C. Beatty, “Color Gamut Mapping and the Printing of Digital Color Images”, in ACM Transactions on Graphics, Vol. 7, No. 3, October 1988. [17] Charles Poynton, A Technical Introduction to Digital Video (New York: John Wiley & Sons, 1996) [18] Agfa Corporation, An introduction to Digital Color Prepress, Volumes 1 and 2 (Mt.Prospect, Ill.: Prepress Education Resources, 800 395 7007, 1990) [19] Robert Ulichney, Digital Halftoning (Cambridge, Mass.: MIT Press, 1988) [20] Peter Fink, PostScript Screening: Adobe Accurate Screens (Mountain View, Calif.: Adobe Press, 1992) [21] Color management systems: Getting reliable color from start to finish, Adobe Systems, <http://www.adobe.com/PDFs/FaxYI/500301.pdf>. [22] Overview of color publishing, Adobe Systems, <http://www.adobe.com/PDFs/FaxYI/500302.pdf>. Despite appear- ances and title, this document is in greyscale, not color.47 Contributors Thanks to Norbert Gerfelder, Alan Roberts and Fred Remley for their proofreading and editing. I learned about color from LeRoy DeMarsh, Ed Giorgianni, Junji Kumada and Bill Cowan. Thanks! This note contains some errors: if you find any, please let me know. I welcome suggestions for additions and improvements.24
  • Home > ArticlesThe Visual System and the Brain:Hubel and Wiesel Redux • By Dale Purves • Dec 21, 2009This chapter is from the bookBrains: How They Seem to WorkDale Purves explains the human visual system and how the work of David Hubel and Torsten Wiesel providedthe greatest single influence on the ways neuroscientists think about the brain during much of the second halfof the twentieth century.I dont think many neuroscientists would dispute the statement that the work David Hubel and Torsten Wieselbegan in the late 1950s and continued for the next 25 years provided the greatest single influence on the waysneuroscientists thought about and prosecuted studies of the brain during much of the second half of thetwentieth century. Certainly, what they were doing had never been very far from my own thinking, even whileworking on the formation and maintenance of synaptic connections in the peripheral nervous system. Toexplain the impact of their work and to set the stage for understanding the issues discussed in the remainingchapters, I need to fill in more information about the visual system, what Hubel and Wiesel actually did, andhow they interpreted it.Presumably because we humans depend so heavily on vision, this sensory modality has for centuries been afocus of interest for natural philosophers and, in the modern era, neuroscientists and psychologists. By the timeHubel and Wiesel got into the game in the 1950s, a great deal was already known about the anatomy of thesystem and about the way light interacts with receptor cells in the retina to initiate the action potentials thattravel centrally from retina to cortex, ultimately leading to what we see. The so-called primary visual pathway(Figure 7.1) begins with the two types of retinal receptors, rods and cones, and their transduction of lightenergy.Figure 7.1 The primary visual pathway carries information from the eye to the regions of the brain thatdetermine what we see. The pathway entails the retinas, optic nerves, optic tracts, dorsal lateral geniculatenuclei in the thalamus, optic radiations, and primary (or striate) and adjacent secondary (or extrastriate) visualcortices in each occipital lobe at the back of the brain (see Figures 7.2 and 7.3). Other central pathways totargets in the brainstem (dotted lines) determine pupil diameter as a function of retinal light levels, organize andmotivate eye movements, and influence circadian rhythms. (After Purves and Lotto, 2003)The visual processing that rods initiate is primarily concerned with seeing at very low light levels, whereas
  • cones respond only to greater light intensities and are responsible for the detail and color qualities that wenormally think of as defining visual perception. However, the primary visual pathway is anything but simple.Following the extensive neural processing that takes place among the five basic cell classes found in the retina,information arising from both rods and cones converges onto the retinal ganglion cells, the neurons whoseaxons leave the retina in the optic nerve. The major targets of the retinal ganglion cells are the neurons in thedorsal lateral geniculate nucleus of the thalamus, which project to the primary visual cortex (usually referred toas V1 or the striate cortex) (Figure 7.2).Figure 7.2 Photomicrograph of a section of the human primary visual cortex, taken in the plane of the face (seeFigure 7.1). The characteristic myelinated band, or stria, is why this region of cortex is referred to as the striatecortex (myelin is a fatty material that invests most axons in the brain and so stains darkly with reagents thatdissolve in fat, such as the one used). The primary visual cortex occupies about 25 square centimeters (about athird of the surface area of a dollar bill) in each cerebral hemisphere; the overall area of the cortical surface forthe two hemispheres together is about 0.8 square meters (or, as my colleague Len White likes to tell students,about the area of a medium pizza). Most of the primary visual cortex lies within a fissure on the medial surfaceof the occipital lobe called the calcarine sulcus, which is also shown in Figure 6.1B. The extrastiate cortex thatcarries out further processing of visual information is immediately adjacent (see Figure 7.3). (Courtesy of T.Andrews and D. Purves)Although the primary visual cortex (V1) is the nominal terminus of this pathway, many of the neurons thereproject to additional areas in the occipital, parietal, and temporal lobes (Figure 7.3). Neurons in V1 also interactextensively with each other and send information back to the thalamus, where much processing occurs thatremains poorly understand. Because of the increasing integration of information from other brain regions in thevisual cortical regions adjacent to V1, these higher-order cortical processing regions (V2, V3, and so on) arecalled visual association areas. Taken together, they are also referred to as extrastriate visual cortical areasbecause they lack the anatomically distinct layer that creates the striped appearance of V1 (see Figure 7.2). Inmost conceptions of vision, perception is thought to occur in these higher-order visual areas adjacent to V1(although note that what occur means in this statement is not straightforward).Figure 7.3 The higher-order visual cortical areas adjacent to the primary visual cortex, shown here in lateral (A)and medial (B) views of the brain. The primary visual cortex (V1) is indicated in green; the additional coloredareas with their numbered names are together called the extrastriate or visual association areas and occupymuch of the rest of the occipital lobe at the back of the brain (its anterior border is indicated by the dotted line).(After Purves and Lotto, 2003)By the 1950s, much had also been learned about visual perception. The seminal figures in this aspect of thehistory of vision science were nineteenth-century German physicist and physiologist Hermann von Helmholtz,and Wilhelm Wundt and Gustav Fechner, who initiated the modern study of perception from a psychologicalperspective at about the same time. However, Helmholtz gave impetus to the effort to understand perception interms of visual system physiology, and his work was the forerunner of the program Hubel and Wiesel
  • undertook nearly a century later.A good example of Helmholtzs approach is his work on color vision. At the beginning of the nineteenthcentury, British natural philosopher Thomas Young had surmised that three distinct types of receptors in thehuman retina generate the perception of color. Although Young knew nothing about cones or the pigments inthem that underlie light absorption, he nevertheless contended in lectures he gave to the Royal Society in 1802that three different classes of receptive "particles" must exist. Youngs argument was based on what humansperceive when lights of different wavelengths (loosely speaking, lights of different colors) are mixed, amethodology that had been used since Isaac Newtons discovery a hundred years earlier that light comprises arange of wavelengths. Youngs key observation was that most color sensations can be produced by mixingappropriate amounts of lights from the long-, middle-, and short-wavelength regions of the visible lightspectrum (mixing lights is called color addition and is different from mixing pigments, which subtractsparticular wavelengths from the stimulus that reaches the eye by absorbing them).Youngs theory was largely ignored until the latter part of the nineteenth century, when it was revived andgreatly extended by Helmholtz and James Clerk Maxwell, another highly accomplished physicist interested invision. The ultimately correct idea that humans have three types of cones with sensitivities (absorption spectra)that peak in the long, middle, and short wavelength ranges, respectively, is referred to as trichromacy, denotingthe fact that most human color sensations can be elicited in normal observers by adjusting the relative activationof the three cone types (see Chapter 9). The further hypothesis that the relative activation explains the colors weactually see is called the trichromacy theory, and Helmholtz spotlighted this approach to explaining perception.Helmholtzs approach implied that perceptions (color perceptions, in this instance) are a direct consequence ofthe way receptors and the higher-order neurons related to them analyze and ultimately represent stimulusfeatures and, therefore, the features of objects in the world. For Helmholtz and many others since that era, thefeature that color perceptions represent is the nature of object surfaces conveyed by the spectrum of light theyreflect to the eye.This mindset that sensory systems represent the features of objects in the world was certainly the way I hadsupposed the sensory components of the brain to be working—and as far as I could tell, it was how prettymuch everyone else thought about these issues in the 1960s and 1970s. By the same token, I took exploringthe underlying neural circuitry (the work Hubel and Wiesel were undertaking) to be the obvious way to solvethe problem of how the visual system generates what we see. The step remaining was the hard work needed todetermine how the physiology of individual visual neurons and their connections in the various stations of thevisual pathway were accomplishing this feat.Using the extracellular recording method they had developed in Kufflers lab at Johns Hopkins, Hubel andWiesel were working their way up the primary visual pathway in cats and, later, in monkeys. At each stage inthe pathway—the thalamus, primary visual cortex, and, ultimately, extratstriate cortical areas (see Figures 7.1–7.3)—they carefully studied the response characteristics of individual neurons in the type of setup that Figure7.4 illustrates, describing the results in terms of what are called the receptive field properties of visual neurons.Their initial studies of neurons in the lateral geniculate nucleus of the thalamus showed responses that weresimilar to the responses of the retinal output neurons (retinal ganglion cells) that Kuffler had described. Despitethis similarity, the information the axons carried from the thalamus to the cortex was not exactly the same as theinformation coming into the nucleus from the retina, indicating some processing by the thalamus. The majoradvances, however, came during the next few years as they studied the responses of nerve cells in the primaryvisual cortex. The key finding was that, unlike the relatively nondescript responses to light stimuli of visualneurons in the retina or the thalamus, cortical neurons showed far more varied and specific responses. On thesurface, the nature of these responses seemed closely related to the features we end up seeing. For example, therather typical V1 neuron illustrated in Figure 7.4 responds to light stimuli presented at only one relatively smalllocus on the screen (defining the spatial limits of the neurons receptive field), and only to bars of light. Incontrast, neurons in the retina or thalamus respond to any configuration of light that falls within their receptivefield. Moreover, many V1 neurons are selective for orientation and direction of movement, respondingvigorously to bars only at or near a particular angle on the screen and moving in a particular direction. Thesereceptive field properties were the beginning of what has eventually become a long list, including selectiveresponses to the lengths of lines, different colors, input from one eye or the other, and the different depthsindicated by the somewhat different views of the two eyes. Based on this rapidly accumulating evidence, itseemed clear that visual cortical neurons were indeed encoding the features of retinal images and, therefore, the
  • properties of objects in the world.Figure 7.4 Assessing the responses of individual neurons to visual stimuli in experimental animals (althoughthe animal is anesthetized, the visual system continues to operate much as it would if the animal were awake).A) Diagram of the experimental setup showing an extracellular electrode recording from a neuron in theprimary visual cortex of a cat (which is more anterior in the brain than in humans). By monitoring theresponses of the neuron to stimuli shown on a screen, Hubel and Wiesel could get a good idea of whatparticular visual neurons normally do. B) In this example, the neuron being recorded from in V1 respondsselectively to bars of light presented on the screen in different orientations; the cell fires action potentials(indicated by the vertical lines) only when the bar is at a certain location on the screen and in a certainorientation. These selective responses to stimuli define each neurons receptive field properties. (After Purves,Augustine, et al., 2008)Important as these observations were, amassing this foundational body of information about the responseproperties of visual neurons was not Hubel and Wiesels only contribution. At each stage of theirinvestigations, they used imaginative and often new anatomical methods to explore the organization of thethalamus, the primary visual cortex, and some of the higher-order visual processing regions. They also madebasic contributions to understanding cortical development as they went along, work that might eventually standas their greatest legacy. Hubel and Wiesel knew from the studies just described that neurons in V1 arenormally innervated by thalamic inputs that can be activated by stimulating the right eye, the left eye, or botheyes (Figure 7.5). What would happen to the neural connections in the cortex if one eye of an experimentalanimal was closed during early development, depriving the animal of normal visual experience through thateye? Although most of the neurons in V1 are activated to some degree by both eyes (Figure 7.5A), when theyclosed one eye of a kitten early in life and studied the brain after the animal had matured (which takes about sixmonths in cats), they found a remarkable change. Electrophysiological recordings showed that very fewneurons could be driven from the deprived eye: Most of the cortical cells were now being driven by the eyethat had remained open (Figure 7.5B). Moreover, the cats were behaviorally blind to stimuli presented to thedeprived eye, a deficit that did not resolve even if the deprived eye was subsequently left open for months. Thesame manipulation in an adult cat—closing one eye for a long period—had no effect on the responses of thevisual neurons. Even when they closed one eye for a year or more, the distribution of V1 neurons driven byone eye and the animals visual behavior tested through the reopened eye were indistinguishable from normal(Figure 7.5C). Therefore, between the time a kittens eyes open (about a week after birth) and a year of age,visual experience determines how the visual cortex is wired, and does so in a way that later experience does notreadily reverse.Figure 7.5 The effect on cortical neurons of closing one eye in a kitten. A) The distribution observed in theprimary visual cortex of normal adult cats by stimulating one eye or the other. Cells in group 1 are activatedexclusively by one eye (referred to here as the contralateral eye), and cells in group 7 are activated exclusivelyby the other (ipsilateral) eye. Neurons in the other groups are activated to varying degrees by both eyes (NRindicates neurons that could not be activated by either eye). B) Following closure of one eye from one weekafter birth until about two and a half months of age, no cells could be activated by the deprived (contralateral)eye. C) In contrast, a much longer period of monocular deprivation in an adult cat (from 12 to 38 months ofage in this example) had little effect on ocular dominance. (After Purves, Augustine, et al., 2008)The clinical, educational, and social implications of these results are hard to miss. In terms of clinicalophthalmology, early deprivation in developed countries is most often the result of strabismus, a misalignmentof the two eyes caused by deficient control of the direction of gaze by the muscles that move the eye. This
  • problem affects about 5% of children. Because the resulting misalignment produces double vision, theresponse of the visual system in severely afflicted children is to suppress the input from one eye (its unclearexactly how this happens). This effect can eventually render children blind in the suppressed eye if they are nottreated promptly by intermittently patching the good eye or intervening surgically to realign the eyes. Aprevalent cause of visual deprivation in children in underdeveloped countries is a cataract (opacification of thelens) caused by diseases such as river blindness (an infection caused by a parasitic worm) or trachoma (aninfection caused by a small, bacteria-like organism). A cataract in one eye is functionally equivalent tomonocular deprivation in experimental animals, and this defect also results in an irreversible loss of visualacuity in the untreated childs deprived eye, even if the cataract is later removed. Hubel and Wieselsobservations provided a basis for understanding all this. In keeping with their findings in experimental animals,it was also well known that individuals deprived of vision as adults, such as by accidental corneal scarring,retain the ability to see when treated by corneal transplantation, even if treatment is delayed for decades.The broader significance of this work for brain function is also readily apparent. If the visual system is areasonable guide to the development of the rest of the brain, then innate mechanisms establish the initial wiringof neural systems, but normal experience is needed to preserve, augment, and adjust the neural connectivitypresent at birth. In the case of abnormal experience, such as monocular deprivation, the mechanisms that enablethe normal maturation of connectivity are thwarted, resulting in anatomical and, ultimately, behavioral changesthat become increasingly hard to reverse as animals grow older. This gradually diminishing cortical plasticity aswe or other animals mature provides a neurobiological basis for the familiar observation that we learn anything(language, music, athletic skills, cultural norms) much better as children than as adults, and that behavior ismuch more susceptible to normal or pathological modification early in development than later. The implicationsof these further insights for early education, for learning and remediation at later stages of life, and for the legalpolicies are self-evident.Hubel and Wiesels extraordinary success (Figure 7.6) was no doubt the result of several factors. First, as theywere always quick to say, they were lucky enough to have come together as fellows in Kufflers lab shortlyafter he had determined the receptive field properties of neurons in the cat retina—the approach that, withKufflers encouragement, they pursued as Kuffler followed other interests (an act of generosity not often seenwhen mentors latch on to something important). Second, they were aware of and dedicated to the importance ofwhat they were doing; the experiments were difficult and often ran late into the night, requiring an uncommonwork ethic that their medical training helped provide. Finally, they respected and complemented each other asequal partners. Hubel was the more eccentric of the two, and I always found him somewhat daunting. He hadbeen an honors student in math and physics at McGill, and whether solving the Rubiks cube that was alwayslying around the lunchroom or learning how to program the seemingly incomprehensible PDP 11 computerthat he had purchased for the lab, he liked puzzles and logical challenges. He asked tough and highly originalquestions in seminars or lunchroom conversations and made everyone a little uneasy by taking snapshots witha miniature camera about the size of a cigarette lighter that he carried around. He was hard to talk to when Isought him out for advice as a postdoc, and I couldnt help feeling that his characterization of lesser lights as"chuckleheads" was probably being applied to me. These quirks aside, he is the neuroscientist I have mostadmired over the years.Figure 7.6 David Hubel and Torsten Wiesel talking to reporters in 1981, when they were awarded that yearsNobel Prize in Physiology or Medicine. (From Purves and Litchman, 1985)Although Wiesel shared Hubels high intelligence and dedication to the work they were doing, he wasotherwise quite different. Open and friendly with everyone, he had all the characteristics of the natural leader ofany collective enterprise. Torsten became the chair of the Department of Neurobiology at Harvard whenKuffler stepped down in 1973 and, after moving to Rockefeller University in 1983, was eventually appointedpresident there, a post he served in with great success from 1992 until his retirement in 1998 at the age of 74.In contrast, Hubel had been appointed chair of the Department of Physiology at Harvard in 1967, but he quitafter only a few months and returned to the Department of Neurobiology when he apparently discovered that
  • he did not want to handle all the problems that being a chair entails. (Other reasons might have contributed,based on the response of the Department of Physiology faculty to his managerial style, but if so, I never heardthem discussed.)This brief summary of what Hubel and Wiesel achieved gives some idea of why their influence on thetrajectory of "systems-level" neuroscience in the latter decades of the twentieth century was so great. Thewealth of evidence they amassed seemed to confirm Helmholtzs idea that perceptions are the result of theactivity of neurons that effectively detect and, in some sense, report and represent in the brain the variousfeatures of retinal images. This strategy seems eminently logical; any sensible engineer would presumably wantto make what we see correspond to the real-world features of the objects that we and other animals mustrespond to with visually guided behavior. This was the concept of vision that I took away from the course thatHubel and Wiesel taught us postdocs and students in the early 1970s. However, I should hasten to add thatfeature detection as an explicit goal of visual processing was never discussed. Hubel and Wiesel appeared toassume that understanding the receptive field properties of visual neurons would eventually explain perception,and that further discussion would be superfluous.In light of all this, it will seem odd that the rest of the book is predicated on the belief that these widely acceptedideas about how the visual brain works are wrong. The further conclusion that understanding what we seebased on learning more about the responses of visual neurons is likely to be a dead end might seem evenstranger. Several things conspired to sow seeds of doubt after years of enthusiastic, if remote, acceptance of theoverall program that Hubel and Wiesel had been pursuing. The first flaw was the increasing difficulty that theyand their many acolytes were having when trying to make sense of the electrophysiological and anatomicalinformation that had accumulated by the 1990s. In the early stages of their work, the results obtained seemed tobeautifully confirm the intuition that vision entails sequential and essentially hierarchical analyses of retinalimage features leading to the neural correlates of perception (see Figure 7.3). The general idea was that theluminance values, spectral distributions (colors), angles, line lengths, depth, motion, and other features wereabstracted by visual processing in the retina, thalamus, and primary visual cortex, and subsequentlyrecombined in increasingly complex ways by neurons at progressively higher stages in the visual cortex. Thesecombined representations in the extrastriate regions of the visual system would lead to the perception of objectsand their qualities by virtue of further activity elicited in the association cortices in the occipital lobes andadjacent areas in the temporal and parietal lobes.A particularly impressive aspect of Hubel and Wiesels observations in the 1960s and 1970s was that thereceptive field properties of the neurons in the lateral geniculate nucleus of the thalamus could nicely explainthe properties of the neurons they contacted in the input layer of the primary visual cortex, and that theproperties of these neurons could explain the responses of the neurons they contacted at the next higher level ofprocessing in V1. The neurons in this cortical hierarchy were referred to as "simple," "complex," and"hypercomplex" cells, underscoring the idea that the features abstracted from the retinal image wereprogressively being put back together in the cortex for the purpose of perception. Although I doubt Hubel andWiesel ever used the phrase, the rationale for the initial abstraction was generally assumed to be engineering orcoding efficiency.These findings also fit well with their anatomical evidence that V1 is divided into iterated modules defined byparticular response properties, such as selectivity for orientation (see Figure 7.4) or for information related tothe left or right eye (see Figure 6.2A). By the late 1970s, Hubel and Wiesel had put these several findingstogether in what they called the "ice cube" model of visual cortical processing (Figure 7.7). The suggestion wasthat each small piece of cortex, which they called a "hyercolumn," contained a complete set of feature-processing elements. But as the years passed and more evidence accumulated about visual neuronal types, theirconnectivity, and the organization of the visual system, the concept of a processing hierarchy in general and theice cube model in particular seemed as if a square peg was being pounded into a round hole.Figure 7.7 The ice cube model of primary visual cortical organization. This diagram illustrates the idea thatunits roughly a square millimeter or two in size (the primary visual cortex in each hemisphere of a rhesusmonkey brain is about 1,000 square millimeters) each comprise superimposed feature-processing elements,
  • illustrated here by orientation selectivity over the full range of possible angles (the little lines) and comappingwith right and left eye processing stripes (indicated by L and R; see Figure 6.2A). (After Hubbel, 1988)A second reason for suspecting that more data about the receptive field properties of visual neurons and theiranatomical organization might not explain perception was the mountain of puzzling observations about whatpeople actually see, coupled with philosophical concerns about vision that had been around for centuries.Taking such things seriously was a path that a self-respecting neuroscientist followed at some peril. But visionhas always demanded that perceptual and philosophical issues be considered, and the cracks that had begun toappear in the standard model of how the visual brain was supposed to work encouraged a reconsideration ofsome basic concerns. One widely discussed issue was the question of "grandmother cells," a term coined byJerry Lettvin, an imaginative and controversial neuroscientist at MIT who liked the role of intellectual and(during the Vietnam War era) social provocateur. If the features of retinal images were being progressively putback together in neurons with increasingly more complex properties at higher levels of the brain, didnt thisimply the existence of nerve cells that would ultimately be ludicrously selective (meaning neurons that wouldrespond to only the retinal image of your grandmother, for example)? Although the question was facetious,many people correctly saw it as serious. The ensuing debate was further stimulated by the discovery in theearly 1980s of neurons in the association areas of the monkey brain that did, in fact, respond specifically tofaces (an area in the human temporal lobe that responds selectively to faces has since been well documented).A related question concerned the binding problem. Even if visual neurons dont generate perceptions byspecifically responding to grandmothers or other particular objects (which most people agreed made littlesense), how are the various features of any object brought together in a coherent, instantaneously generatedperception of, for example, a ball that is round, chartreuse, and coming at you in a particular direction from acertain distance at a certain speed (think tennis). Although purported answers to the binding problem were (andstill are) taken with a grain of salt, most neuroscientists recognized that such questions would eventually needto be answered. Although a lot of my colleagues were not very interested in debates of this sort, I had alwayshad a weakness for them and was glad to see these issues raised as serious concerns in neuroscience. After all,I had been a philosophy major in college and had left clinical medicine because I wanted to understand how thebrain worked, not just how to understand its maladies or the properties of its constituent cells.By the mid-1990s, I began to be bothered by another philosophical issue relevant to perception that wasultimately decisive in reaching the conclusion that mining the details of visual neuronal properties would neverlead to an understanding of perception or its underlying mechanics. Western philosophy had long debatedabout how the "real world" of physical objects can be "known" by using our senses. Positions on this issuehad varied greatly, the philosophical tension in recent centuries being between thinkers such as Francis Baconand René Descartes, who supposed that absolute knowledge of the real world is possible (an issue of somescientific consequence in modern physics and cosmology), and others such as David Hume and ImmanuelKant, who argued that the real world is inevitably remote from us and can be appreciated only indirectly. Thephilosopher who made these points most cogently with respect to vision was George Berkeley, an Irishnobleman, bishop, tutor at Trinity College in Dublin, and card-carrying member of the British "EmpiricistSchool." In 1709, Berkeley had written a short treatise entitled An Essay Toward a New Theory of Vision inwhich he pointed out that a two-dimensional image projected onto the receptive surface of the eye could neverspecify the three-dimensional source of that image in the world (Figure 7.8). This fact and the difficulty itraises for understanding the perception of any image feature is referred to as the inverse optics problem.Figure 7.8 The inverse optics problem. George Berkeley pointed out in the eighteenth century that the sameprojected image could be generated by objects of different sizes, at different distances from the observer, and indifferent physical orientations. As a result, the actual source of any three-dimensional object is inevitablyuncertain. Note that the problem is not simply that retinal images are ambiguous; the deeper issue is that the realworld is directly unknowable by means of any logical operation on a projected image. (After Purves and Lotto,2003)In the context of biology and evolution, the significance of the inverse problem is clear: If the information onthe retina precludes direct knowledge of the real world, how is it that what we see enables us to respond sosuccessfully to real-world objects on the basis of vision? Helmholtz was aware of the problem and argued thatvision had to depend on learning from experience in addition to the information supplied by neural connections
  • in the brain determined by inheritance. However, he thought that analyzing image features was generally goodenough and that a boost from empirical experience (empirical experience, for him, was what we learn aboutobjects in life through trial-and-error interactions) would contend with the inverse problem. This learnedinformation would allow us to make what Helmholtz referred to as "unconscious inferences" about what anambiguous image might represent. Some vision scientists seemed to take Helmholtzs approach to the inverseoptics problem as sufficient, but many simply ignored it. The problem was rarely, if ever, mentioned in thediscussions of vision I had been party to over the years. In particular, I had never heard Hubel and Wieselmention it or saw it referred to in their papers.At the same time, I was increasingly aware in the 1990s, as anyone who delves into perception must be, of anenormous number of visual illusions. An illusion refers to a perception that fails to match a physicalmeasurement made by using an instrument of some sort: a ruler, a protractor, a photometer, or some morecomplex device that makes direct measurements of object properties, therefore evading the inverse problem. Inusing the term illusion the presumption in psychology texts and other literature is that we usually see the world"correctly," but sometimes a natural or contrived stimulus fools us so that our perception and the measuredreality underlying the stimulus fail to align. But if what Berkeley had said was right, analysis of a retinal imagecould not tell the brain anything definite about what objects and conditions in the world had actually generatedan image. It seemed more likely that all perceptions were equally illusory constructions produced by the brainto achieve biological success in the face of the inverse problem. If this was the case, then the evolution ofvisual systems must have been primarily concerned with solving this fundamental challenge. Surprisingly, noone seemed to be paying much attention to this very large spanner that Berkeley had tossed into logical andanalytical concepts of how vision works.I didnt have the slightest idea of how the visual wiring described by Hubel and Wiesel and their followersmight be contending with the inverse problem. But I was pretty sure that it must be by means of a verydifferent strategy from the one that had been explicitly or implicitly dominating my thinking (and mosteveryone elses) since the 1960s. If understanding brain function was going to be possible, exploring howvision contends with the inverse problem seemed a very good place to start.
  • Hearing Colors, Tasting Shapes People with synesthesia— whose senses blend together— are providing valuable clues to understanding the organization and functions of the human brain By Vilayanur S. Ramachandran and Edward M. Hubbard When Matthew Blakeslee shapes hamburger patties with his hands, he experiences a vivid bitter taste in his mouth. Esmerelda Jones (a pseudonym) sees blue when she listens to the note C sharp played on the piano; other notes evoke different hues— so much so that the piano keys are actually color-coded, making it easier for her to remember and play musical scales. And when Jeff Coleman looks at printed black numbers, he sees them in color, each a different hue. Blakeslee, Jones and Coleman are among a handful of otherwise normal as a child and the number 5 was red and 6 was green. This the- people who have synesthesia. They experience the ordinary ory does not answer why only some people retain such vivid world in extraordinary ways and seem to inhabit a mysterious sensory memories, however. You might think of cold when you no-man’s-land between fantasy and reality. For them the sens- look at a picture of an ice cube, but you probably do not feel es— touch, taste, hearing, vision and smell— get mixed up in- cold, no matter how many encounters you may have had with stead of remaining separate. ice and snow during your youth. Modern scientists have known about synesthesia since Another prevalent idea is that synesthetes are merely being 1880, when Francis Galton, a cousin of Charles Darwin, pub- metaphorical when they describe the note C flat as “red” or say lished a paper in Nature on the phenomenon. But most have that chicken tastes “pointy”— just as you and I might speak of brushed it aside as fakery, an artifact of drug use (LSD and a “loud” shirt or “sharp” cheddar cheese. Our ordinary lan- mescaline can produce similar effects) or a mere curiosity. guage is replete with such sense-related metaphors, and perhaps About four years ago, however, we and others began to un- synesthetes are just especially gifted in this regard. cover brain processes that could account for synesthesia. Along We began trying to find out whether synesthesia is a gen- the way, we also found new clues to some of the most mysteri- uine sensory experience in 1999. This deceptively simple ques- ous aspects of the human mind, such as the emergence of ab- tion had plagued researchers in this field for decades. One nat- stract thought, metaphor and perhaps even language. ural approach is to start by asking the subjects outright: “Is thisDAVID EMMITE A common explanation of synesthesia is that the affected just a memory, or do you actually see the color as if it were right people are simply experiencing childhood memories and asso- in front of you?” When we tried asking this question, we did ciations. Maybe a person had played with refrigerator magnets not get very far. Some subjects did respond, “Oh, I see it per- www.sciam.com SCIENTIFIC AMERICAN 53
  • fectly clearly.” But a more frequent reac- with volunteers, the answer was crystal information about these separate fea- tion was, “I kind of see it, kind of don’t” clear. Unlike normal subjects, synesthetes tures is sent forward and distributed to or “No, it is not like a memory. I see the correctly reported the shape formed by several far-flung regions in the temporal number as being clearly red but I also groups of numbers up to 90 percent of the and parietal lobes. In the case of color, know it isn’t; it’s black. So it must be a time (exactly as nonsynesthetes do when the information goes to area V4 in the memory, I guess.” the numbers actually have different col- fusiform gyrus of the temporal lobe. To determine whether an effect is tru- ors). This result proves that the induced From there it travels to areas that lie far- ly perceptual, psychologists often use a colors are genuinely sensory and that ther up in the hierarchy of color centers, simple test called pop-out or segregation. synesthetes are not just making things up. including a region near a patch of cortex If you look at a set of tilted lines scattered It is impossible for them to fake their suc- called the TPO (for the junction of the amid a forest of vertical lines, the tilted cess. In another striking example, we temporal, parietal and occipital lobes). lines stand out. Indeed, you can instantly asked a synesthete who sees 5 tinged red These higher areas may be concerned segregate them from the background and to watch a computer display. He could with more sophisticated aspects of color group them mentally to form, for exam- not tell when we surreptitiously added an processing. For example, leaves look as ple, a separate triangular shape. Similar- actual red hue to the white number unless green at dusk as they do at midday, even ly, if most of a background’s elements the red was sufficiently intense; he could though the mix of wavelengths reflected were green dots and you were told to look instantly spot a real green added to the 5. from the leaves is very different. for red targets, the reds would pop out. Numerical computation, too, seems to On the other hand, a set of black 2’s scat- Visual Processing happen in stages. An early step also takes tered among 5’s of the same color almost CONFIRMATION THAT synesthesia is place in the fusiform gyrus, where the ac- blend in [see illustration on page 57]. It is real brings up the question, Why do some tual shapes of numbers are represented, hard to discern the 2’s without engaging people experience this weird phenome- and a later one occurs in the angular gyrus, in an item-by-item inspection of numbers, non? Our experiments lead us to favor a part of the TPO that is concerned with even though any individual number is just the idea that synesthetes are experiencing numerical concepts such as ordinality (se- as clearly different from its neighbors as a the result of some kind of cross wiring in quence) and cardinality (quantity). (When tilted line is from a straight line. We thus the brain. This basic concept was initial- the angular gyrus is damaged by a stroke may conclude that only certain primitive, ly proposed about 100 years ago, but we or a tumor, the patient can still identify or elementary, features, such as color and have now identified where in the brain numbers but can no longer divide or sub- line orientation, can provide a basis for and how such cross wiring might occur. tract. Multiplication often survives be- grouping. More complex perceptual to- An understanding of the neurobio- cause it is learned by rote.) In addition, kens, such as numbers, cannot do so. logical factors at work requires some fa- brain-imaging studies in humans strong- We wondered what would happen if miliarity with how the brain processes vi- ly hint that visually presented letters of we showed the mixed numbers to synes- sual information [see illustration on op- the alphabet or numbers (graphemes) ac- thetes who experience, for instance, red posite page]. After light reflected from a tivate cells in the fusiform gyrus, where- when they see a 5 and green with a 2. We scene hits the cones (color receptors) in as the sounds of the syllables (phonemes) arranged the 2’s so that they formed a tri- the eye, neural signals from the retina are processed higher up, once again in the angle. If synesthesia were a genuine sen- travel to area 17, in the occipital lobe at general vicinity of the TPO. sory effect, our subjects should easily see the back of the brain. There the image is Because both colors and numbers are the triangle because for them, the num- processed further within local clusters, or processed initially in the fusiform gyrus bers would look colored. blobs, into such simple attributes as col- and subsequently near the angular gyrus, When we conducted pop-out tests or, motion, form and depth. Afterward, we suspected that number-color synesthe- sia might be caused by cross wiring be-Overview/Synesthesia tween V4 and the number-appearance area (both within the fusiform) or be- ■ Synesthesia (from the Greek roots syn, meaning “together,” and aisthesis, or tween the higher color area and the num- “perception”) is a condition in which otherwise normal people experience the ber-concept area (both in the TPO). Oth- blending of two or more senses. er, more exotic forms of the condition ■ For decades, the phenomenon was often written off as fakery or simply might result from similar cross wiring of memories, but it has recently been shown to be real. Perhaps it occurs because different sensory-processing regions. That of cross activation, in which two normally separate areas of the brain elicit the hearing center in the temporal lobes activity in each other. is also close to the higher brain area that ■ As scientists explore the mechanisms involved in synesthesia, they are also receives color signals from V4 could ex- learning about how the brain in general processes sensory information and plain sound-color synesthesia. Similarly, uses it to make abstract connections between seemingly unrelated inputs. Matthew Blakeslee’s tasting of touch might occur because of cross wiring be- 54 SCIENTIFIC AMERICAN MAY 2003
  • MINGLED SIGNALS IN ONE OF THE MOST COMMON FORMS of synesthesia, looking at a number evokes a specific hue. This apparently occurs because brain areas that normally do not interact when processing numbers or colors do activate each other in synesthetes. NEURAL SIGNALS from the retina travel via optic radiation to area 17, in the rear of the brain, where TPO JUNCTION they are broken into simple PARIETAL LOBE shared attributes such as color, form, motion and depth. AREA 17 Color information continues on to V4, near where the visual appearance of numbers is also represented— and thus is a site for cross-linking between the color OPTIC NERVE and number areas (short pink and green arrows). Ultimately, color proceeds “higher,” to an area near the TPO RETINA (for temporal, parietal, occipital ION lobes) junction, which may R ADIAT OPTIC perform more sophisticated color processing. Similarly, a later stage of numerical computation LIGHT V4 occurs in the angular gyrus, a part of the TPO concerned with OCCIPITAL LOBE the concepts of sequence and quantity. This could explain NUMBER- synesthesia in people who link TEMPORAL LOBE APPEARANCE AREA colors with abstract numerical sequences, like days of the week. REAR VIEW of a synesthete’s brain, made with functional magnetic resonance imaging, shows high activity (yellow) in the V4 color-processing area as the subject looks at white numbers on a gray background. This area is not active V4CAROL DONNER (illustration); GEOFF BOYNTON Salk Institute for Biological Studies in people with normal perception viewing the same figures. tween the taste cortex in a region called bers whereas others see colors when they by blocking the action of an inhibitory the insula and an adjacent cortex repre- hear phonemes or musical notes. People neurotransmitter or failing to produce anAND VILAYANUR S. RAMACHANDRAN AND EDWARD M. HUBBARD (inset) senting touch by the hands. who have one type of synesthesia are more inhibitor— would also cause activity in Assuming that neural cross wiring likely to have another, which adds weight one area to elicit activity in a neighbor. does lie at the root of synesthesia, why to this idea. Such cross activation could, in theory, also does it happen? We know that it runs in Although we initially thought in terms occur between widely separated areas, families, so it has a genetic component. of physical cross wiring, we have come to which would account for some of the less Perhaps a mutation causes connections to realize that the same effect could occur if common forms of synesthesia. emerge between brain areas that are usu- the wiring— the number of connections Support for cross activation comes ally segregated. Or maybe the mutation between regions—was fine but the balance from other experiments, some of which leads to defective pruning of preexisting of chemicals traveling between regions also help to explain the varied forms connections between areas that are nor- was skewed. So we now speak in terms of synesthesia can take. One takes advan- mally connected only sparsely. If the mu- cross activation. For instance, neighboring tage of a visual phenomenon known as tation were to be expressed (that is, to ex- brain regions often inhibit one another’s crowding [see illustration on opposite ert its effects) in some brain areas but not activity, which serves to minimize cross page]. If you stare at a small plus sign in others, this patchiness might explain why talk. A chemical imbalance of some kind an image that also has a number 5 off to some synesthetes conflate colors and num- that reduces such inhibition—for example, one side, you will find that it is easy to dis- www.sciam.com SCIENTIFIC AMERICAN 55
  • cern that number, even though you are at the display and made remarks like, “I background, the synesthetic color be-not looking at it directly. But if we now cannot see the middle number. It’s fuzzy came weaker until, at low contrast, sub-surround the 5 with four other numbers, but it looks red, so I guess it must be a 5.” jects saw no color at all, even though thesuch as 3’s, then you can no longer iden- Even though the middle number did not number was perfectly visible. Whereastify it. It looks out of focus. Volunteers consciously register, it seems that the brain the crowding experiment shows that anwho perceive normally are no more suc- was nonetheless processing it somewhere. invisible number can elicit color, the con-cessful at identifying this number than Synesthetes could then use this color to de- trast experiment conversely indicates thatmere chance. That is not because things duce intellectually what the number was. viewing a number does not guaranteeget fuzzy in the periphery of vision. After If our theory is right, this finding implies seeing a color. Perhaps low-contrast num-all, you could see the 5 perfectly clearly that the number is processed in the bers activate cells in the fusiform ade-when it wasn’t surrounded by 3’s. You fusiform gyrus and evokes the appropri- quately for conscious perception of thecannot identify it now because of limited ate color before the stage at which the number but not enough to cross-activateattentional resources. The flanking 3’s crowding effect occurs in the brain; para- the color cells in V4.somehow distract your attention away doxically, the result is that even an “in- Finally, we found that if we showedfrom the central 5 and prevent you from visible” number can produce synesthesia. synesthetes Roman numerals, a V, say,seeing it. Another finding we made also sup- they saw no color— which suggests that it A big surprise came when we gave the ports this conclusion. When we reduced is not the numerical concept of a number,same test to two synesthetes. They looked the contrast between the number and the in this case 5, but the grapheme’s visual appearance that drives the color. This ob- VILAYANUR S. RAMACHANDRAN and EDWARD M. HUBBARD collaborate on studies of synes- servation, too, implicates cross activationTHE AUTHORS thesia. Ramachandran directs the Center for Brain and Cognition at the University of Califor- within the fusiform gyrus itself in num- nia at San Diego and is adjunct professor at the Salk Institute for Biological Studies. He trained ber-color synesthesia, because that struc- as a physician and later obtained a Ph.D. from Trinity College, University of Cambridge. He has ture is mainly involved in analyzing the vi- received a fellowship from All Souls College, University of Oxford, the Ariens Kappers Gold Medal sual shape, not the high-level meaning of from the Royal Netherlands Academy, and the plenary lecture award from the American Acad- the number. One intriguing twist: Imag- emy of Neurology. He gave the BBC Reith Lectures for 2003. This is his fourth article for Sci- ine an image with a large 5 made up of lit- entific American. Hubbard is a fourth-year graduate student in the departments of psycholo- tle 3’s; you can see either the “forest” (the gy and cognitive science at U.C.S.D. His research combines psychophysics and functional mag- 5) or focus minutely on the “trees” (the DAVID EMMITE netic resonance imaging to explore the neural basis of multisensory phenomena. A founding 3’s). Two synesthete subjects reported member of the American Synesthesia Association, he helped to organize its second annual that they saw the color switch, depending meeting at U.C.S.D. in 2001. on their focus. This test implies that even56 SCIENTIFIC AMERICAN MAY 2003
  • though synesthesia can arise as a result of COLOR-CODED WORLD the visual appearance alone— not the high-level concept— the manner in which IN A TEST of visual-segregation capabilities, synesthetes who link a specific hue the visual input is categorized, based on with a given number can instantly see an embedded pattern in an image with black attention, is also critical. numbers scattered on a white page. Whereas a person with normal perception But as we began to recruit other vol- must undertake a digit-by-digit search to pick out, in this example, 2’s amid 5’s unteers, it soon became obvious that not (left), the triangle-shaped group of 2’s pops out for a synesthete (right). all synesthetes who colorize their world are alike. In some, even days of the week or months of the year elicit colors. Mon- day might be green, Wednesday pink, and December yellow. The only thing that days of the week, months and numbers have in common is the concept of numerical sequence, or or- dinality. For certain synesthetes, perhaps it is the abstract concept of numerical se- quence that drives the color, rather than the visual appearance of the number. “INVISIBLE” NUMBERS show up for synesthetes in a perceptual test. When a person Could it be that in these individuals, the stares at a central object, here a plus sign, a single digit off to one side is easy to cross wiring occurs between the angular see with peripheral vision (left). But if the number is surrounded by others (right), gyrus and the higher color area near the it appears blurry— invisible— to the average person. In contrast, a synesthete could TPO instead of between areas in the deduce the central number by the color it evokes. fusiform? If so, that interaction would explain why even abstract number rep- resentations, or the idea of the numbers elicited by days of the week or months, will strongly evoke specific colors. In oth- er words, depending on where in the brain the mutant gene is expressed, it can result in different types of the condition— “higher” synesthesia, driven by numerical concept, or “lower” synesthesia, pro- duced by visual appearance alone. Simi- larly, in some lower forms, the visual ap- pearance of a letter might generate color, activation theory of synesthesia. (Jeffrey and novelists. According to one study, the whereas in higher forms it is the sound, or Gray of the Institute of Psychiatry in Lon- condition is seven times as common in cre- phoneme, summoned by that letter; pho- don and his colleagues have reported sim- ative people as in the general population. nemes are represented near the TPO. ilar results.) On presenting black and One skill that many creative people We also observed one case in which white numbers to synesthetes, brain acti- share is a facility for using metaphor (“It we believe cross activation enables a color- vation arose not only in the number is the east, and Juliet is the sun”). It is as blind synesthete to see numbers tinged area—as it would in normal subjects—but if their brains are set up to make links be- with hues he otherwise cannot perceive; also in the color area. Our group also ob- tween seemingly unrelated domains— charmingly, he refers to these as “Mar- served differences between types of synes- such as the sun and a beautiful young tian colors.” Although his retinal color thetes. One of our subjects with lower woman. In other words, just as synesthe- receptors cannot process certain wave- synesthesia showed much greater activa- sia involves making arbitrary links be- lengths, we suggest that his brain color tion in earlier stages of color processing tween seemingly unrelated perceptual en- area is working just fine and being cross- than occurred in controls. In contrast, tities such as colors and numbers, meta-VILAYANUR S. RAMACHANDRAN activated when he sees numbers. higher synesthetes show less activation at phor involves making links between In brain-imaging experiments we are these earlier levels. seemingly unrelated conceptual realms. conducting with Geoff Boynton of the Perhaps this is not just a coincidence. Salk Institute for Biological Studies in San A Way with Metaphor Numerous high-level concepts are Diego, we have obtained preliminary ev- O U R I N S I G H T S into the neurological probably anchored in specific brain re- idence of local activation of the color area basis of synesthesia could help explain gions, or maps. If you think about it, there V4 in a manner predicted by our cross- some of the creativity of painters, poets is nothing more abstract than a number, www.sciam.com SCIENTIFIC AMERICAN 57
  • COMMON QUESTIONSAre there different types of synesthesia?Science counts more than 100. The condition and yet it is represented, as we have seen,runs in families and may be more common in in a relatively small brain region, the an- gular gyrus. Let us say that the mutation Synesthesia may provide somewomen and creative people; perhaps one we believe brings about synesthesia caus- insights about the evolution ofperson in 200 has synesthesia. In the most es excess communication among differ- thought and languageprevalent type, looking at numbers or listeningto tones evokes colors. In one rare kind, each ent brain maps— small patches of cortex that represent specific perceptual entities, IMAGINE A BAND of ancestral hominids about toletter is associated with the male or female such as sharpness or curviness of shapes invent language. Clearly, they did not begin bysex— an example of the brain’s tendency to or, in the case of color maps, hues. De- having a leader say, “Hey, look at this— let’s callsplit the world into binary categories. pending on where and how widely in the it a banana. All of you say after me, ba-na-na.” brain the trait was expressed, it could Undoubtedly, though, the group had a set ofIf a synesthete associates a color with a lead to both synesthesia and to a propen- capacities that prepared the ground forsingle letter or number, what happens if he sity toward linking seemingly unrelated systematic verbal communication. Our studieslooks at a pair of letters, such as “ea,” or concepts and ideas— in short, creativity. of the neurobiological basis of synesthesiadouble digits, as in “25”? This would explain why the apparently suggest that a facility for metaphor— for seeingHe sees colors that correspond with the useless synesthesia gene has survived in deep links between superficially dissimilar andindividual letters and numbers. If the letters or the population. unrelated things— provided a key seed for thenumbers are too close physically, however, In addition to clarifying why artists eventual emergence of language.they may cancel each other out (color might be prone to experiencing synesthe- Humans have a built-in bias to associatedisappears) or, if the two happen to elicit the sia, our research suggests that we all have certain sounds with particular visual shapes,same color, enhance each other. some capacity for it and that this trait which could well have been important in getting may have set the stage for the evolution of hominids started on a shared vocabulary. InDoes it matter whether letters are abstraction— an ability at which humans addition, specific brain areas that process visualuppercase or lowercase? excel. The TPO (and the angular gyrus shapes of objects, letters and numbers, andIn general, no. But people have sometimes within it), which plays a part in the con- word sounds can activate each other even indescribed seeing less saturated color in dition, is normally involved in cross- nonsynesthetes, causing people to expect, say,lowercase letters, or the lowercase letters modal synthesis. It is the brain region jagged shapes to have harsh-sounding names.may appear shiny or even patchy. where information from touch, hearing Two other types of neural connections and vision is thought to flow together to support our idea. First, the sensory areas forHow do entire words look? enable the construction of high-level per- visual shapes and for hearing in the back of theOften the color of the first letter spreads ceptions. For example, a cat is fluffy brain can cross-activate specific motor areas inacross the word; even silent letters, such as (touch), it meows and purrs (hearing), it the front of the brain that participate in speech.the “p” in “psalm,” cause this effect. has a certain appearance (vision) and A sharp visual inflection or a harsh sound odor (smell), all of which are derived si- induces the motor control area for speech toWhat if the synesthete is multilingual?One language can have colored graphemes, multaneously by the memory of a cat orbut a second (or additional others) may not, the sound of the word “cat.”perhaps because separate tongues are Could it be that the angular gyrus— the lips as they produce the curved “boo-represented in different brain regions. which is disproportionately larger in hu- baa” sound. In contrast, the waveform of mans compared with that in apes and the sound “kiki” and the sharp inflectionWhat about when the person mentally monkeys— evolved originally for cross- of the tongue on the palate mimic thepictures a letter or number? modal associations but then became co- sudden changes in the jagged visualImagining can evoke a stronger color than opted for other, more abstract functions shape. The only thing these two kiki fea-looking at a real one. Perhaps that exercise such as metaphors? Consider two draw- tures have in common is the abstractactivates the same brain areas as does ings, originally designed by psychologist property of jaggedness that is extractedviewing real colors— but because no Wolfgang Köhler. One looks like an ink- somewhere in the vicinity of the TPO,competing signals from a real number are blot and the other, a jagged piece of shat- probably in the angular gyrus. (We re-coming from the retina, the imagined one tered glass. When we ask, “Which of these cently found that people with damage tocreates a stronger synesthetic color. is a ‘bouba,’ and which is a ‘kiki’?” 98 per- the angular gyrus lose the bouba-kiki ef- cent of people pick the inkblot as a bouba fect— they cannot match the shape withDoes synesthesia improve memory? and the other one as a kiki. Perhaps that is the correct sound.) In a sense, perhaps weIt can. The late Russian neurologist because the gentle curves of the amoeba- are all closet synesthetes.Aleksandr R. Luria described a mnemonist like figure metaphorically mimic the gen- So the angular gyrus performs a verywho had remarkable recall because all of his tle undulations of the sound “bouba” as elementary type of abstraction— extract-five senses were linked. Even having two represented in the hearing centers in the ing the common denominator from a setlinked senses may help. — V.S.R. and E.M.H. brain as well as the gradual inflection of of strikingly dissimilar entities. We do MAY 2003
  • THE PUZZLE OF LANGUAGE produce an equally sudden inflection of the IF ASKED which of the two figures below is a “bouba” and which is a “kiki,” 98 tongue on the palate (or consider the spoken percent of all respondents choose the blob as a bouba and the other as a kiki. The authors argue that the brain’s ability to pick out an abstract feature in common— words “diminutive,” “teeny-weeny” and “un such as a jagged visual shape and a harsh-sounding name—could have paved peu,” which involve pursing the lips to mimic the way for the development of metaphor and perhaps even a shared vocabulary. the small size of the object. The brain seems to possess preexisting rules for translating what we see and hear into mouth motions that reflect those inputs. Second, a kind of spillover of signals occurs between two nearby motor areas: those that control the sequence of muscle movements required for hand gestures and those for the mouth. We call this effect “synkinesia.” As Charles Darwin pointed out, when we cut paper with scissors, our jaws may clench and unclench unconsciously as if to echo the hand movements. Many linguists gestures were translated through synkinesia do not like the theory that manual gesturing into movements of the mouth and face could have set the stage for vocal language, muscles, and if emotional guttural but we believe that synkinesia suggests that utterances were channeled through these they may be wrong. mouth and tongue movements, the result psychologist Patricia Greenfield of the Assume that our ancestral hominids could have been the first spoken words. University of California at Los Angeles, we communicated mainly through emotional How would we import syntax, the rules propose that frontal brain areas that evolved grunts, groans, howls and shrieks, which are for using words and phrases in language, into for subassembly in tool use may later have known to be produced by the right hemisphere this scheme? We believe that the evolution been co-opted for a completely novel and an area in the frontal lobes concerned of tool use by hominids may have played an function— joining words into phrases and with emotion. Later the hominids developed a important role. For example, the tool- sentences. rudimentary gestural system that became building sequence— first shape the Not every subtle feature of modern gradually more elaborate and sophisticated; hammer’s head, then attach it to a handle, language is explained by such schemes, but we it is easy to imagine how the hand movement then chop the meat— resembles the suspect that these elements were critical for for pulling someone toward you might have embedding of clauses within larger setting in motion the events that culminated progressed to a “come hither” wave. If such sentences. Following the lead of in modern language. — V.S.R. and E.M.H. not know how exactly it does this job. A broadcast version of this article will air But once the ability to engage in cross- modal abstraction emerged, it might April 24 on National Geographic Today, have paved the way for the more com- a program on the National Geographic plex types of abstraction. The oppor- Channel. Please check your local listings. tunistic takeover of one function for a different one is common in evolution. For example, bones in the ear used for MORE TO E XPLORE hearing in mammals evolved from the The Man Who Tasted Shapes. R. E. Cytowic. MIT Press, 1993. back of the jawbone in reptiles. Beyond Synaesthesia: Classic and Contemporary Readings. S. Baron-Cohen and J. E. Harrison. metaphor and abstract thinking, cross- Blackwell, 1997. Psychophysical Investigations into the Neural Basis of Synaesthesia. V. S. Ramachandran and modal abstraction might even have pro- E. M. Hubbard in Proceedings of the Royal Society of London, B, Vol. 268, pages 979–983; 2001.VILAYANUR S. RAMACHANDRAN vided seeds for language [see box above]. Synaesthesia: A Window into Perception, Thought and Language. V. S. Ramachandran and When we began our research on E. M. Hubbard in Journal of Consciousness Studies, Vol. 8, No. 12, pages 3–34; 2001. synesthesia, we had no inkling of where Synaesthetic Photisms Influence Visual Perception. D. Smilek, M. J. Dixon, C. Cudahy and M. Merikle it would take us. Little did we suspect in Journal of Cognitive Neuroscience, Vol. 13, No. 7, pages 930–936; 2001. Functional Magnetic Resonance Imaging of Synesthesia: Activation of V4/V8 by Spoken Words. that this eerie phenomenon, long regard- J. A. Nunn, L. J. Gregory, M. Brammer, S.C.R. Williams, D. M. Parslow, M. J. Morgan, R. G. Morris, ed as a mere curiosity, might offer a win- E. T. Bullmore, S. Baron-Cohen and J. A. Gray in Nature Neuroscience, Vol. 5, pages 371–375; 2002. dow into the nature of thought. For more on synesthetia, visit www.sciam.com/ontheweb www.sciam.com SCIENTIFIC AMERICAN 59
  • Language regions of brain are operativein color perceptionWai Ting Sioka,b, Paul Kayc,d,1, William S. Y. Wange, Alice H. D. Chana,b, Lin Chenf, Kang-Kwong Lukea,b,and Li Hai Tana,b,1aDepartment of Linguistics and bState Key Laboratory of Brain and Cognitive Sciences, University of Hong Kong, Pokfulam Road, Hong Kong, China;cDepartment of Linguistics, University of California, Berkeley, CA 94720; dInternational Computer Science Institute, 1947 Center Street, Berkeley, CA 94704;eLanguage Engineering Laboratory, Department of Electronic Engineering, Chinese University of Hong Kong, Shatin, Hong Kong, China; and fState KeyLaboratory of Brain and Cognitive Science, Institute of Biophysics, Chinese Academy of Sciences, Beijing 100101, ChinaContributed by Paul Kay, April 2, 2009 (sent for review February 11, 2009)The effect of language on the categorical perception of color is response to colors. A subsequent study (9) with different tasksstronger for stimuli in the right visual field (RVF) than in the left extended this result and showed stronger category effects (i.e.,visual field, but the neural correlates of the behavioral RVF advan- faster responses to between-category color pairs than to within-tage are unknown. Here we present brain activation maps reveal- category color pairs) in the RVF than in the left visual fielding how language is differentially engaged in the discrimination of (LVF), although the LVF did show a significant, if weaker,colored stimuli presented in either visual hemifield. In a rapid, category effect. A third study (12), testing a color term boundaryevent-related functional MRI study, we measured subjects’ brain in Korean that does not exist in English, found CP only in theactivity while they performed a visual search task. Compared with RVF for relatively rapidly responding subjects but CP in bothcolors from the same lexical category, discrimination of colors from visual fields for slowly responding subjects and no CP at thedifferent linguistic categories provoked stronger and faster re- Korean-only boundary for English-speaking subjects. The au- PSYCHOLOGYsponses in the left hemisphere language regions, particularly when thors of that study suggest that LVF color CP in slower-the colors were presented in the RVF. In addition, activation of responding adults probably reflects cross-callosal transfer; thevisual areas 2/3, responsible for color perception, was much stron- same conclusion has been drawn elsewhere (14, 23). Hence it isger for RVF stimuli from different linguistic categories than for possible that in normal adults, color CP is restricted to the leftstimuli from the same linguistic category. Notably, the enhanced hemisphere, with apparent LVF CP an artifact of transcallosalactivity of visual areas 2/3 coincided with the enhanced activity of transfer and/or scanning.the left posterior temporoparietal language region, suggesting Despite growing behavioral evidence for hemifield-dependent NEUROSCIENCEthat this language region may serve as a top-down control source category effects, the neural correlates of these effects remainthat modulates the activation of the visual cortex. These findings unknown. One previous functional MRI (fMRI) study (24)shed light on the brain mechanisms that underlie the hemifield- found that, in comparison with hard-to-name colors, perceptualdependent effect of language on visual perception. discrimination of easy-to-name colors evoked stronger activation in the posterior temporoparietal regions responsible for success-functional magnetic resonance imaging (fMRI) lateralization ful word-finding processes, but the study was not designed to look into neural substrates of the behavioral RVF superiority in colorA typically viewed scene permits multiple visual parses, some of which can be readily mapped onto linguistic terms,whereas others cannot. Does linguistic information play a role in perception, and it did not clarify whether linguistic information aids in the activity of brain regions responsible for color vision. In the current rapid event-related fMRI study, we investigatedvisual perception? For more than half a century, this question neural mechanisms underlying hemifield-modulated Whorfianhas provoked controversy. According to the hypothesis proposed effects in adults. We scanned subjects’ brain activity while theyby Benjamin Lee Whorf (1), by filtering perception, language performed the visual search task used in the original lateralizedaffects our apprehension of the world. This hypothesis has Whorf study (7). The search included colors selected from a setreceived conflicting evidence (2–21); a recent review favors the of 4 (Fig. 1A). These 4 colors form a graded series from greenview that linguistic categories filter some, but not all, perceptual to blue, with the green blue boundary falling between G2 andinputs and that perceptual factors influence, but do not exclu- B1. In the visual search task, each stimulus display consisted ofsively determine, linguistic categories of color (22). a ring of colored squares surrounding a central fixation marker Recent neuropsychological investigations examining visual (Fig. 1B). Except the target, all the squares were of the samefield asymmetries in the categorical perception (CP) of colors color. The target and distractor colors were either from withinhave provided a new perspective on Whorfian effects. In a study the same lexical category (e.g., a blue target and distractors ofusing a visual search task (7), adult English speakers were a different shade of blue, ‘‘within category’’) or from differentrequired to detect a single target color among 11 identical lexical categories (e.g., a green target and blue distractors,distractor colors. Response times for finding the target were ‘‘between category’’). On each trial, participants were asked tofaster when target and distractors were from 2 different lexical indicate whether the target was on the left or right side of thecategories (e.g., a green target among blue distractors) than circle by making timed button-press responses with the corre-when target and distractors were from the same lexical category sponding hand. In this manner, 2 variables were manipulated:(e.g., a particular green among distractors of a different green),but only when the target was exposed in the right visual field Author contributions: W.T.S., P.K., W.S.Y.W., A.H.D.C., L.C., K.-K.L., and L.H.T. designed(RVF). Because the RVF projects to the left cerebral hemi- research; W.T.S., A.H.D.C., and L.H.T. performed research; W.T.S., A.H.D.C., and L.H.T.sphere, the dominant hemisphere for language in most adults, analyzed data; and W.T.S., P.K., A.H.D.C., and L.H.T. wrote the paper.and because the effect was eliminated by a concurrent task Conflict of interest: The authors declare no conflict of interest. .occupying verbal processing resources but not by an equally 1To whom correspondence should be addressed. E-mail: paulkay@berkeley.edu ordifficult task occupying nonverbal resources, the RVF CP find- tanlh@hku.hk.ing suggests that the spontaneous use of lexical codes in the left This article contains supporting information online at www.pnas.org/cgi/content/full/hemisphere may be the origin of the differential visual hemifield 0903627106/DCSupplemental.www.pnas.org cgi doi 10.1073 pnas.0903627106 PNAS Early Edition 1 of 6
  • A B C 540 LVF 520 RVF 500 * 1 3 480 + 460 G1 G2 B1 B2 2 4 440 “green” “blue” 420 400 Between-category Within-categoryFig. 1. Experimental materials and behavioral results. (A) Printed-rendered versions of the 4 colors used. (B) Sample display for the visual search task. The targetoccupied any of the 4 positions (position 1, 2, 3, or 4). This example shows a between-category, LVF pair. (C) Behavioral performance in the 4 conditions. Errorbars indicate SEM. *, significant difference in response (P 0.05).the visual field of the target (LVF vs. RVF) and the categorical fMRI Results. We first calculated an average effect of colorrelationship between the target and distractor colors (between- perception tasks by collapsing and contrasting all of the colorcategory vs. within-category). There were 2 types of target- conditions (LVF within-category, LVF between-category, RVFdistractor pairs: 1-step within-category (G1G2 and B1B2) and within-category, and RVF between-category) against an implicit1-step between-category (G2,B1). baseline available in the fast event-related fMRI design (Fig. 2 We tested 2 predictions of the Whorf hypothesis. First, if and Table 1). Consistent with previous neuroimaging studies oflexical codes of colors are accessed during visual search, dis- color vision (24–31), subjects showed strong activations in thecrimination of colors should evoke activations of cortical regions neural circuitry attributed to color perception, including V2/3contributing to language processes, such as left temporoparietal and V4 bilaterally. The left temporoparietal areas known toareas and the left inferior prefrontal gyrus. Furthermore, activ- mediate lexical processes were activated also. Bilateral inferiority levels of these language regions should be stronger for parietal cortex and motor cortex also showed strong activity,between-category than within-category stimuli, especially in the presumably because of motor responses required by the visualRVF, as predicted by previous behavioral studies (7, 9, 12). search task.Second, if lexical information enhances the perceptual differ- The main effect of categorical relationship (between-categoryence rather than merely being accessed as a byproduct of color versus within-category pairs) was computed by collapsing theidentification, activations of brain regions for color perception, data from the 2 visual fields. As depicted in Fig. 3 (Table 2),such as visual area 2/3 (V2/3) and visual area 4 (V4), should be several language areas involving the left posterior temporopa-altered by the activation of linguistic information, particularly in rietal region [Brodmann areas (BA) 40 and 39], the left middle-the RVF condition. superior temporal gyrus (BA 21 and 22), and the left inferiorResultsBehavior. Trials in which the participant pressed the wrong key or Ain which the reaction time (RT) was 2 SD from the grand meanwere excluded. Two participants’ behavioral data were dis-carded, 1 because of head motion during the brain scan and theother because button responses were recorded inaccurately. Asillustrated in Fig. 1C, with regard to main effects, between-category RTs were significantly faster than within-category RTs[468.80 ms vs. 507.89 ms, F (1, 13) 27.24, P 0.001], and RVFRTs were faster than LVF RTs at a level approaching signifi-cance [481.6 ms vs. 495.09 ms, F (1, 13) 3.41, P 0.088]. The Binteraction of the 2 variables also approached significance [F (1,13) 3.62, P 0.079], with RVF between-category RTs beingthe shortest. For between-category pairs, RVF RTs were signif-icantly faster than LVF RTs (458.9 ms vs. 478.69 ms, t 2.73, P -16 -8 -2 6 120.05). For within-category pairs, LVF RTs were faster by a scant7 ms, not approaching significance (511.497 ms vs. 504.29 ms, t 50.08, P 0.423). For RVF targets, RTs in the between-categorycondition were 45 ms faster than in the within-category condi- ztion (t 5.68, P 0.001). For LVF targets, RTs in thebetween-category condition were 33 ms faster than in the 18 24 32 40 0within-category condition (t 3.914, P 0.005). In general, this Fig. 2. An average effect map of color discrimination tasks. Data from all ofpattern of behavioral data is consistent with previous studies the color conditions (LVF within-category, LVF between-category, RVF within-using the same (7) or similar (9, 12) paradigms, suggesting that category, and RVF between-category) were collapsed. (A) Lateral view. (B)the color CP effects for normal language users are stronger in the Axial sections. The significance threshold is P 0.05 FDR-corrected. L, leftRVF than the LVF (i.e., lateralized Whorf). hemisphere; R, right hemisphere.2 of 6 www.pnas.org cgi doi 10.1073 pnas.0903627106 Siok et al.
  • Table 1. Coordinates of activation peaks: An average effectof color discrimination tasks A B 4.5 Coordinates BrodmannRegions activated area X Y Z Z-ScoreOccipitalLeft V4 28 69 12 5.71 L z -13 -7Left V4 alpha 36 50 19 6.60 46 59 17 6.31Right V4 26 69 13 6.05Right V4 alpha 46 59 19 5.77V1 4 72 4 5.78 0 72 1 5.53 0V2/3 28 84 21 6.60 R 21 31 28 86 25 5.84 Fig. 3. A main effect map of categorical relationship (between-category and 16 64 9 5.54 within-category colors). Data from the 2 visual hemifields were collapsed. (A)Frontal Lateral view. (B) Axial sections. The significance threshold is P 0.05 FDR-Left inferior frontal gyrus 44 44 5 27 3.88 corrected. L, left hemisphere; R, right hemisphere.Left middle frontal gyrus 6 36 3 57 5.09 9 53 9 33 3.10 10 30 53 19 3.65 tant is the significant difference in response delay between colorsLeft precentral gyrus 4 51 13 52 5.16 in the same lexical category and colors in differing lexical 6 32 18 64 5.16 categories, as illustrated in an averaged response delay differ- PSYCHOLOGYLeft postcentral gyrus 2 48 25 44 5.39 ence map (Fig. 4C). On a voxelwise basis, hemodynamic re-Right inferior frontal gyrus 44 48 11 29 4.42Right precentral gyrus 4 48 7 57 5.35 sponses were slower in all 3 language regions for same-category 6 40 3 59 4.93 pairs than for different-category pairs, suggesting that lexical 4 53 13 47 4.85 information speeds up the perceptual processing of the RVF 6 42 1 11 3.99 colors.Cingulate gyrus 24 4 2 46 6.13 Nonetheless, when the color stimuli were displayed in the 32 2 17 36 4.98 LVF, discrimination of colors from different lexical categories NEUROSCIENCELeft insula 32 16 5 3.28 minus colors from the same lexical category did not provokeParietal stronger activation of any of language-related regions such as theLeft superior parietal lobule 7 26 58 51 4.97 left posterior temporoparietal network (BA 40) when the sig-Right superior parietal lobule 7 26 56 45 5.15 nificance threshold was set at P 0.05 FDR corrected. When aTemporal less stringent threshold of P 0.005 uncorrected was used, theLeft superior temporal gyrus 22 57 6 2 4.99 activation of the left posterior temporoparietal regions was seen, 42 61 17 14 4.98 with only 92 voxels totally (see Fig. S1). In addition, differencesRight superior temporal gyrus 22 63 0 4 3.71 in mean hemodynamic delays were not found in this neural 38 55 17 8 3.13 circuitry. These results indicate that differences between the 22 61 10 1 3.05 activation of language regions by the LVF between-categorySubcortical areas stimuli and the LVF within-category stimuli, if any, would beThalamus 12 19 6 5.18 very weak. This finding confirms previous findings that LVF 16 17 10 3.94 stimuli may activate left-hemisphere language areas by virtue of a longer and ‘‘noisier’’ transcallosal pathway (48, 49). To determine whether lexical color categories are used toprefrontal gyrus (BA 47) were strongly activated. These regions sharpen the perceptual difference through enhanced activationhad been shown to govern lexical search and semantic retrieval of brain regions for color perception, particularly for RVF colorin past lesion and neuroimaging investigations of aphasia and stimuli, we performed a whole-brain, voxel-based analysis of thelanguage functions (24, 32–47); their activation in the visual interaction between visual hemifield and categorical relation. Asearch task indicates that linguistic information of colors is small set of regions hypothesized a priori to be involved in colorrapidly activated and represented in the brain. perception on the basis of prior results (24) as well as the lexical To ascertain whether there is a stronger activation category category effect map of this study were defined to determine theeffect in the RVF than in the LVF, we calculated separate significance of predicted peaks. These regions included the leftactivation maps for each visual field, relating between-categorycolor discrimination to within-category color discrimination, visual areas (V2/3 and V4) and the left language regions. Peakswith a significance level for between-condition differences being that survived the whole-brain analysis thresholded at P 0.005set at P 0.05 false discovery rate (FDR) corrected for multiple (uncorrected) and small volume correction with P 0.05comparisons. For the RVF color stimuli, activation produced by FDR-corrected were considered significant. Relevant regionsdiscrimination of colors from different lexical categories minus emerging from this analysis are the left temporoparietal areaactivation from same-category colors was very strong in the left (BA 40) responsible for language processes and V2/3 crucial forposterior temporoparietal region (BA 40), the left middle tem- color vision. Fig. 5 depicts averaged activity levels in the 4poral gyrus (BA 21), and the left inferior prefrontal cortex at BA conditions. The result shows that activity levels in both V2/3 and47 (Fig. 4 A and B). This pattern of data converges with the BA 40 were significantly enhanced when colors from differentaforementioned results from the main effect of categorical lexical categories were exposed in the RVF. Thus, it seems thatrelationship. The total activation volume in these 3 regions, as lexical category information enhances the neuronal response atindexed by number of voxels, is 1007 (Fig. 4D). Equally impor- V2/3 for colors appearing in the RVF.Siok et al. PNAS Early Edition 3 of 6
  • Table 2. Coordinates of activation peaks: Main effectof categorical relationship A Coordinates BrodmannRegions activated area X Y Z Z scoreFrontalLeft inferior frontal gyrus 47 32 21 1 5.97 47 44 28 13 4.33 44 42 5 22 5.27 45 44 17 19 4.41 BLeft middle frontal gyrus 8 38 31 43 3.09Left superior frontal gyrus 8 12 33 48 3.74Left precentral gyrus 6 38 0 37 4.74Left insula 40 2 2 3.38Right inferior frontal gyrus 47 34 21 3 6.06Right middle frontal gyrus 46 42 40 16 3.58Right superior frontal gyrus 8 12 32 52 4.40Medial frontal 11 2 34 19 4.66 6 6 5 61 3.68 CCingulate gyrus 32 6 23 36 4.62ParietalLeft inferior parietal lobule 40 40 43 39 4.77Left precuneus 7 20 62 36 5.82Right inferior parietal lobule 40 44 35 44 4.00Right superior parietal lobule 7 28 56 49 4.44Right precuneus 7 12 67 49 3.53Right supramarginal gyrus 40 46 49 25 3.38TemporalLeft inferior temporal gyrus 20 51 13 30 3.31 DLeft middle temporal gyrus 21 67 29 7 3.86 TalariachLeft superior temporal gyrus 22 65 46 21 3.26 Brain Region BA Voxel coordinates Z valueLeft fusiform gyrus 37 48 49 11 3.85 x y zLeft angular gyrus 39 44 64 31 2.92Right inferior temporal gyrus 20 53 9 25 3.49 L inferior frontal gyrus 47 420 -44 28 -15 5.88Right superior temporal gyrus 38 36 22 21 4.31 L middle temporal gyrus 21 410 -67 -41 2 4.13Right fusiform gyrus 37 46 55 7 4.47 L supramarginal gyrus 40 177 -48 -55 30 3.44Right angular gyrus 39 51 67 29 3.44Occipital Fig. 4. Brain regions with significant activation during the identification ofV2/3 18 8 85 13 3.98 colors from different lexical categories in the right visual field in comparison 18 4 72 28 3.61 with colors from the same lexical category in the right visual field. (A) Lateral 18 8 83 13 3.13 view. (B) Language regions in the brain showing stronger activation in the between-category condition than in the within-category condition. (C) Lan- 19 4 84 32 2.75 guage regions in the brain that exhibited significantly slower hemodynamicRight superior occipital gyrus 19 32 69 26 3.71 responses in the within-category color condition than in the cross-categoryLimbic lobe color condition. (D) Coordinates of activation peaks in the 3 language areas.Posterior cingulate 31 6 61 14 3.32 The significance threshold is P 0.05 FDR-corrected.Discussion strate hemifield-dependent activations of language regions in a color-discrimination task.We have found brain language regions participating in categor- Lexical color information not only was accessed in colorical color perception when subjects performed a visual search discrimination but also enhanced the activation of color regiontask. The activity of the language regions, however, was modu- V2/3. When the colors exposed in the RVF were from differinglated by the visual field in which the stimulus appeared, asdemonstrated by the following findings. First, in the RVF, lexical categories, activation of V2/3 was much stronger thanperception of target and distractor colors from different linguis- with other color conditions. Notably, the increased activity oftic categories (contrasted with target and distractor colors from V2/3 for the RVF between-category colors coincided with thethe same linguistic category) activates language areas including increased activity of the posterior temporoparietal region forthe posterior temporoparietal region, the middle temporal gy- language processes, as demonstrated by the significant interactionrus, and the inferior prefrontal cortex in the left cerebral of visual field and categorical relation, suggesting that CP of colorhemisphere, but in the LVF perception of target and distractor provokes orchestrated cortical activity occurring within subsystemscolors from different lexical categories is not associated with involving the posterior temporoparietal region and V2/3.stronger activity in any language regions. Second, the activation We tentatively infer that the posterior temporoparietal cortexof language regions seems to exhibit a slower hemodynamic serves as a top-down control source that interacts with andresponse for colors from the same lexical category than for modulates the activity of the visual cortex (V2/3) serving data-colors from different lexical categories, but only when the colors driven analysis of visual stimuli. Anatomical studies have foundare presented in the RVF. multiple reciprocal neural pathways between the parietal cortex These findings therefore extend prior neuropsychological and and visual processing areas, and these pathways may govern suchbrain mapping studies (7, 9, 12, 24) and unequivocally demon- control (50–53). Our results are consistent with lesion studies of4 of 6 www.pnas.org cgi doi 10.1073 pnas.0903627106 Siok et al.
  • 0.08 Table 3. CIEL*u*v* values and inter-pair distances Stimulus L* u* v* Pair E % Bold Signal Change 0.04 G1 62.263 52.327 23.044 0.00 G2 62.44 50.447 6.856 (G1,G2) 16.29776282 -0.04 B1 63.054 48.768 10.53 (G2,B1) 17.47767241 B2 56.483 41.453 27.34 (B1,B2) 19.47468526 -0.08 -0.12 170. The brightness and saturation values were adjusted to make them equal, BA 40 based on the independent judgments of 4 observers. The RGB values for the -0.16 background were 210, 210, and 210. CIEL*u*v* values are given in Table 3. The 0.10 inter-pair distances are (G1,G2) 16.3 E, (G2,B1) 17.48 E, and (B1,B2) 19.47 E. The mean within-category distance, 17.89 E, slightly exceeds the % Bold Signal Change between-category (G2,B1) distance, 17.48 E . 0.05 A rapid event-related design was used. During each trial, a ring of 12 colored squares surrounding the fixation marker was presented simulta- 0.00 neously for 200 ms against a gray background (Fig. 1B), followed by a fixation screen against a gray background. Subjects indicated whether the target was on the left or right side of the circle by making button-press responses with the -0.05 corresponding hand as quickly and as accurately as possible. The duration of the fixation screen varied to jitter the blood oxygen level-dependent (BOLD) V2/3 responses. Inter-stimulus intervals of 1800, 2800, or 3800 ms were assigned -0.10 randomly to the trials, resulting in corresponding stimulus onset asynchronies LB LW RB RW of 2000, 3000, and 4000 ms. There were 6 target– distractor pairs formed by PSYCHOLOGYFig. 5. Regions of interest that survive small volume correction with P 0.05 using all 1-step pairwise combinations of the 4 colors (3 pairs: G1G2, B1B2, andFDR-corrected and a whole-brain voxel-based analysis (thresholded at P G2B1) and having each member of a pair serve once as target and once as0.005 uncorrected for multiple corrections) of the interaction between visual distractor. The target occupied any of the 4 positions (position 1, 2, 3, or 4 infield and categorical relationship. LB, left visual field between-category; LW, Fig. 1B), and there were 24 possible stimulus configurations. There were 400left visual field within-category; RB, right visual field between-category; RW, trials in total. In half of the trials, the target was located to the left of centerright visual field within-category. (position 1 or 2), and in the other half of the trials it was located to the right of center (position 3 or 4). In addition, half of the trials presented within- category combinations (G1G2 or B1B2), and the other half presented the NEUROSCIENCEvisual attention (54) indicating that the posterior parietal cortex between-category combination (G2B1).interacts with the response of neurons in the visual areas in ways The stimuli were presented via a liquid crystal display projector and werethat may fundamentally influence object representations. back-projected onto a projection screen placed at the end of the scanner bore. At present a direct functional connection and interaction Subjects viewed the rear projection screen through a mirror attached to the headbetween the posterior temporoparietal region and the color coil. The distance from the projection screen to the mirror was 70 cm, and theregion(s) in color perception has not been established. Future distance from the mirror to the eyes of the subject was 10 cm. The inner edge of the target color was presented 3.9° to the right or to the left of a centrallyresearch may address this question by performing an effective presented ‘‘ ’’. Hence, the stimuli were separated by a visual angle of 7.8°.connectivity analysis of fMRI data and/or by providing more After the fMRI scans, subjects were given a blue– green lexical boundaryrefined time-course information with the event-related potential test. On each trial, a square stimulus (1 of the colors, G1, G2, B1, or B2) wastechnique. Our study nevertheless has identified the neural presented centrally on a gray background for 200 ms, followed by an 1800-mscorrelates of the behavioral RVF advantage in color discrimi- interval. Participants indicated whether the stimulus was green or blue bynation and thus shed light on the mechanisms that underlie pressing 1 of 2 keys, corresponding to the Mandarin Chinese words forWhorfian effects. Language, by enhancing the activation level of ‘‘green’’ and ‘‘blue,’’ respectively. Each stimulus was presented 10 times in athe visual cortex, differentially influences the discrimination of total of 40 randomized trials. Fifteen subjects identified more than 93% of thecolors presented in the left and right visual hemifields. presentations of G1 and G2 as ‘‘green’’ and of B1 and B2 as ‘‘blue.’’ One subject identified only 53% of the presentations in this way; the data from this subjectMaterials and Methods were discarded.Subjects. Beijing college students [8 males and 8 females; mean age, 23.7 years(SD 1.8 years)] participated in the fMRI experiment. The data of 1 subject were Image Acquisition and Data Analysis. Details of image acquisition and datadiscarded because of head motion and a low identification score in the color analysis are given in the SI text.boundary test. Subjects were paid for their participation and gave informedconsent according to guidelines set by the Administrative Panels on Human ACKNOWLEDGMENTS. We thank Liu Haiqi, Zhou Ke, Wei Zhou, Joey Li, andSubjects in Medical Research of the Beijing MRI Center for Brain Research at Liu Zhendong for help with the experiments. This research was supported bythe Chinese Academy of Sciences. They were tested with the Ishihara test for a 973 grant from the National Strategic Basic Research Program of the Ministrycolor blindness; all subjects had normal color vision and no history of neuro- of Science and Technology of China (2005CB522802), the Knowledge Innova-logical or psychiatric illness. All subjects were strongly right-handed. tion Program of the Chinese Academy of Sciences, the University of Hong Kong, Grant 811-5020 from the Shun Hing Institute of Advanced EngineeringStimuli and Experimental Design. The RGB values of the 4 colors were as follows of Chinese University of Hong Kong, and by Grant 0418404 from the U.S.(see Fig. 1 A): G1 0, 171, 129; G2 0, 170, 149; B1 0, 170, 170; B2 0, 149, National Science Foundation. 1. Carroll JB (1956) Language, Thought, and Reality: Selected Writings of Benjamin Lee 7. Gilbert AL, Regier T, Kay P, Ivry RB (2006) Whorf hypothesis is supported in the right Whorf (MIT, Cambridge, MA). visual field but not the left. Proc Natl Acad Sci USA 103:489 – 494. 2. Kay P, Kempton W (1984) What is the Sapir-Whorf hypothesis? Am Anthropol 86:65–79. 8. Winawer J, et al. (2007) Russian blues reveal effects of language on color discrimina- 3. Gilbert AL, Regier T, Kay P, Ivry RB (2008) Support for lateralization of the Whorf effect tion. Proc Natl Acad Sci USA 104:7780 –7785. beyond the realm of color discrimination. Brain and Language 105:91–98. 9. Drivonikou GV, et al. (2007) Further evidence that Whorfian effects are stronger in the 4. Gumperz JJ, Levinson SC (1996) Rethinking Linguistic Relativity (Cambridge Univ Press, right visual field than the left. Proc Natl Acad Sci USA 104:1097–1102. Cambridge, UK). 10. Butterworth B, Reeve R, Reynolds F, Lloyd D (2008) Numerical thought with and 5. Casasanto D (2005) Crying ‘‘Whorf’’. Science 307:1721–1722. without words: Evidence from indigenous Australian children. Proc Natl Acad Sci USA 6. Gordon P (2005) Response. Science 307:1722. 105:13179 –13184.Siok et al. PNAS Early Edition 5 of 6
  • 11. Roberson D, Davidoff J (2000) The categorical perception of colors and facial expres- 34. Oxbury JM, Oxbury SM, Humphrey NK (1969) Varieties of colour anomia. Brain 92:847– sions: The effect of verbal interference. Memory & Cognition 28:977–986. 860.12. Roberson D, Pak HS, Hanley JR (2008) Categorical perception of colour in the left and 35. Damasio AR, Damasio H (1983) The anatomic basis of pure alexia. Neurology 33:1573– right visual field is verbally mediated: Evidence from Korean. Cognition 107:752–762. 1583.13. Heider ER, Olivier DC (1972) The structure of the color space in naming and memory for 36. Davidoff JB, Ostergaard AL (1984) Colour anomia resulting from weakened short-term two languages. Cognit Psychol 3:337–354. colour memory. A case study. Brain 107:415– 431.14. Kay P, Regier T, Gilbert AL, Ivry RB. (in press) Lateralized Whorf: Language influences 37. Gabrieli JD, Poldrack RA, Desmond JE (1998) The role of left prefrontal cortex in perceptual decision in the right visual field. Language, Evolution, and the Brain, eds language and memory. Proc Natl Acad Sci USA 95:906 –913. Minett JW, Wang W.S-Y. (Hong Kong, The City Univ of Hong Kong Press). 38. Hoeft F, et al. (2007) Functional and morphometric brain dissociation between dyslexia15. Franklin A, Clifford A, Williamson E, Davies I (2005) Color term knowledge does not and reading ability. Proc Natl Acad Sci USA 104:4234 – 4239. affect categorical perception of color in toddlers. Journal of Experimental Child 39. Demb JB, et al. (1995) Semantic encoding and retrieval in the left inferior prefrontal Psychology 90:114 –141. cortex: A functional MRI study of task difficulty and process specificity. J Neurosci16. Pinker S (1994) The Language Instinct (Morrow, New York). 15:5870 –5878.17. Bloom AH (1981) The Linguistic Shaping of Thought: A Study in the Impact of 40. Poldrack RA, et al. (1999) Functional specialization for semantic and phonological Language on Thinking in China and the West (Lawrence Erlbaum Associates, Hillsdale, processing in the left inferior prefrontal cortex. NeuroImage 10:15–35. NJ).18. Au TK (1983) Chinese and English counterfactuals: The Sapir-Whorf hypothesis revis- 41. Schlaggar BL, McCandliss BD (2007) Development of neural systems for reading. Annu ited. Cognition 15:155–187. Rev Neurosci 30:475–503.19. Li P, Gleitman L (2002) Turning the tables: Language and spatial reasoning. Cognition 42. Bokde AL, Tagamets MA, Friedman RB, Horwitz B (2001) Functional interactions of the 83:265–294. inferior frontal cortex during the processing of words and word-like stimuli. Neuron20. Levinson SC, Kita S, Haun DB, Rasch BH (2002) Returning the tables: Language affects 30:609 – 617. spatial reasoning. Cognition 84:155–188. 43. Bookheimer S (2002) Functional MRI of language: New approaches to understanding21. Heider ER (1972) Universals in color naming and memory. J Exp Psychol 93:10 –20. the cortical organization of semantic processing. Annu Rev Neurosci 25:151–188.22. Kay P, Regier T (2006) Language, thought and color: Recent developments. Trends in 44. Goswami U (2006) Neuroscience and education: From research to practice? Nature Cognitive Sciences 10:51–54. Reviews Neuroscience 7:406 – 413.23. Franklin A, et al. (2008) Categorical perception of color is lateralized to the right 45. Price CJ (2000) The anatomy of language: Contributions from functional neuroimag- hemisphere in infants, but to the left hemisphere in adults. Proc Natl Acad Sci USA ing. J Anat 197:335–359. 105:3221–3225. 46. Eden G, Moats L (2002) The role of neuroscience in the remediation of students with24. Tan LH, et al. (2008) Language affects patterns of brain activation associated with dyslexia. Nature Neuroscience 5:1080 –1084. perceptual decision. Proc Natl Acad Sci USA 105:4004 – 4009. 47. Wagner AD, Pare-Blagoev EJ, Clark J, Poldrack RA (2001) Recovering meaning: Left25. Martin A, Haxby JV, Lalonde FM, Wiggs CL, Ungerleider LG (1995) Discrete cortical regions prefrontal cortex guides controlled semantic retrieval. Neuron 31:329 –338. associated with knowledge of color and knowledge of action. Science 270:102–105. 48. Grusser OJ, Landis T (1991) Visual Agnosias and Other Disturbances of Visual Percep- ¨26. Zeki S, et al. (1991) A direct demonstration of functional specialization in human visual tion and Cognition (London, Macmillan Press). cortex. J Neurosci 11:641– 649. 49. Kinsbourne M (1970) The cerebral basis of lateral asymmetries in attention. Acta27. Wandell BA (1999) Computational neuroimaging of human visual cortex. Annu Rev Psychologica 33:193–201. Neurosci 22:145–173. 50. Cavada C, Goldman-Rakic P (1989) Posterior parietal cortex in rhesus monkey: I.28. Wade AR, Brewer AA, Rieger JW, Wandell BA (2002) Functional measurements of Parcellation of areas based on distinctive limbic and sensory corticocortical connec- human ventral occipital cortex: Retinotopy and colour. Philos Trans R Soc London B tions. J Comp Neurol 287:393– 421. 357:963–973. 51. Cavada C, Goldman-Rakic P (1991) Topographic segregation of corticostriatal projec-29. Bartels A, Zeki S (2004) Functional brain mapping during free viewing of natural scenes. Hum Brain Mapp 21:75– 85. tions from posterior parietal subdivisions in the macaque monkey. Neuroscience30. Howard RJ, et al. (1998) The functional anatomy of imagining and perceiving color. 42:683– 696. NeuroReport 9:1019 –1023. 52. Distler C, Boussaoud D, Desimone R, Ungerleider LG (1993) Cortical connections of31. Dougherty RF, et al. (2003) Visual field representations and locations of visual areas inferior temporal area TEO in macaque monkeys. J Comp Neurol 334:125–150. V1/2/3 in human visual cortex. Journal of Vision 3:586 –598. 53. Webster MJ, Bachevalier J, Ungerleider LG (1994) Connections of inferior temporal32. Gorno-Tempini ML, et al. (2004) Cognition and anatomy in three variants of primary areas TEO and TE with parietal and frontal cortex in Macaque monkeys. Cereb Cortex progressive aphasia. Ann Neurol 55:335–346. 4:470 – 483.33. Sonty SP, et al. (2003) Primary progressive aphasia: PPA and the language network. Ann 54. Friedman-Hill SR, Robertson LC, Desimone R, Ungerleider LG (2003) Posterior parietal Neurol 53:35– 49. cortex and the filtering of distractors. Proc Natl Acad Sci USA 100:4263– 4268.6 of 6 www.pnas.org cgi doi 10.1073 pnas.0903627106 Siok et al.
  • All Those Seeing Color, Say Eye! Student SheetName:_____________________________________Part IGo to An Eye on Color slideshow (http://www.thetech.org/exhibits_events/online/color/vision/),on the Tech Museum of Innovation website. First, click on the link titled Inside Your Eyeball.Then navigate through the slideshow by clicking on the link within the text of each slide. Stopwhen the slideshow is over. As you read the slides, answer the following questions:1. What happens when waves of light enter your eye? Discuss the role of the pupil, retina, and optic nerve.2. Where are rods and cones located?3. What are rods and cones? What do they allow you to see?4. In what area of the retina are cones concentrated?5. When do cones work best?6. In what area of the retina are rods concentrated?7. When do rods work best?
  • 8. You’ve just read what happens in the eye: light goes through the pupil, then hits the rods and cones and causes a chemical reaction. What understands this reaction and carries the message to the brain?9. What causes the blind spot?10. Why do certain animals (like the owl and bee) see colors differently than humans? Wait for your teacher to discuss what you just learned.Part IIGo to A Big Look at the Eye on the KidsHealth website.http://www.kidshealth.org/kid/body/eye_SW.htmlNavigate through the article by clicking on the “Next Page” link at the bottom of each slide. Afteryou read the “The Eyes Take the Prize” page, click on the link “Want to see the eye in action?”Then click on “The Eye.” As you read the article, answer the following questions:1. What are functions of the eyelid and blinking?2. What is a function of eyelashes?3. Draw a diagram of the eye labeling the sclera, cornea, iris, and pupil.
  • 4. What part of the eye lets light in?5. What happens to the pupil as you enter a dark room? Why?6. Once light enters through the pupil, what part of the eye focuses light on the retina?7. What carries messages from the eye to the brain?Science NetLinks Student Sheet-All Those Seeing Color, Say Eye!All rights reserved. Science NetLinks Student Sheets may be reproduced for educational purposes.
  • Color Constancy based on the Grey-Edge Hypothesis J. van de Weijer Th. Gevers Intelligent Sensory Information Systems Faculty of Science, University of Amsterdam Kruislaan 403, 1098 SJ Amsterdam, The Netherlands {joostw, gevers}@science.uva.nl Abstract Recently, Finlayson and Trezzi [5] showed that the max- RGB method and the Grey-World method can be interpretedA well-known color constancy method is based on the Grey- as the same algorithm applied with different instantiationsWorld assumption i.e. the average reflectance of surfaces of the error function. The max-RGB method is shown toin the world is achromatic. In this article we propose a be equal to applying the L∞ Minkowski norm and Grey-new hypothesis for color constancy, namely the Grey-Edge World is equal to using the L1 norm. They further showhypothesis assuming that the average edge difference in a that the best color constancy results are attained with the L6scene is achromatic. Based on this hypothesis, we propose norm. Although these simple color constancy algorithmsan algorithm for color constancy. are slightly outperformed by more elaborate methods, e.g. Recently, the Grey-World hypothesis and the max-RGB color gamut mapping (for an overview see [1] [2]), they per-method were shown to be two instantiations of a Minkowski form surprisingly well while their computational costs arenorm based color constancy method. Similarly we also significantly lower.propose a more generale version of the Grey-Edge hypoth- In this paper, we pursue color constancy by the Grey-esis which assumes that the Minkowsky norm of deriva- Edge hypothesis, which assumes the average edge differ-tives of the reflectance of surfaces is achromatic. The al- ence in the scene to be achromatic. The method is based ongorithms are tested on a large data set of images under dif- the observation that the distribution of color derivatives ex-ferent illuminants, and the results show that the new method hibit the largest variation in the light source direction. Theoutperforms the Grey-World assumption and the max-RGB average of these derivatives is used to approximate this di-method. Results are comparable to more elaborate algo- rection. The method is tested on a large database of colorfulrithms, however at lower computational costs. objects under varying lighting conditions and different illu- minants. We further extend the method similarly to [5] and also derive color constancy for the error based on the vari-1 Introduction ous Minkowski norms. The paper is organized as follows. In section 2 colorColor constancy is the ability to recognize colors of objects constancy based on the Grey-World hypothesis is explained.invariant of the color of the light source [1], [2] [6]. It gen- In section 3 we propose the Grey-Edge hypothesis for colorerally consists of two steps. Firstly, the light source color constancy computation. Section 4 contains experiments andis estimated from the image data. Secondly, illuminant in- Section 5 finishes with concluding remarks.variant descriptors are computed, which is usually done byadjusting the image for the color of the light source suchthat the object colors resemble the colors of the objects un-der a known light source. 2 The Grey-World Hypothesis A simple color constancy method, called max-RGB, esti- Tmates the light source color from the maximum response of The image values, f = (R, G, B) , for a Lambertian sur-the different color channels [1]. Another well-known color face are dependent on the light source e (λ), where λ isconstancy method is based on the Grey-World hypothesis the wavelength, the surface reflectance s (λ) and the camera[4], which assumes that the average reflectance in the scene sensitivity functions c (λ) = (R (λ) , G (λ) , B (λ))is achromatic. Although more elaborate algorithms exists,methods like Grey-World and max-RGB are still widely f= e (λ) s (λ) c (λ) dλ, (1)used because of their low computational costs. ω0-7803-9134-9/05/$20.00 ©2005 IEEE II-722
  • O3 O3 O3 O2 O2 O2 O1 O1 O1Figure 1: Three acquisitions of the same scene under different light sources [3]. On the bottom line the derivative distribu-tions, where the axes are the opponent color derivatives and the surfaces indicate derivative values with equal occurrenceand darker surfaces indicating a more dense distribution. Note the shift of the orientation of the distribution of the derivativeswith the changing of the light source.where ω is the visible spectrum and bold fonts are applied which yields the normalized light source color :ˆ = efor vectors. The goal of color constancy is to estimate ke/ |ke|. This is indeed a very simple algorithm to findthe light source color e (λ), or its projection on the RGB- the light source color of a scene.kernels, In [5] it is shown that the Grey-World hypothesis can   be improved by replacing the averaging operation by the Re Minkowski norm. In this case Eq. 4 can be rewritten as e =  Ge  = e (λ) c (λ) dλ, (2) Be ω f p (x) dx 1 p = ke. (5)given the image values f (x), where x is the spatial coordi- dxnate in the image. The task of color constancy is not attain- For p = 1 the equation is equal to the Grey-World as-able without further assumptions. sumption. For p = ∞ it is equal to color constancy by Buchsbaum [4] proposes the Grey-World hypothesis, max-RGB, which is based on the assumption that the max-which assumes that the average reflectance in a scene is imum response in the channels is caused by a white patch.achromatic: s (λ, x) dx Hence, the maximum responses yield an estimate of the = k. (3) light source. Finlayson and Trezzi [5] found that the best dx results are obtained with a Minkowski norm with p = 6.The light source color can now be estimated by computingthe average pixel value , since f (x)dx 1 3 The Grey-Edge Hypothesis = e (λ) s (λ, x) c (λ) dλdx dx dx ω , (4) As an alternative to the Grey-World hypothesis, we propose =k e (λ) c (λ) dλ = ke ω the Grey-Edge hypothesis; the average of the reflectance II-723
  • Figure 2: Examples of the images in group A and B [3].differences in a scene is achromatic rate the Minkowsky norm 1 |sx (λ, x)| dx p |fx (x)| dx p = k. (6) = ke. (9) dx dxWith the Grey-Edge assumption the light source color can Color constancy based on this equation assumes that the p-be computed from the average color derivative in the image th Minkowski norm of the derivative of the reflectance in agiven by: scene is achromatic. |fx (x)|dx 1 = e (λ) |sx (λ, x)| c (λ) dλdx dx dx ω (7) 4 Experiments =k e (λ) c (λ) dλ = ke, ω To test the Grey-Edge hypothesis the algorithm is tested on T a large data set of colorful object under varying light sourceswhere |fx (x)| = (|Rx (x)| , |Gx (x)| , |Bx (x)|) . The [3]. The data set is split in two groups. Group A consistsGrey-Edge hypothesis originates from the observation that of 321 images with varying light sources over a total of 32the color derivative distribution of images forms a relatively scenes and group B consists of 220 images of 22 scenes (seeregular, ellipsoid-like shape, of which the long axis coin- examples in Fig. 2). For all images the correct light source iscides with the light source color. In Fig. 1 the color deriva- measured, el . As an error measure we use the angular errortive distribution is depicted for three images. The color between the the estimated light source ee and the measuredderivatives are rotated to the opponent color space light source el Rx −Gx O1x = √ 2 Rx +Gx −2Bx angular error = cos−1 (ˆl · ˆe ) , e e (10) O2x = √ 6 . (8) Rx +Gx +Bx O3x = √ 3 where the (ˆ indicates the normalized vector. Results of .) other color constancy algorithms on this standard data setIn the opponent color space, O3 coincides with the white are available in [2], [7], [5]. For the derivatives Gaussianlight direction. For the scene under white light (the left- derivatives with σ = 3 were applied.most picture) the distribution of the derivatives are centered In Fig. 3 the results for the Grey-World and the Grey-along the O3 or white-light axis. Once we change the color Edge assumption as a function of the applied norm, p, areof the light source as in the second and third picture, the depicted. The results of the Grey-World are taken fromdistribution of the color derivatives no longer align with the [5]. The angular error for the Grey-Edge method outper-white-light axis. Color constancy based on the Grey-Edge forms the Grey-World method for both groups of images.assumption can be interpreted as skewing the color deriva- Whereas the Grey-World method finds a minimum error fortive distribution such that the average derivative is in the O3 the same norm, p = 6 for both groups of images, for theorientation. Grey-Edge method the behavior as a function of p varies Similarly as for the Grey-World based color constancy, for the two groups of images. If we compare p = 6 for thethe Grey-Edge hypothesis can also be adapted to incorpo- Grey-World with p = 16 for the Grey-Edge based method, II-724
  • group A group B ¥ ¥ Figure 3: Angular error of the Grey-World and the Grey-Edge method as a function of the applied Minkowski norm. Mean 5 Conclusions Grey-World (=L1 -norm) 9.8 Max-RGB (=L∞ -norm) 9.2 In this paper we proposed a color constancy algorithm based L6-norm Grey-World 6.3 on the Grey-Edge hypothesis which assumes the average L6-norm Grey-Edge 5.7 edge difference in a scene to be achromatic. Further, an Color by Correlation 9.9 extension based on the Minkowski norm is proposed. The Gamut Mapping 5.6 algorithm is tested on a large data set and is shown to out- GCIE Version 3, 11 lights 4.9 perform color-constancy based on the Grey-World hypoth- esis and the max-RGB assumption.Table 1: Mean angular error (degrees) for various color con-stancy methods on group A images [7]. References [1] K. Barnard, V. Cardei, and B.V. Funt. A comparison of com- putational color constancy algorithms-part i: Methodology and experiments with synthesized data. IEEE transactions onwe attain an improvement of 9% for the images in group A Image Processing, 11(9):972–984, September 2002.and of 10 % for the images in group B. [2] K. Barnard, V. Cardei, and B.V. Funt. A comparison of com- Also the p = ∞ norm, which is the Grey-Edge variant on putational color constancy algorithms-part ii: Experimentsthe max-RGB method, achieves a good performance. The with image data. IEEE transactions on Image Processing,light source is computed from the assumption that the light 11(9):985–996, September 2002.source is equal to the maximum derivatives of the various [3] K. Barnard, L. Martin, B.V. Funt, and A. Coath. A data set forcolor channels. colour research. Color Research and Application, 27(3):147– 151, 2002. Results of more complex color constancy methods, such [4] G. Buchsbaum. A spatial processor model for object colouras gamut mapping and color-by-correlation, have been re- perception. Journal of the Franklin Institute, 310, 1980.ported in [2], [7] for the images in group A. The resultsare comparable to the results reported here and only two [5] G.D. Finlayson and E. Trezzi. Shades of gray and colour constancy. In IS&T/SID Twelfth Color Imageing Conference,methods perform slightly better, see Table 1. For example pages 37–41, 2004.for Gamut mapping an angular error of 5.6◦ was reported(opposed to 5.7◦ for the Grey-Edge based color constancy). [6] D.A. Forsyth. A novel algorithm for color constancy. Inter-These methods are, however, considerably more complex national Journal of Computer Vision, 5(1):5–36, 1990.and therefore require higher computational costs. In con- [7] S.D. Hordley G.D. Finlayson and I. Tastl. Gamut constrainedclusion, the presented Grey-Edge method is an useful alter- illuminant estimation. In Proc. of the Ninth IEEE Interna-native when computational speed is an issue, with a perfor- tional Conference on Computer Vision, Nice, France, 2003.mance comparable to the best results reported in literature. II-725
  • List
  • Research on color books• ‘Josef Albers. Homage to the square’, —, Hochshule fuür Angewandte Kunst Wien, 1992, ISBN 3-85211-018-1• ‘Josef Albers. Werke auf Papier’, —, Bonn, 1998, ISBN 3-929790-28-9• ‘Das Farbenmischbuch’, —, Verlag Berliner Union GmbH, 1954• ‘Elsworth Kelly. In between spaces’, —, Fondation Beyeler, 2002, ISBN 3-905632-21-7• ‘Elsworth Kelly. The years in France, 1984-1954’, —, National Gallery of Art, Washington, 1992, ISBN 0-89468-185-0• ‘Richard Paul Lohse’, —, Stedelijk Van Abbe Museum, Eindhoven, 1971• ‘Richard Paul Lohse. Color becomes form’, —, Annely Juda Fine Art, London, 1997• ‘Interaction of Color’, Josef Albers, Yale University Press, 1963 (1971) > There is an updated edition available via amazon.com• ‘Selling with color’, Faber Birren, McGraw-Hill Book Company, New York, 1945 > This book is NOT available via amazon.com, BUT his ‘Creative Color’ is.• ‘Disruptive Pattern Material. An Encyclopedia of Camouflage’, Hardy Blechman, Firefly Books, 2004, ISBN 1-55407-011-2 > Yes, the book is available via amazon.com• ‘Peter Struycken’, Carel Blotkamp, NAi uitgevers, 2007, ISBN 978 90 5662 605 1• ‘Color’, Betty Edwards, Penguin, 2004, ISBN 978-1-58542-219-7 > Yes, the book is available via amazon.com• ‘Aristoteles over kleuren’, Ferweda & Strucken, Uitgeverij Damon, 2001 > This book is NOT available via amazon.com, BUT ‘Aristotle: Minor Works: On Colours. On Things Heard. Physiognomics. On Plants. On Marvellous Things Heard. Mechanical Problems. On Indivisible Lines. ... Gorgias (Loeb Classical Library No. 307)’ is.• ’Colour. Travels through the paintbox’, Victoria Finlay > Yes, the book is available via amazon.com• ‘Theory of Colours’, Johann Wolfgang von Goethe, MITT, paperback edition, ISBN 0-262-57021-1 > Yes, the book is available via amazon.com• ‘karl gerstner. Review of Seven Chapters of Constructive Pictures’, Eugen Gomringer, Hatje Gantz, 2003, ISBN 3-7757-9151-5 > Yes, the book is available via amazon.com• ‘The Elements of Color’, Johannes Itten, 1961 (1970) > Yes, the book is available via amazon.com• ‘Bart van der Leck’, Toos van Kooten, Museum Kröller Müller, 1994, ISBN 90-73313-08-2
  • • ‘The new art – The new life. The collected writing of Piet Mondrian’, Thames and Hudson, 1986 > Yes, the book is available via amazon.com• ‘The Eye’s Mind. Collected writings 1965-1999’, Bridget Riley, Thames and Hudson, 1999, ISBN 0-500-28165-3 > Yes, the book is available via amazon.com• ‘Frank Stella’, Robert Rosenblum, Penguin New Art 1, 1971, ISBN 0 14 070621 6• ‘Dazling Painting. Kunst als camouflage’, Albert Roskam, Uitgeverij Van Spijk, 1987, ISBN 90 7 1893 02 2 > Yes, the book is available via amazon.com• ‘Frank Stella. Painting 1958 to 1965’, Lawrence Rubin, Stewart Tabori & Chang, 1986, ISBN 0-941434-92-3• ‘Peter Struycken. Trooping the colour’, —, Gorinchem, 2002, ISBN 90-804257-4-5• ‘Orde en harmonie in het rijk der kleuren (an introduction to the color theory by Ostwald)’, Tales & Zoon, Apeldoorn, 1927 on the web– ‘As humans, our color vision influences everything from our art and poetry to the colors’ via http://www.webexhibits.org/causesofcolor/1.html– ‘Below are specifications for recommended color palettes’ via http://www.stanford.edu/group/identity/ug_color.html– ‘Colors are of philosophical interest for two kinds of reason’ via http://plato.stanford.edu/entries/color/– ‘Color can only exist when three components are present’ via http://www.cambridgeincolour.com/tutorials/color-perception.htm– ‘Color Perception Is Not In The Eye Of The Beholder: Its In The Brain’ via http://www.sciencedaily.com/releases/2005/10/051026082313.htm– ‘Color theory encompasses a multitude of definitions, concepts and design applications’ via http://www.colormatters.com/colortheory.html– ‘Friedrich Wilhelm Ostwald was a Baltic German chemist’ via http://en.wikipedia.org/wiki/Wilhelm_Ostwald– ‘International Klein Blue (IKB) is a deep blue hue first mixed by the French artist Yves Klein’ via http://en.wikipedia.org/wiki/International_Klein_Blue Please also look at http://www.international-klein-blue.com/ and ‘Blue Women Art - Yves Klein (1962)’ via http://www.youtube.com/watch?v=x0mYZbYdIpU– ‘Introduction. At the moment, this site are best viewed with Internet Explorer 4.0.’ via http://www.cs.brown.edu/courses/cs092/VA10/HTML/start.html– ‘Johannes Itten (11 November 1888 – 27 May 1967) was a Swiss expressionist painter, designer, teacher, writer and theorist associated with the Bauhaus’ via http://en.wikipedia.org/wiki/Johannes_Itten Please also look at http://www.bauhaus.de/
  • – ‘Josef Albers (March 19, 1888 – March 25, 1976[1]) was a German-born American artist and educator’ via http://en.wikipedia.org/wiki/Josef_Albers– ‘Peter Struycken in the Groninger Musuem’ via http://www.youtube.com/watch?v=1I6kKk_Ua30 Please also look at Skrjabins Vision. Computer art combining music and visual arts. Nine stills images from a dynamic colourspace by visual artist Peter Struycken based upon the symphony Prometheus, poem of fire (1911) opus 60, by Alexander Skrjabin 1872-1915) via http://www.xs4all.nl/~kalden/stru/stru-skrjabin-E.html– ‘Richard Paul Lohse was born in Zürich in 1902’ via http://www.lohse.ch/bio_e.html– ‘The Department of Colour Science, founded in 1878, is an international centre of excellence [. . . ]. It is unique in the UK’ via http://www.colour.leeds.ac.uk/– ‘Theory of Colours is a book by Johann Wolfgang von Goethe published in 1810’ via http://en.wikipedia.org/wiki/Theory_of_Colours– ‘Where to Study Color (updated July, 2008)’ via http://www.colormatters.com/des_studycolor.html– ‘Why study color theory?’ via http://www.worqx.com/color/ papers• D.H. Brainard, ‘Color Vision’• Kate Bukoski, ‘Implications: seeing color, typography and color’• Richard L. Gregory, ‘Images of mind in brain’• C.L. Hardin & Luisa Maffi ‘Color categories in thought and language’• John Harris, ‘How does visual memory work?’• Paul Kay & Luisa Maffi ‘Color Appearance and teh Emergence and Evalution of Basic Color Lexicons’• Edwin H. Land, ‘The Retinex Theory of Color Vision’• Cahrles Poynton ‘Frequently Asked Questions about Color’• Dale Purves, ‘The Visual System and the Brian’• Vilayanur S. Ramachandran & Edward M. Hubbard, ‘Hearing Colors, Tasting Shapes’• Tech Museum, ‘All Those Seeing Color, Say Eye! Student Sheet’• Wal Ting Slok & o, ‘Language regions of brain are operative in color perception’• J. van de Weijer & Th. Gevers, ‘Color Constancy based on the Grey-Edge Hypothesis’