Reading images the grammar of visual design -- van leeuwen and kress
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Reading images the grammar of visual design -- van leeuwen and kress

  • 6,926 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • interesantísimo. Muchas gracias
    Are you sure you want to
    Your message goes here
  • Me gusta y es muy útil
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
6,926
On Slideshare
6,925
From Embeds
1
Number of Embeds
1

Actions

Shares
Downloads
586
Comments
2
Likes
8

Embeds 1

http://www.spundge.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. r e a d a g e s Thissecondeditionof the landmarktextbookReadingImages buildsonitsreputationastheirst systematicand comprehensiveaccountof thegrammarof visualdesign. Drawingonanenormousrangeof examplesfromchildren,s drawingsto textbookillustrations,photo-journalismto fineart, aswellasthree-dimensionalformssuchassculptureandtoys, theauthorsexaminethewaysinwhichimagescommunicate meantng. Featuresof thisfullyupdatedsecondeditioninclude: . newmaterialonmovingimagesandoncolour . a discussionof howimagesandtheiruseshavechanged throughtime . websitesandweb-basedimages . ideasonthefutureof visualcommunication. ReadingImagesfocusesonthestructuresor'grammar'of visual design- colour,perspective,framingandcomposition- and providesthereaderwithaninvaluable'tool-kit'forreading images,whichmakesit a mustfor anyoneinterestedin communication,themediaandthearts. GuntherKressis Professorof Englishat theInstituteof Education,Universityof London.Theovan Leeuwenhas workedasa film andtelevisionproducerinthe Netherlanosano AustraliaandasProfessorintheCentrefor Language& CommunicationResearchat CardiffUniversity.Heiscurrently Deanat theFacultyof HumanitiesandSocialSciences, Universityof Technology,Sydney.Theyhavebothpublished widelyinthefieldsof languageandcommunicationstudies. i n g i m
  • 2. P r a i s e f o r t h e r s t e d 'ReadingImagesisthemostimportantbookin visual communicationsinceJacquesBertin'ssemiologyof informationgraphics.It isboththoroughandthought- provoking;a remarkablebreakthrough.' l(evinG.Barnhurst,SyracuseUniversity,USA 'Freshandstimulating.Thesociocentricapproachisbyfar themostpenetratingapproachto thesubjectcurrently available.' PaulCobley,LondonGuildhallUniversity 'A usefultextfor alI studentswhoareinvolvedin areas whichrelyonbothlanguageandvisualimagesfortheir exoressionandarticulationof ideas.' CatrionaScott,MiddlesexUniversity 'Thisisthebestdetailedandsustaineddevelopmentof the "socialsemiotic"approachto theanalysisof visuals.Clear, informativeandtheoreticallydevelopmental.' Dr S.Cottle,BathHE College 'Excellent- wideranging- accessible- tutors'"Bible".' Jan Mair,EdgeHill UniversityCollegeof Higher Education 'Extremelyattractiveandwelllaidout.Veryuseful bibliography.' Dr IVl.Brottman,EastLondonUniversity 'Veryclearlywritten- it makesgoodconnectionsbetween differentareasof visualpractice- especiallyusefulfor studentsfroma varietyof backgroundsattempting "mixed"coursework.' Amy Sargeant,PlymouthUniversity i t i o nf i
  • 3. readingimage-s G U N T H E R T H E 0 v a n K R E S S a n d L E E U W E N T H E 6 R A M M A R O F V I S U A L D E S I 6 N S E C O N DE D I T I O N Hltiy"'l:s,g:",LONDON AND NEW YORK
  • 4. Firstoublished1995 byRoutledge 2 ParkSquare,MiltonPark,Abingdon,0xon0X144RN SimultaneouslypublishedintheUSAandCanada byRoutledge 270MadisonAve,NewYork,NY10016 Secondeditionpublished2006 Reprinted2007(twice),2008 Routledgeisan imprint of the Taylor& FrancisGroup,an informabusiness @1996,2006GuntherKressandTheovanLeeuwen Typesetin BellGothicbyRefineCatchLtd,Bungay,Suffolk PrintedandboundinGreatBritainby TJ InternationalLtd,Padstow,Cornwall All rightsreserved.Nopartofthisbookmaybereprintedor reproducedor utilizedinanyformor byanyelectronic,mechanical,or othermeans/now knownorhereafterinvented,includingphotocopyingandrecording,or inany informationstorageor retrievalsystem,withoutpermissionin writingfrom theoublishers. British Library Cataloguingin PublicationData A cataloguerecordforthisbookisavailablefromtheBritishLibrary Library of CongressCatalogingin PublicationData l(ress.GuntherR. Readingimages: thegrammarof visualdesign/ Guntherl(ressandTheo vanLeeuwen.- 2nded. 0.cm. Includesbibliographicalreferencesandindex. 1.Communicationindesign. I. VanLeeuwen,Theo,I94T- II. Title. Nl(1510.K642006 70I-dc22 ISBN10:0-415-31914-5(hbk) ISBN10:0-415-31915-3(pbk) ISBN10:0-203-61972-2 Gbk) ISBN13:978-0415-3L9I4-0 (hbk) ISBN13:97844I5-3L915-7 (pbk) ISBN13:978-0-203-6L972-!(ebk) 2006002242
  • 5. C O N T E N T S v i i i x x i I P r e f a c et o t h es e c o n de d i t i o n P r e fa c et o t h ef i r s te d i t i o n A c k n o w l e d g e m e n t s I n t r o d u c t i o n :t h eg r a m m a ro f v i s u a l d e s i g n T h es e m i o t i cl a n d s c a p e :l a n g u a g ea n d v i s u a lc o m m u n i c a t i o n N a r r a t i v er e p r e s e n t a t i o n s :d e s ig n i n g s o c i a la c t i o n C o n c e p t u a lr e p r e s e n t a t i o n s :d e s i g n i n g socialconstructs R e p r e s e n t a t i o na n di n t e t a c t i o n : d e s i g n i n gt h e p o s i t i o no f t h ev i e w e r M o d a l i t y :d e s i g n i n gm o d e l so f r e a l i t y T h em e a n i n go f c o m p o s i t i o n M a t e r i a l i t ya n dm e a n i n g T h et h i r dd i m e n s i o n C o l o u r f u lt h o u g h t s( a p o s t s c r i p t ) R e f e r e n c e s I n d e x 1 6 4 5 7 9 t t 4 t54 L75 2t5 239 266 27t 287 4 5 7 B 9
  • 6. P r e fa c et o t h e s e c o n de d i t i o n Thefirsteditionof ReadingImageshashada positivereceptionamonga widegroupfrom the professionsand disciplineswhichhaveto dealwith real problemsand real issues involvingimages.Thishasgonealongwith a broaderagendaof concernwith'multi- modality',a rapidlygrowingrealizationthatrepresentationisalwaysmultiple.Wedo not thinkfor a momentthatthisbookrepresentsanythinglikea settledapproach,a definitive 'grammar'of images,andat timeswehavebeenworriedbyattemptsto treatit inthatway. Weseeit asanearlyattempt/oneamongmanyothers,andwewouldliketo seeit treated verymuchasa resourcefor beginningto makeinroadsintounderstandingthevisualas representationandcommunication- in a semioticfashion- andalsoasa resourcein the developmentoftheoriesand'grammars'of visualcommunication.In thatspiritwewantto stressthatweseeeverythingwe havewrittenheresimultaneouslyasourfullyseriousand yetentirelyprovisionalsenseof thisfield. Whenwe completedthe first editionof this bookwe wereawareof a numberof 'omissions'-thingswe felt stillneededdoing.Someof thesewe havetakenup in other ways,for instanceinourattemptto developatheoryof multimodality;otherswehavetried to addressin this secondedition.Foremostamongthesehavebeenthe quitedifferent issuesof the movingimageandof colour.Thefirst of thesehasbeenconstantlyraisedby thosewho haveusedthe book,andrightlyso.We hopethat whatwe havesaidherecan beginto integratethe fieldof movingimagesintoour socialsemioticapproachto visual communication.Theissueof colourwaslessfrequentlyraised,yetconstitutedfor usa l<ind of theoreticaltestcase,asmuchto dowiththeissueof colouritselfasto dowitha theory of multimodalsocialsemioticsmuchmorewidelyconsidered.Here,too,wefeelthatwe haveprovidedjust a first attemptfor a differentapproach.In additionwe haveaddeda numberof newexamplesfromCD-R0Msandwebsites,domainsof visualcommunication that hadhardlybegunto developwhenwewrotethefirst edition,andare nowof central importancefor manyusersof thisbook. 0nepersistentcriticismof the first editionfrom a groupof readershasbeenthat the bookwas(too)linguistic.Thefirstcommentwewouldmakeisto saythatforus'formality' inthedomainof representationisnotinanywaythesameas'beinglinguistic'.Soto some extentwethinkthat that criticismrestsonthat kindof misunderstanding.Wealsothink thatthereis a differencebetweenexplicitnessandformality.Wecertainlyhaveaimedfor theformer,andoften(butnotalways)forthelatter.Nordowethinkthateitherexplicitness or formalityarethe enemiesof innovation,creativity,imagination:oftenall theselatter restontheformer.It isthe casethat our startingpointhasbeenthe systemicfunctional grammarof Englishdevelopedby MichaelHalliday,thoughwehadandhaveattemptedto useits generalsemioticaspectsratherthan its specificlinguisticallyfocusedfeaturesas the groundingfor our grammar.As Ferdinandde Saussurehaddoneat the beginningof thelastcentury,weseelinguisticsasa partof semiotics;butwedonotseelinguisticsasthe
  • 7. vil| Preface to the secondedition disciplinethat can furnisha ready-mademodelfor the descriptionof semioticmodes otherthan language.Thenwe hadthought,in our first attempt,that to showhowvisual communicationworksincomparisonto languagemightbehelpfulin understandingeither and both- but that,too,was misunderstoodmaybeas an attemptto imposelinguistic categoriesonthevisual.Wehavethereforetriedto refineandclarifythosesectionsof the bookthatdealwiththerelationbetweenlanguageandvisualcommunication,andto delete or reformulatematerialwhichwethinkmighthavegivenriseto thesemisunderstandings, hopefullywithno lossof clarity.A carefulreadingof thissecondeditionof ourbookwill show,we trust,that we are as concernedto bringout the differencesbetweenlanguage and visualcommunicationas we are the connections,the broadersemioticprinciples that connect,not just languageand image,but all the multiplemodesin multimodal communication. In our growingunderstandingof this domain,reffectedin the reworkingof this book, we owea debtof gratitudefor support,commentandcritiqueto manymorepeoplethan wecanmentionor eventhanweactuallyknow.Butthenamesof somefriends,colleagues, students,fellowresearchersandcriticswhowerenotalreadyacknowledgedinourpreface to the first editionhaveto be mentioned.Amongtheseare CareyJewitt,Jim Gee,Ron Scollon,Paul Mercer,Brian Street,RadanMartinec,Adam Jaworski,DavidMachin, l(las Prytz,TealTriggs,AndrewBurn,Bob Ferguson,PippaStein,DeniseNewfield,Len Unsworth,LesleyLancasterand the manyresearcherswhosework has both givenus confidenceandnewideas,andextendedour understandingof thisfield- andof course, andcrucially,we acknowledgethe supportfrom our publishersandeditorsat Routledge, LouisaSemlyenandChristabell(irkpatrick.
  • 8. P r e fa c et o t h e f i r s t e d i t i o n Thisbool<grewoutof discussionsaboutvisualcommunicationwhichspanneda periodof sevenyears.Bothof ushadworkedontheanalysisof verbaltexts,andincreasinglyfeltthe needof a betterunderstandingof all thethingsthat 9owiththeverbal:facialexpressions, gestures,images,music,andso on.Thiswasnot onlybecausewe wantedto analysethe wholeofthetextsinwhichthesesemioticmodesplaya vitalroleratherthanjusttheverbal part,but alsoto understandlanguagebetter.Justasa knowledgeof otherlanguagescan 0pennewperspectivesonone'sownlanguage,soa knowledgeof othersemioticmodescan 0pennewperspectiveson language. In 1990we publisheda firstversionof our ideason visualcommunication,Reading Images,with DeakinUniversityPress.It waswrittenfor teachers,andweconcentratedon children'sdrawingsandschooltextbookillustrations,althoughwealsoincludedexamples from the massmedia,suchas advertisementsandmagazinelayout.Sincethenwe have expandedour researchto otherfieldsof visualcommunication:a muchwiderrangeof massmediamaterials;scientific(andother)diagrams,mapsandcharts;andthe visual arts.Wehavealsomadea beginningwiththestudyof three-dimensionalcommunication: sculpture,children'stoys,architectureandeverydaydesignedobjects.Thepresentbook thereforeoffersa muchmorecomprehensivetheoryof visualcommunicationthan the earlierbook. In Australia,andincreasinglyelsewhere,ourworkhasbeenusedin courseson com- municationandmediastudies,andasa methodologyfor researchin areassuchasmedia representation,film studies,children'sliteratureandthe useof illustrationsandlayoutin schooltextbooks.Thepresentbookhasbenefitedgreatlyfrom the suggestionsandcom- mentsof thosewhohaveusedourwork intheseways,andof our ownundergraduateand postgraduatestudents,initiallyat the Universityof TechnologyandMacquarieUniversity in Sydney,laterat the Instituteof Educationandthe LondonCollegeof Printingin London,andalsoat theTemasekPolytechnicin Singapore. We beganour work on visualcommunicationin the supportiveand stimulating environmentof the NewtownSemioticsCirclein Sydney;discussionswithourfriends,the membersof thisCircle,helpedshapeour ideasin morewaysthanwe canacknowledge. If anytwo peoplefrom that first periodwereto besingledout,it wouldbeJim Martin, who gaveusmeticulous,detailed,extensiveandchallengingcommentson severalof the chaptersof theearlierbook,andFranChristie,whohadurgedusto write it. But herewe wouldalsoliketo makea specialmentionof BobHodge,whoseideasappearinthisbookin manyways/evenif notalwaysobviouslyso. 0f thosewho usedour bookin teachingand research,andwhosecommentson the earlierbookhavehelpedus rethinkand refineour ideas,we wouldlike to mention theresearchteamof the DisadvantagedSchoolsProgrammein Sydney,in particularRick Iedema,SusanFeez,PeterWhite,RobertVeeland Sally Humphrey;StaffanSelander,
  • 9. x . Preface to the first edition throughwhoseCentrefor TextbookResearchin Hdrnosandourworkcameto betakenup by researchersin the fieldof textbookresearchin SwedenandseveralotherEuropean countries;the membersof the 'Languageand Science'researchteam at the Institute of Education,IsabelMartins,Jon 0gbornand l(ieranMcGillicuddy;PhilipBell;Basil Bernstein;PaulGillenandTeunvanDijk. Threewritersinffuencedour ideasin differentandfundamentalways.Oneis Roland Barthes.Althoughweseeourworkasgoingbeyondhisseminalwritingonvisualsemiotics in severalways,he remainsa stronginspiration.Thereis not a subjectin semioticson whichBartheshasnotwrittenoriginallyandinspiringly.Hehasprovidedfor usa model of what semioticscanbe,in the rangeof his interests,in the depthof hiswork,and in hisengagementwiththe socialandculturalworld.Equallysignificantfor us is Michael Halliday.His viewof languageas a socialsemiotic,andthe widerimplicationsof his theories,gaveusthemeansto gobeyondthestructuralistapproachof 1960sParisSchool semiotics,and our work is everywhereinffuencedby his ideas.Thenthereis Rudolf Arnheim.Themorewe readhiswork,the morewe realizethat mostof whatwe haveto sayhasalreadybeensaidby him,oftenbetterthanwe havedoneit, albeitit usuallyin commentarieson individualworksof art ratherthanintheformof a moregeneraltheory. HeiscommonlyassociatedwithGestaltpsychology:wewouldliketo claimhimasa great socialsemiotician. Wewouldliketo thanoureditor,JuliaHall,for herencouragementandinvaluablehelp in producingthisbook.Jill Brewsterand LauraLopez-Bonillawereinvolvedin various stagesofthebook;theirencouragementandhelpmadetheworkpossibleandenjoyable.
  • 10. A c l t n o w l e d g e m e n t s Plate3 JoshuaSmithbyWilliamDobell,1943,@DACS2004. Plate5 CossacksbyVassilyl(andinsl<y,1910-1911.@ADAGP,ParisandDACS,London 2004.Photography@Tate,London2005. Plate6 HistoricColoursby ColinPoole,reproducedby kindpermissionof PhotoWord SyndicationLtd. Plate7 Palgravecolourschemereproducedwithpermissionof PalgraveMacmillan. 1.1+1.13'My bath' from Baby'sFirst Book by B. Lewis,illustratedby H. Wooley, copyright@ LadybirdBooks1td.,1950. L2 Bird in treefrom 0n My Wallgby Dick Bruna,1988.IllustrationDick Bruna@ Mercisb41972. 1.4 Magazinecoverwith naturalisticphotograph,coverof Newsweek,April9,2004 @ 2004 Newsweek,Inc. Photographby l(arim Sahib-AFP-GettyImages.Reprintedby permission. 1.5 Magazinecoverwithconceptualphotograph,coverof Newsweek,November12,200I O 2001Newsweek,Inc.Reprintedbypermission. 1.6 Imagefrom'lnteractivePhysics'.Courtesyof MSCSoftware. 2.5 CommunicationmodelfromWatson,J. andHill,A.(1980)A Dictionaryof Communi- cationandMediaStudiegLondon,Arnold,p.I43. Reproducedby permissionof Hodder HeadlinePLC. 2.6 TwoCommunicationModelsfrom Watson,J. and Hill,A. (1980)A Dictionaryof CommunicationandMediaStudieqLondon,Arnoldp.I47. Reproducedby permissionof HodderHeadlinePLC. 2.IO Beatthe WhiteswiththeRedWedgeby El Lissitzky,I9I9-20.@ DACS2004. 2.11 l(asimirMalevich(1878-1935)"SuprematistComposition:RedSquareandBlack Square",1914,NewYork,Museumof ModernArt (MolVlA)@ 2004,Digitalimage,The Museumof ModernArt, NewYorl</Scala,Florence. 2.I7 Gulf War Diagram,SydneyMorningHerald,14 February,1991 reproducedby permissionof SydneyMorningHerald. 2.18 Speechcircuitfrom Saussure'sCoursein GeneralLinguistics,I9T4,F.deSaussure, translatedby RoyHarrisbypermissionof GeraldDuckworth& Co.
  • 11. xii . Acknowledgements 2.20 Vitteladvertisementsreproducedbykindpermissionof Nestl6Group. 2.22 CommunicationModel from Watson,J. and Hill, A. (1980) A Dictionaryof CommunicationandMediaStudieELondon,Arnoldp.I47. Reproducedby permissionof HodderHeadlinePLC. 2.23 Arctictundrasystem/fi1.7.5,p.1,72fromSale,C.,Friedman,B.andWilson, G.)ur ChangingWorld,Book1, PearsonEducationAustralia.Reproducedby permissionof the publisher. 2.24 CommunicationModelfrom Watson,J. and Hill,A. (1980) A Dictionaryof Com- municationandMediaStudies,London,Arnoldp.54.Reproducedbypermissionof Hodder HeadlinePLC. 3.1 Guideinterfacefrom'DangerousCreatures',7994.Screenshotreprintedby permis- sionfromMicrosoftCorporation. 3.2 Sekondaadvertisementreproducedbykindpermissionof Sekonda/TimeProducts. 3.3 Sourcesof signsfrom Eco,U. (1976) A theoryof semiotics,Bloomington,Indiana UniversityPress,p.177.Reproducedbypermissionof thepublisher. 3.5 Semanticfielddiagramfrom Eco,U. (I976) A theoryof semiotics,Bloomington, IndianaUniversityPress,p.78.Reproducedbypermissionof thepublisher. 3.7 NetworkfromSharples,M.andPemberton,'Representingwriting:externalrepresen- tationsandthe writingprocess'inN.Williamsand P.Holt,edsComputersand Writing. Reproducedbykindpermissionof IntellectLtd,www.intellectbooks.com 3.13 Resortwear,AustralianWomen'sWeekly,December1987.@ AustralianWomen's Weekly/ACPSyndication.Reproducedwith permission. 3.19 Electricalcircuitdiagramfrom J. Hill,1980,IntroductoryPhysics.Reproducedby permissionof Taylor& FrancisGroup. 3.20 Theplaceof linguisticson the mapof knowledgefrom Halliday,M.A.l(.(1978) Languageas SocialSemiotic,London,Arnold.Reproducedby permissionof Hodder HeadlinePLC. 3.21 'Womenat work',Fig.III-6, p.29,fromPictographsandGraphs:Howto Makeand UseThemby RudolfModleyandDynoLowenstein.@ 1952by Harper& Brothers.Copy- rightrenewed1980by PeterM. ModleyandMarionE.Schilling.Reprintedbypermission of HarperCollinsPublishersInc. 3.28 'Funwithfungi',SydneyMorningHerald,18 June1992,reproducedby permission of SydneyMorning Herald. 4.2 ATMscreenreproducedbykindpermissionof NationalAustraliaBank. 4.4 The murderof Dr Chang,SydneyMorning Herald,5 July 1991, reproducedby permissionof SydneyMorningHerald.
  • 12. Acknowledgements x i i i 4.5 Playstationwebsitereproducedbykindpermissionof SonyComputerEntertainment EuropeLtd. 4.7 NewlookFordMondeofromwww.ford.co.ukreproducedbyl<indpermissionof Ford. 4.8 Fiesta'Rocksolid'websitereproducedby kindpermissionof 0gilvyGroupHoldings LtdandFord. 4.16 'PrisonGuard'byDannyLyon,1969,fromConversationswith thedead.@ Danny Lyon.MagnumPhotos.Reproducedwithpermission. 4.19 Gulfwar map,SydneyMorningHerald,22January1991,reproducedbypermission of SydneyMorning Herald. 4.20 An increasein tourism,SydneyMorningHerald,22January1991,reproducedby permissionof SydneyMorningHerald. 4.22 DeIail from a fourteenth-centurySpanishnativity,from Rudolf Arnheim,Art and Visual Perception:A Psychologyof the Creative Eye. The New Version.@ I974 TheRegentsof the Universityof California.Reproducedby permissionof Universityof CaliforniaPress. 5.1 Speechcircuitfrom Saussure'sCoursein GeneralLinguistics,7974,F.de Saussure, translatedby RoyHarrisbypermissionof GeraldDuckworth& Co. 5.2 Schematizedspeechcircuitfrom Saussure'sCoursein GeneralLinguistics,1974, F.deSaussure,translatedby RoyHarrisbypermissionof GeraldDuckworth& Co. 5.6 Card-players(Van Doesburg,1916-17) photograph/picture:Tim l(oster,ICN, Rijswijk/Amsterdam.Reproducedwithpermissionof InstitutCollectieNederland. 5.7 Composition9 (YanDoesburg,I9I7). Collectionof theGemeentemuseumDenHaag. Reproducedwithpermission. 5.8 ColourProjectfortheSchroderResidence(GerritRietveld,1923-4)@ DACS2005. 5.9 PhotographoftheSchroderResidence,GerritRietveld,@ DACS2004. 5.11 @OxfordUniversityPressfrom TheYoungGeographerInvestigates:Mountainsby TerryJennings(0UP,1986),reprintedbypermissionof 0xfordUniversityPress. 5.12 Drawingby Newton.Bypermissionof theWardenandFellows,NewCollege,Oxford, andTheBodleianLibrary,Universityof 0xford.MS 361,vol.2,fol.45V. 5.13 Drawingof Stretton'sexperiment,figure8.1 (p.141)from TheEyeand Brain: Psychologyof Seeing5/eby RichardGregory,1998,reproducedby permissionof 0xford UniversityPress.Gregory,Richard,EyeandBrain.Reprintedby permissionof Princeton UniversityPress. 6.2 Gold-diggers,AustralianWomen'sWeekly,November1987.@ AustralianWomen's WeeklyiACPSyndication.Reproducedwithpermission.
  • 13. xiv . Acknowledgements 6.3 SonylVliddleEastwebsitereproducedbykindpermissionof SonyGulfFZE. 6.9 Gerbner'scommunicationmodelfromWatson,J.andHill,A.(1980)ADictionaryof CommunicationandMediaStudies,London,Arnold.Reproducedbypermissionof Hodder HeadlinePLC. 6.10 Royalcouple.H.M. The Queen'swedding,photographby Baron,CameraPress, London.H.M.TheQueenandPrincePhilip,photographby H.R.H.PrinceAndrew,Camera Press,London.Reproducedwithpermission. 6.12 Buddhistpaintingfrom RudolfArnheim,Art and VisualPerception:A Psychology of theCreativeEye.TheNewVersion.@1974TheRegentsofthe Universityof California. Reproducedbypermissionof Universityof CaliforniaPress. 6.13 'Goingon holiday'from Prosser,R. (2000) Leisure,Recreationand Tourism, London,CollinsEducational.Reprintedbypermissionof HarperCollinsPublishersLtd@ R.Prosser,2000. 6.14 Anderschet al.'s communicationmodelfrom Watson,J. and Hill, A. (1980) A Dictionaryof Communicationand Media Studies,London,Arnold. Reproducedby permissionof HodderHeadlinePLC. 6.16 Verticaltriptychfromthewebsiteof 0xford Universityreproducedby kindpermis- sionof 0xfordUniversity. 6.23 Screenshotfrom CD-R0M'3D BodyAdventure',l(nowledgeAdventure,1993 providedcourtesyof l(nowledgeAdventure,Inc. 7.1 Roy Lichtenstein,Big Painting,1965.@ The Estateof Roy Lichtenstein/DACS 2004. 8.I Jacoband the Angel Qacob Epstein,1940) O The Estateof Jacob Epstein/Tate, London2005.Imagesuppliedbyandreproducedbypermissionof GranadaTV. 8.2 Peoplein the Wind(l(ennethArmitage,1952). Reproductioncourtesyof l(enneth ArmitageEstate.Photography@Tate,London2005. 8.5 WomanbyJoanMir6,I97O@ SuccessioMiro,DACS2004. 8.6 LesHeuresdesTraces(HouroftheTracelbyAlbertoGiacometti,1930@ADAGP, ParisandDACS,London2004.Photography@Tate,London2005. 8.7 Jacoband the AngelQacobEpstein,1940) O The Estateof JacobEpstein/Tate, London2005.Imagesuppliedbyandreproducedbypermissionof GranadaTV. 8.8 Playmobil'familyset'and'ethnicfamily'fromPlaymobilcatalogue.Reproducedby kindpermissionof PlaymobilUl<Ltd. 8.9 RecumbentFigureby HenryMoore,1938.Illustratedonp.250;hasbeenreproduced bypermissionofthe HenryMooreFoundation.Photography@Tate,London2005.
  • 14. Acknowledgements . xu B.I0 l(ingandQueenby HenrylVloore,1952-3.lllustratedon p.253; hasbeenrepro- ducedbypermissionofthe HenryMooreFoundation.Photography@Tate,London2005. 8.11 Churchof SantaMaria DellaSpinafrom RudolfArnheim,Art and VisualPercep- tion: A Psychologyof the CreativeEye. TheNew Version.@ 7974 The Regentsof the Universityof California.Reproducedbypermissionof Universityof CaliforniaPress. 8.12 Connectedanddisconnectednarrativeprocess/pp.83-84 from S. Goodmanand D. Graddol,RedesigningEnglish- new texts,new identities,London,Routledge,1997. Reproducedbypermissionof thepublisher. 8.13 Overshouldershotincomputergame,DeltaForce.Imagecourtesyof NovaLogicInc. @ 2004.All rightsreserved. 8.14 Dynamicinterpersonalrelationsin the openingsceneol TheBig Sleep(Howard Hawks,1947),pp.9I-92 from S.Goodmanand D.Graddol,RedesigningEnglish- new texts,newidentities,London,Routledge,l997. Reproducedbypermissionofthe publisher.
  • 15. I n t r o d u c t i o n :t h e g r a m m a ro f v i s u a l d e s i g n Thesubtitleof thisbookis'the grammarof visualdesign'.We hesitatedoverthistitle. Extensionsof the term'grammar'oftensuggest'rules'.In bookswith titleslike The Grammarof TelevisionProductiononelearns,for instance,aboutthe rulesof continuity; l<nowingtheserulesisthenwhatsetsthe'professional'apartfromthe'amateur'.Whatwe wishto expressis a littledifferent.In our view,mostaccountsof visualsemioticshave concentratedonwhatmightberegardedastheequivalentof 'words'-whatlinguistscall 'lexis'- ratherthan'grammar',andthenonthe'denotative'and'connotative',the'icono- graphical'and'iconological'significanceof theelementsin images,theindividualpeople, placesandthings(includingabstract'things')depictedthere.In thisbook,bycontrast,we will concentrateon 'grammar'andon syntax,on the way in whichtheseelementsare combinedinto meaningfulwholes.Justas grammarsof languagedescribehowwords Combinein Clauses,sentenceSandtexts,soourvisual'grammar'willdesCribethewayin whichdepictedelements- people,placesandthings- combinein visual'statements'of greateror lessercomplexityandextension. Weareby no meansthefirstto dealwiththissubject.Nevertheless,bycomparisonto the studyof visual'lexis',the studyof visual'grammar'hasbeenrelativelyneglected,or dealtwith from a differentperspective,from the pointof viewof art history,or of the formal,aestheticdescriptionof composition,or the psychologyof perception,or with a focusonmorepragmaticmatters,for instancethewaycompositioncanbeusedto attract theviewer'sattentionto onethingratherthananother,e.g.insuchappliedenvironmentsas advertisingor packaging.All theseare validapproaches,and in manyplacesandmany wayswehavemadeuseof the insightsof peoplewritingfromthesedifferentperspectives. Yetthe resulthasbeenthat,despitethe verylargeamountof work doneon images,not muchattentionhasbeenpaidto themeaningsof regularitiesinthewayimageelementsare used- in short,to their grammar- at leastnot in explicitor systematicways.It is this focusonmeaningthatweseek,aboveall,to describeandcaptureinourbook.Weintendto provideusabledescriptionsof majorcompositionalstructureswhichhavebecomeestab- lishedas conventionsin the courseof the historyof Westernvisualsemiotics,andto analysehowtheyareusedto producemeaningbycontemporaryimage-makers. Whatwehavesaidaboutvisual'grammar'istruealsoof themainstreamof linguistic grammar:grammarhasbeen,and remains,'formal'.It hasgenerallybeenstudiedin isolationfrom meaning.Howeve[the linguistsandthe schoolof linguisticthoughtfrom whichwedrawpartofourinspiration-linguistsfollowingtheworkofMichaelHalliday- havetakenissuewith this view,and seegrammaticalformsas resourcesfor encoding interpretationsof experienceand formsof social(inter)action.BenjaminLeeWhorf arguedthepointinrelationto languagesfromdifferentcultures.In whathecalled'Stand- ard AverageEuropean'languages,termslike'summer','winter','September','morning', 'noon','sunset'arecodedas nouns/asthoughtheywerethings.Hencetheselanguages
  • 16. Introduction makeit possibleto interprettime as somethingyoucancount/use/save/etc.In Hopi,a NorthAmericanIndianlanguage,this is not possible.Timecan onlybe expressedas 'subjectiveduration-feeling'.Youcannotsay'atnoon',or'threesummers,.youhaveto sav somethinglike'whilethesummerphaseisoccurring,(Whorf,1950. Thecriticallinguistsof the EastAngliaSchool,withwhomoneof uswasconnected, haveshownthatsuchdifferentinterpretationsof experiencecanalsobeencodedusingthe resourcesof thesamelanguage,onthe basisof differentideologicalpositions.TonyTrew (1979:106-7) hasdescribedhow,whenthe Hararepolice- in whatwasin 1975still Rhodesia- firedintoa crowdof unarmedpeopleandshotthirteenof them,the Rhodesia Heraldwrote,'A politicalclashhasledto deathandiniury,,whilethe TanzanianDaily Newswrofe,'Rhodesia'swhitesuprematistpolice... openedfire and killedthirteen unarmedAfricans.'In otherwords,thepoliticalviewsof newspapersarenotonlyencoded throughdifferentvocabularies(of the well-known'terrorist'vs'freedomfighter,type), but alsothroughdifferentgrammaticalstructures;that is,throughthe choicebetween codingan eventasa noun('death','injury')or a verb('kill'),whichfor itsgrammatical completionrequiresanactivesubject('police')andanobject(,unarmedAfricans,). Grammargoesbeyondformal rulesof correctness.It is a meansof representing patternsof experience....ltenableshumanbeingsto builda mentalpictureof reality,to makesenseof theirexperienceof what goeson aroundthemandinside them. (Halliday,1985:t0t) Thesameis true for the'grammarof visualdesign'.Like linguisticstructures,visual structurespointto particularinterpretationsof experienceandformsof socialinteraction. To somedegreethesecanalsobe expressedlinguistically.Meaningsbelongto culture, ratherthanto specificsemioticmodes.Andthewaymeaningsaremappedacrossdifferent semioticmodes,thewaysomethingscan,forinstance,be'said'eithervisuallyor verbally, othersonlyvisually,againothersonlyverbally,isalsoculturallyandhistoricallyspecific.In the courseof this bookwe will constantlyelaborateandexemplifythis point.But even whenwecanexpresswhatseemto bethesamemeaningsin eitherimage-formor writing or speech,theywill be realizeddifferently.For instance,what is expressedin language throughthe choicebetweendifferentword classesandclausestructures,may,in visual communication,beexpressedthroughthechoicebetweendifferentusesof colouror differ- entcompositionalstructures.Andthiswillaffectmeaning.Expressingsomethingverbally or visuallymakesa difference. Asfor otherresonancesof theterm'grammar'(grammar,asa setof rulesonehasto obeyif one is to speakor write in'correct',sociallyacceptableways),linguistsoften protestthat theyare merelydescribingwhat peopledo,andthat othersinsiston turning descriptionsintorules.Butof courseto describeisto beinvolvedin producingknowledge whichotherswill transformfromthedescriptiveintothenormative,for instancein educa- tion.Whena semioticmodeplaysa dominantrolein publiccommunication,its usewill inevitablybe constrainedby rules,rulesenforcedthrougheducation,for instance,and
  • 17. Introduction throughall kindsof writtenandunwrittensocialsanctions.0nlya smalleliteof experi- mentersisallowedto breaktherules- afterall,breakingrulesremainsnecessaryto keep openthepossibilityof change.Webelievethatvisualcommunicationiscomingto beless andlessthe domainof specialists,andmoreandmorecrucialin the domainsof public communication.Inevitablythiswill leadto new,and morerules,andto moreformal/ normativeteaching.Not being'visuallyIiterate'willbeginto attractsocialsanctions. 'Visualliteracy'willbeginto bea matterof survival,especiallyintheworkplace. We arewellawarethat work suchasourscanor will helppavethewayfor develop- mentsof thiskind.Thiscanbeseennegatively,asconstrainingthe relativefreedomwhich visualcommunicationhassofar enjoyed,albeitat theexpenseof a certainmarginalization bycomparisonto writing;or positively,asallowingmorepeoplegreateraccessto a wider rangeof visualskills.Nordoesit haveto standinthewayof reativity.Teachingtherulesof writinghasnotmeanttheendof creativeusesof languagein literatureandelsewhere,and teachingvisualskillswill notspelltheendof thearts.Yet,justasthegrammarcreatively employedby poetsandnovelistsis,in the end,the samegrammarwe usewhenwriting letters,memosand reports,so the'grammarof visualdesign'creativelyemployedby artistsis,intheend,thesamegrammarweneedwhenproducingattractivelayouts,images anddiagramsfor ourcoursehandouts,reports,brochures,communiqu6s,andsoon' It isworth askingherewhata linguisticgrammaris a grammarof.fhe conventional answeristo saythatit is agrammarof 'English'or'Dutch'or'French'-therulesthat defineEnglishas'English',Dutchas'Dutch',andsoon.A slightlylessconventionalanswer wouldbeto saythat agrammarisan inventoryof elementsandrulesunderlyingculture- specificformsof verbalcommunication.'Underlying'hereis a shorthandtermfor some- thingmorediffuseandcomplex,morelike'knowledgesharedmoreor lessbymembersof a group,explicitlyandimplicitly'.Thisbringsinsubtlemattersofwhatl<nowledgeisandhow it isheldandexpressed,andaboveallthesocialquestionof whata'group'is.Thatmakes definitionsof grammarverymucha socialquestion,oneof the knowledgesandpractices sharedbygroupsof people. We mightnowask,'What is our "visual grammar"agraffimarof?'First of all we wouldsaythatit describesa socialresourceof a particular9roup,itsexplicitandimplicit knowledgeaboutthisresource/anditsusesinthepracticesof thatgroup.Then,second,we wouldsaythat it isa quitegeneralgrammartbecauseweneeda termthat canencompass oil paintingaswellasmagazinelayout,thecomicstripaswellasthescientificdiagram. Drawingthesetwo pointstogether,andbearingin mindour socialdefinitionof grammar, we wouldsaythat'our'grammaris a quitegeneralgYammarof contemporaryvisual designin'Western'cultures,an accountof theexplicitandimplicitl<nowledgeandprac- ticesarounda resource,consistingof theelementsandrulesunderlyinga culture-specific formof visualcommunication.Wehavequitedeliberatelymadeourdefinitiona socialone, beginningwiththequestion'Whatisthegroup?Whatareitspractices?'andfromthere attemptingto describethegrammarat issue,ratherthanadoptinganapproachwhichsays, 'Hereisourgyammar;dothepracticesandl<nowledgesofthisgroupconformtoitornot?' In the bookwe have,by and large,confinedour examplesto visualtext-objectsfrom ,Western'culturesandassumedthatthisgeneralizationhassomevalidityasit pointsto a
  • 18. Introduction communicationalsituationwitha longhistorythathasevolvedoverthepastfivecenturies or so/alongsidewriting(quitedespitethedifferencesbetweenEuropeanlanguages),asa 'languageof visualdesign'.Its boundariesare notthoseof nation-states,althoughthere are,andverymuchso,cultural/regionalvariations.Rather/thisvisualresourcehasspread, alwaysinteractingwiththespecificitiesof locality,whereverglobalWesterncultureisthe dominantculture. Thismeans,first of all,that jt is not a'universal'grammar.Visuallanguageis not - despiteassumptionsto thecontrary-transparentanduniversallyunderstood;it iscultur- allyspecific.We hopeourwork will continueto providesomeideasandconceptsfor the studyof visualcommunicationin non-Westernformsof visualcommunication.Togivethe mostobviousexample,Westernvisualcommunicationisdeeplyaffectedbyourconvention of writingfrom leftto right(in chapter6 we will discussthismorefully).Thewriting directionsof culturesvary:from rightto leftor from leftto right,fromtopto bottomor in circularfashionfromthecentreto theoutside.Consequentlydifferentvaluesandmeanings areattachedto suchkeydimensionsof visualspace.Thesevaluationsandmeaningsexert their inffuencebeyondwriting,and informthe meaningsaccordedto differentcom- positionalpatterns,theamountof usemadeof them,andsoon.In otherwords,weassume thatthe elements,suchas'centre,or,margin,,'top,or'bottom,,will playa rolein the visualsemioticsof any culture,but with meaningsand valuesthat are likelyto differ dependingonthatculture'shistoriesof useofvisualspace,writingincluded.The'universal, aspectof meaningliesin semioticprinciplesandprocesses,theculture-specificaspectlies in theirapplicationoverhistory,andin specificinstancesof use.Herewe merelywantto signalthat our investigationshavebeenrestricted,by andlarge,to Westernvisualcom- munication.Eventhoughothershavebegunto extendtheapplicationsof theprinciplesof thisgrammar,wemakenospecificclaimsfor theapplicationof our ideasto othercultures. WithinWesternvisualdesign,however,we believethat our theoryappliesto all formsof visualcommunication.Wehopethatthewiderangeof exampleswe usein the bookwill convincereadersof thisproposition. 0urstressontheunityof Westernvisualcommunicationdoesnotexcludethepossibility of regionalandsocialvariation.Theunityof Westerndesignisnotsomeintrinsicfeatureof visuality,butderivesfroma longhistoryof culturalconnectionandinterchange,aswellas nowfrom the globalpowerof the Westernmassmediaandcultureindustriesandtheir technologies.In manypartsof theworld,Westernvisualcommunicationexistssidebyside with localforms.Westernformsmightbeused,for instance,in certaindomainsof public communication,suchaspublicnotices,sitesof publictransport,thepress,advertising,and thevisualarts,aswellasinsomewhatmore'private,domains,inthehome,andin markets andshops,forlnstance.Oftentherelationishierarchical,withoneformoverlaidonanother (seeScollonandScollon,2003;l(ress,2oo3),andoften- asinadvertising,for instance- thetwo aremutuallytransformedandfused.WhereWesternvisualcommunicationbegins t0 exertpressureonlocalforms,therearetransitionalstagesinwhichtheformsof thetwo culturesmix in particularways.In lookingat advertisementsin English-languagemaga- zinesfrom the Philippines,for instance,we werestruckby the way in whichentirely conventionalWesterniconographicalelementswereintegratedintodesignsfollowingthe
  • 19. Introduction rulesof a localvisualsemiotic.In advertisementsontheIVITRin HongKong,someadvert- isementsconformto the'Eastern'directionality,othersto theWestern,yetothersmixthe two.As with the Filipinoadvertisements,discoursesand iconographycan be 'Western', mixedinvariouswayswiththoseof the'East',whilecolourschemescan/at thesametime, bedistinctlynon-Western.Thesituationthereis in anycasecomplicated(asit is,differ- ently,in Japan)bythe fact that directionalityin the writingsystemhasbecomecompli- catedin severalways:bytheadoption,in certaincontexts,of 'Western'directionalityand the Romanalphabetalongsidethecontinueduseof the moretraditionaldirectionalities andformsof writing.Andaseconomic(andnowoftencultural)poweris re-weighted,the trendcango in bothor moredirections:the influenceof Asianformsof visualdesignis becomingmoreandmorepresentinthe'West'.Superimposedonall thisaretheincreas- inglyprominentdiasporiccommunities- of Greeks,Lebanese,Turks,of manygroupsof the Indiansubcontinent,of new and olderChinesecommunities(for instance,Hong l(ongChinesearoundthe PacificRim)- whichseeminglyaffectonlythe membersof this diaspora,andyetin realityarehavingdeepinfluenceswellbeyondthem. WithinEurope,increasingregionalitycounterbalancesincreasingglobalization.Solong asthe Europeannationsandregionsstill retaindifferentwaysof lifeanda differentethos, theywill usethe'grammarof visualdesign'distinctly.It is easy,for example,to find examplesof the contrastinguseof the left and right in the compositionof pagesand imagesinthe Britishmedia.It isharderto findsuchexamplesin,forinstance,theGreekor the Spanishor the Italianmedia.as studentsfrom thesecountrieshaveassuredusand demonstratedin theirworl<- aftertryingto dotheassignmentswe hadsetthemat home duringtheirholidays.in the courseof our bookwe will givesomeexamplesof this,for instanceinconnectionwithnewspaperlayoutindifferentEuropeancountries.However,we arenotableto domorethantouchonthesubject;andtheissueof different'dialects'and 'inffections'needsto beexploredmorefullyinthefuture. In anycase,the unityof languagesis a socialconstruct,a productof theoryandof socialandculturalhistories.Whenthebordersof (a) languagearenotpolicedbyacad- emies,andwhenlanguagesare not homogenizedby educationsystemsandmassmedia, peoplequitefreelycombineelementsfromthe languagestheyknowto makethemselves understood.Mixedlanguages('pidgins')developin thisway,andin timecanbecomethe languageof new generations('creoles').Visualcommunication,not subjectto such policing,has developedmorefreelythan language,but therehas neverthelessbeena dominantlanguage,'spoken'anddevelopedincentresof highculture,alongsidelesshighly valuedregionalandsocialvariants(e.9.'folkart').Thedominantvisuallanguageis now controlledbytheglobalcultural/technologicalempiresofthemassmedia,whichdissemin- atetheexamplessetbyexemplarydesignersand,Ihroughthe spreadof imagebanksand computer-imagingtechnology,exerta 'normalizing'ratherthan explicitly'normative' inffuenceonvisualcommunicationacrosstheworld.Muchasit istheprimaryaimof this bookto describethecurrentstateof the'grammarof visualdesign',wewill alsodiscuss the broadhistorical,socialand culturalconditionsthat makeand remakethe visual 'language'.
  • 20. Introduction A SOCIALSEMIOTICTHEORYOFREPRESENTATION 0urworkonvisualrepresentationissetwithinthetheoreticalframeworkof 'socialsemiot- ics'.It isimportantthereforeto placeit inthecontextof theway'semiotics'hasdeveloped during,roughly,thepast75 years.In Europe,threeschoolsof semioticsappliedideasfrom the domainof linguisticsto non-linguisticmodesof communication.Thefirst wasthe PragueSchoolofthe1930sandearly1940s.It developedtheworkof RussianFormalists by providingit with a linguisticbasis.Notionssuchas'foregrounding,wereappliedto language(e.9.the'foregrounding',forartisticpurposes/of phonologicalor syntacticforms through'deviation'fromstandardforms,for artisticpurposes)aswellasto thestudyof art (Mukarovsky),theatre(Honzl),cinema(Jakobson)andcostume(Bogatyrev).Eachof thesesemioticsystemscouldfulfilthesamecommunicativefunctions(the'referential,and the'poetic'functions).Thesecondwasthe parisSchoolof the 1960sand1970s,which appliedideasfrom de saussureandotherIinguiststo painting(Schefer),photography (Barthes,Lindel<ens),fashion(Barthes),cinema(Metz),music(Nattiez),comicstrips (Fresnault-Deruelle),etc.TheideasdevelopedbythisSchoolarestilltaughtin countless coursesof mediastudies,art anddesign,etc.,oftenundertheheading,semiology,,despite the fact that they are at the sametime regardedas havingbeenovertakenby post- structuralism.Everywherestudentsarelearningabout'langue'and,parole,;the'signifier, andthe'signified',''arbitrary'and,motivated,signs;,icons,,,indexes,and'symbols,(these termscomefromtheworkof theAmericanphilosopherandsemioticianCharlesSanders Peirce,but areoftenincorporatedin theframeworkof 'semiology,),andsoon.Generally thishappenswithoutstudentsbeinggivena senseof,or accessto,alternativetheoriesof semiotics(orof linguistics).Wewill compareandcontrastthiskindof semioticswithour ownapproach,inthisintroductionaswellaselsewhereinthebook.Thisthird,stillffedg- ling,movementin whichinsightsfrom linguisticshavebeenappliedto othermodesof representationhastwo sources/bothdrawingonthe ideasof MichaelHalliday,onegrow- ing out of the'critical Linguistics'ofa groupof peopleworkingin the 1970sat the Universityof EastAnglia,leadingto theoutlineof a theorythatmightencompassother semioticmodes(Hodgeand l(ress),the other,in the later 1990s,as a developmentof Hallidayansystemic-functionallinguisticsbya numberof scholarsinAustralia,insemiot- icallyorientedstudiesof literature(Threadgold,Thibault),visualsemiotics(0,Toole, ourselves)andmusic(vanLeeuwen). Thekeynotionin anysemioticsisthe'sign'.0ur bookisaboutsigns- or,aswewouro ratherput it,aboutsign-making.Wewill bediscussingforms('signifiers')suchascolour, perspectiveandline,aswellasthewayinwhichtheseformsareusedto realizemeanings ('signifieds')inthemakingof signs.Butourconceptionof thesigndifferssomewhatfrom that of 'semiology',andwewishthereforeto comparethetwo viewsexplicitly.In doingso we usethe term 'semiology'to referto the way in whichthe ParisSchoolsemioticsis generallytaughtintheAnglo-Saxonworld,throughthemediationof inffuentialtextbooks suchasthe seriesof mediastudiestextbookseditedby JohnFiske(Fiskeand Hartley, r979; Dyer,r9B2;Fisl<e,r9B2;Hariley,19B2;0'sullivaneta.,1993).In doingthiswedo not seekto repudiatethosewhowentbeforeus.We seea continuitybetweentheirwork
  • 21. Introduction andours,asshouldbeclearfrom ourmaintitle,ReadingImages,whichechoesthatof the firstvolumein Fiske'sseries,ReadingTelevision(FiskeandHartley,I979). Wewouldliketo beginwithanexampleof whatwe understandby'sign-making'.The drawingin figure0.1 was madeby a three-year-oldboy.Sittingon hisfather'slap,he talkedaboutthedrawingashewasdoingit:'Doyouwantto watchme?I'll makeacar . . . g o t t w o w h e e l s . . . a n d t w o w h e e l s a t t h e b a c k . . . a n d t w o w h e e l s h e r e . . . t h a t ' s a f u n n y wheel....'Whenhehadfinished,hesaid,'Thisis a car.'Thiswasthefirsttimehehad nameda drawing,andat firstthenamewaspuzzling.Howwasthisa car?0f coursehehad providedthe keyhimself:'Here'sa wheel.'A car,for him,wasdefinedby the criterial characteristicof 'havingwheels',andhisrepresentationfocusedonthis aspect.What he representedwas,infact,'wheelness'.Wheelsarea plausiblecriterionto chooseforthree- year-olds,andthe wheel'saction/on toy carsas on reai cars/is a readilynoticedand describablefeature.In otherwords,thisthree-year-old'sinterestincarswas,for him,most plausiblycondensedintoandexpressedasan interestinwheels.Wheels,inturn,aremost plausiblyrepresentedby circles,bothbecauseof their visualappearanceandbecauseof thecircularmotionof thehandindrawing/representingthewheel'sactionof 'goinground andround'. Togatherthisupfor a moment,weseerepresentationasa processinwhichthemal<ers of signs,whetherchildor adult,seekto makea representationof someobjector entity, whetherphysicalor semiotic,and in whichtheir interestin the object,at the pointof makingtherepresentation,isa complexone,arisingoutof thecultural,socialandpsycho- logicalhistoryof the sign-maker,andfocusedbythe specificcontextin whichthe sign- makerproducesthesign.That'interest'isthesourceoftheselectionofwhatisseenasthe criterialaspectof the object,and this criterialaspectis then regardedas adequately representativeof theobjectin a givencontext.In otherwords,it isneverthe'wholeobject' butonlyeveritscriterialaspectswhicharerepresented. Thesecriterialaspectsarerepresentedinwhatseemsto thesign-maker,at themoment of sign-mal<ing,themostaptandplausiblefashion,andthemostaptandplausiblerepre- sentationalmode(e.g.drawing,Legoblocks,painting,speech).Sign-makersthus'have'a co O g O Fig0.1 Drawingbya three-year-oldchild
  • 22. Introduction meaning/thesignified,whichtheywishto express,andthenexpressit throughthesemiotic mode(s)that make(s)availablethesubjectivelyfelt,mostplausible,mostaptform,asthe signifier.Thismeansthatinsocialsemioticsthesignisnotthepre-existingconjunctionof a signifieranda signified,a ready-madesignto berecognized,chosenandusedasit is,inthe waythatsignsareusuallythoughtto be'availablefor use'in ,semiology,.Ratherwefocus on the processof sign-making,in whichthe signifier(the form) andthe signified(the meaning)are relativelyindependentof eachotheruntiltheyare broughttogetherbythe sign-makerin a newlymadesign.Toput it in a differentway,usingtheexamplejustabove, the processof sign-makingis the processof the constitutionof a sign/metaphorin two steps:'acaris (mostlike)wheels'and'wheelsare(mostlike)circles,. Puttingit in our terms:the sign-maker'sinterestat this momentof sign-makinghas settledon'wheelness'asthe criterialfeatureof'car'. He constructs,by a processof analogy,twometaphors/signs:first,thesignified'wheel'isaptlyrepresentedbythesignifier 'circle'tomakethemotivatedsign'wheel';second,thesignified'car'isaptlyrepresented by the signifier'manywheels'to makethe motivatedsign'car'. Theresultingsign,the drawingglossed'thisisa car',isthusa motivatedsigninthateachconjunctionof signifier and signifiedis an apt, motivatedconjunctionof the form which bestreoresentsthat whichisto bemeant.Thissignisthustheresultof a doublemetaphoricprocessinwhich analogyistheconstitutiveprinciple.Analogy,inturn,isa processof classification:x is like y (incriterialways).Whichmetaphors(and,'behind'themetaphors,whichclassifications) carrythe dayandpassintothesemioticsystemasconventional,andthenas naturalized, andthenas'natural',neutralclassifications,isgovernedbysocialrelationsof power.Like adults,childrenareengagedin theconstructionof metaphors.Unlikeadults,theyare,on the onehand,lessconstrictedby cultureand its already-existingand usuallyinvisible metaphors,but,onthe otherhand,usuallyin a positionof lesspower,sothat their meta- phorsarelesslikelyto carrytheday. It followsthatweseesignsasmotivated- notas arbitrary-conjunctionsof signifiers (forms)andsignifieds(meanings).In'semiology'motivationis usuallynotrelatedto the act of sign-makingas it is in our approach,but definedin termsof an intrinsicrelation betweenthesignifierandthesignified.It is herethat peirce's'icon',,index,and'symbol, maketheirappearance/incorporatedinto'semiology'ina waywhichin factcontradicts someof the keyideasin Peirce'ssemiotics.The'icon'isthesignin which,thesignifier- signifiedrelationshipis oneof resemblance,likeness'(Dyer,r9g2: r24) - i.e.objective likeness,ratherthananalogymotivatedby'interest',establishestherelation.The,index,is the signin which'thereis a sequentialor causalrelationbetweensignifierandsignified, (Dyer,1982:125);that is,a logicof inference,ratherthananalogymotivatedby,interest,. Thethirdtermin thetriad,'symbol',bycontrast,is relatedto signproduction,asit'rests on convention,or "contract"' (Dyer,r9g2:125), but thisveryfact makesit ,arbitrary,, 'unmotivated'/a caseof meaningbydecreeratherthanof activesign-making. In ourviewsignsareneverarbitrary,and'motivation'shouldbeformulatedin relation to thesign-makerandthecontextinwhichthesignis produced,andnot in isolationfrom theactof producinganalogiesandclassifications.Sign-makersusetheformstheyconsider aptfor theexpressionof theirmeaning,inanymediuminwhichtheycanmakesigns.when
  • 23. Introduction childrentreat a cardboardbox as a pirateship,they do so becausethey considerthe materialform (box)an apt mediumfor the expressionof the meaningtheyhavein mind (pirateship),andbecauseof theirconceptionof thecriterialaspectsof pirateships(con- tainment,mobility,etc.).Languageis no exceptionto this processof sign-making.All linguisticformis usedin a mediated,non-arbitrarymannerintheexpressionof meaning. Forchildrenintheirearly,pre-schoolyearsthereisbothmoreandlessfreedomof expres- sion:more,becausetheyhavenotyetlearnedto confinethemakingof signsto thecultur- ally andsociallyfacilitatedmedia,andbecausetheyare unawareof establishedconven- tionsandrelativelyunconstrainedin the makingof signs;less,becausetheydo not have suchrichculturalsemioticresourcesavailableasdoadults.Sowhena three-year-oldboy, labouringto climba steephill,says,'Thisisa heavyhill',heisconstrainedbynothavingthe word'steep'asanavailablesemioticresource.Thesameisthecasewiththeresourcesof syntacticandtextualforms. 'Heavy',in'heavyhill',is,however,a motivatedsign:thechildhasfocusedonparticular aspectsof climbinga hill (it takesa lot of energy;it isexhausting)andusesanavailable formwhichheseesasaptfor theexpressionof thesemeanings.Theadultwhocorrectsby offering'steep'('Yes,it's a verysteephill') is,fromthechild'spointof view,notsomuch offeringanalternativeasa synonymfor theprecisemeaningwhichhehadgivento'heavy' in that context.Boththe childandthe parentmakeuseof 'what is available';it happens thatdifferentthingsareavailableto each.Butto concentrateonthisisto missthecentral aspectof sign-making,especiallythatof children.'Availability'is notthe issue.Children, likeadults,maketheirownresourcesof representation.Theyarenot'acquired',but made bytheindividualsign-maker. In 'semiology',countlessstudentsacrosstheworldareintroducedto theterms'langue' and'parole',with 'langue'explained,for instance,as'theabstractpotentialof a language system. . . the sharedlanguagesystemout of whichwe makeour particular,possibly unique,statements'(O'sullivanet a|.,1983:t27) or,in ourterms,asa systemof available formsalreadycoupledto availablemeanings,andwith'parole'definedas: an individualutterancethat isa particularrealizationof thepotentialof langue.. . . Byextensionwe canarguethat thetotal systemof televisionandfilm conventions andpracticesconstitutesalangue,andthewaytheyarerealizedineachprogramme or filma parole. (0'Sullivanet al.,l9B3i I27 ) Weclearlyworkwithsimilarnotions,with'availableforms'and'availableclassifications' ('langue')andindividualactsof sign-making('parole'),andweagreethatsuchnotionscan usefullybeextendedto semioticmodesotherthanlanguage.Butfor usthe ideaof'poten- tial' (whatyoucanmeanandhowyoucan'say'it, inwhatevermedium)isnotlimitedbya systemof 'availablemeanings'coupledwith'availableforms',andwewouldliketo usea slightlylessabstractformulation:a semiotic'potential'isdefinedbythesemioticresources availableto a specificindividualin a specificsocialcontext.0f course,a descriptionof semioticpotentialcanamalgamatethe resourcesof manyspeakersand manycontexts.
  • 24. 1 0 Introduction Buttheresulting'langue'(thelangueof 'English'orof 'westernvisualdesign,)is in the endan artefactof analysis.Whatexists,andis thereforemorecrucialfor understanding representationandcommunication,aretheresourcesavailableto realpeoplein realsocial contexts.Andif weconstructa'langue',a meaningpotentialfor'Westernvisualdesign,, thenit is no moreandno lessthana tool whichcanserveto describea varietyof sign- makingpractices,withinboundariesdrawnby the analyst.It followsthat we wouldnot drawthe linebetween'langue'and'parole'assharplyas it is usuallydone.Describinga 'langue'isdescribinga specificset of semioticresourcesavailablefor communicative actionto a specificsocialgroup. Hereare someantecedentsof the car drawing.Figure0.2 is a drawingmadeby the samechild,sometenmonthsearlier.Itscircularmotionisexpressiveof thechild'sexuber- ant,enthusiasticandenergeticactionsin makingthedrawing.In figure0.3,madeabout threemonthslater,the circularmotionhasbecomemoreregular.Theexuberanceand energyarestillthere,butthedrawinghasacquiredmoreregularity,moreinterestinshape: 'circularmotion'is beginningto turn into'circle'.In otherwords,themeaningsof figure O Fig0.2 Drawingbya two-year-oldchitd
  • 25. Introduction I I O Fig0,3 Drawingbyatwo-yeaholdchild 0.2 persistinfigure0.3,transformed,yetwith significantcontinuity:figure0.3gathersup, soto speak,themeaningsof figure0.2,andthentransformsandextendsthem. Figure0.4,finally,showsa seriesof circles,eachdrawnona separatesheet,onecircleto eachsheet.Themovementfrom figure0.2 to figure0.4 is clearenough,as is the con- ceptualandtransformativeworkdonebythechildovera periodof fourteenmonths(figure 0.4 datesfrom the sameperiodasfigure0.1).Togetherthedrawingsshowhowthe child developedtherepresentationalresourcesavailableto him,andwhycirclesseemedsuchan aptchoiceto him:theexpressive,energeticphysicalityof themotionof figure0.2 persisted asthechilddevelopedthisrepresentationalresource/sothatthecircularmotionremained part of the meaningof circle/wheel.Butsomethingwasaddedaswell:thetransformation of representationalresourceswasalsoa transformationof thechild'ssubjectivity,fromthe emotional,physicalandexpressivedispositionexpressedintheactof representing'circular motion'tothemoreconceptualandcognitivedispositionexpressedintheactof represent- inga 'car'. Children,likeallsign-makers,maketheir'own'representationalresources,anddosoas
  • 26. 12 . Introduction O Fig0.4 Drawingbya three-year.oldchild part of a constantproductionof signs,in whichpreviouslyproducedsignsbecomethe signifier-materialto betransformedintonewsigns.Thisprocessrestson lhe interestof sign-makers.Thistransformative,productivestancetowardssign-makingis at the same timea transformationof thesign-makers'subjectivity- a notionfor whichtherewaslittle placein a 'semiology'whichdescribedthe relationbetweensignifiersand signifiedsas restingon inferenceor objectiveresemblance,or onthedecreesof thesocial'contract'. Wehaveusedchildren'sdrawingsasourexamplebecausewe believethat the produc- tionof signsbychildrenprovidesthebestmodelforthinkingaboutsign-making.It applies alsoto fully socializedand acculturatedhumans,with the exceptionof the effectsof 'convention'.As maturemembersof a culturewe haveavailablethe culturallyproduced semioticresourcesof oursocieties,andareawareof theconventionsandconstraintswhich are sociallyimposedon our makingof signs.However,as we havesuggested,in our approachadultsign-makers,too,areguidedby interest,bythat complexcondensationof culturalandsocialhistoriesandof awarenessof presentcontingencies.'Mature'sign- makersproducesignsout ol that interest,alwaysastransformationsof existingsemiotic materials,thereforealwaysinsomewaynewlymade,andalwaysasmotivatedconjunctions of meaningandform.Theeffectof conventionisto placethepressureof constantlimita- tionsof conformityon sign-making;that is,the way signifiershavebeencombinedwith signifiedsin the historyof theculture,actsasa constantlypresentconstrainton howfar onemightmovein combiningsignifierswith signifieds.Conventiondoesnot negatenew making;it attemptsto limitandconstrainthesemioticscopeof thecombinations. This,then,isourpositionvis-d-vis'European'semiology:wheredeSaussurehad(been assumedto have)saidthattherelationof signifierandsignifiedinthesignisarbitraryand conventional,wewouldsaythatthe relationisalwaysmotivatedandconventional.Where hehadseeminglyplacedsemioticweightandpowerwiththesocial,wewishto assertthe effectsof thetransformativeroleof individualagents,yetalsotheconstantpresenceof the social:in thehistoricalshapingof the resources,inthe individualagent'ssocialhistory,in
  • 27. ( 1 ) Introduction the recognitionof presentconventions,intheeffectof theenvironmentinwhichrepresen- tationandcommunicationhappen.Yetit isthetransformativeactionof individuals,along thecontoursof socialgivens,whichconstantlyreshapestheresources,andmal<espossible theself-makingof socialsubjects. 0neof the nowtaken-for-grantedinsightsof sociallyorientedtheoriesof languageis thevariationof languagewiththevariationof socialcontext.Theaccountsofthisvariation differ,rangingfromcorrelation('languageformx relatesto socialcontexty') to determin- ation('languageformx is producedby socialactorsy or in socialcontexty').A social semioticapproachtakesthelatterview,alongthefollowinglines. Communicationrequiresthat participantsmaketheir messagesmaximallyunder- standablein a particularcontext.Theythereforechooseformsof expressionwhich theybelieveto be maximallytransparentto otherparticipants.0nthe otherhand, communicationtakesnlacein socialstructureswhichareinevitablymarkedbypower differences,andthisaffectshoweachparticipantunderstandsthenotionof'maximal understanding'.Participantsin positionsof powercanforceotherparticipantsinto greatereffortsof interpretation,andtheirnotionof'maximalunderstanding'isthere- foredifferentfrom that of participantswhodo their bestto producemessagesthat will requirea minimaleffortof interpretation,or from that of participantswho, throughlackof commandof the representationalsystem,producemessagesthat are harderto interpret(e.g.children,learnersof a foreignlanguage).Theotherpartici- pantsmaytheneithermaketheeffortrequiredto interpretthesemessagesor refuse to doso,whetherin a schoolor in a railwaystationin a foreigncountry. Representationrequiresthat sign-makerschooseformsfor the expressionof what theyhavein mind,formswhichtheyseeasmostaptandplausibleinthegivencontext. Theexamplesaboveinstantiatethis: circlestostandfor wheels,andwheelsto stand for cars; heavyto standfor significanteffort, andsignificanteffort to standfor climbinga steeps/ope.Speakersof a foreignlanguageuseexactlythe samestrategy. Theychoosethe nearest,mostplausibleform theyknowfor the expressionof what theyhavein mind.Therequirementsof communicationarenodifferentin moreusual circumstances,they are simplylessapparent.The interestof sign-makers,at the momentof makingthesign,leadsthemto chooseanaspector bundleof aspectsof the objectto berepresentedasbeingcriterial,at thatmoment,for representingwhatthey want to represent,andthen choosethe most plausible,the mostapt form for lts representation.Thisappliesalsoto the interestof thesocialinstitutionswithinwhich messagesare produced,andthereit takestheform of the (historiesof) conventions andconstraints. APPLICATIONS In theprevioussectionwehavefocusedonthetheoreticalbackgroundof ourwork,butour aimsarenotjusttheoretical.Theyarealsodescriptiveandpractical.Weseekto developa ( 2 )
  • 28. 14 . Introduction descriptiveframeworkthatcanbeusedasa toolfor visualanalysis.Sucha toolwill have its usefor practicalaswellasanalyticalandcriticalpurposes.Togivesomeexamplesof theformer,educationalistseverywherehavebecomeawareof the increasingroleof visual communicationin learningmaterialsof variouskinds,andtheyareaskingthemselveswhat kind of maps,charts,diagrams,picturesandformsof layoutwill be mosteffectivefor learning.Toanswerthisquestiontheyneeda languagefor speakingabouttheformsand meaningsofthesevisuallearningmaterials.Withinthemedia,visualdesignislessandless the provinceof specialistswhohadgenerallyseenlittleneedfor methodicalandanalytic- ally explicitapproaches,and had reliedinsteadon creativesensibilitieshonedthrough experience.But wheremediaformsare relativelyrecentlyintroduced- as isthecase.for example,withadvertisingin EasternEuropeandpartsof Asia- thereisnosuchresistance to combiningsystematicanalysisandpractice.Andwith theadvanceof easyto usesoft- warefor desktoppublishing,the productionof diagramsandcharts,imagemanipulation, etc.,visualdesignbecomeslessof a specialistactivity,somethingmanypeoplewill do alongsideotheractivities.Thishasalreadyledto rapidgrowthinthenumberof coursesin thisarea- anddesigningsuchcoursesrequiresmoreof an analyticalgraspof principles thanlearningonthejob byexampleandosmosis.Last,andmaybeat bottomat therootof muchofthischange,is'globalization',which- maybenearlyparadoxically- demandsthat theculturalspecificitiesof semiotic,social,epistemologicalandrhetoricaleffectsof visual communicationmustbeunderstoodeverywhere,sincesemioticentitiesfromanywnerenow appearandare'consumed'everywhere. Analysingvisualcommunicationis,or shouldbe,an importantpart of the 'critical' disciplines.Althoughin thisbookwe focuson displayingthe regularitiesof visualcom- munication,ratherthan its ('interested',i.e.political/ideological)uses,we seeimagesof whateverkindasentirelywithinthe realmof the realizationsandinstantiationsof ideol- ogy/asmeans- always- for thearticulationof ideologicalpositions.Theplainfactof the matteristhat neitherpowernor itsusehasdisappeared.It hasonlybecomemoredifficult to locateandto trace.In that contextthereis an absoluteneedin democratictermsfor makingavailablethe meansof understandingthearticulationsof poweranywhere,in any form.Thestillgrowingenterpriseof 'criticaldiscourseanalysis'seeksto showhowlan- guageis usedto conveypowerandstatusin contemporarysocialinteraction,andhowthe apparentlyneutral,purelyinformative(linguistic)textswhichemergein newspaperreport- ing,governmentpublications,socialsciencereports,and so on, realize,articulateand disseminate'discourses'as ideologicalpositionsjust as muchas do textswhich more explicitlyeditorializeor propagandize.To do sowe needto beableto'read betweenthe lines',in orderto geta senseof whatdiscursive/ideologicalposition,what'interest',may havegivenriseto a particulartext,and maybeto glimpseat leastthe possibilityof an alternativeview.It is this kindof readingfor whichcriticaldiscourseanalysisseeksto providethe waysandmeans.Sofar,however,criticaldiscourseanalysishasmostlybeen confinedto language,realizedasverbaltexts,or to verbalpartsof textswhichalsouse othersemioticmodesto realizemeaning.Weseeourbookasa contributionto a broadened criticaldiscourseanalysis,andwe hopethat our exampleswill demonstrateits potential for thiskindof work.
  • 29. Introduction . 5 0ur examplesinclude'text-objects'ofmanyI<inds,fromworksof art to entirelyordin- ary,banalartefactssuchas maps,charts,pagesof differentkinds,includingthoseof websites,etc.Wehaveincludedworl<sof art notjustbecauseof theirkeyroleinthehistory of conventionsandconstraints,hencein theformationof the'grammarof visualdesign', butalsobecausethey,too,articulateideologicalpositionsof complexandpotentkinds,and they,too,shouldbeapproachedfromthepointof viewof socialcritique. As is perhapsalreadyobviousfrom what we havesaidso far,we believethat visual design,likeallsemioticmodes,fulfilsthreemajorfunctions.TouseHalliday'sterms,every semioticfulfilsbothan'ideational'function,a functionof representing'theworldaround andinsideus'andan'interpersonal'function,a functionof enactingsocialinteractionsas socialrelations.All messageentities- texts- alsoattemptto presenta coherent'worldof thetext',whatHallidaycallsthe'textual'function- a worldin whichall theelementsof thetextcohereinternally,andwhichitselfcohereswith itsrelevantenvironment.Whether we engagein conversation,producean advertisementor playa pieceof music,we are simultaneouslycommunicating,doingsomethingto,orfor,or with,othersin thehereand nowof a socialcontext(swappingnewswitha friend;persuadingthereaderof a magazine to buysomething;entertainingan audience)and representingsomeaspectof the world 'out there',be it in concreteor abstractterms(thecontentof a film we haveseen;the qualitiesof theadvertisedproduct;a moodor melancholysentimentor exuberantenergy conveyedmusically),andwebindtheseactivitiestogetherin a coherenttextor communi- cativeevent.Thestructureof our book reffectsthis.Chapters2 and3 dealwith the patternsof representationwhichthe 'grammarof visualdesign'makesavailable,and hencewith the wayswe canencodeexperiencevisually.Chapters4 and5 dealwith the patternsof interactionwhichthe'grammarof visualdesign'mal<esavailable,andhence with thethingswe cando to, or for,eachotherwith visualcommunication,andwith the relationsbetweenthe makersandviewersof visual'texts'whichthis entails.Chapter6 dealswiththe'textual'function,withthewayinwhichrepresentationsandcommunicative actscohereintomeaningfulwholes.Chapter7 dealswiththematerialityof visualsigns- the toolswe makethemwith (ink,paint,brushstrokes,etc.)andthe materialswe make themon (paper,canvas/computerScreens/etc.);these,too,contributeto the meaningof visualtexts.Chapter8 extendsthepreviouschaptersintothedomainofthree-dimensional visualsand movingimages.Againwe assumethat there is somethinglil<ea Western 'grammarof three-dimensionalvisualdesign',a setof availableformsandmeaningsused in sculptureaswellas,for instance,inthree-dimensionalscientificmodels,or inchildren's toys- anda Western'grammarof themovingimage'. Wewill begin,however,bydiscussingsomeof thebroaderthemeswehavetouchedonin thisintroduction.
  • 30. I T h es e m i o t i cl a n d l a n g u a g ea n dv i s u a l s c a p e : c o m m u n i c a t i o n In theearlyyearsof schooling,childrenareconstantlyencouragedto produceimages,and to illustratetheirwrittenwork.Teacherscommentontheseillustrationsas muchasthey doonthewrittenpartof thetext,thoughperhapsnotquiteinthesamevein:unlikewriting, illustrationsarenot'corrected'norsubjectedto detailedcriticism('thisneedsmorework,, 'not clear','spelling!','poorexpression',and so on).Theyare seenas self-expression, ratherthan as communication- as somethingwhichthe childrencando arreaoy,spon- taneously,ratherthanassomethingtheyhaveto betaught. Bythetimechildrenare beyondtheirfirst two yearsof secondaryschooling,illustra- tionshavelargelydisappearedfromtheirownwork.Fromhereon,in a somewhatcontra- dictorydevelopment,writingincreasesin importanceandfrequencyandimagesbecome specialized.Thisis mademoreproblematicbythe factsof the presentperiod,in which writingand imageare in an increasinglyunstablerelation.We mightcharacterizethe situationof saytwentyor thirtyyearsagointhisway:textsproducedfor theearlyyearsof schoolingwererichlyillustrated,but towardsthe lateryearsof primaryschoolimages beganto givewayto a greaterandgreaterproportionof writtentext.In asmuchasimages continued,theyhadbecomerepresentationswith a technicalfunction,maps,diagramsor photographsillustratinga particularlandformor estuaryor settlementtype in a geog- raphytextbook,for instance.Thuschildren'sownproductionof imageswaschannelledin thedirectionof specialization- awayfrom 'expression'andtowardstechnicality.In other words,imagesdidnotdisappear,buttheybecamespecializedintheirfunction. In manywaysthe situationin schoolremainsmuchthe same,with two profoundly importantprovisos.0nthe onehandall schoolsubjectsnowmakemuchmoreuseof images,particularlyso in the yearsof secondaryschooling.In manyof thesesubjects, certainlyin themoretechnical/scientificsubjectssuchas(in England)Science,Informa- tion Technologyor Geography,imageshavebecomethe major meansof representing curricularcontent.In the morehumanisticsubjects- for example,History,Englishand ReligiousStudies- imagesvary in theirfunctionbetweenillustration,decorationand information.Thistrendcontinues,andit isthecasefor worksheets,intextbooksandin CD- R0Ms.0ntheotherhand,thereisnoteachingor'instruction'inthe(new)roleof images (thoughin England,in the schoolsubjectInformationTechnology,thereis teachingin desktoppublishing).Mostimportantly,assessmentcontinuesto bebasedonwritingasthe majormode.Studentsarecalleduponto makedrawlngsin Science,GeographyandHis- tory; but,asbefore,thesedrawingstendnot to bethe subjectof the teacher,sattention, iudgingby their (written)commentson the children'swork.In otherwords,materials providedforchildrenmakeintenserepresentationaluseof images;in materialsdemanded fromchildren- invariousformsof assessmentparticularly- writingremainstheexpected anddominantmode. Outsideschool,however,imagesplayan ever-increasingrole,andnotjust in textsfor
  • 31. Thesemiotic landscape . 1,7 children.Whetherintheprintor electronicmedia,whetherin newspapers,magazines,CD- R0Ms or websites,whetheras publicrelationsmaterials,advertisementsor as infor- mationalmaterialsof all kinds,mosttextsnowinvolvea complexinterplayof writtentext, imagesandothergraphicor soundelements,designedascoherent(oftenat thefirst level visualratherthanverbal)entitiesby meansof layout.Butthe skillof producingmulti- modaltextsof this kind,howevercentralits rolein contemporarysociety,is not taught in schools.Toputthispointharshly,intermsof thisessentialnewcommunicationability, this new'visualliteracy',institutionaleducation,underthe pressureof oftenreactionary politicaldemands,producesilliterates. 0f course,writingis itselfa form of visualcommunication.Indeed,andparadoxically, thesignof thefully literatesocialpersonis the abilityto treatwritingcompletelyasa visualmedium- for instanceby not movingone'slipsandnot vocalizingwhenoneis reading,noteven'subvocalizing'(asilent'speakingaloudinthehead',to bringoutthefull paradoxof thisactivity).Readerswhomovetheir lipswhenreading,whosubvocalize,are regardedasculturallyandintellectuallytaintedbyhavingto takerecourseto theculturally lessvaluedmodeof spol<enlanguagewhenreadingvisualscript.This'old'visualliteracy, writing,hasfor centuriesnowbeenoneof the mostessentialachievementsandvaluesof Westernculture,andoneof the mostessentialgoalsof education,so muchsothat one majorandheavilyvalue-ladendistinctionmadebyWesterncultureshasbeenthatbetween literate(advanced)andnon-literate(oralandprimitive)cultures.Nowonderthatthemove towardsa newliteracy,basedon imagesandvisualdesign,cancometo beseenasa threat, a signof thedeclineof culture,andhencea particularlypotentsymbolandrallyingpoint for conservativeandevenreactionarysocialgroupings. Thefadingout of certainkindsof textsby andfor children,then,is not a straight- forwarddisvaluationof visualcommunication,but a valuationwhichgivesparticular prominenceto onekindof visualcommunication,writing,andto onekindof visualliteracy, the'old'visualliteracy.Othervisualcommunicationis eithertreatedasthedomainof a verysmalleliteof specialists,or disvaluedasa possibleformof expressionfor articulate, reasonedcommunication/seenasa'childish'stageonegrowsoutof.Thisisnota valuation of languageassuchovervisualcommunication,becauseevennowthestructures,meanings and varietiesof spokenlanguageare largelymisunderstood,and certainlynot highly valuedrn their varietyin the educationsystem(withsomeexceptions,suchas in formal 'debating')or in publicforumsof power. Tosumup:theoppositionto theemergenceof thevisuafasa full meansof representa- tionisnotbasedonanoppositionto thevisualassuch,butonanoppositionin situations whereit formsan alternativeto writingandcanthereforebeseenasa potentialthreatto thepresentdominanceof verballiteracyamongelitegroups. In thisbookwetal<ea freshlool<at thequestionof thevisual.Wewantto treatformsof communicationemployingimagesasseriouslyaslinguisticformshavebeen.Wehavecome to this positionbecauseof the nowoverwhelmingevidenceof the importanceof visual communication,andthe nowproblematicabsenceof the meansfor talkingandthinking aboutwhatisactuallycommunicatedbyimagesandbyvisualdesign.In doingso,wehave to moveawayfromthepositionwhichRolandBarthestook in his1964essay'Rhetoricof
  • 32. I8 Thesemiotic landscape theimage'(1977i32-51).ln thisessay(andelsewhere,asintheintroductionto Elements of Semiology;Barthes,1967d, he arguedthat the meaningof images(and of other semioticcodes,likedress,food,etc.) is alwaysrelatedto and,in a sense,dependenton, verbaltext.Bythemselves,imagesare,hethought,too 'polysemous',too opento a variety of possiblemeanings.Toarriveat a definitemeaning,languagemustcometo the rescue. Visualmeaningistooindefinite;it isa'ffoatingchainof signifieds'.Hence,Barthessaid,'in everysocietyvarioustechniquesaredevelopedintendedto fx theffoatingchainof signi- fiedsinsucha wayasto countertheterrorof uncertainsigns;the linguisticmessageisone of thesetechniques'i977'.39). Hedistinguishedbetweenan image-textrelationinwhich theverbaltextextendsthemeaningof theimage,or viceversa,asisthecase,for example, withthespeechballoonsin comicstrips,andan image-textrelationin whichtheverbal textelaboratestheimage,orviceversa.lntheformercase,whichhecalledrelay,newand differentmeaningsareaddedto completethe message.In the lattercase,thesamemean- ingsare restatedin a different(e.g.moredefiniteand precise)way,as is the case,for example,whena captionidentifiesand/orinterpretswhatisshownin a photograph.0f the two,elaborationisdominant.Relay,saidBarthes,is'morerare'.Hedistinguishedtwotypes of elaboration,oneinwhichtheverbaltextcomesfirst,sothattheimageformsan illustra' tionof it,andoneinwhichtheimagecomesfirst,sothatthetextformsa moredefiniteand preciserestatementor'fixing'ofit (a relationhecallsanchorage). Beforeapproximately1600(thetransitionis,of c0urse,verygradual),Barthesargued, 'illustration'wasdominant.Imageselaboratedtexts,morespecificallythefoundingtexts of theculture- mythology,the Bible,the'holywrit'of theculture- texts,therefore,with whichviewerscouldbeassumedto befamiliar.Thisrelation,inwhichverbaltextsformeda sourceof authorityin society,andin whichimagesdisseminatedthe dominanttextsin a particularmodeto particulargroupswithinsociety,graduallychangedto onein which nature,ratherthandiscourse,becamethesourceof authority.In theeraof science,images, evermorenaturalistic,beganto functionas'thebookof nature',as'windowsontheworld', as'observation',andverbaltextservedto identifyandinterpret,to'loadtheimage,burden- ingit witha culture,a moral,an imagination'. Thispositiondoesexplainelementsof communication.Anyoneof theimage-textrela- tionsBarthesdescribesmayat timesbedominant,althoughwefeelthattodaythereis a moveawayfrom 'anchorage'.Compare,for example,the'classic'documentaryfilm in whichthe vieweris first confrontedwith'imagesof nature',thenwith the authoritative voiceof a narratorwho identifiesand interpretsthe images,with the modern'current affairs'item,in whichthe vieweris first confrontedwith the anchorperson'sverbaldis- courseand,eithersimultaneouslyor followingon fromtheverbalintroduction,withthe 'imagesof nature'thatillustrate,exemplifyandauthenticatethediscourse.But Barthes' accountmissesan importantpoint:the visualcomponentof a text is an independently organizedandstructuredmessage,connectedwiththeverbaltext,butin nowaydependent onit - andsimilarlytheotherwayaround. Oneimportantdifferencebetweenthe accountwe developin this bookandthat of earliersemioticiansis our useof work in linguistictheoriesanddescriptions.Thisis a difficultargumentto make,but worth makingclearly.Wethinkthatthis bookwouldnot
  • 33. Thesemiotic landscape 1 9 havebeenpossiblewithouttheachievementsof linguistics,yetwedo not,inthewaysome criticsof ourapproachhavesuggested,seeourapproachasa linguisticone.Sowhathave we usedfrom linguistics,andhowhavewe usedit? And,equally,what havewe not used from linguistics?Tostartwiththe latterquestion,we havenot importedthetheoriesand methodologiesof linguisticsdirectlyintothe domainof the visual,as hasbeendoneby othersworkinginthisfield.Forinstance,wedonotmakea separationof syntax,semantics andpragmaticsin the domainof thevisual;we do not lookfor (theanaloguesof) sen- tences,clauses,nouns,verbs,andso on,in images.We takethe viewthat languageand visualcommunicationcan bothbe usedto realizethe'same'fundamentalsystemsof meaningthat constituteour cultures,but that eachdoesso by meansof its ownspecific forms,doessodifferently,andindependently. To givean example,the distinctionbetween'subjective'and'objective'meaningshas playedan importantrole in Westerncultureeversincethe physicalsciencesbeganto developin the sixteenthcentury.Thisdistinctioncanbe realized(that is,givenconcrete, materialexpression,hencemadeperceivableandcommunicable)withlinguisticaswellas visualmeans.Theterms'subjective'and'objective'canthereforebeappliedto both:they belongto the meaningpotentialof a cultureanditssociety.Butthewaythedistinctionis realizedin languageisquitedifferentfromthewayit isrealizedin images.Forexample,in languagean ideacanberealizedsubjectivelybyusinga'mentalprocessverb'likebelieve in the first person(e.9. We believethat thereis a grammar of image); or objectively throughtheabsenceof sucha form (e.9.Thereis a grammarof image).Visualrepresen- tation,too,canrealizebothsubjectivity,throughthepresenceof a perspectivalangle,and objectivity,throughitsabsence,a pointwhichwill bediscussedmorefully in chapter4. Mentalprocessclausesandnominalizationareuniqueto language.Perspectiveis unique to images.Butthe kindsof meanrngexpressedarefrom the samebroaddomainin each case;andtheforms,differentastheyare/weredevelopedinthesameperiod,in responseto the sameculturalchanges.Bothlanguageandvisualcommunicationexpressmeanings belongingto andstructuredbyculturesin the onesociety;the semioticprocesses,though notthesemioticmeans/arebroadlysimilar;andthisresultsin a considerabledegreeof congruencebetweenthetwo. At the sametime,however,eachmediumhasits ownpossibilitiesand limitationsof meaning.Noteverythingthatcanberealizedin languagecanalsoberealizedbymeansof images,or viceversa.Aswellasa broadculturalcongruence,thereissignificantdifference betweenthetwo (andothersemioticmodes,of course).In a languagesuchas Englishone needsto usea verbin orderto makea full utterance(believe,rs);andlanguagehasto use namesto referto whateveris to berepresented(a grammarof images,believe,wd.But languagedoesnothaveor needanglesof visionto achieveperspective,nordoesit haveor needspatialdispositionsof elementsto achievethemeaningsof syntacticrelations:images haveandneedboth.Themeaningpotentialsof thetwo modesareneitherfullyconflated norentirelyopposed.Wedifferfromthosewhoseethemeaningof languageasinherentin the formsandthe meaningof imagesas derivedfrom the context,or the meaningsof languageas'conscious'andthemeaningsof imagesas'unconscior.ls'. To returnto thefirst of our two questions- What havewe usedfrom linguistics,and
  • 34. 20 . Thesemiotic landscape howhaveweusedit?- perhapsthemostsignificantborrowingisouroverallapproach,an 'attitude'whichassumesthat,asa resourcefor representation,images,likelanguage,will displayregularities,whichcanbemadethesubjectof relativelyformaldescription.Wecall thisa'grammar'todrawattentionto culturallyproducedregularity.Morespecifically,we haveborrowed'semioticorientations',featureswhichwetakento begeneralto all human meaning-making,irrespectiveof mode.Forinstance,wethinkihat thedistinctionbetween 'objectivity'and'subjectivity'isa generalcultural/semioticissuewhichcanbe realized linguisticallyas well as visually,thoughdifferentlyso,as we havesaid.0r,as another instance,we havetaken MichaelHalliday'ssocialsemioticapproachto languageas a model,asa sourcefor thinkingaboutgeneralsocialandsemioticprocesses/ratherthanas a minefor categoriesto applyin the descriptionof images.His modelwith its three functionsis a startingpointfor our accountof images,not becausethe modelworkswell for language(whichit does,to anextent),butbecauseit workswellasa sourceforthinking aboutall modesof representation. Maybemostto thepointisthis:ourapproachto communicationstartsfroma social base.In ourviewthemeaningsexpressedbyspeakers,writers,printmakers,photographers, designers,paintersandsculptorsarefirst andforemostsocialmeanings/eventhoughwe acknowledgetheeffectandimportanceof individualdifferences.Giventhat societiesare nothomogeneous,butcomposedof groupswithvarying,andoftencontradictory,interests, themessagesproducedbyindividualswill reffectthedifferences,incongruitiesandclashes whichcharacterizesociallife.It is likely,and in our experienceoftenthe case,that the differentmodesthroughwhichtextsareconstructedshowthesesocialdifferences,sothat in a multimodaltextusingimagesandwritingthewritingmaycarryonesetof meanings andthe imagescarryanother.In an advertisement,for instance,it maybethattheverbal textisstudiously'non-sexist',whilethevisualtextencodesovertlysexiststereotypes.Given the still prevalentsenseaboutthe meaningof images,it is possibleto pretendthat the meaningcarriedin the imageisthereonly'intheeyeof the beholder',somethingthat it wouldnotbepossibleto assertaboutverballyrealizedmeanings. Ourexamplesin this bookarequitedeliberatelydrawnfrom verymanydomains,and from differenthistoricalperiods.We hopethat our ideaswill helpanyoneinterestedin communicationto seein imagesnot onlytheaestheticandexpressive,but alsothe struc- turedsocial,politicalandcommunicativedimensions.We will drawexamplesfrom the kindsof textswhicharealreadyfullybasedonthenewvisualliteracyandplaya dominant rolein anypublicsphere,magazinearticles,advertisements,textbooks,websitesandsoon. Thisisnotbecausewewantto promotethesetextsasa kindof modelwhichshouldreplace otherkindsof texts,butbecausetheirroleinthelivesof childrenandadultsissoimportant thatwesimplycannotaffordto leavetheabilityto ihinkandtalk aboutthem(and,indeed, to producethem)to a handfulof specialists.Wehavea particularinterestin the placeof thevisualinthelivesof children,andwehopeto showthatchildrenveryearlyon,andwith verylittlehelp(despitealltheencouragement),developa surprisingabilityto useelements of the visual'grammar'- an abilitywhich,we feel,shouldbe understoodbetterand developedfurlher,ratherthan beingcut off prematurelyas is,too often,the caseat present;andanabilitythatshouldalsobeavailableto adults.
  • 35. Thesemiotic landscape 2 7 AN UNCONVENTIONALHISTORYOFWRITING Thedominanceof theverbal,writtenmediumoverothervisualmediaisfirmlycodedand buttressedin conventionalhistoriesof writing.Thesegosomethinglikethis.Languagein itsspokenformisa naturalphenomenon,commonto all humangroups.Writing,however, istheachievementof onlysome(historically,byfartheminorityof)cultures.At a particu- lar stagein the historyof certaincultures,theredevelopedthe needto makerecordsof transactionsof variouskinds,associatedusuallywithtrade,religionor (governing)power. Theserecordswereinitiallyhighlyiconic;that is,the relationbetweenthe objectto be recordedandtheformsandmeansof recordingwascloseandtransparent.For instance, thenumberof notchesina stickwouldrepresentthenumberof objectsstoredor tradedor owed.Therepresentationof the objectwouldusuallyalsobe transparent:a wavy line eventuallybecamethe Chineseideogramfor'water';the hieroglyphicimageof the ox's headwhichinitially'stoodfor''ox'eventuallybecametheletteraleph( N),alpha(a),a.This exampleillustrateswhatinthesehistoriesisregardedastherarestof all achievements,the inventionof alphabeticwriting. Alphabeticwritingdeveloped,it seemsclear,outof iconic,image-basedscripts.In these originalscriptforms,an objectwasinitiallyrepresentedbyan imageof that object.Over time,intheuseofthescriptbydifferentgroups,speakingdifferentlanguages,theimageof theobjectcameto standforthenameof theobjectandthenfor itsinitialletter.Aleph,'ox' in Egyptianhieroglyphics,aftercenturiesof travelandconstanttransformationthrough the culturesand languagesof the easternMediterranean,becamethe letteralpha,and eventuallythe lettera in the Romanalphabet.Clearlythiswasa processwhereeachstep involvedconsiderableabstraction,somuchsothat,seemingly,alphabeticwritinghasbeen inventedonlyoncein thehistoryof humancultures.All presentalphabeticscripts,from Indiato the MiddleEastto Europe,aredevelopmentsof that initialstepfrom Egyptian (or possiblySumerian)iconichieroglyphicrepresentationto the Phoenicianalphabet, and from therewestwardto the Greek-speakingworld,and eastwardto the Indian subcontinent,or,in the regionof its origin,developingintothe Arabicversionof the alphabet. Thisis indeedan impressiveculturalhistory,impressiveenoughto havestoodasthe acceptedhistoricalaccountof the achievementof (alphabetic)writing,unquestionedfor centuries.Withinthisaccount,all cultureswithformsof visualrepresentationthatarenot directlyconnectedto languagearetreatedascultureswithoutwriting.However,it isworth investigatingthis history,andin particularthe crucialstepfrom visualrepresentationto the linkwith language,a littlemoreclosely.Priorto thisstep(in realitya development spanningmillennia)thereweretwoseparateandindependentmodesof representation.0ne waslanguage-as-speech;the other,the visualimage,or visualmarks.Eachserveda par- ticularsetof purposessuchasthe constructionof historiesandmyths,the recordingof genealogiesandtransactions,andtherecordingandmeasurementof objects.In thecaseof somecultures,however,the oneform of representation'took over'the other,asa means of recording;that is,visualrepresentationbecamespecialized- onecouldsay,reduced- to functionasa meansof thevisualrepresentationof speech,perhapsin highlyorganizedand
  • 36. 22 Thesemiotic landscaPe bureaucratizedsocieties.At thispointthevisualwassubsumed,takenover,bytheverbalas itsmeansof recording.Consequentlyitsformerpublicuses,possibilitiesandpotentialsfor independentrepresentationdisappeared,declinedandwitheredaway' In the caseof othercUltures,however,thisdevelopmentdid not occur.Herethe visual continued,alongwith theverbalmeansof representation.Instancesof thisabound:from theoneextremeof theIncaquipustrings(sensorilythetactilemodeof representation)to AustrallanAboriginaldrawings,sand-paintingsandcarvings.Theseencode,in a manner not at all directlydependentontor a 'translation'of,verballanguage,meaningsof the culturewhicharedeemedto bebestrepresentedin visualform.Theyareconnectedwith language,or languagewiththem,sothatwall-paintingsor sand-paintings,for instance,are accompaniedbyverbalrecountsof geographicalfeatures,journeystancestormyths,andso on. However,in thesecasesthereis no questionof the priorityof the oneoverthe other mode,andthe visualhascertainlynot becomesubsumedto the verbalas its form of representation. In thisconnectionit is interestingto considerthehistoryof twowordswhichina sense are synonymouswith Westernnotionsof literacy,the wordsgrammarandsyntax'Gram- mar derivesfrom the Greekgrammatike('the art of readingandwriting','qrammar', 'alphabet'),'relatedwordSweregramma('Sign','letter','alphabet'),grammatikosCliter- ate,,,(primary)teacher','grammarian').Thisetymologyrecordsthestateof thingsin the Hellenisticperiod(fromapproximately3008c);in earliertimesthe meaning'sign',as in 'paintedor drawn[etc.] mark'wastheprimarymeaning.In Homer,for example,theverb grapheinstill means'Scratch','scratchin',as in engraving/andfr6m thereit comesto meanboth'writing'and'drawing','painting'.Syntaxis,in pre-Hellenistictimes,meant 'Contract','Wage','OrganizatiOn','System','battlefOrmatiOn',with SyntagmatfOrinStanCe, 'contingentof troops','constitution(ofa state)','bookor treatise'.0nlyinthe Hellenistic perioddoessyntaxiscometo mean(amongits othermeanings)'grammaticalconstruc- tion'.The verbsyntasso,again/meansboth'arrangebattleformations'and 'concentrate (one'sthoughts)','organize','write','compose'. Whilewedonotwishto placetoomuchemphasisonetymology,neverthelessthehistory of thesetwowordswhicharesocrucialto our notionsof literacypointsto formsof social organizationandorder,on the onehand,andto visual'markings'on the other'Together theyindicatethe initiallyquiteindependentorganizationof the modeof imagesandthe modeof verballanguage.At thesametime,the subsequenthistoryof theword grammar bringsout clearlythe subordinationof the visualmediumto the mediumof verballan- guage.Cultureswhichstill retainthefull useof bothmediaof representationare,fromthe oointof viewof'literate cultures',regardedas illiterate,impoverished,underdeveloped, whenin fact theyhavea richerarrayof meansof representationthanthat overtlyand consciouslyavailableto literatecultures.Nevertheless,aswe pointedout earlier,literate culturesdo makeuseof meansof visualcommunicationotherthanwriting,beit thatthey areseenasuncodedreplicasof realityor asa meansof individualexpressionbychildrenor artists.In otherwords,theyare nottreatedaseitherthe expressionsof,or accessibleto meansof readingbasedon,articulated,rationalandsocialmeanings' 0ur unconventionalhistoryof writingis onethat treatsthe comingtogetherof visual
  • 37. Thesemiotic landscape 2 3 andverbalrepresentationasonlyonepossibility/andone/furthermore,thatbringswith it notjustthosebenefitsof writingwhicharewelI enoughunderstood,butalsothenegative aspectsincurredin the lossof an independentform of representation,the diminutionof modesof expressionandrepresentation.Fromthat pointof viewculturessuchasAustral- ianAboriginalculturesareseenas havingbothmodesof representation:thevisual(or perhapsa wholesetof visualformsof representation)andtheverbal.Thepointof this historyis not onlythe politicaloneof underminingthe notionof illiterateculture'(or 'merelyoralculture'),butalsotheattemptto seeto whatextenttheconventionalhistory blindsusto thefactsandusesof visualcommunicationinso-calledliteratecultures. In thisbookwe developthe hypothesisthat in a literateculturethe visualmeansof communicationare rationalexpressionsof culturalmeanings,amenableto rational accountsandanalysis.Theproblemwhichwe faceis that literatecultureshavesystem- aticallysuppressedmeansof analysisof thevisualformsof representation,sothatthereis not,at the moment,an establishedtheoreticalframeworkwithinwhichvisualformsof reoresentationcanbediscussed. THE'OLD'ANDTHE'NEW'VISUALLITERACYIN BOOKSFORTHEVERY Y O U N G Sofar wehavedistinguishedtwo kindsof visualliteracy:oneinwhichvisualcommunica- tionhasbeenmadesubservientto languageandinwhichimageshavecometo beregarded asunstructuredreplicasof reality(the'oldvisualliteracy',inourterms);andanotherin which(spoken)languageexistssideby sidewith,and independentof,formsof visual representationwhichare openlystructured,ratherthan viewedas moreor lessfaithful duplicatesof reality(the'new',in ourterms).Wehavelookedat theseashistoricaland culturalalternatives.But theyalsoexistsideby side,at leastin contemporaryWestern culture,andwesuggestthatweareinthemiddleof a shiftin valuationandusesfromthe onemodeto theother,fromthe'old'tothe'new'visualliteracy,in manyimportantsocial contexts.Theexampleswe will now discusssuggestthat the veryfirst bookschildren encountermayalreadyintroducethemto particularkindsof visualliteracy. Figure1.1 showsa typicaltwo-pagespreadfrom Baby'sFirstBook,a bookwhich,on its insidecover,declaresthat'thetextandillustrations,thoughoversimpleto grown-ups, will satisfytheirti.e.thetoddlers'lcravingsfor therepetitionof whattheyalreadyknow, andwill helpthemassociatethewordswiththeobjects'.Whenwewrotethefirstversionof thischapter,in 1989,it wasstill widelydistributed,andtodayit is alreadymakinga comebackasanobjectof nostalgia. Figure1.2 showsa typicalpagefrom Dicl<Bruna's0n My Walk.Thisbookis oneof a setof four,theothersbeingIn My Home,In My ToyCupboardand0n theFarm.lt consists of eightpagesand,with theexceptionof thefront andbackcovers,the pagescontainno wordswhatsoever. Comparedto the pictureof the bird in the tree,the pictureof the bath is realistic, detailedand complex.If we wereto analyseit into its components,if we wereto try
  • 38. Thesemiotic landscape {-l - , ' - - - , , . - , , i . - _y * i ' 1 d -,J i;*iir*o** i'i:u **:,::1 f: i,; i: i: , -"i i i'lii !ii:l f-Ttili 3 r !,-,,t E "r O Figl.l My bath(from Bahls Fitst Boo&Ladybird) andidentifyall thedifferentelementsof thispicture,we mightencounterproblems.Are the ripplesin the waterto becountedascomponents?Are the shadows,castbythetub andtowel?And if we wereto try and identifythe relationsbetweenthesecomponents, what wouldwe haveto say,for example,aboutthe relationbetweenthe duckandthe soap?We ask thesequestionsbecausetheyare the kindsof questionswith whichone might start if one wantedto showthat imagesare structuredmessages,amenable to constituentanalysis.Isn't the structureherethat of the culturalobject'bathroom', ratherthanoneimposedbytheconventionsof a visualcode?Isn'tthispictureunproblem- atically,transparentlyreadable(recognizable),providedoneknowswhatbathroomslook like? Thisisthe lineParisSchoolsemioticianssuchas RolandBarthesandChristianMetz tookinthe1960s.Commentingonphotography,Barthessaid: In orderto movefromtherealityto itsphotographit isin nowaynecessaryto divide upthisrealityintounitsandto constitutetheseunitsassigns,substantiallydifferent fromtheobjecttheycommunicate....Certainly,theimageis nottherealitybutat leastit is its perfectanalogonandit isexactlythisanalogicalperfectionwhich,to commonsense,definesthe photograph.Thuscanbe seenthe specialstatusof the photographicimage:.it is a messagewithouta code. (Barthes.1977:l-7) !-:*I i
  • 39. Thesemiotic landscape . 25 Q fig f.Z Birdin tree(Bruna,t988) Andheextendsthisargumentto otherpictorialmodes,albeitwitha qualification: Are thereothermessageswithouta code?At first sight,yes:preciselythe whole rangeof analogicalreproductionsof reality- drawings,paintings,cinema,theatre. However,eachof thosemessagesdevelopsin an immediateand obvious supplementarymessage... whichis what is commonlycalledthe style reoroduction. (Barthes,1977:17) Thepictureof the bird in thetree,onthe otherhand,is muchlessnaturalistic,muchless detailedandmuchsimplerthanthepictureofthebathroom.It isstylizedandconventional, andquiteclearlya'coded'image.No depth,no shadows,no subtlenuancesof colour: everythingisplainandboldandsimple.Andthestructureoftheimage,withitsonecentral andfourmarginalimages,doesnotimitateanythingintherealworld.It isa conventional visualarrangement,basedona visualcode.Asa resultthecomponentsof thewholestand out asseparate,distinctunits,andthe picturewouldseemquiteamenableto constituent analysis.Thisis not just a matterof style:the structureof this picturecouldalsobe realizedin moredetailedstyles.Bruna'sbookdatesfrom 1953,well beforethe era of computer'imaging',butthepictureofthebirdinthetreecouldhavebeencomposedwitha computer,aligningready-madesimpleiconsina compositionalconfiguration- it isinfact quitesimilarto thecomputer-drawndinnerinvitationinfiqure1.3. way a of the
  • 40. Thesemiotic landscape @ fig f.: Computer-drawndinnerinyitation Second,the pictureof thebathroomis part of a two-pagelayout,andaccompaniedby words.Languagecomesfirst,authoritativelyimposingmeaningon the image,turningit intoa typicalinstanceof a bathroombymeansof thegenericlabel'Bath'.Asa resultthe picturecouldbe replacedby otherimagesof bathroomswithoutmuchlossof meaning (oneverbaltext,manyimages,manypossibleillustrations).Herelanguageis general, bestowingsimilarityandorderon the diverse,heterogeneousworldof images.Thusthe bookpresents,ontheonehand,an'uncoded',naturalisticrepresentation('theworldasit is'- empirical,factual,specific)and,on the otherhand,a specific,authoritativelypre- scribedwayof readingthis'uncoded'naturalisticpicture.Wewillshowlaterthat,contrary to what Barthesandothersarguedinthe 1960s,picturesof thiskindarealsostructured, whetherthey are photographs,drawings,paintingsor otherkindsof pictures.For the moment,however,theimportantpointisthattheyarenotusuallyinterpretedassuch,that awarenessof thestructurednessof imagesof thiskindis,in oursociety,suppressedandnot oartof tcommonsense'. In DickBruna's0n My Walk,bycontrast,therearenowordsto authoritativelyimpose meaningon the image,andthe imageis no longeran illustration:the imagecarriesthe meaning,thewordscomesecond.Parentswhoreadthisbookwiththeirchildrencouldall tell a differentstory,couldevenusedifferentlanguages(oneimage,manyverbaltexts). irrulNer{1'
  • 41. Thesemiotic landscape Theworldof '0ne image,manydifferentverbaltexts'('commentariesi)imposesa new modeof controlovermeaning,and turnsthe image,formerlya recordof natureor a playgroundfor childrenandartists,intoa morepowerful,but alsomorerigorouslycon- trolledandcodifiedpubliclanguage,whileit giveslanguage,formerlycloselypolicedin manysocialinstitutions,a moreprivateandlesscontrolled,butalsolesspowerful,status. The'readings'whichparentsproducewhentheyread0n My Walkwiththeirchildrenmay all bedifferent,yetthesedifferentreadingswill necessarilyhavecommonelements,deriv- ing from their commonbasis- the elementsincludedin the image,andthe way these elementsarecompositionally broughttogether. Whateverstoryparentswilltellaboutthepagewiththebirdinthetree,it will necessar- ily haveto bea storythat createsa relationbetween,for instance,birdsandaeroplanes (natureandtechnology)andbirdsandcats(preyandpredator).It will alsohaveto bea storyinwhichthebird,safelyin itstree,isthecentralcharacter,literallyandfiguratively.In howmanywayscancatsandbirdsberelated?Notthat many,at leastnot if oneassumes that bookslike0n My Walkserveto introducechildrento theworldaroundthem,rather thanto the possibleworldsof fantasiesandutopias.Catscan'hunt,,'torture,,.kill,and'eat' birds.Birdscan'escape'catsor fail to do so.Thereare not that manystoriesto choosefrom.0ntheotherhand,parentsandtheirchildrencanchoosetheorderin which theywantto dealwiththevariouselements:the pageis'non-linear,.It doesnot imposea sequentialstructure.Andtheycanchoosewhetherto tellthestoryof thebirdandthecat as a politicaistory,a story of powerfulpredatorscomingfrom anothercontinentand nativebirdskilledandthreatenedwithextinction(asmightbedone,for instance,in Aus- tralia),or asa storythat legitimizesthesurvivalof thefittest.Thestoryof thebirdandthe aeroplane,similarly,maybetoldfroman environmentalistpointof view,or asa storyof evolutionarytriumphsand humantechnologicalprogress.Evenwheresuchdiscourses are not explicitlyinvoked,theywill still communicatethemsejvesto childrenthrouqhthe parents'attitudestowardsthecharactersandtheactions. Not onlytheelementson the individualpages,but alsothe pagesthemselvesmustbe broughtin relationto eachother.Thebookas a wholemustbe readableas a conerent sequence.Thisispromptedbythetitle(0n My Walb aswellasbythepictureonthefront cover,whichshowsall the elementstogether.We haveinvestigatedthis a littlefurtherin connectionwith anotherbool<in the Brunaseries,0nthe Farm.Thisbookcontainsthe followingcentralpictures:house,farmer,cat,dog,appletree,rooster,lamb,cow.Listingthe waysinwhichthesepicturescanplausiblybelinkedto eachother,wefoundthatsome(e.g. theappletreeandthehouse)canonlybelinkedin spatial,locativeterms(e.g.theapple treeisnextto thehouse).Others(e.g.theanimalsandthehouse)canberelatedbyverbsof'dwelling'(e.9.thecowliesundertheappletree)or by theverbsof 'motion,(e.g.thecat climbsuptheappletree).Twoof theanimals(thecatandthedog)canrelateto theother animalsandto eachotherby meansof antagonisticor co-operativeactions(e.g.thedog barksat the cow;the dog leadsthe sheep).only the farmercan relateto all the other elementsin an agentiveway.He canbuythem,ownthem,buildthem,growthem,keep them,raisethem,harvestthem,shearthem,slaughterthem,andso on.In otherwords, whateverwaythe parentsreadthesepictures,theywill, in the end,haveto dealwith the
  • 42. Thesemiotic landscape themeof spatialorder,thethemeof socialinteraction(projectedonto animals)andthe themeof humanmasteryovernature(aswellas,viathemarginalpictures,withthetheme of procreation),andtheywill haveto doallthisintermsof theelementspre-selectedbythe book.An analysisof the way the elementscan be opposedto eachothershowsthat, whatevertheclassificationsparentsmayconstruct,theywill notbeableto avoidengaging withtheWesternculturaldistinctionsbetween'untamednature','domesticated/cultivated nature'and'humantechnology'.And theywill alsohaveto recognizethe distinction betweenanimateandinanimate,floraandfauna,andbetweenpets,farmanimalsandwild animals. It shouldbenoted,however,that everypagein the book(andin Bruna'sotherbooks) containsat leastonerelationthatdoesnoteasilyfit thereceivedclassifications,thatforms somewhatof a challengeandapuzzle.What,for example,istherelationbetweena rabbit anda basketof ffowers?A beetleanda fence?Suchvisualenigmascanchaliengeparents andchildrento exercisetheirimagination,to includeintheirthinkingelementsthatdonot easilyfit in with the traditionalorderof things,to toleratesomeambiguity,to allowthe inclusionofthe'other'intheirconstructionof theworld. Thetwo books,then,areverydifferentin their stancetowardsthe image.TheBruna stancepresentshighlyprocessed/essentializedandidealizedrepresentations,andprovides parentsand,later,childrenwiththeopportunityto talk abouttheimagesinwayswhichare or seemappropriateto them,to applyspecificvalues,specificdiscoursesto theserelatively abstractimages.TheLadybirdstancepresentsostensiblylessprocessed,morenaturalistic visualrepresentationsandprovidesparents(and,lateron,children)witha specificverbally realizedway of readingthe image.The Ladybirdbookis openand interactivefrom the perspectiveof the image,and authoritarianfrom the perspectiveof writing;the Bruna bookworksin the oppositeway.Theclosurein the Brunabookliesin the limitswhich selection,formandstructureof theimagesimposeontheapparentlyopenreadings-these enterthediscourseswhicharealready'in'thesocializedparents,sothatthewhole,once orallytransmittedto thechildren,will appearspontaneousand'natural'toparentsand childrenalike.Aretheynot,afterall,merelyengagedin an innocentreadingof 'whatis there'in the pictures?Thusthetwo booksreoresenttwo differentformsof socialcontrol overmeaning.0neisopenlyandexplicitlylocatedinthetextitself;theotherlies,perhaps morecovertlyand implicitly,in the way the bookpresentsitselflessas a text than as an organizedresourcefor makingtexts,jointlywiih the parentaldiscoursesthat will inevitablyenterthetextaswell. Thesediscourses,however,arenotthemselvespart of Bruna'sbooks,of thepublictext meantto transcendtheirdiversity.Insteadtheyarerelegatedto therealmof theprivate,of 'lifestyles'wheretheydo notthreatentheorderof the largersocialworld.Thereis never just'heteroglossia'(manymeanings),noreverjust'homoglossia'(oneauthoritativemean- ing).Insteadthereisa roledistributionamongthedifferentsemiotics,a roledistributionin whichsomesemioticsare givena greatdealof socialpower,but at the priceof being subjectedto greaterinstitutional(andtechnological)control,whileothersare allowed relativefreedomfrom control,but payfor thiswith diminishedpower.Today,we seemto movetowardsa decreaseof controloverlanguage(e.9.the greatervarietyof accents
  • 43. Thesemiotic landscaPe ' 29 allowedonthepublicmedia,theincreasingproblemsin enforcingnormativespelling),and towardsan increaseincodificationandcontroloverthevisual(e.g.theuseof imagebanks from whichready-madeimagescan be drawnfor the constructionof visualtexts,and, generally,theeffectof computerimagingtechnology). Thetwo formsof controlovermeaningcan be foundelsewherealso.Compare,for example,theclassicdocumentaryfilm- in whichanauthoritative'voiceof God'narrator explainsandinterpretsimagesof recordedreality- to the moremodern'directcinema' documentary,in whichcontrolovermeaningliesin the selectionof imagesand in the sometimeshardlynoticeablewaysinwhichtheseimagesareeditedtogether'0r thinkofthe wayinwhich,inthefieldof 'culturalstudies',anemphasisonanalysing'whatthetextsays' isgraduallybeingreplacedbyanemphaslson'howdifferentaudiencesreadthesametext', anemphasis/in otherwords,ontheapparentfreedomof interpretationwhich,bydiverting attentionawayfrom thetext itself,allowsthe limitationswhichthe text imposeson this 'freedomof reading'toremaininvisible,andtherefore,perhaps,all the moreefficacious andpowerful. In thisconnectionthebackgroundoftheBrunabookisworthbriefmentlon.It wasfirst printedin 1953,inAmsterdam,andreprintedmanytimesin itscountryof origin.Thefirst Britishprintingwasin 1978.Thetimelagis perhapsnoaccident.Unlikethe British,the Dutchhad,earlyin the twentiethcentury,recognizedthat their countrydid not havea 'commonculture',butwasdividedintogroupscharacterizedbydifferentandoftenoppos- ingldeologies,zuilen(literally'columns','pillars'),asthe Dutchhavecalledthem.Dutch broadcasting,for example,hadfrom its inceptionin the late 7920sa systemin which differentlevensbeschouwelijkegroups(i.e.'groupsorientatedtowardsa particularviewof life,)ran broadcastingorganizationswhichwereallottedair time accordingto the size of their membership.Thusthe sameeventswould,on radioand lateron television,be interpretedfrom a varietyof differentdiscursiveideologicalpositions,whilemostother Europeancountrieshadcentralized,usuallygovernment-run,broadcastingorganizations withoneauthoritativemessage.Fora messageto reach,inthiscontext,thewholepopula- tion,it hadto beadaptableto a varietyof culturalandideologicalconstructionsand,aswe haveseen,the Brunabool<susethe visualmediumto achieveexactlythat. Perhapsthe belatedsuccessof theseriesin countrieslikeBrltalnandAustraliashowsthatthereisnow, in thesecountriestoo,an increasingawarenessthattheyno longerhavea 'commonCul- ture',andthat,instead,theyhavebecomecomplex,diverseanddiscursivelydivided,and thereforein needof newformsof communication(althoughthe Dutchzuilensystemwent indeclinefromthe1960sonwards). Thechanglngdistributionsof meaningbetweenlanguageandimage,whichwesuggest is nowin full flow,wasforeshadowedby variousexperimentsin the SovietUnionof the earlyI920s.Whilelinguistsand literaryscholarslil<eVoloshinovand Bakhtinwroteof languageas sociallydivided,'multi-accentuat'and'heteroglossic',constructivistartists likeMalevich,El LissitzkyandRodchenko,andfilm-mal<erslikeEisenstein,rejectednat- uralismanclbeganto elaboratea newvlsuallanguage,capableof communicatingnew, revolutionaryideasvisually.Then,asnow,imagesbecamemorestylized,moreabstractand moreobviouslycoded:the newvisuallanguagewasexplicitlycomparedwith language,
  • 44. 30 Thesemiotic landscape with hieroglyphicwriting,with the stylizedmasksof kabukitheatre.Then,asnoWvisual communicationwasalsoseenas transparent:coloursandshapeswerethoughtto havea direct,unmediated,'psychological'impact,a non-semioticcapacityfor stirringtheemo- tionsof the 'masses'.Then,as now,visualcommunicationwasto be removedfrom the sphereof art,to becomepart of the morepowerfulandmorepublicsphereof industrial production,of typography,design,architecture.Thissemioticrevolutionwasalliedto the politicalrevolution:constructivistpostersandfilmshada propagandisticpurpose- they soughtto helpbringabouta culturalrevolution,andtheyhadto gettheirmessageacross to a sociallyandlinguisticallyheterogeneouspopulation.Thevisual,thoughtto beableto produceanemotiveimmediacy,wasto bethemediumthatcouldachievethis.In theend, thenewsemioticorderfailedto establishitselfpermanently.It wascrushedbyStalin.0ld- fashionedcentralistandrepressivecontrolovermeaning(andwith it a returnto naturalist, 'bourgeois'art)prevailedovercontrolbymeansof a formof propagandathatcouldallow pluralismandideologicalcohesionto coexist.Thistime-thoughwithverydifferentpolit- ical,social,technologicalandeconomicconditions- it maynotfail. Thesemioticshiftswehaveexemolifiedinourdiscussionofthetwochildren'sbookscan beobservedelsewhere,too.Theshiftfrom 'uncoded'naturalisticrepresentationsto styl- ized,conceptualimagescanbeseen,for instance,onthecoversof newsmagazines,which usedto be dominatedby documentaryphotographs- photographsrecordingevents,or portrayingnewsworthypeople.Occasionallythisstillhappens,asinfigure1.4,butincreas- inglythe photographson magazinecoversare contrivedand posed,usingconventional symbolsto illustratetheessenceof an issue,ratherthandocumentingnewsworthyevents, as in figure1.5,wherea padlockanda UnitedStatesflag,againsta neutralbackground, illustratethe issueof tightenedbordercontrol.Unlikethe pictureof the 'Bird in Tree' (figure1.2),thesearestillphotographicimages,buttheymightaswellbedrawings. As an exampleof the changingrelationbetweenlanguageand image,consideran extractfrom a ScienceCD-ROMfor the loweryearsof highschool(figure1.6).Here languagehasherebeendisplacedbythevisualasdecisivelyasinthe Brunabook.Instead of the majormediumof information,with the visualas'illustration',it hasbecomea mediumforcommentor labelling,withthevisualasthecentralsourceof information.Two questionsneedasking:oneisthequestionof implicitchangesin notionsandpracticesof reading,andof readingsciencein particular;the otheris the questionof changesin the constitutionof what is representedhere,scienceitself.The students/viewers/usersof the scienceCD-R0Mare no longeraddressedvia the hierarchicallycomplexstructures of scientificwriting,with its specificdemandsfor cognitiveprocessing,and its needto 'translate'verbalformsto theirthree-dimensionalor visualequivalents(asonthe page reproducedin figure1.7).Theyare addressedlargelyin the visualmode,andeitheras 'scientists'whounderstandabstractionfromtheempiricallyreal,or aspeoplefocusingon the empiricallyrealwiththe intentionto understandthe regularitieslying'behind'that reality.In otherwords,eventhoughthevisualmodemightseemto providedirectaccess to theworld,it isasamenableto realizingtheoreticalpositionsasistheverbal. Morecomplexis the questionwhether,in thisdramaticshiftfromthe verbalto the visual,theveryconstitutionof theschoolsubjectScienceis undergoinga transformation.
  • 45. Thesemiotic landscape . 3l @ fiq f,l Magazinecoverwith naturalisticphotograph(Newsweek,lgApril2004) Caneverythingthat wascommunicablein the formationof scientificwritingbe saidin thesevisuallyconstructedforms?Conversely,aretherepossibilitiesof scientificcommuni- cationinthevisualwhichwerenotavailableinthemodeof writing?Andwhichof theseis a moreapt mediumfor scientifictheory?Will scientifictheorieschangeastheformof expressionshiftsfromthewrittento thevisualmode?Wecannottal<eupthesequestions here,but if we areto makeourselvesconsciousof the far-reachingimplicationsof these changesinthesemioticlandscape,theyneedat leastto beasked. Implicitinthisisa centralquestion,whichneedsto beputopenly,anddebatedseriously: isthe movefromtheverbalto thevisuala lossor a gain?0ur answerat thisstagein our thinkingis multiple.Thereare losses,andtherearegains.0urargumentthroughoutthis bookisthatdifferentsemioticmodes- thevisual,theverbal,thegestural,etc.- eachhave theirpotentialitiesandtheirlimitations.A movefroma centralrelianceononemodeto a centralrelianceon anotherwill thereforeinevitablyhaveeffectsin bothdirections.But that isnottheendof thestory.Wealsohaveto considerwhatisrepresented.It maybethat visualrepresentationis moreapt to the stuffof sciencethan languageeverwas/or even thata sciencewhichis constructedvisuallywill bea differentkindof science.Theworld representedvisuallyonthescreensofthe'newmedia'isa differentlyconstructedworldto thatwhichhadbeenrepresentedonthedenselyprintedpagesof the printmediaof some
  • 46. Thesemiotic landscape @ fi9 f.S Magazinecoverwith conceptualphotograph(ltewsweek,l2November200I) thirtyor fortyyearsago.Theresourcesit offersfor understandingandfor meaning-making differfromthoseof theworldrepresentedin language,andsodothecitizensit produces. Theseare far-reachingquestionsandtheycan onlybe answeredby consideringthe interconnectionsbetweenthe changingpolitical,economicandculturalconditionsgath- eredup underthe labelof globalizationand the newpossibilitiesfor representation affordedbythenewmediaof productionanddissemination.Wehavebarelyhintedat these kindsof questionsin ourdiscussionof theBrunaandLadybirdbooks.Couldit bethecase that informationis nowso vast,so complex,that perhapsit,hasto be handledvisually, becausetheverbalisno longeradequate? Merenostalgia/meresocialandculturalregretsor pessimismcannothelphere.We,all of us,haveour particularstandpointsand our particularvaluescarriedforwardfrom yesterdayor from the day beforeyesterday.Thefirst most importantchallengeis to understandthisshift,inallof itsdetail,andinallof itsmeaning.Fromthatunderstanding, wecanhopeto beginthetaskof constructingadequatenewvaluesystems. Tosummarize: (1) Visualcommunicationis alwayscoded.It seemstransparentonlybecausewe know thecodealready,at leastimplicitly- butwithoutknowingwhatit isweknow,without
  • 47. Thesemiotic landscape O fin f.O ContemporaryscienceCD-ROM Q fig f.Z Earlytwentieth-centurysciencetextbook(McKenzie,Ig3g) havingthemeansfor talkingaboutwhatit iswedowhenwereadan image.A glance at the'stylized'artsof otherculturesshouldteachusthatthemythof transparencyis indeeda myth.Wemayexperiencetheseartsas'decorative,,,exotic,,'mysterious,or (e)
  • 48. Thesemiotic landscape 'beautiful',but we cannotunderstandthemascommunication/asformsof 'writing' unlessweare,or become,membersof thesecultures. (2) Societiestendto developexplicitwaysfor talkingonlyaboutthosesemioticresources whichtheyvaluemosthighly,andwhichplaythemostimportantrolein controlling thecommonunderstandingstheyneedin orderto function.Untilnow,language,espe- ciallywrittenlanguage,hasbeenthemosthighlyvalued,themostfrequentlyanalysed, themostprescriptivelytaughtandthemostmeticulouslypolicedmodein oursociety. If, as we haveargued,this is now changingin favourof moremultiplemeansof representation,with a strongemphasison the visual,then educationalistsneedto rethinkwhatwill needto be includedin the curriculaof 'literacy',whatshouldbe taughtunderitsheadingin schools,andconsiderthenewandstillchangingplaceof writingasa modewithinthesenewarrangements. If schoolsareto equipstudentsadequatelyfor the newsemioticorder,if theyare notto producepeopleunableto usethenewresourcesof representationactivelyandeffectively, thentheoldboundariesbetweenthemodeof writingontheonehand,andthe'visualarts' onthe other,needto be redrawn.Theformerhadtraditionallybeenthat form of literacy withoutwhichpeoplecouldnot adequatelyfunctionascitizensor asworkers;the latter hadbeeneithera marginalsubjectfor the speciallygifted,or a subjectwith limitedand specializedapplications,as in'technicaldrawing'.Thenewlydefinedareawill haveto involvethetechnologiesof the'newscreens'-theelectronictechnologiesof information andcommunication,centralnowto thesemioticlandscape.But aboveall,suchacurric- ulumiscruciallydependentonhavingthemeansof analysis,themeansfor talkingabout the'new lileracy',aboutwhat it is we do whenwe produceandreadimages.As Iedema (I994: 64) notes,inthe'post-Fordist'workpIace, Workersmustbe multi-skilled,articulateand'self-steering'.. .. tTheyJnegotiate theirjobsasmembersof 'qualitycircles'andconsultativecommittees.Thisrequires that workersare not merelycapableoI doingtheir work,but alsothat theyare capableof talkingandthinkingabouttheirworkanditseffectiveness. Elsewhere(l(ress,2000;l(ressandvanLeeuwen,200L)wehavetalkedof theneedforthe introductionof the conceptof design,both as a categorywith generalsignificancein representationandcommunication,andasa crucialcategoryfor developingthecurricula of institutionalizededucation,whetherin thetraditionalschoolor otherformalsitesof learning.Thisis implicitalsoin thedescriptionwe havegivenearlierinthischapterof the newformsof reading.Thisis notthe placeto developthat point,thoughit is essentialto drawattentionto its unavoidablesignificanceas part of the urgentneedfor developing adequatewaysandtalkingaboutthevisual.
  • 49. Thesemiotic landscape t a THE SEMIOTICLANDSCAPE Theplaceof visualcommunicationina givensocietycanonlybeunderstoodinthecontext of,onthe onehand,therangeof formsor modesof publiccommunicationavailableinthat societyand,onthe otherhand,theirusesandvaluations.Wereferto thisas'thesemiotic landscape'.Themetaphorisworthexploringa little,asis itsetymology.Thefeaturesof a landscape(afield,a wood,a clumpof trees,a house,a groupof buildings)onlymakesense in the contextof theirwholeenvironmentandof the historyof its development('waste land'hasmeaningonlyinthatcontext,ashas'field'or'track';'village'hasmeaningonlyas a groupof buildingsthat ispartof a historyof waysof workingtheland).In thesameway, particularfeaturesandmodesof communicationshouldbeseenin the historyof their develooment.and in the environmentof all the othermodesof communicationwhich surroundthem.Theuseof thevisualmodeis notthesamenowasit wasevenfifty years agoin Westernsocieties;it is notthesamefrom onesocietyto another;andit is notthe samefromonesocialgroupor institutionto another. Eachfeatureof a landscapehasitshistory,asdoesthelandscapeasa whole,andeach is subjectto constantremaking.It is herethat the etymologyof the word landscapeis revealing.To the casualbeholdera landscapesimplyis,andmayevenhavea timeless appearance('thetimelessbeautyoftheEnglish,or Spanish,countryside').Yetit isinfacta productof socialactionandof a socialhistory,of humanwork on the land,on nature: -scape,withitsrelationto shapein Englishandschaffen(both'to work'and'to create')in German,indicatesthis.Andthisappliesalsoto the'semioticlandscape'.Metaphoricexcur- sionsofthiskindcanbestretchedtoofar;however,wewillallowourselvesoneotherpoint of comparison.Landscapesarethe result,notjust of humansocialwork,but alsoof the characteristicsof the landitself.Theflat landbythe riveris mostsuitablefor thegrazing of cattleor thegrowingof wheat;the hillsidesfor vineyardsor forestry.At thesametime, thecharacteristicvaluesof a culturemaydeterminewhichof thepotentialusesof theland are realized,whetherthe hillsidesare usedfor vineyardsor forestry,for example.And culturalvaluesmayeveninducepeopleto goagainstthegrainof theland,to usethesteep hillsidefor growingrice,for example,whichopposesthe'naturalpotential'ofthe land almostto thelimit. Semioticmodes,similarly,areshapedbothbytheintrinsiccharacteristicsandpotential- itiesof the mediumandbythe requirements,historiesandvaluesof societiesandtheir cultures.Thecharacteristicsofthemediumof airarenotthesameasthoseofthemedium of stone,andthepotentialitiesof thespeechorgansarenotthesameasthoseofthehuman hand.Nevertheless,culturalandsocialvaluationsandstructuresstronglyaffecttheusesof thesepotentialities.It isnotanaccidentthatin Westernsocietieswrittenlanguagehashad theplacewhichit hashadforthelastthreeor fourmillennia,andthatthevisualmodehas in effectbecomesubservientto language,as its modeof expressionin writing.Western linguistictheorieshavemoreor lessnaturalizedtheviewthattheuseof air andthevocal organsisthe natural,inevitablesemioticmeansof expression.But evenspeechis,in the end,cultural.We are not biologicallypredisposedto usespeechas our majormodeof communication.Thepartsof thebodythatwecallthetspeechorgans'areanadaptationof
  • 50. Thesemiotic landscape physicalorgansinitiallydevelopedto preventhumansfromchokingwhilebreathingand eating.Whentheneedarises,wecananddouseothermeansof expression,asinthehighly articulateddevelopmentof gesturein signlanguages,andalsoin theatricalmimeand certainEasternformsof ballet.And,whiletheseare at presentrestrictedto relatively marginaldomains,whoisto saythatthiswill alwaysremainsointhefuturedevelopment of humankind?It issalutaryto considerhowothercultures'rank'modesof communica- tion,and to bringthat knowledgeinto the mainstreamof 'Western'thinking(see,for instance,Finnegan,2002). Thenewrealitiesof thesemioticlandscapearebroughtaboutby social,culturaland economicfactors:by the intensificationof linguisticand culturaldiversitywithinthe boundariesof nationstates;bytheweakeningof theseboundarieswithinsocieties,dueto multiculturalism,electronicmediaof communication,technologiesoftransportandglobal economicdevelopments.Globalflowsof capitalandinformationof all kinds,of commod- ities,andof people,dissolvenotonlyculturalandpoliticalboundariesbut alsosemiotic boundaries.Thisisalreadybeginningto havethe mostfar-reachingeffectsonthecharac- teristicsof English(andEnglishes)globally,andevenwithinnationalboundaries. Theplace,use,functionandvaluationof languageinpubliccommunicationischanging. It ismovingfromitslormer,unchallengedroleasthemodeof communication,toa roleas onemodeamongothers,to thefunction,for instance,of beinga modefor comment,for ratification,or for labelling,albeitmoreso in somedomainsthan in others,andmore rapidlyinsomeareasthaninothers.Althoughthisisa relativelynewphenomenoninpublic communication,childrendoit quite'naturally' intheirtext-making. Newwaysof thinkingare neededin this field.Herewe use/oncemore,children,s representationas a metaphorto suggestsomedirections.Thedrawingsreproducedin figure1.8 weremadeby a five-year-oldboy.0na summerSundayafternoon,whilehis parentswereentertainingfriends,the childtook a small,squarenotepadfrom nearthe telephoneanddrewa pictureoneachof sixpages.Hisfatherhadnotnoticedthisuntilhe cameacrosshiminthehallof theirhouse,wherethechildwasputtingthecards'in order', asshownin figure1.8.Askedwhat hewasdoing,the child'saccountwasasfollows:for picturesI and2 together'Meandthe dogare in life,sothey'rein thecorrectorder,;on pictures3 and4 'Theffyingbombis intheair andtheplaneis intheair,sothey,reinthe correctorder';andon5 and6'The patternsareinthecorrectorder,. Thewholeprocess/involvingsign-making,representationandclassification,hadpro- ceededthroughthe visualmedium.It wasonlywhenthe parentcamealongwith his questionthatthechildwasforcedto usewords.Themetaphoricprocessesof sign-making, theactsof representationandclassification,eachinvolvingquitecomplexanalogies,took placein the visualmode.Language,as speech,enteredwhencommunicationwith the parentbecamenecessary.Speechwasthemodeusedfor'ratifying'andfor describingwhat hadtakenplacewithoutit. Sometwo weekslater,at the endof the summerterm of hisprimaryschool,the child broughthomesomeof hisexercisebooks.Amongthesewasthepageshownin figure1.9. Clearly,herethetaskwasoneof classification,andit hadbeenundertakenat school,prior to the makingandorderingof thedrawingsin figure1.8,at home.A wholesequenceof
  • 51. Thesemiotic landscape 3 7 4,,./ t t p l*gL a'r fu" uu^' ..MEANOTHEDOGAREIN LIFE,SOTHEY'REIN THECORRECTORDER" ..THEFLYINGBOMBIS IN THEAIRANDTHEPLANETSIN THEAIR,SOTHEY'REIN THECORRECTORDER'' Y- r,zN!,1 /U,N ,,THEPATTERNSAREII{ THECORRECTORDER'' O Fig1.S Sixdrawingsbya five-year-oldboy semioticactivitiesisthusinvolved,a sequenceof production,transformationanddevelop- ment,movingfromthe initialtaskof joiningimagesof thesameobjects- a classificatory, cognitive,conceptual,semioticandmanualtask- to that of producingcomplexanddis- similarimages,andfindinglikenessin them(or imposinglikenesson them)throughan intermediarytaskof abstractionandgeneralization.If wethinkaboutthis periodof two weeks,thechild'sproductionof signsinvolveda seriesof distinctsemioticmodes,andof translationsbetweensuchmodes.Firsttheteachersnokewiththechildrenaboutthetask (mode:languageasspeech);thensheintroducedthe bookandshowedthemwhatwasat issue(mode:3Dphysicalobject,andvisualmode);thenthechildrenusedtheirpencilsto drawthe connectinglines(mode:manualactionandvisualmodeof drawing);thenthe teacherengagedthechildreninspokendiscussion,andmadeevaluativecommentsontheir
  • 52. 38 Thesemiotic landscape 1 5 { / - - 1 / s . .L ll APtDrowo linet6jbinthethingswhich Q fig f.e Schoolexercisebookof afive-year-otdboy work.Thiswasfollowedby a longperiodof 'silence',a fortnightor sowhennothingwas seenor heard,butwhen,weassume,theseriesoftransformativeactsofthechildcontinued 'internally','mentally'. Finallythe internalactivitybecamevisible,literally,throughthe child'sunpromptedproductionof the drawings,his unpromptedclassificatoryactivity (spatiallyshown)andhisspokencommentaryin responseto hisfather,squestion. 0f course,whileall thistook placethe child,as do all of us,wouldno doubthave experiencedconstantlyshiftingaffective,emotionalstates.Hemighthavebeenenthusedby thetaskintheclassandpraisedbytheteacherfor hissuccess;hemighthavehada difficult timewithhisfriendsintheplayground,or at home,andsoon,andall of thiswouldhave inffuencedhowhe'read'theactivityandhowit was'takenup,byhim.If weseeit likethis,
  • 53. Thesemiotic landscape it makesit impossibleto thinkof affectandcognitionasdistinct,asseparable.In other words,here- asalways- theaffectiveaspectsarealwaysonewith,andactcontinuouslyas a 'modality'on,cognitivesemioticprocesses. In part in responseto the representational,semioticand cognitiveresourcesmade availablebytheteacher,andherdemandsmadeintheclass,thoughafterwardsprompted byhisowninterests,thechilduseda seriesof differentrepresentationalmodes(including, of course,'internalrepresentations')ina constantlyproductivesequenceof semioticactiv- ities.Somehappenedwithinthesamemode(linkingtheimagesof theobjectbya line,for instance),sometookplacebya shiftacrossmodes(theshiftfromthespatiallyperformed classificationto thespol<encommentaryonit). Suchprocessesareconstantlytransforma- tive(thenameweusefor suchorocesseswithinonemode)andtransductive(ournamefor suchprocessesacrossmodes).All these,we assume/haveeffectson 'innerresources', whichconstantlyreshape(transform)thesubjectivityof thechild. Aswehaveindicated,thevisual,actionalandspatialmodes,ratherthanspeech/seemed to bethecentralrepresentationalandcognitiveresources.Speechwasusedfor communi- cationwithadults,asa meansfor translation,forcommenlandfor ratification.It maywell bethatthe complexitiesrealizedin the six imagesandtheirclassificationwereinitially beyondthechild'scapacityof spokenexpression,conceptionandformulation,butthatthe visualmodeofferedhimsemioticandcognitiveresourceswhichwerenotavailableto him intheverbalmode.Howeverlonceexpressedinthevisualmode,onceclassifiedthroughthe visual/spatialmode,the meaningswhichthe childhad producedbecameavailableas externalized,objectiveexpression;this in turn mayhavemadethemdifferentlyavailable for verbalexpression,for theverbalratificationof semiotic,affective/cognitiveprocesses that hadalreadytakenplace. This incessantprocessof 'translation',or 'transcoding'- transduction- betweena rangeof semioticmodesrepresents,wesuggest,a better,a moreadequateunderstandingof representationandcommunication.In theexamplewehavediscussedhere,languageisnot at thecentre.In manyareasof publiccommunicationthesameiseitherthecasealready, or rapidlycomingto bethecase.Andclearlyit matterswhichsemioticmodesof represen- tationandcommunicationaredominant,mostfrequent,mostvaluedinthepublicdomains inwhichweact. Figure1.10comesfromworkdonebytwo (13-year-old)studentsin Science,in the earlyyearsof secondaryschoolin England.Twoquestionscanbeasked.Thefirst is,What istheeffectofthemodeof representationontheepistemologyof science?,andthesecond: 'Do differentmodesof representationfacilitate,or ruleout,differentaccountsof natural phenomena?'Toanswerboth,we needto comparetwo differingmodalrepresentationsof 'thesame'issue. Thistimeourquestionisnot'Whatisthestatusofwrittenlanguageinthesetexts?'but rather'Whatistheeffectof thedifferentmodalrealizationsintermsof epistemolog,in termsof thestudents'perspectiveonknowledge?'If wecomparefigure1.10withjustone examplefromanotherexercisefor assessment- thetask'to writea storyof thejourneyof a redbloodcellaroundthebody'- wecanseetherudimentsof thatdifference.Hereisa briefextractfrom onesuch'story',inthiscasewritteninthegenreof'diary':
  • 54. Thesemiotic landscape y" I sr4oKrNG I o'oo'""" Y EilI I rroNoxrpE I '""un" Y i n [;;I ,, ,o,'"0 i o t [;;;] I HEARrI F;;;l - rhehead 1, I cELLs I pumpsrhe ,/ , ' bloodaround ,/ .. . . v. ooav ,,/ ne oo@ conrarns I ruruesI I l----.-.-- | eLooo |* |*",r. u.*o I serslxysenr,om/a-= -=> | cELis Ithe lunos / | t;;] + |-;I I maKe Y F'--I I "n"ro"f maoJin |;;;I + G;t ..I DroxrDE I w;; ' ' productof respiration n;ii,;:a:" // I ---a / + contains,/ {"on,u,n" I ,*" broodsetsl I t--_l ',1""Yrfl"l"*tto I lnNrraooresl intestines d @ fiq f.fO Conceptmapr'bloodcirculation' DearDiary,I havejustlefttheheart.I hadto comefromthetopoftherightchamber of theheart(Rightatrium)andsqueezemywaythroughto the Rightventriclewhere theheartbeatgotstronge4andI lefttheheart. Dear Diary,I am currentlyin the lungs,it is terriblycrampedin hereas the capillariesaretinyandtherearemillionsof us.Wehavejustdroppedoff oxygenand wepickedupsomecarbondioxide. DearDiary,wehaveenteredtheIiverwherewehada thoroughwash. DearDiary,wehavejust leftthe kidneywherewedroppedoff somewaterwhich will beturnedintourine. DearDiary,I havefinishedmy journeyaroundthe bodyby stoppingoff at the heart. In thediary,thefundamentalorganizationalprinciple,or logic,isthatof 'sequenceintime,, andthefundamentalrepresentationalprincipleisthatof actionor eventor,lessfrequently, statesof affairs.Objectsare relatedto otherobjectsby actionsrepresentedbyverbs('we haveenteredthe liver','wehavejust leftthe kidney').Theactionsandeventsthemselves arearrangedintemporalsequence/mirroringthatof theimaginedeventsastheyhappened in the world.In the conceptmap,on the otherhand,the fundamentalorganizational principleis that of a conceptualorder,realizedby the spatialarrangementsof the /Drooogoes /- the bloodrerums 1Yi{1'9T / to the hean th@sh the hean / rheverns Inaneites
  • 55. Thesemiotic landscape 4 t individualconcepts.Hereobjectsare related,not by actions,but by hierarchy,by signifi- cancederivingfromrelationof'priority'of variouskinds. All theseexamplesrevealwhathasin factalwaysbeenthecase:language,whetherin speechor writing,hasalwaysexistedasjust onemodein theensembleof modesinvolved in the productionof texts,spokenor written.A spokentext is neverjust verbal,but also visual,combiningwith modessuchasfacialexpression,gesture,postureandotherforms of self-presentation.A writtentext,similarly,involvesmorethan language:it is written on something,on somematerial(paper,wood,vellum,stone,metal,rock,etc.)and it is writtenwith something(gold,ink,(en)gravings,dotsof paint,etc.);with lettersformed as typesof font, influencedby aesthetic,psychologicai,pragmaticand otherconsider- ations;and with layoutimposedon the materialsubstance,whetheron the page,the computerscreenor a polishedbrassplaque.Yetthe multimodalityof writtentextshas, by andlarge,beenignored,whetherin educationalcontexts,in linguistictheorizingor in popularcommonsense.Today,in the ageof 'multimedia',it cansuddenlybe perceived again. We cansummarizethisdiscussionin theform a setof hypotheses:(a) humansocieties usea varietyof modesof representation;(b) eachmodehas,inherently,differentrepre- sentationalpotentials,differentpotentialsfor meaning-making;(c) each mode has specificsocialvaluationin particularsocialcontexts;(d) differentpotentialsfor meaning- makingmayimplydifferentpotentialsfor theformationof subjectivities;(e) individuals usea rangeof representationalmodes,andthereforehaveavailablea rangeof meansof meaning-making,eachaffectingtheformationof theirsubjectivity;(f) thedifferentmodes of representationare not helddiscretely,separately,as stronglyboundedautonomous domainsinthebrain,or asautonomouscommunicationalresourcesinculture,norarethey deployeddiscretely,eitherin representationor in communication;(g) affectiveaspectsof humanbeingsandpracticesare not discretefrom othercognitiveactivity,andtherefore neverseparateor absentfrom representationalandcommunicativebehaviour;(h) each modeof representationhasa continuouslyevolvinghistory,inwhichitssemanticreachcan contractor expandor moveintodifferentareasof socialuseas a resultof the usesto whichit isput. None of these hypotheseswould, we imagine,attract significantdisagreement, especiallywhenputsingly.Jointlytheyrepresenta challengeto theexistingcommonsense onthe relationsbetweenlanguageandthoughtandin mainstreamtheoriesandpractices in all areasof publiccommunication.This is a crucialfeatureof the new semiotic landscape. A NOTEONA SOCIALSEMIOTICTHEORYOFCOMMUNICATION in orderto functionasa full systemof communication,thevisual,likeall semioticmodes, hasto serveseveralrepresentationalandcommunicationalrequirements.Wehaveadopted the theoreticalnotionof 'metafunction'fromthe work of MichaelHallidayfor this
  • 56. 42 . Thesemiotic landscape purpose.Thethreemetafunctionswhichhepositsaretheideational,theinterpersonaland the textual.lntheform in whichwe glossthemheretheyapplyto all semioticmodes, andarenotspecificto speechor writing. The ideationalmetafunction Anysemioticmodehasto beableto representaspectsof theworldasit isexperiencedby humans.In otherwords,it hasto beableto representobjectsandtheirrelationsina world outsidethe representationalsystem.Thatworldmayof coursebe,andmostfrequentlyis, alreadysemiotically represented. In doingso,semioticmodesofferanarrayof choices,of differentwaysinwhichobjects, andtheirrelationsto otherobjectsandto processes,canberepresented.Twoobjectsmay berepresentedasinvolvedin a processof interactionwhichcouldbevisuallyrealizedby vectors: O Fiql.tl vector Butobjectscanalsorelatedin otherways,for instanceintermsof a classification.They wouldbeconnected,notbya vectorbut,for instance,bya'tree'structure: O rlg f.fZ Treestructure In chapters2 and3 wewill investigatepreciselywhichideationalchoicesareavailable forvisualsign-makinginthisway. The interpersonalmetafunction Any semioticmodehasto be ableto projectthe relationsbetweenthe producerof a (complex)sign,andthereceiver/reproducerof thatsign.Thatis,anymodehasto beableto representa particularsocialrelationbetweenthe producer,the viewerand the object represented. As in the caseof the ideationalmetafunction,modesoffer an arrayof choicesfor representingdifferent'interpersonal'relations,someofwhichwill befavouredinoneform of visualrepresentation(say,in the naturalisticimage),othersin another(say,in the
  • 57. Thesemiotic landscape . 43 diagram).A depictedpersonmaybeshownasaddressingviewersdirectly,bylookingatthe camera.Thisconveysa senseof interactionbetweenthe depictedpersonandthe viewer. Buta depictedpersonmayalsobeshownasturnedawayfromtheviewer,andthisconveys the absenceof a senseof interaction.It allowsthe viewerto scrutinizethe represented charactersasthoughtheywerespecimensina displaycase. In chapters4 and5 wewill discusstheseandotherinterpersonalchoices,bothinterms of thekindsof interactionsthatcanberepresented,andintermsof thevisualfeaturesthat realizetheseinteractions. Thetextualmetafunction Anysemioticmodehasto havethecapacityto formfexfs,complexesof signswhichcohere bothinternallywith eachotherandexternallywiththecontextin andfor whichtheywere produced.Here,too,visualgrammarmakesa rangeof resourcesavailable:differentcom- positionalarrangementsto allowthe realizationof differenttextualmeanings.In figure 1.1,for example,thetextis onthe leftandthepictureonthe right.Changingthe layout (figure1.13) wouldcompletelyalter the relationbetweenwrittentext and imageand themeaningof thewhole.Theimage,ratherthanthewrittentext,wouldnowserveaspoint of departure,as'anchor'forthe message.In chapter6 we will discusssuchleft-right relationshipsandothercompositionalresources. : e . , " , . - " .' " , i : " j t ' i . , ' Y .j h*{* r*. i ir*,r* I* **lj eli*hr lY:.y ** il: 1 qr: @ fig f.fl Alteredlayoutof tigurel.I (left-rightreversal)
  • 58. 44 . Thesemiotic landscape Ourfocusis onthedescriptionof theseideational,interpersonalandtextualresources as they are realizedin the visualmode.We recognizethat in doingthis work we are engagedin morethan'meredescription',andparticipateourselvesinthe reshapingof the semioticlandscape;andwerealizealsothatthisisa highlypoliticalenterprise.
  • 59. 2 N a r r a t i v er e p r e s en t a t i o n s : d e s i g n i n gs o c i a la c t i o n INTRODUCTION Thepicturesshownin figure2.1 are takenfrom an Australianprimary-schoolsocial studiestextbool<(Qakleyet a1.,1985).0nerepresentsthetraditionaltechnologyof the AustralianAborigines,theotherthesuperiortechnologyof thosewhoinvadedtheirterri- tory ('TheBritishhada technologythat wascapableof changingthe faceof the earth. Theirtoolswereableto workfasterthanthoseof theAboriginesandtheirweaponswere muchmorepowerful').Theformerhasthreemainelements(anaxe,a basketanda wooden sword),the latterfour (the'British',as theyare calledin the caption,their guns,the Aboriginesandthelandscape).Butthetwo picturesdiffernotonlyinwhateachincludes andexcludes(theleft picture,for instance,excludestheusersof thetechnology,the right pictureincludesthem),theydifferalso in structure:theyrelatetheir elementsto each otherdifferently.Theelementsof the left pictureare arrangedsymmetrically,againsta neutralbackground:axe,basketandwoodenswordarerepresentedasequalinsize,placed at equaldistancefrom eachotherandorientedin the samewaytowardsthe horizontal andtheverticalaxes,sothatthepictureasa wholecreatesa relationof similaritybetween thethreeelements.Thepicturesays,asit were,thatthisaxe,thisbasketandthiswooden O Fig2.1TheBritishusedguns(0akleyeta/.,1985)
  • 60. Narrative representations swordall belongto thesameoverarchingcategory(a category,incidentally,whichis only implied,andwhichconflatesthenotionof 'tools,andthenotionof 'weaoons,). Therightpicturerepresentstechnologyin action.Wherethe leftpictureis impersonal, thispictureis personal.Wherethe left pictureis static,thispictureis dynamic.Where the left pictureis dry and conceptual,this pictureis dramatic.It relatesthe British andtheAboriginesthrougha transactionalschemain whichthe Briiishplaythe roleof 'Actor',theoneswhodothedeed,andtheAboriginestheroleof 'Goal,,theonesto whom thedeedis done-the BritishstalktheAborigines,onecouldsay.It alsorelatesthe land- scapeto the BritishandtheAboriginesin a'locative'way(theBritishandtheAborigines are in lhe landscape),andthe gunto the Britishin an 'instrumental'wav (the British stalkthe Aborigineswiththeir gun). Theserelationscanbetransformedinto linguisticform,aswe havejust done,butthe pointisthatheretheyarerealizedbyvisualmeans.Thetransactionalrelationbetweenthe BritishandtheAboriginesisrealizedbythevectorthatlinksthem,namelytheobliquelines formedbytheglancesandoutstretchedarmsof the Britishandbytheirguns.Thelocative relationis realizedbyoverlapping,bythegradientsof focus,the degreesof coloursatur- ationandsoon,whichcreatethe contrastbetweenforegroundandbackground.Andthe instrumentalrelationisrealizedbythegestureof holding,wheretheobjectheldisa tool. Theimportantpointat thisstageis notthe detailof the analysis,butthe observation that the semioticmodesof writingandvisualcommunicationeachhavetheir ownquite particularmeansof realizingwhat may be quitesimilarsemanticrelations.What in Ianguageisrealizedbywordsofthecategory'actionverbs'isvisuallyrealizedbyelements that canbeformallydefinedas vectors.What in languageis realizedby locativepreposi- tionsis visuallyrealizedby the formalcharacteristicsthat createthe contrastbetween foregroundandbackground.Thisis notto saythat all the relationsthat canbe realized linguisticallycanalsoberealizedvisually- or viceversa,thatalltherelationsthatcanbe realizedvisuallycanalsoberealizedlinguistically.Rather,a givenculturehasa rangeof general,possiblerelationswhichisnottiedto expressioninanyparticularsemioticmode, althoughsomerelationscanonlyberealizedvisuallyandothersonlylinguistically,or some moreeasilyvisuallyandothersmoreeasilylinguistically.Thisdistributionof realization possibilitiesacrossthesemioticmodesisitselfdeterminedhistoricallyandsociallyaswell asbythe inherentpotentialitiesandlimitationsof a semioticmode. Toreturnto thetwo picturesinfigure2.I,theycanbesaidto representanaspectof the experientialworld,technology.Butthroughthe differentdesignpatternsselectedin each, throughthemannerin whicheachbringsitsindividualelementstogetherintoa coherent andmeaningfulwhole,theyrepresentthe technologyof Aboriginesverydifferentlyfrom the technologyof the British.NothingaboutAboriginaltechnologynecessitatesit to be representedasa static,conceptualtaxonomy,nor isthereanythingintrinsicaboutBritish technologythat requiresit to berepresentedina personalizedanddramatizedway.British technologyis just ascapableof beingrepresentedby a classificatoryschemeas is Abo- riginaltechnology,andit isjustaspossibleto telldramaticstoriesof Aboriginesstalking their invaderswith spearsor woodenswordsasit isto tell suchstoriesaboutthe British. evenif the formerkind of story doesnot usuallyform part of mainstreamAustralian
  • 61. Narrativerepresentations' 47 history.In otherwords,the representationis mediated,visually,throughtwo distinctdis- courses:thatof anthropologyfor Aboriginalpeople- who'knowno history';andthatof historyfor thewhites- whoarenotsubjectsof anthropology. imaginea reversalof thoserelations.Imagineonthe left a catalogueof Britishtools andweapons,andontherighta pictureinwhichAboriginespointtheirwoodenswordsat a smallgroupof Britishinthebackground.Suddenlya representationof colonizationasthe transitionfrom a fixed,stable('primitive')orderof thingsto the dynamicunfoldingof historyischangedintosomethingliketherevengeof the'primitive'ontheWest'stechno- logicalorder.Thismaybesuitablefor a fictionfilm,perhaps,setsafelyin an apocalyptic future,but notfor a primary-schooltextbookin contemporaryAustralia. Thetwo designpatternsin figure2.I,the classificatorypatternandthetransactional pattern,are onlytwo of severalpossiblepatterns.In the courseof this chapterwe will introduceothers,andtry to giveanoverviewof thevisualstructuresthatcanrealizeways of representingtheworld.0uremphasisisnotondepiction,noronthequestionof recogni- tion,on howwe cometo seeconfigurationsof pencilmarksor brushstrol<esor pixelsas picturesof trees,or on howpicturesof treesmayconnoteor symbolizemeaningsand valuesoverandabovewhattheyliterallyrepresent.Thisaspectof thepictorialhasalready receiveda gooddealof attentionin the writingsof philosophers(e.g'Goodman,1969; Hermeren,1969), semioticians(e.g. Eco, I976a; Barthes,I977), mediaanalysts (Williamson,1978)andart historians(e.g.Panofsky,I97O).At thisstageof ourwork onvisualcommunicationwehavelittleto addto whathasbeensaidintheseareas. Thequestionofvisualstructuring,ontheotherhand,has,inouropinion,beendealtwith lesssatisfactorily.Visualstructuringhaseitherbeentreatedas simplyreproducingthe structuresof reality(e.g.Metz,I974a,7974b),ratherthanascreatingmeaningfulpro- positionsby meansof visualsyntax,or it hasbeendiscussedin formaltermsonly(e'9. Arnheim,1974t1982,who in hisactualanalysesoffersmanyinsightson the semantic dimensionof visualstructuring).0urexampleof the representationof Aboriginaland Britishtechnologyhas,we hope,madeclearwhyneitherof theseapproachessatisfiesus. Visualstructuresdo notsimplyreproducethestructuresof'reality'.0n thecontrary,they produceimagesof realitywhichareboundupwith the interestsof thesocialinstitutions withinwhichthe imagesare produced,circulatedandread.Theyare ideological.Visual structuresarenevermerelyformal:theyhavea deeplyimportantsemanticdimension. PARTICIPANTS In chaptert we definedthe ideationalmetafunctionasthe abilityof semioticsystemsto representobjectsandtheir relationsin a worldoutsidethe representationalsystemor in the semioticsystemsof a culture.In the introductionto the presentchapterwetriedto showhowtwo designpatternscanproducetwo differentrepresentationsof broadlythe sameaspectof theworld.Insteadof 'objects'or'elements'wewill,fromnowon,usethe term'participanfs'er,moreprecisely,'representedparticipants'.Thishastwo advantages: it pointsto the relationalcharacteristicof 'participantrn something';and it draws
  • 62. 48 . Narrative representations attentionto thefactthattherearetwo typesof participantinvolvedin everysemioticact, interactiveparticipantsandrepresentedparticipants.Theformerare the participantsin theact of communication- theparticipantswhospeakandlistenor writeandread,make imagesor viewthem,whereasthe latterarethe participantswho constitutethe subject matterof the communication;that is,the people,placesandthings(includingabstract 'things')representedin and by the speechor writing or image,the participantsabout whomor whichwearespeakingor writingor producingimages. Thesituationisof coursemorecomplexthanthis,for the realinteractiveparticipants, the reallmage-producersand-viewers,cannotbetakento beidenticalwith the'implied, producerwho'silentlyinstructsus,throughthedesignofthewhole'(chatman,r97B:r4B) andthe'implied'viewer.It mayalsobethatthe producersand/orviewersarethemselves explicitlyrepresentedin the image,causingthe two categoriesto shadeintoeachother, complexitieswhichhavebeenstudiedextensivelyin the fieldof literarynarratology(e.g. Iser,l97B; Bal,1985;Rimmon-l(enan,l-9B3).we will returnto theseinthenextchapter; forthepurposesof thischapterthebasicdistinctionwillsuffice. In the caseof abstractvisualssuchas diagrams,it doesnot seemtoo difficultto determinewho or whatthe representedparticipantsare.ShannonandWeaver'sfamous 'communicationmodel'(figure2.2),for instance,is madeupof boxesandarrows(Shan- nonandWeaver,1949).Theboxesrepresentparticipants(peopleand/orthings,the dis- tinctionbeingblurredby objectifyinglabelslike'informationsource'and'destination'); the arrowsrepresentthe processesthat relatethem.If we wantedto translatethis into language,wecouldsaythattheboxesarelikenouns,thearrowslikeverbs(e.g.'send,or 'transmit'),andthat,together,theyformclauses(e.g.'aninformationsourcesendsIinfor- mationlto transmitter'). In thecaseof moredetailednaturalisticimages,however,it maybedifficult,evenfutile, to try andidentifythe representedparticipants.Take'TheBritishusedguns',for instance (figure2.1).Doweincludethehatsandkerchiefswornbythetwomen?Everysingletree, everysingleoneof the rocksstrewnaboutin theforeground?Theanalogywith language losesits relevancehere.In language,wordssuchasman/gun,tree,rockygroundabstract awayfromdetailsof thiskind.In naturalisticimagesthisdoesnothappen.Theyare'worth a thousandwords'. O Fig2.2 ShannonandWeayer'scornmunicationmodel
  • 63. Narrativerepresentations 49 Yetwethinkthat naturalisticimagescanbeanalysedintoparticipantsandprocesses muchinthesamewayasdiagrams.Therearetwodifferent,butintheendcompatible,ways of arguingthis.Thefirstisthewayof formalart theory(e.9.Arnheim,1974,1982).fhe languageof this kindof theoryis,for the mostpart,formalistic,andgroundedin the psychologyof perception.Participantsarecalled'volumes'or'masses',eachwitha dis- t i n c t ' w e i g h t ' o r ' g r a v i t a t i o n a lp u l l ' .P r o c e s s e sa r e c a l l e d ' v e c t o r s ' o r ' t e n s i o n s ' o r 'dynamicforces'.But,andthisiswhatmattersfor thepurposeof identifyingparticipants, these'volumes'areperceivedasdistinctentitieswhicharesalient('heavy')to different degreesbecauseof their differentsizes,shapes,colour,andso on.Thusthe two menin figure2.1 standout as a distinctentity becauseof the tonal contrastbetweentheir silhouettesandthe lightof thefire.Andwhatis more,we recognizetheirshapesonthe basisof visualschemasnotunlikethosethatarerealizedin diagrams.Artistshavelong learnedtheircraftbyreducingthevisibleworldto simplegeometricforms(seeGombrich, 1960).Accordingto Arnheim(1974:ch.4), childrenlearnto draw in the sameway, buildingup a repertoireof basicforms,andthen,gradually,'fusingthe parts'.If the perceptionof picturesdoesindeedoperateon the basisof the sameprinciplesas the productionof pictures,we mightgrasptheoverallstructureof a picturelil<e'TheBritish usedguns'accordingto a schemathat is not so differentfrom that of Shannonand Weaver'scommunicationmodel,asshowninfigure2.3. Thesecondwayof identifyingparticipantsis that of functionalsemiotictheory(see Halliday,1978,1985).Theconceptualapparatusof thiskindof theoryhas,sofar,been 'd O Fig2.3 Schematicreductionof tigure2.1,showingvector
  • 64. 50 . Narrative reoresentations appliedonlyto language,the mostfrequentlyandmethodicallyanalysedsemioticsystem. It isorientedtowardsthesemanticfunctionsratherthantowardstheformsof thepartici- pants.It usestermslike'Actor','Goal'and'Recipient'ratherthantermslike'volume'and 'mass'.Yetthetwoapproachesarecompatible.Themostsalient'volumes'in'TheBritish usedguns'arenotonlyperceptuallymostconspicuous,theyalsoplaythemostcruciaroles inthegrammaticalstructurethatconstitutesthemeaningof thepicture:thetwo men(the participantfrom whichthe vectoremanates)havethe roleof Actor,andthe Aborigines (the participantat whichthe vectorpoints)havethe role of Goalin a structurethat representstheir relationasa Transactio4assomethingdonebyan Actor/o a Goal.The sameterms('Actor','Goal','Transaction')areusedin functionallinguistics.Thisis pos- siblebecausetheyaresemantic-functional,ratherthanformal,terms.0uruseof these termsdoesnot implythat imagesanddiagramswork in the samewayas language;only that theycan'say' (someof) thesamethingsaslanguage- in verydifferentways:whatin languageis realizedby meansof syntacticconfigurationsof certainclassesof nounsand certainclassesof verbsis visuallyrealized,madeperceivableandcommunicable,by the vectorialrelationsbetweenvolumes.In Arnheim'swords,'Weshalldistinguishbetween volumesandvectors,betweenbeingandacting'(1982:1,54). Thetransactionalstructureis notthe onlykindof structurethat canberealizedvisu- ally.We havealreadydiscussedan exampleof a classificatorystructure(a subjectwhich wewill takeup in moredetailin a latersection;seepp.79-87).lnthe picturein figure 2.4,takenfrom the samesocialstudiestextbookasfigure2.1 (Oakleyet al.,1985),the structureis'analytical'.Heretheparticipantshavetherolesnotof'Actor'and'Goal'but of 'Carrier'and'Attribute'.Thispictureis not aboutsomethingwhichparticipantsare doingto otherparticipants,butaboutthewayparticipantsfit togethertomakeupa larger whole.It hasthe structureof a map.Justas in mapsa largerparticipant,the 'Carrier', representsthe'whole'(say,Australia),anda numberof otherparticipants,the'Possessive Attributes',representthe 'parts' (say,the statesof Australia),sothe Antarcticexplorer functionsas'Carrier',andthebalaclava,thewindprooftop,thefur mittens,etc.functionas 'PossessiveAttributes',asthepartsthatmakeupthewhole.Theclosestlinguistictransla- tionhere- wereweto attemptone- wouldnotbeanactionclauselike'TheBritishpoint their gunsat the Aborigines',but a 'possessiveattributive'clauseliketTheoutfit of the Antarcticexplorerconsistsof a balaclava,a windprooftop,fur mittens. . . [etc.]' We cannowlookat the abundanceof detailin naturalisticimagesin a newway.The naturalisticimage,whateverelseit maybeabout,isalwaysalsoaboutdetail.It Containsa multitudeof embedded'analytical'processes.It may,at the mostsalientlevel,say,'The Britishpointtheirgunsat iheAborigines',butit willalso,at lessimmediatelyconspicuous levels,saythingslike'Themen'soutfitsconsistof hats,kerchiefs. . . [etc.]'and'Thetrees haveclumpsof leaves.'Inlanguage,prepositionalphrases(twomenwith hatsandker- chiefs)and subordinateclauses(two men,wearinghatsand kerchiefs)fulfil the same functionof addingdetailat a'secondary'or evenmoredeeplyembeddedlevel. Embeddingcanalsooccurindiagrams.Takethe'communicationmodel'infigure2.5,a modeldrawnup,not bytwo telecommunicationengineers,as in the caseof Shannonand Weaver(figure2.2),but by two sociologists,Rileyand Riley(1959).As a whole,the
  • 65. Narrativerepresentations 5 l llr-t "r. r11' rdqqv *i3-: i::t !rf1*{},:x :iq l';!* *a $$:!jh* t1]!i1*:a1 O Fig2.4 Antarcticexplorer(oakleyet at,,LgPjs) O Fig2.5 C0mmunicationmodel(fromWatsonandHill,1980:143) diagramis'analytical';it is a kindof abstractmap.It showsthat the'over-allsocial system'consistsof 'largersocialstructures'whichin turnconsistof 'primarygroups'.It alsofeaturestwo individuals,'C'('Communicator')and'R' ('Recipient').Theseare depictedas half in,half out of the 'largersocialstructures',andtheyare connectedto, thoughnot partof,the'primarygroups'.Embeddedwithinthisanalyticalstructureis a ; i n :l;i i1 .,*.J ;'.'.:. .i:";: 'r . trjs':h; F"!,..,,;i . 1 : ; ; ' r . : : - ; - . , i . !
  • 66. Narrativerepresentations O Fig2.5 Twocommunicationmodels(tromWatsonandHill,1980:147) transactionalstructure:the'largersocialstructures/andthe individuals'C' and'R' are representedasinvolvedin anactiveprocessof communication,realizedbyvectors. Whenwelookat'TheBritishusedguns/(infigure2.1)asa transactionalstructure,the two menformoneparticipant:togethertheyhavetheroleof 'Actor'.Whenwe lookat the two menasan'analytical'structure,theyformtwo distinctparticipants,linkedbythelines formedbythehandof the manonthe rightandthe gunof the manonthe left.Diagrams allowfurtherpossibilities,ascanbeseeninthetwocommunicationmodelsinfigure2.6, bothdrawnbySchramm(1954),a socialpsychologistwritingaboutmasscommunication. In thefirst model,'source'and'encoder'areseparateentities,conjoinedbya line,justas arethetwo menin'The Britishusedguns'.Wewill arguelaterthat lineswithoutarrow headsrealizea particularkindof 'analvtical'structure: O Fig2.7 Conjoiningparticipants In thesecondmodel,'source'and'encoder'are distinctcomponentsof thewhole: compounded,weldedtogetheryetstill O Fig2.8 Compoundingof participants
  • 67. Narrativerepresentations 53 Thethird possibilitywouldbea completefusionbetweensoLrrce'and'encoder'.The shapeof RileyandRiley's'overallsocialsystem/(figure2.5)canbeinterpretedassucha fusion- a fusionof two circlesanda box.Apparentlyparticipantscanlosetheirseparate identityto differentdegrees.Whentheyareconjoined,the process,the act of connecting them,isstillexplicit,realizedbya line.Whentheyarecompounded,theiridentitiesremain distinct,butthereis no longeranexplicitlyexpressedprocessto connectthem.Whenthey are fused,eventheir separateidentitieshavedisappeared.In speechand writing,with somewhatdifferentmeans- for instance,stressandintonation- we canmovefrom,say, Thebirdisblack,whichhastwo distinctparticipantsaswellasa connectingprocess('is'),' to theblackbird,whichhasblackandbird stillasdifferentwords,but removestheprocess,' to theblackbird,in whichtwowordshavebeenfusedto becomeonesemanticentity/noun. Eachsuccessivestepfurtherobscurestheactof predication,theexplicitactof bringingthe twoparticipantstogether,untilthestructureisnolonger'analytical',nolongeranalysedor analysable.Wemakethepointat somelengthbecauseof the(ideological)significanceof thissemioticresourceinconfiguringtherepresentedworld. O Fig2.9 Fusionof participants As with manyotherkindsof diagram,the communicationmodelswe haveusedto illustratethis sectionare explainedor paraphrasedin the writtentextsthat accompany them.Butbynomeanseverythingthat isexpressedinthediagramsisalsoexpressedinthe writtentexts.Not all of the meaningsconveyedvisuallyare alsoconveyedverbally.The meaningsofthevisualshapes,theboxesandcirclesandtrianglesthatgivetheparticipants theirvolume,for example,are almostalwaysleft unexplained.0lder,commonsenseor theoreticalnotions,suchas'illustration'(images'illustrating'verbaltexts)or'explan- ation'(words'explaining'diagrams)are no longeran adequateaccountof the relations betweenwordsandpictures,hereasin otherinstances.Why,infigure2.6,isthe'signal'a circle,the'source'a rectangle,the'encoder'a triangle?Why,in figure2.5,isthe'primary group'arectangle,whilethe'largersocialstructure'andtheindividuals'C'and'R'are circles?WhydoShannonandWeaver(figure2.2)preferangularity,whileRileyandRiley prefercurvature (figure2.5)? Therecanbelittledoubtthatsuchchoicesarechargedwithmeaning.Basicgeometrical shapeshavealwaysbeena sourceof fascination,evenof religiousawe.0ur scientificageis no exception.Circles,squaresandtriangleshavebeenregardedas pure/quasi-scientific 'atoms'ofthevisibleworld,a'puremanifestationof theelements',the'universal-as-the- mathematical',as Mondriansaid (quotedin Jaff6,1967: 54-5). And they havebeen thoughtto havethe powerto directlyaffect our nervoussystem,for instanceby the constructivistartistGabo:'Theemotionalforceof an absoluteshapeis unioueandnot
  • 68. Narrativerepresentations replaceablebyanyothermeans.. . . Shapesexultandshapesdepress,theyelateandmake desperate'(quotedin Nash,I974i 54). As we are hereprimarilyconcernedwith the relationsbetweenparticipants,this subjectfalls somewhatoutsideour mainconcern;it deservesa separatestudy.Butgiventhesemioticandideological(mythical)significanceof theseaspects,we will at leastindicatethe issueswith which sucha studymight be concerned. In contemporaryWesternsociety,squaresandrectanglesaretheelementsof themech- anical,technologicalorder,of theworldof humanconstruction.Theydominatetheshapeof ourcities,ourbuildings,ourroads.Theydominatetheshapeof manyof theobjectsweuse in dailylife,includingour pictures,whichnowadaysrarelyhavea roundor ovalframe, thoughotherperiodswerehappyto usetheseto framemoreintimateportraitsin particu- lar.Unlikecircles,whichare self-contained,completein themselves,rectangularshapes canbestacked,alignedwitheachotheringeometricalpatterns:theyformthemodules,the buildingblockswith whichwe constructour world,andtheyarethereforethe dominant choiceof buildersandengineers,andofthosewhothinklikebuildersandengineers.Inart, theyarethechoiceof geometricalabstractionists,artistsfor whomart hasto be,aboveall, rational.As Mondrianwroteinthe1920s, In allfieldslifegrowsincreasinglyabstractwhileit remainsreal.Moreandmorethe machinedisplacesnaturalpower.In fashionweseea characteristictensingof form andintensificationof colour,signifyingthedeparturefromthenatural. In moderndancesteps(boston,tango,etc.)thesametensingisseen:thecurved lineof theolddance(waltzetc.)hasyieldedto thestraightline,andeachmovement is immediatelyneutralizedbya counter-movement- signifyingthe searchfor equi- librium.)ur social/ifeshowsthistoo:autocracy,imperialismwith its (natural)rule of power,isaboutto fall- if it hasnotfallenalready- andyieldsto the(spiritual) oowerof law. Likewisethenewspiritcomesstronglyforwardin logic,scienceandreligion.The impartingof veiledwisdomyieldsto the wisdomof purereason;andknowledge showsincreasingexactness.The old religion,with its mysteriesand dogmas,is increasinglythrustasidebya clearrelationshipto theuniversal. (quotedinJaff6,1967:64) Glossesin'dictionariesof visualsymbols'andsimilarpublicationstendto expressthe meaningof geometricalshapesintermsof intrinsic,abstractqualities,buttheypointinthe samedirection.Accordingto Dondis(1973:,44),thesquarerepresents'honesty,straight- nessandworkmanlikemeaning';accordingto ThompsonandDavenport(1982:110),it 'representstheworldanddenotesorder'. Circlesare glossedvery differentlyin suchdictionaries,as denoting'endlessness, warmth,protection'(Dondis,I973:44), or as'thetraditionalsymbolof eternityandthe heavens'(ThompsonandDavenport,I9S2:110).Butsuchdescriptionscanbemultiplied endlessly:themoreabstractthesign,thegreateritssemanticextension;or,to put it in our terms,thegreateritspotentialrangeof usesasa signifierinsigns.Weneedto lookfor the
  • 69. Narrativerepresentations. 55 principlesthat unitethesemeanings,andfor thefundamentaloppositionsbetweensquare and circle,betweenthe angularand the curved.In nature,squarenessdoesnot exist. Mondrianadmittedthathismethodof 'abstractingthecurve'madeit difficultto represent nature:'1npaintinga tree,I progressivelyabstractedthe curve;youcanunderstandthat verylittle"tree" remained'(quotedinJaff6,1967:120).Circlesandcurvedformsgener- ally arethe elementswe associatewith an organicand naturalorder,with the worldof organicnature- andsuchmysticalmeaningsasmaybeassociatedwiththemderivefrom this.Angularitywe associatewiththe inorganic,crystallineworld,or with theworldof technology,whichisa worldwehavemadeourselves,andthereforea worldwecan/at least in principle,understandfully and rationally.Theworld of organicnaturels not of our making,andwill alwaysretainan elementof mystery.Curvedformsare thereforethe dominantchoiceof peoplewhothinkin termsof organicgrowthratherthanmechanical construction,intermsof whatisnaturalratherthanintermsof whatisartificial.In art, it isthechoiceof whatissometimescalled'biomorphicabstractionalism'-thecurves,blobs andbulgesinthepaintingsof HansArpor thesculpturesof HenryMoore. Thevaluesattachedto thesepolesof meaning- that is,the actualsignsproduced with the signifiersof the'technological'andthe'natural'- do,of course,differ.The squarecanconnotethe 'technological'positively,as a sourceof powerandprogress/or negatively,as a sourceof oppressionwhich,literallyandfiguratively,'boxesus in'. In Rileyand Riley'scommunicationmodel(figure2.5) societyis representedasa natural order,organicallyevolvedratherthanhumanlyconstructed.Butthe'primarygroups'are depictedas rectangles.Perhapsthis betraysan unconsciousbias in favourof modern urbansociety,a viewin whiChthesmall,close-knitcommunityin whicheveryoneknows everythingabout everyoneis seenas oppressive,and the 'larger socialstructure'as liberating,providingthe individualwithanonymity,andtherebywith autonomyandself- containment,choiceand freedom.For Rileyand Riley,individualshail from'primary groups'(andare still connectedto them,albeittenuously),but thengo theirownway, leavingthe'primarygroups'behind,andmovingfreelyin andout of the'largersocial structures',in a sociallymobileworld.Thisexampleshowsthat diagrams,rationaland scientificasthey mayseem/canconveymeaningsvisuallythat are not necessarilyalso conveyedverbally. Thetriangleis angular,likethesquare- an elementof the mechanical,technological order.But,unlikethesquare/thetriangle,especiallywhentilted,isa (fusedstructureof a) participantanda vector,becauseit canconveydirectionality,pointat things.Themeanings it attractsare thereforelesslil<e'qualitiesof being'thanlil<eprocesses/as in the well-knownrevolutionaryposterby El Lissitzky(figure2.10),in whichthe revolution, representedby a redtriangle,is an active,dynamicforce,wedgingitselfintothe inert, self-contained,'organic'societyof WhiteRussia. In diagrams,trianglescansimilarlyintroducea senseof process.Thetrianglesinfigure 2.6,forinstance,couldbeseenas'focusing'or'aiming'the'message'(wehavealready indicatedwhyit issodifficultto giveverbaltranscodingsof visualprocessesandwewill discussthismorefullylater).Notsurprisingly,glossesofthemeaningsoftrianglesinvisual dictionariesreflectthis dynamicquality.Trianglesare 'a symbolof generativepower'
  • 70. Narrativerepresentations O Fig2.LOBeattheWhiteswiththeRedryedge(ElLissitzky,lglg-20)(fromilash,1974) (ThompsonandDavenport,1982:110),andrepresent'action,conffict,tension'(Dondis, I 9 7 3 : 4 4 ) . Themeaningsof the basicgeometricalshapes,then,are motivatedin two ways.First, theyderivefrom the propertiesof the shapesort rather,from the valuesgivento these propertiesin specificsocialandculturalcontexts.Thestraightline,for instance,means whatit literallyis:'straight'.This'straightness'maythenbeusedto carryanyoneof a vast rangeof meaningscompatiblewiththat.It maybepositivelyvaluedinonecontext(e.g.the 'straightandnarrowpath',or Mondrian'sassociationof straightnesswith the'spiritual powerof law'),lesspositivelyin another(e.9.'thestraightman'asopposedto the'funny man',or 'straight'asopposedto 'gay').Theproducersof an imagehavetheir interestsin makingthe visualsign,andthis makesthe meaningof the imagequitespecificfor the producer;it coloursin and makesspecificthe abstractmeaningsthat derivefrom the inherentpropertiesof theshapesandfromthe historiesof theirculturaluses.Rectangles can be stacked- and,again,this maybe positivelyvaluedin onecontext,say in urban planning,or in geometricabstractionism;and lesspositivelyin another,say in counter- culturesthatdreamof livingin geodesicdomes,or in biomorphicabstractionism. Second,thesemeaningsderivefrom the commonqualitieswe may detectin such objectsin ourenvironmentaswouldbecircularor rectangularwhenabstractedto their underlyingbasicshape,andfrom thevaluesattachedto thesequalitiesin differentsocial &i,'..i.,
  • 71. Narrativerepresentations. 57 contexts.Thesun,the moon,the bellyof the pregnantwoman,arecurved.Theskyscraper, theexecutivedesk,theexpensivebriefcase,arerectangular.Suchcommonqualitiesaswe mayseeinthesegroupsof objects(say,'nature'scycles'and'malepower')willevidentlybe readandvalueddifferentlyin differentsocialcontexts- andthegroupingsof objectsfrom whichwe derivetheserneaningsare likelyto be madeselectively,so as to obtainthe commonqualitiessought. Finally,our argumentsuggeststhatthe semioticaenesisof diagramslies,notjust in technicaldrawing,but alsoin art, andspecificallyin the abstractart movementsof our century,whichnolongerfill outandcorporealizetheschematicreductionswhichnatural- isticartistshaveusedfor centuries,andwhich,if the Gestaltistsare right,underlieall visualreoresentation. Fromthe basicshapesothergeometricalshapescan be derived:square/circleand trianglecanbehorizontallyor verticallyelongatedto differentdegrees;andthesquare,the triangleandall elongatedshapescanbetilted,eithertowardstherightor towardsthe left. Verticalelongationcreatesa morepronounceddistinctionbetweentop andbottom,and hencea biastowardshierarchy,andtowards'opposition'generally(whatis mostimport- ant or otherwisedominantgoesontop,what is lessimportantor dominantis relegatedto thebottom).Horizontalelongationcausesa shapeto leantowardsthekindof structurein whichwhatispositionedontheleftispresentedas'Given',asinformationthatisalready familiarto the readerandservesas a'departurepoint'forthe message/whilewhat is positionedonthe rightis presentedas'New',asinformationnotyetknownto the reader, andhencedeservinghisor herspecialattention.Theshapeof Schramm's'fieldof experi- ence'and'signal'in the seconddiagramof figure2.6,for instance,suggeststhat these participantsare,at leastpotentially,endowedwithsuchan informationstructure.Tilting, finally,createsobliquelinesandhencea senseof vectoriality.In Malevich'sSupremacist Composition:RedSquareandBlackSquare(fi9ure2.11)theparticipantsarerepresented assquares.Butbecausetheredsquareistilted,thepaintingisstructurallymoresimilarto El Lissitzky'sBeatthe Whiteswith the Red Wedge(figure2.10) than,for instance,to Mondrian'scompositionsof red,yellowand bluesquares:it is aboutdynamic'action, whereasMondrian'scompositionsareabouta stableorder,a'searchforequilibrium'-the redsquareseemsto moveawayfromtheoppressivelylargeblacksquare.t Finally,it is importantto stressthe essentialinterchangeabilityof visualandverbal participantsin diagrams,and,indeed,in manyothervisualgenres.Althoughtheprocesses andstructuresin diagramsare alwaysvisual,the participantswhichtheyrelateto each othermaybeof differentkinds:pictures,naturalisticor schematic;abstractshapes,with or withoutverballabels;words,eitherenclosedor notenclosedin boxesor othershapes; letters;andsoon.Thesamethingcanbeseenin pagelayout:theparticipantsarehetero- geneous.Theycanbeverbal(headlines,blocksof copy,etc.),butthesemioticmeanswhich bringthem togetherinto a coherentsemanticstructureare alwaysvisual.The keyto understandingsuchtextsthereforeliesaboveall inanunderstandingof thevisualsemiotic meanswhichareusedto weldtheseheterogeneouselementsintoa coherentwhole,intoa text.Visualstructuresrelatevisualelementsto eachother;thesevisualelements,however, maythemselvesbeheterogeneous- a wordasa visualelement,a blockof writtentextas
  • 72. Narrativerepresentations O Fig2.ll SuprematistComposition:Red Sqaareand BlackSquare(Kasimir Matevich,I9l4) (from Nash,1974)
  • 73. Narrative representations . 59 a visualelement,an imageas a visualelement,a numberor an equationas a visual element. NARRATIVEPROCESSES Whenparticipantsareconnectedbya vector,theyare representedasdoingsomethingto or for eachother.Fromhereonwewill callsuchvectorialpatternsnarrative-in l(ressand vanLeeuwen(1990)weusedtheterm'presentational'-andcontrastthemto conceptual patterns(seefi9ure2.12).Whereconceptualpatternsrepresentparticipantsin termsof theirclass,structureor meaning/in otherwords,intermsof theirgeneralizedandmoreor lessstableandtimelessessence/narrativepatternsserveto presentunfoldingactionsand events,processesof change,transitoryspatialarrangements. Thehallmarkof a narrativevisual'proposition'is the presenceof a vector:narrative structuresalwayshaveone,conceptualstructuresneverdo.In pictures,thesevectorsare formedbydepictedelementsthat forman obliqueline,oftena quitestrong,diagonalline, asin 'TheBritishusedguns'(infigure2.1),wherethegunsandtheoutstretchedarmsof the Britishformsucha line.Thevectorsmaybeformedby bodiesor limbsor tools'in action',buttherearemanyotherwaysto turn representedelementsintodiagonallinesof action.A roadrunningdiagonallyacrossthe picturespace,for instance,is alsoa vector, andthecardrivingon it an'Actor'intheprocessof'driving'.In abstractimagessuchas diagrams,narrativeprocessesare realizedby abstractgraphicelements- for instance, lineswith an explicitindicatorof directionality,usuallyan arrowhead.Suchfeaturesof directionalitymustalwaysbepresentif thestructureisto realizea narrativerepresenta- tion:connectinglineswithoutan indicatorof directionalityform a particularkindof analyticalstructure,andmeansomethinglike'isconnectedto','isconjoinedto','isrelated t0'. The'Actor'istheparticipantfromwhomor whichthevectordeparts,andwhichmay be fusedwith the vectorto differentdegrees.In Beat the Whiteswith the Red Wedge (figure2.10),for instance,the redtriangleis bothparticipantandvector,andrepresents boththe'wedge'andtheactof wedgingin (or'beating',asthetitlehasit).Infigure2.I3, takenfroma factualchildren'sbookaboutFrance(Bender,19BB),theActorsarerealized bythecolourofthearrows(the'GulfStream'vectorisred,the'Mistral'vectorblue),and bytheirvolume(theyare notjustthin lines),andthe narrativeprocessesare realizedby rI Reoresentational I structures I I f classificatorv L conceptuar-----*t ilfl;:f:: O Fig2.12 Maintypesof visualrepresentationalstructure
  • 74. 60 Narrativerepresentatrons EnglishChannel Biscay ffi Mediterranean ,'-.,',]::' i:,,i",:,:i,rll!'t'Gulf Stream LEso+,f*,ftrtistrat { il O Fig2.13 culf StreamandMistrat(Bender,t988) theirvectoriality.In Shannonandweaver'si949) communicationmodel(figure2.D, on theotherhand,Actorandprocessarerealizedbyseparatevisualelements,theActorbya box('informationsource'),theprocessbyan arrow.Pictureslike'TheBritishusedquns' (infigure2.1)occupy,perhaps,anintermediateposition. In thecaseof 'realist'images,thecontextusuallymakesclearwhat kindof actionthe vectorsrepresent.'TheBritishusedguns/(in figure2.1) canbetranslated- shouldwe wishor needto doso- notsomuchwith'used',asthecaptionhasit,aswithsomethinglike 'TheBritishstalktheAborigineswiththeirguns'.Otherpossibilitiesexist,butihefieldis limited.Thevectorsin abstractpicturesaremoredifficultto transcode.Onefirst needsto formulatewhatthey,literally,formally,do.Thiscanthenhelpto circumscribethefieldof possiblereadings.Thetrianglein El Lissitzky'sBeat the whites with the Redwedge (figure2.10)literallywedgesitselfintothewhitecircle.Thisopensupa perhapslarge,but by no meansinfinite,rangeof possiblereadings:thetrianglecanbesaidto,pierce,or 'infiltrate'or'destabilize'the circle.In factit doesallthesethings-the processrepresents a fieldof possiblemeanings.In the caseof Malevich,sSuprematistComposition:Red Squareand BlackSquare(frgure2.7I), the redsquareliterallytilts awayfrom the black squareaboveit. Howcanwetranscodethis?Doestheredsquare'ffeefrom,or 'pointaway
  • 75. Narrativerepresentations. 6I from'the blacksquare?Is it'ejectedfrom' or'discardedby' it? All thesereadingsare legitimate.Thepointis that eachreadingwill havethe smallredsquareasthe mobile participant,howevermuchit is dominatedby the big,heavyblacksquareiandthe black squareasstatic,motionlessandmonolithic.Thepicturetellsthestoryof an'underling' escapingfrom,or beingexpelledfrom,themonolithicpowerof theblacksquare.Whoor whatthisunderlingor theseforcesarethereaderswill produceintheprocessof reading, althoughthe shapesand colourswill pointthemin a certaindirection,mal<ecertain readingsmoreplausible,morelikethoseof theproducer'sthanothers. In thecaseof diagramsit isalsodifficultto sayinwordsjustwhatkindof actionthe vectorsrepresent.Thecommonsenseviewof thefunctionof imagesas'illustration'and 'explanation'wouldbethattheaccompanyingverbaltextexplainswhatisnotmadeclear visually.Butusuallythisisnotso.Usuallytheprocessisrepresentedonlyvisually,andthe writtentexteitherdoesnotparaphraseit at all,or providescontradictoryor evenmislead- ingglosses.TheShannonandWeavercommunicationmodelhasbeen'quoted'andver- ballyexplainedin manybooksandarticles,forinstanceinWatsonandHill'sDictionaryof CommunicationandMediaStudies(1980),whichhasa 250-wordentryaboutthemodel, andina ScientificAmericanarticleby Pierce,a telecommunicationsengineer(I972).The first givesthefollowingindicationsof themeaningof thevectors:theauthorssaythatthe model'can be appliedto any informationtransfersystem'andthey call it a'process centredmodel'(1980;I49,ouritalics)- tworatherobliquereferences,onea broadgloss ('process'),the otheronlyslightlymorespecific('transfer').Pierce'sexplanationsare contradictory.0ntheonehand,hecallsthemodela'system'(asdoWatsonandHillinthe dictionary)andparaphrasesit in termsof ananalyticalratherthana narrativestructure: 'Thesystemconsistsof an informationsource,a transmitter,a communicationchannel,a noisesource,a receiveranda messagedestination'0972'.32,ouritalics)- notethathe listsonlytheparticipants,nottheprocesses.0n theotherhand,whenhegivesanexample, heparaphrasesthevectorsbymeansof activeverbs: A humanbeingmay typea messageconsistingof the lettersand spaceson the keyboardof a teletypewriter.Theteletypewriterservesasa transmitterthatencodes eachcharacterasa sequenceofelectricalpulses,whichmaybe'on'or'off','current' or'no current'.Theseelectricalpulsesare transmittedbya pairof wiresto another teletypewriterthatactsasa receiverandprintsoutthe lettersandspaces. (Pierce,I972133,ouritalics) Thesearejusttwo exampleswhereothersarepossible.Themeaningpotentialof diagram- maticvectorsis broad,abstractandhencedifficultto put intowords.Theaccompanying textstendto bemuchmoreexplicitaboutparticipants,aboutthingsexistingin space(or representedasthoughtheyare),thanaboutprocesses/eventsandactions.Scientificand bureaucraticwriting,andmanyformsof expositorywritinggenerally,put mostof their meaningin the nounsratherthan in theverbs.Verbs,in theseformsof language,remain restrictedto a relativelysmallsetof (logical)connectors('is','has','leadsto','causes', 'generates','developsinto',andsoon),almostasthoughtheywere'functionwords',like
  • 76. Narrativerepresentations articlesandpronounstralherthan'contentwords/.In hisbookFactualWriting(1985: 40),Martingivesanexample: Thereis littledoubtthat televisioncoverageof a domesticslaughteringoperation, conductedina governmentapprovedabattoir,whichinvolvedtheslaughterof lambs, calvesandswine,wouldgeneratea gooddealof publicrevulsionandprotest. In this extractmostof the specificactionsappearin nounIorm, andsomehavebeen nominalized;that is,turnedintonounsfrom priorfull clausalforms('doubt','coverage', 'slaughteringoperation','slaughter','revulsion','protest').In a textof thirty-sevenwords, thereareonlytwo mainverbs('is' and'generate'),bothverygeneral.Doingsandhappen- ingshavebeenturnedintothings.Thedynamicsof actionhasbeenchangedintoa staticof relations.Diagramsdosomethingsimilar.Theyrepresenteventswhichtakeplaceovertime asspatialconfigurationgandsoturn 'process'into'system'- or intosomethingambigu- ouslyin between,somethingthatcanbecalledeither'system-centred'or'process-centred'. In thisrespectdiagramsareakinto certainformsof nominalizingwriting,whilenatural- isticimages,withtheirhumanparticipantsandtheirmoreconcrete/specificprocesses,are moreakinto story-writing.Like manynaturalisticimages,storiesare abouthumanor animatebeingsandthethingstheydo,andin storiesmuchmoremeaningis put intoverbs thanin mostnon-narrativeformsof writing. Becausetheir meaningis so abstractand general,vectorscan representfunda- mentallydifferentprocessesasthoughtheywerethe same(for instance,'humanstyping lettersand spaceson a keyboard'and'teletypewriterstransmittingelectricalpulses'). Diagramsof the ShannonandWeavertypecan imposetwo modelsof interpretationon onesituationor perhapsonemodelon many,'herethe two modelsare 'transport'and 'transformation'.Figure2.2 representswhat is goingon eitheras transport,movement from oneplaceto another,or as the moreor lesscausallydeterminedtransformation from onething into another.And becauseonesign,the arrow,can representboth,the two meaningsoftenbecomeconffated:movement,transportrstransformation;mobility rsthe causeof,andconditionfor,change,growth,evolution,progress.TheShannonand Weavermodel,for instance,representscommunicationastransport,as movinginforma- tion from oneplaceto another,but it alsoand at the sametime representscommuni- cation as the transformationof messagesinto signals,of 'letters and spaces'into 'electricalpulses'. Thearrowsin the 'systemnetworks'of systemic-functionalgrammar(ourdiagramin figure2.I2 issucha'systemnetwork')areusuallytranscodedby'choose'or'select'(e.g. ' "Conceptual"selects"Classificational","Analytical"or "Symbolical" '). But visually the processis, again,a combinationof transportand transformation.And whensuch networksareturnedintocomputerprograms/as indeedtheyhavebeen,the visualmeta- phorbecomesa realityin whichthereare not people'choosing'between'options',but pulsestransportedto pointsat whicha changeof stateoccurs.At thatpointtheschematic reductionof onesemioticrealityhasturnedinto a blueprintfor another,newsemiotic reality,andpeoplewill havebeenreducedto the roleof'source'and'destination'in an
  • 77. Narrativerepresentations 63 autonomizedandexteriorizedsemioticprocess,justastheyarein ShannonandWeaver's communicationmodel. Differentkindsof narrativeprocesscanbe distinguishedon the basisof the kindsof vectorandthenumberandkindof participantsinvolved, I Actionprocesses TheActoristheparticipantfromwhichthevectoremanates,or whichitself,inwholeor in part,formsthevector(aswiththetrianglein figure2.10).In imagestheyareoftenalso the mostsalientparticipants,throughsize,placein the composition,contrastagainst background,coloursaturationor conspicuousness,sharpnessof focus,andthroughthe 'psychologicalsalience'whichcertainparticipants(e.9.thehumanfigureand,evenmore so,thehumanface)havefor viewers.In figure2.l,for instance,the Britisharelargerthan theAborigines,andplacedintheforeground.In theShannonandWeavercommunication model(figure2.2),the'informationsource'andthe 'noisesource'are Actors(wewill commentontheirdifferentpositionin the diagram,andonthe differentdirectionalityof theirarrows,inchapter6): Whenimagesor diagramshaveonlyoneparticipant,this participantis usuallyan Actor.Theresultingstructurewecall non-transactional.Theactionin a non-transactional processhasno'Goal',is not'doneto'or'aimed at/ anyoneor anything.Thenon- transactionalactionprocessisthereforeanalogousto theintransitiveverbin language(the verbthat doesnot take an object).Theprocessesin figure2.I3 arenon-transactional: thewaterof the GulfStreamdoesnot movesomething,it just moves;andthewindof the Mistraldoesnotblowsomething,itjustblows.Thisvisualrepresentationisakinto theway meteorologicalprocessesarerepresentedin English;it rains,orit snows.As Hallidayhas pointedout(1985:102),otherlanguagesdonotnecessarilydothis.In oneChinesedialect, for instance,onehasto saysomethinglike'the sky is droppingwater';in otherwords, raininghasto berepresentedasa transactiveprocess.In figure2.15,thegestureof theold manformsa vector,buthedoesnotgesturetowardsanyoneor anything,at leastnotsofar aswecanseeinthispicture.Asa result,thevieweris leftto imaginewhoor whathemay becommunicatingwith.Is healreadyintouchwithwhatliesbeyondlife?Isthatwhythe youngboylooksat himwithsuchconcentratedfascination? O Fig2.t4Actors
  • 78. Narrative representations O Fig2.I5 NewYork,1955(RobertFrank) At othertimes,thereis onlya vectoranda Goal(figure2.16).TheGoalisthepartici- pantat whomor whichthevectoris directed,henceit is alsothe participantto whomor whichtheactionisdone,or at whomor whichtheactionisaimed. Representationsof actionswhichincludeonlytheGoalwewillcallEvents:somethingis happeningto someone,butwecannotseewhoorwhatmakesit happen.Figure2.17shows a diagramwhichappearedin the SydneyMorningHeraldduringthe first Gulf War.A vectorrepresentstheactionof movingtowardsthetownof l(hafji,andtheGoalisthetown of l(hafjiitself,representedby a blackdot.But nothingrepresentsthewar planeswhich are movingtowardsl(hafji.Closelyrelatedis the casein whichjust a smallpart of the Actor ls visible,a hand,or a foot,sothat the Actor becomesanonymous.In bothcases there rs in fact an Actor,as in figure2.I7, but the Actor is eitherdeletedfrom the representationor madeanonymous/a visualanalogue,perhaps,of'passiveagentdeletion', a linguisticformof representationthatplaysan importantrolein criticallinguisticsand criticaldiscourseanalysis,as whena newspaperheadlinesays,'FifteenDemonstrators Shot in Riots',therebyomittingto mentionthat theywereshotby police(frew,I979: eTff). Whena narrativevisualpropositionhastwo participants,oneistheActor,theotherthe Goal.fheActorin sucha transactionalprocessisnotsomuchtheparticipantwhichmoves (as in the non-transactionalprocess)asthe participantwhichinstigatesthe movement, andif wehadto givea verbalparaphraseof a transactionalprocesswewouldprobablyuse O Fig2.16Goat
  • 79. NarrativerePresentations 65 O Fig2.17 Gulf War Diagnn (SydneyMoming Herald,I4 Fehruary1991) a transitiveverb,a verbthattal<esanobject(e.g.'transport'or'send'insteadof Iintransi- tivel'move'). In'The Britishusedguns/(figure2.1),thetwo menaretheActor,theAboriginesarethe Goal.Thereis,in fact,a secondtransactionalprocess:thereare alsovectorsformedby linesthat canbedrawnfrom the headsof the Aboriginesto thefire,andsoconstitutea processin whichthe Aboriginesare the Actorandthe fire the Goal.'TheAborigines surroundthefire',onecouldtranscode.Butthepointisnotto finda verbalequivalent;the pointis to establishthat the Aboriginesare representedas Actor andthe fire as Goal. Becauseof itssmallersizeandplacementfurthertowardsthebackground,thisprocessisa ,minorprocess,,embeddedin the majorprocess.Thewholecouldbetranscodedas'The BritishstalktheAborigines,whosurroundthefire'. it ispossibleto arguethatstructuressuchastheShannonandWeaverdiagram(figure 2.2) havebeenaffectedbythefactthat Westernculturegivessuchcentralityto language that the structureof English,wlth its lexicaldistinctionof verbs/processesand nouns/ objects,mayhaveactedasa modelfor a semioticschema.Soarrowsasvectors/processes and boxesas participants/nounsmay be a moreor lessunconscioustranslationfrom
  • 80. Narrativerepresentations languageintothevisual.However,it isimportanthereto insistonthedistinctorganization of the two modes.Thevisualstructureof arrowsand boxesconveysa strongsenseof 'impacting'or'targeting',whichisquiteabsentintheverbaltranslationswhichcomemost immediatelyto mind.Thevisualstructureforegroundsprocedureoversubstantivecontent, the act of impacting'overwhat makesthe impact,moreor lessin the way that,for instance,marketingexpertsare oftenmoreconcernedaboutstrategiesfor reachingcon- sumersthan aboutthe goodsand servicesthat shouldreachthem,or that pedagogic expertsare moreconcernedaboutthe formatof classroominteractionthan aboutthe contentof lessons. i--._:--* -- , - . z : - - F /a-- - ------'--" O Fig2.l8 Speechcircuit(deSaussurelgT4tlgf6D Sometransactionalstructuresarebidirectional,eachparticipantplayingnowtherole of Actor,nowtheroleof Goal,asfor instancein deSaussure'swell-known'speechcircuit' diagram(figure2.18)in which'A'and'B'are nowspeaker,nowlistener.It is notalways clearwhetherbidirectionaltransactionsarerepresentedasoccurringsimultaneouslyor in succession,althoughthereisa tendencyto useonearrowwithtwo headsto signifysimul- taneity,andtwo arrowspointingin differentdirectionsto signifysequentiality.In figure 2.I8,for instance,sequentialityis realizedbytwo separatedottedlines(lineandarrow- head,connectionanddirectionality,areseparateelementsinthisdiagram). O Fig2.19 Simultane0usandsequentialbidirectionality We will referto the participantsin such doublerole. structuresas Interactors.to indicatetheir
  • 81. Narrativerepresentations. 67 2 Reactionalprocesses Whenthevectorisformedbyaneyeline,bythedirectionof theglanceof oneor moreofthe representedparticipants,theprocessis reactional,andwewill speaknotof Actors,but of Reacters,andnotof Goals,bul of Phenomena.TheReacter,theparticipantwhodoesthe looking,mustnecessarilybehuman,or a human-likeanimal- a creaturewithvisibleeyes thathavedistinctpupils,andcapableof facialexpression.ThePhenomenonmaybeformed eitherbyanotherparticipant,theparticipantat whomor whichthe Reacteris looking,or by a wholevisualproposition,for example,a transactionalstructure.In figure2.!5,for instance,the old manandhisgestureform the Phenomenon,whilethe youngboyis the Reacter.In figure2.20,an advertisementfor mineralwater,the manis Actor in a trans- actionalactionprocessin whichthe water is Goal('Themandrinl<swater',onemight transcode):thewholeangleof hisbodyformsa strongvectorbetweenthetwo represented participants.Thisprocess('Mandrinkswater')thenbecomesthe Phenomenonof a reac- tionalstructurein whichthewomanis Reacter- a vectotformedbythe directionof her glanceandtheangleof herleftarm,leadsfrom herto thedrinl<ingman.Shereactsto his actionwith a smileof approval(theprecisenatureof reactionsis colouredin by facial expression).Themanasdoer,thewomanasfaithfuladmirerof hisactions,isa distribution of roleswhich,asGoffmanhasshowninhis GenderAdvertisements(I976), isverycom- mon in advertisements(but not onlvin advertisements):'Whena man anda woman T . . - : : lfffi tjesi?;s j f;w$$$ryrrwr-i;r*;Fi{ies-i*$-ir-;:,i.i}frii"*r'-*,tfiil.,;T .*.*"'*a{&,:si O Fig2.2O Vittel advertisement(/Verlde4 5 Decenber 1987)
  • 82. Narrat i verepresentations collaboratein an undertaking,themanis likelyto performtheexecutiverole'(Goffman, 7976:3D. Like actions,reactionscan betransactionalor non-transactional.In the lattercase thereisnoPhenomenon,asisthecasewiththelookoftheoldmaninfigure2.15.It isthen leftto theviewerto imaginewhatheor sheisthinkingaboutor lookingat,andthiscan createa powerfulsenseof empathyor identificationwith the representedparticipants. Sometimesphotographersor pictureeditorscrop photosback to close-upsof non- transactionalReacterswho lookbored,or animated,or puzzled,at somethingwe cannot see.This can becomea sourceof representationalmanipulation.A captionmay,for instance,suggestwhatthe Reacteris lookingat,but,needlessto say,it neednot bewhat the Reacterwasactuallylookingat whenthepicturewastaken.StuartHall(1982)has describedhowthiskindof manipulationisusedin pressphotographsof politicians. 3 Speechprocessandmentalprocess A specialkindof vectorcanbeobservedin comicstrips:the obliqueprotrusionsof the thoughtballoonsanddialogueballoonsthatconnectdrawingsof speakersor thinkersto theirspeechor thought.Untilrecentlytheywereconfinedto the comicstrips,although therehave,of course,alsobeenspeechprocessesin medievalart,forinstanceintheformof ribbonsemanatingfrom the speaker'smouth.Todaytheyincreasinglycropup in other contexts,too;for instance,in connectionwithquotesin schooltextbooksor onthescreens of automaticbanktellers.Liketransactionalreactions,theseprocessesconnecta human (oranimate)beingwith'content',butwhereintransactionalreactionsit isthecontentof a perception,inthecaseof thoughtbubblesandsimilardevicesit isthecontentof an inner mentalprocess(thought,fear,etc.),andin the caseof speechvectorsthe contentof the speech.Halliday(1985:227ff.)callsthiskindof structure'projective'.ThePhenomenon of thetransactionalReaction,andthecontentof thedialogueballoonor thoughtballoon are not representeddirectly,but mediatedthrougha Reacter,a'Senser'(in the caseof a thoughtballoon)or a'Speaker'(inthecaseof thedialogueballoon). 4 Conversionprocesses The Shannonand Weavercommunicationmodel(figure2.2) formsa chainof trans- actionalprocesses.Thischainingresultsina thirdkindof participant,a participantwhich istheGoalwith respectto oneparticipantandtheActorwith respectto another.Shannon andWeaver's'transmitter'issucha participant,actingasGoalwithrespectto the'infor- mationsource'andasActorwithrespectto the'receiver'.Wewillcallthiskindof oartici- panta Relay(ouruseof thistermobviouslydiffersfromthat of Barthesll977l, whouses it to denotean image-textrelationin whichthe text extends,ratherthan elaborates ['anchors'J,thevisualinformation).Relaysdo notjustpasson/in unchangedform,what theyreceive;theyalwaysalsotransformit - for instance,in the caseof communication models,from'lettersandspaces'into'electricalpulses',or,asin figure2.23,from'grasses, sedgesandffowers'into'urine,droppingsanddeadcarcasses'.
  • 83. NarrativerepresentatIons O Fi92.2lRetay WhiletheShannonandWeavermodelrepresentscommunicationasa chainedprocess with a beginningandanend,henceasan agentiveprocess/a processsetintomotionby an Actor,othermodels,suchasthatshowninfigure2.22,representcommunicationasa cycle, andinthatcasealltheparticipantsare Relays,andagencyis moreweal<lysignified'This kindofprocess,whichwewillcallaConversionprocess,isespeciallycommoninrepresen- tationsof naturalevents;for instance,foodchaindiagramsor diagrammaticrepresenta- tionsof thehydrologicalcycle.But,asfigure2.22shows,it canalsobeappliedto human (inter)action,andwhenthishappenshuman(inter)actlonisrepresentedasthoughit wasa naturalprocess. ("..") /en"oo") /o""oo") Interpreter Interpreter t"'o0"7 =*"0"7 t*"? O Fig2.22Communicationmodel(fromwatsonandHill,l9S0:147)
  • 84. 70 . Narrative representations ! . i l i'atitilt t'i* !:i3t*IeS i ! / " - ilr*r****rrs l?,i i r+' i:i)3::r!i,,p,1i*::* ..-..-..-..-..._.,"_.-,-,".-.-.-..,"_..-_,1 !:ji-1{ta,!1jir!:ll .si:i]#*+'d*ihan. & i! *;t 'q]wsG i,i:|,$irn iltiii$i #4 AiSSal l;trlf,i! lr: :14v* riii4 qr$ii!{ ia!t*.i 1+!s Y*rf$ ls.P.**sl n.!,i. :-3iirriis s:!*,Sr#!!{q ilillf" *"'***uusll*. ff" i*'-'q * x l ' : i----" ---------"----l I Ptr{.*s$*s$fs l3'.: i i-----,_.--.--.---.:-i O Fig2.23 Arctictundrasystem(Saleetal.,1980) 5 Geometricalsymbolism Figure2.24,another'communicationmodel//doesnot includeanyparticipants.Thereis onlya vector,indicatingdirectionalitybymeansof an'infinity'sign,ratherthanbymeans of an arrowhead.Dance'sdiagramis in fact not so mucha communicationmodelas a 'metadiagram'whichshowsusa processin isolation,inorderto discusswhyhelicalvectors aremoresuitablefor the representationof communicationthan,for instance,straightor ., 'lri&#4:S!ilqt:l '' :'cq$e{ &6aS:$ O Fig2.24 Communicationmodel(from Dance,1967)
  • 85. Narrative representations . 7l curvedarrows.Dancedoesthisbypointingat thesymbolicmeaningsof intrinsicproperties of thehelix.Accordingto Dance,theshapeof thehelix: combinesthedesirablefeaturesof thestraightlineandof thecircle,whileavoiding theweaknessof either....lt givestestimonyto the conceptthat communication, whilemovingforward,isatthesametimecomingbackuponitselfandbeingaffected by itspastbehaviour,for thecomingcurveof thehelixisfundamentallyaffectedby thecurvefromwhichit emerqes. (Dance,1967) Imagesof this kindusepictorialor abstractpatternsas processeswhosemeaningsare constitutedbytheirsymbolicvalues,andsoextendthevectorialvocabularybydrawingour attentionto possibilitiesbeyondthediagonalactionlineor thesimplearrow:coils,spirals, heIixes. Variantsof the arrowmayaffectthe meaningof the processin narrativediagrams.A curvedarrow,for instance,partakesof thesymbolicvalueof thecircle,sothattheprocess is representedas'natural'and'organic'(seefigure2.22).Vecforsmayalsolrcattenuated, bytheuseof dottedlines,by makingthearrowheadsmalleror lessconspicuousin other ways/or byplacingit inthemiddle,ratherthanatthefrontoftheline,whichdiminishesthe senseof impacting'and'targeting',andcausesthe meaningof thevectorto movein the directionof mereconnectivitv(seefiqure2.25). Attenuatedvectors O Fig2.25 Attenuatedvectors Thevectorialrelationmayalsobe amplified,by meansof bolderarrows(seefigure 2.26),whichperhapssuggesta certaindensityof 'traffic',asinfigure2.17,thediagramof i # Amplilied v!ctor indicating cenSlty O Fig 2.26 Amplitied vectors indicating density and frequency Ampliliedvector indicatjng frequency
  • 86. 72 . Narrative representations theattackon l(hafji,or bytheuseof a numberof arrows,whichmaysuggestthefrequency or multiplicitywithwhichtheprocessoccurs. In imagesa similareffectcanbeachievedbymakingthediagonalactionlinesmoreor lessconspicuoustmoreor lessdominantinthecompositionasa whoie. 6 Circumstances Asweindicatedin ourdiscussionof 'TheBritishusedguns'(figure2.I),narrativeimages maycontainsecondaryparticipants,participantsrelatedto the mainparticipants,not by meansof vectors,but in otherways.Wewill referto theseparticipants,followingHalliday (1985),as Circumstances.Theyare participantswhichcouldbeleft out withoutaffecting thebasicpropositionrealizedbythenarrativepattern,eventhoughtheirdeletionwouldof courseentaila lossof information. LocativeCircumstancesrelateotherparticipantsto a specificparticipantwewill call Setting.Thisrequiresa contrastbetweenforegroundandbackground,whichcanbereal- izedin oneor moreof thefollowingways:(1) the participantsin theforegroundoverlap, andhencepartiallyobscurethe Setting;(2) the Settingis drawnor paintedin lessdetail (or,in the caseof photography,hassofterfocus),'G) the Settingis more mutedand desaturatedincolour,withthevariouscoloursalltendingtowardsthesamehue,usuallythe blueof distance;(4) theSettingisdarkerthantheforeground,or lighter,sothat it acquires an 'overexposed',ethereallook.Theseformalfeaturescanoccurin variouscombinations, andtheyareall gradients-'more-or-less,,ratherthan'either-0r,,features. Aswewill discussin moredetailin chapter5, settingshaveimportancefor the realiz- ation of visualmodality.The Settingsthemselvescan of coursebe readas embedded analyticalprocesses('Thelandscapeconsistsof grass,treesandrocks,). Thetoolsusedin actionprocessesareoftenrepresentedas Circumstancesof Means.If thisisthecase,thereisnoclearvectorbetweenthetoolanditsuser.Thetoolsthemselves may,of course/constitutethevectorswhichrealizetheactionprocesses/aswiththegunsin 'TheBritishusedguns'(infigure2.r),andtheyneednotbeobjects.Forinstance,wewould interpretthegestureof theoldmaninfigure2.15asa non-transactionalaction('Theold manaddressesan unseenparticipant'),andhishandsasa circumstanceof Means('The oldmanaddressesan unseenparticipantwlthhishands,). Figure2.27showsa penguinwith herbaby.Thereis againnovectorto relatethetwo. Yetthe penguinandherbabyclearlyformtwo distinctparticipants:this is a pictureof a penguinwith ababy.In sucha casewe will interpretthe relationas a Circumstanceof Accompaniment.Asthepicturecontainsnovectoranddisplaysthepenguinmoreor less frontally,againsta de-emphasizedbackground,weinterpretit as'analytical,,asthekindof picturemorelikelyto illustratea textgivingdescriptiveinformationaboutpenguinsthana storyaboutwhatpenguinsdo.
  • 87. Narrativerepresentations' 73 O Fig2.27 Penguinwith baby(oakleyefal.,1985) SUMMARY Figure2.28summarizesthe distinctionswe haveintroducedin thissection.'Following Halliday(1985),wehavecalledprocessesthatcantakea wholevisual(or verbal)prop- ositionas their 'object' proiective,and the othersnon-proiective.The squarebrackets indicatea singlechoice;for instance,'atransactionalactionis eitherunidirectionalor bidirectional'.0urclaimis that the'choices'infigure2.28 chartthe principalwaysin whichimagescan representthe world'narratively'-that is, in termsof 'doing'and 'happening'.
  • 88. 74 , Narrativerepresentations ^**,*{ lon-lrolective { - Unidirectional I Transactional -----N Action -----N L Bidirectional L Non-transactional ( Non-transactional)reaction ,a"a"rra,-*l P . o j ( t i w - { Mental process VerbaI process .u"rr",'""*fstructures L Non agentive:Conversion Setting lvleans Accompaniment O Fig2.28 Narrativestructuresin visualc0mmunication Cirauartunaar{ REALIZATIONS UnidirectionaI transactional action BidirectionaI transactionaI action Non-transactionaI action Actor Goal Interactors TransactionaI reaction Non-transactionaI reaction A vector,formedbya (usuallydiagonal) depictedelement,or anarrowlconnectstwo participants,anActoranda Goal. A vector,formedbya (usuallydiagonal) depictedelement,or a double-headedarrow, connectstwo Interactors. A vector,formedbya (usuallydiagonal) depictedelement,or anarrowtemanatesfrom a participant,theActor,butdoesnotpointat anyotherparticipant. Theactiveparticipantin anactionprocessis theparticipantfromwhichthevector emanatesor whichisfusedwiththevector. Thepassiveparticipantin anactionprocess istheparticipantat whichthevectoris directed. Theparticipantsin a transactionalaction processwherethevectorcouldbesaidto emanatefrom,andbedirectedat,both participants. An eyelinevectorconnectstwo participants,a Reacteranda Phenomenon. An eyelinevectoremanatesfroma participant, the Reacter,butdoesnotpointat another participant.
  • 89. Reacter Phenomenon Conversion Mentalprocess Senser Verbalprocess Sayer Utterance Qailinn Means Accompaniment Narrativerepresentations Theactiveparticipantin a reactionprocessis theparticipantwhoselookcreatestheeyeline. Thepassiveparticipantin a (transactional) reactionistheparticipantat whichtheeyeline isdirected;in otherwords,theparticipant whichformstheobjectof the Reacter'slook. Thesametermis usedfor theparticipant (verbaior non-verbal)enclosedbya'thought bubble'. A processinwhicha participant,the Relay,is theGoalof oneactionandtheActorof another.Thisinvolvesa chanqeof stateinthe participant. A vectorformedbya'thoughtbubble'ora similarconventionaldeviceconnectstwo participants,theSenserandthe Phenomenon. Theparticipantfromwhomthe'thought bubble'vectoremanates. A vectorformedbythearrow-likeprotrusion of a'dialogueballoon'or similardevice connectstwo participants,a Sayerandan Utterance. Theparticipantina verbalprocessfromwhom the'dialogueballoon'emanates. The(verbal)participantenclosedinthe 'dialogueballoon'. TheSettingof a processis recognizable becausetheparticipantsintheforeground overlapandhencepartiallyobscureit; because it isoftendrawnor paintedin lessdetail,or,in thecaseof photography,hasa softerfocus; andbecauseof contrastsin coloursaturation andoveralldarknessor lightnessbetween foregroundandbackground. TheMeansof a processisformedbythetool withwhichtheactionisexecuted. It usuallyalsoformsthevector. An Accompanimentisa participantina narrativestructurewhichhasnovectorial relationwithotherparticipantsandcannot beinterpretedasa SymbolicAttribute(see chapter3).
  • 90. 76 . Narrative representations VISUALSTRUCTURESAND LINGUISTICSTRUCTURES Wehavedrawnattentionto thefactthat,whilebothvisualstructuresandverbalstructures canbeusedto expressmeaningsdrawnfroma commonculturalsource,thetwo modesare notsimplyalternativemeansof representing'thesamething'.It iseasyto overemphasize eitherthesimilarityor thedifferencebetweenthetwo modes.0nlya detailedcomparison canbringouthowinsomerespectstheyrealizesimilartypesof meaning,thoughindifferent ways,while in other,perhapsmost respectsthey representthe world quite differently, allowingthe developmentof the differentepistemologieswe discussedin the previous chapter.In this brieffinal sectionwe wishto explorethis in somedetailwith respectto narrativevisualstructures. Bycomparisonto thestructureswewill discussinchapter3,narrativevisualstructures arecomparativelyeasyto'translate';though,aswewill see/therecertainlyis no one-to- onecorrespondence.Like'non-transactionalactions','one-participantmaterialprocesses' (Halliday,1985:103ff.)representeventsasthoughtheybearno relationto,andhaveno consequencesfor,participantsotherthanthe Actor.In Manypeoplemigratedonecannot adda secondparticipantto this clauseandsay,for instance,Manypeoplemigratedtheir relatives,althoughonecanof courseaddcircumstances:Manypeoplemigratedto Aus- tralia. And,like 'transactionalactions','two-participantmaterialprocesses'(Halliday, 1985:103ff.) involvetwo participants(e.9.MigrantsinvadedAustralid. But linguistic 'Events'andvisual'Events'arequitedifferent.LinguisticEventshaveprocessesthatare 'happenings'whichcannothavean Actor,as in Manyof my relativesdied.Inthecaseof visualEvents,the Actor is left out,but couldhavebeenused.Theyarethe equivalentof passiveswith agentdeletion,of clauseslike Manyof my relativeswerekilled,ratherthan of clauseslike Manyof my relativesdied.Toshowsomeonedyingii is necessaryto show someonebeingkilled,or to showsomeoneperformingan actionthat representshisor her death.Also,whilein Englishmanyprocessescantakea thirdparticipant,the'Beneficiary' (traditionally'indirectobject'in,e.9.,Marygavehimtheboob,in imagesthepossibilityof sucha thirdparticipantdoesnotexist.Whatisa Beneficiaryin Englishbecomesa Goalin images('shemessage-sendshim'insteadof 'shesendshima message'). 0n the otherhand,Englishlacksthe visualmode'sstructuraldevicesto represent eventsascyclical(althoughlinguisticparticipantscanhavea doublerolein English- for instance,in exampleslike Hemadethemdo it,wherethemisGoalof makedoaswellas Actorof do;cf. Halliday,1985:153).Nor is therean'interactional'process.To realize whatwe havecalled'lnteractors',Englishwouldhaveto makeuseof reffexivepronouns. Consider,for instance,the problemof trying to 'translate'figure2.18,de Saussure,s 'speechcircuit'diagram,into English.A singlevisualprocessindicatessomethingfor which,in English,weneedat leastfourclauses:'Aspeaksto B','B speaksto A,,'A listens to B','B listensto A'. Howcanonerenderthisin onec{ause?'A andB communicatewith eachother'?But that causes'A' and'B'to losetheir separateidentity,transforminga reciprocal,bidirectionaltransactionintoa jointlyauthorednon-transactionalaction. Whatwe havecalledthe'non-transactionalreaction'isin somewaysakinto what Hallidaycallsthebehaviouralprocess(1985:I2B),aprocesstypewhichcantakeonlyon
  • 91. Narrativerepresentations. 77 participant(whomustbehuman)andservesto realizea restrictedfieldof action,thefield of'physiologicalandpsychologicalbehaving'(1985:128).But the meaningsof visual 'non-transactionalreactions'forma morerestrictedfield,tiedupastheyarewithonekind of behaviour- looking. Projectiveprocesses-that is,mentalandverbalprocesses- playan importantpart in English,andit ispossibleto distinguisha numberof differenttypesof eachonthebasisof formalgrammaticalcriteria(Halliday,1985:106ff.,129).Mentalprocesses/for instance, includeprocessesof perception('see','hear',etc.),processesof affection('like','fear', 'wish',etc.)andprocessesof cognition('know','think','believe',etc.).Eachhasa Senser, the personwho doesthe seeing,or liking,or knowing(thishasto be a person/or a participantrepresentedas human),anda Phenomenon,someoneor somethingseen/or liked,or known,by the Senser.Phenomenamay be realizedby participantsor whole structures,just asinthe caseof images.In Manypeoplewantto migrateto Australia,the clausefo migrateto Australiais Phenomenon;this setsmentalprocessesapart from actionsandtransactions,whichcannothavea clauseasGoal.Butwhatwehavecalledthe 'transactionalreaction'can,if onewishesor needsto, be relatedto onlya subsetof the perceptionprocess,becausenon-visualPhenomenacannotdirectlyberealizedinthevisual semiotic.Mentalprocessesform,aswe haveshown,onlya minorcategoryin the visual semiotic;asfar aswecansee,therearenostructuralvisualdevicesfor makingthestrong distinctionbetween'cognition'and'affection'processesthathascometo characterizethe ideationalresourcesof English.Thecinema,however,hasdevelopedafairlyextensivesetof projectiveconventionsfor realizingdifferentkindsof mentalprocessessuchasmemories, dreams,hallucinations,andsoon. In English,verbalprocessesdifferfrom mentalprocessesin that theydo not needa human'Sayer'(onecansay Thedocumentsaidthat..., but not Thedocumentthought that. . .).0n theotherhand,likementalprocesses,theycantakewholeclausesastheir object,andthisintwodifferentways- intheformof ReportedSpeechGs in Hesaidtthatl hehadno ided andin theform of QuotedSpeech(e.sinHesaid'l haveno idea).fhere seemsto benodirect,structuralwayof expressingthiskindof differencevisually.'Dia- logueballoons'alwaysquote. Wehaveidentifiedonlythreedifferenttypesof circumstancein images:location,means andaccompaniment.Allthreeexistin English(Halliday,7985:1,371f.),buttheretheyare bynomeanstheonlytypes.Englishallowsall kindsof informationto beaddedto thebasic narrativepropositionconveyedby the process('What happened?')andthe participants ('Whoorwhatwasinvolved?');informationabouttime('Whendidit happen?';'Howlong didit last?');aboutpurpose('Whatdidit happenfor?');cause('Whydidit happen?')and so0n. Thefollowingtablegivesanoverviewof someof thecorrespondencesbetweenlinguistic andvisualnarrativeorocesses:
  • 92. Narrat ive representations Table2.7 Nafiatiueprocessin languageandyisualcommunication VisuaI narrati veprocesses Linquistic narrativecIauses Non-transactionalaction Unidirectionaltransactionalaction Event Bidirectionaltransactionalaction Non-transactionalreaction Transactionalreaction Mentalprocess Verbalprocess Conversion 0ne-participant(Actor)materialprocess('action') Two-participantmaterialprocess Passivetransactionalclausewithagentdeletion Behaviouralprocess(fieldof looking) Mentalprocess:perception(visualonly) Mentalprocess(cognitionandaffection) Verbalprocess(quotation) Verbalprocess(affection) Comparisonssuchasthesecanhighlightwhichwaysof representingtheworldcanbe realizedlinguistically,whichvisuallyandwhich(moreor less)inbothways.This,inturn,is usefulasa backgroundfor analysingrepresentationin multimodaltexts:photographsand their captions,diagramsand their verbalglosses,storiesandtheir illustrations.If, for instance,a diagramshowsan arrowemanatingfrom a participantlabelled'environment' anddirectedat anotherparticipantlabelled'message',thena'literal'translationwouldbe 'the environmentactsuponthe message/.If the accompanyingtext saysthat the'com- municationprocess'is'interactingwith factors(or stimuli)from the environment' (WatsonandHill,1980:14),it'mistranslates',andthe mistranslationis notdueto the limitationsof eitherEnglishor visualcommunication.A literal'translation'of'thesource sendsa messageto thereceiver',ontheotherhand,isnotpossible:thespatialrepresenta- tion of verballyconceivedideaschangesthe ideasthemselves,andviceversa.In the next chapterwewillshowfurtherexamplesofthisproblem. Notes 1 We are gratefulto the Danish previouseditionof thisbook,this degrees. 2 Wewouldliketo thankBenteFogedMadsenfor someusefulcommentsthathaveledto improvementsof thisdiagram. art historianLiseMark for pointingout that,in the paintingwasreproduced'on its side',i.e.tilted by 90
  • 93. 3 C o n c e p t u a lr e p r e s e n t a t i o n s : d e s i g n i n gs o c i a lc o n s t r u c t s CLASSIFICATIONAL PROCESSES In the previouschapterwe notedthat visualstructuresof representationcaneitherbe narrative,presentingunfoldingactionsandevents,processesof change,transitoryspatial arrangements,or conceptual,representingparticipantsin termsof theirmoregeneralized andmoreor lessstableandtimelessessence/intermsof class,or structureor meaning. It is to the lattercategoryof representationalstructuresthat we nowturn,beginning with classificationalprocesses.Classificationalprocessesrelateparticipantsto eachother in termsof a'l<indof' relation,a taxonomy:at leastonesetof participantswill playthe roleof Subordinafeswithrespectto at leastoneotherparticipant,the Superordinafe.We havealreadycomeacrossanexampleintheleft-handpictureof figure2.1,wherethethree participants- theaxe,thebasketandthewoodensword- wererepresentedas'species'of thesame'genus',asall belongingto thesameoverarchingcategory.In thisexamplethe overarchingcategorywasnot shownor named.Thestructurewasa CovertTaxonomy,a taxonomyin whichthe Superordinateis inferredfrom suchsimilaritiesastheviewermay perceiveto existbetweenthe Subordinates,or onlyindicatedin theaccompanyingtext,as infigure3.1. 0nevisualcharacteristiciscrucialintherealizationofcoverttaxonomies:theproposed equivalencebetweenthe Subordinatesis visuallyrealizedbya symmetricalcomposition. TheSubordinatesare placedat equaldistancefrom eachother,giventhe samesizeand the sameorientationtowardsthe horizontaland verticalaxes.To realizethe stable, timelessnatureof the classification,the participantsare oftenshownin a moreor less objective,decontextualizedway.Thebackgroundis plainand neutral.Depthis reduced or absent.The angleis frontal and objective.And frequentlythereare words inside the picturespace.Thesefeatureswill be discussedin a laterchapterunderthe heading 'modality'. Classificationprocessesdonot,of course,simplyreffect'real','natural'classifications. Forparticipantsto beputtogetherina syntagmwhichestablishestheclassificationmeans thattheywerejudgedto bemembersof thesameclass,andto bereadassuch.'Naturaliza- tion' is not natural,whetherin imagesor in language.Theorderingin the imageitself producesthe relations.Thismakesit possiblefor the producerof an imageto classify basketsandweapons(frgure2.I), or zoologists,wildlifephotographersandAboriginal storytellers(figure3.1) asbeingof thesameorder. Coverttaxonomiesare oftenusedin advertisements,wherethe photographsmay,for instance,showarrangementsof bottlesthat representthe varietyof productsmarketed undera brandname,or arrangementsof differentpeoplewho all usethe sameproduct. Figure3.2 is a pagefrom Cosmopolitanmagazine.Whatdo thesewatcheshavein com, mon?Theyall belongto thesameXposerange.
  • 94. Conceptual representations @ rig f.f cuideintertace(Microsoft,1994) Othertaxonomiesshowa higherdegreeof (explicit)orderingincludea Superordinate. Theyrepresentand namethe Superordinatewithinsomekindof treestructure.In that structurethe orientationis vertical,andthe Superordinateis placedaboveor belowthe Subordinates,asin figure3.2.fhe participantsmayberealizedverbally,visually,or both verballyandvisually,buttheprocessisalwaysvisual. Taxonomiesdo not haveto berepresentedbyformaldiagramswith simplelines;they mayberealizedin morerealistfashion,for instance,by an actualtreein a'family tree'. 0verttaxonomiesare usually'chained',sothat the'intermediate'participants(e.9.the 'inorganicsubstances'and'organicsubstances'offigure3.3) will beSuperordinatewith respectto someof the otherparticipants,and Subordinatewith respectto others.To indicatethiswe will cointhe term Interordinate.In otherwords,overttaxonomieshave levels,andparticipantsat the samelevelare representedasbeing,in somesense,'ofthe samekind'. Treestructures,however,are not onlyusedto realize'kind of' relations.'Reporting diagrams',showingthe hierarchicalstructureof companiesand otherorganizations, andgenealogicalor evolutionarytrees,usethe samestructure.Thismeansthat visual grammarconflates,or at leastrepresentsasverycloselyrelated,whatwould,in language, be expressedby differentmeans.Conceptualclassificationis representedby the same
  • 95. Conceptualrepresentations. 8), rFrr'"&p*{f 'sAr,{$c dF v.".1';..s{;iifiy s .{.r.'",;s:r ; *tr i:?}J;1s1f.a.6+rI Ir"5i1'a+lsr @ fig f,Z Xposerange(Cosmopafifau,November2O0l: g4)
  • 96. 82 . Conceptualrepresentations Sourcesof siqns I Inorganicsubstances organic substances - t l Natural Manufactuied Extraterrestrial Terrcstrial I Homosapiens I componetrts organisms Sp!echlesscreatures components ofgdisms O fis f.: Sourcesof signs(Eco,l976b:177) structureassocialhierarchy;that is,the moregeneralideais representedas similarto greaterpower.AsVirginiaWoolfhassaid,'GeneralideasarealwaysalsoGenerals'ideas.' Furfher,hierarchiesof conceptsandhierarchiesof socialpowerarerepresentedassimilar to genealogies.In otherwords,theidentityof an individual(or a species)isrepresentedas being'subordinate'toits'origins'or'ancestors'inthesamewayasspecificconceptsare subordinatedto moregeneralandabstractconcepts,andloweremployeesor localbodies to managersor centralorgans. Diagrammatictreestructurescantakedifferentforms.Thebranchesof the treemay be parallelor oblique,straightor curved,andso on (seefigure3.4).Obliquebranches tre structure with parallelbranches tree slructure wilh cuded bianches - /nA A A A A A A A O piSf.l Treestructureswith parallel(straight)0bliqueandcuryedbranchet
  • 97. Conceptualrepresentations. 83 abstractsomewhatlessfromtheshapeof thetreethanparallelbranches,sothat moreof the symbolicmeaningof the treecanbe preserved.Hencetheyare commonin contexts wherea senseof'generation'and'growth'isconnoted,asfor instanceingenealogiesor in thediagramsusedin'generativegramrnar',a formof linguisticswhichpositsthat'surface structures'aregeneratedfrom (possiblyinnateandmaybeuniversal)'deepstructures'. Thecontrastbetweenstraightand curvedbranchesis perhapssimilarto that between participantsrepresentedas boxesand participantsrepresentedas circlesor ovals (seechapter2),a contrastbetweenthe'mechanical'and'technological'andthe'natural' and'organic'. Manytreediagramsareinverted('bottom-up'),andwhenthebranchesareobliquethe overallshapewill tendtowardsthat of a pyramid.Suchstructuresare concernedwith hierarchyandhierarchicaldifferenceratherthanwith clarityaboutlevels:a readingof levelsispossible,butnotreadilyfacilitated.Notalltrees,however,areinverted.Sometimes the specificor,in thecaseof genealogiesandevolutionarytrees,the present,is placedon thetop,depicting,for instance,humankindasthepinnacleof evolutionratherthanasbeing foreverdominatedbyits lowlyorigins. Althoughclassificationalstructuresrepresentparticipantsin termsof theirplacein a staticorderitheverballabelsandexplanationswhichmayaccompanythemdo notalways do so.Theterm'reportingdiagram',for instance,usesan activeprocess('report')rather thana staticonesuchas'issubordinateto'.Genealogiesandevolutionarytrees,similarly, may be glossedby verbslike'generates','evolvesinto','begets',and so on. Visually, however,a hierarchicalorderis signified,a system.Thusthe visualrepresentationcan blurthe boundariesbetweenthe dynamicandthe static.Is the dynamicin realitythe instantiationor enactmentof an underlyingsystem?0r is the staticthe systemization andobjectificationof a dynamicandever-changingreality?Suchquestionsbecomedif- ficultto answerin a modewhichhaserasedthe boundariesbetweenthe schemaandthe bluenrint. A similarblurringmayoccurbetweenanalytical('partof') andclassificational('kind of') processes.The'stackdiagram'infigure3.5 couldbecalledclassificational,andis in fact so renderedin the accompanyingtext:'to the Latin word /mus/correspondtwo differentthingswhichweshallcallx, andxr' (Eco,r976b:78).In otherwords,a'mouse' isa kindof 'mus'.Butit canalsobeseenasanalytical.0necanalsosaythatthemeaning 'mouse'ispartof themeaningof 'mus'.And,mostimportantly,Ecohaschosentheformof theanalyticaI diagram. Mous Mus Rat O fis f.l Semantictielddiagram(Eco,l976b:78)
  • 98. 84 . Conceptualrepresentations Finally,classificationaldiagramsmaybe rotatedthroughninetydegreessothat their mainorientationis alongthe horizontalaxis.Theyhavethenthe orientationtypicalof narrativediagrams,andhencea dynamicconnotation.Buttheyretainthestructureof the classificationaldiagram.Theystill representthe relationbetweenthe participantsas a system.Featuresfromdifferenttypesof structureareabstractedandrecombinedto create patternsthat are ambiguouslyin betweenthe dynamicandthe conceptual.Whensuch diagramsacquirearrows- as,for example,in systemnetworks(seefigure3.8) - they becomeinfact dynamicandnarrative.Yettheystill movefromthegeneralto thespecific, in contrast(forexample)to flowcharts. Taxonomiesandffowchartsclearlyprovidetwo differentkindsof knowledge.Theone representsthe worldin termsof a hierarchicalorder.Its mainconcernis the rankingof phenomenafromtheperspectiveof a singleunifyingterm,beit thatof theoriginof things, the mostgeneralizinggeneralization,or that of the highestpower.Theotherdescribes the worldin termsof an activelypursuedprocesswith a clearbeginningandan end(or 'input'and'output','source'and'destination','rawmaterials'and'finishedproduct').It hasa sequentialprogressionandis goal-oriented.And,aswe havenotedalready,system networkssuchasweuseinthisbookattemptto combinethetwo perspectives. Recentlyanotherkindof diagramhasbegunto gainascendance- the'network'.Net- worksseekto showthemultipleinterconnectionsbetweenparticipants.Anyparticipantin a network('node')canform an entry-pointfrom whichits environmentcanbeexplored, andthe vectorsor lines('links')betweentheseparticipantscantakeon manydifferent values,thevalueof signification('a meansb'),of combination('a goeswithb'),of com- position('a containsb'):theessenceof the linkbetweentwo participantsisthattheyare, in somesense,nextto eachother,or closeto eachother,associatedwith eachother.To demonstratethedifferencewitha linguisticexample,a taxonomywouldshow,for instance, a hierarchyof words,a 'ffowchart',for instance,a wayof generatinga clausebyfollowinga precisesequenceof instructions',anda networkmightshowlhecollocationof words- the otherwordswith whichanygivenwordtypicallycombines/regardlessof the structural relationsbetweenthewords(seefigure3.6). Figure3.7 showsa'linear' (ffowchart)and'non-linear'(network)representationfrom anarticleona'writer'sassistant'computerprogram.Thenetwork,itsauthorssay/allows thewriter'to form ideasintoan associativenetwork'(Sharplesand Pemberton,I992i 22).Theprinciplebehindsuchnetworksclearlyrelatesto theideaof the'non-linear'text to whichwehavealludedinourdiscussionof DickBruna's0n My Walk(figure1.2)andto whichwe will returnin greaterdetailin chapter6. In discussinga pagefrom Brunawe stressedthatsuchpagesontheonehandprovidethereaderwithmanychoices,manypaths to follow,but on the other handtend to obscurethe fact that the rangeof choices is ultimatelypre-designedand limited.As such,networksare,in the end,just as much modelledonformsof socialorganizationastaxonomiesandffowcharts.Thetaxonomyis modelledon a static,hierarchicalorganizationin whicheverythinghasits pre-ordained placein a grandschemeunifiedbya singlesourceof authority.Theflowchartis modelled onthe principleof authoritativelyprescribed,structured,goal-orientedactivity.Thenet- work is modelledona formof socialorganizationwhichisa vastlabyrinthof intersecting
  • 99. ConceptuaI representations (D fis f,O Taxonomy,flowchartandnetwork localrelationsin whicheachnodeis relatedin manydifferentwaysto othernodesin its immediateenvironment,but in whichit is difficult,if not impossible,toforma coherent viewof thewhole.Perhapsit is notaccidentalthat this kindof networkis comingto the fore in an ageof increasingsocialfragmentationand regionalization.But regionsare neverthelessconnectedto thewholeandthenetworkmodelmayobscuretheglobalizing tendencies,whichare alsoandsimultaneouslyat work in contemporarysociety.Thisis recognizedby thosewho pioneerthe applicationof this modeof representation,for instancein connectionwith hypertexts,whicharealsonetworks: Readerswho browsehypertextnetworksbecome'lost',unawareof wheretheyare in relationto the document,andthus unableto achievea senseof text, i.e.an affection Pleasure displeasure physical psychological (etc pleasure pleasure / l / r--+---r(etc.) contentment joy delight
  • 100. 86 ConceptuaI representations lrtii:ri !4 iJl iXf;r?;i il @ fig f.Z Network(fromSharplesandPemberton,1992) intellectualfeelfor the substanceof the document.Thisis analogousto startingat a particularword in a thesaurusand passingfrom onesimilarassociativeword to another.Soonit is likelythat the readerwouldbe examininga word that is completelydifferentfrom the originalword,with no notionof anymeaningto the derivationthat hasjusttakenplace. (Ghaouiet al.,1992:7I0) In the pronouncementsof networkgurul(evinl(elly,as reportedbyJim McClellanin the )bserver,thepoliticalimplicationsof thenetworkbecomeevenclearer: Theimageof the atom. . . stoodfor powerandknowledge.Its sure,regularorbits anddefinedspacesrepresented'law-abidingsolarsystemsof energy'undercentral direction.In contrastthe Net- a tangleof apparentdisorder- wasan iconof no beginning,nocentre,noend(orall beginning,all centre,allend).It wastheemblem of ournewunderstandingof thecomplexlogicof natureandcomputers;thebanner of systemswhich,in somesenses,organizedthemselves.Theatom had beenthe iconof the 26ih century(theAtomAge),heconcluded,butthe Netwouldbethe archetypeof thecomingnetworkcultureof the2lst. (McClellan,0bserverLife Magazine,21 September1994,p. 62) in figure3.8 we summarizethedistinctionswe havemadein thissection.Networksare not includedin the summaryastheyare'analytical'ratherthan'classificational'.The differencebetween'single-'and'multi-levelled'ismarkedbytheabsenceor presenceof 'lnterordinates'.0ur discussionabovehas,wehope,madeit clearthatweseethesedistinc- tionsastoolswith whichto describevisualstructuresratherthanthat specific,concrete visualscannecessarilyalwaysbedescribedexhaustivelyanduniquelyintermsof anyoneof ourcategones.
  • 101. ConceptuaI representat ions a7 Coverttaxonomy - Single-levelled Overttaxonomy - L Ntulti-leve|ed O fig f.g Chssificationatimagestructures REALIZATIONS Coverttaxonomy A setof participants(,subordinates,)is distributedsymmetricallyacrossthepicture space,at equaldistancefrom eachother, equalin size,andorientedtowardsthe verticalandhorizontalaxesinthesame way. A participant('Superordinate,)is c0nnectedto two or moreotherparticipants ('Subordinates')througha treestructure withtwo levelsonly. A participant('Superordinate,)is connectedto otherparticipantsthrougha treestructurewith morethantwo levels. Theparticipantswhichoccupyintermediate levelsareinterordinates,whilethosewhich occupythelowestlevel(if the Superordinateisontop)or thehighestlevel (if theSuperordinateisat thebottom)are Subordinates. Single-leveIIedoverttaxonomy Multi- |eve||edovert taxonomy ANALYTICALPROCESSES Analyticalprocessesrelate participantsin terms of a part-wholestructure.They involvetwo kindsof participants:oneCarrier(thewhole) andanynumberof possessive Attributes(theparts).We havealreadygivenan exampleof an analyticalstructure,the pictureof the Antarcticexplorer(figure2.4),whichanalysedthe explorerin termsof his'outfit'. Fashionshots,too,are analytical.Likethe pictureof the Antarcticexplorer, theyclearlydisplaythe parts of an 'outfit',and labelboththe Carrier('easy-wearing, inexpensivecottonsteamedwiththerightaccessories'-seefigure3.9) andthe possessive Attributes('Laura Ashleytrenchcoat,Stuart Memberysweater,Benettonjodhpurs,), albeit in a caption,ratherthan insidethe picturespace,and in the ffowerylanguageof fashionwritingwhichhasbeendescribedsowellbyBarthes(r967b).
  • 102. ConceptuaI representations O fis f.S Easy-wearingC0ttons(yogue,November1987) Differentas they mayseemat first sight,mapshavethe samestructure:thereis a Carrier,for instance'Australia',and thereare PossessiveAttributes,for instance'the statesof Australia',andbotharelabelled,eitherinsidethepicturespaceor in a legendor caption.Mapsmayprovidequitedistinctanalysesof what seemsto bethe samecarrler' Somemapsfocuson geographicalfeaturessuchaswaterways/altitude,etc.,whileothers concentrateon socialandpoliticalboundaries.Analysisalwaysinvolvesselection.Some attributesor characteristicsof the Carrieraresingledout ascriterialinthegivencontext orlgeneralltwhileothersareignored,treatedasnon-essentialandirrelevant' Thedifferencebetweenthe mapandthefashionshotliesnot in their ideationalbut in their interpersonalstructures- for instance,in their modality(seechapter5)' Many analyticalvisualshavelow modality,from the naturalisticpointof view.Toomuchlife- likeness,too muchdetail,woulddistractfromtheiranalyticalpurpose'0nlytheessential featuresof the possessiveAttributesare shown/andfor this reasondrawingsof various degreesof schematizationareoftenpreferredoverphotographsor highlydetailedartwork' Therepresentationof depthis reducedor absent,asisthedetailedrepresentationof llghi andshadow,andof subtletonaldistinctions.Colour,if it is usedat all, is restrictedto a reducedpalette,or usedconventionally- for instance,to distinguishparticipants,suchas differentsocioeconomicgroups/or landforms.Backgroundis left out,or onlysketched in lighly.Andthe PossessiveAttributesarelabelled.Notethatthearrowsinfigure2.4 do not realizenarrativeprocessesbut a relationof identitybetweena verbalanda visual
  • 103. ConceptuaI representat ions realizationof the samePossessiveAttribute.Yetphotographs,particularlyposedphoto- graphs,canalsobeanalytical,asin thecaseof fashionshots,or of advertisementswhich givea detaileddepictionoftheadvertisedproduct,or of press'mugshots'ofpoliticiansand othernewsworthypersons.Theschoolsocialstudiestextbookfrom whichwetookfigure 2.4 shows- onthe pageoppositeto that showingthe drawingin figure2.4 -the picture reproducedinfigure3.10. It is an analyticalpicture;there is neithera vector (narrativeprocess)nor com- positionalsymmetryand/oratreestructure(classificationalprocess).ltservesto identify a Carrierandto allowviewersto scrutinizethis Carrier'sPossessiveAttributes.Some minordegreeof loweredmodalityisalsopresent:thebackgroundisplain,andthefactthat the pictureis posedaddsfurtherartificiality.Yet,althoughit is analytical,its purposeis more interactionaland emotivethan representational.The interactionalsystemof the gazedominates:the gazeof representedparticipantsdirectlyaddressesthe viewersand soestablishesan imaginaryrelationwiththem,whilemoreschematicanalyticalpictures inviteimpersonal,detachedscrutiny.A similarargumentcanbe madeaboutadvertise- mentsshowingthe advertisedproduct- the overallimpressionof an abundanceof parts(or ingredients,or varietiesof the product),or the alluringsensoryqualityof the advertisedproductas a whole(thestreamlinedsheenof the car the vividcolourand @ fiq f.fO Sir DouglasMawson(0akleyetal.,1985)
  • 104. 9O . ConceptualrePresentations textureof the ingredientsof a cannedsoup)take precedenceovermoredispassionate scrutinyof the PossessiveAttributes.Persuasionis foregrounded,instructionand expositionare backgrounded.Thetextbookincludesbothfigure2.4 andfigure3.10 becauseit seeksnotonlyto teachchildrenobjectivefactsaboutAntarcticexplorationbut alsoto makethememotionallyidentifywith anadventuroushero.Untilrecentlyrthisthen decreasedin the lateryearsof education.Advancedtextbooksaddressedtheir readers as'no longerneedingpictures',as havingbeenweanedoff everydaynaturalism,andas havingacquiredthe abstractand impersonalattitudethat characterizedhigherlearning and higherart appreciationin Westernculture.However,the tendencyis nowfor the interpersonalto enterthe textbooksof lateryears,aswell- throughimageandthrough writing.In the languageof the seniorsecondaryschooltextbook,as in the languaqeof scienceandbureaucracy,thepassivevoicehadbeennormal,realizinga moreimpersonal anddetachedformof address: Theseoutlinesmaybeabstractedfromtopographicmapsandstreetdirectories. In thelanguageof theprimary-schooltextbook,asalsoinadvertisinglanguage,'you'wasa keyword,andthereaderwasalwaysdirectlyaddressed: Hereis a pictureof Antarctica.If you landedherein a spaceship,howwouldyou describethisPlace? Today,the personalandthe informalenterincreasinglymanydomainswhichformerly werecharacterizedbyimpersonalandformalmodesof address,verballyaswelI asvisually' 0f course,somephotographsremainalmostas objectiveand detachedas traditional diagramsandmaps.Thisis still trueof manyscientificphotographsand,for instance,of aerialphotographs:their top-downangleensuresthe absenceof depthandbackground and,dependingon the heightfrom whichtheyaretaken,detailis moreor lessremoved. But,as in our primary-schooltextbookexample/theywill nowoftenbe combinedwith lessformal pictures,even in the PowerPointconferencepresentationsof scientists (Rowley-Jolivet,200Q. Abstractart may also be analytical.The structureof Theovan Doesburg'sPure Painting,forinstance(figure3.11), is likethe structureof a map'It analysesreality in termsof possessiveAttributes,highlyabstractones:rectanglesof differentsizeand colour.But it doesnot labeleitherthe Carrieror the PossessiveAttributes.It leavesit upto theviewerto do so,andasa resultthe paintingcanbereadin manydifferentways' We couldseeit, quiteconcretely,asthe mapof the moderncity (greenfor recreational areas,yellowfor housing,redfor industrialareas,andso on) - figure3.12 ls in fact a mapof a city,andlooksquitesimilar,if we ignorethewriting.we couldalsoseeit, more abstractly,asa mapof desirablehumanqualitiesor activities(greenfor contemplation, red for passion,etc.),and of the spacethey shouldoccupyin our lives.The point is, thepaintingrepresentsboththesethings,andmanymore.It isopento manyreadings,and that constitutesits powerto shapereality,a powerwhich,however,cannotbe unleashed
  • 105. I I t Conceptual representations 9 1 Q fig f.ff PurePainting(TheovanDoesburg,1920)(fromJaff6,1967) until the boxesare givenconcretereference,so that the schemacan be turnedinto a bIueprint. Lowerednaturalisticmodality,however,is not a definingcharacteristicof analytic visuals.Modalityformsa separate(interactional)systemwhichis presentin visuals simultaneouslywiththekindsof structurewearedescribinginthischapter.At mostwecan saythat,in specificsocialcontexts(cartography,educationat differentlevels,advertising fashion),thereis a tendencyfor certainmodalitychoicesto go togetherwith certain representationalchoices,certainkindsof processes.Thedefiningcharacteristicof the analyticalprocessisinfacta'default'one.It liesintheabsenceof vectorsandtheabsence of compositionalsymmetryand/ortreestructures,andalsoin theabsenceof thefeatures that markthe symbolicprocesseswe will describein the nextsection.Thereare more positiveand specificcharacteristics,but thesepertainto specifickindsof analytical process.Asa whole,theanalyticalprocessistheusual,the'unmarked',andthereforealso the mostelementaryoptionin the visualsystemof representation- a visual'this is'.
  • 106. 92 ConceptuaI representations WGww wrc ffiWffi iWuffiffiffiW ffiffiffiffiffi Wwffiffiffiffiffi}rcry]I 1 wffi 13r::irf;*y* ffi*u.,." W**.,,'",,.,",,o I'.'.**.*q4::.:'i.ni'i,"rnl.r1 ffi81*.o**o-*.*0"*,n.,''-,"' Q fig Z.tZ Mapof the centralbusinessdistrictof Melbourne(PaskandBryant,1982) Experimentalstudiesof the productionof drawingswouldseemto bearthis out: the principaltaskthat drawersmustmasteristhe representationof objectsin termsof their minimaldefiningcharacteristics,and the principalpurposefor whichnon-specialists actuallyuse drawingin everydaylife is the productionof pragmaticallymotivated 'descriptions',sketchesof localities,clothesand hairstyles,mechanicaldevices,and so on,aswellasthe productionof doodles,whichoftenare alsoanalytical(vanSommers, I9B4 234ff.). 1 Unstructuredanalyticalprocesses Someanalyticalprocessesareunstructured;thatis,theyshowusthe PossessiveAttributes of the Carrier,but notthe Carrieritself,theyshowusthe parts,but notthewaythe parts fit togetherto makeupa whole.Thepagereproducedinfigure3.).3,Iorinstance,isa kind of unstructuredmap,accuratein scale,but withoutanyvisualindicationof the location of the PossessiveAttributesrelativeto eachother.We seethe partsof the garment,but notthewaytheyareto beassembled- althoughverballabels('coatfront','bikinitop back',etc.)providean indication.Thepagedoesalsoincludea photographof thefinished garment.This photograph,however,would haveto be seenas a separateanalytical structure,with highernaturalisticmodalitythanthe patternitself.Themagazineusesthe m t:
  • 107. Conceptualrepresentations. 93 - " l ' ' ' ' - ' - ' . " ' - t " . " : " ' . : 1 : " : . - . ; . I : : : ' : : _ . Q fiq f.ff Escape(Austnlian Women'sWeekl!,December1987) samestrategyasthesocialstudiestextbool<:onepictureto enticeyou/andanotherto give youa m0reobjectiveanalysisof theCarrier. EspeciallywhentheCarrierisabstract,theremaynotbea singleprinciplefortheway inwhichPossessiveAttributesshouldbeassembled.TheCarriercannotbevisualizedinan assembledstate,andanyarrangementof the PossessiveAttributesisthereforepossible:an unstructuredanalyticalprocessis likea moreor lessunorderedlist.Thiscanbeseen,for instance,inadvertisementswhichdisplayallthepartsthatmakeuptheengineof acar,or
  • 108. Conceptual representations the full rangeof productsmanufacturedby an advertiser,in orderto impressthe viewer withtheirsheerabundance. 2 Temporalanalyticalprocesses Theexampleswehavediscussedsofar representthethingstheyrepresentasobjects.They focusonspatiality.Butthereisalsothecategoryof thetimeline,a processwhichseemsto occupyanintermediatepositionbetweenthenarrativeandtheanalytical.Timelinesinvolve the temporaldimension,and this suggestsnarrative.Yet theselinesare not vectorial and,ratherthanrepresentinghistoryasa gradualunfoldingof events,theyanalyseit into successivestageswithfixedandstablecharacteristics,stageswhichcanthenbetreatedas thoughtheywerethings.Thetypicalevolutiontimeline,showinga seriesof figuresstarting with an ape0n the left andendingwith homosapienson the right,wouldbeanalytical in this sense,'saying':'the evolutionof humankindconsistsof the "Ape" stage,the "Apeman"stage,the "Australopithecus"stage(etc.)',with 'theevolutionof humankind' asCarrier,andthestagesas PossessiveAttributes. Theessentialcharacteristicof temporalanalyticalprocessesisthattheyarerealizedby timelines:the participants(sometimeswholestructures,'scenes')are arrangedon an actualor imaginaryline,usuallyhorizontal,sometimesvertical.Thetimelinemaybetopo- graphical,drawnto scale,or topological,assemblingtheparticipantsintherightsequence, butnotdrawingthetimeintervals'toscale'. Timelinesneednotbestraight,andtheymayinvolveall kindsof geometricalsymbolism. Figure3.I4, takenfroma Swedishjuniorhigh-schoolhistorytextbook,is a particularly interestingexample.Heretimedoesnot progresslinearly,but takesturns,goesdownhill, movestowardsthe viewer,andchangescolou[from a crystallineblueto a drabbrown (thecolourcodeemployedbytheseriesof fourbooksof whichthebookcontainingthis illustrationis part,hasblueasthe colourof historyandbrownasthe colourof social studies).Theresultis a ratherethnocentricandpatriarchalstorywhichcouldbesaidto reveal,behindthesurfaceof Sweden'smodernandegalitarianwelfarestate,a deeplong- ingfortheprimitiveuncertainlifeofthenomadichunter.When'Man'firstappearedonthe horizonof time,sothisstorygoes,hewasa hunteranda makeroftools.Hethenacquireda wife,but she,the womanof 'homoerectus',wasnot yet'erectus'herselfand,whereher manfixedhisgazeuponthefuture,shelookedbacktowardsthepast.Then'Man'invented fire- a maleachievement,for womanisnotshownto bepartof it,notevenasanadmiring onlooker.Next,he beganto coverhisbodyandbegota child.But at this point,the point wherehistorytakesa backwardturn,a curiousinversionoccurred.It was now he who crouchedandshewhowas'erectLrs',hewholookedbackwardsandshewhofixedhergaze upona futurewhich,however,wasno longerthere,astimehadalreadyturnedbackwards. Butallwasnotlost.Theunionofthetwoprovideda very'erect',verymaleandveryNordic homosapiensasthe outcomeof evolution,as'our'ancestor,hereplacedin the centreof thecompositionandcloseto theviewer.ThisstoryaboutSwedishidentityistoldbyvisual meansonly,andit illustratesa textwhichistold,notasSwedishhistory,butasa historyof theworld.Thetwistingandturningof thetimelineinfigure3.14does,of course,giveit a
  • 109. ConceptuaI representations - *ffi ffiffiffi @ rig f.f+ Evolution(ohman,1989) l * .H - & s s a * l f f i q q r @ iisq!{i!r4ejd;ii*ii&etd6 *{"! * a iR s- L*i*lw*:&:*- a ' d r d # W # q @ ' S eFe4;" dts 4'. 'h dlb "i . : q 3 ? @ 4 a q " e . ! ! @ ? : b F r Wqiftir*r4!!&ltsi!& !M * * #4+'*F; q *aw rs4 jk;s.4* " #& $d{.1 sS ffi;6htu"tutrF, {i;!i!{saii,iid.istd;.'l ' ,r: i r' _,.. r" .: . :r' dynamicqualityandremindsof thehelixinfigure2.24.Thecategoriesof visualgrammar donothaveclear-cutedges,andspecificrepresentationscanmergetwo or morestructures - for instance,thenarrativeandtheanalytical. 3 Exhaustiveand inclusiveanalyticalprocesses (Spatial)structuredanalyticalprocessescanbeexhaustive,thatis,theycanexhaustively representthe PossessiveAttributesof a Carrier,sothat all of the Carrierisaccountedfor, all of itsspacetakenupby PossessiveAttributes.Toput it inanotherway,inanexhaustive analyticalstructurethereis assembly:the PossessiveAttributesare joinedtogetherto makeup a complexshape.Structuredanalyticalprocessescanalsobe inclusive,that is, theycanshowus onlysomeof the PossessiveAttributesof a Carrier,leavingmuchof the Carrierunaccountedfor,as blankspacenot takenup by PossessiveAttributes.The circlesontheleftandrightof Schramm'sthirdcommunicationmodel(figure2.22)form exhaustiveanalyticalstructures,embeddedinthe largernarrativestructure:the'sender' andthe'Receiver'arehereanalysedasbeingmadeupout of thesamethreecomponents ('Encoder','lnterpreter'and'Decoder'),assembledindifferentways,andeverypartofthe spaceof thecirclesiscoveredby PossessiveAttributes.Thepointisnot,of course,thatthe analysisis in factexhaustive.'Senders'and'Receivers,maywellhaveothercomponents besidethesethree.Thepointis that in the analysisthe world is treatedas thoughit is exhaustivelyrepresented,as thoughthe Carriershavethesemajorcomponentsand no
  • 110. 96 . Conceptualrepresentations others.Justaswe haveto assumethat a mapshowingthe statesof Australiashowsus a//thestatesof Australia,anda pictureof theoutfitof anAntarcticexplorera//theparts of this outfit,so we haveto assumethat this diagramshowsus all the componentsof the'Sender'andthe'Receiver'.0fcourse,'encoding','interpreting'and'decoding'migh be seenas activitiesratherthan as components.However,Schrammhas represented them hereasthoughtheywerepartsof a machine,asthough'Senders'and'Receivers' havetheseelements,or consistoI theseelements.Assooftenindiagrams,doingshavebeen turnedintothings,spatializedandobjectified. In inclusiveanalyticalstructures,the PossessiveAttributesdo not exhaustivelydivide upthespaceof theCarrier.TheyarecontainedwithintheCarrier,andsotakeuppartof its space,but not all - otherpartsare left blank,unanalysed.Thisstructurealsoentailsthe possibilityof partial inclusionand exclusionIn figure3.15, A and B are Possessive Attributesof the Carrierformedbythe largestcircle,C is a PartialPossessiveAttribute (in part belongingto the Carrier,in part outsideit), and D is excluded,hencenot a PossessiveAttribute.In otherwords,whereexhaustivestructuresare formedby the weldingtogetherof PossessiveAttributes,inclusivestructuresare formedby the full or partial overlappingof the participants.And the structureis recursive'.a Possessive AttributecanbecometheCarrierof otherPossessiveAttributes,as in the casewith B in fi9ure3.15:it isa PossessiveAttributewith respectto the largercircle,buta Carrierwith respectto E.Wecanseethisstructureat workin RileyandRiley'scommunicationmodel (figure2.5).It doesnottell usthat 'over-allsocialsystems'exhaustivelyconsistof two 'largersocialstructures'and 'largersocialstructures'of two and not morethan two 'primarygroups',butthat'over-allsocialsystems'contain(besidethemanyotherthings theymayalsocontain)at leasttwo'larger socialstructures',andthat'largersocial structures'containat leasttwo'primarygroups'.Individualssuchas'C'and'R'are partial PossessiveAttributesof the CarrierlAttribute'larger social structure'and fully PossessiveAttributesof the Carrier'over-allsocialsystem'.Togivea glossonthis structure,accordingto Rileyand Riley,individualscannotescapethe'over-allsocial @ fi9 f.fS Inclusiveanalyticalstructures
  • 111. Conceptualrepresentations' 97 system//buttheycan/at leastin part,freethemselvesfromtheconstraintsof'largersocial structures'. Manymapshavea similarstructure.A mapof a state0r nationwhichshowscitiesand towns,for instance,is not read as meaningthat everycity and everytown has been included,orthatthestateornationcontainsnoPossessiveAttributesotherthancitiesand towns.It is readasmeaningthatthesearesome(themajor)citiesandtownsof thatstate or nationor,ralher,thatthesearethecitiesandtownsof interestandrelevanceto you,the readerof thisparticularmap.Inclusivestructuresarehereembeddedwithinexhaustive structures:the mapas a wholemay be primarilyconcernedwith the boundariesthat exhaustivelydividethe Carrierinto PossessiveAttributes(Europeintonations,Australia into states,etc.),but thesePossessiveAttributesthemselvesare Carriersin embedded inclusivestructureswith PossessiveAttributessuchasmajorcities,rivers,lakes,etC' 4 Conjoinedandcompoundedexhaustivestructures In conjoinedexhaustivestructures,PossessiveAttributesare eitherconnected,by a line lackinga featureof directionality,or disengaged,bya layoutof the PossessiveAttributes whichseparatesthem,yet clearlyshowshowtheyfit together.Thelatteris the case,for instance,of the 'pie chart' in figure3.16,wherethe disengagementof the Possessive Attributesacquiressymbolicvalue,showing,literallyandfiguratively,the riftsthat divide thenation.Theformeristhecaseinthecommunicationmodelontopof fi9ure2.6,where all theparticipantsareconnectedby lines.Suchconnectinglinesmaybepurelyabstract connectors,asinfigure2.6,or havea featureof conductivity: *&{x*v* xs",4 &*hi*q*s* *?YE ryffi* Srs$lti*eal *{*% tl*ar*igncd1S% Q fig f,fe Australiarthe segmenls(Builetin,lo January1989)
  • 112. 98 Conceptualrepresentations Conductorsareconnectorsthat alsorepresenta PossessiveAttribute,a physicalentity - for instance,wiring,a pipeline,a road,a railwaytrack- andtheymayalsobeabstract,as in somecommunicationmodels.Realizedbya doubleline,conductorsindicatea potential for dynamicinteractionbetweenthe PossessiveAttributesthey connect.As such,they are both participantand process,connectedelementand connector,compoundedand conjoining. In compoundedstructuresthe PossessiveAttributesareweldedtogether,whileat the sametimeretainingtheirdistinctidentities.Thisisasmuchthecasein simplepiecharts as in a technicaldrawing,for instance,the drawingof a machinefor crushingore,in figure3.18. 5 Topographicalandtopologicalprocesses Whenanalyticalstructuresare topographicaltheyare read as accuratelyrepresenting the physicalspatialrelationsandthe relativelocationof the PossessiveAttributes.AII of Q fig f.fZ Connectorsandconductors ,:#qr ; i:,: lr:!1i: 1:i;gli O fiS f.fg Fourth.stagecrusher(Merritt,1984)
  • 113. ConceptualrePresentations' 99 thetypesof structuredanalyticalprocesseswehavediscussedsofar maybetopographical: we readthe pictureof the ore crusherin figure3.18 as accuratelyscalingdownthe dimensionsandrelativelocationof thepartsof themachine,justaswereadtopographical mapsasaccuratelyscalingdown,for example,the dimensionsof a lake,andits distance from other PossessiveAttributes(mountains,rivers)and from the boundariesof the Carrier. Whenanalyticalstructuresaretopological,theyarereadasaccuratelyrepresentingthe 'logical'relationsbetweenparticipants,theway in whichparticipantsareconnectedto eachother(whethertheyhavecommonboundaries,or are partiallyor whollyincluded in eachother,in whichsequencetheyareconnected,etc.),but nottheactualphysicalsize of theparticipantsor theirdistancefromeachotherorlinthecaseof inclusivestructures, from the boundariesof the Carrier.An electricalcircuit diagram,for instance,is topological(figure3.19).It doesnot signify,say,that lampsa andb are'above'and'at the rightof,the battery,andit doesnot accuratelyscaledownthe distancebetweenthe batteryanda lampa, or betweenthe two lamps.But it doessignifythat theyare con- nectedinthisparticularsequence,justasevolutiontimelinessignifythat'Ape','Apeman', 'Australopithecus',etc.madetheirfirstappearancein historyintheordershown' Maps,too,mayeitherbetopographicalor topological,as,for instance,in mapsof urban transportsystems.The digitalnetworkswe discussedin the previoussectionare als0 topologicaldiagrams,abstractmapsrastheyarebasedonadjacency,co-location'In other words,abstractdiagrams,too, can be eithertopologicalor topographical.Schramm's communicationmodel(figure2.6),forinstance,istopological.ltdoesnottellusthatthe ,Source,isontheleft,or to thewest,of the'Encoder';it tellsusonlythattheparticipants areconnectedinthissequence.Thisisnotto saythattheplacementof the'Source'tothe left of the 'Encoder'is not significant,onlythat its significancederives,not from the ideational,but from the textualstructureof the diagram(seechapter6). Halliday's diagramof the'nature of linguisticstructuresand their relationsto otherfieldsof scholarship'|1-978:10),onthe otherhand,is an intricatepieceof abstracttopography (seefigure3.20).Thediagramusesdistanceto indicatehowclosethe variouskinds of languagestudyarefrom what ls hereseenasthe centralandmostimportantform of languagestudy,thestudyof'languageassystem'.It usesdistancein a figurative,yetfinely calibrated,way:'phonetics',forexample,iscloserto 'languageassystem/thanthestudyof dialectsandregisters.Andit usesrelativesizeinthesameway:thestudyof languagetakes upa muchgreaterareathanall theother'fieldsof scholarship'together.As in medieval O fig:.rf Electriccircuitdiagram(Hi11,1980)
  • 114. i 0 0 ConceptuaI representations logicand mathematics communications engineering , ; ) i (psycho linguist language language typology universals I I t lt t t l- - l - - - - - - - - - 1 - ' V ? culture human biology language pathology aphasia etcinternalization productionand O fiS f.ZO Theplaceof linguisticsonthe mapot knowledge(Halliday,lgTg) maps/the cartographer/s'hometown'is both exaggeratedin sizeand represented internallymoreaccuratelyandwithmoredetailthanthesurroundinq'countrvside,. 6 Dimensionaland quantitativetopography Liketopographicalvisuals,chartsaredrawnto scale.Thescale,however,is based,noton thephysicaldimensionsof theparticipants,butonthequantityor frequencyof aggregates of participantsthataretakento beidentical.piecharts,for instance(seee.g.figure3.ro, dividea Carrier(thepopulationof Australia)intocomponents,PossessiveAttributesthat are in fact aggregatesof participantsanalysedasbeingthesamein somerespect,andit tellsus,notthat'Achievers'arefoundnextto,andto thesoutheastof,,Adapters,,butthat thenumberof 'Achievers'standsto thenumberof 'Adapters'asthesizeof the possessive Attribute labelled'Achievers'standsto the size of the PossessiveAttribute labelled 'Adapters'. Quantityistranslatedintorelativesize- althoughit is,of course,alsopossible to representquantitywith quantity,asin figure3.21,a nowperhapsratherold-fashioned languagevarieties: graphlc dialect torn: grammar and
  • 115. Conceptual representat ions 1 0 1 Q figf.zf womenatwork(ModleyandLowenstein,1952) examplewhich usesOtto von Neurath's'picture language'Isotype(seee.g. Lupton, 1989). Pie chartsand bar charts(seefigure3.23) are one-dimensionalitheyshowus one (abstract)Carrier(inthecaseof figure3.16,theconcreteCarrier'Australia' isa metonym for theabstractCarrier'the populationof Australia')andits PossessiveAttributes.Their structureis quitesimilarto that of stackdiagrams(seefigure3.5).Theyareexhaustive, compoundedanalyticalstructures.The differenceliesonly in their peculiarkind of abstractionandpeculiarkindof topographicalaccuracy,bothof whicharequantitative. 7 Spatio-temporalanalyticalstructures Two-dimensionalchartscreatea conjunctionbetweena sef of such(exhaustive,com- pounded,quantitativelyabstractandtopographical)analyticalstructuresanda timeline, for thesal<eof comparativeanalysisalonganorderedtimescale.In thecaseof linegraphs WOMENAT WORK ttti titti AAAAAAAA AAAAAAAA AAAAAAAAAAA AIAAATIITIIAAA AAAAAAAAAAAAAAAAAAA 1890 l900 r 9 1 0 1920 I930 r940 l950
  • 116. r02 ConceptuaI representations this can resultin the quasi-vectorial,quasi-narrativestructuretypifiedby temperature charts,profitcharts,companygrowthcharts,etc.(seefigure3.22).In otherwords,the substitutionof a linefor discreteentitiescreatessomethinglikea dynamicprocess(with meaningssuchas'change','grow','decrease',etc.),andbackgroundsor evenerasesthe analyticalstructuresthatunderliethegraph,sothatthegraphnolongersuggeststhat'ln 19BBthe Carrier("incidenceof AIDS") consistedof 500,000PossessiveAttributes ("AI DScases")andin 1990the Carrierconsistedof 800,000PossessiveAttributesand . . . (etc.)',butthat'TheActor ("theepidemic")acts("spreadsrapidly,'),.Tocomplicate matters/one-dimensionalstructurescan be representedas though they were two- dimensional,andsoproducea senseof progressor decline,dependingontheorderinwhich the PossessiveAttributesare plottedalongthe horizontalaxis- an ordernot givenby relationsof conjoinednessor compoundness,or of inclusionor exclusion,but solelyby quantity.Thebarchartin figure3.23caneasilybe(mis)takenassuggesting,notthatmore peoplelivein ffatsthan in largehouses,but that people,astime progresses/increasingly liveinffats. Somethingquitesimilarcanhappenin language.Hodgeandl(ress(1978,1993)drew attentionto the effectsof the transformationof nominalization,whichturns clauses (reportsof events)suchas peoplelearnedinto nominals(namesof objects)suchas people'slearningwhichmaythenbecomeactorsin newevents(thenewlearningspread. Martinet al. describedthisprocessin historytextbooks.Theycommentedthatsuchtrans- formationsfavourthe'anthropomorphicmetaphorof birth,growthanddeath,(1988: 157)andallowhistoriansto analysehistoryin successiveperiodsinwhichsimilarthings go on.In otherwords,despitetheir apparentstatusas active,dynamicprocesses,these structuresstillserveto establishstable,conceptualorders. &iilstflis th*"'1rutrl}rlln $a rliJl# Wrrrtd "qtxl$ l$ rig rZz c lobalepidemic(Bafletin,26 Junetgg})
  • 117. ConceptuaI representations 1 0 3 Q figZZ Typesof dwelling(BindonandWilliams,1988) is possible, but nol 6 risl.z+ Dynamizationof analyticalprocesses This dynamizationof two-dimensionalanalyticalprocessescannotoccurwhenthe arrangementisvertical(seefigure3.24).ThePossessiveAttributesthenremaindiscrete, andthe graphsasa wholesuggesta stableorderratherthana dynamicprocess.Thisis borneout,forinstance,bythefactthatonecannot'fillin'thespaceoftheCarrier. It isour impressionthattherehasbeen,in Westerncultures,a shiftfroma focusonthe verticalto a focusonthehorizontal,a changefroma concernwith'Whatisthestateof affairs?'and'Wherearewe?'to'Wherearewegoing?'and'Haveweprogressedor arewe in decline?'Thisisborneoutalsobythefactthatscriptswhichweretraditionallywritten top-downarenowbeginningto bewrittenfrom leftto right. In figure3.25,wesummarizethedistinctionsintroducedinthissection.Notethatthe superscript'l'means'if'andthesuperscript'T'means'then'.In otherwords,if thereisa topographicaltimeline,thenthetopographymustbequantitative. 34 f2 3 A 28 ct 26 2 t t - 2 2 = ' "o 1 8 u ! 4 = ^ 6 2 0 Vi l l a unitshouses Smal houses Semi- detached RoWterface Leavrn9a space betweeneachbaf is optional
  • 118. I 0 4 ConceptuaI representations Q fiq f.ZS Analyticimagestructures ,-***r*, -{ Anarvticar | | ""tt""'-Lrnrr.u.,u."o L REALIZATIONS Unstructured anaIyt icaI process Temporalanalyticalprocess Exhaustiveanalyticalprocess Dimensional topographical accuracy QuantitativetopographicaI accuracy Topologicalaccuracy An unorderedsetof participants('Possessive Attributes')is interpretedasthesetof parts of a wholewhichitselfisnotrepresented. A setof participants('PossessiveAttributes,) isorderedlinearlyona (horizontalor vertical)timelineandinterpretedasthesetof successivestagesof a temporallyunfolding process. A participant('Carrier')isdepictedasmade upof a numberof parts('Possessive Attributes')andthestructureisinterpretedas showingallthepartsfromwhichthewholeis madeup. TheCarrierandthe PossessiveAttributesof an analyticalprocessaredrawnto scale. Thesizeof the PossessiveAttributesinan analyticalprocessaccuratelyrepresentsthe numberor someotherquantitativeattribute of the PossessiveAttributes. TheCarrierandthe PossessiveAttributesof ananalyticalprocessarenotdrawnto scale, butthewaytheyareinterconnectedisdrawn accurately. Theparticipantsin ananalyticalprocessmay beconcrete. Abstraction
  • 119. ConceptuaI representations t 0 5 SYMBOLICPROCESSES Symbolicprocessesareaboutwhata participantmeansor rs.Eithertherearetwo partici- pants-the participantwhosemeaningor identityisestablishedintherelation,theCarrier, andtheparticipantwhichrepresentsthemeaningor identityitself, theSymbolicAttribute - or thereis onlyoneparticipant,the Carrier,andin that casethe symbolicmeaningis establishedin anotherway,tobedescribedbelow.Theformertypeof processwewill call SymbolicAttributive; the latter,SymbolicSuggestive. Art historianshavechartedtheformalpictorialcharacteristicswhichcanrealizethe SymbolicAttributerelation(seeespeciallyHermeren,1969).Symbolicattributesare objectswithoneor moreof thefollowingcharacteristics. (1) Theyare madesalientin the representationin oneway or another;for instance,by beingplacedintheforeground,throughexaggeratedsize,throughbeingespeciallywelI lit,throughbeingrepresentedinespeciallyfinedetailor sharpfocus,or throughtheir conspicuouscolouror tone. (2) Theyare pointedat by meansof a gesturewhichcannotbeinterpretedasan action otherthantheactionof 'pointingoutthe symbolicattributeto theviewer'- herewe canincludealsothearrowswhichcanconnectvisualrealizationsof participantswith verbalrealizationsof the sameparticipant,or viceversa/as in figure2.4,for these alsoestablisha relationof identitythrough'pointing'. (3) Theylookoutof placeinthewhole,insomeway. (4) Theyareconventionallyassociatedwithsymbolicvalues. Suchconventionalsymbolswereverycommonin the MiddleAgesandthe Renaissance: see,for instance,figure3.27,wherethe apple,lookingsomewhatout of placein St Jerome'sstudy,symbolizesthe Fall,Temptation,0riginalSin, and so bringsthese immediatelyto mindfortheviewerofthepainting.Totakea modernexample,thescientist depictedin figure3.28 is clearlynot doinganythingwith,or to, the fungiwhichare displayedintheforegroundandof whichheisholdingonein hisrighthand.Hispositionin relationto thefungiseemscontrivedandposed.ThefungifunctionhereastheAttributes thatestablishhisidentityasanexpertonfungi. Humanparticipantsin SymbolicAttributiveprocessesusuallyposefor the viewer, ratherthan beingshownas involvedin someaction.Thisdoesnot meanthat theyare necessarilyportrayedfront-onandat eyelevel,or thattheynecessarilylookat theviewer, Symbolic 5tructures @ figl.ZO Typesof symbolicimagestructure
  • 120. 106 ConceptuaI representat ions Q ligl.Zt St Jeromein his Stud! (JanvanEyck,1434)(from Hermeren,1969) eventhoughall of thesewill oftenbethecase.It meansthattheytakeupa posturewhich cannotbeinterpretedasnarrative:theyjustsit or standthere,for no reasonotherthanto displaythemselvesto theviewer. SymbolicSuggestiveProcesseshaveonlyoneparticipant,the Carrier.Theycannotbe interpretedasanalytical,becausein this kindof imagedetailtendsto bede-emphasized infavourof whatcouldbecalled'mood'or 'atmosphere'.Thiscanberealizedina number of ways:thecoloursmayall blendtogether,intoa hazyblue,for instance,or a softgolden glow;thefocusmaybesoft;or the lightingmaybeextreme,renderingtheparticipantsas outlinesor silhouettes.It isthiswhichlendsSymbolicSuggestivepicturestheirgenericity, their qualityof depictingnot a specificmomentbut a generalizedessence.rhe wayin whichtheblurringof detailoccursthenlendssymbolicvaluetotheCarrier-asoftgolden glow,for instance,wouldconferonthe Carrierall thevaluesassociatedwith softness,and with gold,as in the Bushellsadvertisementreproducedin plate2. As a resultSymbolic Suggestiveprocessesrepresentmeaningand identityas comingfrom within,as deriving fromqualitiesof theCarrierthemselves,whereasSymbolicAttributiveprocessesrepresent meaningandidentityasbeingconferredto theCarrier. Expressionistlandscapes(e.9.Nolde'sTropicalSun,seeplate1) alsodiminishthe detailof therepresentation,infavourof overallcoloureffectsevolvinga strongmood,and imbuingtheCarrier('autumnevening')withsymbolicmeanings.
  • 121. Conceptualrepresentations ' I07 {Fsx{e"';-a$*le}dru &g .&eeg,g;,!Ets,.eiEre.B{i{'q" b**+'u; . tTi"'*li-*X ffi O f in l.zg Funwith fungi (s/dney Moming Henld,lS June1992) EMBEDDING In language,senten6escanbesimple(consistingof onlyoneclause/process)or complex (containingseveralclauses,eachwiththeirownprocess/coordinatedwith or subordinated to eachother).Pictures,too,canbesimpleor complex.We havealreadydiscussedhow in figure2.1 the relationbetweentheAboriginesandthefire constitutesa second,minor, transactionalprocess/subordinatedto the 'major' process,which is constitutedby the relationbetweenthe Britishandthe Aborigines;and howparticipantssuchas the Britishor the landscapecanthemselvesbe readas analyticalstructures,with a Carrier (e.g.the landscape)and PossessiveAttributes(e.g.rocksand trees).which of these structuresaremaJorandwhichareminoris,invisuals,determinedbytherelativesizeand conspicuousnessof theelements.Thecoverof thebookfromwhichwetook'TheBritish usedguns'providesanexampleof thiskindof complexity(fiTure3.29).Wecanrecognlze fourdifferentprocessesinthispicture: (1) A classificationalprocessin whichthefivechildrenareSubordinatesof theclassof 'youngAustralians"in what we havecalleda covert Taxonomy.Theyare shown againsta neutralbackgroundandarrangedina symmetricalfashion,ina circlewhich
  • 122. 1 0 8 ConceptuaI representations Q fig f.Zf OurSociet! antt Others(0akteyet at.,Lg}5) is closed(andreiterated)by the globe.(we will discussthe meaningof circularity morefullyinchapter6.) A numberof analyticalprocessesinwhicheachchildisCarrierinrelationto a number of (prototypical,'essential')PossessiveAttributes(skincolour,colourand kindof hair,colourof eyes,itemsof clothing)- attributeswhichcreatevisualconceptsof theirdifferentethnicities. A symbolicattributiveprocessinwhichtheglobeisa conventionalsymbolwithstrong associations.It isplacedintheforegroundof thepicture,andtwoof thechildrentouch it with a gesturewhichcannoteasilybeinterpretedasan actionotherthanonethat drawstheviewer'sattentionto theglobe.Thechildrenarethusshownto bepartof the world,a microcosmof theworldinAustralia. Transactionalprocessesinwhichthreeofthechildrenhaveputtheirarmsroundother children.Theboyontheleftwouldseemto bethemajorActorhere,whilethetwogirls ( 2 ) 3 ) G )
  • 123. ConceptuaI representations r09 are Goalwith respectto hisaction.Thetwo girls,who havetheir armsroundeach other,ontheotherhand,relateina morereciprocalway.Theyarewhatwehavecalled Interactors. Fromthis multipleanalysiswe canseethat the picture(evenwhenwe disregardthe interactiveandcompositionalstructureswhichare alsopresent,andwhichwill bedis- cussedinthenextthreechapters)formsa powerful,multidimensionalstructure. Turningto the relationbetweentheseprocesses,we mightsaythat becauseof itssheer sizeandconspicuousnessthecirculararrangementof thechildrenis dominant,whilethe analyticalprocessesareembeddedin it: fivechildrenareco-classified,andeachof them can be further analysedin termsof PossessiveAttributes.In otherwords,the major messageisthatthesechildrenbelongto thesamecategory,despitethedifferenceingender andethnicity;the minormessageis that theyare,nevertheless,different.As far as the transactionalandsymbolicprocessesareconcerned,wewouldarguethatthewhitelineson theboy'stracksuitformstronganddominantvectors. Perhapsthe'transactional'processof whichthisboyisActorandthesymbolicprocess of whichheis Carrierweighasheavilyinthescaleastheclassificationalprocess/andalso formmajorprocesses.Thetransactionalprocessesof whichthegirlsintheforegroundare bothActorandGoal,andthesymbolicprocessof whichthegirlontherightistheCarrier, however,are considerablylessconspicuousandmightbe interpretedas minorprocesses. In otherwords,asActorsof thegestureof solidarity,andasCarriersof thesymbolicvalue of 'representingthe world',the 'ethnic'girlshave,in this representation,a muchless significantroleto playthanthewhiteAustralianboyin histracksuit. CONCEPTUALSTRUCTURESIN LANGUAGE Thereare somepointsof contactbetweenthe wayconceptualstructuresare realizedin languageandinimages.Thecomparisonwouldhaveto bemadewiththekindsof linguistic structuresHallidaycalls'relational'and'existential'processes(seeHalliday,1985: 1l-2ff.).Thesehaveat leastthisincommonwithconceptualimagesthattheyrepresentthe worldintermsof moreor lesspermanentstatesof affairsor generaltruths,ratherthanin termsof actionsor mentalprocesses.Hallidayrecognizestwomaincategoriesof relational process,theAttributiveandthe Identifyingprocess.Themeaningof anAttributiveprocess clausecan be schematicallydescribedas 'a is an attributeof x'. The attribute'a' is thensimplycalledAttribute,andthe participantwhoseattributeit is,isthe Carrier-we haveborrowedthesetermsin our analysisof images.In Somepeopleare racist,for instance,somepeopleis Carrier,areis the Relational(Attributive)process/andracistis the Attribute.Attributiveprocessescanbe Intensive- that is,theycan beaboutwhat a Carrieris,as in the examplejust given;theycanbe Circumstantial-that is,theycanbe about'where'or'when'or'whatwith'a Carrieris(e.9.in Theirhomewasin HoChiMinh City, theirhomeis Carrier,razasanAttributive[Circumstantial]Relationalprocess,andin Ho Chi Minh Cityan Attribute);andtheycanbe Possessive- that is,theycan beabout
  • 124. 1 1 0 ConceptuaI representations whata Carrierhas(e.9.inI didn'towna thingin theworld,I isCarrier,owntheAttributive IPossessive]Relationalprocess,anda thingin theworld,theAttribute). Themeaningof anldentifyingRelationalprocessclausecanbeschematicallydescribed as'a istheidentityof x'. Theidentity'a'isthentermedtheValueandtheparticipant'x' whoseidentityit is, is calledthe Token.In RevPeter Nyanginguis the Minister of the Uniting Churchat Ernabella, Rev Peter Nyanginguis Token,rs is the Identifying Relationalprocess/andthe Ministerof the UnitingChurchat Ernabellaisthe Value.The Value,then,tendsto bea'status' or 'function'or 'meaning,whichservesto identifythe Token,andthe Tokenis the nameor somedescriptionof the holderof the status,or of theoccupantof thefunction,or of thesignwhichhasthe meaning.Picturecaptionsare oftenidentifyingclauses,with a referenceto the pictureasTokenandthe meaningof the Pictureas Value(for instance,Thisis oysterfarmingin PalmIslana.ldentifyingand Attributiveclausescanbedistinguishedfromeachotherbymeansof thereversibilitytest: in Identifyingclausesthe orderof the participantscanbereversedG.g. TheMinisterof the UnitingChurchat Ernabellais RevPeterNyanginguisjust asacceptableas RevPeter Nyanginguis the Minister of the Uniting Churchat Ernabelld, whereasin Attributive clausesthis is not the case(Racistare somepeopleis muchmoreunusualthan Some peopleareracis). Existentialclauses,finally,simplystatethat'somethingexists,.Theyhaveonlyone participant,the Existent- that is,the participantwhoseexistencethe structureaffirms. Existentsmayeitherbe EventsG.g.in It's terribleweatheAor EntitiesG.g.in Therewas a public librarfl. The presenceof a 'dummysubject'(thereor referentlessi/) is the principalidentifyingcharacteristicof existentialclauses(seeHalliday,1985:130). VisualClassificationalandAnalyticalstructurescouldthereforebesaidto beakinto, respectively,IntensiveandPossessiveAttributiveclauses,andSymbolicAttributivestruc- turescouldbeseenasakinto Identifyingclauses,andthereis perhapsalsosomeaffinity betweenSymbolicSuggestivestructuresand Existentialclauses.But the differencesare greaterthanthe similaritiesand,especiallyin the areaof ClassificationalandAnalytical images,thevisualsemiotichasa rangeof structuraldeviceswhichhavenoequivalentin language:thedifferencebetweenvisualconceptualizationandlinguisticconceptualization isevidentlyquitelarge.All the moreimportantto havea vocabularyfor expressingwhat canbedoneandisdonewith eachin concretetextsthatcombinethetwo semiotics,texts suchas 'The0verland'(figure3.30),a schoolprojectby an eight-year-oldboy.Looking first at thewords,the structureof theverbaltextonthe left pageis PossessiveAttribute throughout.Thereis a Carrier('TheOverland'),andthereare six PossessiveAttributes ('fivefirst classcarriages','threesecondclasscarriages','twodiningcar,s,,,twoengines, and'oneclubcar','onemotorrailcart').Theprocessitselfiselided. The picture,too, is concernedwith the relationof partsto a whole.It hasall the hallmarksof an exhaustive,structuredanalyticalprocess:the pictureis front-onandeye level,thereis no Setting,andthe differentPossessiveAttributes(signs,windows,wheels, suspensionandrails)are shownclearly,but withoutunnecessarydetail:it is relatively 'abstract'.Therelationbetweenpictureandtextisnotoneof illustration.Theoicturedoes not duplicatethe text,it doesnot representvisuallywhat hasalreadybeenrepresented
  • 125. ConceptuaI representations 1 1 1 Q rigs.lo'Theovertand' linguistically.Nor is therea relationof 'anchorage'(Barthes,1977) in whichthe text elaboratesthe informationgivenin the picturewithoutprovidingnewinformation.It is true that bothtext and pictureare aboutpart-wholerelationsbut this doesnot mean theyduplicateeachother,becauseinthetexttheCarrieristhetrainasa whole,whereasin thepicturetheCarrierisoneo/thecarriagesofthetrain.Thechildhastakenatopic,the '0verlandExpress',andtreatedit verballyandvisuallyin sucha waythateachpartof the textsupplementstheotherpart. Letusanalysethesecondpage.Thewordsareasfollows: It goesfromADELAIDEto MelbournerailAustralianNationalroot. Herewe havefirst of all a non-transactionalaction,with an Actor ('it') andan Action ('goes'),as well as two Circumstancesof place('fromAdelaide,and'to Melbourne,), Therefollowsan identificationof the route,with the Tokenandthe Processelided('rail AustralianNationalroot'):the childobjectifiesthe action'going,,turnsit intoa thing,a 'route',by meansof an identifyingrelationalprocessin whichthe'root'is Value.Finally the routeis describedin moredetail,througha visualanalyticalprocessin whichthe relevantsectionof Australiais the carrier,andAdelaide,Melbourneandthe'route'the
  • 126. r72 ConceptuaI representat ions PossessiveAttributes,in a processwhich usesmanyof the structuraldevicesof the analyticalvisualto showpreciselyhowthe PossessiveAttributesarespatially related,how they'fit together'. 'My Adventure'(figure3.31)isanotherschoolproject,writtenbya childfromthesame schoolandyearasthe authorof'The 0verland'.Mostof the verbalprocessesaretrans- actionalandnon-transactionalactionsin whichthe narratoris Actor:hewalks,findsa cave/findssomenails,and so on. He is not describingsomethingbut tellinga story, narratinga particularevent: Not longagoI wentfor a journeydownto thebeach.I walkedall alongthebeach untilI founda cave.Thatcavewasbig.Thentherewassomewoodbut I couldn,t makea raftwithoutnails.JustthenI foundsomenailsinthecave.I saidto myself 'Somebodymusthavebeenhere'.So I gota pieceof woodandstartedmakingthe raft.I useda pieceof woodfor a. . . . Still,thereisalsoa conceptualelement.in thispartof thestorythewriterispreoccupied withthe materialsneededfor buildinga raft.Thereis a hiddenattributiveprocess,some- thinglike'thematerialsfor araft arewoodandnails',butit istransformedintothestory of thefindingof thesematerials(whichemergemysteriouslyina bigcave).As inthestory @ rigr.3l ,MyAdventure'
  • 127. ConceptuaI representations 1 r3 that opensthe bool<of Genesis,whichis alsoa narrativeconcernedwith conceptualizing the elementsthat makeup a whole,the world,the writer alternatesactionalandtrans- actionalprocesses('l foundsomenails')with moreconceptualexistentialprocesses('then therewassomewood'). Theverbaltextstopsabruptly,in the middleof the Circumstanceof Purpose,andit is continued,notonthefollowingpage,but bythepicture,whichshowswhatthesematerials wereusedfor:the partsof a raft.It is an UnstructuredAnalyticalprocess,with the sail, andthe raft itself (alreadycompletewith mastand rudder),as PossessiveAttributes. Theseare,in thechild'sconceptualization,the partsfrom whicha raft is made.However, thereisalsoa strongnarrativeelementinthispicture:it hasa Setting(theshoreline,the cave)and it doesnot usethe frontal,eye-levelperspectivetypicalof most analytical images(andalreadysuccessfullyusedby the authorof 'TheOverland')but a highand obliqueangle.In boththepictureandthewords,thewriterblendstheconceptualandthe narrativeor,rafher,narrativizesthe conceptual.In the verbaltext he doesthis through conjunctions('then','justthen',etc.),throughtheuseofthepasttenseandthroughtheuse of transactionaland non-transactionalAction processeswith the narratoras a (first- person)Actor.In the picturehe doesit throughthe useof a Settingand a narrative perspective.Finally,thewordsandthepictureagaincomplementeachother.Thepicture doesnotillustratethestorybutcontinuesit. It is clear,we hope,that childrenactivelyexperimentwith the representational resourcesof word and image,andwith the waysin whichtheycan be combined.Their drawingsarenotjustillustrationsof a verbaltext,notjust'creativeembellishment';they arepartof a'multimodally'conceivedtext,a semioticinterplayin whicheachmode,the verbalandthevisual,isgivena definedandequalroleto play.
  • 128. 4 R e p r e s e n t a t i o na n d i n t e r a c t i o n : d e s i g n i n gt h e p o s i t i o no f t h e v i e w e r In thepreviouschapterwediscussedvisualresourcesfor therepresentationof interactions andconceptualrelationsbetweenthe people,placesandthingsdepictedin images.But visualcommunicationalsohasresourcesfor constitutingandmaintaininganotherkindof interaction,theinteractionbetweentheproducerandtheviewerof theimage.Anotherway of sayingthisisthat images(andotherkindsof visual)involvetwo kindsof participants, representedparticipants(thepeople,theplacesandthingsdepictedin images)andinter- activeparticipanfs(the peoplewho communicatewith each other throughimages,the producersand viewersof images),andthree kindsof relations:(1) relationsbetween representedparticipants;(2) relationsbetweeninteractiveand representedparticipants (the interactiveparticipants'attitudestowardsthe representedparticipants);and (3) relationsbetweeninteractiveparticipants(thethingsinteractiveparticipantsdoto or for eachotherthroughimages). Interactiveparticipantsare thereforereal peoplewho produceand makesenseof imagesinthecontextof socialinstitutionswhich,to differentdegreesandindifferentways, regulatewhat maybe'said'with images,howit shouldbe said,and howit shouldbe interpreted.In somecasesthe interactionis directand immediate.Producerandviewer knoweachotherand are involvedin face-to-faceinteraction,as whenwe makephoto- graphsof eachotherto keepin walletsor pin on pinboards,or drawmapsto giveeach otherdirections,or diagramsto explainideasto eachother.But in manycasesthereis no immediateanddirectinvolvement.Theproduceris absentfor the viewer.andthe viewer is absentfor the producer.Thinkof photographsin magazines.Who isthe producer?The photographerwhotooktheshot?Theassistantwhoprocessedandprintedit? Theagency who selectedanddistributedit? Thepictureeditorwhochoseit? Thelayoutartist who croppedit anddetermineditssizeandpositiononthe page?Mostviewerswill notonly nevermeetall thesecontributorsto theproductionprocessfaceto face,butalsohaveonly ahazy,andperhapsdistortedandglamorized,ideaoftheproductionprocessesbehindthe image.All theyhaveis the pictureitself,as it appearsin the magazine.And producers, similarly,canneverreallyknowtheirvastandabsentaudiences,andmust,instead,create a mentalimageof 'the' viewersand'the' way viewersmakesenseof their pictures.In everydayface-to-facecommunicationit is easyenoughto distinguishinteractivepartici- pantsfromrepresentedparticipants:thereisalwaysanimage-produceranda viewer(who, dependingonthesituation,mayswaproleswiththeproducer,addto thescribbledffoorplan or diagram,for instance),andthentherearetherepresentedparticipants(for instance,the peopleonthequicksketchof thedinnertablearrangement,or the landmarksonthehand- drawnmap),andthesemay,of course/includetheproducerand/ortheviewerthemselves. Producerandviewerarephysicallypresent.Theparticipantstheyrepresentneednot be. But whenthereis a disjunctionbetweenthe contextof productionand the contextof reception,theproducerisnotphysicallypresent,andtheviewerisalonewiththeimageand
  • 129. Representation and interaction 1 1 5 cannotreciprocate- an illuminatingexceptionisthecaseof the'defacement'of billboard advertisements,whengraffiti artists'respond'tothe initial'turn' or statementof the image. Somethingsimilaroccursin writing.Writers,too,are not usuallyphysicallypresent whentheir wordsare read,and mustaddresstheir readersin the guiseof represented participants,evenwhentheywrite in the first person.Readers,too,are alonewith the writtenword,andcannotusuallybecomewritersin turn. Literarytheorists(e.9.Booth, 1961; Chatman,!978) haveaddressedthis problemby distinguishingbetween'real' and'implied'authors,andbetween'real'and'implied'readers.The'impliedauthor'isa disembodiedvoice,or even'a set of implicitnormsratherthan a speakeror a voice' (Rimmon-l(enan,t983:87):'he,or better,it hasnovoice,no directmeansof communi- cating,but instructsussilently,throughthedesignof thewhole,with alI thevoices,byall the meansit haschosento let us learn'(Chatman,I978:148). The'impliedreader', 'preferredreadingposition',etc.,similarly,is'animageof a certaincompetencebroughtto the text and a structuringof suchcompetencewithinthe text' (Rimmon-l(enan,I9B3'. 118):thetextselectsa'model reader'throughits'choiceof a specificlinguisticcode,a certainliterarystyle'andbypresupposing'aspecificencyclopediccompetence'onthepart of the reader(Eco,I97g:7). Thiswecanknow.0f thiswe haveevidenceinthetextitself. Realauthorsand real readerswe cannotultimatelyknow.Thisbracketingout of real authorsandrealreaderscarriestheriskof forgettingthattexts,literaryandartistictexts as muchas massmediatexts,are producedin the contextof realsocialinstitutions,in orderto playa veryrealrolein sociallife- in orderto do certainthingsto or for their readers,andin orderto communicateattitudestowardsaspectsof sociallifeandtowards peoplewhoparticipateinthem,whetherauthorsandreadersareconsciouslyawareof this or not.Producers,if theywantto seetheirwork disseminated,mustworkwithinmoreor lessrigidlydefinedconventions,andadhereto the moreor Iessrigidlydefinedvaluesand beliefsofthesocialinstitutionwithinwhichtheirworkisproducedandcirculated.Readers will at leastrecognizethesecommunicativeintentionsand thesevaluesand attitudes for what theyare,evenif theydo not ultimatelyacceptthem as their ownvaluesand beliefs.Theycan'recognizethe substanceof what is meantwhilerefusingthe speaker's interpretationsandassessments'(Scannell,1994:11). Howeverimportantand realthis disjunctionbetweenthe contextof productionand the contextof reception,the two do haveelementsin common:the imageitself,and a knowledgeof thecommunicativeresourcesthatallowitsarticulationandunderstanding, a knowledgeofthewaysocialinteractionsandsocialrelationscanbeencodedin images.It isoftensaidthattheknowledgeof theproducerandtheknowledgeof theviewerdifferina fundamentalrespect:theformerisactive,allowingthe'sending'aswellasthe'receiving' of 'messages';the latteris passive,allowingonlythe'receiving'of'messages'.Producers areableto'write'aswellas'read',viewersareableonlyto'read'.Uptoapointthisistrue, at leastin the sensethat the productionof imagesis still a specializedactivity,sothat producers'write' moreffuentlyandeloquently,andmorefrequently,thanviewers.Butwe hopeourattemptsto makethat knowledgeexplicitwill showthatthe interactivemeanings arevisuallyencodedin waysthat reston competenciessharedby producersandviewers.
  • 130. 1 1 6 Representation and interacti on Thearticulationandunderstandingof socialmeaningsin imagesderivesfromthevisual articulationof socialmeaningsin face-to-faceinteraction,the spatialpositionsallocated to differentkindsof socialactorsin interaction(whethertheyareseatedor standing,side by sideor facingeachotherfrontally,etc.).In this sensethe interactivedimensionof imagesisthe'writing'ofwhatisusuallycalled'non-verbalcommunication',a'language' sharedbyproducersandviewersalike. Thedisjunctionbetweenthecontextof productionandthecontextof receptionhasyet anothereffect:it causessocialrelationsto be representedrather thanenacted.Because the producersare absentfrom the placewherethe actualcommunicativetransactionis completed,fromthelocusof reception,theycannotsay'l' otherthanthrougha substitute 'l'. Evenwhenthe viewerreceivesan imageof the 'real author'or a contributorto the productionprocess- thepresenterin a televisionprogramme,thepainterina self-portrait, theownerof thecompany(or theworkerin thecenturies-olddistillery)in an advertise- ment- that imageis only an image,a doubleof the 'real author',a representation, detachedfromhisor heractualbody.Andthe'realauthors'mayalsospeakintheguiseof someoneelse,of a 'character',aswhen,insteadof theownerof a company,it is UncleSam, or a larger-than-lifewalkingandtalkingteddybear,whoaddressesusinanadvertisement. Thisdimensionof representationis anotheronewhichhasbeenstudiedextensivelyin literarytheory(e.9.Genette,7972). The relationbetweenproducerand viewer,too, is representedratherthan enacted.In face-to-facecommunicationwe must respondto a friendlysmilewitha friendlysmile,to anarrogantstarewitha deferentialloweringof the eyes,andsuchobligationscannoteasilybeavoidedwithoutappearingimpolite,unfriendly or impudent.Whenimagesconfrontuswithfriendlysmilesor arrogantstares,wearenot obligedto respond,eventhoughwedorecognizehowweareaddressed.Therelationisonly represented.We are imaginarilyratherthan reallyput in the positionof the friend,the customer,the lay personwho mustdeferto the expert.And whetheror not we identify with that positionwill dependotherfactors- on our realrelationto the produceror the institutionheor sherepresents,andon our realrelationto the otherswhoform part of the contextof reception.All thesame,whetheror notwe identifywith thewayweare addressed,we do understandhowwe are addressed,becausewe do understandthe way imagesrepresentsocialinteractionsandsocialrelations.It isthebusinessof thischapter to try andmakethoseunderstandingsexplicit. THEIMAGEACTANDTHEGAZE In thepreviouschaptersweshowedtwo picturesof an Antarcticexplorer,takenfromthe Australianprimary-schoolsocialstudiestextbook)ur Societyand)thers (}akley et al., 1985).Figure3.10 wasa photographin whichthe AustralianAntarcticexplorerSir DouglasMawsonlookeddirectlyat theviewer.Theschematicand'generalized'explorerin figure2.4, on the otherhand,did not look at the viewer.Thetwo imagesare in fact positionedsidebyside,thephotoonthe left page,thedrawingonthe right.Together,they combinetwo differentcommunicativefunctions.Thephotoseeksaboveallto brinqabout
  • 131. Representation and interact ion t 1 7 an imaginaryrelationbetweenthe representedexplorerandthe childrenfor whomthe bookis written,a relationperhapsof admirationfor,andidentificationwith,a national hero.Andthismeansalsothattheimage-producer(theinstitutionof educationalpublish- ing)addressesthechildreninthevoiceof thenationalheroandmakesthatnationalhero an 'educational'voice.Thedrawing,ontheotherhand,seeks,firstof all,to bereadasa pieceof objective,factualinformation,andin thiswayaimsto setintomotiontheactual processof learning. Thereis,then,a fundamentaldifferencebetweenpicturesfrom which represented participantslookdirectlyat the viewer'seyes,andpicturesin whichthis is notthe case. Whenrepresentedparticipantslool<at theviewer,vectors,formedbyparticipants'eyelines, connectthe participantswith theviewer.Contactis established,evenif it is onlyon an imaginarylevel.In additiontheremaybea furthervector,formedbya gestureinthesame direction,asinfigure4.1. Thisvisualconfigurationhastwo relatedfunctions.In thefirst placeit createsa visual form of directaddress.It acl<nowledgesthe viewersexplicitly,addressingthemwith a visual'you'.In thesecondplaceit constitutesan'imageact'.Theproducerusestheimage Q fiq e.f Recruitmentp0ster(AtfredLeete,I9l4) (ImperialWarMuseum)
  • 132. 118 . Representationand interaction to dosomethingto theviewer.It isfor thisreasonthatwe havecalledthiskindof imagea 'demand',followingHalliday(1985):theparticipant'sgaze(andthegesture,if present) demandssomethingfrom the viewer,demandsthat the viewerenterinto somekind of imaginaryrelationwith himor her.Exactlywhatkindof relationisthensignifiedbyother means,for instanceby the facialexpressionof the representedparticipants.Theymay smile,inwhichcasetheviewerisaskedto enterintoa relationof socialaffinitywiththem; theymaystareat theviewerwithcolddisdain,inwhichcasetheviewerisaskedto relateto them,perhaps,asan inferiorrelatesto a superior;theymayseductivelypoutat theviewer, inwhichcasetheviewerisaskedto desirethem.Thesameappliesto gestures.A handcan pointat the viewer,in a visual'Hey,youthere,I meanyou',or invitethe viewerto come closer,or holdtheviewerat baywitha defensivegesture,asif to say,'Stayawayfrom me'. In eachcasethe imagewantssomethingfrom theviewers- wantsthemto do something (comecloser,stayat a distance)or to forma pseudo-socialbondof a particularkindwith therepresentedparticipant.Andindoingthis,imagesdefineto someextentwhotheviewer is (e.9.male,inferiorto the representedparticipant,etc.),andin that wayexcludeother viewers. In thehistoryof art,thislookwasa significantinnovation.Althoughin Italianpainting smallfiguresamongthebystandersoftheCrucifixionandotherbiblicalscenescanbeseen to lookat the viewerfrom the fourteenthcenturyonwards,the 'demand'picturecomes intoitsownin thefifteenthcentury.Accordingto Panofsky(1953:190),it originatedin self-portraits,andJanvan Eyckwasthefirst to useit in Man in a RedTurban(1433), whichis regardedbymostart historiansasa self-portrait. In 1.433Jan van Eyckmadeoneof the greatdiscoveriesin portraiture.In the portraitof a 'Manina Redfurban', completedin October21 ofthatyear,theglance of thesitteristurnedoutof thepictureandsharplyfocusedonthebeholderwith an air of skepticismintensifiedby the expressionof the thin mouthwith its slightly compressedcorners.For the first time the sitterseeksto establishdirectcontact withthespectator.. . . Wefeelobservedandscrutinizedbya wakefulintelligence. (Panofsky,1953:798) 0therstraceit backfurther.Accordingto Belting(1990:57),'thesuggestionof reciprocity betweentheviewerandthepersondepictedintheimage'hada devotionalpurpose.Bythe thirteenthcentury,monksintheircells'hadbeforetheireyesimagesof theVirginandher crucifiedson,sothatwhilereading,prayingandsleeping,theycouldlookuponthemandbe lookeduponwiththeeyesof compassion'(ouritalics). Representedparticipantswholookattheviewerareusuallyhuman(oranimal),butnot always:theheadlightsof a carcanbedrawnaseyeslookingat theviewer,for instance,and onthescreenof oneautomaticbankteller,a creaturewhosecombinedheadandbodyhas the box-likeshapeof a machine,smilesat theviewer,holdingout hishandin an inviting gesture,thus'demanding'a friendlyrelationbetweenthemachineanditsuser(figure4.2). Thepointis,whethertheyarehumanor not,bybeingrepresentedaslookingat theviewer, theyarerepresentedashuman,anthropomorphizedto somedegree.
  • 133. Representation and interaction I 1 9 O riqa.zATMscreen Otherpicturesaddressus indirectly.Herethe vieweris not object,but subjectof the look,andthe representedparticipantis the objectof the viewer'sdispassionatescrutiny. Nocontactismade.Theviewer'sroleisthatof an invisibleonlooker.All imageswhichdo notcontainhumanor quasi-humanparticipantslookingdirectlyat theviewerareof this kind.Forthisreasonwehave,againfollowingHalliday(1985),calledthiskindofimagean 'offer'- it'offers'the representedparticipantsto the vieweras itemsof information, objectsof contemplation,impersonally,asthoughtheywerespecimensina displaycase. It isalwaysinterestingto studywhichkindsof representedparticipantsare,in a given context,depictedasdemandinganimaginarysocialresponseof somel<indfromtheviewer, andwhicharenot.In OurSocietyand0thers(Oal<leyetal.,).985),theAustralianprimary- schooltextbookfromwhichwedrewmanyof our keyexamplesinthefirstversionof this book,immigrantfamiliessmileat theviewer.However,thehumanparticipantsin pictures fromtheseimmigrants'countriesof origindo not lookat theviewer,notevenin close-up portraits,as,for instance,inthe portraitof an Italiangrandmotherwhostayedbehind.In thechapteronAborigines,bycontrast,hardlyanyoftheAboriginalparticipantslookat the viewer.TheAboriginalpoetOodgerooNoonuccal,referredto inthebookas'l(athWalker', anddepictedin close-upin the lastillustrationofthat chapter,istheonlyexception(see figure4.3 below).Herexpression,hermake-up,herhairstyleanddresshardlydistinguish herfrom non-Aboriginalwomenof herage.At mostherskinissomewhatdarker,buteven that is not verypronouncedin the black-and-whiteshot.OtherAboriginalpeoplein the chapterare muchmorecleariydepictedas'other', and evenif they do, occasionally, lookdirectlyat the viewer,theydo sofrom a longdistance,whichgreatlydiminishesthe impactof theirlook,or arefiguresin the bacl<ground,lookingblanklyandmoreor less
  • 134. 120 Representation and interaction @ fig l.f OodgerooNoonuccal(0akleyetal.,1985) accidentallyinthedirectionof thecamera.Aboriginalpeople,inthisprimary-schooltext- book,aredepictedasobjectsof contemplation,notassubjectsforthepupilto enterintoan imaginarysocialrelationwith.Immigrants,bycontrast,at leastoncetheyarein Australia, are portrayedas peoplewith whom the pupilsshouldengagemoredirectly,and in a friendlyway,asequals. Thechoicebetween'offer' and'demand',whichmustbe madewheneverpeopleare depicted,is not onlyusedto suggestdifferentrelationswith different'others',to make viewersengagewith someand remaindetachedfrom others;it can alsocharacterize pictorialgenres.In somecontexts- for instance,televisionnewsreadingandthe posed magazinephotograph-the'demand'pictureispreferred:thesecontextsrequirea senseof connectionbetweentheviewersandtheauthorityfigures,celebritiesandrolemodelsthey depict.In othercontexts- for example,featurefilm andtelevisiondramaandscientific illustration-the'offer' ispreferred:herea realor imaginarybarrieriserectedbetweenthe representedparticipantsandthe viewers,a senseof disengagement,in whichthe viewer musthavetheillusionthattherepresentedparticipantsdo notknowtheyarebeinglooked at,andinwhichtherepresentedparticipantsmustpretendthattheyarenotbeingwatched. Andwhat in onecontextis acceptedconventionmayin anotherbea startlingmistakeor an innovativeexperiment.Filmtheorists(e.9.Allen,1977;Wollentlg92)havehailedthe look at the cameraas a daring,Brechtian,'self-reflexive'-stylefigure,but in television newsreadingthelookat thecameraiscommonplaceand,wewouldthink,notexactly'self- reflexive'- at leastfor the oresenters:an intervieweewho looksat the camerain a televisionnewsprogrammebreaksthe rulesin an unacceptableway.Not everyonemay
  • 135. Representation and interact ion 121 addressthe viewerdirectly.somemayonlybe lookedat, othersmaythemselvesbethe bearersof the look.Thereis an issueof communicativepoweror'entitlement'(Sacks, 1992) involvedin this,notonlyin pictures,butalsoin everydayface-to-facecommunica- tion,for instancein interactionsbetweenmenanowomen: Asheanswersthegirl'slaststatementhebeginstalkingandreachesthepointwhere normallyhewouldlookaway,but insteadhe is still staringat her.Thismakesher uncomfortable,becausesheis forcedeitherto lockeyeswith him,or to lookaway fromhimwhileheistalkinq.If hecontinuesto talkandstarewhileshedeflectsher eyes,it putsher intothe 'shy'category,whichsheresents.If shebo/dlylockseyes withhim,hehasforcedherintoa'lover,sgaze,,whichshealsoresents. (Morris,1977i76) Diagrams,mapsandchartsaremostoftenfoundincontextsthatoffera kindof knowledge which,in Westernculture,hastraditionallybeenvaluedhighly- objective,dispassionate l<nowledge,ostensiblyfreeof emotiveinvolvementandsubjectivity.Hencethe'demand'has beenrareinthesevisualgenres.Buttherewerecontextsinwhichthetwoformsof address werecombined.Schooltextbooksof the kindwe usedas datawhenwe wrotethe first editionof this book,for instance,constructeda progressionfrom 'demand,to 'offer, pictures,and this not only in the courseof a chapter,as in the chapteron Antarctic exploration,butalsoin thecourseof a wholebookor seriesof booksand,indeed,in the courseof educationasa whole- illustrationsthatservedto involvestudentsemotivelyin thesubjectmatterthengraduallydroppedout ashigherlevelsofeducationwerereached. In seniorhigh-schooltextbookswefound'demand'picturesat mostinthecartoonswhich, in almostapologeticfashion,soughtto alleviatethe seriousnessof thetextfrom timeto time,asin a cartoonin a geographytextbook(Bindonandwilliams,rgBB)wherea girl lookeddespondentlyat the viewer,with the words'what doeshypothesismean?,in a dialogueballoonemanalingfrom her mouth.In the contextof education,the'demand, pictureplayedanambivalentrole.0ntheonehand,it wasnota highlyvaluedform,buta form deemedsuitableonlyfor beginners,a form onegrewout of as oneclimbedthe educationalladder;ontheotherhand,it playedan indispensablerolein educationalstrat- egy: objectiveknowledgehad to be built, apparenily,upon a foundationof emotive involvement,of identificationwith celebratorymythologies,for instance.Thisfoundation wasthen,gradually,repressed,for if it wasnot repressed,the l<nowledgebuilton it could not beseento be objective.Outsidethe sphereof education,the valueof the,demand, picturedependedon theassumededucationallevelof the reader.When,for instance,the massmedia(orautomatictellermachines)beganto use'demand'pictures,thoseeducated in the linguisticandvisualgenresof objectiveknowledgeandimpersonaladdresswould havefelt patronized,'addressedbelowtheirclass'.Thosenotsoeducated(or thosewho contestedthevalueof suchaneducation)wouldhavefeltthatcommunicationhadbecome moreeffective(andmorefun)thanwasthecaseinthe eraof moreformalandimpersonal publiccommunication.Aswe alreadydiscussedinthe previouschapters,thissituationis nowchangingand,withthegradualdisappearanceofthesemioticdistinction,theclassand
  • 136. Renresentation and interaction agedistinctionsit supported(andthe differentvaluesandattitudesthat wereassociated withthesedistinctions)alsobeganto erode. It ispossibleto relatethemeaningsconveyedby'demands'and'offers'tothelinguistic systemof person.Aswehaveseen,'demand'picturesaddresstheviewerdirectly,realizing a visual'you'.Butthisisnotmatchedbya visual'l'. The'l' isabsentin picturesQr,rather, objectified,hidingbehinda he/she/they.The'demand'picturethereforeremindsmoreof the languageOf,for instance,advertisementsandinstruCtions,Where'yoLl'S'abOundbut 'I'5' arerare,than,say,of the languageof personalletters,where'l's'and'you's'arelikely to beequallycommon.'Realproducers'cannotreferto themselvesdirectly.Theymust speakimpersonally,as traditionallyin bureaucraticandscientificlanguage,where'I's' were,andinmanycasesstillarelalsorepressed.Thepublic,ontheotherhand,isaddressed directly.Andyet,aswe haveseen/the distinctiOnbetween'offers'and'demands'derives historically from attemptsof Renaissancepaintersto findwaysof saying'l'in theself- portraitswhich expressedtheir new-foundself-confidenceand statusof independent artistsratherthanhumblecraftsmen' Buttheconceptsof'offer' and'demand'canalsoberelatedto anotherkeyconceptin linguistics,thatof the'speechact'.Asmentioned,wehavetakenthetermsfrom Halliday's descriptionof fourbasicspeechacts(or'speechfunctions'ashecallsthemin hisIntro- ductionto FunctionalGrammar,1985).Eachof thesespeechacts,saysHalliday,is part of an interactionaldyad,andhasits'expected'andits'discretionary'(alternative)social response.Thusspeechactscan(1) 'offerinformation',that is,forma statement,inwhich casethe responsesoughtis'agreement',althoughthe statementmayof coursealsobe contradicted;theycan (2) 'offer goods-and-services'(e.g.Wouldyou like a drink), in whichcasetheexpectedresponseis'acceptance',althoughtheoffermayalsoberejected; theycan(3)'demandinformation',that is,forma question,in whichcasetheexpected responseisananswer,althoughthe listenermayalsodisclaimthequestion,for instanceby saying1don'tknoworI can'ttellyouthat;andG),theycan'demandgoods-and-services', that is,constltutesomekindof command,in whichcasetheexpectedresponseis for the listenerto undertakewhatheor shehasbeenaskedto do,althoughlistenersmayof course alsorefuseto doso. Thesespeechactsare realizedbythe linguisticsystemof 'mood',that is,by syntactic permutations,permutationsof theorderof thesubjectandthefiniteelementof theverbal group(i.e.theelementof theverbalgroupthatexpressestenseandmodality).The'offerof information'(statement),for instance,isrealizedbytheindicativemood,inwhichthefinite elementfollowsthesubject,asinthissentencefrom Vogue" Women(subject)cannot(finite) liveby diamondsalone. 'Demancfinginformation',thequestion,isrealizedbytheinterrogativemood,whichhasth subjectfollowingthefiniteinthecaseofthe'polarquestion'(thekindof questionto which onecanjustanswer'yes'or 'no'),and,inthecaseof a 'WH-question'a WH-subjectfollowed bythefinite,or thefiniteelementfollowedbythesubject:
  • 137. Representation and interact ion 1 2 3 Can(finite) women(subject)liveby diamondsalone? Who(wn-subjecil could(finite) live by diamondsalone? Whatcould(finile) women(subject)liveby? The imperativemood/the 'command,,has positive,nofiniteeither: Don't (finite) live by diamondsalone Liveby diamondsalone no subjectat all and,whenthe polarityis Thespeechact of'offeringgoods-and-services',finally,is notrealizedbya permutationof subjectandfinite,but by variousidioms(e.g.Hereyou ard,by questions,in conjunction with specificmentalprocessverbsG.g.Doyou wanta drink) or by commands(Havea drinb and,indeed,invariousotherways. Thereare manysubtypesof thesefour basicl<indsof speechact. Theyare realized throughspecificcombinationsof additionallinguisticfeatures.A'prediction',forexample, isan'offerof information'withfuturetenseandeithersecond-or third-personsubject,or first-personsubjectwitlra'non-volitional'verb(e.g.Youwill liveby diamondsalone,or I will dieyoung).A'promise'isan'offerof information'withfuturetense,a first-person subjectanda volitionalverbG.g.I will buyyou diamondd.lt wouldtake ustoo far to discussthesein detail.Thepointto rememberisthat in languagetherearea few'core' typesanda verylargenumberof furthertypeswhichareconstructedoutof thecoretypes. Thesameistruein thecaseof images.A visual'invitation,isa'demand,picturewitha beckoninghandanda smilingexpression;a visual'summons',a'demand'picturewith a beckoninghand andan unsmilingexpression,'avisual'warning',a 'demand'picture witha raisedforefingeranda sternexpression;andsoon. Despitethesebroadsimilarities,it wouldseemthat'imageacts,do notwork in the samewayasspeechacts.Whenimages'effer',theyprimarilyofferinformation.0f course, an image,sayan advertisingimage,mayshowsomeoneofferingsomethingto theviewer, andthisoffermayin fact bea realoffer,whichcanbeobtainedbywritingto an address specifiedin the advertisement.But if thereis suchan 'offer of goods-and-services'in images,it musttakethe formof an'offer of information'.It mustberepresented.Itcannol beenacteddirectly. when images'demand',they demand,one couldsay,the 'goods-and-services'that realizea particularsocialrelation.0fcourse,an imagecouldshowa gestureof puzzle- ment,a'silent'question,buttheexampleis somewhatcontrivedandwouldneedverbal reinforcement,or reinforcementby a conventionalvisualsign,for instance,a question mark.Thereisnoimageactfor everyspeechact.Butthisneednotbesoforever.Although languageandimagedohavetheirspecificaffordances,whatcanbe,said,and'done,with images(andwith language)doesnotonlydependonthe intrinsicanduniversalcharac- teristicsof thesemodesof communication,butalsoonhistoricallyandculturallyspecific
  • 138. r l q Representationandinteraction socialneeds.It is quitepossibleto extendthe semanticreachand useof imagesinto domainswhichformerlyweretheexclusiveprovinceof language,asisalreadydone,ona smallscale,in placeswherepeopleareunlikelyto haveanygivenlanguageincommon(for example,internationalairports). SIZEOFFRAMEANDSOCIALDISTANCE Thereis a seconddimensionto the interactivemeaningsof images,relatedto the'sizeof frame',to the choicebetweenclose-up,mediumshotand longshot,and so on.Justas image-producers,in depictinghumanor quasi-humanparticipants,mustchooseto make themlookat thevieweror not,sotheymustalso,andat thesametime,chooseto depict themascloseto or far awayfromtheviewer- andthisappliesto thedepictionof objects also.And,lil<ethechoicebetweenthe'offer'andthe'demand',thechoiceof distancecan suggestdifferentrelationsbetweenrepresentedparticipantsandviewers.In handbool<s aboutfilm andtelevisionproduction,sizeof frameis invariablydefinedin relationto the humanbody.Eventhoughdistanceis,strictlyspeaking,a continuum,the'languageof film andtelevision'hasimposeda setof distinctcut-offpointsonthiscontinuum,in thesame wayas languagesimposecut-offpointsonthecontinuumof vowelswecanproduce.Thus thecloseshot(or'close-up')showsheadandshouldersof thesubject,andtheveryclose shot('extremeclose-up','bigclose-up')anythinglessthanthat.Themediumcloseshot cutsoff the subjectapproximatelyat the waist,the mediumshotapproximatelyat the knees.Themediumlongshotshowsthe full figure.In the longshotthe humanfigure occupiesabouthalftheheightof theframe,andtheverylongshotisanything'wider'than that.Stylisticvariantsarepossible,buttheyarealwaysseenandtalkedaboutintermsof thissystem,aswhenfilm andtelevisionpeopletalk of 'tightcloseshots'or'tight framing', or abouttheamountof'headroom'ina picture(i.e.spacebetweenthetopof theheadand theupperframeline). In everydayinteraction,socialrelationsdeterminethe distance(literallyandfigura- tively)we keepfrom oneanother.EdwardHall (e.9.1966: 170-20) hasshownthat we carrywith usa setof invisibleboundariesbeyondwhichwe allowonlycertainkindsof peopleto come.Thelocationof theseinvisibleboundariesisdeterminedbyconfigurations of sensorypotentialities- bywhetheror nota certaindistanceallowsusto smellor touch the otherperson/for instance,andby howmuchof the otherpersonwe canseewith our peripheral(sixty-degree)vision.'Closepersonaldistance'isthedistanceat which'onecan holdor grasptheotherperson'andthereforealsothedistancebetweenpeoplewhohavean intimaterelationwitheachother.Non-intimatescannotcomethiscloseand,if theydoso,it will beexperiencedas an act of aggression.'Farpersonaldistance'is the distancethat 'extendsfrom a pointthat isjust outsideeasytouchingdistancebyonepersonto a point wheretwo peoplecantouchfingersif theybothextendtheirarms',thedistanceat which 'subjectsof personalinterestsand involvementsare discussed'.'Closesocialdistance' beginsjustoutsidethisrangeandis thedistanceat which'impersonalbusinessoccurs'. 'Farsocialdistance'is'thedistanceto whichpeoplemovewhensomebodysays"Stand
  • 139. Representation andinteraction L25 awaysoI canlookatyou"'-'businessandsocialinteractionconductedat thisdistance hasa moreformalandimpersonalcharacterthanin theclosephase'.'Publicdistance', finally,is anythingfurtherthan that,'the distancebetweenpeoplewho are and are to remainstrangers'.Thesejudgementsapply,of course,withina particularculture,and Hallcitesmanyexamplesof the misunderstandingswhichcanarisefrom intercultural differencesinthe interpretationof distance. With thesedifferencescorresponddifferentfieldsof vision.At intimatedistance,says Hall (1964),weseethefaceor headonly.At closepersonaldistancewetakeinthe head andtheshoulders.At far personaldistanceweseetheotherpersonfromthewaistup.At closesocialdistanceweseethewholefigure.At far socialdistanceweseethewholefigure 'with spacearoundit'. Andat publicdistancewecanseethetorsoof at leastfouror five people.It isclearthatthesefieldsofvisioncorrespondcloselyto thetraditionaldefinitions of sizeof framein film andtelevision;in otherwords,that the visualsystemof sizeof framederivesfrom the'proxemics',as Hallcallsit, of everydayface-to-faceinteraction. Hall is awareof this and in fact acl<nowledgesthe inffuenceof the work of Grosser, a portraitpainter,on hisideas.Accordingto Grosser(quotedin Hall,I966i7I-Z), at a distanceof morethan 13 feet(4m),peopleare seen'ashavinglittleconnectionwith ourselves',andhence'thepaintercanlookat hismodelasif hewerea treeina landscape or anappleina stilllife'.Fourto eightfeet(r.25-2.5m),ontheotherhand,isthe'portrait distance': the painteris nearenoughsothat hiseyeshaveno troublein understandingthe sitter'ssolidforms,yet he is far enoughawayso that the foreshorteningof the formspresentsnorealproblem.Hereat thenormaldistanceof socialintimacyand easyconversation,the sitter'ssoul beginsto appear.. . . Nearerthan threefeet [90cmJ,withintouchingdistance,thesoulisfartoomuchinevidencefor anysortof disinterestedobservation. Thedistancespeoplekeep,then,dependontheirsocialrelation- whetherthisisthemore permanentkind of socialrelationon whichHall mainlyconcentrates(thedistinction betweenintimates,friends,acquaintances,strangers,etc.)or the kindof socialrelation thatlastsforthedurationof a socialinteractionandisdeterminedbythecontext(someone intheaudienceof a speechgivenbyanacquaintanceor relativewouldneverthelessstayat publicdistance,thedistanceof the'stranger').Butthesedistancesalso,andat thesame time,determinehowmuchof the otherpersonis in our fieldof vision- just asdoesthe framingof a personina portraitor filmshot. Likethe'demand'picture,theclose-upcameto theforeinthe Renaissance.Ringbom (I965:48) arguesthat it hasitsoriginin devotionalpictures,whereit servedto provide 'the "near-ness"so dearto the God-seekingdevout,.In Italianand Dutchpaintings of theearlysixteenthcenturyit acquireda'dramatic'function,allowing'thesubtlestof emotionalrelationshipswitha minimumof dramaticscenery,(p.4B). Thepeopleweseein imagesarefor themostpartstrangers.It istruethatweseesome of them(politicians,film andtelevisionstars,sportsheroes,etc.)a gooddealmorethan
  • 140. L 2 6 Representation and interaction others,butthiskindol lamiliaritydoesnot of itselfdeterminewhethertheywill beshown in closeshotor mediumshotor longshot.Therelationbetweenthe humanparticipants representedin imagesandthe vieweris onceagainan imaginaryrelation.Peopleare portrayedasthoughtheyarefriends,or as thoughtheyarestrangers.Imagesallowusto imaginarilycomeascloseto publicfiguresasif theywereourfriendsandneighbours- or to lookat peoplelikeourselvesasstrangers,'others'.In theprimary-schoolsocialstudies textbookfromwhichwehavequotedseveralexamples,threeAboriginalboysareshownin longshot,occupyingonlyabouta quarterof theheightof the'portrait'formatframe.The captionreads,'Thesepeopleliveat Redfern,a suburbof Sydney.'Theyare shown impersonally,asstrangerswith whomwe do not needto becomeacquaintances,as'trees ina landscape'.Althoughtheydo lookat theviewer,theydosofromsucha distancethat it barelyaffectsus.Indeed,theyare so smallthat we can hardlydistinguishtheir facial features.'Theirsouldoesnot yet beginto appear',to useGrosser'swords.Thecaption, significantly,givesthemnoname;in fact,wherethemorefriendly'boys'couldhavebeen, thequiteformal'people'hasbeenused. The portrait of the Aboriginalpoet OodgerooNoonuccal(figure4.3), already mentionedin theprevioussectionof thischapter,isa tightcloseshot.Sheis depictedin a personalway.If thiswasall we couldseeof herin reality,wewouldbecloseenoughto touchher.As mentioned,the sectionin whichthe photooccursconcludesa chapteron AboriginesinwhichnootherAboriginesmilesat theviewerinthisway.0neof herpoems is quoted:'DarkandwhiteuponcommongroundlIn clubandofficeandsocialround! Yoursthefeelof a friendlylandlThegripof the hand'(Oakleyet al.,1985:164).But Noonuccal'smessageis not borneout bytheway'darkandwhite'areportrayedin the chaoter. Patternsof distancecan becomeconventionalin visualgenres.In currentaffairs television,for example,'voices'of differentstatusare habituallyframeddifferently: the camera'movesin for biggerclose-upsof subjectswho are revealingtheirfeelings, whereasthe set-upfor the "expert" is usuallythe sameasthat for the interviewer- the breastpocketshot'.Both kindsof 'statusedparticipants'tendto be 'nominated'(their namesappearon the screenin superimposedcaptions)and 'havetheir contributions framedandsummedup' (Brunsdonand Morley,l97Bi 65).ln otherwords,distanceis usedto signifyrespectfor authoritiesof variouskinds,on televisionas in face-to-face interaction. In diagramsthehumanfigureis almostalwaysshownin mediumlongor longshot- objectively,'asif hewerea tree in a landscape'.Thepicturesin figure4.4 illustrateda front-pagenewspaperstoryabouta murdercasein Sydney.Thediagramsshowexactly what happened,from an objectifyingand impersonaldistance(andfrom a highangle). Theclose-upphotosaccompanytestimoniesby formerpatientsof the victim,but are representedasalso'friends'of wereadersof the SydneyMorningHerald,andthereforeas peoplewhoserelationwith the victim we shouldidentifywith. As in the chapteron Antarcticexploration,thepersonalandthe impersonal,theemotiveandthedetached,are combined. Sofar wehavediscussedsocialdistancein relationto human-representedparticipants,
  • 141. Representationand interaction . I2-l }II5 PATIENTS @ fig l.+ The murderof Dr Chang(SydneyMotning Heratd,sJuly799L) butunlikethesystemof'offer'and'demand',thesystemof socialdistancecanapplyalso to the representationof objectsandof the environment.As sizeof frameis traditionally definedin termsof specificsectionsof the humanbody,beginningstudentsof film and televisionareoftenat a lossasto whichtermsto usefor describingshotsof objectsand landscapes.Thescaleof sevensizesof frameseemstoofine-grained.Therearenoclear-cut equivalentsfor the shoulder,the waist,the knees.And objectscomein manydifferent shapesandsizes.Wewouldneverthelesssuggestthat at leastthreesignificantdistances canbedistinguished,andthattherearecorrespondencesbetweenthesedistancesandour everydayexperienceof objectsandthe environment,'in otherwords,that sizeof frame canalsosuggestsocialrelationsbetweentheviewerandobjects,buildingsandlandscapes. At closedistance,wewouldsuggest,the objectis shownas if thevieweris engagedwith it as if heor sheis usingthe machine,readingthe bookor the map,preparingor eating thefood.Unlessthe objectis verysmall,it is shownonlyin part,andoftenthe picture includestheuser'shand,or a tool- for instance,a knifescrapingthesoftmargarinein an advertisement.Filmandtelevision'cutaways'('overshoulders')of objects,in whichthe objectsshownareintegratedintoanactionthroughtheediting,usethisdistance.Atmiddle distance,theobjectisshowninfull,butwithoutmuchspacearoundit. It is representedas
  • 142. r2a Representation and interacti on withinthe viewer'sreach,but not as actuallyused.Thistypeof pictureis commonin advertising:the advertisedproductis shownin full, but from a fairly closerange,anda steepangle,asif theviewerstandsjust infrontof thetableonwhichit isdisplayed.At long distancethereisan invisiblebarrierbetweentheviewerandtheobject.Theobjectisthere for our contemplationonly,out of reach,as if on displayin a shopwindowor museum exhibit.Thescreenshotof the EuropeanPlayStationwebsite,in figure4.5, usesboth middledistanceandclosedistance,significantlyputtingthecloseshotonthe right,asthe 'New'(seechapter6). Thesamekindof distinctionscanbemadewith respectto representationsof buildings andlandscapes.Wecanseea buildingfrom the distanceof someoneaboutto enterit, in whichcasewewill notseethewholeof the building,as is (again)oftenthecasein film shotsinwhichthebuildingisrelatedto someaction.Wecanalsoseeit fromthedistanceof someonewhojustidentifiedit ashisor herdestination,andissurveyingit for a moment, beforemovingtowardsit.In thatcasetheframewillincludeonlythebuildingandleaveout thesurroundingenvironment.0rwecanseeit,soto speak,frombehindthegatesthat keep the publicat a respectfuldistancefromthe palace/or thefortress,or the nuclearreactor, andin thatcasetherepresentationwill includealsothespacearoundthebuilding.Land- scapes/too,canbeseenfrom within;from a kindof middledistance,with a foreground objectsuggesting,perhaps,that the vieweris imaginarilylocatedwithinthe landscape, but stoppingfor a moment,as if to takestockof what is ahead;or from a longdistance, fromtheair,perhaps,or froma'lookout'position,a placenotitselfin the landscapebut affordingan overviewof it, as,for instance,in manyof the photographicillustrationsin geographytextbooks. @ fig a.S Playstationwebsite(http://eu.playstation.c0m/europe-select.jhtml)
  • 143. Representation andinteraction r29 We will endwith somebriefcommentson the verydifferentway in whichsocial distanceis realizedinthe Englishlanguagemainlythroughpermutationsintheformality of style(seeJoos,i-967).Intimatelanguageisa kindof personallanguage,spokenperhaps onlybythemembersof a coupleor family,or bya groupof schoolfriends.Thespeal<ersof sucha'languageof intimates'oftenhavespecialnamesfor eachother,nameswhich outsidersdonotgetto use.Andthelanguageitselfisminimallyarticulated:a half-wordis enoughto understandeachother.Facialexpressions,eyecontact,intonation,voicequality, etc.carrymostof themeaning,andpeoplewhoarein an intimaterelationwitheachother becomefinelyattunedto thereadingof meaningsconveyedinthlsway.'Personal/language is casual,witha gooddealof colloquialismandslang.Non-verbalexpressionstillcarries muchof the meaningbut not so muchthat 'half a word is enough'.'Sociallanguage', thoughstillcolloquial,alreadybeginsto introducea hintof formality.Andthereis,inthis kindof situation,lesssharingof informationandassumptions.Thelanguageneedsto be morearticulate,moreverballyexplicit,so that non-verbalexpressionis no longeras importantasin intimateandpersonalstyle.Publiclanguage,finally,isthe languageused in moreor lessformaladdress.Herelanguagebecomesmonologic:listenersno longer participateastheydo in the otherstylesof speech.Speechis no longerimprovised,but thoughtout in advance,perhapsevenfullyor partiallywrittenout.Intonationandother formsof non-verbalexpressionbecomeasformal,asmuchsubjectedto controlassyntax and word usage.Speechmust be fully explicit,meaningsfully articulatedverbally. Colloquialismsareoutof placeanda moreformalvocabularymustbeemployed.Writers canof courseusethesestylesto addressusasfriendsor evenintimates,evenwhenweare not,justas picturescangiveusclose-upsof peoplewho,in reality,areandwill remain strangersto us- thinkof thecolloquial,chummyinformalitywithwhichweareaddressed in manyadvertisements. PERSPECTIVEANDTHE SUBJECTIVEIMAGE Thereisyetanotherwayin whichimagesbringaboutrelationsbetweenrepresentedpar- ticipantsandthe viewer:perspective.Producingan imageinvolvesnot onlythe choice between'offer'and'demand'andtheselectionof a certainsizeof frame,butalso,andat thesametime,theselectionof anangle,a'pointof view',andthisimpliesthepossibilityof expressingsubjectiveattitudestowardsrepresentedparticipants,humanor otherwise.By saying'subjectiveattitudes',wedonotmeanthattheseattitudesarealwaysindividualand unique.Wewill seethattheyareoftensociallydeterminedattitudes.Buttheyarealways encodedasthoughtheyweresubjective,individualandunique.Thesystemof perspective whichrealizes'attitude'wasdevelopedinthe Renaissance,a periodinwhichindividuality andsubjectivitybecameimportantsocialvalues,andit developedpreciselyto allowimages to becomeinformedby subjectivepointsof view.Paradoxically,whilethesewerethe meaningsencoded,perspectiverestsonan impersonal,geometricfoundation,a construc- tionwhichisa quasi-mechanicalwayof 'recording'imagesof reality.Sociallydetermined viewpointscould,in thisway,benaturalized,andpresentedas'studiesof nature',faithful
  • 144. 130 Representationandinteraction copiesof empiricalreality.0nlyrecentlyhasit becomepossibleagainto seethat per- spectiveis also'a daringabstraction'(Hauser,1962: 69), andto discussits semiotic effects,for instance,infilm theory(e.g.Comolli,I97I). Pre-Renaissanceforms,frescoesonthewall of a churchnave,for example,or mosaics in the domedroofof a church,didnot haveperspectiveto positiontheviewer.Viewersof suchworkswerepositioned,notbythe internalstructureof thework,but bythestructure of its environments,boththe immediateenvironmentof the church,its proximityto the altar,for instance,andthe widersocialenvironments.In otherwords,the syntaxof the objectdependedfor itscompletion,itsclosure,notona particularrelationwiththeviewer butona particularrelationwith itssurroundings,andthepointof viewwasthepositionthe vieweractuallytookupin relationto theimage:'Theworldinthepicturewasexperienced asa directcontinuationof theobserver'sownspace'(Arnheim,!974:274.As a result, the viewerhada certainfreedomin relationto the object,a degreeof what,today,we wouldcall 'interactiveuse'of thetext,albeitin thecontextof a highlyconstrainedsocial order.Fromthe Renaissanceonwards,visualcompositionbecamedominatedbythesystem of perspective,with its single,centralizedviewpoint.Thework becamean autonomous object,detachedfrom its surroundings,movable,producedfor an impersonalmarket, ratherthanfor specificlocations.A framebeganto separatethe representedworldfrom the physicalspacein whichthe imagewasviewed:at thetimeperspectivewasdeveloped, picturesbeganto beframedpreciselyto createthisdivision,to markofftheimagefrom its environment,andturn it intoa kindof 'windowon theworld,.At the sametime,images becamemoredependenton the viewerfor theircompletion,theirclosure,andviewers becamemoredistancedfrom the concretesocialorderin whichthe worldhadformerly beenembedded:theynowhadto learnto internalizethesocialorder.Thisyieldeclgreater freedomwith respectto theimmediate,concretesocialcontext,butdiminishedfreedomin relationto thework.A parallelcanbemadewiththedevelopmentswhich,moreor less simultaneously,took placein music(seeshepherd,r977).lnmedievalmodes,basedas theywereonthepentatonic,anynoteof thescalecouldstandin onlyintervallicrelationto anyothernote.Henceanynotecouldprovidea senseof resolution,of closure.In thenew diatonicmusica strict hierarchywasestablishedbetweenthe fundamentals,sothat any melody,whateverthe harmonicprogressionsit traversed,hadto return,ultimately,to the samepredeterminednote,the'tonic',inthekeyof whichthepiecewasscripted.Thenotes in musicthusrelateto the keycentrein thesamefixedwayin whichviewersrelateto the perspectivalcentreof thevisualwork. Thereare,then,sincethe Renaissance,two kindsof imagesin westerncultures:sub- jectiveandobjectiveimages,imageswithkentral)perspective(andhencewitha'built-in, pointof view)andimageswithout(central)perspective(andhencewithouta'built-in, pointof view).In subjectiveimagesthe viewercanseewhatthereis to seeonlyfrom a particularpointof view.In objectiveimages,the imagerevealseverythingthereisto know (orthatthe imageproducedhasjudgedto beso)abouttherepresentedparticipants,even if,to doso,it isnecessaryto violatethelawsof naturalisticdepictionor indeed,thelawsof nature.Thehistoryof art hasmanystrikingexamplesof this- for example,thesculptures of wingedbullsandlionswhichffankedthedoorsof Assyriantemples:fromthesidethese
  • 145. Representation and interact ion I 3 1 hadfour movinglegs,and from the front two stationarylegs,five altogether,so as to provide,from everyside,a viewfrom which no essentialpartswere missing.Modern technicaldrawingsmaystill showwhat we knowaboutthe participantstheyrepresent, what is objectivelythere,ratherthanwhat we wouldseeif we werelool<ingat them in reality,ratherthanwhat is subjectivelythere.If wewere,in reality,to seethefront of the cubeinfigure4.6thewaywe knowit 'objectively'is (a square),wewouldnotat thesame timebeableto seethetopandtheside.It isanimpossiblepicture(ora possiblepictureof a highlyirregularhexahedron,ratherthana cube)fromthe pointof viewof whatwe can seein reality.Yet in manycontexts(for instance,assemblyinstructionsfor a pieceof furniture)an 'objective'picturelikethis is entirelyacceptable.0bjectiveimages,then, disregardthe viewer.Theysay,as it were,'l am thisway,regardlessof who or whereor whenyouare.' Bycontrast,thepointof viewofthesubjective,perspectivalimagehasbeenselectedfor the viewer.As a resultthereis a kindof symmetrybetweenthewaythe image-producer relatesto therepresentedparticipants,andthewaytheviewermust,willy-nilly,alsoreiate to them.Thepointof viewis imposednot onlyonthe representedparticipants,but also ontheviewer,andtheviewer's'subjectivity'isthereforesubjectivein theoriginalsenseof the word,the senseof 'beingsubjectedto somethingor someone'.In a shortessayon Chineseart,BertoltBrechthascommentedonthis: As we know,the Chinesedo not usethe art of perspective.Theydo not liketo see everythingfrom a singlepointof view.Chinesecompositionthuslacksthe com- pulsionto whichwe havebecomealtogetheraccustomed... and rejectsthe subjugationof theobserver. (Brecht,967:278-9) Thesystemof perspectiveisfundamentallynaturalistic.It developedin a periodinwhich theworldof naturewasno longerseenasmanifestinga divineorder(whichwasalso/and at thesametime,a socialorder),butasanautonomousandultimatelymeaninglessorder Q fig l.o Frontal-isometriccube
  • 146. 132 Representation and interacti on whoselawsalsogovernedthe conductof people.It wasexplicitlygroundedin the new scientificspirit,legitimizedbytheauthorityof scientificobservationandthephysicallaws of nature.Thenewmusic,similarly,wasconstructedascongruent,notwith a (divineand) socialorder,butwiththephysicallawsof sound. In the late nineteenthcentury,after centuriesof hegemony,bothsystemscameinto crisis,inthehigharts(Cubism,twelve-tonemusic)aswellasinthepopulararts.Film,for example,stillusesperspectivalimages,but,ina near-Cubistfashion,providesmultipleand constantlyshiftingviewpointsin itsediting.Moderntelevision,especiallyin programmes not basedon the modelof film,suchas newsprogrammes,hasgonea stepfurther,and challengesperspectivealsowithintheimage.A newsreadermayhave,behindhimor her,on thelefta verbaltext,andontherighta chroma-keyedmovingpictureonthewall (a wall whichisinfacta kindof two-dimensionalscreenonwhichto projecta'layout',andinfront of whichto positionthenewsreader).Modernmagazineandwebsitelayoutsformanother categoryof visualworkswhichareno longerbasedsolelyonthecompositionalprinciples of perspective.0f course,theystill containmanyperspectivalimagesbutthesehavebeen subordinatedto a structurethat canno longerbesaidto beperspectival.Twoexamples mayillustratethis. Thepictureonthe FordMondeowebsite(figure4.7) isnaturalistic.Whatweobserve herecouldalsobeobservedin reality.Therecouldbea carpositionedinthisway,infrontof thisparticularcoupleandthisparticularbuilding.As a resultof theangleandthesocial distance(a low-angle'longshot',withthecarintheforeground),viewersarethenmadeto relateto therepresentedparticipantsina certainway.Theyaremadeto'lookupto'them, andtheyaremadeto seethemasif theynoticethecarandthestylishcouplefrom across thestreet,withenvy.In thepictureonthe FordFiestapage(figure4.8),ontheotherhand, the viewer,ratherthan beingpositionedin the naturalworld,is confrontedwith a world ffi []]e*i#'l*tltrN"l**:S.lans*c{rr*{rillr}s}st p*.l""r"*"w[i6'ffi- !il hle$*-l.xlk Msmd*rail T}}* Mwd*a h$$ I f&r|1, d{e.tldattre d6r{Ss lsp.?iratin! s** t*vd;$ s? {ra*qpsr*ship sd k{hsol*q?. e *!* fa*il1 of reglx*r *Sei sltrlr*dlrS pd$oeas# &ed t!61 s{ticrs*art clf isatcl* }rs C4t doirrlesr. Aid v'tUblte f+-ffinFree;$4 treq*$l*ene P?*tq$io^ Sf!18a" t|:r f$d l4eed*6 i5 oee *{ tiie 6tf<!t pl*ei t$ be. @ fig a.Z NewJookFordMondeo(www.ford.co.uMe/mondeo)
  • 147. Representationandinteraction . I33 qffi {..m*u'gr**tir;rluvrr,6gi:r'{.pt$*+itr6q}'r.}t,a'st ' i . . . ' P x a l i _ . neaixr|.oc***rffi a l , * , t ' . . 1 - . e 3 { * ; : . . . + : Q fig n.A FordFiestaRockSolid(http://www.f0rd.co,uk) which openlypresentsitself as a semioticconstruct,mixing perspectivaland non- perspectivalelementsinsucha wayasto givetheappearanceof a continuumofformsfrom the representationalto the significational,whilethe visualas a wholeremainsnon- naturalistic:thecar,onthispage,cannotbesaidto be'behind'theLasVegasroadsign or'behind'theword'rocksolid'inthewaythatthecoupleandthebuildingin figure4.7 canbesaidto bebehindthe FordMondeo.'lnfrontof'and'behind'losetheirideational dimension,andbecometextualprinciplesonly.Thetwo pagesthusexemplifya shiftfrom thedominanceof natureto thedominanceof signification,andfromthedominanceof the perceptualto the dominanceof the conceptual- in a wayverysimilarto that whichwe observedin chapterl whenwecomparedBaby'sFirstBook(figure1.1)to Dicl<Bruna's 0n My Walkand0n TheFarm(figureI.2). INVOLVEMENTANDTHE HORIZONTALANGLE Whenweprolongtheconvergingparallelsformedbythewallsof thehousesinfigure4.9, theycometogetherin two vanishingpoints.Bothpointsare locatedoutsidethe vertical
  • 148. I34 . Representationand interaction @ rig e.r Aborigines(oakleyet a1.,L985) boundariesof the image/as shownin figure4.10.Thesevanishingpointsallowus to reconstructwhat we can seeevenwithoutthe aid of geometricalprojection:the scene hasbeenphotographedfrom an obliqueangle.Thephotographerhasnotsituatedhimself or herselfinfrontof theAborigines,buthasphotographedthemfromtheside. Figure4.10 showshow the positionfrom which the photo was taken can be reconstructedbydroppinglinesfromthevanishingpointsin sucha waythattheymeetto forma 90oangleonthelinedrawnthroughtheclosestcornerofthecottages.Figure4.11 showsthescenefromabove.Theline(ab)representsthefrontalplaneof thesubjectof the photograph:the lineformedbythefront of thecottages,which,as it happens,is alsothe linealongwhichthe Aboriginesare linedup.Theline(cd)representsthe frontalplane of the photographer(andhenceof the viewer).Hadthesetwo linesbeenparallelto one another,the horizontalanglewouldhavebeenfrontal- in otherwords,the photographer wouldhavebeenpositionedin front of the Aboriginesandtheir cottages,facingthem. Instead,the two linesdiverge:the angleis oblique.The photographerhasnot aligned himselfor herselfwith the subject,not facedthe Aborigines,but viewedthem'from the sidelines'. Horizontalangle,then,is a functionof the relationbetweenthe frontalplaneof the image-producerandthefrontalplaneof the representedparticipants.Thetwo caneither beparallel,alignedwith oneanother,or formanangle,divergefrom oneanother.
  • 149. Representati on and interact ion t 3 5 @ fiq e.fO Schematicdrawing:vanishingpointsof'Aborigines'(figure4.9) a *"-9oot>5 (zDr r^ - !>46>Cn-(--e-I @ fiq +.ff Schematicdrawing:topyiew0f'Aborigines'(figure4.9) Theimagecanhaveeithera frontalor anobliquepointof view.It shouldbenotedthat thisisnotstrictlyaneither/ordistinction.Therearedegreesof obliqueness,andwewill,in fact,speakof a frontalangleso longas the vanishingpoint(s)still fall(s)withinthe verticalboundariesof theimage(theymayfall outsidethehorizontalboundaries).
  • 150. 136 . Reoresentationand interaction Figure4.I2 hasa frontalangle.As shownin figure4.I3, thereis onlyonemajor vanishingpoint,andit liesinsidetheverticalboundariesof theimage.Figure4.14shows howthefrontalplaneofthephotographer(lineab)andthefrontalplaneoftherepresented participants(linecd) run parallel- that is, if oneonly considersoneset of represented participants,theteachers,the blackboardandthe readingchart.Thefrontalplaneof the Aboriginalchildren(lineef) makesan angleof ninetydegreeswith thefrontalplaneof theteachersandwiththefrontalplaneof the photographer.TheAboriginalchildrenhave beenphotographedfroma veryobliqueangle. Thedifferencebetweenthe obliqueand the frontal angleis the differencebetween detachmentand involvement.Thehorizontalangleencodeswhetherthe image-producer (andhence,willy-nilly,theviewer)is'involved'withthe representedparticipantsor not. Thefrontalanglesays,as it were,'Whatyouseehereis part of our world,somethingwe areinvolvedwith.'Theobliqueanglesays,'Whatyouseehereis nolpartof ourworld;it is theirworld,somethingweare notinvolvedwith.' Theproducersof thesetwo photographs have,perhapsunconsciously,alignedthemselveswiththewhiteteachersandtheirteaching tools,but not with the Aborigines.Theteachersare shownas 'part of our world',the Aboriginesas 'other'.And as viewerswe haveno choicebut to seetheserepresented @ fig l.fe Aboriginalchildfenat scho0l(Oakleyetal.,1985)
  • 151. Representation andinteraction L37 @ fig e.ff Schematicdrawing:vanishingpoint0f'Aboriginalchildrenat school'(figure4.12) () fiO C,f+ Schematicdrawing:topviewof'Aboriginalchildrenat school'(figure4.12) participantsastheyhavebeendepicted.Weareaddressedasviewersfor whom'involve- ment'takestheseparticularvalues.In reality,theymightnot - we mightbeAboriginal viewers,for example.It is onethingfor the viewerto be limitedbywhatthe photograph shows(andto understandwhat this means/for exampleexclusion,in the caseof an Aboriginalviewer);it is anotherthingto actuallyidentifywiththeviewpointencodedin the photo.We canacceptor reject,but eitherwaywe first needto understandwhat is meant. Theprimary-schoolsocialstudiestextbook)ur SocietyandOthersprovidesa further =l I-. I I d --_ ffi :- z - - - @ - - - - g ' - - J r M0 l iolloil : l l ^ I plEl i t f &
  • 152. r38 Representation and interacti on illustration.A shotof the NewSouthWalesParliamentHousein Sydneyis frontal,and takenfroma lowangle.A shotofthechurchistakenfromanobliqueandsomewhathigher angle.Theformerillustratesa sectionaboutSydneyinthechapter'Whatisa City?';the latter,a sectionabouta Maorifamilyinthechapteronimmigrants.Religionisdepictedas somethingwhich,in thecontextof primary-schoolsocialstudies,doesnot belongto'our society'-the bookcontainsstatementslike,'TheBritishbelievedin oneGod'(notethe pasttense)andquestionslike,'Doyouthinka churchor a cemeteryislikea sacredsite?'It fostersa detached,outsider'sattitudetowardsthe Christianreligion. In thedepictionof humans(andanimals),'involvement'and'detachment'caninteract with'demand'and'offer'incomplexways.Thebodyof a representedparticipantmaybe angledawayfromtheplaneoftheviewer,whilehisorherheadandlorgazemaybeturned towardsit (seee.g.figure4.24 below)- or viceversa.Theresultis a doublemessage: 'althoughI am not part of yourworld,I neverthelessmakecontactwith you,from my own,differentworld';or'althoughthispersonis partof ourworld,s0meonelil<eyouand me,we neverthelessoffer hisor her imageto youas an objectfor dispassionatereflec- tion.'Thelatteristhecase,for example,inanillustrationfroma Dutchjuniorhigh-school geographytextbook(Bolse/ al.,l,996i21). In a sectionentitled'De DerdeWereldin onzesfraat'('TheThirdWorldin our Street'),two picturesareshownsidebyside.0n the leftweseethreeolderwomen,theirheadscarvesanemblemof theirstatusasimmigrants. Theyarephotographedfromanobliqueangle,henceas'notpartof ourworld'andin long shot,henceas 'others','strangers'.0n the rightwe see,left in the foreground,a blonde girl,clearlymeantto betakenas Dutch,witha blackfriend,whohashisarmaroundher. Theangleis a gooddealmorefrontalthanthat of the shotof thethreewomen,andthe shotis a close-up:sheis shownas like'us',Dutchhigh-schoolstudents,andfrom'close personal'distance.But shedoesnot makecontactwiththeviewers.Shedoesnot invite the viewersto identifywith her,andwith her relationshipto a blackman.Instead,the vieweris invitedto contemplateherrelationshipdetachedly,to ponderthefact that some peoplelike'us'haverelationshipswith blackpeople,but not,it is implicitlysuggested, 'we'viewersourselves.Sheis a phenomenonto beobserved,nota personaddressingthe viewer. Equallycomplexand ambivalentis the backview.Oneof the authors,at age21, photographedhisparentsin a snow-coveredpark,just outsideBrussels(figure4.15)and, perhapsmoreimportantly,it wasthispicturehechoseto pinonthepinboardof hisstudent roominAmsterdam,ratherthanoneoftheother,morefrontalpictureshehadtakenonthe sameday.At thetime,hisfeelingsfor hisparentswerecomplex.Deepattachmentmixed with onlyhalf-understooddesireto distancehimselffrom the worldin whichhe was broughtup.Perhapsthepicturecrystallizedtheseconfusedemotionsfor him.0ntheone hand,it showedhisparentsturningtheirbackonhim,walkingawayfromhim(a reversal, of course,of the actualsituation);on the otherhand,it showedthis gestureof'turning one'sback',in a sense,'frontally',in a maximally'confronting'way.Butto exposeone's backto someoneis alsoto makeoneselfvulnerable,andthisimpliesa measureof trust, despitethe abandonmentwhichthe gesturealsosignifies.Perhapsthe picturereminded himof a passagefroma Dutchnovelhelikedat thetime:
  • 153. Representationandinteraction . I39 i , r i . . . , ' f f : i @ fiq e.fl Photographof author'sparents,1968 Throughthewindowheseesthemwalkaway.'HowmuchI lovethatman',hethinks, andhowimpossiblehehasmadeit for meto expressthat.. . . Hismotherhaslinked armswith him.With hesitantstepsshewalksbesidehimonthefrozenpavement.He keepslookingat themuntiltheyturnthecorner,nearthetall featheredpoplars. (Wolkers.1965:6I) Howis 'involvement'realizedin language?Perhapsthe systemof possessivepronouns comesclosestto realizingthe kindsof meaningswe havediscussedhere.But the two systems,the visualsystemof horizontalangleand the linguisticsystemof possessive pronouns/differin manyways.Involvement,aswehaveseen,isalwaysplural,amatterof 'mine'and'his/herlits';a matterof distinguishingbetweenwhatbelongsto 'us'andwhatto 'them'.And,whilein languageonecannoteasilyhavedegreesof 'ourness'and'theirness', in imagessuchgradationisan intrinsicpartof thesystemof involvement.Finally,thereis no'yours'in thesystemof horizontalangle.Thevisual'you-relation'is,aswehaveseen, realizedby the systemof'offer'and 'demand'.Perspectiveputsa barrierbetweenthe
  • 154. 140 Representat ion and interact ion viewerandthe representedparticipants,evenin the caseof a frontalangle:the viewer looksat the representedparticipantsand hasan attitudetowardsthem,but doesnot imaginarilyengagewiththem. POWERANDVERTICALANGLE Textbooksof film appreciationneverfailto mentioncameraheightasan imporuanrmeans of expressionin cinematography.A highangle,it issaid,makesthesubjectlooksmalland insignificant,a lowanglemakesit lookimposingandawesome:'Lowanglesgenerallygive an impressionof superiority,exaltationandtriumph... highanglestendto diminishthe individual,toffattenhimmorallybyreducinghimto groundlevel,torenderhimascaught in an insurmountabledeterminism'(Martin,1968:37-B).Butthisleavestheviewerout of the picture.We wouldrathersay it in a somewhatdifferentway: if a represented participantisseenfroma highangle,thentherelationbetweentheinteractiveparticipants (theproducerof the image,andhencealsothe viewer)andthe representedparticipants is depictedas one in whichthe interactiveparticipanthaspoweroverthe represented participant- the representedparticipantis seenfrom the pointof viewof power.If the representedparticipantisseenfrom a lowangle,thentherelationbetweenthe interactive andrepresentedparticipantsis depictedas onein whichthe representedparticipanthas powerovertheinteractiveparticipant.If,finally,thepictureisat eyelevel,thenthepointof viewisoneof equalityandthereisnopowerdifferenceinvolved. Thisis,again,a matterof degree.A representedparticipantcantowerhighaboveusor lookdownon useversoslightly.In manyof the illustrationsin schooltextbookswe look downrathersteeplyonpeople- workersinthehall;childrenina schoolyard.In suchbooks thesocialworldliesat thefeetof theviewer,soto speak:knowledgeispower.Themodelsin magazineadvertisementsandfeatures,andnewsworthypeopleandcelebritiesinmagazine articles,onthe otherhand,generallylookdownontheviewer:thesemodelsaredepicted asexercisingsymbolicpoweroverus.As shownin figure4.5,productsadvertisedin the advertisementsmaybephotographedbothfroma lowangle,ashavingsymbolicpowerover us,andfrom a highangle,as beingwithinreachandat the commandof the viewer.The photographreproducedinfigure4.16showsa guardinthe'deathrow'sectionof a prison in Texas.The angleis low,to makehim look powerful.But what makesthis picture extraordinaryisthat nottheguard,butthehorseisclosestto theviewer,andthat it is not the guard,but the horse,whoseeverymovementis commandedby this guard,who is lookingat theviewer.what canthishorse'demand'fromus?Thereinliesthemysteryand theforceof thispicture.Empathywitha fateof beingsubjugatedto thepowerrepresented bytheguard?0r witha fateof suffering? Howis powerrealizedin language?Herewe need,again,to rememberthe difference betweenface-to-facecommunicationand mediatedcommunication.In the classroom. for example,powerwill manifestitselffirst of all in the relationbetweenteacherand pupil.This,ascate Poyntonhasshown(1985:ch.6),is inthemainrealizedthroughthe differencebetweenthe linguisticformsthat maybeusedbytheteachersandthe linguistic
  • 155. Representation and interact ion @ rig e,fO Prisonguard(DannyLyon,1969)
  • 156. r42 Representation and interacti on formsthatmaybeusedbythepupils;inotherwords,througha lackof reciprocitybetween the choicesavailableto eachparty in the interaction.Theteachermay usefirst names in addressingthe pupils;the pupilsmaynot usefirst namesin addressingteachers.The teachermayuseimperativesto 'demandgoods-and-services';thepupilswouldhaveto use politeforms,for instance,questions.Thislackof reciprocityhasitseffecton everylevel of language:phonology,grammartvocabulary,discourse,andon ideational,interpersonal, aswell astextualmeanings.If thereis,in face-to-facecommunication,anyquestionof powerrelationsbetweenrepresentedparticipantsandthepupils,thenthisresultsfromthe powerrelationbetweentheteacherandthepupils. Tosomeextentthis isthe casein writingalso,andnotjust becausein writing- as in mediatedcommunicationgenerally- the absenceof the writercauses/from the start,a fundamentallackof reciprocity(youcannottalk backto the writer),but alsobecause thewriterandthereaderareoftenunequalin a numberof otherways.Thereadermaybe addresseddirectly,by meansof the second-personpronounyou,whilethe writer hides behindimpersonalforms.Mentalprocessesmay be attributedto the readerwhilethe writer'smentalprocessesare neverreferredto. Imperativesmaybeused,as modulated processespredicatedof the reader(youcan,youshould,you need,etc.),whilesuchforms are not usedof thewriter.Herearesomeexamplesof textsin whichpoweris encodedin thisway- thefirstfrom a Revlonadvertisement,thesecondlrom OurSocietyand0thers (Oakleyet al.,I9B5): Wrinkles.Theydon'tstartwhereyouthinktheydo.Theystartunderneathyourskin. That'swhyAnti-AgingDailyMoisturizergoesbeyondmeresurfacetreatment. Whenyoustudyplacesandpeopleyouneedto havea wayof keepingtheinformation youcollect.Onewayof doingthisisto takenotesfromthebookswhichyouread.You cannotwritedownallthethingsyoureadasthiswouldmeanwritingoutthewhole book.Notesarea shortwayof recordingthemostimportantinformation. In the first textthe writerdoesnot directlyreferto himselfor herself,but writesas an impersonalauthority,in termsof relationalprocesses('Theystart underneathyourskin'). Thereaderis referredto directly('you'),andthe writernot onlyknowswhatthe reader thinks('whereyouthinktheydo'),but alsothatthe reader'sthoughtsare misguided:the authorityof thewriter is firmlybasedonthe reader'signorance.In the secondtext,too, the writerdoesnot directlyreferto himselfor herself,but writesimpersonally,in terms of relationalprocesses('Notesare a shortway of recordingthe most importantinfor- mation'),whilethereaderisaddresseddirectly('you').Theprocesseswithwhichthereader is associatedare modulatedin variousways('youneed','yor.rcannot').In bothcasesthe lackof reciprocitywhichrealizespowerisencodedinthetextitself. Butthisomniscientknowledgeof thereader'smind,thisdirectpostulationof whatthe readerneeds(mustdo,shouldthink,will feel,andso on) andthis lackof reciprocity betweenthewriterandthereaderor thespeakerandhearer,cannotberealizedinthesame wayin images.In images,thepowerof an image-producermust,asit were,betransferred
  • 157. Representation and inleract ion t43 onto oneor morerepresentedparticipants- thepowerof theadvertiseronthemodel,the powerof the producerof thetextbool<ontheordinarypeoplerepresentedin thetextbook. Thenearestequivalentin speechwouldbethe useof evaluativeadjectives.Wemight,for example,transcodea pictureinwhichwe lookdownonfactoryworkersor refugeesas'the humbleworkerl or'the downtroddenrefugees'.lnthe issueof the Australian Women's Weeklyfrom whichwe tool<manyof our originalexamples,this kind of transcoding occursa numberof times.Themagazinecontainsphotographsof a bejewelledQueen Elizabeth,andtheactorMichaelDouglas- bothtakenfroma lowangle.0nthecoverthe relevantarticlesareannouncedbythefollowinglines:'Dnzzlrrue- TheQueen'sjewels'; 'FASctNATING- MichaelDouglas'"FatalAttractien"'.But thereremainsa verybig difference.Whatintheimageisanattitudetowardstherepresentedparticipantsbecomes in languagea characteristicof therepresentedparticipants:it objectifiestheattitude. NARRATIVIZATIONOFTHE SUBJECTIVEIMAGE In manycasesthereisno immediatelyapparentmotivationfor pointof view(andfor size of frame).Theanglemaybehighandfrontal,andsoconveypoweroverandinvolvement with the representedparticipants,but the precisenatureof the relationof powerand involvementisnotgiven.Thusa high-anglepictureof workersina factorycouldbesaidto betakenfromtheviewpointof a supervisorinanelevatedoffice,witha windowoverlooking the factory,but this remainsa metaphor.We do not seethe officein the picture.Other possibilitiesmightalsoserveto makeconcretetherelationof powerandinvolvement.In othercasesthe (imaginary)viewerintrudesin the pictureto a greateror lesserdegree. In anadvertisingcampaignthat ranat thetimeweworl<edonthefirstversionof thisbook, thiswasdoneby includingthe handsof the imaginaryviewerin the foregroundof the picture.Thesecouldthenbemaleor female,andgroomedin differentways- theycould weardrivinggloves,expensiverings,andsoon.In figure4.).7theycreatedtheviewpointof a couple. In filmsthe sequencingof imagescanfulfil thisfunction.Theshotof the factory, showingtheworkersfroma highangle,canbeprecededbya low-angleshotof theelevated office,witha supervisorbehindthewindowlookingdownat theworkers.In suchcasesthe textnarrativizesthepointof viewandimposesa fictionalviewerbetweenthe represented andtheinteractiveparticipants.Butevenwhentheiroriginsarenotshown,viewpointscan alwaysberelatedto concretesituations.0necan,andperhapsshould,alwaysask,'Who couldseethissceneinthisway?','Wherewouldonehaveto beto seethissceneinthisway, andwhatsortof personwouldonehaveto beto occupythatspace?' OBJECTIVEIMAGES Scientificandtechnicalpictures,suchas diagrams,mapsandcharts,usuallyencodean objectiveattitude.Thistendsto be donein oneof two ways:by a directlyfrontal or
  • 158. t44 Representation and interact ion 'rWhYdon'twe takl nnine?" oXlfkata SMt*g td*d' cil :iti;r:1., !!l.til::ts:l.ii' r!i r ;; :rj :,i r: ri: : xj. ljl !..1. ; i'; i. i! rjl i 1!;1. Q fiq +.fZ Sterlinqadvertisement(ivewldea,Novembel1987) perpendiculartop-downangle.Suchanglesdo suggestviewerpositions,but specialand privilegedones,which neutralizethe distortionsthat usuallycomewith perspective, becausetheyneutralizeperspectiveitself.Toillustratethiswith a simpleexample,whena cubeis drawnperspectivallyitssidesarenot of equallength,andthedegreeof distortion dependsontheangle,ontheencodedviewerposition.Thecubedoesnotlook'asweknow it is',withall itssidesof equallength,but'aswe seeit',froma particularposition.But fromdirectlyin frontthethirddimensiondisappears,andthecubeappearsflat,withall itssidesof equallength.Fromabove,exactlythe sameeffectoccurs.Perspectiveandits attitudinizinqeffecthavebeenneutralized: @ rig +.fe Cuheseenfrom anangle,frontallyandfrom above
  • 159. o z E .s = q
  • 160. "[,Ove"n*f]lp' 'ullflavouro[atrulvxtisfring granulatcdcoflce...lovcacupof BushcllsiVlastcrRoastCo{Icc. @ ftate Z Bushellsadyertisemenl(Woman'sWeekt!,1987)
  • 161. Plalet JoshuaSmith(William Dobell,1943)(Art callery of NewSouthWales)
  • 162. @ etat. I Patick White(LouisKahan,1963)(Art Galleryot NewSouthWales)
  • 163. *,, Ia .rl s,a ': gl t . tI f 3a * J E !
  • 164. :w re : : Q etatez PalgraYecolourscheme
  • 165. w Q CtateI Colourfulthoughls (transparency)
  • 166. Representationand interaction - 145 Frontalandtop-downangles,however,are notobjectivein entirelythesameway.The frontal angleis the angleof maximuminvolvement.It is orientedtowardsaction. Thepicturesof the Antarcticexplorer(figure2.4 couldbetranscodedas'Thesearethe clothesyoushouldwearandthisisthewayyoushouldwearthemif youwantto explorethe Antarctic.'Thefrontalangleistheangleof 'thisishowit works','thisis howyouuseit', 'this is howyoudo it'. Thetop-downangle,onthe otherhand,is the angleof maximum power.It is orientatedtowards'theoretical',objectiveknowledge.It contemplatesthe worldfrom a god-likepointof view,putsit at yourfeet,ratherthanwithinreachof your hands.Abstractdiagramscansometimesbereadin bothways.A communicationmodel, for instance(e.9.figure2.2), canbe readas a map ('top-down',a schema,a'theory of communication':'thisis whatcommunicationlookslike,fromthe pointof viewof a disinterestedobserver'),or as a frontalview,a blueprint,a'practicalmanualof com- munication'('thisiswhatyoudowhenyoucommunicate')- andthisisperhapsoneof the sourcesof itssocialpower. A third objectiveviewpoint,the cross-section,andthe,X-ray,view,shouldalsobe considered:its objectivityderivesfrom the fact that it doesnot stopat appearances,but probesbeyondthe surface,to deeper,morehiddenlevels.In Westerncultureit is almost exclusivelyusedindiagrams,althoughonecansometimesalsoobserveexperimentswithit inchildren'sdrawings. Nof all diagrams/mapsandcharts,however,arecompletelyobjective.Theverticalangle oftheGulfWarmapinfigure4.79,ishigh,butnotcompletelytop-down,anditshorizontal USplana dcrtroy 28anks,26 othCr wtdder thr.i lItiugry Dlrcar andtiici rrnmrniricn {urer "*dchi lEri !t tranrport planordrop ' 'dairycsttcr' botnbsofi lslandln oreoaradonfsr @ fig l.ff cuff War map(s/dr e! MorningHetatd,22Januaryt99I)
  • 167. r46 Representation and interact ion angleisoblique,causingusto lookat thetheatreof warfromthesidelines,in a relatively detachedway.In booksaboutscienceforyoungchildrenwefindsimilarangles(seefigure 5.11,for example),their obliquenessperhapssuggestingthat theyare not (or not yet quite)meantas'howto do it' pictures. Elementsof perspectivemayalsobeaddedto graphsandcross-sections,to givea sense of reality,of physicalexistence,to abstract,two-dimensionalvisuals.Havingfirst been abstractedfrom the concrete,three-dimensionalworldof people,placesandthings,they arenowrestoredto it, but in a transformedway,asneryhuman-madel<indsof thingsand places.Thuswe cansee- for instance,in lavishlyproducedannualcompanyreports- three-dimensionalbargraphs,lookinglikeskyscrapersor monoliths,againsta background of cleanandsmoothhillsin ffat,primarycolour.In figure4.20graphsbecomea setting for action:touristsmovethroughthe abstractlyrepresented,but neverthelessthree- dimensional,worldof the internationaltouristbusiness,justas mayalsobethe casein televisionnewsgraphics,whereafurthersenseof realitymaybegivento suchpicturesby meansof animation. The additionof perspectiveaddsnothingto the representationalmeaningof these diagrams,mapsandcharts;butit doesaddattitudinalmeanings.In alltheseexamplesthe angleishigh,explicitlyattitudinalizingtheobjectivestanceof thegod-liketop-downview, andoftennarrativizingit astheviewfromthesatellite,that moderntoolof theproduction of visualknowledgeandsymbolof informationalpower.Thehorizontalangle,ontheother hand,mayvary:withthe'increasein tourism'wearedirectlyinvolved;theeventsof the GulfWar (figure4.19),onthe otherhand,wewatch'fromthesidelines',as bystanders. Thisprocessof attitudinalizationhappens,not so muchin the contextswherethis new oun ?ounttlt sttLLAR ?Al(Gs (lrr ANDryHO'LLgEBRiltGtNGtTtN* ym" @ fig e.ZOAn increasein tourism (Sydne! Moming Heratd,23January]-ggll
  • 168. Representation andinteraction r47 visualknowledgeis producedand this new informationalpowerexercised,but in the contextsin whichit is disseminatedin popularizedform,andcelebrated:hereconceptual andschematicimagesare dressedup in the clothesof visualreality,and literallyand figuratively'animated'. To concludethissectionwe addsomenoteson different,less'subjective'kindsof perspective.If, incentralperspective,thekindof perspectivewehavebeendiscussingsofar, somethingisseenfromthefrontandat eyelevel,thesides,top andbottomwill behidden fromviewA cubewouldappearas in drawing1 of figure4.2I.If thesamecubeis seen fromanobliqueangle,oneofthesideswillcomeintoview,buttheotherwillremainhidden. If theangleis high,sothatwe lookdownonthecube,thetopwill alsocomeintoview,as indrawing4 of figure4.2I. Butinthiscasethefrontwill nolongerbea square.It will be distorted.Thehorizontalparallelsin an imagein centralperspectiveconvergetowards oneor morevanishingpoints- andsodotheverticalparallels,althoughthisisoftenless obvious,asverticaldistancesarenotsogreat,andasverticaldistortionisoften'corrected' indrawingsandpaintings. Drawing2 in figure4.2It on the otherhand,is an exampleof 'frontal-isometric' perspective.Herethefrontof thecubeisnotdistorted,yetwecanseethesideandthetop. Andthehorizontalparallelsdo notconvergetowardsa vanishingpoint.Frontal-isometric perspectiveis basedon the 'objective'dimensionsof the representedparticipants,on what we knowthesedimensionsto be,ratherthan on howthey appearto us.For this reasonfrontal-isometricperspectiveis usedin technicaldrawings,whereit is important to be ableto measurethe dimensionsof the representedobjectsfrom the drawing.In frontal-isometricperspective,then,thereis not,asyet,a choicebetweeninvolvementand detachment.It is the analogyin visualtermsof the'impersonality'characteristicof scientificlanguage. Theperspectiveusedin drawing3 of fi9ure4.27 is calledangular-isometricperspec- tive.Herethe front is distorted,the squareno longerrepresentedas a square.But the horizontalandverticalparallelsdo not converge.Thereis no endto spacein this kind of perspective- it stretcheson indefinitely.Angular-isometricperspectivewas used, for example,in eighteenth-centuryJapanesewoodcuts- Japaneseartistsof this period alwayschoseanobliquepointof view,aswellasa relativelyhighangle.Theylookedat the worldwithouta senseof involvement,from a detachedpointof view,from a meditative distance. r f l Q fig +.Zf (1) Cubeseenfrom the front; (2) cubein frontal-isometricperspective;(3) cubein angular-isonetricpersp!ctive; (4) cubeseenfrom anangleIn centralperspective
  • 169. 148 . Representationand interaction Q fig +.ZZ Detailfiom a fourt!enth-centurySpanishnativity(fromArnheim,1974) Thisbriefsurveydoesnotexhaustthepossibilities.In medievalart'invertedperspec- tive'wassometimesused(seefigure4.22).Thisallowsbothsidesof an objectto beseen andcausesthe perspectivalvectorsto divergeratherthanconverge.It canoftenbefound in children'sdrawings(youngchildrenalsotendto drawtheworldastheyknowit to be, ratherthan as theyseeit) and,in recenttimes,hasbeentakenup by painterssuchas PicassoandBraque,wholookedfor moreobjectivewaysof representingtheworld,regard- ingthe simpleviewpointof centralperspectiveas one-sidedandrestrictive,andviewing realityas multifaceted,a complexwholeof oftenincompatibleand mutuallyclashing viewpoints.In thisway,asArnheimnotes(1974:I32),'Ihey makethe contradictionsof whichMarxistssoeakvisual'. A SUMMARY Figure4.23summarizesthe mainkindsof interactivemeaningwe havediscussedin this chapter.It shouldbe rememberedthat theseare'simultaneoussystems/(as indicatedby thecurlybrackets):anyimagemusteitherbea'demand'oran'offer'andselectacerta REALIZATIONS Demand 0ffer Intimate/personal Social Impersonal Involvement Detachment Viewerpower Equality Representedparticipant power gazeat theviewer absenceof gazeat theviewer closeshot mediumshot longshot frontalangle obliqueangle highangle eye-levelangle lowangle EtsBEE
  • 170. T Demand Contact ___________>l L 0""' Social distance @ fig C.Zf Interactivemeaningsin images sizeof frameandselectacertainattitude.In the nextfew sectionswewill discusssome examplesat greaterlength,to showhowthe systemsof 'contact,,'socialdistance,and'attitude'interact to createmore complexand subtlerelationsbetweenreoresented andinteractiveparticipants. TWOPORTRAITSANDTWOCHILDREN'SDRAWINGS Rembrandt'sfamousSelf-portraitwith Saskiadatesfrom1634.JohnBerger(1972: 111) callsit'an advertisementof the sitter,sgoodfortune,prestigeandwealth,and,headds,'likeall suchadvertisementsit is heartless'.Yet,fromthe pointof viewof the interactive meaningswe havediscussedin thischapterithe paintingis perhapsa littlemorecomptex thanBerger'sremarkssuggest.0ntheonehand,it isa'demand,picture- Rembrandtand Saskiasmileat theviewer,Rembrandtperhapsa littlemoreeffusivelyandinvitinglythan Saskia:heevenraiseshisglassina gesturedirectedat theviewer.0ntheotherhand,hehas shownhimselfand Saskiafrom behind,andfrom what Hall wouldcall 'closesocial, distance,with Saskiaa littlefurtherawayfromtheviewerthan Rembrandt- herheadis considerablysmallerthan Rembrandt'seventhoughsheis sittingon hislapandshould therefore,strictlyspeaking,becloserto theviewerthanRembrandt(theangleat whichher headisturnedio acknowledgethevieweralsoseemsunnaturaD.IsRembrandtdistancing himself(andSaskiaevenmore)from the viewer,excludingthe viewerfrom rnvolvement and intimacywith his new-found(and Saskia'salreadyestablished)socialstatus,thus Representationand interaction . I49 Intimate/personal Social Intefactive , i - meanrngs Impersonal T Involvement L Detachment f- subjectivity I f viewerpower | | -+f Equaritv Auitude----rl L - Representationpower I I |.- Action orientatlon L objectivitv ---1 Knowredge orientation
  • 171. 1 5 0 Representation andinteraction contradictingtheinvitation?Perhaps- buttheportraitisalsoa self-portrait.Rembrandt, the miller'sson,nowmarriedintoa wealthyandrespectablefamilyandlivingin grand style,alsodistanceshimselffrom hisnewself(andto someextentfrom Saskia),asif he cannotfeelfully involvedand intimatewith his newenvironment.As a self-portraitthe picturemaybeself-congratulatoryandsmug,'heartless',but it alsobetraysa degreeof alienation,positioningtherepresentedRembrandtin a complexandcontradictorysocial classposition,betweenthe world of his origins,which is alsothe pointof viewof the picture,andtheworldof Saskiaintowhichhehasmoved.This,wethink,makesit a little lesssmug,anda littlemoretouchingthanBergergaveit creditfor. Figure4.25showsa laterself-portrait,paintedin 1661.Bythistime,Saskiahasdied, and Rembrandthasgonebankrupt.Henowliveswith hisformerhousekeeper,Hendrickje, ina moredownmarketneighbourhood,andinmuchreducedcircumstances.In thisportrait Q fig +Zt Setf-poftrait with Sasftia(Rembrandt,1634)(Pinakotek,Dresden)
  • 172. Representation and interaction 1 5 1 Q fiq+.ZS Self-portnit(Rembrandt,166I)(Kunsthist0rischesMuseum,Vienna) heisableto comeface-to-facewith himself,to confronthimself(andtheviewer)squarely andintimatelywith himself:'Heis an old mannow.All hasgone,excepta senseof the questionof existence,of existenceasa question'(Berger,I972:112). Thepictureonthecoverof 'lVlyAdventure'(figure4.26),thestorybyaneight-year-old boywhichwe havealreadyfeaturedin the previouschapter,constitutesa'demand':the littleboyis lookingat us,andsmiling.Heseeksourrecognition.Hewantsto beacknow- ledged.0ntheotherhand,theangleisoblique,andhigh,andtheboyisshownfroma great distance.Not onlydoesthe writerof thisstoryshowhimselfin the roleof beingship- wrecked,healsoshowshimselfas'other'(theobliqueangle),assomeoneoverwhomthe viewerhaspower(thehighangle)andassociallydistant,a'stranger'(thelongshot).In
  • 173. I52 . Representationand interaction otherwords,heusesthe interactiveresourcesof thesubjectiveimage(quiteprecociously, wefeel)to showhimselfassmall,insignificantandalienated,yetdemandingrecognition from theviewer.At thesametimethe act of drawinghimselflikethisaffordshim.asthe producerof the image,somepoweroverthat imageof himself,anoutletfor hisfeelings.In supportof this interpretationit canbenotedthat the boydoesnot exactlyplaya heroic role in the story.After creatingthe raft, andjust as the raft 'startedto be goodfun', everythinggoeswrongfor him: he loseshis moneyand neverfinds it again,the raft collapsesandis lostirretrievably,andtheherohasto walkall thewayhome,wetandcold. It is an unhappyendingfor a herounableto controlthe unpleasanteventsthat happento h i m . Figure4.27isthefrontcoverof a'story'onsailingboatsbya childfromthesameclass asthe authorof 'My Adventure'.Its subjectis similar:peopleon a boat.But thesystems of imageact','socialdistance'and'attitude'takeonverydifferentvalues.Thecharacters do not lookat us;the pictureis an 'offer'.Theangleisfrontalandeyelevel,andthetwo figuresin the boatare neitherparticularlydistant,nor particularlyclose.Thereis no setting,notexture,nocolour,nolightandshade.Thesailingboatisdrawnwithgeometrical accuracy.But for the two figures- simplydrawn,andmoreor lessidentical,exceptfor theirsize(a fatherandson?)- thiscouldbe a technicaldrawinq.As suchit suitsthe O Fiq4,26 Coyerillustrationof MyAdventure'
  • 174. Representation and interacti on r53 ($ figl.Zl Coverillustrationol'sailing Boats' objective,generictitle,'SailingBoats',just asthe coverillustrationof 'My Adventure' suitsthat story'ssubjective,specifictitle.In mostof the illustrationsinsidetheessay,no humanfiguresare seen/as thoughthe childalreadyunderstandsthat the 'learning'of technicalmattersshouldbeorecededbva'humanelement'toattractnon-initiatesto the subject. Clearly,chlldrenactivelyexperimentbothwiththeinteractiveresourcesof languageand withthe interactiveresourcesof visualcommunication.Theyareactivesign-makers.And the differentwaysin whichthesetwo childrenrepresentboatsshowtwo verydifferent subjectivitiesat work.
  • 175. 5 M o d a l i t y : d e s i g n i n gm o d e l so f r e a l i t y MODALITYANDA SOCIALTHEORYOFTHE REAL 0neof thecrucialissuesin communicationisthequestionof the reliabilityof messages. Is whatweseeor heartrue,factual,real,or is it a lie,a fiction,somethingoutsidereality? To someextentthe form of the messageitselfsuggestsan answer.We routinelyattach morecredibilityto somekindsof messagesthanto others.Thecredibilityof newspapers, for instance,restsonthe'knowledge'thatphotographsdo not lieandthat'reports'are morereliablethan'stories',thoughsincewewrotethefirsteditionof thisbooktheriseof Photoshopand'spin'havebegunto undermineboththesetypesof knowledge. Moregenerally,andwithparticularrelevanceto thevisual,weregardoursenseof sight asmorereliablethanoursenseof hearing,'lsawit with my owneyes'asmorereliable evidencethan'l heardit with myownears'. Unfortunately,we alsoknowthat,whilethe cameramaynot lie- or not much,at any rate-thosewhouseit anditsimagescananddo.Thequestionsof truthandrealityremain insecure,subjectto doubtanduncertaintyand,evenmoresignificantly,to contestationand struggle.Yet,asmembersof a society,wehaveto beableto makedecisionsonthebasisof theinformationwereceive,produceandexchange.Andin sofar aswearepreparedto act, wehaveto trustsomeof theinformationwereceive,anddoso,to quitesomeextent,onthe basisof modalitymarkersinthemessageitself,onthebasisof textualcuesfor whatcanbe regardedas credibleand what shouldbe treatedwith circumspection.Thesemodality markershavebeenestablishedbythegroupswithinwhichwe interactasrelativelyreliable guidesto thetruth or factualityof messages,andtheyhavedevelopedout of the central values,beliefsandsocialneedsof thatgroup. In thischapterwewill discussthesemodalitycues.As throughoutthe book,we take themto bemotivatedsigns- signswhichhavearisenout of the interestof socialgroups whointeractwithinthestructuresof powerthatdefinesociallife,andalsointeractacross the systemsproducedby variousgroupswithin a society.As we havediscussedin the Introduction,the relationbetweenthe signifiersand signifiedsof motivatedsignsis, in principle,oneof transparency.Sign-makerschoosewhat they regardas apt, plausible meansfor expressingthemeaningstheywishto express.Wearethereforefocusingonthe rangeof signsfromwhichsuchchoicescanbemade- someof themspecializedmodality markers,otherspart of a muchwiderand moregeneralrangeof meansof expressing meaningsof truth and falsehood,fact and fiction,certaintyand doubt,credibilityand unreliability. A socialsemiotictheoryoftruthcannotclaimto establishtheabsolutetruthor untruth of representations.It canonlyshowwhethera given'proposition'(visual,verbalor other- wise)is representedastrueor not.Fromthe pointof viewof socialsemiotics,truth is a constructof semiosis,andassuchthetruthof a particularsocialgrouparisesfromthe
  • 176. Modality 155 valuesandbeliefsof that group.As longasthe messageformsanapt expressionof these beliefs,communicationproceedsin an unremarkable,'felicitous'fashion.Thisdoesmean, however,that ourtheoryof modalityhasto accountfor a complexsituation:peoplenot onlycommunicateand affirm as true the valuesand beliefsof their group.Theyalso communicateandaccorddeqreesof truth or untruthto the valuesandbeliefsof other gr0ups. Theterm'modality'comesfrom linguisticsandrefersto thetruthvalueor credibility of (linguisticallyrealized)statementsabouttheworld.Thegrammarof modalityfocuses onsuchmodalitymarkersastheauxiliaryverbswhichaccordspecificdegreesof modality to statements,verbslikemay,will andmustGf.the differencebetweenHemaycomeand He will comd andtheir relatedadjectivesG.g.possible,probable,certain)andadverbs (seeHalliday,1985: B5-9). But modalityis not only conveyedthroughthesefairly clear-cutlinguisticsystems(seeKressand Hodge,t978,7993'.127).Tal<ethisexample, from )ur Societyand0thers(}akleyet a1.,1985).Clearly,it notonlycontainsstatements suchas'Aboriginalpeoplehadnoreligion'and'thewholelandwasa cathedral',butalso indicationsof thetruthvalueof thesestatements: GovernorPhillip,thesettlers,andtheconvictscouldfindnochurchesor cathedrals or worksof art likethoseof Brltain.PerhapsthismadethemthinkthatAboriginal peoplehadnoreligion.In fact,theAborigineshadverycomplicatedreligiousbeliefs. Thesehadbeenpasseddownfrom onegeneratlonto the nextthroughDreamtime storiesfor thousandsof years.Forthe Aboriginesthe wholelandwasa cathedral. Theirart wasjoinedto their religion.Muchof their art hadbeenkeptsafelyfor thousandsof years. (Oakleyetal.,I9B5:I42) Thestatement'Aboriginalpeoplehadno religion'isgivenlowmodality.Thewritersdis- tancethemselvesfrom it by attributingit to 'GovernorPhillip,the settlers,andthe con- victs'byformulatingit asa subjectiveidea('thismadethemthinkthat . . .') andbyusing the pasttense(afterall,whatwastrue in the pastneednot betrue in the present).The writers'ownstatements(e.9.'theAboriginalshadverycomplicatedreligiousbeliefs')are not qualifiedin thisway,'theyareformulatedas objectivefacts('In fact,the Aborigines had . . .') and not attributed(it is curious,however,that they are nol extendedto the presenttimel).Thestatementsthat embodythe'beliefs'ofthe Aborigines,finally,are givenlowermodality.Theyare explicitlyattributedto the Aboriginesandthereforenot subscribedto by the writers,andtheyare qualifiedby termslike'story','dream'and 'belief' - termswhich,in Westernculture,signifylow modalityandare contrastedwith high-modalitytermssuchas'reality','fact' and'truth'. Theexampleshowsthat modalityis 'interpersonal'ratherthan 'ideational'.It does not expressabsolutetruthsor falsehoods;it producessharedtruthsaligningreadersor listenerswith somestatementsanddistancingthemfrom others.It servesto createan imaginary'we'.It says,asit were,thesearethethings'we'considerIrue,andthesearethe things'we'distanceourselvesfrom,forinstance:'we'havenoreligion,buttheAborigines
  • 177. r56 Modality do,andalthoughthisreligionistruefor'them',it isnottruefor'us'.Nevertheless,as(for 'us')art andreligionarenot'joined','we'canappreciateAboriginalreligionas'art',as beautiful'stories'and'dreams'(art,in Westernculture,haslowermodalitythan,for instance,science- hencethe greaterlicencegivento artists).We call the 'we'the text attemptsto produce'imaginary'becausemanyof the childrenwhoaremadeto readthe bookmayinfact 'havereligion'.However,we realizethatthesocialgroupingsdiscursively institutedinthiswaymaybeveryrealandmayhaveveryrealeffectsonchildren'slives. Theconceptof modalityis equallyessentialin accountsof visualcommunication. Visualscan representpeople/placesandthingsasthoughtheyare real,as thoughthey actuallyexistin this way,or as thoughthey do not - as thoughthey are imaginings, fantasies,caricatures,etc.And,heretoo,modalityjudgementsare social,dependenton whatisconsideredreal(ortrue,or sacred)inthesocialgroupfor whichtherepresentation isprimarilyintended.Consider,forinstance,the'speechcircuit'diagramfromdeSaussure's famousCoursein GeneralLinguistics(I974ll9I6J), shownoncemoreinfigure5.1 (see alsofigure2.18).It depictstwo humans,'A'and'B',anda process/circularandcon- tinuous,describedas the'unlockingof sound-imagesin the brain',followedby the 'transmittingof an impulsecorrespondingto the imageto the organsusedin producing sounds',followedby the 'travellingof the soundwavesfrom the mouthof A to the ear of B'(1974 t19161:11-12).In anotherversion(figure5.2),de Saussureschematizes thediagramevenfurther,makingit lookalmostlikeanelectricalcircuit. Thephotographin figure5.3 aisorepresentsthe speechprocess,or ratherpart of it, sinceweseeonly'A'speaking,andonly'B"unlockingsound-imagesin hisbrain'.It isa scenefrom RobertAldridge'smovieTheBig lhife 0955), starringRodSteigerandJack Palance. Thethreerepresentationsof thespeechprocessdifferin a numberof ways.First while thephotographrestrictsitselfto representingwhatwouldnormallybevisibleto thenaked eye,the diagramsdo not:theymakevisiblewhat is normallyinvisible(mentalprocesses, 'sound-imagesin thebrain')andtheydo showwhatcannormallyonlybeheard('sound waves').Todosotheytakerecourseto abstractgraphicelements(dottedandcontinuous lines,arrows)andto language.Second,whilethe photographpresentsuswith a moment --I_ .-j-:- - - - - - - / - - - z A O pisf.f Speechcircuit(deSaussure,t974n9l6l) B
  • 178. Modality 157 phonation @ fig S.Z Schematizedspeechcircuit(deSaussure,1974[1916]) O fiS S.: RodSteigerandJackpatancein fhe Big Knife(Atdridqe,1955) frozenintime,thediagramsdepicta processthattakesa certainamountoftimeto unfold: oneutteranceof 'A'as werras oneutteranceof 'B', at the veryreast.Third,whirethe photographdepicts'A'and'B'ingreatdetail,showingstrandsof hair,wrinkres,glimmers of lightin Steiger'sdark glasses,the diagramsreducethe two to schematicprofiles,or evencircles,minimalgeometricshapes,abstractelements.And,whiiethe photograph showsdepth,modellingcausedbytheprayof rightandshade,anda setting,a background, the diagramsomitall of these.Theyareabstractandschematicwherethephotographis
  • 179. 158 Modality concreteanddetailed;conventionalizedandcodedwherethephotographpresentsitselfas a naturalistic,unmediated,uncodedrepresentationof reality. Doesthis meanthat diagramsare less'real' than photographs,and hencelowerin modality,and that photographyis moretrue than diagrammaticrepresentation?Not necessarily.Totheviewersfor whomdeSaussure'sdiagramsareintended,theymayinfact bemorerealthanthe photograph,in the sensethat theyreveala truth whichrepresents moreadequatelywhatthespeechprocessis reallylike. Realityis in the eyeof the beholder;or rather,what is regardedas realdependson howrealityis definedby a particularsocialgroup.Fromthe pointof viewof naturalism realityis definedon the basisof howmuchcorrespondencethereis betweenthe visual representationof anobjectandwhatwenormallyseeof thatobjectwiththenakedeye(or, in practice,on the capacityof 35mm photographyto resolvedetailandrendertonal or colourdifferentiation:images,includingphotographs,canbeexperiencedas'hyper-real', asshowing'toomuchdetail','too muchdepth','toomuchcolour'tobetrue).Scientific realism,ontheotherhand,definesrealityonthebasisof whatthingsarelikegenericallyor regularly.It regardssurfacedetailandindividualdifferenceasephemeral,anddoesnot stopat whatcanbeobservedwiththenakedeye.It probesbeyondthevisualappearanceof things.In otherwords,realitymay be in the eyeof the beholder,but the eyehashada culturaltraining,and is locatedin a socialsettingand a history;for instance,in the communityof linguists,or of semioticiansin de Saussure'sday,a communitywhichsaw realityin thatform,in termsof abstractionsanddeeperregularities.A'realism'ispro- ducedby a particulargroup/as an effectof the complexof practiceswhichdefineand constitutethat group.In thatsense,a particularkindof realismis itselfa motivatedsign, inwhichthevalues,beliefsandinterestsof that groupfindtheirexpression. As the examplessuggest,definitionsof realityare also boundup with technologies of representationandreproduction.Therelativelyrecentchangefrom the dominanceof blackandwhiteto the dominanceof colourin manydomainsof visualcommunication showshowquicklythesehistoriescandevelop,andhowcloselytheyarerelatedto techno- logicalchange.For us,now as commonsenseviewers,everydaymembersof societyat large,the definingtechnologyis perhapsstill that of 35mm colourphotography,as we suggestedabove.But the shiftto digitalphotographyis alreadycreatinga newstandard for naturalism,whichstillaimsat everhigherresolution,naturalisticcolourrendition,and soon,but hasin fact madea decreasein resolutionandcontrastto becomeacceptableas thenormin manydomains. Eachrealismhasitsnaturalism- thatis,a realismisa definitionof whatcountsasreal - a setof criteriafor the real,and it will find its expressionin the 'right',the best,the (most)'natural'form of representingthat kindof reality,be it a photograph,digitalor otherwise,or a diagram.Thisis notto saythatall realismsareequal.Althoughdifferent realismsexistsidebysidein our society,the dominantstandardbywhichwejudgevisual realism,andhencevisualmodality,remainsfor themoment,naturalismasconventionally understood,'photorealism'.In otherwords,the dominantcriterionfor what is realand what is not is basedon the appearanceof things,on howmuchcorrespondencethereis betweenwhatwe can'normally'seeof an object in a concreteandspecificsetting,and
  • 180. Modality I59 whatwecanseeof it in a visualrepresentation- again,atleastintheory,forineffectit is basedon currentlydominantconventionsandtechnologiesof visualrepresentation.We judgean imagerealwhen,for instance,itscoloursareapproximatelyassaturatedasthose inthestandard,themostwideryusedphotographictechnorogy.whencorourbecomesmore saturated/we judgeit exaggerated,'morethan real',excessive.When it is lesssatur- atedwe judgeit'less than real,,'ethereal,,for instancgor 'ghostly,.And the samecan besaidaboutotheraspectsof representation,the renditionof detail,the representation of depth,andsoon.Pictureswhichhavethe perspective,thedegreeof detail,the kindof colourrendition,etc.of thestandardtechnorogyof colourphotographyhavethehighest modality,andareseenas 'naturalistic'. As detail,sharpness,colour,eic.are reducedor amplified,astheperspectiveflattensor deepens,somodaritydecreases. Likemanyotheradvertisements,the advertisementin plate2 is a compositetext.It showsa pictureof the product(thejar of instantcoffee),with a verbarcaption,anda picturewhichvisualizesthe preasurethe productwiil afford.Thispicture,showingtwo loverssharingan intimatemoment,usessoftfocusandsoftcolours,tendingtowardsthe samegolden-brownhue,andsodeliberatelyloweringmodality,representing,whatusingthe productwill be like'asfantasyor promise,as'what mightbe,,ratherthanas reality,as'what is'. Thepictureof the productitself,however,is in sharperfocusand usesmore saturatedanddifferentiatedcolours:the productis givenhighermodality,higherreality value,thanthe promiseof blissattachedto it, andtheadvertisementasa wholetherefore accordsvaryingdegreesof'credibility'tothedifferentrepresentationsit contains,justlike thetexton'Aboriginalreligion'.Thelowermodalityof thephotoof the romantrcscene, however,is not a matterof the sceneitselfbeingimprobableor fantastic(althoughthaf too,oftenhappensinadvertisingphotographs).Probableaswellasimprobabteeventsmay havehighor lowmodality.Justasonecan sayTherecertainlyare ghtosts(highmodality) andI believeghostsmayexist(ow modalitfl,soonecanalsoshowrealisticandnot-so- realisticdepictionsof ghosts. what isthedifferencebetweentheseusesof colour?we wouldput it thisway:themore that is takenaway,abstractedfrom the coloursof the representation,tne morec0lour is reduced,the lowerthe modality.Thereis a continuumwhichrunsfrom full colour saturationto theabsenceof colour,blackandwhite,inwhichonlythebrightnessvaluesof thecolours,their'darkness'or'lightness',remains.Thereisalsoa continuumwhichruns from full colourdifferentiationto a'reducedpalette'andeventuallymonochrome.For example,eighteenth-centurylandscapepainting(e.g.ClaudeLorrain)wasoftenrestricted to variousshadesof brownfor the foregroundandto desaturated,silverybluesfor the distance'Thisisnottheonlywayinwhichabstractionfrom'naturalistic, colourispossible. Thecolourof manyobjectsis not even.pareskin,for instance,may varyin redness, mayhavethe blueveinsshowingthrough,andsoon,andsuchdifferencesmayeitherbe renderedor abstractedfrom.In otherwords,colourmaybeidealizedto a greateror lesser degree- a scalewhichrunsfromnaturalisticphotographyviathechoiceof differentvalues of a colourfor the representationof lightandshade,to theffat,unmodulatedcolourused by childrenin their drawings,or,for example,in the work of painterssuchas Matisse. Matissewasnot a chird,of coursgwhenhe producedthe paintingswe nowadmire.His
  • 181. 1 6 0 . M o d a l i t y unmodulatedcoloursexpresseda differentview of what countsas real, as do the unmodulatedcoloursinchildren'sdrawings- wewillcommentonthisinmoredetaillater. Fromthe pointof viewof naturalism,however,modalityis decreasedin suchimages.The continuumfrommodulatedto ffatcolourisat thesametimea continuumfromhighto low modality.And in both casesthe rule applies:the greaterthe abstraction(awayfrom saturation,differentiationandmodulation),the lowerthemodality. It shouldbestressedthat whatwe aretalkingaboutis not abstractionfrom whatwe actuallysee,from 'the realworld'.The literatureof otheragesand culturesatteststo thefactthat peoplehavemarvelledat the'lifelikeness'of workswhich,byourstandards, arefar from 'naturalistic'.Whatwe aretalkingaboutat thispointisabstractionrelative to thestandardsof contemporarynaturalisticrepresentation. MODALITYMARKERS Sofarwehavediscussedtheroleof colourasa markerof naturalisticmodalitv.intermsof threescales: (I) Coloursaturation,ascalerunningfromfullcoloursaturationtotheabsenceofcol that is,to blackandwhite. (2) Colourdifferentiatio4a scalerunningfrom a maximallydiversifiedrangeof colours to monochrome. (3) Colourmodulation,ascalerunningfromfullymodulatedcolour,with,forexample useof manydifferentshadesof red,to plain,unmodulatedcolour. At oneendof thesescalesthe particulardimensionof colouris maximally reduced. At theotherendit ismostfullyarticulated,usedto itsmaximumpotential.Eachpointof the scalehasa certainmodalityvaluein termsof the naturalisticstandard.However,the pointof highestmodalitydoesnotcoincidewitheitherextremeof thescale:naturalistic modalityincreasesasarticulationincreases,but at a certainpointit reachesits highest valueandthereafterit decreasesagain.Naturalisticmodalityscalescouldthereforebe representedasinthefollowingexample: Eiackand white lvlaximumcolour saturation Lowestmodalrty Q fig s.+ Modality scale for colour saturation Highestmodality Low(er) modality We will nowdiscussthe otherkeymarkersof visualmodalityon whichwe already touchedinourdiscussionof fiqure5.3.
  • 182. Modality r6 1 (4 Contextualization,a scalerunningfromthe absenceof backgroundto the mostfullv articulatedanddetailedbackground. Withinthenaturalisticcodingorientation,theabsenceof settinglowersmodality.Bybeing'decontextualized', shownin a void,representedparticipantsbecomegeneric,a'typical example',ratherthanparticular,andconnectedwith a particularlocationanda specific momentin time.Thescaleof 'contextualization' runsfrom'full contextualization,,to'plain,unmodulatedbackground'.Onestepawayfrom'full contextualization,we find settingswhichareout of focusto a greateror lesserdegree,or whichlosedetailthrough overexposure/resultingin a kind of etherealbrightness,or underexposure,resultingin muddydarkness,or throughthe lossof visualdetailinthedepiction.Furtherdecontextu- alizationcan be achievedthroughellipsis:a few 'props,sufficeto suggesta setting, or a small,irregularlyshapedpatchof greenunderthefeetof a figurewitha few lines suggestinggrassindicatesthesetting,whilethe restof the paperis leftblanl<.0r perhaps the backgroundmay merelyshowan irregularpatternof lightandshade,or a fieldof unmodulatedcolour,or black,or white. Again,the mostfully articulatedbackgrounddoesnot havethe highestnaturalistic modality.The limitationsimposedby the resolutionof standard35mm photographic emulsionsandby the depthof fieldof standardlenseshaveaccustomedusto imagesin whichthe backgroundis lessarticulatedthanthe foreground.Whenthe backgroundis sharperandmotedefinedthanthis,a somewhatartificial,'morethanreal,impressionwill result- as,for instanceinolderHollywoodmoviesshotina studiowithbackprojection(a close-upof an actorin acar,in frontof a rearwindowbehindwhichweseethereceding landscapein sharpfocus),or in muchSurrealistpainting,suchasintheworkof Salvador Dali. (5) Representation,a scalerunningfrom maximumabstractionto maximumrepresenta- tionof pictorialdetail. An imagemayshoweverydetailof the representedparticipants:the individualstrands of hair,the poresin the skin,the creasesin the clothes,the individualleavesof thetree, andso on,or it mayabstractfrom detailto a greateror lesserdegree.Again,thereis a pointbeyondwhicha furtherincreaseof detailbecomes'hyper-real,andhencelowerin modalityfrom the pointof viewof 'photographic, naturalism.Similarly,in discussing decontextualization,above,wehavepointedoutthatreducedrepresentationof detailmay form oneof thewaysin whichthe modalityof backgrounds,of what is .distant,,is lower thanthemodalityof theforeground(thereisa parallelherewiththelowermodalityofthe pasttensein language). In photographyit is not onlysharpnessof focus,but alsoexposurewhichcanreduce detail.In artworka varietyof techniquescouldbe rankedon a scalefrom maximumto minimumdetail'Texturecanbecomestylized,renderedbylineswhichtracethefoldsinthe clothes,for example,andtheselinesmaybe manyandfine,as in detailedengravings,or fewandcoarse/asin quickandreadystylesof drawing.In medicaldrawings,for example,
  • 183. r62 Modality texturemaybecomeentirelyconventional:dotsto indicatethetextureof onelayerof skin, short,curvedlinesto indicatethetextureof another.Texturecanalsobeomittedaltogether - theparticipantisthenrepresentedmerelybythelinesthattraceitscontour.Beyondthis, the contourmaybesimplifiedto differentdegrees:a headmaybecomea circle,theeyes two dots,the moutha short,straightline.Diagramsandgeometricalarttakeabstraction evenfurtherandreducetheshapeof thingsto a smallvocabularyof abstractforms,asin the paintingsof Mondrian,or in figure5.2,de Saussure'sschematized'speechcircuit, diagram. 6) Depth,a scalerunningfromtheabsenceof depthto maximallydeepperspective. Bythecriteriaof standardnaturalism,centralperspectivehashighestmodality,followed by angular-isometricperspective,followedby frontal-isometricperspective,followedby depthcreatedbyoverlappingonly.Again,perspectivecanbecome'morethanreal,,aswhen strongconvergenceof verticallinesisshown,or a'fish-eye'perspectiveisused. (7) Illumination,a scalerunningfrom thefullestrepresentationof the playof lightand shadeto itsabsence. Naturalisticdepictionsrepresentparticipantsastheyareaffectedbya particularsourceof illumination.Lessnaturalisticimages,ontheotherhand,mayabstractfromillumination, andshowshadowsonlyin sofar astheyare requiredto modelthe volume,especiallyof roundobjects.Theyhave'shading'ratherthanshadow.0rtheyuseshadingto indicate recedingareasandhighlightsto indicateprotrudingareas,oftenin wayswhichhaveno explanationintermsofthelogicof illumination.Thiscanbedoneto differentdegrees:with a fully modulateddarkeningof the areasof shadow;with justtwo degreesof brightness, onefor the'lit'areasandonefor thosein shadow;with moreor lessdensehatchingor dottingoftheshadedareas;andsoon.At theextremeendofthescale,Iightandshadeareab- stractedfromaltogether,andlineratherthanshadingisusedto indicaterecedingcontours. (B) Brightness,a scale runningfrom a maximumnumberof differentdegreesof brightnessto justtwo degrees:blackandwhite,or darkgreyandlightergrey,or two brightnessvaluesof thesamecolour. Brightnessvaluescan also contrastto a greateror lesserdegree:in one picturethe differencebetweenthedarkestandthe lightestareamaybeverygreat(deepblacks,bright whites),in anotherthedifferencemaybeminimal,sothat a misty,hazyeffectis created. Thepaintingsof Rembrandtareinterestingfromthispointof view,in partbecausehisuse of illuminationandbrightnessis soofteninvokedasa paradigmexampleof naturalism, and in part becauseof the way his subtledivergencesfrom naturalismoftenacquire ideationalfunctions,ina broadlyallegoricalsense. Theabilityof photographyto renderblackandwhiteis limited,as is its abilityto differentiatebrightnessvalues.Again,a contrastrangeanda rangeof brightnessvalues
  • 184. Modality .163 whichexceedsthisabilitymaybeexperiencedasmorethan real,andhenceasbeinqof lowermodality. It followsfrom our discussionthat modalityis realizedby a complexinterplayof visualcues.Thesameimagemaybe 'abstract'in termsof oneor severalmarkersand 'naturalistic'in termsof others.Impressionistpaintings,for example,oftenhavea narrow brightnessrange,andabstractfrom lightandshadow,but theyhavea highiynaturalistic approachto colour.Yet,from this diversityof cuesan overaliassessmentof modalitvis derivedbytheviewer. From all this it mightseemthat the realizationof modalityin imagesis muchmore complexandfinelygradedthanthe realizationof modalityin language.yet language,too, allowscomplexcombinationsof differentmodalitycues.Take,for instance,the sentence I absolutelydon't think he couldpossiblyhavedoneif. Is this 'low,, 'middle,or ,high, modality?How doesone 'compute'thesevariousmodalitycuesinto one 'degreeof credibility'?Frequentlythereareevencontradictions:1fisprobablydefinitelytruethat. . . Andin language,too,thevalueof modalitycuesdependsoncontext.In academicwriting, for example,qualificationssuchasIt maywellbethe case. . .or It isquitepossiblethat. . . (bothlow modality,strictlyspeaking)servein fact to increasethecredibilityof thetext, as indicatorsof thecarewithwhichthewriter,sjudgementsweremade,andhenceof the reliabilityof thesejudgements. CODINGORIENTATION Sofar wehavedescribedthevalueof modalitymarkersintermsof thenaturalisticcriteria for'what countsasreal'.Wehavehypothesizedthattheabilityof moderncotourphotgg- raphyto renderdetail,brightness,colour,etc.constitutesfor ourculturetodaya kindof standardfor visualmodality.whenthisstandardis exceeded,an imagebecomes'more thanreal'- an effectwhichcanbe achievednot only in art (and is oftenthe favoured modalityin Surrealism),butalsobymeansof thespecialtechniques,materialsandequip- mentof studiophotography.A certainstandardof photographicnaturalism,dependenton the stateof photographictechnologyandon currentphotographicpractices,henceever evoiving,hasbecomethe yardstickfor what is perceivedas ,real,in images,evenwhen theseimagesare not photographs.Underpinningthis is the beliefin the objectivityof photographicvision,a beliefin photographyas capableof capturingrealityas it is, unadulteratedby humaninterpretation.Behindthis,in turn, is the primacywhichis accordedto visualperceptionin our culturegenerally.Seeinghas,in our culture,become synonymouswith understanding.we'look'at a problem.we'see,thepoint.we adopta'viewpoint'. we 'focus'onan issue,we 'seethingsin perspective,.Theworld,asweseeit, (ratherthan'asweknowit', andcertainlynot,aswehearit, or'as wefeelit,) hasbecome themeasurefor whatis'real,and'true,. Sovisualmodalityrestsonculturallyandhistoricallydeterminedstandardsof whatis realandwhat is not,and not on the objectivecorrespondenceof the visualimageto a realitydefinedin somewaysindependentlyof it. At the momenthologramsareprobably
  • 185. r64 Modality still seenby mostpeopleas'morethan real'.In the imageswe are mostusedto,the absenceof thethirddimension,theffatnessof thepicture,doesnotfunctionasanindicator of lowmodality,justastheabsenceof perspectivein cultureswhoseart doesnotemployit doesnotfunctionasan indicatorof lowmodalityfor membersof thosecultures. Aswehavealreadydiscussedin relationto deSaussure's'speechcircuif',however,even withinourownculturethesamestandardsforwhatis'real'andwhatisnotdonotapply in everycontext.In technologicalcontexts,a differentconceptof reality underlies visualmodality,a conceptwecouldcal| 'Galileanreality'.In theearlyseventeenthcentury, Galileowrote: I do not find myselfabsolutelycompelledto apprehendtobjectsJas necessarily accompaniedbysuchconditionsasthat theymustbewhiteor red,bitteror sweet, sonorousor silent,smellingsweetlyor disagreeably....lthinkthat thesetastes, smells,colours,etc.with regardto the objectin whichtheyappearto resideare nothingmorethan merenames.. . . I do not believethat thereexistsanythingin externalbodiesfor excitingtastes,smells,sounds,etc.,exceptsize,shape,quantity andmotion. (quotedin Mumford,t936i 48) Here'real'means'whatcanbe knownby meansof the methodsof science';that is,by meansof counting,weighingandmeasuring.Bythisstandardof whatis real,a technical linedrawing,withoutcolourortexture,withoutliqhtor shade,andwithoutperspective,can havehighermodalitythana photograph.Everydaycommonsensenaturalismandrealism no longermergehere.Therealism(andhencethe'naturalism')of scientific-technical imagesis of a differentkind,based,in the end,on the questions,Canwe useit?,,,Can we measurethe real dimensionsfrom it?', 'Canwe find out from it howto set up the experiment?',and so on. Whateverdoesnot contributeto this purposemerelyaddsa dimensionof illusionism'tothepicture,anddilutes'Galileanrealism'withcommonsense 'naturalism'.Thelatter,of course,is sometimesdonefor the purposeof communicating scientificideasor technologicalcomplexitiesto a publicof non-initiates.In his book writing Biology(1990)GregMyerscomparesthe reportingof the'same'researchfind- ingsin specialistandpopularjournalssuchasScientificAmerican,andvisualrepresenta- tionsinthelattertendto belavish,full-colourand'hyper-real',whileintheformersparse linedrawingsarethe onlyform of visualimage.Furthermore,we haveto beawarethat therearecompetingtheoriesof realityin today'sscience,despitethe fact that for many practicalpurposesGalileanrealityremainsof overridingimportance.Alternativetheories mightleadto differentstandardsfor highandlowmodality. In othercontextsthe'hyper-real'doesnothavethedecreasedmodalityit hasin'photo- graphic'naturalism.Magazinephotosof foodareoneexample.A differentprinciplefor whatcountsasrealoperateshere,theconverseof Galileanreality:themorea picturecan createan illusionof touchandtasteandsmell,the higherits modality.In suchimages everythingisdoneto appealto'sensory'qualities:realityhereisconstitutedpreciselyby thosesensationswhichGalileobrandedasillusions:texture,colouri'feel'.It isherethatthe
  • 186. Modality L65 affectivevaluesof colourscomeintotheirown,for example.Theemotivevalueof colour is sometimesseenas a generalcharacteristicof colour.But in scientific-technological contexts,colourmaybeconventional(moreor lessarbitrary'colourcodes'tofacilitatethe readingof complexdiagrams),andin naturalismcoloursarethere'becausetheyarethere in reality'.Fromthe pointof viewof the 'sensory'definitionof reality,ontheotherhand, coloursare thereto be experiencedsensuallyandemotively- it is for this reasonthat peopleenjoythehighlysaturatedandunmodulatedcoloursof,say,Matisse,or thatchildren enjoythehighlysaturatedandunmodulatedcoloursof theirplastictoys.Withinnaturalism thesecoloursare'lessthanreal',butwithina realismthattakessubjectiveemotionsand sensationsasthecriterionfor whatis realandtrue,theyhavethehighestmodality. Thereis,finally,a third areain whichthe standardof 'photographic,naturalismdoes notapply,theareaof'abstractreaiism'-bothinscience(e.9.the'speechcircuit'diagram infi9ures5.1and5.2)andinabstractart.Highereducationinoursocietyis,to quitesome extent,an educationin detachment,abstractionand decontextualization(and against naturalism),andthisresultsin anattitudewhichdoesnotequatetheappearanceof things withreality,but looksfor a deepertruth'behindappearances'.Justasacademicallytrained personsmay accordgreatertruth to abstractexpositorywritingthan to storiesabout concrete,individualeventsand people,so theymay alsoplacehighervalueon visual representationswhichreduceeventsandpeopleto the'typical,,andextractfromthemthe 'essentialqualities'. Whileour ideasherearedrawnto a largeextentfrom thetheoreticalwork of Jurgen Habermas(especiallyhis Theoryof communicativeAction,1984),andto someextent fromthat of Bourdieu(1986),wewill useBernstein'sterm'codingorientation,(1981) for thesedifferentrealityprinciples.Codingorientationsare setsof abstractprinciples whichinformthewayin whichtextsarecodedbyspecificsocialgroups,or withinspecific institutionalcontexts.Wedistinguishthefollowing: Technologicalcoding orientations,which have,as their dominant principle,the 'effectiveness'of the visualrepresentationas a 'blueprint'.whenevercolour,for example,is uselessfor the scientificor technologicalpurposeof the image,it has,in thiscontext,lowmodality. Sensorycodingorientations,whichareusedincontextsinwhichthepleasureprinciple is allowedto be the dominant:certain kindsof art, advertising,fashion,food photography,interiordecoration,andsoon.Herecolouris a sourceof pleasureand affectivemeanings,andconsequentlyit conveyshighmodality:vibrantreds,soothing blues,andsoon- a wholepsychologyof colourhasevolvedto supportthis. Abstractcodingorientations,whichareusedbysocioculturalelites- in'high'art, in academicandscientificcontexts,andso on.In suchcontextsmodalityis higherthe morean imagereducesthe individualto thegeneral,andtheconcreteto itsessential qualities.Theabilityto produceand/orreadtextsgroundedinthiscodingorientation isa markof socialdistinction,of beingan'educatedperson'ora'seriousartist'. Thecommonsensenaturalisticcodingorientation,whichremains,for thetime being, thedominantonein our society.It isthe onecodingorientationall membersof the ( 1 ) ( 2 ) ( 3 ) G )
  • 187. r66 M odality culturesharewhentheyarebeingaddressedas'membersof ourculture',regardless of how much educationor scientific-technologicaltrainingthey have received. Individualswithspecialeducationor groupallegiancemaydrawon non-naturalistic codingorientationsin certaincontexts,buttheyarelikelyto revertto thenaturalistic codingorientationwhentheyare'just beingthemselves'.Theymay,for example,use the abstractcodingorientationwhenvisitinga gallery,andthe naturalisticcoding orientationwhenwatchingtelevisionor readinga magazine.Forthosewithoutsuch education,however,abstractandtechnologicalimageswill neverhavehighmodality andalwaysremain'unreal'.Today,however,naturalismis comingintocrisis,as a resultof newwaysof thinkingandnewimagetechnologies.In thiscontexttheroleof someor allofthenon-naturalisticcodingorientationsislikelyto becomeof increasing importance. The diagramin figure5.5 showshowthe samecolourcontinuum,runningfrom 'no abstraction'to'full abstraction'(abstractionalwaysbeinga matterof degree)canhave differentmodalityvalues,accordingto the four codingorientations.It is drawnherefor coloursaturation,but it couldalsohavebeendrawnfor anyof theothermodalitymarkers wediscussedintheprevioussection. MODALITYIN MODERNART Theissueof modalitybecomesparticularlycomplexin modernart, becauseit has,to a largeextent,beenthe projectof modernart to redefine'reality',andto do so in contra- distinctionto photographicnaturalism.In thissectionwewill attemptto discussa fewof theissues,beginningwithsomeAustralianexamples. FulI colour saturation (Somewhatlessthan full coloursaturation) Blackandwhite Scientific/technoIogical Abstract Naturalistic Sensory L 0 w E S T + - _ _ _ _ _ _ _ _ _ _ _ MODALITY > H I G H E S T MODALITY (a0ove + - - - - - - - - - + modality) (below maxtmum + - - - - - - - - - + mooailty) L o w E S T + - - - - _ _ _ _ _ + H I G H E S T MODALITY IVIODALITY H I G H E S T + _ _ - _ _ _ _ _ _ +L 0 w E S T M O D A L I T Y M O D A L I T Y H r G H E I I+ - _ _ - _ _ _ _ - _ _ _ L o w E S r IVIODALITY MODALITY @ fig S.S Modalityyaluesof coloursaturationin lour codingoiientations
  • 188. Modality I67 Thepicturein plate3 showswilliam Dobell'sportraitof Joshuasmith,a painting whichwonthe annualAustralianArchibaldPrizecompetitionfor portraitpaintingin 1943.Itwasthefirstmodernpaintingto doso.All previouswinnershadbeenconventional 'academic'portraitists, stayingwellwithinthe boundsof naturalisticdepiction.After Dobellhadbeenawardedthe prize,a numberof conservativepainterstook thetrustees of the prizeto courtfor givingit to a paintingwhich,theyargued,wasnot eligible,as it was not a portrait but a caricature.The prosecutor,GarfieldBarwick,interrogated Dobellabouteverydetailof the painting,askinghimwhetherhe hadfaithfullyrepre- sentedthe ears,the neck,the arms,and so on. Thepainter,in exasperation,answered: 'Yes,within the limitsof arf.' Froma naturalisticpoint of view,the paintingdoesof coursehavecomparativelylowmodality,bothin the directionof 'lessthanreal,andin the directionof 'morethanreal'.Colourdifferentiationis greatlyreduced,to a palette of orange,yellow and brown.The representationof detail, on the other hand, is amplified,exaggerated,'morethanreal'.Froma naturalisticpointof view,theprosecutor wasright.But heapplieda criterionwhichwasno longervalidin the contextof modern art, a way of matchingmodalityvaluesto the scalesof colourdifferentiation,represen- tation and so on which,in the historyof modernart, had beensuccessfullycontested decadesearlier.In modernart, the truth of paintingno longerliesin beingfaithfulto appearances/but in beingfaithfulto somethingelse- for example,to somemodern abstracttruth, in the caseof this rather'expressionist,painting,to .the spirit of the man',and'the essenceof what he lookslike',as Dobellhimselfformulatedit durinqthe trial. Attemptsto alter definitionsof realityare alwayslikelyto producescandal,and thereforeresisted.whetherinthetrialsof D.H.Lawrence,snovelor Dobell,spainting,the issuesarefar largerandmorefar-reachingthanaestheticsor artisticconvention.Changes in the definitionof realityhaveprofoundculturaland socialeffects,and this helpsto explainextremereactiotrssuchasthe proscriptionof'entarteteKunst,or the burninqof books. Despitethetrial,futurewinnersof the ArchibaldPrizewould,withoutexception,be modernartistsratherthanconventionalportraitpainters.Thepictureshownin plate4, a portraitof thewriter Patricl<Whiteby Louisl(ahan,is oneof them.It showsa different modalityconfiguration,a differentsetof abstractionsandamplifications.UnlikeDobell, l(ahandoesnotdeviatefromtlrenaturalisticrepresentationofcletail;ifonedisregardsthe strangelyunseeingeyes,White'sfeaturesare renderedwith naturalisticfaithfulness.But textureis amplified:the figureof Whiteseemsto havebeencarvedout of somepgrous, chalkyrock,andthe renditionof its surfaceis so detailedthat onecanalmostfeelits cold,wettouch- a'sensory'orientation,but in the directionof displeasureratherthan pleasure.colour,ontheotherhand,isgreatlyreduced:inwhatisat oncea punonwhite,s nameanda symbolicgesture,Whiteis drainedof all colour.ThusKahandepictshimas cold,hard,almostrepulsiveto the touch,andthe expressionof this'truth, hastaken precedenceoverthe faithful renderingof outwardappearances;indeed,has become possibleonlybymeansof these'deviations'fromthenaturalisticstandard.Thevisualoun remindsus that modalityis alwaysrelatedto the values,meaningsand beliefsof a
  • 189. 168 Modality particulargroup/inthiscasean'ordering'of thefigureof PatrickWhitewithinthesystem of Australianhighculture,andof Australiansocietygenerally. 0ur secondexamplepertainsto the geometricabstractionismof the 1920s.When Europeanpainters/afterstudyingvisiblerealityfor centuries,beganto conceiveof it as madeupof abstract,geometricalelements(circles,cones/squares,triangles),realitywas redefinedasa configurationof basicelements,justashadalreadyhappened,for instance, in physics.Withinthis newdefinitionof reality,paintersat first still soughtto produce recognizablerepresentations,asshowninfigure5.6.Mondrian,whotriedto painttreesin thisway,complainedthat it wasdifficultto representtreesasarrangementsof rectangular shapes.But soonthesepainterswent a step further and abandonedthe attemptto reconcilethevisiblesurfaceappearanceof thingswiththeirgeometricinnerstructure(see figure5.7).Fromhereitwasonlyonestepto,forinstance,GerritRietveld'sColourProject for theSchroderResidence(t923-4). Rietveld'swork (figure5.8) is no longera reduced,abstractrepresentationof reality, but a designfor a newreality,yetto beconstructed.0f course/blueprintsandplanshad existedalongsidevisualrepresentationlongbeforethe1920s,but in separatedomains.In the twentiethcentury,however,they becameintertwined.Art becameintertwinedwith design,just as sciencehadalreadybecomeintertwinedwith technology.Theboundaries betweenrepresentingrealityand constructingrealitybecameblurred.And when real thingswereproducedfrom designssuchasthese,theprocessesof abstractioncouldcome fullcircleandyield'naturalistic'imagesagain(figure5.9). In theworkof Ryman,a contemporaryAmericanartist,abstractionisperhapstakento its limit.Manyof hispaintingsare,at leastat first glance,whitesurfaces.Everythingis reduced,everythingabstracted.Thereis no colour,no line,no background.And,in terms of our earlierchapters,thereis neitherrepresentationof actionor of socialconstructs, nor yet any indicationof textuality,of composition.Thistruly is the degreezeroof representation. @ fi9S.O Cad-playerc(TheovanD0esburg,lg16-17)(DoesburgArchive,TheHague)
  • 190. o risl'z compositiong(ahstractversionofcad-ptayers)(TheovanDoesburg,rgl6-u)(fromJaff6,1967) @ rig s'a cobur Prciect fot theschrfuderResidence(Gerrit Rietyeld,lgz3.4) (catal0gue81,stedetijk Museum,Amsterdam)
  • 191. I 7 0 Modality , t',1" ' ' -:l'::li:j::t:! Q figS,f PhotographoftheSchrdderresidence('RietYeldHouse')(fromBrown'1958) Butif reductionandabstractionserveto revealotherwisehidden,innertruths,thesame mightbesaidaboutRyman'spaintings.indeed,likesomanyothernon-naturalisticartists, he seeshiswork as realistic,andcallshispaintings'realisticpaintings':theyaspireto presentthe reality and the truth of the processof representationand the processof perception,andtherebyperhapsalsoof thesocialandculturalworld' Thereis,however,another,muchlessabstractfeatureof Ryman'spainting:hisconcern with texture.Hisworkshowsa constantpreoccupationwith the materialsof representa- tion,andwith the materialityof the processesof representation.Someof the paintings leavea patchof the canvasuncoveredand onlythinlycoverthe rest.Othersdisplaya varietyof brushstrokesor,by contrast,completelyde-emphasizethe way in whichthe paint is applied,resultingin totallyffat, presumablysprayedsurfaces.Againothers emphasizelheframe,or the meansbywhichframesareattachedto walls,or theflatness of the painting,by rotatingit throughninetydegrees,and so foregroundingits two- dimensionality. In otherwords,thereisa strongrepresentationalconcerninthesepaintings,but it is a concernwith representingthe processof representation.Doesthlssuggestlow modality, giventhe enormousdistancefrom everydaynaturalism?0r doesit suggestthe highest modality,in whichthe negationof representationformsan ultimatetruth,or in whichthe highestmodalityisaccordedto therepresentationwhichdoesnotrepresentbutsimplyis?
  • 192. Modality As before,our answeris onewhichrefersto the social.Whethera representationis judqedcredibleor notisnotnecessarilyamatterof absolutetruth.Whatonesocialgroup considerscrediblemaynot beconsideredcrediblebyanother.Thisiswhyweseemodality as interactive/ratherthan ideational,as social,ratherthan as a matterof someinde- pendentlygivenvalue.Modalitybothrealizesandproducessocialaffinity,throughaligning theviewer(or reader,or listener)withcertainformsof representation,namelythosewith whichthe artist (or speaker,or writer)alignshimselfor herself,and not with others. Modalityrealizeswhat'we'considertrueor untrue,realor notreal.In thisliessomeofthe powerof art. Tothe extentthat peopleare drawnintothis,we,,newvalues,newmodes of thinkingandperceivingcanestablishthemselves.Andwhenenoughpeoplearedrawnin, theorgansof popularfizing)culturgsuchasadvertising,will quicklymovein to amplify thenewforms,andmovethemintothemainstreamof culture. MODALITYCONFIGURATIONS Theexamplesintheprevioussectionshowthatthemodalityvaluesin arlcan0ecomptex. A paintingcanreducenaturalisminthewayit treatscolour,amplifyit inthewayit treats texture,andyetrepresentitssubjectin a naturalisticway,asin plate4. It canbeabstract in respectof onemodalitymarker,naturalisticin respectof anotherandsensoryInrespect of yetanother,andthisallowsa multiplicityof possiblemodalityconfigurations,andhence a multiplicityof waysin whichartistscanrelateto the realitytheyare depictingand'define'realityin general.In manyotherkindsof images,too,'modalitymarkers,do not moveen bloc in a particulardirectionacrossthe scales,sayfrom the abstractto the sensory/but behavein relativelyindependentways.Most glossymagazinefood photo- graphs,for instance,are highlysensoryin theirdepictionof the food.Thecoloursare intense.Thetextureof the food is shownin sharpdetail.Lightingenhancesthe fresh dropletsofwaterona bunchof grapes,ortheviscosityof a sauce,ortheglazingoftheham andthe cherriesin a pie.But the surroundingobjectstendto havelowermodality.The weaveof thetableclothonwhichthefoodis displayed,for instance,maybeonlyjust be visibleand oftenthe settingis absentaltogether,with the foodshownagainsta black background.In otherwords,suchpicturesarenotonlysensory,theyarealsoabstract.The'sensorily'depicted foodis takenout of its context,idealizedandessentialized.Andthis showsthat eachof the modalitychoicesin sucha modalityconfigurationisexpressiveof specificmeanings,whichthencometogetherinthewhole. From our inventoryof modalitymarl<erswe could construct'modalityprints, (borrowingthe metaphorof 'voiceprint,,'DNAprint,,etc.)to characterizethemodality configurations,andshowwhichmodalitymarkersare reduced,made'lessthanreal,,and whichareamplified,made'morethanreal'- andthiseitherinrelationto ananchoring pointof commonsensehighnaturalisticmodality(asonemightdofor anaudienceof 'lay, peopleat anart exhibition,tal<ingtherepresentationalfunctionof art asa commonsense pointof departure)or in relationto an anchoringpointsituated[n someotherrealism. Figure5.10 isanattemptto showwhatwehavein mind.
  • 193. t72 Modality Colourmodulation Colourdifferentiation Backgroundrepresentation Detailrepresentation Tonality etc. @ fiq S.fO Modalitycontiguration (abstraction).r+ (exaggeration) Suchmodalityconfigurationswoulddescribewhat,in a specificgenreor a specific work/isregardedasreal/asadequateto reality.Andit wouldalsodemonstratethat images arepolyphonic/weavingtogetherchoicesfromdifferentsignifyingsystems,differentrepre- sentationalmodes/intoonetexture.In thisview,a termsuchas'painting'is an artificial constructwhichbringstogetherandtreatsas a homogeneousunit what is in realitya complexconfigurationof differentvoices,differentrepresentationalmodes.(ln the same way it can be saidthat 'grammar'is an artificeof theory,describingwidelydifferent representationalmodes- phonicsubstance,intonation,lexis,syntax,etc.) And it is of coursefrom herethatthe interestingquestionscanbeasked.Arethere,or couldtherebe, socialandhistoricalexplanationsfor thesemodalityconfigurations? Hereis an example,from a sciencetextbookfor the upperyearsof primaryschool, producedin Australia(figure5.11).Thisis a scientific-technicalpicturefor children. As suchit formsa compromisebetweenthe naturalisticand the technologicalcoding orientation,perhapsbecausea'pure'technologicalpicturewouldhavebeenregardedby thewriterasbeyondtheunderstandingof youngchildren.0n theonehand,it isa drawing and not a photographand it lacksa Setting;on the other hand,it usesperspective (angular-isometric),colour(idealized,ffatcolour),andit showsat leastsomethingof light andshade(thoughin a rathersimpleand,in part,inconsistentway),andof texture(the grainof thewood,thetextureof theheadof thenail,thecreasesinthepieceof cloth).The producerof this imageperhapsoperateswith the assumptionthat childrenare familiar with the naturalisticcodingorientation(that is,'wheretheycomefrom') andhaveto be inductedintothetechnologicalcodingorientation(thatis,theprogressionintodisciplinary knowledge).Theimagecapturesthistransitionalphase. Diagrams,mapsand chartsfor lay readersmay be 'naturalized'in similarways. Newspaperdiagramsandmaps/for instance,maybedrawnin perspective(seethe Gulf Warmapin figure4.19).lvlagazinesmayaddcolourandpictorializepiecharts.In com- panybrochuresor annualreports,the barsof bar graphsmaybecomethree-dimensional andrise,likefeaturelessskyscrapers,froma cleanlandscapeof undulatinghillsinstrong, ffat colour.Thisshowsthat modalityis a systemof socialdeixiswhich'addresses'a particularkindof viewer,or a particularsocial/culturalgroup,and providesthroughits systemof modalitymarkersan imageof thecultural,conceptualandcognitivepositionof theaddressee.At the sametime it showsthetransitionacrossandbetweensuchgroups, NATURALISM
  • 194. Modality 173 C) fig s.ff Compass(Jennings,1986) andin doingsodemonstratesthe socialaspectof modality.Mostcrucially,it showshow modalityis motivated,intheclosematchingof modalityandthemodaladdress(location) of specific(andassumed)aspectsof theviewer'ssubjectivity. Figure5.12showsa drawingby Newton,illustratingtheset-upfor oneof hiscolour experiments.Figure5.13 is a modernscientificillustrationshowingthe set-upfor an experimentbyStrattonwhichcausedhimto seehimselfstretchedout inspaceasindicated in the drawing.To moderneyes,Newton'sdrawinghasnot yet advancedveryfar in the directionof hightechnologicalmodality:he uses(inverted)perspective,andshowsthe Setting.Themoderndrawing,bycontrast,leavesoutthesettingandsimplifiestheforms, concentratingon the relationbetweenthem,ratherthan on the representationof the experimenterandthemirrors. As Hallidayhasshown(HallidayandMartin,I993i54-6$, Newton'swritingdidnot yet havethe objective,impersonalstanceand the lexicaldensityof modernscientific writing.At the sametime he madesomedecisivemovestowardsdevelopingthe gram- maticalresourcesthatwouldbecomecharacteristicof scientificwriting.Clearlythesame canbesaidof hisscientificdrawings.Andthat showsthat,howevergreatthe differences betweenthe verbal and the visualgrammar,they derivefrom similar concernsand orientations.
  • 195. @ fig l.fZ Drawingby Newton(BodteianLibrary) Q fiq S.ff Drawingof Stratton'sexperiment(Gregory,1970)
  • 196. 6 T h em e a n i n go f c o m p o s i t i o n COMPOSITIONANDTHE MULTIMODALTEXT In previouschapterswe haveconsideredtheway imagesrepresentthe relationsbetween thepeople,placesandthingstheydepict,andthecomplexsetof relationsthatcanexist betweenimagesandtheirviewers.Anygivenimagecontainsa numberof suchrepresen- tationalandinteractiverelations.In figure6.I,an imagefrom Bergman'sThrougha Glass Darkly(1961),weseel(arin(HarrietAndersson),whosuffersfroman incurablemental disease,andheryoungerbrotherlVinus(LarsPassgard).Fromthepointof viewof repre- sentation,theshotcontainswhatwehavecalleda'non-transactivereaction'(l(arinlooks outof theframe,at somethingtheviewercannotsee)anda'transactivereaction'(Minus looksupat hissister).Thesechoicesrelateto thethemesof thedramaticaction:l(arinhas visions,seesthingsotherpeoplecannotsee;Minusis caughtin the hereandnowof his problematicrelationswith the othercharactersin the film. Fromthe pointof viewof interactivemeaning,the vieweris positionedcloserto l(arin('mediumshot')thanto Minus('1ongshot');and,whileMlnusisseenfrombehind,l(arinfacestheviewerfrontally. ,w'wk, @ figO.f HarrietAnderssonandLarsPassgardinfhroughaGtassDarkt!(Belgman,1960)
  • 197. 1 7 6 Themeaning of composition Clearly,thevieweris meantto bemostcentrallyinvolvedwith l(arin,andwith hermental lurmoil. Thesepatternsdonotexhausttherelationssetupbytheimage.Thereisathirdelement: the compositionof the whole,the way in which the representationaland interactive elementsare madeto relateto eachother,thewaytheyare integratedintoa meaningful whole.Minus,for instance,isplacedontheleft,andl(arinontheright.If thiswereturned around,the representationaland interactivemeaningswould not be affected.l(arin,s reactionwouldstill be 'non-transactive'and Minus,reaction'transactive,,and l(arin wouldstillbeinmediumshot,Minusstillin longshot.Butthemeaningof thewholewould no longerbethesame.In otherwords,theplacementof theelements(of the participants andof thesyntagmsthatconnectthemto eachotherandto theviewer)endowsthemwith specificinformationvaluesrelativeto eachother.Wewill discussthe valueof ,left,and 'right'inthenextsection. In addition,l(arinisthemostsalient,themosteye-catchingelementinthecomposition, notjustbecausesheisplacedintheforegroundandbecausesheformsthelargest,simplest elementin the picture,but alsobecausesheis in sharperfocusandreceivesthe greatest amountof light.Throughoutmuchof thefilm l(arinisdressedin lightcoloursandmadeto bathein light,in an almostsupernaturalfashion,this in contrastto the othercharacters. Forthesereasonssheis alsothe mostsalientelementin theshotswhereoneof theother characters,for exampfeherhusband,isplacedintheforeground.Herwhiteclothesandthe lighton herpalefacedrawattentionto her,evenwhensheis placedinthe background.To generalize,pictorialelementscanreceivestrongeror weaker'stress'thanotherelements intheirimmediatevicinity,andsobecomem0reor lessimportant'itemsof information,in thewhole. A verticallineformedby the left edgeof the doorof the shed,andcontinuedby the dividinglinebetweena particularlylightanda darkerboardontheroofof theshed,runs throughthe middleof the picture,dividingit intotwo sections,literallyandfiguratively 'drawinga line'betweenthespaceof l(arin,whocan'look intothebeyond,,andthespace of Minus,whocannot.Theworldof l(arinisthusseparatedfromtheworldof Minus,in thispictorialcompositionasin thedramaticactionof thefilm asa whole,whereMinus, desirefor contactandcommunionwithhissisterremainsunfulfilled.Thereisyetanother demarcationlinein the picture:the horizon,whichdividesthe pictureintothe zoneof 'heaven'andthezoneof 'earth'.ln hisdiscussionof ritian's Noli Me Tangere,Arnhejm (1982: lr2-r3) describeshowthe staff of christ formsa 'visualboundary,between christ,who is already'removedfrom earthlyexistence',andMagdalen,who is not;and how'thelowerregionisseparatedbythehorizonfromtheupperregionof freespirituality, in whichthetreeandthebuildingsonthehillreachheavenward'.In figure6.1,similarly, l(arinstraddlesthe two zones,half still of the earth,half alreadyin the realmof 'free spirituality',whileMinusis'helddownbythehorizonintotheregionof theearth,.More generally,compositionalsoinvolvesframing(or itsabsence),throughdeviceswhichcon- nector disconnectelementsof thecomposition,soproposingthatweseethemasjoinedor asseparatein someway,where,withoutframing,we wouldseethemas continuousand complementary:therewouldbenovisual'directive,of thiskind.
  • 198. ( 1 ) ( 2 ) 3 ) Themeaning of composition L77 Composition,then,relatesthe representationalandinteractivemeaningsof the image lo eachotherthroughthreeinterrelatedsystems: Inforntationvalue.Theplacementof elements(participantsandsyntagmsthat relate themto eachotherandto the viewer)endowsthemwith the specificinformational valuesattachedto the various'zones'of the image:left andright,top andbottom, centreandmargin. Salience.The elements(participantsas well as representationaland interactive syntagms)aremadeto attracttheviewer'sattentionto differentdegrees,asrealized bysuchfactorsasplacementintheforegroundor background,relativesize,contrasts intonalvalue(orcolour),differencesinsharpness,etc. Framing.Thepresenceor absenceof framingdevices(realizedby elementswhich createdividinglines,or byactualframelines)disconnectsor connectselementsof the image,signifyingthattheybelongor donotbelongtogetherinsomesense. Thesethreeprinciplesof compositionapplynotjustto singlepictures,asintheexamplewe havejustdiscussed;theyapplyalsoto compositevisuals,visualswhichcombinetextand imageand,perhaps,othergraphicelements,beit ona pageor ona televisionor computer screen.In theanalysisof compositeor multimodaltexts(andanytextwhosemeaningsare realizedthroughmorethanonesemioticcodeis multimodal),thequestionariseswhether the productsof thevariousmodesshouldbeanalysedseparatelyor in an integratedway; whetherthe meaningsof thewholeshouldbetreatedasthesumof the meaningsof the parts,or whetherthe parts shouldbe lookeduponas interactingwith and affecting oneanother.It isthelatterpathwewill pursueinthischapter.In considering,for example, thepictureof thetrain(figure3.30)wedonoseekto seethepictureasan'illustration'of the verbaltext,therebytreatingthe verbaltext as prior and moreimportant,nor treat visualandverbaltextasentlrelydiscreteetements.Weseekto beableto lookat thewhole pageasan integratedtext.0ur insistenceon drawingcomparisonsbetweenlanguageand visualcommunicationstemsfromthisobjective.Weseekto breakdownthedisciplinary boundariesbetweenthestudyof languageandthestudyof images,andweseek/asmuch aspossible,to usecompatiblelanguage,andcompatibleterminologyto speal<aboutboth, for in actualcommunicationthe two,and indeedmanyothers,cometogetherto form integratedtexts. In our viewthe integrationof differentsemioticmodesis thework of an overarching codewhoserulesandmeaningsprovidethe multimodaltextwith the logicof its inte- gration.Therearetwosuchintegrationcodes:themodeof spatialcomposition,withwhich wewill beconcernedinthischapter;andrhythm,themodeof temporalcomposition.The formeroperatesin textsln whlchall elementsare spatiallyco-present- for example, paintings,streetscapes,magazinepages.Thelatteroperatesintextswhichunfoldovertime - for example,speech,music,dance(seevanLeeuwen,1999).Sometypesof multimodal text utilizeboth,for examplefilm andtelevision,althoughrhythmwill usuallybe the dominantintegrativeprincipleinthesecases. It followsthattheprinciplesof lnformationvalue,salienceandframingapply,notonly
  • 199. L78 Themeaning of composition to pictures,butalso,for example,to layouts.Plate2,anadvertisementfor Bushellsinstant coffee,containstwo photographsanda smallamountof verbaltext.Thelargerphotoisa pictorialrepresentationof the'prornise'oftheproduct,andit isplacedinthetop section. Thephotoof theproductissmaller,andplacedbelowthe largerphotograph,togetherwith thetext.Reversingthiswouldproduceanentirelydifferenteffect,andprobablyresultin a ratheranomalouslayout.Justwhat informationvaluesthis arrangementaccordsto the two sectionsof the pagewill bediscussedbelow.As far assalienceis concerned,we can notethatthispageisnotdividedintotwo equalhalves.Thetop sectionisthemostsalient, notonlybecauseof itssizebutalsobecausethesalienceof thewoman,whoispositioned ontherightandcatchesmostof thegoldenglowof thelight.Thustheadvertisementgives greaterstressto the promiseof the productthan to the productitself,or the verbal information.Finally,a sharplinecreatesa boundarybetweenthephotoandtheverbaltext, dividingthepageintotwoseparatesections,twospaces/reservedfor twodifferentkindsof meaning- onefor thepromiseof theproduct,enhancedintimacybetweenlovers;theother for theproductitself.Justasthereis a dividinglinebetweenheavenandearthin Titian's Noli Me Tangere,andin thestillfrom Bergman'sThrougha GlassDarkly,sothereis,in thisadvertisementtoo,a dividinglinebetweentheworldof 'whatmightbe',thehappiness the productmight bring,and the world of 'what is', the productitself- and,just as in the two earlierexamples,this product,the jar of instantcoffee,straddlesthe two domainsof meaning,forminga bridgebetweenthem.Thehomepageof Sony'swebsite (http://www.sony.com)hasa similarstructure.Thetop part showsthe pleasurederived fromusingthecompany'sproducts,andwelcomestheuserto the'worldof Sony',whilethe bottompart showsa rangeof actualproductsandallowsthe userto clickon the pages fromwheretheproductscanbeordered. Earlyprintedpagesstilltreatedtextas'visualmaterial'.Walter0n9 (1982:119ff.) describeshowsixteenth-centurytitle pagesbrokeup wordswithoutregardfor syllable boundaries,and useddifferenttypesizesin a way that was not relatedto the relative importanceof words,but servedto createpleasingvisualpatterns.However,the printed pagesoondevelopedintothe'denselyprintedpage'inwhichreadingis linearandtextual integrationachievedbylinguisticmeans(conjunctions,cohesiveties,etc.).In booksof this kindit seemsthat the pagehasceasedto bea significanttextualunit.Thepageshownin plate2,ontheotherhand,isa semioticunit,structured,notlinguistically,butbyprinciples of visualcomposition.In sucha pageverbaltext becomesjust one of the elements integratedbyinformationvalue,salienceandframing,andreadingisnotnecessarilylinea4 whollyor inpart,butmaygofromcentreto margin,or incircularfashion,orvertically,etc. Andthis isthe case,not onlyin contemporarymagazinesandwebsites,but alsoin many othercontexts- for instance,in modernschooltextbooks,aswewill showlater(e.g.figure 6.6). It shouldbenoted,of course,that the layoutof thedenselyprintedpageisstill visual, still carriesan overallculturalsignificancgasan imageof progress.Thedenselywritten pagesof otherculturaltraditionsare laidout differently- as,for examplein the Talmud, whichhastheoldesttext,theMishna,inthecentre,theGemarawrittenaroundit;andlater, medievalcommentariesagain,aroundthe Gemara,in concentriclayers.In suchcases,
  • 200. Themeaning of com?osition ' u9 however,everypageisstill readthesameway.In thecaseof magazinepagesandthepages of moderncomputerscreens,eachsuccessivepagemayhavea differentreadingpath. Thisdevelopmentbeyondthedenselyprintedpagebeganinthe latenineteenth-century masspress,in a contextin whichthe rulingclass,itselfstronglycommittedto thedensely printedpage,attemptedto maintainits hegemonyby takingcontrolof popularculture, commercializingit,andsoturningthemediao/thepeopleintothemediaforthepeople(see Williams1977:295).Theirowncomparablemedla-'high' literatureandthehumanities generally- becameevenmorefirmlyfoundedonthesinglesemioticofwriting.Layoutwas not encouragedhere,becauseit underminedthe powerof the denselyprintedpageas/ literally,the realizationof the mostliteraryandliteratesemioticmode.Thegenresof the denselyprintedpage,then,manifesttheculturalcapital('high'culturalforms)controlled bythe intellectualandartisticwingof themiddleclass,to useBourdieu'sterms(1986). Yetit isthissamesocialgroupwhichhasbeeninstrumentalin spreadingthenewvisual literacyto thosewhowerenot,or notyet,to be initiatedintotheformsof literacywhich constitutedits ownmark of distinction(the'masses',or children),andto embraceit, for examplein 'high'cultureavant-gardemanifestations/asanexpresslonof theiroppos- itionalrolewithinthe middleclassasa whole.As so oftenin thetwentiethcentury,they turnedout,intheend,to havebeensawingoff the branchonwhichtheyweresitting.The distinctionbetween'high'and'low'formsis noweverywherein crisis,and newways of maintainingculturalhegemonyarerequired,for instancethedevelopmentof different and differentlyvaluedwaysof talkingaboutformswhich,themselves,are no longer differentiatedintheoldway(the'discourses'ofdifferent'audiences').Butthemosthighly valuedwaysoftalking(andsemioticsisoneofthem)remainthemselvesboundto methods that cannotadequatelydescribethe newforms.If weareto understandthewayin which vital text-producinginstitutionslikethe media,educationandchildren'sliteraturemake senseof theworldandparticipateinthedevelopmentof newformsof socialstratification, a theoryof languageisno longersufficientandmustbecomplementedbytheorieswhich can makethe principlesof the newvisualliteracyexplicit,and describe,for instance, theroleof layoutintheprocessof socialsemiosisthattakesplaceonthepagesofthetexts producedbytheseinstitutions- aswewill try to doin thischapter. GIVENAND NEW:THE INFORMATIONVALUEOFLEFTAND RIGHT Manyof thedouble-pagespreadsintheAustralianwomen'smagazineswe usedasoneof ourdatasetswhenwewrotethefirstversionof thischapterusethe layoutshowninfigure 6.2.Theirrightpagesaredominatedbylargeandsalientphotographsfromwhichthegaze of oneor morewomenengagesthe gazeof the viewer(what,in chapter4, we called 'demand'pictures).Thesepagesshowwomenin specificandsometimescontradictory roles,withwhichthereadersof themagazineareinvitedto forma positiveidentification:a mother;a former'soapiestar'turnedhousewifeandhappyin that role;workingwomen capableof copingwith'tough','masculine'jobs.Theirleft pagescontainmostlyverbal text,withgraphicallysalientphotographsontheright.Thespreadshowninfigure6.2 hasa
  • 201. Themeaning of composition fD fiS e.Z Gold-diggers(Iustra lian Women'sWeekry,Noyember1987) photographonthe leftalso,butthis photois smallerand,in contrastto the photoonthe rightpage,it isa'ffyonthewall'photograph,whichdoesnotacknowledgethepresenceof the photographer,nor thereforethat of the viewer.It is what,in chapter4, we calledan 'offer' picture.0n suchpagesthereoftenis a senseof complementarityor continuous movementfrom leftto right,asin figure6.2,wherethe photographonthe left istiltedto forma vectorthat leadstheeyesto thephotographontheright,andwherethecolourgold, with itsobviousconnectionsto thethemeof thestory,isusedasanotherintegratingdevice: it occursinthephotographasthecolourof thehelmetsandof theliquidbeingpoured,and is usedalsoasthebackgroundagainstwhichtheverbaltextisprinted. 0n suchpagesthe rightseemsto bethesideof the keyinformation,of whatthereader mustpayparticularattentionto,of the'message'-whetherit isthe invitationto identify with a rolemodelhighlyvaluedin the cultureof the magazineor somethingelse;for example,an instanceof what isto be learnedin a textbook.It followsthat the left isthe sideof the'alreadygiven',somethingthereaderisassumedto knowalready,aspartof the culture,or at leastas part of the cultureof the magazine.In figure6.2, goldminingis Given,andthefact that womencanengagein it, andthat you,the reader,shouldidentify withsuch'tough'women/is New,themessage,the'issue'. Lookingat what is placedon the left andwhat is placedon the right in otherkinds of visualshasconfirmedthisgeneralization:whenpicturesor layoutsmakesignificantuse {i. l:::l
  • 202. Themeaning of composition 1 8 1 of thehorizontalaxis,positioningsomeoftheirelementsleft,andother,differentonesright ofthecentre(whichdoesnot,of course,happenineverycomposition)/theelementsplaced onthe leftarepresentedasGiven,theelementsplacedontherightas New.Forsomething to beGivenmeansthat it ispresentedassomethingthevieweralreadyknows,asa familiar andagreed-uponpointof departurefor themessage.Forsomethingto beNewmeansthat it is presentedas somethingwhichis not yet known,or perhapsnot yet agreeduponby the viewer,henceas somethingto whichthe viewermustpayspecialattention.Broadly speaking,themeaningofthe Newistherefore'problematic','contestable','theinformation "at issue"',whiletheGivenispresentedascommonsensical,self-evident.Thisstructureis ideologicalin the sensethat it may not correspondto what is the caseeitherfor the produceror for the consumerof the imageor layout.The importantpoint is that the informationis presentedas thoughit hadthat statusor valuefor the reader,andthat readershaveto readit withinthatstructure,evenif thatvaluationmaythenberejectedby a oarticularreader. A similarstructureexistsin spokenEnglish(seeHalliday,I9B5:274ff.).As in visual communication,thestructureof a'tonegroup',anintonationalphrase,isnota constituent structure,with strongframingbetweenelements,buta gradual,wave-likemovementfrom left to right (or,rather,from 'before'to 'after',sincein languagewe are dealingwith temporallyintegratedtexts),andit is realizedbyintonation.Intonationcreatestwo peaks of saliencewithineachttonegroup'- oneat thebeginningof thegroup,andanother,the majorone(the'tonic',inHalliday'sterminology),astheculminationoftheNew,attheend. Justasinfigure6.2wehaveonepeakof salienceonthe left,intheboldheadlineandthe redbarwhichseparatesit fromthearticleitself,andanotheronthe right,inthephotoof thetwowomen/sowewouldhaveonepeakof salienceonthesyllablegoldandanotheron thesyllablewo- of womenin: colD-diggingcannowbedonebywomen And just as the imageof the two womenis the Newin figure6.2,sothe word women wouldbethe New,thekeypointof themessage.intheclauseabove.In otherwords,thereis a closesimilaritybetweensequentialinformationstructurein languageandhorizontal structurein visualcomposition,andthisatteststo theexistenceof deeper,moreabstract codingorientatlonswhichfind thelr expressiondifferentlyin differentsemioticmodes. Suchcodingorientationsareculturallyspecific,certainlywherethehorizontaldimension is concerned.In cultureswhichwritefrom rightto left,the Givenis onthe rightandthe Newontheleft,asshowninfigure6.3,wheretheEnglishandtheArabiclanguageversions of Sony'sMiddleEastwebsitearecompared. So far we havetakena compositetext as our example,but the Given-Newrelation appliesalsowithinan image.Figure6.4 showsa fourteenth-centuryreliefdepictingthe creationof Eve.Godis the Given,agreedoriginanddeparturepointof all that exists. 'Woman',onthe otherhand,is Newand,in thecontextof the Genesisstory,problematic, thetemptresswho leadsAdamintosin.Michelangelo,on the otherhand,in hisfamous paintingTheCreationof Adamon the ceilingof the SistineChapel,placedGodon the
  • 203. Themeaning of composition Q riq e.r EnglishandArabiclanguageyersionsot sony'swebsite(http://www.sony-middleeast.com)
  • 204. Themeaning of composition 183 @ fig e.a TheCreationof Eve(LorenzoMaitani,lourteenthcentury)(from Hughes,1969) right,in l<eepingwiththe neryhumanisticspiritof the Renaissance.in thisperiodGod suddenlybecameNew,andproblematic.Generationsof philosopherswereto attemptto redefineHim inwayscommensuratewiththenewscience,andto try to proveHisexistence bytheuseof logic.In thispicturethemovementisnolongerfromGodto'Man',butfrom 'Man'to God.'Man'reachesout,aspiringto divinestatus,andalmostachievingit - but notquite. In magazinelayoutssuchastheoneshowninfigure6.2,thespaceof theGivenisfilled by verbaltext,andthe spaceof the New,or at leasta largepart of it, by oneor more images.Butthisis notalwaysthecase.A double-pageadvertisementfor Mercedes-Benz showed,on the Ieft,a Mercedesphotographedobjectively(ratherthan,for example, fromthedriver'spointof view),andwiththewell-knownMercedesembleminthecentre of the composition.Therightpagecontainedonlyverbaltext,with a headlinesaying, 'Mercedes-Benzagreeswith itscompetitors.Youshoulddrivetheircarsbeforeyoudrivea Mercedes-Benz.'In otherwords,the advertisementtreatedthe Mercedesasan already- known,'Given'symbolof status,andthe messagethat'you,too,mightowna lvlercedes' asthe New.Moregenerally,if the leftcontainsa pictureandthe right is verbaltext,the pictureispresentedasGiven,asa well-establishedpointof departurefor thetext,andthe text containsthe New.If the left pagehastext andthe right pagea picture,the text
  • 205. Themeaning of composition containsthe Given,andthepicturethe New.Theexamplepointsto the socialeffectsand usesof this structure.what is takenfor grantedby onesocialgroupis not takenfor grantedby another.We mightexpectto find,therefore,systematicdifferencesin the dis- positionsof materialin layoutacrossdifferentmagazines- for instance,accordingto their readershio. Theconceptsof Givenand Newcan be appliedalsoto the designof diagrams.In ShannonandWeaver's(1949)communicationmodel(figure2.2) it mightseemthatthe horizontalorderof the elementsis motivatedrepresentationally:the processof'sending information',for instance,musttakeplacebeforetheinformationcanbereceived.Butthe leftdoesnotalwayssignify'before',nordoesthearrowof timealwayspointto theright.A diagramfrom a 1990 issueof TimeMagazinewhichwe werenot allowedto reproduce hereshowed,ontheright,a stickfigurewhoseverylargeheadwasa piechartrepresenting thecompositionof theworkforceintheyear2000(i.e.tenyearsintothefutureat thetime of publication).Anotherpie chart,on the left,was superimposedon a massiveoffice buildingandrepresentedthepresentcompositionof theworkforce.An arrowshowedthat the stickfigurewaswalkingtowardsthe door of the massiveofficebuilding,i.e.that a changein the compositionof theworkforcewasgraduallycomingcloserto the present, but it was not movingtowardsthe right,becausethe currentcompositionof the work forcehadto betreatedas Givenandthe futureadditions(morewomen,minoritiesand immigrants)as Newandproblematic.Thisshowshowthe Given-Newstructurecanbe ideologicalevenin diagrams.If the horizontalorderof the communicationmodelwere rearrangedina similarway(seefigure6.5),it wouldnolongerdepictcommunicationfrom the pointof viewof the'sender',with the'receiver'asNew,andproblematic(Willthe message'hitthe target'?Wifl it havethe intendedeffect?).Instead,the readerwould becometheoriginanddepartureofthecommunicationprocess/andthe'sender'('author') problematic,ashasindeedhappened,for instance,in literaryreceptiontheory. Given-Newstructurescanalsobefoundin film andtelevision.Mediainterviews,for examplgoftenplacethe intervieweron the left of the interviewee(from the viewer's pointof view).Thusinterviewersare presentedas peoplewith whoseviewsandassump- tionsviewerswill identifyandarealreadyf amiliar,indeed,asthepeoplewhoaskquestions onbehalfof theviewers.Theinterviewees,ontheotherhand,present'New,information- and are situatedon the right (seeBelland van Leeuwen,r994i 160-4). The relation betweenGivenand Newmay beemphasizedby horizontalcameramovements('pans'). In a currentaffairsitemfrom an ABC7.30 Report(MarchI9B7),the childrenfrom a MuslimCommunitySchoolwereinitiallyshownas'ethnic','different'from'us,,viewers- @ fig e.S Reversedcommunicationmodel
  • 206. Themeaning of comPosition r85 therewas muchemphaslson their non-Westerndress,andtherewas Arabicmusicin thebackground.But it wasthepointof theprogrammeto establishthattheywere,despite this,'justlikeordinaryAustralianchildren',playful,spontaneous/creative/etc.Thiswas realized,amongotherthings,by varioushorizontalcamerarnovements:a ShotWhich pannedfrom childrenin non-Westernclothesto theteacher,a youngwomanin a Western dress,tyinga bowin thehairof a littlegirl;a shotwhichpannedalonga classroomwall from an Arabicsignto a pictureof a clown,etc.In otherwords,'difference',ethnic prejudice,wastreatedasGiven;thefactthatat leastthesechildrenshouldbeacceptedas 'like us'wastreatedas NeWandformedthe messagethe programmewastryingto get acroSs. In ongoingtexts,eachNewcan,in turn,becomeGivenfor the nextNew.Theopening pagesof thechapter'lnsearchof a straw'fromthe Dutchjuniorhigh-schoolgeography textbookWerk aande Wereld(Bolsef al.,i1986)have,onthefar left,a singlecolumnof text,occupyingabouta fifth of the page,whichhasa'landscape'format,favouringa horizontallyorientedlayout.Thetextcontainsassertionssuchas'ManypeopleintheThird Worldhavenothing'andquestionssuchas'What prospectsdo all thesepeoplehave?' Givenis,inthefirstplace,theThirdWorldasa problem.Theremainderoftheleftpagehas a largecolourphotoof a mansleeplngon the street,coveredby a blanket(thereis no indicationwherethisphotoistaken).Thismoreemotivewayof presentingthe'problem' thereforefunctionsas Newin relationto thetext.Therightpagefeaturesa singlephoto- graph,showinga largecrowdof peoplesearchinga rubbishdump,armedwithcanebags andbaskets.In relationto thisphoto,theimageof thehomelessman(animagenowalso familiarin Europe)becomesGlven,whilethephotoitself,with its (in NorthernEurope) lessfamiliar imageof shockingmasspoverty,is presentedas New.Togetherwith the introductorytext,thetwo imagesconstitutetheGivenof thechapterasa whole.Thuseach newitemof information,oncereceived,becomes,inturn,Givenfor the informationwhich follows,asshowninfigure6.6.Thispatternof the NewbecomingGivenischaracteristicof languagea1so,bothin speechandinwriting. GivenI (ln the Third World peoplehavenothing What can be done?) New1 (thehomelessindividual, instantiatingand dramatizing the given) Given3 O fis O.OCumulativeGiven-Newstructure
  • 207. 186 Themeaning of composition IDEALANDREAL:THEINFORMATIONVALUEOFTOPAND BOTTOM Likemanyothermagazineadvertisementsandmarketingorientedwebsites(seeMyers, 1994: r39), the Bushellsadvertisement(plateD andthe Sonywebsiteare structured alongthe verticalaxis.In suchtextsthe uppersectionvisualizesthe prsmi5se1th. product',the statusof glamourit can bestowon its users,or the sensoryfulfilmentit canbring.Thelowersectionvisualizesthe productitself,providingmoreor lessfactual informationaboutit,andtellingthereadersor userswhereit canbeobtained,or howthey canrequestmoreinformationaboutit, or orderit. Thereis usuallylessconnection,less ongoingmovement,betweenthetwo partsof thecompositionthaninhorizontallyoriented compositions.Instead,thereis a senseof contrast,of oppositionbetweenthe two. The uppersectiontendsto makesomekindof emotiveappealandto showus'whatmightbe,; thelowersectiontendsto bemoreinformativeandpractical,showingus'whatis,.A sharp dividinglinemayseparatethetwo,although,at a lessconspicuouslevel,theremayalsobe connectiveelements.In plate2 thisis createdbythewaythe jarof coffeeformsabridge betweentheupperandthelowersectionof thead,whileintheSonywebsiteit iscreatedby thecolourschemewhichunitesthepageasa whole:inboththetopandthebottompartof thepagethedominantcoloursareshadesof beige,withsomeblueandblue-greyelements added(thejacketof the girl,the picturesin the bottompart of thepage)aswelI assome red elements(e.g.the girl's lips and the words,what,s new,in the top half,and the headingsof thefoursectionsinthebottomhalf). Overall,however,theoppositionbetween topandbottomisstronglyemphasized,withproductsplacedfirmlyintherealmofthereal, asa solidfoundationfortheedificeof promise,andwiththetopsectionastherealmof the consumer'ssupposedaspirationsanddesires. In othercontexts,theoppositionbetweentop andbottomtakesonsomewhatdifferent values.in a fairlyconservativebut(intheearly1990s)stillwidelyusedDutchgeography textbook(Dragtef al.,1986),theupperhalfof thefirstpageof a chapteron,again,.The ThirdWorld',isfullyverbal,presentinggeneralizedassertionsanddefinitionssuchas,A largepartof theworldhasa lowdevelopment'and'Theseunderdevelopedcountrieswe callpoorcountriesor developingcountries'.Thisprovidesa moreneutralandlessemotive (but not lessideological)kindof idealization,a representationof the worldwhichis divestedof contradictions,exceptionsand nuances.The lowerhalf of the pageis given overto a map of the worldwhichusescolour-codingto dividethe world into regions accordingto the averageincomeof the inhabitants,thusprovidingspecificanddetailed evidenceto supportthe assertionsin the top half.Directionsfor action- for instance, couponsfor orderinga productinadvertisements,or assignmentsor questionsintextbooks - alsotendto befoundonthelowerhalfof thepage,usuallyat thebottomriqht(hencealso New). Theinformationvalueof top andbottom,then,canperhapsbesummarizedalongthe followinglines.If, in a visualcomposition,someof theconstituentelementsareplacedin the upperpart,andotherdifferentelementsin the lowerpart of the picturespaceor the page,thenwhat hasbeenplacedonthetop is presentedasthe Ideal,andwhat hasbeen placedat thebottomisputforwardasthe Real.Forsomethingto beidealmeansthat it is
  • 208. Themeaning of comPosition r87 presentedas the idealizedor generalizedessenceof the information,hencealsoas its, ostensibly,mostsallentpart.The Realis thenopposedto this in that it presentsmore specificinformation(e.g.details),more'down-to-earth'information(e.9.photographsas documentaryevidence,or mapsor charts)/or morepracticalinformation(e'g'practical consequences,directionsfor action). As is alreadyevidentfromtheexamplesgivensofar,theoppositionbetweenIdealand Realcanalsostructuretext-imagerelations.If theupperpartof a pageisoccupiedbythe textandthe lowerpartbyoneor morepictures(or mapsor chartsor diagrams),thetext plays,ideologically,the leadrole,andthe picturesa subservientrole(which,however,is importantin its ownway,asspecification,evidence,practicalconsequence/andsoon)'If the rolesarereversed/sothatoneor morepicturesoccupythetop section/thentheldeal, theideologicallyforegroundedpartof themessage,iscommunicatedvisually,andthetext servesto elaborateon it. As withthe Givenand NeWthe ldeal-Realstructurecanbeusedin thecomposition bothof singleimagesandof compositetextssuchaslayouts.Figure6.7,reproducedfrom oneof the Dutchgeographytextbool<swe havediscussed(Bolse/ al.,1986),includesa photowhichmayhavebeentakeninIndia- itsoriginisnotmentioned,butto theleftofthe picture,asitsGiven,weseea mapof india.A youngmother,carryinga bab, occupies/by herself,thetopsectionof theverticallycomposedphoto,asa'ThirdWorld'Madonnawith *"w*t ffi*'$*i*'- #it:kffi; l:J.r'S,-;:.-.1 ffi.ffiffit: @ figO.Z overpopulation(Bolsef ar.,1986)
  • 209. I88 Themeaningof composition child.Thebottomsectionshowsa groupof womenandchildren,sittingon the ground, tightly packedtogether.The youngmotherlooksat this group,a worriedexpression crossingherface.In thiswaythepictureasa wholeexpressesa contradictionbetweenthe deep-rootedIdealof motherhoodandthe Realof overpopulation.Immediatelybelowthe photowefinda collageof newspaperheadlines('lndiastrugglesagainstoverpopulation,,'Unemploymentnightmarein India')as Real(thenewspaperassourceof ,hardfacts,,of evidence)with respectto the moresymbolic,idealizedandemotiverepresentationof the probleminthepicture. Idealand Realcan alsoplaya role in diagrams.It is striking,for instance,that diagramsbasedona verticaltimelinesometimesidealizethepresent,sometimesthepast. Thealready-mentionedDutchgeographytextbookWerkaande Wereld(Bolsetal.,1986) featuresa diagramwhichrepresentsthedecreaseof livingspaceperheadof the popula- tion,by meansof a verticalarrangementof what looklikechessboardsof differentsizes. 0n these'chessboards' standcartoonfigures.0n top we seea gentlemanfrom 1900, completewithtop hat,ona large('6285m2')'chessboard'.At thebottom,onthesmallest'chessboard', we seea'punk' characterfrom 19g0.Here,as in manyadvertisements, the past,the 'goodold days',is presentedas Ideal.TheotherDutchgeographytextbook we mentioned(Dragtef al.,1986)featuresa 'geologicalcalendar,in whichthe present ('developmentof vertebrates',completewitha smalldrawingof a nakedwoman)becomes theIdeal,theculminationof progressandevolution. Manyvisualscombinehorizontalandverticalstructuring.In figure6.g (asin figure 6'4) Godis Given,andAdamandEveare New.Buttheirfall fromgracehasintroduced a (New)opposition,betweentheIdealof Paradise,of theGardenof Eden,andthe Realof deathanddecay- andthe two are visuallyseparatedby the riverwhichsurroundsthe Gardenof Eden. Thecommunicationmodelin figure6.9 alsocombineshorizontalandverticalstruc- turing,inanintricatepieceof visualthinkingabouttheimpossibilityof knowingreality'as it is',objectively.Givenisthe'event',asit exists'out there,,separatefrom our perception of it. New,andthereforeproblematic,isourperceptionof theeventand,at the lowerlevel, thewaywe communicateour perceptionsthroughlanguage.Idealis the'empirical,,the world'asit is',andourperceptionof it,unmediatedbycommunication,culture,language (whicharepositionedin the lowersection).Realareour interpretationsof thesepercep- tions,as mediatedby communication.Clearly,thisdiagramcouldhavebeenvertical,or horizontal.Butit isnot.Communicationispositionedbelowthe,event,anditsperception. The'empirical'worldand'pure'observationareIdeal.ButthisIdealis alsodepictedin isolationfrom our'statements'abouti! andour perceptionsof thesestatements.Thisis whatthe lowersectionof thediagram,the Real,tellsus.Perceptionissecondhand,filtered throughcultureand language,which,as the double-headedarrowsindicate,feedback intoourperceptionof nature,andhenceintonatureitself.Thediagramtellsusthat reality doesexist,but that our perceptionof it can onlybe 'subjective, selective,variableand unpredictable'(McQuailandWindahl,1993:25). Figure6'10 is anotherone of our originalexamplesfrom late t980s Australian women'smagazines,but it remainsa goodexampleof the combinationof horizontal
  • 210. Themeaning of composition ' I89 O fis t.A GottShows Deathto Adamand Eye (French,titteenlh-century miniature from ms'ol De civitate ,eD (from Hughes' r969) andverticalstructuring.Idealisthe moment,onemightsay,that 'marriagewasmadein Heaven'.Modalityis'distant',representingthe'not now',the'out of time'.Thebottom section,bycontrast,representstheworldof is','now','inourtime'. Givenis the royalcouple,presentedasthe quintessentialcouple,the well-established symbolof family values.New are Sydney'sGwenand Ray l(inkade,an instance,an exampleof thesevalues.Hencewhatis Givenis the pre-eminence(historically,socially, semiotically)of theroyalcoupleastheparadigmexampleof themarriedcouple.Whatis Newisoneinstanceof theparadigm- wheremanyothersof a setof acceptableinstances wouldhaveservedequallywell.Thisdistinctionisperhapssharpestinthebottompart.The two picturesinthetoppartbecome,at thesafedistanceof fortyyears,almostidentical,an eouationbetweeneoualterms. In one,anda veryrealsense,the placeof the Newseemsmerelyperfunctory:it isthe placeof the replicationof theparadigm,of the reproductionof theexistingclassifications oftheculture,theplacewheretheunderlyingvaluesoftheculturearereaffirmed.TheNew
  • 211. 190 Themeaning of composition meansandcontrol (orcommunicating) dim!nsion lQ fiq O.e Gerbner,scommunicationmodel(Watsonand Hilt,t9g0: 7i) instantiatesand 'naturalizes'these values.But that veryfact alsomakesthe position problematic/for it isat thesametimetheplaceof theaffirmationof whatis,theplaceof thereproductionof socialmeanings,andtheplacewherethecontestationof paradigmatic valuescantakeplace,the placethereforeof the constantproductionof socialmeanings (e.9.of newdefinitionsof 'women'swork'infigure6.2),evenwhenthatproductionseems to be merereproductionandhenceconservativein its effects.Couldthere,for instance, havebeena Vietnameseor Lebaneseor AboriginalcoupleinthispositionOnI9B7),notto mentiona gaycouple?Thiscontestationover,established,,,Given,valuesmayhappenin one0r twoways:a readerwhoisnotAnglo-Australianwill eitheridentifywiththe syntagm of Anglo-Australianness,'assimilate',in otherwords;or will refusethesyntagmashaving norelevanceor valueto himor her.In thelattercasetherewill bepressureonthisplacein thesyntagm,andthisinturnwill resultinpressureontheparadigmasa whole. Thereis anotheraspectto this;whilethe syntagmdeclaresitselfas unquestionably established,itsappearancepointsat thesametimeto a problemwiththeparadigm,to the needpreciselyfor a testingandke)affirmationof its legitimacy.Readfromthe rightto the left,the syntagmdeclaresthat it is thewillingnessof readersto readit as a relationof identity(withina hyponymicstructure)whichgiveslegitimacyto theroyalcouple.Royalty is the established,the Given.What hasto be reaffirmedanewis that subjectsare still preparedto enterintothis paradigmaticrelation.A monarchytryingto establishitself, on the otherhand,mightneedto utilizea structurewherethe powerof the peopleis representedas Given,andthe identityof the monarchis to beestablished- that is.the royalcouplewouldappearontheright.
  • 212. Themeaning of composition 1 9 1 'r/{;rgugry ttrifl.#{tys#,y'lf rJ ${} 'f h*i'r* f relr dilr'.r$if srrkj*. hut tl* lorg a*l rt*ngtlr ir rlif,if nrfila$es d* !l!: lafia , loesft&rX!,1t{ti I Prihoe* eiirrbeit srd I Klip Marntb'ln sk I (*ltd$idg4e{hand' lsm qd.e* HadE& rbt! n6lh4.hrod aMn! @e. rhd )o!. !nglD.dr tunrd Pd.ce Phllo. ond Srdq'3 Gsn 6d FryrtudcSdo d dt ,oddln! sn.lvaBry - rd lolt iod !o(t .n 40ts6 oltupll|* @ fiq e.fO Royalcouple(Australian Women,sWeekty,Nouember1987) Thusthis syntagmrevealsa numberof socialfacts:what is regardedas established andGiven;whattheculturalclassificationsystemiswith respectto a certainfeature,'and whetherthesystemisprogressiveor reactionary.It isaboveall a syntagmwhichdoesnot permitdeviance;or,rather,oncean itemis inthesyntagm,it hasto bereadasbeinginthe paradigm.Whereit doespermitdevianceisonthepartof the reader,whocanrefuseto be partof thecommunitydefinedbythisparadigm. In the Westernvisualsemiotic,then,the syntagmaticis the realmof the processof semiosis,andthe top-bottomstructurethe resultand recordof semiosis,the realmof
  • 213. 192 Themeaning of composition order,theparadigm,the mimeticrepresentationof culture(Hodgeandl<ress,19gB).To maintainandunsettletop-bottomstructures,onehasto workonthe left-rightstructures. That this systemgoesbacka longway in westernart can be seenin genressuchas fifteenth-centuryFlemishdiptychs,which,for instance,mayhavetheVirginandthe child as Given,anda donoror Saintas New,as in the diptychbythe masterof Brugesin the courtauldGallery,and polyptychsfrom the sameperiod,whichmay parallela Real (earthly)andIdeal(heavenly)versionofthesamethemeinthelowerandupperpartofthe panels,asin Bosch'sLastJudgement,wherethe lowerpartof oneof the leftpanelsshows AdamandEvebeingdrivenfromtheGardenof Edenandtheupperparttheexpulsionof the RebelAngelsfromheaven. As we havesaidin the Introduction,we are largelyconcernedwith the descriptionof the visualsemioticof Westerncultures.Cultureswhichhavelong-establishedreading directionsof a differentkind(rightto left,or top to bottom)are likelyto attachdifferent valuesto thesepositions,asshownin figure6.3.In otherwords,readingdirectionsmay bethe materialinstantiationsof deeplyembeddedculturalvaluesystems.Directionality as such,however,is a semioticresourcein all cultures.All cultureswork with margin andcentre,leftandright,topandbottom,evenif theydonotalI accordthesamemeanings andvaluesto thesespatialdimensions.And the waytheyusethemin their signifying systemswill haverelationsof homologywith otherculturalsystems,whetherreligious, philosophicalor practical. WewillendthissectionwithonefurtherexampleoftheusesofGivenandNew,theway in whichRembrandtusedGivenandNewfor theexpressionof affectiveaspectsof mean- ing,andthisespeciallyin relationto the sourceand directionof lightandthe effect producedby that. In many,perhapsthe majority,of Rembrandt,spaintings,whetherin landscapessuchas Landscapewith a stone Bridge or in portraits such as A young Womanin Bedor DoublePortrait of the MennonitePreacherCorneliusClaeszAnsloand his WifeAeltie GerritsdrSchouten(figure6.71),the lightsourceisoutsidethe left frame of the pictureandilluminatesmainlythe left part,leavingthe rightof the paintingin greateror lesserdarkness.Iconographicallyspeakingthemetaphoricrangeof lightiswide - lightcansignify'thedivine','illumination','hope',etc.In thesepaintingslight,whatever its meaning,is in the areaof the Given,thetakenfor granted,thenow/present.,Light,is Given,'darkness'New. Theheightof theunseenlightsourcealsovaries:in YoungWomanit isonor justbelow the centre;in DoublePortraitit comesfrom a positionabovethe halfwaymark,perhaps two-thirdsof thewayup;andin Landscapeit comesfromsomewherehighup,nearthetop cornerandinthenearmiddledistance.Thatis,lightmaybeintheareaoftheRealcoming from a'mundane'sourceor it maybe'divine,.In otherpaintingsthe lightcomesfrom withinthepainting;for instance,in TheHolyFamilyontheFlightto Egypt,whereit forms the(divine)light'intheworld'(thereisalsoa second,faintlightcomingfromoutside,in the skyabove).In Belshazzar'sFeast,by a mostunusualcontrast,thelightsource(the glowingscriptannouncingthedoomof theking)issituatedinthetop-rightquadrant- the spaceof the Newandthe'Ideal?divine'.Thevariationin the sourceanddirectionality of lightthushasa complexsetof meanings.It cancontrastthesecular/mundaneandthe
  • 214. Themeaning of composition 193 O fis O.ff Douhle Portmit of the Mennonite Preachet Cornelius Claesz Anslo and his Wile Aeltje Gerritsdr Schouten (Rembrandt,1641)(StaatlicheMuseen,PreussischerKulturbesitz,Gemaldegalerie,cat.no.828L) divine/ideal;lightasGivenandtakenfor granted,andlightasNewandastonishing:andall thesein variablecombinations.In DoublePortrait,for example,the light comesfrom outsidethe depictedworld,is situatedin theareaof the Given(theareawherethescrip- turesaredepicted),andcomesfromjustabovethemidwaypointbetweenIdealandReal, sothatit couldbeinterpretedas'divine',yetcloseto the Real.0neoverwhelmingeffectis thebrightnessof theareaof the Given,andthetotal darknessof theareaof the New(the future?)to whichthetwo figureshave,in anycase,turnedtheirbacks.Areweentitledto readfromthisautobiographical,affectivemeanings- perhapsa deep,pervasivepessimism aboutboththe future,the New,andthe present,the Real,whichthencontrastswith a feelingof securityaboutwhatwas,a faith in a divinelightfromthe pastcertainty,which entailsthat we mustturn our back on the New,on the future?If so,theseaffective, personalmeaningsaresurelyassignificantassocialandculturalmeaningsand,of course/ relatedto them.
  • 215. L94 Themeaningof composition THEINFORMATIONVALUEOFCENTREAND MARGIN Visualcompositionmayalsobestructuredalongthedimensionsof centreandmargin.The mosttypicalmanifestationsof thiscanbefoundin children'sdrawingsor,for example,in Byzantineart.AsArnheim(1982:73)notes, In theByzantinechurchesthedominantimageof thedivinerulerholdsthecentreo{ the apse.In portrait paintings,a popeor emperoris oftenpresentedin central position.Moregenerally,whentheportraitof a manshowshimin themiddleof a framedareatwe seehim detachedfrom the vicissitudesof his life'shistory,alone with his own beingand his ownthoughts.A senseof permanencegoeswith the centralposition. Figure6.I2 isanexample- a Buddhistpaintinginwhichthecentralfigureissurrounded by a circleof subordinates.Arnheimin fact makesthecentrethecrucialelementof his theoryof composition,conceivingofthevisualobjectsina compositionas'somanycosmic bodiesattractingandrepellingoneanotherinspace'Q9B2:207). In contemporaryWesternvisualizationcentralcompositionis relativelyuncommon, thoughheretoo theremaybechangesin train. Mostcompositionspolarizeelementsas O fiq O.fZ Burldhistpainting(Arnheim,1982)
  • 216. Themeaningof composition 195 GivenandNewand/orIdealandReal.Butwhenoneof uswasteachingona mediadesign coursein Singapore,hefoundthat centralcompositionplayedan importantrole in the imaginationof youngAsiandesigners.Perhapsit is the greateremphasison hierarchy, harmonyand continuityin Confucianthinl<ingthat makescentringa fundamental organizationalprincipleinthevisualsemioticof theirculture.Muchof theworkproduced by thesestudentshad strongdominantcentres,surroundedor ffankedby relatively unpolarizedmarginalelements. WhilemanyAnglo-Westerntabloidnewspaperstendto adhereto a basicleft-right structureinthe layoutof theirfrontpages,othersplacethemainstoriesandphotographs in thetop section.Thefront pagesof the businesssectionsof the SydneyMorningHerald, however,for a time invariablyusedcentralcomposition/featuringa large photo (or, frequently,drawing)in the centreof the page:for instance,Asianstudentsenteringthe neo-GothicQuadrangleof the Universityof Sydneywhenthe pagefeaturedarticleson educationasa moneyearnerforthecountry'seconomy;a cartoon-lil<edrawingoftwo men playingMonopoly(basedon Van Doesburg'sCard-players-seefigure5.6),whencor- poratetakeoversdominatedthenews;andsoon.Suchpicturesprovideda symbolickernel fortheissuesoftheday,anda centrefortheelementsarrangedaroundthem- newsstories at thetop andto the left,as,still,the ldealandthe Givenof the newspaper,evenif now somewhatmarginalized;advertisementsasthe Real;anda columnof expertcommentary asNew,henceastheelementto whichreadersshouldpayparticularattention.Figure6.13 Q fig e.ff Goingon holiday(from Prosser,2000:ll7)
  • 217. 196 Themeaning of composition showsa diagramfroma tourismstudiestextbookin which'goingon holiday'isthecore issue,andinwhicha rangeof reasonsfor goingonholidayisarrangedaroundthisCentre, withoutanysenseof polarization. Togeneralize,then,if a visualcompositionmakessignificantuseof the Centre,placing oneelementin the middleandthe otherelementsaroundit. we will referto the central elementasCentreandto theelementsaroundit asMargins.Forsomethingto bepresented as Centremeansthat it is presentedasthe nucleusof the informationto whichall the otherelementsare in somesensesubservient.TheMarginsaretheseancillary,dependent elements.In manycasesthe Marginsareidenticalor at leastverysimilarto eachother,so thatthereisnosenseof a divisionbetweenGivenandNewand/orIdealandRealelements amongthem.In othercases- for instance,thenewspaperpageswediscussedabove- Centre andMargincombinewithGivenandNewand/orIdealandReal. Not all Marginsare equallymarginal.Circularstructurescancreatea gradualand gradeddistinctionbetweenCentreandMargin,asfor instanceinthecommunicationmodel byAnderschet al. infigure6.I4, wheretheprocessof 'structuring'is moremarginalthan theprocessof 'evaluating'.In thismodel,nature(the'environment')is Centre,originand primemoverof communication.Comparedto thedominantpositionof nature,communica- tion is a marginalphenomenon,just as,in the medievalmapsof citieswe discussedin chapter3,thecitiesthemselveswereplacedinthecentreanddepictedwithtopographical -i't'i **)W,'*"91 @ figO.f+ Anderscheta/.'scommunicationmodel(fronWatsonandHill,lgS0:14)
  • 218. Themeaning of composition 197 accurac,whilethesurroundingcountrysidewasrepresentedona smallerscale,andwith lessaccuracy.Verbalcommentarlesdo not necessarilytry to 'translate'suchmeanings. WatsonandHill(1980:76),forexample,saythatinthismodelthe'message'is'interact- ingwithfactorsin theenvironment'.Yet,the modelitselfrepresentsthe relationbetween communicationandthe'environment'/notasinteraction,butasa one-wayprocess/a'non- transactivereaction',accordingto our terminologyin chapter2 (thereis an arrowonly fromthe'environment'tothecommunicativeprocessesthatsurroundit).And'interacting' suggestsgreaterequalitybetweenthe'message'andthe'factorsintheenvironment'than doesthecentredcompositionof themodel. As we haveseen,Given-NewandIdeal-Realcancombinewith Centreand Margin. Dividingvisualspaceaccordingto thesedimensionsresultsin thefigureof the Cross,a fundamentalspatialsymbolin Westernculture(seefigure6.15).Justhowmarginalthe marginsarewill dependonthesizeand,moregenerally,onthesalienceof the Centre.But evenwhenthe Centreis empty,it continuesto existin absentia,asthe invisible(denied) pivotaroundwhicheverythingelseturns,the placeof the 'divineruler'.Therelative infrequencyof centredcompositionsin contemporaryWesternrepresentationperhaps signifiesthat,inthewordsof thepoet,'thecentredoesnothold'anylongerin manysectors of contemporarysociety. Onecommonmodeof combiningGivenandNewwlthCentreandMarginisthetriptych. In manymedievaltriptychsthereis no senseof Givenand New.TheCentreshowsa key religioustheme,suchasthe Crucifixionor theVirginandChild,andthesidepanelsshow Saintsor donors,kneelingdownin admiration.Thecompositionis symmetricalrather thanpolarized,althoughthe leftwasregardedasa slightlylesshonorificposition.In the sixteenthcenturyaltarpiecesbecomemorenarrative,showing,for instance,the birth of Christor theroadto Golgothaintheleftpanel,theCrucifixiononthecentrepanel,andthe Resurrectionontherightpanel.Thiscouldinvolvesomepolarization,albeitsubordinated to thetemporalorder,with the leftasthe'badside'(e.9.thetransgressionof Adam),the Q rig O.fs Thedimensionsotvisualspace
  • 219. 198 Themeaningof composition rightasthe'goodside'(e.9.the Resurrection)andthemiddlepanelrepresentingChrist's roleas MediatorandSaviour(e.9.theCrucifixion).Bosch'sLastJudgement(e.ndalsohis EarthlyDelightl invertsthis,showingonthe leftthe Gardenof Edenandonthe righta cataclysmicvisionof Hellinwhichthereisnoplaceforthe'ascentof theblessed'. Thetriptychsin modernmagazinesandnewspaperlayoutsaregenerallypolarized,with a'Given'left,a'New'right,anda centrewhichbridgesthetwoandactsas'Mediator'.In August2004,thetop bannerof Nokia'swebsiteshowedonthe leftan imageof a fashion- ablewomanandon the righta Nokiaimagingphone.Thetext in the Centreconnected thetwo.In fact it consistedof two alternatingtexts.Firstwe read'lnspiringly.Welcome to LondonFashionWeek',thenthisfirsttextmadewayfor 'lnspiringlydifferent.Thenew officialimagingphoneof LondonFashionWeek'.Theconceptof fashionis Given,andthe Nokiaimagingphoneasa fashionaccessoryNew. Triptychscanalsobe usedto structurediagrams.Iedemaet al. (1994:217)shows howthe left columnof the organizationalchartof a localcouncilliststhat council's 'CorporateServices',so makingthe 'administrativeand financialbackboneof the organization'Given,whilethe council's'Developmentand EnvironmentalServices',the departmentwhich'connectsall otherdepartments',isdescribedinthecentralcolumn,as the Mediator.In a lectureon socialcognitionattendedbythe authors,the lecturerused theblackboard(conventionalblackboardsalsohaveatriptychstructurel)to list,onthe left panel,a numberof keyissuesin linguistics(thiswasGivenbecausemostof those presentwerelinguistsandstudentsof appliedlinguistics),on the rightpanela number of issuesin sociology(thiswas New,asthe linguistsweremeetingto discussthe social relevanceof discourseanalysis),andonthecentral'panel'anoutlineof hisowntheoryof socialcognition,whichhepresentedasthenecessarylinkbetweenthetwofields,andasan issuethatshouldbethecentralconcernof thosepresentat themeeting. Verticaltriptychsare alsocommonin websites.Thetriptychfrom the Universityof 0xfordwebsite(figure6.16)canbeinterpretedasa simpleMargin-Centre-Marginstruc- ture,thoughthereissomepolarizationinthatthetopimageisa'longshot'andthebottom imagea closeup.0verall,however,thestudentisCentrehere,whileimagesof historyand traditionsurroundandsupporther.Thetriptychinfigure6.17comesfroma Germanjunior high-schoolpoliticstextbook(Nitzschke,1990).As the Ideal,we see(in colour)immi- grants(Ausltinder,'foreigners')in high-statusprofessions.Asthe Real,wesee'foreigners' in low-statusprofessions.ThisRealisdividedintoa Givenanda New,with a colourphoto as Givenanda black-and-whitephotoas New,asthough,in the 1990s,the lowstatusof immigrantsshouldbelookedat in a moresoberlight,andno longeras'Given'asit once was.In the Centrewesee,againin blackandwhite,a singleimmigrantworkercleaninga train.Theaccompanyingtextencouragesstudentsto explorewhatwouldhappenif 'one dayall foreignworkershadto leave'.What,it asks,wouldbethe consequencesfor the buildingindustry,thechildrenof theworkers,theownersof hostels,theworkersthemselves, themanagersof hospitalsandcleaningfirms?In otherwords,thistriptych(itselftheNew on the doublepagein whichit appears)tellsusthat foreignworkersshould,perhaps, ideallybe ableto moveintohigh-statuspositions,but in realityare neededto do'our' menialjobs.The centralimageis an attemptto overcome,or at leastmitigate,this
  • 220. Themeaning of composition 199 @ fig O.fOVerticaltriptychfromthe Universityof 0xfordwebsite(www.oxford.ac.uk) contradiction.It showsa workerwho,likethe high-statusimmigrantsin the ldeal,is depictedasan individual,andas involvedin'clean'work,butwho,alsoliketheworkers showninthe Real,hasa low-statusjob* andisshowninthesober,documentarymodality of blacl<-and-whiterealism. Thestructureof the triptych,then,canbe eithera simp[eandsymmetricalMargin- Centre-Marginstructureor a polarizedstructureinwhichtheCentreactsasa Mediator betweenGivenandNewor betweenIdealandReal(seefigure6.18). In this and the precedingsectionof this chapter,we havenot drawnany parallels with language.Thoughspol<enEnglishhasits ownGiven-Newstructure,thisis notthe casewith the ldeal*Realand the Centre-Marginstructures.This is not to say that the meaningsthesestructuresexpresscannot,in someform,be expressedin language, but rather that they are more readilyand frequentlyexpressedvisually,and that language,unlikevisualcommunication,hasnotdeveloped'grammatical'formsto express them.Aswehaveemphasizedthroughoutthisbook,sometimeslanguageandvisualcom- municationexpressthe samekind of semanticrelations,albeit in very differentways, but thereare also manytypesof semanticrelationwhich are more oftenand more easilyexpressedvisually,just asthereare otherswhichare moreoftenand moreeasily expressedlinguistically,withepistemologicalconsequencesof thekindwediscussedinthe introductionandchapter1.
  • 221. 200 Themeaning of composition { wt"ui"l" xinder ausliin- dircher[itern lind in eufer Sc"r,le? Fragtsie,seit wannihre Elternin Deutschland5ind,wassiearbea- ten warumsiehierhdrkamen,ob sie bleibenwdhn undwarum. Ste$tim Atlasi:st woherausl{n- discheArteiter komrnen,rioher Fliidrdinge.taBt euchenihlert wie esdort aussieht"Lestin Ed- kundebiichemoder Reiseliihrern rach. Achtetdabeibesonder auf die Arbeits-undLebenssit'ratisn. If n grr*a,iut eurerSdlule gibt esZahleoiiber ausliindis<ire fthtiler Vorschlligezum Cesprach: . Cibt esProblememit ausliin- dischenSchiilerinnenund Schii- lern{ o WelcheLdsungsmdglidrkeiten sehtihr? o W.s denkenausliindische Schiler tiber die Eundesrepublik? o Was denkendeutscheSchiiler iiber Auslinder? /l stettt euclrvor,da8 an einsmTag!alleausliindisclcn Arbeiter die Fuodesrepublikver- lassenmii8ten.EildetGruppen undiiberlegt wase.B. o Bauunternehmer, o Kinderder Arbeiter. o caststiittenbesitzer, o die Arbeiterselbst o Leitervon l*artl<enhiius!m, o LeitervonReinigungsfrnenfiir ProHerneUithn, trn miitlten ... Tragt die lrgebdsse zusanrnen und besprerl* sle. f) fiq e.fZ Verticaltriptychfrom a Germanschooltextbook(Nitzschke,1990)
  • 222. Themeaningof compositton 207 Given Mediatof @ fiq e.fe Horizontalandverticaltriptychs SALIENCE Thefundamentalfunctionof integrationcodessuchascompositionistextual.Integration codesserveto producetext,to placethemeaningfulelementsintothewhole,andto provide coherenceand orderingamongthem.So far we havediscussedhowcompositiondeter- mines'wherethingscango' andhowthe positioningof the elementsin a composition endowstheseelementswith differentinformationvaluesin relationto otherelements. But the compositionof a pictureor a pagealsoinvolvesdifferentdegreesof salienceto its elements.Regardlessof wherethey are placed,saliencecan createa hierarchyof importanceamongthe elements,selectingsomeas more important,moreworthy of attentionthan others.TheGivenmaybemoresalientthanthe New,for instance,or the NewmoresalientthantheGiven,or bothmaybeequallysalient.Andthesameappliesto IdealandRealandto CentreandMargin. Thesamephenomenonoccursin temporallyintegratedtexts.Rhythmalwaysinvolves cycleswhichconsistof an alternationbetweensuccessivesensationsof salience(stressed syllabies,accentednotes,etc.)andnon-salience(unstressedsyllables,unaccentednotes) andthesecyclesrepeatthemselveswiththetimeintervalsthatareperceivedasequaleven when,measuredobjectively,theyarenot.Theperceptionof salience,in speechasin music, resultsfroma complexinterplaybetweena numberof auditoryfactors:thedurationof the strongandweakelementsofthecycle('long'-'short'),thepitchofthestrongandtheweak elements('high'-'low')theirloudness('loud'-'soft'),andin speechalsothevowelcolour (vowelsmaybefully pronounced,for instancethefirst'e' in element,or pronouncedasa 'schwa',likethesecond'e'inelement,or thesecond'a'inalabaster).Indeedanythingthat cancreatean auditorycontrastbetweensuccessivesoundscanserveto realizesalience. Andevenwhenobjectivecluesfor salienceareabsent,thefirst elementof eachcyclecan beperceivedas'stronger':perceptionimposesrhythm,wavesof salienceandnon-salience onsound(andonmovement)evenwhen,strictlyspeaking,thereisnone.
  • 223. 202 Themeaning of composition Whencompositionis the integrationmode,salienceis judgedon the basisof visual clues.Theviewersof spatialcompositionsareintuitivelyableto judgethe'weight'of the variouselementsof a composition,andthegreatertheweightof anelement,thegreaterits salience.Thissalience,again,is not objectivelymeasurable,but resultsfrom complex interaction,a complextrading-offrelationshipbetweena numberof factors:size,sharp- nessof focus,tonalcontrast(areasof hightonalcontrast- for instance,bordersbetween blackandwhite- havehighsalience),colourcontrasts(for instance,thecontrastbetween stronglysaturatedand'soft'colours/or thecontrastbetweenredandblue),placementin thevisualfield(elementsnotonlybecome'heavier'astheyaremovedtowardsthetop,but alsoappear'heavier'thefurthertheyaremovedtowardsthe left,dueto an asymmetryin thevisualfield),perspective(foregroundobjectsaremoresalientthanbackgroundobjects, andelementsthatoverlapotherelementsaremoresalientthantheelementstheyoverlap), andalsoquitespecificculturalfactors,suchasthe appearanceof a humanfigureor a potentculturalsymbol.And,just asrhythmcreatesa hierarchyof importanceamongthe elementsof temporallyintegratedtexts,sovisualweightcreatesa hierarchyof importance amongtheelementsof spatiallyintegratedtexts,causingsometo drawmoreattentionto themselvesthanothers. Beingableto judgethevisualweightof theelementsof a compositionis beingableto judgehowthey'balance'.Theweighttheyputintothescalesderivesfromoneor moreof the factorsjust mentioned.Takentogether,the elementscreatea balancingcentre,the point,onemightsay,fromwhich,if oneconceivedof theelementsaspartof a mobile,this mobilewouldhaveto besuspended.Regardlessofwhetherthispointisintheactualcentre of the compositionor off-centre,it oftenbecomesthe spaceof the centralmessage,and thisatteststo the'powerof thecentre'(Arnheim,1982)to whichwehavealludedalready, a powerwhichexertsitselfevenif the Centreis an emptyspacearoundwhichthetext is organized- cf.Barthes'remarksaboutthe'emptyheartof fokyo' (I970:44). Perspectiveproducescentresof itsown,andbydoingsocontributesto thehierarchiza- tionoftheelementsincompositions.Asa resultviewersmayrelateto compositionsintwo ways:perspectivally,inwhichcasethecompositionisostensiblybasedontheviewer'sper- spective/position;or non-perspectivally,in whichcasethecompositionis not basedonthe viewer'sposition/perspective.In theformercasetheviewers,face-to-facewiththe infinite recessof perspective,becomethemselvesthe centreof the composition,thustakingthe placeof,forexample,thedeitiesin Byzantineor Buddhistpaintings.In thelattercasethe representationiscodedfroman internalpointof view,asisborneoutbythefactthatwhat is left andwhat is rightis judgedfrom the pointof viewof the representedparticipants ratherthanfromthe pointof viewof theviewer.Uspensky(1975:33-9) hasdocumented thiswithrespectto icon-painting.Hecitestraditionalguidesfor icon-painterswhichstate, for instance,'0nthe right,or goodside,is MountSinai,onthe left,or badside,Mount Lebanon',andthenshowshow,fromtheviewer'spointof view,MountSinaiisontheright andMountLebanononthe left.Headdsthatthis isa generalfeatureof pre-Renaissance art, andalsoof primitivecartographicdrawing. In the theoryof art, compositjonis oftentalkedaboutin aestheticandformalterms ('balance','harmony',etc.).In thepracticeof newspaperandmagazinelayoutit is more
  • 224. Themeaning of composition oftendiscussedin pragmaticterms(doesit'grab the readers/attention'?).In our view thesetwo aspectsare inextricablyintertwinedwith the semioticfunctionof composition. As we haveseen,in manymagazineadvertisements(e.g.plate2) the top section,the 'promiseof theproduct',isthemostsalientelementdueto itssize.Thissuggests,notjust that suchadvertisementsattemptto makereadersnoticethe attractivepicturefirst,so asto 'hook'them,but alsothat IdealandRealareranl<edin importanceandopposedto eachotherinthisway.Compositionis notjusta matterof formalaestheticsandof feeling, or of puilingthereaders(althoughit isthataswell);it alsomarshalsmeaningfulelements intocoherenttextsandit doesthisin wayswhichthemselvesfollowthe requirementsof mode-specificstructuresandthemselvesproducemeaning. Rhythmandbalancealsoformthe mostbodilyaspectsof texts,the interfacebetween our physicalandsemioticselves.Withoutrhythmandbalance,physicalcoordinationin timeandspaceis impossible.Theyforman indispensablematrixfor the productionand receptionof messagesandare vital in humaninteraction.Moreover,it is to quitesome degreefromthesenseof rhythmandthesenseof compositionalbalancethatouraesthetic pleasureintextsandouraffectiverelationsto textsarederived. FRAMING Thethirdkeyelementin compositionisframing.In temporallyintegratedtextsframingis, again,broughtaboutby rhythm.Fromtimeto timethe ongoingequal-timedcyclesof rhythmaremomentarilyinterruptedbya pause,a rallentando,a changeof gait,andthese juncturesmarkoff distinctunits,disconnectstretchesof speechor musicor movement from eachotherto a greateror lesserdegree.Wheresuchjuncturesare absent,the elementsare connectedin a continuousffow.In spatiallyintegratedcompositionsit is no different.The elementsor groupsof elementsare eitherdisconnected,markedoff from eachother,or connected,joinedtogether.And visualframing,too, is a matterof degree:elementsof thecompositionmaybestronglyor weaklyframed. Thestrongertheframingof anelement,the moreit is presentedasa separateunitof information.Contextthencoloursin the moreprecisenatureof this'separation'.The membersof a group,for instance,maybeshownin a groupportrait(asin groupphotos of schoolclassesor employeesof a company)or in a collageof individualphotos,marked off by frame linesand/oremptyspacebetweenthem (aswith photosof the managers of a companyin a companybrochure).Theabsenceof framingstressesgroupidentity, its presencesignifiesindividualityand differentiation.In figure6.1, framingacquires dramaticsignificance.Theleftpostof thedoorandthedividinglinebetweenthe lightand dark boardson the roof createa frame linewhich,literallyandfiguratively,separates Minusfromhissister,expressingthecommunicativegapbetweenthem.lnfilmandvideoa similareffectcanbecreatedbythechoicebetweenshowingtwoor moreactorstogetherin oneshot,or editingbetweenindividualshotsof theactorsin whicheachis isolatedfrom theothersbyframelines. Themorethe elementsof the spatialcompositionare connected,the moretheyare
  • 225. 204 . Themeaning of composition presentedas belongingtogether,as a singleunit of information.In the Nokiatriptych referredto above,for instance,therearenoframelinesto demarcatetheelementsof the triptychstronglyfromeachother.Thereisa senseof continuousffowfrom leftto right.But in figure6.16the'panels'of thetriptychareseparateunits- thereisa sharpdemarcation herebetweenpastandpresent.Thesameappliesto figure6.17,wherethe emptyspace betweenthetop andthecentral'panel'andthecolourcontrast(thetop panelis in colour, the middlepanelin blackandwhite)createa strongdivisionbetweenthe Idealandthe realityof immigration.Theexamplealsoillustratesthemanywaysinwhichframingcanbe achieved- by actualframelines,by whitespacebetweenelements,by discontinuitiesof colour.andsoon. Connectedness,too,canberealizedin manyways.It canbeemphasizedbyvectors,by depictedelements(structuralelementsof buildings,perspectivallydrawnroadsleadingthe eyeto elementsin the background,etc.)or by abstractgraphicelements,leadingthe eye fromoneelementto another,beginningwiththemostsalientelement,theelementthatfirst drawstheviewer'sattention.In figure6.2 thetiltingof the left-handphotoformsa vector leadingtheeyefrom leftto right,andtherepetitionof thecolourgoldinall theelementsof thetwo pagesprividesa strongsenseof unityandcohesion- visual'rhymes'of thiskind, repetitionof coloursandshapesin differentelementsof the composition,form another keyconnectiondevice,oftenusedin advertisementsto stresstheconnectionbetweenthe 'promiseoftheproduct'andtheproductitself(cf.alsothecolour-coordinationintheSony homepage). It shouldfinallybenotedthat,at a deeperlevel,thereis alsoanelementof framingin stylesof drawingandpainting.In linedrawings,for instance,theoutlinesof objectsstrictly demarcatethemfromtheirenvironment,whereasincertainstylesof painting(e.9.Impres- sionism)theyaresetapartfromtheirenvironmentonlybysubtletransitionsof colour. LINEARAND NON-LINEARCOMPOSITIONS In denselyprintedpagesof text,readingis linearandstrictlycoded.Suchtextsmustbe readthewaytheyaredesignedto beread- from leftto rightandfromtopto bottom,line by line.Anyotherform of reading(skipping,lookingat the lastpageto seehowthe plot will beresolvedor whattheconclusionwill be)isa formof cheatingandproducesa slight senseof guiltinthereader.Otherkindsof pages(e.g.traditionalcomicstrips)andimages (e.g.timelinediagrams)arealsodesignedto bereadinthislinearway. Thepageswe havedescribedin thischapterare readdifferently- andcanbereadin morethan oneway.Theirreadingpathis lessstrictlycoded.Readersof magazines,for instance,may ffick thoughthe magazine,stoppingeverynow and againto look at a pictureor reada headline,andperhapslaterreturningto someof thearticleswhichdrew theirattention,andwebsitesarespecificallydesignedto allowmultiplereadingpaths.Yet in manypagescompositiondoesset up particularhierarchiesof the movementof the hypotheticalreaderwithinandacrosstheirdifferentelements.Suchreadingpathsbegin withthemostsalientelement,andfromtheremoveto thenextmostsalientelement.andso
  • 226. Themeaning of composition on.Theirtrajectoriesarenot necessarilysimilarto that of thedenselyprintedpaggleft- rightandtop-bottom,but maymovein a circle,as in figure6.2,wherethe goldbeing pouredisthemostsalientelement,becauseof itsextremebrightness(somewhatreducedin reproduction),the photoof the two gold-diggersthe nextmostsalient,the headlinethe third mostsalient,andthetextthe nextmostsalient- but it mayalsobethatthevector formedbythetiltingof the left photographleadstheeyebackto the largerphoto,andso on,incircularfashion.Whetherthereaderonly'reads'thephotosandtheheadline,or also part or all of the verbaltext,a complementarity,a to andfro betweentextandimage,is guaranteed.Foranyonereaderthephotographor theheadlinemayformthestartingpoint of thereading.0ur assumptionisthatthe mostplausiblereadingpathistheonein which readersbeginbyglancingat thephotos,andthenmakea newstartfrom leftto right,from headlineto photo,afterwhich,optionally,theymoveto the bodyof the verbaltext.Such pagescanbe'scanned'or read,justaspicturescanbetakenin at a glanceor scrutinized for theireverydetail.Wedeliberatelymakea modestclaimhereandspeakof the'most plausible'readingpath,forthistypeof readingpathisnotstrictlycoded,notasmandatory, asthat of thedenselyprintedpageor theconventionalcomicstrip.Differentreadersmay followdifferentpaths.Giventhatwhatismadesalientisculturallydetermined,membersof differentculturalgroupingsarelikelyto havedifferenthierarchiesof salience,andperhaps textsofthiskindarethewaytheyarepreciselyto allowforthepossibilityof morethanone readingpath,andhencefor theheterogeneityanddiversityof theirlargereadership. As non-lineartextsbecomemorecommon,evendenselyprintedpagesof text beginto bereaddifferently.Thescientist,readinga journalof organicchemistry,will glanceat the diagrammaticrepresentationsof organiccompoundsbeforedecidingwhetheror not to readthe paperorrwhenreadingthat onlyonerat hasbeenusedin the experiment,skipto findoutfirstwhythiswasdone(Gledhill,1994).Studentspreparingfor theirexamswill usethe indexof thetextbookto findout andhighlightthepassagestheyneed,ratherthan readthetextbool<fromcoverto cover.Themorea textmakesuseof subheadings,emphatic devices(italics,boldtype,underlining),numberedlinesof typicalelementsor charac- teristicsof somephenomenon,tables,diagramsand so on,the more likelyit is to be scanned,skip-read,'used'ratherthanread:linearreadingisgraduallylosingground. Wenotedthat readingpathsmaybecircular,diagonal,spiralling,andsoon.Assoonas thispossibilityisopenedup/assoonasthereisa choicebetweendifferentlyshapedreading paths,theseshapescanthemselvesbecomesourcesof meaning.If the readingpath is circular,onereadsoutwards,inconcentriccircles,froma centralmessagewhichformsthe heart,soto speak,of theculturaluniverse.If thereadingpathis linearandhorizontal,it constitutesa progression,movinginexorablyforwardstowardsthefuture(or backwards, towardsthe'origin'of all things).If it is vertical,a senseof hierarchyis signified,a movementfromthegeneralto thespecific, fromthe'headline'to the'footnote'.Theshape of thereadingpathitselfconveysa significantculturalmessage. Sixteenth-centurybooksof emblemsexplicitlydescribedthe meaningsof different kindsof readingpath.Thereadingpathof figure6.19,anillustrationfroma Flemishbook of emblems,isa spiral,whichwasanemblemfortheinexorableprogressof time.It isalso a serpent,sothat the readingproceedsfrom thetail, a [ow,baseelement,to the head,a
  • 227. Themeaning of composition s f f { F { . : : - : f ; . t } . : t : L ' r . : " tr5 ;'.,'r| la ::"r " " r'"'v:!ii'$r { i r ' ; r " ' ! . ' 1 " d . d . : : : : C) fis O,rs A pagefrom Alciato'sBookof Emhtems(fromBassy,1975) superiorelement.Alain-MarieBassy(1975:303-5)explainsthesequenceof theemblem- aticallyexpressedmeaningsoneencountersas onefollowsthe spiralfrom the centre outwards:the hand('work'),the head('intelligence'),the tail of the serpent(a'base' element),thehandwhichholdsdownandimposesitswill onthetail.Joiningthesemean- ingstogetherresultsinthevisualpropositionalsoexpressedinthetitleof thepicture:'Ex literatumstudiisinmortalitemacquiri'('ThroughintellectualendeavoursI gainedimmor- tality').Today,ihestudyof themeaningof newkindsof readingpathshasbarelybegun. Analysingreadingpathswithstudents,wefoundthatsomeareeasyto agreeon,others harder,againothersimpossible.Thiswas not,we think,becauseof a lackof analytical abilityonour partor onthepartof ourstudents,but becauseof thestructureof thetexts themselves.Textsencodereadingpathsto differentdegrees.Some,thoughno longer denselyprintedpages,still takethe readersbythe hand,guidingthemfirmlythroughthe text.Others(wemightcallthem'semi-linear'texts)at bestprovidereaderswitha fewhints andsuggestions,andfor therestleavethereadersto theirowndevices.In againotherswe can,withthebestwill intheworld,notdetectanyreadingpaththat ismoreplausiblethan any numberof others.In figure6.20, a comicstrip from the magazineCracked,the headlinestandsout andthis,togetherwith the strongvectorformedby the waterslide
  • 228. t s s t a 1s s 1 A : r I i r * @*:* ffi *ffi #ffi ffi @ ffiffi ffi# Wffi ffi* ffie ffi ffi MSWffi W #ffi* ffi ffiffi @ g% ffi@ o o o fq = o .! i;
  • 229. 208 . Themeaning of composition onthe leftpage,predisposesusto start our readingtop left.Butwheretheeyewill move fromhereisdifficultto predict.Thereisneitherchronology(despitetheresemblanceto a ffowchart)nora clearhierarchyof salience. Increasinglymanytexts (newspapers,billboards,comicstrips,advertisements,web- sites)areofthiskind.Theyofferthereadera choiceof readingpathand,evenmoresothan inthecaseof textswherea plausiblereadingcanbediscerned,leaveit upto thereaderhow to traversethetextualspace.Theyare'interactive'-andit isperhapsnoaccidentthatthey havetheirclearestantecedentinthegenreof the'activities'bookswhichofferchildrena choiceof puzzles,riddles,colouring-inpictures,etc.fora rainydayduringtheholidays.This is notto saythat theorderof theelementsonsuchpagesis random.Thecomicstrip,for instance,stillhasits'welcome'signat top left,anditsmostgruesomeimagesinthe Real,a divisionbetweendepictionsof holidayfunandof sadistictorturethatrecallsthedivision betweentheGardenof EdenandDeathinfigure6.8. Lineartexts,then,are like movies,wherethe viewershaveno choicebut to seethe imagesin an orderthat hasbeendecidedfor them,or likean exhibitionin whichthe paintingsarehungin longcorridorsthroughwhichthevisitorsmustmove,followingsigns perhaps,to eventuallyendupat the exit.In non-lineartextsviewerscanselecttheirown imagesandviewthemin an orderof theirownchoosing.Theyare likeanexhibitionin a largeroomwhichvisitorscantraversein anyway they like.But,again,the way these exhibitsarearrangedwill not berandom.It will not berandomthata particularmajor sculptureisplacedinthecentreof theroom,or thata particularmajorpaintinghasbeen hungonthewalI oppositetheentrance,to benoticedfirstbyalI visitorsenteringtheroom. Lineartextsthusimposea syntagmaticsonthereader,describethesequenceof,andthe connectionbetween,theelements.Asa resultthemeaningsof individualelementscanbe lessstrictlycoded,asfor instancein documentaryfilms,wherethe meaningof the indi- vidualshotscanbelargelydeterminedbytheediting, ratherthanbytheintrinsicmeanings of theshots.Non-lineartextsimposea paradigmatics.Theyselecttheelementsthatcanbe viewedandpresentthemaccordingto a certainparadigmaticlogic- theIogicof Centreand Marginor of GivenandNew,for instance- but leaveit to thereaderto sequenceandcon- nectthem.In thedesignof suchtextstherewill bepressureto putmoreof themeaningin theindividualelementsofthecomposition,to usemorehighlycodedimages- symbolicand conceptualimages,tightlywritten,self-containeditemsof information,stereotypedcharac- ters,drawingsor highlystructuredimagesratherthan realisticphotographs,andso on. Linearandnon-lineartextsthusconstitutetwomodesof readingandtworegimesof control overmeaning,exactlyin the samewayaswe discussedin chapterl, in connectionwith Baby'sFirstBook(figure1.1)andthepagefrom DickBruna's0n My Walk(figure1.2). A SUMMARY Figure6.2I providesa summaryof the distinctionswe haveintroducedin thischapter. The double-headedarrows(f) standfor gradedcontrasts('moreor less,,ratherthan 'either-or').Thesuperscript'l'means'if'andthesuperscript'T'means,then,.In other
  • 230. Themeaning of composition . 209 words/if thereis no horizontalpolarization,thentheremustbeverticalpolarization'- theoppositefollowsfromthis.In thenextsectionwewilldiscussa numberof examplesin greaterdetail. REALIZATIONS Centred Anelement(theCentre)isplacedinthecentre of thecomoosition. Polarized Thereisnoelementinthecentreof the composition. Triptych Thenon-centralelementsin a centred compositionareplacedeitherontherightand leftor aboveandbelowtheCentre. Circular Thenon-centralelementsin a centred compositionareplacedbothaboveandbelow andto thesidesof theCentre,andfurther elementsmaybeplacedin betweenthese polarizedpositions. Margin Thenon-centralelementsin a cenrreo compositionareidenticalor near-identical,so creatingsymmetryinthecomposition. Mediator TheCentreof a polarizedcentredcomposition formsa bridgebetweenGivenandNewand/or IdealandReal,soreconcilingpolarized elementsto eachotherin someway. Given Theleftelementina polarizedcompositionor theleftpolarizedelementin a centred composition.Thiselementisnotidenticalor near-identicalto thecorrespondingright element. New Therightelementina polarizedcomposition or therightpolarizedelementina centred composition.Thiselementisnotidenticalor near-identicalto thecorrespondingleft element. Ideal Thetopelementina polarizedcompositionor thetoppolarizedelementina centred composition.Thiselementisnotidenticalor near-identicalto thecorrespondingbottom element. Real Thebottomelementin a polarizeo compositionor thebottompolarizedelement
  • 231. 2to Themeaning of composition Salience Disconnection Connection ina centredcomposition.Thiselementisnot identicalor near-identicalto the correspondingtopelement. Thedegreeto whichanelementdraws attentionto itself,dueto itssize,itsplacein theforegroundor itsoverlappingof other elements,itscolour,itstonalvalues,its sharpnessor definition,andotherfeatures. Thedegreeto whichanelementisvisually separatedfromotherelementsthroughframe lines,pictorialframingdevices,emptyspace betweenelements,discontinuitiesof colour andshape,andotherfeatures. Thedegreeto whichanelementisvisually joinedto anotherelement,throughthe absenceof framingdevices,throughvectors andthroughcontinuitiesor similaritiesof colourvisualshape,etc. | - Circular | ------l | - TriPtYch Centred { I f Centre-Margin | -----l | - Mediator-Pola ( elements- PoIarized Maximumsalience +Y Minimumsalience Maximumdisconnection +Y Maximum connection ator-PoIarized Composition O fis O.Zf Themeaningot composition
  • 232. Themeaning of composition 2r) GIVENAND NEWIN CHILDREN'SDRAWINGSAND CD-ROMs In anysequentialstructure,thatwhichisaboutto besaidor shownisbydefinitionalways New,not yet known.By contrast,what has (just) beenseen,heard,discoveredis, by comparison,nowknown,Given.In visualmedia,sequencecanof courseberepresentedina numberof dimensions,rightto left,bottomto top,in a spiralfrom outside,etc.(andin medievalpaintingperspectivecan indicatesequence,with the foregroundasthe present andthe backgroundasthefuture).Suchdimensionshavebeenusedthroughouthistory, andare still usedby differentcultures,as primaryvisualsequencingorientations.The mediumof thebook,bringingthepossibilityof turningthepage,addsa furthermeansof reprintingsequencevisually,the left page/rightpagestructureandthe possibilityof the two-pagestructures(rightpageandfollowingleftpage). Figure6.22 showsa doublepagefrom a bookproducedby a six-year-oldboywhile stayingin Parisfor halfa yearwith hisparents.It recordseventsandexperienceshewas involvedin,andsightsandobjectsheencounteredduringhisstayin Paris.Clearly,inthis situationeverythingwasNewfor thechild,literally.Hewasfacedwiththequestionof how to representnewinformation,newideas,newobjects,withoutthe possibiiityof relating themto already-established,knowndomains. Thebookopenswith the nameandaddressof the author,on the first lefthandpage. Thisis the Givenfor the bookasa whole,an elementof securityandfamiliarityin the new environment.0nthe first righthandpagethat new environmentis represented visually:a pictureof the EiffelTower.It isonlywhenthispageisturnedthatthepictureis named,commutedintolanguage.Oncenamed,the EiffelTowerbecomesGiven,andonthe adjoiningrighthandpagethechildfacesthenextaspectof hisnewenvironment.Thusthe bookcontinues:thenewpicture,too,is onlyidentifiedonthe nextlefthandpage- theArc deTriomphe.Thechildobviouslyrealizedthatthisstructurecouldbemisunderstood,and usedleft-facingarrowsto referthe readerto the pictureon the previouspage.But his impulsewasto first visuallyrepresentParisasthe New,andthento masterit, makeit knownandGivenbymeansof language,bymeansof namingit. Hisattitudewasempirical and he usedlanguageas an 'anchorage'in his effort to cometo termswith his new experiences. We will endwith an examplethat bringsall the elementsof this chaptertogether. Figure6.23showsthefirstscreenof an'edutainment'CD-R0Mfor children,titled'3D BodyAdventure'.Thetop of the screenshowsa rangeof mediaon a desktop.A slideis projectedon a screen.A videomonitorshowsan animatedsequence.Half-hiddenbehind themonitor,a loudspeakerplayssoftmusic.In otherwords,theldealhereiswhatwemight call'informationmedia',mediatoread,lookat andlistento.TheReal,ontheotherhand, presentsthingstheusercando.It of{ersgamesto play,mediato interactivelyengagewith. 'Emergency',for instance,isa gamewhichmixeslasersurgeryandtheshootinggallery- the playerzapsbraincellsin a raceagainsttime ('Hurrydoctor,savethe patient').And in'Body Recall'bodypartsmustbe matchedwith their names.Thusthe composition of the screenusesthe verticaldimensionto separateinformation-as-knowledgefrom information-as-action,or information-as-knowledgefrom information-as-entertainment.
  • 233. 2I2 . Themeaningof composition .:r.z*r {' l, r'{sr$r **r,.f *Xt/ Q figO.ZZ Parisdiaryof a six-year-oldboy And,whileit continuesto puttheformer,literallyandfiguratively,ona pedestal/it places reallearningsquarelyinthezoneof interactiveactivities.Wemightsaythat'entertaining activities'arehererepresentedas'consolidating'(givinga firm'footing'or'grounding'to) authoritativelypresented,'high'knowledge.Reversingthetwo - puttingthegamesontop ,iL "1rgc ie Tnr*flptjg
  • 234. Themeaningof composition 2 1 3 Q fig O,Zf Pagefrom3D BodyAdventure(KnowledqeAdventures,lgg3) andtheinformationmediaat thebottom- wouldcreatea verydifferentmeaning,perhaps somethinglike 'knowledgeprovidesa "foundation"fer ("highly,, regarded)active exoeriences'. Thescreenalsousesthe horizontaldimension,andthis in two ways.First,the left is thedomainof thestill image,andthe rightthedomainof the animated'3D,images,of the movefrom two-dimensionalrepresentationto 'virtualreality'.Second,the left is the domainof what hasalreadybeenformulatedfor the users,whilethe rightis the domain of what userscando themselves:theycanrotatethe skeletonwith their mousesoasto viewtheimagefromwhicheverangletheychoose,andtheycanexitthescreenat will.Note that the monitorstraddlesthe boundarybetweenIdealand Real:likeinteractivegames, user-activated3D viewinghas(still)someentertainmentvalue,becauseof itsnovelty;but, likeinformationmedia,it alsohasinstructionvalue- the animatedskeletoncanserveas a stand-infor a realor reproductionskeletonandmakea goodlearningaidfor students.In otherwords,aswemovefrom leftto right,wemovefromthetraditional2Ddiagramto the newanimated3Ddiagramor drawing,andfromthetraditional'passivelearner'to thenew 'interactive'modeof learning. Anotherdimensionusedhereisthatof foregroundandbackground.Theloudspeakeris placedbehindthe monitor,whichiscongruentwiththe roleplayedbysoundandimagein
  • 235. 2I4 . Themeaning of composition this CD-R0M:all informationis providedvisually,andthe soundtrackonlyofferssoft backgroundmusic. Mostsalientonthescreenisthemonitorimageof themovingskeleton,andthisfortwo reasons:it moves,andit displaysthegreatesttonalcontrast.Nextmostsalientareperhaps the namesof the qames.Althoughtheydo not occupymuchspace,theircolours- bright redandyellow- contraststronglywiththecoolwhites,bluesandgreysthat dominatethe restofthescreen.Andtheimages(thedoorsoftheEmergencyWardandthe'BodyRecall' keyboard)are both sharperand moresaturatedin colourthan the rest of the screen. Relativesizecanalsoestablishsalience,andasa resultthe'slide'withtheX-Raypicture of thebodyandthetitleof theCD-R0Mis perhapsthenextmostsalientelement.Which leavestheloudspeakerandthe'exit'sign. Fromthe pointof viewof lraming,finally,the mostsignificant'disconnection'isthat betweenthespaceof theinteractivegamesandtherestof thescreen.Thegames,againsta brighter,moregarishbluethancanbefoundelsewhereonthescreen,insertthemselvesinto the moretraditional,naturalisticcontinuity(and naturalpalette)of the desktop.They couldhavebeenplacedonthe desktop.Buttheyarenot.Theyarerepresentedasa quite separate/'alien'element,disruptingthenaturalperspectivalhomogeneityof thesemiotic space.Withinthe pictureof thedesktoponthe otherhand,thereis a senseof continuity, bothbecauseof the harmonyof the mutedcolours,andbecauseof thewaythe elements form part of a continuous,homogeneous,non-fragmentedspace.Thusthe traditional mediaare representedas naturalisticand complementaryto eachother,but also as radicallydifferentfromthenew'interactive'media. Theexampleshowsthatthecompositionof thisscreenpositionsthecomponentmodes of themultimodaltextin relationto eachother,makingsomeplaya foregroundrole,some a backgroundrole,presentingsomeascomplementarylo eachother,othersaseachother's opposites/andsoon.It visuallyrealizesa discourseof 'edutainment',andvisuallydefines its characteristicrelationsand values,and the part playedin it by differentsemiotic modes.
  • 236. 7 M a t e r i a l i t ya n d m e a n i n g MATERIALPRODUCTIONAS A SEMIOTICRESOURCE Thesemioticresourceswe havediscussedinthisbookabstractawayfromthemateriality ofthesignifier.Theycanbeapplied,wehaveclaimed(andtriedto demonstratethroughour examples)to the productionand understandingof visualswhich,materially,are quite differentfrom eachother:photographs,movies,websites,drawings,paintings,andsoon. Oneofthemajorfeatures- explicitlyandimplicitly- of thedevelopmentof ourideassince wewrotethefirst editionof thisbookhasbeento paymoyeattentionto thesemioticrole of the materialproductionof thesign.In music,the performanceof a compositioncon- tributesa greatdealto itsmeaning,andin manycasesit isdifficult,if notimpossible,to separatecompositionandperformance.In visualcommunication,similarly,the material productionof a designis not just the executionof somethingalreadycomplete,but a vital part of meaning-making.Herewe will focuson that aspectof semiosisin some moredetailby lookingat the materialsusedin what,in the first editionof this bool<, wecalled'inscription',andhavesincecometo call'production'(l(ressandvanLeeuwen, 2 0 0 1) . When,sometimein1988,wefirstpresentedourinitialideasonthevisualto theSydney SemioticsSalon,oneof our friendssaid,'Butwhat aboutbrushstrokes?Howcanyou describebrushstrokesassemioticunits?'0urfumbledresponsewasto say,'Youhaveto start somewhere.We'llgetto brushstrokeslater,whenwe arefurtheronwith ourwork.' Butthequestionstayedwithus,andregularlykeptcomingupin ourdiscussions.It wasa questionthat respondedto our view- thenan oddone- that in a messageall aspects matterandmean,andat thesametimeshoweda profoundscepticismaboutthatassump- tion. However,it is and remainsan importantquestion.Nearlytwentyyearslater,our answerwouldin principlebethesame,thoughmaybea littlelessfumbledandsomewhat morethoughtthrough.In our'grammar'of visualdesign,wewantedto moveawayfroma totalizingviewof semioticresources,a viewinwhichsemioticresourcesarehomogeneous systemsinwhichtheremaybedifferencesinthe'size'ofunits,butinwhichalltheunitsare of thesamekind,all 'belong'to'thesamesystem',sothatalltextsare,intheend,builtup fromasinglekindof'minimalunit',beitthebrushstroke,the'iconicfigure'(Eco,I976d, the'coloureme'(Saint-Martin,J-987),orthephonemeandmorpheme- intheirrespective 'tactic'arrangements.Bycontrast,wewantedto maintainthata givenformof semiosis- for instance,'painting'-involvesa rangeof signifyingresources.Someofthesearelikethe signifyingsystemswe havediscussedinthisbook,resourceswhichcanbeused,notjust in painting,butalsoin photography,or indrawing,to mentionjustsomeexamples.Anygiven typeof productionmediumcan,at leastin principle,realizemostof thechoicesfromthe ideational,interpersonalandtextualnetworkswehavepresentedinthisbook,thoughthere are, in practice,historicallyand culturallyspecificrestrictionson the combinationof
  • 237. 2 1 6 MateriaIity and meaning choicesfromthesesignifyingsystems;for example,restrictionsonwhatcanbepaintedand how.But othersemioticresourcesaremorespecificallytiedto specificformsof material productionandcanberealized,for instance,onlyin the mediumof paintor onlyin the mediumof thephotograph. In the realmof art thisisa relativelyuncontentiouspointof view.Materialitymatters: oil-andwater-basedpaintsofferdifferentaffordances,andhencedifferentpotentialsfor makingmeaning.The mannerof productionalso matters,as we discussedwith the examplesof RobertRymaninchapter5.ln therealmof linguisticsit hasbeenlessobvious. If we askthe seeminglysimplequestion'What is a text?'or 'ls a writtentextthe same objector a differentonewhenit is writtenwith a pencilor with penandink or is word- processed?/,theanswerof most linguistswouldbe,'Noquestion.It isthesametext.'The material,graphicexpressionof thetextwouldnotbeseenasa relevantissue.If weaskeda non-linguistthesamequestion,theanswermightbedifferent- theteacherwhoresponds negativelyto anessaypresentedonscrappybitsof paper,badlyhandwritten(perhapsbadly spelled),but respondsfavourablyto a'well-presented',typedversionof thesametext,uses a quitedifferentcriterion.Sodoesthemarketingexecutivewhenpresentinga proposalto a client.Theirnotionof whata textisdiffersfromthatofthelinguist.Likeus,theywouldsee 'presentation'asa significantpartof themakingof thetext,increasinglyoftenequalto,or evenmoreimportantthan,otheraspects.Forthem,asfor the painteror the viewerof a painting,themediumof inscriptionchangesthetext. It is our impressionthat this aspectof text is rapidlygainingin importance,perhaps aidedbynewtechnologiesofwriting.Theboundariesbetweenthecriteriaprevailingin'art' andthoseprevailingineverydaywritingareno longerassharplydrawnastheyoncewere. Wedo notwantto engagein an argumentwith linguisticshere,andthe linguistictheory fromwhichwedrawmuchinspirationis in anycasesemioticallyoriented.Butwedowant to saythatthe linguisticnotionof textisanartefactof linguistictheory;as,indeed,isour notionof text- whetherwrittenandlinguisticor paintedandvisual,or both.Thequestion aboutthesignificanceof brushstrokescomes,wethink,out of a viewin whicheverything representationalis seenas belongingto the sameunified,homogeneousrepresentational system(language,or painting).Theboundariesaroundwhatis'in'andwhatis'out'used to bestrictlypatrolled:in the linguistictrainingof oneof the authors,phoneticswasnot partof linguistics,andeveryoneknewwhatwasextra-,para-or simplynon-linguistic.The materialaspectsof handwritingandtypographywerenoteventouchedupon. In our approachthe materialexpressionof signs,andthereforeof the text,is always significant;it is what constitutes'signifiermaterial'atonelevel,and it is thereforea crucialsemioticfeature.So isthe processof sign-(andthereforetext-)production.Texts are materialobjectswhich resultfrom a varietyof representationaland production practicesthat makeuseof a varietyof signifierresourcesorganizedassignifyingsystems (we havecalledthese'modes'),anda varielyof 'media',of 'signifiermaterials'- the surfacesof production(paper,rock,plastic,textile,wood,etc.),thesubstancesof produc- tion (ink,gold,paint,light,etc.)andthetoolsof production(chisel,pen/brush/pencils, stylus,etc.). Everyculturehassystemsof meaningscodedin thesematerialsandmeansof produc-
  • 238. Materiality and meaning . 2I7 tion.Here,asin all areasof semiosis,signsintheirmaterialityarefullymotivated,though asalwaysthemotivationsarethoseof a particularcultureina particularperiod,andthose of themakerofthesign;theyarenotglobal,noraretheya-historical.Preciousmetalsare preciousbecauseof theirscarcity,andperhapsbecauseoftheir malleability.Butscarcityis nota globallyuniformcharacteristic,andthepreciousnessof onemetalneednotbeequally markedinanotherculture.Itwasoneoftheparticularcalamitiesoftheculturesof Central and SouthAmericathat they had attacheddifferentsemioticvaluesto the material signifiergoldfromthoseof theinvadingSpaniards. We regardmaterialproductionas particularlysignificantbecauseoften it is in its processesthat unsemioticizedmaterialityis drawnintosemiosis.At timesproductionis thereforesomewhatlesssubjectto the variousformsof semioticpolicingthanare other regionsof thesemioticlandscape,andthusleavesmoreroomfor individualpossibilities of expressionthan thoseregionswhichhavebetter-knownculturalhistories,are more foregroundedandhavebetter-understoodconventions.Toexplorematerialproductionis thereforealsoto explorethe boundariesbetweenthe semioticandthe non-semiotic,and betweenindividualexpressionandsocialsemiosis. PRODUCTIONSYSTEMSANDTECHNOLOGY Likeall culturaltechnologies,formsof productionareentirelyrelatedto theoverallstate of a society'stechnologies.Indeed,dependenceontechnologymaybeoneof thestrongest featuresof graphically realizedsemiotics;it distinguishesthemfrom semioticmodesin whichsignsare articulatedby the bodywithoutanytechnologicalaids(as,for instance, inspeech,singing,'non-verbalcommunication',dance).Modeslikemusicstraddlethetwo categories;yetthe boundariesbetweenthemare in anycasealwaysfuzzy:onecandraw or writewith one'sfingerin sand,usingonlythebodyanda naturalsurface.Butgenerally the surfaces,substancesand tools of the visualsemioticare madeavailableby tech- nologies,as muchin the caseof penciland paperas in the caseof the modernword processor.Technologyentersfundamentallyintothe semioticprocess:throughthe kinds of meanswhichit facilitatesor favours,andthroughthedifferentialaccessto themeansof productionandreceptionwhichit provides. Wedistinguishthreemajorclassesof productiontechnologies:(1) productionin the narrowersense- that is,technologiesof the hand,technologiesin whichrepresentations are,in all theiraspects,articulatedbythehumanhand,aidedbyhand-heldtoolssuchas chisels,brushes,pencils,etc.;(2) recordingtechnologies- that is,technologiesof theeye (andear),technologieswhichallowmoreor lessautomatedanalogicalrepresentationof whattheyrepresent,for instance,audiotape,photographyandfilm; and(3) synthesizing technologieswhichallowthe productionof digitailysynthesizedrepresentations.While remainingtiedto theeye(andear),thesereintroducethehumanhandviaa technological 'interface',at presentstill in the shapeof a tool (keyboard,mouse),thoughin future perhapsincreasinglythroughdirectarticulationbythebody(e.g.throughissuingspoken commandsto thecomputer,or throughothergestures).
  • 239. 218 Materiality and meaning Theboundariesbetweenthesecategoriesare not clear-cut;andarealwayssubjectto furthertransformativesemioticwork.A photographcanbehand-colouredonceit hasbeen printed,for instance,or digitallyaltered,andmanyartistsexperimentwith preciselythese mixedproductionsystems.It shouldalsobe notedthat the possibilityof 'mechanical reproduction',to useBenjamin'sterm,is not uniquelytiedto anyof thethreecategories. Printingcanbedonefroma hand-carvedmaster,photographically,or witha modernlaser printer.But we thinkthe categoriesare useful,particularlyastheycanbetiedto major periodsinthehistoryof productionandto theepistemologiesthatwentwiththem. Whileproductiontechnologies- technologiesof the hand- havecontinuedto playa role,thedevelopmentof recordingtechnologieshasdominatedthevisualsemioticfrom themomentRenaissanceartistsbeganto usethecameraobscuraasanaidinpainting,and particularlyduringthe lasttwo centuriesor so,whena varietyof recordingtechnologies weredeveloped,beginningwith photography.They,in theirturn,are nowbeginningto be supersededby synthesizingtechnologies.Quitedifferentontologicalorientationsgowith thesedifferenttechnologies.Walter Benjamin(1973) commentedon the transltion betweenmanualproductionand recording,stressingreproduceabilityratherthan the modesof representationthemselves,andlinkingit to the dissolutionof traditionalforms of socialorganizationin 'masssociety'andto thedisappearanceof the'aura'of thework of art. Todaythetransitionfrom recordingto synthesizingtechnologiesisthe morepressing issue.The 'crisis of representation'which has characterizedtheoreticaldebateover the lasttwo decadesor so maybean indicationof this.'Recording'leads,we believe,to ontologiesof referentiality,a viewof representationbeingfoundedon direct,referential relationsbetweentherepresentationsandtheworld.In anearlierpublicationwedeveloped thisideain moredetail(vanLeeuwenandl(ress,1992).Synthesizingtechnologiesunder- mineor evenabolishsuchnotionsof referentiality,whereasas recentlyas in the 1,970s, 'ElectronicNewsGathering'wasubiquitousenoughto havedevelopedanacronym,'ENG', a deceptivelynaivemetaphorremlniscentof otherunproblematicgatherings- wildmush- rooms,apples,the children.Thatmetaphoris nowentirelyuntenable;not onlybecause newsneverwassimply'outthere'tobegathered,butevenmoresobecausethetechnology nowexistsliterallyto produceit - a developmentanticipatedbythecriticalmediatheory of the 1970s.'Reference'hasgivenwayto 'signification',theproduction,out of existing semioticresources,of newsemioticmeans,newsigns,newtexts,newimages,newvisions, newworlds.Thisdoesnot meanthat representationhas ceased.Rather,the formerly naturalizedrelation,the identityof representationand reference,has brokendown, irreparablyfor the time being.A newrelationis becomingestablishedinstead,between representationandsignification.If presentsocialandtechnologicaldevelopmentscon- tinue,thisrelationwill,in itsturn,firstbecomenaturalizedanddominant,andthencome intocrisis.In the yearthat we revisedthis bookfor its secondedition,the 'production' of photographsof abusesof prisonersby Britishtroopsin Iraq wasone(notorious)case in point.(Leavingthe crucialmatterof veracityaside,it is interestingto notethatthe muchearlier'production'ofthesame'newsevent'bymeansof writingproducednooutcry of anvkind.)
  • 240. MateriaIity and meanIng 2r9 Thesetechnologicalandsemioticdevelopmentsperhapshelpusunderstandcontemp0r- arytheoreticaldevelopments.With'recording'goesanoldersemioticsof 'representation' andof naturalisticmodality,whichitselfgaveriseto particularontologiesof truth, of fact (hencetheinterminabledebatesof priordecadesaround'blas').Todeconstructrepre- sentationas recording,representations('texts')themselveshadto be 'deconstructed', whichwasdoneby emphasizing'production'andthereforedisplayingthe 'constructed- ness/of representations,intexts,images,etc.Thiscanbeseenastheperiodof'critique'- withtheadjective'critical'usedasa descriptorof manypractices.Inherentinthisthere wasalreadya theorizingof onenewstageof semioticpractice,namelysynthesis,through newpracticesof constructionandproduction- mal<ingnewrepresentationsout of (con- structed)representations- ln the visualarts maybesomeof the practicesof'Brit Art' or of theAmericanartistJeff l(oons,andin musicthecurrentlyubiquitouspracticesof scratching,mixing. Fromsucha historyand perspective,whichbothheraldsand legitimizesthe present stageof synthesis,we couldprojecta furtherdevelopment,whichwill beto deconstruct currentpracticesof productionby showingthat 'underneath'productionthere is an already-existing/already-produced'programme',a systemwhichdefinesthe limitsof pro- duction.Thissystem,of course,is still a systemof representation,a representationof the social/culturalsystem.Wecouldalsopointto thepresenttendenciesin semiotictheories (inculturalstudies,ineducation,in literacytheory)to collapsereadingintowriting,or vice versa.Thedissolutionof that distinction('readingis writing')wasinitiatedin theoryin the 1960sby RolandBarthes,thoughit is nowenactedin electronictechnologieswhich combinetheactsof readingand(re)writlngsomeoneelse'stext,whetherin changestoan attachmentto the emailor,differently,in playingcomputergames.In otherwords,when theanalogicallybasedmodeof 'recording'wasdominant,thetendencyof criticaltheory wasto deconstructrepresentation-as-reference/andto emphasizethe'constructedness'of thesign,or thetext. As thesyntheticallybasedmodeof productionis becomingthedominanttechnology, critical theory will have to turn to deconstructingrepresentation-as-programme, representation-as-design;that is,deconstructingthecombinatorlalpossibilitiesandlaying baretheir cultural/socialsources.It is for this reasonthat we concentrateon represen- tationalresourcesinthisbook,ratherthan(only)ontexts.However/giventhedeconstruc- tionofformerlystableframes- whethersemioticor social/culturalandeconomic- forthe timebeingthereexistsnewlytheneedfor conceivingof socialsemioticpracticeintermsof rhetoricand.design',wheretheterm'rhetoric'focusesonthesocialrelatlonswhichobtain intheprocessof communication,andtheterm'design'focusesonthearrangementof the availablesemioticresourcesinthemakingof therepresentationasa message' Beforewe leavethissubject/weshouldnotetwo otheraspectsof the relationbetween productionandtechnology.0ur classificationof productionmediawasbasedontheway representationsare produced,whetherby hand,by moreor lessautomatedrecordingor byelectronicsynthesis.Butproductionmediaalsofavourmodesof reception,andherethe surfaceplaysa particularlyimportantrole.Somesurfaces(wal1s,clnemascreens)favour publicreception,for instance,andothers(pages,andpapergenerally,thecomputerscreen)
  • 241. 2 2 0 Materiality and meaning favourindividualreception.Also- andmoredifficultto describe- thereistheeffectof the physicality,the tangibilityof the surface,the differencebetweenthe formscarvedin the hardrockandtheffeetingffickersof lightontheglassscreen(wereturnto thisinthenext section).What mattersis the site as muchas the kindof surfaceon whichthe text is received.Now,unlikein previousperiods,thesurfaceof receptionis no longernecessarily at all thesameastheoneonwhichthetextwas/isproduced.Transcodings/transpositions of a widevarietymaytakeplace.An imagemaybeproducedinonemedium- asa painting, say- andbereceivedin a differentmedium- asa photograph,for instance.0r it maybe producedin a recordingmedium,asa photograph,for instance,andreceivedin a svnthesis medium,retrievedfromtheimagebankof a computer. Finally,technologyhasalsodevelopeddifferentdistributionmedia,andit is herethat the issueof (mass)reproduceabilitybelongs,togetherwith that of communicationat (long)distance.Thelatter,althoughof crucialsocialimportance,bearslessdireclyonour subiect'Whetherimagesaredistributedviaelectricalwires,opticalfibresor theairwaves is irrelevant,semiotically,at the levelof representation- thoughnot at the levelof dis- semination'Thefactthattheinternetiscrammedfull with imagesis in largeparta matter of availabletechnoJogy;andit hasprofoundsemioticconsequences.At anotherlevelwhat mattersmostistheproductionmediuminwhichimagesareproduced,andthedistribution mediumin whichtheyarereceived,if the latterisdifferentfromthe former,beit because of transcoding(e.g.the photographicreproductionof paintings)or becauseof recodingat theotherendof thetelecommunicationchannel.0r,to bemoreprecise,wewouldsaythat the modeof transmissionis relevantonlyin relationto the potentialswhichit offersfor receptionas(re-)production. BRUSHSTROKES If one looksat Rothko'sseagramMuralsfrom a distanceof 4-5m,the boundaries betweenthelargeblocksof colourseemsharpandclear-cut.Thecloseronemovesto the paintings,however,the moreuncertainandfuzzythe boundariesbecome,the morethey overlapandrun intoeachother.yet a postcardof oneof the paintings,takenfrom much morethan5m distance,showsnothingof this.An aspectof meaningis lost,becauseof the distancefrom whichthe photographwastaken,and becauseof the transcoding,from paintingto photograph,from oneproductionmediumintoanother.Thisbringsusbackto thestartingpointof thischapter,to theargumentaboutbrushstrokesandaboutthestatus of'text'. Paintingallowstheviewera choicebetweendifferentwaysof relatingto thetext,even thoughthischoicemayberestrictedin practice,aswhena lineonthefloor in front of a paintingprohibitsthe galleryvisitorfrom comingtoo closeto the painting.I maywish to viewthe paintingas'a representation',concentratingon whatthe painting'is about,, or view it in termsof its varioustechniques('the effectiveuseof colour,),or effects ('depression').In eachcaseI will standat therequisitedistance.I maywishto engagewith itsmaterialityandwiththewayinwhichthehandof theartist,inscribed,thecanvas- the
  • 242. MateriaIity and meaning 2 2 f matterof theapplicationof thepainUthebrushstrokes- andinthatcaseI wouldneedto moveverycloseup.Photographyallowsmechoiceof distanceof viewing,butwithoutany of theseeffectsin meaning.This'recording'mediumis representationalandcanonlybe representational.It abstractsawayfromtheimprintof thehandthatmadeit,evenwhenit reproducesart. Certaindiscoursesaboutart putmuchemphasisonmaterialproduction,particularlyin relationto a handfulof greatpainters:the finebrushstrokesof Rembrandt,the violent brushstrokesof VanGogh,thetreatmentof volumeby the Pointillists.Thisencourages a focuson the 'graphology'of the paintingasa symptom,a traceof the individualtem- peramentof theartist.Thatis,it tal<esa semioticapproachto thematterof brushstrokes. Mostof the descriptionsof paintlngsln art galleriesandin catalogues,however,focuson representationratherthanon inscription.Thepreferenceof art historiansfor black-and- whitereproductionsor evenfor drawingsof paintingsalsopointsto anoverridingconcern with representation. The questionof transcodingis closelyrelatedto this,What doesit slgnifywhenI purchasea printof Monet'sPoppiestohangonmyliving-roomwall?ThatI wanttoshow my appreciationand admiratlonof Monet?That t havebeento a gallerywherethe paintingwasexhibited?ThatI likethethemeof the painting?ThatI am familiarwith the intellectualhistoriesof whichImpressionlsmis a part?Theprintwill allowme to signifyallofthese,butitwillnotallowmetosignifymyinterestinthematerialproduction of thework,simplybecauseit doesnotenablemeto focusonthat;it doesnotevenmakeit availableasa question. Totakeanotherexample,in anoriginalpaintingby Mondrianthelinesare,in close-up, notstraight,but overpaintedandthecolourof thevariousrectanglesis modulatedrather thanplainandflat.Postcardsandotherreproductionsof thesamepaintingmakethelines appearstraight,removethe overpaintingandpresentffat,unmodulatedcolour.It isthese reoroductionswhichproducetheMondrianof innumerablehigh-schoolart lessons,andso reinforceand reproducea particular(incorrect)versionof Mondriananda particular (ideological)versionof abstractpainting.It is throughtheseideologizedreadingsthat suchpaintingshavetheireffecton,andin,otherpractices. Asa finalexample,considerl(andinsl<y'sCossacks(plate5). Representationisstrongly reducedin this painting:onecan'see'Cossackson horsebacl<anddrawnandffashing sabres,but figurationis not foregrounded;or,we mightsay that what is represented ideationallyis'violentaction'.Colourdifferentiation,ontheotherhand,isamplified.And colouris used,not in a representational/referentialfunction(shadesof whiteto represent the'real'shadesof thewhiteof uniforms,for instance),butto allowthesubstanceof the productionmediumandthetracesoftheactof productionthemselvesto signify.Colouris, in this painting,the keysemioticresource:jaggedblocksof white,flashesof red,curves of yellowand blue.Action,energy/movementandviolenceare representedthroughthe way the productionmediumhasbeenhandled.And thistakesus backto the issueof individualitywithwhichweendedthefirst sectionof thischapter,andperhapsalsoto the distinctionbetween'art'and'design'.Thehandlingof paintiscloseto theindividualityof handwriting,andit waspreciselythismarkof individualitywhichcameto belaudedasthe
  • 243. 222 . Materialityandmeaning distinguishingcharacteristicof art andas its markof differencefrom 'recording,.0nthe otherhand,thiscelebrationof individualitywasalsosomewhatof a last-ditchstandfor art, in its losingbattlewith photography,andthe verysameart whichcameto stand for individualexpressionhaditsprincipalsocialeffectsintheformof photographicrepro- ductions- that is,withoutthe marksof individuality- whilethe originalsbecamethe pricelessrelicsof a pastethos.In mostotherdomainssuchmarksof individualitybecame proscribed.Handwriting,for instance,hasnowbecomeunacceptablein all but the most privateformsof writing,despitethe increasedemphasison 'presentation,whichwenoted earlier:thisnewvaluationof 'presentation'is in nowaya returnto the kindof individual expressivenessthat hoverson the borderbetweenthe individualand the social,the ostensiblyunsemioticizedandthesemioticized;it isthoroughlysemioticandsocial. Theindividualityof thebrushstrokenotonlybecamea symbolof individualexpression, of 'theessenceof thingsseenthroughan individualtemperament,,but it alsocameto be drawnintothe domainof the semiotic,the domainof culture.It'made school,.It was reproduced,faked,developed,imitated,and so enteredinto the world of semiosisas a transformativeelement,in a processwhichthentransformedthe brushstrokeitself.as witnessedbyRoyLichtenstein'spopart parody(figure7.l). Thusthebrushstrokebecomes a paradigmcaseof howinscription(the'how'ofpainting)isallowedto playa keyrolein @ riq z.r Big painting, t965(Roy Lichtenstein)
  • 244. Materiality and meaning 223 somedomains- for instance,thedomainof thegreatModernistart of the immediatepast - but requiredto playa humblerrole in others,whererepresentation(the'what'of painting)dominates,and whereproductmattersmorethan processand practice.The divisionexistswithinpaintingalso.A paintersuchas RobertRyman,whosework we discussedin chapter5, focuseson the 'how',on the inscriptionalpracticesof painting, whereasa paintersuchas Gainsboroughfocusedon the'what',on the analysisof social arrangements,therecordingofsocialstatesofaffairs;inshort,onthereferential.(Though maybethebetterwayof describinglt wouldbeto saythatfor Rymanthe'how'hasbecome the'what'.In thatviewthequestionthatpaintersaskis,'Whatispaintingfor?'andthat ouestlonhasdifferentanswersin dlfferentperiodsandin dlfferentcultures.) Suchshiftsof emphasisare themselves'signs'.Interestin the materialityof repre- sentationandrepresentationalpracticesreflectswidersocialandculturalconcernswith questionsof substanceandmaterialityin a worldinwhichtheconcretebecomesabstract, thematerialimmaterial,thesubstantialinsubstantialandreality'virtual'.Todaywehave, sidebyside,a hankeringfor the individual,thesubjective,theaffective,the non-semiotic andnon-social'punctum'ofthe photographor the'grain'of thevoice(Barthes,1977, I9B4),andat thesametime(andin largepartasa result)the increasingsemioticization of all thesephenomenaandmore.Aswehavenotedalready,it isasrepresentationsrather thanasmaterialproductionsthat modernart hasinformedandshapedpracticesin other domains,thatpaintingssuchasthoseof Mondriancameto betransformedintoblueprints for designedobjects,buildings,cities,andthat paintingssuchasthoseof l(andinskyhad their effecton the layoutof Europeannewspaperssuchas the Bildzeitungand TheSun, which are further translations/reductionsof othertranslations/reductions,but no less potentfor that. Any systematizingsemioticsof materiality(of the brushstroke,or of handwriting) wouldfollowthetrendof semioticizingthenotyetsemioticized.Reluctanceto dosowould followthe opposingtrend set by Barthesto 'protect'the non-semiotic,to protectthe ,unsooiltnature'afterwhichthetouristhankers.In thischapter,we(likehim)arealsoat leasta littlereluctantto followthe pathof relentlesslymakingeverythingsemiotically accounted.In reality,wethinkthat the choiceis not onebetweenturningthe unsemiotic intothesemiotic- if I asan interpretertakemeaningfrom somephenomenon,thenit is semioticbyvirtueof thataction.Thechoiceratheris onebetweenassistingin layingbare, in makingovertlyvisiblemeaningswhichareasyetnotvisibleandmadesystematic;and beingclearaboutthe consequencesof that process.Nevertheless,as materialproduction issemioticized,it becomesmoreimportantto beableto talkaboutit.Simplyassertingthe valueof the non-semioticat the very momentwheneverywherearoundus it is being semioticized,marginalizedor repressedseemsat leastequallyproblematic. THE MEANINGOFMATERIALITY Fromthe1920sonwards,therehasbeena'functionalist'currentin Modernism,a trendto 'let materialsspeakfor themselves',whlchis onlynowbeginningto change.Thishad
  • 245. 224 Material ity and meaning variousroots/but it culminatedinnowhappilyclich6dstylesof 'plainness,:whetherof steel or timberfurniture,or of 'brutalist'architecture,with its loveof unadornedconcrete. Behindthistrendwerenotionsof 'authenticity',themselvesexplicitor implicitcritiquesof the 'distortions/of representation,the 'falsities'introducedby 'decorative,art and its ideologies.Thistrendhaditsoriginsin art,whereit couldevenbecomethesubjectmatter of artworks,as in Ryman'spaintings.In someperiodsor genresof art, artistshaveno choiceof materials:all paintingsarepaintedoncavewallswith ochres,or oncanvaswith oil paints,all photographsare printedon paper.In otherperiodsor genres,the material becomesa fullyexploitableandexploitedresource.Modernsculptureis perhapsthebest example.ThesmoothlyturnedwoodofBrancusi'sHeadQgIg-23)ortheveinedrednessof thealabasterof Epstein'sJacobandtheAngel1940) (seefigure8.1) becomepartof the meaning. In somecasesit is preciselythe oppositionbetweenthe materialityof the material andthe mimeticqualityof the productwhichbecomesthe issue.In Rodin,sThe Kiss (1880), the figuresare perfectlyworkedwhen seenfrom the front. The material resistancesof marblehavebeenentirelyand successfullyovercome.The materialhas become'invisible',justasthe materialityof thecanvasis invisiblein mostpaintings.If we changeourviewingpositionbywalkingaroundthesculpture,however,wearepermittedto seethat thisfirst impressionis'produced',andthereforeideological.Thecontrastposes the questionof the sculptor'swork,his semioticaction.It forcesus to reflecton the borderlinesbetweenthe seeminglyunsemioticizedmaterialityof the representationand the semioticized,fully culturalwork of the sculptor,and on the dialecticbetweenthe expressionof individualityandthesocialsemioticframeworkinwhichit takesplace.The two areconnected:the'handwriting'effect,soclearlyvisiblein the less,polished,parts of Rodin'ssculpture,becomesincreasinglypossibleaswe movefromthe highlypoliced, thehighlyconventionalized,thefullysemiotic,towardsthelesssemioticized,thereforeless policedand lessconventionalized.The uncertainlinesof a Mondrianpaintingsignal individuality,affect and art as clearlyas the certainlinesof a blueprintsignalcon- ventionality,referenceanddesign.A paintingby BenNicholson(1945)makesthesame pointin a differentway.It consists,quitesimply,of two circles:a perfect,compass-aided circleonthe left,asthe Given;anda hand-drawncircleonthe right,asthe New.Drawing by hand,with all its subtlemarksof individuality,mayoncehavebeenunproblematic, as therewas no otherway of drawing,but hasnow becomeproblematic,an issuefor c0ncern. Architectswho developblueprintsfor buildingsalso work with 'unsemioticized, materials.Theirintentionsareusuallylesssemioticor,to putit anotherway,it isharderfor themto putthesemioticintheforeground.Otherconsiderationsmayweighmoreheavily: functionalconsiderations(thefactthattheyaredesigninganofficebuilding,for instance), financialconsiderations,thewishesof a client.Art,ontheotherhand,has,sincethe1960s, attemptedcritiquesof masssocietyandin particularof massproductionmethods,which ledit towardstheverypracticesit soughtto critique.WhentheAmericanartistJeff l(oons commissionssculpturesfrom factories,fromartisansor fromotherartists,andthensigns them with his name,he workswith attenuated,abstract,surfaces,:the wholeqlobal
  • 246. Materialityandmeaning ' 225 domainof culturalproductionbecomesthe materialof the work.Choiceservesastne productiontool,andindustrializedculturalproductionasthe materialsurface'0ur focus onphysicalmaterialityshouldthereforebetakenasa metaphor.In thehighlysemioticized worldwe livein,Jeffl(oons'workmaybebothmorerelevantandmoreusualthanthat of Giacomettior Moore,of Rothkoor Ryman.Thenotionof artistsproducingobjects'with theirownhands'datesbackto thepre-industrialperiod,andhadalreadycomeintocrisis in the periodof industrialization.It maybethat in the post-industrialworldit haslost muchof itsrelevance. The relativefreedomof the artist,perhapsgreaterin the visualthan in the verbal, remains,for the time being,in force,albeiton the margins.It liesin the possibilityof foregrounding- eitherof the materialityof the meansof productionor of the object produced,or of thesemioticizationof thismaterialin (referentialor significatory)actsof representation,Wallpaperdrawsattentionawayfrom the materialityof a wall' A room withouta hintof decorationonconcreteor pinewallsdrawsattentionawayfromthefacts of thewall itselfandonto themateria[ityof thematerial' What,then,is the meaningof material?Qur assumptionremainsthat signsare motivated.It is no accidentthat the statueserectedto commemorateheroicfiguresare madeof durablematerials,or that tombstonesare still carved:the durabilityof the materialsmakesthemusablesignifiersforthemeaningsof permanentfeelingsweintendto produce.Norisit anaccidentthatcertainffowersor stonesbecomesignifiersfor love:their rarity may makethem precious/or elsetheir colour,shapeor perfumemay mal(ethem suitablesignifiers.Bone-chinateacupsdo not producethe samemeaningsastin mugs' Australianmoneyis printedon plastic,a shockstill to the returningexpatriate,asif it is tooboldlya signifierof whatshouldnotbesignified. To summarize,materialproductioncomprisesthe interrelatedsemioticresourcesof surface/substanceandtoolsof production.Eachhasitsownsemioticeffects,andintheir interactiontheyproducecomplexeffectsof meaning.Productionexistson manyplanes; that is,thereare serialrelationsbetweensurfaces.As with Barthesinotionof the sign, signsat onelevelbecomeavailableas signifiersat a higherlevel.Andthereare serial relationsof translationbetweenproductionmediaalso,as in the caseof the relation betweenpaintingsandtheirphotographicreproductions. GOLOURAS A SEMIOTICMODE Ourfocussofar hasbeenonmaterialityof thesign- onsurfaces,substancesandtools'We now want to turn to the questionof materialityas a meansfor representationmore centrally.0f coursewehavetouchedonthismanytimes,indirectlyor moredirectty,butour questionnowis,'Howdoesmaterialityactuallyenterintoandshapethe resourcesfor representation,themodes?'Wereferredto thiswhenwepointedoutthatour'grammar'of thevisualcouldnotbe,simply,a transpositionof thetermsof a grammarof thelinguistic mode,becauseit hadto paydueattentionto the materialdifferencesof the resourcefor reoresentation- notthematerialof soundasinspeech,butthematerialof graphicstuffas
  • 247. 226 . Materiality and meaning in images,nottheorderinglogicof timeasin speech,butthai of the spaceof surfacesof images. 0netheoreticalissueto be'got out of theway,,soto speak,isthat of'abstraction,.If transpositionsof linguisticterminologyto othermodeshavebeenquitecommonplace in the past,it hasbeenbecause'grammars,of languagepaidattentionto an abstract entity,'language',anddidnotseeit or itselementsintermsof their materiality.Thatisone reasonwhyphonetics(asa discipline)hadbeenexcludedfromlinguistics(asa discipline). It hadbeentoo muchconcernedwith the physicalityof its domain.It is alsoonereason why for muchof the twentiethcenturymuchof mainstreamlinguisticswasconcerned with 'language' as such,as an abstraction,ratherthan with the distinctivenessof the grammaticalorganizationof speechandwriting.At that levelof abstractionit seemed possibleto movefromonemediumto another:materialitywasnotseen,andhencedidnot figureanddidnotneedto beaccountedfor. In ourapproachthetwinfactorsof materialityandof culture(alwayssetinthesocial organizationsinwhichculturesexist)arethemeanstoanexplanationof theresourcesfor representation.Materialityentersinto semiosisthroughtangiblephysicalfacts:speech happensassound,andsoundhappensin temporalsequence.However,whatculturedoes withthese'factsof nafure'isthenanothermatter.In anytemporalsequence,somethingis first andthereforesomethingelseis secondandsomethingelselast;culturescannotget aroundthat fact. But what meaningsmaybeattachedto that orderingis quiteanother malter,a matterfor makersof signsintheircultures.'Beingfirst,mayhaveanynumberof meanings:'thatwhichis mostimportantto methespeaker,,or ,thatwhichis mystarting point,fromwhereI canproceed',or'the entitywhichis responsiblefor theactionwhichis represented',andso on.Using'speechsounds,meansusingthe possibilitiesof changes inpressureintheair,aswellasthepotentialof changesinthefrequencyof vibrationof the vocalcordsandchangesin thevolumeandfrequencyof oscillationof a columnof airin the'vocaltract'to fashiona semioticresource- speechsoundsandintonation.Again,the sameconsiderationsapply:someculturesusethepotentialsof pitchforsyntactic/semantic means/t0 makequestions;othersuseit to producedifferencesin lexis- in the so-called tone languages,suchas cantonese,Mandarin,Igbo,etc.differencein pitch produces differencein lexis. Whatmateriala culturechoosesto fashionintoa resourcefor makingrepresentations. intoa mode,isa matterof thecontingenciesof thatcultureandof itshistory;thoughit is alsoa matter- a fundamentalpointneglectedintraditionallinguistics- of thebodilyness of humansasmakersandas receivers/remakersof signs.A modeis a meansfor making representations,throughelements(sounds,syllables,morphemes,words,clauses)andthe possibilitiesof theirarrangementastexts/messages.Colourissucha material,andherewe willexplorethequestionof colourasmodeinsocialsemioticterms,lettingcolourstandin for all otherinstancesof therelationsof material,cultureandmodeassemioticresource. Webeginwith a bit of history,to demonstratethepointwe madeabove.In the Middle Ages,pigmentshadvaluein themselves.Ultramarine,as the nameindicates,hadto be importedfromacrosstheseaandwasexpensive,notonlyfor thisreason,butalsobecause it wasmadefrom lapislazuli.Henceit wasusedfor high-valuemotifs,suchasthe manle
  • 248. Materiality and meaning ' 227 of theVirginMary.Suchpigmentswerenot mixed,but usedin unmixedform,orat most onlymixedwithwhite.Thematerialidentitiesof specificpigmentshadto remainvisible, andit wasthesematerialqualitieswhichmotivatedtheir use,their meaning'As a result 'coloLlr'wasa collectionof distinctmaterialsubstances,ratherthana'system'-'lexis' ratherthan'grammar'. Around1600,in Dutchpainting,the technologychanged.Newtechniquesallowed eachoarticleto becoatedin a film of oil,whichinsulatedit againstchemicalreaction with otherpigmentsandmademoreextensivemixingpossible.As a resultthestatusand priceof specificpaintswentdownandcolourbecameto someextentdisengagedfrom its materiality.Colourwas no longerusedandthoughtof as a collection,an extensive catalogueof distinctlydifferentindividualpigments,eachwiththeirownaffordances,and hencesemioticpotentialities,but asa systemwith fiveelementarycolours(black,white, yellow,redandblue)fromwhichall othercolourscouldbemixed'Butthissystemwasnot a semioticsystem.It wasa (practical)physicsof colour,just as phoneticsis a physics of speech.Neverthelessit involvedconsiderableabstraction,andthis madeit possibleto applythe systemto differentmedia,just as the systemof languageis appliedto the mediumof speechaswellasto themediumof writing.Newtondrewa comparisonbetween the elements(tones)and rulesof combination(harmony)of musicand the elements (colours)andrulesof combination('colourharmony')of colour.Colourswouldbe'con- sonant,or'dissonant'onthebasisof thesameintervalsbetweensevenordered'elements' (green,blue,incligo,violet,red,orangeandyellow)asmusic(thesevendifferenttonesinthe octave).In histimeit inspiredCastelto builtan'ocularharpsichord',a typeof experiment whlchhascroppedup againandagainsince- for instance,in the interestin synaesthesia andauditioncoloreeof earlytwentieth-centurypsychologistsand,morerecently/in the workofabstractfilmandcomputeranimationartists.Thepointis,'colour'wasn0longera collectionof materialsubstances/pigments;it became'colour'.As l(andinskywouldlater say,,Colourisonlylooselyattachedto objects.. . . It hasa grammarof itsown,al<into the grammarof music.' What,then,of meaning?Ofcourse,colourhasalwaysbeenusedasa semioticresource. In the MiddleAgesthereweremanytheoreticalandpracticaldebatesaboutcoloursym- bolism.Shouldmonkswearblack(penitence,humility)orwhite(glory,joy)?Buttherewas nounifiedsystem.Greencouldmean'justice'aswellaS'hope',red'charity'aswellaS'life' andsoon.LearnedtomessuchasF.P.Morato's0n theMeaningof Colours(1528)argued withandagainsteachotheraboutthesymbolicmeaningsof colour,andintheirarguments the meaningsof colourwerealwaysmotivated.Greencouldbe the colourof unity,for instance,becauseit wasusedasa backgroundin representationsof theTrinity'Redcould mean'life,becauseit is the colourof blood.Somemodernartistshavetriedto revive thiskindof symbolism.For Malevichblackdenoteda worldlyviewof economy;red,the revolution;andwhite,action.Withtheseelements,hethought,morecomplexideascould be constructed.But as in the MiddleAges,contemporary colourcodes/havelimited domainsof application,andspecificcolourscanhaveverydifferentmeaningsin different contexts.Thework of Malevich,Mondrian,l(andinskyand otherswas an attemptto explorethepossibilityof a broader,morewidelyapplicable'languageof colour'.Butthey
  • 249. 228 Materi ality and meaning didnotmanageto bringsucha ranguageintobeing.As Gagehassaid,theirexperiments 'offeredtheprospectof universality,tbut becamelthoroughlyhermetic,i:-999:24$. So colourhas,on the onehand,developedintoa'mode,,a systematicallyorganized resource.But onthe otherhand,thissystemis a physical,ratherthana semioticsystem, a kind of 'phonetics',althoughthe basicelementsof the system,the 'primary,and'secondary'colours, haveplayeda keyrolein visualsemioticpracticesandin accountsof themeaningof colour.semiotically,a single,system,hasnotdeveloped.'whatpeopledo, withcolourvariesenormously,andsocialgroupswhichsharecommonpurposesaround usesof colourareoftenrelativelysmaliandspecialized- theydo not constitutealarge group/as is the casewith speech,or with the systemsof visualcommunicationwe have discussedinthisbook.Butif westaywiththenotionthat'whatpeopledo,shapesthetools, andbearinmindthatverydifferentthingsaredonebydifferentgroups,wemightbeableto makesomesenseof howcolourbecomesa usableresourcefor makingmeaning.If we relatethemeaningsof colourbothto theirmaterialityandto whatpeopledowiththat,we mightbeableto asktherucial questions:Is coloura modeof representationin itsown right?Doesit offerthefull affordancesof mode? So the task is to discoverthe regularitiesof the resourceof colourastheyexistfor specificgroups;to understandthemwellenoughto beableto describewhattheprinciples for the useof the resourcein signsare (that is,to understandhowa specificgroup,s interestin colourshapestheirsignsof colour).Fromthatwe mightbeginto understand generalprinciplesof the semiosisof colourandof semiosisgenerally,andthesein turn mightprovidea principledunderstandingof all usesof colourinallsocioculturaldomains. THE COMMUNICATIVEFUNCTIONSOFCOLOUR In Halliday'smetafunctionalsemiotictheory,a communicationalsystemsimultaneously fulfilsthreefunctions:theideationalfunction,thefunctionof constructingrepresentations of the world;the interpersonalfunction,the functionof enacting(or helpingto enact) interactionscharacterizedbyspecificsocialpurposesandspecificsocialrelations;andthe textualfunction,thefunctionof marshallingcommunicativeactsintolargerwholes,into the communicativeeventsor textsthat realizespecificsocialpractices.We canaskthe questionsthat we haveaskedof imagesgenerallyof colourspecifically.can it create specificrelationsbetween'participants'; thatis,betweenrepresentedpeople,places,things andideas?Canit representsocialrelationsandhelpenactsocialinteraction?Andcanit realizetextualmeanings- for instance,in a systemof referenceor in creatingcohesionin an text? In theprecedingchapterswehave,wehopereasonablyplausibly,appliedthismodelto a numberof resourcesof visualcommunication(compositiont gazetangleandsizeof frame, andsoon),thereby(re)constitutingtheseresourcesaspartofthe'grammaticalsystem,of images,in Halliday'sterms.we didnot,however,inthefirstversionof thisbook,dealwith colourinthisway,becausewefoundit difficultto assigncolourplausiblyto justone_ and onlyone- of Halliday'sthreemetafunctions.It istruethatthereisa dominantdiscourseof
  • 250. Materiality and meaning ' 229 colourinwhichcolourisprimarilyrelatedto affect,andHallidayandothers(e.9.Poynton, 1gg5;Martin,IggD seeaffectasan aspectof the interpersonalmetafunction.Butthe communicativefunctionof colouris notrestrictedto affectalone.Wethinkthatcolouris usedmetafunctionally,andthat it isthereforea modein itsownright. Startingwith the ideationalfunction,colourclearlycan be usedto denotepeople, placesandthingsaswell asclassesof people,placesandthings,andmoregeneralideas. Thecoloursof ffags,for instance,denotestates,andcorporationsincreasinglyusespecific coloursor colourschemesto denotetheir uniqueidentities.Car manufacfurers,for instance,ensurethat the dark blueof a BMW is quitedlstinctfrom the darkblueof a VW or a Ford,andtheylegallyprotect'their'colours,sothatotherswill notbeableto use them.Evenuniversitiesusecolourto signaltheir identities.The 0pen University,for example,stipulates: Twocolours. . . for formalapplicationssuchashigh-qualitystationeryanddegree certificates- blue(referencePMS 300) for the shieldand lettering,and yellow (PMS123)forthecircularinset.Singlecolourstationeryshouldbein blue(PMS 300)if possible. (ouotedin GoodmanandGraddol,r996tll9) 0n maps,colourscanserveto identifywater,arableland,deserts,mountains,andsoon.In uniforms,colourcansignalrank.In the safetycodedesignedby US colourconsultant FaberBirren(Lacy,I996:75),'9reen'identifiesfirst-aidequipment,while'red'identifies hosesandvalves(whichplaya role,of course,in fire protection).0nthe LondonUnder- ground,'green'identifiesthe DistrictLineand'red'theCentralLine,andonbothUnder- groundmapsandin Undergroundstationsmanypeoplelool<for thosecoloursfirst,and speakof the'greenline'andthe'redline'. Colouris alsousedto convey'interpersonal'meaning:it allowsusto realize'colour acts' (as languagepermits'speechacts').It can be and is usedto do thingsto or for eachother:to impressor lntimidatethrough'power-dressing"to warnagainstobstructions andotherhazardsby paintingthemorange,or to subduepeople- apparentlythe Naval CorrectionalCentrein Seattlefoundthat 'pink,properlyapplied,relaxeshostileand aggressiveindividualswithln15 minutes'(Lacy,1995:89).Accordingto the Guardiads 'OfficeHours'supplement(3 September2001:5): ,Coloursare verypowerfulandcanreduceor raisestresslevels,'believesLllian Verner-Bonds,authorof ColourHealing.Brightredsare energisingandare good for officesin the bankingor entertainmentfields.Greenis usefulif there'sdiscord or disharmonyasit issoothing.Blueisratedasthebestcolourfor promotingcalm andpastelorangeisgoodfor gentlyencouragingactivity. Elsewherein thesamearticlewe learnthat addingcolourto documentscanincreasethe reader'sattentionspanby morethan eightyper centandthat'an invoicethat hasthe amountof moneyin colouris thirty per centmorelikelyto be paidon time than a
  • 251. 230 Materiality and meaning monocol0urone/.In allthesecasescolourrepresents/projects,enablesor constructssocial relations- it is interpersonal.It is not just the casethat colour'expresses/or 'means' thingssuchas'calm'or'energy';rather,peopleactuallyusecolourto try to energizeor calmdownotherpeople.Puttingit moregenerally,colourisusedto actonothers,to send managerialmessagesto workers,or parentalmessagesto children,aswehaveshowninan analysisof a child'sroom(l(ressandvanLeeuwen,200t).ltisusedbypeopleto present themselvesandthevaluestheystandfor,to sayinthecontextof specificsocialsituations,'l am calm/or'l amenergetic',andto project'calm'or'energy'aspositivevalues.Wewill addressthisin moredetailbelow,whenweanalysetheuseof colourinhomedecoration. Colouralsofunctions,maybeevenmostobviously,atthetextuallevel.In manybuildings, the differingcoloursof doorsand otherfeatures- the colourschemesof floors- dis- tinguishdifferentdepartmentsfrom eachotherontheonehand,whilecreatingunityand coherencewithinthesedepartmentsontheother.Colourcanbeusedto createcoherencein texts.Textbooksmakewideuseof this,whetherin 'readingschemes'or in mathematics textsto indicate'levels'of difficulty,or in sciencetextbooksto providetopicalunity.In Pasos,a Spanishlanguagetextbook(Martfnand Ellis,200I), the chapterheadingsand pagenumbersof eachchapterhavea distinctcolour,all sectionheadingsCVocabulario encasa','Gramdtica',etc.)are redthroughoutthe book,andall'activities'(e.9.'Make phraseswith esor est6')havea purpleheadingandnumber.In an issueof the German editionof Cosmopolilan(November200I), film reviewshaveorangeheadlinesandother usesof orangeinthetypography,asthe backgroundof textboxes,etc.Theart reviewsuse greenin a similarway,bookreviewsusered,andso on.In somecasesthis is cuedby a salientcolourin the keyillustrationof thefirst pageof the relevantreviewsection;for instance,CateBlanchett'sorangehair in a still from thefilm Banditsin the film review section. 'Colour-coordination',ratherthanthe repetitionof a singlecolour,can be usedto promotetextualcohesion.In thiscasethevariouscoloursof a page,or alargersectionof a text (or of an outfit,or a room),mayhaveroughlythesamedegreeof brightnessand/or saturation.lncomputersoftwaresuchasPowerPoint,thisfeatureisalreadybuiltin,a kind of analogueofthespell-checker,showingjustthedevelopmentinthedirectionof a broader useof grammar.Choosingtheinitialbackgroundautomaticallyselectsa rangeof colours asa colourscheme.Forinstance,if theinitialcolourisa pastel,thentheothercolourswill alsobepastels.It is possibleto overridethisbyselectinganothercolourfroma Munsell atlastypedisplay,butthistakesmoreeffortandskill. Thereare two pointsto make.First,colourfulfils the three metafunctionssimul- taneously.Thecoloursona mapretaintheirideationalandtheirinterpersonalvalue,their appealingbrightness,or stuffydullness;on mapscoloursare coordinatedto enhance textualcohesion.And contemporaryscientificvisualizationsarethoughtof as primarily ideational:veinsandarterieswill be representedusingdifferentcoloursto indicatethe amountof oxygen- or the levelof itsdepletion.Second,we arenotarguingthatcolour alwayshas and alwayswill fulfil all threeof thesefunctionsequally.Colourdoes whatpeopledowith it, in makinga signandin remakingthesignin itsreception.Weare not'discovering'universalandsuprahistoricalfactsaboutcolourhere.Wearetryingto
  • 252. Material ity and meaning 23f documentwhatkindof communicativeworkcolouris madeto do in todav'sincreasingly globalsemioticpractices,and how.The examplesprovidean indicationthat someof theseusesof colourhavefairlyspecific,limiteddomains,wheretheyclearlyrelateto the specificinterestsof sign-makers(e.g.map-making,subduingprisoners)whileothers may havewiderdistribution(e.g.the useof colour-codingin magazinesas a meansof cohesion). Finally,thecentralquestion:If weareright,if colourfulfilsallthreemetafunctions,isit a semioticmodein itsownright,alongwithspeech,image,writing,music?Maybe.But is therenotalsoa difference?Language,imageandmusichavebeenconceivedof (andhave in various'purist'practicesoftenoperated)as relativelyindependentsemioticmodes. Althougha novelisa materialobject,anda pagea visualartefact,itscommunicativework is doneprimarilythroughwriting.In anart galleryimagesusuallycomewithnowords- thesmalldescriptivesignonthewallnearbyis nota partof theimagebuta partof the environmentof displayin the galleryor museum.In the concerthalleverythingis con- centratedon the music,whileexpressionthroughsemioticmodessuchas dress,bodily performance,etc.,is heldback,certainlyby comparisonto contemporarypopularmusic shows.Thereisa choicefortheaudienceto focusonthemusic'assuch'oronthewholeas 'performance',the'concert'. Isthisthecasewithcolour?Paintershavetriedto makepaintingsthatuseonlycolour and nothingelse('fieldpainting',Rothko,etc.),but this doesnot appearto haveledto a wholenewartform.Thenagain,maybecolouris a characteristicmodefor the ageof multimodality.It cancombinefreelywith manyothermodes,witharchitecture,typography, productdesign,documentdesign. Letusstepbackfor a moment.Asoneof uswritesthis- it isa dayin mid-August,just beforelunchtime- I amsittinglookingoutthroughtheopenFrenchwindows,ona tranquil Frenchcountryside.I seelowhills,trees,forestinthebacl<ground,someCharolaiscattlein thepasturebeyondthefence.Therearea veryfewfluffyclouds,thoughaboveandbeyond theforestedhillthereisa denserbankof cloud,justappearing,mal<ingmethinkof a late afternoonthunderstorm.I am describingthissceneby selectingout specificelements, namingthem,puttingthemhereonthescreenof mylaptopaswordsin a particularorder. I haveavoidedusinganycolourwords. Nowletmetry thisagain.I amlookingoutthroughtheopenFrenchwindowsat a world of colour.Overwhelmingly,greensof themostvariedhuesdominate,thoughtherearegreys of variouskindsandbrowns,darkpurples,blues,off-white.Everythingthat I seeI seeas colour.And if I representit hereon my screenagainas words,it is becauseI have translatedthe world as I seei/ intothe modethat my culturehasmademostreadily availableto me.UsuallyI don'tevenregardthat actionastranslation,but asrepresenta- tion:thatishowmyculturehastaughtmeto understandit.Theelementsofthistranslation codehaveto beinscribed- firstonto thisscreeninwaysI donotunderstand,lateronto a printedpage- thoughuntilabouteightyearsagoor soI wouldhavetranscribedthemonto thepaper-pagedirectly,usinga pen. I might,however,havelearned(or mightstilllearn)to paint.I wouldbeableto repre- sentthisscene- it wouldof coursenot bea recording(evena photographwouldnot be
  • 253. 232 Materi ality and meaning that) - in a modewhichis closerto the mannerof my perception.Colourwouldbe representedbycolour,whereasat themomentcolouris representedbywords(insyntactic order).Thecolourswouldof coursebeorganized,asblocks,splashes,lines,dots:thegreys andbrownswouldappearasthinner,differentlyverticalelements- asthe trunksof the treesI see,or asthe ffecksof greyin the bankof cloud,the greensof varioushuesand brightnessas leavesandbladesof grass,andthe purplesasthe dotsof varioussizesof plumsandripeningelderberries.Colourwouldappearentirelybyitself- onaninscriptional surface,of course,noless,butalso(arewerightinsaying?)nomoresothanthewordsI used aboveneedthe inscriptionalsurfaceof pageor,if spoken,the inscriptionalsurfaceof air and,if heard,thereceptororgansof theear. In my banalaccountof this framedsegmentof the landscapeI usedwordsas my descriptionalresource/havingbecomeso usedto it that it alsoservedas my meansof analysisof thecountryside.Themodegavemethetermswithwhichto analysethatwhich I saw,andit gavemethe meansfor itsdescription.Themodeof colour- if weseeit asa mode- wouldgivemedifferentterms(notof courseasherein mytranscriptionaswords), if I wereableto paintwhatI seehavingmixedmyowncolourson mypalette,usingnowa quitedifferentsetof analyticalanddescriptive'terms'. Noneof this is new;andthe Impressionistswerejust one'school'ofpainterswho workedwith ideassuchasthese,evenif moresubtlythoughtandexpressed,andfocusedon thematerialityof lightratherthanof colour.However,whatwouldbenewforusnowisto seecolourfor what it is andwhat it does.Doescolourhereexiston its own?Well,yes, of course- at leastas muchas do wordsspokenor written.Onceposedand seenin thiscontexttheanswerbecomessomewhatoddlyself-evident.Cancolourbeor becomea modeonlyina multimodalenvironment?Well,yes,inthesameway- nomorenoless,even if differently- as everyothermode.And the experimentsof Mondrian,of Rothkoor Nicholson,asof others,wouldnowbeseennotsomuchasexperimentsin turningcolour intomode,butasexperimentsinabstractingawayfromthe(attempted)realismsof blocks (astreetrunks),slashes(asbladesof grass),lines(asedgesof all kinds)anddots(berries or plums),turningrealismof theideationalkindintoitsabstraction. A DISTINCTIVEFEATUREAPPROACHTOTHE SEMIOTICSOFCOLOUR In l(ressandvanLeeuwen(2001),we arguedthat signifiers- andcoloursaresignifiers, not signs- carry a set of affordancesfrom whichsign-makersand interpretersselect accordingto their communicativeneedsand interestsin a givencontext.In somecases theirchoicewill be highlyregulatedby explicitor implicitrules,or by the authorityof expertsandrolemodels.In othercases- for instance,intheproductionandinterpretation of art - it will be relativelyfree.In our briefanalysisof the useof colourin home decorationbelow,weshowhowinmostsituationsthesetwopoles,constraintandcreativity, arebothinevidenceandmixedincomplexways. Like l(andinsky,we distinguishtwo typesof affordancein colour,two sourcesfor makingmeaningwith colour.Firstthereis association,or provenance- the questionof
  • 254. Materiality and meaning . 233 'wherethecolourcomesfrom','wherewehaveseenit before'.Theassociationstakenupin manyof the communicativeusesof colour,suchas in advertisingor the entertainment media,will usuallybewithsubstances,objects,etc.thatcarrysignificantsymbolicvaluein the givensocioculturalcontext.Whilethe affordancesof a colourmaybewidein theory, in practicetheyare not whenthe contextof productionand interpretationis takeninto account,aswewill try to do intheanalysisbelow. Thesecondtypeof affordanceis that of the 'distinctivefeatures'of colour.Herewe wantto showsomeaspectsof theaffordancesof thematerialityof colour,andhencemake a connection,notwiththe'grammar',butwith phoneticsandphonology.In Jakobsonand Halle'sdistinctivefeaturephonology(1956),thefeaturesnamedrealmaterialphenomena - suchas the pointof articulationof a consonanl,or the apertureof the mouthin the makingof a vowel- andin descriptiontheyweredeployedasoperatingin opposition.So oneconsonantcouldbedistinguishedfrom anotherbyan opposition,suchas/+voiced/as againstl-voiced/,an oppositionwhichwoulddistinguish/b/ from/p/, or /d/ from /t/, in English.Wefocuslessonoppositionthanonthequality,thecharacteristicsof thefeatures, andtalk aboutvalueson a rangeof scales,suchasthe scalethat runsfrom lightto dark, thescalethat runsfrom saturatedto desaturated,andsoon.UnlikeJakobsonand Halle, we seethesefeaturesnot just as servingto distinguishdifferentsoundsor coloursfrom eachother,but aboveall as meaningpotentials;that is,as their potentialto become signifiers.Anyspecificinstanceof colourcanbedefinedasa combinationof specificvalues oneachof thesescales- andhencealsoasa complexandcompositemeaningpotential,as wewilltry to showbelow. Value Thescaleof valueis the greyscale,thescalefrom maximallylight(white)to maximally dark(black).In thelivesof all humanbeingslightanddarkarefundamentalexperiences, andthereis no culturewhichhasnot builtan edificeof symbolicmeaningsandvalue systemsuponthisfundamentalexperience- eventhoughdifferentculturesmayhavedone soin differentways.Painterswhoemphasizevalue- for instance,Rembrandt- areoften ableto exploitthismeaningpotentialincomplexandprofoundways. Saturation Thisis the scalefromthe mostintenselysaturatedor'pure'manifestationsof a colour to its softest,most'pale'or'pastel',or dull anddark manifestations,and,ultimately, to completedesaturation,to blackand white.Its key affordancelies in its abilityto expressemotive'temperatures',kindsof affect.It is the scalethat runsfrom maximum intensityof feelingto maximallysubdued,maximallytoned-down,indeedneutralized feeling.In contextthis allowsmany differentmore preciseand stronglyvalue-laden meanings.Highsaturationmay be positive,exuberant,adventurous,but alsovulgaror garish.Lowsaturationmaybesubtleandtender,but alsocoldandrepressed,or brooding andmoodv.
  • 255. 234 MaterlaIity and meaning Purity Thisisthescalethatrunsfrommaximum'purity,tomaximum,hybridity,,andit hasbeenat the heartof colourtheoryas it developedoverthe last few centuries.Manydifferent systemsof primaryandmixedcolourshavebeenproposed- somephysical,somepsycho- logicalandsomea mixtureof both- andthissearchfor primariesor basicshasnotresulted in a generallyacceptedsystem,but'hasprovedto beremarkablyinconsequentialand. . . freightedwiththeheavyburdenof ideology'(Gage,1999:107).Somewritershaveseenthe issueascloselyrelatedto thequestionof colournames.Colourswithcommonlyusedsingle names/suchasbrownandgreen,wouldbeconsideredpure.Thenamesof othercolours,like cyantaremainlyusedbyspecialists,andnon-specialistswouldreferto thembymeansof a compositename,forinstance,blue-green.Suchcolourswouldthenbeperceivedasmixed. Termslike'purity'and'hybridity'alreadysuggestsomethingof themeaningpotential of thisaspectof colour.The'pure'brightreds,bluesandyellowsof the'Mondrian,colour schemehavebecomekeysignifiersof theideologiesof modernity,whilea colourschemeof pale,anaemiccyansand mauveshasbecomea key signifierof the ideologiesof post- modernism,inwhichtheideaof hybridityispositivelyvalued.Thisisbynomeanstheonly wayin whichtheaffordancesof thisscalehavebeentakenup,but it isa culturallysalient one,andhenceonewhichiscurrentlyquitewidelyunderstood. Modulation Thisisthescalethat runsfromfullymodulatedcolour(forexample,froma bluethat is richlytexturedwith differenttintsandshades,as in paintingsbyClzanndto flat coJour, (as in comicstrips,or paintingsby Matisse).It wasalreadyrecognizedas a featureof colour in Goethe'sFarbenlehre(Theoryof Colour)(1970 tlBt0l). The affordances of modulationarevariousand,again,stronglyvalue-laden.Flatcolourmaybeperceivedas simpleandboldin a positivesense,or asoverlybasicandsimplified.Modulatedcolour, similarly,maybeperceivedassubtleanddoingjusticeto the richtextureof realcolour.or as overlyfussyanddetailed.And,aswe havediscussedin chapter5, modulationis also closelyrelatedto theissueof modality.Flatcolourisgenericcolour,it expressescolouras an essentialqualityof things('grassis green'),whilemodulatedcolouris specificcolour ('thecolourof grassdependsonthetimeof dayandtheweather'),it attemptsto showthe colourof people,placesandthingsasit isactuallyseen,underspecificlightingconditions. Hencethe truth of ffat colouris an abstracttruth,andthe truth of modulatedcoloura naturalistic,perceptualtruth. Differentiation Differentiationisthe scalethat runsfrom monochrometo the useof a maximallyvaried palette,and its verydiversityor exuberanceis oneof its keysemioticaffordances,as is the restraintinvolvedin its opposite,lackof differentiation.In our analysisof an article froma homedecorationmagazinebelow,a couple'usesnearlythewholespectrumintheir
  • 256. Materiality and meaning . 235 house'andcommentthat'it'sgreatthattherearesomanybrightshadesinthehouse.It's a shamepeoplearen'tmoreadventurous.It's whenyoustart beingtimid that thingsgo wrong' (HouseBeautiful,SeptemberI99B: 2I). So here high differentiationmeans 'adventurousness'andlowdifferentiation'timiditv'.but it isclearthatin anothercontext restralntmighthavea morepositivevalue. Hue Thisisthescalefromblueto red.In a distinctivefeaturetheoryof colourit becomesonly oneof thefactorsconstitutingthecomplexandcompositemeaningsof colour,andmaybe noteventhemostimportantone.Nevertheless,although'the'meaningof red-in-general, of the abstractsignifier'red',cannotbe established,the red end of the scaleremains associatedwith warmth,energy,salience,foregrounding,andthe blueendwith cold,calm, distance,backgrounding.Thecold-warmcontinuumhasmanycorrespondencesanduses. Itten(1970)liststransparenVopaque,sedative/stimulant,rareldense,airy/earthy,farlnear, lighVheavyand wet/dry.In an actualred,meanwhile,its warmthcombineswith other features.An actualred may,for instance,beverywarm/mediumdark,highlysaturated, pureandmodulated,andits affordancesfor sign-makersandsigninterpretersflowfrom all thesefactors.In thenextsectionwewill seehowsuchsetsof affordancesareactually takenupin a specificcontext,andwhatcontext-specificinterestsandvaluesareat workin thisprocess. H0ME DEC0RATION:G0L0UR.CHARACTERANDFASHION Whatcoloursareusedin homedecorationandwhy?Theanswerdependsonthe socio- culturalcontext.Therehavebeenmanydifferenttraditions,including,for instance,regional differences,suchasthebrightbluesandgreensof thedoorsandwindowsof farmhousesin Staphorst,a villagein the Netherlandswheretraditionaldressis stillworn.Buttodaya newapproachhasdeveloped,in whichthe expertiseof the colourconsultantplaysa key role.Accordingto Lacy(J996l,29),theentrancehallof a homesignalsthe identityof its owneror owners: A yellowentrancehallusuallyindicatesa personwhohasideasanda widefieldof interests.A homebelongingto an academicwouldprobablycontaina distinctive shadeof yellowasthiscolourisassociatedwiththeintellect,ideasanda searching mind.... A greenentrancehall- say,a warmapplegreen- indicatesa homein whichchildren,familyandpetsareheldin highimportance.. . .A blueentrancehall indicatesa placeinwhichpeoplehavestrongopinions- therecouldbea tendencyto appearaloofastheycanbeabsorbedtoo muc.{rintheirownworld. In expertdiscourseof thiskindthecoloursof a homeaboveall expresscharacter,express theidentity,thepersonalcharacteristics,andthevaluesandinterestsof thehomeowneror
  • 257. 236 MateriaIity and meaning owners/whilethecoIoursofworkpIaces(andprisons,schooIs,etc.) aremoreoftendiscussed intermsof theireffectsonworkers(prisoners,students,etc). Mostpeoplewill encounterthis discoursein magazinesandtelevisionmakeoverpro- grammeswhereit is mediatedbyjournalists,althoughtheexpertiseof colourconsultants and interiordecoratorsis oftenexplicitlydrawnon. In magazinesaimingat different sectorsof themarket,differenttypesof homeownersor celebritiesmaybeintroduced(for instance,the ownerof a Londonart galleryversusan actor in a popularsoapopera). Comparethefollowingtwoquotes: Herlatesthabitat(shemovesas regularlyandhappilyasa nomad)is surprisingly spareandelegant,asyoumightexpectfromsomeonewitha senseof theaestheticin hergenes.Afterall,Jane'sgreatauntwasNancyLancaster,of ColefaxandFowler fame,whileherbrother,HenryWyndham,ischairmanof Sotheby,s. (ldealHomeandLifestyle,September1998:60) Guessingwhat HamishandVanessaDowsdofor a livingisn'ttoo difficult- a pair of feet on the housenumberplateis a deadgiveawayfor a couplewho are both chiropodists,but it'salsoan indicationof thefunthey'vehaddecoratingtheirhome. (HouseBeautiful,September1998:20) In sucharticlescolourchoiceis presentedas an originalanduniqueexpressionof the characterandvaluesofthehomeowners-asfullypersonal,ratherthanmediatedbysocial codes.Thetwofun-lovingchiropodistsabove,for instance, usenearlythewholespectrumin their house,from mustardyellowand leafgreen in the sittingroom,to brickredandbluein the diningroom.Theirbedroomis a softbutteryyellowcombinedwith orange,there'slemonand limein the breakfast roomandcornffowerandWedgwoodbluesonthestairs.'lt'sgreatthatthereareso manybrightshadesin the house',saysHamish.,It,sa shamepeoplearen,tmore adventurous.It's whenyoustartbeingtimidthatthingsgowrong., (HouseBeautiful,September1998:21) Thisshowsthereaderhowcoloursemiosisworks,butat thesametimeavoidsthesugges- tionthatsuchmodelscanbeslavishlyfollowed,andsuggeststhatcoloursemiosisshould naturallyffowfrompeople'suniquecharacterandvalues.It isherethattheaffordancesof colouraretakenup.Highcolourdifferentiationandhighsaturationbecomesignifiersof 'adventurousness',with differentiationstandingfor theabsenceof monotonyandroutine, andsaturationfor anintensityof feeling,for'livingto thefull'andnotbeing'timid'. A lookattheactualcoloursintheillustrationsofthearticle(e.g.plate6) showsthatthe distinctivefeaturesareselectivelyusedin the discourse.Thereare,onthosebrightwalls, paintedgold leavesand sunffowerswhich,Hamishand Vanessasay,'givesucha lovely Victorianfeel'.Indeed,the photosshowa veryclutteredinterior,with manyretroobjects, includingfringedlampshadesandstatuettesof servileblackservants.But.evenwithout
  • 258. Materiality and meaning . 237 thequoteandwithouttheseobjects,theprovenanceof the leavesandsunffowerswouldbe clear.Whilethecoloursmaybehighlysaturated,theyarealsorelativelydarkandrelatively impure,certainlyby referenceto Modernistbrightandlightinteriors(andMondrian-type purecolours), andthisaspectof thecolours,theirprovenanceas'historic'colours,isnot explicitlydiscussedinthearticle. Such'historic'colourswereverymuchin fashionin the 1990s:'Thespecialistpaint firm Farrow& Ball whosecolourswere usedto recreateeighteenth-and nineteenth- century Englandin televisionadaptationsof Pride and Preiudiceand Middlemarch, reportsthat its saleshaveconsistentlyrisenby 40o/oeachyearoverthe pastten years' (GuardianWeekendMagazine,19 January2002: 67).lt may be that Hamishand Vanessa'sinterioris not just an originalexpressionof their character,but alsofollows fashion,andalsotakesits cuesfrom the media.It maybethat HamishandVanessanot only usethe affordancesof the distinctivefeaturesof colourto expresstheir unique interestsandvalues,but alsobasetheirchoiceof colouron'provenance',andthereby alsoexpressthevaluesof the placeandthetimewherethesecolourscomefrom.It may be that Hamishand Vanessa,throughthe way they decoratetheir home,symbolically identifywiththevaluesof thatera,andwiththe nostalgiafor a'lost' Englishnesswhich hadbeensosalientthroughoutthe 1990s.In thisarticle,thisisexpressedin a covertway in whichcolourplaysan absolutelycrucialrole.It maybethatwhatseemssouniquely theirownissocialtyconstructedin andthroughthemedia,in discoursesrealizedthrough colour. COLOURSCHEMES In this lastsectionwe will discussonefinalexamplein orderto focusbrieffyon the questionofthecolourscheme.Theexampleisa pamphletproducedto describeandexplain thecorporateidentitychangeof a majorpublishinghouseinihe Ul( (plate7). A numberof modesareinvolved- colour,typeface,iconsof severalkinds.Thepamphlet brieffydescribesthe functionof each.In the caseof colour,a captionstatesthat 'The colourpaletteprovidesa harmoniousselectionof 16 colours,alI carefullychosento com- plementthecorporatecolourPalgravesilver,andtheyshouldbeusedwhereverpossible.' Sothe deliberatenessandlntentls clear.Ratherthanthetraditionallayoutof thecolour chart,herethecorporatecolouriscentral,to indicateitsstatusandrole,andthesubsidiary coloursclusteraroundit ina regulardisplay.Thisclusteringisorganized- inpart- onthe principleof gradationsin hue,thoughgiventhe colourschosenthiscannotbeachieved entirelyin the mannerof the traditionalchart; there are gaps.Coherencehas been deliberatelyaimedfor:allthehueshaveto beableto collocatewiththecorporatec0lour, in its support.Thereis,consequently,alreadya strongsenseof structure- both in the explicithierarchyof coloursandin the delimitingof the rangeof permissible'units'.But onecolourchangesthe overalleffectof the scheme,the bright-yellowtop left.Its intro- ductionineffectmakesallthecoloursin itspalettedifferentintheirmeaning-potentials'It putsthiscolourpaletteintothedomainof thesharp,thebright,theupbeat- inthewords
  • 259. 238 MateriaIity and meaning of thepamphlet,'activepursuitof ideas','rapidchange','a worrdof challengesto bemet,, 'a newcompanyanda globalforcein publishing,,etc. Today,coloursincreasinglyare coloursin a'colour scheme,,coloursin systemsof colourwhichcan be definedon the basisof specificusesof the distinctivefeatureswe havediscussed.We havecomeacrossseveralsuchschemesalready:the 'historic,colour scheme,basedondifferentiation,relativelyhighsaturationanddarkvalue;the modernist 'Mondrian'colour scheme,basedon purityandhighsaturation;the postmoderncolour scheme,basedon hybridityand pastelvalues.All thesecolourschemeshavedistinct historicalplacements.Buttheyliveon beyondtheir historicalperiodas recognizedsemi- oticresourceswhichcancontinueto beusedandcombined(for instance,thebright-yellow accentin the overallpostmodernschemeof Palgrave)to realizedistinctlvdifferent ideologicalpositions.
  • 260. I T h et h i r d d i m e n s i o n THETHIRDDIMENSI0NTFROMnREADER'T0 'USER' So far we haverestrictedourselvesto still ratherthan movingimages,and to two- dimensionalformsof visualcommunicationratherthanthree-dimensionalonessuchas sculpture,productdesign,architectureor stagesetdesign,In thischapterwewill explore to whichdegreethedescriptiveframeworkwe havedevelopedin thisbookcanbeapplied also to the three-dimensionalvisualand the movingimage.0ur three-dimensional exampleswill mainlybedrawnfromthefieldsof sculptureandchildren'stoys.Toysareof particularinterestastheyoccupya spacesomewherein betweensculptures,whichare primarilysymbolicobjects,objectsfor contemplationand veneration,and'designed objects',whichare primarilyobjectsfor use,eventhoughtheymayalsoconveysymbolic messages.In additionwewill considereverydayobjectssuchascupsandmotorcars,and architecture.Thechapteris intendedas a first exploration.We are not able,withinthe spaceof this book,to presenta systematicaccountof all aspectsof three-dimensional visualcommunication,asthiswouldrequiretoo manyadditionalconcepts(someof these arediscussedinvanLeeuwen,2OO3).Tnstead,wewill concentrateontheconceptswehave describedin the precedingchapters,to showwhat rolethey play in three-dimensional visualcommunication.In thiswaywe will at leastbe ableto indicatein whichways three-dimensionalvisualcommunicationissimilarto anddifferentfromtwo-dimensional communication,andto outlinethetheoreticalissueswhichfollowfromthis. Startingwith the issueof visualrepresentation,we foundthat the key categories we introducedin chapters2 and3 canbeappliedalsoto three-dimensionalvisuals,and do indeedseemsufficientto describea widerangeof suchobjects.Manysculptures,for example,havewhatwe calledin chapter2 a'narrative'structure.TakeEpstein'sJacob andtheAngel(n946J,showninfigure8.1:thearmsof JacobandtheAngelformpowerful vectors,relatingthetwo participantsin a complexandinterestingway.TheAngel'saction istransactional.It hasa Goal,asheholdsJacobina firmgrip.ButJacob'saction,though foregrounded,is non-transactional- hisarm hangslimply,andhedoesnot holdor grab anything. l(ennethArmitage'sPeoplein the Wind(1952),shownin figure8.2,alsohasstrong vectors,formedbythewaythefiguresarebentforwardsastheystruggleagainstthewind. But here(as in sculpturesof discusthrowers,ballerinasand otheractivesubjects)the actionis'non-transactional'.Thevectorsdo not pointat or leadto anotherparticipant,a Goal.Thefigures,it seems,'strainforwards',buttheydo not'straintowardssomething'' 'Reactions'arealsocommon/althoughin sculpturethe eyesdo not usuallyformas stronga focusof attractionastheydo in two-dimensionalimages,becausetheylackthe strongtonalcontrastbetweenthewhitesof theeyesandthe pupils,which,in picturesas in nature,makeseyessosalient.It isasif theaddednaturalismofthethirddimensionmust
  • 261. 240 . Thethird dimension Q fig e.f Jacoband theAnge,f(JacobEpstein,1940)(cranadaTelevisionLtd) be counteractedby greaterabstractionin othermeansof expression/suchas colourtto preventsculpturefromcrossingthelinebetweenart andtheuncannilyrealmake-believeof thewaxworksshow(or of certaincontemporaryformsof art). Nevertheless,ina sculpture likeRodin'sTheKiss(lBBO),themanandthewomannotonlyholdeachotherwiththeir arms,theyholdeachotherwiththeirgazeaswell,ina'transactionalreaction/,andit isnot difficultto findexamplesof 'non-transactionalreactions'also.Jacob(figureg.1)looksup, ina non-transactionalreaction,whiletheAngellooksat Jacob,ina transactionalreaction. Again,theAngelactsonJacob,butJacobdoesnotactontheAngel.Arnheim,sdescription of Michelangelo'sMosesprovidesanotherexample.'Thedeffectionof thelawgiver,shead andthefierceconcentrationof hisglanceintroduceanobliquevectorthatmoves0urwaros likethebeamof a lighthouse.Butnogoalobjectisincluded,(Arnheim,r9B2:4(). Eventhe designof objectsand buildingscan be vectorialand hence,narrative,, as shownin figure8.3.Thevectorformedbythetailfinsof 1950scars,for instance,repre- sented(in an abstractway) the ideaof dynamicmotion,as if it was not enoughthat carsare in fact dynamicmovingobjects,regardlessof whethertheyhavetailfinso( not. Thefactthatcarsdonothavetailfinsnowpointsto theideologicaldimensionof sculptural representation:themeaningsrepresentedwerethoseofthe'jet age,.Buthereanimportant
  • 262. Thethird dimension O fis O.z Peoptein the tyird (KennethArmitaqe,1952)(TateGallery)
  • 263. 242 The third dimension V E C T O R I A L D E S J G N Lightshades @ fiq e.: Vectorialandnon.vectoriallightshades,cups,carsandbuildrnqs complicationoccurs.In the caseof objects,the ideationalrelationswe havediscussedin chapters2 and3 canberealizedintwoways:theycanberealizedbythedesigne4asforms to be'read'by a viewer,as whena cup has a'dynamic',vectorialhandle(andalso,of course/inthecaseof thepictorialor decorativedesignsprintedor paintedon,or moulded or carvedin,thecup);or theycanberealizedbytheuserof theobject,aswhenthecupis heldor drunkfrom,ina'transactionalaction'withitsuser-thevectorialhandleisthena 'non-transactionalaction'fromthe pointof viewof thedesignof thecup,anda potential for transactionalaction,a means,fromthepointof viewof itsuse. Reactionscanevenoccurin objects.Thetoy telephoneshownin figureg.4 not only includesthedog'stongue,asanobliquevectorsignifyinga non-transactionalspeaking;it alsohaseyes.Theinclusionof eyesis in fact quitecommonin toysfor youngchildren, particularlyin toyswiththethemesof time(clocks),communication(toytelephones)and transport(toy locomotivesandcars),as if to encouragethe childto form an emouve, personalizedbondwiththesethreekeytechnologiesasearlyaspossible. N O NV E C T O R I A L DE S I GN E= ,"/'77I tJ.-.H Buildings J -?(:!>
  • 264. The third dimension 743 Q riq e.nToytelephon! In contrastto two-dimensionalvisuals,sculpturesrarelyincludea Setting.Theirsetting istheenvironmentin whichtheyaredisplayed,a gallery,a nichein a church,or a public square.It isnota representedSetting.0fcourse,sculpturescanincludea Setting,asinthe worksof EdwardI(ienholzandGeorgeSegal,for instance,or Asiansculpturegardens, suchastheTigerBalmGardensin Singapore.ButincontemporaryWesternsculpturethe inclusionof a Settingis relativelyrare.Decontextualization,it seems,hasto counteract the addednaturalismof the third dimension.Sculpturesdo,however,oftenhaveanother participant,the pedestalon whichthey stand.Suchpedestalscan be (mere)framing devices,creatinga degreeof separationbetweenthesculptureanditsenvironment,andso enhancingits statusasa representation,an objectfor contemplation,setapartfrom its environment.Buttheycanalsoandat thesametimeformpartof therepresentation,asin Canova'sPaoIinaBorghese(1805),whichhasPaolinarestingona couchwhichformsalso a sarcophagus-likesupportfor herrecliningbody.Theabsenceof suchframingcanhavea strongeffect,as in the lifesizebronzeof a middle-aged,corpulentmanin a raincoatand hatplacedasif minglingwiththeshoppersinthemiddleofthefootpathof a busyshopping streetin Amsterdam. Turningnowto 'conceptual'ratherthan'narrative'structuresof representation,these, too,canbefoundinsculpture.Mir6's Woman(I970), reproducedinfigure8.5,iswhat,in chapter3, wecalledan'analytical'representation.Thesculpturedoesnotjustplaywith theformsof foundobjects,mal<ingeyesof the headlightsanda mouthof thewindscreen of a car;it isalsoan'analysis'of'Woman'.'Woman',in all itsgenerality,isthe'Carrier', andtheparts,the'PossessiveAttributes',are,in Mi16'sconception:a headwhichisalso the emptyshellof a car;an upperbodywhichis alsoa tray on whichtwo aggressively pointedbreastsare presentedto the viewer;anda lowerbodywhichis a barrel-shaped containerwith a vagina-lil<eslit andtwo handlesto holdher by.Whenwe visitedan
  • 265. 244 Thethird dimension @ fig A.S Woman(JoanMi16,1970)(Parellada,Barcelona) exhibitionof Mir6'ssculptureswenotedthatthemisogynisticqualityof Mir6,s'analysis, wasnot loston theviewers.Therewasa guestbookin whichthevisitorsof the gallery couldwrite downtheir impressions.lvanyhad usedthe opportunityto draw a quick caricatureof a Mi16womanwitha contemptuouscommentsuchas'wounru????,These readingscontrastedsharplywiththeartspeakinthecatalogue,whichdescribedtheformal qualitiesofthesculpturesonly,anddidnotdwellon Mir6'swayof representingthefemale gender. Giacometti'sHour of the Traces(1930),shownin figure8.6, is an analysisof the genderlesshumanbeing,of thehumanconditionin general.Theparts:a kindof antenna, with an abstracteye,formsan activesensorytentacleandprotrudesfrom the sculpture at an angle,constitutingan obliquevector;a rigidrustyframe,thebody;and,withinthe
  • 266. Thethird dimension . 245 O fis g.e Hourof the Traces(AlbertoGiacometti,1930)(TateGallery)
  • 267. 246 . The third dimension openframe/a plasterheart,suspendedona thinstring,andmovingslightlyto andfro.Thus twoactionsareembeddedintheanalysis,bothnon-transactional:theactionofthesensory apparatus,andthe movementof the heart- the humanbeingasa skeletalframethat is aliveandsurveysitsenvironment. Wemightaddthat analyticalsculptureis usednotonlyin art but alsoin science- for instance,to showthe constructionof a molecule;or as a teachingaid,for instancein anatomy,inwhichcasethepartscanoftenbedetachedfromthewhole.Thekineticdesign of sculpturesandotherobjects,the waytheycanmoveor bemadeto move,takenapart andputbacktogetheragain,andsoon,isa subjectto whichwecannotdojusticeinthis chapter,as it wouldagaindemandthe introductionof a newsetof concepts(butseevan LeeuwenandCaldas-Coulthard,2004). Thethird dimensioncreatesan additionaloptionin representation,a relationbetween therepresentationalstructureandthepositionof theviewer.SeenfromthesideEpstein's JacobandtheAngel(figure8.1) hasa narrativestructure('transactionalaction').It is in thefirstplaceaboutwhatJacobandtheAngeldo.ButifwelookattheAngelfrombehind (figure8.7),wearefacedwithan'analysis'of theAngel,anda verystrikingone:thethree O fiS g.Z JacobandtheAngelfrombehind(JacobEpstein,1940)(GranadaTeteyisionLtd)
  • 268. Thethird dimension 247 principal'PossessiveAttributes'EpsteinemphasizesaretheAngel'slonghair,hiswings- andhisballs. Not all sculpturesusethis possibility.0necan imaginea continuumrunningfrom reliefs,whichperhapsdifferfrom two-dimensionalimagesonlyin termsof modality,to fully 'multifaceted'sculpturessuch as Jacob and the Angel.ln betweenthere are sculptureswhich,thoughfree-standing,areclearlynotdesignedto beseenfrombehindand leavetheback'unworked',perhapsbecausetheyweremeantto beplacedagainsta wall or ina niche.Andevenwhena sculptureisa fullymultifacetedrepresentation,likeJacob andtheAngel,its placementin a particularenvironmentcanblocl<accessto alternative viewingpositions,and henceto alternativereadings.Thismay be becausethe work is placedwith itsbacl<againstthewall,or becauseabarrierpreventstheviewerfromaccess to otherthanmoreor lessfrontalviewingpositions.But it mayalsobedonein subtler ways.Whenwe first analysedJacoband theAngel,it was placedin the centreof the octagonalentrancehallof theTateGallery,in sucha waythattheviewerfirstsawit from the side,with the Angelon the left.In otherwords,its positionfavouredthe narrative reading,the dramaof thesculpture,ratherthanEpstein'sstril<ing'analysis'of theAngel. Buttheviewerdidhaveaccessto theothersides,asthesculpturewasplacedinthecentre of thehall. PlaymobilfiguressuchasthoseshowninfigureB.Barealsoanalyticalstructures.They showthesignificantattributes,thesignificantcharacteristics,of (for example)an'ethnic family'.Thefamilyhasfivemembers- a father,a motherandthreechildren.Eachmember of thefamilyhasblackhairanddarksl<in.NotethedifferencefromtheneutrallyIabelled 'familyset':thecompositionof thefamilyisthesame,andall themembersof thefamily havepinkskin,but theydifferin the colourof theirhairandthereforehaveindividual characteristicsaswellassocialcharacteristics,whereas(lessonnumberone)themembers of the'ethnicfamily'are'others'who'alllookthesame'.Thechildrenof thetwofamilies are dressedidentically,but the parentsare not.Lessonnumbertwo: second-generation immigrantsarealready'morelikeus'.ThePlaymobilcompanybrochuresaysthatthese @ fig e.e Playmobil'familyset'and'ethnicfamily'
  • 269. 248 f he third dimension toys'formanaidto thetrainingof yourminds'andwill'acquaintchildrenwithwhatthey will meetinthebigrealworld'- butnotinanentirelyneutralfashion.Aswehavesaidin chapter3,anyanalyticalstructureisonlyoneof the manywaysinwhicha given'carrier, canbeanalysed. Wehaveto remember,of course,that Playmobilfiguresenterintorepresentationintwo ways,likeourearlierexampleofcupswithvectorialhandles(figure8.4.0ntheonehand, theyarelikesculptures,pre-designedrepresentations,to be'read'bythechild;ontheother hand,theyhavemovablelimbsanddetachablepartsandtheycan holdobjectsin their hands.Childrencanthereforeusethemto createa varietyof representationalstructures, narrative'scenes',andtheycanevensubvertthepre-designedrepresentations,for instance by givingan'ethnic'childred hair.Theycan alsocreatetheir own classificatoryor analyticalarrangements,for instancebymakinga displayof differentkindsof Playmobil childrenontheirtoyshelfor bycreatinga newanalysisof the'family',withonlya motheri perhaps,or withfivechildrenof different'ethnic'origins(thearrangementof )ur Society and0thers,figure3.29,couldbereconstructedwith Playmobilfigures!).lnthesameway a cupcanbeused,notonlyinthe'transactions'of holdingor drinking,butalso:to create ananalyticalstructure,aswhenthecupisarrangedona sideboardtogetherwiththeother partsof the setto whichit belongs,or on the kitchenshelf,to becomea'possessive Attribute'of the 'carrier' 'dishes';or to createa classificationalstructure,as whena numberof differentcupsare arrangedsymmetricallyin a shopwindowor in a design exhibition. The vitrinen(displaycases)of the Germanavant-gardeartist JosephBeuysare an intriguingexampleof theclassificationalsculpture.Theyarethekindof sculpturewhichis usuallyreferredto asan'installation'-glassdisplaycasescontaininga varietyof objects, someof themalteredby Beuys.vitrine2 (1960-7o, for instance,containsa film can (itselfcontaininga film whichfeaturesa performanceby Beuys),a pairof boxinggloves,a sausage/a cassetteplayer(with a tapeof musicperformedby Beuys),a Beethovenscore with a smallblackboarderaseron it,twowineglasses(onecoatedwith a whitesubstance, the otherlookingas thoughsomethinghasbeenburntin it), anda zincbox.clearlya work like this raisesthe questionof what theseobjectshavein common(whattheir 'superordinate'is,in our terms),andthis is exactlythe questionBeuys,interpretershave asked.Accordingto Theewen(1993: r39), verwandschaft('relatedness,)is the keyto understandingBeuys' vitrinen:'By bringingrelatedobjectstogetheran association betweenthem is created',and in the caseof the vitrinewe havejust describedthis associationisthat all theobjects'haveplayeda rolein performancesof Beuys,andtthat theylallcontainsomething'i993: 29). Thethirdkindof conceptualstructurediscussedinchapter3 wasthesymbolicrelation. we sawhowin picturesan overallcolour- a bluehaze,or a goldenglow- couldrealize what we calleda'suggestivesymbolic'process,endowingthe depictedscenewith an overallsignificance,'saying',as it were,'thissceneis coldanddesolate,(in the caseof thebluehaze)or'theseobjectsareveryvaluable'(inthecaseof thegoldenglow).Colour canplaythisrolealsoin thecaseof three-dimensionalobjects- thinkof thecoloursof cars,for instance,thedifferencebetweena black,a bright-redanda whiteMercedes,sav.
  • 270. The third dimension 249 Butinadditionto colourthereareotherfactors,suchasthematerialfromwhichanobject is made,thewaythesurfaceof a sculptureor otherobjectis'worked',or theoverallshape of the objects,in so far as theseare not determinedby considerationsof naturalistic representation,or bythefunctionsservedbyanobject.Giacometti'sManPointing(1947) ismuchless'analytical'thanthesculptureswehavesofardiscussed.It placesnoemphasis on the distinctnessof the partsof the humanbody.Eventhe facialfeaturesare hardly stated- smallindentationsto indicatea mouthandeyes.Thesculpturedoesof course havea clear vector,as the man is makingan expansiveoratoricalgesture.The most strikingcharacteristicof this sculpture,however,is its rough,black,craggysurface.It is difficultto put into words exactlywhat is suggestedby this surface,but whatever transcodingwe attemptit will haveto expresssomehowthat this figure is'weather- beaten',affectedbyexposureto - butherewecanfill in a numberof things- theelements suffering,ageingandsoon.Thesculptureis abstractenoughto allowall thesereadings andmore. The sameapplies,again,to otherl<indsof three-dimensionalobjects.Cupscan be smooth,madeof delicatechina,suggesting,perhaps,an overallqualityof eleganceand refinement;or theycanbesturdyandsolidandmadeof brick-redterracotta,suggesting, perhaps,an overallquality of down-to-earthsimplicity.Cars can be elongatedand streamlined,suggestingpowerandspeed,or,as in the caseof the currentlyfashionable 'retro'look,roundedandegglike,suggestinga safecocoon,a l<indof womb. Thesecondkindof symbolicrelationwediscussedwasthe'symbolicattributive'pro- cess,whereonerepresentedparticipanthasnootherfunctionthanto endowanotherwith symbolicsignificance.Thisoccurs,for instance,in someof Mir6'ssculptures,wherebirds andeggsattributesymbolicqualitiesto the figuresdepicted(usually'women').But it occursalsointoysfor youngchildren,wheretelephonescanhavewheels(anearlylesson abouttheconceptof communicationas'transportof information'and'bridgingthedis- tance'ratherthan'sharingof information'),and'interactivelearningcentres'forchildren agedzt/zto 5 ('6 built-infunctionswhichteachthe alphabet,numbers,shapes,colours, soundeffectsand nurseryrhymes',and all this for 829.50)havea steeringwheeland dashboard,asa symbolof the powerandcontrolaffordedby knowledge.0r in adulttoys: emblemsandotherdecorationsoncars/for instance.0rin architecture,wheresculptures andmuralscanbecomesymbolicattributesfor buildings. 0n thewhole,then,wefeelthattheaccountof visualrepresentationwe havepresented in chapters2 and3 canbeappliedto three-dimensionalvisualcommunication.Yetthere aresomesignificantdifferences.First,three-dimensionalobjectscanbeplacedon a con- tinuumwhichrunsfromobjectsthatallowonlyonereading(byofferingthe readeronly oneaspect,usuallythefront)to objectswhichallowmorethanonereading,dependingon thepositionof theviewerrelativeto theobject. Second,three-dimensionalobjectscan be placedon a continuumwhichrunsfrom objectswhichhavebeendesignedonlyto belookedat,onlyto be'read',to objectswhich enterintorepresentationalrelationsinthreeways:(1) therelationsencodedinthedesign of the objectitself,to be'read'onlybythe viewer;(2) interactiverelationsbetweenthe objectandils user(e.g.holdingthecup,or drinkingfrom it); and(3) conceptualrelations
  • 271. 2 5 0 The third dimension createdby the user(e.9.creatinga classificationalsyntagmwith a numberof different C U P S ) . Third,evenwhenanobjectdoeshavea potentialfor multifacetedrepresentationand/or for being'used'aswell as'read',externalconditionscan inhibitthis potential,block theviewer'saccessto alternativereadingpositions,or to interactiveengagementwiththe representationalpotentialof theobject. INTERAGTIVEVIEWING Wewill nowturnto the interactiverelationswe discussedin chapter4,tryingto explore, again,how applicablethey are to three-dimensionalvisualcommunication.In that chapterwedistinguishedbetween'demand'picturesfromwhichrepresentedparticipants addressthe viewerdirectlywiih theirgazeand'wantsomethingfrom the viewer,,and 'offer' pictureswhichpositionthe vieweras an observeronly,andofferthe represented participantsas'information'tobetakeninbytheviewer. Clearly,thisdistinctioncanbeappliedalsoto sculpture- but,again,witha difference. HenryMoore'sRecumbentFigure(7938),shownin figure8.9, addressesthe viewer @ fig a.e RecumhentFigurc(HenryMoore,1938)(Tatecailery)
  • 272. Thethird dimension . 25I powerfully.Althoughtheeyesarelittlemorethanindentationsinthesurfaceof thestone, thewholeattitudeof thefiguresuggestsa concentratedlook.But,asviewers,wecantake upa positionfromwhichthatlookwilldirectlyaddressus(asdidthephotographer,inthe caseof figure8.9),sothatthe pictureformsa'demand';or a positionfromwhichthe figurelookspastus,at somethingelse,or at nothingin particular,inanycase,at something not includedin our view,and in that casethe lookwill becomea'non-transactional reaction'.In thetwo-dimensionalmediumwecannot,asviewers,decidewhetheror notwe willallowourselvesto bedirectlyaddressedbya representedparticipant;thedecisionhas beenmadefor us.Inthethree-dimensionalmediumwecan- thatis,if theplacementofthe sculptureallowsusto doso.In theTateGallery,Moore'ssculpturecouldhavebeenplaced in sucha waythat the figure'sgazewouldfix the viewerimmediatelyuponenteringthe room.Butthiswasnotdonewhenweviewedthesculpturethere,andasa resultthefigure becamejust oneof a numberof Moore'sworks,presentedas part of a classificational syntagm,andfavouringthe'offer'ratherthanthe'demand'. Thesamewouldbetrueofthetoytelephone(figure8.4).Thegazeofthistelephonecan onlybecomea'demand'byvirtueof anactivedecisiononthepartof itsuser.Sometoys,of course,lendthemselvesmoreto thisthanothers.Playmobilcharactershavesmallblack dotsfor eyes.Theyare biasedmoretowardsthe'offer for informatlon'thantowards the interactive'demand'.Andthe eyesof many'boys"dolls(Batmen,Crash-dummies, Megazords,etc.)areoftenobscuredbyhelmets,masksor darl<glasses.Theeyesof 'girls'' dolls(andof manycuddlyanimaltoys),on the otherhand,tendto be largeandhighly detailed.Whileboysaresteeredtowardsa moremanipulativerelationto theirdolls,for girlsthe look,the interactivedimension,is madeto mattermore.Andthe sameis true forveryyoungchildren:eventheirbedclothes,pillowcases,cups/platesmayhaveeyes,and arethuspersonalized,animated,capableof enteringintoa'directaddress'relationwith the child.Aswith manyotherthings,someof thismaywellliveon,unconsciously,in the adultrelationwithobjects. Thesamereasoningcanbeappliedto theotherinteractivedimensionswediscussedin chapter4. In principletheviewercandecidewhetherto seetheobjectfromcloseup or froma distance,frontally(hencewith'involvement')or fromanobliqueangle(hencewith 'detachment');fromabove(hencefroma positionof powerovertheobject)or frombelow (hencefroma positioninwhichtheobjecthaspowerovertheviewer).Wesay'inprinciple', becausehere,too,theviewer'schoicemayberestrictedbyexternalfactors,bybarriersthat preventviewersfromcomingupcloseor seeingtheobjectfroma differentangle.Andlarge objectscanmakethehigh-angleviewpointandtheclosedistanceimpossible.Whattowers overus has,by design,poweroverus,and is, by design,sociallydistant:the vertical dimensionis the dimensionof powerandreverentialdistance,the dimensionof 'highly placed'people,placesandthings.In thisconnectionit isalsosignificantthatsculptures,as worksof 'high'art,cannotusuallybeapproachedfromthe mostintimatedistance,the distancethat makestouchingpossible:assoonasthe galleryvisitorcomestoo close,a guardwill becomealert. Whensculpturesaretakenout of theiroriginalcontextandmovedintoanother,their interactivemeaningsmaychangesignificantly.Theymaybe,literally,tal<endownfromthe
  • 273. 252 The third dimension pedestal- in a churchperhaps- wherethey wereto be lookedat from below,with reverence/to bemovedintoa gallery,wheretheyarepositionedat a levelof equality,and viewedfroma more'familiar'distance:Michelangelo'sDavid,removedto therotundaof a museum/nolongercallsto thecitizensof Florenceandisunawareof theircallingonhim, andnow'canbeexploredbytheviewer,but makesnoadvancesto him,(Arnheim,1982: 50;alsoHodgeandl(ress,1988:2OI-3). MODALITYIN THREEDIMENSIONS In chapter5 we describedvisualmodalityas resultingfromthedegreeto whichcertain meansof pictorialexpression(colour,representationaldetail,depth,tonalshades,etc.) are used.Eachof thesedimensionscanbeseenas a scale,runningfrom the absenceof any renditionof detailto maximalrepresentationof detail,or from the absenceof any renditionof d