Stadon Digital Simulations and Real Time Applications Conference 2011
Syncretic Post-Biological Digital Identity: Hybridizing Mixed Reality Data Transfer Systems Julian Stadon Raphael Grasset Curtin University Graz University of Technology way that requires rethinking everything from spirituality, time andABSTRACT space, esotericism, agency, emergence, quantum coherenceThis paper offers a contribution to an emerging culturally through to eroticism .orientated discourse regarding mixed reality interaction. It seeksto analyse syncretic, hybridized agency, particularly in mixed Innovative actualisations of real-time data transfer systems thatreality data transfer systems. Recent developments in bridging incorporate biological information and mixed reality applicationsautonomous relationships with digital representation through as artistic mediums are a creating a new notion of post-biologicalmixed reality interfacing, have brought about the need for further digital identity. Post-biological, in this sense, refers to aanalysis of these new ‘post-biological’, hybridized states of being redefinition of the embodied subject which encompasses theirthat traverse traditional paradigms of time and space. Roy location in virtual environments as well as in the physical. ThisAscott’s concept of syncretism may facilitate further involves the creation, through art practice, of what we might termunderstanding of multi-layered world views, both material and autonomous agents that are born from data but which take on themetaphysical, that are emerging from our engagement with such appearance of bio-forms and thus become embodied. At the samepervasive computational technologies and post-biological time these agents are a differential embodiment of the ‘bodies’,systems. Syncretism has traditionally been regarded as an attempt which first generated that data in their everyday activities.to harmonise and analogise  Citing recent examples of practicalresearch outcomes, this paper will analyse what Gilles Deleuze This interrogates the meaning and consequences of the possibilityand Fèlix Guattari have called ‘deterritorialisation’ of the human of such ‘agents’ and, in doing so, enables us to question the notionbody through its dispersion throughout multiple reality that information, once extracted from the embodied self andmanifestations and how mixed reality data transfer might placed within a computer system, becomes ‘bodiless’. In posingconstitute a ‘reterritorialising’ effect on syncretic post-biological that question we discover that, contrary to what we might at firstdigital identity construction . assume, data is also embodied. The existence of ‘embodied information’, linked to and yet not the same as embodied selves,KEYWORDS: Art, Media art, Performing Arts, Metaverse, dual creates an interface through which humans negotiate theirreality, cross-reality, transhumanism, post-humanism. identities across the boundaries of different reality states, more or less virtual, and yet always involving the mapping or writing ofINDEX TERMS: H.5.2 [Information Interfaces and Presentation] that identity onto ‘a body’. By having bodies both material andMulti Media Information Systems-Artificial, Augmented and virtual, humans have become post-biological even as their biologyVirtual Realities; I.3.6 [Computer Graphics] Methodologies and remains the primary point of reference for the data gathering,Techniques-Interaction Techniques; J.5 [Computer Applications] which enables this transition to occur.Arts and Humanities-Fine Arts.1 INTRODUCTION 2 BACKGROUNDIt is a popular belief that we are now, through a media convergent, Current research in mixed reality and interactive workspaces thatparticipatory culture (that is integrated socially through a sub uses the concept of a bridge for data transfer have continued thenetwork of platforms) creating what was first coined in 1997 as development of new knowledge in this field, however the majoritycollective intelligence by Pierre Lévy , which exists in a ‘global of previous research in this area has been in the field of computervillage’  of knowledge (data) transfer. This perspective evades science. The application of cultural and philosophical discourse totraditional mythological notions of anthropomorphic interaction as recent developments in computer science will propose new modesit moves beyond the individual and into a universal model of open of representation that concern themselves with the affectiveaccess. Networked agency destabilises traditional orthodoxies of capacities of art in order to articulate a sense of dispersedthought through challenging notions of representation, embodiment. Unlike traditional sites for communication andconfronting materialism, accelerating and smoothing social cultural exchange, digital platforms rely on actions andengagement and most importantly, demanding participation in conversations to shape not only the social and culturalthese open systems of collaborative engagement. This has environments, but also the spatial environments. Such systemsredefined our understandings of consciousness and presence in allow participants to physically interact with virtual (deterritorialised) biological representations and mediate 18 Ethel Street North Perth, Western Australia 6050 (reterritorialise) through physical engagement, rather than firstname.lastname@example.org entering traditional text or numerical based data sets and command sequences. A good example for mixed reality would be would be the Layar application for smartphones. This application, COPYRIGHT BLOCK available for free download to any smartphone user, provides an advanced augmented reality platform capable of reliably
delivering many different AR experience, though largely focusing manner. The hypersurface is the site on which bridges are built:on using geolocation to augment the user’s physical surroundings. where the real and virtual, material and textual, author and agentFor example, Manifest.AR, an international artist’s collective can meet and interact with each other. Performance technologyworking with emergent forms of augmented reality as public art, theorist Gabriella Giannachi states that, “The hypersurface is ause the technology to transform public space for users. They zone of exchange between consciousness (language and text) andinstall virtual objects and artworks that respond to and overlay the levels of the inorganic… Able to present dichotomousconfiguration of located physical meaning. The application uses relationships, between representation and matter, inside andgeolocation software to superimpose computer generated three- outside, organic and inorganic, the hypersurface is the site ofdimensional art objects, enabling the public to see the work virtual performance .” For the construction and exploration ofintegrated into the physical location as if it existed in the real mixed reality to occur interfacing is required to bridge the virtualworld. Thus, the Layar application reterritorialises information environment with the physical so that both spaces can be mediatedprimarily through geolocating individuals, in order to provide a in an autonomous manner. The hypersurface is the site on whichricher engagement with their physical surrounds through the bridges are built: where the real and virtual, material and textual,layering of virtual content over real time video. author and agent can meet and interact with each other.Massively Multi User Online Worlds (MMOs) are another 3 EXAMPLES FROM COMPUTER SCIENCEexample of an open virtual environment that allows for thecontribution to and manipulation of private and publicly owned The earliest example of a research project that proposed avirtual space through a variety of methods. The content of these hypersurfaced system for data transfer using mixed reality was inenvironments is dependent on the participants, due to this open 1999 Butz et al.  proposed a drag and drop technique betweeninteraction, and therefore relies on the quality of information an augmented reality space to a screen space within the EMMIEtransfer methods being used. Through collaborative creative system. Using a mirror metaphor, virtual objects would change representation and dimensionality by passing through screenproduction MMOs facilitate social engagement and further boundaries, their approach focused on transferring documents.collaborative production by its participants. Spatial developmentsdefine the environments and the (real or virtual) individuals Recently Lang et al.  from Georgia Tech University modifiedinhabiting such spaces through their participation in and response Second Life to create mixed reality experiences the purpose beingto them. The collective construction of such virtual meeting sites, the creation of a novel augmented reality environment forfor remote interpersonal interaction acts as an instrument of entertainment. This example bridged reality states in a way thatlocation and orientation, referential to the real world of facilitated a further inquiry into the socio-cultural implications ofknowledge. such systems, but was never addressed in the research publications. The VTT Technical Research Center Finland hasImplementing biological and physical data into MMOs through also recently worked with hypersurfacing Second Life avatarsaugmented reality, contributes new knowledge in regards to within physical experience through the Meeting Avatars jointbridged mixed reality states, under a paradigm of post-biological project with IBM and Nokia.  By using the Second Life engine,deterritorialisation and reterritorialisation of the body. Deleuze virtual avatars had the same appearance and behavior as in theand Guatarri discuss deterritorialisation in terms of dispersed virtual world but in their context be represented in a physicalresemblance and identity. In Difference and Repetition Deleuze meeting room. Barakonyi and Schmalstieg  created two pilotintroduces the notion of deterritorialisation (through dispersion) as systems in order to facilitate proactive multi-user interfacea “dark precursor” that “relates heterogeneous systems and even adaptation and user interface migration. The system wascompletely disparate things .” In order for deterritorialisation to developed in order to migrate tasks across a range of autonomousoccur there must be some form of agent that can remain constantand self-referent. Deleuze and Guatarri state that: “The alignment agents and a number of users, rather than a single avatar being used by each individual. The goal was to increase the versatility ofof the code or linearity of the nucleic sequence in fact marks a ubiquitous agency through mixed reality data bridgingthreshold of deterritorialisation of the “sign” that gives it a newability to be copied and makes the organism more deterritorialised (hypersurfacing). By increasing the number of agents (in various reality states) that can autonomously perform tasks set by users,than a crystal: only something deterritorialised is capable of the bridge defines a dedicated space where the viewer can transferreproducing itself .” objects and images between worlds, spaces, and contexts. KolevaVirtual reality’s hybridization with physical and biological et al  explored navigation between real, augmented and virtualarchitecture is constructed by the methods used to connect the worlds by establishing “mixed reality boundaries” and proposed aenvironments. The combination and cohesion of heterogeneous model of how space, boundaries can be represented.elements is generally problematic, particularly when a three Schnädelbach et al  further generalized the concept to anydimensional space is primarily viewed on a two dimensional architectural construct, how collaboration and communication canplane. The integration of virtual elements and physical be established in this type of environment. Finally, Grasset et alenvironments relies on bridging the two spaces with dynamic proposed in  and  a general conceptual model how tointerfaces that are simultaneously accessible and able to be openly represent spaces, navigation and the different step of a transitionengaged with, edited and developed. To create integration systems between contextsthat network physical and virtual data shared location are required These research examples articulate a range of different solutionsin order to represent the data in a meaningful way, that is that have been proposed for technological developments in theinclusive of both environments. field of computer science, and often neglect the philosophical and theoretical impact of such technologies on human subjectivity,For the construction and exploration of mixed reality to occur representation, identity and social discourse. The collaborationinterfacing is required to bridge the virtual environment with the between computer science and art seeks to establish a hybridizedphysical so that both spaces can be mediated in an autonomous
practice capable of traversing fields in order to provide a richerdialogue. Emerging technologies often develop faster than we The work uses real-time motion-tracking technologies with ahave the ability to understand them. When these technologies, unique pipeline application to create a mixed-reality soundscape.particularly imaging systems across science and lived experience This audio environment is mediated through interactions betweenbecome creative mediums, they redefine the ways by which we the viewer, the physical environment, and other participantsdefine humanity. within a hypersurfaced mixed-reality feedback loop. As visitors negotiate a traditional public environment -the entrance and4 EXAMPLES FROM ARTISTIC PRACTICE surround to the Somerville Auditorium at UWA- data regarding their movements and interaction with others present is gatheredAn example of a researcher that creates shared mixed reality and translated into sonic outputs, both in the physical and virtualsystems of exchange is telematic artist Paul Sermon. Sermon’s environments. In the physical, the output is via stereo speakersearly work explored the emergence of user-determined narrative installed in the space and in the virtual, a three dimensionalby bringing remote participants together in a shared telepresent representation shadows and echoes sonic and visual traces of theenvironment. Through the use of live chromakey and video real-time dialogues into UWAs Second Life Environment.conferencing technology, two public rooms or installations andtheir audiences are joined in a virtual duplicate that turns into a The individual experiences an intimate interaction with the workmutual space of activity. Currently Sermon’s practice examines and social environment where they control the soundscapethe concepts of presence and performance within Second Life and through their actions, thereby conducting their own personal song.what he calls ‘first life, and attempts to bridge these two spaces Meanwhile, each community that forms also produces uniquethrough mixed reality techniques and interfaces . The notion tones. Movements of individuals between groups results in a sonicof telepresence is explored through a blurring between ‘online’ symphony of social interaction that shifts dynamically accordingand ‘offline’ identities, and the signifiers and conditions that make to the social dialogues that occur in the space. Stepped tonalus feel present in this world. His research questions how outputs are produced by audience movement in space: a scroll tosubjectivity is articulated in relation to embodiment and the right will cause the pitch to drop with each step. The pace ofdisembodiment. Sermon creates hypersurfaces through which data the movement determines the speed of the notes; lingeringcan oscillate between two reality states in an autonomous way: conversations produce long lingering sounds while the rush ofpresent and telepresent. The development of such a method of busy passersby results in fleeting melodies that come and go justdata exchange creates an interesting situation where both the user as quickly.and an autonomous agent (their avatar) can now affect visual datain a mixed reality environment. Co-author Julian Stadon’s ownresearch practice is inspired largely by the work of theaforementioned examples.4.1 TERRA(SOCIO)SONIATerra(socio)sonica is a mixed-reality interface which realises asonic soundscape constructed via the movements of communitiesthat inhabit two current landscape realities that constitute theUniversity of Western Australia (UWA) cultural precinct.Through the translation of movement into sound in both thephysical and virtual realms, the work explores the notion ofunspoken ‘silent dialogues’ under a paradigm of socialengagement. Individuals and large clusters of people produceamplified sounds and shadows based on their oscillatingmovements within a defined social landscape. Figure2. Technical Specifications Diagram for Terra(socio)sonica 4.2 ORGANTRADER2010 organtrader2010 is a novel mixed reality interface that allows for the transfer of real CT scanned organs into augmented reality and Second Life. Using the metaphor of organ trade to allude to Figure 1. Second Life view of virtual ‘shadows’ from physical data traditional gallery hierarchies, organtrader2010 allows the
participant to donate, sell, buy or steal virtual organs acrossplatforms including an interactive mixed reality system, standardSecond Life interfaces and mobile platforms. organtrader2010uses the organ trade metaphor to question the meaning ofownership and the relationship between content and property. Inregards to (unregulated) machines of production and thesubversion of power hierarchies, organtrader2010 examines theroles of media artist/supplier, gallery/distributor andparticipant/trader. In doing so, the project exploresdeterritorialisation of the body and posthuman identity in mixedrealities.organtrader2010 uses a narrative representational structure in amixed reality context where a participant, wearing a cameramounted HMD (head mounted display) can transfer real CTscanned organs to an augmented organ ‘trader.’ This augmentedSecond Life avatar can exist in both physical and virtual spacesimultaneously, so when the participant hands over one of theirorgans to the ‘augmented trader’ they are also giving their organsunsuspectingly to the ‘in-world avatar.’ This avatar is linked to anetwork of organ trader avatars that all have ownership Figure 4. Technical Overview Diagram for organtrader2010permissions to clone and steal organs from the augmented traderand sell them to other Second Life avatars. The system uses an XML RPC to link an augmented reality application with Second Life via a PHP server. This pipeline allows for a method of real-time transfer of 3D visual material, linking the body with the augmented and virtual representations of itself. The use of real CT scanned organs with fiducial marker and proximity tracking adds to the viewer’s experience of agency within the processes involved in the simulated organ trade and in the process of media art creation, display and dissemination. 5 SYNCRETIC POST-BIOLOGICAL IDENTITY At the recent First International Conference on Transdisciplinary Imaging at the Intersections between Art, Science and Culture (TIIC) Roy Ascott gave a keynote in which he described Syncretism as a possible method by which to classify mixed reality interaction . He used Second Life as an example of a metaverse that allowed for an embodied syncretic participatory experience. Second Life, like all virtual environments uses an avatar (agent) to navigate users through the space. While these are usually controlled by the user (avatars can be automated and left on their own, plus there are bots being regularly created and used), they function as independent to their ‘master’ and are therefore autonomous. Returning to the example of Facebook, this can also be said for user profiles on that platform. Who we represent Figure 3. Installation setup for organtrader2010 ourselves as on social networks is not necessarily a true articulation of our identity by any means and therefore it isAs mentioned above, the organs in organtrader2010 are obtained autonomous. Avatars represent a transient, continually alteredthrough real CT scans and are made by converting data into a 3D identity, usually that of its author and acts as an agent, throughmodel, then converting this model to Open Scene Graph. They are which users can engage with virtual platforms. This becomesthen included in a Python-based application that uses the particularly interesting in unique autonomous systems whereOSGSWIG python wrapper for the ARToolkit to enable the participants can physically interact with a virtual deterritorialisedaugmented reality system to occur. To bridge this application with ‘self’ and mediate it through physical engagement. The dispersionSecond Life, data is streamed in to the Linden Scripting Language of multiple autonomous virtual agents via mixed reality constructsvia the PHP server using XML. PHP provides the potential to in expands deterritorialisation to include reterritorialisation, byextend the application network to include mobile devices and facilitating a dispersive relationship between the body and itsmultiple reality environments with the system. The actual virtual self-referent. In the same way that a digital deviceorgantrader2010 application can even be installed to Python deterritorialises and reterritorialises information through binaryenabled platforms for multiple mixed reality participation. code, the augmentation of an autonomous agent into a shared space with the body, creates new opportunities for investigation into technology, the body and identity.
Critical literary theorist Donna Haraway relates the body’s of thought through challenging notions of representation,augmentation through digital technology to the notion of the confronting materialism, accelerating and smoothing socialcyborg. In A Cyborg Manifesto she argues that the body can be engagement and most importantly, demanding participation inviewed as a conglomerate where its components can be separated, these open systems of collaborative engagement.combined with new elements and put together again in ways thatviolate its traditional boundaries . This rhetoric implies a As art is fundamentally an articulation of the human condition itfractured identity that articulates a ‘cyborg’ reality. In Chaos can therefore be said that syncretism is also a valid method forBound, literary theorist N. Katherine Hayles refers to the notion of analysing identity within the post-biological discourse. If we aredispersed self in light of virtual bodies and narrative, arguing that indeed post-biological then we must exist in syncretic mixedby turning bodiless information into narratives, the teleology of reality state. The hybridisation of augmented reality and virtualdisembodiment is replaced with contests with ambiguous environments with physical/biological systems calls for aoutcomes: “As I have argued, human being is first of all embodied rethinking of not only posthuman ideologies, but also the way thatbeing, and the complexities of this embodiment mean that human cybernetic systems function. This paper has scoped a range ofawareness unfolds in very different ways than intelligence in examples from varying fields of inquiry that have influenced thecybernetic machines .” author’s own practice, which is articulated in order to provide a The advent of nanobiology has called for a rethinking of range of practical outcomes to what has been discussed. ThroughHayles and Harraways’ post-human discourse through it shifting the creation of systems that engage the viewer in a hybridizedour perception of organisms from micro to nano scale. Charles participatory interaction with mixed reality data transfer, theseOstman suggests: "[T]he very definition of life itself may be notions of deterritorialisation, reterritorialisation, syncretism andperched on the edge of the next great revolution in medicine- post biological identity can be explored in a more intuitive andnanobiology. What is emerging now are technologies and involved fashion.applications in the arenas of biomolecular components integratedinto microscale systems, . . . synthetically engineered quasi-viral REFERENCEScomponents, modified DNA and related pseudoproteins,  Roy Ascott, 2010. Syncretic Dialogues: Keynote address atbiomolecular prosthetics, and biomolecular organelle component Proceedings of The first International Conference onentities . . . [that] will redefine the very essence of what is Transdisciplinary Imaging at the Intersections between Art,commonly referred to as life .” Critical theorist Colin Science and Culture. Published by Transdisciplinary ImagingMilburn relates nanotechnology to virtual environments, stating: Conference 2010 Sydney, ISBN: 978-0-9807186-6-9“Nanotechnology thrives in the realm of the virtual. Throughout  Deleuze and Guatarri, A Thousand Plateaus: Capitalisim andits history, the field has been shaped by futuristic visions of Schizophrenia, 1980,p.66)technological revolution, hyperbolic promises of scientific  Levy, P. (1997), Collective Intelligence. trans. Bononno, R.convergence at the molecular scale, and science fiction stories of Perseus Books, Cambridge,the world rebuilt atom by atom .” Today nanotechnology  McLuhan, Marshall (1994) Understanding Media.exists as a living sensation of the future of human existence: In Massachusetts. MIT PressMilburn’s words, “[A] bodily registration of potential for global  MANIFEST.AR@ICAHTTP://WWW.LAYAR.COM/LAYERS/MANIFESTARchange .” Brian Massumi states, “The body,sensor of change, ICAis a transducer of the virtual .” Through existing in these  Gabriella Giannachi, Virtual Theatres: An Introduction,virtual representations, that are directly linked to living bio- Routledge, London and New York, 2008.systems, we effectively sense, feel and think in a way that  A. Butz, T. H¨ollerer, S. Feiner, B. MacIntyre, and C.hybridizes the virtual with scientific inquiry, and therefore we Beshers. (1999). Enveloping Users and Computers in arequire a discourse that addresses whether this does in fact make Collaborative 3d Augmented Reality. IWAR’99:us post-biological. Proceedingsof the 2nd IEEE and ACM International Workshop on Augmented Reality,CONCLUSION page35,Washington,DC,USA,1999  Lang, T. MacIntyre, B. and Zugaza, I. (2008). Massively Art has, historically had a strong preoccupation with the Multiplayer Online Worlds as a Platform forbody and with consciousness. When dealing with such a massive Augmented Reality Experiences. Virtual Realityarea of inquiry there is a need to look to a wide variety of fields, Conference, 2008 (VR’08 IEEE), March 8-12 2008.As Ascott has stated, “[H]owever eccentric or esoteric, any  Kantonen T., Woodward C., Katz N., "Mixed reality inculture, immediate or distant in space or time, any technology, virtual world teleconferencing", Proc. IEEE Virtual Realityancient or modern, to find ideas and processes that allow for the 2010 Waltham, Massachusetts, USA, March 20 - 24, pp.navigation of mind and its open-ended exploration .” Roy 179-182Ascott proposes a syncretic approach to this issue: “Just as  (Barakonyi and Schmalstieg, 2008)cybernetics analogizes differences between systems, so  Boriana Koleva, Holger Schnädelbach, Stevesyncretism finds likeness between unlike things. Syncretic Benford, and Chris Greenhalgh. 2000. Traversable interfacesthinking breaches boundaries and subverts protocols. Thinking out between real and virtual worlds. In Proceedings of theof the box, testing the limits of language, behaviour and thought SIGCHI conference on Human factors in computing systemsputs the artist on the edge of social norms but at the centre of (CHI 00). ACM, New York, NY, USA, 233-240.human development .” Second-order cybernetics did very  Schnädelbach, H, Penn, A., Benford, S., Steadman, P.,well to explain our early relationships with machines in terms of Koleva, B. Moving Office: Inhabiting a Dynamic Building,interactivity and connectivity, however the incorporation of more CSCW 2006 conference, Banff, Canada, pp.313-322open networked systems of autonomous/anthropomorphic based  Raphael Grasset, Philip Lamb, and Mark Billinghurst. 2005.interactions have created a less physical and paradigmatic Evaluation of Mixed-Space Collaboration. In Proceedings ofsituation. Networked agency destabilises traditional orthodoxies
the 4th IEEE/ACM International Symposium on Mixed and Science and Culture. Published by Transdisciplinary Imaging Augmented Reality (ISMAR 05). IEEE Computer Society, Conference 2010 Sydney, ISBN: 978-0-9807186-6-9 Washington, DC, USA, 90-99.  Haraway, 1991,Simians Cyborgs and Women, The Raphael Grasset, Julian Looser, and Mark Billinghurst. 2006. reinvention of Nature, p.149-181 Routledge, New York. Transitional interface: concept, issues and framework. In  Hayles, N.K. (1990). Chaos Bound: Orderly Disorder in Proceedings of the 5th IEEE and ACM International Contemporary Literature and Science. Ithaca, New York. Symposium on Mixed and Augmented Reality (ISMAR 06). Cornell University Press IEEE Computer Society, Washington, DC, USA, 231-232.  Charles Ostman, 2003. Nanobiology? Where Sermon, P and Gould, C (2011). Liberate your Avatar: Nanotechnology and Biology Come Together. The Revolution Will Be Socially Networked , Creating www.biota.org/ostman/nanobio.htm (accessed June 3, 2004). Second Lives , pp. 15-31 Routledge, Taylor & Francis  Colin Milburn, “Atoms and Avatars: Virtual Worlds as Massively-Multiplayer Laboratories,” Spontaneous Group, New York, USA. Generations, University of Toronto, 2:1 2008. Roy Ascott, 2010. Syncretic Dialogues: Keynote address at  Brian Massumi, Parables for the Virtual: Movement, Affect, Proceedings of The first International Conference on Sensation, Duke University Press, Durham, NC, 2002. Transdisciplinary Imaging at the Intersections between Art,  ROY ASCOTT, 2008 “C, TECHNOETIC, SYNCRETIC THE PROSPECT FOR ART” LEONARDO VOL 41, #3,MIT PRESS P. 20