ISMAR 2014 AR Workshop: Designing Location Based Experiences

1. Location Based Experience Design Workshop Mark Melnykowycz ISMAR 2014 Sept. 9th 2014 1

2. Story Structure Prototyping User Experience Story Location A/MR Technology Mark Melnykowycz mark@idezo.ch Location Hero’s Journey Changing structures Books, movies, ebooks, games IxD / UX Fundamentals of IxD Consumption Patterns User Journey Game Mechanics Metaio SDK Tracking / Positioning AR / MR Teaching Users How to Play Fundamental differences Technology World Building Game Space Rules 2014 ISMAR Workshop: Location based experience design Ghosts of Venice Mixed-reality film Lost In Reality Mobile storytelling 2

3. Location Mark Melnykowycz mark@idezo.ch 3

4. Generic Location App Use Case User goes to a location Content or interaction is triggered Mark Melnykowycz mark@idezo.ch 4

5. Designing Location Experiences Why is the person in that location? How are they interacting with the location? Why are they motivated to engage in this experience? How are experiences planned? What technologies are available? Mark Melnykowycz mark@idezo.ch 5

6. Key Physical Elements Location Position / Orientation Interaction Content Mark Melnykowycz mark@idezo.ch 6

7. Sketching out user flows / prototyping / product definition Technology trends User motivations / immersion Communication patterns Building interaction / defining app world Mark Melnykowycz mark@idezo.ch 7

8. Tech and Society Trends IoT/M2M Devices Device Fragmentation Evolving Communication Mark Melnykowycz mark@idezo.ch 8

9. Technology and Device Trends Mark Melnykowycz mark@idezo.ch 9

10. Connected Device Evolution 50 Billion! By 2020*! Location Interaction Connected City Quantified User! Quantified Self! IoT! Large connected ecosystem http://share.cisco.com/internet-of-things.html Mark Melnykowycz mark@idezo.ch 10

11. Location Time Heart Rate Quantified User Sensor Fusion User State Mark Melnykowycz mark@idezo.ch 11

12. Defined UX in Closed Systems Interaction Device Interaction and consumption patterns are learned making the transition to new devices easy Mark Melnykowycz mark@idezo.ch 12

13. Fragmented UX Patterns Distributed UX Ubiquitous Computing Defined UX Space Mark Melnykowycz mark@idezo.ch 13

14. Location as a Device Experience How are users trained to interact with locations? Maps Mark Melnykowycz mark@idezo.ch 14

15. Determine orientation in location using navigation aids Maps Identify physical anchor points to define position Use technology to identify information around you Future actions based on understanding where you are and what you can explore Mark Melnykowycz mark@idezo.ch 15

16. Communication Trends Mark Melnykowycz mark@idezo.ch 16

17. Short Evolution of Communication Face to Face Copier Printing Writing Web Location Services Email Static HTML Blogging Instant Messaging Fragmented Lines! Push/Pull! Transmedia Social Networks Telephone Mobile SMS MMS Mark Melnykowycz mark@idezo.ch 17

18. Content Container Consumption Velocity Mark Melnykowycz mark@idezo.ch User Experience Design The User 18 Consumption Concepts

19. Consuming Media Content The User Information transferred between containers and consumed by people Mark Melnykowycz mark@idezo.ch 19

20. Physical Container The User A real, tangible thing used to hold, and through which content can be consumed Has been designed, has structure and fills up physical space Mark Melnykowycz mark@idezo.ch 20

21. The User Virtual Container A virtual thing used to hold, and through which content can be consumed Has structure, fills up space and can scale Defined through graphic design and UI engineering Mark Melnykowycz mark@idezo.ch 21

22. Consumption The User Vocabulary Content Understand/Retain information Make it part of your knowledge base How fast is this? Velocity Mark Melnykowycz mark@idezo.ch 22

23. Consumption Patterns Print Web Mobile Landing Page Grid Content Home Linear progression Pull out bits of information Scan - look for anchors Save an idea and move on Mark Melnykowycz mark@idezo.ch 23

24. Push/Pull Consumption Mark Melnykowycz mark@idezo.ch Patterns 24

25. Digital Consumption App/Webpage Social Media Video Augmented Reality Push/Pull Curator Audience Mark Melnykowycz mark@idezo.ch 25

26. SnowFall New York Times / http://www.nytimes.com/projects/2012/snow-fall/ Scroll page, easy to pull out bits of the story and be immersed in graphics, story, characters. Mark Melnykowycz mark@idezo.ch 26

27. Text, animations, videos, different themes, consistent narrative Pull-based consumption and reading of content Mark Melnykowycz mark@idezo.ch 27

28. Parallax, easy to explore Pull consumption Easy to design/prototype https://www.scrollkit.com Mark Melnykowycz mark@idezo.ch 28

29. Easy to integrate, drag/drop images, embedded media (video/ audio/iframes) Custom URL or publish on scrollkit.com Mark Melnykowycz mark@idezo.ch 29

30. Social Media Curation Flow/steam Slideshow More powerful than Tumblr, connect to and search social media channels, present content in sequence Mark Melnykowycz mark@idezo.ch 30

31. Connect to social media channels Arrange elements to group together Add text and links Mark Melnykowycz mark@idezo.ch 31

32. Publish as landing page on BrickFlow, share link to social media channels, remix Embed ex ter nal l y as iframe element, view as slideshow experience Mark Melnykowycz mark@idezo.ch 32

33. Connect to and search social Add content, text, etc. media channels Mark Melnykowycz mark@idezo.ch 33

34. Mixed Video/Interaction Non-linear video navigation Clickable elements Easy assembly Combine multi-media elements View in video/slide format Videos, audio, animations, text, etc. Mark Melnykowycz mark@idezo.ch 34

35. http://www.raptmedia.com/product Add clickable elements to videos, seamlessly jump from one video to another. Allow choices for jumping between storylines/ elements Mark Melnykowycz mark@idezo.ch 35

36. Interactive layer allows clicking/hyperlinks Different video clips can be linked together User can make decisions at plot points Easy to create alternative story lines or allow exploration Mark Melnykowycz mark@idezo.ch 36

37. Connect to channels, ability to upload Arrange elements Share www.zeega.com Mark Melnykowycz mark@idezo.ch 37

38. Designing Experiences Mark Melnykowycz mark@idezo.ch 38

39. Discovery High-Level Solution Product Solution Empathy Define Ideation Prototype Test Mark Melnykowycz mark@idezo.ch Product/Experience Design Phases User Needs Focus / Flare App Prototype 39 Experience Planning Interactivity Prototyping Story/Location Technology

40. Story Consumption Container What is the story, the world it takes place in, the character, etc. How will this story world be consumed by the audience or users? How will the consumption be realized, physically/ virtually designed, etc? User Experience/Interaction Know your User! Wireframe/prototyping Webpage/App Video/ebook Know your Audience! Story boarding Pre-Visualization Mark Melnykowycz mark@idezo.ch 40

41. Creative Product Creates stories, art, movies, etc. Builds physical/ virtual products User Experience/ Interaction Find your audience The Audience The Users Prototype/Test/Explore Mark Melnykowycz mark@idezo.ch 41

42. How will the consumption be realized, physically/ virtually designed, etc? Wireframe/prototyping Webpage Video ebook Consumption Container User Experience/Interaction User Journey Paper/Digital Mockup Interactive Prototype Final Experience Mark Melnykowycz mark@idezo.ch 42

43. How does the user get into your story and engage in your experience? User Journey User sees meme via Facebook/Tumblr Searches for Zurich Dada Game on Google Starts reading/ learning about the Dada ghost in Zurich Need to have good SEO base on landing page Need fast consumption on desktop/mobile Parallax page design User is push/pull fragmented reader Mark Melnykowycz mark@idezo.ch 43

44. Design representation of your app/container to communicate and test the user experience Paper/Digital Mockup/Prototype Communicate with potential partners / investors / users Testing tool for fast user feedback before development Communicate between design / development Demo App Mockup Paper Mark Melnykowycz mark@idezo.ch 44

45. Web/App Prototyping Keynote Powerpoint OmniGraffle GUIToolkits AppCooker BluePrint iMockups HotGloo JetStrap Stencils Visio Balsamiq Axure iRise MX Photoshop Illustrator iPad Desktop Online Mark Melnykowycz mark@idezo.ch 45

46. Mark Melnykowycz mark@idezo.ch Storytelling 46

47. Story •Story, a recounting of a sequence of events* •An account of imaginary or real people and events told for entertainment: "an adventure story"** Narrative •A narrative (or story) is any account that presents connected events*** •A spoken or written account of connected events; a story: "a bare narrative of the details"** *Wikipedia **Google dictionary ***Oxford English Dictionary (online) Mark Melnykowycz mark@idezo.ch 47

48. Emotions Knowledge Society Story Mark Melnykowycz mark@idezo.ch 48

49. Shared emotions help form the fabric of society ...storytelling is a vehicle used to combine emotions and information ...and communicate those feelings and information in a structure which gives us comfort or instills feelings of fear, optimism, etc. Mark Melnykowycz mark@idezo.ch 49

50. Emotions Memories Connections Structure Mark Melnykowycz mark@idezo.ch 50

51. Storytelling gives emotional context to our linear lives Mark Melnykowycz mark@idezo.ch 51

52. Physical time is linear Emotional memory is non-linear Birth Death Mark Melnykowycz mark@idezo.ch 52

53. Story Connect current events to good feelings from the past Hope in the future and our confidence to meet challenges Play out emotions we can’t explore in reality Mark Melnykowycz mark@idezo.ch 53

54. Die Hard ...makes me feel good about my chances of surviving a dramatic terrorist attack Die Hard, 20th Century Fox ...statistically it will probably never happen, but I’m sort of mentally prepared if it does Mark Melnykowycz mark@idezo.ch 54

55. Mark Melnykowycz mark@idezo.ch What is traditional story structure? Does it conflict with our fragmented communication patterns? 55 Story Structure

56. Many different models of story structure for different media 56

57. Linear Story Basics Mark Melnykowycz mark@idezo.ch 57

58. Generic Story Ingredients Villain Conflict Mark Melnykowycz mark@idezo.ch 58

59. Challenge Transformation Over-coming extreme odds...to succeed in the end… Mark Melnykowycz mark@idezo.ch 59

60. Linear Story Structure http://www.musik-therapie.at/PederHill/Structure&Plot.htm Mark Melnykowycz mark@idezo.ch http://wandertones.deviantart.com/art/Standard-Story-Structure-178276496 60 Rising drama leading to an apex with essential plot points and local challenges/resolutions

61. Joseph Campbell “Campbell explores the theory that important myths from around the world which have survived for thousands of years all share a fundamental structure, which Campbell called the monomyth.” https://en.wikipedia.org/wiki/The_Hero_with_a_Thousand_Faces Mark Melnykowycz mark@idezo.ch 61

62. Joseph Campbell THE HERO IS INTRODUCED IN HIS ORDINARY WORLD THE CALL TO ADVENTURE THE HERO IS RELUCTANT AT FIRST THE HERO IS ENCOURAGED BY THE WISE OLD MAN OR WOMAN THE HERO PASSES THE FIRST THRESHOLD THE HERO ENCOUNTERS TESTS AND HELPERS THE HERO REACHES THE INNERMOST CAVE THE HERO ENDURES THE SUPREME ORDEAL THE HERO SIEZES THE SWORD THE ROAD BACK RESURRECTION RETURN WITH THE ELIXIR http://www.cs.uu.nl/docs/vakken/b2go/docs/hero_journey.pdf Mark Melnykowycz mark@idezo.ch 62

63. The Hero’s Journey repeated in countless media… Mark Melnykowycz mark@idezo.ch 63

64. Suspension of Disbelief Suspension of disbelief or willing suspension of disbelief is a term coined in 1817 by the poet and aesthetic philosopher Samuel Taylor Coleridge, who suggested that if a writer could infuse a "human interest and a semblance of truth" into a fantastic tale, the reader would suspend judgement concerning the implausibility of the narrative. http://en.wikipedia.org/wiki/Suspension_of_disbelief Mark Melnykowycz mark@idezo.ch 64

65. Why is the Hero’s Journey structure so popular in movies and media? Fairy Tales, myths, Star Wars, The Matrix, etc. Do we like it because we expect it and are conditioned for it? Is it inherent in our minds and DNA? Does it hold us back in life? Must we be chosen to do great things by someone else? Mark Melnykowycz mark@idezo.ch 65

66. Games and Story Mark Melnykowycz mark@idezo.ch 66

67. Play Confront a challenge Interact with other Understand your progress players Games Interactive with the world Actions have consequences Conquer small quests Mark Melnykowycz mark@idezo.ch 67

68. Players Story Game Elements Conflict / Mechanics Challenge Resources Game world / field of play Mark Melnykowycz mark@idezo.ch 68

69. Game Mechanics Players overcome challenges Players understand the rules Players understand their place in the context of the game Reason for the experience to exist Know how to interact with the experience Points System Levels Time Mark Melnykowycz mark@idezo.ch 69

70. Game Enjoyment 70 The Challenge The Story The Game Play http://en.wikipedia.org/wiki/Doom_(1993_video_game) http://z2.com/game/metalstorm-online/ http://en.wikipedia.org/wiki/Puzzle_Bobble http://en.wikipedia.org/wiki/World_of_Warcraft

71. World Building Mark Melnykowycz mark@idezo.ch 71

72. A Generic Story Character presented with challenge Generic Threat Finds courage... Triumphs! Character evolves Generic Hero Character overcomes challenges Mark Melnykowycz mark@idezo.ch 72

73. Character Change Through Conflict Character evolves A character evolves/ changes due to conflict inherent in the story Relationships Environment Mark Melnykowycz mark@idezo.ch 73

74. Who is the Main Character? Is the user a participant or an outside viewer in a location game or experience? Mark Melnykowycz mark@idezo.ch 74

75. Story Development Character Driven World Driven Generic environment, used as a setting for character interaction to take place. The story of a character in a given environment, which forces action / conflict ...become a different person How will the world affect character development? ...character traits stay the same Mark Melnykowycz mark@idezo.ch 75

76. Contemporary super hero Traditional action hero Die Hard Arnold Schwarzenegger Ironman, Thor, Spiderman, etc. Watchmen Dark Night Man of Steel X-men Simple dialogue, hero never dies, always triumphs Question their actions and place in the story, supporting characters die, like real life. Good vs. Evil Good? vs. Grey Zone Mark Melnykowycz mark@idezo.ch 76

77. World Location Time Physics Where is the world? Is it a city, a new planet, a coffee house? Take a character from another time or use a world from a different time? If it’s a new world, does it respect the physics of Earth as we understand them? Mark Melnykowycz mark@idezo.ch 77

78. 78 X | Media | Lab Switzerland 2013: Transmedia Alex McDowell – Transmedia’s Renaissance Man: Narrative Designer for Minority Report; Fight Club; Man of Steel; Associate Professor, USC – School of Cinematic Arts; Visiting Artist, MIT Media Lab (Hollywood)

81. How does a character move around the world? Character Journey Character becomes sick Must go to hospital How would she die? Transportation Never invented gasoline, all hydrogen logistics Mark Melnykowycz mark@idezo.ch 81

82. Physical Virtual World Building / Prototyping Unity 3D! • Multi-platform Game development • Terrain building • Visualization • Integration of buildings, people, etc. Mark Melnykowycz mark@idezo.ch 82

83. Sketch out worlds Physical story worlds define logic of story progression The Hobbit Map http://tolkiengateway.net/wiki/File:Christopher_Tolkien_-_Map_of_Wilderland.jpg Mark Melnykowycz mark@idezo.ch 83

84. Paint terrain and topography features ! Add textures for grass, rock, water, etc. ! Add buildings, people, tanks, etc. Mark Melnykowycz mark@idezo.ch 84

85. Unity User Interface Paint on topography Load new models Use different brushes Load textures Mark Melnykowycz mark@idezo.ch 85

86. Augmented and Mixed Reality Technologies Mark Melnykowycz mark@idezo.ch 86

87. Generic AR Experience Mark Melnykowycz mark@idezo.ch App tracks feature in reality Marker Marker-less Point-cloud Edge-based Content positioning User engaged with app Virtual content 87

88. Mark Melnykowycz mark@idezo.ch Creator / Studio SDKs AR Software AR Browsers 88

89. Depth Cameras Desktop Laptop Mobile Structure.io Project Tango Mark Melnykowycz mark@idezo.ch 89

90. 90 Ghosts of Venice ghostsofvenice.it

91. Augmented Reality Ghost Stories of Venice ghostsofvenice.it Mark Melnykowycz mark@idezo.ch

92. 92 Film Maker / Director Marco Gatti AR Glass Expert Story Tech Sara Mautino UX / AR coding Mark Melnykowycz Visual Effects Production ghostsofvenice.it

93. Take users on a ghost story through Venice, mixing story elements and 93 Place Ghosts in the Environment ghostsofvenice.it 3D animations into the local environment

94. Film vs. App 94 C o n s u m e d b y watching/listening while stationary L o c a t i o n a n d content can be mixed to provide more immersive experience ghostsofvenice.it

95. Augmented Reality 95 Technology used to slightly modify the perception of the environment of the user GPS location Media overlay Envi ronment or user triggers new actions in the software Image recognition Feature tracking 4 Image 3 Video 2 Audio Text 1 ghostsofvenice.it

96. Story - Technology Development 96

97. Transmedia Narrative Mark Melnykowycz mark@idezo.ch Main point of contact between user, story, and location Online Story Hard Copy Mobile App Educate users of the background stor y before starting Give users something tangible, objects from t h e gh o s t s , t h e i r letters, etc. 97 ghostsofvenice.it

98. Story Development Mark Melnykowycz mark@idezo.ch App Design App Development Media Production How do all of these key elements come together in the transmedia development process? 98 ghostsofvenice.it

99. Story/App Workflow Mark Melnykowycz mark@idezo.ch Test/Iterate Story App Design Prototype App Development User Experience User Interaction Background stories of ghosts How stories are delivered Implement UX/UI in App Development Production App 99 ghostsofvenice.it

100. Story/Media Creation Story Locations Ghosts Mark Melnykowycz mark@idezo.ch Choose locations Choose trigger elements Background story for each ghost Visual Design Casting Filming Post-production Mobile Media Ready to distribute on mobile device Test App Code into data model for viewing/testing Production App 100 ghostsofvenice.it

101. Transmedia Landscape A transmedia project integrates a larger narrative across different distribution channels, we need to develop a plan for how this would be realized Sets the stage, gives tangibility to the story experience, being in the same physical place as the ghosts App is used to view the ghosts, to connect with the “being” the user has learned about Mark Melnykowycz mark@idezo.ch Interaction How can the story and location be interactive with the participants? This increases the emotional connection to the experience The impact of seeing the ghosts comes from the emotional connection to the story behind them Location Mobile App Story Channels Real book/diary of ghosts Wikipedia page each individual Artifacts of ghosts in the city/museums...what else? Establish emotional connection and empathy to ghost 101 characters ghostsofvenice.it

102. Augmented Reality Technology Needs 102

103. User Walks Through Area User Flow Location Aware Image recognition Event Playing User sees they are in the area where a trigger is User searches for physical trigger Location triggers haptic/vibration on device Content plays through device ghostsofvenice.it Mark Melnykowycz mark@idezo.ch 103

104. Alpha Mask Overlay in video feed Transparent background ghostsofvenice.it 104

105. Image Recognition for Location? Software recognizes specific features of the environment Image pre-programmed int o the device or queried from a server Recognition may not be automatic, depends on hardware speed ghostsofvenice.it 105

106. Install Junaio Access GOV channel View content on image trigger 106 ghostsofvenice.it

107. Changing light conditions will impact tracking ability Image target not reliable enough for spatial location use 107 ghostsofvenice.it

108. Feature Recognition Software recognizes specific features of the environment P o i n t - c l o u d m a p created to aid in feature recognition CAD or edge-based tracking can be tested as well ghostsofvenice.it 108

109. Metaio Edge-based Tracking https://dev.metaio.com/sdk/documentation/tracking-config/optical-tracking/cad-model/ CAD model overlay on environment Better in low-light tracking scenarios Where do I get building CAD? www.metaio.com 109

110. App Development Flow 110 Initial Test Functional Prototype Initial Release Layar/Junaio Metaio SDK Unity Beta App Test the concept of playing audio o r v i d e o fi l e s based on GPS location with AR Browser I m p l e m e n t v i d e o overlay concept in a nat i v e iOS/Androi d application, testing with users to understand the UX better With a developed s t o r y l i n e / p l o t , producing the full m i x e d r e a l i t y exper ience and broad testing with users ghostsofvenice.it

111. Mobile 3D Scanning Use 3D scan of objects to create CAD for model positioning Experiment with 3D mapping for model positioning Mark Melnykowycz mark@idezo.ch 111

112. Combine Person and Environment Models Ghosts as model scans Match to environmental features Mark Melnykowycz mark@idezo.ch 112

113. Film Maker / Director Marco Gatti AR Glass Expert Sara Mautino UX / AR Development Mark Melnykowycz Visual Effects Team www.ghostsofvenice.it 113

114. Lost In Reality 114

115. Lost In Reality Mobile App Web App Search Consume Create Share Create Edit Storytelling Social Network 115 lostinreality.net mark@lostinreality.net

116. Francisco Campelo José Rodrigues Mark Melnykowycz Ismail Cimen App Developer Partner and Story Development Project Lead UX/UI Design UX/UI Team Lisbon, Portugal Zurich, Switzerland 116 lostinreality.net mark@lostinreality.net

117. People Loca8ons Emo8ons How can we use story to add to the information context of locations? 117 lostinreality.net mark@lostinreality.net

118. For Storytellers Stories saved Authors can write loca8on stories on the web app Users find stories based on loca8on and tags Users can write stories on the mobile app Story server 118 lostinreality.net mark@lostinreality.net

119. Loca8on used as integral element of Tourists have more unique experiences in new ci8es For Authors Authors can write loca8on stories on the web app Users can download stories Locals have new perspec8ves on their city stories 119 lostinreality.net mark@lostinreality.net

120. For Readers Users search and access loca8on-‐stories on demand with mobile app Story elements triggered based on loca8on in ci8es Context of stories gives more value to loca8ons 120 lostinreality.net mark@lostinreality.net

121. Location Stories vs. Social Media Social Networks Context? Location Services Social media evolved from blogging as faster way to publish ideas Social media updates are only useful for a short time frame Location stories give structure and context to information and combats the social media noise “Facebook/Twitter shows people by who they are, what they have done and said” Location services give an idea of what a person can do in a place (irregardless of when) “Locat ion s tor ies show people the experience they’ve had” “Location stories let people follow experiences and have their own” lostinreality.net mark@lostinreality.net 121

122. Mobile App Flow Focus on location-based storytelling Users can create, search, and follow location stories Emotional engagement to location Primary Goals of LIR Smart phone application Dashboard Story Library Local Story Search Story Element Story elements delivered to user at GPS location (triggered) lostinreality.net mark@lostinreality.net 122

123. Maker Web App lostinreality.net mark@lostinreality.net 123

124. Maker Web App lostinreality.net mark@lostinreality.net 124

125. 125 User can follow different stories in a city as they walk around. lostinreality.net mark@lostinreality.net

126. Sketching out user flows / prototyping / product definition User motivations / immersion Mark Melnykowycz mark@idezo.ch Building interaction / defining app world Technology trends Communication patterns 126

ISMAR 2014 AR Workshop: Designing Location Based Experiences

You are reading a preview.

Check these out next

ISMAR 2014 AR Workshop: Designing Location Based Experiences

More Related Content

Slideshows for you (20)

Viewers also liked (20)

Similar to ISMAR 2014 AR Workshop: Designing Location Based Experiences (20)

More from Mark Melnykowycz (10)

Recently uploaded (20)