Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

East of the Rockies: Developing an AR Game


Published on

Presented at FITC Toronto 2019
More info at

Amelie Rosser

For the past two years Jam3 worked alongside Joy Kogawa and the NFB to create East of the Rockies, an augmented reality storytelling experience.

East of the Rockies is the first interactive AR game of its kind. The story takes users through a piece of Canadian history where Japanese Canadians were forced to leave their homes and live at internment camps during WWII.

This talk will cover the creation of the game: from concept and storyboarding, to the development process in Unity and various challenges and questions to consider from a creator’s perspective.

To let the audience in on the behind the scenes of developing an AR experience like East of the Rockies.

Target Audience
For those interested in Augmented Reality storytelling and game development.

Five Things Audience Members Will Learn
AR techniques using Unity
Storytelling in AR
Prototyping interactions in AR
Game state management using Unidux
Game optimization techniques in Unity

Published in: Technology
  • Be the first to comment

  • Be the first to like this

East of the Rockies: Developing an AR Game

  1. 1. FITCToronto2019 1 TORONTO / LOS ANGELES / MONTEVIDEO East of the Rockies: Developing an AR Game
  2. 2. FITCToronto2019 2 Intro - Historical context - Internment 1942 - Imperial Japanese Army bombed Pearl Harbour - The Canadian government declared Japanese Canadians as the enemy - In British Columbia they were forced to leave their homes and businesses - They were pushed into internment camps
  3. 3. FITCToronto2019 3 Joy Kogawa Author, poet, activist, order of Canada - Joy Kogawa was one of those interned Japanese Canadians. - She was able to turn these negative experiences into creative outputs - She wrote Obasan and was rewarded with Order of British Columbia and Japan's Order of the Rising Sun. FITCToronto2019
  4. 4. FITCToronto2019 4 Intro - Jason and Dirk’s initiative Jason Legge and Dirk van Ginkel wanted to do something for the 150th anniversary of Canada. - Wanted to highlight Canada's injustices as well as celebrate their achievements - Jason was always a fan of Joy - They met and played video games together - Joy was on board!
  5. 5. FITCToronto2019 6 East of the Rockies Partners
  6. 6. FITCToronto2019 7 Our goal was to shed light and acknowledge Canada’s past injustices We wanted to use immersive storytelling to educate and empower our audience.
  7. 7. FITCToronto2019 8 Our Audience Gen Z’ers are digital natives ready to embrace brands and services that understand their needs and interests. “When we match a digital relationship with the influence of storytelling, we have the ability to empower young people to make a difference.” – Huffington Post Gen Z’ers - 55% are willing to pay for products and services that push forward social impact issues –Neilson, 2015 - The most diverse and inclusive generation – Business Insider - Want to make a difference and change the world – Forbes - Crave innovative ways to consume content and embrace new forms of storytelling – Forbes
  8. 8. FITCToronto2019 9 Augmented Reality What I find amazing about this technology is the fact it’s integrated into your phone or tablet, so there’s no extra hardware barrier for people to start using the content. Augmented Reality - Augmented Reality has become a popular way for people to experience content - There’s no hardware barriers
  9. 9. FITCToronto2019 10 Intro - EOTR Script We started with a script. - Joy Kogawa teamed up with Walt Williams - Jason and Dirk adapted the script for gameplay
  10. 10. FITCToronto2019 11 - The narrative is told from the perspective of Yuki, a 17-year-old girl - The story follows Yuki as she and her family experience good times as well as the bad - Joy’s granddaughter Anne voices Yuki
  11. 11. FITCToronto2019 12 Our next step as a team was to breakdown the script into a storyboard. Storyboarding
  12. 12. FITCToronto2019 13 Storyboarding - Legend - We created a legend to breakdown the narrative and gameplay - A pattern started to develop for the main gameplay objectives and interactions
  13. 13. FITCToronto2019 14
  14. 14. FITCToronto2019 15 Storyboarding - Act 1, Scene 2
  15. 15. FITCToronto2019 16 UX - With the initial storyboarding in place we would then begin to map out a clear user journey path - UX then starts to define the gameplay journey - We’d define key interactions - Optional interactions to enrich the storytelling UX MappingFITCToronto2019
  16. 16. FITCToronto2019 17 Gameplay structure We started with another legend, this time breaking down the gameplay - Critical path for the users - Side and optional interactions
  17. 17. FITCToronto2019 18 Gameplay structure
  18. 18. FITCToronto2019 19
  19. 19. FITCToronto2019 20 Gameplay breakdown by location - Gameplay was then broken down by location - This helped us understand the order interactions would go in
  20. 20. FITCToronto2019 21 Draw Skippy With Tomio 🎯 Align device UI on the dots to connect them and reveal the drawing (x5 dots) 5 5 Open Luggages 👆 Tap on locks to open the luggages (x2 locks) 1 4 Light On Stove 👆 Tap on stove to light it on 4 1 Open Curtain ⚡ Swipe on curtain to open & close it 2 2 3 3 Chop Logs ⚡ Swipe on logs to chop them (x5 logs) Act 1 - Scene 2 - Yuki’s House
  21. 21. FITCToronto2019 22 6 1 2 5 3 4 Interact With Mrs Tucker 👆 Tap on Mrs Tucker’s to trigger news & radio 6 Raise Union Jack Flag 🎯 Align device UI on the dots to raise the flag (x3 dots) 1 Swipe Steam ⚡ Swipe to remove steam from the screen 5 Close Bag With Japanese Doll ⚡ Swipe to the bottom to close the bag 2 3 Look Through Walls 🔍 Get closer to look through the walls of the Bathhouse & the Odd Fellows Hall. 4 Interact With NPC Characters 👆 Tap on NPC characters to trigger their actions (x3 characters) Act 1 - Scene 5 - Camp Scene
  22. 22. FITCToronto2019
  23. 23. FITCToronto2019 24 FITCToronto2019 - Harshness of winter - Only sources of heat was the stove
  24. 24. FITCToronto2019
  25. 25. FITCToronto2019
  26. 26. FITCToronto2019 27 UI Discovery - Initial exploration into how we would overlay graphical elements within the game
  27. 27. FITCToronto2019 28
  28. 28. FITCToronto2019 29
  29. 29. FITCToronto2019 30
  30. 30. FITCToronto2019 31 Technical Discovery When we reached technical kickoff we had a lot of areas to explore. Our key areas to focus on were: - Game engines - Augmented Reality - Interactions - Lighting and rendering - Visual fx Our team already had knowledge of these areas but not within Unity. We needed to figure these things out to give us insight on the right way forward
  31. 31. FITCToronto2019 32 Development - Technical Discovery - Deciding on a Game Engine - Our team had the most experience with it - Team workflow was important - two options Unity Collaborate and Git - Big active and open community Choosing Unity
  32. 32. FITCToronto2019 33 We used Vuforia and it was hell. Development - Technical Discovery - Vuforia Pros Cons Cross Platform In Editor Simulator Updating Unity would often break Vuforia API and examples were not straightforward Performance was bad at times, drifting occured In-Editor simulation eventually broke from an update
  33. 33. FITCToronto2019 34 - We prototyped with it to test the api and performance - We noticed less drifting and a performance increase - The API is completely open source - Unity are also working on a remote AR tool preview.573229/ - When removing the Vuforia Framework from our project, XCode failed to build . . . Development - Technical Discovery - AR Foundation Meanwhile, Unity released AR Foundation
  34. 34. FITCToronto2019 35 We needed a way to focus the players attention within the world Development - Technical Discovery - Content scaling - We couldn’t scale the models as it would break the lighting - However, we found an article by Unity that documented two approaches we could use
  35. 35. FITCToronto2019 36 Development - Technical Discovery - Content scaling
  36. 36. FITCToronto2019 37 Interactions needed a manager - We used a web inspired approach to handle touches / mouse interactions - Attach a script to a gameobject to make it interactable - Raycasting is expensive Development - Technical Discovery - Interactions
  37. 37. FITCToronto2019 38 Inspect interaction - Object would animate towards the camera - User could turn the item around and find details - Seperate camera attached to the main camera
  38. 38. FITCToronto2019 39 Development - Technical Discovery - Lighting explorations Unity Concept art
  39. 39. FITCToronto2019 40 Rig lighting explorations continued
  40. 40. FITCToronto2019 41 World Masking - Mask to focus the region of the world - We called it the “circle of occlusion” - Mask happened in the fragment shader - Radius of the mask needed to scale with the camera zoom - Unity Post processing caused us problems
  41. 41. FITCToronto2019 42 Assets - We worked with two Technical Artists for the project that created the 3D game assets - Models were created in Meters to align with Unity’s measurement space - Avoid textures where possible for performance - Houdini, Zbrush and Maya Game Assets
  42. 42. FITCToronto2019 43 Development - Assets - Import settings - Import assets with the right settings - Track memory usage with the profiler - Use streaming sounds to reduce memory overload - Set max texture size - Use the memory profiler & look for unusual memory sizes Unity Asset Import Tips
  43. 43. FITCToronto2019 44 Colour Palette Creating a cohesive colour tone - We created a reusable colour palette of multiple materials to use throughout the game
  44. 44. FITCToronto2019 45 Assets - Characters High poly sculpt of Yuki (that turned out we couldn’t use)
  45. 45. FITCToronto2019 46 Assets - Characters The Tanaka Family
  46. 46. FITCToronto2019 47 Development - Assets - Character animations - A decision early on in the project was to use motion capture suits for the character animation - In theory it sounded like a great idea since we required a lot of animations for the game
  47. 47. FITCToronto2019 48 Development - Assets - Character animations - We later discovered though a lot of the mocapped animations would require cleaning up. - We came to the conclusion we should have opted for an animator instead Oh dear.
  48. 48. FITCToronto2019 49 Assets - Buildings - Models separated into prefabs - Combed prefabs together to build the world - Static objects together, dynamic game objects into gameplay prefab
  49. 49. FITCToronto2019 50 Assets - Buildings - Gameplay prefab - Objects require realtime lights - Interactive objects would appear here
  50. 50. FITCToronto2019 51 Scene Structure This is the typical structure for a scene in the game - Gameplay prefab progresses user through the story - Scene prefab attached to the AR anchor point
  51. 51. FITCToronto2019 52 Assets - Photographs
  52. 52. FITCToronto2019 53 Assets - Photographs
  53. 53. FITCToronto2019 54 We then needed a solution for the team to collaborate on the project - We had up to 5 developers on the project - It needed to be based on Git Collaboration Strategy
  54. 54. FITCToronto2019 55 Development - Workflow - Unity Collaborate vs Git Unity Collaborate - Team licensing - Basic interface - Limited branch features (no git flow) Github - No limit on contributors - Use the tools we know and love - Git flow workflow
  55. 55. FITCToronto2019 56 Development - Setting up Git - Github for Unity was still in it’s early stages - Use .gitattributes and .gitignore - Used Gitflow Branch Structure: - Master - Develop - Feature - [Jira-ticket-id]-feature-name Setting up Git = lots of trial and error!
  56. 56. FITCToronto2019 57 Development - Game Development Every Game Developers nightmare. - We wanted our game architecture to be clean!
  57. 57. FITCToronto2019 58 - Our team has a strong web development background - We needed a robust approach to managing the core logic of our game - We came across Unidux, a Redux inspired application architecture - Features: - Includes a useful UI inspector for your store’s state. - State is serialised to json - We built on top of it to add in the missing features we need Development - Game Development - Unidux
  58. 58. FITCToronto2019 59 Development - Game Architecture - UniduxUnidux Inspector Unidux Inspector - Redux dev tools equivalent
  59. 59. FITCToronto2019 60 Development - Game Architecture - Unidux Game reducers used
  60. 60. FITCToronto2019 61 Action Creators Action Creators for a gameplay scene
  61. 61. FITCToronto2019 62 Development - Game Architecture - Unidux Reducer for a gameplay scene
  62. 62. FITCToronto2019 63 Subscribe - Subscribing to the store was simple - Unity OnDestroy would automatically unsubscribe
  63. 63. FITCToronto2019 64 Handle Props Changes - This was the manual part - Handle prop changes ourselves to see what changes
  64. 64. FITCToronto2019 65 Props Changed If props have changed, do something.
  65. 65. FITCToronto2019 66 The Game Prefab is the brain - Contains all core logic - Every scene in the game needs it - A singleton and never gets destroyed Development - Game Development - Game Prefab
  66. 66. FITCToronto2019 67 Development - Closed Captions The game required both English and French - We used as our CMS - Exported JSON files to use in-game
  67. 67. FITCToronto2019 68 Development - Localised Text Script We have a localised text script which fetches the text and renders it
  68. 68. FITCToronto2019 69 Development - Voice Over Voice over required a different approach - Youtube for captions - Node scripts to fetch VTT and export to json
  69. 69. FITCToronto2019 70 We stored game assets in Asset Bundles - Included in the app and loaded from Streaming Assets - Unity depreciated their ABM so we created our own to suit our needs Development - Game Development - Asset Bundles
  70. 70. FITCToronto2019 71 Scene Flow Structure - Each scene would start with a voice over and a scripted timeline - Once the voice over ends interactions would activate - We created our own solution to subscribe to timeline events - Sometimes things would go wrong... Development - Scene Structure
  71. 71. FITCToronto2019 72 Development - Game Development - Building for testing Building for devices would require a few steps: - If content changed in the asset bundles, asset bundles would need to be re-generated which would take ~4 mins to build - If the changes we’re only scripts we could build the Xcode project straight away - Then building the Game’s Xcode project would take 1 minute - It could take up to 5 mins to test changes on a device
  72. 72. FITCToronto2019 73 Unity & UI - NO HTML CSS :( - Unity’s UI system takes time to learn - How we approached it Unity’s UI
  73. 73. FITCToronto2019 74 Our main game UI lived in our Game Prefab - We used Canvas scale with resolution setting - We should of made the UI more component based Development - UI - Canvas
  74. 74. FITCToronto2019 75 Key Learnings & Insights - Start with a solid game architecture - Define a good naming convention for Assets - Always keep an eye on memory - Make sure the Unity project is kept clean, avoid messy folders - Ensure code is reviewed by a lead developer - Keep the workflow optimised and fast for device testing
  75. 75. FITCToronto2019 76 IDFA - We showed the game off at IDFA - International Film and Documentary festival in Amsterdam - It gave us a good insight to how people played the game - It allowed us time for final UX amends for release
  76. 76. FITCToronto2019 77
  77. 77. FITCToronto2019 78 We reached game release! 2+ Years in development 2000+ Hours worked on the project ~87 Pizzas eaten (we can’t be too sure)
  78. 78. FITCToronto2019 79 Results and Reception Game Launch - We finally launched the game - Apple promoted us for App of the day - Maintaining a rating of 4.5 or higher 100,000+ Downloads in the first two weeks Rank in Education category #3
  79. 79. FITCToronto2019 80 FITCToronto2019
  80. 80. FITCToronto2019 81 FITCToronto2019
  81. 81. FITCToronto2019
  82. 82. FITCToronto2019 The Bathhouse
  83. 83. FITCToronto2019 84 Launch - JCC We showed the app of at the Japanese Canadian Centre - It was the first time showcasing the full experience to members of the Japanese Canadian community - Some had been in internment camps themselves as young children