This grant proposal requests funding for a two-semester studio at NC State combining design, development, and storytelling using mixed reality technologies and physical crafting. Students will develop novel game concepts in teams during the first semester and complete 2-3 games in the second semester for public release. The games will be exhibited at NC State and Epic Games. The grant would support research assistants, materials, and internships to advance research at the intersection of physical and digital crafting through interactive storytelling.

1. Todd Berreth (PI)
Assistant Professor of Media Arts, Design and Technology
Chancellor’s Faculty Excellence Program for Visual Narrative
NC State College of Design, Media Arts, Design and Technology Department
Computer Science Games and Visual Narrative Group
Raleigh, NC, USA
Hand-crafting and Storytelling in
Mixed Reality Games
Epic Games design research studio, 2024-2025
MegaGrant Proposal 2023 - Education

2. Narrative

3. This proposal is requesting grant funding to support an
innovative two-semester multi-disciplinary design/development
research studio at NC State, a collaborative effort between
faculty in the department of Media Arts, Design and Technology
and Computer Science, and Sound Design faculty at UNC
School of the Arts.
Student designers, artists and engineers will work in teams to
develop novel and engaging concepts and initial prototypes
involving the use of mixed reality technologies combined with
physical hand-craft, digital gameplay and storytelling.
model building,
miniatures &
dioramas
wood
working
fibers / soft
construction
papercraft /
origami
some examples of handcrafts

4. Story-Go-Round
Berreth et al.

5. The most successful concepts and prototypes will advance to
a production phase in the second semester, which will
culminate with a public release of the games.
There will be a public exhibition of conceptual development
and art materials, physical artifacts and environments, and
working prototypes, hosted at Epic Games and NCSU at the
completion of the effort.

6. The game development will be Unreal-based, combined with
a range of mixed-reality hardware platforms and software
libraries.
I would like to involve Epic Games staff in occasional design
critiques, depending on interest by Epic, as part of the
educational and project-development process.

7. Some of the products of the studio, and findings, will also be
publicized and discussed in prominent academic research
venues such as ACM SIGGRAPH, UIST and IEEE
Transactions on Games.
The grant will support one graduate research assistantship,
and a number of student teaching assistantships/ internships
to support the work and output of the studio.

8. Background

9. Many people love hand-crafting, the artistry of analog
physical making and building, and the “retro” aesthetic
of stop-motion animation storytelling and gameplay.

10. Wes Anderson’s Isle of Dogs (2018)

11. State of Play’s Lumino City (2015)
State of Play’s Lume (2011)

12. A synthesis of digital and physical crafting domains
offers unique:
● aesthetic and storytelling potentials
● interactive affordances
● strategies to engage players and builders.
● maker expression in digital domains

13. Papertown VR (2016), NC State Art + Design (Polyak and FitzGerald)

14. Wallace and Gromit: The Big Fix Up AR Story (2020), Aardman Studios
Grand Day Out (1989), Nick Park

15. type of project produced in studio (among others)
Hortiquest - A Plantformer Game
teaching plant biology through papercraft, origami and gameplay
My Paper World AR (Signal Space Labs/Kiwa Digital) ● students learn plant biology lessons though game-based play
● download additional plants, lessons & papercraft instructions
● community makers upload additional “levels”, plants, insects, etc.
+
(concept developed in previous class using Story-Go-Round)
(preferable MR/AR platform - HoloLens or Magic Leap headset,
analog or digital paper/origami plants, social/multiplayer play)

16. hypothesis - physical hand-crafting, storytelling and
mixed reality can draw students to STEM,
computational media and game development fields,
and uniquely teach creativity and foundational
concepts in all of these areas.

17. what are the affordances of a physical space and its artifacts
combined with a range of technologies and techniques
(MR - headsets/mobiles/PCs, computer vision,
electronics/physical computing, animation etc.)
for storytelling and interactive gameplay?

18. Calendar
Spring 2024
Summer 2024
Fall 2024
Spring 2025
May-June 2025
Summer 2025
Recruit and hire graduate research/teaching assistants
to support studio and research (PhD or Masters-level student), technical artist with interactive
media chops, and Unreal software development capabilities
Prepare studio and production space at NCSU College of Design
+ gather and test necessary equipment, cameras, lighting, robotics
+ produce Unreal development templates to speed prototyping
+ acquire materials and supplies in advance of studio
+ if necessary, prepare distributed/virtualized studio environment, due to COVID-19
Epic Studio I - Mixed Reality & Hand-craft (Conceptual Design / Art Direction) (24 students)
Ideate, research and develop, via team-based exploration, 6-8 game concepts, with mood boards, UX storyboards,
diagrams of game mechanics, early art assets and environments, and simple functional prototypes.
(in-class support by Grad A+D TA & CS Game Design TA) (Design and Computer Science students)
(6 credit course - 16 weeks)
Epic Studio II - Mixed Reality & Hand-craft (Development / Production) (24 students)
Develop 3 or 4 game concepts generated in Studio I, into complete, publicly-releasable games.
Teams of one project manager, 2-3 developers, 2-3 designers/animators/artists
(in-class support by Grad A+D TA & CS Game Design TA) (Design and Computer Science students)
(6 credit course - 16 weeks) (co-taught with CS faculty A. Jhala, UNCSA Sound Design faculty W. Wilson)
Exhibition hosted at Epic Games and NCSU
public exhibition of conceptual development and art materials, physical artifacts and environments, and working prototypes,
hosted at Epic Games and NCSU, student-produced
Final Project Documentation, Polish & Release, and Research Scholarship
wrap up, document and release final artifacts, produce a number of academic research papers
provide 3-4 summer internships to support this work, offered to “superstars” from studio sequence

19. Target Objectives
+ release 2-3 quality student-produced games to
prominent public games stores (Epic Store, Steam, etc),
which demonstrate novel and/or artful approaches to
Mixed Reality and Hand-Craft.
+ produce shared Epic Games / NCSU exhibition, which
strengthened ties between communities, and connects
Art + Design students to local computer game industry.
+ disseminate 2-3 research publications to top-tier
academic venues, highlighting efforts and research
findings (PI Berreth and collaborators)
+ release open-source Unreal Engine templates for
Story-Go-Round platform, along with refined versions of
pedagogic materials and fabrication instructions

20. Budget (estimate)
graduate research assistantship (PhD or Masters student)
(salary, health, benefits, tuition support)(1.25 years - 15 mos.)
2 x teaching assistantships (.5 TA)
($20/hr, 10 hours/week * 16 weeks * 2 semesters * 2 students)
53,906
12,800
3 x summer internships (NCSU VN / Epic Games)
($20/hr, 32 hours/week * 10 weeks * 3 students)
19,200
equipment
computers, MR-related devices, cameras, software, etc.
(NCSU COD, DELTA, CS Dept have significant pre-existing
equipment and resources for studio’s use)
studio project supplies
supplies for studio, fabrication materials
7,500
5,000
exhibition costs
costs associated with staging public exhibition(s), rentals, etc.
5,000
misc expenses
travel between Epic/NCSU, refreshments for studio events,
admin supplies, conference fees etc.
2,500
105,906
Base
F + A (52%) 55,071
Total Costs $160,977
(NCSU overhead)

21. Story-Go-Round
A Carousel-based Platform for AR Storytelling in Education
(Berreth, Polyak and FitzGerald)
early potential prototyping tool within studio
current research project

22. Story-Go-Round
(r)evolving platform
mythotrope
(from Greek mythos ‘story’ + -tropos ‘turning)
similar to “zoetrope“
( ‘life’ + ‘turning’)

23. SIGGRAPH 2020
Edulearn20
ICERI2020
recent talks/papers
featuring this work:
complete talk (17:20)

24. diorama-making as common shared skill
storytelling hand-crafting

25. & LIGHTING
DESIGN

26. “Naut” - Micaelah Scott

27. early environment
testing with
game character

29. real-time game engine colliders (spatial mapping) from depth camera
Microsoft Azure Kinect – Mixed RealityToolkit

30. DEPTH OR
Story-Go-Round platform

34. develop and feature Unreal Engine template project
(currently Unity only)
Story-Go-Round supported game engines

35. multi-disciplinary courses taught by Berreth and
co-instructors combining designers, software developers
and storytellers working on projects together
Previous studios

36. INTERACTIVE INSTALLATIONS / MOBILE APPS / GAME DEVELOPMENT
ADN 460 - Multimedia Studio (S 2018)
Berreth w/Arnav Jhala (CS) and Pat Fitzgerald (A+D)

37. ADN 460 - Multimedia Studio (S 2018)
Berreth w/Arnav Jhala (CS) and Pat Fitzgerald (A+D)
EXAMPLE INTERDISCIPLINARY PROJECT - 12 WEEK DESIGN / DEVELOPMENT
PROJECT INTERACTIVE SCIENCE EXHIBIT, REAL CLIENT
MUSEUM OF LIFE + SCIENCE, DURHAM

38. ADN 411/561 - Visual Narrative Project Course (F 2018)
w/Frederico Freitas (History) and Arnav Jhala (CS)

39. Kirsten Benson,
Man Wong, et al.
ADN 411/561 - Visual Narrative Project Course (F 2018)
w/Frederico Freitas (History) and Arnav Jhala (CS)

40. Sara Fisher,
Andrew Wiedner, et al.
ADN 411/561 - Visual Narrative Project Course (F 2018)
w/Frederico Freitas (History) and Arnav Jhala (CS)

41. Prof. Berreth with Art + Design Masters students demoing projects @ ECGC 2019