Virtual Choreographies: a new kind of semantics for immersive environments The document discusses using virtual choreographies to provide meaningful semantics for actions in immersive environments like serious games, simulations, and interactive videos. It notes choreographies can describe character behaviors across platforms and help identify behaviors for intervention or assessment. Open questions are discussed around matching behaviors to known choreographies, inferring choreographies from clustered behaviors, and how human-machine cooperation could maximize effectiveness of virtual choreographies.

1. Virtual Choreographies:
a new kind of semantics for
immersive environments
Leonel Morgado
Leonel.Morgado@uab.pt
November
18th, 2020
Future of AI in business in business
analytics.

2. Story choreographies go
from staging diagrams
to prototypes,
until final rendering

3. In serious games, training simulations, data analysis, and
after-action review, the most valuable information is not the
visual quality: it’s the meaning of actions.

4. The semantics of the story choreographies depend on
the actual purpose of the
game/simulation/animation/interactive video…

5. “Initially, instructors may make time for using a
new training game. However, if the game cannot
be modified according to their experience, their
changing training needs, or the emerging
conditions of the current operational
environment, then the game will not be utilized.”

6. Someone Does something Somewhere

7. XAP
I
Actor Verb Place Context

8. Lacet et al. (2020) Preserving story choreographies across multiple platforms: An approach to platform-
independent reuse of characters' behaviors for games, simulations, animations and interactive videos,
Proceedings of the 9th International Conference on Digital and Interactive Arts

9. Anything that renders the
story choreography.

11. Darth Vader Luke Skywalker
Cloud City Core
Story Recipe
ACTOR:
VERB:
PLACE:
Darth Vader
Luke Skywalker
being at
saying
looking at
going to
rotating
getting up
climbing down
gesturing
Upper platform
Center of beam
End of beam
Pole
Lower platform
shouting
rotating
climbing up
climbing down
expressing
crying
denying
fighting

12. Outcome: Platform-Adapted Choreography
Lacet et al. (2020) Preserving story choreographies across multiple platforms: An
approach to platform-independent reuse of characters' behaviors for games,
simulations, animations and interactive videos, Proceedings of the 9th International
Conference on Digital and Interactive Arts

13. Lopes et al. (2009) Use of a virtual world system in sports coach education for
reproducing team handball movements, Journal of Virtual Worlds Research
The trigger:
Sports tactics training for coaches

14. Lacet et al. (in press) Magic Board GO! interactive stories that can be reused and played
on multiple game boards, to appear in Proceedings of Videojogos 2020.
The current approach:
multiplatform stories

15. The challenge: it’s not “Go to position X,Y,Z”
but “go to the available screw”, for instance.
Vilela et al. (2012) Aplicação de avatares autónomos para desempenhar o papel de
membros na execução de trabalhos em equipa. Proceedings of CISTI 2012
Pinheiro et al. (2014) Development of a mechanical maintenance training simulator in
OpenSimulator for F-16 aircraft engines. Entertainment Computing

16. But for evaluation, assessment, and
feedback, we also need to understand what
is occurring or happened
…and consider that dealing with hundreds
of trainees is not the same as dozens.

17. Cassola et al. (2017). Using choreographies to support the gamification process on the
development of an application to reduce electricity costs. Lecture Notes in Computer Science
Choreographies can help identify and target behaviours for intervention

18. Beaconing provided massification structure
with Gamified Lesson Plans
Cardoso et al. (2020). Authoring game-based learning activities that are manageable by
teachers. ERCIM News
Coelho et al. (2020). Serious Pervasive Games. Frontiers in Computer Science

19. Baptista et al. (2015). Relation between game genres and competences for in-game
certification. International Conference on Serious Games, Interaction, and Simulation
Activity tracking: teachers get analytics and feedback to decide how to act
- Virtual Choreographies can help cluster behaviours and identify them -

20. Schlemmer, Morgado, Beck, Moreira, Bittencourt, Coelho, Cardoso
(unpublished draft) Inven!RA Platform for Inventive Activity Plans

21. Virtual choreographies:
opportunities and challenges
• How to match actual behaviours to a known choreography?
• How to infer choreographies from clusters of behaviours?
• How to provide visual analytics for choreography matching and proximity?
• How can joint human-machine cooperation be employed to maximize
effectiveness of choreography matching and identification?
• What software architectures and data structures are more adequate for
choreography data?
• What data-harvesting methods are adequate for immersive systems, IoT
environments, heavy industry, etc.?

22. Virtual Choreographies:
a new kind of semantics for
immersive environments
Leonel Morgado
Leonel.Morgado@uab.pt
November
18th, 2020
Future of AI in business in business
analytics.
Get in touch for
research cooperation,
joint supervisions, etc.