This document proposes a standard called AR-LEM for Augmented Reality Learning Experience Models. It describes models for activities, workplaces and augmentations that can be used to create augmented reality learning experiences. The activity model defines action steps, triggers, instructions and flow. The workplace model defines tangible people, places and things that can be activated or deactivated. It also describes how to attach augmentations like overlays, videos and 3D models to tangibles. The document provides examples of using these models to assemble an augmented reality activity for assembling a cabinet, including activating overlays on detected markers and transitioning between action steps.

1. A Standard for
Augmented Reality
Learning Experience Models
(AR-LEM)
Fridolin Wild1), Christine Perey2), Kaj Helin3), Jaakko Karjalainen3),
Paul Lefrere4)
1) The Open University, UK
2) Perey Research and Consulting, CH
2) VTT, Finland
4) CCA, UK

2. World Knowledge
2
Activity Knowledge
http://bit.ly/arlem-input

3. Embedding knowledge into experience
3
augmentation

4. The Activity Model
4
“find the spray
gun nozzle size
13”
Messaging in the
real-time
presence
channel and
tracking to xAPI
onEnter/onExit
chaining of
actions and
other
activations/deact
ivations
Styling
(cascading) of
viewports and
UI elements
Constraint modeling:
specify validation
conditions and model
workflow branching
e.g. smart player;
e.g. search widget
http://bit.ly/arlem-input

5. The Workplace Model
5
The ‘tangibles’:
Specific persons,
places, things
The ‘configurables’:
devices (styling),
apps+widgets
The ‘triggers’:
Markers trigger
Overlays; Overlays
trigger human action
Overlay ‘Primitives’:
enable re-use of e.g.
graphical overlays
http://bit.ly/arlem-input

6. Action steps
<action
id=‘start’
viewport=‘actions’
type=‘actions’
>
</action>

7. Instructions for action
<instruction><![CDATA[
<h1>Assembly of a simple cabinet</h1>
<p>Point to the cabinet to start…</p>
]]></instruction>

8. Defining flow: Entry, Exit, Trigger
<enter removeSelf="false">
</enter>
<exit>
<activate type="actions" viewport="actions"
id="step2"/>
<deactivate type="actions" viewport="actions"
id="start"/>
</exit>
<triggers>
<trigger type="click" viewport="actions" id="start"/>
</triggers>
Nothing (for
now)
On exit:
launch step2
On exit: remove
dialogue box ‘start’
This action step
shall be exited by
‘clicking’ on the
dialogue box

9. Sample script
<activity id="assembly" name="Assembly of cabinet" language="english"
workplace="http://crunch.kmi.open.ac.uk/people/~jmartin/data/workplace-AIDIMA.xml"
start="start">
<action id=‘start’ viewport=‘actions’ type=‘actions’>
<enter removeSelf="false">
</enter>
<exit>
<activate type="actions" viewport="actions" id="step2"/>
<deactivate type="actions" viewport="actions" id="start"/>
</exit>
<triggers>
<trigger type="click" viewport="actions" id="start"/>
</triggers>
<instruction><![CDATA[<h1>Assembly of a simple cabinet</h1><p>Point to the cabinet
to start ... </p>]]></instruction>
</action>
<action id="step2" viewport="actions” type=“actions”>
<enter></enter>
<exit removeSelf="true”></exit>
<triggers>
<trigger type="click" viewport="actions" id="step1"/>
</triggers>
<instruction><![CDATA[<h1>step2</h1><p>do this and that.</p>]]></instruction>
</action>
</activity>

10. Working with ‘tangibles’
 Utilise computer vision engine to
detect things/places/people
(=tangibles)
 Define tangibles in the
workplace model
 Then activate (or deactivate) what
shall be visible and relevant in each
action step

11. Points of interest on <thing>s
11
· Default POI (0, 0, 0)
· Handle (0, -5, 0)
· Tip (-3, 5, 0)
· Default POI (0, 0, 0)
· Pick
· Handle (0, -5, 0)
· Tip (-3, 5, 0)
1. Configure routines for
detection by providing
either (pre-packaged)
fiducial markers or loading
image targets from a URL
provided
2. Bind the tangibles of choice
(things, persons, places) to
the instantiated markers
3. Add points-of-interest
(POIs) configuration, set
offsets, set scale
4. Configure and attach event
handlers for additional
functionality (such as
internet-of-things data
handling) to specified
tangibles
5. Flag to a workflow
controller component that
the experience is
configured and can now be
executed

12. In the workplace model
 We open the workplace model and define a new thing (under
resources/tangibles/things):
<thing id="board1" name="Cabinet"
urn="/tellme/object/cabinet1" detectable="001">
<pois>
<poi id="leftside" x-offset="-0.5" y-offset="0"
z-offset="0.1"/>
<poi id="default" x-offset="0" y-offset="0" z-
offset="0"/>
</pois>
</thing>
The id is what we
will reference
The detectable
specifies, which
marker (or
sensor state)
will be bound
to the thing
Poi = point of interest:
specify locations relative
to centre of marker
(x=y=z=0: centre)

13. Triggers and tangibles
 If you add a tangible trigger (for ‘stareGaze navigation’), an target icon
will be overlaid, rotating in yellow, turning green when the stare
duration (3 secs) has been reached
<trigger type="detect" id="board1" duration=”3"/>

14. Markers and pre-trained markers
 Marker must be defined in the workplace model
 Possible to provide pretrained markers (and their PDF file
to print): named, e.g., 001 to 050
 Markers shall be specified via their id in the computer
vision engine (under resources/triggers/detectables):
<detectable id="001" sensor="engine" type="marker"/>
<detectable id=”myid" sensor="engine"
type=”image_target”
url=“myurl.org/marker.zip”
/>

15. Activates and deactivates
 Now we have defined a thing called ‘board1’ and we have tied it to the
marker 001
 We can start referring to it now from the activity script: we can, e.g.,
activate pictogram overlays for the verbs of handling and motion
<activate
tangible="board1"
predicate="point"
poi="leftside"
option="down”
/>

16. <activity id="assembly" name="Assembly of cabinet" language="english"
workplace="http://crunch.kmi.open.ac.uk/people/~jmartin/data/workplace-AIDIMA.xml"
start="start">
<action id=‘start’ viewport=‘actions’ type=‘actions’>
<enter removeSelf="false”>
<activate tangible="board1" predicate="point" poi="leftside" option="down"/>
<activate tangible="board1" predicate="addlabel" poi="default"
option="touchme"/>
</enter>
<exit>
<deactivate tangible="board1" predicate="point" poi="leftside"/>
<deactivate tangible="board1" predicate="addlabel" poi="default"/>
<activate type="actions" viewport="actions" id="step2"/>
<deactivate type="actions" viewport="actions" id="start"/>
</exit>
<triggers>
<trigger type="click" viewport="actions" id="start"/>
</triggers>
<instruction><![CDATA[<h1>Assembly of a simple cabinet</h1><p>Point to the cabinet
to start ... </p>]]></instruction>
</action>
<action id="step2" viewport="actions” type=“actions”>
<enter></enter>
<exit removeSelf="true”></exit>
<triggers>
<trigger type="click" viewport="actions" id="step1"/>
</triggers>
<instruction><![CDATA[<h1>step2</h1><p>do this and that.</p>]]></instruction>
</action>
</activity>
Display an
arrow pointing
downwards on
the point of
interest ‘leftside’
Display a label
‘touchme’ at the
centre of the
marker
Remove both
visual overlays
when this action
step is exited

17. Non-normed overlays
<activate tangible=”board1"
predicate="add3dmodel"
poi="leftside"
option=”augmentation"/>
<augmentations>
<augmentation id="cube" scale="1" y_angle="180.0"
url="http://myurl.org/cube.unity3d"
/>
</augmentations>
<activate tangible=”board1”
predicate=”addvideo”
option=“http://myurl.org/myvideo.mp4"
/>
<activate tangible=”board1"
predicate=”addimage”
option=“http://myurl.org/myvideo.png"
/>

18. Normed overlays – verb primitives
All verbs need the ‘id’ of the
tangible, some of them have ‘POIs’
that they need as input, few have
‘options’
 'point': poi + options = up, upperleft, left, lowerleft,
down, lowerright, right, upperright
 'assemble’, ‘disassemble’
 ‘close’
 ‘cut’: poi
 'drill': poi
 'inspect': poi
 'lift':
 'lower’:
 'lubricate':
 'measure': poi
 'open’
 ‘pack’
 ‘paint’
 ‘plug’
 'rotate-cw’, 'rotate-ccw': poi
 'screw': poi
 'unfasten': poi
 'unpack
 'unplug’:
 'unscrew': poi
 'forbid':
 'allow':
 'pick':
 'place':

19. 19
Viitaniemi et al. (2014):
Deliverable d4.2,
TELLME consortium

20. Warning signs
Add an enter activation:
<activate tangible=”board1" poi=“leftside” warning="p030"/>
…

21. Internet of things (workplace model)
21

22. IoT in the activity ML
22

23. Example Implementation

24. Towards a Component Reference
Architecture
24
Enquire
Repository
Mix
WorkplaceModel
ActivityModel
ExperienceSpeechOutput
Component
FlowControl
Component(andtextual
Instruction)
Augmentation
Component
Match
Tracking
Component
ConstraintTesting
Optimise
Analytics

25. The END