Andrey Smorkalov, Mikhail Fominykh, and Ekaterina Prasolova-Førland: "Virtualizing Real-life Lectures with vAcademia and Kinect", Workshop on Off-The-Shelf Virtual Reality IEEE Virtual Reality Conference, Orlando, FL, USA, March, 16 2013.

1. Virtualizing Real-life
Lectures with vAcademia
and Kinect
Andrey Smorkalov
Volga State University of Technology, Russia
Mikhail Fominykh and Ekaterina Prasolova-Førland
Norwegian University of Science and Technology, Norway
Workshop on Off-The-Shelf Virtual Reality
IEEE Virtual Reality Conference
March, 16 2013 | Orlando, FL, USA
1

2. Goal
o A low-cost technological setup for translating real-
life presentations and lectures into a 3D virtual
environment
– Streaming real-life lectures into 3D virtual environment
– Automatically creating immersive 3D recordings
2

3. Motivation: learning with VR
o Virtual worlds have recognized affordances for learning,
but also many challenges
o Cost is a limiting factor for learning with virtual worlds
and other VR
o Industry, military, and healthcare are the major areas
where VR is currently used for educational purposes
o Exploring new ways of using 3D virtual worlds for
learning: capturing lectures and creating asynchronous
content out of synchronous learning activities
3

4. Motivation: capturing lectures
o „Traditional‟ video recording of lectures and web
conferences change the context of learning and do not
provide immersion or sense of presence as in 3D virtual
worlds
o „2D‟ recordings, including Machinima, do not provide a
possibility for collaborative work or a method for further
developing the content
o Kinect was previously used to improve video recording
of presentations by designing an automatic camera
control system
o => Combining 3D recording in vAcademia with Kinect
for advanced, immersive capturing of lectures
4

5. First prototype: system
implementation
Kinect plugin Animation library Cal3D
Script Executing
Library
vAcademia
graphic engine
Scripts
vAcademia
Virtualizing real-life lectures mode interface of vAcademia
5

6. First prototype: system
implementation
o Five body parts: left arm, right arm, left leg, right leg,
and head
o Standing mode (all body parts) and sitting mode (only
arms and head)
o “Adequately recognized” status for each part
o If a body part is not recognized adequately, last
adequate state is used for 0.2–0.5 sec, and then the
default state
6

7. First prototype: system
performance
o Requirements of the components
– vAcademia requires and actively uses one CPU core.
– Kinect requires a dual-core CPU, but uses only one core, as the
second is reserved for the application that uses Kinect data.
o The process of animating the lecturer’s avatar
based on the data from Kinect is not
computationally complex.
o System’s performance is satisfactory if component
requirements are satisfied, which has been
confirmed during the evaluation.
7

8. First prototype: system
evaluation
o Non-systematic evaluation during iterative
development process
– Several evaluation sessions two-three different courses
– Auditoriums of different configurations and lightning
– Involving different teachers
o Data
– Short interviews with the lecturer while watching the 3D recording
created vAcademia
o Most common feedback
– Too many restrictions on the lecturer’s movements
– Suggestions on how to increase the educational value
8

9. Applying Kinect Motion Capture in
vAcademia: Challenges
1. Low accuracy in capturing gestures
– We could not build a reliable avatar model that can move without
unnatural poses
2. Kinect does not recognize the turn of the lecturer
– Left and right arms are mixed up, unnatural pose is returned
3. Kinect cannot capture parts of the body that are
covered by other body parts or foreign objects
– Additional requirements to the setup
– Lower recognition accuracy
9

10. Applying Kinect Motion Capture in
vAcademia: Solutions for 1
o Positioning Kinect device and the lecturer
– < 1.8 m. for standing mode
– < 1.3 m. for sitting mode
– Kinect device at 0.5 m. from the floor
– Software-based turn into a zero-degree position by the vertical axis
o Additional filtration mechanism for sorting out
unnatural positions of the body parts
– Limited the acceptable values of Euler angles between the bones
– Separated hands as distinct body parts
10

11. Applying Kinect Motion Capture in
vAcademia: Solutions for 2
o The turn is recognized relatively as a function of the
position of the pelvis end points
– The resultant value is valid within the range from -110 to 110
degrees against the “facing Kinect device” direction.
o Colored markers
– Two markers are placed on the body of a lecturer on the left and on
the right side, facing the Kinect device.
– The colors should be different from the lecturer’s clothing and the
material should not be shimmering.
– If they are recognized, the system considers that the lecturer is in
the acceptable turn range. If not -> last correctly recognized state -
> default state.
11

12. Applying Kinect Motion Capture in
vAcademia: Solutions for 2
o Testing colored
markers
12

13. Applying Kinect Motion Capture in
vAcademia: Proposal for 3
o Multiple Kinect devices
– Three Kinect devices: to the left, to the right, and in front of the
lecturer
o New challenges:
– Increased price of the system
– Data from the multiple Kinect devices should be adjusted to a
single coordinate system => increased requirements for the
accuracy of locating Kinect devices
– Additional requirements to the auditorium (>7 m. across)
– Merging the data from multiple Kinect devices
13

14. Supporting Slide Presentations:
Challenges
1. Matching relative positions in real and virtual
worlds
– The position of the lecturer against the whiteboard should match
the position of the avatar against the virtual whiteboard.
2. Capturing a physical pointer
– It is an important part of lecture experience, but Kinect cannot
capture it.
3. The gestures switching slides in real world do not
have the same meaning in 3D virtual world
14

15. Supporting Slide Presentations:
Solutions for 1
o Precise match between the physical whiteboard and
the virtual one.
– Performed once after installing the physical whiteboard and the
Kinect device in the classroom.
– Capturing left and right edges of the physical whiteboard in Kinect
coordinate system.
– Installing the Kinect device and the physical whiteboard on a
specified distance from the floor.
o Further improvement
– Recognizing the borders of the physical whiteboard and creating
the replica in the 3D virtual world keeping the proportion
automatically.
15

16. Supporting Slide Presentations:
Solutions for 2
o Directing the virtual pointer based on the position of
the lecturer’s hand.
– If the half line that extends from the lecturer’s hand towards the
physical whiteboard crosses it, the avatar in the 3D VW directs a
virtual pointer to the same point.
– In order to keep the lecturer aware of his or her hand being
captured, we display a semi-transparent yellow area on the physical
whiteboard on top of the slides.
16

17. Supporting Slide Presentations:
system in work
o Capturing lecture o Streaming lecture
17

18. Supporting Slide Presentations:
Solutions for 3
o Switching slides functionality in PowerPoint by
recognizing standard gestures Swipe Left and Swipe
Right
18

19. Learning Scenarios
o Scenario 1: Lecturing as a synchronous mixed
reality activity
– Interactions between students in the physical and virtual
classrooms
– Recording student and lecturer activities in the same context
o Scenario 2: Round-table discussion as a
synchronous mixed reality activity
– Participants joining through the 3D virtual world or captured from
the real world
– Multiple Kinect-based systems can be installed in remote locations,
each of them can capturing two participants
– The designed system provides a significant advantage over pure 3D
virtual worlds in the non-verbal communication support
19

20. Learning Scenarios (2)
o Scenario 3: Motion capture for synchronous
mixed reality educational role plays
– Taking turns in the physical classroom or letting the users
captured by Kinect play the roles of facilitators
o Scenario 4: Creating immersive 3D recordings
out of live lectures
– Easy and low-cost creation of educational content for later
(asynchronous) use, such as lectures and simulations
– Any activity, including streaming Kinect-captured lectures, in
the 3D virtual world can be easily saved and revisited later
– The resultant 3D recordings combine the convenience of video
and immersive qualities of 3D virtual worlds
20

21. Questions?
Feedbacks?
Andrey Smorkalov Mikhail Fominykh Ekaterina Prasolova-Førland
smorkalovAY@volgatech.net mikhail.fominykh@ntnu.no ekaterip@ntnu.no
Acknowledgments
Mikhail Morozov
morozovMN@volgatech.net
Multimedia Systems Laboratory Virtual Spaces LLC
Volga State University of Technology vAcademia
http://mmlab.ru http://vacademia.com
21