Successfully reported this slideshow.
Your SlideShare is downloading. ×

Understanding user interactivity for immersive communications and its impact on QoE

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 24 Ad

Understanding user interactivity for immersive communications and its impact on QoE

Download to read offline

A major challenge for the next decade is to design virtual and augmented reality systems for real-world use cases such as healthcare, entertainment, e-education, and high-risk missions. This requires immersive systems that operate at scale, in a personalized manner, remaining bandwidth-tolerant whilst meeting quality and latency criteria. This can be accomplished only by a fundamental revolution of the network and immersive systems that has to put the interactive user at the heart of the system rather than at the end of the chain. With this goal in mind, in this talk, we provide an overview of our current researches on the behaviour of interactive users in immersive experiences and its impact on the next-generation multimedia systems. We present novel tools for behavioural analysis of users navigating in 3-DoF and 6-DoF systems, we show the impact and advantages of taking into account user behaviour in immersive systems. We then conclude with a perspective on the impact of users behaviour studies into QoE.

A major challenge for the next decade is to design virtual and augmented reality systems for real-world use cases such as healthcare, entertainment, e-education, and high-risk missions. This requires immersive systems that operate at scale, in a personalized manner, remaining bandwidth-tolerant whilst meeting quality and latency criteria. This can be accomplished only by a fundamental revolution of the network and immersive systems that has to put the interactive user at the heart of the system rather than at the end of the chain. With this goal in mind, in this talk, we provide an overview of our current researches on the behaviour of interactive users in immersive experiences and its impact on the next-generation multimedia systems. We present novel tools for behavioural analysis of users navigating in 3-DoF and 6-DoF systems, we show the impact and advantages of taking into account user behaviour in immersive systems. We then conclude with a perspective on the impact of users behaviour studies into QoE.

Advertisement
Advertisement

More Related Content

Slideshows for you (19)

Similar to Understanding user interactivity for immersive communications and its impact on QoE (20)

Advertisement

More from Alpen-Adria-Universität (20)

Recently uploaded (20)

Advertisement

Understanding user interactivity for immersive communications and its impact on QoE

  1. 1. Understanding user interactivity for immersive communications and its impact on QoE  Laura Toni UCL - University College London MPEG Workshop on Quality of Immersive Media: Assessment and Metrics 05th October 2021
  2. 2. A massive thanks to Silvia Rossi (UCL) .. the Phd Student behind this work Cagri Ozcinar (TCD) Our collaborators Aljosa Smolic (TCD) +CWI Pablo Cesar (CWI) Irene Viola (CWI)
  3. 3. How to assess/ define quality of experience in immersive realities? How do we actually behave in immersive realities? • Can we identify dominant behaviours (e.g., experiences)? • Can we quantify users’ similarity in their navigation? • Can we profile users? • How much the virtual experience is affected by external factors (e.g., video content features, video quality)? Image Credit: https://upload.wikimedia.org/wikipedia/commons/0/04/Mobile_World_Congress_2017_%2838277560286%29.jpg
  4. 4. Why? 4 VR therapists Live performance Coding-streaming optimisation Mu Mu et al, “User attention and behaviour in virtual reality encounter”, 2020 WHIST, AoE 2019
  5. 5. Do we have good tools already to study users’ behaviour?
  6. 6. User Behaviour Analysis in VR system Traditional metrics Scenario A Scenario B Do these metrics fully capture users’ behaviour? 6 • Mean exploration angles • Heat map • Angular velocity • Frequency of fixation FAIL
  7. 7. Our Main Goal To design metrics and methodologies to analyse users’ behaviour in 360-degree videos aiming at • identifying dominant behaviours of immersive navigation • quantifying similarities across contents and across users • analysing and quantify the level of interaction of the user with the content 7
  8. 8. 8 User Behaviour Analysis in VR system D) User’s Trajectories Analysis v1 v2 vj . . . ui ui A) Experiments B) Raw Data Collected u s e r s video C) Pre-Processing ui = < (x1, t1), . . , (xn, tn) > u s e r s video Intra-user behaviour analysis: Actual Entropy Fixation map Entropy To characterise the navigation of each user over time against different video contents. Inter-user behaviour analysis User Affinity Index To study the behaviour of a single user in correlation with others in the same content.
  9. 9. User Navigation in 3-DoF 9 v1 v2 v3 [1] S. Rossi, F. De Simone, P. Frossard, and L. Toni. 2019. Spherical Clustering of Users Navigating 360◦ Content. In IEEE International Conference on Acoustics, Speech and Signal Processing. Distance as metric to assess user similarity Clique-based clustering to detect user with similar behaviour (looking at the same viewport) Distance between viewport centres as proxy of viewport overlap [1] rities reveal general and useful features of users’ behaviour, however nswer to one simple and yet crucial question: “Can we predict users’ to fully answer to this question with the following data analysis, mation to grasp is “Do users behave similarly?". This is the key as navigation are highly challenging to predict. This motivates the ifying behaviour similarities among users, across video content rtance of developing metrics able to capture this information. et with the clique-based clustering algorithm presented in [44], usters based on their consistency in the navigation. In practice, ogether users that consistently display similar viewports over content. Also, this is done by taking into account the spherical ore introduce a novel metric (based on the clique-based clustering rity among users’ navigation trajectories within the same given e User A�nity Index (UAI), given as follows: UAI = ÕC i=1 xi · wi ÕC i=1 wi (1) detected in a frame by the clique-clustering2, xi is the % of users sers sampled) in cluster i andwi is the number of users in cluster i. s the weighted average of cluster popularity (i.e., how many users • C: number of clusters detected in a frame by the clique-clustering • xi : % of users in cluster i • wi : number of users in cluster i Affinity metric
  10. 10. Results - Clustering of Trajectories - - /2 0 /2 theta 3* /4 /2 /4 0 phi - - /2 0 /2 theta 3* /4 /2 /4 0 phi Spectral clustering of trajectories Proposed Clique-Based clustering
  11. 11. Chapter 4. Toward User Prediction in Virtual Reality (a) Rollercoaster video 11 Analysis based on Clusters ➡ Users similarly behaving
  12. 12. Viewport angular velocity • Users dynamically navigate more the content with laptop • Movie are explored slower with all devices • HMD has the lowest speed across devices and video categories 360 Video Renderer Scene Te t re ie port trajectories Scene objects Camera Mesh Sphere Geometr Sensors ODV Te t re 360 180 M SQL 360 Video Renderer Scene Te t re ie port trajectories Scene objects Camera Mesh Sphere Geometr Sensors ODV Te t re 360 180 M SQL 360 Video Renderer Scene Te t r Scene objects Camera Mesh Sphere Geometr Sensors Users’ behaviour changes not only based on the video content categories but also on the selected viewing devices 12
  13. 13. • In contents with no main focus of attention, users experience a low affinity, which is interestingly not perturbed by the viewing device. • Users tend to explore content characterised by a dominant focus of attention in a very similar way. • In content with a main focus of attention, the user affinity is strongly related to the selected viewing device. In particular, the HMD leads to quite similar navigation among users. Take Home Message 13
  14. 14. User Navigation in 6-DoF 14 • The head is the only “interface” for interacEvity •The media is displayed from an inward posiEon EE, John Doe, Fellow, OSA, and Jane Doe, Life Fellow, IEEE pdated} Thanks to ble technology, and in our daily life. A h has turned from roving any aspect rts with the need we investigate the F VR environment. es and similarities ting methodologies settings. Our sim- ths of users while F conditions, show 3-DoF in assessing tions, we state the analysis of 6-DoF L A TEX, paper, tem- ionised how users nt, going beyond technology, and nd interaction. In s provided with a lay (HMD) – and scene and display im/herself, named ion functionalities assified as 3- or 6- nario, the de-facto r spherical video, ment on a virtual (a) 3-DoF (b) 6-DoF Fig. 1. Viewing paradigm in 3- and 6-DoF VR. point clouds) which are observed from an outward position (Fig. 1 (b)). This extra-level brings the virtual experience even closer to reality: a higher level of interactivity makes the user feels more immersed within the virtual environment [1]. Despite their differences, the common denominator of both 3- and 6-DoF systems is the user as the main driver of the content being displayed. In other words, both type of environ- ments define a user-centric era, in which content preparation, streaming, as well as rendering need to be tailored to the users’ interaction to remain bandwidth-tolerant whilst meet- ing quality and latency criteria. Media codecs, for example, are optimised in such a way that the quality experienced by the user is maximised [2], [3]. Analogously, streaming platforms should also ensure smooth navigation in the scene to make the user experience real as much as possible [4]. However, each user within an immersive environment might have a different interaction with the content thus, maximising the experience per single viewer is highly challenging. The • The user has now the freedom to move inside the VR space • The media is displayed from an outward posiEon Michael Shell, Member, IEEE, John Doe, Fellow, OSA, and Jane Doe, Life Fellow, IEEE ct—{SR: from ICIP paper still to be updated} Thanks to vances in computer graphics, wearable technology, and ity, Virtual Reality (VR) has landed in our daily life. A ty in VR is the role of the user, which has turned from assive to entirely active. Thus, improving any aspect oding–delivery–rendering chain starts with the need rstanding user behaviour. To do so, we investigate the n trajectories of users within a 6-DoF VR environment. ly, we investigate the main differences and similarities 3 and 6-DoF navigation through existing methodologies to study user behaviour in 3-DoF settings. Our sim- esults, based on real navigation paths of users while g dynamic volumetric media in 6-DoF conditions, show ations of clustering algorithms for 3-DoF in assessing larity in 6-DoF. Given these observations, we state the developing new solutions for the analysis of 6-DoF es. Terms—IEEE, IEEEtran, journal, L A TEX, paper, tem- I. INTRODUCTION TUAL reality technology has revolutionised how users age and interact with media content, going beyond ive paradigm of traditional video technology, and higher degrees of immersiveness and interaction. In Reality (VR) settings, the viewer is provided with a ce – typically a head-mounted display (HMD) – and d to freely navigate the immersive scene and display portion of the environment around him/herself, named Depending on the enabled locomotion functionalities space, VR environments can be classified as 3- or 6- of-Freedom (DoF). In the first scenario, the de-facto dia content is an omnidirectional or spherical video, (a) 3-DoF (b) 6-DoF Fig. 1. Viewing paradigm in 3- and 6-DoF VR. point clouds) which are observed from an outward position (Fig. 1 (b)). This extra-level brings the virtual experience even closer to reality: a higher level of interactivity makes the user feels more immersed within the virtual environment [1]. Despite their differences, the common denominator of both 3- and 6-DoF systems is the user as the main driver of the content being displayed. In other words, both type of environ- ments define a user-centric era, in which content preparation, streaming, as well as rendering need to be tailored to the users’ interaction to remain bandwidth-tolerant whilst meet- ing quality and latency criteria. Media codecs, for example, are optimised in such a way that the quality experienced by the user is maximised [2], [3]. Analogously, streaming platforms should also ensure smooth navigation in the scene to make the user experience real as much as possible [4]. However, each user within an immersive environment might have a different interaction with the content thus, maximising the experience per single viewer is highly challenging. The Is the position of viewport center over time enough to identify user behaviour ? How to assess user behaviour similarity in 6-DoF?
  15. 15. Distance Metrics 15 To verify if the overlap ra<o can be subsEtuted with a distance between users, we consider 4 different distance metrics: Oi,j t • → euclidean distance between user posiEons in the space • → euclidean distance between viewport centres on PC • → geodesic distance between viewport centres on PC • → cityblock distance between viewport centres on PC L2 x xi t, xj t L2 p pi t, pj t Gp pi t, pj t L1 p pi t, pj t
  16. 16. Why do we need a new clustering? 16 JOURNAL OF L A TEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 (a) Ground-truth (Oth = 75%) (b) w1 (single feature metric) (c) w2 (sin (d) w3 (single feature metric) (e) w4 (single feature metric) (f) w5 (mu d-truth (Oth = 75%) (b) w1 (single feature metric) (c) w2 (single feature metric) ngle feature metric) (e) w4 (single feature metric) (f) w5 (multi-feature metric) ulti-feature metric) (h) w7 (multi-feature metric) (i) w8 (multi-feature metric) ts in frame 50 of sequence PC1 (Longdress). Each dot represent a user on the virtual floor while the blue star stands for the volumetric d, per each cluster with more than 2 users are reported on brackets the following values: number of users included in the cluster, averaged verlap and corresponding variance within the cluster.{SR: not sure if having this legend or remove the variance of the overlap in order each proposed similarity metric (Fig. 4 (b- , each user is represented by a point on the exception is given by w1, which generates a variable set of clusters with consistent values of overlap ratio, over 0.64. Let 7 metric) (c) w2 (single feature metric) Ground truth clustering (75% overlap threshold) Clustering from 3DoF (based on viewport center only) New clustering for 6DoF (taking into account users’ position)
  17. 17. 17 User Behaviour Analysis in VR system D) User’s Trajectories Analysis v1 v2 vj . . . ui ui A) Experiments B) Raw Data Collected u s e r s video C) Pre-Processing ui = < (x1, t1), . . , (xn, tn) > u s e r s video Intra-user behaviour analysis: Actual Entropy Fixation map Entropy To characterise the navigation of each user over time against different video contents. Inter-user behaviour analysis User Affinity Index To study the behaviour of a single user in correlation with others in the same content.
  18. 18. A key quantity in information theory that measures the uncertainty associated with an event. Intra-User behaviour metrics Entropy H(X) = − ∑ x∈X p(x)log(p(x)) Actual Entropy Introduced as a proxy of predictability of human mobility patterns [1], the actual entropy quantifies the information carried within a given trajectory. [1] C. Song, Z. Qu, N. Blumm, and A. Barabási. 2010. Limits of predictability in human mobility. Science. Hact (X) ≈ ( 1 n n ∑ t=1 λt ) −1 log2(n) 18
  19. 19. Intra-User behaviour analysis A B X. Corbillon, F. De Simone, and G. Simon. 2017. 360-degree video head movement dataset. In Proceedings of the 8th ACM on Multimedia Systems Conference. 19
  20. 20. Intra-User behaviour analysis A B X. Corbillon, F. De Simone, and G. Simon. 2017. 360-degree video head movement dataset. In Proceedings of the 8th ACM on Multimedia Systems Conference. 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 5860 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 User 30: = 0.12 = 0.21·10−2 Hact (X) H(M) User 48: = 0.65 = 0.43·10−2 Hact (X) H(M) User 49: = 0.28 = 0.32·10−2 Hact (X) H(M) ➡ High indicates more randomness in the navigation Hact
  21. 21. Intra-User behaviour analysis A B X. Corbillon, F. De Simone, and G. Simon. 2017. 360-degree video head movement dataset. In Proceedings of the 8th ACM on Multimedia Systems Conference. ➡ Users profiling (high- low- interaction) despite the content
  22. 22. • We need to study, understand, and predict users behaviour in immersive tools New tools needed for this study • Clusters are meaningful if identifying users looking at the same portion of content • Deeper analysis showed us correlation between content-device and level of interactivity • User interactivity can be a good metric for system design and QoE assessment Conclusions 22
  23. 23. What’s the link? (Future Directions) 23 How do assess/ define quality of experience in immersive realities? ? How do we actually behave in immersive realities? • When studying QoE, should we focus on dominant behaviours? (Should we discard outliers?) • What are the key trajectories/interactions experienced by the users? • Does quality impact behaviour and QoE in immersive systems? • Is the similarity in users behaviour related to the quality of the experience?
  24. 24. Thank You! Questions? Learning and Signal Processing Lab UCL https://laspucl2016.com

×