Recommended
More Related Content
Similar to 360° Video Viewing Dataset in Head-Mounted Virtual Reality
Similar to 360° Video Viewing Dataset in Head-Mounted Virtual Reality (20)
Recently uploaded
Recently uploaded (20)
360° Video Viewing Dataset in Head-Mounted Virtual Reality
- 1. Wen-Chih Lo¹, Ching-Ling Fan¹, Jean Lee¹, Chun-Ying Huang², Kuan-Ta Chen³, and Cheng-Hsin Hsu¹ ¹National Tsing Hua University, HsinChu, Taiwan ²National Chiao Tung University, HsinChu, Taiwan ³Institute of Information Science Academia Sinica, Taipei, Taiwan 1 ACM MMSys’17, Dataset Track, Taipei, Taiwan, June 22, 2017
- 2. Introduction Basic statistics Dataset structure Content trace collection Sensor trace collection How to choose Sample applications Teaser 2
- 4. A 360° video is a view that every direction is recorded at the same time With planar monitors is passive experiences Head-Mounted Displays (HMDs) offer more immersive experiences 4
- 5. VR/AR deliver a total $3.9 billion, including $2.7 billion VR and $1.2 billion AR, revenue in 2016 [1] 5[1] After mixed year, mobile AR to drive $108 billion VR/AR market by 2021, Digi-capital, Jan 2017. https://goo.gl/Blcv2f
- 6. Latency Extremely high resolution The distortion while stitching or projecting videos/images Compress tremendous amount of video data in real-time Reduce the computational cost using tile-based viewing method 6 Distortio n
- 7. 7
- 8. We recruit 50 subjects, each of them is asked to watch ten 360°videos 52% are male Most of them are in early twenties 56% of them for the first time 8 56% 42% 2% How often do you watch 360°video using HMDs? Never Seldom Often
- 10. Video traces Image saliency map Motion map 10
- 11. We collect ten 360° videos from YouTube 1 minute long, 4K resolution, and 30fps 11 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Content
- 12. Convolutional Neural Networks (CNN) Based on a pre-trained VGG-16 networks Gray-scale image (from 0 to 255) 12 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile [1] M. Cornia, L. Baraldi, G. Serra, and R. Cucchiara. 2016. A Deep Multi-Level Network for Saliency Prediction. In International Conference on Pattern Recognition (ICPR’16). Content
- 13. Relative motions Lucas-Kanade optical flow Black-and-white images (0 or 1) 13 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile [2] B. Lucas and T. Kanade. 1981. An iterative image registration technique with an application to stereo vision. In Proc. of the International Joint Conference on Artificial Intelligence (IJCAI’7) Content
- 14. Collect sensor data from HMDs while viewers are watching 360° videos Frame Capturer: GamingAnywhere[1] Sensor Logger: OpenTrack[2] 14 [1] http://gaminganywhere.org/ [2] https://github.com/opentrack/opentrack 360° video Sensor data with timestamp250Hz Video frame with timestamp 30Hz Oculus DK2
- 15. Raw data Orientation data Tile data 15
- 16. Raw sensor data from HMDs Timestamp with epoch time Position (x, y, and z) Orientation (yaw, pitch, and roll) 16 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Sensor
- 17. Raw sensor data from HMDs Timestamp with epoch time Position (x, y, and z) Orientation (yaw, pitch, and roll) 17 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Sensor
- 18. Raw sensor data from HMDs Timestamp with epoch time Position (x, y, and z) Orientation (yaw, pitch, and roll) 18 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile x y z Sensor
- 19. Raw sensor data from HMDs Timestamp with epoch time Position (x, y, and z) Orientation (yaw, pitch, and roll) 19 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile x y z roll pitchyaw Sensor
- 20. Align the sensor data and video frames Different viewers introduce different bias A 35-sec calibration procedure 20 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Sensor
- 21. Align the sensor data and video frames Different viewers introduce different bias Design a calibration procedure 21 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Sensor
- 22. Align the sensor data and video frames Different viewers introduce different bias Design a calibration procedure 22 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Sensor
- 23. Align the sensor data and video frames Different viewers introduce different bias Design a calibration procedure 23 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Sensor
- 24. Align the sensor data and video frames Different viewers introduce different bias Design a calibration procedure 24 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Sensor
- 25. Field-of-View (FoV) are 100° x 100° circle We divide each frame into 192x192 tiles We number the tiles from upper-left to lower-right (from 0 to 199) 25 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile 100° 100° Sensor
- 26. Field-of-View (FoV) is 100° x 100° circle We divide each frame into 192x192 tiles We number the tiles from upper-left to lower-right (from 0 to 199) 26 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Sensor
- 27. Field-of-View (FoV) is 100° x 100° circle We divide each frame into 192x192 tiles We number the tiles from upper-left to lower-right (from 0 to 199) 27 Dataset Content Trace Video Trace Saliency Map Motion Map Sensor Trace Raw Orientation Tile Sensor 0
- 28. Head moveme nt Eye moveme nt Content -driven data Open- source software Applicati on-driven Lo et al. [1] Rai et al. [2] Corbillon et al. [3] Wu et al. [4] 28 [1] W. Lo, C. Fan, J. Lee, C. Huang, K. Chen, and C. Hsu. “360° Video Viewing Dataset in Head-Mounted Virtual Reality.” In Proc. of the 8th ACM on Multimedia Systems Conference (MMSys'17). 2017. [2] Y. Rai, J. Gutiérrez, and P. Callet. “A Dataset of Head and Eye Movements for 360 Degree Images.” In Proc. of the 8th ACM on Multimedia Systems Conference (MMSys'17). 2017. [3] X. Corbillon, F. Simone, and G. Simon. “360-Degree Video Head Movement Dataset.” In Proc. of the 8th ACM on Multimedia Systems Conference (MMSys'17). 2017. [4] C. Wu, Z. Tan, Z. Wang, and S. Yang. “A Dataset for Exploring User Behaviors in VR Spherical Video Streaming.” In Proc. of the 8th ACM on Multimedia Systems Conference (MMSys'17). 2017.
- 29. Viewed tile predictions Bitrate allocation α β θ FoV 0° 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 011 1 1 1 1 1 1 1 1 1 1 1 1
- 30. NOSSDAV’17 Tomorrow (6/23) 2:10pm - 3:10pm at 2nd Conference Room C. Fan, J. Lee, W. Lo, C. Huang, K. Chen, and C. Hsu, “Fixation Prediction for 360˚ Video Streaming in Head- Mounted Virtual Reality” 30
- 31. Wen-Chih Lo wchih.lo@gmail.com Dataset link address: https://nmsl.cs.nthu.edu.tw/dropbox/3 60dataset.zip 31
Editor's Notes
- Hello everyone, I am Wen-Chih Lo from NTHU. Today I am going to present a 360-degree video dataset. This work is joint with my collaborators from NTHU and Prof. Huang from NCTU and Prof. Chen from Academia Sinica.
- In this talk, I will briefly describe our dataset structure and the two major components in our 360-degree video viewing dataset. Besides, I will also show the basic statistics of our dataset and a sample application using our dataset.
- A 360-degree video is known as spherical video or immersive video. Watching these videos using traditional planar monitors gives viewers passive experience. Nowadays, HMDs are widely available. Using HMDs give viewers more immersive experience than previous one.
- In digi-capital report, they says AR/VR will keep growing up. In the future, watching 360-degree video using HMDs might be a common experience.
- However, it is challenging to deliver such a high resolution video over today’s network. For example... bla bla bla. More and more researchers, engineers are jumping into this topic. At that time, there are no public standard dataset can be used to evaluate their system performance and develop the new algorithms. But I guess that we don't need to worry about this problem. There are several dataset have been published here.
- At the beginning, I want to show you some basic statistics of our dataset. We recruit 50 subjects in our dataset. (description) Each of them is asked to watch ten videos. During the experiments, all of them are asked to stand when watching videos.
- Our dataset is unique, because it contains not only the sensor-driven data, but also the content-driven data.
- First, I want to show you the content-driven data, we called it content trace. The content trace means the impact of video content on viewer’s attentions. (click) For example, the video traces, the saliency map and motion map of 360-degree video. I will explain each of them later.
- Here, I want to show you the video traces we collected from YouTube. We divide the 10 videos into three groups, including fast-paced Natural Images (NI), slow-paced NI, and fast-paced Computer-Generated (CG) All of them are 1 minute 4k resolution videos.
- The other one is saliency map. We develop a deep neural network based on pre-trained VGG networks. We use this network to produce the saliency map. (demo video) A saliency map indicates the attraction levels of the video. It means which part of video that attracts viewer’s attention. As many previous presenters mentioned the saliency map. Which FoV should we stream to meet the viewer’s needs in the next moment.
- The last one is motion map. We analyze the optical flow of the video frames to produce the motion map using OpenCV library. (demo video) Motion map indicate the relative motions between the objects in video and the viewers.
- Second, I want to show you the sensor-drive data. It is the viewing orientation data from HMDs when a viewer watching videos. (click) In our testbed, we render the videos to Oculus DK2 and Oculus Video using Oculus SDK. (click) We use GamingAnywhere to be our Frame capturer. It records the video from HMDs. The rate of frame capturer is 30 Hz. (click) We modified and enhanced the opentrack to be our Sensor logger. Opentrack allows you to track user's head movements. It captures and timestamps the orientation data from HMDs. The rate of sensor logger is 250 Hz
- (click) The sensor trace contains 3 different data, the raw data from HMDs, the processed viewing orientation data, and the processed tile data.
- Here, I want to show you the raw sensor data from HMDs. There are 7 fields in the raw sensor data, including timestamp (epoch time), position (x, y, and z), and orientation (yaw, pitch ,and roll).
- Here, I want to show you the raw sensor data from HMDs. There are 7 fields in the raw sensor data, including timestamp (epoch time), position (x, y, and z), and orientation (yaw, pitch ,and roll).
- Here, I want to show you the raw sensor data from HMDs. There are 7 fields in the raw sensor data, including timestamp (epoch time), position (x, y, and z), and orientation (yaw, pitch ,and roll).
- Here, I want to show you the raw sensor data from HMDs. There are 7 fields in the raw sensor data, including timestamp (epoch time), position (x, y, and z), and orientation (yaw, pitch ,and roll).
- To simplify the usage of our dataset, we align the timestamps in the raw sensor data from HMD and video frames which captured from GA. However, in our pilot experiments, we find that different viewers tend to introduce different amount of bias. We then insert a 35-sec calibration video before each viewer starts watching 360° videos. This is not only the calibration procedure, but also help viewers to familiar how to watch the 360-degree video.
- There are 10 fields in the orientation data, Including index, position (x, y, and z), raw orientation (yaw, pitch ,and roll), and calibrated orientation data.
- There are 10 fields in the orientation data, Including index, position (x, y, and z), raw orientation (yaw, pitch ,and roll), and calibrated orientation data.
- There are 10 fields in the orientation data, Including index, position (x, y, and z), raw orientation (yaw, pitch ,and roll), and calibrated orientation data.
- There are 10 fields in the orientation data, Including index, position (x, y, and z), raw orientation (yaw, pitch ,and roll), and calibrated orientation data.
- We measure that the FoV of HMD is about 100°x100° circles. Therefore, we process the view orientation data, and generate viewed tile data to further simplify the usage of our dataset. There are 2 fields in the tile data, including index and the tiles which are watched by viewer.
- We measure that the FoV of HMD is about 100°x100° circles. Therefore, we process the view orientation data, and generate viewed tile data to further simplify the usage of our dataset. There are 2 fields in the tile data, including index and the tiles which are watched by viewer.
- We measure that the FoV of HMD is about 100°x100° circles. (click) Therefore, we process the view orientation data, and generate viewed tile data to further simplify the usage of our dataset. There are 2 fields in the tile data, including index and the tiles which are watched by viewer.
- Now, there are several dataset are available. Here is a table summarizing different dataset efforts. Each of them is unique and contains different features. For example, if you need to take content-driven data into consideration, you can choose the first dataset. If you need to take eye movement into consideration, you can choose the second one. I need to emphasize that this is preliminary, and if you think differently, please don't hesitate to let us know.
- Our dataset can be used in various 360-degree video applications. (click) For example, when a viewer with HMD rotates his/her head to watch some new tiles which have not been requested. (click) It may take several seconds to deliver these new tiles. To reduce the buffer time, (click) predict a viewer who watches a tile-based 360-degree video using HMDs is important. Our dataset can be used for developing and evaluating new algorithms for viewed tile predictions. Besides, our dataset also has video content with diverse characteristics, for example, it also can be used for bitrate allocation for 360° video streaming to HMDs.
- Here, I want to take a quick teaser of our prediction work. Tomorrow we will present our fixation prediction network using our dataset. If you are interested in this topic or how to use our dataset, Please feel free to join us tomorrow afternoon at the 2nd conference room. See you tomorrow.
- That is. Thank you. I am ready to take questions. I am not quiet sure. I guess….blablabla. Here is what I propose to do for deeper investigations. I will let you know my findings. I hope that we can stay in touch with each other. 1. We tiled the video in the equirectangular domain. 2. We provide original video sequences in our dataset, and they are protected with password to avoid copyright issues. 3. It is our individual judgment. 4. Saliency maps are computed based on equirectangular images using a classical image-based saliency mapping approach. There is no research that indicates with sufficient confidence that this can be done. If we tiled the video into multi-level sphere window, I guess it will reduce the distortion influence. 5. If we use the cubic map projection and process the 6 sides of the cubes separately using the same saliency mapping software. It seems to me that it won't achieve the same result.