This document describes experiments using crowdsourced virtual reality. The researchers validated a panel of 242 virtual reality-eligible workers. They implemented a user flow between desktop and VR experiments to replicate three previous studies remotely, examining place illusion, embodiment illusion, and plausibility illusion. Some limitations included device constraints like battery life and overheating. The work demonstrates the potential of crowdsourcing to conduct cheaper, faster, and more reproducible VR experiments at larger scale.
WSDM 2018 Tutorial on Influence Maximization in Online Social NetworksCigdem Aslay
In this tutorial, we extensively survey the research on social influence propagation and maximization, with a focus on the recent algorithmic and theoretical advances. To this end, we provide detailed reviews of the latest research effort devoted to (i) improving the efficiency and scalability of the influence maximization algorithms; (ii) context-aware modeling of the influence maximization problem to better capture real-world marketing scenarios; (iii) modeling and learning of real-world social influence; (iv) bridging the gap between social advertising and viral marketing.
Comparing Real and Virtual Quality of Life Data Using a Virtual Assisted Sel...Mark Bell
Please use this to cite presentation:
Bell, M., Castronova, E. & Wagner, G. 2009, May). Comparing Real and Virtual Quality of Life Data Using a Virtual Assisted Self-Interview Method. Presented at 59th Annual ICA Conference , Chicago, Illinois.
In this talk we detail the step to creating a Visual Search engine for 1M Amazon product using MXNet Gluon and the K-Nearest Neighbor search library HNSW.
For implementation details, check this repository: https://github.com/ThomasDelteil/VisualSearch_MXNet
Video available here:
https://www.youtube.com/watch?v=9a8MAtfFVwI
Demo website available here:
https://thomasdelteil.github.io/VisualSearch_MXNet/
This thesis constitutes one of the first investigations that lie at the intersection of social influence propagation, viral marketing, and social advertising. The objective of this thesis is to take the algorithmic aspects of viral marketing out of the lab, and further enhance these aspects to account for the real world social advertisement models, by drawing on the viral marketing literature to study social influence aware ad allocation for social advertising. To this end, we take a first step towards enabling social influence online analytics in support of viral marketing decision making, and propose efficient influence indexing framework that can accurately answer topic-aware viral marketing queries with milliseconds response time. We then initiate investigation in the area of social advertising through the viral marketing lens, aligned with real world social advertisement models, and introduce two fundamental optimization problems, regarding the allocation of ads to social network users under social influence. We devise greedy approximation algorithms with provable approximation guarantees for the novel problems introduced. We also develop scalable versions of our approximation algorithms by leveraging the notion of reverse reachability sampling on social graphs, and experimentally confirm that our algorithms are scalable and deliver high quality solutions.
WSDM 2018 Tutorial on Influence Maximization in Online Social NetworksCigdem Aslay
In this tutorial, we extensively survey the research on social influence propagation and maximization, with a focus on the recent algorithmic and theoretical advances. To this end, we provide detailed reviews of the latest research effort devoted to (i) improving the efficiency and scalability of the influence maximization algorithms; (ii) context-aware modeling of the influence maximization problem to better capture real-world marketing scenarios; (iii) modeling and learning of real-world social influence; (iv) bridging the gap between social advertising and viral marketing.
Comparing Real and Virtual Quality of Life Data Using a Virtual Assisted Sel...Mark Bell
Please use this to cite presentation:
Bell, M., Castronova, E. & Wagner, G. 2009, May). Comparing Real and Virtual Quality of Life Data Using a Virtual Assisted Self-Interview Method. Presented at 59th Annual ICA Conference , Chicago, Illinois.
In this talk we detail the step to creating a Visual Search engine for 1M Amazon product using MXNet Gluon and the K-Nearest Neighbor search library HNSW.
For implementation details, check this repository: https://github.com/ThomasDelteil/VisualSearch_MXNet
Video available here:
https://www.youtube.com/watch?v=9a8MAtfFVwI
Demo website available here:
https://thomasdelteil.github.io/VisualSearch_MXNet/
This thesis constitutes one of the first investigations that lie at the intersection of social influence propagation, viral marketing, and social advertising. The objective of this thesis is to take the algorithmic aspects of viral marketing out of the lab, and further enhance these aspects to account for the real world social advertisement models, by drawing on the viral marketing literature to study social influence aware ad allocation for social advertising. To this end, we take a first step towards enabling social influence online analytics in support of viral marketing decision making, and propose efficient influence indexing framework that can accurately answer topic-aware viral marketing queries with milliseconds response time. We then initiate investigation in the area of social advertising through the viral marketing lens, aligned with real world social advertisement models, and introduce two fundamental optimization problems, regarding the allocation of ads to social network users under social influence. We devise greedy approximation algorithms with provable approximation guarantees for the novel problems introduced. We also develop scalable versions of our approximation algorithms by leveraging the notion of reverse reachability sampling on social graphs, and experimentally confirm that our algorithms are scalable and deliver high quality solutions.
https://github.com/telecombcn-dl/lectures-all/
These slides review techniques for interpreting the behavior of deep neural networks. The talk reviews basic techniques such as the display of filters and tensors, as well as more advanced ones that try to interpret which part of the input data is responsible for the predictions, or generate data that maximizes the activation of certain neurons.
We build a network of locations from Twitter geolocalized tweets. We then analyze the small-world and scale-free properties of such spatial network. We also use modularity to reveal the structre of neighborhood communities.
Community search is the problem of finding a good community for a given set of query vertices.
In this work we propose a novel method that it is in general more efficient and effective than state-of-the art, it can handle multiple query vertices, it yields optimal communities, and it is parameter free.
Workshop on Virtual Worlds: Second Life and OpenSim (Nov. 8, 2011) UC DavisBernadette Daly Swanson
Workshop on Virtual Worlds: Second Life & OpenSim: simulations and research
Searching the peer reviewed literature across the disciplines (2 hours) with Bernadette Daly Swanson, MLIS (HVX Silverstar in Second Life and OSGrid (OpenSim grid).
MHI 289h 4 units: Virtual Reality, Simulation, and Robotics
with Alberto Odor, MD
Health Informatics Master's Program, University of California, Davis.
http://www.ucdmc.ucdavis.edu/informatics/masters/
Fall, 2011
On A Quest for Combating Filter Bubbles and Misinformation.
Invited Talk, Chinese University of Hong Kong at Shenzhen, Dec 13, 2022.
Social media have greatly facilitated access to information and news and have enhanced users' ability to share with peers their views on issues. However, they have unfortunately led to increased societal polarization. At the center of this phenomenon are filter bubbles and misinformation. Filter bubbles are the result of excessive personalization which enhances relevance of content at the price of limiting exposure to a specific viewpoint. These bubbles are amplified by the so-called echo chambers that exist in social media, whereby members of a community mutually reinforce a fixed opinion or viewpoint on an issue. Misinformation as well as disinformation, on the other hand, tends to propagate through the network, often faster and more virally than truth.
Both problems manifest themselves in the form of groups of actors working in concert and providing mutual reinforcement. How can we recognize these groups? Having detected them, how can we counteract these problems? The first question can benefit from an examination of techniques developed to search for dense subgraphs in an underlying network. As for the second question, a natural approach for countering filter bubbles is to launch some kind of counter-campaign to balance users' exposure to viewpoints. Countermeasures for misinformation propagating through a network depend on the party planning the countermeasure. The network host can intervene and take steps to limit the propagation of misinformation, but these actions come with a cost. Besides the political sensitivity and cost of limiting freedom of expression, what if the intervention was by mistake done on genuine information? On the other hand, a third party interested in countering the propagation of misinformation may launch a counter-campaign. Some of the ideas behind designing such campaigns have strong connections to a classic problem called Influence Maximization, studied in a very different context, driven by different applications like viral marketing, infection containment, and revenue or welfare maximization. In this talk, we will examine research on detecting dense subgraphs as well as competitive influence maximization and discuss how that can inspire techniques for addressing the two problems above.
NG2S: A Study of Pro-Environmental Tipping Point via ABMsKan Yuenyong
A study of tipping point: much less is known about the most efficient ways to reach such transitions or how self-reinforcing systemic transformations might be instigated through policy. We employ an agent-based model to study the emergence of social tipping points through various feedback loops that have been previously identified to constitute an ecological approach to human behavior. Our model suggests that even a linear introduction of pro-environmental affordances (action opportunities) to a social system can have non-linear positive effects on the emergence of collective pro-environmental behavior patterns.
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...Ruofei Du
Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.
Social media in virtual reality is in a high-growth market segment with influential products and services in virtual tourism, remote education, and business meetings. Nevertheless, previous systems have never achieved an online platform which renders a 6DoF mirrored world with geotagged social media in real time. In this paper, we introduce the technical detail behind Geollery.com which reconstructs a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local street views and depth maps. When viewed from afar, we apply projection mappings to 3D geometries extruded from building polygons for a coarse level of detail. In contrast to teleportation, our system allows users to virtually walk through the mirrored world at the street level. Our system integrates geotagged social media from both internal users and external sources such as Twitter, Yelp, and Flicker. We validate our real-time strategies of Geollery.com on various platforms including mobile phones, workstations, and head-mounted displays.
2019년 파이콘 한국에서 진행된 튜토리얼 자료입니다. 최재식 교수님께서 설명가능인공지능이란 무엇인가에 대해 발표해주신 Part 1 발표자료입니다. 아래 링크를 통해 행사 관련 정보를 확인하실 수 있습니다.
http://xai.unist.ac.kr/Tutorial/2018/
https://github.com/OpenXAIProject/PyConKorea2019-Tutorials
Part 1: https://www.slideshare.net/OpenXAI/2019-part-1
Part 2: https://www.slideshare.net/OpenXAI/2019-lrp-part-2
Part 3: https://www.slideshare.net/OpenXAI/2019-shap-part-3
Calit2: Experiments in Living in the Virtual/Physical WorldLarry Smarr
11.02.24
Pat Ledden Memorial Luncheon
Faculty Luncheon Seminar
Title: Calit2: Experiments in Living in the Virtual/Physical World
UC San Diego Faculty Club
https://github.com/telecombcn-dl/lectures-all/
These slides review techniques for interpreting the behavior of deep neural networks. The talk reviews basic techniques such as the display of filters and tensors, as well as more advanced ones that try to interpret which part of the input data is responsible for the predictions, or generate data that maximizes the activation of certain neurons.
We build a network of locations from Twitter geolocalized tweets. We then analyze the small-world and scale-free properties of such spatial network. We also use modularity to reveal the structre of neighborhood communities.
Community search is the problem of finding a good community for a given set of query vertices.
In this work we propose a novel method that it is in general more efficient and effective than state-of-the art, it can handle multiple query vertices, it yields optimal communities, and it is parameter free.
Workshop on Virtual Worlds: Second Life and OpenSim (Nov. 8, 2011) UC DavisBernadette Daly Swanson
Workshop on Virtual Worlds: Second Life & OpenSim: simulations and research
Searching the peer reviewed literature across the disciplines (2 hours) with Bernadette Daly Swanson, MLIS (HVX Silverstar in Second Life and OSGrid (OpenSim grid).
MHI 289h 4 units: Virtual Reality, Simulation, and Robotics
with Alberto Odor, MD
Health Informatics Master's Program, University of California, Davis.
http://www.ucdmc.ucdavis.edu/informatics/masters/
Fall, 2011
On A Quest for Combating Filter Bubbles and Misinformation.
Invited Talk, Chinese University of Hong Kong at Shenzhen, Dec 13, 2022.
Social media have greatly facilitated access to information and news and have enhanced users' ability to share with peers their views on issues. However, they have unfortunately led to increased societal polarization. At the center of this phenomenon are filter bubbles and misinformation. Filter bubbles are the result of excessive personalization which enhances relevance of content at the price of limiting exposure to a specific viewpoint. These bubbles are amplified by the so-called echo chambers that exist in social media, whereby members of a community mutually reinforce a fixed opinion or viewpoint on an issue. Misinformation as well as disinformation, on the other hand, tends to propagate through the network, often faster and more virally than truth.
Both problems manifest themselves in the form of groups of actors working in concert and providing mutual reinforcement. How can we recognize these groups? Having detected them, how can we counteract these problems? The first question can benefit from an examination of techniques developed to search for dense subgraphs in an underlying network. As for the second question, a natural approach for countering filter bubbles is to launch some kind of counter-campaign to balance users' exposure to viewpoints. Countermeasures for misinformation propagating through a network depend on the party planning the countermeasure. The network host can intervene and take steps to limit the propagation of misinformation, but these actions come with a cost. Besides the political sensitivity and cost of limiting freedom of expression, what if the intervention was by mistake done on genuine information? On the other hand, a third party interested in countering the propagation of misinformation may launch a counter-campaign. Some of the ideas behind designing such campaigns have strong connections to a classic problem called Influence Maximization, studied in a very different context, driven by different applications like viral marketing, infection containment, and revenue or welfare maximization. In this talk, we will examine research on detecting dense subgraphs as well as competitive influence maximization and discuss how that can inspire techniques for addressing the two problems above.
NG2S: A Study of Pro-Environmental Tipping Point via ABMsKan Yuenyong
A study of tipping point: much less is known about the most efficient ways to reach such transitions or how self-reinforcing systemic transformations might be instigated through policy. We employ an agent-based model to study the emergence of social tipping points through various feedback loops that have been previously identified to constitute an ecological approach to human behavior. Our model suggests that even a linear introduction of pro-environmental affordances (action opportunities) to a social system can have non-linear positive effects on the emergence of collective pro-environmental behavior patterns.
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Soc...Ruofei Du
Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.
Social media in virtual reality is in a high-growth market segment with influential products and services in virtual tourism, remote education, and business meetings. Nevertheless, previous systems have never achieved an online platform which renders a 6DoF mirrored world with geotagged social media in real time. In this paper, we introduce the technical detail behind Geollery.com which reconstructs a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local street views and depth maps. When viewed from afar, we apply projection mappings to 3D geometries extruded from building polygons for a coarse level of detail. In contrast to teleportation, our system allows users to virtually walk through the mirrored world at the street level. Our system integrates geotagged social media from both internal users and external sources such as Twitter, Yelp, and Flicker. We validate our real-time strategies of Geollery.com on various platforms including mobile phones, workstations, and head-mounted displays.
2019년 파이콘 한국에서 진행된 튜토리얼 자료입니다. 최재식 교수님께서 설명가능인공지능이란 무엇인가에 대해 발표해주신 Part 1 발표자료입니다. 아래 링크를 통해 행사 관련 정보를 확인하실 수 있습니다.
http://xai.unist.ac.kr/Tutorial/2018/
https://github.com/OpenXAIProject/PyConKorea2019-Tutorials
Part 1: https://www.slideshare.net/OpenXAI/2019-part-1
Part 2: https://www.slideshare.net/OpenXAI/2019-lrp-part-2
Part 3: https://www.slideshare.net/OpenXAI/2019-shap-part-3
Calit2: Experiments in Living in the Virtual/Physical WorldLarry Smarr
11.02.24
Pat Ledden Memorial Luncheon
Faculty Luncheon Seminar
Title: Calit2: Experiments in Living in the Virtual/Physical World
UC San Diego Faculty Club
How Successful Crowdsourcing Depends on asking 'Interesting Questions'Crowdsourcing Week
Writing Interesting Questions is as much art as as science. Here are some 100%Open has written recently. How can we double the fun of the LEGO play experience? How can I wash my home, myself, or my clothes with a single cup of water? (Unilever) How can we enable all Detroiters to travel more easily, safely and reliably? (Ford) How can we empower investors and their advisers to consider the CO2 impact of their investment decisions? (UBS) How can we help people do good by using their mobile phone in 3 minutes or less? (EE) Our Interesting Question methodology (https://www.100open.com/toolkit_2/interesting-question/) ensure that questions are accessible, contagious and as inspiring to the Challenge Holder organisation as they are to the Innovator target group.
Contestant Centered Design: creative approaches to designing competitionsCrowdsourcing Week
Creativity is critical to solving complex problems, developing new strategies, facilitating innovation, and driving organizational change. NIST’s Public Safety Communications Research Division’s open innovation efforts focuses on advancing wireless communications for America’s first responders by leveraging expertise and innovative solutions through crowdsourcing and collaboration. Success relies on creating competitions that achieve NIST’s organizational goals, incentivize world class science, remove barriers to entry, and maximize participation. Not an easy equation to balance. This session will discuss how design tradeoffs are considered for a variety of competition elements as concepts develop into a competition and as competitions are implemented. The goal of this interactive session is to provide a behind-the-scenes view of our process, engage audience ideas, and dive into a discussion about crowdsourcing and contestant-centered design.
Ethan will talk about the opportunity to reward crowdsourcing participants through crypto assets/tokens that allows the possibility of performing many micro transactions, saving costs for both the business and the users. In addition, the topic of transparency coming from the blockchain sector where business are now becoming more open to have the public help with tough R&D questions that in the past would have been kept internal. The blockchain industry is in fact growing communities as their branding strategy from the start, and rely on transparency for their community to trust them. All in all, we are seeing the tools in the making to ignite crowdsourcing’s future potential within decentralized business models. Lastly, we will dive into current use case studies from crowdholding.com, on creating a crowd rewarding mechanism for both crowd intelligence and crowd marketing.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
5. @infoxiao
COMBINING THE BEST OF BOTH WORLDS
5
Virtual Reality
(as of 2017)
Crowdsourcing
Crowdsourced VR
(this work)
Manipulation
Realism
Measurement
Granularity
Participant
Diversity
Reproducibility
6. @infoxiao
COMBINING THE BEST OF BOTH WORLDS
6
Virtual Reality
(as of 2017)
Crowdsourcing
Crowdsourced VR
(this work)
Manipulation
Realism
+ High - Low
Measurement
Granularity
+ High - Low
Participant
Diversity
- Low + High
Reproducibility - Low + High
7. @infoxiao
COMBINING THE BEST OF BOTH WORLDS
8
Virtual Reality
(as of 2017)
Crowdsourcing
Crowdsourced VR
(this work)
Manipulation
Realism
+ High - Low + High
Measurement
Granularity
+ High - Low + High
Participant
Diversity
- Low + High + High
Reproducibility - Low + High + High
9. @infoxiao
RESEARCH QUESTIONS
10
RQ1: Are there VR-eligible reachable workers?
RQ2: What is a good user flow?
RQ3: What types of experiment manipulations can we
deliver remotely?
RQ4: What are the limitations and challenges?
10. @infoxiao
CONTRIBUTIONS
11
1. Validated a VR-eligible panel of 242 workers
2. Implemented a user flow between desktop and VR
3. Replicated three previous studies with different
experiment manipulation remotely
4. Limitations and challenges
Source code and data are available at:
bit.ly/VRCrowdExperiments
11. @infoxiao
OUTLINE
12
1. Validated a VR-eligible panel of 242 workers
2. Implemented a user flow between desktop and VR
3. Replicated three previous studies with different
experiment manipulation remotely
4. Limitations and challenges
Source code and data are available at:
bit.ly/VRCrowdExperiments
12. @infoxiao
Please take a picture of your device with lasts 4 digits of your worker ID
handwritten on a piece of paper in view.
CONSTRUCTING VR-READY PANEL
13
14. @infoxiao
N = 242
(MORE) DIVERSE PARTICIPANTS
16
70%
14%
6%
10%
0% 25% 50% 75% 100%
White
Asian
Black
Other
61%
39%
0% 25% 50% 75% 100%
Male
Female*
52%
30%
18%
0% 25% 50% 75% 100%
Suburban
Urban
Rural
90%
10%
0% 25% 50% 75% 100%
U.S.
Other
14%
29%
29%
13%
-13% 10% 33%
High school or below
Some / 2-year college
Bachelor's
Master's and above
21%
57%
22%
0% 25% 50% 75%
Below $30K
$30K - $80K
Above $80K
Age: 18 - 78 (Median 32)
* One worker self-identified as “other”.
15. @infoxiao
OUTLINE
17
1. Validated a VR-eligible panel of 242 workers
2. Implemented a user flow between desktop and VR
3. Replicated three previous studies with different
experiment manipulation remotely
4. Limitations and challenges
Source code and data are available at:
bit.ly/VRCrowdExperiments
16. @infoxiao
DESIGN GOALS OF USER FLOW
18
Usable Direct workers to participate in VR and complete survey on desktop
Web-Based Allows for online data collection and thus remote participation
Low Technical Barrier Allows for easier replication and adaptation
18. @infoxiao
FLOW
20
Worker accepts task
through Amazon
Mechanical Turk
Worker completes
experiment in VR
VR Web App URL
Worker answers survey in
Qualtrics
Verification Code 1
Experimenter approves
payment Verification Code 2
19. @infoxiao
OUTLINE
21
1. Validated a VR-eligible panel of 242 workers
2. Implemented a user flow between desktop and VR
3. Replicated three previous studies with different
experiment manipulation remotely
4. Limitations and challenges
Source code and data are available at:
bit.ly/VRCrowdExperiments
21. @infoxiao
MODELS OF ILLUSIONS IN VR
23
Gonzalez-Franco, M., & Lanier, J. (2017). Model of Illusions and Virtual Reality.
Frontiers in psychology, 8, 1125.
Place Illusion A user’s feeling of being transported into the rendered environment
Embodiment Illusion A user’s feeling of experiencing the virtual world through an avatar
Plausibility Illusion A user’s feeling that events happening in the virtual world are real
22. @infoxiao
MODELS OF ILLUSIONS IN VR
24
Place Illusion A user’s feeling of being transported into the rendered environment
Embodiment Illusion A user’s feeling of experiencing the virtual world through an avatar
Plausibility Illusion A user’s feeling that events happening in the virtual world are real
Study 1: Restorative Effects of Virtual Environments [Valtchanov et al. 2010], N = 22 (original) v.s. 55 (ours)
Study 2: Proteus Effect [Yee and Bailenson, 2007], N = 50 (original) v.s. 59 (ours)
Study 3: Drawing Power of Crowds [Milgram et al. 1969], N = 1,424 (original) v.s. 56 (ours)
Gonzalez-Franco, M., & Lanier, J. (2017). Model of Illusions and Virtual Reality.
Frontiers in psychology, 8, 1125.
23. @infoxiao
MODELS OF ILLUSIONS IN VR
25
Place Illusion A user’s feeling of being transported into the rendered environment
Embodiment Illusion A user’s feeling of experiencing the virtual world through an avatar
Plausibility Illusion A user’s feeling that events happening in the virtual world are real
Gonzalez-Franco, M., & Lanier, J. (2017). Model of Illusions and Virtual Reality.
Frontiers in psychology, 8, 1125.
Study 1: Restorative Effects of Virtual Environments [Valtchanov et al. 2010], N = 22 (original) v.s. 55 (ours)
Study 2: Proteus Effect [Yee and Bailenson, 2007], N = 50 (original) v.s. 59 (ours)
Study 3: Drawing Power of Crowds [Milgram et al. 1969], N = 1,424 (original) v.s. 56 (ours)
25. @infoxiao
STUDY 2: PROTEUS EFFECT
27
Van Der Heide, Brandon, et al. "The Proteus effect in dyadic communication: Examining the effect of avatar appearance
in computer-mediated dyadic interaction." Communication Research 40.6 (2013): 838-860.
27. @infoxiao
STUDY 3: DRAWING POWER OF CROWDS
29
Milgram, S., Bickman, L., & Berkowitz, L. (1969). Note on the drawing power of crowds of different size. Journal of
personality and social psychology, 13(2), 79.
N = 1,424
28. @infoxiao
STUDY 3: DRAWING POWER OF CROWDS
30
Milgram, S., Bickman, L., & Berkowitz, L. (1969). Note on the drawing power of crowds of different size. Journal of
personality and social psychology, 13(2), 79.
N = 1,424
32. @infoxiao
GAZE* DISTRIBUTION
34Size of stimulus crowd
Zero (N = 15) Low (N = 15) Medium (N = 13) High (N = 13)
0% 100%
* Head rotation used as a proxy for gaze.
Participant
Male avatar
Female avatar
33. @infoxiao
GAZE* DISTRIBUTION
35Size of stimulus crowd
0% 100%
60%
8%
14% 18%
* Head rotation used as a proxy for gaze.
Zero (N = 15) Low (N = 15) Medium (N = 13) High (N = 13)
Participant
Male avatar
Female avatar
34. @infoxiao
GAZE* DISTRIBUTION
36Size of stimulus crowd
0% 100%
60%
8%
14% 18%
54%
12%
12% 22%
47%
13%
16% 24%
47%
14%
16% 23%
*** statistically significant
* Head rotation used as a proxy for gaze.
Zero (N = 15) Low (N = 15) Medium (N = 13) High (N = 13)
Participant
Male avatar
Female avatar
35. @infoxiao
RECAP
37
Place Illusion A user’s feeling of being transported into the rendered environment
Embodiment Illusion A user’s feeling of experiencing the virtual world through an avatar
Plausibility Illusion A user’s feeling that events happening in the virtual world are real
Study 1: Restorative Effects of Virtual Environments [Valtchanov et al. 2010], 22 (original) v.s. 55 (ours)
Study 2: Proteus Effect [Yee and Bailenson, 2007], 50 (original) v.s. 59 (ours)
Study 3: Drawing Power of Crowds [Milgram et al. 1969], 1,424 (original) v.s. 56 (ours)
Source code and data are available at:
bit.ly/VRCrowdExperiments
36. @infoxiao
REPRODUCIBILITY
38
• $8 per task (study participation)
• $1,374 for 201 participants (50%
reduction, estimating $3,600 for the
same in-lab studies)
• No equipment cost
Cheaper
• 201 study participation within a week
• Compared to three weeks estimation
(67% reduction, estimating 10 in-lab
studies per day)
• ~20 min per task
Faster
• Pure web-based
• JavaScript easy to learn
• Open sourced
Easier
Source code and data are available at:
bit.ly/VRCrowdExperiments
37. @infoxiao
CONTRIBUTIONS
39
1. Validated a VR-eligible panel of 242 workers
2. Implemented a user flow between desktop and VR
3. Replicated three previous studies with different
experiment manipulation remotely
4. Limitations and challenges
Source code and data are available at:
bit.ly/VRCrowdExperiments
38. @infoxiao
THE (LONG-DUE) PROMISE OF VIRTUAL REALITY
40
Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N. (2002). Immersive virtual
environment technology as a methodological tool for social psychology. Psychological Inquiry, 13(2), 103-124.
… replications, or at least near-perfect replications,
become quite possible.
”
“
… allow for more representative sampling. Whole
experiments can be carried out concurrently in
multiple laboratories via networked collaboratories.
... provide a compelling sense of personal, social, and
environmental presence for users, while allowing the
investigator near-perfect control over the experimental
environment and actions within it.
39. @infoxiao
LIMITATION & CHALLENGES
41
• Device battery exhaust
• Device overheating
• Limitation on measurement
• Switching between desktop and VR not smooth
Device ConstraintSize Constraint
• N = 242
144
46
18
18
0 40 80 120 160
Samsung Gear VR
Google Cardboard
HTC Vive
Sony Playstation
• 98% at home - safety is important
• 89% seated, 81% have space to walk around
Physical Environment
41. @infoxiao
THANK YOU
44
Mor Naaman[1,2]
[1] Social Technologies Lab, Cornell Tech
[2] Cornell University
Megan Cackett[2] Leslie Park[2]Xiao Ma[1,2] Eric Chien[1,2]
We thank the crowdworkers who participated in our studies, and Oculus for providing the equipment used for
developing the studies. We thank Dan Goldstein, Jake Hofman and Sid Suri for early feedback and direction.
Source code and data are available at:
bit.ly/VRCrowdExperiments
This work is partially supported by
Oath through the Connected
Experiences Lab at Cornell Tech.
Editor's Notes
Hi. I’m Xiao, 5th year PhD at Cornell Tech.
Happy to present this work, with my colleagues at Cornell, Megan, Leslie, Eric, and my advisor Mor Naaman.
I’m going to tell you a bit about Virtual Reality and Crowdsourcing.
This work is first published at The Web Conference 2018, WWW, one of the premier academic conference for the future direction of web technologies. You can find the full paper on my website, or a medium write-up on twitter.
Now just out of curiosity, by show of hands, how many of have experienced virtual reality?
As you probably experienced, or heard,
Virtual reality is an immersive technology that can generate highly realistic simulated experiences. }}
This paper is about combining the power of virtual reality, with that of crowdsourcing.
We believe that we have ran the world’s first three crowdsourced VR experiments.
In this talk, I am going to tell you why this is important, how we did it, and how you can run you own crowdsourced experiments going forward. }}
To understand why it is important to combine virtual reality with crowdsourcing, we draw this table of comparison for both areas of research, focusing on four things:
manipulation realism
measurement granularity,
participant diversity
and reproducibility }}
Virtual reality is strong in delivering highly realistic experience, making it suitable for strong experiment manipulation.
In addition, a lot of the VR devices are mobile, and the built in sensors in these devices provide high granularity for measuring things, such as head orientation.
But at the same time, current virtual reality research lacks participant diversity, relying heavily on college students as participants. Finally, it is very hard to replicate a VR study, making VR research low in reproducibility.
These last two weakness, participant diversity and reproducibility, are actually the very strengths of crowdsourcing — a vibrant area of research that many of you in the room have contributed to. }}
Another area of research is crowdsourcing — again very rich area for the past decade.
In particular, our work builds on Mason and Suri, 2012, [CLICK] who first articulated the benefits of conducting behavior experiments using crowdsourcing.
The virtual lab concept was inspired by a recent work by Mao et al, [CLICK], a longitudinal virtual synchronized experiment that studied trust. It is our hope that we begin to bring these two areas of research together through this work.
By making VR experiments crowdsourced, we can have the best of both worlds — meaning…
the ability to run reproducible online experiments, with high experimental realism and fine-grained measurement, combining desktop based surveys with mobile based interactions, as well as reaching more diverse participants.
This is why crowdsourced VR is important.
Now the next question is, [CLICK] how? }}
(… how.)
How do we combine VR and crowdsourcing to create an effective paradigm for conducting experiments on the web.
More concretely… }}
(More concretely…,) we ask 4 research questions.
First, are there workers out there with their own VR devices and are willing to participate in our experiments?
Second, once we find them, how do we direct them to do tasks in VR?
Third, what types of experiment manipulations can we deliver remotely in VR, without being in a lab and maybe not even being online at the same time with the participant?
And lastly, what are the limitations and challenges that we still face? We addressed these questions in our work. Here are our main contributions: }}
(here are our main contributions…)
First, we validated a VR-ready panel on Mechanical Turk, and we surveyed that population.
Second, we designed and implemented an effective user flow to direct workers to participate in VR and come back to desktop for exit surveys and payment.
Third, we replicated three previous studies in VR and without VR, each successfully delivering a different kind experiment manipulation.
Last but not least, we got a deeper understanding of the limitations and challenges of crowdsourced VR, and open source our data and code for easy replication and adaptation for you to run new experiments. Which we quite excited to see.
I will use this slide as the outline for the rest of my talk, and will go into each point in more detail. }}
So first, validating VR-ready panel.
To see if there are any crowdworkers available with their own VR devices and are willing to participate in our experiments, we designed a screening task and posted it through Amazon Mechanical Turk. [CLICK] }}
We invited crowdworkers who own VR devices from a list of approved device to participate in a survey.
In the survey, to prove that they have access to the VR device, we asked the worker to take a picture of the device, but together with a piece paper with their last four digits of Turker ID handwritten on it in view as well.
We accepted the qualifying ones; [CLICK]
and rejected the ones that appear to be scamming us. [CLICK]
As you can see, it’s not very difficult to tell them apart.. }}
We received two hundred forty two valid submissions in total.
In the same survey, we also asked the worker about their demographics, and we report it here. }}
I will give you a second to read the graphs. [PAUSE]
There are two key takeaways from this slide.
First, we observe that this population does not appear to differ significantly from the overall turker population tracked by literature over the years.
Second, this VR-panel is more diverse than the usual college student population, which is promising in addressing one of the key weakness of current VR research, participant diversity. }}
Next, now that we have a potential population of crowdworker participants that are relatively diverse, can we design a good user flow to direct them to complete VR tasks? }}
There are several design goals for this user flow:
First, it should usable at least.
Second, it needs to be web-based to allow for remote participation.
Third, ideally, the technical barrier for implementing such a user flow should be relatively low to make it easy replication and adaptation. }}
To achieve these design goals: we chose the following technical platform.
A JavaScript based UI framework for VR on the client side, and node.js for the server.
In other words, everything is in JavaScript and can easily be picked up and deployed by a web developer without special knowledge about game engines such as Unity, that is usually required to develop VR applications.
Again code and data are open sourced for replication and adapation. }}
Here is how the flow works. A worker accepts a task through Amazon Mechanical Turk, [CLICK]
It contains a URL for VR web app, and the worker joins the URL to complete VR part in their own headset. [CLICK] At the end of the VR experience, the participant receives the first verification code in VR, which unlocks a desktop-based survey. [CLICK]
After they finish the survey, worker receives a second verification code which unlocks the payment on Mechanical Turk. The task is then complete. }}
Now we have the participants. We also have the flow.
Let’s run some experiments. }}
(7:00min estimate)
When we were choosing experiments to run, we wanted to be very strategic about the first few experiments for this paper. First of all, we want to run replication studies rather than brand new experiments, because we want to have some baseline results to compare against than to create something in vacuum.
Second, there are thousands of VR studies out there. How do we choose smartly to maximize the demonstration power of these few studies. To do that, we relied on a framework of models of illusions in VR as a map. }}
This framework states that VR’s capacity can be broken down into three main types of illusions.
Place illusion, embodiment illusion, and plausibility illusion. The definitions of the illusion are here on the right if you want to take a second to read. [PAUSE]
Based on this framework of illusions, we chose three studies to replicate. Some in VR, some not, but one for each type of illusion. }}
For place illusion, we chose a study in 2010 on the restorative effects of virtual environment.
For embodiment illusion, we chose one of the most iconic and well-cited work in VR, the proteus effect, on how one’s virtual representation of self can influence their behavior.
For plausibility illusion, we chose a seminal study by Milgram et al. in 1969 on the study of crowds. }}
Due to time limitation, I will show you demo videos of all three studies, but only go in depths for results for Study 3. You can of course refer to our paper for details. }}
I am about to show you the VR portion of the first study, the restorative effects of virtual nature environment. In this study, we first show participants a thriller video to increase their psychological stress, and then play clips of nature to see if we decrease their negative affect, and increase positive affect.
Each video is 2-minute long, which I trimmed here, and also omitted the audio. [CLICK]
The dot here shows the center of participant gaze location. }}
Moving on to the second study, proteus effect. Again this is a very iconic and well-cited work in VR, investigating how virtualized self can impact one’s behavior.
In one of the experiments, the paper shows that the participant plays more dominantly in a negotiation game of splitting money when assigned to a taller avatar compared to a shorter one.
There are mixed results when other researcher tried to replicate the Proteus effect.
We did our own version, but did not observe an effect. Here is the video. }}
And then, finally, the Milgram study.
In 1969, Milgram and students conducted a study using pedestrians of New York as participants on the street.
They hired actors as stimulus crowd to stand in the middle of the sidewalk, and looked up to the sky. Then they measure the percentage of pedestrians who looked up, or stopped. }}
(Target time: 11min)
Here is a graph showing the percentage of people who looked up and stopped on the Y-axis, and the size of the stimulus crowd on the x-axis. As you can see, as the size of stimulus crowd increased, the percentage of passers bys who were influenced in their behavior also increased. More of them looked up and stopped. I’m going to show you how we replicated this study in VR, using avatars as stimulus crowd.
One caveat is that we had the stimulus crowd looking to the back, rather than looking up per feedback from early pilot user studies. }}
[CLICK] There are ten avatars, half male and half female who acted as the stimulus crowd.
Some of them looking to the back of the participant, which varies per condition.
The participant was instructed to freely explore the space to look for an object, which always appeared at the last 10 seconds of a 3-minute time limit window.
During the 3 minutes, we log the participants’ head rotation five times per second as our depend variable. }}
To simplify our analysis, we focus only on one axis of head orientation, [CLICK] which is yawning, looking left to right. Here are some notation for our illustration of results of this rotation.
You are looking from top down at the participant. [CLICK] The default field of view of a VR headset is 100 degrees, which is this zone to the front. [CLICK] Based on this, we divide the space into four zones. [CLICK]
[Gestures..] Zone 1 to the front, 2 to the left, 3 to the back, and 4 to the right. Later we will plot heat maps of gaze distribution based with these four zones.}}
We had a between subject design with four experiment conditions. Zero, low, medium and high. Indicating the number of avatars who face to the back of the participant.
For zero condition, [CLICK], all avatars are looking in the same direction as the participant. The arrow indicates the direction the avatar is facing.
For low condition, we had 2 randomly selected avatars [CLICK] facing to the back; 4 for medium condition [CLICK], and 8 for high condition [CLICK]. }}
We received 56 valid submissions in total, which was roughly distributed evenly across conditions.
Based on how we divided the space into four zones earlier,
now we can plot the head orientation distribution averaged across all participants in the same condition as a proxy for their gaze distribution. }}
Here is a gaze distribution heatmap for zero condition.
As you can see, participants spend most of the time looking to the front, and least time to the back, which is not surprising and nice validation. }}
For other conditions, we plot the same heatmap.
Here is the interesting part.
As the size of the stimulus crowd increases, participants spend less time to the front, and more exploring other areas. In fact, if we compare the percentage of time spent in zone 1, which is to the front, we found significant differences [CLICK] between Medium Zero, and High Zero condition. This replicates Milgram’s finding of the drawing power of crowds with a much small number of participants, as we were able to observe the participants in a much granular way. }}
So there you go, our three replication studies, done with crowdsourcing and remotely, for the first time, in VR.}}
(Target time: 15min)
In addition, cost analysis showed that we were able to drive VR experimentation to be 50% cheaper, 60% faster, and easier to implement.
We are also among the firsts to open source data and code for VR research, which further contributes to higher reproducibility.}}
Our work is not without limitations.
The main challenge we still face is participant size, which limits the number of experiments that we could run. You can refer to the paper for more discussion on limitations.
But for now, I want to close with a paper that served as the most direct inspiration for this work, 2002 Blascovich et al. }}
(Target time 16:00min).
(2002 Blascovich…,) Sixteen years ago, Blascovich et al. imagined a world where VR can be used to run replicable experiments with diverse participants. It is not until today that we finally made it work for the first time. }}
Nonetheless, our work still has key limitations.
Although we were able to get a few hundred participants, the scope of the experiments is still limited.
We also face issues with device performance such as battery exhaustion and overheating.
Finally, there are constraints with the physical environment people participate in, and need to consider important safety issues.
Some of these issues will be mitigated with newer devices, and we hope the research and industry communities continue to build on our work, and strive towards a shared vision of
Combing VR and crowdsourcing to run scalable, and reproducible VR studies.
And with that, }}
I conclude my talk, and thank my collaborators again as well as Oath for supporting this work.
Thank crowdsourcing week for having me and feel free to reach out.
Thank you very much for your attention. }}