This presentation introduces the most important quantitative research methods: questionnaires, biometrics and data analysis. It discusses several case studies in which these methods are employed.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
User Experience 6: Qualitative Methods, Playtesting and InterviewsMarc Miquel
This presentation introduces the most fundamental qualitative methods: the playtesting and the interview. It discusses when to use it and the possible bias the researcher may incur.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
GAMES USER RESEARCH: Guest Lecture in UX Design Class at Wilfried Laurier Uni...Lennart Nacke
In this talk, I describe several games user research methods from the Oxford University Press book: Games User Research. I talk about UX maturity levels of game development companies and the game design iterative development cycle and where Game UX fits into that space. I finally present several games user research methods.
Research Design constitute blue print for the collection, measurement and analysis of data.
Types of Research Designs and How to select good Research Design.
User Experience 6: Qualitative Methods, Playtesting and InterviewsMarc Miquel
This presentation introduces the most fundamental qualitative methods: the playtesting and the interview. It discusses when to use it and the possible bias the researcher may incur.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
GAMES USER RESEARCH: Guest Lecture in UX Design Class at Wilfried Laurier Uni...Lennart Nacke
In this talk, I describe several games user research methods from the Oxford University Press book: Games User Research. I talk about UX maturity levels of game development companies and the game design iterative development cycle and where Game UX fits into that space. I finally present several games user research methods.
Research Design constitute blue print for the collection, measurement and analysis of data.
Types of Research Designs and How to select good Research Design.
Research methods for engineering students (v.2020)Minh Pham
Beginning students who start doing research may face to many difficulties from choosing a good research topic to start, how to develop new ideas to how to implement models to test their ideas and write papers. Research skill is a craft skill. You only learn it by doing. However, it is good to learn know-how in doing research. In this lecture, I share information of how-to-do research for engineering students with the hope that it will help students to save time at the beginning state of doing research.
Experimental methods are widely used in industrial settings and research activities. In industrial settings, the main goal is to extract the maximum amount of unbiased information regarding the factors affecting production process form few observations, whereas in research, ANOVA techniques are used to reveal the reality. Drawing inferences from the experimental result is an important step in design process of product. Therefore, proper planning of experimentation is the precondition for accurate conclusion drawn from the experimental findings. Design of experiment is powerful statistical tool introduced by R.A. Fisher in England in the early 1920 to study the effect of different parameters affecting the mean and variance of a process performance characteristics
Taguchi's orthogonal arrays are highly fractional orthogonal designs. These designs can be used to estimate main effects using only a few experimental runs.
Consider the L4 array shown in the next Figure. The L4 array is denoted as L4(2^3).
L4 means the array requires 4 runs. 2^3 indicates that the design estimates up to three main effects at 2 levels each. The L4 array can be used to estimate three main effects using four runs provided that the twthree-factoro factor and three factor interactions can be ignored.
This presentation is an introduction and all about the Positivist and Post-positivist perspective in Educational Research and how these perspectives link to Quantitative Research. Determining a personal research perspective is an important move before deciding on writing the first chapters of a study.
Research Design: Quantitative, Qualitative and Mixed Methods DesignThiyagu K
A Research Design is simply a structural framework of various research methods as well as techniques that are utilized by a researcher. This presentation slides explain the resign design of quantitative, qualitative, and mixed-method design.
User Experience 5: User Centered Design and User ResearchMarc Miquel
This presentation introduces the user-centered design paradigm and the field of game user research. It includes some hypothetical case studies which are later discussed in the following presentations.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
Research methods for engineering students (v.2020)Minh Pham
Beginning students who start doing research may face to many difficulties from choosing a good research topic to start, how to develop new ideas to how to implement models to test their ideas and write papers. Research skill is a craft skill. You only learn it by doing. However, it is good to learn know-how in doing research. In this lecture, I share information of how-to-do research for engineering students with the hope that it will help students to save time at the beginning state of doing research.
Experimental methods are widely used in industrial settings and research activities. In industrial settings, the main goal is to extract the maximum amount of unbiased information regarding the factors affecting production process form few observations, whereas in research, ANOVA techniques are used to reveal the reality. Drawing inferences from the experimental result is an important step in design process of product. Therefore, proper planning of experimentation is the precondition for accurate conclusion drawn from the experimental findings. Design of experiment is powerful statistical tool introduced by R.A. Fisher in England in the early 1920 to study the effect of different parameters affecting the mean and variance of a process performance characteristics
Taguchi's orthogonal arrays are highly fractional orthogonal designs. These designs can be used to estimate main effects using only a few experimental runs.
Consider the L4 array shown in the next Figure. The L4 array is denoted as L4(2^3).
L4 means the array requires 4 runs. 2^3 indicates that the design estimates up to three main effects at 2 levels each. The L4 array can be used to estimate three main effects using four runs provided that the twthree-factoro factor and three factor interactions can be ignored.
This presentation is an introduction and all about the Positivist and Post-positivist perspective in Educational Research and how these perspectives link to Quantitative Research. Determining a personal research perspective is an important move before deciding on writing the first chapters of a study.
Research Design: Quantitative, Qualitative and Mixed Methods DesignThiyagu K
A Research Design is simply a structural framework of various research methods as well as techniques that are utilized by a researcher. This presentation slides explain the resign design of quantitative, qualitative, and mixed-method design.
User Experience 5: User Centered Design and User ResearchMarc Miquel
This presentation introduces the user-centered design paradigm and the field of game user research. It includes some hypothetical case studies which are later discussed in the following presentations.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
A framework for designing small bite-sized educational games for use in learning institution for more effective learning through Experiential Learning.
Intro to Games User Research Methods - March 2013Ben Lewis-Evans
An update to my Introduction to Games User Research lecture (http://www.slideshare.net/Gortag/an-introduction-to-games-user-research-methods). Due to a changing course design this version focuses a bit more on questionnaire design and interviews. A few other changes have been made and the aesthetics have also been changed.
Overview of an idea for a science puzzle game. The project is currently undergoing development, and will be presented in Summer 2010 for my Learning, Design and Technology Master's project.
Game features of cognitive training (Michael P. Craven and Carlo Fabricatore)
Interactive Technologies and Games (ITAG) Conference 2016
Health, Disability and EducationDates: Wednesday 26 October 2016 - Thursday 27 October 2016 Location: The Council House, NG1 2DT
A Primer On Play: How to use Games for Learning and ResultsSharon Boller
Discover the power games have to produce learning and business results. View the latest research and case studies on game-based learning and gamification. See a demo of Knowledge Guru, a game engine your team can use to quickly build your own games.
Primer on Play: Case Study for Knowledge GuruMarlo Gorelick
As shared in #GE4L, great structure of how and why to create game based learning. Prime case study to use when discussing possibilities of gamification for business
Figuring out the right metrics for your gameSaurav Sahu
This is a talk I gave at IGDA Conference 'Industry Speaks' on 1st April'17. I talked about how one should go about thinking the metrics to track in their games. Also, stressed on the fact that Analytics should not be an after-thought but should be squeezed in during the game production phase itself.
The slide discusses Google's HEART framework and Pirate Metrics while sharing an approach Goals/Signals/Metrics to make it easy to list down metrics once you have your goals.
The latter part of the slides talks about the generic biases one should be aware of.
Feel free to reach out incase of any query.
User Experience 8: Business, Ethics and MoreMarc Miquel
This presentation introduces the topic of ethics in video games from the user experience perspective. The implications of a f2p monetizations are enumerated. Dark User experience is defined in relation to the company and the user. Its main examples (Dark Patterns) are ellaborated for both websites and video games. Finally, a clear case study of Dark UX in video gambling is developed.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
User Experience 4: Usable User InterfaceMarc Miquel
This presentation introduces several interfaces and discusses what their usability and user experience depend on.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
User Experience 3: User Experience, Usability and AccessibilityMarc Miquel
This presentation introduces the most important usability models among other concepts (affordances, heuristics, etc.).
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
This is an introduction to the most important psychology concepts from the perspective of UX and their application to video games and software.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
User Experience 1: What is User Experience?Marc Miquel
This is an introduction to this course on User Experience in video games and web.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
Quality Assurance 2: Searching for BugsMarc Miquel
In this presentation we introduce the most useful testing techniques in order to find bugs (ad hoc testing, combinatorial testing, test flow diagram, cleanroom testing and testing trees).
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
In this presentation we introduce the concept quality assurance in video games along with the most important concepts, team members and testing phases.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
In this presentation we introduce the game balance "interesting strategies". It is especially important as games with a single dominant strategy are boring. No strategy must be much better than others and without drawbacks.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
Game Balance 3: Player Equality and FairnessMarc Miquel
In this presentation we introduce the game balance type "player equality and fairness". It is essential so the players do not feel the game is unworthy of playing. All the players must feel they are given the chances to win.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
In this presentation we introduce the game balance type 'sustained uncertainty'. Uncertainty is usually understood as related to randomness and difficulty. It is essential to keep the game interesting to the user.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
In this presentation we introduce the concept game balance, its different types, and the most useful methods to study it.
These slides were prepared by Dr. Marc Miquel. All the materials used in them are referenced to their authors.
public presentation of "Calçotada Wars" the card gameMarc Miquel
This is a presentation I gave in FNAC Plaça Catalunya in order to explain and show "Calçotada Wars" (the card game) for promotional purposes.
For more info about the project, check out marcmiquel.com
public presentation of "La Puta i la Ramoneta" the card gameMarc Miquel
This is a presentation I gave in Ateneu Igualadí in order to explain and show "La Puta i la Ramoneta" (the card game) for promotional purposes.
For more info about the project, check out marcmiquel.com
Towards a User-Centered Wikipedia - Viquitrobada, 26 de novembre 2016, ValènciaMarc Miquel
En aquesta presentació faig dues observacions; la primera sobre com s'ha construït Viquipèdia, quins són els seus valors relacionats amb la cultura hacker i com poden obstruïr el disseny centrat en l'usuari; la segona sobre com pel viquipedista és fonamental desenvolupar una identitat de comunitat i com s'ha d'ajudar als nouvinguts a crear-la. Per altra banda i vinculat amb les observacions, faig dues propostes per centrar el disseny i la cultura de Viquipèdia en els editors per tal de millorar l'engagement (participació).
Cultural Identities in Wikipedia (Wikimania 2016)Marc Miquel
Unlike in most social network platforms, in Wikipedia editors are not encouraged to disclose personal traits, hobbies or affiliations. In fact, I think the identity issue has not been discussed enough. Since the project is dedicated to promote a common good, there is no content ownership, and the personal aspects become uncomfortable, or partly taboo. However, I defend that identity matters, in terms of building a Wikipedian reputation, and that editors' identities are tightly related to the content. As a Wikipedian, would you contribute equally if you couldn't choose the topics?
In this presentation I want to address the creation process and composition of Wikipedia language editions as a matter of identity. Our research on the issue has shown us that an identity-based motivation allows editors to conciliate the Wikipedian identity in the community along with their other identities. Therefore, in order to act congruently with each of such identities, they contribute with content related to them. To assess the influence of this motivation type, we developed a method and identified articles related to each Wikipedia language edition's Cultural Identities. The results on 40 Wikipedias show that this kind of content represents almost a quarter of each language edition. We analyze the content in terms of topical coverage and find that different specific topics emerge as important for each of them, although the most important topics are generally Geography, People and Culture. Inspecting how articles related to each language edition's cultural identities are exported to other languages, we show relationships between Wikipedias.
The selection of articles reflecting each Wikipedia language based cultural identities is a rich source for research, but can be also a useful base to establish an intercultural exchange between Wikipedia language editions. We propose the diversity of content across languages to be seen as an asset, and the spread of content specific to a language edition to be facilitated by automatic tools. The main point is to recognize the power of identity as a motivator for action and as a driver for change. Finally, we present a project called Wikiidentities in which we will disseminate the results of the research, make the datasets available, and provide some ideas and debate on how identities can be key to bridge the culture gap in any Wikipedia.
Happiness Has To Do With Clarity - World Information Architecture Day '15Marc Miquel
Most of the times we hear design for engagement or for better user experiences. Why don’t we design for happiness? Who is interested in happy users? I will give various examples of games and websites whose success depends on many things but joy and pleasure. Probably the key is in their information architecture and consequently in their interaction design. We as designers have an enormous responsibility for users’ behaviours. How much aware are we of our designs implications? And how much are the users?
To me, happiness in UX is the absence of frustration. Let's fight 'dark patterns' to make a more free Internet.
If you want to learn about Dark Patterns: www.darkpatterns.org
The Elements of Videogambling ExperienceMarc Miquel
For more information: http://uxmag.com/articles/dark-ux-the-elements-of-the-video-gambling-experience
This is a presentation I gave in La-Salle University (Barcelona) on April 12th about Videogambling Design and deceptive user experience. I include some of the most used dark patterns in the business and the tricks companies use to keep gamblers playing for longer sessions.
Its material is complementary to the deceptive UI designs in www.darkpatterns.org.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
20 Comprehensive Checklist of Designing and Developing a WebsitePixlogix Infotech
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data Analysis
1. Unit 7: Quantitative Methods:
Questionnaires, Biometrics and Data Analysis
Second term, January 2019 Dr. Marc Miquel Ribé
Course in User Experience
Bachelor Degree in Video Game Design and Production
Computer Engineering for Information System Management
2. UX
Our aim is to understand the player experience.
There are four key aspects to understanding
players and gameplay:
• User Experience: How does this make her feel
now?
• Physiological reactions: How does this make
her react?
• Recalled Experience: What does she think it
happened?
• User behaviour / interactions: What is she
doing?
Physiological reactions
User behaviour / interactions
Recalled UX (later)
Now
Quick Reminder. What are we studying in Games User Research?
We need to decide the method according to what we are studying. For each method, we have to
be very careful. Answering these two questions can be useful:
• Where and when can we bias or spoil the data?
• What are the limits of these data to reach conclusions?
Remember, we cannot know about their behaviour by just asking, or we cannot know about
the experience by just analysing behavioural data.
3. Our aim is to understand the player experience.
There are four key aspects to understanding
players and gameplay:
• User Experience: How does this make her feel
now?
• Physiological reactions: How does this make
her react?
• Recalled Experience: What does she think it
happened?
• User behaviour / interactions: What is she
doing?
Quick Reminder. What are we studying in Games User Research?
We need to decide the method according to what we are studying. For each method, we have to
be very careful. Answering these two questions can be useful:
• Where and when can we bias or spoil the data?
• What are the limits of these data to reach conclusions?
Remember, we cannot know about their behaviour by just asking, or we cannot know about
the experience by just analysing behavioural data.
UX
Physiological reactions
User behaviour / interactions
Recalled UX (later)
Now
4. Goal of the Unit
Learn when and how to use the methods aimed at quantifying
data from a group of users in order to draw general
conclusions.
5. Overview of the Lesson
7.1 Questionnaires
7.2 Biometric Methods
7.3 (Behaviour) Data Analysis
Goal of the Lesson: Learn when and how to use the methods aimed at obtaining
user’s quantitative data.
6. 5.4 Questionnaires
Interviews and focus groups may be subject to some specific kind of bias introduced by the
interviewer and the group dynamics. To put it simply: questionnaires are a list of questions in
regards of your game. They have with different sorts of limitations but without the previous
interfering biases.
Questionnaires are also called ‘self-reported methods’ because it is the participant who
provides the data with no need of a moderator.
The questionnaire method is a research method, a tool, used to collect information from a
representative sample of a population (we aim at the target more than with an interview or
playtesting).
Differently than the interview, the survey is very good at extracting general conclusions.
Also, it is a great method for when we know what we want to know.
7. UX
Physiological reactions
User behaviour / interactions
Recalled UX (later)a) What are we studying with a questionnaire?
In a questionnaire, we only study the recalled
experience – What does she think it happened?
The recalled UX are the emotions, motivations,
beliefs, perceived usability, perceived difficulty and
mechanics, perceived fairness, among others.
5.4.1 Purpose and problems (what)
We cannot know about the real gameplay (game functioning and usability) with just the survey.
We cannot know about the user behaviour and interactions – intensity or repetition.
This is why sometimes questionnaire are conducted after the playtesting – as a complementary
but not essential or as ‘core’ as playtesting or data analytics.
We use questionnaire when we know quite clearly what we want to know.
8. a) What are we studying with a questionnaire?
In a questionnaire, we only study the recalled
experience – What does she think it happened?
The recalled UX are the emotions, motivations,
beliefs, perceived usability, perceived difficulty and
mechanics, perceived fairness, among others.
5.4.1 Purpose and problems (what)
We cannot know about the real gameplay (game functioning and usability) with just the survey.
We cannot know about the user behaviour and interactions – intensity or repetition.
This is why sometimes questionnaire are conducted after the playtesting – as a complementary
but not essential or as ‘core’ as playtesting or data analytics.
We use questionnaire when we know quite clearly what we want to know.
UX
Physiological reactions
User behaviour / interactions
Recalled UX (later)
Now
9. b) What kind of method it is Mainly quantitative and attitudinal
Datasource
eye tracking
data analysis
interviews & focus groups surveys
physiological data
Questionnaire gives valuable data sorted in different questions that can be closed-questions
(only few answers are available) or opened-questions (then it would also be qualitative).
playtesting/usability
playtesting/usability
think-aloud
10. c) Experiment structure
Purpose: A survey or questionnaire are conclusive for games.
What to ask:
Specific concept or aspect in particular with a theoretical construct. The main purpose of a
survey is to obtain useful data about a particular aspect.
The main topics that a questionnaires can be designed for can be: emotional aspects,
preferences, motivation, comprehension (cognitive), game aspects (balance, narrative, goal
clarity, etcetera.). With a questionnaire we are not going to validate behaviour or ask for
particular choices.
Research process
e.g. Motivation > Self-Determination Theory > Dimensions (Intrinsic, Extrinsic; Needs:
competence, relational, autonomy) > Indicators (Intrinsic = no rewards) > Hypothesis (the game
is dominated by intrinsic) > Method (Questionnaire) > Questions (‘You had the most fun while
maneuvering the car’> Results > Conclusions (the game is dominated by intrinsic but…).
Research Question > Theoretical Framework > Survey > Data Analysis with Statistics > Conclusions
11. c) Experiment structure
Purpose: A survey or questionnaire can be explorative to understand the context of a user when
we are designing a new service.
What to ask:
Surveys are useful to identify:
Who your users are; What your users want; What they purchase; Where they shop;
What they own; and What they think of your brand or product.
Research process
e.g. Motivation > Self-Determination Theory > Dimensions (Intrinsic, Extrinsic; Needs:
competence, relational, autonomy) > Indicators (Intrinsic = no rewards) > Hypothesis (the game
is dominated by intrinsic) > Method (Questionnaire) > Questions (‘You had the most fun while
maneuvering the car’> Results > Conclusions (the game is dominated by intrinsic but…).
[https://uxmastery.com/better-user-research-through-surveys/]
12. Necessarily, the target player. It makes no sense to propose a survey to a non-target player.
We need to extract conclusions from a sample population from the target player.
Since the method is no qualitative, introducing noise in the sample would be devastating.
The sample size will be determined by the final number of players and the confidence interval
and level we want to obtain. The bigger the sample, the less error.
5.4.2 Production pipeline (when) Here! Here!
Some questionnaires are handed after the
playtesting session and are directly related to the
recent experience. Surveys are not usually for
product release.
Other questionnaires are more about the general
use of the game. It is usual to use questionnaires
after the game release in order to know better the
target player and understand the game success or
failure.
5.4.3 Participants (who)
13. 5.4.4 Question formulation (how)
• Each question must be mapped or related to a particular concept or problem you are
trying to understand (research question).
Let us say we want to study emotional attachment or motivation (again).
Validity: A questionnaire experiment is valid if it measures what it claims to measure. For
example, a test measuring the emotional attachment to a set of characters should not ask
something else such as the UI/UX design. Another way to think about it is if I’m trying to
measure a player’s attitude of a game’s level difficulty, am I asking the correct questions to be
able to gauge that?
In order to be sure that we are asking questions about the concepts or topics we are thinking
about, we should give the questionnaire to someone to try to identify the concepts and check
the agreement.
The number of questions will determine the time to answer (it should not exceed 15-20 min.).
[http://www.gamasutra.com/view/feature/169069/finding_out_what_they_think_a_.php?print=1]
[http://www.gamasutra.com/blogs/NeferDoane/20150316/238780/How_to_Design_a_Survey_for_User_Feedba
ck.php?print=1]
14. General writing advices
• Use clear, concise everyday, simple language. Avoid jargon, technical terms, and acronyms.
• Refrain from using double negatives in a survey.
• Avoid double-barreled questions. A double-barreled question is a single question that asks
about more than one issue but is limited to one answer. “I found this game to be challenging
and fun.” The matters of challenge and fun are two separate issues.
• Beware of leading or loaded questions. We don’t want participants to say what we want
them to say, we want their own opinions.
15. Generally, the most common type of close-ended survey questions is the single-answer
multiple-choice question.
In order for close-ended questions to be comfortably answerable by the respondents, the
answers should be specific, exhaustive, and mutually exclusive.
You can apply this to ask their favorite ‘weapons’, their enemies, their avatar purchases, among
others. In a questionnaire questions must be more precise than in an interview.
For instance: if you are asking how much they spend playing. The answers should not be some,
a lot, every day. They should be ‘none’, ‘0 to 5 hours’, ‘6 to 10 hours’, ‘11 to 20 hours’, ‘more
than 20 hours’.
5.4.5 Question formats (how)
16. • Likert Scale: this method was
designed to 1) measure attitudes or
opinions and 2) the intensity of these
attitudes or opinions in 3) a response
format of fixed choice.
Control
Avoid asking duplicate questions in
different way (unless it is for control). For
example, using reverse scores. This asks
to rate the direction (strength of feeling)
of the statement e.g. positive to negative.
Generally, avoid questions that are
phrased negatively. But, in small doses,
counterbalance including reversed score
items. This ensures that the participant is
consistent with his ideas.
Some are five-point and some are seven-point.
Each answer in a scale has a value (1-5)
Example of Reverse Scores:
This game was fun to play.
I would recommend this game to friends.
This game was boring.
Using scales:
17. I was made a survey on satisfaction after a usability testing and made several mistakes. All
the answers were very positive (5/5).
What is satisfaction? “The state of satisfaction may include a variety of emotions and …
their intensity may vary according to how much you care….”
Mistakes I made:
• Be aware of biases. Their relationship with the facilitator/company.
• If you ask about satisfaction, never use 5 scale as it has not resolution enough.
Advice:
Comparing it to other experiences may be very helpful. Otherwise, what does 7 mean?
How was your experience with our company?
Using scales to understand user satisfaction with a product
18. • Use balanced ratings scales. Use an equal number of positive and negative options—this
relates to probability. With 4 options the natural spread would be 25% per answer, therefore
if we have more positive options than negative we would increase the chances of getting
positive feedback.
Using balanced rating scales:
With a balanced rating scale there is a greater chance of the results reflecting a participant’s true
beliefs.
19. If the questionnaire is aiming at understanding a very specific thing and not motivation or
emotion, you can use game elements or scenes to help ensure the player knows what you mean.
Be aware of the biases you introduce (for instance, showing just specific images and then not
others may influence the answers in further unrelated questions).
There are scales up to 10,
sometimes more distance is
not an advantage.
In order for the
questionnaire to be reliable
(this means that all the
questions represent well the
concepts). You need to keep
the same scale (either with
5, 7 or 10 levels) for the
entire questionnaire.
20. Order matters. Previous questions can influence further questions, especially when they are
long. You have two options. In case you are asking about a playtesting session, you can segment
the questions and introduce them like a story (beginning, development, ending), so that each
question helps remembering the recent experience. In case you are asking about a general
experience about the game (for instance, we again want to know what motivates the player),
we are going to provide very different sort of questions and shuffle them.
Anonymity. Some personalities avoid ’extremes’. Allowing anonymous answers may help.
21. 5.4.6 Pros / Cons
Pros
• Cost-efficiency: Ease of gathering large amounts of data, practically no constraints on
geographical location and demographics, and quick results — in other words… time and
money. While a survey’s construction can take considerable effort, the player’s role in filling it
out usually takes a short amount of time.
• Consistent and reliable. You know that participants are answering to the same questions and
you can work out this data together.
• Quantifiable. You can compare different concepts and see their magnitude much more easily
(in comparison with interview) and in a large scale.
Cons.
• Individuals might be unfamiliar with or have difficulty remembering relevant information. An
additional reason for this is that individuals may feel societal pressure to respond in a manner
that will be viewed favorably by others, a social science term called social desirability bias.
• Another challenge when using survey design is that surveys have inflexible design (you can’t
change anything about the survey once you’ve started to administer it). The reasons for this
inflexibility is to maintain the validity of collected data. (for instance, it is no possible to add
follow up questions). Besides, it takes some time to create them in a proper way.
• Need a large(ish) sample to be sure that the data is representative.
• It reflects what users say instead of what users do (typical limits of subjective data).
[http://www.gamasutra.com/blogs/NeferDoane/20150316/238780/How_to_Design_a_Survey_for_User_Feed
back.php]
22. Case Study: Pokémon Go players’ motivation
1. Research Question: What types of motivation exist while playing Pokémon Go?
2. Concepts/Theoretical Framework: Self-determination theory, identity-based motivation, etc.
3. Methodology: Surveys (e.g. PENS), Interview, etc.
23. Case Study:
Research Question: What features would users want from an alarm clock app for the
metro?
Context: Alarm clock App.
24. 5.4.7 Standardized Questionnaires
Some games user researchers create questionnaires to understand specific concepts (motivation,
engagement, etc.) and publish them so they can be used with very different players and games.
• Scott Rigby and Richard Ryan created Player Experience of Need Satisfaction (PENS), an
standardized questionnaire to examine players motivation. They also created a company
(Immersyve) to test their model with video games and give consultancy to developers.
• Their research shows that this underlying motivational energy takes the form of three basic
psychological needs: Those of competence, autonomy, and relatedness.
You can measure for these motivators in a playtest session by having players complete short
questionnaires immediately following important events in the game, such as a difficult puzzle or
a pitched battle. For example, you might ask players to rate the extent to which they agree or
disagree with these statements:
“The game kept me on my toes but did not overwhelm me.” (to measure competence)
“I felt controlled and pressured to act a certain way.” (to measure autonomy)
“I formed meaningful connections with other people.” (to measure relatedness)
With PENS, you can see the relationship between: 1) the player’s psychological
needs triggered by your game, 2) specific outcomes (e.g. Fun/enjoyment; will
buy more of developer’s game; will recommend to others).
25. Rigby and Ryan have demonstrated
there exist a relationship between
among the three measures of
motivation (autonomy, relational and
competence) and the player
outcomes (in enjoyment,
recommending the game….) for
specific genres of games.
For example, the PENS measures
were found to be very accurate at
predicting the likelihood that players
of an adventure or role-playing game
would purchase more games from
the same developer.
[http://www.gamasutra.com/view/feature/130155/rethinking_carrots_a_new_method_.php?print=1]
[http://immersyve.com/white-paper-the-player-experience-of-need-satisfaction-pens-2007/]
Playful Design. John Ferrara. 2012. Chapter 8. Playtesting. Evaluating Motivation: The PENS Moodel. (p. 108)
26. • The development of the Game Engagement Questionnaire: A measure of engagement in
video game-playing Jeanne H. Brockmyer, Christine M. Fox, Kathleen A. Curtiss, Evan
McBroom, Kimberly M. Burkhart, Jacquelyn N. Pidruzny.
• Game Experience Questionnaire (GEQ), IJsselsteijn, W.A.; de Kort, Y.A.W.; Poels, K.
Once a questionnaire is published and other scientific peers find it useful, they start using it for
their research. For instance, Chek Tien Tan et al. (2014) use GEQ to understand the relationship
between Facial Expressions and game experience. (Correlation between Facial Expressions and
the Game Experience Questionnaire).
Other questionnaires are more focused in understanding specific emotions.
IMPORTANT! Standardized and published questionnaire are often easy to find and
you can use them to obtain good feedback. You can be certain the questions and all
the details are designed in the most rigorous way.
27. 6.1 Biometrics Methods
Biometrics is the technical term for body measurements and calculations. Biometrics methods
are those aimed at obtaining these measurements.
They may remind of you a ‘lie detector’. When utilizing lie detector machines, the body of a test
subject is hooked to several sensors capable of recording changes in a range of physiological
processes such as her heartbeat, the electrical conductivity of her skin, the frequency of her
respiration, and so forth.
[http://www.gamasutra.com/view/feature/183887/the_case_for_casual_biometrics.php?print=1]
28. a) What are we studying with psycho-physiological
methods?
Physiological reactions are good proxies for certain
aspects of the UX. We can really relate body
reactions to emotions and psychological states more
generally.
Through psycho-physiological methods we can detect
emotions (among many other UX aspects) that the
player is not aware of. They are produced
involuntarily.
It is not what they say about what they feel, it is what
they feel directly. Their measurement is independent
from the user talking.
6.1.1 Purposes and problems (what)
UX
Physiological reactions
User behaviour / interactions
Recalled UX (later)
They are based on the fact that the body gives clues on cognition and emotion.
29. a) What are we studying with psycho-physiological
methods?
Physiological reactions are good proxies for certain
aspects of the UX. We can really relate body
reactions to emotions and psychological states more
generally.
Through psycho-physiological methods we can detect
emotions (among many other UX aspects) that the
player is not aware of. They are produced
involuntarily.
It is not what they say about what they feel, it is what
they feel directly. Their measurement is independent
from the user talking.
6.1.1 Purposes and problems (what)
They are based on the fact that the body gives clues on cognition and emotion.
UX
User behaviour / interactions
Recalled UX (later)
Now
Physiological reactions
30. b) What kind of method it is Quantitative and behavioural
Datasource
playtesting/usability eye tracking
data analysis
Interviews & focus groups surveys
physiological data
playtesting/usability
think-aloud
They answer well ‘what’ and ’how much' questions (but not why or how)
31. Physiological methods need to be used with a theory-driven approach. The experiment must be
designed very carefully.
1. Context/Object
2. Problem/Research Question/Hypothesis
3. Concept / Theoretical Framework
4. Methods (Methodology)
5. Measurements and test design (Methodology)
6. Results analysis
7. Conclusions
8. Reporting
c) Experiment structure
Important:
• We select the concept in relation to the question.
• We select the method that can provide data useful to answer the question.
• We study the specific measures and how to interpret the data to answer the question.
32. Research Questions
Employing biometric experiments and methodologies to analyze a video game, you can obtain
answers to several questions related to the user experience that are crucial for its development
and commercial success. Examples of such questions include:
• Is the initial speed of our video game too stressing for our target audience?
• Did we reach a climax in emotional involvement where and when intended (this is likely to
be at the end of our free demo)?
• Does the tutorial of our video game succeed in keeping our players engaged while
empowering them to perform well?
• What are the emotions our target audience respond with to the experience of their first
Game Over?
• Is our game too stressful in a cognitive sense for them? Is it perceived as too punishing?
While asking these questions we should have clear the concepts and theoretical framework.
The next steps are choosing the method (6.1.2), studying the measurements and designing the
test (6.1.3), and finally, analyzing the data (6.1.4).
33. 6.1.2 Methods: types of sensor (what)
• Electroencephalography (EEG): electrical brain activity measured with electrodes.
• Electromyography (EMG): recording electric potential generated by muscles. Measure of
valence (positive/negative emotion).
• Galvanic skin response (GSR): also electrodermal response (EDR) or activity (EDA),
psychogalvanic reflex (PGR), skin conductance response (SCR). Electrical resistance of the
skin. Measure of arousal (high/low).
• Electrocardiography (EKG): recording the electrical impulses and heart rate variability (HRV).
• Eye Tracking (ET): recording either the point of gaze (where one is looking) or the motion of
an eye relative to the head.
Chapter 14. Physiological Measures for Game Evaluation. Regan Mandryk. Game Usability.
Katherine Isbister and Noah Schaffer. (p. 207)
34. Choosing appropriate sensors depends on what concept you want to study, and what your
setting is like. Use the following as a guideline:
• Mental effort: depending on your setting, decreasing heart rate variability (HRV) or greater
pupil dilation can be used to measure increases in mental effort. Increases in jaw clenching
(through EMG sensors on the face) or brow-raising (EMG of the forehead) may also be
indicative of increased mental effort. Increased respiration rate and a decrease in the
variability of respiration rate are also associated with mental effort. Electrodes (EEG) can
also measure certain physiological aspects of mental activity.
• Positive versus negative emotions: The valence of an emotion (whether it is positive or
negative) can be measured through facial muscle analysis (EMG) over the brow (frowning)
and cheek (smiling). Some potential has been shown in the use of heart rate, irregularity of
respiration, and pupil diameter as indicators of valence.
• Arousal: increases in psychological arousal are best measured by increases in galvanic skin
response (GSR), but can also be seen in increased respiration, decreased blood volume
pulse (BVP), and increased heart rate (HR).
• Perception and attention: the time it takes to look at an element or the time spent looking
at particular elements can be indicators of perception and attention. We can measure them
with an eye tracking (ET).
35. Test design. Obtain the right data:
How can we obtain data that allows us to have reliable conclusions.
Once you know what you want to measure (e.g. emotion) and what sensors to use, you need to
design the test. The game user researcher usually follows an experimental psychology approach
to design the research, that is characterized by 4 features:
1. Comparing controlled conditions (always test an scenario without ’that feature’).
2. Comparing controlled participant sample (always test participants with other scenarios).
3. Representative results (right tests for measures: number of participants).
4. Counterbalanced design to remove order effects (change the order no to accumulate
undesired effects from previous interactions).
Physiological data is sensitive, variable and difficult to interpret without a high level of
experimental control.
Therefore, we are at risk to oversimplify some interpretations of physiological data and do not
keep in mind its one-to-many relationship to psychological effects.
6.1.3 Measurements: data and experiment design (how)
36. Test design. Some aspects to take into account:
• Sensitivity to movement: Blood volume pulse (BVP), respiration via stretch sensor, and
galvanic skin response can be sensitive to movement.
• Sensitivity to physical activity: Most physiological measures are sensitive to fluctuations in
physical activity. Be aware of this when testing users.
• Individual differences: Most physiological measures show large differences between users
and between the same user on different days or at different times of day. Use normalization
procedures to correct for fluctuations.
Physiological measures are not easy to conduct and data can be easily biased.
We need to correct deviations by applying some procedures.
37. What are the specific measures for each method?
Electroencephalography (EEG) measures
In terms of these frequencies, the bands of interest are usually the:
• Alpha band (8-14 hz) that reflects calm, mental work.
• Beta band (14-30 hz) that reflects focused, engaged mental work.
• Delta band (1-4 hz) that reflects sleep, relaxation and fatigue.
• Theta band (4-8 hz) that reflects emotions and sensations.
Electromyography (EMG) measures
EMG is all about detecting the activation of muscles through the use of electrodes, which are
attached to the relevant muscle (or muscles). So again, like EEG, (and like most of the measures
I am mentioning) this method relies on detecting electric current. However, unlike EEG, EMG is
a direct indication of activation in the peripheral nervous system.
• Brow (Corrugator supercilii) that register negative emotion (unpleasant valence)
• Cheeks (Zygomaticus major) that register positive emotion (pleasant valence)
• Area around the eyes (Orbicularis oculi) that are said to register expressions of enjoyment
and "genuine pleasure" (whatever that is).
[http://www.gamasutra.com/view/feature/6341/game_testing_and_research_the_.php?print=1]
6.1.3 Data analysis (how)
38. An eye tracker measures eye positions and eye movement. These are some of the most usual
measures and metrics that are employed:
• Fixations and gaze points
• Fixation sequences (saccades)
• Areas of Interest (AOI)
• Time to first fixation (TTFF)
• Time spent
• Pupil size (interest and attention)
Eye Tracking provides data from the gaze, in regards of perception, attention and mental processes.
Eye Tracker (ET) measures
This is a heat map created with the fixation points.
39. The Eye Tracking is easy to use in comparison with the presented biometrics methods.
Eye tracking can be definitive to test if something is visible enough (enemies, visual interface
elements, etc.). The time spent in an area is measure that explains well attention.
It happens often that players complaint during a post-playtesting interview something is not
visibile enough. Then during an eye tracking session it appears they can see it, although they
struggle at other parts of the interaction. Saying “it’s not visible enough is just part of the
complaint for a challenge”.
Heat map Fixations and gaze points
40. Case Study:
Research Question: Do all users have the same intent and pay attention to the same
places while looking at search results?
Research Question: What places do they pay more attention to?
Context: Google search results.
41. Case Study:
Research Question: How long does it take to users to find the Credit Card page of the
site on the home page?
Context: Google search results.
42. Pros
• Gives objective quantifiable data unable by any other means (it cannot be faked).
• Allows for continuous data recording without interrupting the player.
• The relationship between the player’s psychology (emotion and mental effort especially)
and some signals like heart rate and face expressions is very clear.
Cons
• The data acquisition devices are typically expensive, and sufficient attention and time should
be given to personnel training and device maintenance.
• Often invasive or intrusive.
• Problems with specificity, artifacts, inference and validity can make it difficult to interpret.
6.1.4 Pros / Cons
Eye Tracker pros (the cons are basically the same as other biometric methods)
• Its data can also be commented in a qualitative way.
• Effective metric of player attention/gaze.
• Excellent tool for interface design.
• Provides good understanding of scene interpretation.
Mixed-Methods! We want to examine the relationship between game events,
physical responses and survey results.
43. [https://link.springer.com/article/10.1007/s11257-017-9192-3]
Flow could be research with Eye Tracking and surveys.
• With an eye tracking we would see a sustained attention (“Focal attention is reduced to
one area and is stable along time”).
• With a survey we would see the degree of immersion they recall from their experience.
Case Study: Flow in Need for Speed 2015
1. Research Questions: Does the player experiment a Flow state of mind while playing Need for
Speed?
2. Concepts/Theoretical Framework: Flow theory, attention, physiological effects of attention.
3. Methodology: Biometric sensors: e.g. heart rate (HR), galvanic skin response (GSR),
respiration (RESP), temperature (T), blood volume pulse (BVP)
4. Results: Players experienced flow. Both data from the gameplay and the player physiological
responses could be used to understand what creates flow and then automatically generate
effective content variations to stimulate it.
44. Case Study: Fear in Resident Evil 7 and Agony
1. Research Question: Which game creates the strongest emotion of fear in the player, Resident
Evil 7 or Agony?
2. Concepts/Theoretical Framework: Fear, physiological effects of fear.
Interview could tell us their reasoning, how they explain fear,…Survey could tell us how much
they think they experienced fear. A survey would even be useful to see how general is the
emotion of fear among the participants.
But they are not the same things as taking physiological measurements, as we do not obtain
precise measures about their emotions. We can see better how much they experience fear.
3. Methodology: Biometric sensors: e.g. heart rate (HR), galvanic skin response (GSR),
respiration (RESP), temperature (T), blood volume pulse (BVP)
45. The Effectiveness of Casual Video Games in Improving Mood and Decreasing Stress. Carmen V. Russoniello,
Kevin O’Brien and Jennifer M. Parks. 2009.
[https://www.supercheats.com/articles/262/video-games-as-stress-relief/2]
Case Study: Relax in Casual Video Games
1. Research Question: Does playing casual video games such as Bejeweled 2, Bookworm
Adventures and Peggle improve mood and decrease stress?
2. Concepts/Theoretical Framework: Emotion theories.
3. Methodology: Biometric sensors: Electroencephalography (EEG), Heart Rate Variability
(HRV).
4. Results: The effects were consistent with increased mood and corroborated findings on
psychological reports. These sort of games help in releasing stress.
46. Why aren’t biometrics more adopted by video game developers?
Because of the “cons” and,…
1. It does not substitute playtesting, and the subjective methods (surveys, interviews) can
provide data which is useful “enough” to iterate in game design.
2. Only certain games like First-person shooters, racing cars, action-RPG, have clear
patterns in terms of subjective behavior (emotional arousal, cognitive workload). Many
other games are more fuzzy and do not put the player in these high states.
Valve’s game researcher opinion (Dr. Mike Ambinder, experimental psychologist)
"Some publishers are going down this route, but I'm not sure biometrics is the way to go. We've
had a lot of time to experience what works and what doesn't, and biometrics doesn't tend to
add a lot to the techniques we're already using. ”
“You could learn most of this stuff by just asking people. I mean, what we want to know is
whether people are having fun or not. And just asking them, or watching them play the game,
can determine that. We don't need super-precise accuracy on the emotional state. ”
Sometimes, more complex scientific methods do not add enough to justify their use.
[http://www.gamespot.com/articles/the-science-of-playtesting/1100-6323661]
47. 6.2 (User Behaviour) Data Analysis
Telemetry n. The science and technology of automatic measurement and transmission of data
by wire, radio, or other means from remote sources, as from space vehicles, to receiving
stations for recording and analysis.
Game development telemetry – … automatic measurement and transmission of data from
game executable, build pipeline and development tools for recording, analysis and continual
improvement. What kind of data are we obtaining? Events, player small interactions,
achievements, etcetera. this is the behavioral data that we want to import from gameplay.
What is behavior? It is the user interactions along time. We can define interactions, we can
define time. By analyzing data related to the user behavior we can understand certain things
about motivation, but not completely.
Game analytics can thus be understood as the application of analytics to game development
and research (El-Nasr, Drachen and Canossa, 2013, p.5).
[http://www.gameanalytics.com/blog/what-is-game-telemetry.html]
48. a) What are we studying with game data
analysis?
Playtesting can shred light into ”what” and
”why”, but with game analytics we can
understand “how much” something happens.
It gives us a wider perspective, in order to
evaluate if the what happens very often and
what is extensive to all players.
Besides, we can ‘cross data’ with economical
data (revenue, conversions, etcetera.). But this
is not the scope of our study in UX.
6.2.1 Purposes and problems (what)
UX
Physiological reactions
User behaviour / interactions
Recalled UX (later)
Data analytics is used to study user behaviour and monetization.
49. a) What are we studying with game data
analysis?
Playtesting can shred light into ”what” and
”why”, but with game analytics we can
understand “how much” something happens.
It gives us a wider perspective, in order to
evaluate if the what happens very often and
what is extensive to all players.
Besides, we can ‘cross data’ with economical
data (revenue, conversions, etcetera.). But this
is not the scope of our study in UX.
6.2.1 Purposes and problems (what)
Data analytics is used to study user behaviour and monetization.
UX
Physiological reactions
User behaviour / interactions
Recalled UX (later)
Now
50. IMPORTANT: Once you have logs, you will never ask
again about interaction when there is limited amount of
time in an interview or in a survey.
Behavioral data is unbiased data
Which gives you the clearest picture of your game? Surveys or data logs?
Data science is the set of methods and practices to extract knowledge from data.
It mainly provides analysis and visualization. We apply data science to games.
Behavioral data does not depend on the ‘observer’ or
the player (unless there are coding bugs). It comes
directly from the game functioning. Like biometrics, it
cannot be faked by the player.
51. b) What kind of method it is Quantitative and behavioural
It answers well some ‘how much’ questionsDatasource
playtesting eye tracking
data analysis
Interviews & focus groups surveys
physiological data
playtesting
think-aloud
52. c) Experiment structure
Theory-driven means that the entire research process is based on a theory which gives a model of
interpretation of the reality, then it allows you to find indicators, to create hypothesis, to run a
test and finally obtain results and conclusions. This is the scientific method explained in previous
classes. With theory-driven we want to prove a hypothesis right (or wrong) and still want an
explanation.
Research Question > Concept / Theoretical Framework > Methods > Data Analysis > Conclusions
Instead, data-driven strategies rely on solely pay attention to the data without having a strong
theory behind. In this second case, it is possible to gather data without a theory in mind, and then
‘work out’ the data until finding interesting insights. I have this data… What can I do with it?
Perhaps, later, we can find a theory which helps us at understanding the data.
Data analysis (telemetry) allows data-driven research while biometrics and surveys do not.
53. d) Most common concepts we study with data analytics
The most usual specific purposes to use game analytics are to balance economy, to
understand challenge level, to study game mode preferences, to study motivation and
participation, to catch cheaters.
We cannot study how users understand narrative, whether they find the controls or the user
interface usable, or the emotions they experience. Just the behavioural part.
• If you know where your players are getting
stuck, you can change the difficulty so more
players can continue playing.
• If you know where players die too easily,
you can redesign the level to prevent that
from happening.
• If you know that some game items are too
expensive, you can change the prices so
more players can afford them.
Analytics may give you a hint on why this is probably too hard! (So many deaths).
54. Game telemetry data can be thought of as the raw units of data that are derived remotely from
somewhere, for example an installed client submitting data about how a user interacts with a
game, transaction data from an online payment system or bug fix rates. In the case of user
behavior data, code embedded in the game client transmits data to a collection server; or the
data is collected from game servers (as used in e.g. online multi-player games like Fragile
Alliance, Quake and Battlefield) (Derosa, 2007; Kim et al., 2008; Canossa and Drachen, 2009).
6.2.2 Production pipeline (when and where)
So… it can be from the game developer
headquarters or from anywhere else in the world.
We are here Production and post-production
55. 6.2.3 Data acquisition and pre-processing (how)
Game telemetry happens in different
phases: the first one is attribute definition.
In the following sections we are going to
see more about each phase.
1.
2.
3.
4.
5.
6.
7.
8.
An Introduction to Gameplay Data Visualization. Günter Wallner and Simone Kriglstein. Game Research
Methods. 2015.
56. 1. Attribute definition
The essence is that telemetry is measures of the attribute of objects – the latter which
should be understood broadly to include people and processes.
For example, the location of a player character as it navigates a 3D environment. In this case
the location is the attribute, the player character the object.
In order to work with telemetry data, the attribute data needs to be operationalized. This
means deciding a way of expressing the attribute data: 0-1, numerical, etc.
Raw telemetry data can be stored in databases.
Game metrics are, in essence, interpretable measures of something, as long as this
something is related to games.
Metrics can be directly attributes or created from attributes (for instance, a ratio, the
number of events in a determined time or place).
57. 1) General attributes: The attributes that are shared for users (players) across all games. These
form the core metrics which can always be collected, for any computer game, e.g. when a user
starts playing a game, stops playing, a userID, etc.
2) Core mechanics/design attributes: The essential attributes related to the core of the
gameplay and mechanics of the game. For example attributes related to time spent playing,
number of opponents killed, etc. Defining the core mechanics attributes should be based
directly on the key gameplay mechanics of the game, and provide information that allows
inferences to be made about the user experience. For example, whether players are
progressing as planned, if flow is sustained, death ratios, level completions, point scores, etc.
3) Core business attributes: The essential attributes related to the core of the business model
(e.g. F2P) of the company. For example, logging every time a user purchases a virtual item,
establishes a friend connection in-game, country of origin, recommends the game to a
Facebook friend, attributes related to retention, virality and churn, etc.
These other attributes
can be regarding
business,
performance, errors…
We are interested in
these two! (1 and 2)
58. 2. Data acquisition
During this second step, incoming telemetry data are transformed and loaded into a database
structure, from where they are accessible for analysis. Additionally, data are cleaned and
otherwise made ready for analysis.
1. Start at the accounts database (in case of an external database provider). This will be the
first step to economy, since the accounts database has the ID of every record that you want
information about. Then, you can store the data. Once it is in, you can validate it.
2. Validate which data is relevant and clean. This eliminates garbage as soon as possible, so
that you are not storing or analyzing unusable data. Starting at the accounts database,
exclude unregistered accounts or administration accounts. For example, exclude test and
admin characters that have artificial attributes. For each valid character in an account, query
for activity in the log database. If the character has not been active during the previous
week, then its record contains no player performance information.
3. Backup valid user, log, and accounts records into an archive database. This will be a useful
warehouse that you may return to in the future to mine for data you have not considered
yet. Treat this backup preciously; if you were an archaeologist, this would be your find; if you
were a detective, this would be your forensic sample.
59. The data acquisition comprises the different computing systems and processes dedicated
to obtain and storage the data.
• The code installed in the back-end of the game to store the .
• The communications at network level (TCP-UDP)
• The database where the data is stored (MySQL, etc.)
This is the usual network architecture to do data analysis
[https://www.raywenderlich.com/7559/game-analytics-101]
System Architecture
60. The transmission of a piece of information via a telemetry system – irrespective of whether this is in the
context of user, process or performance measures – in games can occur in three fundamental ways:
1) Event: A pre-specified event occurs, for example, a user starts a game, a designer submits a bug fix
request, a unit of a game is sold, a player fires a weapon, buys an item, etc. – any action initiated by a
person or system forms an event. Event-based telemetry is based on tracking such actions and
transmitting this information to a collection server.
2) Frequency: Rather than being triggered by the occurrence of a specific event, information can be
recorded following a specific frequency. For example, when tracking the trajectory of player avatars
through virtual environments, we can record the location of the avatar once per second, as a
compromise between precision and bandwidth constraints. Frequency-based recording of telemetry is
generally used when the attribute of the object being tracked is always present, e.g., a player character
in an MMORPG always has a position in the world when playing.
3) Initiated: Sometimes the game analyst wants to enable and disable the tracking of a specific attribute,
rather than having a telemetry system autonomously submitting tracked information based on some
pre-defined command. For example, it may not be necessary to record player avatar trajectories all the
time, but only when updates or patches are pushed to the users. Having the ability to turn on and off
recording of specific attributes can be useful in these situations. There are different strategies available
for the recording, transmission and storage of game telemetry.
Chapter 12: Game Analytics – The Basics. Drachen et al. Game Analytics (Book).
Tracking Strategies
Strategies are important in order not to saturate the game performance and database.
62. Valve has a platform for recording gameplay metrics: Kills, Deaths, Hero Selection, In-
Game Purchases, Matchmaking wait times, Bullet trajectories, Friends in Party, Low-
Priority Penalties, etcetera.
Data sent at relevant intervals: Daily, Monthly, Lifetime Rollups, Views, Aggregations.
This is an important point: it is not useful to send data at all times. At the same time, it is
necessary to collect data for different periods of time (days, weeks, months). Take into
account the circadian patterns, holidays, etc.
Valve and its Data Collection OGS (Operational Game Stats)
[https://www.youtube.com/watch?v=HQwL6zh7AgA&list=PLckFgM6dUP2hc4iy-IdKFtqR9TeZWMPjm]
63. To summarize, many alternate methods can do this. Here is a simple method that economizes
storage space and reduces mining computation. This preprocess has five general steps:
1. Take a snapshot of the database.
2. Validate that the data is clean and appropriate for analysis.
3. Integrate the data into a central archive.
4. Reduce the data down to just the fields you need.
5. Transform the reduced data into a form that is easy to analyze for player performance.
We split attribute definition, data acquisition and metrics development because… we can
always create better more developed metrics. If we obtain the data and process the metric at
the same time, we can never go back to the original data.
Its best to obtain the attributes simple and then process more complex metrics.
Metrics development
3. Data pre-processing
64. Engagement metrics. This are usually created from general attributes which can be found in
all games. They are also very usual in websites to measure user engagement.
These are usually about the session duration, the number of return (retention or loyalty).
They usually take into account variables related to time.
4. Metrics development
Up to this point the discussion about user attributes has been at a fairly abstract level,
because it is impossible to develop classes of which user metrics it makes sense to develop
for all types of games.
FPS, TPS, Racing, Adventure Games, Arcade, Beat ’em up, Family games, Fitness games,
Music games, Platformer games, RPG, Simulation, Sports Games, Strategy Games.
Exercise! Let’s play with metrics
65. • FPS: Useful gameplay metrics: Weapon use, trajectory, item/asset use, character/kit choice,
level/map choice, loss/win [quota], heatmaps, team scores, map lethality, map balance,
vehicle use metrics, strategic point captures/losses, jumps, crouches, special moves, object
activation. AI-enemy damage inflicted + trajectory. Possibly even projectile tracking.
• TPS: Useful gameplay metrics: as for FPS + camera angle, character orientation.
• Racing: Useful gameplay metrics: Track choice, vehicle choice, vehicle performance, win/loss
ratio per track and vehicle, completion times, completion ratio per track and player, upgrades
[if possible], color scheme [if possible], hits, avg. speed different types of tracks/track shapes.
• Adventure games: Useful gameplay metrics: story progression [e.g. node based], NPC
interaction, trajectory, puzzle completion, character progression, character item use, world
item use, AI-enemy performance, damage taken and received + source (player, mob).
• Advance: Useful gameplay metrics: trajectory, powerup usage, special ability usage, session
length, stages completed, points reached, unlocks, opponent type damage dealt/received,
player damage dealt/received [as applicable].
• Beat’em up: Useful gameplay metrics: Character selection, ability use, combo use, damage
dealt, damage received (per ability, character etc.), weapon usage, arena choice, win/loss
ratio as a feature of character, player skill profiles.
Chapter 12: Game Analytics – The Basics. Drachen et al. Game Analytics (Book).
66. • Family games: Useful gameplay metrics: varies substantially – subgame selection,
character/avatar selections, game mode used, in-game selections, asset use, number of
players, etc. form some of the possibilities
• Fitness games: Useful gameplay metrics: session length, calories burned, exercises chosen,
match between exercises shown and player actions, player accuracy in performing exercises,
total playtime over X days, player hardware/exercise equipment [usually registered], player
demographics [usually entered during profile creation], music tracks selected, backgrounds
selected, avatar selection, powerups/content unlocked [common feature], total duration of
play per user.
• Music games: Useful gameplay metrics: Points scored, song/track chosen, match with
rythm/auditory mechanics, difficulty setting, track vs. difficulty, track vs. errors, track vs.
choices.
• Platformer games: Useful gameplay metrics: jumping, progression speed, items collected,
powerups/abilities used, AI-enemy performance, damage taken + sources of damage
• RPGs: Useful gameplay metrics: character progression, quest completions, quest time to
complete, asset use (resources), character ability/item use [including context of use], combat
statistics, AI-enemy performance, story progression [including choices], NPC interactions [e.g.
communication], ability/item performance, damage taken + sources of damage, cutscene
viewed/skipped, items collected [including spatial info].
67. • Sports games: Useful gameplay metrics: match types, win/loss ratios, team selection, color
schemes, country chosen, management decisions [if game includes management aspects],
in-match events [e.g. goal scored, fouls, tackles, length of hit], item use [e.g. club type],
heatmap [density of player time spent on sections of the field], team setup/strategy, player
[in-game] selection, player commands to team/team members.
• Strategy games: Useful gameplay metrics: all features related to player strategy and control.
Generally two types of things players can build: building and units. Selections and order of
selection are crucial metrics. Commands given to units, upgrades purchased, trajectory,
win/loss ratio, team/race/color selection, maps used, map settings, match/game settings
(usually strategy games have some settings that affect the core mechanics).
Race/aspect/team chosen, time spent on building tasks vs. unit tasks.
Chapter 12: Game Analytics – The Basics. Drachen et al. Game Analytics (Book).
How many metrics did you match?
68. We are at the data analysis and
evaluation phase and we want to
extract some interesting conclusions
from our data.
During this phases, cases and stored
metrics are selected as required by
the analysis in question.
Some basic operations are:
• To identify or classify
• To compare
• To relate
6.2.4 Data analysis (how)
An Introduction to Gameplay Data Visualization. Günter Wallner and Simone Kriglstein. Game Research
Methods. 2015.
1.
2.
3.
4.
5.
6.
7.
8.
69. Player Modeling using Self-Organization in Tomb Raider: Underworld. Anders Drachen, Alessandro
Canossa and Georgios N. Yannakakis. (PAPER)
Modelling the entire population of players can
give a good idea in order to understand if the
game is balanced for “the most important type of
player”.
Case Study: Identifying Tomb Raider players
1. Research Question: Are there distinct player profiles who prefer
each of the different mechanics and goals provided in the game
Tomb Raider Underworld?
2. Concepts/Theoretical Framework: Bartle’s Taxonomy, game design
concepts, MMOO theory,…
3. Methodology: Data Analysis (neural networks). Metrics: causes of
death (opponent, environment, falling), number of death,
completion time, help-on-demand. In this study we identify players.
4. Results:
They identified the player types with cluster
analysis. The algorithms trained on the data reveals
four clusters of playing behavior — labeled as
Veterans, Solvers, Pacifists and Runners.
70. Case Study: Enabling cooperation in Left 4 Dead
1. Research Question: Does introducing a GUI marker make player cooperation more effective
in Left 4 Dead?
Problem: players letting teammates die.
Hypothesis: Give better visual cues to teammate location will increase cooperation and
reduce team death rate.
2. Concepts/Theoretical Framework: GUI design, usability, perception.
3. Methodology: Data Analysis (metric: high death rates), Surveys, Q&As. Setting “No Mercy –
The apartments”. In this study we compare metrics in different scenarios.
4. Results:
Death decreased a 40%
Survey ratings of enjoyment/cooperation increased
71. Case Study: Improve Player Communication in DOTA 2
1. Research Question: Does introducing alert messages and automatic bans in DOTA 2 decrease
the level of negative communication?
Hypothesis: Automating communication bans will reduce negativity in-game
Iterative (future): Will this work in Team Fortress 2? Do these systems scale?
2. Concepts/Theoretical Framework: Theory-Driven: Operant conditioning. No feedback loop
to punish negativity.
3. Methodology: Data Analysis, Chat, reports, forums, emails, quitting.
Measurements: Chat, reports, ban rates, recidivism. In this study we relate.
4. Results:
35% fewer negative words used in chat • 32% fewer communication reports • 1% of active player
base is currently banned • 61% of banned players only receive one ban.
72. Case Study: CS:GO weapon selection
1. Research Question: Does a wider selection of weapons increase a longer gameplay in CS:GO
multiplayer?
Iterative: Inform future design choices.
Hypothesis: M4A4 usage is high; few choices in late-game. Creating a balanced alternative
weapon will increase player choice and playtime.
2. Concepts/Theoretical Framework: Game design, balance theory (greater tactical choice ->
Player retention).
3. Methodology: Data Analysis (purchase rates, playtime). In this study we relate.
4. Results:
~ 50/50 split between new and old favorites • Increase in playtime.
Does weapon variety increase player retention? Still to answer.
73. Visualizations are representations of
data to perceive, use, and
communicate information.
In context of gameplay data analysis,
the interest to use and develop
visualization techniques increased in
the last years among industry
professionals and researchers. Visual
representations of gameplay can
support game developers and
designers to analyze recorded player
behavior to, for example, identify
interaction or design problems or to
understand the effects of design
decisions.
6.2.5 Reporting and data visualization
An Introduction to Gameplay Data Visualization. Günter Wallner and Simone Kriglstein. Game Research
Methods. 2015.
[http://www.gamasutra.com/view/feature/170332/?print=1]
1.
2.
3.
4.
5.
6.
7.
8.
74. An Introduction to Gameplay Data Visualization. Günter Wallner and Simone Kriglstein. Game Research
Methods. 2015.
[http://www.gamasutra.com/view/feature/170332/?print=1]
The power of visualization: on the left is where people are standing when they make kills with a
weapon and on the right is deaths by this weapon in Halo Reach. With just a basic knowledge of
FPS games, you can still probably work exactly what kind of weapon this is and where the
elevated and the open spaces are in the level.
They are less precise than statistical analysis
but sometimes more helpful. Easier to
understand and find interesting insights.
75. Charts are pictorial representations of information. Charts exist in a variety of forms, like bar
charts, pie charts, or scatter plots to name but a few with each of them having different
advantages and disadvantages.
Four general types of data visualization
Pie-charts show which types of towers have been built on the different building lots in Team Fortress.
The radius of the pie-chart is proportional to the number of towers built (Kayali, et al., 2014).
Advantage: they can summarize variables and display trends more easily.
Disadvantage: they can lead to false conclusions if the chart is inappropriate.
Charts; HeatMaps; Movement Visualizations; Node-Link Representation
76. Heatmaps are used to visualize aggregated data from huge data sets but can also be used to
provide players with individualized visual feedback for the purpose of post-gameplay analysis.
Heatmap of death locations on the Team Fortress 2 map Goldrush.
[http://www.gamasutra.com/view/news/125213/Opinion_Balance_and_Flow_Maps.php]
Advantage: they can show spatial patterns more easily.
Disadvantage: only one variable can be shown at a time and a third dimension is lost.
78. Movement visualizations can help you understand how players actually move around in a game can
thus provide valuable information for level design.
It can be interesting to use colors in order to clear
what is the achievements the player finally
obtains or how he died.
Advantage: they show exploration patterns.
Disadvantage: a lot of data can be difficult to
visualize.
79. Node-link approaches provide an intuitive way to visualize the relational structure of data items.
Left: Player movement between regions, cities, and battlegrounds on the World of Warcraft
continent Outland. Right: Corresponding matrix view with cells colored according to the number
of players moving from one area to another.
Advantage: suitable for multidimensional or abstract data.
Disadvantage: dense graphs suffer from visual clutter, layout can look confusing without something
to orient the data.
80. Case Study: Bioware’s Star Wars Area Balancing
1. Research Question: Are there unbalanced and conflicted areas in Star Wars: The Old Republic
that drive players to a bad experience?
Content interation can help at balancing.
1. Concepts/Theoretical Framework: Game balance theories
2. Methodology: Communication Channels, Data Visualization, Content Tracking.
81. The interesting point is that BIOWARE not only has good metrics but every item in the game
(asset) tracked. So they can cross even more data.
Georg Zoeller MMO Content Iteration
[http://twvideo01.ubm-us.net/o1/vault/gdconline11/Georg_Zoeller_Rapid_MMO_Content_Iteration.pdf]
[http://gdc.gulbsoft.org/2011-gdc-online-talk]
82. “Almost all actionable content feedback is more useful when you look at from a spatial or
temporal perspective. In order to create an efficient iteration process, we need to look at all
three elements together.” Zoeller, BIOWARE
With excellent tools
Georg Zoeller MMO Content Iteration
[http://twvideo01.ubm-us.net/o1/vault/gdconline11/Georg_Zoeller_Rapid_MMO_Content_Iteration.pdf]
[http://gdc.gulbsoft.org/2011-gdc-online-talk]
83. We want to make it possible for people in the trenches to analyze and suggest course of action to their
leads. We also want them to be able to spot mistakes on their own – something the tool can help with by
highlighting common mistakes (2 strong enemies on a single encounter, etc.)
Georg Zoeller MMO Content Iteration
[http://twvideo01.ubm-us.net/o1/vault/gdconline11/Georg_Zoeller_Rapid_MMO_Content_Iteration.pdf]
[http://gdc.gulbsoft.org/2011-gdc-online-talk]
84. Pros
• Objective data that can be collected remotely and discretely.
• It can be used to see trends and provide data that supports other methods.
• It allows for continuous data recording without interrupting the player
• You can take robust conclusions from an entire population of players.
• You can start research from the data without the need for a theory.
• They are very useful for post-production (mobile games) or multiplayer games (MMORG).
Cons
• Time and resources-consuming.
• It requires good professionals with programming and statistics skills.
• Needs large sample sizes to get meaningful data. And at the same time, you can get too
much data.
• No subjective feedback, so you can never really tell what is going if you have no other
source of data. You can make inferences about usability or motivation, but not with the
same certainty that other data would provide you.
• It does not clearly substitute any other methodology (playtesting, survey, interviews).
• Behavioral data is not good at explaining ’why’ or giving new ideas.
6.2.6 Pros and cons
85. Hodent (2017) reminds us of some classical statistical fallacies. There are things to keep in mind:
• Sample representativeness. You cannot generalize from a small sample.
• Is the result statistically significant? Average are not enough. You need statistical tests.
• Correlation is not causation. Be cautious. You do not know the nature of the relationship
with a correlation. Perhaps it is a coincidence and there is no pattern.
• Data are not information and information is not insight. Tons of data is not the answer if we
have no questions or we are not good at interpreting it.
• Bad data are worse than no data. If you have a bug in the telemetry it can be fatal.
• Data analysis is good at telling what is going on but not why. You should complement it with
other methods.
6.2.6 Statistical fallacies and limitations
86. [http://www.gamasutra.com/view/feature/5827/starcraft_ii_building_on_the_beta.php?print=1]
Analyzing behavioral data without a feel on
playtesting can be misleading.
If we look at the stats and we say,
"This doesn't actually back anything
we're experiencing online," I'm very
suspicious of that number.
We look at another source and say,
"You know what? What they're saying
online matches my play experience,
and it matches the stats. This seems
real. Let's talk about what some
possible fixes can be."
Data-driven assumptions wrong: Starcraft II carriers
Design director Dustin Browder warns that caution is required when analysing the data.
• “With unit stats, I can tell you that, for example, in a Protoss versus Terran game, 12 percent
of the time the Protoss build carriers. And when they build carriers, they win 70 percent of the
time. You could say, "That must mean carriers are overpowered!”. Not, it just means that you
get towards the end of the game. If they have extra resources to waste, they’re going to win
anyway.”
87. Investigate suspicious player performance starting at the top.
Data-driven assumptions right: Preventing Cheaters
• A cheater in a MMOG does not just cheat himself. He performs an injustice to all honest
players. Cheating short-circuits gameplay, so it achieves exceptionally high performance.
88. You can only check Google Analytics in post-production.
Fidelity
Frustration
Classic usability
Conversions
tasques més freqüents, cerques més freqüents...
rati de conversió.
rati de compra.
Search analytics
6.2.7 Web: Google Analytics
90. Key Questions and Concepts (TakeAways)
• Quantitative data is not possible to bias with the gathering procedures.
However, you need expertise in obtaining and analysing them.
• Biometrics may be useful to understand very specific aspects of user
experience, but playtesting and interviews may be enough just to provide
insight to game design.
• Data analytics is fundamental to study the characteristics of the players
that end up playing your game. It provides valuable data but it also needs
to be contextualized with qualitative data.
91. References and Bibliography
• All the references provided in the Powerpoint are valuable.
Books
• Game Analytics, Maximizing the Value of Player Data. El-Nasr et al. 2013.
• Chapter 12: Game Analytics – The Basics. Anders Drachen, Magy Seif El-Nasr, Alessandro Canossa. Game
Analytics (Book).
• Game Usability: Advancing the player experience. Isbister, Katherine, and Noah Schaffer. CRC Press. 2015.
• Game Research methods: An overview. Lankoski, P., & Björk, S. 2015.
• Games User Research: A Case Study Approach. Miguel Angel Garcia-Ruiz. AK Peters/CRC Press. 2016.
• Others
• Methods for Game User Research - Studying Player Behavior to Enhance Game Design. Heather Desurvire
and Magy Seif El-Nasr. 2013. (PAPER)
• Articles available in the site Gamasutra [gamasutra.com]
• Blog gameanalytics
– [https://andersdrachen.com/2013/10/31/10-great-reads-on-gamef-analytics/]
– [https://andersdrachen.com/category/game-user-research]
• CASE STUDY: "Game Analytics" book. Chapter 16: Better Game Experience Through Game Metrics: A Rally
Videogame Case Study. Pietro Guardini and Paolo Maninetti. (PAPER)
• CASE STUDY: Game Metrics for Evaluating Social In-game Behavior and Interaction in Multiplayer Games.
Katharina Emmerich. (PAPER)
• The Game Life-Cycle and Game Analytics: What Metrics Matter When? | Mark GAZECKI
[https://www.youtube.com/watch?v=C5lx4L0iJQI]
All images used in these slides belong to the cited sources.
92. • Nacke, L. E. (2013). An introduction to physiological player metrics for evaluating games. In Game
Analytics (pp. 585-619). Springer, London.
• “Beyond Thunderdome: Debating the effectiveness of different user-research techniques”
[https://vimeo.com/26733185]
• [http://www.gamasutra.com/blogs/TrevorMcCalmont/20130208/186075/5_Common_Pitfalls_for_Mobil
e_Game_Analytics.php]
• Game Usability: Advancing the player experience. Isbister, Katherine, and Noah Schaffer. CRC Press. 2015.
• Game Research methods: An overview. Lankoski, P., & Björk, S. 2015.
• Games User Research: A Case Study Approach. Miguel Angel Garcia-Ruiz. AK Peters/CRC Press. 2016.
• Playful Design. John Ferrara. Rosenfeld Media, 2012.
• The Art of Game Design: A Book Of Lenses. Jesse Schell. Carnegie Mellon University. 2008.
• King, R., Churchill, E. F., & Tan, C. (2017). Designing with Data: Improving the User Experience with A/B
Testing. " O'Reilly Media, Inc.".
All images used in these slides belong to the cited sources.