While traditional task-based usability research provides invaluable insights, sometimes expanding your practice to include additional methodologies allows usability to have greater influence in an organization. In this session, you will learn how adding remote observational studies enabled the team at AutoTrader.com to become more involved in projects from the beginning.
These set of slides will explain the importance of maintaining the user friendly features of any website. It contains statistics gathered from researches that have been already conducted and stats collected over the internet to quantitatively prove the importance of the aforementioned title.
Introduction to usability and usability testing as a discipline, followed by how to do guerilla usability testing. Presented at Duke Tech Expo April 13, 2018 with co-author Lauren Hirsh, with content from a prior collaborative presentation of hers.
Day 2 slides from a two-day workshop on UX Foundations by Meg Kurdziolek and Karen Tang. Day 2 covered research methods that can be used throughout the design process to evaluate and validate design.
These set of slides will explain the importance of maintaining the user friendly features of any website. It contains statistics gathered from researches that have been already conducted and stats collected over the internet to quantitatively prove the importance of the aforementioned title.
Introduction to usability and usability testing as a discipline, followed by how to do guerilla usability testing. Presented at Duke Tech Expo April 13, 2018 with co-author Lauren Hirsh, with content from a prior collaborative presentation of hers.
Day 2 slides from a two-day workshop on UX Foundations by Meg Kurdziolek and Karen Tang. Day 2 covered research methods that can be used throughout the design process to evaluate and validate design.
User Experience Design + Agile: The Good, The Bad, and the UglyJoshua Randall
There's a rumor going around that user experience design (UXD) and Agile don't play well together. In this talk, I'll explain that they do -- most of the time! Learn about the historical reasons for why these two disciplines sometimes butt heads, as well as the good/bad/ugly of various approaches to integrating design and development.
Usability and User Experience Training Seminarlabecvar
This presentation describes a day-long seminar for giving participants an overview of best practices in usability design and research. Also included are several hand-on exercises to be done throughout the day to solidify participants' understanding of course concepts.
Preference and Desirability Testing: Measuring Emotional Response to Guide De...Paul Doncaster
(From UPA 2011-Atlanta) Usability practitioners have a variety of methods and techniques to inform interaction design and identify usability problems. However, these tools are not as effective at evaluating the visceral and emotional response generated by visual design and aesthetics. This presentation will discuss why studying visual design is important, review considerations for preference and desirability testing and present two alternative approaches to user studies of visual designs in the form of case studies.
Choosing the Right Research Methods for Your Project (webinar)Susan Mercer
It’s very easy for User Experience researchers to get stuck in the rut of using your favorite research methods for gathering information and getting user feedback. But, are you really gathering the best information that you can? Or are there other methods that are better suited for your project’s specific needs?
Or, if you’re just starting out – how do you know whether you should conduct interviews, run a survey or a card sort, or something different all together?
Don’t stress – in this webinar, we’ll cover the most popular user research methods and discuss their strengths and weaknesses. Each method shines in different circumstances, and we’ll highlight the factors that will make each successful. We will also present a structured approach to helping you choose the best method or methods for a particular situation.
TestHive sahipliğinde gerçekleşen Usability Testing Workshop etkinliğinde kullanıcı deneyimi ve kullanılabilirlik testi üzerine genel bir teorik bilgiye sahip olduk. Sonrasında, card sorting, tree testing, usability testing planlama, usability testing ve usability testing raporlama workshopu yaptık.
Genel olarak şu konulardan bahsettik:
Kullanıcı deneyimi ve kullanıcı alışkanlıkları
Do's and don'ts
Test etikleri
Teste hazırlanma
Senaryo seçimi
Test sonucunu raporlama
Test sonucunu yorumlama
Esra Yalman
Experience Design Consultant at ThoughtWorks
Content Strategy and Product Management (in science education)Roger Hart
Presentation from Content Strategy Applied 2017
When your product is mostly content, product management looks a lot like content strategy. The Royal Society of Chemistry is an academic publisher, and a major provider of educational resources for schools and teachers. So that's certainly true here. Having worked in content strategy and product management, and now helping the RSC develop its product management function, I'll talk about how the disciplines interact.
We'll cover:
- What makes a good strategy, and what it means to be a product
- Innovation, roadmapping, and thinking about services
- Measurement and value when your goals are both charitable and commercial
Intimidated by conducting your own usability study? This session will give you the tools you need to conduct effective usability tests whether your participants are in the room or in a different country. The session includes practical techniques to successfully plan, prepare, and conduct your test and activities to help you become more confident with the entire process of usability testing. Finally, you’ll get tips on how to get the most useful results from your study.
Participants will also learn about:
Testing protocols
Types of usability testing and required vs. optional resources
Recruiting and scheduling usability tests
Non-disclosure and consent forms and their purposes
Pilot testing
Techniques for interacting with test participants
Current usability testing issues of interest (e.g. testing internationally, moderated vs. un-moderated, etc.)
This was a 4-hour workshop that was given at World Usability Day Colombia. #wudco14
Summary:
Now more than ever is the survival of the easiest. Whether the product is a website or a handheld device, success depends largely on how easy it is to use. Usability testing is one of the most effective for creating an intuitive methods. By observing actual people when they use the product, you can get valuable insights if your design is easy to use. Attendees will learn how to conduct a usability test with end users of a product. This workshop is highly interactive and includes several practical exercises to give participants practical experience.
You will learn:
- How to plan a usability testing study
- How to define the goals and objectives
- Explore options (unmoderated usability testing vs. unmoderated & remote vs. in-person)
- How to recruit the right participants
- How to create tasks (Interview-based vs. predefined tasks)
- How to moderate a usability test
- How to analyze and report the results
UXPA DC Redux 2013 Notetaker Perspective 10-25-2013.pptUserWorks
Kristen Davis and Dick Horst from UserWorks presentation slides on the "Notetaker's Perspective During Usability Testing: Recognizing What's Important, What's Not" from UXPA-DC Conference Redux 2013
User Experience Basics for Product ManagementRoger Hart
User Experience (UX) has matured as a discipline and radically changed how products are delivered. It touches workflows, usability, customer needs, and of course visual design and UI. Product managers can't ignore it, even if they want to... and if they want to, they're probably wrong. The tools of User Experience can help us get closer to our customers and differentiate our products.
Day 1 slides from a two-day workshop on UX foundations by Meg Kurdziolek and Karen Tang. Day 1 covered the building blocks of design process and design research methods.
Remote usability testing and remote user research for usabilityUser Vision
From User Vision's presentation on remote usability testing describing some of the main methods, challenges, tools and tips for successful remote usability testing for user experience
User Experience Design + Agile: The Good, The Bad, and the UglyJoshua Randall
There's a rumor going around that user experience design (UXD) and Agile don't play well together. In this talk, I'll explain that they do -- most of the time! Learn about the historical reasons for why these two disciplines sometimes butt heads, as well as the good/bad/ugly of various approaches to integrating design and development.
Usability and User Experience Training Seminarlabecvar
This presentation describes a day-long seminar for giving participants an overview of best practices in usability design and research. Also included are several hand-on exercises to be done throughout the day to solidify participants' understanding of course concepts.
Preference and Desirability Testing: Measuring Emotional Response to Guide De...Paul Doncaster
(From UPA 2011-Atlanta) Usability practitioners have a variety of methods and techniques to inform interaction design and identify usability problems. However, these tools are not as effective at evaluating the visceral and emotional response generated by visual design and aesthetics. This presentation will discuss why studying visual design is important, review considerations for preference and desirability testing and present two alternative approaches to user studies of visual designs in the form of case studies.
Choosing the Right Research Methods for Your Project (webinar)Susan Mercer
It’s very easy for User Experience researchers to get stuck in the rut of using your favorite research methods for gathering information and getting user feedback. But, are you really gathering the best information that you can? Or are there other methods that are better suited for your project’s specific needs?
Or, if you’re just starting out – how do you know whether you should conduct interviews, run a survey or a card sort, or something different all together?
Don’t stress – in this webinar, we’ll cover the most popular user research methods and discuss their strengths and weaknesses. Each method shines in different circumstances, and we’ll highlight the factors that will make each successful. We will also present a structured approach to helping you choose the best method or methods for a particular situation.
TestHive sahipliğinde gerçekleşen Usability Testing Workshop etkinliğinde kullanıcı deneyimi ve kullanılabilirlik testi üzerine genel bir teorik bilgiye sahip olduk. Sonrasında, card sorting, tree testing, usability testing planlama, usability testing ve usability testing raporlama workshopu yaptık.
Genel olarak şu konulardan bahsettik:
Kullanıcı deneyimi ve kullanıcı alışkanlıkları
Do's and don'ts
Test etikleri
Teste hazırlanma
Senaryo seçimi
Test sonucunu raporlama
Test sonucunu yorumlama
Esra Yalman
Experience Design Consultant at ThoughtWorks
Content Strategy and Product Management (in science education)Roger Hart
Presentation from Content Strategy Applied 2017
When your product is mostly content, product management looks a lot like content strategy. The Royal Society of Chemistry is an academic publisher, and a major provider of educational resources for schools and teachers. So that's certainly true here. Having worked in content strategy and product management, and now helping the RSC develop its product management function, I'll talk about how the disciplines interact.
We'll cover:
- What makes a good strategy, and what it means to be a product
- Innovation, roadmapping, and thinking about services
- Measurement and value when your goals are both charitable and commercial
Intimidated by conducting your own usability study? This session will give you the tools you need to conduct effective usability tests whether your participants are in the room or in a different country. The session includes practical techniques to successfully plan, prepare, and conduct your test and activities to help you become more confident with the entire process of usability testing. Finally, you’ll get tips on how to get the most useful results from your study.
Participants will also learn about:
Testing protocols
Types of usability testing and required vs. optional resources
Recruiting and scheduling usability tests
Non-disclosure and consent forms and their purposes
Pilot testing
Techniques for interacting with test participants
Current usability testing issues of interest (e.g. testing internationally, moderated vs. un-moderated, etc.)
This was a 4-hour workshop that was given at World Usability Day Colombia. #wudco14
Summary:
Now more than ever is the survival of the easiest. Whether the product is a website or a handheld device, success depends largely on how easy it is to use. Usability testing is one of the most effective for creating an intuitive methods. By observing actual people when they use the product, you can get valuable insights if your design is easy to use. Attendees will learn how to conduct a usability test with end users of a product. This workshop is highly interactive and includes several practical exercises to give participants practical experience.
You will learn:
- How to plan a usability testing study
- How to define the goals and objectives
- Explore options (unmoderated usability testing vs. unmoderated & remote vs. in-person)
- How to recruit the right participants
- How to create tasks (Interview-based vs. predefined tasks)
- How to moderate a usability test
- How to analyze and report the results
UXPA DC Redux 2013 Notetaker Perspective 10-25-2013.pptUserWorks
Kristen Davis and Dick Horst from UserWorks presentation slides on the "Notetaker's Perspective During Usability Testing: Recognizing What's Important, What's Not" from UXPA-DC Conference Redux 2013
User Experience Basics for Product ManagementRoger Hart
User Experience (UX) has matured as a discipline and radically changed how products are delivered. It touches workflows, usability, customer needs, and of course visual design and UI. Product managers can't ignore it, even if they want to... and if they want to, they're probably wrong. The tools of User Experience can help us get closer to our customers and differentiate our products.
Day 1 slides from a two-day workshop on UX foundations by Meg Kurdziolek and Karen Tang. Day 1 covered the building blocks of design process and design research methods.
Remote usability testing and remote user research for usabilityUser Vision
From User Vision's presentation on remote usability testing describing some of the main methods, challenges, tools and tips for successful remote usability testing for user experience
This presentation was provided by Serena Rosenhan of ProQuest, during Session Four of the NISO event "Agile Product and Project Management for Information Products and Services," held on June 4, 2020.
EffectiveUI's Ari Weissman (Lead Experience Architect) and Lys Maitland (Senior Experience Planner) spoke at Denver Startup Week 2016. Discussion description:
Test early, test often.
It’s a mantra that’s been proven successful time and again when it comes to innovation and design. So why aren’t you doing it? In the start-up world, when everything is moving so quickly, it can be easy to overlook or postpone collecting feedback from real people because of cost, time, or lack of preparation. Don’t let those things stop you. Valid data can be captured cheaply, quickly, and with half-finished products and strategies.
This talk will cover:
What is user testing and why is it important
How to plan for user testing
What are ways to make testing cheaper
What are ways to make testing quicker
How to test with different fidelities of concept and design
How to collect data more frequently
Opportunities for getting the whole team engaged
What to do with the insights/outcomes of research
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...BrittanyRubinstein
As part of Denver's 2019 Startup Week, Crownpeak's Director of UX, Ari Weissman and Lys Maitland, Experience Research Manager at a national healthcare organization, presented a joint session on "Choosing a direction: Learning how to test ideas and designs."
Introduction to Usability Testing for Digital MarketeersLennart Overkamp
These slides provide an introduction to usability testing for digital marketeers. This well-known method in user-centred design is used to improve products, by having participants interact with these products and by measuring their performances and responses.
I presented this topic as a guest lecturer to students attending the Minor Digital Marketing at the Fontys ICT Eindhoven at April 5th, 2017. Providing examples and best practices from Dutch digital design agency Mirabeau, I explained to them the required steps for the preparation, the moderation, and the analysis of usability tests.
Bazley Developing And Evaluating Online ResourcesMartin Bazley
Slides used as an introduction to E-Learning Resources: Evaluation course at Appleby Magna on 11 May 2009 run by Martin Bazley on behalf of Renaissance East Midlands
090511 Appleby Magna Overview PresentationMartin Bazley
Slides used as an introduction to E-Learning Resources: Evaluation course at Appleby Magna on 11 May 2009 run by Martin Bazley on behalf of Renaissance East Midlands
The agency's guide to effective user researchUserTesting
Aiden Bordner, co-founder and Principal Designer at Parade, an experience design firm, draws upon over six years of research experience with clients to discuss some of the tools and processes he’s found to make this process easier. From allocating and protecting budget, to lean research techniques, to setting expectations and reducing client anxiety about test day, these tools will help you work research into more project plans and run successful studies that provide valuable design insights.
Best Practices on Recruiting Participants for Remote User TestingUserZoom
Recruiting the right participants is often the most difficult and challenging aspect of conducting research studies. A well-managed recruitment process can be pivotal to the success of achieving your research objectives.
Join JonDelina (“JD”) Buckley, Manager of User Experience Research at Kelley Blue Book, and Aaron Rager, Client Development Manager at Research Now, to learn about some of the best practices on recruiting the right participants for remote unmoderated user testing.
A mini workshop designed to prepare teams with the knowledge and practice they need to better understand their problems and project gaps, determine appropriate participants, ask the right qualitative questions, and gather information in an unbiased and thoughtful way.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
4. State of Our Practice
In early 2011, our usability practice…
- Conducted mostly evaluative, task-based usability tests
- Kept very, very busy doing those studies
- Had a dedicated team, an in-house lab and a budget
- Did the majority of our studies remotely *
* Want to know more? Take a look, it’s in a book.
Specifically, Remote Research by Nate Bolt and Tony Tulathimutte.
5. But We Wanted More
We wanted to…
- Do a wider variety of studies
- Be more involved in projects from the beginning
So that we could…
- Provide more value to our organization
- Continue to grow as a practice and as professionals
6. We Needed an Opportunity
• The right problem
• The right solution
• The right attitude
7. The Right Problem
A big, new not-yet-project that needed to be done
“the right way”
9. Figuring Out a New Solution
Key Goals:
- Something that is most valuable when done early
to inform a not-yet-project
- Less task-focused, more observational
Restrictions:
- Time - Money
- Resources - Fear
13. The Right Solution
Observe the user in their natural environment remotely, using
the same tools we use for remote usability studies and the same
general approach used in contextual inquiry
14. The Right Solution
Why it was a great idea (for us, at the time):
- No more expensive than our average remote study
- Gave us the opportunity to see users’ actual behavior
- Allowed for many observers to see the study first-hand
- Not too far a departure from what we already did
15. A Perfect Storm
• The right stage in the right project
• Project stakeholders who liked working with us
and trusted us
• The right attitude
• Willpower
16. Checkpoint
Introduction
- Our usability practice
- Our ambitions
The Perfect Storm
Planning the Study
Conducting the Study
Communicating Findings
Reception & Benefits
Getting Started
17. Checkpoint
Introduction
The Perfect Storm
- The right problem
- The wrong solution
- The right solution
Planning the Study
Conducting the Study
Communicating Findings
Reception & Benefits
Getting Started
19. Planning the Study
Step 1: What do we hope to learn?
- How do people decide on major purchases online?
- How do they use various interface tools as they come
across them?
- Which design patterns support decision making?
Which hurt it?
- What don’t we know that we don’t know?
20. Planning the Study
Step 2: How can we answer those questions?
- Interview to learn what starts the decision making process,
and a bit about how users proceed through that process
- Observation to see how users interact with sites that they find
themselves visiting and note how they use the interfaces
21. Planning the Study
Step 3: Who can help us answer those questions?
- Not just automotive
- Individuals who made a major purchase decision using online
tools in the past 2 weeks in one of five verticals:
- Real estate
- Major appliances
- Major electronics
- Travel
- Automotive
22. Planning the Study
Typical Study This Study
Task-based Open-ended
Remote
Defined interface Any interface
Conducted from lab
Evaluative Observational
Think aloud
Specific audience Wider audience
23. Planning the Study
Step 4: The Standard Stuff
- Recruit
- Write the detailed test plan
- Set up web conferences
- Email participants
- Invite observers
24. Conducting the Study
Facilitator (me) in the lab
- One computer running GoToMeeting to observe the
participant’s screen and Morae to record the session
- Another computer for note taking and to talk to the
notetaker in the observation room
- A phone to call the participant
25. Conducting the Study
Observers in the observation room
- A TV showing the computer with GoToMeeting to see
the participant’s screen
- Speakers to hear the conversation
- Post-it notes to write down things they want to remember
26. Conducting the Study
Notetaker in the observation room
- Taking notes
- Facilitating the observers
- Keeping me in the loop
30. Conducting the Study
What we observed:
- How users started
- What they did when they got to a site
- How and why they used specific interface tools
- When and why they left a site, and what they did next
- At what point they had made their decision
32. Conducting the Study
Observer Debrief
- Share any notes taken
- Discuss observations
- Less discussion of “findings”
33. Checkpoint
Introduction
The Perfect Storm
Planning the Study
- What we hoped to learn
- How we planned on learning it
- Who would help us
- The standard stuff
Conducting the Study
Communicating Findings
Reception & Benefits
Getting Started
38. Communicating Findings
Workshop structured around five questions:
- What are users trying to do?
- What information do they need to do that?
- How do they start?
- When are they done?
- How do they use different tools and interfaces?
43. Initial Reception
Key feedback:
- Right research at the right time
- Worthwhile to see other verticals and websites
- Desire for more dedicated observation
44. Long-Term Benefits
The long-term benefits were of far more value to
our usability team.
- Showed clear proof that we could and should do more than
task-based evaluative usability studies
- Established us as an active part of the project team
- Gave us the opportunity to do other new types of studies
45. You can do it, too!
• You have more power than you think you do
• Just do it!
• Work within the confines you can…
• And ignore the ones you can’t.
• Set proper expectations
• Learn
48. The End
Introduction
The Perfect Storm
Planning the Study
Conducting the Study
Communicating Findings
Reception & Benefits
Getting Started
Editor's Notes
Hi. My name is Emily Schroeder, and I’m a Sr. Usability Analyst at AutoTrader.com.Today I’m going to share with you how we tried a new approach – “Remote Fieldwork” – and how it has helped elevate our usability practice beyond the immediate value the research provided.Here’s what you can expect from today:
I’ll start with a quick introduction about our usability practice at AutoTrader.com, and the type of work we were doing early last year. I’ll then share how a perfect storm of events set us up to try something new that was a positive experience for us and our internal customers, while continuing to elevate our practice in ways we couldn’t even predict at the time. I’ll share the details of our study planning and how we conducted the study, as well as and how we communicated those findings. Finally, I’ll share the immediate positive reception our approach received, the long-term benefits our practice has seen, and how you can try something new in your own practice, no matter what challenges you think you may face.Throughout this presentation, I’ll take breaks to take any questions you may have. There will also be a chance for questions at the end, for you to ask anything you forgot or were too afraid to ask earlier on.
As I mentioned, I work at AutoTrader.com. Our site is a consumer-facing site of automotive classifieds that allows people who sell cars – dealers and private sellers – to reach people who buy cars.
When our story takes place, way back in early 2011, we were doing mostly task-based evaluative usability tests: Can the user do that thing we want them to do using this interface?Because we got the chance to test most changes before they went live on our site, we kept very busy with these types of studies.We also were lucky to have a small but dedicated usability team, our own lab and a budget that made it possible to do so many studies and expose lots of people to how real users use websites and other UIs.We had also recently begun doing most of our studies remotely from our lab, and this had become standard practice primarily for recruiting purposes.
Despite how well things were going for us, we wanted more!We wanted to…Do a wider variety of studies, so that we could utilize a larger part of the UX research toolkit-and-Be more involved in projects from the beginningSo that we couldProvide more value to our organization – our team, our department, and our company-and-Continue to grow as a practice and as usability professionals
In order to do the things we wanted to do, we needed to identify the right opportunity.We needed the right type of problem to solve, specifically, the type of questions that need to be asked at the beginning of a projectWe needed the right solution, one that would work within our organization and with our teamFinally, we needed the right attitude to actually do what we wanted to do and step out of our comfort zonesLuckily, last spring, we had the perfect opportunity to make that move.
The business was looking to kick off a huge not-yet-project, and wanted to do it “the right way” by considering user needs based on research from the beginning.This problem aligned very well with our desire to do more up-front UX research, and we had the perfect solution…
We wanted to be able to follow people at home and at dealerships while they shopped for and bought vehicles, known as contextual inquiry.We knew that this type of research would give us the early insights into how users use tools – online and off – to decide what car to buy and follow through with that purchase. It would help the product managers and designers ground their solutions in the actual needs and behaviors of usersIt was a great idea, but we knew it wasn’t feasible in our environment.Things that got in the way:Time – We are typically in the lab at least two weeks a month, and traveling would cut in to our ability to do other projectsMoney – While our budget easily supports in lab studies, it would have been harder to maneuver travel costsResources – Our team wasn’t large enough to keep other things running while we were in the field conducting this study Lack of clarity between this and the ethnographic interviews that Market Research does in the field andToo big of a departure from our existing methodsSo we took a step back and began figuring out a new solution.
As with everything else, we wanted to start with our most important goals :Something that can and should be done early to inform the very beginning of a projectSomething more observational in nature so that we can see how users actually behaveAnd then consider our restrictions, the afore mentioned time, money, resources, and fear of the vastly unknown.
We were already doing lots of remote studies. We had the logistics pretty much figured out, they were an accepted part of our practice, and they were definitely time, money, and resource-friendly.So, why couldn’t we do REMOTE! Contextual Inquiry
Which we also called Remote Fieldwork
And Remote Observational User StudiesAnd probably a lot of other silly things, but they all meant the same thing:
Observe the user in their natural environment remotely, using the same tools we use for remote usability studies and the same general approach used in contextualinquiry
There were a few things that this solution accomplished that made it the right solution for us, with this project, at this time. It was no more expensive than our average remote study, where we just ad recruiting and incentive costs It gave us the opportunity to see users completing their own tasks, using their own web browser, their own computer, and whatever sites they chose Using our lab, we were able to let many observers see user behavior first-hand Finally, it wasn’t too far a departure from the setup we were used to and that our stakeholders were used to seeing from us, so much less scary fear-inducing for everyone involved.
We were lucky enough to find ourselves in the perfect storm to try something new.We were at the right stage in the right project – Not only did the project happen to exist, but we recognized it for the opportunity it presented us to conduct early UX researchProject stakeholders who liked working with us and trusted us – In retrospect, this was way bigger than I realized at the time, and had been a product of many months of working to build collaborative relationships with product managersThe right attitude – We were prepared to not have a perfect study, if it meant that we could try something new and learn from that experienceWillpower – We just did it! We made room in our schedule to make this happen, and while we kept lots of people informed along the way, we didn’t wait for explicit permission to move forward
So far, we’ve talked about our usability practice and our ambitions, and then the situation we found ourselves in that allowed us to act on those ambitions: the right problem, followed first by the wrong solution, then the right solution, all in the right situation for us to make our move.Any questions about any of that so far?Next up, planning our study.
And then the situation we found ourselves in that allowed us to act on those ambitions: the right problem, followed first by the wrong solution, then the right solution, all in the right situation for us to make our move.Any questions?Next up, planning and conducting our study.
Next up, planning and conducting our study.
Once we decided, in general, what we were doing, we went ahead and did it.In the beginning, it looked a lot like the start of any study looked.First, we needed to figure out what, exactly, we planned to do. The questions that would help inform the project we were working on were:How do people decide on major purchases online?How do they use various interface tools as they come across them?Which design patterns support decision making? Which hurt it?What don’t we know that we don’t know? – A key for us with any type of exploratory study. Be open to new learnings that you never expected.
And then structured a study around answering those questions. We built a test plan around two main “tasks.”To understand what prompts the need to make a decision and how the decision making process proceeds, we conducted a short interview with participants before observing their behaviors. This helped us to understand where they were coming from, what they brought to the decision making process, and the larger scope beyond what we could learn in an hour phone call. While we typically begin all usability studies with an interview, those are focused more on confirming screener questions. For this study, we took a bit more time with the goal of learning a bit more about the user beyond confirming if they were qualified to participate.The bulk of the study would be focused on observing users interacting with the actual interfaces they used to accomplish their goal. We used a lot of the same methods we use in our usability studies: asking the user to think aloud while completing a task and keeping our mouths shut! The only difference was that we asked participants to show us what they had done on whatever sites they used, rather than assigning them discrete tasks to complete on an interface we chose for them. If there was time at the end of the study, we would sometimes direct them to a site that they hadn’t used and may not have been familiar with to see how they would use different types of interfaces to accomplish their goals.
Next, we figured out who can best help us answer these questions. When we do evaluative studies, we are always talking to people who are either buying or selling a car. Finally, a chance to talk to people who aren’t doing something car related! Our big requirement was people who had made major purchase decisions online recently.The really great thing about opening things up is that we would get a chance to see other sites beyond ours and our competitors, which honestly, we knew pretty well.
So if we think about how this study differed from our typical studies,A typical study for us was task-based, where we defined specific tasks for participants to complete,Whereas this study was planned to be much more open-ended.Our typical study looked at a defined interface, usually a new design that was planned for our site,While in this study, we expected to see a wide variety of interfacesMost of our studies were evaluative, where we were hoping to determine how successful a new design was,While this was far more observational, with no goal of evaluating any specific interfaceFinally, we would recruit a very specific audience – new car buyers, used car sellers, etc. – for our typical studies,But for this study, we were open to a variety of different verticalsDespite these differences, both our typical study and this study wereConducted remotelyFrom our labUsing think aloud methodology
Once we knew what we were doing, we had to go through the standard setup steps that we do for every usability study.Recruit: We use external recruiters, and have a good relationship with one. We sent our account manager our detailed screener, and they took care of the rest. (Thank God.) We recruited 12 scheduled participants and 6 floaters. The floaters were expected to be available for two sessions, should we run into cancellations or technical difficulties with a scheduled participant.Write the detailed test plan, and review it with stakeholdersSet up web conferences – we were using GoToMeeting, but any web conference software will workEmail participants with links to our online release form and a link to the web conferenceInvite observers to come down to the lab and see as many sessions as they could – those directly involved attended most, if not all of the sessions, but we had room for others to come and observe as wellOnce all of that was done, we were ready to conduct our study.
Then we did the study! It was all very similar to our standard remote studies once the planning was done and the study was underway.I got comfortable in the lab while our observers and note taker got comfortable in the observation room.In the lab I had two computers. The first was running the web conference and Morae to record the session. The second was for me to take notes and communicate with the note taker in the observation room.
In the observation room, observers could see the computer with the web conference thanks to a big TV screen and could hear me thanks to microphones and, when those failed, conference phones.I also left post-it notes and pens out on the tables so that observers would have an easy way to record things they wanted to remember to share with the rest of the group after the session.
Also in the observation room was another usability analyst who served as my primary note taker. She typed detailed notes, answered questions that came up from the observers, and communicated any questions that the observers had to me via insistent messenger.
When it’s time, I just used the speaker phone to call the participant and followed my detailed test plan.Introduce myself, make sure I am talking to the right personConfirm availability – Can you talk for an hour, are you at a computer with high-speed internet accessReview release – If they haven’t already signed it, get it signed; Explain that it allows me to record the session and use their feedback; As soon as I have the signed release, I begin recording the session using MoraeSet expectations – That I’m going to ask them some questions, and then have them walk me through the process they went through to decide on a recent purchase online by thinking aloud; while the details were novel to me, they were no more novel to my participants than a standard usability study isRemote setup – Make sure they have the link and have clicked it; coach them through any technical problems
Once we were all set up and ready to go, I spent about 10 minutes on the interview.Confirm screener information: I understand that you recently bought an [item.] What did you buy? – Checking to see that they know enough about their purchase to be believableWhen did you buy your new [item]? – Should have completed the purchase within the past monthHow did you decide which [item] to buy? – Expect them to at least mention online sourcesHistory:Why did you decide to buy a new [item]?Would you consider this purchase a want or a need?When you decided to buy a new [item], did you have any ideas about what one you’d chose to buy?Decision making processDid you consult with anyone else (when deciding which [item] to buy)? If so, when? Who did you consult?Priorities: What was more important to you: the [item] or the price?Next steps:How long was it from the time that you decided to buy a new [item] and when you made the purchase?Where did you buy your [item])?After you made your purchase, did you do anything online as a result, like right a review, sign up for email updates, or anything else?
Once all of that was out of the way and I understood where they were approaching the decision making process from, I went to the one big “task” of this study.As with traditional usability studies, I simply let the participant take it from there with as little guidance as necessary. If they needed help getting started, I asked them to think back to the very first thing they did online and start there. I reminded them to think aloud, and stayed quiet as much as possible, though I did ask probing questions as necessary and appropriate.This part took 30 minutes, and we saw all kinds of interesting things, like…
How users started: Did they start with a specific website or retailer in mind? Did they type that URL into their browser, or use a search engine? What did they search for?What they did when they got to a site: How did they navigate to the things they were looking for? Search? Global navigation? Did they get lost?How and why they used specific interface tools: Search, filters, sort, change views, images, compare, etc.When and why they left a site: met goal, gave up, had new goalAnd what they did next: talked to someone, went to a store, went to another websiteAt what point they had made their decision
The way we wrap up our studies is short and sweet.Thank the participant for their timeClose the web conference, confirm that it’s closed on their end and that I will never be able to see their screen againConfirm when and how they will be paidBid them adieu
Once the session is done, I went back to the observation room to discuss what we saw with everyone.Observers got a chance to share any notes they had taken and discuss their observations with me and with each other.Given the nature of this study, there tended to be very robust discussions after each session that were very different in nature from what’s discussed during task-based usability tests. Specifically, we talked more general observations – what we saw that participant to and why they did it – and less about discrete findings, where the goal is to provide a clear example of how a specific interface element doesn’t work and needs to be changed.In total, we did that 12 times, with participants who had made a decision in one of our five verticals
Now we’ve talked about how we planned for the study, from the goals, general approach, and participants to the nuts and bolts of recruiting, plan writing, and setup.
As well as the details of how the study was conducted, form the setup, to the interview and observation, and finally the closing and the observer debrief.Any questions about any of that so far?
Next, I’ll talk about how we communicated the study findings.
When we were done, it was time to figure out how to communicate our findings.For the vast majority of our studies, we do very detailed findings reports that take the screens that we tested and enumerate the findings on each one. Due to the breadth and unfocused nature of this study, that approach wasn’t appropriate.I was honestly unsure of the best way to document findings. This study was entirely about actually observing user behavior, which I was struggling with how to communicate in a readable report.
I decided instead to conduct an analysis workshop with the stakeholders – product manager, designers – all of whom had seen a majority of, if not all of, the sessions and were familiar with the general goals of user observation.
I structured the workshop around five questions:What are users trying to do? – Get a lay of the land? Pick a specific product? Figure out where to go to make the purchase?What information do they need to do that? – Specs, pictures, retail locations?How do they start? – Google? Specific manufacturer? Specific retailer?When are they done? – They met their goal; they gave up; they ran into new goals that can’t be done onlineHow do they use different tools and interfaces?
While going through those questions, the discussion ended up focusing on:Key findings, as intended, specifically how users did and didn’t use specific interface elementsUnanswered questions, such as how frequency of certain behaviors we saw and the divers behind that different behavior (audience segment, drivers, level of comfort online, etc.)Next steps – quantitative research to determine how prevalent some of our observations are; qualitative research to evaluate design iterations
We also discussed if formal documentation was important for this study. The stakeholders agreed that the most value was had by observing the sessions directly, but they did state that having a formal documentation of key findings and planned actions would help later on as it became time to defend the designs.But as with many things we say that we’ll do later, it never got done, lost in a sea of other, more urgent work. This is one of my biggest regrets about this study. Even though I was unsure of how to document such rich information, I wish I had just taken a stab at it. Like everything else, the format could have been evolved over time.
Any questions about our analysis workshop and our lack of a report?
Any questions about our analysis workshop and our lack of a report?Next, I’ll share how this study was received at its conclusion and the long-term benefits our team saw as a result of trying something new.Finally, I’ll share some tips on taking the leap and trying something new in your own practice.
After we discussed findings and next steps, I took a few minutes to discuss what worked and didn’t work about this approach with the stakeholders.What we heard:The designers felt that this was the right research at the right time: worthwhile to actually see users interact with different interface elements, and early enough to impact the very earliest designsEveryone liked that we looked at different verticals and saw websites that we typically don’t evaluate; in fact, some people wished we had cast our net wider, looking at other verticals and smaller purchasesProduct managers especially wished they had been more dedicated to observing the sessions, rather than having the opportunity to multi-task and attend other meetings.First-hand observation was keyThis was the first time that having our own lab was a hindrance, because it made it too easy for observers to do other thingsWhen this is done in the future, wants them to last longer and possibly happen off site, even if still conducted remotely
While the positive reception after the study was great, the real benefits our team saw were long term.Proved that we have value beyond task-based, evaluative usabilityEstablished us as an active part of the project team – attended bi-weekly team meetings, providing design feedback throughout and staying in the loopBoth of which allowed us the opportunity to propose and conduct other new types of research
All in all, despite making some mistakes (no findings report!) and doing some guessing, we have no regrets about trying something new.You may think that’s fine and dandy, but it just won’t work for you. That’s what we thought, too. And then we ignored that thought and did it anyway.You have more power than you think you do, soJust do it! Try something new! If you have good relationships with your stakeholders, they will forgive you for moving full speed ahead on something new, even if they don’t quite understand what’s happening in the beginningTo get buy-in, work within the confines you can – for us, that was controlling costs and making this study not too different in setup from other studies we doBut ignore the confines you can’t – or are unwilling – to work with, like the false believe that change isn’t possibleSet proper expectations – “Let’s try this. If it doesn’t work, we won’t do it again.” Position it as an experiment, and build the experiment in such a way that there is little real riskFinally, learn. Not just from the study itself, but from the experience of doing the study. Evolve based on those learnings.
I have shared how we not only got positive feedback coming out of the study, but how the experience propelled our practice to get to try new things.
I’ve also hopefully given you some inspiration to try some new things yourself, even if you feel like it’s not possible
It looks like we have [x] minutes or so for questions. Any questions about the reception, benefits, how to get started, or anything else we’ve discussed today?Thank you!