SlideShare a Scribd company logo
1 of 34
Download to read offline
Test Pilot
Jono Xia - Rolling your own tool: test pilot
Why/How We Built It
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Data is available and understandable to
users, BEFORE they submit it.




    Data in charts     All raw data
Data is published as public resources
- Anonymous and aggregated user data samples




- Explanation on how to use the data
Surprises
Jono Xia - Rolling your own tool: test pilot
80%-20% rule?



More like 80%-7%
Jono Xia - Rolling your own tool: test pilot
20% used only one:
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
92% kept tabs on top
70% kept new menu
New tab: 12% -> 88%
Resulting Changes in
     Firefox UI
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Dealing with Self-
 Selection Bias
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
Jono Xia - Rolling your own tool: test pilot
testpilot.mozillalabs.com
blog.mozilla.com/metrics
heatmap.mozillalabs.com

    @MozTestPilot

More Related Content

More from bolt peters

Ben McAllister - The Science of Good Design: A Dangerous Idea
Ben McAllister - The Science of Good Design: A Dangerous IdeaBen McAllister - The Science of Good Design: A Dangerous Idea
Ben McAllister - The Science of Good Design: A Dangerous Ideabolt peters
 
Sara Summers - Invisible Me: Emotional Intelligence & Observational Research
Sara Summers - Invisible Me: Emotional Intelligence & Observational ResearchSara Summers - Invisible Me: Emotional Intelligence & Observational Research
Sara Summers - Invisible Me: Emotional Intelligence & Observational Researchbolt peters
 
Aaron Irizarry - Bootstrapping User Research
Aaron Irizarry - Bootstrapping User ResearchAaron Irizarry - Bootstrapping User Research
Aaron Irizarry - Bootstrapping User Researchbolt peters
 
Gini Keating - Getting Out There: Research for Mobile Augmented Reality
Gini Keating - Getting Out There: Research for Mobile Augmented RealityGini Keating - Getting Out There: Research for Mobile Augmented Reality
Gini Keating - Getting Out There: Research for Mobile Augmented Realitybolt peters
 
On demand v1 webvisions
On demand v1 webvisionsOn demand v1 webvisions
On demand v1 webvisionsbolt peters
 
Stop Listening to Your Customers
Stop Listening to Your CustomersStop Listening to Your Customers
Stop Listening to Your Customersbolt peters
 
Remote Research, The Talk.
Remote Research, The Talk.Remote Research, The Talk.
Remote Research, The Talk.bolt peters
 
Remote Research at IxD10
Remote Research at IxD10Remote Research at IxD10
Remote Research at IxD10bolt peters
 
Escape The Lab Tri Upa Slides
Escape The Lab Tri Upa SlidesEscape The Lab Tri Upa Slides
Escape The Lab Tri Upa Slidesbolt peters
 
When Design Research is Evil
When Design Research is EvilWhen Design Research is Evil
When Design Research is Evilbolt peters
 
Future of Library User Experience
Future of Library User ExperienceFuture of Library User Experience
Future of Library User Experiencebolt peters
 
Researching Spore: A New Approach to Player Testing
Researching Spore: A New Approach to Player TestingResearching Spore: A New Approach to Player Testing
Researching Spore: A New Approach to Player Testingbolt peters
 

More from bolt peters (14)

Ben McAllister - The Science of Good Design: A Dangerous Idea
Ben McAllister - The Science of Good Design: A Dangerous IdeaBen McAllister - The Science of Good Design: A Dangerous Idea
Ben McAllister - The Science of Good Design: A Dangerous Idea
 
Sara Summers - Invisible Me: Emotional Intelligence & Observational Research
Sara Summers - Invisible Me: Emotional Intelligence & Observational ResearchSara Summers - Invisible Me: Emotional Intelligence & Observational Research
Sara Summers - Invisible Me: Emotional Intelligence & Observational Research
 
Aaron Irizarry - Bootstrapping User Research
Aaron Irizarry - Bootstrapping User ResearchAaron Irizarry - Bootstrapping User Research
Aaron Irizarry - Bootstrapping User Research
 
Gini Keating - Getting Out There: Research for Mobile Augmented Reality
Gini Keating - Getting Out There: Research for Mobile Augmented RealityGini Keating - Getting Out There: Research for Mobile Augmented Reality
Gini Keating - Getting Out There: Research for Mobile Augmented Reality
 
On demand v1 webvisions
On demand v1 webvisionsOn demand v1 webvisions
On demand v1 webvisions
 
Stop Listening to Your Customers
Stop Listening to Your CustomersStop Listening to Your Customers
Stop Listening to Your Customers
 
Remote Research, The Talk.
Remote Research, The Talk.Remote Research, The Talk.
Remote Research, The Talk.
 
ETL May 2010
ETL May 2010ETL May 2010
ETL May 2010
 
Remote Research at IxD10
Remote Research at IxD10Remote Research at IxD10
Remote Research at IxD10
 
Escape The Lab Tri Upa Slides
Escape The Lab Tri Upa SlidesEscape The Lab Tri Upa Slides
Escape The Lab Tri Upa Slides
 
When Design Research is Evil
When Design Research is EvilWhen Design Research is Evil
When Design Research is Evil
 
Future of Library User Experience
Future of Library User ExperienceFuture of Library User Experience
Future of Library User Experience
 
Researching Spore: A New Approach to Player Testing
Researching Spore: A New Approach to Player TestingResearching Spore: A New Approach to Player Testing
Researching Spore: A New Approach to Player Testing
 
In The Moment
In The MomentIn The Moment
In The Moment
 

Recently uploaded

Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 

Recently uploaded (20)

Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 

Jono Xia - Rolling your own tool: test pilot

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n
  34. \n