SlideShare a Scribd company logo
1 of 40
Download to read offline
A formula foran experiment driven culture
Ruben de Boer
Manager Conversie Optimalisatie & UX
ruben.deboer@sanoma.com
www.linkedin.com/in/rgdeboer/
Sanoma Group Bereik
3
80%
4
An experiment driven culture starts with
the first proof of success
5
6
Achieving that first success is not that
easy
7
1. Most companies (68.2%) do not
execute more than four tests per month
8
2. …
Success = chance *frequency
A B
A B
more articles read
Success
Qualitative research
Quantitative research
Design
Coding
QA
Analyse of experiments
Meetings…
etc.
Number of
experiments
= chance *frequency
13
3 winners = 30% * 10 experiments per month
14
4 winners = 20% * 20 experiments per month
Qualitative research
Quantitative research
Design
Coding
QA
Analyse of experiments
Meetings…
etc.
Number of
experiments
Success = chance * frequency
Qualitative research
Quantitative research
Design
Coding
QA
Analyse of experiments
Meetings…
etc.
Number of
experiments
Success = chance * frequency
17
Scenario 1
Hours per week 200
Hours per experiment 40
# experiments p/w 5
# experiments p/y 260
% winners 25%
# winners 65
Value per winner €100K
Revenue uplift €6,5M
18
Scenario 1 2
Hours per week 200 200
Hours per experiment 40 25
# experiments p/w 5 8
# experiments p/y 260 416
% winners 25% 20%
# winners 65 83
Value per winner €100K €100K
Revenue uplift €6,5M €8,3M
Extra revenue € 1,8M
Lower chance, increase frequentie
Success= chance *frequency
• Does the design have to be pixel perfect?
• Does the experiment need beautiful code?
• Do you need hours of data mining?
• Do you really need to attend that meeting, or use that time to setup an experiment?
• Etc.
Success = (chance * frequency)mindset
21
Biggest CRO/ Optimization challengein 2018
22
Bottleneck?
23
First success
24
Chance
Frequency
Google Optimize Free
HTML, CSS, Js course €10,-
Google Analytics Free
Scientific research Free
Website of competition Free
Information from customer support Free
1,5 month Hotjar €30,-
CRO/ optimization course €10 or educational budget
Total €50,-
25
Success = (chance * frequency)mindset
26
27
“Niet lullen maar toetsen”
TomvandenBerg, DiDo#27
28
29
30
31
Data & Performance
Business Intelligence Traffic acquisition Data Science CRO & UX
32
CRO & UX team
33
• Weekly meetings to discuss research findings, hypotheses, tests and
learnings
• Brainstorm sessions for smaller brands
• Rebuild/ redesign checklist
• Document progress and learnings
34
35
A/B testing Personalization Personalization UX lab
A.I.
A.I.
Innovation
New functionalities
Qualitative research
New businessUX/UI designer
Optimization meeting
36
A/B testingSEOData analistHead of
product
Chief editor New functionalities
Qualitative research
37
38
Success = (chance * frequency)mindset
Questions?
www.linkedin.com/in/rgdeboer/
ruben.deboer@sanoma.com
40

More Related Content

Similar to A formula for an experiment driven culture

Разработка эффективной тестовой стратегии, Антон Семенченко
Разработка эффективной тестовой стратегии, Антон СеменченкоРазработка эффективной тестовой стратегии, Антон Семенченко
Разработка эффективной тестовой стратегии, Антон СеменченкоCOMAQA.BY
 
How to Effectively Experiment in PM by LendingTree Sr PM
How to Effectively Experiment in PM by LendingTree Sr PMHow to Effectively Experiment in PM by LendingTree Sr PM
How to Effectively Experiment in PM by LendingTree Sr PMProduct School
 
World Growth Forum Disrupt: Implementing a High-Tempo Experimentation Process...
World Growth Forum Disrupt: Implementing a High-Tempo Experimentation Process...World Growth Forum Disrupt: Implementing a High-Tempo Experimentation Process...
World Growth Forum Disrupt: Implementing a High-Tempo Experimentation Process...Johannes Radig
 
Lessons From the DesignOps Journey of the World’s Largest Travel Site (Eniola...
Lessons From the DesignOps Journey of the World’s Largest Travel Site (Eniola...Lessons From the DesignOps Journey of the World’s Largest Travel Site (Eniola...
Lessons From the DesignOps Journey of the World’s Largest Travel Site (Eniola...Rosenfeld Media
 
Webinar: True Cost of Calling Every Lead
Webinar: True Cost of Calling Every LeadWebinar: True Cost of Calling Every Lead
Webinar: True Cost of Calling Every LeadFliptop
 
Toolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanToolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanUXPA UK
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROUXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROCraig Sullivan
 
What Are the Product & Design Principles by FindMyPast PM
What Are the Product & Design Principles by FindMyPast PMWhat Are the Product & Design Principles by FindMyPast PM
What Are the Product & Design Principles by FindMyPast PMProduct School
 
5 Steps To Measure ROI On Your Data Science Initiatives - Webinar
 5 Steps To Measure ROI On Your Data Science Initiatives - Webinar 5 Steps To Measure ROI On Your Data Science Initiatives - Webinar
5 Steps To Measure ROI On Your Data Science Initiatives - WebinarGramener
 
How to Measure the ROI of User Experience
How to Measure the ROI of User ExperienceHow to Measure the ROI of User Experience
How to Measure the ROI of User ExperienceUserZoom
 
Design research for a quality product
Design research for a quality productDesign research for a quality product
Design research for a quality productCarmen Brion
 
Ryerson Startup School 2015 - Growth Hacking 2015
Ryerson Startup School 2015 - Growth Hacking 2015Ryerson Startup School 2015 - Growth Hacking 2015
Ryerson Startup School 2015 - Growth Hacking 2015Cristian Contreras
 
Key Success Factors in New Product Efforts
Key Success Factors in New Product EffortsKey Success Factors in New Product Efforts
Key Success Factors in New Product EffortsAtul Setlur
 
Lead Tracking Guide | Pirate Skills
Lead Tracking Guide | Pirate SkillsLead Tracking Guide | Pirate Skills
Lead Tracking Guide | Pirate SkillsPirate Skills
 
Navigating Competing Priorities as a PM by Optimizely PM
Navigating Competing Priorities as a PM by Optimizely PMNavigating Competing Priorities as a PM by Optimizely PM
Navigating Competing Priorities as a PM by Optimizely PMProduct School
 
How to Use Data to Drive Product Decisions by PayPal PM
How to Use Data to Drive Product Decisions by PayPal PMHow to Use Data to Drive Product Decisions by PayPal PM
How to Use Data to Drive Product Decisions by PayPal PMProduct School
 
Final waca free trial ppt
Final waca free trial pptFinal waca free trial ppt
Final waca free trial pptArun Mani
 
UX Webinar: Always Be Testing
UX Webinar: Always Be TestingUX Webinar: Always Be Testing
UX Webinar: Always Be TestingCharity Dynamics
 

Similar to A formula for an experiment driven culture (20)

Beta-start Program
Beta-start Program Beta-start Program
Beta-start Program
 
Разработка эффективной тестовой стратегии, Антон Семенченко
Разработка эффективной тестовой стратегии, Антон СеменченкоРазработка эффективной тестовой стратегии, Антон Семенченко
Разработка эффективной тестовой стратегии, Антон Семенченко
 
How to Effectively Experiment in PM by LendingTree Sr PM
How to Effectively Experiment in PM by LendingTree Sr PMHow to Effectively Experiment in PM by LendingTree Sr PM
How to Effectively Experiment in PM by LendingTree Sr PM
 
World Growth Forum Disrupt: Implementing a High-Tempo Experimentation Process...
World Growth Forum Disrupt: Implementing a High-Tempo Experimentation Process...World Growth Forum Disrupt: Implementing a High-Tempo Experimentation Process...
World Growth Forum Disrupt: Implementing a High-Tempo Experimentation Process...
 
Lessons From the DesignOps Journey of the World’s Largest Travel Site (Eniola...
Lessons From the DesignOps Journey of the World’s Largest Travel Site (Eniola...Lessons From the DesignOps Journey of the World’s Largest Travel Site (Eniola...
Lessons From the DesignOps Journey of the World’s Largest Travel Site (Eniola...
 
Webinar: True Cost of Calling Every Lead
Webinar: True Cost of Calling Every LeadWebinar: True Cost of Calling Every Lead
Webinar: True Cost of Calling Every Lead
 
Toolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig SullivanToolkits and tips for UX analytics CRO by Craig Sullivan
Toolkits and tips for UX analytics CRO by Craig Sullivan
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CROUXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
 
What Are the Product & Design Principles by FindMyPast PM
What Are the Product & Design Principles by FindMyPast PMWhat Are the Product & Design Principles by FindMyPast PM
What Are the Product & Design Principles by FindMyPast PM
 
5 Steps To Measure ROI On Your Data Science Initiatives - Webinar
 5 Steps To Measure ROI On Your Data Science Initiatives - Webinar 5 Steps To Measure ROI On Your Data Science Initiatives - Webinar
5 Steps To Measure ROI On Your Data Science Initiatives - Webinar
 
How to Measure the ROI of User Experience
How to Measure the ROI of User ExperienceHow to Measure the ROI of User Experience
How to Measure the ROI of User Experience
 
Design research for a quality product
Design research for a quality productDesign research for a quality product
Design research for a quality product
 
Ryerson Startup School 2015 - Growth Hacking 2015
Ryerson Startup School 2015 - Growth Hacking 2015Ryerson Startup School 2015 - Growth Hacking 2015
Ryerson Startup School 2015 - Growth Hacking 2015
 
Key Success Factors in New Product Efforts
Key Success Factors in New Product EffortsKey Success Factors in New Product Efforts
Key Success Factors in New Product Efforts
 
Lead Tracking Guide | Pirate Skills
Lead Tracking Guide | Pirate SkillsLead Tracking Guide | Pirate Skills
Lead Tracking Guide | Pirate Skills
 
Navigating Competing Priorities as a PM by Optimizely PM
Navigating Competing Priorities as a PM by Optimizely PMNavigating Competing Priorities as a PM by Optimizely PM
Navigating Competing Priorities as a PM by Optimizely PM
 
How to Use Data to Drive Product Decisions by PayPal PM
How to Use Data to Drive Product Decisions by PayPal PMHow to Use Data to Drive Product Decisions by PayPal PM
How to Use Data to Drive Product Decisions by PayPal PM
 
Pitch clinic NW
Pitch clinic NWPitch clinic NW
Pitch clinic NW
 
Final waca free trial ppt
Final waca free trial pptFinal waca free trial ppt
Final waca free trial ppt
 
UX Webinar: Always Be Testing
UX Webinar: Always Be TestingUX Webinar: Always Be Testing
UX Webinar: Always Be Testing
 

More from UXDXConf

Breaking Silos_The Shift from a Software Delivery to a Product Development Mi...
Breaking Silos_The Shift from a Software Delivery to a Product Development Mi...Breaking Silos_The Shift from a Software Delivery to a Product Development Mi...
Breaking Silos_The Shift from a Software Delivery to a Product Development Mi...UXDXConf
 
How Intercom built ‘Fin’, a GPT-4 powered chatbot_Fergal Reid_UXDX_EMEA_2023
How Intercom built ‘Fin’, a GPT-4 powered chatbot_Fergal Reid_UXDX_EMEA_2023How Intercom built ‘Fin’, a GPT-4 powered chatbot_Fergal Reid_UXDX_EMEA_2023
How Intercom built ‘Fin’, a GPT-4 powered chatbot_Fergal Reid_UXDX_EMEA_2023UXDXConf
 
Leveling Up Design Maturity in a Large-Scale Organisation_ Daniel Heaslip_ U...
Leveling Up  Design Maturity in a Large-Scale Organisation_ Daniel Heaslip_ U...Leveling Up  Design Maturity in a Large-Scale Organisation_ Daniel Heaslip_ U...
Leveling Up Design Maturity in a Large-Scale Organisation_ Daniel Heaslip_ U...UXDXConf
 
Continuous-Research_Mike Brown_UXDX_ EMEA_2023
Continuous-Research_Mike Brown_UXDX_ EMEA_2023Continuous-Research_Mike Brown_UXDX_ EMEA_2023
Continuous-Research_Mike Brown_UXDX_ EMEA_2023UXDXConf
 
Crafting Digital Products for Connected Appliances and Other Stories_ Alexis ...
Crafting Digital Products for Connected Appliances and Other Stories_ Alexis ...Crafting Digital Products for Connected Appliances and Other Stories_ Alexis ...
Crafting Digital Products for Connected Appliances and Other Stories_ Alexis ...UXDXConf
 
Integrating AI _King's journey of Technology Transformation_Steven Collins_ U...
Integrating AI _King's journey of Technology Transformation_Steven Collins_ U...Integrating AI _King's journey of Technology Transformation_Steven Collins_ U...
Integrating AI _King's journey of Technology Transformation_Steven Collins_ U...UXDXConf
 
Seamless UX: Invisible Transactions_Sudev Balakrishan_UXDX_EMEA_2023
Seamless UX: Invisible Transactions_Sudev Balakrishan_UXDX_EMEA_2023Seamless UX: Invisible Transactions_Sudev Balakrishan_UXDX_EMEA_2023
Seamless UX: Invisible Transactions_Sudev Balakrishan_UXDX_EMEA_2023UXDXConf
 
Roadmaps Unveiled_ Lessons from successes, failures and the crucial role of s...
Roadmaps Unveiled_ Lessons from successes, failures and the crucial role of s...Roadmaps Unveiled_ Lessons from successes, failures and the crucial role of s...
Roadmaps Unveiled_ Lessons from successes, failures and the crucial role of s...UXDXConf
 
Operational Evolution_ Harnessing Design for Internal Business Transformation...
Operational Evolution_ Harnessing Design for Internal Business Transformation...Operational Evolution_ Harnessing Design for Internal Business Transformation...
Operational Evolution_ Harnessing Design for Internal Business Transformation...UXDXConf
 
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...UXDXConf
 
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023UXDXConf
 
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023UXDXConf
 
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...UXDXConf
 
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023UXDXConf
 
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...UXDXConf
 
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023UXDXConf
 
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023UXDXConf
 
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023UXDXConf
 
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...UXDXConf
 
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023UXDXConf
 

More from UXDXConf (20)

Breaking Silos_The Shift from a Software Delivery to a Product Development Mi...
Breaking Silos_The Shift from a Software Delivery to a Product Development Mi...Breaking Silos_The Shift from a Software Delivery to a Product Development Mi...
Breaking Silos_The Shift from a Software Delivery to a Product Development Mi...
 
How Intercom built ‘Fin’, a GPT-4 powered chatbot_Fergal Reid_UXDX_EMEA_2023
How Intercom built ‘Fin’, a GPT-4 powered chatbot_Fergal Reid_UXDX_EMEA_2023How Intercom built ‘Fin’, a GPT-4 powered chatbot_Fergal Reid_UXDX_EMEA_2023
How Intercom built ‘Fin’, a GPT-4 powered chatbot_Fergal Reid_UXDX_EMEA_2023
 
Leveling Up Design Maturity in a Large-Scale Organisation_ Daniel Heaslip_ U...
Leveling Up  Design Maturity in a Large-Scale Organisation_ Daniel Heaslip_ U...Leveling Up  Design Maturity in a Large-Scale Organisation_ Daniel Heaslip_ U...
Leveling Up Design Maturity in a Large-Scale Organisation_ Daniel Heaslip_ U...
 
Continuous-Research_Mike Brown_UXDX_ EMEA_2023
Continuous-Research_Mike Brown_UXDX_ EMEA_2023Continuous-Research_Mike Brown_UXDX_ EMEA_2023
Continuous-Research_Mike Brown_UXDX_ EMEA_2023
 
Crafting Digital Products for Connected Appliances and Other Stories_ Alexis ...
Crafting Digital Products for Connected Appliances and Other Stories_ Alexis ...Crafting Digital Products for Connected Appliances and Other Stories_ Alexis ...
Crafting Digital Products for Connected Appliances and Other Stories_ Alexis ...
 
Integrating AI _King's journey of Technology Transformation_Steven Collins_ U...
Integrating AI _King's journey of Technology Transformation_Steven Collins_ U...Integrating AI _King's journey of Technology Transformation_Steven Collins_ U...
Integrating AI _King's journey of Technology Transformation_Steven Collins_ U...
 
Seamless UX: Invisible Transactions_Sudev Balakrishan_UXDX_EMEA_2023
Seamless UX: Invisible Transactions_Sudev Balakrishan_UXDX_EMEA_2023Seamless UX: Invisible Transactions_Sudev Balakrishan_UXDX_EMEA_2023
Seamless UX: Invisible Transactions_Sudev Balakrishan_UXDX_EMEA_2023
 
Roadmaps Unveiled_ Lessons from successes, failures and the crucial role of s...
Roadmaps Unveiled_ Lessons from successes, failures and the crucial role of s...Roadmaps Unveiled_ Lessons from successes, failures and the crucial role of s...
Roadmaps Unveiled_ Lessons from successes, failures and the crucial role of s...
 
Operational Evolution_ Harnessing Design for Internal Business Transformation...
Operational Evolution_ Harnessing Design for Internal Business Transformation...Operational Evolution_ Harnessing Design for Internal Business Transformation...
Operational Evolution_ Harnessing Design for Internal Business Transformation...
 
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
 
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
 
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
 
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
 
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
 
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
 
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
 
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
 
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
 
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
 
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
 

Recently uploaded

Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemSafe Software
 
full stack practical assignment msc cs.pdf
full stack practical assignment msc cs.pdffull stack practical assignment msc cs.pdf
full stack practical assignment msc cs.pdfHulkTheDevil
 
Dynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientationDynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientationBuild Intuit
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
Why Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionWhy Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionDEEPRAJ PATHAK
 
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023Joshua Flannery
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfHCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfROWELL MARQUINA
 
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2DianaGray10
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfROWELL MARQUINA
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementDianaGray10
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
Women in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automationWomen in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automationDianaGray10
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...BookNet Canada
 

Recently uploaded (20)

Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
full stack practical assignment msc cs.pdf
full stack practical assignment msc cs.pdffull stack practical assignment msc cs.pdf
full stack practical assignment msc cs.pdf
 
Dynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientationDynamical Context introduction word sensibility orientation
Dynamical Context introduction word sensibility orientation
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
Why Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile EvolutionWhy Agile? - A handbook behind Agile Evolution
Why Agile? - A handbook behind Agile Evolution
 
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
THE STATE OF STARTUP ECOSYSTEM - INDIA x JAPAN 2023
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfHCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
 
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
Efficiencies in RPA with UiPath and CyberArk Technologies - Session 2
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdf
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions management
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
Women in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automationWomen in Automation 2024: Career session - explore career paths in automation
Women in Automation 2024: Career session - explore career paths in automation
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
 

A formula for an experiment driven culture

Editor's Notes

  1. Question: Who has an experiment driven culture? I dare to say NU.nl has an experiment driven culture. We can still make a lot of improvements, but the basis, the culture is definitely there.
  2. Ruben – manager CO & UX Sanoma at the central data & performance department. NU, CRO & U team Free time, own business for courses in CRO and setting up experiments, writing book on psychology and websites and help charity
  3. For those who dont know Sanoma, these are a number of famous brands. In the next 20 minutes tell something about how an experiment driven culture started at Sanoma ad NU, how to get the ball rolling and how we are organized today.
  4. An experiment driven culture starts with the first proof of success. Proof that experimentation contributes to the development of an organization. Without the first proof of success, companies get stuck on a few tests a month or even a year. This is how it started at NU and Sanoma. A few years ago there was one manager who hired a CRO specialist at NU en he had rapid success. NU is a brand of Sanoma which gets noticed, and so did the success and proof that experimentation works. From there on, experimentation became bigger and bigger at Sanoma.
  5. But what is success, what does it look like, and how do you get to the first success? How do you get the ball roling, and keep roling, without hitting obstacles?
  6. I belief that realising the first success is of great importance, and not all companies realize this. Recently report CXL and VWO about status of CO world wide. And that really proofs getting to first success is not that easy Two important conclusions…
  7. With less than 5 tests -> no cultural shit. This is not enough, you won’t get enough proof. Especially realize in most cases a team or at least one dedicated person responsible
  8. This is literally stated in the research Somewhere wrong choice. But we do have lots of traffic and a really fun team to work with. But lets stick to the first point, where companies dont test enough
  9. First success and success of experimentation -> simple formula Success – winning experiments Change to get success – spend as much time as you want. Perfect code, perfect design, research And frequency Higher chance & higher frequency is more success
  10. For those who dont know about experimentation
  11. For those who dont know about experimentation
  12. First success and success of experimentation -> simple formula Success – winning experiments Change to get success – spend as much time as you want. Perfect code, perfect design, research And frequency Higher chance & higher frequency is more success
  13. Must sound very straigh forward, but by being aware of this, we will start spending our time differently and we are more aware of success. For instance: do you really need to attent meeting or set up an experiment
  14. QA at NU
  15. Inspiration section vtwonen
  16. Tom van den Berg – online dialogue
  17. Bit less research, code less perfect, analysis afwards bit shorter. Bit lower chance, more time for experiments Of course also otherway around. No success = spend more time on chance
  18. If you take this into account during every day work, much more succesful tests But I think you can get even more out of this, which will lead to even more success.
  19. Happens with the right mindset. Seen in GH blogs, speak about growth mindset
  20. Growth mindset is ‘I dont fail, I learn’. When others successful = not jealous, inspire. I want to challenge myself
  21. We really need a growth mindset, and see it as a fun challenge. Research shows we have many challenges / frustrations. Another one I often here is IT capacity
  22. Are you creative with bottlenecks? high chance low frequency due to IT capacity. Lower fr for two weeks and learn coding yourself Of course not most fun thing to do, but helps get started and build business case for dedicated front-end developer We can elaborate on this.
  23. For €50,- start experimenting and not dependent on IT or budgets This is how we kickstart a culture and get the ball roling
  24. By being aware of our success, and being creative with this regarding bottlenecks, I truly belief there are many possibilities to start an experiment driven culture
  25. As Tom said at DiDO 27…
  26. Stop talking and start testing Make sure that with the right chance, as many experiments as possible, gather proof and start rolling that ball
  27. That is how it starts, and we are still working with this mindset. So how do we do this today?
  28. Add NU we test almost everything, not just design, but also algoritms, AI, features and new businesses
  29. Not just A/B testing but also in our own usability lab. UX lab is extremely useful to get valuable insights, but also great for marketing and getting colleagues to think more from a user perspective
  30. That doesnt mean it is going well for every brand at Sanoma, but we are trying our best. Two redesign gone wrong – no testing. Step by step we are learning
  31. How? We have a centralized D&P team. See as inhouse agency, with 20 experts helping the top online brands, including NU. Nu gets noticed, what works there can be tested or rolled out to other brands Work together with product teams, performance teams designers and developers, to improve most important KPIs
  32. Wekelijkse meetings, waarbij we testen en inzichten bespreken. Brainstorms voor kleinere titels In EE houden we alles bij en documenteren we alles
  33. Someone in the team responsible for sharing knowlegde with the whole organization: Newletters, cases, workshops, presentations and online lectures at Sanoma Academy Consultation hour every Tuesday
  34. Several people are responsible for setting up experiments. Besides CO and redesign, also personalization AI video. Also web analists and development
  35. Weekly Optimization meeting Occasionally invite others, like chefs
  36. Test results, upcoming tests and innovation newletter Everyone is encouraged to contribute and come up with ideas.
  37. And of course we celebrate our failures big, with campaign and glitters.
  38. In other words, at NU there is a real experiment driven culture. But is all started with one manager who hired one CRO specialist who got to that first success. The first proof that experiments contribute to the success of the organization. And I belief when we are aware of obtaining this success. With the right mindset, we get the ball rolling and in time, create an experiment driven culture.