[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

•

0 likes•5,608 views

A/B testing is a great way to really learn from user behavior, but this is mostly not the main objective of an optimization program. It needs to improve revenue! If you're only implementing significant results, you're leaving a lot of money on the table. We will explain how we run optimization at the biggest hotel chain in the Netherlands.

Marketing

SUBTITLE BELOW
Optimize for €€€
HOW WE OPTIMIZE AT THE BIGGEST HOTEL CHAIN IN THE NETHERLANDS

@AM_Klaassen / @tomvdberg1
How we got here…

@AM_Klaassen / @tomvdberg1
The quest for female speakers…

@AM_Klaassen / @tomvdberg1
Female speakers…
?

@AM_Klaassen / @tomvdberg1
Our lovely colleagues

@AM_Klaassen / @tomvdberg1
Conversion rate optimization
Analytics Psychology

@AM_Klaassen / @tomvdberg1
Adding
direct value
Learning
user behavior

@AM_Klaassen / @tomvdberg1
Van der Valk
 Unique website for each hotel
 1 centralized team
 Paid by % of their turnover

@AM_Klaassen / @tomvdberg1
Optimization of Valk
 Increase revenue through CRO
 Reduce costs per experiment

“We’re not in the business of
science, we’re in the business
of making money”

@AM_Klaassen / @tomvdberg1
Jorrin Quest
GTM testing
http://gtmtesting.com

@AM_Klaassen / @tomvdberg1
GTM testing
 Easy setup
 Free
 Preview mode

@AM_Klaassen / @tomvdberg1
Focus on finding proof

@AM_Klaassen / @tomvdberg1
Example test: Order flow

@AM_Klaassen / @tomvdberg1
Example test: Set completion
A B

@AM_Klaassen / @tomvdberg1
• No effect on conversion
 Don’t implement
 Re-test with higher volumes
Conclusion

@AM_Klaassen / @tomvdberg1
What’s the alternative?
Frequentist
statistics
Bayesian
statistics

@AM_Klaassen / @tomvdberg1
Bayesian Test evaluation

@AM_Klaassen / @tomvdberg1
Focus on risk assessment
88,4%
A test result is the probability that B outperforms A:
ranging from 0% - 100%

@AM_Klaassen / @tomvdberg1
IMPLEMENT B PROBABILITY * EFFECT ON REVENU
Expected risk 11,6% - € 69.791
Expected uplift 88,4% € 213.530
Contribution € 180.733
* Based on 6 months and an average order value of € 175
Make a risk assessment

@AM_Klaassen / @tomvdberg1
Implement B
B

@AM_Klaassen / @tomvdberg1
abtestguide.com/bayesian/
Roy Schieving
Annemarie Klaassen

@AM_Klaassen / @tomvdberg1
Conclusion Bayesian
 Easier to understand and
communicate
 Better suits the business
 Don’t throw away good ideas
(indicatively significant)
 Higher implementation rate and
revenue
€ 0
€ 1.000.000
€ 2.000.000
€ 3.000.000
€ 4.000.000
€ 5.000.000
€ 6.000.000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Expected impact on revenue in 1 year
€ Frequentist € Bayesian

@AM_Klaassen / @tomvdberg1
50%
50%
A/B-test

@AM_Klaassen / @tomvdberg1
Traffic allocation over time
A
B
Traffic
Time
Profit or loss?

@AM_Klaassen / @tomvdberg1
What’s the alternative?
A/B-test Bandits

If you have multiple options:
which bandit
will pay-out the
most?

@AM_Klaassen / @tomvdberg1
Explore Exploit
?
Explore / Exploit dilemma

@AM_Klaassen / @tomvdberg1
Bandit
80%
20%
10%
90%

@AM_Klaassen / @tomvdberg1
Traffic allocation over time
A
B
Traffic
Time
More Profit!

@AM_Klaassen / @tomvdberg1
Limited time frame Automation
When use Bandits?
Earn over learn

@AM_Klaassen / @tomvdberg1
Example 1
 Older target group
 Comparing price on
other sites
 Behavior of others

@AM_Klaassen / @tomvdberg1
Example 1
Control No message
Variation B Copy default
Variaton C
Variation D

@AM_Klaassen / @tomvdberg1
Example 1 – Effect on revenue
A
B
Traffic
Time
+ € 6.637

@AM_Klaassen / @tomvdberg1
Example 2
Control New control
Variation B Copy control
Variation C
Variation D

@AM_Klaassen / @tomvdberg1
Example 2 – Effect on revenue
A
B
Traffic
Time
+ € 19.862

@AM_Klaassen / @tomvdberg1
Conclusion Bandits
 Less regret during testing
 No implementation costs
 Optimizing per user segment

1. Free test software
2. Bayesian statistics
3. Bandit testing

TÄNAN VÄGA!
Bayesian calculator: abtestguide.com/bayesian/
Slide deck: ondi.me/elitecamp2016
@AM_Klaassen
annemarie@onlinedialogue.com
nl.linkedin.com/in/amklaassen
@tomvdberg1
tom@onlinedialogue.com
nl.linkedin.com/in/vandenbergtom

So you’ve read the book and you know everything there is to know about CRO but nobody is listening! It doesn’t matter how good you are because without buy in and a data-driven testing culture embedded across your digital organisation you will never reach your full potential. I’ll show you how one of the oldest banks in the world went from a standing start all the way through to testing and optimisation Superstars.

[Elite Camp 2016] Martijn Scheijbeler - Optimization for Content Sites

CXL

[Elite Camp 2016] Johnathan Dane - The PPC Thermometer - Why You Should Care ...

CXL

[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...

CXL

Stating your offer is just not enough. Your copy has to command your prospects’ attention and drive them seamlessly to your call-to-action. You need to drill down into human desire and motivation if you want someone to look at your service and say ‘tell me more!’ Great screenwriting does just that - it encourages readers to turn the page and viewers to stay in their seats. Trained scriptwriter and copywriter Amy Harrison shows you how businesses are already using secrets of the screen to create high-converting landing pages, web content and more. You’ll also take away scriptwriting techniques you can implement in your own copy straight after the session.

[Elite Camp 2016] Chris Mercer - How Top Marketers Are Using Google Tag Mana...

CXL

[Elite Camp 2016] Mogens Møller - How To Compete Against The Big Ones (Zaland...

CXL

You can't compete with Amazon, Zalando, and the biggest ecommerce websites on price, delivery and return terms, large selection, usability, etc. Because of that, some people might consider this a lost game! What else can you compete at? I've done hundreds of user tests and customer interviews on ecommerce websites the last year. I've told users they need e.g. new shoes, a new washing machine, a new computer, new bedding, new toys for their kids etc. and started them at Google. Most of the times, they don't end up buying at the biggest ecommerce websites. They end up buying at smaller niche ecommerce sites. Why? Because these sites truly understand their visitors, and they understand how to motivate them to make a buying decision. In my presentation at Elite Camp I'll reveal how to beat these big websites by understanding how to motivate and upgrade your visitors and by competing on parameters where the big ones suck!

10 Unconventional, Proven, Data-Backed CRO Hacks

HubSpot

So you've changed your copy, rearranged your landing page, shortened your form but still can’t get more conversions? Join the club. It’s time to try something new, something guaranteed to drive more leads. Join this revolutionary on-demand webinar with conversion experts WordStream Founder Larry Kim and HubSpot Psychologist Lanya Olmsted to learn our top 10 new and unconventional CRO hacks backed by psychology principles that can DOUBLE your leads. We’ve run dozens of tests to prove these changes will make a huge impact on your campaigns, and better yet, they’re easy to replicate! You’ll learn amazing strategies like: - Why you should rethink the conversion formula - How to overcome your audience's anxiety over providing personal information on the landing page form - When to throw out your landing page altogether - And much more!

Digital Pulse Summit 2013 - The Road to Delight in the Age of the Customer

ISITE Design is now Connective DX

Let's make testing your new text and design ideas for your landing page as straightforward as possible. Our landing page and tracking template add the last piece to the growth marketing puzzle: A/B Testing with Google Optimize. Follow our step-by-step guide and upgrade your setup. You can show 50% of your visitors your new and hopefully better version of your landing page from here on out. Build-in reports will show you which version has a statistically significant higher conversion rate. Back up your intuition and creativity with data and build up your confidence to make the right choices. You'll be rewarded with lower cost-per-acquisitions and higher customer-lifetime-values, aka happier customers.

[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...

CXL

The conversion optimisation industry is in a mess. Poor practice and the spread of misinformation is rife. The agencies and practitioners who truly understand what intelligent conversion optimisation is and what it looks like are currently part of what can only be described as “the secret society”. Nearly every business claims they are "customer centric", yet reality proves they’re far from it and more often than not, they’re clueless as to what it means. Millions is spent every month on enterprise testing tools which promise the world, only to be used by untrained teams and produce inconclusive results. To add to the problem, there are new shiny toys in the guise of big data & personalisation stealing all the headlines and clamouring for marketing budget to be thrown their way. In his opening keynote talk, Paul will be discussing the struggles that the conversion optimisation industry has faced (and will continue to) and how collectively, the people within the industry can prove that conversion optimisation is the best force for growth available to businesses all around the world.

Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect...

Optimizely

Featuring speakers from Rue La La, Princess Cruises Abby Lokshin, Product Manager, Mobile, Rue La La Abel Ramos, Product Owner, Shopping and Booking Web Scrum Team, Princess Cruises Testing is an iterative process, oftentimes with confusing or inconclusive results. However, every experiment has something to teach you, and all too often, critical information is left unnoticed! Learn how to look for insights hidden behind the numbers and maximize the utility of every experiment you run.

Technical Marketing and Growth Hacking Low Hanging Fruit

David Arnoux . Growth

Growth Hacking, Growth Marketing, Technical Marketing... whatever you want to call it. I presented this deck at 2016's Digital Elite Camp in Tallinn, Estonia. In my talk, I covered the definition of growth hacking, or technical marketing (if you're sick of hearing that word), went through all the attributes that technical marketing is compiled of and shared some low hanging fruit so that your company or startup reaches growth as quickly as possible. Marketers should be held to the same level of quality as developers. That's why we must teach more technical growth skills. Check out growthtribe.io for crash courses, no-bullshit workshops and a free email course.

How to identify your highest impact growth opportunities

Kissmetrics on SlideShare

In many ways, customer engagement has become the most important metric for any modern software business. Not too long ago the software marketers job was to optimize the funnel for impression, traffic and pipeline generation and it usually ended when a sale closed. Corks were popped and parties were thrown. But to a modern software marketer, this old funnel looks like something out of Mad Men. Today the modern marketer or growth hacker needs to optimize the funnel for lifetime value. And it requires a dramatically different process and approach to growing your business. In this webinar, Sean Ellis and Brian Kelly will be showing you how to: Find you North Star Metric, which will help you increase true value to your customers Identify your greatest opportunities for sustainable growth Set up a growth process based on experimentation Analyze your customer behavior and increase customer engagement

Opticon 2015-The Power of Empathy in Driving Conversion

Optimizely

Featuring speakers from FunnyBizz, Elevated Rachman Blake, Co-Founder, FunnyBizz Keith Lovgren, Director of Conversion Rate Optimization, Elevated Talia Wolf, CEO and Founder, Conversioner Moderated by: Cara Harshman, Content Marketing Manager, Optimizely Do your digital experiences convey your unique value proposition? What about creating customer delight with humor and wit? Learn how to cut through the noise and create meaningful audience connections that drive conversions.

Webinar with Aaron Ross, Author of Predictable Revenue and Jim Williams, VP M...

Influitive

PPC Lead Generation Strategies for Quantity, Quality, and Revenue

Utah Digital Marketing Collective

Michelle Morgan Utah Digital Marketing Conference 2019 Learn more at: utahdmc.org Lead generation presents different challenges than Ecommerce. The sale isn’t made in an instant on a website. We have to offer, show, and then deliver on value through a sometimes lengthy buyer journey, all while competing for our audience’s attention against distractions and competitors and also making sure they’re a good fit for our customer base. In this session, we take a step away from the keyword and bidding strategies and focus on the parts of lead generation campaigns that can help drive higher lead counts, better quality leads, and propel users through to customer.

Conversion Tracking Template | Pirate Skills

Pirate Skills

Setting up analytics tools and keeping track of all critical user journey steps can be confusing and often done half-heartedly. The problem with this is that you see less clearly where you are wasting time and money. Maybe even worse, you might miss your next scalable and profitable funnel because your analytics setup wasn't delivering your crucial metrics. You need to tell which campaign had what cost per acquisition and its revenue value. We don't want you to compromise on that, so we build a complete conversion tracking template for you.

905 keynote massey

Rising Media, Inc.

Conversion Hotel 2014: Michael Aagaard (DK) keynote

Webanalisten .nl

Key CRO Metrics to Analyze for Successful Landing Pages

Kissmetrics on SlideShare

In order to have a successful CRO program, you need to focus in on the data behind your landing pages. However, you might find yourself logging into Google Analytics with your heart rate at 250 because you know there are a million metrics you could be analyzing. Which ones do you need to spend your time on? In this new live webinar, Hanapin Marketing’s CRO Manager Samantha Kerr will help you narrow down your focus. She’ll walk you through the key metrics she analyzes when creating landing page optimizations. You’ll learn tips like: Why bounce rate is important to analyze and what it means for your site Why it’s important to look at the engagement metrics for specific users On which areas of your site you should focus your optimizations

How to Build an Effective Facebook Funnel

Utah Digital Marketing Collective

Susan Wenograd Utah Digital Marketing Conference 2019 Learn more at: utahdmc.org When someone discovers a new method to reach an audience, everyone starts doing the same thing and oversaturates the audience. We should not be pounding the bottom of the funnel with the hard sell. We have to change what we have been doing in the past. Learn how to approach creating effective Facebook ad funnels. This session will be filled with actionable content that will help you change what you've been doing for the better.

Opticon 2015-UX That Doesn’t Make You Think: Removing Friction and Cognitive ...

Optimizely

Featuring speakers from Volusion, Optimizely Merritt Aho, Sr. Digital Analyst, Volusion Ryan Lillis, Strategic Consultant, Optimizely When words like ‘intuitive’ and ‘convenient’ mean different things to different people and both are often a far cry from ‘effective’, how do you think about the user experience? How do you develop test hypotheses in a way that sets you up to beat the 20% industry-average win rate? Join for an in-depth look at methods and case studies that will help you get inside consumers’ heads to win the optimization game.

VWO-GetResponse Webinar - 10 Conversion Killers And How to Fight Them

VWO

How To Create High-Converting Landing Pages With Conversion Centered Design [...

Oli Gardner

[Webinar]: eCommerce Insights Generation For a Super Holiday Season

VWO

Slides from the eCommerce marketing webinar where Johann van Tonder from AWA Digital and Sushant Sharma from VWO talk about eCommerce insight generation for the holiday season and how to form a marketing strategy around the same. *Key Takeaways From the Talk* 1- Generating eCommerce insights based on user behavior 2- Using generated insights to plan time-boxed campaigns 3- Data backed case studies with proven success rates

[CXL Live 16] From WTF to Hell Yes - How to Come Up With Copy That Persuades ...

CXL

Conversion optimization for startups

Experiments

Friends of Search 2016 - Building a Bridge to GTD

Wil Reynolds

[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven

CXL

[Elite Camp 2016] Karsten Lund - Master the Moment of Decision

CXL

Learn how accurately measuring attention, emotion & memory can help you discover new ways to optimise your website, landing pages and campaigns. At the end of the day, whether a campaign, website or landing page is successful or not comes down to one thing - A human making a decision. The science behind decision making has been researched by psychologists and neuroscientists for a decade, but only in the recent years has technology, science and business merged interest in validating methods and metrics. With the rise of wearables the foundation for a scalable way to gather valid data on decision making emerged.

What's hot

A/B Testing Blueprint | Pirate Skills

Pirate Skills

[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...

CXL

Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect...

Optimizely

Technical Marketing and Growth Hacking Low Hanging Fruit

David Arnoux . Growth

How to identify your highest impact growth opportunities

Kissmetrics on SlideShare

Opticon 2015-The Power of Empathy in Driving Conversion

Optimizely

Webinar with Aaron Ross, Author of Predictable Revenue and Jim Williams, VP M...

Influitive

PPC Lead Generation Strategies for Quantity, Quality, and Revenue

Utah Digital Marketing Collective

Conversion Tracking Template | Pirate Skills

Pirate Skills

905 keynote massey

Rising Media, Inc.

Conversion Hotel 2014: Michael Aagaard (DK) keynote

Webanalisten .nl

Key CRO Metrics to Analyze for Successful Landing Pages

Kissmetrics on SlideShare

How to Build an Effective Facebook Funnel

Utah Digital Marketing Collective

Opticon 2015-UX That Doesn’t Make You Think: Removing Friction and Cognitive ...

Optimizely

VWO-GetResponse Webinar - 10 Conversion Killers And How to Fight Them

VWO

How To Create High-Converting Landing Pages With Conversion Centered Design [...

Oli Gardner

[Webinar]: eCommerce Insights Generation For a Super Holiday Season

VWO

[CXL Live 16] From WTF to Hell Yes - How to Come Up With Copy That Persuades ...

CXL

Conversion optimization for startups

Experiments

Friends of Search 2016 - Building a Bridge to GTD

Wil Reynolds

What's hot (20)

A/B Testing Blueprint | Pirate Skills

[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...

Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect...

Technical Marketing and Growth Hacking Low Hanging Fruit

How to identify your highest impact growth opportunities

Opticon 2015-The Power of Empathy in Driving Conversion

Webinar with Aaron Ross, Author of Predictable Revenue and Jim Williams, VP M...

PPC Lead Generation Strategies for Quantity, Quality, and Revenue

Conversion Tracking Template | Pirate Skills

905 keynote massey

Conversion Hotel 2014: Michael Aagaard (DK) keynote

Key CRO Metrics to Analyze for Successful Landing Pages

How to Build an Effective Facebook Funnel

Opticon 2015-UX That Doesn’t Make You Think: Removing Friction and Cognitive ...

VWO-GetResponse Webinar - 10 Conversion Killers And How to Fight Them

How To Create High-Converting Landing Pages With Conversion Centered Design [...

[Webinar]: eCommerce Insights Generation For a Super Holiday Season

[CXL Live 16] From WTF to Hell Yes - How to Come Up With Copy That Persuades ...

Conversion optimization for startups

Friends of Search 2016 - Building a Bridge to GTD

Viewers also liked

[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven

CXL

[Elite Camp 2016] Karsten Lund - Master the Moment of Decision

CXL

[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...

CXL

Most optimizers aren’t big fans of the traditional redesign cycle. They promote a continuous optimizing process. And that’s fine. But sometimes websites just suck donkey balls. And a new website is a must. Also: new companies need new websites too. So how can we stop those clients from making shitty websites? Which research methods and tools can you use in which stage of the development cycle? And how do you this?

[Elite Camp 2016] Nilan Peiris Product == People

CXL

Nilan will be talking through how finding its cause helped TransferWise build an evangelical brand and drive product driven growth. TransferWise is the leader in international money transfer transferring over £0,5bn a month, doubling in size every 3 to 4 months mainly through Word of Mouth. Nilan will be talking through how defining its purpose helped TransferWise unlock “NPS driven growth”.

[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...

CXL

[Elite Camp 2016] Stacey MacNaught - Nobody Pays the Bills in "Social Shares"...

CXL

Content marketing has a tendency to be a bit of a "fluffy" term, too frequently measured against things like "shares." Nobody pays the bills in shares. In this talk, Stacey will be talking about content marketing that delivers against tangible marketing goals. Stacey will share tactics her team are using to come up with content ideas that deliver relevant (converting!) traffic and tips for executing the ideas for maximum impact.

[Elite Camp 2016] Craig Sullivan - Elite Camp Summary Session

CXL

[Elite Camp 2016] Yehoshua Coren - Strategic And Tactical Implementation And ...

CXL

Hands on session focused on how to get the most out of Google Analytics. Yehoshua will show you how he approaches implementation of GA's enhanced features, including Custom Dimensions, Custom Metrics, Data Import, and Enhanced Ecommerce. Yehoshua will show you how to access a beautifully crafted data set via custom reports and segments and put it to use. Go home with a list of "things to do tomorrow" and practical knowledge about how to do it!

[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and Tactics

CXL

Social Proof Persuasive Content Framework

Angie Schottmuller

Thinking: People are doing it wrong!

DreamGrow Digital

Lean content: An effective web writing workflow

Ian Lurie

An Executive Summary: Social Media and Information and Communications Technol...

The University of Texas (UTRGV)

Content and Strategy

DreamGrow Digital

Construct a Lean Content Marketing Machine in 7 Steps

Omar Mohout

The fast track to credibility is content that vouches for your expertise. With just eight hours and $135, you can create SEO-optimized content for multiple distribution channels and jumpstart your position as the Go-To Person in your field. Building material for your audience takes a simple seven steps. By outsourcing key elements, you not only increase the appeal -- and the likelihood you capture viewers -- but you also save time. Extra time allows for extra content. Assuming the quality is top notch, the more content you produce, the greater your chance to cast yourself as an expert. In an easy Seven Steps you, too, can construct a Lean Content Building Machine that showcases your ability and converts you into a bona-fide expert.

Branded Interactions - Martin NedbalMartin Nedbal

Viewers also liked (16)

[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven

[Elite Camp 2016] Karsten Lund - Master the Moment of Decision

[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...

[Elite Camp 2016] Nilan Peiris Product == People

[Elite Camp 2016] Marie Polli - Taking Your Customers on a Journey That Match...

[Elite Camp 2016] Stacey MacNaught - Nobody Pays the Bills in "Social Shares"...

[Elite Camp 2016] Craig Sullivan - Elite Camp Summary Session

[Elite Camp 2016] Yehoshua Coren - Strategic And Tactical Implementation And ...

[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and Tactics

Social Proof Persuasive Content Framework

Thinking: People are doing it wrong!

Lean content: An effective web writing workflow

An Executive Summary: Social Media and Information and Communications Technol...

Content and Strategy

Construct a Lean Content Marketing Machine in 7 Steps

Branded Interactions - Martin Nedbal

Similar to [Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

Optimize for money

Annemarie Klaassen

Digital Elite Camp 2016: Optimize for €€€ - Annemarie Klaassen & Tom van den ...

Online Dialogue

Test for business growth

Annemarie Klaassen

Test for business growth - analyzing A/B-test with a Bayesian approach

Online Dialogue

Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...

Craig Sullivan

A summary of my 4 years of A/B and Split testing, with case studies of work, photography guidelines, and key advice on which elements of the page to test for quick wins. I enjoyed giving this talk at the Online Retailer Conference in Sydney, which is a fine place to visit. The presentation c.overs a really good 'pizza' analogy for explaining testing to senior management and budget holders. Then covering how you go about discovering good places to test on your site, and what tools will help you get that data. Lastly, I explore what worked for me in testing, show some examples of how similar our winners are across the globe and then cover some cross channel testing. The last one here is a big growth area and involves optimising contact centres and channels, using the web as a tool. Some interesting work going on here and I show some of ours, as well as new things in the pipeline. There are some great resources attached, including a list of remote user testing services and the best 'guides' I could find on 'Conversion Rate Optimisation'. Hope you enjoyed the talk and thank you Sydney.

The Agile Testing Survival Guide

TechWell

As innovative businesses reduce their time-to-market and shorten release cycles, the need for continuous delivery methods becomes inevitable. Testers are constantly asked to rapidly and reliably deliver comprehensive test results in ever-decreasing or continuous cycles. Meanwhile, testing is butting heads with hard budget, tight time, and limited resource constraints, causing frustration for everyone involved. How can we turn these pains into gains? First, Ingo Phillip shows how to measure test coverage in terms of risk coverage on both the requirement and test case level—and why this matters. Second, Ingo shows how novel test design techniques help us test smarter, not harder, while keeping the entire test case portfolio straightforward. Third, he demonstrates how this effort results in a robust synthetic test data approach with repeatable, stable test execution. Finally, Ingo explains how to enable continuous delivery through continuous testing without drowning in test cases.

Network Downtime: How to Calculate the Business Cost

tvgconsulting

The trouble with Marketing

Jalin Somaiya

What are the odds of making that number risk analysis with crystal ball - O...

p6academy

VC, Funding & Pitching - Startup Talks June 2015

Amazon Web Services

Keynote Ton Wesseling at the Web Analytics Wednesday Copenhagen #wawcph at Se...

Online Dialogue

Ton was asked to talk about things that get him excited as a web analyst looking at conversion rate optimization. He picked 5 things: - The real fun part of web analytics is analyzing how user behavior is changing (analyzing experiments), not creating campaign reports... - Win: inject your website feedback form responses into your analytics and be able to segment behavior based on goals. - Run your experiments with an automated and free GTM / GA / EXCEL results set-up! - To make sure business gets it: apply Bayesian statistics on experiment results, don't report on P values, confidence levels etc. - Bandit algoritms: use www.smartnotifications.com for automated persuasive messaging on your website.

New Product Marketing: What Radiohead Knows... The Art of Pricing & Making a ...

Bryan Cassady

User Research: The Superpower Behind Experimentation Programs | VWO Webinars

VWO

User research is often a side-lined activity within the experimentation space, but it’s crucial for keeping your testing velocity up and your win rates high. Many experimentation and optimization teams fail to make this connection due to a lack of time, resources, knowledge or the siloed structure of their organization. In this talk, Chris will take you through some practical examples of how user research can drive the quantity of ideas, their quality, and originality, which in turn leads to a much more successful overall experimentation program.

Market Testing, The Agile Way

GYK Antler

Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!

Ton Wesseling

Decision Making masterclass - Radu Atanasiu, MSM Romania, April 2019

LauraMSM

Growth for Early Stage Startups

Chandini M Ammineni

Is Growth appropriate for startup of any stage? What must very early stage or pre-product market fit stage startups focus on. Practical tips to acquire the first 1000 customers or potential customers. The right questions to ask to collect qualitative data. About engagement and retention. About Growth Process, Lean Stack/Lean Startup, SCRUM, Growth Mindset, AARRR Funnel, Pirate Metrics, Funnel Optimization.

Growth hacking: 5 growth tactics that you can try right now!

Eveline Smet

Evolving Beyond Analytics - Getting To Meaningful Measurement

The Content Advisory

The Perfect Firm

Rob Nixon

The Perfect Accounting Firm acts differently. It performs differently. Its financial metrics are different. Its people perform differently. Its services and pricing are different. Its marketing and sales are different. Everything is different. After 23 years advising firms across the globe, leading advisor to the Accounting profession, Rob Nixon, has created what he believes is the model of a perfect Accounting firm. For more information go to www.robnixon.com

Similar to [Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths (20)

Optimize for money

Digital Elite Camp 2016: Optimize for €€€ - Annemarie Klaassen & Tom van den ...

Test for business growth

Test for business growth - analyzing A/B-test with a Bayesian approach

Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...

The Agile Testing Survival Guide

Network Downtime: How to Calculate the Business Cost

The trouble with Marketing

What are the odds of making that number risk analysis with crystal ball - O...

VC, Funding & Pitching - Startup Talks June 2015

Keynote Ton Wesseling at the Web Analytics Wednesday Copenhagen #wawcph at Se...

New Product Marketing: What Radiohead Knows... The Art of Pricing & Making a ...

User Research: The Superpower Behind Experimentation Programs | VWO Webinars

Market Testing, The Agile Way

Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!

Decision Making masterclass - Radu Atanasiu, MSM Romania, April 2019

Growth for Early Stage Startups

Growth hacking: 5 growth tactics that you can try right now!

Evolving Beyond Analytics - Getting To Meaningful Measurement

The Perfect Firm

More from CXL

Pauline Marol - From Fantasy to Reality – How to Do AB Testing in a Small Com...

CXL

Jeff Sauer - Attribution is Bullshit

CXL

Sujan Patel - Content as a Growth Machine

CXL

Stephen Pavlovich - Testing Beyond the User Experience

CXL

Nicolas Visiers - User Experience Testing

CXL

Morgan Brown - Building a Growth Organization

CXL

Kristoffer Ewald - Modern Measurement Strategies

CXL

Guido Jansen -How to Involve the Whole Team in Optimization

CXL

Els Aerts - Advanced User Research

CXL

Daria Nepriakhina - Problem-Solution Fit

CXL

Christoph Luetke Schelhowe - Data for Everyone

CXL

Bob Ruffalo - How Impact Used ResearchXL to 3X Conversions

CXL

Andre Morys the magic of combining neuromarketing & data sciences

CXL

How to Execute and A/B Test Cross-Channel Personalization

CXL

According to Forrester Research, 47% of customers report that they’ve engaged in at least one cross-channel interaction in the past 90 days. Yet only 16% of marketers can actually recognize the same visitors and understand their interests and intent across channels - a critical capability required to deliver a consistent multichannel experience. How do you close this customer experience gap, integrate cross-channel into your digital marketing strategy and get ahead of your competitors? How do you run proper A/B tests that lead to meaningful insights, thus improving your personalization efforts? Join Shanelle Mullin of ConversionXL and Andy Zimmerman, CMO of Evergage, to learn how to offer a superior, personalized customer experience and run meaningful, insightful A/B tests.

Changing the Game: Always On Testing with Artificial Intelligence

CXL

A/B testing is an powerful way to improve conversion rates on your site, but only about 1 out of 7 tests are winners. In addition, mature organizations are always looking for ways to squeeze more value out of their optimization programs. Enter Sentient Ascend. They perform massively multivariate tests using the power of artificial intelligence. Join ConversionXL founder, Peep Laja, in his discussion with Sentient's Chief Marketing Officer, Jonathan Epstein.

How Eye Tracking Research Helps You Run Better A/B Tests

CXL

More from CXL (16)

Pauline Marol - From Fantasy to Reality – How to Do AB Testing in a Small Com...

Jeff Sauer - Attribution is Bullshit

Sujan Patel - Content as a Growth Machine

Stephen Pavlovich - Testing Beyond the User Experience

Nicolas Visiers - User Experience Testing

Morgan Brown - Building a Growth Organization

Kristoffer Ewald - Modern Measurement Strategies

Guido Jansen -How to Involve the Whole Team in Optimization

Els Aerts - Advanced User Research

Daria Nepriakhina - Problem-Solution Fit

Christoph Luetke Schelhowe - Data for Everyone

Bob Ruffalo - How Impact Used ResearchXL to 3X Conversions

Andre Morys the magic of combining neuromarketing & data sciences

How to Execute and A/B Test Cross-Channel Personalization

Changing the Game: Always On Testing with Artificial Intelligence

How Eye Tracking Research Helps You Run Better A/B Tests

Recently uploaded

Marketing Management Notes Philip Kotler

LaveenaShanker

Your Path to Profits - The Game-Changing Power of a Marketing OS for Your Bus...

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

Most small businesses struggle to see marketing results. In this session, we will eliminate any confusion about what to do next, solving your marketing problems so your business can thrive. You’ll learn how to create a foundational marketing OS (operating system) based on neuroscience and backed by real-world results. You’ll be taught how to develop deep customer connections, and how to have your CRM dynamically segment and sell at any stage in the customer’s journey. By the end of the session, you’ll remove confusion and chaos and replace it with clarity and confidence for long-term marketing success. Key Takeaways: • Uncover the power of a foundational marketing system that dynamically communicates with prospects and customers on autopilot. • Harness neuroscience and Tribal Alignment to transform your communication strategies, turning potential clients into fans and those fans into loyal customers. • Discover the art of automated segmentation, pinpointing your most lucrative customers and identifying the optimal moments for successful conversions. • Streamline your business with a content production plan that eliminates guesswork, wasted time, and money.

5 Big Bets for 2024 - Jamie A. Lee, Stripes Co

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

My Personal Brand Exploration by Mariano

marianooscos

The New Era Of SEO - How AI Has Changed SEO Forever - Danny Leibrandt

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

In this presentation, Danny Leibrandt explains the impact of AI on SEO and what Google has been doing about it. Learn how to take your SEO game to the next level and win over Google with his new strategy anyone can use. Get actionable steps to rank your name, your business, and your clients on Google - the right way. Key Takeaways: 1. Real content is king 2. Find ways to show EEAT 3. Repurpose across all platforms

Generative AI - Unleash Creative Opportunity - Peter Weltman

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

Adapt or Die - Jon Lakefish, Lakefish Group LLC

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

A.I. (artificial intelligence) platforms are popping up all the time, and many of them can and should be used to help grow your brand, increase your sales and decrease your marketing costs.In this presentation:We will review some of the best AI platforms that are available for you to use.We will interact with some of the platforms in real-time, so attendees can see how they work.We will also look at some current brands that are using AI to help them create marketing messages, saving them time and money in the process. Lastly, we will discuss the pros and cons of using AI in marketing & branding and have a lively conversation that includes comments from the audience. Key Takeaways: Attendees will learn about LLM platforms, like ChatGPT, and how they work, with preset examples and real time interactions with the platform. Attendees will learn about other AI platforms that are creating graphic design elements at the push of a button...pre-set examples and real-time interactions.Attendees will discuss the pros & cons of AI in marketing + branding and share their perspectives with one another. Attendees will learn about the cost savings and the time savings associated with using AI, should they choose to.

The Old Oak - Press Kit - Cannes Film Festival 2023

Pascal Fintoni

Mastering Multi-Touchpoint Content Strategy: Navigate Fragmented User Journeys

Search Engine Journal

Digital platforms are constantly multiplying, and with that, user engagement is becoming more intricate and fragmented. So how do you effectively navigate distributing and tailoring your content across these various touchpoints? Watch this webinar as we dive into the evolving landscape of content strategy tailored for today's fragmented user journeys. Understanding how to deliver your content to your users is more crucial than ever, and we’ll provide actionable tips for navigating these intricate challenges. You’ll learn: - How today’s users engage with content across various channels and devices. - The latest methodologies for identifying and addressing content gaps to keep your content strategy proactive and relevant. - What digital shelf space is and how your content strategy needs to pivot. With Wayne Cichanski, we’ll explore innovative strategies to map out and meet the diverse needs of your audience, ensuring every piece of content resonates and connects, regardless of where or how it is consumed.

Digital Marketing Trends - Experts Insights on How to Gain a Competitive Edge

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

Offissa Dizayn - Otel, Kafe, Restoran Kataloqu_240603_011042.pdf

offisadizayn

Winning local SEO in the Age of AI - Dennis Yu

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

Mastering Local SEO for Service Businesses in the AI Era is tailored specifically for local service providers like plumbers, dentists, and others seeking to dominate their local search landscape. This session delves into leveraging AI advancements to enhance your online visibility and search rankings through the Content Factory model, designed for creating high-impact, SEO-driven content. Discover the Dollar-a-Day advertising strategy, a cost-effective approach to boost your local SEO efforts and attract more customers with minimal investment. Gain practical insights on optimizing your online presence to meet the specific needs of local service seekers, ensuring your business not only appears but stands out in local searches. This concise, action-oriented workshop is your roadmap to navigating the complexities of digital marketing in the AI age, driving more leads, conversions, and ultimately, success for your local service business. Key Takeaways: Embrace AI for Local SEO: Learn to harness the power of AI technologies to optimize your website and content for local search. Understand the pivotal role AI plays in analyzing search trends and consumer behavior, enabling you to tailor your SEO strategies to meet the specific demands of your target local audience. Leverage the Content Factory Model: Discover the step-by-step process of creating SEO-optimized content at scale. This approach ensures a steady stream of high-quality content that engages local customers and boosts your search rankings. Get an action guide on implementing this model, complete with templates and scheduling strategies to maintain a consistent online presence. Maximize ROI with Dollar-a-Day Advertising: Dive into the cost-effective Dollar-a-Day advertising strategy that amplifies your visibility in local searches without breaking the bank. Learn how to strategically allocate your budget across platforms to target potential local customers effectively. The session includes an action guide on setting up, monitoring, and optimizing your ad campaigns to ensure maximum impact with minimal investment.

Your Path to Profits - The Game-Changing Power of a Marketing - Daniel Bussius

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

BLOOM_May2024 (r). Balmer Lawrie Online Monthly Bulletin

BalmerLawrie

How to Run Landing Page Tests On and Off Paid Social Platforms

VWO

Join us for an exclusive webinar featuring Mariate, Alexandra and Nima where we will unveil a comprehensive blueprint for crafting a successful paid media strategy focused on landing page testing.With escalating costs in paid advertising, understanding how to maximize each visitor’s experience is crucial for retention and conversion. This session will dive into the methodologies for executing and analyzing landing page tests within paid social channels, offering a blend of theoretical knowledge and practical insights. The Pearmill team will guide you through the nuances of setting up and managing landing page experiments on paid social platforms. You will learn about the critical rules to follow, the structure of effective tests, optimal conversion duration and budget allocation. The session will also cover data analysis techniques and criteria for graduating landing pages. In the second part of the webinar, Pearmill will explore the use of A/B testing platforms. Discover common pitfalls to avoid in A/B testing and gain insights into analyzing A/B tests results effectively.

Metaverse Marketing in the Generation of the Internet - Eugene Capon

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

A Guide to UK Top Search Engine Optimization

Brand Highlighters

Unleash the power of UK SEO with Brand Highlighters! Our guide delves into the unique search landscape of Britain, equipping you with targeted strategies to dominate UK search engine results. Discover local SEO tactics, keyword magic for UK audiences, and mobile optimization secrets. Get your website seen by the right people and propel your brand to the top of UK searches. To learn more: https://brandhighlighters.co.uk/blog/top-seo-agencies-uk/

The What, Why & How of 3D and AR in Digital Commerce

PushON Ltd

Vladimir Mulhem has over 20 years of experience in commercialising cutting edge creative technology across construction, marketing and retail. Previously the founder and Tech and Innovation Director of Creative Content Works working with the likes of Next, John Lewis and JD Sport, he now helps retailers, brands and agencies solve challenges of applying the emerging technologies 3D, AR, VR and Gen AI to real-world problems. In this webinar, Vladimir will be covering the following topics: Applications of 3D and AR in Digital Commerce, Benefits of 3D and AR, Tools to create, manage and publish 3D and AR in Digital Commerce.

De-risk Your Digital Evolution - Hannah Grap

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

It's another new era of digital and marketers are faced with making big bets on their digital strategy. If you are looking at modernizing your tech stack to support your digital evolution, there are a few can't miss (often overlooked) areas that should be part of every conversation. We'll cover setting your vision, avoiding siloes, adding a democratized approach to data strategy, localization, creating critical governance requirements and more. Attendees will walk away with actions they can take into initiatives they are running today and consider for the future.

May 2024 - VBOUT Partners Meeting Group Session

Vbout.com

Recently uploaded (20)

Marketing Management Notes Philip Kotler

Your Path to Profits - The Game-Changing Power of a Marketing OS for Your Bus...

5 Big Bets for 2024 - Jamie A. Lee, Stripes Co

My Personal Brand Exploration by Mariano

The New Era Of SEO - How AI Has Changed SEO Forever - Danny Leibrandt

Generative AI - Unleash Creative Opportunity - Peter Weltman

Adapt or Die - Jon Lakefish, Lakefish Group LLC

The Old Oak - Press Kit - Cannes Film Festival 2023

Mastering Multi-Touchpoint Content Strategy: Navigate Fragmented User Journeys

Digital Marketing Trends - Experts Insights on How to Gain a Competitive Edge

Offissa Dizayn - Otel, Kafe, Restoran Kataloqu_240603_011042.pdf

Winning local SEO in the Age of AI - Dennis Yu

Your Path to Profits - The Game-Changing Power of a Marketing - Daniel Bussius

BLOOM_May2024 (r). Balmer Lawrie Online Monthly Bulletin

How to Run Landing Page Tests On and Off Paid Social Platforms

Metaverse Marketing in the Generation of the Internet - Eugene Capon

A Guide to UK Top Search Engine Optimization

The What, Why & How of 3D and AR in Digital Commerce

De-risk Your Digital Evolution - Hannah Grap

May 2024 - VBOUT Partners Meeting Group Session

[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

1. SUBTITLE BELOW Optimize for €€€ HOW WE OPTIMIZE AT THE BIGGEST HOTEL CHAIN IN THE NETHERLANDS

2. @AM_Klaassen / @tomvdberg1 How we got here…

3. @AM_Klaassen / @tomvdberg1 The quest for female speakers…

4. @AM_Klaassen / @tomvdberg1 Female speakers… ?

5. @AM_Klaassen / @tomvdberg1 Our lovely colleagues

6. What we do…

7. @AM_Klaassen / @tomvdberg1 Conversion rate optimization Analytics Psychology

8. Lots of A/B-tests

9. @AM_Klaassen / @tomvdberg1 Our clients

10. @AM_Klaassen / @tomvdberg1 Adding direct value Learning user behavior

11. @AM_Klaassen / @tomvdberg1

12. @AM_Klaassen / @tomvdberg1

13. @AM_Klaassen / @tomvdberg1 Van der Valk  Unique website for each hotel  1 centralized team  Paid by % of their turnover

14. @AM_Klaassen / @tomvdberg1 Optimization of Valk  Increase revenue through CRO  Reduce costs per experiment

15. “We’re not in the business of science, we’re in the business of making money”

16. 3 Tactics

17. 1. Free test software

18. @AM_Klaassen / @tomvdberg1 Jorrin Quest GTM testing http://gtmtesting.com

19. @AM_Klaassen / @tomvdberg1 GTM testing  Easy setup  Free  Preview mode

20. 2. Change of statistics

21. @AM_Klaassen / @tomvdberg1 Focus on finding proof

22. @AM_Klaassen / @tomvdberg1 Example test: Order flow

23. @AM_Klaassen / @tomvdberg1 Example test: Set completion A B

24. @AM_Klaassen / @tomvdberg1 Test result

25. @AM_Klaassen / @tomvdberg1 • No effect on conversion  Don’t implement  Re-test with higher volumes Conclusion

26. @AM_Klaassen / @tomvdberg1

27. @AM_Klaassen / @tomvdberg1 What’s the alternative? Frequentist statistics Bayesian statistics

28. @AM_Klaassen / @tomvdberg1 Example test: Set completion A B

29. @AM_Klaassen / @tomvdberg1 Bayesian Test evaluation

30. @AM_Klaassen / @tomvdberg1 Focus on risk assessment 88,4% A test result is the probability that B outperforms A: ranging from 0% - 100%

31. @AM_Klaassen / @tomvdberg1 IMPLEMENT B PROBABILITY * EFFECT ON REVENU Expected risk 11,6% - € 69.791 Expected uplift 88,4% € 213.530 Contribution € 180.733 * Based on 6 months and an average order value of € 175 Make a risk assessment

32. @AM_Klaassen / @tomvdberg1 Implement B B

33. @AM_Klaassen / @tomvdberg1 abtestguide.com/bayesian/ Roy Schieving Annemarie Klaassen

34. @AM_Klaassen / @tomvdberg1 Conclusion Bayesian  Easier to understand and communicate  Better suits the business  Don’t throw away good ideas (indicatively significant)  Higher implementation rate and revenue € 0 € 1.000.000 € 2.000.000 € 3.000.000 € 4.000.000 € 5.000.000 € 6.000.000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 Expected impact on revenue in 1 year € Frequentist € Bayesian

35. 3. Change of test method

36. @AM_Klaassen / @tomvdberg1 50% 50% A/B-test

37. @AM_Klaassen / @tomvdberg1 Traffic allocation over time A B Traffic Time Profit or loss?

38. @AM_Klaassen / @tomvdberg1 What’s the alternative? A/B-test Bandits

39. If you have multiple options: which bandit will pay-out the most?

40. @AM_Klaassen / @tomvdberg1 Explore Exploit ? Explore / Exploit dilemma

41. @AM_Klaassen / @tomvdberg1 Bandit 100%

42. @AM_Klaassen / @tomvdberg1 Bandit 80% 20% 10% 90%

43. @AM_Klaassen / @tomvdberg1 Traffic allocation over time A B Traffic Time More Profit!

44. @AM_Klaassen / @tomvdberg1 Limited time frame Automation When use Bandits? Earn over learn

45. How?

46. @AM_Klaassen / @tomvdberg1 Example 1

47. @AM_Klaassen / @tomvdberg1 Example 1  Older target group  Comparing price on other sites  Behavior of others

48. @AM_Klaassen / @tomvdberg1 Example 1 Control No message Variation B Copy default Variaton C Variation D

49. 11,1% 10,6% 10,4% 10,5%

50. @AM_Klaassen / @tomvdberg1 Example 1 – Effect on revenue A B Traffic Time + € 6.637

51. @AM_Klaassen / @tomvdberg1 Example 2 Control New control Variation B Copy control Variation C Variation D

52. 9,2% 8,9% 7,8% 8,4%

53. @AM_Klaassen / @tomvdberg1 Example 2 – Effect on revenue A B Traffic Time + € 19.862

54. @AM_Klaassen / @tomvdberg1 Conclusion Bandits  Less regret during testing  No implementation costs  Optimizing per user segment

55. 1. Free test software 2. Bayesian statistics 3. Bandit testing

56. revenue Maximize

57. @AM_Klaassen / @tomvdberg1

58. “We’re not in the business of science, we’re in the business of making money”

59. TÄNAN VÄGA! Bayesian calculator: abtestguide.com/bayesian/ Slide deck: ondi.me/elitecamp2016 @AM_Klaassen annemarie@onlinedialogue.com nl.linkedin.com/in/amklaassen @tomvdberg1 tom@onlinedialogue.com nl.linkedin.com/in/vandenbergtom

[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to [Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

Similar to [Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths (20)

More from CXL

More from CXL (16)

Recently uploaded

Recently uploaded (20)

[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths