- Beerwulf aims to create a culture where all decisions are data-driven by making data accessible, coaching teams to use data, and encouraging experimentation.
- Step 1 is to make data easily accessible and insightful through tools that provide the right data in the right context.
- Step 2 is to coach, teach, and train teams to find the best solutions for users by looking at data insights and examples of experiments.
- Step 3 is for everyone at Beerwulf to use data when making decisions by understanding user behavior at different points to test improvements through experiments.
This document discusses optimizing digital marketing return on investment (ROI) through various online channels like blogs, social media, and email. It emphasizes the importance of defining outcomes, valuing different customer actions, avoiding biases when analyzing data, identifying weaknesses in the customer funnel, creating optimized practices through testing, and continually refining processes to drive better results over time. The overall message is that marketing ROI improvement requires a data-driven approach, focus on meaningful metrics, and a long-term process of experimentation and mastery.
The Wall Street Journal - Optimizing MembershipOptimizely
For the past three years, The Wall Street Journal has used experimentation to help create their membership experience. Starting with acquisition and expanding to engagement, product, and retention testing, WSJ has built a data-driven culture that leverages these experiments to deliver a premium experience that grows their base and keeps members coming back for more.
In this webinar, Olivia Simon from The Wall Street Journal’s optimization team shares how WSJ created their robust and successful testing program.
You’ll learn:
How to think about testing in a membership ecosystem
How WSJ prioritizes tests and earns stakeholder buy-in
About WSJ’s emphasis on testing throughout the customer acquisition funnel
How WSJ expanded their testing funnel beyond acquisition, and the key testing moments they target for consumer engagement and retention
Learn how to transform from a mild-mannered online organizer into a true data-driven mastermind! What to track, how to test, and methods for creating a data-driven culture at your nonprofit.
Growth Hacking, Growth Marketing, Technical Marketing... whatever you want to call it. I presented this deck at 2016's Digital Elite Camp in Tallinn, Estonia. In my talk, I covered the definition of growth hacking, or technical marketing (if you're sick of hearing that word), went through all the attributes that technical marketing is compiled of and shared some low hanging fruit so that your company or startup reaches growth as quickly as possible.
Marketers should be held to the same level of quality as developers. That's why we must teach more technical growth skills.
Check out growthtribe.io for crash courses, no-bullshit workshops and a free email course.
Technical Marketing and Growth Hacking Low Hanging FruitGrowth Tribe
Growth Hacking, Growth Marketing, Technical Marketing... whatever you want to call it. This deck was presented by our head of growth, David Arnoux, at 2016's Digital Elite Camp in Tallinn, Estonia. In his talk, David covered the definition of growth hacking, or technical marketing (if you're sick of hearing that word), went through all the attributes that technical marketing is compiled of and shared some low hanging fruit so that your company or startup reaches growth as quickly as possible.
As David says, "Marketers should be held to the same level of quality as developers. That's why we must teach more technical growth skills."
Check out www.growthtribe.io for crash courses, no-bullshit workshops and a free email course.
This document discusses strategies for measuring social media success. It emphasizes the importance of (1) planning measurable executions, (2) ensuring measurements influence strategies, and (3) reporting on successes. It provides tips for setting goals around reach, reaction, and response. It also discusses qualitative and comparative measurement techniques as well as integrating offline and online measurement.
This document discusses strategies for measuring social media success. It emphasizes the importance of (1) planning measurable executions, (2) ensuring measurements influence execution, and (3) reporting on success. It provides tips for setting goals based on reach, reaction, and response metrics. It also discusses qualitative and comparative measurement techniques as well as integrating offline and online measurement.
The Marketer's Blind Spot, presented by Flint McGlaughlin at the Jacksonville Business Journal's Advertising and Marketing Summit 2017.
View the full session as presented at MarketingSherpa Summit 2016: https://www.marketingsherpa.com/video/flint-mcglaughlin-marketers-blind-spot?_ga=1.82094229.220271385.1471291473
This document discusses optimizing digital marketing return on investment (ROI) through various online channels like blogs, social media, and email. It emphasizes the importance of defining outcomes, valuing different customer actions, avoiding biases when analyzing data, identifying weaknesses in the customer funnel, creating optimized practices through testing, and continually refining processes to drive better results over time. The overall message is that marketing ROI improvement requires a data-driven approach, focus on meaningful metrics, and a long-term process of experimentation and mastery.
The Wall Street Journal - Optimizing MembershipOptimizely
For the past three years, The Wall Street Journal has used experimentation to help create their membership experience. Starting with acquisition and expanding to engagement, product, and retention testing, WSJ has built a data-driven culture that leverages these experiments to deliver a premium experience that grows their base and keeps members coming back for more.
In this webinar, Olivia Simon from The Wall Street Journal’s optimization team shares how WSJ created their robust and successful testing program.
You’ll learn:
How to think about testing in a membership ecosystem
How WSJ prioritizes tests and earns stakeholder buy-in
About WSJ’s emphasis on testing throughout the customer acquisition funnel
How WSJ expanded their testing funnel beyond acquisition, and the key testing moments they target for consumer engagement and retention
Learn how to transform from a mild-mannered online organizer into a true data-driven mastermind! What to track, how to test, and methods for creating a data-driven culture at your nonprofit.
Growth Hacking, Growth Marketing, Technical Marketing... whatever you want to call it. I presented this deck at 2016's Digital Elite Camp in Tallinn, Estonia. In my talk, I covered the definition of growth hacking, or technical marketing (if you're sick of hearing that word), went through all the attributes that technical marketing is compiled of and shared some low hanging fruit so that your company or startup reaches growth as quickly as possible.
Marketers should be held to the same level of quality as developers. That's why we must teach more technical growth skills.
Check out growthtribe.io for crash courses, no-bullshit workshops and a free email course.
Technical Marketing and Growth Hacking Low Hanging FruitGrowth Tribe
Growth Hacking, Growth Marketing, Technical Marketing... whatever you want to call it. This deck was presented by our head of growth, David Arnoux, at 2016's Digital Elite Camp in Tallinn, Estonia. In his talk, David covered the definition of growth hacking, or technical marketing (if you're sick of hearing that word), went through all the attributes that technical marketing is compiled of and shared some low hanging fruit so that your company or startup reaches growth as quickly as possible.
As David says, "Marketers should be held to the same level of quality as developers. That's why we must teach more technical growth skills."
Check out www.growthtribe.io for crash courses, no-bullshit workshops and a free email course.
This document discusses strategies for measuring social media success. It emphasizes the importance of (1) planning measurable executions, (2) ensuring measurements influence strategies, and (3) reporting on successes. It provides tips for setting goals around reach, reaction, and response. It also discusses qualitative and comparative measurement techniques as well as integrating offline and online measurement.
This document discusses strategies for measuring social media success. It emphasizes the importance of (1) planning measurable executions, (2) ensuring measurements influence execution, and (3) reporting on success. It provides tips for setting goals based on reach, reaction, and response metrics. It also discusses qualitative and comparative measurement techniques as well as integrating offline and online measurement.
The Marketer's Blind Spot, presented by Flint McGlaughlin at the Jacksonville Business Journal's Advertising and Marketing Summit 2017.
View the full session as presented at MarketingSherpa Summit 2016: https://www.marketingsherpa.com/video/flint-mcglaughlin-marketers-blind-spot?_ga=1.82094229.220271385.1471291473
Agilia Budapest - Driving Business Decisions with Pirate MetricsOri Bendet
Pirate metrics is a customer lifecycle framework used to optimize business processes by measuring key metrics at different stages of the customer journey. It was presented using examples from product betas and a job searching process. The framework involves measuring acquisition, activation, retention, revenue, and referrals. The presentation emphasized starting by reviewing current data, mapping user flows, identifying the biggest drop-off point, and focusing on improving that stage through the "magic moment" which moves users further in the funnel.
Tami Dalley SES Chicago Advanced B2B Search MarketingTamidalley
The document discusses value-based optimization in analytics for B2B marketers. It describes value-based optimization as identifying online user behaviors that correlate with leads, assigning each behavior a value based on its relationship to leads, and using this point system to optimize marketing efforts by focusing on behaviors with the highest points rather than just one key metric like leads. The process involves identifying behavioral indicators, confirming their relationship to leads, assigning values to calculate a point total for each behavior, and then optimizing marketing like PPC campaigns, SEO, and content based on the cost per point of different tactics.
Growth Tribe Academy: Growth Hacking Power Session @ The Next Web 2016Growth Tribe
This document summarizes a presentation on growth hacking. It discusses what growth hacking is, why it is important, and the key skills and techniques involved. Growth hacking focuses on acquiring, activating, and retaining customers through low-cost marketing experiments rather than traditional campaigns. It is data-driven and emphasizes continuous testing and optimization. The presentation covers frameworks for growth, customer acquisition channels, behavioral psychology principles, and ways to improve activation, retention and referrals through techniques like onboarding, habit-forming hooks, and reducing steps to a "wow moment." It emphasizes taking an iterative, experiment-based approach to growth.
Agile metrics can be used to the advantage or the detriment of teams and an organisation’s Agile success. This session looks at several of the core Agile metrics used to measure success to help you understand what success looks like, why the metric is desirable and what the metrics can tell us.
Understanding why we want these metrics is critical to capturing something of value, rather than just doing 'because'. What will leaders and decision makers do with these metrics? What value do they add?
Steve will also dive into the negative impacts of some of the Agile metrics we are sometimes forced to capture, how chasing velocity leads to gaming the system etc. He’ll look at bad metrics such as the seven deadly sins of Agile measurement and how to avoid them in your enterprise.
Meetup 16 dec data driven process to optimizationAndra Baragan
This document outlines the data-driven process for conducting conversion optimization experiments on a website. It discusses determining key performance indicators, gathering insights from data, developing testable hypotheses about ways to improve conversions, prioritizing hypotheses, designing and testing experiments, and learning from the results. The goal of conversion optimization is to better understand customers and provide a better experience to simplify completing desired actions like signups, purchases or downloads. It emphasizes establishing a process, using data to inform hypotheses rather than assumptions, and continuously testing and improving based on results.
Funnels Workshop Web Summit 2014 @geckoboard @GASofia Quintero
The document discusses customer funnels and analytics. It defines a funnel as visualizing a customer's journey from stranger to follower through various stages like acquisition, activation, retention, revenue, and referral. It emphasizes measuring key metrics at each stage of the customer journey. It distinguishes between vanity metrics that don't change behavior and actionable metrics that can help improve the funnel. The document provides resources for testing and optimizing the funnel through techniques like A/B testing and cohort analysis.
Google Analytics Workshop - Steps To Better MeasurementBarry Hand
The document outlines an agenda for a Google Analytics workshop. It will cover what website analytics is and how it works, followed by four steps to better measurement using analytics: 1) measure what matters, 2) proper implementation, 3) setting up goals, events and revenue tracking, and 4) tracking marketing campaigns. The workshop will also address three critical questions analytics can answer and include a Q&A session.
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...Percussion Software
Websites that leverage peer participation and allow visitors to interact socially drive more results. 79% of online retailers reported that consumer-generated rating and reviews improved site conversion rates (eMarketer). And the trend doesn’t affect only consumer-facing businesses: 90% of B2B buyers first turn to the internet, including user-generated content (TechTarget/CMO Council).
Learn:
• How leveraging peer endorsement can lift conversation rates and drive more sales, leads, or revenue.
• Different ways of fostering participation on your website - comments, ratings, reviews, and polls.
• Actionable steps you can take and best practices for implementing community features.
SXSW Workshop on Designing for Behavior Change (2014)Stephen Wendel
Slides from my 2.5 hour SXSW workshop on how to design products to support behavior change among users. The toolkit that accompanies it is up on actiondesign.hellowallet.com.
Hotspot ALPHA Camp_Setting Course with MetricsALPHA Camp
This document discusses the importance of metrics for businesses and provides examples of different types of metrics. It covers:
- Why businesses need metrics to monitor health, make decisions, and set focus.
- Examples of business, product, and marketing metrics like revenue, daily/monthly active users, click-through rate, cost-per-click.
- How to establish success formulas using metrics at the business, product, and marketing levels.
- The concept of a metric framework to define key events, tactics to drive those events, and metrics to measure performance for an AARRR strategy.
- Important early stage metrics like retention rates and cohort analysis, as well as the idea of a "North
Hear first-hand accounts from top marketers as they discuss redesigns, drivers for change, metrics for success, and best practices acquired along the way.
How to Know When Metrics Are Not Right by Pinterest PMProduct School
In this presentation, Will Hamlin, covers the following:
- How to choose the right metric
- How to balance different metrics in opposition to each other
- How to choose the right time horizons for your metrics
- When to use other tools, e.g., human evaluation, to measure instead
- When to trust your gut. This will not be a scientific answer
So you've learned the Results-Based Accountability framework. The next step is to build systems of accountability within the organization? This short course offers the "brass tacks" in building a data collection, presentation and analysis assembly-line with your staff. Michael Moser, from the Vermont State Data Center and Shelagh Cooley from Common Good Vermont provide examples, tools and concrete next steps that you can implement immediately. Watch the video here: http://www.cctv.org/watch-tv/programs/make-data-work-you#
Shek Viswanathan, Product Manager at Qualtrics who previously built Words with Friends at Zynga, explains the product development loop and growth framework. His talk gives examples of north star metrics and other metrics to drive growth.
Why Product Managers Must Relearn Their CustomersAggregage
We are living in unprecedented times that have changed the way we live and work. The impact on businesses cannot be understated, and product managers have felt the brunt of it. So how do you adapt your product development process knowing that your customer's behaviors and expectations have completely changed over the past year? Integrating their feedback is the first step! Join Janelle Estes, Chief Insights Officer, at UserTesting, as she highlights the critical areas during the product design and development process when you need to reach out to customers, understand needs, test hypotheses, and refine your approach through ongoing feedback. We'll provide real-life examples as well as practical tips for pulling in customer feedback without delaying your process.
Growth marketing for corporates - Intro session - ING innovation leadersGrowth Tribe
A recent talk give by Growth Tribe to the heads of innovation at ING... the talk covers growth hacking for corporates, growth marketing for corporates and how growth marketing fits within a digital transformation strategy,
Metrics and Analytics, Guest Lecture, UCLADarren Levy
This document summarizes a guest lecture on analytics and metrics given by Darren Levy. The lecture covered:
1) Definitions of metrics and why they are important for measuring business performance and customer satisfaction.
2) What specific metrics to focus on, including vanity metrics to avoid and key startup metrics like CPA, ARPU, churn, and LTV.
3) Methods for collecting and optimizing metrics, including A/B testing, cohort analysis, and heatmaps.
4) Case studies demonstrating how Dollar Shave Club and Netflix used metrics and analytics to optimize pricing and content.
Creating a culture that provokes failure and boosts improvementBen Dressler
Everyone fails - but not everyone uses failed attempts as a source of learning and improvement. This talk outlines a framework to turn failure into gaining knowledge by understanding IF, HOW and WHY something fails.
This presentation by Professor Giuseppe Colangelo, Jean Monnet Professor of European Innovation Policy, was made during the discussion “The Intersection between Competition and Data Privacy” held at the 143rd meeting of the OECD Competition Committee on 13 June 2024. More papers and presentations on the topic can be found at oe.cd/ibcdp.
This presentation was uploaded with the author’s consent.
More Related Content
Similar to Emerce GAUC 2020 - Lars Harmsen - Beerwulf - Brewing a culture of data-driven decisions
Agilia Budapest - Driving Business Decisions with Pirate MetricsOri Bendet
Pirate metrics is a customer lifecycle framework used to optimize business processes by measuring key metrics at different stages of the customer journey. It was presented using examples from product betas and a job searching process. The framework involves measuring acquisition, activation, retention, revenue, and referrals. The presentation emphasized starting by reviewing current data, mapping user flows, identifying the biggest drop-off point, and focusing on improving that stage through the "magic moment" which moves users further in the funnel.
Tami Dalley SES Chicago Advanced B2B Search MarketingTamidalley
The document discusses value-based optimization in analytics for B2B marketers. It describes value-based optimization as identifying online user behaviors that correlate with leads, assigning each behavior a value based on its relationship to leads, and using this point system to optimize marketing efforts by focusing on behaviors with the highest points rather than just one key metric like leads. The process involves identifying behavioral indicators, confirming their relationship to leads, assigning values to calculate a point total for each behavior, and then optimizing marketing like PPC campaigns, SEO, and content based on the cost per point of different tactics.
Growth Tribe Academy: Growth Hacking Power Session @ The Next Web 2016Growth Tribe
This document summarizes a presentation on growth hacking. It discusses what growth hacking is, why it is important, and the key skills and techniques involved. Growth hacking focuses on acquiring, activating, and retaining customers through low-cost marketing experiments rather than traditional campaigns. It is data-driven and emphasizes continuous testing and optimization. The presentation covers frameworks for growth, customer acquisition channels, behavioral psychology principles, and ways to improve activation, retention and referrals through techniques like onboarding, habit-forming hooks, and reducing steps to a "wow moment." It emphasizes taking an iterative, experiment-based approach to growth.
Agile metrics can be used to the advantage or the detriment of teams and an organisation’s Agile success. This session looks at several of the core Agile metrics used to measure success to help you understand what success looks like, why the metric is desirable and what the metrics can tell us.
Understanding why we want these metrics is critical to capturing something of value, rather than just doing 'because'. What will leaders and decision makers do with these metrics? What value do they add?
Steve will also dive into the negative impacts of some of the Agile metrics we are sometimes forced to capture, how chasing velocity leads to gaming the system etc. He’ll look at bad metrics such as the seven deadly sins of Agile measurement and how to avoid them in your enterprise.
Meetup 16 dec data driven process to optimizationAndra Baragan
This document outlines the data-driven process for conducting conversion optimization experiments on a website. It discusses determining key performance indicators, gathering insights from data, developing testable hypotheses about ways to improve conversions, prioritizing hypotheses, designing and testing experiments, and learning from the results. The goal of conversion optimization is to better understand customers and provide a better experience to simplify completing desired actions like signups, purchases or downloads. It emphasizes establishing a process, using data to inform hypotheses rather than assumptions, and continuously testing and improving based on results.
Funnels Workshop Web Summit 2014 @geckoboard @GASofia Quintero
The document discusses customer funnels and analytics. It defines a funnel as visualizing a customer's journey from stranger to follower through various stages like acquisition, activation, retention, revenue, and referral. It emphasizes measuring key metrics at each stage of the customer journey. It distinguishes between vanity metrics that don't change behavior and actionable metrics that can help improve the funnel. The document provides resources for testing and optimizing the funnel through techniques like A/B testing and cohort analysis.
Google Analytics Workshop - Steps To Better MeasurementBarry Hand
The document outlines an agenda for a Google Analytics workshop. It will cover what website analytics is and how it works, followed by four steps to better measurement using analytics: 1) measure what matters, 2) proper implementation, 3) setting up goals, events and revenue tracking, and 4) tracking marketing campaigns. The workshop will also address three critical questions analytics can answer and include a Q&A session.
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...Percussion Software
Websites that leverage peer participation and allow visitors to interact socially drive more results. 79% of online retailers reported that consumer-generated rating and reviews improved site conversion rates (eMarketer). And the trend doesn’t affect only consumer-facing businesses: 90% of B2B buyers first turn to the internet, including user-generated content (TechTarget/CMO Council).
Learn:
• How leveraging peer endorsement can lift conversation rates and drive more sales, leads, or revenue.
• Different ways of fostering participation on your website - comments, ratings, reviews, and polls.
• Actionable steps you can take and best practices for implementing community features.
SXSW Workshop on Designing for Behavior Change (2014)Stephen Wendel
Slides from my 2.5 hour SXSW workshop on how to design products to support behavior change among users. The toolkit that accompanies it is up on actiondesign.hellowallet.com.
Hotspot ALPHA Camp_Setting Course with MetricsALPHA Camp
This document discusses the importance of metrics for businesses and provides examples of different types of metrics. It covers:
- Why businesses need metrics to monitor health, make decisions, and set focus.
- Examples of business, product, and marketing metrics like revenue, daily/monthly active users, click-through rate, cost-per-click.
- How to establish success formulas using metrics at the business, product, and marketing levels.
- The concept of a metric framework to define key events, tactics to drive those events, and metrics to measure performance for an AARRR strategy.
- Important early stage metrics like retention rates and cohort analysis, as well as the idea of a "North
Hear first-hand accounts from top marketers as they discuss redesigns, drivers for change, metrics for success, and best practices acquired along the way.
How to Know When Metrics Are Not Right by Pinterest PMProduct School
In this presentation, Will Hamlin, covers the following:
- How to choose the right metric
- How to balance different metrics in opposition to each other
- How to choose the right time horizons for your metrics
- When to use other tools, e.g., human evaluation, to measure instead
- When to trust your gut. This will not be a scientific answer
So you've learned the Results-Based Accountability framework. The next step is to build systems of accountability within the organization? This short course offers the "brass tacks" in building a data collection, presentation and analysis assembly-line with your staff. Michael Moser, from the Vermont State Data Center and Shelagh Cooley from Common Good Vermont provide examples, tools and concrete next steps that you can implement immediately. Watch the video here: http://www.cctv.org/watch-tv/programs/make-data-work-you#
Shek Viswanathan, Product Manager at Qualtrics who previously built Words with Friends at Zynga, explains the product development loop and growth framework. His talk gives examples of north star metrics and other metrics to drive growth.
Why Product Managers Must Relearn Their CustomersAggregage
We are living in unprecedented times that have changed the way we live and work. The impact on businesses cannot be understated, and product managers have felt the brunt of it. So how do you adapt your product development process knowing that your customer's behaviors and expectations have completely changed over the past year? Integrating their feedback is the first step! Join Janelle Estes, Chief Insights Officer, at UserTesting, as she highlights the critical areas during the product design and development process when you need to reach out to customers, understand needs, test hypotheses, and refine your approach through ongoing feedback. We'll provide real-life examples as well as practical tips for pulling in customer feedback without delaying your process.
Growth marketing for corporates - Intro session - ING innovation leadersGrowth Tribe
A recent talk give by Growth Tribe to the heads of innovation at ING... the talk covers growth hacking for corporates, growth marketing for corporates and how growth marketing fits within a digital transformation strategy,
Metrics and Analytics, Guest Lecture, UCLADarren Levy
This document summarizes a guest lecture on analytics and metrics given by Darren Levy. The lecture covered:
1) Definitions of metrics and why they are important for measuring business performance and customer satisfaction.
2) What specific metrics to focus on, including vanity metrics to avoid and key startup metrics like CPA, ARPU, churn, and LTV.
3) Methods for collecting and optimizing metrics, including A/B testing, cohort analysis, and heatmaps.
4) Case studies demonstrating how Dollar Shave Club and Netflix used metrics and analytics to optimize pricing and content.
Creating a culture that provokes failure and boosts improvementBen Dressler
Everyone fails - but not everyone uses failed attempts as a source of learning and improvement. This talk outlines a framework to turn failure into gaining knowledge by understanding IF, HOW and WHY something fails.
Similar to Emerce GAUC 2020 - Lars Harmsen - Beerwulf - Brewing a culture of data-driven decisions (20)
This presentation by Professor Giuseppe Colangelo, Jean Monnet Professor of European Innovation Policy, was made during the discussion “The Intersection between Competition and Data Privacy” held at the 143rd meeting of the OECD Competition Committee on 13 June 2024. More papers and presentations on the topic can be found at oe.cd/ibcdp.
This presentation was uploaded with the author’s consent.
This presentation by Tim Capel, Director of the UK Information Commissioner’s Office Legal Service, was made during the discussion “The Intersection between Competition and Data Privacy” held at the 143rd meeting of the OECD Competition Committee on 13 June 2024. More papers and presentations on the topic can be found at oe.cd/ibcdp.
This presentation was uploaded with the author’s consent.
This presentation by OECD, OECD Secretariat, was made during the discussion “The Intersection between Competition and Data Privacy” held at the 143rd meeting of the OECD Competition Committee on 13 June 2024. More papers and presentations on the topic can be found at oe.cd/ibcdp.
This presentation was uploaded with the author’s consent.
The importance of sustainable and efficient computational practices in artificial intelligence (AI) and deep learning has become increasingly critical. This webinar focuses on the intersection of sustainability and AI, highlighting the significance of energy-efficient deep learning, innovative randomization techniques in neural networks, the potential of reservoir computing, and the cutting-edge realm of neuromorphic computing. This webinar aims to connect theoretical knowledge with practical applications and provide insights into how these innovative approaches can lead to more robust, efficient, and environmentally conscious AI systems.
Webinar Speaker: Prof. Claudio Gallicchio, Assistant Professor, University of Pisa
Claudio Gallicchio is an Assistant Professor at the Department of Computer Science of the University of Pisa, Italy. His research involves merging concepts from Deep Learning, Dynamical Systems, and Randomized Neural Systems, and he has co-authored over 100 scientific publications on the subject. He is the founder of the IEEE CIS Task Force on Reservoir Computing, and the co-founder and chair of the IEEE Task Force on Randomization-based Neural Networks and Learning Systems. He is an associate editor of IEEE Transactions on Neural Networks and Learning Systems (TNNLS).
Gamify it until you make it Improving Agile Development and Operations with ...Ben Linders
So many challenges, so little time. While we’re busy developing software and keeping it operational, we also need to sharpen the saw, but how? Gamification can be a way to look at how you’re doing and find out where to improve. It’s a great way to have everyone involved and get the best out of people.
In this presentation, Ben Linders will show how playing games with the DevOps coaching cards can help to explore your current development and deployment (DevOps) practices and decide as a team what to improve or experiment with.
The games that we play are based on an engagement model. Instead of imposing change, the games enable people to pull in ideas for change and apply those in a way that best suits their collective needs.
By playing games, you can learn from each other. Teams can use games, exercises, and coaching cards to discuss values, principles, and practices, and share their experiences and learnings.
Different game formats can be used to share experiences on DevOps principles and practices and explore how they can be applied effectively. This presentation provides an overview of playing formats and will inspire you to come up with your own formats.
1.) Introduction
Our Movement is not new; it is the same as it was for Freedom, Justice, and Equality since we were labeled as slaves. However, this movement at its core must entail economics.
2.) Historical Context
This is the same movement because none of the previous movements, such as boycotts, were ever completed. For some, maybe, but for the most part, it’s just a place to keep your stable until you’re ready to assimilate them into your system. The rest of the crabs are left in the world’s worst parts, begging for scraps.
3.) Economic Empowerment
Our Movement aims to show that it is indeed possible for the less fortunate to establish their economic system. Everyone else – Caucasian, Asian, Mexican, Israeli, Jews, etc. – has their systems, and they all set up and usurp money from the less fortunate. So, the less fortunate buy from every one of them, yet none of them buy from the less fortunate. Moreover, the less fortunate really don’t have anything to sell.
4.) Collaboration with Organizations
Our Movement will demonstrate how organizations such as the National Association for the Advancement of Colored People, National Urban League, Black Lives Matter, and others can assist in creating a much more indestructible Black Wall Street.
5.) Vision for the Future
Our Movement will not settle for less than those who came before us and stopped before the rights were equal. The economy, jobs, healthcare, education, housing, incarceration – everything is unfair, and what isn’t is rigged for the less fortunate to fail, as evidenced in society.
6.) Call to Action
Our movement has started and implemented everything needed for the advancement of the economic system. There are positions for only those who understand the importance of this movement, as failure to address it will continue the degradation of the people deemed less fortunate.
No, this isn’t Noah’s Ark, nor am I a Prophet. I’m just a man who wrote a couple of books, created a magnificent website: http://www.thearkproject.llc, and who truly hopes to try and initiate a truly sustainable economic system for deprived people. We may not all have the same beliefs, but if our methods are tried, tested, and proven, we can come together and help others. My website: http://www.thearkproject.llc is very informative and considerably controversial. Please check it out, and if you are afraid, leave immediately; it’s no place for cowards. The last Prophet said: “Whoever among you sees an evil action, then let him change it with his hand [by taking action]; if he cannot, then with his tongue [by speaking out]; and if he cannot, then, with his heart – and that is the weakest of faith.” [Sahih Muslim] If we all, or even some of us, did this, there would be significant change. We are able to witness it on small and grand scales, for example, from climate control to business partnerships. I encourage, invite, and challenge you all to support me by visiting my website.
• For a full set of 530+ questions. Go to
https://skillcertpro.com/product/servicenow-cis-itsm-exam-questions/
• SkillCertPro offers detailed explanations to each question which helps to understand the concepts better.
• It is recommended to score above 85% in SkillCertPro exams before attempting a real exam.
• SkillCertPro updates exam questions every 2 weeks.
• You will get life time access and life time free updates
• SkillCertPro assures 100% pass guarantee in first attempt.
This presentation by Katharine Kemp, Associate Professor at the Faculty of Law & Justice at UNSW Sydney, was made during the discussion “The Intersection between Competition and Data Privacy” held at the 143rd meeting of the OECD Competition Committee on 13 June 2024. More papers and presentations on the topic can be found at oe.cd/ibcdp.
This presentation was uploaded with the author’s consent.
Emerce GAUC 2020 - Lars Harmsen - Beerwulf - Brewing a culture of data-driven decisions
1.
2. Brewing a culture of
data-driven decisions
Lars Harmsen
Product Owner Experimentation
lars.harmsen@beerwulf.com
3. About Beerwulf
Disruption2: Craft x Digital
x
1955
1957
1959
1961
1963
1965
1967
1969
1971
1973
1975
1977
1979
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
2007
2009
2011
2013
2015
September 2015
4,000+ active breweries
according to the
Brewers Association
1987
H.R. 1337 signed by
Jimmy Carter
legalizes homebrewing
1995
Boston Beer Co.
IPO (Sam Adams)
2005
The last annual drop
in operating breweries
was a decade ago
Publicadmin.
Chemicals
Agriculture
Energy
Construction
Education
Food
Manufacturing
Healthcare
Professionalsvcs
Home&Garden
Insurance
Banking
Logistics
FMCG
Automotive
Mobility
Fashion
Consumer electronics
Media
Travel
Stateofdisruption
Point of disruption journey
14. • The data is not accessible
• It takes too much time
• It doesn’t tell me what I want
• I am not sure what I’m looking at
• I don’t think it is possible to get the data I want
• I don’t need data
• Not aware of potential damage if wrong decision is made
• And many, many more…
Data-driven decision making
Why we weren’t making data-driven decisions
“
”
15. Data-driven decision making
Game plan – The ‘Simple’ Version
1 42 3
Make data easily
accessible & insightful
Coach, teach, train,
& challenge
to use data
Everyone uses data
to make decisions
Beerwulf becomes
#1 beer platform
16. How do we make data accessible
and insightful?
17. Step 1 - Make data easily accessible & insightful
Start with the basics
2017
2017
2017
18. Step 1 - Make data easily accessible & insightful
Start with the basics
Why
What & Where
2017
2017
2017
19. Step 1 - Make data easily accessible & insightful
Your tooling landscape evolves over time
Why
What & Where
202020202019
2019
2019
2018 2017
20. Step 1 - Make data easily accessible & insightful
Your tooling landscape evolves over time
2020
Easy to use &
easy to get insights
Hard to use &
hard to get insights
20202019
2019
2019
2018 2017
21. Step 1 - Make data easily accessible & insightful
Often, the data doesn’t really tell you that much…
Accessible data
22. Step 1 - Make data easily accessible & insightful
Often, the data doesn’t really tell you that much…
Rainbow
effect
Accessible data
23. Step 1 - Make data easily accessible & insightful
Often, the data doesn’t really tell you that much…
Rainbow
effect
Accessible data
24.
25. Step 1 - Make data easily accessible & insightful
Don’t look “closlier”, look smarter!
Accessible
26. Step 1 - Make data easily accessible & insightful
Don’t look “closlier”, look smarter!
Insightful
Rainbow
effect
Valuable
element
Exposure
rate
Conversion
rate
Accessible
27. Step 1 - Make data easily accessible & insightful
Money makes the world go ‘round
Revenue per click
29. Test
variant
Control Experiment
Inconclusive
- No increase in subscriptions
- No impact on conversion rate
- Higher exit rate
- More sessions per user
Step 1 - Make data easily accessible & insightful
Experiment to see the impact on behavior
N = 145.889
30. Step 1 - Make data easily accessible & insightful
Often, the data doesn’t really tell you that much…
- Is this a high click rate?
31. Step 1 - Make data easily accessible & insightful
Data becomes insightful in the right context
Event action % of sessions
Click color selector 7,32%
Click ‘Explore’ 3,20%
Click ‘Add to cart’ 2,31%
32. Step 1 - Make data easily accessible & insightful
Combining insightful data with insightful data
33. Control Experiment
Step 1 - Make data easily accessible & insightful
Experiment to see the impact on behavior
=
34. Control Experiment
Step 1 - Make data easily accessible & insightful
Experiment to see the impact on behavior
+ 3,3%
Add to cart rate
=
Winner!
N = 103.112
36. Having insightful data doesn’t
mean you will always win
But you sure as hell
will win more often
37. Data-driven decision making
Game plan – The ‘Simple’ Version
✓ 42 3
Make data easily
accessible & insightful
Coach, teach,
train, & challenge
to use data
Everyone used data
to make decisions
Beerwulf becomes
#1 beer platform
40. Step 2 - Coach, teach, train, & challenge
What we tell people to look for in data
41. Use data to find
the best solution
for the user
Step 2 - Coach, teach, train, & challenge
What we tell people to look for in data
42. Step 2 - Coach, teach, train, & challenge
How we teach to find the best solutions
Workshops & training
Self study Explain and teach
Share examples (and the data behind it)
Learningbydoing
43. Workshops & training
Contentsquare University Explain and teach
Share examples (and the data behind it)
Learningbydoing
Step 2 - Coach, teach, train, & challenge
How we teach to find the best solutions
Self study
44. Step 2 - Coach, teach, train, & challenge
How to find better solutions for the user..
45. Step 2 - Coach, teach, train, & challenge
How to find better solutions for the user..
46. Step 2 - Coach, teach, train, & challenge
How to find better solutions for the user..
• Rage click rate of 2,35% (3 clicks in <2 seconds)
• Scroll is never ending
• Only 1 or 2 additional images in the scroll
• We are hiding the images from our users
47. Control Experiment
Step 2 - Coach, teach, train, & challenge
Experiment to see the impact on behavior
=
48. Control Experiment
Step 2 - Coach, teach, train, & challenge
Experiment to see the impact on behavior
=
Inconclusive
- No impact on add to cart &
conversion rate
- More thumbnail images viewed
per visitor
- Higher time on page
N = 93.410
49. Step 2 - Coach, teach, train, & challenge
Providing the right information at the right time
Journey: Step 1 Step 2 Step 1
50. Control Experiment
Step 2 - Coach, teach, train, & challenge
Providing the right information at the right time
51. Decrease of 122% in Step 1 > 2 > 1 behavior
Step 2 - Coach, teach, train, & challenge
Providing the right information at the right time
Journey Group % Δ
Step 1 > Step 2 > Step 1
Control 6,34%
-122%
Variant 2,86%
Control Experiment
+ 4,02%
conversion rate
N = 64.603
52. Step 2 - Coach, teach, train, & challenge
What we tell people to be cautious ofUse data to find
the wrong solution
for the user
53. Step 2 - Coach, teach, train, & challenge
“We think that blog is clearer and better!”
Control Experiment
54. Control Experiment
Step 2 - Coach, teach, train, & challenge
“We think that blog is clearer and better!”
55. Control Experiment
The use of the word blog
decreased clicks on the menu
item by 51%
& decreased conversion rate by
15% for those who clicked
Step 2 - Coach, teach, train, & challenge
“We think that blog is clearer and better!”
N = 47.199
56. Data-driven decision making
Game plan – The ‘Simple’ Version
✓ 4✓ 3
Make data easily
accessible & insightful
Coach, teach, train,
& challenge
to use data
Everyone uses data
to make decisions
Beerwulf becomes
#1 beer platform
57. Data-driven decision making
Game plan – The ‘Simple’ Version
✓ 4✓ 3
Make data easily
accessible & insightful
Coach, teach, train,
& challenge
to use data
Everyone uses data
to make decisions
Beerwulf becomes
#1 beer platform
58. Step 3 - Everyone uses data to make decisions
Understanding behavior where it matters most
59. Step 3 - Everyone uses data to make decisions
Understanding behavior where it matters most
60. Step 3 - Everyone uses data to make decisions
Understanding behavior
Step 1
61. Step 3 - Everyone uses data to make decisions
Understanding behavior
Step 1 Step 2 Step 3
62. Step 3 - Everyone uses data to make decisions
Understanding behavior
63. Step 3 - Everyone uses data to make decisions
Experiment to see the impact on behavior
Control Experiment
64. Step 3 - Everyone uses data to make decisions
Experiment to see the impact on behavior
Page Group Pageviews Δ
Step 1
Control 5,01%
-156%
Variant 1,96%
Step 2
Control 2,15%
-357%
Variant 0,47%
Step 3
Control 3,32%
-121%
Variant 1,50%
Control Experiment
View homepage directly after checkout page…
65. Winner
• Effect on behavior visible in all steps
• Decreased ‘Checkout > Homepage’
behavior by 169% (from 10,5% to 3,9%)
• Conversion rate increased by 0,75% (from
69,5% to 70,1%)
Step 3 - Everyone uses data to make decisions
Experiment to see the impact on behavior
Control Experiment
+ 0,75%
Conversion rate
N = 84.764
66. Step 3 - Everyone uses data to make decisions
Data helps you to solve your business challenges
67. Step 3 - Everyone uses data to make decisions
When it all comes together
Category
manager
UX
analyst
UX designer Experiment
lead
Company
wide
“14,6% of users
order full boxes
of one beer”
“12,8% of users
click add to cart
several times”
“15% of users
wants to order
in bulk”
“2 experiments
showed positive
impact on CTA”
Free delivery on
full box, saves
time and money
68. Step 3 - Everyone uses data to make decisions
Experiment with different solutions
69. Orders
(96,5% probability)
+5,72%
9,23% ➝ 9,75%
Add to cart
(99,2% probability)
+5,88%
30,83% ➝ 32,65%
Avg. order
quantity
+ 6,4%
2,04 ➝ 2,17
Add box to
cart button
7,25%
Revenue
per visitor
+ 6,7%
↧
Avg. order
value
+ 2,6%
Step 3 - Everyone uses data to make decisions
Improve for both Beerwulf and the user
Cost of running
the test
€ 1890
ROI on test
33656%
72. • Have an amazing team of talented people
• Have insightful data guiding us
• Know how to make the right decisions
• Know how to prevent the wrong decisions
• Learn from our mistakes and take action
• Focus on making our customers happy!
Step 3 - Everyone uses data to make decisions
Doing the right thing
73.
74. ✓ ✓✓ ✓
Make data easily
accessible & insightful
Coach, teach, train,
& challenge
to use data
Everyone uses data
to make decisions
Beerwulf becomes
#1 beer platform
Step 4
Beerwulf will become THE #1 beer platform
76. Bedankt voor je aandacht!
Ga naar de Emerce GAUC-event app
of naar glsr.live/GAUC2020
en laat weten wat je van de sessie vond
Editor's Notes
Almost 3 years ago, on March 2nd, 2017. Beerwulf was born. And we went live in the Netherlands.
Starting off with a team of six, the first orders started rolling in.
Now, with a team of 75, we are active in 11 countries.
In 2019 we sold over xxx million euros worth of beer.
Today I’m going to tell you the story how we achieved all this in only 3 years, using data-driven decision making.
I will show you how you can get quality insights with a very basic setup of GA in combination with other tools…
The initial idea for Beerwulf started several years ago,
When Heineken recognized 2 disruptions in the market
Beerwulf was set up as a corporate venture
Organized for disruption
Sole investor Heineken
Trying to be as independent as possible
Our goal as a company is to…
Ambition is to become the #1 beer platform in Europe!
But how to achieve this?...
We asked ourselves..
..
and it didnt take us very long to realize that its data…
It’s that simple
They know how to improve on and profit from measuring every single thing
Knowing that data = the key to growth,
And wanting to become the Amazon of beer
We said to ourselves..
..
And nowadays…
Back in 2017, small team
…smart, talented, people. ..
All specialists in their respective fields.
Some not used to using data.
So, back in 2017, when we told people on the team that data is super important, it can make their lives super easy, and businesses that use data are more successful, their response often is something like…
And then we asked them: “So.. you use data to make decisions on a daily basis?”
And this was the response we got…
So we know how important data is. BUT
We don’t actually use it to make decisions
Why is that?...
And there were lots of reasons…
…
...To overcome these obstacles
we made a GAME PLAN…
3 simple steps to world domination
Sounds simple, right!?
Lets look at step 1
What actually is accessible and insightful data?
Well, we started off using the basics: GA and Hotjar…
Because…
Google Analytics is going to tell you the what and where,
And Hotjar will help you uncover the why. (next to usability testing)
And this is a very basic setup, but as I will show you in a lot of examples later on, this will help you to uncover plenty of insights already.
And what you’ll see is that…
…your tooling landscape will evolve over time..
Once your team grows and gets more experienced and skilled, their data needs change
But even in later stages of maturity...
…it is important to keep in mind that the ease of use,
And the ease of getting insights = higher ADOPTION rate
But the real power lies in combining the insights from all tools
Let me show you a couple of examples…
The behavior flow, personally I feel its very difficult to get good insights from it:
The flow I am observing here:
Is that GOOD or BAD?
I cannot put a PRICE on it.
Same goes for heatmaps
…
BUT…
Is that GOOD or BAD?
I cannot really QUANTIFY it.
I cannot put a PRICE on it.
And we can enrich it with scroll tracking data
But still its not very insightful
And at this point, I try to crawl into my monitor, to see the insight...
But no matter how closely I look, I cannot see it.
Its really hard to get a true insight from data that is not insightful.
So…
Instead of looking closlier,
We should look smarter
Insightful data tells you more
...
And as you all know...
‘money’ makes it tangible
helps to convince people of the importance
So we did an experiment…
…
That must be big win, right?...
Well it was not…
…
Let’s take a look at another example…
And again, often, the data doesn’t really tell you that much…
Is this a high click rate?
I cannot really tell.
But…
After adding click events for interactions with these elements…
We said: YES! Its the highest click rate on the page, it must be an important element!
Maybe...
...its worth it to take a look into the user behavior with this element?
...
That’s super frustrating right?
And if this would happen to me, I would probably do something like this...
Because it is the worst user experience!
So to fix this user experience, we did a very simple experiment…
And it was a big winner.
We improved the user experience and the add to cart.
These examples show us that..
…
Lets go back to the game plan.
Once your tooling is set up right
the data is accessible and insightful
How do we get people to actually use it?
I'm sure you all know this proverb..
But assuming you are not actually fishermen teaching people to fish all day long…
I’ve tweaked it a little to make it more applicable to the work we do on a day to day basis...
…
That's what you want, isnt it?
You don't want to provide all teams with all insights all the time. Thats not scalable..
You want people to use data themselves to find insights...
> 1st thing we teach
Look for the user experience in all parts of the organization.
Instead of looking at what we have designed ourselves,
look at what the user is trying to tell you with his behavior…
And use that user behavior to find the best solution.
But how to teach?...
We believe...
...
And its important to remember...
To repeat endlessly
To get everyone involved
To show your colleagues the added value
For example...
When we were analyzing the behavior on our beer tap PDP in Google Analytics, we saw…
that this element here had a high number of rage clicks.
What we saw is that 2,35% of users rage clicked on this element
…
And of course we did an experiment…
To see what happens if we show all thumbnails right away
...
Another example…
Another example of how we try to find better solutions
When looking at behavior in the navigation summary, we saw a looping behavior
...
Why would they go back, when the only thing they see in step 2 are the payment methods?
WAIT A MINUTE...
So we did an experiment, showing the payment methods in step 1 of the checkout...
....
And thus we had found a better solution for our users..
But decision making is not only about making the right decisions and improving the user experience
There's a 2nd thing we teach
...
So the 2nd thing we teach in Beerwulf,
is to find the wrong solution for the user and prevent that it will be implemented...
To show you an on-site example for this..
...
...
But when the experiment was done… this was me
And of course that wasnt a great response from my end, but..
what we saw in the data is that…
...
With this example,
We prevented the wrong decision being made, thanks to data.
Time to go back to the game plan...
Data is insightful, we have coached and taught
So now everyone will use data to make decisions…
This is where we are now
You see it in the maturity of the data landscape, requests & data analyses
I’ll explain this by showing you two final examples..
First, we were trying to understand user behavior in the checkout, where it matters most.
Because what we saw…
…
And we saw the same thing in step 2 and 3
First assumption was that people click the logo to go back to home…
So when we looked at the clickrate in this step, we saw that they weren't clicking on the logo, but on the back button
And the same thing happened in step 2 and 3
If you look at it from a user perspectiven what would you expect to happen when clicking this arrow?
Have you ever seen a back arrow on the internet, that was NOT sending you to the previous page?
...
So what did we do?
Did an experiment, of course!
We just removed the back button, a super small change on screen
But a super big change in behavior...
The user behavior changed completely...
...
You can probably guess what the outcome of this experiment was…
…
This shows us once again that
The smallest change on screen
Can make the biggest change in behavior
Final example
And at this stage, data is really helping us solve business challenges
One of the challenges = logistics
We can’t be customer centric
But by bringing all the insights from all the tools together..
And by interpreting the data correctly...
We came up with an experiment, to find a better solution for the user,
where we let them add a full box with 1 click...
Which resulted in improving the user experience
AND the Beerwulf KPI's...
To finish things up…
The answer is simple, its..
But we’ve got everything in place
To do the right thing,
...
And as long as we stay humble,
and realize that,
just like Jon Snow...
We all know nothing,
And we need data to inform our decision making..
If we do that I'm sure that...