Experiment driven product - LAST 2014

•

0 likes•334 views

Redbubble is an online marketplace with 10 million monthly visits based in Melbourne that has 4 engineering teams deploying updates up to 12 times per day. They test assumptions through A/B testing new experiences and features, like offering free shipping for a week which resulted in a $3500 loss but $50,000 saved, or adding a "fake door" for Amazon Payments only that increased conversions by 8%. The key is to isolate changes, measure impacts on the user funnel, and establish causal relationships between tests and outcomes.

Experiment Driven Product
How do we evolve our product at Redbubble

Who is Redbubble
Artwork Marketplace
~10M visits a month
Melbourne based
San Francisco office
2 http://www.redbubble.com/people/luckyvegetable

Engineering at
Redbubble
4 teams
Very-very Lean
Even more Agile
up to 12 deploys / day
We test assumptions
3
http://www.redbubble.com/people/luckyvegetable

Why A/B test
Isolate changes
!
Establish causal
relationship
6
http://www.redbubble.com/people/joust/works/567864-e-mc-bananas

Funnel
1. Identify major
steps
2. Measure each
3. Profit
7
http://www.redbubble.com/people/anfa/works/8412967-i-put-the-fun-in-funnel

Free Shipping Experiment
1 week
No significant change
~$3500 loss
~$50000+ saved
9

Amazon Payment Fake Door
10
FAIL
SU
C
C
ESS

Amazon Payment Fake Door
Minimal cost to run
People who use only Amazon Payments exist!
Looks like ~8% increase in conversion
11

Stop assuming - start testing
13
Duncan Hull - https://flic.kr/p/iVLZt

Running Head: SCM GLOBE SIMULATION 7 SCM Globe Simulation Collaborative Supply Chains ASCM 631 University of Maryland University College Professor Jim Bryant November 11, 2018 I was given the task to evaluate the SCM globe simulation tool for a new company the Just Born Candy Company. As the testing evaluator it is my job to come up with a way to properly manage two different companies and evaluate how well the simulation is able to handle the changes the occur during the simulation. This will be done by figuring out which combination of different supply chain operations will allow it to run with the most effective approach. As the evaluator for the simulation I will have to fix all demand issues, route issues, supply issues, storage issues and any other possible object that may present itself while trying to come up with an approach that will allow both organizations to run smoothly. Then provide them with a recommendation on how I felt the simulation ran and its possible benefits. By accomplishing this it will allow the simulation to meet the objective that has been put into place. Based on the data and information that will be provided in this paper you will be able to see what issues were found that became a problem for both companies and what steps were taken to fix the problem. It will also show what changes had to be made during the simulation that either made things worse or better for the simulation. The main reason for the entire simulation other than a teaching tool, is to get two companies that were not able to compete as individuals against the bigger brand companies so they had to figure out how to come together so that both companies could still survive. This simulation proves that they can do that and draw in the necessary revenue to compete. The two companies need to run as a unit in order to run successfully. They must find a way to balance every aspect associated with running the companies individually and as a unit. This will mean balancing all known components and the demands that they produce per day must go with the production per day as well as the quantity per day. The operation cost and the storage must also be balanced. During the first week of the simulation (week 2) the task was to load the simulation (collaborative supply chains) case study. Given how SCM globe is designed to function the data will allow for adjustments in the supply chain to ensure proper operations. This includes adjustments with truck routs, changing to different modes of transportation, storage, adding and removing facilities, inventory and the quantities being produced and delivery schedules. The first four graphs will show the results of the changes made to the simulation as a result of trying to merge the two companies and get them to run together. Error one occurred when I first downloaded the simulation at only three days, there was a storage issue at Just Born Candy Factory. This would be a problem because it would ...

7 ways you are doing your A/B testing wrong by Côme Courteault

TheFamily

As as startup, you should be running tests all the time. Making sure you are drawing the right conclusions from those tests is also a vital part of this: it helps you get those customers and save precious time. There are lots of mistakes you can make when A/B testing. In this 45mn workshop, Côme tells you about them, why they happen, and how to avoid them. He is also share tips and tricks to get your A/B testing right. Côme Courteault is Growth Hacker at TheFamily. He helps many of our startups build sustainable growth practices, and is also a teacher at our Growth Hacking school.

Preview: The NEW 4-Hour Workweek

Tim Ferriss

A/B Testing and the Infinite Monkey Theory

UseItBetter

How We Make Apps And Services

culturengine

Funsize Sprint School: Product Design Sprints Workshop

Anthony Armendariz

Design sprints are all the rage. It may sound like a trendy buzzword but the reality is that flavors of agile methodologies and design sprints are already the status quo for designing and developing digital software. How can you deliver the perfect product for a client in a set time frame, budget with limited revisions? Design is never perfect or done and design sprints allow you to incrementally enhance a product over time. If you’re designing web and mobile applications and you’re not using an agile or sprint process, you’re probably hitting road blocks. Get ready to learn why agile is the best methodology to craft and ship great digital products and maintain a balanced studio and work life. We’ll be reviewing Funsize’s design sprint model and organize into teams to run through a workshop using an example native mobile design project. We’ll then discuss outcomes-based design sprints (as popularized by Google Ventures Design) and work as a team through a web design challenge.

Language of Conversion

Mike Roberts

2017 - JITF - Agile survival kit - how to save the marriage

Roko Roić

Madison+ UX 2014: A/B Testing - The Good, The Bad, and The Ugly

coreyloose

German Testing Day Keynote - Testing at ebay - a look at a rather unconvent...

Michael Palotas

Great! another bug

AgileCoach.net

House Of Chefs - Incubator Hybrid for F&B

Matt Wu

Figuring that fund-raising is hard and near impossible, and that ideas are basically worth shit without execution. I've decided to upload my fairly long pitch deck and share it with the rest of the world! I am fairly confident in the little bets on FADS theory and believes this is an IPO material across different markets. As they often say, capital gain is the only way to get filthy rich. This is your chance to get on early, friends, family and investors!! But I also bet that nobody really gives a rats ass.

ADMINISTRACION III - TEACHER - SEMANA 11.pptx

aderlincanovicente34

Taking your Startup from Free to PaidDave Schappell

Testing the BBC Website

shudderfix

A Tale of Two Workflows - ChefConf 2014

Pete Cheslock

Watch this talk here: https://www.youtube.com/watch?v=L__8o02od6Q For an example of the code we used in our CI pipeline to make a Chef Environment from a Berksfile.lock - check out this project: https://github.com/petecheslock/berks2env One of the biggest advantages of Chef is it's flexibility, allowing you to customize it at-will to fit your infrastructure needs. While this makes Chef incredibly powerful, it can also be challenging to develop a workflow to manage the day-to-day usage of chef. Should I use a single repo for all my cookbooks? One cookbook per repo? Berkshelf? Librarian? Test-Kitchen? Where does Jenkins(CI) fit it? What about Testing? How does this work with my small team? What about my large team? What about my * Distributed Team? Over the past few years I have been a part of two distinct Chef workflows that take opposite paths about how to solve issues around collaboration, versioning, testing, etc. During the course of this talk I will share: Details about the requirements that lead us down these 2 paths. What worked. What didn't. How we use many of the tools available to safely test code changes. How we deploy cookbook changes safely and quickly (and keep uptime our highest priority).

10 Deploys a Day - A Case Study of Continuous Delivery at Envato

John Viner

How to Run A/B Tests Successfully by Vice Media Product Manager

Product School

Introduction to CHERI technology - Cybersecurity

mikeeftimakis1

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

nkrafacyberclub

Similar to Experiment driven product - LAST 2014

Mobile App Feature Configuration and A/B Experiments

lacyrhoades

Make Money In Crypto Without Buying Coins | The Loci cycle Book

Fat to Fit

Running Head SCM GLOBE SIMULATION7SCM Globe Simul.docx

jeanettehully

7 ways you are doing your A/B testing wrong by Côme Courteault

TheFamily

Preview: The NEW 4-Hour Workweek

Tim Ferriss

A/B Testing and the Infinite Monkey Theory

UseItBetter

How We Make Apps And Services

culturengine

Funsize Sprint School: Product Design Sprints Workshop

Anthony Armendariz

Language of Conversion

Mike Roberts

2017 - JITF - Agile survival kit - how to save the marriage

Roko Roić

Madison+ UX 2014: A/B Testing - The Good, The Bad, and The Ugly

coreyloose

German Testing Day Keynote - Testing at ebay - a look at a rather unconvent...

Michael Palotas

Great! another bug

AgileCoach.net

House Of Chefs - Incubator Hybrid for F&B

Matt Wu

ADMINISTRACION III - TEACHER - SEMANA 11.pptx

aderlincanovicente34

Taking your Startup from Free to PaidDave Schappell

Testing the BBC Website

shudderfix

A Tale of Two Workflows - ChefConf 2014

Pete Cheslock

10 Deploys a Day - A Case Study of Continuous Delivery at Envato

John Viner

How to Run A/B Tests Successfully by Vice Media Product Manager

Product School

Similar to Experiment driven product - LAST 2014 (20)

Mobile App Feature Configuration and A/B Experiments

Make Money In Crypto Without Buying Coins | The Loci cycle Book

Running Head SCM GLOBE SIMULATION7SCM Globe Simul.docx

7 ways you are doing your A/B testing wrong by Côme Courteault

Preview: The NEW 4-Hour Workweek

A/B Testing and the Infinite Monkey Theory

How We Make Apps And Services

Funsize Sprint School: Product Design Sprints Workshop

Language of Conversion

2017 - JITF - Agile survival kit - how to save the marriage

Madison+ UX 2014: A/B Testing - The Good, The Bad, and The Ugly

German Testing Day Keynote - Testing at ebay - a look at a rather unconvent...

Great! another bug

House Of Chefs - Incubator Hybrid for F&B

ADMINISTRACION III - TEACHER - SEMANA 11.pptx

Taking your Startup from Free to Paid

Testing the BBC Website

A Tale of Two Workflows - ChefConf 2014

10 Deploys a Day - A Case Study of Continuous Delivery at Envato

How to Run A/B Tests Successfully by Vice Media Product Manager

Recently uploaded

Introduction to CHERI technology - Cybersecurity

mikeeftimakis1

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

nkrafacyberclub

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™

UiPathCommunity

In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni. 📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath: Autopilot per Studio Web Autopilot per Studio Autopilot per Apps Clipboard AI GenAI applicata alla Document Understanding 👨‍🏫👨‍💻 Speakers: Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath Andrei Tasca, RPA Solutions Team Lead @NTT Data

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

Quantum Computing: Current Landscape and the Future Role of APIs

Vlad Stirbu

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

PHP Frameworks: I want to break free (IPC Berlin 2024)

Ralf Eggert

In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development. This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

Recently uploaded (20)