Scaling Product Experimentation

•Download as PPTX, PDF•

0 likes•938 views

Experiments and A/B tests are at the heart of innovation at Twitter. A culture of experimentation empowers the organization to try new things with minimal downside. Twitter uses Confluence and JIRA to manage hundreds of experiments at any given time. Learn Twitter's philosophy on experimentation, how we enhance organizational learning, and uncover deeper insights that help everyone become smarter with every new experiment.

Technology

Scaling Product Experimentation
with JIRA and Confluence
Z A N E S A L I M • S E N I O R T E C H N I C A L P M • TWI T T E R • @Z A N E S

Technology is disrupting
companies faster than ever

Companies that stop
innovating fizzle… fast

1. Our approach @twitter
2. Challenges at scale
3. How we scaled
4. What got better
5. Final takeaways

@Twitter we innovate
through experimentation

Confluence: Experiment Brief Templates
Experiment Brief Experiment Write-up Retrospective Analysis
Describe the problem Describe the experiment Present retrospective analysis
Define a hypothesis Present the results Present the results
Design the experiment Offer take-away’s Offer take-away’s

Confluence: Reviewed Experiments & Learning

Smart use of tools can make
you 4x more effective

The modern SEO is like "An Army of One", always taking on new roles and responsibilities. But there are many problems and limitations with this model for SEO in large organizations. We must do better to achieve greater results -- enter Agile development methodology! In this presentation for the ad:Tech 2012 Marketing Masters panel chaired by Rhonda Hanson from Getty Images, I show how REI used Agile development practices to complete numerous SEO efforts that results in greatly increased organic visibility and traffic. If you like SEO, the stormtroopers from Star Wars, and shipping iteration after iteration of valuable experiences to your customers, then this is the presentation for you! You can learn more about Jonathon Colman at http://www.jonathoncolman.org/

High Output Tech Management

Tu Pham

Take ithomeVijaya M

N=10^9: Automated Experimentation at Scale

Optimizely

Wojciech Galuba, Decision Tools Lead, Facebook Experimentation is a valuable tool for supporting product decisions, iterating on features and gaining actionable insights into people's behavior. In this session Wojciech Galbua, Data Scientist at Facebook, presents an overview of Facebook's experimentation framework and how it is used to make day-to-day data-driven decisions at global scale. The talk focuses on the challenges of building and scaling the analytics infrastructure, designing the tools for ease-of-use and ensuring broad adoption of sound experimentation methodologies across all the teams.

Implementing and analyzing online experiments

Sean Taylor

Randomized experiments are the gold standard for understanding and quantifying causal relationships. This talk is divided into two parts corresponding to before and after the experiment is run. In the first section, we discuss how to design and implement online experiments using PlanOut, an open-source toolkit for advanced online experimentation used at Facebook. We will show how basic “A/B tests”, within-subjects designs, as well as more sophisticated experiments can be implemented. In the second section, we cover methods to estimate causal quantities of interest and construct appropriate confidence intervals. Particular attention will be given to scalable methods suitable for “big data”, including working with weighted data and clustered bootstrapping.

Ab testing work flowJainul Khan

The Experimentation Imperative

Clay Parker Jones

A/B Testing at Pinterest: Building a Culture of Experimentation

WrangleConf

Presenter: Andrea Burbank, Pinterest A successful experimentation program consists of much more than mere randomization and measurement. How do you help stakeholders understand the right things to measure, avoid common pitfalls, and learn to rely on A/B tests as the best way to measure a new system or feature? In this talk, Andrea will explain how building a culture of experimentation and the right tools to support it is just as important as the statistics behind the comparisons themselves - and potentially much trickier to get right.

Field Guide to Rapid Experimentation

Intuit Inc.

The slide deck we used to raise half a million dollars

Buffer

Managing Product Growth Experiments With JIRA and Confluence - Zane Salim

Atlassian

Help! I need an empirical study for my PhD!

Walid Maalej

Lean User Research - UXPA 2013 Workshop

Cassy Rowe

Polychrome Testing: Ideas & Practices to Improve Software Testing | QASymphon...

QASymphony

Stanford Peace Innovation Lab: ITBA Argentina workshop day 2Margarita Quihuis

Experimentation for PMs: A Primer by Amazon Director of Product

Product School

More Than Usability

Razan Sadeq

AISA Leadership Retreat Ghana - Leading complex change 2013

Chris Jansen

What's hot

What Awesome Sauce Tastes Like: Getting & Keeping Your Teams Healthy... the A...

Atlassian

An Engaging First Impression - Growth at Atlassian

Benjamin Humphrey

10 Commandments of Virtual Teams

PGi

Collaboration at Scale with the Product Fair

Phil Wolpers

Learn Agile Marketing & SEO from Star Wars Stormtroopers

Jonathon Colman

High Output Tech Management

Tu Pham

Take ithomeVijaya M

What's hot (7)

What Awesome Sauce Tastes Like: Getting & Keeping Your Teams Healthy... the A...

An Engaging First Impression - Growth at Atlassian

10 Commandments of Virtual Teams

Collaboration at Scale with the Product Fair

Learn Agile Marketing & SEO from Star Wars Stormtroopers

High Output Tech Management

Take ithome

Viewers also liked

N=10^9: Automated Experimentation at Scale

Optimizely

Implementing and analyzing online experiments

Sean Taylor

Ab testing work flowJainul Khan

The Experimentation Imperative

Clay Parker Jones

A/B Testing at Pinterest: Building a Culture of Experimentation

WrangleConf

Field Guide to Rapid Experimentation

Intuit Inc.

The slide deck we used to raise half a million dollars

Buffer

Viewers also liked (7)

N=10^9: Automated Experimentation at Scale

Implementing and analyzing online experiments

Ab testing work flow

The Experimentation Imperative

A/B Testing at Pinterest: Building a Culture of Experimentation

Field Guide to Rapid Experimentation

The slide deck we used to raise half a million dollars

Similar to Scaling Product Experimentation

Managing Product Growth Experiments With JIRA and Confluence - Zane Salim

Atlassian

Help! I need an empirical study for my PhD!

Walid Maalej

Lean User Research - UXPA 2013 Workshop

Cassy Rowe

Polychrome Testing: Ideas & Practices to Improve Software Testing | QASymphon...

QASymphony

Stanford Peace Innovation Lab: ITBA Argentina workshop day 2Margarita Quihuis

Experimentation for PMs: A Primer by Amazon Director of Product

Product School

More Than Usability

Razan Sadeq

AISA Leadership Retreat Ghana - Leading complex change 2013

Chris Jansen

User Testing: Adapt to Fit Your Needs

Effective

EffectiveUI's Ari Weissman (Lead Experience Architect) and Lys Maitland (Senior Experience Planner) spoke at Denver Startup Week 2016. Discussion description: Test early, test often. It’s a mantra that’s been proven successful time and again when it comes to innovation and design. So why aren’t you doing it? In the start-up world, when everything is moving so quickly, it can be easy to overlook or postpone collecting feedback from real people because of cost, time, or lack of preparation. Don’t let those things stop you. Valid data can be captured cheaply, quickly, and with half-finished products and strategies. This talk will cover: What is user testing and why is it important How to plan for user testing What are ways to make testing cheaper What are ways to make testing quicker How to test with different fidelities of concept and design How to collect data more frequently Opportunities for getting the whole team engaged What to do with the insights/outcomes of research

Lean Startup 301

Lean Startup Co.

Aubrey Smith, Sparked Advisory In this training, we will build on the foundation established in Lean Startup 101 and 201 by delving into examples and cases of the Lean Startup concepts in action. Attendees of Lean Startup 301 will be exposed to cutting edge work from thought leaders and experts using Lean Startup in practice today — at startups and within the enterprise. Participation in this session is essential: You will be asked to help design an MVP and experiment to test critical Leap of Faith Assumption(s) in groups and will be encourage to share experiences. The session is designed to allow attendees to stretch their skills and to push one-another to ‘learn by doing’. The session will also include: Sample cases and live interviews with practitioners highlighting the application of core concepts; Exercises designed to bring the concepts to life and challenge participants to deepen their skills; Discussion of advanced topics such organizational culture and governance as well as industry-specific concepts such as using Lean Startup in heavily regulated markets. Thanks to Lean Startup Co.’s law firm, Orrick, for being the sponsor for this track.

Research and Commercialisation Challenges

Dr. Mazlan Abbas

Sprint school slidepack final

Design Lab

Testing the unknown: the art and science of working with hypothesis

Ardita Karaj

Testing what we know, or have a clear understanding of, is relatively straight forward, as is making decisions based on the expected result. But today’s world is presenting us with the Unknown and the Ambiguous, which can only be approached by hypothesizing and experimenting - a lot! This requires intentional thinking, and a different strategy to observe in context. This session will uncover how testers are helping their teams and product owners, by basing their testing on the science behind creating hypotheses and running experiments. A testing mindset and probing the context around use cases are some of the most valuable competencies testers bring to the team in order to enable decisions based on data.

Entreprenuership_ppt4.pptx idea generat

ShivaniTiwari24572

501 Talks Tech: Design Thinking Workshop by Dupla Studios

501 Commons

The future of market research

InSites on Stage

Continuous Learning

Dashlane

Presentation on Design Thinking teaching pedgagogy

Eric Morrow

Formative Evaluation for Educational Product Development

Vanessa Gennarelli

Running Lean Startup Experiments

Lifehack HQ

Similar to Scaling Product Experimentation (20)

Managing Product Growth Experiments With JIRA and Confluence - Zane Salim

Help! I need an empirical study for my PhD!

Lean User Research - UXPA 2013 Workshop

Polychrome Testing: Ideas & Practices to Improve Software Testing | QASymphon...

Stanford Peace Innovation Lab: ITBA Argentina workshop day 2

Experimentation for PMs: A Primer by Amazon Director of Product

More Than Usability

AISA Leadership Retreat Ghana - Leading complex change 2013

User Testing: Adapt to Fit Your Needs

Lean Startup 301

Research and Commercialisation Challenges

Sprint school slidepack final

Testing the unknown: the art and science of working with hypothesis

Entreprenuership_ppt4.pptx idea generat

501 Talks Tech: Design Thinking Workshop by Dupla Studios

The future of market research

Continuous Learning

Presentation on Design Thinking teaching pedgagogy

Formative Evaluation for Educational Product Development

Running Lean Startup Experiments

Recently uploaded

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

Peter Spielvogel

Building better applications for business users with SAP Fiori. • What is SAP Fiori and why it matters to you • How a better user experience drives measurable business benefits • How to get started with SAP Fiori today • How SAP Fiori elements accelerates application development • How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities • How SAP Fiori paves the way for using AI in SAP apps

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Aggregage

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Accelerate your Kubernetes clusters with Varnish Caching

Thijs Feryn

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

Quantum Computing: Current Landscape and the Future Role of APIs

Vlad Stirbu

The Future of Platform Engineering

Jemma Hussein Allen

UiPath Community Day Dubai: AI at Work..

UiPathCommunity

Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking. 📕 Curious on our agenda? Wait no more! 10:00 Welcome note - UiPath Community in Dubai Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank 10:20 A UiPath cross-region MEA overview Ashraf El Zarka, VP and Managing Director MEA, UiPath 10:35: Customer Success Journey Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank 11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more Boris Krumrey, Global VP, Automation Innovation, UiPath 12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services. Brendan Lingam, Director of Sales and Business Development, Marc Ellis

DevOps and Testing slides at DASA Connect

Kari Kakkonen

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Free Complete Python - A step towards Data Science

RinaMondal9

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

SOFTTECHHUB

The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing. One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.

The Metaverse and AI: how can decision-makers harness the Metaverse for their...

Jen Stirrup

The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives? How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.

Recently uploaded (20)

FIDO Alliance Osaka Seminar: Overview.pdf

The Art of the Pitch: WordPress Relationships and Sales

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

Generative AI Deep Dive: Advancing from Proof of Concept to Production

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

Accelerate your Kubernetes clusters with Varnish Caching

Leading Change strategies and insights for effective change management pdf 1.pdf

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

Quantum Computing: Current Landscape and the Future Role of APIs

The Future of Platform Engineering

UiPath Community Day Dubai: AI at Work..

DevOps and Testing slides at DASA Connect

Video Streaming: Then, Now, and in the Future

Removing Uninteresting Bytes in Software Fuzzing

Epistemic Interaction - tuning interfaces to provide information for AI support

Free Complete Python - A step towards Data Science

Monitoring Java Application Security with JDK Tools and JFR Events

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

The Metaverse and AI: how can decision-makers harness the Metaverse for their...

Scaling Product Experimentation

1. #atlassiansummit

2. Scaling Product Experimentation with JIRA and Confluence Z A N E S A L I M • S E N I O R T E C H N I C A L P M • TWI T T E R • @Z A N E S

3. Technology is disrupting companies faster than ever

5. Companies that stop innovating fizzle… fast

6. How are you innovating?

7. 1. Our approach @twitter 2. Challenges at scale 3. How we scaled 4. What got better 5. Final takeaways

8. 1. Our approach @Twitter

9. @Twitter we innovate through experimentation

10. Anyone can launch an experiment

11. Unleashes creativity

12. Drives user insights at scale

13. 100s every quarter

14. Experimentation is the new black

15. … and the old black

16. Both refinement and exploration

17. Example: which drives more clicks?

18. Example: which drives more retention?

19. 2. Challenges at scale

20. Visibility

21. Learning

22. Collaboration

23. Rigor

24. 3. How we scaled

25. COMMON JIRA WORKFLOW FOR COLLABORATION

26. Lifecycle of an Experiment

27. Lifecycle of an Experiment in JIRA

28. Lifecycle of an Experiment in JIRA

29. KNOWLEDGE MANAGEMENT IN CONFLUENCE

30. Confluence: Knowledge Management

31. Confluence: Experiment Brief Templates

32. Confluence: Experiment Brief Templates Experiment Brief Experiment Write-up Retrospective Analysis Describe the problem Describe the experiment Present retrospective analysis Define a hypothesis Present the results Present the results Design the experiment Offer take-away’s Offer take-away’s

33. Confluence: Reviewed Experiments & Learning

34. HYPOTHESIS TREES

35. Visualizing Focus Areas & Insights

36. 4. What got better

37. Doubled completed experiments

38. Reduced cycle time up to 50%

39. Learning org  smarter experiments

40. 5. Final takeaways

41. Experimentation drives innovation

42. Collaboration & learning essential

43. Smart use of tools can make you 4x more effective

44. Thank you!