Mozilla: Continuous Deploment on SUMO

MOZILLA Continuous Deployment on Sumo support.mozilla.com Matt Brandt, QA Engineer mbrandt@mozilla.comTuesday, November 22, 2011

[Nearly] MOZILLA Continuous Deployment on Sumo support.mozilla.com Matt Brandt, QA Engineer mbrandt@mozilla.comTuesday, November 22, 2011

Bring questionsTuesday, November 22, 2011

Bring questions What’s wrong with our current small iterations?Tuesday, November 22, 2011

Bring questions What’s wrong with our current small iterations? Trust developers to test the app??!!Tuesday, November 22, 2011

Bring questions What’s wrong with our current small iterations? Trust developers to test the app??!! What happens to our Selenium test suite?Tuesday, November 22, 2011

Bring questions What’s wrong with our current small iterations? Trust developers to test the app??!! What happens to our Selenium test suite? We’ll be relegated to manual testers?Tuesday, November 22, 2011

Bring questions What’s wrong with our current small iterations? Trust developers to test the app??!! What happens to our Selenium test suite? We’ll be relegated to manual testers? How do we measure quality?Tuesday, November 22, 2011

But first a little history...Tuesday, November 22, 2011

Short IterationsTuesday, November 22, 2011

Simplified What the team did before Trunk/ Stage Prod DevTuesday, November 22, 2011

Simplified What the team did before Trunk/ Stage Prod Dev •Verify new features •Is this the right thing? •AutomateTuesday, November 22, 2011

Simplified What the team did before Trunk/ Stage Prod Dev •Verify new features •IT pushes •Is this the right thing? •Regression test •Automate •Exploratory Testing •Is this the right thing?Tuesday, November 22, 2011

Simplified What the team did before Trunk/ Stage Prod Dev •Verify new features •IT pushes •IT pushes •Is this the right thing? •Regression test •Verify •Automate •Exploratory Testing •Wahoo! •Is this the right thing?Tuesday, November 22, 2011

The brave new world Continuous deployment http://cliptank.com/funny/dog-costumes.phpTuesday, November 22, 2011

Tools & attitudeTuesday, November 22, 2011

Discuss what is acceptable risk https://wiki.mozilla.org/QA/Execution/Web_Testing/Continuous_Deployment#Risks_.26_Plans_.26_ToolsTuesday, November 22, 2011

Discuss what is acceptable risk Risk: Less critical areas will not be covered by automation https://wiki.mozilla.org/QA/Execution/Web_Testing/Continuous_Deployment#Risks_.26_Plans_.26_ToolsTuesday, November 22, 2011

Discuss what is acceptable risk Risk: Less critical areas will not be covered by automation •If the feature breaks, it is acceptable to wait up to an hour for a fix. https://wiki.mozilla.org/QA/Execution/Web_Testing/Continuous_Deployment#Risks_.26_Plans_.26_ToolsTuesday, November 22, 2011

Discuss what is acceptable risk Risk: Less critical areas will not be covered by automation •If the feature breaks, it is acceptable to wait up to an hour for a fix. • Example: the ability to answer a question. https://wiki.mozilla.org/QA/Execution/Web_Testing/Continuous_Deployment#Risks_.26_Plans_.26_ToolsTuesday, November 22, 2011

Acceptable riskTuesday, November 22, 2011

What’s our manual strategy?Tuesday, November 22, 2011

Manual strategyTuesday, November 22, 2011

Manual strategy Feature veriﬁcation & exploratory testingTuesday, November 22, 2011

Manual strategy Feature veriﬁcation & exploratory testing Testing can occur in productionTuesday, November 22, 2011

Manual strategy Feature veriﬁcation & exploratory testing Testing can occur in production Community involvement on new featuresTuesday, November 22, 2011

Automation is fun https://github.com/mozilla/sumo-testsTuesday, November 22, 2011

Automation strategyTuesday, November 22, 2011

Automation strategy Selenium tests for areas that are clumsy to cover with unit tests or QUnitTuesday, November 22, 2011

Automation strategy Selenium tests for areas that are clumsy to cover with unit tests or QUnit Qualify as deployment blockers: run in dev & staging envsTuesday, November 22, 2011

Automation strategy Selenium tests for areas that are clumsy to cover with unit tests or QUnit Qualify as deployment blockers: run in dev & staging envs Non-volatile production tests verify env & services workTuesday, November 22, 2011

Automation strategy Selenium tests for areas that are clumsy to cover with unit tests or QUnit Qualify as deployment blockers: run in dev & staging envs Non-volatile production tests verify env & services work [All] Tests will be deleted if they don’t cover critical areasTuesday, November 22, 2011

Waffling Failure is always an optionTuesday, November 22, 2011

ControlTuesday, November 22, 2011

Gathering dataTuesday, November 22, 2011

Now Trunk/ Stage Prod Dev •Automation runs •Fast iterations •Early stage testingTuesday, November 22, 2011

Now Trunk/ Stage Prod Dev •Automation runs •Auto-updated •Fast iterations •No post-commit hooks •Early stage testing •Bug & feature testingTuesday, November 22, 2011

Now Trunk/ Stage Prod Dev •Not push-button •Automation runs •Auto-updated •Automation •Fast iterations •No post-commit hooks •Monitoring •Early stage testing •Bug & feature testing •Feedback tool •Wahoo!Tuesday, November 22, 2011

QuestionsTuesday, November 22, 2011

References •https://support.mozilla.com •https://wiki.mozilla.org/QA/Execution/Web_Testing/ •http://docs.jquery.com/QUnit Continuous_Deployment •https://github.com/mozilla/sumo-tests •https://github.com/jsocol/django-wafﬂe •http://coffeeonthekeyboard.com/the-future-of-sumo- •http://coffeeonthekeyboard.com/introducing-wafﬂe-for- django-541/ development-511/ •http://coffeeonthekeyboard.com/acronyms-you-should-know- •https://github.com/etsy/statsd mttd-and-mttr-597/ •http://codeascraft.etsy.com/2011/02/15/measure-anything- measure-everything/ •http://www.slideshare.net/stephendonner/continuous- deployment-9363895Tuesday, November 22, 2011

Mozilla: Continuous Deploment on SUMO

by Matt Brandt on Nov 22, 2011

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 2

Statistics

Mozilla: Continuous Deploment on SUMO — Presentation Transcript