This presentation on Continuous Delivery is from the November 2013 Automated Testing San Francisco meetup that took place at Constant Contact. The author/presenter is Matt Wilson, CTO of Lab Zero. Matt has advised clients at various industries including consumer brands, non-profits, start-ups, and financial services on Agile development, web application development, and other technology leadership challenges. This overview on Continuous Delivery highlights some of the best practices that Lab Zero has distilled, based on their many client engagements. --- About Matt Wilson: Matt is an enthused agile developer, architect, and consultant. He enjoys building elegant web services in Ruby. He believes that high-fives are underrated and measures the success of his day by how many he's seen. Prior to joining Lab Zero, Matt's work history includes: Co-founder/Architect at Earfl.com, Architect at Kodak Gallery, Developer at Westwave Communications, Engineer at Motorola, and Developer at Coldwell Banker. About Lab Zero: Lab Zero Innovations, Inc. provides web application development and technology leadership consulting. Our client relationships include staff augmentation, pure software development, project management, system integration, advisor/leadership roles. Contact us about your next project.

1. Don’t Hate, Automate.
Lessons learned from implementing Continuous Delivery.
Matt Wilson, CTO Lab Zero
LAB ZERO INNOVATIONS, INC.
PIER 9, THE EMBARCADERO SAN FRANCISCO, CA 94111
T:415.839.6861 F:415.839.6901 INFO@LABZERO.COM HTTP://LABZERO.COM

2. Overview
Who we are
Our evolution toward Continuous D*
What we’ve learned along the way
Questions & Answers
LAB ZERO INNOVATIONS, INC.

3. Who We Are
Product development, design, and strategy team based in SF.
Early-stage entrepreneurs and startup founders: earfl.com, meez.com, gofish.com, echosign.
com, veodia.com.
Held senior product design & dev roles at Apple, Google, Yahoo, Kodak, Accenture.
Our clients are often startups and Fortune 500s.
Founded in 2008
Many of our clients hire us just to build out their Continuous D* rigs.
LAB ZERO INNOVATIONS, INC.

4. Our path to Continuous D*
and what we’ve seen

5. In the beginning there was
Just a build…
We had code and compiled it. We even had some unit tests for some areas of the code.
The official build ran nightly but we certainly weren’t using it for fast feedback.
Then we started deploying with automation. “cap production deploy” Those were some fancy
pants.
Then we started running our tests on every check in.
Then we needed to automate the validation of our app running in different environments
Then we started automating the creation of our environments. Just can’t have enough.
LAB ZERO INNOVATIONS, INC.

6. Necessity struck
We have to start new projects very often.
The teams we collaborate with want to see progress often.
Projects started without a clear delivery process were often slower to deliver over time.
At some point manually running our deployment scripts every time we wanted to share our work
seemed like too much hassle. </first_world_problem>
We need to know that the feature works in the QA environment.
We figured out how to get Continuous D* running on our projects in less than a day.
LAB ZERO INNOVATIONS, INC.

7. Our Webapp “Build”
Rspec:
Unit tests and generates code coverage reports.
Jasmine + PhantomJS:
Javascript unit testing.
Cucumber:
We write our ATDD suites in Cucumber and use Capybara to actually drive browsers.
Brakeman:
Static security analyzer to help us find obvious security vulnerabilities.
Capistrano:
Automated deployments of RoR, PHP, Java... applications.
LAB ZERO INNOVATIONS, INC.

8. Our Mobile “Build”
Xcodebuild, Ant+javac:
iOS and Android command line tools.
OCUnit, JUnit:
iOS and Android unit testing frameworks.
Cucumber:
Using Calabash to actually drive the native devices and emulators.
TestFlight:
Deploy every passing build to TestFlight for easy distribution to team devices.
LAB ZERO INNOVATIONS, INC.

9. The story starts at your laptop and ends in production.
LAB ZERO INNOVATIONS, INC.

10. TDD
the first “d” is for “driven”

11. Build Failed

12. “When the light is green, the trap is clean.” --Ghostbusters

13. It’s hard to keep the light green or meaningful if tests
are written afterwards.
Build your culture of automation awareness.
There is a correlation between writing tests and running them. Those that write them tend to run
them more often.
Developers and QA both need to run tests before committing.
If developers aren’t running the tests because they take too long, it’s worth the investment to
speed them up.
CD can fall apart quickly if the team doesn’t react early to failed builds.
LAB ZERO INNOVATIONS, INC.

14. Speed will matter
when you’re doing it right

15. How long does it take to deliver a feature to QA?
LAB ZERO INNOVATIONS, INC.

16. Ouch, my tests!
We want to run as many tests as we can but once we’ve invested in a fair amount of browser
and emulator testing they get really slow.
When they get slow we become reluctant to run them.
Tests that don’t run, can’t find bugs.
The longer you wait to run them all, the less likely they are to still pass.
Tests that are broken and remain unfixed become dead weight.
Releasing under-tested products often means we’re asking too much of our QA teams and
usually won’t find bugs until they are in production.
LAB ZERO INNOVATIONS, INC.

17. Usual suspects for slowness in the pipe
Heavy use of fixtures and db cleaning (factory-girl and database-cleaner)
Consider using mocks/stubs instead of db’s when possible.
Android emulator
Get a real device and plug it into your CI server.
Assets compiled on each server at deploy time
Pre-compile your static assets on the build server and deploy archives instead.
LAB ZERO INNOVATIONS, INC.

18. Parallulala, parallaluh, parallelize them
Selenium tests are just too slow to not do this
Remote browser tests will need you to run multiple tests at once. Especially when you start
considering running your tests in more than one browser/OS combo.
To do this well you should lean on the PaaSs out there like
Solano CI (beaucoup parallelization-fu), Sauce Labs (browsers and mobile emulators, can be
used from Solano), Travis CI, etc...
Unit tests will need it too
If you’re doing TDD, you likely already have this problem.
LAB ZERO INNOVATIONS, INC.

19. Continuous D* wants to use your test automation
across many environments

22. You’ll need a few environments to pull this off.
CI Server
Run tests here first to find anything that should prevent us from deploying the app. Unit tests,
static analysis,
DevInteg
First deployed spot for the app, the environment will be volatile and will serve little use other than
for CI. May also serve as QA server if the numbers of commits are low.
QA1, 2, 3..
You may need one of these for each product team.
Staging
Runs in production and is often used as a place to validate the app in production before going
live.
Production
Live.
LAB ZERO INNOVATIONS, INC.

23. We want to get the features out, but they have to be
verified in QA and Staging before we go live.
LAB ZERO INNOVATIONS, INC.

24. Co-own yer ATDD
with QA

25. Force multiplier
Share the same ATDD code
Devs and QA should collaborate on their functional testing investment. Many organizations
separate these groups activities which leads to redundant investments.
Devs need to lead in the spirit of TDD
Relying only on QA for functional automation causes flapping red->red->green->red->green
builds.
It works better when you work together
Devs can do the heavy lifting on writing reusable steps and tests. QA can add more cases if
they see fit. Devs should expect to train and support QA on the tools.
Allow QA to audit
With our QA hats on, let’s have an open conversation about what should be tested and what is
missing. This is how we can focus our beams to make the light brighter.
LAB ZERO INNOVATIONS, INC.

26. Running tests
just one thing your CI server can do

27. Robots, ACL’s and users, what could go wrong?
Empower folks with a magic button
There are likely things that your team is always asking someone else to do for them. De-spof it
by making a job that folks can click whenever needed, even from their phones. Zomg seriously.
Database Refreshes
Nearly every webapp project needs to be able to reset/refresh the dev/qa/staging databases
easily.
Production deploy jobs
This is your parent’s liquor cabinet: make sure the kids don’t get into it, but make it easy to open
when needed.
LAB ZERO INNOVATIONS, INC.

28. Thank You!
Questions?
matt@labzero.com
twitter/aim/yim/skype: w1150n
LAB ZERO INNOVATIONS, INC.
PIER 9, THE EMBARCADERO SAN FRANCISCO, CA 94111
T:415.839.6861 F:415.839.6901 INFO@LABZERO.COM WWW.LABZERO.COM