Until 2009, the Atlassian JIRA team shipped a major release of its software every nine to twelve months. Everything was tested—every story and every bug fix—and everything still contained serious bugs. Story development moved quickly, but after the feature-complete date, several month-long hardening periods were required to make the software actually shippable. Integrating the release into Atlassian’s hosted platform took another three or four months. Now, the team ships JIRA from master directly to the hosted platform every two weeks, with features being written and tested within an iteration, and then bundled into a shippable release at the end. Penny Wyatt describes how to move the responsibility for quality back to developers, find bugs before development work even starts through early communication, use an active feedback loop to prevent classes of bugs, and reinforce the concept that “done means done”—ruthlessly backing out changes that aren’t fully completed by the end of the iteration. You, too, can learn how to ship fast with high quality.

1.
AW12
Session
6/5/2013 3:45 PM
"Transforming Your Agile Test
Process"
Presented by:
Penny Wyatt
Atlassian Software
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Penny Wyatt
Atlassian
With ten years of software industry experience, Penny Wyatt works at Atlassian as the QA
team lead for JIRA, an issue tracker used by more than 11,000 organizations worldwide. Penny
started her career as a software developer, but after joining Microsoft in Redmond as a
developer in test, she discovered that breaking software is much more enjoyable than building
it. After a few years of developing testing tools, Penny realized that hiring a small test team to
educate developers and prevent bugs is much more efficient than hiring a large test team
to find bugs. Penny focuses on white-box testing, security training, and ruining developers' fun.

3. Transform Your Agile Process –
Ship Fast with High Quality
Penny Wyatt
Atlassian
Who am I?
Dev
Dev‐in‐Test
Test
0
5
Experience (Years)
10
QA

4. What do I do?
Why am I here?

5. JIRA, 2010
• About 15 Developers
• 2 QA Engineers
JIRA, 2010
Development
QA
Hardening
9‐12 months
Release

6. Everything I code gets
rejected over and over,
months later
Why can’t you add
this feature? Surely
it’s not hard.
I want to see increased
sales now, not in 9
months
The harder I work,
the people criticise
me for slowing
down shipping
Let’s hire more QA!

7. Assistance
Quality Assurance
JIRA, 2011
• About 20 Developers
• 2 QA Engineers

8. DoTing
• DoS – Developer on Support
• DoT – Developer on Test
DoTing
• Aims
– Remove QA bottleneck
– Make developers responsible for quality
– Remove the dev/QA wall

9. JIRA, 2011
Development
QA
QA DoT
Hardening
Release
6‐9 months
The results
• QA was more scalable
But…
• Bugs found later
• Hardening became more important
• Stories in “Done” state weren’t really done
• Inconsistent quality across JIRA

11. Time
Scope
Quality
Improving DoTing
• Better DoTing means less hardening
• We were testing experts, they weren’t
• “No one asks the developers to teach the tech
writers how to code”

12. DoTing Training
• Testing technique
• JIRA‐specific hints
• Practical examples
JIRA, 2012
• About 35 Developers
• 2 QA Engineers

13. Crisis Time!
•
•
•
•
QA team becoming irrelevant
Hardening phase getting longer
Focus on SaaS platform
Goal – to ship from master to production
every two weeks

14. Goals
•
•
•
•
•
•
Scalability of QA
Better usability
Consistent high quality
Developer education
Done means Done
Early intervention

15. Where are the bugs?
DoT
DoT
DoT
DoT
DoT
DoT
D T
DoT
D T
DoT
D T
DoT
D T
DoT
D T
QA
DoT
DoT
DoT
DoT
DoT
Story 1
Story 2
Story 3
Story 4
Story 5
Story 6
QA
Usability issues
Usability issues
QA
Non‐obvious bugs
QA
Obvious bugs

16. Breadth, not Depth
DoT
DoT
DoT
DoT
DoT
QA
QA
QA
QA
QA
QA
DoT
DoT
D T
DoT
D T
DoT
D T
DoT
D T
DoT
D T
DoT
DoT
DoT
DoT
DoT
DoT
Story 1
Usability issues
Usability issues
DoT
Story 2
Story 3
Story 4
Story 5
Story 6
Non‐obvious bugs
Obvious bugs
Goals
•
•
•
•
Scalability of QA
Better usability
Consistent high quality
Developer education
Done means Done
Early intervention

17. QA Magic
Developers can’t test?
• “QA Magic”
– Detailed product knowledge
– SaaS knowledge
– Keeping all testing aspects in mind
– Feedback loop from bug history
– Exploratory testing skills
Exploratory testing skills

18. Testing Notes
• Hints for the testing of a feature
• Specific to a story
• Written by a QA engineer after a quick pairing
session with the dev
• Consumed by a DoT while testing
Testing Notes
• Hints and tips
• Not checklists
• Assume the competence and intelligence of
the reader

19. Testing Notes
“As an administrator, I want to invite
users via email to sign up to JIRA
users via email to sign up to JIRA”
• Click “Invite”.
• Check that the invite popup appears.
• Insert an email address.
• Click “Send”.
• Check that the email arrives.
• Click “Invite”.
• Check that the invite popup appears.
• Insert an invalid email address.
• Click “Send”.
• Check that an error appears.
Testing Notes
“As an administrator, I want to invite
users via email to sign up to JIRA
users via email to sign up to JIRA”
• What if we’re connected to a read‐only
LDAP server?
• What is outgoing mail is disabled?
• Check security of the invite tokens – can
they be reused? Do they expire?
th b
d? D th
i ?

20. Benefits of Testing Notes
• No more “QA Magic”
• Easy dissemination of knowledge
• More efficient DoTing
Developers can test!
DoT
DoT
DoT
DoT
DoT
DoT
Usability issues
Usability issues
QA
QA
QA
QA
QA
QA
Non‐obvious bugs
DoT
DoT
DoT
DoT
DoT
DoT
Obvious bugs
DoT
DoT
DoT
DoT
DoT
DoT
Story 1
Story 2
Story 3
Story 4
Story 5
Story 6

21. Done means Done
• Story is Done ‐> Ready to release
• Release is always in a known state
• Miss the deadline?
– Finish the work
– Or back it out
Goals
Scalability of QA
Better usability
Consistent high quality
Developer education
Done means Done
• Early intervention

22. JIRA, 2012
Development
DoT
Release
Dev/QA Pairing
2 weeks
Room for Improvement
Development
Dev/QA Pairing
DoT
Release

23. I’m going to add a
new action!
I bet you forget to
handle session
timeouts, again…
It’s ready for
DoTing!
Hey DoT, please test
session timeouts
Session timeouts are
broken, please fix.
OK! Fixed now.
I’m going to add a
new action!
Don’t forget session
timeouts!
It’s ready for DoTing!
I made sure session
timeouts work.
Sweet.
Sweet

24. Pre‐ to Post‐Dev Testing Notes
• Implementation choices affect risks
– Need to add some testing hints
– Need to remove some testing hints
• QA/Dev Pairing sessions still critical
• Avoid duplication of testing
Goals
Scalability of QA
Better usability
Consistent high quality
Developer education
Done means Done
Early intervention

25. A Welcome Surprise
• Empty post‐dev testing notes!
DoT
Development
Dev/QA Pairing
Goals
Scalability of QA
Better usability
Consistent high quality
Developer education
Done means Done
Early intervention
Unicorns
Release

26. JIRA, 2013
• 60 Developers
• 4 QA Engineers
JIRA, 2013
DoT
Development
Dev/QA Pairing
Release
Usability
feedback
Pre‐dev
testing notes
Post‐dev
testing notes

27. My features don’t get
rejected!
We get high‐quality
features fast!
I see results in weeks,
not months!
I get to make
people happier,
not angrier!
not angrier!
What else do we do?

28. What else do we do?
• Spot‐check features
• Pick up stories from the DoTing backlog
What else do we do?
• Write testing tools
• Set up automation frameworks
• Create test data sets

29. What else do we do?
• ShipIt days
– Quarterly programming competition
– 24 hours to “ship” a feature
– QA team works on product features or test tools
– Many QA victories
What else do we do?
• Work with Support
• Analyse bug history
• Collect metrics

30. What else do we do?
• Speak at conferences ☺
Take‐Away Lessons
•
•
•
•
•
•
Question your assumptions
Constraints encourage efficiency
Unreasonable goals can trigger great change
Don’t be afraid to reinvent yourselves
Teach people to fish, don’t hand them fish
Software QA is about people, not code

31. Q&A
More questions?
Email: pwyatt@atlassian.com