Arylee McSweaney speaks at the QA Ecommerce Forum at the Harvard Club. Timeline of QA at Etsy Quality Advisory Board

1. Etsy’s New Culture of QA:
How a Quality Advisory Group Can
Accelerate Code Coverage
Arylee McSweaney
Thursday, November 14th 2019

2. 2
• Two-sided, global E-commerce marketplace with
a focus on custom, handcrafted and vintage
goods.
• Web
• Mobile
• Opened June 18th 2005
• Keeping Commerce Human
• Over 2M sellers and 44M buyers worldwide
• Strong engineering presence via Code as Craft
• Agile CI/CD (Continuous Integration/Continuous
Deployment)
About
Footnote or attribution

3. 3
• Quality Assurance Professional, > 10 years
• Leading Etsy’s Quality Initiative (2 years)
• Keeping Commerce Human at Etsy (6 years)
• Quality background at corporate, mid-size & small
startups
• Agile, Waterfall and everything in between
• BS, Computer Science & MBA - Information
Systems
• Quality works well when it’s a shared
responsibility.
About Me
Footnote or attribution

4. 4
• Quality Assurance Professional, > 10 years
• Keeping Commerce Human at Etsy (6 years)
• Leading Etsy’s Quality Initiative (2 years)
• Quality background at corporate, mid-size & small
startups
• BS, Computer Science & MBA - Information
Systems
• Agile, Waterfall and everything in between
• Quality works well when it’s a shared
responsibility.
Fun fact: I love having goldfish as pets (but I don’t
have one right now)
About Me
Footnote or attribution

5. Agenda
1
Quality at Etsy
2
Quality Advisory Group
3
How to Start a Quality
Advisory Group
4
Wrap Up & Thanks

6. Software Quality at Etsy
Quality is not just the act of testing prerelease (by a
specific group of people), it also includes our failure
detection capabilities, our response time and our
ability to learn from production incidents.
PP Jeff Sussna, ALM Forum 2014

7. Footnote or attribution
Quality at Etsy - A Timeline
2012 - 2013
Continuous Integration
Continuous Deployment
Config Flag Testing
Unit/Integration Tests
Monitoring/Alerting
PHP/JavaScript
Code Coverage
PHP
Mobile
Cultural Shift to testing
(really) becoming
everyone’s responsibility
Test Automation Team
2017
Code Coverage
> 10 Repositories
Quality Advisory
Group
2018+
2014 - 2016
Manual Testing
Complex product
development
Mobile Apps (BOE &
SOE)

8. Quality at Etsy - 2012 • Continuous Integration
• Continuous Deployment
• Config Flag “Production” Testing
• Unit/Integration Tests
• Monitoring/Alerting
• PHP/JavaScript
“The goal here is to
constantly emphasize
shipping, and get over
any deployment fears
early.”
Code as Craft: How Does Etsy Manage Development & Operations?

9. Footnote or attribution
Quality at Etsy - A Timeline
2012 - 2013
Continuous Integration
Continuous Deployment
Config Flag Testing
Unit/Integration Tests
Monitoring/Alerting
PHP/JavaScript
Code Coverage
PHP
Mobile
Cultural Shift to testing
(really) becoming
everyone’s responsibility
Test Automation Team
2017
Code Coverage
> 10 Repositories
Quality Advisory
Group
2018+
2014 - 2016
Manual Testing
Complex product
development
Mobile Apps (BOE &
SOE)
Tech Debt

10. Quality at Etsy - 2014 • Manual Testing Team
• Complex/new product development
• Mobile Apps (BOE & SOE)
• Tech Debt
“At the start of 2014, we
were at an impasse. After
years of iteration by
many different teams,
our seller tools were
starting to show their
age.”
Code as Craft: Rebuilding the Foundation of Etsy’s Seller Tools

11. Footnote or attribution
Quality at Etsy - A Timeline
2012 - 2013
Continuous Integration
Continuous Deployment
Config Flag Testing
Unit/Integration Tests
Monitoring/Alerting
PHP/JavaScript
The TEST Team
Code Coverage
PHP
Mobile
Cultural Shift to testing
(really) becoming
everyone’s responsibility
2017
Code Coverage
> 10 Repositories
Quality Advisory Group
2018+
2014 - 2016
Manual Testing
Complex product
development
Mobile Apps (BOE &
SOE)
Tech Debt
Inflection Point

12. Quality at Etsy - 2017 • The Inception of the TEST Team
• Code Coverage and Measurement
frameworks
• PHP
• Mobile
• Quality truly became everyone’s
responsibility
“One of our first goals was to
find a way to establish a
quantifiable quality
“baseline” and track our
progress against this
baseline over time.”
Code as Craft: Culture of Quality: Measuring Code Coverage at Etsy

13. What is Code Coverage?
“ Code coverage is the measure of how much
code is being executed by your test suite (unit,
integration, ui tests). ”
Code as Craft: Culture of Quality: Measuring Code Coverage at Etsy

14. What is Code Coverage Reporting?
“The act of monitoring and measuring the
amount of code, in a given repository, that is
executed by tests.”

15. What is the Value of Code Coverage?
● The value we place in code coverage is the building of a habit (across engineering) for
covering our code with tests.
● The trend (increasing) matters more than the actual code coverage percentage.
● The quality of the tests matter more than increasing coverage by a certain percentage.
● A mechanism for reporting progress both to engineering teams as well as engineering
leadership.

16. Code Coverage is valuable
when everyone does it.

17. Footnote or attribution
Quality at Etsy Timeline
2012 - 2013
Continuous Integration
Continuous Deployment
Config Flag Testing
Unit/Integration Tests
Monitoring/Alerting
PHP/JavaScript
Code Coverage
PHP
Mobile
Cultural Shift to testing
(really) becoming
everyone’s responsibility
Test Automation Team
2017
Code Coverage
> 10 Repositories
Quality Advisory Group
2018+
2014 - 2016
Manual Testing
Complex product
development
Mobile Apps (BOE &
SOE)

18. Quality at Etsy
Today
• Quality and testing are top of mind for
engineers.
• Quality is a point of discussion at
Engineering Leadership meetings.
• Coverage on new classes meet or
exceed 75% coverage each week.
• Track coverage on > 10 code
repositories
• Annual Engineering Quality Survey
• A holistic view of Quality, pain points
and gaps to testing are surfaced via
the Quality Advisory Group.

19. Quality Advisory
Group

20. Footnote or attribution
About the Quality Advisory Group
What is It?
The Quality Advisory
Group guides
conversations and
contributes diverse,
engineering perspectives
that steer Etsy’s Quality
narrative.
Directors/Engineering Leaders
Engineers from various disciplines
Test Automation/Quality team
members
Who Should Participate?
Why Create One?
Easily & frequently identify
and prioritize pain points
across engineering
disciplines.
Articulate recommendations
for investment to engineering
leadership.
Drive buy-in/adoption for new
processes and tools.
Increase visibility of QA and
build a stronger QA presence.

21. Footnote or attribution
Quality Advisory Group - How it Works
Leadership
The Quality Advisory
Group is best led by the
QA Manager/Director or
Head of Quality.
Collaborate with
Engineering Leadership
Develop Quality focused
communications and
updates.
Engineering Leadership should buy
in to the Quality Advisory Group in
order to be prepared to receive
recommendations and requests for
technical investment that
represent broad engineering
needs.
Engineers from various disciplines
should participate in the group to
ensure their needs are heard.
The group is a perpetual fixture in
engineering and members may
rotate out after a quarter.
Buy In
Workload Impact
The Quality Advisory Group
typically meets bi-weekly.
An agenda should be
circulated before the meeting.
Attendees at each meeting
talk through the agenda,
weigh in on decisions or
suggest ideas to drive more
engagement.

22. Footnote or attribution
Quality Advisory Group
● Feedback on quality investments
● Quality as a priority for engineering
leadership
● Visibility for Quality metrics like Code
Coverage
● Validation of roadmap priorities
● Ongoing discussions about quality with
various stakeholders
● Submitting recommendations to
leadership
● Ability to measure the impact of this group
in collaboration with the QA/Quality team
over time.
● Administrative burden falls on Quality
personnel.
○ Can be mitigated by having more than
one QA team member participate.
● Technical work to be done by other teams
needs to be prioritized according to their
roadmaps.
Advantages Disadvantages

23. Quality Advisory Group - Recommendations
Quality Coach - Mobile Data Science QA Engineer Test Account Tool
● Held accountable for the
quality of our mobile
releases.
● Liaises with third party
testing services.
● Instill a Quality
perspective upstream in
the Sprint Planning
Phase.
● Data Scientist team
members often validated
changes to data sets
manually.
● Lacked appropriate
tooling to increase
efficiencies in their
workflows.
● Have different needs
pertaining to test
automation.
● Validating testing on
Etsy required the use of
test accounts configured
to specific use cases.
● Creating this data was
time consuming and
there was no centralized
mechanism for surfacing
or sharing test accounts
across engineers.

24. Quality Advisory Group - Key Wins
Quality Coach - Mobile Data Science QA Engineer/SME Test Account Tool
● Engineering teams
working on the apps plan
for quality during Sprint
planning.
● Identified other guides
needed to support app
delivery.
● Creates a visibility and
presence for Quality that
signals the need for us
to expand this capability
to our Web practice.
● Surfaced critical gaps in
how data teams engaged
with each other.
● Developed a Big Data
Diff Tool that helps
product engineers and
data engineers validate
their changes in minutes
(not hours).
● Identified clear tracks of
work that help to
demystify Data Science.
● Phase 1 launched in
June of 2019.
● Phase 2 to launch in
December 2019.
● Etsy team members now
have a safe, compliant
means for testing on
production.

25. Quality Advisory Group - Key Wins
Footnote or attribution
● Annual, engineering quality survey
● Expanded QA practice beyond Web into Mobile, Data Science, Test tooling
● Introduction of the Quality Scorecard
● Introduction of 3rd party tools and services

26. How Do I Get
Started?

27. 7 Steps to Begin a Quality Advisory Group
Footnote or attribution
1. Introduce the Idea. Talk to the Head of QA or CTO to align on the vision for Quality in
your organization and how it should support overarching engineering goals.
2. Secure Buy In. Leverage that conversation as an opportunity to discuss the benefits
of introducing a Quality Advisory Group.
3. Discuss ROI and KPIs. Identify and agree on how the impact of the group will be
measured.
4. Grow Membership. Start with a group of 5-7 members. Ideally, members will
represent different engineering disciplines.
5. Assess. Group members will meet to discuss, assess and formally articulate the
state of quality in the organization to leadership.
6. Recommend. Make recommendations to engineering leadership that solve the key
pain points.
7. Implement Approved Recommendations. Then measure and repeat.

28. Thank You

29. Thank You
Join the Fun!!
etsy.com/careers
Contact Me!!
LinkedIn: Arylee McSweaney
Email: amcsweaney@etsy.com
Twitter: @adrainbowsend
Keep Reading!!
codeascraft.com
Twitter:
@codeascraft