Talk given by Kelly Currier, Agile Senior Director and Vladimir Gerasimov, Product Management Senior Manager at Salesforce, at STPCon in April 2016 Salesforce adopted agile methodologies over 7 years ago. Over the years, it has helped us to drive innovation, productivity and become the world’s #1 CRM solution. Salesforce has taken agile methodologies and created a unique approach called the Adaptive Delivery Methodology (ADM). During this session, we will provide an ADM overview and how it helps us deliver 3 major releases with hundreds of features every year. We will also cover how we approach testing and quality through ADM. At Salesforce, there is no such thing as throwing code over the fence for someone else to test. Developers and Quality Engineers, we all work together to ensure release quality.

1. Quality Testing and Agile at
Salesforce
Delivering three major releases a year
Kelly Currier, Director Agile Delivery
kcurrier@salesforce.com
@kcducks
/in/kcurrier
Vladimir Gerasimov, Sr. Software Engineer, Quality
vgerasimov@salesforce.com
/in/vgerasimov87

2. Safe harbor statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties
materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed
or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-
looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any
statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new,
planned, or upgraded services or technology developments and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new
functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our
operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any
litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our
relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our
service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger
enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our
annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These
documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our
Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available
and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features
that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
Safe Harbor

3. Vladimir Gerasimov
Sr. Software Engineer, Quality
Kelly Currier
Director Agile Delivery

4. Our Goals Today:
• Show you how Salesforce Technology Organization leverages Agile methodologies
to deliver 3 major releases a year
• Explain the testing practices used to maintain the highest quality standards
• Show you how Developers and Testers collaborate
• Provide practical tips on how to start your own agile transformation today

5. In the Early Days…
• Delivering consistently
• Easy with everyone in same office
• No hand-offs
• Clear communication

6. • Incredible growth
• Unpredictable releases
• Bottlenecks
• Lack of visibility
• Changing priorities
Growing Pains Caused…

7. 2000 2001 2002 2003 2004 2005 2006
Features
Delivered per
Team
Days between
Major
Releases
Time in between major releases increased

8. A Solution
was
Needed

9. Salesforce’s Agile Implementation:

10. Respect people
Eliminate waste
Create knowledge
Build quality in
Deliver fast
Optimize the whole
Just in time decisions
7 Principles are the Foundation of our Agility

11. Vision
V2MOM
2 Release Review
1 Release
Review
Sprint
Daily
Continuous Planning & Prioritization at Every Level
Spring Summer
3-5 Years
1 year
R1
2 Releases
1 Release
~ 2 Weeks
Daily
Vision
Values
Method
Obstacles
Metrics

12. It all starts with V2MOM
CEO
Technology
Organization
Cloud
CloudCloud
Cloud
Team
Team
Team
TeamTeam
Team
Team
Team
Team
Team
Connected MeasurementIterative PrioritizationTotal Alignment
Vision: What impact will you have this year?
Values: What is most important to you?
Method: Actions required to achieve vision?
Obstacles: What might stand in your way?
Measures: Results you’re aiming to achieve?

13. Roles within our Software Engineering Organization
Team
• Cross functional (5-9)
(Dev, QE, Doc, UX, Perf, DevOps*)
• Self organizes around how to deliver
• Determines how much to take on
Product Owner
• Defines project goals
• Gathers/prioritizes requirements
• Writes stories & acceptance criteria
• Makes scope/schedule trade-offs
• Provides continuous feedback
during sprint
Scrum Master/Kanban Master
• Enable self-organization
• Facilitate continuous improvement
• Coach team and stakeholders
• Remove obstacles
• Provide visibility
Functional Manager
• Technical excellence & standards
• Hiring, people development
• Removing blockers
• Empowering team members
Technical Program Manager
• Drives delivery for a program or cloud
• Visibly & transparently reports progress
• Aligns everyone to common goals

14. 2007 2008 2009 2010 2011 2012 2013
Features
Delivered per
Team
Days between
Major
Releases
Agile Enables 3 Predictable Major Releases a Year

15. 400 +
Agile Teams @ Salesforce
We’ve Built a Scalable Model

16. Innovation Programs Retain Top Talent
Opportunity Open Market
What:
Opportunity for employees to
easily move positions to
another role, product, feature
area.
Goal:
Retain top talent. Spread
knowledge.
Timing: 3/year
PTOn!
What: Paid time per month to
innovate, explore, learn, hack
and step out of the day to day.
Goal: Allow employees to learn
and innovate - factors that drive
them and their success.
Timing: Ongoing.
Shark Tank
What: Pitch an idea to a panel
of “investors”.
Goal: Inspire employees to
take risks, work with others
and provide time for innovation
Timing: 3/year

17. Quality Engineering

18. Quality Engineers at Salesforce are Responsible for:
• Test Design and Test Planning
• Identifying Test Coverage gaps
• Developing Test Frameworks
• Writing Test Automation

19. User Story: Collaboration Between Devs and QEs
New Triaged
In
Progress
Ready For
Review
Fixed
QE in
Progress
Closed
• User Story cannot be closed without QE
• Devs and QEs working on the stories together
Devs and QEs
discussed the
work
Devs and QEs
reviewed the
code
Code
checked in
Definition of
Done is met

20. Dog-Fooding our own product
Bug and User Story tracking
Internal and External
Communication
Helps discover issues earlier
Using your product on daily basis

21. Definition of Done
• Single Definition of Done
• Establishes shared, explicit understanding
within & across teams
• Increases team accountability for quality
and commitment
• Reduces waste
• Encourages cross-team collaboration at
multiple touch points
Criteria Features
1 2 3
Code checked in and follows department standards
☑ ☑
No open regressions. Automated tests written and
reviewed for all regressions
☑ ☑
No open P1 & P2 bugs
☑
Code Coverage 78% 80%
100% of test cases logged and executed in a QA
environment, and all P1/P2 cases passing
☑ ☑
All resolved bugs verified and closed
☑ ☑ ☑
Performance/scalability impact ascertained and sys
testing scheduled if required
☑ ☑ ☑
UE has reviewed any new features; P1 and P2 UI bugs
fixed
☑ ☑ ☑
Usability testing completed when necessary, and
feedback incorporated into backlog
☑ ☑
Code and UI reviewed for 508 compliance; UE team
notified of any non-compliant features
☑ ☑
All UI labels ready for localization vendors
☑ ☑
User documentation complete and checked in
☑

22. Frequent Product Demos & Reviews
• Demo completed work with team every sprint
• Demo and review release plan progress
monthly with Executive participation
• Provide radical transparency
• No surprises come release time

23. Freedom to adjust the process
Teams decide what work or doesn’t work for them
• Team has the freedom to decide best practices
• Continuous improvement part of the DNA
• Encourages everyone to own the process
• Reduces amount of bureaucracy

24. Test Failure Management

25. Test Failures, Flappers and Long Running Tests
Lock teams with Test Failures older than 7
days
Opening Bugs for non-stable tests
Opening Bugs for Long Running Tests
Keeping test healthy and stable

26. Lock Out Email
Team gets locked out if they don't fix their Test Failures

27. Test History
Providing reports to help analyze test failure
Change List that most likely caused the failure
Stacktrace information

28. Test Failure Auto Assign
Auto-Assign manages test failures effectively
It can link Test Failure to the change list that most
likely caused the failure
It groups similar test failures under one Bug
It eliminates manual work of running reports and
opening bugs
Even when wrong, it still opens bugs and starts
the process of fixing Test Failure
Helping teams to address test failures

29. Build Management

30. Bug Triage
Helps to assign bugs to engineers
• Usually happens on weekly basis
• Helps address Bugs that weren’t auto-assigned
• Helps keep bug-log clean
• Devs and QEs attend

31. Pre-Checkin Validation
Keeps the Build healthy
• Sharing the same code branch for current
release work
• Important to keep build healthy at all times
• Pre-checkin validation ensures the change you
are submitting is not breaking the build and
major functionality
• Takes about 1.5 hours

32. Frequent Major Releases
Feature
Freeze
Release
Freeze
Done Done Done Release to Internal
Sandbox &
Production Instances
SB/R0 Release
• 2/3 sandbox instances
• Production instance
where Salesforce has
largest orgs
R1/R2 Release
• 25% of prod instances
• All remaining instances
• Branch locked
• Check-in approval required
• Incomplete features disabled
• Code line open for next release
Monthly Sprint Reviews Release Sprint Staggered Release
Scrum Teams
and Functional Areas
Sign Off
Scrum Teams
Sign Off
Dec Jan Feb Mar MayApr Jun
• Continuous integration w/ 600k
JUnit and Selenium tests
• Performance testing
• 110M Apex customer tests
• Other production tests
• Final performance testing

33. Next Steps
What you can do today
Get Developers to work closer with Quality Engineers
• Discuss User Story scope and requirements together
• Participate in Code Reviews
• Have Dev and QE pair on story completion
Test Automation
• Add Automated Test Coverage to Definition of Done
• Create backlog to track remaining Automation work
• Involve Developers in writing Automation and helping QEs with
Test Frameworks
• Test Failures locks when too high
• Enforce Test Automation health
• Invest in Auto-Assign tools and reports

34. thank y u