The document summarizes dependency management practices at Salesforce.com for their large agile development environment with over 30 Scrum teams. Key practices include a release kickoff meeting to share team plans, a dependency identification exercise to map dependencies, release open space discussions, functional design reviews, a virtual architecture team, continuous integration processes, and allocating team time to address dependencies. These practices aim to improve visibility, collaboration and coordination across teams.

1. Agile 2008 Conference
Dependency Management in a Large Agile Environment
Eric Babinet Rajani Ramanathan
salesforce.com salesforce.com
ebabinet@salesforce.com rajani@salesforce.com
Abstract
2. Team Structure
Salesforce.com’s R&D organization has over 30
Scrum teams working simultaneously in a single Product development at salesforce.com is organized
release code branch. This report highlights practices into three business units: Applications, Platform, and
that salesforce.com has been using successfully to Core Infrastructure. There are 30 plus Scrum teams
scale Scrum and to manage inter-team dependencies. distributed across these three business units. Each
Scrum team typically has dedicated developers, quality
assurance engineers and a Product Owner. Each team
1. Introduction also has a ScrumMaster, who is usually a program
manager, development manager, or QA manager.
In October 2006, salesforce.com's R&D Depending on the complexity of the features being
organization embarked on a large scale transformation developed, the Scrum team may also have dedicated or
from a waterfall software development approach to an part-time team members from other functional teams
agile approach based on Scrum. It had been 10 months such as System Testing, Documentation, UI Design,
since the last major release, the planned release date of Usability, Technology Operations, and Release
the next release had moved five times, and many Engineering.
people were frustrated by the inability to deliver Typically all Scrum teams are working on the same
frequently and on schedule. Instead of waiting until the major release and in the same codeline/branch. Given
release was completed, we reconfigured existing teams the integrated nature of our products, the features
into Scrum teams and used the Scrum process to implemented by these various teams can be very
deliver the in-progress release to our customers in tightly interwoven, both functionally and technically.
February 2007. Since then we have delivered five From a technical implementation perspective many
major releases (at 3-4 month intervals) of our teams may be using common shared code. From a
Software-as-a-Service (SaaS) application suite and functional perspective, teams working on features
Force.com platform using our new agile approach. targeting the same end user need to coordinate closely
Every release has gone into production exactly on the to provide a consistent and coherent user experience.
pre-planned release date.
During the transition, Scrum offered significant 3. Why is Dependency Management
guidance for individual teams, but not a lot with
Difficult?
respect to coordinating between teams. We organized
teams to minimize dependencies, but the code hadn’t
With so many teams working together on the same
changed overnight and there were still significant
release and on shared features and code, it is
dependencies between teams. We implemented scrum-
impossible to anticipate all the issues, surprises,
of-scrums meetings early and they were helpful for
changes, failures and successes that teams will
discussing issues and status, but not sufficient. Over
encounter during software development. That
the last five releases we've tried out and refined a
unpredictability is one reason that dependency
number of additional practices to improve coordination
management is difficult. This section highlights some
between teams. This report highlights challenges we’ve
other reasons why dependency management is
encountered with respect to dependency management
difficult. Any solution must address these underlying
in a large agile environment and how we overcame
tensions and challenges.
them.
978-0-7695-3321-6/08 $25.00 © 2008 IEEE 401
DOI 10.1109/Agile.2008.58

2. The ability of teams to change priorities each sprint is a
benefit of Scrum, however, it can make managing
3.1. System Complexity
dependencies more difficult. Dependency identification
and analysis cannot just happen once at the beginning
The first step in managing dependencies is to
of the release, but rather must be an ongoing process.
identify them. Like most mature software,
Over the course of a release dependencies come and go
salesforce.com's systems are complex enough that one
as teams refine what they can deliver, and any process
person or team cannot possibly know about all
for managing those dependencies needs to be equally
dependencies. Increasingly we must rely on the
dynamic.
collective knowledge of many people to identify and
understand dependencies. Pooling this collective
3.4. Short and Overlapping Release Cycles
knowledge and connecting those who have the
knowledge with those that need the knowledge is
Short release cycles mean that dependencies and
essential. Experts with broad knowledge of the system
impact must be understood early as there is less time to
are highly valued and play a key role in identifying the
coordinate with other teams. Salesforce.com’s release
dependencies and impact of proposed changes.
cycles are designed with some overlap, meaning that
However, as the system grows in complexity it is hard
the planning process for the next release starts before
to avoid more specialization and knowledge
the prior release is completely finished. This is
fragmentation. Given the volume of change it is also
intentional as some teams are completely finished and
no longer practical for these experts to review the
need to be able to move on to the next release.
detail of every feature being developed. We need a
However, teams that are not finished may fall behind
method to identify changes with the highest likelihood
in their release planning and be less available to
of impacting or depending on other teams, so we can
discuss next release dependencies with other teams.
focus our attention on those areas. Of course this is not
This is problematic given the importance of collective
easy, as deceptively small changes can have a big
knowledge in understanding dependencies.
impact.
4. Specific Practices
3.2. Conflicting Priorities
Dependencies can lead to conflict between teams. If This section describes the specific practices we are
one team needs another team to do something, the using to manage dependencies and address the above
product owners for those teams must negotiate the challenges. Some common strategies incorporated into
relative priority of that work. If they negotiate these practices include:
effectively and reach a common understanding for • Create visibility: for release plans,
when the work can be done, then managing that dependencies, designs, build status, etc.
dependency should be relatively easy. If they don’t • Provide forums to stimulate collaboration and
reach a clear agreement, or if the work is still a lower knowledge sharing between teams.
priority for the other team, there will be greater • Promote self-organization and decentralized
uncertainty and risk around that dependency. dependency management.
Another type of conflict occurs when one team • Automation, automation, and more automation.
makes a change that imposes work on another team. A
common example is when one team does architectural
4.1. Release Kickoff
refactoring that requires supporting changes by other
teams. The impacted teams may get no short-term
About one month prior to the current major release
benefit from these changes, and may resent having to
going live in production, we hold a release kickoff
make them, especially if the need for them is
meeting for the next major release. This is an all-hands
unexpected. We attempt to identify these types of
one hour meeting in which the Senior Vice President
changes early in the release, but inevitably there are
for each business unit gives a high level presentation of
some that show up later in the release, either because
the features that each team is planning to build in the
they arise opportunistically or due to a change in
next release. This meeting has a number of benefits
priorities.
with respect to dependency management. First and
foremost, it motivates all teams to complete their initial
3.3. Dynamic Scope release plan by a specific date. If a team has leftover
work from the current release they can fall behind in
planning for the next release, but the Release Kickoff
402

3. helps teams stay on track with that planning. It is each Scrum team in a room with a large white board.
difficult to identify dependencies or negotiate Usually the representative is the ScrumMaster or
commitments with other teams if those teams don't yet Product Owner from the team, but it can be anyone.
know what they're doing in the release. The Release The first part of the meeting involves all team
Kickoff meeting acts as an important synchronization representatives going up to the whiteboard
point for teams, and helps ensure more productive simultaneously and creating two diagrams representing
discussions around inter-team dependencies. the dependencies between teams. One diagram
The second major benefit results from the captures teams that need another team to do something
visibility that the meeting creates around team release for them. The second diagram captures teams that are
plans. Our goal is to generate a high level of awareness doing something that will impact other teams. For
of what teams are doing in the release, as we believe these diagrams we use a simple notation:
that this visibility will naturally cause people to think • A circle represents a team
about and discuss dependencies. In order to encourage • An arrow between two circles represents a
attendance and focus attention on the release plans, the dependency. On the first diagram the arrow
meeting is given a high-profile and all R&D executives points from the team doing the work to the team
attend. The high level of participation helps align that needs the work done. On the second
expectations and create broad awareness of what is diagram the arrow points towards the team that
planned for the release. is impacted.
Of course, team release plans can and do change • A label on the arrow describes the dependency
throughout the release. We recently started reviewing • If a dependency is committed (i.e. the other
an updated version of the Release Kickoff presentation team has agreed to do the work) we put a box
during the monthly sprint reviews. The updated around the dependency label.
presentation highlights features that have been dropped
or added to the release, and has proven to be very At first there is a commotion as everyone draws
effective at maintaining release plan visibility their dependencies on the whiteboard and onlookers
throughout the release. start asking questions and pointing out dependencies
that are missing. After about 20 minutes the diagram
4.2. Dependency Identification Exercise stabilizes and comments subside. At that point we ask
each team representative to briefly describe their
Once teams have their initial release plans, they can dependencies.
have a more meaningful discussion about In a very short amount of time we generate a
dependencies. There is a lot of informal discussion comprehensive view of the dependencies in the release.
between teams to negotiate dependencies, but we have Figure 1 shows the typical output of this meeting after
found it very valuable to facilitate a more formal it has been copied into a more readable format. Hot-
Dependency Identification exercise. This is a highly spots (i.e. teams with a lot of dependencies) are
interactive and collaborative event that involves immediately visible. Teams that have not yet obtained
representatives from all teams. We have found it most commitments for their dependencies are also visible.
effective to do the exercise shortly before the Release These teams have a higher risk of not delivering on
Kickoff as it improves the quality of the release plans their release plans until the dependencies are
presented at the Release Kickoff. committed.
The actual logistics of the Dependency Identification
exercise are simple. We gather representatives from
403

4. Figure 1. Example team dependency diagram
organization we are experimenting with some
4.3. Release Open Space
variations to see what works best, including:
We recently started holding an open space style
• Generating topic ideas prior to the open space
meeting during the week after the Release Kickoff. The
•
purpose of this open space is to create a forum for Encouraging senior management participation
discussing questions or concerns regarding team • Changing the number and length of breakout
release plans. We ask that at least one representative sessions
from each Scrum team attend, but otherwise the
meeting is optional. In standard open space style, 4.4. Functional Design Reviews
attendees propose topics at the beginning of the
meeting. Then we have 45 minutes for breakout The goal of the functional design review meetings is
discussions and 30 minutes for reports from the to improve the overall quality and usability of our
breakout groups. Popular topics include wanting to products by:
know more about functionality that is new and which • leveraging the collective wisdom and creativity
can benefit other teams, and wanting to know more of our product teams
about changes that will impact other teams. Attendees • improving design coherence across products
have reported the open space to be educational and
• creating synergies through cross-team
have found it interesting to be in discussions with
knowledge sharing
people with whom they do not normally associate.
Since the open space meeting format is new to the
404

5. These are cross-functional and cross-scrum team least 20% of their time every release towards
meetings that focus on the functional and user implementing changes recommended by the VAT.
interaction design of features deemed to be higher risk, The VAT meets for two hours twice a week to
higher complexity, or higher impact. These meetings review the technical implementation of products and
are intended to break down team silos and surface features being built by the Scrum teams. The teams
design inconsistencies and dependencies of which the building the most complex features in the release are
Scrum team may be unaware. asked to present to the VAT. The group provides
There are two different parts to the functional valuable feedback to the Scrum team on how their
design review meetings. Early in the release cycle the technical design will impact or be impacted by other
discussions focus on design concepts and aim at areas. The VAT focuses primarily on the technical
achieving functional design coherence and synergy implementation, especially scalability and performance
between existing and new features. These discussions considerations. Teams asked to make significant
are usually attended by the product owners of each changes must present again in the same release cycle
Scrum team and UI designers, and there is little and provide details on how they modified their design.
representation from other functional teams such as
development or QA. 4.6. Continuous Integration
Starting near the middle of the release cycle the
participants change to primarily include QA and some We have a web-based automated build, test and
development team members, and the focus is on triage infrastructure that provides a continuously
reviewing implementation details. These discussions integrated build system and allows us to track the
provide insight and clarity around: health of any given codeline. This infrastructure is
• Additional dependencies between teams critical for enabling all developers and QA engineers to
• Additional integration testing that needs to be work in a shared clean codebase.
added to the test plan The primary test suite is built using an extension of
• The release risk of a particular feature, and the JUnit framework and provides a mechanism for
perhaps the need to restrict availability of that implementing functional tests through the Force.com
feature API. We also have a UI testing framework using
• Deployment specific tasks Selenium for automating test cases that need to be
executed using the UI.
Some fundamental tenets guiding our automation
4.5. Virtual Architecture Team
approach are:
1. Provide fast feedback to developers so they
The Virtual Architecture Team (VAT) is virtual
understand the effects of their changes. Each check-
as it is made up of developers from every Scrum team.
in triggers a build and a subset of tests to be run. Build
Members work on the VAT in addition to being on a
failure notifications are sent to the responsible
Scrum team from the Application, Platform or Core
engineers immediately. The basic test suite runs within
Infrastructure business units.
half an hour and the developer is notified of the results
The VAT owns maintaining and extending our
of their change. Extended test suites run periodically
industry-leading software architecture. They do this by
throughout the day.
defining the architectural road map, by reviewing
2. Fix build and test failures quickly. To ensure
architecturally significant changes to the code, and by
this, we have a rotating “build master” role that
defining coding standards to ensure consistent and
changes each week. This role is typically filled by two
maintainable code.
people, a dev manager and a senior developer, who
The VAT maintains a backlog of major architectural
coordinate closely. The build master is responsible for
projects or refactoring changes that need to be
monitoring build results, tracking test history, tracking
implemented. As the product keeps evolving, we need
failures across different code lines, and assigning bugs
to redesign features and remove old code that is no
to fix failures. The test results are logged in a database
longer optimal. Developers are sometimes reluctant to
and metrics are collected on a central dashboard (% of
tamper with programs that already work, even when
tests passing, build times, number of failures, etc.) The
they know it would be better if coded differently.
build master also uses this data to send out status
Product Owners share their reluctance as they would
emails.
rather see new features developed than have something
3. High automated test coverage. Scrum teams
that already works be rewritten. To counteract these
typically aim to have over 70-80% of code covered by
tendencies, the Application, Platform, and Core
automated tests. Our large collection of automated tests
Infrastructure business units are asked to allocate at
405

6. is one of our biggest assets and a key enabler for 5. Conclusion
delivering high quality on-time releases every 3-4
months. Salesforce.com has demonstrated that it is possible
to scale Scrum and we feel very confident in our ability
4.7. Scrum-of-Scrums to continue scaling as the organization grows.
Dependency management and cross-team coordination
Each business unit has a weekly scrum-of-scrums are a significant challenge as the number of teams
meeting and there is also a weekly scrum-of-scrum-of- increases. Having a robust build and automated test
scrums. Occasionally we will form additional scrum- infrastructure is absolutely critical. Increasing visibility
of-scrums if there is a cluster of teams working closely and alignment through practices such as the release
together on a common goal. We have experimented kickoff, the dependency identification exercise, scrum-
with a few different formats for the scrum-of-scrums. of-scrums, and lightweight status reports is very
We started out using the standard 4 question format helpful. Ultimately it comes down to effective
where teams report on what they've done, what they're communication, collaboration, and knowledge sharing
going to do, what is blocking them, and if they are between teams. Practices such as the virtual
doing anything that will impact another team. That architecture team, the functional design reviews, the
worked fine initially, but became a bit tedious due to release open space, and the scrum-of-scrums are all
the large number of teams. We have since moved to a designed to open communication and promote
more open and self-organizing format, in which collaboration.
attendees propose discussion topics by writing them on
the whiteboard at the beginning of the meeting. This
approach leads attendees to take responsibility for the
content of the meeting and has resulted in more
productive and collaborative discussions. The topic of
cross team dependencies does come up during the
scrum-of-scrums, especially early in the release cycle.
The Dependency Identification Exercise is held during
a scrum-of-scrums meeting and for the next two to
three weeks after that we will typically discuss changes
to dependencies during the scrum-of-scrums.
4.8. Status Reports
Written status reports are often discarded when a
team moves to Scrum as visibility into team status is
provided by other means (e.g. sprint review, sprint
burndown chart, daily standup). When we adopted
Scrum we decided to keep a lightweight status report
written weekly by each ScrumMaster. When we were
using the 4 question format for the scrum-of-scrums
there was definitely some redundancy between the
team's scrum-of-scrums update and the team's status
report. However, now that we use the open format
scrum-of-scrums, the weekly status report is actually
an important complement to the scrum-of-scrums. We
don't discuss individual team status in the scrum-of-
scrums unless specifically raised as a discussion topic,
however, the status is always available in the weekly
report. Dependencies, blockers, and risks are all
highlighted in the report. While the report is a little
extra work for ScrumMasters we feel that the visibility
it provides is worth the cost.
406