This document discusses DevOps practices at Salesforce, including: - Using continuous delivery pipelines to plan, build, test, release, deploy and monitor code changes. - Organizing teams using a Scrum model with sprints every two weeks and an average size of 7 engineers per team. - Maintaining sandbox environments like development, test, UAT and production for each project. - Following a branching strategy in version control and using continuous integration to test and deploy code changes.

1. Scott Glaser
salesforce, Sr. Manager Software Engineering
Vamshi Gandham
salesforce, Lead Software Engineer
DevOps in App Cloud
Salesforce IT

2. Forward-Looking Statements
Statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any
of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking
statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or
service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for
future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts
or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our
service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth,
interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible
mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our
employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com
products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of
salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most
recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information
section of our Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not
be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available.
Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.

3. DevOps Vision
Continually improving the application lifecycle, making teams more productive,
improving collaboration, and delivering high quality products faster

4. • Salesforce Ohana
• Families are bound together and members cooperate and remember one another
• Social Intelligence is a combination of mental, emotional, physical and community
intelligence
Collaborative Culture

5. Continuous Delivery Pipelines
Plan DeployBuildCode Test Release Monitor
One pipeline per technology stack (apex, java, ruby, mobile, etc)

6. Continuous Delivery Transformation
Organize team
Environments
Version Control System
Continuous Integration
Test Automation
Metrics & Dashboard
Continuous Improvement
Trail Starts Here

7. Team Organization
• A project consists of one or more scum teams
o Average team size is 7, sprint every 2 weeks
o 250 Engineers, 50 Teams
o Use a project playbook
• We use Agile Accelerator (GUS)
• We store documents in google docs
• Run large projects with smartsheet
• We use salesforce chatter

8. Sandbox Environments
DEV TEST UAT
Dev
UAT
Dev
TeamA
Dev
QA
TeamB
Dev
TeamC
Dev
TeamDTeamE
Prod
PRD
D
Confi
g
Sand-
box
full
Sand
box
Produ
ction
Per Project
Sandboxes
INT
Perf
Per Org Sandboxes
Restricted Access
E2E
(opt)

9. VCS Branching Strategy
Feature /
Hot Fix
local
Dev 1 Dev 2
git.soma.salesforce.com
UAT
Project
CICI CICI CICI
Dev 3 Dev 4 Dev 1 Dev 2
Multi Feature or Feature Dependency Independent Feature
QA
INT QA
R
1.0
tag
Prod
Master
Dev
CI CI
Dev
CI

10. Continuous Delivery Flow
Artifacts
Commit
Test UAT ProdDev
DeployBuild
Logs
Test
ApexUnit
Checkmarx
Kumonium
iPromote
Grinder
Status
Tools

11. Metrics Dashboard

12. Apex CD Pipeline
ApexUnit Tests
(open sourced)
Kumonium UI Tests
Checkmarx Scan
(async)
Code
Coverage
Results
Results
Results
Results
Burp Scan
(async)
Results
Grinder Perf Tests
QA &
Perf
Results
Build &
Deploy
Check-in
VCS
JenkinsiPromote
GUS
Hadoop
UAT

13. CD Metrics
Lead Time and Cycle Time
Change Success Rate
Deployment Success Rate
Mean Time To Recover
CD Maturity Index

14. Stories and sprint acceptance met, definition of done
Requirements traceability, no open sprint stories
All project code on master branch
Complete runlist created and tested
Designated tests executed and logged
All P1 and P2 bugs are closed, open vs closed
Deciding When To Release

15. Continuous Improvement - CD Maturity
How do you know your continuously improving?
How do you measure team effectiveness?
How do you provide consistency across teams?

16. • Frequent Daily Check-
ins to VCS
• Integration Builds on
Dedicated Environment
• Force.com Security
Essentials (SECDEV2)
• Use a CD Pipeline
Status Dashboard
• Trend Bug Rates
• Trend Manual vs Auto
Test Cases
• % of SECDEV2
Completed
• 100% Auto Unit Test
• Post Deploy Smoke
Tests
• 50% Automated
Regression Testing
• Passive Integration
with Static Analysis
• Automated
Deployments
• Triggered Upon
Successful Builds
• Post Deploy Smoke
Tests
• VCS Triggered Build
and Testing
• Builds Stored In Artifact
Repository
• Designated Security
Owner Established
• Security Code Reviews
• Story Level Sec Bugs
• Generate Build Id to
Track in CD Pipeline
• Dependency
Management
• Feature Flags
• Automatic Code
Promotion & Build
From Dynamic VMs
Building TestingDeployment Reporting
Level2Level5Level4Level3
• Auto Deploys From
Artifact Repository
• Auto Deployment to
Any Environments
• Auto Env Data
Generation
• Deploy To Any Env
• Partial Automated
Runlist Deploys
• Auto Provisioning of
Any Environment
• Auto Rollbacks Upon
Unsuccessful
Deployments
• Blue-Green/Canary
Deployments
• Dark Launches
• Fully Automated
Runlist Deploys
• Automatic Rerun Test
Failures
• Pre Check-in
Verification
• Manual Penetration
Testing
• 100% Automated
Acceptance Testing
• Automated Perf
Testing
• Integration with
Penetration Testing
Suites (Burp, Zap)
• 100% Automated
Regression Testing
• Use Test Suites to
Categorize Tests
• Mocked Testing
• Active Integration with
Static Code Analysis
• Continuous APM
• Penetration Testing
Bug Trends
• Req, Code, Test
Traceability
• Auto Create Bugs into
Bug Tracking System
upon Failure
• Monitor & Report
Production App KPIs
• Automated Penetration
Testing
• Trend Testing
Coverage Percentage
• Track Code Changes
to Pipeline Failures
• Security Bug Trending
Per Sprint
AdvancedInt
• Use a Branching
Strategy and Check All
Code into VCS
• Code Reviewed Before
Every Check-in
• Secure Development
(SECDEV1)
• Auto Notification of CD
Pipeline Stages
• Track Story Completion
and Sprint Velocity
• Various Logging Levels
For Each Pipeline
• % of SECDEV1
• All Test Cases
Documented and
Execution Tracked
• 75% Code Coverage
• All Bugs Tracked In A
Bug Tracking System
• Separate Dev, Test,
UAT Environments
• Streamlined
Deployment Runlist
Level1
BeginnerBaseExpert
CD
Maturity
Matrix

17. Follow a scrum practice
Use source control & branching strategy
Continuously integrate (CI)
Automate testing & deployment
If anything fails stop the line
Immediate visibility & feedback
Track and measure
Use a maturity model
Continuous Delivery Guiding Principles

18. Open Source Projects
ApexUnit:
https://github.com/forcedotcom/ApexUnit
Kumonium:
Planned Release March 2017

19. Thank Y u