Self-service secure test and release pipelines: Our Journey to get Jenkins to deploy and deliver software to production.

1. Self-Service Secure Test and
Release Pipelines
Andrey Falko - Lead Engineer - Diagnostics, Visibility, and Analytics Cloud
afalko@salesforce.com
Our journey to get Jenkins to deploy and deliver software to production

2. Forward-Looking Statements
Statement under the Private Securities Litigation Reform Act of 1995:
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties
materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed
or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed
forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items
and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning
new, planned, or upgraded services or technology developments and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new
functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our
operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any
litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our
relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our
service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger
enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our
annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These
documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our
Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available
and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features
that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.

3. Outline
Once Upon A Time…
From Guilds to a Factory
Pipeline Phases
Happily Ever After?
Presentation Content

4. Once Upon A Time...
● One App
● No public cloud
Imagine being a startup in 1999

5. Once Upon A Time...
● Customer data
○ Lives and livelihoods depend on it!
● Need
○ Secure chain of trust
■ Release process from source control to production (CI/CD)
■ Test and release pipelines (TNRP)
With success comes great responsibility
VCS
Test and Package
Package Store
Customer Data

6. Once Upon A Time...
● 400+ developers
● App and Platform
● CI and CD mature:
○ Automation
○ Two-man rule
○ Java, java, and more java!
Fast forward a decade: 2009

7. Once Upon A Time...
● 25 Acquisitions
● Countless apps and infrastructures
● Infrastructure gets complicated
Fast forward five years: 2014

8. Once Upon A Time...
● One of many clouds under infrastructure
● Platforms for Monitoring and Alerting
● Lots of software components
○ Some open source, some not, most somewhere in between
○ Java, Scala, Python, Go, Ruby, Erlang…
○ Ant, Maven, Gradle, Make, Rake, Tox
Formation of Diagnostics, Visibility, and Analytics (DVA) Cloud

9. Once Upon A Time...
● Same release requirements
○ Everything going into any datacenter must follow two-man rule and other compliance rules!
● Mature process?
○ It was complicated...
Guess how we coped?

10. Once Upon A Time...
● Jenkins was widely used
○ Fifty software packages shipped
○ Three different installations
○ Inconsistent authentication and authorization setups
○ Inconsistent release process enforcement
○ Lovingly hand-crafted FreeStyle jobs
The Strata team formed!
Security
Compliance Deliver Features
Service Ownership

11. Outline
Once Upon A Time…
From Guilds to a Factory
Pipeline Phases
Happily Ever After?
Presentation Content

12. From Guilds To Factory
Solution
● Generate Jenkins jobs for developers
○ Engineers-only CRUD per team
○ Test and Release Pipelines
○ Parameterized Jenkins build jobs
■ See DEMO

13. From Guilds To Factory
● Regardless of language, project, to tools:
○ How do I unit test?
○ How do I package?
○ How do I integration test my components?
○ What defines my application health?
○ Is my application stateful or stateless?
○ Where do I deploy first?
○ How large is the footprint and where?
Developers have to do all these things

14. CI: Big Picture
Our Continuous
Integration Flow
Git Master
Branch
Unit Tests
Create
Package
Integration
Tests
Promote To
Production
Git Branch
Unit Tests
Create
Package
Integration
Tests
Code Review

15. Continuous Deployment
Completing the
feedback loop Back to developer
for fixes
Production
Tests
Production
Deployments
Production
Tests
Rollback
Deployment
Monitoring
System
Canary
Deployments
Jenkins

16. Outline
Once Upon A Time…
From Guilds to a Factory
Pipeline Phases
Happily Ever After?
Presentation Content

17. Unit Testing and Static Analysis
● User input
○ Unit test command(s)
○ Docker image to use for unit test commands
■ Create pipeline for new docker image
■ Automated static analysis via SonarQube
■ Never give access to Docker daemon
How do I test?

18. Packaging
● Had to support multiple packaging formats
○ RPM, TAR, JAR, Docker
● RPM and TAR also allow users to package Docker
○ Provide a better future
○ Make legacy components more testable
● CI system controls package signing
○ Sign on separate physical machines
Package what you test, test what you package

19. Integration Testing
● Declare how components are wired together
○ See EXAMPLE
● Users create integration test that is run from a container and executed in
docker-compose.yaml
○ Output JUNIT formatted test result info to Jenkins
Test what you ship, ship what you test
Deployment Declaration
Pipeline diagram by Lauren Padia

20. Integration Testing
● Integration test failures difficult to troubleshoot
○ Only change one component at a time
○ Slow
■ Force containerization
■ Limit quantity of tests
■ Large footprints
Overcome the pitfalls

21. Delivery To Production
Master Docker Registry
R&D DMZ
Notary Master
Trusted
TNRP
PROD
Mirrored Registry
Servers
Pull
Validate
Push
Pull
Pull
Sign
Release Engineer
presses release
buttons
Data flow
VCS

22. Delivery To Production
● Fail if “FROM” contains non-approved image
● Trigger pipeline runs when parent image updated
○ Dockerfile Image Update tool
○ See DEMO
Overcome Docker pitfalls

23. Monitoring in Production
● Monitoring Driven Deployments
○ Deploy if in good state
● Time Series Metrics and Alerts
○ Continuously test
■ Why not use those integration tests?
■ Think like a customer
○ Application metrics
○ System metrics
■ CPU, Memory, Disk I/O, etc.
From TDD to MDD

24. Outline
Once Upon A Time…
From Guilds to a Factory
Pipeline Phases
Happily Ever After?
Presentation Content

25. Happily Ever After?
● 150 pipelines
● 50 pipelines runs per day
● 80 developers
● Current challenges:
○ Feedback to Jenkins on
deployments in production
○ Better Jenkins agent
support
○ Pipeline declaration in VCS
Fast-forward to today

26. Happily Ever After?
● Use Pipeline Multibranch and Github Organization plugins:
○ Don’t allow users full use of the DSL
○ Represent the same fields we have, but in “Stratafile” in VCS
● Never run a single job on master
○ Start with agents from day 1
If we were to start again today

27. Happily Ever After?
● Jenkins feature wishlist
○ User-controlled credentials for integration tests
○ AWS/Openstack plugins
■ Spin up containers for Jenkins on VMs
■ Autoscale
● Swarm or dynaslave plugins don’t fully fit our needs
Still not automated out of a job...

28. Takeaways
● Limit access to critical systems
● No Freestyle Jobs
● Split your agents by security boundaries
● Integrate Jenkins with your monitoring systems

29. Acknowledgements
● Project contributors
○ Nari Mulakala
○ Jinesh Doshi
○ Nelson Wolf
○ Justin Harringa
○ Min Ho Park
● Project management
○ Vaishali Nandal
○ Fergus Sullivan
● Chief Beard Officer
○ Ian Varley
● Doc Writer
○ Lauren Padia
● Security Reviewer
○ Travis Emmert

30. Thank Y u