Choosing & building the appropriate infrastructure to run your applications in production can be a daunting task. Typically we revisit previous choices as we approach different scales. In this talk we'll cover the two architectures we've been through at Workday so far and speak about our third major architectural change as we ramp up to larger numbers of servers and users. Topics will include: deployment, configuration management, automation tools, build & test pipelines, immutable infrastructure.

1. Evolving Infrastructure
Louis Dunne, Platform Engineering
louis.dunne@workday.com

2. Workday Introduction
• Background / Our Own Cloud
2.0 Architecture
• Config Management / Chef Build Pipeline /
Chef Development Tools
3.0 Architecture
• Release & Deployment Changes / Image Build Pipeline /
Release Management / Planned vs. Unplanned Changes

3. Enterprise HCM &
Financials software
in the cloud
First release
November 2006

6. Workday Introduction
• Background / Our Own Cloud
2.0 Architecture
• Config Management / Chef Build Pipeline /
Chef Development Tools
3.0 Architecture
• Release & Deployment Changes / Image Build Pipeline /
Release Management / Planned vs. Unplanned Changes

8. One of the DevOps mantras is
Infrastructure is Code

9. If it’s code…
→ you need to test it
→ you need a build & test pipeline

10. Local
Development
Code
Review
C.I.
Unit Tests
All triggered by
developers
pushing code
Bronze
Cookbook
Artefacts
Local
Development
Code
Review
C.I.
Unit Tests
Local
Development
Code
Review
C.I.
Unit Tests

11. C.I.
System Tests
C.I.
System Tests
C.I.
System Tests
Failures
(back to dev)
System Tests
Triggered on
the Hour
Bronze
Cookbook
Artefacts
Silver
Cookbook
Artefacts

12. C.I.
Integration Tests
Gold
Cookbook
Artefacts
Silver
Cookbook
Artefacts
Failures
(back to dev)
Integration Tests
Triggered Several
Times a Day

13. reekChefSpec RSpec

14. System &
Integration
reekChefSpec RSpec

15. Where Do We Run Our ServerSpec Tests?
• Lab Hardware?
• Vagrant?
• AWS?

16. The Lab?
• Hardware Based
• Can’t set machine state before the test run
• Can’t reset machine state after the test run

17. Run In Vagrant?
• Good for simple cases
• Harder for integration testing a few dozens
Chef roles
• Prefer a hosted platform with longer running
nodes for some services like artefact repos

19. Unit System Integration

20. https://www.chef.io/delivery/
https://downloads.chef.io/chef-dk/

21. Workday Introduction
• Background / Our Own Cloud
2.0 Architecture
• Config Management / Chef Build Pipeline /
Chef Development Tools
3.0 Architecture
• Release & Deployment Changes / Image Build Pipeline /
Release Management / Planned vs. Unplanned Changes

22. Platform Services
Appliances
Workday
Linux Servers
Image Based
Deployment
Chef / Cobbler
Based
Deployment

23. ■ Cobbler for the OS
■ Chef based deployments
of system / infrastructure
changes
■ Custom tooling for
applications deployments
■ Cobbler / Chef for bare
metal
■ Most services moving to
image based deployments
■ Custom deployment tools
to manage VM lifecycle
2.0 Deployments 3.0 Deployments

24. 1. Where in the build & test pipeline do
the Machine Images get created?
1. What technology & processes are
used to create them?

25. • Early in the pipeline
• Application teams → image artefact
• Image artefact → build & test pipeline

26. • Lots of tools to choose from:
• Diskimage-builder
• VMBuilder
• Box Grinder
• Packer
• Imagefactory
• We use Oz (https://github.com/clalancette/oz)

27. KickStart
File
OZ
Template
Base
Image
OZCentOS
Core
Image
Manifest
Unit + System
Tests
System
RPMs

28. OZ
Template
Application
Image
OZ
Base
Image
Image
ManifestApplication
RPMs
Image Build Service

29. Gold
Cookbook
Artefacts
Gold
Application
Artefacts
Promote
to Staging
UNIT SYSTEM INTEGRATION
UNIT INTEGRATION
I N F R A S T R U C T U R E
A P P L I C A T I O N S
SYSTEM
Promote to
Production

30. Image Build
Service
SYSTEM TESTS
UNIT INTEGRATION
I N F R A S T R U C T U R E
SYSTEM
Promote
to Staging
A P P L I C A T I O N S
Base
Image
Base
Image
Application
RPMs
Application
Image
Application
Image

31. The ability to push code to environments
easily and quickly - push button deploys
A stable framework for development,
testing, deployment and auditing
Deploy applications and operational
environments in the same way each time
Source code, tools and all components
that make up releaseIdentifiability
Reproducibility
Consistency
Agility

32. ■ Images
■ Startup Properties
■ Cookbooks
■ Roles
■ Data Bags
■ Encrypted Data Bags
■ RPMs
■ Other artefacts (e.g. ruby
& python packages)
Config Management Image Deployment

33. ■ Destroy and recreate
rather than change in
place
■ All facets of the OS are
captured by the image
artefact
■ Server state mutated over
time as updates are
applied
■ Impractical to manage
every last detail of the OS
with config management
Config Management Image Deployment

34. • The ideal is…
• Push all changes through the image pipeline for
both planned and unplanned changes
• If your pipeline is reliable with a quick
turnaround you can use this for all changes
• We also like to have a Break The Glass option
Planned Vs Unplanned Changes

35. • For low impact changes only
• Bash script → RPM
• Agent on each VM can deploy RPMs
• Trigger remotely via secure channel
• Testable, easy to roll out, good for auditability
Break The Glass

36. MonitoringDeployment
(*)
(*) developed in house

37. Stats & MetricsLogging

38. “If it hurts,
do more of it”

39. http://www.beatcleaver.com/portfolio/
https://www.flickr.com/photos/cote/
DC Image
Melissa Stolberg
Michael Coté
Paul McAuley