This workshop was delivered by Conjur at ChefConf 2015

1. 2015 #ChefConf Workshop
Building a Self-Documenting Compliant Application
www.conjur.net | @conjurinc

2. © Confidential
Set-up Prerequisites
1.Sign up for the Conjur demo appliance
http://info.conjur.net/secrets-management-demo
2. Login to the demo appliance
If you’re on Windows, use Putty
http://bit.ly/1NAHh3Y
Ready To Deep Dive?

3. © Confidential
Welcome!
I. Agenda
II. Workshop Goals
III.Meet Your Workshop Hosts, Conjur Introduction
IV.Ready To Deep Dive? Set-up Conjur Prerequisites
● Part I: Introduction to DevOps and Compliance / Q&A
● Part II: Chef Secrets Management Overview / Q&A
Agenda

4. © Confidential
- Break -
● Part III: Practicum - Build a Sample App
● Part IV: Compliance & Chef; Good Practice / Q&A
● Part V: Traceability, Auditability and Chef / Q&A
V. Wrap up
Agenda

5. © Confidential
Workshop Goals
After this workshop, you will:
● Have a deeper understanding of how compliance with
Chef, access to the Chef Server, and Secrets
Management are interrelated.
● Know how to implement a delivery pipeline that
combines the two topics in a self-documenting and
highly automated fashion.

6. © Confidential
Your Workshop Hosts
● Kevin Gilpin, CTO and Co-Founder, Conjur - Kevin has 19 years of technical
leadership in enterprise software, delivering solutions to the Fortune 500. He has held technical
positions at Infinity Pharmaceuticals, Praxeon Inc, i2 Technologies, and Trilogy, where he developed
technology that delivered pioneering interactivity on the web and has ultimately been adopted across
the industry. @kegilpin
● Elizabeth Lawler, CEO & Co-Founder, Conjur - Elizabeth Lawler is CEO and Co-
founder of Conjur, Inc. Elizabeth has over 20 years of experience working in highly regulated and
sensitive data environments. Prior to founding Conjur, Elizabeth was Chief Data Officer of Generation
Health and held a leadership position in research at the Department of Veterans Affairs. She has
been a programmer herself and is constantly working to make software development and IT systems
easier to manage for people working in regulated industries. @elizabethlawler

7. © Confidential
Your Workshop Hosts
● Dustin Collins, Developer Advocate, Conjur - Dustin is a polyglot engineer and
Conjur's developer advocate. He organizes the Boston DevOps meetup and is most interested in how
we as a community can iterate on our processes and tools to enable continuous delivery of quality
software. @dustinmm80
● Christopher Farnham, Novartis - Chris is a proven software development and operations
team leader within diverse domains, including banking, publishing, military and biotechnology. He has
demonstrated deep and broad knowledge of SDLC processes within highly regulated, demanding (but
ultimately person driven) environments. @chrisfarnham

8. © Confidential
Conjur Introduction
We believe that security success for all organizations
is determined at the business and infrastructure levels
● Conjur is a privileged access security platform for large and fast-
growing cloud and web-based enterprises using next generation
technologies.
● It provides a smarter, transparent, automated way to scale and
manage critical secrets and access control. It’s used on-premise or
in your cloud (private, public, hybrid).

9. © Confidential
Conjur Introduction
Application Auth
Systems Access
Internal Network
Physical Infrastructure
Firewall
Control Plane
We provide the modern access
management and audit solution
for the missing pieces: Systems
Admin and the Systems Control
Plane.
Conjur Resolves The
Access Control Gap

10. © Confidential
Conjur Introduction
● Conjur solves three critical infrastructure security growth
issues:
> Secrets Management
> Access Control (SSH) (e.g., machines, robots,
humans)
> Audit (Compliance)
● Integration-Friendly! We work with all technology stacks.

11. © Confidential
Conjur Clients

12. © Confidential
Part I.
Introduction to DevOps and Compliance
Compliance for People who Cringe at “Compliance”

13. © Confidential
How the Business Views DevOps
How does DevOps
work?
Magic.

14. © Confidential
Compliance and DevOps
Source: DevOps: The Worst-Kept Secret to Winning in the Application Economy CA Technologies
October 2014 cite: http://rewrite.ca.com/us/~/media/rewrite/pdfs/white-papers/devops-winning-in-
application-economy.pdf

15. © Confidential
Goal: Reproducible, Reliable Processes
● Same packages
● Same configuration
… and…
● Same security and
compliance controls

16. © Confidential
● Automated and traceable authorizations for promotion of code to
production
● Role-based access controls that acknowledge when DevOps personnel
have access to production systems and document the specific use cases
● Encryption and logical access controls which essentially “lock-out”
unauthorized access (to systems or data)
Adapted From Brightline https://www.brightline.com/2012/12/auditing-devops-developers-with-
access-to-production/)
Auditors/Compliance want to see

17. © Confidential
● File integrity monitoring (and alerting) on changes to production code
versus the traditional focus on critical operating system executable
● File access monitoring on the source code itself with appropriate alerting
● Extensive logging and daily, if not real-time, log review of the above data
sources
Adapted From Brightline https://www.brightline.com/2012/12/auditing-devops-developers-with-
access-to-production/)
Auditors/Compliance want to see

18. © Confidential
● Single source for all change requests
○ No change is deployed without a request
○ Intent is that most problems can be diagnosed via the change reporting systems
● Clear separation of roles between developer and deployer
○ Or at least developer cannot deploy without review
● Clear metrics to demonstrate improvement
○ Mean Time To Recovery
○ Percentage of failed deployments/changes
Goals for delivery automation

19. © Confidential
Dev, Test, & Prod
Environments
Code Review
Infrastructure
Source Code
Goal automate documentation of this
Infrastructure
Code Developer
deploy
Continuous
Build & Unit
Test
Config, Release,
Deployment
commit on branch build
check
approval
tests pass

20. © Confidential
Part II
Chef and Access Management

21. © Confidential
● Access control is determined by “gates” which can be opened and closed
by human or automated processes
● SSH gates access to production machines by developers and code
● Secrets gate access to production services by developers and code
○ Access to a secret constitutes access to a production service
The Importance of SSH and Secrets

22. © Confidential
State of Chef + SSH
Combining users + sudo cookbook is a popular solution.
Workflow
● Create a data bag for your users and place it in a my_users
cookbook
● Create JSON entries for each user w/ group membership
and public SSH keys
● Upload the data bag and cookbook to Chef server
Include the sudo recipe in my_users cookbook to set
passwordless sudo for your users

23. © Confidential
State of Chef + Secrets
chef-vault + IAM is a popular solution
workflow
● Create IAM JSON file defining access permissions to
S3 buckets
● Script out creating IAM users/groups/policies for
specific access to a set of secrets
● create a vault and encrypt the JSON output of the
previous step, tied to a specific Chef query

24. © Confidential
State of Chef + Audit
Chef Analytics
● Tracks changes to Chef objects (clients, nodes, etc)
and who made them in real-time
● You write controls into your recipes, compliance tests
that report back to analytics after Chef run
Guardrail
● SAAS that tracks changes to infrastructure in real-time,
can trigger alerts on specific changes

25. © Confidential
We will create a simple, but real infrastructure using Chef, Conjur, and
Docker. Infrastructure activity and access will be fully managed and
audited.
Workshop Flow - Overview
Chef Chef Server is the definitive source of server configuration.
Chef deploy and validation keys provide access to Chef Server.
Chef provides audit and verification of server installs.
Conjur Provides User, Group, and Host identity.
Stores and distributes secrets.
Manages SSH access to machines.
Records audit of all permission changes, access to secrets, and access to
hosts, plus custom audit records of cookbook pushes.
Docker Used to run infrastructure machines:
deploy can push cookbooks to Chef Server. Accessible only to ops.
sentry application server. Accessible to ops and developers.

26. © Confidential
◁ Move to the source directory and fetch all the code
Workshop Flow - Get the Code
cd chefconf15
git submodule update --init --recursive
Submodule 'cookbooks/foundation' (https://github.com/conjurdemos/chefconf15-
base.git) registered for path 'cookbooks/foundation'
Submodule 'cookbooks/sentry' (https://github.com/conjurdemos/chef-sentry)
registered for path 'cookbooks/sentry'
…

27. © Confidential
◁ Create an ops group to own the key
◁ You’re a member of security_admin, so you’re also a member of ops.
group
conjur group create --as-group security_admin ops
{
"id": "ops",
"ownerid": "demo:group:security_admin",
"gidnumber": null,
"roleid": "demo:group:ops",
"resource_identifier": "demo:group:ops"
}

28. © Confidential
◁ Store the public key of the ‘demo’ user (that’s you) in Conjur.
Workshop Flow - Load your Public Key
conjur pubkeys add demo @~/.ssh/id_rsa.pub
Public key 'demo@ip-10-97-161-35' added

29. © Confidential

30. © Confidential
◁ Copy deploy.pem and validation.pem from your Chef
Server
◁ Save to the local files : deploy.pem and validation.pem
◁ deploy.pem has permission to manage cookbooks
◁ validation.pem used to register new machines with
Chef
Workshop Flow - Download the Chef Keys

31. © Confidential
Workshop Flow - Store the Chef deploy.pem
conjur variable create --as-group ops -m application/x-pem-file
chef-server/deploy/private_key
cat deploy.pem | conjur variable values add chef-server/deploy/private_key
{
"id": "chef-server/deploy/private_key",
"mime_type": "application/x-pem-file",
"kind": "secret",
"version_count": 0
}
Value added
◁ Create the variable and store the key

32. © Confidential
◁ Store the Chef server validation key in Conjur
Workshop Flow - Store the Chef validation.pem
conjur variable create --as-group ops -m application/x-pem-file
chef-server/validation/private_key
{
"id": "chef-server/validation/private_key",
"mime_type": "application/x-pem-file",
"kind": "secret",
"version_count": 0
}
cat validation.pem | conjur variable values add chef-server/validation/private_key
Value added

33. © Confidential

34. © Confidential
◁ Foundation image cookbook is pre-loaded into the Chef server. Along with
Conjur auth and RBAC, this is the foundation of trust in the infrastructure.
⊃ Base OS.
⊃ Packages - Chef, Conjur
⊃ SSH access - Conjur SSH - login with your own private key, always
⊃ /etc/conjur.conf + /etc/conjur-demo.pem - Verified TLS for everything
⊃ Only public info! No secrets.
https://github.com/conjurdemos/chefconf15-base
Workshop Flow - Foundation Image

35. © Confidential
“Deploy” server builds on the foundation image
◁ .conjurenv → Chef validation.pem
◁ client.rb → Chef client configuration
◁ deploy.sh → Docker entrypoint, uploads all cookbooks to Chef server.
Cookbooks should be volume-mounted to Docker
Workshop Flow - Build Deploy Server
docker build -t deploy deploy/

36. © Confidential
Workshop Flow - Create Deploy Host
conjur host create --as-group ops deploy-01 | tee deploy-01.json
◁ Conjur - Create a Host identity for the Deploy server
◁ Assign ownership to the ops group
cat << ENV > deploy-01.env
CONJUR_AUTHN_LOGIN=host/deploy-01
CONJUR_AUTHN_API_KEY=$(cat deploy-01.json | jsonfield api_key)
ENV

37. © Confidential
◁ Launch the deploy host
◁ Host does not have access to the secret … yet
Workshop Flow - Test the Deploy Host
docker run --env-file deploy-01.env --entrypoint conjur deploy env check
error: Some variables are not available
CLIENT_PEM: unavailable

38. © Confidential
◁ Give access to the Chef key to the deploy host
Workshop Flow - Grant Access to Chef
conjur resource permit variable:chef-server/deploy/private_key
host:deploy-01
execute
Permission granted

39. © Confidential
◁ Run the deploy box as a daemon. We will use SSH to access it.
Workshop Flow - Launch the deploy box
docker run -d --env-file deploy-01.env
--name deploy
-v $PWD/cookbooks/:/cookbooks
-p 2200:22
deploy
*** Running /etc/my_init.d/10_foundation-configure.sh...
[2015-03-28T20:56:58+00:00] INFO: Forking chef instance to converge...
[2015-03-28T20:56:58+00:00] INFO: *** Chef 12.0.3 ***
…
*** Runit started as PID 336
Mar 28 20:57:09 613420620e92 syslog-ng[345]: syslog-ng starting up;
version='3.5.3'
docker logs -f deploy

40. © Confidential

41. © Confidential
◁ Login to the deploy host
◁ On the deploy host, display user info. Group membership is controlled by
Conjur.
Workshop Flow - Login to the deploy host
ssh -p 2200 localhost
demo@e9de200cc9da:~$
id
uid=1100(demo) gid=50000(conjurers) groups=50000(conjurers)
This icon means, run this command from inside a container

42. © Confidential
◁ Use the deploy host to list cookbooks
◁ Group ‘conjurers’ is whitelisted to passwordless-sudo this command.
Workshop Flow - List Cookbooks
sudo list-all
apt 2.7.0
build-essential 2.2.1
conjur-client 0.3.1
foundation 0.1.1
sshd-service 1.1.0
terminal-login 0.2.3
yum 3.5.3

43. © Confidential
◁ Exit the container with Control-D and navigate to the cookbooks directory
◁ Rename the sentry folder to end with your initials
Workshop Flow - Build the sentry Cookbook
cd cookbooks
mv sentry sentry-yourname ← changeme!
cd sentry-yourname

44. © Confidential
Edit metadata.rb
◁ Change the cookbook name to end
with your initials
◁ Increment the version to 0.2.0
Workshop Flow - Build the sentry Cookbook

45. © Confidential
◁ Edit recipes/default.rb
Workshop Flow - Build the sentry Cookbook

46. © Confidential
◁ Login to the deploy host
Workshop Flow - Login to the deploy host
ssh -p 2200 localhost
demo@e9de200cc9da:~$

47. © Confidential
◁ This sequence will vendor the Berksfile in /cookbooks/sentry, then upload
the cookbook with dependencies.
Workshop Flow - Upload Cookbooks
sudo upload sentry-yourname
# ./upload-all /cookbooks/sentry/
Resolving cookbook dependencies...
Fetching 'sentry-keg' from source at .
Using build-essential (2.2.2)
Using python (1.4.6)
Using yum (3.5.3)
Using sentry-keg (0.1.6) from source at .
Using yum-epel (0.6.0)
Vendoring build-essential (2.2.2) to /cookbooks/sentry//.vendor/build-essential
Vendoring python (1.4.6) to /cookbooks/sentry//.vendor/python
Vendoring sentry-keg (0.1.6) to /cookbooks/sentry//.vendor/sentry-keg
Vendoring yum (3.5.3) to /cookbooks/sentry//.vendor/yum
Vendoring yum-epel (0.6.0) to /cookbooks/sentry//.vendor/yum-epel
Uploading sentry-keg [0.1.6]
Uploaded all cookbooks.

48. © Confidential
◁ Audit activity shows access to secrets, as well as the upload event
Workshop Flow - Inspect the audit
conjur audit all -s
[2015-03-30 18:28:25 UTC] demo:host:deploy-01 checked that they can execute
demo:variable:chef-server/deploy/private_key (true)
[2015-03-30 18:28:30 UTC] demo:host:deploy-01 created resource demo:chef-cookbook:foundation
owned by demo:host:deploy-01
[2015-03-30 18:28:31 UTC] demo:host:deploy-01 reported cookbook-deploy:upload; message:
Uploaded cookbook 'foundation' with command options: --include-dependencies

49. © Confidential
◁ Run a MySQL server
◁ Display the connection info
Workshop Flow - Launch database server
cd ~/chefconf15/database
source start.sh
Launching database
Database launched! Check $DB_HOST and $DB_PASSWORD
echo $DB_PASSWORD
eee5377d52c6b033

50. © Confidential
◁ Store the DB password in Conjur
Workshop Flow - Store the database password
conjur variable create --as-group ops
sentry/mysql/password
$DB_PASSWORD
{
"id": "sentry/mysql/password",
"mime_type": "text/plain",
"kind": "secret",
"version_count": 1
}

51. © Confidential
◁ Return to $HOME/chefconf15
Workshop Flow - Chdir
cd ~/chefconf15

52. © Confidential
◁ Create the host
◁ Permit secrets access
Workshop Flow - Configure Sentry host
conjur host create --as-group ops sentry-01 | tee sentry-01.json
cat << ENV > sentry-01.env
CONJUR_AUTHN_LOGIN=host/sentry-01
CONJUR_AUTHN_API_KEY=$(cat sentry-01.json | jsonfield api_key)
ENV
conjur resource permit variable:chef-server/validation/private_key
host:sentry-01
execute
conjur resource permit variable:sentry/mysql/password
host:sentry-01
execute

53. © Confidential
◁ Build the image
Workshop Flow - Build Sentry app
docker build -t sentry sentry/

54. © Confidential
◁ Run the Sentry box as a daemon.
Workshop Flow - Run Sentry
docker run -d --env-file sentry-01.env
--name sentry
-e DB_HOST=$DB_HOST
-e COOKBOOK_NAME=sentry-keg
-p 2201:22
-p 8080:9000
sentry
*** Running /etc/my_init.d/10_foundation-configure.sh...
[2015-03-28T20:56:58+00:00] INFO: Forking chef instance to converge...
[2015-03-28T20:56:58+00:00] INFO: *** Chef 12.0.3 ***
…
*** Runit started as PID 336
Mar 28 20:57:09 613420620e92 syslog-ng[345]: syslog-ng starting up;
version='3.5.3'
docker logs -f sentry
Change this to match your cookbook name

55. © Confidential
◁ Login to the sentry box
◁ On the sentry box, run chef-client
Workshop Flow - Configure sentry
ssh -p 2201 localhost
… chef-client run happens here
$ sudo conjur env run -c /.conjurenv -- sudo -E chef-client
-c /etc/chef/client.rb
-o $COOKBOOK_NAME

56. © Confidential
◁ You can now view the Sentry UI on port 8080 of your endpoint.
Workshop Flow - View Sentry UI

57. © Confidential
● Create a developers group and a user.
● Try to deploy, as this user.
Workshop Flow - Add New Team Members

58. © Confidential
Part IV
Regulatory Environments and
Chef Server
Presented by Chris Farnham, Novartis
@chrisfarnham

59. © Confidential
DevOps Mission
Development Operations strives to shorten
and amplify feedback loops, thereby bringing
software development and IT operations
closer to end users.

60. © Confidential
Macro Trends
● Self-service
● Virtualization
● Automation
● Manage sets of assets not individual assets
● External collaborations
● Data Explosion
● Globalizaton
● Mobile devices
● Internet of Things

61. © Confidential
In order for Chef to manage system configurations in our regulated
environment it needed to be regulated itself. This is daunting; the following
high-level strategy really helped me deliver
First: Describe the system and list all activities
Second: Go through each activity and ask:
◁ Can I declare this activity out of scope?
◁ Can I reuse (or map it to) a pre-existing policy or activity?
◁ Can I leverage automation (in lieu of documentation)?
◁ Document the activity at an appropriate level
The above is generally ordered best to worst (but ultimately not that bad)
Chef in a regulated environment

62. © Confidential
◁ Gather product documentation
◁ List all components and integrations to the system
◁ List all activities with the system with careful attention to status updates
◁ Describe and list the roles and responsibilities associated with the system
Example activities:
◁ Create/Modify/Update Cookbooks in Chef Server
◁ Bootstrap (add)/Remove Node
Protip: Declare a “maintenance” activity with a change tracking process of
“none” ; because nothing changes when this activity occurs
System activities

63. © Confidential
Always look to manage scope and be aware of which integrated systems are
particularly sensitive (read: important) and which integrated systems are
managed by other groups
◁ Declare which products and systems are “in scope”
◁ Explicitly declare what is “out of scope”
Examples:
◁ Repository development activities (e.g., Berkshelf, cookbook writing) were
declared “out of scope”
◁ Repository changes (e.g., upload, remove) were declared “in-scope”
◁ Supporting infrastructure (e.g., monitoring, Jenkins) was declared “out of
scope”
Managing scope

64. © Confidential
Documentation and processes already existed for many things, we just
reused them.
Examples:
◁ Server provisioning and retirement ← On-premise
◁ Incident management (we just needed to declare roles and list points of
contact)
◁ Backup and recovery
Reusing existing activities

65. © Confidential
Leveraging automation
Automated processes, where everything happens in a traceable and
auditable manner (driven out of assets managed in source control) provide
great flexibility and insulation within the change control process.
Example doc excerpt: “change tickets must be filed for all medium risk
updates unless performed via source control driven automation.”

66. © Confidential
Figuring out what should be documented at a high-level and what should be
detailed is an art but important
Control related documentation is usually tracked and signed (a rigorous
process). Do yourself a favor and only put core, less frequently changed
information within the core doc (the operations manual). Then make heavy
use of referenced “working instructions” that can be updated without going
through a heavy process
Documenting activities

67. © Confidential
◁ Really get to know the compliance group
⊃ Ask to attend meetings, request 1 on 1s, have lunch with them
⊃ Goal is to learn their language, concerns and establish a personal relationship
◁ Learn the groups, communication paths, workflows and activities within
the organization
⊃ How are change management decisions made?
⊃ Who are the operational and compliance stakeholders?
Homework

68. © Confidential
We hope you learned A LOT today!
Next Steps...
> Test drive our on-demand demo at conjur.net
> Check out our dev docs at developer.conjur.net
> Set up a live 1:1 demo to learn more
> Subscribe to our blog & monthly newsletter
> Follow us @conjurinc
Thank You!