How to let IaC support Unit Testing by Terratest, Integration Testing by Open Policy Agent, and GitOps by Atlantis

1. Mastering IaC
the DevOps Way

2. 2006-2010
Hello!
I am smalltown
MaiCoin Site Reliability Engineer
Taipei HashiCorp UG Organizer
AWS UG Taiwan Staff

3. Taipei HashiCorp User Group
● Vault Workshop：幫 Credentials 找個
窩 (2019/06)
● DevOpsDays Taipei: AWS
Kubernetes Service Mesh Workshop
(Tomorrow 13:00~17:00)
● Hack 'n' Roll: Kubernetes From Zero
To Hero (2019/11/09)

4. About 2017...

6. But Maybe Not So Far...

9. What Problem We Facing
● Permission Control
● Infrastructure Quality
● Collaboration
● Efficiency & Productivity

10. Post-IaC Era Permission Unit Testing Integration Testing GitOps

11. Post-IaC Era Permission Unit Testing Integration Testing GitOps

12. Infrastructure as Code
● The Process of Managing and Provisioning
Computer Data Centers Through
Machine-Readable Definition Files

13. Iac First Generation
~$ apt-get update
~$ apt-get install
-y tar=1.16.1
package 'tar' do
version '1.16.1'
action :install
end
👉 Record Your Provision Procedure with
CM Tool, Not Document !

14. R.I.P. Configuration Management

15. Why?
Stateful Service
Stateless Service
Cloud Provider
Orchestrator

16. IaC Second Generation
👉 Record Your Cloud Resource with IaC
Tool, Not Document !
resource "aws_s3_bucket" "b" {
bucket = "my_tf_test_bucket"
acl = "private"
tags { Name = "My_bucket" }
}

17. IaC Second Generation is Hot Now!
… etc
AWS CDKPulumiTerraform

18. Post-IaC Era Permission Unit Testing Integration Testing GitOps

19. Do you know the Prod.
environment broken ? 😠
I think I execute
terraform destroy
in Alpha...maybe 😱

20. We need the multiple
accounts or projects
architecture with
cloud provider 🍯
I feel it’s complicated,
why we need that 🤔

21. Who Can Cause a Service Broken?
Developer!
Who Can Ruin Entire Infrastructure?
SRE!

22. Multiple Accounts/Projects Architecture?!
● There are Multiple Environments for One
General Application
● Due to Application Need to be Tested, but
not Impact the Real Users
● Production Environments Don’t Allow
Access at Will
● Infrastructure Becomes Code Now, Hence,
It Need to be Treated as The Same Way

23. Multiple Accounts/Projects Architecture
Beta
RobotRead Write
Prod
RobotRead Write
Alpha
RobotRead Write

24. Multiple Accounts/Projects Architecture
● The Write Role is For Human Usage
if Necessary
● Production Write and Robot Role
Only can be Permitted Through
Change Management Process
● Don’t Manage Infrastructure By
Your Laptop
Prod
RobotRead Write

25. Don't Repeat Yourself (DRY)
● Keep Code DRY
● Keep Configuration DRY
● Keep CLI Flags DRY
● No Matter What IaC Tools Been Used, You
Could Take a Look at Terragrunt

26. Keep Code DRY
● Separate Different
Account or Project by
Folder
IaC
├── aws
│ ├── alpha-ac
│ │ ├── us-east-1
│ │ │ ├── database
│ │ │ ├── elastic
│ │ │ ├── ...
│ │ │ └── kubernetes
│ │ └── us-west-2
│ ├── beta-ac
│ └── prod-ac
├── azure
└── gcp

27. Keep Code DRY
● Separate Code and Configuration!
● Separate Code and Configuration!
● Separate Code and Configuration!
● Code Modularize Then
Configuration can Reference
Different Module Version
module "consul" {
source = "consul/aws"
version = "0.0.5"
servers = 3
}

28. Keep Configuration DRY
● The Same Account,
Environment...etc,
Share the Same
Configuration
aws
├── alpha-ac
│ ├── alpha-ac.conf
│ ├── us-east-1
│ │ ├── database
│ │ ├── elastic
│ │ ├── ...
│ │ ├── kubernetes
│ │ └── us-east-1.conf
│ └── us-west-2
├── aws.conf
├── beta-ac
└── prod-ac

29. Keep CLI Flags DRY
● Flag Like -var 'foo=bar' or -var-file=foo can be
Stored in the Shared Configuration

30. Post-IaC Era Permission Unit Testing Integration Testing GitOps

31. Everything As Code
Could You Write Code Without Tests?

32. What is Terratest?
● A Go Library That Makes it Easier to Write
Automated Tests for Your Infrastructure
Code
● It Provides a Variety of Helper Functions
and Patterns for Common Infrastructure
Testing Tasks

33. How to Test IaC By Terratest
Setup
- Compose Configuration
- Create Resource
- Wait Resource Ready
Verification
- Leverage Helper Function
- Write Golang Directly
Teardown
- Destroy Resource
- Generate Report

34. Rich Helper Function
● Testing Terraform Code
● Testing Packer Templates
● Testing Docker Images
● Executing Commands on Servers Over SSH
● Working With AWS APIs
● Working With GCP APIs
● Working With Kubernetes APIs
● Testing Helm Charts
● Making HTTP Requests
● Running Shell Commands

35. IaC Testing Tools Comparison
● XXX-Spec ←→ Terratest ←→ Pure Programming
Language
● Learning Curve is Between XXX-Spec and Pure
Programming Language
● Not Only Check Server Properties, But Also The
Service Functionality
● Testing Scope Include Entire Systems

36. IaC Module Structure With Testing
● Modules: The Terraform
to Create Cloud Resource
● Examples: Illustrate how
to Use the Module
● Test: Test the Module by
Executing Examples
tf-aws-iam
├── examples
│ ├── iam-roles
│ └── iam-users
├── modules
│ ├── roles
│ └── users
└── test
├── iam_roles_test.go
└── iam_users_test.go

37. IaC Module Unit Test CI/CD Flow
Developer
IaC Tool
Terratest
Git Service
CI/CD Framework
Cloud
#1
#2
#3
#4
#5
#6
Unit Test

38. Post-IaC Era Permission Unit Testing Integration Testing GitOps

39. What Need To Take Care In Production?
● Operational Excellence: Prevent Service
Outages or Degradation
● Security Policy: Adopted Internally Protect
Data Privacy and Infrastructure Integrity
● Compliance Policy: Ensure Compliance with
External Standards (PCI-DSS, SOC, or GDPR)

40. How to Verify Them?
Auditing

41. Excel Engineer
● Lots of Spreadsheets
● Lots of Manual Process
● Takes Weeks to Months to
Complete Review and Fix
● Policy Document not Ready
yet
● But The Most Terrible ...

42. Shift Left Testing
Testing is Performed Earlier in the Life Cycle

43. Development Life Cycle
Local
Development
Continuous
Integration
Production
Environment
Fast Slow Slower

44. Development Life Cycle
Auditing
Local
Development
Continuous
Integration
Production
Environment

45. Development Life Cycle
Automation Auditing
Local
Development
Continuous
Integration
Production
Environment

46. How to Achieve Automation Auditing?
Audit CodePolicy
☝ Policy as Code

47. What is Open Policy Agent?
OPA
Service
Policy
(Rego)
Data
(JSON)
Request,
Event, etc
Query
Decision

48. OPA Features
● Declarative Policy Language (Rego)
● Library, Sidecar, Host-Level Daemon
● Management API for Control &
Observability
● Tooling to Build, Test and Debug Policy

49. OPA Integrations
Admission
Control
API AUthorization
SSH & sudo
Data Protection
Data Filtering
Linux PAM

50. https://github.com/smalltown/policy-as-code

51. Someone Create A Server in AWS...
0.0.0.0/0

52. Test Terraform With OPA
Policy
(Rego)
1. Terraform Output Plan
Result as Json File
2. OPA Test The JSON
Input Through Policy
Terraform
OPA

53. Benefits
● Help Individual Developers Sanity Check
Their Terraform Changes
● Auto-Approve Run-Of-The-Mill
Infrastructure Changes and Reduce the
Burden of Peer-Review
● Help Catch Problems That Arise When
Applying Terraform to Production After
Applying it to Staging

54. IaC Integration Test CI/CD Flow
Developer
IaC Tool
Terratest
Git Service
CI/CD Framework
Cloud
#1
#2
#3
#4
#5
#6
PaC Tool
#1
Unit Test
Integration
Test
#2
#3
#5
#4

55. Post-IaC Era Permission Unit Testing Integration Testing GitOps

56. A Normal Day at office
Why Alpha
is Broken?
I apply something
just now @@
(15 mins later...) You
need to merge my PR
first, or blabla...
👉 Don’t Manage Infrastructure By Your
Laptop

57. How to Achieve it?
Developer
IaC Tool
Terratest
Git Service
CI/CD Framework
Cloud
#2
#3
#4
#5
#6
PaC Tool
Unit Test
Integration
Test
#2
#3
#5
#4
#1
#1

58. What GitOps Want to Resolve Here?
● The Bridge Between IaC, Developer and
Cloud Resource
● Avoid Execute IaC From Developer’s Laptop
● Avoid Multiple Developers Modify the Same
IaC
● E.g. Atlantis, Terraform Cloud

59. Take Atlantis & Terraform for Example
● Display Detail Changes in Git PR Page
● Only When the PR is Reviewed/Merged,
The Changes Can be Applied
● No One Can Modify The Same IaC in The
Same Time (Permission)
● Reference Article

60. Add GitOps Feature
Developer
IaC Tool
Terratest
Git Service
CI/CD Framework
Cloud
#1
#2
#3
#4
#5
#6
PaC Tool
#1
Unit Test
Integration
Test
#2
#3
#5
#4
GitOps
GitOps
#1
#2
#3
#4

61. Key Takeaways
Permission Control
Unit Test
(Terratest)
Integration Test (PaC)
Collaboration (GitOps)

62. THANKS!
Any questions?
You can find me at:
● facebook.com/smalltown0110
● smalltown@awsug.tw

63. We’re Hiring!!
Software Engineer in Test
Software Engineer