Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way

893 views

Published on

How to let IaC support Unit Testing by Terratest, Integration Testing by Open Policy Agent, and GitOps by Atlantis

Published in: Software
  • Be the first to comment

DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way

  1. 1. Mastering IaC the DevOps Way
  2. 2. 2006-2010 Hello! I am smalltown MaiCoin Site Reliability Engineer Taipei HashiCorp UG Organizer AWS UG Taiwan Staff
  3. 3. Taipei HashiCorp User Group ● Vault Workshop:幫 Credentials 找個 窩 (2019/06) ● DevOpsDays Taipei: AWS Kubernetes Service Mesh Workshop (Tomorrow 13:00~17:00) ● Hack 'n' Roll: Kubernetes From Zero To Hero (2019/11/09)
  4. 4. About 2017...
  5. 5. But Maybe Not So Far...
  6. 6. What Problem We Facing ● Permission Control ● Infrastructure Quality ● Collaboration ● Efficiency & Productivity
  7. 7. Post-IaC Era Permission Unit Testing Integration Testing GitOps
  8. 8. Post-IaC Era Permission Unit Testing Integration Testing GitOps
  9. 9. Infrastructure as Code ● The Process of Managing and Provisioning Computer Data Centers Through Machine-Readable Definition Files
  10. 10. Iac First Generation ~$ apt-get update ~$ apt-get install -y tar=1.16.1 package 'tar' do version '1.16.1' action :install end 👉 Record Your Provision Procedure with CM Tool, Not Document !
  11. 11. R.I.P. Configuration Management
  12. 12. Why? Stateful Service Stateless Service Cloud Provider Orchestrator
  13. 13. IaC Second Generation 👉 Record Your Cloud Resource with IaC Tool, Not Document ! resource "aws_s3_bucket" "b" { bucket = "my_tf_test_bucket" acl = "private" tags { Name = "My_bucket" } }
  14. 14. IaC Second Generation is Hot Now! … etc AWS CDKPulumiTerraform
  15. 15. Post-IaC Era Permission Unit Testing Integration Testing GitOps
  16. 16. Do you know the Prod. environment broken ? 😠 I think I execute terraform destroy in Alpha...maybe 😱
  17. 17. We need the multiple accounts or projects architecture with cloud provider 🍯 I feel it’s complicated, why we need that 🤔
  18. 18. Who Can Cause a Service Broken? Developer! Who Can Ruin Entire Infrastructure? SRE!
  19. 19. Multiple Accounts/Projects Architecture?! ● There are Multiple Environments for One General Application ● Due to Application Need to be Tested, but not Impact the Real Users ● Production Environments Don’t Allow Access at Will ● Infrastructure Becomes Code Now, Hence, It Need to be Treated as The Same Way
  20. 20. Multiple Accounts/Projects Architecture Beta RobotRead Write Prod RobotRead Write Alpha RobotRead Write
  21. 21. Multiple Accounts/Projects Architecture ● The Write Role is For Human Usage if Necessary ● Production Write and Robot Role Only can be Permitted Through Change Management Process ● Don’t Manage Infrastructure By Your Laptop Prod RobotRead Write
  22. 22. Don't Repeat Yourself (DRY) ● Keep Code DRY ● Keep Configuration DRY ● Keep CLI Flags DRY ● No Matter What IaC Tools Been Used, You Could Take a Look at Terragrunt
  23. 23. Keep Code DRY ● Separate Different Account or Project by Folder IaC ├── aws │ ├── alpha-ac │ │ ├── us-east-1 │ │ │ ├── database │ │ │ ├── elastic │ │ │ ├── ... │ │ │ └── kubernetes │ │ └── us-west-2 │ ├── beta-ac │ └── prod-ac ├── azure └── gcp
  24. 24. Keep Code DRY ● Separate Code and Configuration! ● Separate Code and Configuration! ● Separate Code and Configuration! ● Code Modularize Then Configuration can Reference Different Module Version module "consul" { source = "consul/aws" version = "0.0.5" servers = 3 }
  25. 25. Keep Configuration DRY ● The Same Account, Environment...etc, Share the Same Configuration aws ├── alpha-ac │ ├── alpha-ac.conf │ ├── us-east-1 │ │ ├── database │ │ ├── elastic │ │ ├── ... │ │ ├── kubernetes │ │ └── us-east-1.conf │ └── us-west-2 ├── aws.conf ├── beta-ac └── prod-ac
  26. 26. Keep CLI Flags DRY ● Flag Like -var 'foo=bar' or -var-file=foo can be Stored in the Shared Configuration
  27. 27. Post-IaC Era Permission Unit Testing Integration Testing GitOps
  28. 28. Everything As Code Could You Write Code Without Tests?
  29. 29. What is Terratest? ● A Go Library That Makes it Easier to Write Automated Tests for Your Infrastructure Code ● It Provides a Variety of Helper Functions and Patterns for Common Infrastructure Testing Tasks
  30. 30. How to Test IaC By Terratest Setup - Compose Configuration - Create Resource - Wait Resource Ready Verification - Leverage Helper Function - Write Golang Directly Teardown - Destroy Resource - Generate Report
  31. 31. Rich Helper Function ● Testing Terraform Code ● Testing Packer Templates ● Testing Docker Images ● Executing Commands on Servers Over SSH ● Working With AWS APIs ● Working With GCP APIs ● Working With Kubernetes APIs ● Testing Helm Charts ● Making HTTP Requests ● Running Shell Commands
  32. 32. IaC Testing Tools Comparison ● XXX-Spec ←→ Terratest ←→ Pure Programming Language ● Learning Curve is Between XXX-Spec and Pure Programming Language ● Not Only Check Server Properties, But Also The Service Functionality ● Testing Scope Include Entire Systems
  33. 33. IaC Module Structure With Testing ● Modules: The Terraform to Create Cloud Resource ● Examples: Illustrate how to Use the Module ● Test: Test the Module by Executing Examples tf-aws-iam ├── examples │ ├── iam-roles │ └── iam-users ├── modules │ ├── roles │ └── users └── test ├── iam_roles_test.go └── iam_users_test.go
  34. 34. IaC Module Unit Test CI/CD Flow Developer IaC Tool Terratest Git Service CI/CD Framework Cloud #1 #2 #3 #4 #5 #6 Unit Test
  35. 35. Post-IaC Era Permission Unit Testing Integration Testing GitOps
  36. 36. What Need To Take Care In Production? ● Operational Excellence: Prevent Service Outages or Degradation ● Security Policy: Adopted Internally Protect Data Privacy and Infrastructure Integrity ● Compliance Policy: Ensure Compliance with External Standards (PCI-DSS, SOC, or GDPR)
  37. 37. How to Verify Them? Auditing
  38. 38. Excel Engineer ● Lots of Spreadsheets ● Lots of Manual Process ● Takes Weeks to Months to Complete Review and Fix ● Policy Document not Ready yet ● But The Most Terrible ...
  39. 39. Shift Left Testing Testing is Performed Earlier in the Life Cycle
  40. 40. Development Life Cycle Local Development Continuous Integration Production Environment Fast Slow Slower
  41. 41. Development Life Cycle Auditing Local Development Continuous Integration Production Environment
  42. 42. Development Life Cycle Automation Auditing Local Development Continuous Integration Production Environment
  43. 43. How to Achieve Automation Auditing? Audit CodePolicy ☝ Policy as Code
  44. 44. What is Open Policy Agent? OPA Service Policy (Rego) Data (JSON) Request, Event, etc Query Decision
  45. 45. OPA Features ● Declarative Policy Language (Rego) ● Library, Sidecar, Host-Level Daemon ● Management API for Control & Observability ● Tooling to Build, Test and Debug Policy
  46. 46. OPA Integrations Admission Control API AUthorization SSH & sudo Data Protection Data Filtering Linux PAM
  47. 47. https://github.com/smalltown/policy-as-code
  48. 48. Someone Create A Server in AWS... 0.0.0.0/0
  49. 49. Test Terraform With OPA Policy (Rego) 1. Terraform Output Plan Result as Json File 2. OPA Test The JSON Input Through Policy Terraform OPA
  50. 50. Benefits ● Help Individual Developers Sanity Check Their Terraform Changes ● Auto-Approve Run-Of-The-Mill Infrastructure Changes and Reduce the Burden of Peer-Review ● Help Catch Problems That Arise When Applying Terraform to Production After Applying it to Staging
  51. 51. IaC Integration Test CI/CD Flow Developer IaC Tool Terratest Git Service CI/CD Framework Cloud #1 #2 #3 #4 #5 #6 PaC Tool #1 Unit Test Integration Test #2 #3 #5 #4
  52. 52. Post-IaC Era Permission Unit Testing Integration Testing GitOps
  53. 53. A Normal Day at office Why Alpha is Broken? I apply something just now @@ (15 mins later...) You need to merge my PR first, or blabla... 👉 Don’t Manage Infrastructure By Your Laptop
  54. 54. How to Achieve it? Developer IaC Tool Terratest Git Service CI/CD Framework Cloud #2 #3 #4 #5 #6 PaC Tool Unit Test Integration Test #2 #3 #5 #4 #1 #1
  55. 55. What GitOps Want to Resolve Here? ● The Bridge Between IaC, Developer and Cloud Resource ● Avoid Execute IaC From Developer’s Laptop ● Avoid Multiple Developers Modify the Same IaC ● E.g. Atlantis, Terraform Cloud
  56. 56. Take Atlantis & Terraform for Example ● Display Detail Changes in Git PR Page ● Only When the PR is Reviewed/Merged, The Changes Can be Applied ● No One Can Modify The Same IaC in The Same Time (Permission) ● Reference Article
  57. 57. Add GitOps Feature Developer IaC Tool Terratest Git Service CI/CD Framework Cloud #1 #2 #3 #4 #5 #6 PaC Tool #1 Unit Test Integration Test #2 #3 #5 #4 GitOps GitOps #1 #2 #3 #4
  58. 58. Key Takeaways Permission Control Unit Test (Terratest) Integration Test (PaC) Collaboration (GitOps)
  59. 59. THANKS! Any questions? You can find me at: ● facebook.com/smalltown0110 ● smalltown@awsug.tw
  60. 60. We’re Hiring!! Software Engineer in Test Software Engineer

×