Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Terraform modules and some of best-practices - March 2019

794 views

Published on

Slides from my talk at DevOps Singapore meetup in March 2019

Published in: Technology
  • Real Money Streams ~ Create multiple streams of wealth from your home! ★★★ http://scamcb.com/ezpayjobs/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Terraform modules and some of best-practices - March 2019

  1. 1. Terraform modules and some of best- practices Anton Babenko @antonbabenko March 2019
  2. 2. Anton Babenko Terraform AWS fanatic since 2015 Organiser of HashiCorp UG, AWS UG, DevOps Norway, DevOpsDays Oslo I 💚 open-source: terraform-community-modules + terraform-aws-modules antonbabenko/pre-commit-terraform — clean code and documentation antonbabenko/terraform-docs-as-pdf antonbabenko/modules.tf-lambda — generate Terraform code from visual diagrams www.terraform-best-practices.com medium.com/@anton.babenko @antonbabenko — Twitter, GitHub, Linkedin
  3. 3. Collection of open-source Terraform AWS modules supported by the community. More than 2 mil. downloads since September 2017. (VPC, Autoscaling, RDS, Security Groups, ELB, ALB, Redshift, SNS, SQS, IAM, EKS, ECS…) github.com/terraform-aws-modules registry.terraform.io/modules/terraform-aws-modules
  4. 4. Write, plan and manage infrastructure as code www.terraform.io
  5. 5. Google Cloud Deployment Manager Azure Resource Manager
  6. 6. +morethan100providers
  7. 7. Why Terraform and not AWS CloudFormation, Azure ARM, Google Cloud Deployment Manager? • Terraform manages 100+ providers, has easier syntax (HCL), has native support for modules and remote states, has teamwork related features, is an open-source project. • Provides a high-level abstraction of infrastructure (IaC) • Allows for composition and combination • Supports parallel management of resources (graph, fast) • Separates planning from execution (dry-run)
  8. 8. Terraform — universal tool for everything with an API GSuite Dropbox files and access New Relic metrics Datadog users and metrics Jira issues All Terraform providers
  9. 9. Let’s start!
  10. 10. Terraform modules
  11. 11. Modules in Terraform are self-contained packages of Terraform configurations that are managed as a group.
  12. 12. Resource modules Create resources in a very flexible configuration Open-source
  13. 13. Resource modules
  14. 14. Resource modules
  15. 15. Resource modules
  16. 16. Resource modules
  17. 17. Q: Why use resource modules instead of resources? A: Resources can’t be versioned, but modules can. Resource modules
  18. 18. Infrastructure modules Consist of resource modules Enforce tags and company standards Use preprocessors, jsonnet, cookiecutter
  19. 19. Infrastructure modules
  20. 20. Infrastructure modules
  21. 21. Infrastructure modules
  22. 22. Types of Terraform modules Resource modules (github.com/terraform-aws-modules , for eg) Infrastructure modules
  23. 23. - [ ] How to write modules? - [ ] How to call modules? - [ ] How to work with the code?
  24. 24. Tip №0 Check Terraform Registry before writing resource modules
  25. 25. Hide implementation details
  26. 26. Size
  27. 27. Size https://github.com/mbtproject/mbt
  28. 28. Things to avoid in modules
  29. 29. Exception: logical providers (template, random, local, http, external) Providers in modules — evil
  30. 30. Provisioner — evil Avoid provisioner in all resources
  31. 31. Provisioner — evil Avoid provisioner in all resources
  32. 32. Provisioner — evil Avoid provisioner even in EC2 resources
  33. 33. Provisioner — evil Avoid provisioner even in EC2 resources
  34. 34. null_resource provisioner — good
  35. 35. Traits of good Terraform modules Documentation and examples Feature rich Sane defaults Clean code Tests Read more: http://bit.ly/common-traits-in-terraform-modules
  36. 36. - [x] How to write modules? - [x] Do not write, if you can - [x] Avoid providers in modules, provisioners - [ ] How to call modules? - [ ] How to work with the code?
  37. 37. Call Terraform modules Amount of resources and code keeps growing How to organize and call? How to orchestrate calls?
  38. 38. All-in-one Good: Declare variables and outputs in fewer places Bad: Large blast radius Everything is blocked at once Impossible to specify dependencies between modules (depends_on)
  39. 39. 1-in-1 Good: Smaller blast radius Possible to join invocation Easier and faster to work with Bad: Declare variables and outputs in more places
  40. 40. Which way do you group your code? All-in-one or 1-in-1?
  41. 41. Correct MFA (Most Frequent Answer): Somewhere in between
  42. 42. What kind of orchestration tool do you use? -target Makefile …
  43. 43. Orchestration in Terraform
  44. 44. No really, do not try this at home!
  45. 45. Orchestration = Terragrunt https://github.com/gruntwork-io/terragrunt/
  46. 46. Orchestration = Terragrunt
  47. 47. Orchestration = Terragrunt
  48. 48. Orchestration = Terragrunt
  49. 49. Orchestration = Terragrunt
  50. 50. tfvars can’t contain dynamic values :( Orchestration = Terragrunt
  51. 51. Orchestration = Terragrunt tfvars can’t contain dynamic values, so I have fixed it :)
  52. 52. before_hook + shell script https://github.com/antonbabenko/modules.tf-lambda/blob/master/templates/ terragrunt-common-layer/template/common/scripts/ update_dynamic_values_in_tfvars.sh or use modules.tf
  53. 53. - [x] How to write modules? - [x] How to call modules? - [x] 1-in-1 works beter over time - [x] Orchestration = Terragrunt - [x] Dynamic values in tfvars - [ ] How to work with the code?
  54. 54. Add new features Usually it is easy…
  55. 55. Create new or use existing
  56. 56. Create new or use existing
  57. 57. Create new or use existing
  58. 58. Create new or use existing
  59. 59. Work with lists
  60. 60. Work with lists
  61. 61. Work with lists
  62. 62. https://jsonnet.org/ Work with stateful lists
  63. 63. Work with stateful lists
  64. 64. Work with stateful lists
  65. 65. Work with stateful lists
  66. 66. Work with stateful lists
  67. 67. Integration
  68. 68. Integration
  69. 69. Auto-integration
  70. 70. Edge cases Different AWS regions (version of S3 signature, EC2 ClassicLink, IPv6) Date of creation of AWS account Limits on resources in AWS Services and features availability
  71. 71. Avoid in Terraform Not secret arguments should not be specified as command line arguments => put them in tfvars Reduce usage of "-target" and "-parallelism" "Terraform workspaces" evil in=> separate by directories Dependency hell in modules
  72. 72. - [x] How to write modules? - [x] How to call modules? - [x] How to work with the code? - [x] Lists in Terraform 0.11 can be painful - [x] Perceive Terraform easier
  73. 73. - [x] How to write modules? - [x] How to call modules? - [x] How to work with the code? - [x] Terraform 0.12 beta!
  74. 74. Terraform 0.12 HCL2 — simplified syntax Loops ("for") Dynamic blocks ("for_each") Correct conditional operators (… ? … : …) Extended types of variables Templates in values Links between resources are supported (depends_on everywhere) Read more — https://www.hashicorp.com/blog/announcing-terraform-0-1-2-beta
  75. 75. Summary Write less and simpler (Terraform 0.12 won’t fix your code for you!) Use existing modules and utilities
  76. 76. How to handle secrets in Terraform? • Can you accept secrets to be saved in state file in plaintext? Probably not. • AWS IAM password & access secret keys — use PGP as keybase.io • AWS RDS — set dummy password and change after DB is created • AWS RDS — use iam_database_authentication_enabled = true • EC2 instance user-data + AWS KMS • EC2 instance user-data + AWS System Manager’s Parameter Store • AWS Secrets Manager • https://github.com/opencredo/terrahelp • Other options: • Secure remote state location (S3 bucket policy, KMS key)
  77. 77. What are the tools/solutions out there? • Terraform Registry — collection of public Terraform modules for common infrastructure configurations for any provider — https://registry.terraform.io/ • Terraform linter to detect errors that can not be detected by `terraform plan` — https://github.com/wata727/tflint • Terraform version manager — https://github.com/kamatama41/tfenv • A web dashboard to inspect Terraform States — https://github.com/ camptocamp/terraboard • Jsonnet — The data templating language — http://jsonnet.org • terraform-compliance - BDD style Terraform validation/compliancy check — https://github.com/eerkunt/terraform-compliance
  78. 78. Atlantis — Start working on Terraform as a team A unified workflow for collaborating on Terraform through GitHub, GitLab and Bitbucket https://www.runatlantis.io https://github.com/terraform-aws-modules/terraform-aws-atlantis
  79. 79. Bonus
  80. 80. ✓ cloudcraft.co — design, plan and visualize ✓ terraform-aws-modules — building blocks of AWS infrastructure ✓ Terraform — infrastructure as code
  81. 81. Infrastructure as code generator — from visual diagrams to Terraform https://github.com/antonbabenko/modules.tf-lambda Demo video: https://www.youtube.com/watch?v=F1Ax1zfZbiY
  82. 82. 1. Go to cloudcraft.co 2. Sign up, sign in (free account) 3. Draw your AWS infrastructure 4. Click "Export" 5. Click "Terraform code export" Try it yourself!
  83. 83. modules.tf — generated code ✓ Potentially ready-to-use Terraform configurations ✓ Suits best for bootstrapping ✓ Enforces Terraform best-practices ✓ Batteries included (terraform-aws-modules, terragrunt, pre-commit) ✓ 100% free and open-source (https://github.com/antonbabenko/ modules.tf-lambda) ✓ Released under MIT license
  84. 84. Thanks! Questions? github.com/antonbabenko twitter.com/antonbabenko

×