Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Terraform AWS modules and some best-practices - May 2019

444 views

Published on

Slides from my meetup talks during meetups in Germany. Follow me:
https://twitter.com/antonbabenko
https://github.com/antonbabenko

Published in: Technology
  • Earn $90/day Working Online. You won't get rich, but it is going to make you some money! ➤➤ http://ishbv.com/ezpayjobs/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Terraform AWS modules and some best-practices - May 2019

  1. 1. Terraform AWS modules and some best- practices Anton Babenko @antonbabenko May 2019
  2. 2. Anton Babenko Terraform AWS fanatic since 2015 Organiser of HashiCorp UG, AWS UG, DevOps Norway, DevOpsDays Oslo I 💚 open-source: terraform-community-modules + terraform-aws-modules antonbabenko/pre-commit-terraform — clean code and documentation antonbabenko/tfvars-annotations — update terraform.tfvars using annotations antonbabenko/modules.tf-lambda — generate Terraform code from visual diagrams antonbabenko/terragrunt-reference-architecture — Terragrunt reference architecture www.terraform-best-practices.com medium.com/@anton.babenko @antonbabenko — Twitter, GitHub, Linkedin
  3. 3. What do I do? All-things Terraform + AWS + DevOps Consulting Workshops Trainings Mentorship My email: anton@antonbabenko.com LinkedIn: https://www.linkedin.com/in/antonbabenko
  4. 4. Collection of open-source Terraform AWS modules supported by the community. More than 2 mil. downloads since September 2017. (VPC, Autoscaling, RDS, Security Groups, ELB, ALB, Redshift, SNS, SQS, IAM, EKS, ECS…) github.com/terraform-aws-modules registry.terraform.io/modules/terraform-aws-modules @antonbabenko
  5. 5. Cloudcraft.co — the best way to draw AWS diagrams @antonbabenko
  6. 6. cloudcraft.co features • Manage components in browser (EC2 instances, autoscaling groups, RDS, etc) • Connect components • Import live AWS infrastructure • Calculate the budget • Share link to a blueprint • Export as image • Embed drawing to wiki, Confluence, etc @antonbabenko
  7. 7. Infrastructure as code makes DevOps possible Key benefits: • Treat infrastructure like application code • Always know what changed • Validate infrastructure before deployment https://dzone.com/articles/infrastructure-as-code-the-benefits @antonbabenko
  8. 8. Tool for building, changing and versioning infrastructure safely and efficiently. www.terraform.io @antonbabenko
  9. 9. @antonbabenko
  10. 10. @antonbabenko
  11. 11. @antonbabenko
  12. 12. @antonbabenko
  13. 13. @antonbabenko
  14. 14. Google Cloud Deployment Manager Azure Resource Manager @antonbabenko
  15. 15. @antonbabenko
  16. 16. +morethan100providers @antonbabenko
  17. 17. Why Terraform and not AWS CloudFormation, Azure ARM, Google Cloud Deployment Manager? Terraform manages 100+ providers, has easier syntax (HCL), has native support for modules and remote states, has teamwork related features, is an open-source project. Provides a high-level abstraction of infrastructure (IaC) Allows for composition and combination Orchestration, not merely configuration Supports parallel management of resources (graph, fast) Separates planning from execution (dry-run) @antonbabenko
  18. 18. Terraform — universal tool for everything with an API Google G Suite Dropbox files and access New Relic metrics Datadog users and metrics Jira issues Minecraft, or even order Domino’s pizza All Terraform providers — https://www.terraform.io/docs/providers/index.html @antonbabenko
  19. 19. Let’s begin — everything fits in main.tf @antonbabenko
  20. 20. @antonbabenko
  21. 21. Project grows — main.tf: 20+ resources and data sources @antonbabenko
  22. 22. Why? @antonbabenko
  23. 23. Resources, regions, providers, … @antonbabenko
  24. 24. @antonbabenko
  25. 25. @antonbabenko
  26. 26. @antonbabenko
  27. 27. @antonbabenko
  28. 28. @antonbabenko
  29. 29. main.tf: 10-20 Kb 300+ LOC @antonbabenko
  30. 30. Emerging issues Code size is increasing Dependencies between resources become complicated @antonbabenko
  31. 31. Solution — Terraform modules @antonbabenko
  32. 32. Modules in Terraform are self-contained packages of Terraform configurations that are managed as a group. @antonbabenko
  33. 33. Resource modules Create resources in a very flexible configuration Open-source @antonbabenko
  34. 34. Resource modules @antonbabenko
  35. 35. Resource modules @antonbabenko
  36. 36. Resource modules @antonbabenko
  37. 37. Resource modules @antonbabenko
  38. 38. Would you use Terraform module to manage AWS EC2 security group? @antonbabenko
  39. 39. @antonbabenko
  40. 40. Would you use Terraform module to manage AWS EC2 security group? Yes :) @antonbabenko
  41. 41. Infrastructure modules Consist of resource modules Enforce tags and company standards Use preprocessors, jsonnet, cookiecutter @antonbabenko
  42. 42. Infrastructure modules @antonbabenko
  43. 43. Infrastructure modules @antonbabenko
  44. 44. Infrastructure modules @antonbabenko
  45. 45. @antonbabenko
  46. 46. Types of Terraform modules Resource modules (github.com/terraform-aws-modules , for eg) Infrastructure modules @antonbabenko
  47. 47. - [ ] How to write modules? - [ ] How to call modules? - [ ] How to work with the code? @antonbabenko
  48. 48. Tip №0 Check Terraform Registry before writing resource modules @antonbabenko
  49. 49. Hide implementation details @antonbabenko
  50. 50. @antonbabenko
  51. 51. @antonbabenko
  52. 52. Size @antonbabenko
  53. 53. Size https://github.com/mbtproject/mbt @antonbabenko
  54. 54. Things to avoid in modules @antonbabenko
  55. 55. Exception: logical providers (template, random, local, http, external) Providers in modules — evil @antonbabenko
  56. 56. @antonbabenko
  57. 57. Provisioner — evil Avoid provisioner in all resources @antonbabenko
  58. 58. Provisioner — evil Avoid provisioner in all resources @antonbabenko
  59. 59. Provisioner — evil Avoid provisioner even in EC2 resources @antonbabenko
  60. 60. Provisioner — evil Avoid provisioner even in EC2 resources @antonbabenko
  61. 61. @antonbabenko
  62. 62. @antonbabenko
  63. 63. null_resource provisioner — good @antonbabenko
  64. 64. Traits of good Terraform modules Documentation and examples Feature rich Sane defaults Clean code Tests Read more: http://bit.ly/common-traits-in-terraform-modules @antonbabenko
  65. 65. - [x] How to write modules? - [x] Do not write, if you can - [x] Avoid providers in modules, provisioners - [ ] How to call modules? - [ ] How to work with the code? @antonbabenko
  66. 66. How to structure Terraform configurations? How to call them? @antonbabenko
  67. 67. Call Terraform modules Use Terraform modules, because amount of resources and code is increasing How to organize Terraform configurations and invoke them? How to orchestrate modules? @antonbabenko
  68. 68. All-in-one Good: Declare variables and outputs in fewer places Bad: Large blast radius Everything is blocked at once Impossible to specify dependencies between modules (depends_on) @antonbabenko
  69. 69. 1-in-1 Good: Smaller blast radius Possible to join invocation Easier and faster to work with Bad: Declare variables and outputs in more places @antonbabenko
  70. 70. Which way do you group your code? All-in-one or 1-in-1? @antonbabenko
  71. 71. All-in-one 1-in-1 or @antonbabenko
  72. 72. Correct MFA (Most Frequent Answer): Somewhere in between @antonbabenko
  73. 73. All-in-one Undefined project scope Fast prototyping and initial development phase Small number of resources & developers Tightly connected resources 1-in-1 @antonbabenko Defined project scope Different types of developers involved * Code reuse is encouraged (across organization and environments) Use Terragrunt
  74. 74. What about Terraform workspaces? @antonbabenko
  75. 75. Problems with Terraform workspaces Terraform Workspaces aren’t infrastructure-as-code friendly. You can’t answer straight from the code: "How many workspaces do you have?" "What infrastructure has been deployed in workspaceX?" "What is the difference between workspaceX and workspaceY?" Introducing complexity almost in all cases. @antonbabenko
  76. 76. Solution — use re-usable modules instead of workspaces @antonbabenko
  77. 77. What kind of orchestration method do you use? -target Makefile … @antonbabenko
  78. 78. Orchestration in Terraform @antonbabenko
  79. 79. No really, do not try this at home! @antonbabenko
  80. 80. Orchestration = Terragrunt https://github.com/gruntwork-io/terragrunt/ @antonbabenko
  81. 81. Orchestration = Terragrunt @antonbabenko
  82. 82. Orchestration = Terragrunt @antonbabenko
  83. 83. Orchestration = Terragrunt @antonbabenko
  84. 84. Orchestration = Terragrunt @antonbabenko
  85. 85. tfvars can’t contain dynamic values :( Orchestration = Terragrunt @antonbabenko subnet_id = "???"
  86. 86. Orchestration = Terragrunt tfvars can’t contain dynamic values, so I have fixed it :) @antonbabenko subnet_id = "???"
  87. 87. before_hook + shell script Go binary https://github.com/antonbabenko/tfvars-annotations — Update values in terraform.tfvars using annotations (WIP) or take a look at modules.tf @antonbabenko
  88. 88. Orchestration = Terragrunt @antonbabenko subnet_id = "" # tfvars:terragrunt_output.vpc.public_subnets
  89. 89. Orchestration = Terragrunt @antonbabenko subnet_id = ["subnet-1a2b3c4d"] # tfvars:terragrunt_output.vpc.public_subnets
  90. 90. - [x] How to write modules? - [x] How to call modules? - [x] 1-in-1 works beter over time - [x] Orchestration = Terragrunt - [x] Dynamic values in tfvars - [ ] How to work with the code? @antonbabenko
  91. 91. Work with stateless lists @antonbabenko
  92. 92. @antonbabenko Work with stateless lists
  93. 93. @antonbabenko Work with stateless lists
  94. 94. https://jsonnet.org/ Work with stateful lists @antonbabenko
  95. 95. Work with stateful lists @antonbabenko
  96. 96. Work with stateful lists @antonbabenko
  97. 97. Work with stateful lists @antonbabenko
  98. 98. Work with stateful lists @antonbabenko
  99. 99. Integration @antonbabenko
  100. 100. Integration @antonbabenko
  101. 101. Auto-integration @antonbabenko
  102. 102. Edge cases Different AWS regions (version of S3 signature, EC2 ClassicLink, IPv6) Date of creation of AWS account Limits on resources in AWS Services and features availability @antonbabenko
  103. 103. Avoid in Terraform Not secret arguments should not be specified as command line arguments => put them in tfvars Reduce usage of "-target" and "-parallelism" "Terraform workspaces" evil in=> separate by directories Dependency hell in modules @antonbabenko
  104. 104. - [x] How to write modules? - [x] How to call modules? - [x] How to work with the code? - [x] Lists in Terraform 0.11 can be painful - [x] Perceive Terraform easier @antonbabenko
  105. 105. Summary Write less and simpler (Terraform 0.12 won’t fix your code for you!) Use existing modules and utilities @antonbabenko
  106. 106. How to handle secrets in Terraform? • Can you accept secrets to be saved in state file in plaintext? Probably not. • AWS IAM password & access secret keys — use PGP as keybase.io • AWS RDS — set dummy password and change after DB is created • AWS RDS — use iam_database_authentication_enabled = true • EC2 instance user-data + AWS KMS • EC2 instance user-data + AWS System Manager’s Parameter Store • AWS Secrets Manager • https://github.com/opencredo/terrahelp • Other options: • Secure remote state location (S3 bucket policy, KMS key) @antonbabenko
  107. 107. What are the tools/solutions out there? • Terraform Registry (https://registry.terraform.io/) — collection of public Terraform modules for common infrastructure configurations for any provider. • Terraform linter to detect errors that can not be detected by `terraform plan` — https://github.com/wata727/tflint • Terraform version manager — https://github.com/kamatama41/tfenv • A web dashboard to inspect Terraform States — https://github.com/camptocamp/ terraboard • Jsonnet — The data templating language — http://jsonnet.org @antonbabenko
  108. 108. Atlantis — Start working on Terraform as a team A unified workflow for collaborating on Terraform through GitHub, GitLab and Bitbucket https://www.runatlantis.io @antonbabenko
  109. 109. Bonus
  110. 110. ✓ cloudcraft.co — design, plan and visualize ✓ terraform-aws-modules — building blocks of AWS infrastructure ✓ Terraform — infrastructure as code
  111. 111. Infrastructure as code generator — from visual diagrams to Terraform https://github.com/antonbabenko/modules.tf-lambda Demo video: https://www.youtube.com/watch?v=F1Ax1zfZbiY
  112. 112. 1. Go to cloudcraft.co 2. Sign up, sign in (free account) 3. Draw your AWS infrastructure 4. Click "Export" 5. Click "Terraform code export" Try it yourself!
  113. 113. modules.tf — generated code ✓ Potentially ready-to-use Terraform configurations ✓ Suits best for bootstrapping ✓ Enforces Terraform best-practices ✓ Batteries included (terraform-aws-modules, terragrunt, tfvars- annotations, pre-commit) ✓ 100% free and open-source (https://github.com/antonbabenko/ modules.tf-lambda) ✓ Released under MIT license
  114. 114. Thanks! Questions? github.com/antonbabenko twitter.com/antonbabenko

×