Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Terraforming

798 views

Published on

My team heavily uses both Serverless and Terraform to build infrastructure, but as the number of projects we managed grew, it got harder to know how components were glued in different projects.
So we decided to try different ways to structure our projects in order
to :
– make better use terraform modules
– run terraform in a more secure manner
– share serverless components in a reliable way
– abstract complex infrastructure gluing
– faster project bootstrap times

Published in: Software
  • Be the first to comment

  • Be the first to like this

Terraforming

  1. 1. Terraforming David Przybilla dav009 dav009
  2. 2. ! コロンビア
  3. 3. first time at JAWS
  4. 4. Tokyo
  5. 5. until recently
 Data Engineering / NLP
  6. 6. Slowly get into
 Infrastructure
  7. 7. ..Ops + Golang + Python..
  8. 8. first week..started looking at projects..
  9. 9. first week..started looking at projects.. Github repos…
  10. 10. first week..started looking at projects.. Github repos… New project I had to work on…
  11. 11. code
  12. 12. code terraform apply
  13. 13. code aws infrastructureterraform apply
  14. 14. lots of small services
  15. 15. lots of infrastructure
  16. 16. ..lambda functions.. ..kinesis.. ..sqs.. ..ecs..
  17. 17. early adopters of terraform
  18. 18. many terraform versions
  19. 19. many different project structures
  20. 20. bootstrapping a project is also tough
  21. 21. where do I run terraform?
  22. 22. what role should run terraform? where do I run terraform?
  23. 23. what role should run terraform? where do I run terraform? ..credentials..
  24. 24. bootstrapping my project took longer than I wanted
  25. 25. ..complex project structure..
  26. 26. sample_old_project/
  27. 27. sample_old_project/ tf/
  28. 28. sample_old_project/ tf/ lambda_functions.tf
  29. 29. sample_old_project/ tf/ lambda_functions.tf iam.tf
  30. 30. sample_old_project/ tf/ lambda_functions.tf iam.tf apigateways.tf
  31. 31. sample_old_project/ tf/ lambda_functions.tf iam.tf apigateways.tf policies.tf ecs….
  32. 32. sample_old_project/ tf/ lambda_functions.tf iam.tf apigateways.tf policies.tf ecs….
  33. 33. api gateways
  34. 34. ..when it gets bigger tho..
  35. 35. many api gateways many lambdas many iam roles.. many everything..
  36. 36. many api gateways many lambdas many iam roles.. many everything.. even they are not exactly related to each other
  37. 37. I had to keep all pieces in my head at the same time
  38. 38. ..experiment a bit..
  39. 39. my_new_project/ tf
 /modules /stg /prd
  40. 40. my_new_project/ tf
 /modules /stg /prd
  41. 41. my_new_project/ tf
 /modules /stg /prd ..shared infrastructure.. between prd and stg
  42. 42. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion
  43. 43. my_new_project/ tf
 /modules /stg main.tf .tfversion /prd main.tf .tfversion …just import modules..
  44. 44. module: should be like a function
  45. 45. function/module: purpose: does only one thing
  46. 46. module: parameters outputs inside is invisible to importer
  47. 47. module: abstracts complexity
  48. 48. module: abstracts complexity don’t need to know how things are connected
  49. 49. module: abstracts complexity don’t need to know how things are connected don't need to see all pieces
  50. 50. database_synchornizer: main.tf vars.tf output.tf readme.md lambda.tf iam.tf … module connect infrastructure together input parameters output parameters
  51. 51. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module all related
  52. 52. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module all related less pieces
  53. 53. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module
  54. 54. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module api: main.tf vars.tf output.tf …. apigateways …
  55. 55. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module api: main.tf vars.tf output.tf …. apigateways … ecs-cluster: main.tf vars.tf output.tf ….
  56. 56. less to remember
  57. 57. less to remember other modules connect via output variables
  58. 58. we realise we could reuse modules
  59. 59. so we made our own GitHub repo only for modules, similar to: we realise we could share modules
  60. 60. github.com/terraform-community-modules github.com/segmentio/stack we realise we could reuse modules so we made our own GitHub repo only for modules, similar to:
  61. 61. ..looking at other projects..
  62. 62. ..looking at other projects.. ..how to run terraform..
  63. 63. what terraform version is this using?
  64. 64. what role should I assume to run terraform for this project?
  65. 65. ..Don’t forget to change environment..
  66. 66. project A: do this but don’t do that
  67. 67. project B: do this other thing but don’t do that other one
  68. 68. ..running terraform is scary…
  69. 69. ..take down service..
  70. 70. starting a new project?
  71. 71. starting a new project? difficult to bootstrap
  72. 72. bootstrapping should:
  73. 73. bootstrapping should: be fast
  74. 74. bootstrapping should: take little effort be fast
  75. 75. bootstrapping should: take little effort be fast be easy for a new team member
  76. 76. bootstrapping should: take little effort be fast be easy for a new team member complexity should be hidden
  77. 77. what to do?
  78. 78. ..a module.. .. a terraform automated pipeline that takes 10 min to set up..
  79. 79. ..a trigger..
  80. 80. codepipeline codecommit codebuild
  81. 81. how do we use it?
  82. 82. 1. import module: terraform_pipeline 2. follow project structure
  83. 83. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
  84. 84. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
  85. 85. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
  86. 86. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } no need to remember this role anymore! importing this module
  87. 87. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
  88. 88. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
  89. 89. .. details are hidden for importer ..
  90. 90. .. details are hidden for importer .. importer does not need to worry about :
  91. 91. importer does not need to worry about : how to send notifications to slack .. details are hidden for importer ..
  92. 92. .. details are hidden for importer .. importer does not need to worry about : how to run terraform how to send notifications to slack
  93. 93. .. details are hidden for importer .. importer does not need to worry about : how to run terraform module knows that (inside) how to send notifications to slack
  94. 94. .. details are hidden for importer .. importer does not need to worry about : how to run terraform module knows that (inside) how to send notifications to slack a new team member can import it quickly
  95. 95. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion
  96. 96. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion terraform ci module : 1. downloads terraform version specified in .tfversion 2. knows how to run a terraform binary 3. knows how to get git credentials to fetch to our private modules 4. knows how to send notifications to slack
  97. 97. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion want to run terraform for stg? go to stg folder
  98. 98. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion want to run terraform for stg? go to stg folder want to run terraform for prd? go to prd folder
  99. 99. reduced the complexity of bootstrapping a project
  100. 100. reduced the complexity of bootstrapping a project means a faster team
  101. 101. reduced the complexity of bootstrapping a project means a faster team also I am less scared to run terraform
  102. 102. reduced the complexity of bootstrapping a project means a faster team also I am less scared to run terraform all projects use the same pipeline
  103. 103. whats the next step to hide complexity?
  104. 104. whats the next step to hide complexity?
  105. 105. github.com/serverless/serverless
  106. 106. serverless serverless “applications”
  107. 107. serverless serverless “applications” code + glue + infrastructure i.e: serverless service to get a slack bot via FaaS
  108. 108. serverless serverless “applications” > serverless install --url <service-github-url> > sls deploy code + glue + infrastructure i.e: serverless service to get a slack bot via FaaS
  109. 109. please be aware a few things have changed:
  110. 110. - Terraform community modules - Serverless application repository (aws)
  111. 111. how are you running terraform?
  112. 112. how are you structuring your terraform projects ? how are you running terraform?
  113. 113. how are you structuring your terraform projects ? how are you running terraform? my team is using terraform + github.com/serverless/serverless
  114. 114. how are you structuring your terraform projects ? how are you running terraform? my team is using terraform + github.com/serverless/serverless feel free to talk to me! I would like to learn more
  115. 115. Thanks 🙇 dav009 dav009 Q & A bit.ly/jd2018-sls

×