Terraform for Serverless - Best Practices
•Download as PPTX, PDF•
1 like•586 views
Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Best practices include collaborative approach to infrastructure provisioning, use of version control systems and prevent manual changes, as well as efficient management of boundaries between different teams, roles, applications and deployment tiers. In this session we will walk you through our journey from early serverless "never thought our dreams would ever come true" adopters to enterprise "why are we still dealing with this crap" developers. We will focus on lessons learned and best practices that goes above and beyond official documentation.
Recommended
Recommended
More Related Content
Similar to Terraform for Serverless - Best Practices
Similar to Terraform for Serverless - Best Practices (20)
More from Mitoc Group
More from Mitoc Group (18)
Recently uploaded
Recently uploaded (20)
Terraform for Serverless - Best Practices
- 1. DevOps Pro Europe, Vilnius March 2019 Terraform for Serverless. Best Practices. Lessons Learned.
- 2. DevOps Pro Europe, Vilnius March 2019 DevOps Landscape. Daily Challenges.
- 3. DevOps Pro Europe, Vilnius March 2019 If That Was Not Enough… Introducing Serverless Landscape
- 4. DevOps Pro Europe, Vilnius March 2019 Serverless Landscape. Daily Challenges.
- 5. DevOps Pro Europe, Vilnius March 2019 Please excuse my honesty, but... this feels like TOO MUCH to keep up with
- 6. DevOps Pro Europe, Vilnius March 2019 Eugene ISTRATI @eistrati About Presenter • CTO, Tech Partner @ Mitoc Group • Ex-AWS, ex-Hearst, ex-GrubHub • Certified AWS Solutions Architect • 20 Years in IT; 10 Years in Cloud Computing; 5 Years in Enterprise IT • Focusing on: Automation, DevOps, Serverless
- 7. DevOps Pro Europe, Vilnius March 2019 Agenda Exponential increase in services and tools for cloud & serverless Solution: we are striving to make IT simpler, reusable & cloud native The Devil is in details
- 8. DevOps Pro Europe, Vilnius March 2019 We Are Striving To… Make IT Simpler
- 9. DevOps Pro Europe, Vilnius March 2019 We Are Striving To… Make IT Simpler Yeah, Good Luck With That!
- 10. DevOps Pro Europe, Vilnius March 2019
- 11. DevOps Pro Europe, Vilnius March 2019 Prerequisites: Terraform For Serverless 1. Understand IT-as-a-Service Spectrum 1. Understand DevOps Spectrum 2. Understand Scope & Boundaries
- 12. DevOps Pro Europe, Vilnius March 2019 1. Understand IT-as-a-Service Spectrum On-Prem Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center Colocation Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center Hosting Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center IaaS Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center PaaS Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center SaaS Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center Managed by Customer Managed by Provider
- 13. DevOps Pro Europe, Vilnius March 2019 What Is Serverless?
- 14. DevOps Pro Europe, Vilnius March 2019 Serverless in IT-as-a-Service Spectrum On-Prem Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center Colocation Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center Hosting Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center IaaS Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center PaaS Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center SaaS Data Application Databases Operation System Virtualization Physical Servers Network & Storage Data Center Managed by Customer Managed by Provider Serverless Architecture not in scope
- 15. DevOps Pro Europe, Vilnius March 2019 2. Understand DevOps Spectrum
- 16. DevOps Pro Europe, Vilnius March 2019 3. Understand Scope & Boundaries A B C
- 17. DevOps Pro Europe, Vilnius March 2019 Terraform For Serverless A B C B == Terraform A + B + C == Terraform For Serverless
- 18. DevOps Pro Europe, Vilnius March 2019 Terraform For Serverless Best Practices. Lessons Learned.
- 19. DevOps Pro Europe, Vilnius March 2019 Best Practice #1 (of 8) Adopt microservices architecture; aim for 1-to-1 relationship between serverless resources and terraform configurations
- 20. DevOps Pro Europe, Vilnius March 2019 Best Practice #1 (of 8) Adopt microservices architecture; aim for 1-to-1 relationship between serverless resources and terraform configurations
- 21. DevOps Pro Europe, Vilnius March 2019 Best Practice #1 (of 8) Adopt microservices architecture; aim for 1-to-1 relationship between serverless resources and terraform configurations
- 22. DevOps Pro Europe, Vilnius March 2019 Best Practice #2 (of 8) Pass variables between resources using terraform remote state
- 23. DevOps Pro Europe, Vilnius March 2019 Best Practice #2 (of 8) Pass variables between resources using terraform remote state
- 24. DevOps Pro Europe, Vilnius March 2019 Best Practice #2 (of 8) Pass variables between resources using terraform remote state
- 25. DevOps Pro Europe, Vilnius March 2019 Best Practice #3 (of 8) Avoid code build using local provisioner or external data; instead use hooks provided by terraform orchestration tools
- 26. DevOps Pro Europe, Vilnius March 2019 Best Practice #3 (of 8) Avoid code build using local provisioner or external data; instead use hooks provided by terraform orchestration tools
- 27. DevOps Pro Europe, Vilnius March 2019 Best Practice #3 (of 8) Avoid code build using local provisioner or external data; instead use hooks provided by terraform orchestration tools
- 28. DevOps Pro Europe, Vilnius March 2019 Best Practice #4 (of 8) Execute in parallel your automated terraform workflows; don’t ignore terraform configurations dependencies
- 29. DevOps Pro Europe, Vilnius March 2019 Best Practice #4 (of 8) Execute in parallel your automated terraform workflows; don’t ignore terraform configurations dependencies
- 30. DevOps Pro Europe, Vilnius March 2019 Best Practice #4 (of 8) Execute in parallel your automated terraform workflows; don’t ignore terraform configurations dependencies
- 31. DevOps Pro Europe, Vilnius March 2019 Best Practice #5 (of 8) Optimize automated terraform workflows with git diff
- 32. DevOps Pro Europe, Vilnius March 2019 Best Practice #5 (of 8) Optimize automated terraform workflows with git diff
- 33. DevOps Pro Europe, Vilnius March 2019 Best Practice #5 (of 8) Optimize automated terraform workflows with git diff
- 34. DevOps Pro Europe, Vilnius March 2019 Best Practice #6 (of 8) Logically separate environments using terraform workspace
- 35. DevOps Pro Europe, Vilnius March 2019 Best Practice #6 (of 8) Logically separate environments using terraform workspace dev == default
- 36. DevOps Pro Europe, Vilnius March 2019 Best Practice #6 (of 8) Logically separate environments using terraform workspace dev == default
- 37. DevOps Pro Europe, Vilnius March 2019 Best Practice #7 (of 8) Overwrite environment specific values using variables precedence
- 38. DevOps Pro Europe, Vilnius March 2019 Best Practice #7 (of 8) Overwrite environment specific values using variables precedence https://www.terraform.io/docs/configuration/variabl es.html#variable-precedence
- 39. DevOps Pro Europe, Vilnius March 2019 Best Practice #7 (of 8) Overwrite environment specific values using variables precedence https://www.terraform.io/docs/configuration/variabl es.html#variable-precedence
- 40. DevOps Pro Europe, Vilnius March 2019 Best Practice #8 (of 8) Get comfortable with lots of terraform code; or use terrahub cli
- 41. DevOps Pro Europe, Vilnius March 2019 Best Practice #8 (of 8) Get comfortable with lots of terraform code; or use terrahub cli $ find . -name ‘*.tf*’ | xargs wc -l | grep total 33998 total $ find . -name ‘.terrahub*.yml’ | xargs wc -l | grep total 22118 total
- 42. DevOps Pro Europe, Vilnius March 2019 Best Practice #8 (of 8) Get comfortable with lots of terraform code; or use terrahub cli $ find . -name ‘*.tf*’ | xargs wc -l | grep total 33998 total $ find . -name ‘.terrahub*.yml’ | xargs wc -l | grep total 22118 total
- 43. DevOps Pro Europe, Vilnius March 2019 Best Practices Summary 1. Adopt microservices architecture; aim for 1-to-1 relationship 1. Pass variables between resources using terraform remote state 1. Avoid code build using local provisioner or external data 1. Execute in parallel your automated terraform workflows; don’t ignore terraform configurations dependencies 1. Optimize automated terraform workflows with git diff 1. Logically separate environments using terraform workspace 1. Overwrite environment specific values using variables precedence 1. Get comfortable with lots of terraform code; or use terrahub cli
- 44. DevOps Pro Europe, Vilnius March 2019 Lessons Learned Summary 1. Adopt microservices architecture; aim for 1-to-1 relationship 1. Pass variables between resources using terraform remote state 1. Avoid code build using local provisioner or external data 1. Execute in parallel your automated terraform workflows; don’t ignore terraform configurations dependencies 1. Optimize automated terraform workflows with git diff 1. Logically separate environments using terraform workspace 1. Overwrite environment specific values using variables precedence 1. Get comfortable with lots of terraform code; or use terrahub cli https://github.com/TerraHubCorp /terrahub
- 45. DevOps Pro Europe, Vilnius March 2019 Terraform For Serverless DEMO https://github.com/TerraHubCorp/demo- terraform-automation-aws
- 46. DevOps Pro Europe, Vilnius March 2019
- 47. DevOps Pro Europe, Vilnius March 2019 Terraform for Serverless. Best Practices. Lessons Learned. Eugene Istrati @eistrati eugene@mitocgroup.com Thank You!
Editor's Notes
- Good afternoon!
- This picture describes best what kind of challenges I’m dealing with on a daily basis. And I believe I’m not the only one. But, if you are lucky enough to use only 1 tool or service from each square, please reach out to me at the end, I’ll buy you a drink.
- Like that was not enough, welcome to the landscape of serverless.
- Invented by AWS, and joined by all other major providers like Microsoft Azure and Google Cloud, serverless is quickly becoming a parallel universe.
- Please excuse my honesty but this feels like too much to keep up with, right?
- My name is Eugene. I’m the chief technology officer of Mitoc Group. We are cloud native automation company, born in the cloud about the same time when AWS Lambda was launched in preview. These are my credentials.
- And today I would like to discuss about how we felt overwhelmed by these constantly evolving services and tools, and what we did in order to keep up with latest emerging solutions that are compatible with legacy, as well as cutting edge technologies. The Devil is in details.
- We strive to make IT simpler.
- Yeah, right. I hope you’re smiling, otherwise my humor is outdated and I would be in a really awkward situation.
- Nevertheless, a smart guy once said: When something is important enough, you do it even if the odds are not in your favor. History shows that he was right.
- So, after a short description of the problem and a small explanation into the way we think about this problem, I’ll dive into Terraform for Serverless. Next, let’s make sure that we have the same understanding on the following 3 concepts: IT as a service, DevOps, and overall scope and boundaries.
- First, IT as a service spectrum spreads across on-premises, collocation, hosting, infrastructure as a service, platform as a service and software as a service. As you can see, services are more or less the same in every category, but the difference is whether the customer is managing these services or not.
- Now, speaking of serverless, I have seen several definitions out there, but my favorite one is this tweet by Netflix former chief architect, nowadays vice president at AWS, Adrian Cockroft. If your platform as a service can efficiently start instances in 20 milliseconds that run for half a second, then call it serverless.
- Therefore, in serverless universe, we generally assume and agree among us that PaaS or SaaS are in scope, everything else is out of scope.
- Second, DevOps spectrum includes all development operations that we as engineers must go through in order to manage our IT needs end to end. Therefore agile development, CI and CD are part of DevOps, and definitely not optional or out of scope.
- And third, if we go back to devops landscape, mark deployment services as A, configuration management and provisioning as B, orchestration and scheduling as C.
- Terraform is B, but Terraform for Serverless is a combination of A + B + C.
- And there we go. This is the core of our presentation. So thank you for your patience so far. Now we are ready to talk about best practices and lessons learned from using Terraform for Serverless.
- Best practice number 1 is to adopt microservices architecture. This might sound trivial to many of you, but I still come across engineers who don’t apply this in practice.
- Heads up disclosure: I will provide examples for each best practice. To be clear, these examples are not wrong, but rather an indication that something can be done better. So, in this case of microservices architecture, instead of putting all your resources for a service in 1 terraform configuration file.
- Break it down into smaller pieces or components, aim for 1-to-1 relationship between serverless resources and terraform configuration files.
- After you break down into microservices, use terraform built in functionality to pass variables between resources. This built in feature is called terraform remote state.
- So, if you have 2 services that need to reference to the same variable, instead of hardcoding it in 2 different configuration files...
- Define it in service1, enable terraform backend like we do it in this example via S3, expose it in output and reference it in service2 using data terraform remote state.
- Third best practice – avoid using local provisioner or external data to execute external scripts. In our experience, it was extremely difficult to identify issues when something goes wrong.
- In this example above, data external will trigger the execution of build.js node script every time lambda function resource is created or updated. Fun fact, our team established as an internal rule to avoid this kind of functionality.
- Instead, we use terraform orchestration tools like terragrunt, or atlantis, or terrahub – our own home grown orchestration and automation tool.
- When we developed serverless applications, we ended up with a lot of terraform components. Unfortunately, automated terraform workflows are not built into this open source tool. Either you do it yourself, use one of previously mentioned orchestration tools or switch to using terraform enterprise.
- To be factually correct, I must mention that terraform has a built in option called parallelism, but it does not work across different terraform components.
- We ended up adding this feature into terrahub cli. This is one of the most used commands in this tool.
- Very often, we don’t need to run automated terraform workflows for all terraform components. So instead of specifying a static list of components, we use git diff to dynamically see what changed between commits or in a specific pull request, and execute automated workflow on the map of components that changed.
- Like in the example above. Instead of running automation for all 9 components in my repository...
- We used git diff and reduced the list of components down to 2. If you ask me, this is a very nice optimization of almost 80%
- One way to separate environments is to have separate code base for each environment. We don’t do that. Instead, we use terraform workspace to logically separate environments.
- Like in the example above, instead of keeping 4 different versions of pretty much the same configs for our dev environment, and test, and stage and prod...
- We use default terraform workspace as our de facto dev environment and values or variables that change for specific environment, we store them in workspace folder. When using terraform orchestration tools like terrahub cli, switching between environments is as easy as adding --env option in command line.
- Another best practice is built around terraform variables precedence.
- For example, if you have same variables used across different terraform workspaces, instead of storing the same values across different tfvar files...
- We store everything in default tfvar file corresponding to default terraform workspace, and only environment specific values that are different from default terraform workspace, we add them into separate tfvar files. When running terraform commands like plan or apply, default tfvar file comes first and specific tfvar file comes second. Please follow this link to learn more about variable precedence.
- And, finally, get used to a lot of terraform. Or don’t.
- As you can see in the example above, we have almost 34 thousand lines of code for around 100 terraform components.
- But when we switched to YAML based version using terrahub cli, we ended up with a little bit over 22 thousand lines of code. That’s over 30% reduction of the original terraform codebase. Another benefit of using terrahub cli is that we are seamlessly transforming YAML to HCL and the other way around.
- To summarize, today we have discussed 8 best practices. All lessons that we have learned while working with terraform for serverless...
- We have included them into this open source tool for terraform automation and orchestration called terrahub cli. If you like this project, please show some support and leave us a star on GitHub.
- Now, if we have some time left, let’s try to do a demo.
- At the end, I’d like to leave you with this quote from Andy Jassy, CEO of AWS. There is no compression algorithm for experience.
- Thank You!