More Related Content
Similar to Azure Infrastructure as Code and Hashicorp Terraform
Azure Infrastructure as Code and Hashicorp Terraform
- 1. @AlexMags Microsoft Azure Infrastructure asCode and Hashicorp Terraform Alex Magnay - @alexmags
- 2. @AlexMags Alex Magnay Twitter: @alexmags Email:alex@alexmags.com LinkedIn: linkedin.com/in/amagnay
- 3.
- 4.
- 5. @AlexMags The hard way AWSCloudFormation & custom scripts
- 6.
- 7. @AlexMags Hashicorp -> defacto standard https://aws.amazon.com/blogs/devops/how-to-create-an-ami-builder-with-aws-codebuild-and-hashicorp-packer-part-2/
- 8.
- 9. @AlexMags “Take the Devout of DevOps” Don’t DIY infrastructure management tools
- 10.
- 11.
- 12. @AlexMags Thankyou Hashicorp! (and HUGUK Meetup!)
- 13. @AlexMags This talk • AzureInfrastructure as a Service • Hashicorp Terraform • Terraform Workflow • Demo • Operations, Security, Development teams • Microsoft & Hashicorp News
- 14.
- 15. @AlexMags IaaS: Azure catchingup with AWS https://azure.microsoft.com/en-gb/blog/gartner-names-microsoft-azure-as-a-leader-in-the-cloud-iaas-mq/
- 16.
- 17.
- 18. @AlexMags Healthy competition hasbenefits https://aws.amazon.com/blogs/aws/new-per-second-billing-for-ec2-instances-and-ebs-volumes/
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30. @AlexMags What is Terraform? •A way to manage Azure • Domain Specific Language • Declarative • Easy to read and write • Drives the Azure API • Runs on Windows & Linux • Open Source • Free • Yes, seriously, it’s free
- 31. @AlexMags What is TerraformNOT? • Not OS configuration management • Not an abstraction layer for any cloud
- 32. @AlexMags https://www.terraform.io/docs/providers - September2017 Alicloud Archive Arukas AWS Bitbucket CenturyLinkCloud Chef Circonus Cloudflare CloudStack Cobbler Consul Datadog DigitalOcean DNS DNSMadeEasy DNSimple Docker Dyn External Fastly GitHub Gitlab Google Cloud Grafana Heroku HTTP Icinga2 Ignition InfluxDB Kubernetes Librato Local Logentries Mailgun New Relic Nomad NS1 Microsoft Azure MySQL 1&1 Oracle Public Cloud OpenStack OpsGenie OVH Packet PagerDuty PostgreSQL PowerDNS ProfitBricks RabbitMQ Rancher Random Spotinst Template Terraform Terraform Enterprise TLS Triton UltraDNS Vault VMware vCloud Director VMware vSphere
- 33. @AlexMags Resource Groups App Service(web apps) App Insights Content Delivery Network Containers CosmosDB (Document DB) DNS records Event Hubs Key vault Event Hub Virtual Network Resources Load Balancers Managed Disk Redis cache Azure Search ServiceBus Azure SQL Storage ARM templates Virtual Machines https://www.terraform.io/docs/providers/azurerm - September 2017 Terraform these Azure Resources
- 34.
- 35.
- 36.
- 37. @AlexMags Terraform Workflow Edit CodeTerraform.exe Plan Terraform.exe Deploy Execution Plan
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45. @AlexMags Demo Time Shut upand prove it!
- 46. @AlexMags Terraform For Operations •Deploy, change, manage IaaS (any cloud!) • With source control you can roll back to previous state • Delegate dev environments to dev teams • Give your execution plan to someone else to apply out of hours
- 47. @AlexMags Terraform For Security •Enforce configuration • Git commit history - See WHO changed WHAT and WHY • Delegate Azure access to a scheduler (Jenkins/Teamcity) • Security concerns – long lived API access keys with privileged access • Don’t store keys in code or source control • Don’t store keys in config files in default locations • Don’t store keys in user or machine environment variables • Use short key expiry times (1 hour)
- 48. @AlexMags Beware: Long livedAPI access keys https://www.terraform.io/docs/providers/azurerm/index.html
- 49. @AlexMags Plain text keysin default locations unsafe http://theburningmonk.com/2017/07/slides-for-my-serverless-security-talk (65)
- 50. @AlexMags Terraform For Developers OpsTerraform • Resource groups • vNets • Subnets • VPNs • Shared infra services • Security groups • Ops state file Dev Terraform • Read only Ops state file • Dev VMs and Apps • Dev state file
- 51. @AlexMags Terraform For Developers OpsResource Group Dev Resource Group
- 52. @AlexMags Windows PowerShell Copyright (C)2016 Microsoft Corporation. All rights reserved. PS H:> cd MyEnvironment PS H:MyEnvironment> terraform apply PS H:MyEnvironment> terraform destroy
- 53. @AlexMags Terraform For YourBudget • Terraform is open source and free • Tear up & tear down easily – only pay when required • Let terraform clean up. Avoid wasteful cruft • Don’t write your own cloud infra management tool!
- 54.
- 55.
- 56.
- 57. @AlexMags August 2017 “I amexcited to announce that we are greatly increasing our investment in Terraform, partnering closely with HashiCorp, a well-known voice in the DevOps and cloud infrastructure management space.” Corey Sanders, Director of Program Management, Azure, Microsoft Corp. HashiCorp, a leader in cloud infrastructure automation, today announced a multi-year collaboration with Microsoft to deepen support for the provisioning of Microsoft Azure cloud services with HashiCorp Terraform. http://www.marketwired.com/press-release/hashicorp-extend-work-with-microsoft-multi-year-collaboration-that-enables-hashicorp-2230675.htm
- 58.
- 59. @AlexMags Takeaways & TipsFrom the Field • Don’t mix manual deploy and Terraform • Start simple and build up iteratively • Establish a resource naming convention quickly • Tag everything ‘deployed_by=terraform’ • Use comments liberally • Use modules, variablise everything, set sensible defaults • Use remote backend/remote state file • Ops need to learn source control tools (Git) • Stay safe: Avoid long lived API access keys
- 60. @AlexMags Resources terraform.io/docs GitHub Hashicorp Terraformexamples github.com/hashicorp/terraform/tree/master/examples TerraformBook.com meetup.com/London-HashiCorp-User-Group
- 61.
- 62. @AlexMags Thanks! Questions? Alex Magnay(hire me!) Twitter: @alexmags Email:alex@alexmags.com
Editor's Notes
- #2 CTO at Hentsu We consult, deploy and manage public cloud for our customers. Specialise in the Asset Management, hedge fund space. High availability, high security, regulatory compliance. London and NY. Come off big multi region azure deployment. Used terraform
- #3 Last seen working at consultancy Hentsu spinning up new hedge funds and migrating hedge funds to public cloud infra Background engineering teams investment banking, asset management regulatory compliance, high security, high availability, high tech certifications
- #4 1st slice of cloud was replacing an aging NetApp filer and a bunch of HP g1 blade servers with VMs on AWS. This was a grid computing cluster provided researchers with 128 CPUs and a few TB of shared storage. If you’re going to go to the board and tell them you’re going to put their secret sauce code and their IP on “someone else’s servers” Start with a calm, restful image like this while you convince them “it’s all going to be ok”
- #6 Yes we felt like banging head against wall (slide 5 and we’re already at cats) Some sort of record.. Why here today Cloudformation and bunch of scripts to deploy.
- #7 2015 maybe packer wasn’t quite ready?
- #8 How to create an AMI builder on AWS blog
- #9 Microsoft tells you how to use packer
- #11 With our public cloud skills we can do Hedge Fund IT as a service
- #12 IaaS deployments using Infrastructure as Code Spin up new environments for customers VERY quick Repeatable, reliable, managable
- #13 Thanks for Hashicorp for making great products and sharing them with us Thanks to this meetup for showing us the light I went to the 1st HUG Meetup March 2016 Mitchell Hashimoto
- #14 On prem vs IaaS Terrafrom Why youre here. WHAT it is Terraform workflow HOW to use it Demo Terraform for Dev, Sec, and Ops News Warning: Fetish for excruciating PowerPoint transitions.
- #15 This is AWS sumit 2014 – FDLFBDC This was the mantra because maintaining DCs is a huge drag on IT and doesn’t add value We had a customer. They wanted to get out of someone else’s datacentre So called “private cloud” which what used to be called hosting. They didn’t want to build and maintain their own DCs They wanted to use public cloud specifically Azure not AWS
- #16 AWS grew first, lots of nimble web start-ups. Big corp is also getting the bug now The services are comparable, virtualisation, storage, networking. Office365 and Azure AD why go off the reservation and bring yet another supplier into the mix.
- #17 Azure Security Centre is amazing Anyone here “I don’t care I hate anything from Microsoft”
- #18 Listen to Pre-CGI yoda – hate leads to suffering Like developers didn’t care if servers were Dell or HP Developers don’t care if their VM / container spins up on Vmware, AWS or Azure As long as they can run the frameworks they want and get their releases out with minimum friction, they’re happy One of the most sophisticated investment banks is dynamically sending workloads to on prem, AWS or Azure based on cost and capacity.
- #19 AWS cuts billing granularity from per-hour to per-second Every time you turn on a VM you got charged a minimum of an hour Because Azure was already per-minute (This doesn’t include Windows or Enterprise Linux or anything from Marketplace billing)
- #20 Azure gets you virtual datacentres, anywhere you want
- #21 Ireland, London, Cardiff, Frankfurt, Netherlands Two more coming in France because Pourquoi Pas? Marseille, Paris “by end of year”. Sweden in 2018. Put infra– where your staff are, where your customers are, or just where it happens to be cheaper to run at the moment Terraform is tremendous for configuring the software defined networking (virtual networks, subnets, routing tables) and then dropping VMs into them.
- #22 Azure datacenters are positioned on laylines of tremendous connectivity If you’re an international organization , investigate if you can ditch your point to point international lease lines and use public cloud provider as a hub to link your offices and datacenters. CLICK When comparing the cost of on prem vs public cloud You can connect to Azure, break out to internet from there and use Azure as hub to connect offices. No international leased lines. $$$
- #23 Azure datacenters are positioned on laylines of tremendous connectivity If you’re an international organization , investigate if you can ditch your point to point international lease lines and use public cloud provider as a hub to link your offices and datacenters. CLICK When comparing the cost of on prem vs public cloud You can connect to Azure, break out to internet from there and use Azure as hub to connect offices. No international leased lines. $$$
- #24 Now blend other pri cloud providers into the mix Snapshot Nov 2016 Azure had nearly twice the number of locations as AWS Azure has more regions than AWS and Google put together (double)
- #25 This is the news no ops guy wants to hear worst has happened product is wildly successful With public cloud “you got this” Scale up to bigger VMs, scale out to more, go global You can also contract.
- #26 Oh yes. I’m going there. “I need the trading chain spun up in Frankfurt” “I need Corp IT env for new Paris office” Again: “youve got this” DCs, file servers, VDI in Paris, some trading servers in Frankfurt.
- #27 Once you have connectivity to public cloud And you have your security in place (IAM policies and federation, policies)
- #28 Dragging things back to the point of talk May I proudly present!!
- #29 May I present Terraform (finally..) So what is it
- #30 Don’t panic!! It looks like programming but Ops guys I promise, you’ve totally got this I’ll come back to this
- #31 This is key WHAT it should look like Not HOW to get there Think desired state configuration Diff to imperative like powershell commands think order, not repeat
- #32 Azure VM Extentions, AWS user data 1st boot strips Install chef/puppet agents or configure DSC on new VMs Or enrole new system in config management You can be useful on AWS/Azure & you can recruit
- #33 Terraform has a plugin system of “providers”. AWS, Google Cloud, Microsoft Azure Bitbucket and Github Template, TLS, Random, HTTP VMware vSphere -
- #34 Terraform has a plugin system of “providers”. The azure one can manage all this stuff so far Virtual network resources Vnet peering (spin up a new virtual data centre, wire it to existing virtual data centre (hub vnet with connection to on prem), configure all the routing ARM Templates
- #35 Resource group Virtual network (virtual DC) 1 subnet called subnet 1
- #36 Rg name lookup/cross reference “interpolation” No messing with Azure object IDs or AWS ARNs Type of resource – resource identifier – some property West US is repeated
- #37 After WHAT, not the HOW we use terraform
- #38 Bust out your favourite editor new favourite Use editor with assistance for Hashicorp Language (intellisense) I started on IntelliJ, Microsoft VScode also has HCL plugin now Git support is also useful
- #39 Run terraform in PLAN mode Reads the code it finds in current directory Compares the code to your Azure subscription and works out differences (It also tells you if you’ve got any errors in your code that would prevent it from running)
- #40 Produces a report of what WOULD change IF you ran this. 1) What resources would be added (Green) -example 2) What resources would be modified (yellow) -example 3) What resources would be removed (red) –example
- #41 Terraform in Deploy mode Executes the plan and drives Azure API to make changes
- #42 VMs, SQL instances, security groups, vnets But wait there’s more – checkout this transition…..
- #43 PowerPoint acrobatics ladies and gentlemen This is a cycle Start small and build. Drop in incremental changes
- #44 When you’re done with the environment – terraform destroy will tear down for you stop the billing clock $$$ You can also hook scripts “destroy provisioners” Remove machine from monitoring, clean up AD and DNS records, remove from config management Easy Tear up and tear down the latest version of the infra code – great for development environments. Easy reset.
- #45 This is how you use terraform on Windows Ha ha…. PLAN, APPLY, DESTORY That’s all you need to know to use terraform………..
- #46 !!!!PC SCREEN ONLY SWAP!!!!
- #47 Infrastructure as code - more consistent. Less manual errors and troubleshooting
- #48 Demo crazy dave. Undo changes See WHAT changed and along with audit log you can see WHO changed it but no indication WHY. If changes are linked to git commit with reference to trouble ticket (JIRA/ServiceNow) which links to an APPROVED change request. Now the change control process becomes audit trail of WHO changed WHAT and WHY You can build a release pipeline where you don’t have change access but the scheduler does. Git commit, terraform validate, Security concerns: Long lived API keys. Give the CI system permission to push changes on behalf of staff.
- #50 AWS UG July Yan Cui Burning monk. Exploiting weak passwords on public package repos. Added dependencies to packages that read creds when installed. AWS client CLI ini file. Don’t store keys on unmanaged devices (random bring your own macbook). Changes pushed from a hardened admin/management machine Least rights privilege.
- #51 Dev 3 things Safely partition access to resources, allowing development freedom to manage own deployments/labs
- #52 Azure Resource groups PERFECT for delegating access Resource Tags for tracking and billing Make this slide build out?
- #53 This easy for developer If developers can spin up & tear down their own environments develops a culture of experimentation Terraform (because it's code) fits well in a continuous delivery pipeline
- #54 All these things save you money
- #56 There’s some love going on between MS and HC March 2016 MS and Hashicorp announced Azure PROPERLY supported
- #57 Around Build2017 May Terraform has momentum Microsoft is embracing terraform and collaborating with Hashicorp to ensure Terraform support for Azure keeps pace with new shiny stuff on Azure
- #58 August 2017 Multi-year collaboration Opensource Terraform definitely a safe bet
- #60 Do or do not! Modules can enforce naming and tagging conventions Modules can enforce storage encryption
- #63 I’m doing talk one more time so feedback gratefully received