Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Declarative & workflow based infrastructure with Terraform

DevOps Glasgow; 26th April

Declarative & workflow based infrastructure with Terraform

  1. 1. @radeksimko Declarative & workflow based infrastructure
  2. 2. @radeksimko Radek Simko @radeksimko
  3. 3. @radeksimko
  4. 4. @radeksimko
  5. 5. @radeksimko Terraform Goals ● Describe infrastructure as code ● Provision efficiently (get as far as APIs allow at a time) ● Cover the whole lifecycle of the infrastructure ● Provide composability within and across multiple tiers (IaaS, PaaS, SaaS)
  6. 6. @radeksimko Terraform is NOT solving ● OS-level provisioning ○ Chef ○ Puppet ○ Ansible ○ SaltStack ○ … ● But integrates with these tools
  7. 7. @radeksimko Terraform 101
  8. 8. File resource "google_compute_instance" "server" { name = "server" machine_type = "g1-small" zone = "us-central1-a" disk { image = "ubuntu-1404-trusty-v20160114e" } network_interface { network = "default" } } resource "dnsimple_record" "hello" { domain = "example.com" name = "server" value = "${google_compute_instance.server.network_interface.0.address}" type = "A" }
  9. 9. Terminal $ terraform plan + google_compute_instance.server can_ip_forward: "false" create_timeout: "4" disk.#: "1" disk.0.auto_delete: "true" disk.0.disk_encryption_key_sha256: "<computed>" disk.0.image: "ubuntu-1404-trusty-v20160114e" machine_type: "g1-small" metadata_fingerprint: "<computed>" name: "server" network_interface.#: "1" network_interface.0.address: "<computed>" network_interface.0.name: "<computed>" self_link: "<computed>" tags_fingerprint: "<computed>" zone: "us-central1-a" ... Plan: 2 to add, 0 to change, 0 to destroy.
  10. 10. Terminal $ terraform apply google_compute_instance.server: Creating... can_ip_forward: "" => "false" create_timeout: "" => "4" disk.#: "" => "1" disk.0.auto_delete: "" => "true" disk.0.disk_encryption_key_sha256: "" => "<computed>" disk.0.image: "" => "ubuntu-1404-trusty-v20160114e" machine_type: "" => "g1-small" metadata_fingerprint: "" => "<computed>" name: "" => "server" ... zone: "" => "us-central1-a" google_compute_instance.server: Still creating... (10s elapsed) google_compute_instance.server: Still creating... (20s elapsed) ... Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
  11. 11. @radeksimko
  12. 12. @radeksimko Infrastructure as Code
  13. 13. @radeksimko https://www.youtube.com/watch?v=SLh_6Ievi7A Infrastructure as Code Might be Literally Impossible
  14. 14. @radeksimko high-level language language for dataDSL JSON {} YAML -- :
  15. 15. @radeksimko http://jsonnet.org
  16. 16. @radeksimko HCL ● github.com/hashicorp/hcl ● Used in various HashiCorp projects ○ Consul ○ Vault ○ Nomad ○ Terraform ● JSON compatible ○ Useful for generated code
  17. 17. Terminal @radeksimko service { key = "hi ${var.world}" }
  18. 18. @radeksimko https://github.com/alphagov/govuk-dns
  19. 19. @radeksimko Efficiency
  20. 20. Terminal @radeksimko #!/bin/sh aws rds create-instance ... aws ec2 run-instance ... aws ec2 run-instance ... aws ec2 run-instance ... aws s3 create-bucket ... aws route53 create-recordset ...
  21. 21. Terminal @radeksimko resource "aws_s3_bucket" "b" { ... } data "aws_ami" "ubuntu" { ... } resource "aws_instance" "web" { count = 3 ... } resource "aws_db_instance" "default" { ... } resource "aws_route53_record" "www" { ... }
  22. 22. @radeksimko
  23. 23. Terminal @radeksimko $ terraform graph | dot -Tpng > graph.png $
  24. 24. @radeksimko Applying Graph Theory ● Build a graph ● Transform, find cycles ● Run operations in parallel
  25. 25. @radeksimko 1. 2.
  26. 26. @radeksimko Applying Graph Theory The only limit is the sky (Cloud API throttling)
  27. 27. @radeksimko https://www.youtube.com/watch?v=Ce3RNfRbdZ0
  28. 28. @radeksimko Life of Pi Resource
  29. 29. Terminal @radeksimko #!/bin/sh # Creation aws rds create-instance ... aws ec2 run-instance ... aws ec2 run-instance ... aws ec2 run-instance ... aws s3 create-bucket ... aws route53 create-recordset ...
  30. 30. Terminal @radeksimko #!/bin/sh # Renaming DB instance aws rds create-instance … # new one aws rds restore-from-snapshot … aws route53 update-recordset … aws rds destroy-instance … # old one
  31. 31. Terminal @radeksimko #!/bin/sh # Renaming DB instance aws rds create-instance … # new one # wait until state == launched aws rds restore-from-snapshot … # wait until restored aws route53 update-recordset … # wait until DNS in sync aws rds destroy-instance … # old one # wait until instance gone
  32. 32. Terminal @radeksimko "identifier": { Type: schema.TypeString, Optional: true, ForceNew: true, ConflictsWith: []string{"identifier_prefix"}, ValidateFunc: validateRdsIdentifier, }, "username": { … "password": { …
  33. 33. Terminal @radeksimko stateConf := &resource.StateChangeConf{ Pending: []string{"creating", "backing-up", "modifying", "resetting-master-credentials", "maintenance", "renaming", "rebooting", "upgrading"}, Target: []string{"available"}, Refresh: resourceAwsDbInstanceStateRefreshFunc(d, meta), Timeout: 40 * time.Minute, MinTimeout: 10 * time.Second, Delay: 30 * time.Second, }
  34. 34. @radeksimko Resource Schema ● More or less mapped to API structures ● Allows TF make decisions during the lifecycle ○ Creation ○ Updates ○ Destruction ● Allows TF to present accurate plan ahead of applying it ● Non-updatable fields
  35. 35. Terminal @radeksimko $ terraform plan ~ aws_instance.app tags.%: "0" => "1" tags.Name: "" => "HelloWorld" Plan: 0 to add, 1 to change, 0 to destroy.
  36. 36. Terminal @radeksimko $ terraform plan -/+ aws_instance.app ami: "ami-408c7f28" => "ami-4d795c5a" (forces new resource) associate_public_ip_address: "true" => "<computed>" availability_zone: "us-east-1d" => "<computed>" ebs_block_device.#: "0" => "<computed>" ephemeral_block_device.#: "0" => "<computed>" instance_state: "running" => "<computed>" instance_type: "t1.micro" => "t1.micro" key_name: "" => "<computed>" network_interface_id: "eni-8d666460" => "<computed>" ... Plan: 1 to add, 0 to change, 1 to destroy.
  37. 37. @radeksimko Schema + DSL + graph theory allow Terraform to make the right decisions during the whole lifecycle
  38. 38. @radeksimko Composability
  39. 39. @radeksimko ● S3 ● CloudFront ● Route 53 ● RDS ● ... ● Cloud DNS ● Cloud CDN ● Cloud Storage ● Spanner ● BigTable ● ...
  40. 40. @radeksimko
  41. 41. @radeksimko
  42. 42. @radeksimko What is infrastructure then? A lot of things
  43. 43. @radeksimko Terraform
  44. 44. @radeksimko Providers Amazon Bitbucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato Microsoft Azure MySQL OpenStack Packet PostgreSQL SoftLayer UltraDNS VMware vSphere and more...
  45. 45. @radeksimko Providers Amazon Bitbucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato Microsoft Azure MySQL OpenStack Packet PostgreSQL SoftLayer UltraDNS VMware vSphere and more...
  46. 46. @radeksimko Providers Amazon Bitbucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato Microsoft Azure MySQL OpenStack Packet PostgreSQL SoftLayer UltraDNS VMware vSphere and more...
  47. 47. @radeksimko Provider ● Group of resources ● Responsible for validating creds, setting up connection to API(s) ● Works in isolation ○ Terraform core is responsible for relationships between providers ● Is pluggable ○ Binary which talks RPC ○ Can be written in any language ■ But nobody has written helper libraries outside of Go land so far
  48. 48. Terminal @radeksimko provider "aws" { region = "eu-west-1" profile = "prod" }
  49. 49. @radeksimko Resource ● CR(U)D ○ Create() ○ Read() ○ Update() ○ Delete() ● API specifics ● Responsible for eventually making API calls ● Arguments (e.g. tags to assign to EC2 instance, AMI ID to use) ● Attributes (e.g. instance ID, ...)
  50. 50. Terminal @radeksimko resource "aws_instance" "web" { ami = "${data.aws_ami.ubuntu.id}" instance_type = "t2.micro" tags { Name = "HelloWorld" } }
  51. 51. @radeksimko Data Source ● The “R” from resources (CRUD) ● Useful for ○ building on existing infrastructure ○ Keeping configs reusable ● Never modify loaded resource
  52. 52. File data "aws_availability_zones" "available" {} # Create 1 subnet per availability zone resource "aws_subnet" "primary" { count = "${length(data.aws_availability_zones.available.names)}" availability_zone = "${data.aws_availability_zones.available.names[count.index]}" ... }
  53. 53. File data "aws_vpc" "selected" { cidr_block = "10.0.1.0/20" } resource "aws_subnet" "primary" { vpc_id = "${data.aws_vpc.selected.id}" }
  54. 54. File data "aws_cloudformation_stack" "network" { name = "my-network-stack" } resource "aws_instance" "web" { ami = "ami-abb07bcb" instance_type = "t1.micro" subnet_id = "${data.aws_cloudformation_stack.network.outputs["SubnetId"]}" tags { Name = "HelloWorld" } }
  55. 55. @radeksimko State Management
  56. 56. @radeksimko POST / HTTP/1.1 Host: ec2.eu-west-2.amazonaws.com User-Agent: aws-sdk-go/1.8.13 Authorization: ... Content-Type: ... Action=RunInstances&... ImageId=ami-63342007&InstanceType= t2.micro... HTTP/1.1 200 OK Connection: close Content-Type: ... Server: AmazonEC2 <?xml version="1.0" encoding="UTF-8"?> <RunInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-1 5/"> … <instanceId>i-02228e28962ee478b</instanceId>
  57. 57. @radeksimko Plz change this tag from old to new, Ta! For which instance? Few days later...
  58. 58. @radeksimko Existing resources ● Lookups via tags or names ○ Not everything AWS supports tags ○ Not every provider supports tags ○ Tags may not be unique ○ Unique names don’t allow graceful recreation ○ 1 extra lookup API call to get the real unique ID for modify/delete API calls ● Storing unique ID in a state (Terraform) ○ Guaranteed to be unique ○ Provider-agnostic solution
  59. 59. @radeksimko Worst case scenario ● Duplicate/wrong tag/name ■ Changed wrong resource ■ Destroyed wrong resource ● Drifted/decoupled state ○ Duplicate infrastructure ○ Original (production?) resources intact
  60. 60. @radeksimko State Mgmt in Terraform ● Part of Remote Backend ○ Remote State ○ Locking ○ Environments ● Many supported backends ○ Consul, etcd ○ Azure Storage, S3, Google Cloud Storage ○ TFE ○ ...
  61. 61. Terminal @radeksimko # main.tf resource "aws_instance" "app" { count = 5 ami = "ami-408c7f28" instance_type = "t1.micro" }
  62. 62. Terminal @radeksimko terraform { backend "s3" { bucket = "my-tfstate-prod" key = "terraform.tfstate" region = "eu-west-2" } }
  63. 63. @radeksimko Summary ● Infrastructure as Code ● Efficiency ● Full lifecycle ● Composability ● State Management
  64. 64. Thanks! Questions? @radeksimko

    Be the first to comment

    Login to see the comments

  • AnthonyGiza

    May. 11, 2017
  • JunshanHe

    Jun. 17, 2017
  • rorybramwell

    Jul. 18, 2017

DevOps Glasgow; 26th April

Views

Total views

1,011

On Slideshare

0

From embeds

0

Number of embeds

2

Actions

Downloads

36

Shares

0

Comments

0

Likes

3

×