Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

London Hashicorp Meetup #8 - Testing Programmable Infrastructure By Matt Long

3,681 views

Published on

With Hashicorp tools like Terraform, Packer and Vagrant, programmable infrastructure is reaching widespread adoption. However, although automated testing of software is becoming ever more common, the same cannot be said with testing programmable infrastructure. With microservices making our deployments more and more complex, we can no longer afford to ignore this type of testing. This talk will cover some experiences we have had testing the programmable infrastructure of complex applications, especially Terraform, and the lessons we have learned.

Published in: Software

London Hashicorp Meetup #8 - Testing Programmable Infrastructure By Matt Long

  1. 1. MATT LONG TESTING PROGRAMMABLE INFRASTRUCTURE
  2. 2. PROGRAMMABLE INFRASTRUCTURE IS GREAT, BUT WE'RE MISSING SOMETHING. TESTING.
  3. 3. I'M A TESTER HELLO, I'M MATT
  4. 4. I WORK HERE ↑
  5. 5. I AM NOT A SYSADMIN
  6. 6. WHAT IS PROGRAMMABLE INFRASTRUCTURE?
  7. 7. TESTING PROGRAMMABLE INFRASTRUCTURE THE APPLICATION OF METHODS AND TOOLING FROM SOFTWARE DEVELOPMENT TO MANAGEMENT OF IT INFRASTRUCTURE PROGRAMMABLE INFRASTRUCTURE IS.. THE INTERNET
  8. 8. TESTING PROGRAMMABLE INFRASTRUCTURE EXAMPLES OF PROGRAMMABLE INFRASTRUCTURE ▸ Automated provisioning & configuration ▸ Configuration as code ▸ Version / source controlled
  9. 9. TESTING PROGRAMMABLE INFRASTRUCTURE TOOLING EXAMPLES
  10. 10. PROGRAMMABLE INFRASTRUCTURE IS AWESOME! Credit: Vault Boy, Bethesda Softworks
  11. 11. IT'S FAST!
  12. 12. IT'S AUTOMATIC!
  13. 13. IT'S ALL CODE!
  14. 14. BUT IT GETS COMPLEX
  15. 15. TESTING IS USED TO MITIGATE COMPLEXITY & RISK
  16. 16. BUT TESTING IS RARE Credit: Gunshow, KC Green
  17. 17. TESTING PROGRAMMABLE INFRASTRUCTURE WHAT I'M GOING TO TALK ABOUT ▸PART 1: Testing a cloud broker ▸PART 2: Building a Kubernetes cluster ▸CONCLUSIONS
  18. 18. TESTING A CLOUD BROKER AN INFRASTRUCTURE HEAVY PRODUCT
  19. 19. THE PROBLEM
  20. 20. TESTING PROGRAMMABLE INFRASTRUCTURE WE WANT TO MOVE TO THE CLOUD... BUT WE'RE WARY OF LOCK IN Large organisation
  21. 21. TESTING PROGRAMMABLE INFRASTRUCTURE USERS USE MULTIPLE CLOUD PROVIDERS
  22. 22. TESTING PROGRAMMABLE INFRASTRUCTURE PROBLEMS ▸ Different interfaces, feature sets & lingo ▸ Can't switch easily ▸ Spending difficult to track ▸ Temptation to fall back on most popular
  23. 23. TESTING PROGRAMMABLE INFRASTRUCTURE USERS CLOUD BROKER
  24. 24. TESTING PROGRAMMABLE INFRASTRUCTURE BENEFITS ▸ Quick, easy provisioning ▸ one team previously took 3 months ▸ Common interface to cloud features ▸ Templates for common dev environments ▸ Built in best practice: monitoring, security ▸ Track spending
  25. 25. THIS IS A REALLY COMPLICATED APPLICATION
  26. 26. TESTING PROGRAMMABLE INFRASTRUCTURE
  27. 27. TESTING PROGRAMMABLE INFRASTRUCTURE WORKFLOW ▸ Log into Web UI ▸ Fill in information about environment ▸ Broker creates and bootstraps resources ▸ SSH into resources
  28. 28. TESTING PROGRAMMABLE INFRASTRUCTURE WEB TESTING ▸ Log into Web UI ▸ Fill in information about environment
  29. 29. TESTING PROGRAMMABLE INFRASTRUCTURE ??? ▸ Broker creates and bootstraps resources ▸ SSH into resources
  30. 30. HOW DO YOU TEST INFRASTRUCTURE?
  31. 31. TESTING PROGRAMMABLE INFRASTRUCTURE WHAT TO TEST? Do our deployment 
 scripts work? Does the VPN server work? Can instances 
 access one another? Are services running? Can I SSH into a server?
  32. 32. THIS SEEMS FAMILIAR..
  33. 33. TESTING PROGRAMMABLE INFRASTRUCTURE Does the VPN box work?
 Can I SSH into a server? Do our deployment scripts work? Are services running? ANOTHER TESTING PYRAMID? credit: Ubuntu dev quality guide
 https://developer.ubuntu.com/en/phone/platform/quality/ Can instances access one another?
  34. 34. TOOLING
  35. 35. TESTING PROGRAMMABLE INFRASTRUCTURE TOOLS AVAILABLE ▸ Bats ▸ ShUnit2 ▸ Goss ▸ ServerSpec / Inspec / TestInfra ▸ Test Kitchen
  36. 36. UNIT TESTING
  37. 37. TESTING PROGRAMMABLE INFRASTRUCTURE BATS ▸ "Bash Automated Testing System" ▸ Unit testing for bash ▸ Like JUnit
  38. 38. TESTING PROGRAMMABLE INFRASTRUCTURE SH UNIT 2 ▸ Shell unit testing framework ▸ Runs on all Bourne shells ▸ sh, BASH, DASH, ksh, zsh ▸ No activity or support?
  39. 39. INTEGRATION TESTING OR: SERVER VALIDATION
  40. 40. TESTING PROGRAMMABLE INFRASTRUCTURE GOSS ▸ Go based ▸ Specs in YAML ▸ Minimal, fast, and simple ▸ Some neat features ▸ .. have to run on the server ▸ .. no Windows support
  41. 41. TESTING PROGRAMMABLE INFRASTRUCTURE SERVERSPEC ▸ Server based assertions ▸ Ruby/RSpec based ▸ Probably the most famous ▸ Can SSH into instances
  42. 42. TESTING PROGRAMMABLE INFRASTRUCTURE INSPEC ▸ Written & maintained by Chef ▸ Very similar to ServerSpec ▸ Different feature set ▸ More focused on compliance
  43. 43. TESTING PROGRAMMABLE INFRASTRUCTURE TESTINFRA ▸ ServerSpec, but in Python
  44. 44. TEST HARNESS
  45. 45. TESTING PROGRAMMABLE INFRASTRUCTURE TEST KITCHEN ▸ Orchestrates setup, test, teardown ▸ Runs BATS, shUnit2, RSpec, Serverspec ▸ Popular in the Chef community ▸ Not suitable for our cloud broker
  46. 46. OUR SOLUTION
  47. 47. TESTING PROGRAMMABLE INFRASTRUCTURE USERS CLOUD BROKER
  48. 48. TESTING PROGRAMMABLE INFRASTRUCTURE USERS WEB TEST FRAMEWORK
  49. 49. TESTING PROGRAMMABLE INFRASTRUCTURE USERS INFRASTRUCTURE TEST FRAMEWORK
  50. 50. TESTING PROGRAMMABLE INFRASTRUCTURE USERS WEB TESTS https://github.com/opencredo/test-automation-quickstart
  51. 51. TESTING PROGRAMMABLE INFRASTRUCTURE INFRASTRUCTURE TESTS Serverspec
  52. 52. TESTING PROGRAMMABLE INFRASTRUCTURE INFRASTRUCTURE TESTING STACK / Serverspec ???
  53. 53. TESTING PROGRAMMABLE INFRASTRUCTURE WHY RUBY? ▸ Fantastic testing community ▸ More suitable for SSHing into boxes ▸ "Win RM" gem ▸ Ops already familiar with it ▸ Reduces tech stack
  54. 54. TESTING PROGRAMMABLE INFRASTRUCTURE SERVERSPEC SMOKE TESTS ▸ Run before everything else ▸ Really quick ▸ Catches obvious errors ▸ Not complex tasks
  55. 55. TESTING PROGRAMMABLE INFRASTRUCTURE SERVERSPEC EXAMPLE describe package('jenkins') do it { should be_installed } end describe service('jenkins') do it { should be_enabled } it { should be_running } end describe port(8080) do it { should be_listening } end
  56. 56. TESTING PROGRAMMABLE INFRASTRUCTURE Background:
 Given environment has been created
 And the following user details:
 | user_alias | username | public_key |
 | userA | envoy | test | 
 
 
 Scenario: IPA - Login via SSH Key authentication succeeds
 Given user "userA" is authorised to access environment vms
 When user "userA" starts ssh session in host "env" 
 Then I should be able to echo "hello world"
 CUCUMBER FOR ACCEPTANCE TESTING
  57. 57. TESTING PROGRAMMABLE INFRASTRUCTURE Background:
 Given environment has been created
 And the following user details:
 | user_alias | username | public_key |
 | userA | envoy | test | 
 
 
 Scenario: IPA - Login via SSH Key authentication succeeds
 Given user "userA" is authorised to access environment vms
 When user "userA" starts ssh session in host "env" 
 Then I should be able to echo "hello world"
 CUCUMBER FOR ACCEPTANCE TESTING Cloud broker APIs
  58. 58. TESTING PROGRAMMABLE INFRASTRUCTURE Background:
 Given environment has been created
 And the following user details:
 | user_alias | username | public_key |
 | userA | envoy | test | 
 
 
 Scenario: IPA - Login via SSH Key authentication succeeds
 Given user "userA" is authorised to access environment vms
 When user "userA" starts ssh session in host "env" 
 Then I should be able to echo "hello world"
 CUCUMBER FOR ACCEPTANCE TESTING Standard Ruby
  59. 59. TESTING PROGRAMMABLE INFRASTRUCTURE Background:
 Given environment has been created
 And the following user details:
 | user_alias | username | public_key |
 | userA | envoy | test | 
 
 
 Scenario: IPA - Login via SSH Key authentication succeeds
 Given user "userA" is authorised to access environment vms
 When user "userA" starts ssh session in host "env" 
 Then I should be able to echo "hello world"
 CUCUMBER FOR ACCEPTANCE TESTING RSpec assertions
  60. 60. TESTING PROGRAMMABLE INFRASTRUCTURE UNDER THE CUCUMBER, PLAIN RUBY Then(/^I should be able to echo "([^"]*)"$/) do |text| cmd = "echo #{text}" output = @session.exec!(cmd) close_ssh(@session) expect(output.to_s.strip).to eql(text) end
  61. 61. THOUGHTS
  62. 62. TESTING PROGRAMMABLE INFRASTRUCTURE THE GOOD ▸ Specialised tests for each layer ▸ Really quick, expressive ServerSpec tests ▸ Power of a full programming language for user tests
  63. 63. TESTING PROGRAMMABLE INFRASTRUCTURE THE BAD ▸ Over reliance on acceptance tests ▸ Awkward switching between two suites ▸ Out of my comfort zone
  64. 64. TESTING PROGRAMMABLE INFRASTRUCTURE THE UGLY ▸ Starting infrastructure is SLOW. ▸ It's expensive...
  65. 65. IT WAS WORTH IT DESPITE ALL THAT
  66. 66. BUILDING A KUBERNETES CLUSTER APPLYING TDD TO INFRASTRUCTURE
  67. 67. INTERNAL DEVOPS TRAINING COURSE
  68. 68. I LEARNED A LOT! Credit: The Simpsons, Fox
  69. 69. TESTING PROGRAMMABLE INFRASTRUCTURE BUILD THIS: WITH THESE:
  70. 70. TESTING PROGRAMMABLE INFRASTRUCTURE NOT A STRAIGHTFORWARD TASK
  71. 71. TESTING PROGRAMMABLE INFRASTRUCTURE BUT HOW TO TEST IT? ▸ This is a dev activity ▸ Want fast feedback ▸ Complexity is mitigated by testing!
  72. 72. TESTING PROGRAMMABLE INFRASTRUCTURE ▸ Provisions cloud infrastructure ▸ Declarative files ▸ Some support for variables TERRAFORM
  73. 73. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM COMMANDS ▸ terraform plan ▸ Tells you what will change ▸ terraform apply ▸ Applies changes ▸ terraform validate ▸ Lints terraform syntax
  74. 74. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM FILE EXAMPLE resource "aws_instance" "etcd-node" { count = 3 ami = "ami-7abd0209" # centos availability_zone = "eu-west-1a" # ireland instance_type = "t2.micro" subnet_id = .... private_ip = .... key_name = "${aws_key_pair.my-key.key_name}" }
  75. 75. TESTING PROGRAMMABLE INFRASTRUCTURE LINT WITH 'TERRAFORM VALIDATE' COMMAND Omitting a variable:
  76. 76. TESTING PROGRAMMABLE INFRASTRUCTURE BUT IT DOESN'T CATCH ALL PROBLEMS Duplicate subnet CIDRS:
  77. 77. TESTING PROGRAMMABLE INFRASTRUCTURE LINTING ISN'T ENOUGH ▸ Devs don’t just rely on compilers ▸ We need something more powerful Credit: Nick Cave, "Soundsuit"
  78. 78. UNIT TESTING
  79. 79. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM_VALIDATE ▸ Python based unit testing ▸ NOT to be confused with 'validate' command ▸ Builds map of resources & properties ▸ Totally offline ▸ New and incomplete https://github.com/elmundio87/terraform_validate
  80. 80. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM_VALIDATE FORK OC has forked the terraform validate repo https://github.com/opencredo/terraform_validate
  81. 81. INTEGRATION TESTING
  82. 82. TESTING PROGRAMMABLE INFRASTRUCTURE GOSS ▸ Easy to get up and running ▸ Doesn’t support remote # example usage: ./goss-test.sh 34.248.91.167 TARGET='centos@'$1 SSH_KEY_PATH=~/.ssh/aws ssh -t -t -i $SSH_KEY_PATH $TARGET 'curl -fsSL https://goss.rocks/install | sudo sh' scp ./goss.json $TARGET:~/goss.yaml ssh -t -t -i $SSH_KEY_PATH $TARGET 'goss validate' https://gist.github.com/burythehammer/081d6ee11cc33c2f4c4729ae67622f5b
  83. 83. TESTING PROGRAMMABLE INFRASTRUCTURE ▸ Terraform compatibility ▸ Already a talk about this ▸ “Untangling Infrastructure Code” by Nell Shamrell-Harrington TEST KITCHEN + INSPEC
  84. 84. TESTING PROGRAMMABLE INFRASTRUCTURE TestCreate Config Destroy ]TEST KITCHEN MANAGES YOUR TEST LIFECYCLE
  85. 85. TESTING PROGRAMMABLE INFRASTRUCTURE TestCreate Config Destroy TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS
  86. 86. TESTING PROGRAMMABLE INFRASTRUCTURE TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS https://github.com/test-kitchen/test-kitchen/issues/329
  87. 87. TESTING PROGRAMMABLE INFRASTRUCTURE TERRAFORM 'NULL RESOURCE' resource "null_resource" "ansible" { triggers { instance_ids = "${join(",", aws_instance.etcd-node.*.id)}" } provisioner "local-exec" { command = "sleep 20 && cd ../ansible/ && ansible-playbook etcd.yaml" } }
  88. 88. TESTING PROGRAMMABLE INFRASTRUCTURE TestCreate Config Destroy
  89. 89. THOUGHTS
  90. 90. TESTING PROGRAMMABLE INFRASTRUCTURE THE GOOD ▸ Tooling exists! ▸ You can totally get a test suite working Credit: Overwatch, Blizzard Entertainment
  91. 91. TESTING PROGRAMMABLE INFRASTRUCTURE THE BAD ▸ Unit testing extremely immature ▸ Tools immature in general Credit: Overwatch, Blizzard Entertainment
  92. 92. TESTING PROGRAMMABLE INFRASTRUCTURE THE HACKY ▸ Be prepared to hack ▸ It might not even be possible Credit: Overwatch, Blizzard Entertainment
  93. 93. THIS IS BRAND NEW GROUND REMEMBER:
  94. 94. TESTING TOOLS DEPEND ON YOUR STACK
  95. 95. CONCLUSIONS
  96. 96. TESTING IS IMPORTANT BUT OFTEN IGNORED
  97. 97. TESTERS AND OPS SHOULD WORK TOGETHER WE NEED TO GET OUT OF OUR COMFORT ZONES
  98. 98. TOOLS EXIST BUT BE PREPARED TO HACK
  99. 99. FINALLY...
  100. 100. TESTING PROGRAMMABLE INFRASTRUCTURE THE APPLICATION OF METHODS AND TOOLING FROM SOFTWARE DEVELOPMENT TO MANAGEMENT OF IT INFRASTRUCTURE PROGRAMMABLE INFRASTRUCTURE IS..
  101. 101. TESTING IS A SOFTWARE DEVELOPMENT METHOD WE SHOULD APPLY IT TO INFRASTRUCTURE
  102. 102. THANKS QUESTIONS? @burythehammer matt.long@opencredo.com

×