Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Integration testing for salt states using aws ec2 container service

1,143 views

Published on

A SaltConf16 use case talk by Steven Braverman of Dun & Bradstreet. Testing configuration changes for multiple server roles can be time consuming when real instances or legacy container systems are used. Applying configuration changes to each role in parallel can be difficult. So what's the best way to test configuration changes efficiently, quickly, and securely prior to applying them? See how an integrated test setup using AWS EC2 Container Service (ECS), AWS AutoScaling Group, and SaltStack simplifies the application of configuration changes and allows you to test configuration changes in parallel to reduce the time spent testing.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Integration testing for salt states using aws ec2 container service

  1. 1. Integration Testing for Salt States Using AWS EC2 Container Service Steven Braverman
  2. 2. DevOps Engineer at D&B Graduated from UCLA with B.S. Cognitive Science Developer Background Love to surf STEVEN BRAVERMAN
  3. 3. • pd2jira_python • SIT Created: Contributed to: • SaltStack • Troposphere • Placebo OPEN SOURCE PROJECTS
  4. 4. ~175 year old company that prides on its capability to nurture invaluable relationships in business by uncovering truth and meaning from data.
  5. 5. DEVOPS
  6. 6. It is important to test applying states to roles efficiently, quickly, and securely prior to applying them to production servers. TESTING SALT STATES
  7. 7. MASTER SALT MASTER/MINION SETUP
  8. 8. MINIONS MASTER SALT MASTER/MINION SETUP
  9. 9. MINIONS BANANAGRAPESPEARWATERMELONAPPLE MASTER SALT MASTER/MINION SETUP
  10. 10. MINIONS BANANAGRAPESPEARWATERMELONAPPLE MASTER SALT MASTER/MINION SETUP
  11. 11. MINIONS BANANAGRAPESPEARWATERMELONAPPLE MASTER SALT MASTER/MINION SETUP
  12. 12. SALT MASTER/MINION SETUP MASTER MINIONS BANANAGRAPESPEARWATERMELONAPPLE
  13. 13. MINIONS MASTER SALT MASTER/MINION SETUP
  14. 14. MASTER PHP JENKINS NODES SALT MASTER/MINION SETUP JAVA DATABASE
  15. 15. MASTER PHP JENKINS NODES SALT MASTER/MINION SETUP JAVA DATABASE
  16. 16. MASTER PHP JAVA DATABASE JENKINS NODES SALT MASTER/MINION SETUP
  17. 17. MASTER PHP JAVA DATABASE JENKINS NODES SALT MASTER/MINION SETUP OPENVZ VIRTUAL MACHINES
  18. 18. SALT MASTER/MINION SETUP OPENVZ VIRTUAL MACHINES Pull-Request Pull-request builder Results
  19. 19. PROBLEMS • Linear integration testing is slow • Have to maintain legacy virtual machines
  20. 20. PROBLEMS • Linear integration testing is slow • Have to maintain legacy virtual machines No generic ways to run integration tests for SaltStack exist
  21. 21. Applying Salt states to a series of Docker images will: HYPOTHESES • Speed up the time it takes to apply state changes to roles • Allow for concurrent builds • Be easier to maintain
  22. 22. SALT INTEGRATION TESTING
  23. 23. MASTER PHP JAVA DATABASE JENKINS NODES SIT Flow ASG
  24. 24. JENKINS NODES SIT Flow ASG 1. Create a pull-request 2. Pull salt repository and run unit tests 3. “Initiate” Jenkins node to be a Master 4. Launch Integration Test 5. “Teardown” node back to a Minion 6. Rinse and Repeat
  25. 25. INITIATION AND TEARDOWN • Pull down your Salt repo into the workspace • Run lint tests • Run unit tests/coverage Run SIT Teardown SITInitiate SITPre-SIT Tasks
  26. 26. INITIATION AND TEARDOWN • SED to configure Jenkins node master config • Start Salt-Master service • Start Redis service • Install SIT requirements Run SITPre-SIT Tasks Initiate SIT Teardown SIT
  27. 27. INITIATION AND TEARDOWN Pre-SIT Tasks Run SIT Teardown SITInitiate SIT 1. SIT requests AutoScaling Group to provision a new instance 2. Instance provisions & SIT discovers it 3. ASG instance gets registered into ECS cluster 4. ECS Tasks begin (runs Docker container); states are applied to the minions 5. Tasks return applied states’ results to the Jenkins node 6. When all tasks have stopped, ASG instance gets terminated 7. Results are analyzed
  28. 28. 1. SIT requests AutoScaling Group to provision a new instance 2. Instance provisions & SIT discovers it 3. ASG instance gets registered into ECS cluster 4. ECS Tasks begin (runs Docker container); states are applied to the minions 5. Tasks return applied states’ results to the Jenkins node 6. When all tasks have stopped, ASG instance gets terminated 7. Results are analyzed INTEGRATION TEST ASGJENKINS NODE Run SIT
  29. 29. 1. SIT requests AutoScaling Group to provision a new instance 2. Instance provisions & SIT discovers it 3. ASG instance gets registered into ECS cluster 4. ECS Tasks begin (runs Docker container); states are applied to the minions 5. Tasks return applied states’ results to the Jenkins node 6. When all tasks have stopped, ASG instance gets terminated 7. Results are analyzed INTEGRATION TEST ASGJENKINS NODE ASG INSTANCE Run SIT
  30. 30. 1. SIT requests AutoScaling Group to provision a new instance 2. Instance provisions & SIT discovers it 3. ASG instance gets registered into ECS cluster 4. ECS Tasks begin (runs Docker container); states are applied to the minions 5. Tasks return applied states’ results to the Jenkins node 6. When all tasks have stopped, ASG instance gets terminated 7. Results are analyzed INTEGRATION TEST Run SIT JENKINS NODE ECS CLUSTER
  31. 31. 1. SIT requests AutoScaling Group to provision a new instance 2. Instance provisions & SIT discovers it 3. ASG instance gets registered into ECS cluster 4. ECS Tasks begin (runs Docker container); states are applied to the minions 5. Tasks return applied states’ results to the Jenkins node 6. When all tasks have stopped, ASG instance gets terminated 7. Results are analyzed INTEGRATION TEST Run SIT JENKINS NODE ECS CLUSTER
  32. 32. 1. SIT requests AutoScaling Group to provision a new instance 2. Instance provisions & SIT discovers it 3. ASG instance gets registered into ECS cluster 4. ECS Tasks begin (runs Docker container); states are applied to the minions 5. Tasks return applied states’ results to the Jenkins node 6. When all tasks have stopped, ASG instance gets terminated 7. Results are analyzed INTEGRATION TEST JENKINS NODE Run SIT ECS CLUSTER
  33. 33. 1. SIT requests AutoScaling Group to provision a new instance 2. Instance provisions & SIT discovers it 3. ASG instance gets registered into ECS cluster 4. ECS Tasks begin (runs Docker container); states are applied to the minions 5. Tasks return applied states’ results to the Jenkins node 6. When all tasks have stopped, ASG instance gets terminated 7. Results are analyzed INTEGRATION TEST JENKINS NODE Run SIT ECS CLUSTER
  34. 34. 1. SIT requests AutoScaling Group to provision a new instance 2. Instance provisions & SIT discovers it 3. ASG instance gets registered into ECS cluster 4. ECS Tasks begin (runs Docker container); states are applied to the minions 5. Tasks return applied states’ results to the Jenkins node 6. When all tasks have stopped, ASG instance gets terminated 7. Results are analyzed INTEGRATION TEST JENKINS NODE Run SIT
  35. 35. INITIATION AND TEARDOWN • SED to configure Jenkins node master back to easily editable state • Flush redis of data • Remove Salt-keys • Stop redis service • Stop Salt-master Pre-SIT Tasks Teardown SITInitiate SIT Run SIT
  36. 36. VPC Security Groups EC2 AutoScaling Group Launch Configuration AWS INFRASTRUCTURE Instance Profile Policy Role ECS Cluster CloudFormation
  37. 37. JENKINS NODES Results ASG • Time saved drastically • Maintenance • Open-source generic SIT tool DandB/Salt-Integration-Testing
  38. 38. Competitors ?
  39. 39. TOPIC Body text THANKS TO: Jeff Stibel Aaron Stibel Konstantin Antselovich DevOps Team Creative Team
  40. 40. TOPIC Body text THANK YOU (you may now stand)
  41. 41. Use the SaltConf16 event app to provide feedback for this presentation. (we’re all ears) QUESTIONS AND FEEDBACK

×