Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jason Shawn, Senior Director of DevOps, Ellucian...
Who are we?
Jesse Gigler Jason Shawn
DevOps Engineer Sr. Director of DevOps
@jsin@jessegigler
Ellucian: Enabling Student Success
How we define DevOps
People working together with a common set of tools & goals
to achieve the best customer experience
Did we mention DevOps is culture?
Our DevOps mission statement
Continuous delivery
Source: https://github.com/red-gate/continuous-delivery-periodic-table
Prior to DevOps
Mostly lift-and-shift into AWS
Very little test coverage
Security scans ad-hoc
Sparse CI, no real CD proce...
Our DevOps toolchain
Jenkins – orchestration layer
• Amazon EC2 Plugin allows Jenkins to spin up slaves
dynamically as needed.
• Folder per pro...
Packer to create immutable AMIs
• Immutable AMIs ensure all tools and
components are included in the development
lifecycle...
Terraform for infrastructure as code
• Remote Amazon S3 state-file allows sharing of
resource values across modules and te...
Ansible for configuration management layer
• Ansible serves as standard format to write
and share server-level automation....
Automated tests and scans
• Unit, smoke, and functional tests
ensure environment is operating as
expected.
• ServerSpec te...
Challenges and pain points
Cross-account deployments
AWS resource tag management
• Lots and lots of teams.
• One central “billing” account.
• Defined “required” tags – but how...
Some select child AWS resources do not support
a “tag flow down”.
Requirement: Automate a way to flow the
resource tags do...
AWS resource soft limits
Each resource class in each region of all of our accounts has a soft limit but:
• How can we moni...
AWS scheduling for cost optimizations
"ScheduledActionUp": {
"Type": "AWS::AutoScaling::ScheduledAction",
"Condition": "DE...
Emerging patterns
Blue/Green deployment
Self-healing CI/CD environment
• Deployment and configuration of
Jenkins pipeline is fully automated.
• Can seamlessly dep...
DevSecOps
Thank you!
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Upcoming SlideShare
Loading in …5
×

Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016

4,225 views

Published on

Ellucian has been migrating its entire organization from a myriad of software delivery mechanisms, many of them manual, to a highly automated and advanced suite of DevOps tools. Using tools such as Jenkins, Terraform, and Ansible along with native AWS tooling, we have built a highly customized DevOps pipeline on top of the AWS platform. In this talk, we go over some of the challenges we have faced and also discuss our thoughts on the evolution of DevOps and the emerging patterns of managing AWS-based environments.

Published in: Technology
  • Be the first to comment

Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Jason Shawn, Senior Director of DevOps, Ellucian Jesse Gigler, DevOps Engineer, Ellucian June 21, 2016 Enterprise DevOps at Scale with AWS
  2. 2. Who are we? Jesse Gigler Jason Shawn DevOps Engineer Sr. Director of DevOps @jsin@jessegigler
  3. 3. Ellucian: Enabling Student Success
  4. 4. How we define DevOps People working together with a common set of tools & goals to achieve the best customer experience
  5. 5. Did we mention DevOps is culture?
  6. 6. Our DevOps mission statement
  7. 7. Continuous delivery Source: https://github.com/red-gate/continuous-delivery-periodic-table
  8. 8. Prior to DevOps Mostly lift-and-shift into AWS Very little test coverage Security scans ad-hoc Sparse CI, no real CD processes New node deployments took man-weeks Comparison Current state Refactoring into cloud-native apps Improved test coverage Security scans in DevOps pipeline 1500+ Jenkins jobs running daily New node deployments took ~4 hours automated
  9. 9. Our DevOps toolchain
  10. 10. Jenkins – orchestration layer • Amazon EC2 Plugin allows Jenkins to spin up slaves dynamically as needed. • Folder per product team. Product teams restricted to their folder based on AD Group. • CloudBees Folder Plus Plugin allows us to constrain projects/folders to specific slave pools. Separate slave pools for different instance profiles for assume-role access.
  11. 11. Packer to create immutable AMIs • Immutable AMIs ensure all tools and components are included in the development lifecycle. • AMIs can be spun up in a different account to audit software and licensing without direct access to the product environment. • New AMIs are rolled out by updating the launch configuration in terraform.
  12. 12. Terraform for infrastructure as code • Remote Amazon S3 state-file allows sharing of resource values across modules and teams. • Count, split, element design pattern to scale resources. • Jenkins serves as a middleware wrapper to handle dynamic variables and configuration across AWS accounts. • Allows us to remain “cloud agnostic”.
  13. 13. Ansible for configuration management layer • Ansible serves as standard format to write and share server-level automation. • Playbooks are pushed to S3 from Jenkins, then downloaded from S3 and executed in local mode in user_data to provision the instance. • PowerShell Desired State Configuration for Windows platforms to adhere to the same Ansible principles (push, pull, local).
  14. 14. Automated tests and scans • Unit, smoke, and functional tests ensure environment is operating as expected. • ServerSpec tests to validate infrastructure is configured properly. • Results proxied through bastion server back to Jenkins for reporting and tracking.
  15. 15. Challenges and pain points
  16. 16. Cross-account deployments
  17. 17. AWS resource tag management • Lots and lots of teams. • One central “billing” account. • Defined “required” tags – but how do you enforce this?
  18. 18. Some select child AWS resources do not support a “tag flow down”. Requirement: Automate a way to flow the resource tags down from the parents to the children. Answer: Leverage AWS Lambda to regularly scan the environment and copy tags from parent resources to the appropriate child resources. Resource tag “flow down”
  19. 19. AWS resource soft limits Each resource class in each region of all of our accounts has a soft limit but: • How can we monitor our soft limits? • How can we automatically request an increase? • How can we ensure production isn’t affected?
  20. 20. AWS scheduling for cost optimizations "ScheduledActionUp": { "Type": "AWS::AutoScaling::ScheduledAction", "Condition": "DEV", "Properties": { "AutoScalingGroupName": { "Ref": ”us-east-1-DevApp" }, "MaxSize": "1", "MinSize": "1", "DesiredCapacity": "1", "Recurrence": "0 12 * * 1-5" } }, "ScheduledActionDown": { "Type": "AWS::AutoScaling::ScheduledAction", "Condition": "DEV", "Properties": { "AutoScalingGroupName": { "Ref": "us-east-1-DevApp" }, "MaxSize": "0", "MinSize": "0", "DesiredCapacity": "0", "Recurrence": "0 22 * * 2-6" } }
  21. 21. Emerging patterns
  22. 22. Blue/Green deployment
  23. 23. Self-healing CI/CD environment • Deployment and configuration of Jenkins pipeline is fully automated. • Can seamlessly deploy to new regions or recreate an existing environment. EBS volume snapshot is taken and reattached after recreating. • Self-healing - Jenkins will recreate itself based on certain Amazon CloudWatch alarms. Events
  24. 24. DevSecOps
  25. 25. Thank you!

×