Implementing DevOps AutomationBest Practices and Common Mistakes

Implementing DevOps Automation
Best Practices and Common Mistakes
Given by Derek C. Ashmore
Agile+DevOps East 2023
November 8, 2023
©2023 Derek C. Ashmore, All Rights Reserved 1

Who am I?
• Professional Geek
since 1987
• AWS since 2010
• Azure since 2017
• Specialties
• Application
Transformation
• Infrastructure
Automation
• Yes – I still code!

Discussion Resources
• This slide deck
– https://www.slideshare.net/derekashmore/presentations
• Slide deck has hyper-links!
– Don’t bother writing down URLs
• I take questions
– For those online, contact me on LinkedIn

Agenda
Intro and
Level Set
DevOps Best
Practices
DevOps
Management
Summary /
Q&A

DevOps Managed Infrastructure
• 99+% Infrastructure as Code (IaC)
• Manual changes
– Increase errors
– Increase unwanted differences between
environments
– Increase admin workload
• Scripted/Coded changes
– Larger upfront cost, but…..
– Less busywork
– Leverage Others Work
– Decreases Errors
– Errors fixed in one place
– Eliminates unwanted differences
– Change history (with source control)

Tales from the Field
• Large Consumer Product Firm
– Rebuilds entire cloud footprint every
two weeks
• Large Fast-food Franchise
– Easy to add new business unit spokes
– Security / Guardrails built in
– Internet ingress/egress
– On premises network connectivity
• Large Financial Institution
– Mobile App Cloud and application
footprint
• blue/green capability
©2023 Derek C. Ashmore, All Rights Reserved

DevOps Automation Categories
➢Network / non-application specific infrastructure
▪ Virtual Networks/VPCs and subnets
▪ Route tables, Network peering
▪ Security groups / NSGs
➢Application infrastructure
▪ Relational databases
▪ Serverless constructs
➢Security privileges and policies
▪ IAM Roles and privilege grants
➢Virtual machine image production
▪ Produce machine images for teams to use
▪ Docker image production is similar conceptually

Agenda
Intro and
Level Set
DevOps Best
Practices
DevOps
Management
Summary /
Q&A

Discipline is Key
➢Discipline required differs per maturity level
➢Source Code Management
➢Source Code Structure
➢Deployment Management (CI/CD Pipelines)
➢Avoid Manual Changes
➢Testing Strategy

Automation Usage Evolution
• In the beginning
– Use Source Control
• As #Coders grow
– Feature branches
– CI/CD Pipelines
• As #Configurations grow
– Separate repo for modules
– Implement versioning
• Never use main/master!
• Further reading

Feature Branching
• DevOps Team Discipline is Key
• Feature Branches
– Never edit main/master directly!
– Update using Pull Requests
• Should live less than one day!
– Single targeted enhancement
– One developer only
– Long-lived branches prone to
merge conflicts
– Squash commits on merge
• Further reading

Code in Reusable Modules
➢Advantages are
▪ Small blast radius
▪ More easily tested
▪ Economies of scale
➢Example reusable modules
▪ Kubernetes Cluster
▪ Virtual Machine
▪ Virtual Networks and Subnets
▪ S3/Storage accounts
▪ Serverless services/functions
➢100+ Modules in all
▪ Used in 400+ pipelines
➢Tested in merge to master

CI/CD Pipelines
• Provides consistent runtime
environment
– Software version
– Cloud security policy
• Audit history / Admin security
• Pipeline approvals
– Force Plan/Dry-Run execution
– Force manual approval before
changing the environment

Manual Intervention Requirements
➢Some companies require manual intervention
▪ Often dictated by company policy
▪ Examples include
▪ Requiring DNS entries to be manually entered
▪ Separate group allocates security privileges
▪ On-premises connectivity
➢IaC depending on manual intervention cannot have automated tests
▪ Localize the manual intervention requirements

Accommodating Manual Processes

Infrastructure Code Testing
➢IaC is code!
▪ Housed in source control
▪ Often changed and released
▪ Needs testing like any other code
➢IaC change can have negative impact
▪ Environment outages
▪ End-user internet connectivity outage
▪ Application outages
▪ Testing team delayed for four days
➢Testing IaC can minimize negative impact

Infrastructure Code Testing Differences
➢IaC != Application Code
▪ IaC requires external resources (e.g. Cloud) to run
▪ In-process unit testing often not possible
➢Limited localized (in-process) testing
▪ Generally limited to syntax checks
▪ Terraform validation
▪ Ansible Dry Runs
▪ IDE syntax checks
➢Most testing is “integration” testing

Infrastructure Code Testing Challenges
➢Friction
▪ Harder to write/maintain
➢Dependencies
▪ Managed by other teams
➢Testing costs
▪ Use Sandbox – tear-down after tests
➢Manual intervention requirements
▪ Not possible to automate tests

Lack of Discipline Causes…
➢Unplanned Work
▪ Change due to automatic upgrades
▪ Unintended consequences
➢Accidental over-writing changes of others
▪ Merge conflicts
▪ Changes deployed from unmerged branches
➢Increased defect rate
➢Configuration Drift caused by manual change

Frequent Mistakes in the Field…
➢Lack of testing for common IaC code
▪ Testing for one use of common code is not sufficient!
▪ Often breaks other consumers of the common code
➢Creating a blast radius for IaC that’s too large
▪ Can’t make targeted changes without unintended consequences
➢Treating common IaC code as an enforcement mechanism
▪ Decouple policy enforcement and naming conventions
▪ Common IaC is a productivity enhancer only

Agenda
Intro and
Level Set
DevOps Best
Practices
DevOps
Management
Summary /
Q&A

Management is different too!
• Instead of
– Manual reviews/approvals
• Automate guardrails
• Automate testing
• Whitelist cloud services
• Consider continuous
delivery/deployment
– Capacity planning up front
• Monitor cost increases and
investigate
– Mandating policy changes by
edict
• Automate policy enforcement

Things that Don’t work
• Adding Manual
Approvals/Reviews
– Kills velocity and productivity
– Stops innovation
– Creates bottlenecks
• Forcing manual procedures
– Attempt to “expedite”
– Creates technical debt

Things that Work!
• Declare War on manual approvals
– Favor automated guardrails
– Automate oversight
• Decentralize Cloud Management
– Let app teams manage app infrastructure
• Don’t be a bottleneck
– Leave App teams to innovate
• Create a Service Catalog
– Automate whitelisting of services
– Create reasonable process for new services
• Legal reviews (HIPPA, GDPR, etc.)
• Fund automation
– You won’t make it manually!
• DevOps Team Discipline
– Automation needs SDLC just like applications
– Source management is key

Thank you!
• Derek Ashmore:
– Blog: www.derekashmore.com
– LinkedIn: www.linkedin.com/in/derekashmore
• Connect Invites from attendees welcome
– Twitter: https://twitter.com/Derek_Ashmore
– GitHub: https://github.com/Derek-Ashmore
– Book: http://dvtpress.com/
• Please fill out the evaluation form!

Implementing DevOps AutomationBest Practices and Common Mistakes

Recommended

Recommended

More Related Content

Similar to Implementing DevOps AutomationBest Practices and Common Mistakes

Similar to Implementing DevOps AutomationBest Practices and Common Mistakes (20)

Recently uploaded

Recently uploaded (20)