SlideShare a Scribd company logo
1 of 38
Download to read offline
Design Review Best Practices 
Velocity Barcelona 2014 
Mandi Walls 
Consulting Director, EMEA, Chef 
mandi@getchef.com
whoami 
• Mandi Walls 
• Professional Services for Chef 
• @lnxchk
Why Design Reviews? 
• Your opportunity to get a look at an incoming project 
• Hopefully early enough to have influence! 
Talk to development before it’s too late to make changes to support Ops
What You Want 
• Maximize the conversation 
• Get a good feel for important details 
• Don’t get too in the weeds, focus on what alleviates the most 
headaches
Set Your Goals 
• Initial deployment of the application 
• Runtime management and day-to-day operations 
• Releases and upgrades 
• Handling Outages
Have A Checklist 
• Combination of global requirements and needs specific to the project 
• Doesn’t have to be exhaustive, or hit everything the first time 
• Sample: http://bit.ly/lnxchk_reviewcklist 
• Sets goals by layer 
• You can also set goals by activity 
• It’s a living document 
• Make changes as you learn, launch more projects
Some Application 
FE 
Web Tier 
LB 
App Tier 
External 
Services 
Data Tier BI
Investigating Layer by Layer 
• Different components will require different focus! 
• Some topics will hit all layers
Frontend Entry Point 
• To look at during deploy 
• Gathering all FQDNs 
• Determining SSL Requirements 
• Looking at Geographic Topologies
FE Gotchas: Rewrite Rules, Divided Farms 
• Sophisticated load balancing equipment allow for lots of cleverness 
• Documentation of intent is key 
• Later migrations, requirements changes are more difficult 
/foo 
/bar
Web Tier 
• Server type and version 
• Access methods to other tiers 
• Caching? 
• Locations 
• Modules and their configurations
Web Insanity: Custom HTTP Servers 
https://www.flickr.com/photos/16210667@N02/13973604274
Application Tier 
• Basics: server type and version, port requirements, protocols 
• Outbound connections to outside services 
• User affinity 
• Necessary libraries and their release cycles
App Tier Gotchas 
• Things that affect scaling, replacement of hosts 
• Whitelisting! Why!? 
• What changes require restarts? 
• New FE connections? New BE connections to cache? To DB?
Data Tier – Our Data 
• In multi-location topologies, where is the master? 
• Is there whitelisting or other app authorization? 
• What are compliance/reporting requirements? 
• SOX? HIPAA? PCI? 
• Will schemas be versioned?
Doing Horrible Things to Data 
• Premature optimization can hamstring a project 
• Are the data tiers susceptible to queuing? 
• Management of data should be automated
Data Tier – Imports and ETL 
• Schedules 
• Measuring successful load 
• Contact information for upstream provider 
• Storage locations and intended size
Operationalizing ETL 
• Some of the fiddliest fiddly bits 
https://www.flickr.com/photos/joming/2182022195
Data Tier – Exports, Reporting, BI 
• Packaging 
• Scheduling 
• Live replica versus full dump and export 
• Should have no impact on production operation of the datastore!
External APIs - Consumer 
• Relying on services you don’t own is tricky business 
• Account name and owner 
• SLA of service provider 
• Contact info for outages
Graceful Degradation 
• What does the App do if the external service is unavailable? 
• Shouldn’t hang 
• Should log some message 
• Should have some reasonable timeout, plus a hold down 
• Users shouldn’t notice
Red Button 
• Can you completely disconnect the code from a broken service? 
• Who can make the call? 
https://www.flickr.com/photos/bluetsunami/535781178
Gotchas for All Tiers 
• Host firewalls 
• Application user accounts – in the central service? 
• Logging policy 
• Metrics collection 
• Backups 
• OS Versions
Alerts 
• A stacktrace is not an alertable event 
• Log hygiene is important – what’s in the prod logs needs to be useful 
• If it’s worth writing to disk, someone should know what to do with it 
2014-06-01:16:44:00:UTC ERROR: $$$$$$$$ 
2014-06-01:16:44:07:UTC ERROR: Host Not Found
Releases 
• Schedule – Over night? Weekends? Rolling continuous deploy? 
• Getting operations requirements into Dev cycle 
• Security requirements 
• Upgrades and patches 
• Service restarts 
• Graceful horizontal restarts 
• Full-zone downtime
Nightmare: Attaching Prod App to Dev DB 
• All artifacts should ship production ready 
• Localized configurations ready in advance 
• Lean on config management tools for other environments 
https://www.flickr.com/photos/garon/121923087
Software Library 
• Set organizational standards 
• Rely on your OS vendors when possible 
• Pre-check that all dependencies are available in all repos for all 
environments 
• Suppress the urge to build special packages for absolutely 
everything 
• Streamlined, repeatable, reliable installs
Spelunking Through History 
• Keeping up with OS releases makes later scaling, replacement of 
dead hosts smoother 
• Always be current or no more than one release back in all envs
Performance and Tuning 
• Hard to be concise with new projects 
• Performance regression should be part of the testing process! 
• Know in advance what tunables exist, what their indicators are 
• Heap 
• GC 
• Network stacks 
• In-memory caching
Outage Planning 
• Known failure patterns and indicators 
• Who is oncall from Dev? 
• Who is oncall from Product or other decision-making stakeholders?
The Dark Art of Healthchecks 
• At some point, there should be a check that works back through the 
whole stack for status 
• These can be expensive 
• You have to know, when you hit the FE, that it is able to serve all the 
components of the app
Static Failover Pages 
• Mechanism for not serving up 500s or blanks during downtime 
• Set to non-cachable, show a picture, give a better experience
Other Team Info 
• Know who’s who 
• Dev teams should have some sort of oncall in case of problems Ops 
can’t fix 
• Intake process for bug reports, feedback, getting Ops issues fixed 
• Like those crazy log messages! 
• Get invited to team meetings! Go occasionally!
Success Metrics 
• What will success actually look like after launch? 
• Is the product team watching a particular set of KPIs? 
• How will the data for success be collected 
• BI, beacons, third party collectors
Things That are Challenging to Find in Review 
• Bottlenecks 
• That flat table-top graph isn’t a good thing 
• Impact of third-party inclusions 
• Rigor of the integration testing environment 
• What parts of the system users are really going to use 
• People might like something you thought was going to be minor
Learn From Experience 
• Over time, the infrastructure build out will get easier 
• Operations teams develop their own heuristics for difficulty 
• Always be learning and improving 
• Give everyone a chance to learn and improve too 
• Create standards so you don’t have to investigate all dark corners 
• Update your checklist periodically
Thanks!
Design Reviews for Operations - Velocity Europe 2014

More Related Content

What's hot

Change management in hybrid landscapes
Change management in hybrid landscapesChange management in hybrid landscapes
Change management in hybrid landscapesChris Kernaghan
 
VMworld 2013: Examining vSphere Design Through a Design Scenario
VMworld 2013: Examining vSphere Design Through a Design Scenario VMworld 2013: Examining vSphere Design Through a Design Scenario
VMworld 2013: Examining vSphere Design Through a Design Scenario VMworld
 
Pain points of agile development
Pain points of agile developmentPain points of agile development
Pain points of agile developmentPerforce
 
Integration strategies best practices- Mulesoft meetup April 2018
Integration strategies   best practices- Mulesoft meetup April 2018Integration strategies   best practices- Mulesoft meetup April 2018
Integration strategies best practices- Mulesoft meetup April 2018Rohan Rasane
 
Operating a High Velocity Large Organization with Spring Cloud Microservices
Operating a High Velocity Large Organization with Spring Cloud MicroservicesOperating a High Velocity Large Organization with Spring Cloud Microservices
Operating a High Velocity Large Organization with Spring Cloud MicroservicesNoriaki Tatsumi
 
Success recipe for new IT projects-Agile way. Fail Fast, Fail Early
Success recipe for new IT projects-Agile way. Fail Fast, Fail EarlySuccess recipe for new IT projects-Agile way. Fail Fast, Fail Early
Success recipe for new IT projects-Agile way. Fail Fast, Fail EarlyJoseph Vargheese PMP CSM CSP
 
Application Performance Management
Application Performance ManagementApplication Performance Management
Application Performance ManagementNoriaki Tatsumi
 
Agile Secure Cloud Application Development Management
Agile Secure Cloud Application Development ManagementAgile Secure Cloud Application Development Management
Agile Secure Cloud Application Development ManagementAdam Getchell
 
SAP TechEd 2013 session Tec118 managing your-environment
SAP TechEd 2013 session Tec118 managing your-environmentSAP TechEd 2013 session Tec118 managing your-environment
SAP TechEd 2013 session Tec118 managing your-environmentChris Kernaghan
 
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good ServerICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good ServerSerdar Basegmez
 
DevOps+Data: Working with Source Control
DevOps+Data: Working with Source ControlDevOps+Data: Working with Source Control
DevOps+Data: Working with Source ControlEd Leighton-Dick
 
How Mature is Your Infrastructure?
How Mature is Your Infrastructure?How Mature is Your Infrastructure?
How Mature is Your Infrastructure?Gary Stafford
 
SharePoint and the Lean Enterprise
SharePoint and the Lean EnterpriseSharePoint and the Lean Enterprise
SharePoint and the Lean EnterpriseDave Healey
 
2015 Mastering SAP Tech - Enterprise Mobility - Testing Lessons Learned
2015 Mastering SAP Tech - Enterprise Mobility - Testing Lessons Learned2015 Mastering SAP Tech - Enterprise Mobility - Testing Lessons Learned
2015 Mastering SAP Tech - Enterprise Mobility - Testing Lessons LearnedEneko Jon Bilbao
 
Challenges and best practices of database continuous delivery
Challenges and best practices of database continuous deliveryChallenges and best practices of database continuous delivery
Challenges and best practices of database continuous deliveryDBmaestro - Database DevOps
 
Go Faster with Lightning Process Builder
Go Faster with Lightning Process BuilderGo Faster with Lightning Process Builder
Go Faster with Lightning Process BuilderSalesforce Developers
 

What's hot (20)

Change management in hybrid landscapes
Change management in hybrid landscapesChange management in hybrid landscapes
Change management in hybrid landscapes
 
VMworld 2013: Examining vSphere Design Through a Design Scenario
VMworld 2013: Examining vSphere Design Through a Design Scenario VMworld 2013: Examining vSphere Design Through a Design Scenario
VMworld 2013: Examining vSphere Design Through a Design Scenario
 
Pain points of agile development
Pain points of agile developmentPain points of agile development
Pain points of agile development
 
Extreme Makeover OnBase Edition
Extreme Makeover OnBase EditionExtreme Makeover OnBase Edition
Extreme Makeover OnBase Edition
 
Integration strategies best practices- Mulesoft meetup April 2018
Integration strategies   best practices- Mulesoft meetup April 2018Integration strategies   best practices- Mulesoft meetup April 2018
Integration strategies best practices- Mulesoft meetup April 2018
 
Operating a High Velocity Large Organization with Spring Cloud Microservices
Operating a High Velocity Large Organization with Spring Cloud MicroservicesOperating a High Velocity Large Organization with Spring Cloud Microservices
Operating a High Velocity Large Organization with Spring Cloud Microservices
 
Success recipe for new IT projects-Agile way. Fail Fast, Fail Early
Success recipe for new IT projects-Agile way. Fail Fast, Fail EarlySuccess recipe for new IT projects-Agile way. Fail Fast, Fail Early
Success recipe for new IT projects-Agile way. Fail Fast, Fail Early
 
Application Performance Management
Application Performance ManagementApplication Performance Management
Application Performance Management
 
Agile Secure Cloud Application Development Management
Agile Secure Cloud Application Development ManagementAgile Secure Cloud Application Development Management
Agile Secure Cloud Application Development Management
 
SAP TechEd 2013 session Tec118 managing your-environment
SAP TechEd 2013 session Tec118 managing your-environmentSAP TechEd 2013 session Tec118 managing your-environment
SAP TechEd 2013 session Tec118 managing your-environment
 
Resolving problems & high availability
Resolving problems & high availabilityResolving problems & high availability
Resolving problems & high availability
 
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good ServerICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
ICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server
 
DevOps for Database webinar
DevOps for Database webinarDevOps for Database webinar
DevOps for Database webinar
 
DevOps+Data: Working with Source Control
DevOps+Data: Working with Source ControlDevOps+Data: Working with Source Control
DevOps+Data: Working with Source Control
 
How Mature is Your Infrastructure?
How Mature is Your Infrastructure?How Mature is Your Infrastructure?
How Mature is Your Infrastructure?
 
SharePoint and the Lean Enterprise
SharePoint and the Lean EnterpriseSharePoint and the Lean Enterprise
SharePoint and the Lean Enterprise
 
2015 Mastering SAP Tech - Enterprise Mobility - Testing Lessons Learned
2015 Mastering SAP Tech - Enterprise Mobility - Testing Lessons Learned2015 Mastering SAP Tech - Enterprise Mobility - Testing Lessons Learned
2015 Mastering SAP Tech - Enterprise Mobility - Testing Lessons Learned
 
PM, Scrum and TFS - Ivan Marković
PM, Scrum and TFS - Ivan MarkovićPM, Scrum and TFS - Ivan Marković
PM, Scrum and TFS - Ivan Marković
 
Challenges and best practices of database continuous delivery
Challenges and best practices of database continuous deliveryChallenges and best practices of database continuous delivery
Challenges and best practices of database continuous delivery
 
Go Faster with Lightning Process Builder
Go Faster with Lightning Process BuilderGo Faster with Lightning Process Builder
Go Faster with Lightning Process Builder
 

Similar to Design Reviews for Operations - Velocity Europe 2014

When small problems become big problems
When small problems become big problemsWhen small problems become big problems
When small problems become big problemsAdrian Cole
 
Datapolis Guest Expert Presentation: Limitations of SharePoint Designer by Bj...
Datapolis Guest Expert Presentation: Limitations of SharePoint Designer by Bj...Datapolis Guest Expert Presentation: Limitations of SharePoint Designer by Bj...
Datapolis Guest Expert Presentation: Limitations of SharePoint Designer by Bj...Datapolis
 
158 - Product Management for Enterprise-Grade platforms
158 - Product Management for Enterprise-Grade platforms 158 - Product Management for Enterprise-Grade platforms
158 - Product Management for Enterprise-Grade platforms ProductCamp Boston
 
Project Sherpa: How RightScale Went All in on Docker
Project Sherpa: How RightScale Went All in on DockerProject Sherpa: How RightScale Went All in on Docker
Project Sherpa: How RightScale Went All in on DockerRightScale
 
Patching is Your Friend in the New World Order of EPM and ERP Cloud
Patching is Your Friend in the New World Order of EPM and ERP CloudPatching is Your Friend in the New World Order of EPM and ERP Cloud
Patching is Your Friend in the New World Order of EPM and ERP CloudDatavail
 
Lifecycle Management with SharePoint Apps and Solutions
Lifecycle Management with SharePoint Apps and SolutionsLifecycle Management with SharePoint Apps and Solutions
Lifecycle Management with SharePoint Apps and SolutionsSPC Adriatics
 
Building enterprise platforms - off the beaten path - SharePoint User Group U...
Building enterprise platforms - off the beaten path - SharePoint User Group U...Building enterprise platforms - off the beaten path - SharePoint User Group U...
Building enterprise platforms - off the beaten path - SharePoint User Group U...Andy Talbot
 
Software development planning and essentials
Software development planning and essentialsSoftware development planning and essentials
Software development planning and essentialsRajesh P
 
Software development planning and essentials
Software development planning and essentialsSoftware development planning and essentials
Software development planning and essentialsRajesh P
 
Change Management in Hybrid landscapes 2017
Change Management in Hybrid landscapes 2017Change Management in Hybrid landscapes 2017
Change Management in Hybrid landscapes 2017Chris Kernaghan
 
ALM with TFS: From the Drawing Board to the Cloud
ALM with TFS: From the Drawing Board to the CloudALM with TFS: From the Drawing Board to the Cloud
ALM with TFS: From the Drawing Board to the CloudJeremy Likness
 
Why retail companies can't afford database downtime
Why retail companies can't afford database downtimeWhy retail companies can't afford database downtime
Why retail companies can't afford database downtimeDBmaestro - Database DevOps
 
Making the Transition from Suite to the Hub
Making the Transition from Suite to the HubMaking the Transition from Suite to the Hub
Making the Transition from Suite to the HubBlack Duck by Synopsys
 
Keeping in Touch -- Collaborative Technologies
Keeping in Touch -- Collaborative TechnologiesKeeping in Touch -- Collaborative Technologies
Keeping in Touch -- Collaborative TechnologiesIABC Houston
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting examplecorehard_by
 
Strategy vs. Tactical Testing: Actions for Today, Plans for Tomorrow​
Strategy vs. Tactical Testing: Actions for Today, Plans for Tomorrow​Strategy vs. Tactical Testing: Actions for Today, Plans for Tomorrow​
Strategy vs. Tactical Testing: Actions for Today, Plans for Tomorrow​Eggplant
 
Building SharePoint Enterprise Platforms - Off the beaten path
Building SharePoint Enterprise Platforms - Off the beaten pathBuilding SharePoint Enterprise Platforms - Off the beaten path
Building SharePoint Enterprise Platforms - Off the beaten pathAndy Talbot
 
cloud session uklug
cloud session uklugcloud session uklug
cloud session uklugdominion
 
Transforming Enterprise Teams to DevOps Workflows
Transforming Enterprise Teams to DevOps WorkflowsTransforming Enterprise Teams to DevOps Workflows
Transforming Enterprise Teams to DevOps WorkflowsMandi Walls
 

Similar to Design Reviews for Operations - Velocity Europe 2014 (20)

When small problems become big problems
When small problems become big problemsWhen small problems become big problems
When small problems become big problems
 
Datapolis Guest Expert Presentation: Limitations of SharePoint Designer by Bj...
Datapolis Guest Expert Presentation: Limitations of SharePoint Designer by Bj...Datapolis Guest Expert Presentation: Limitations of SharePoint Designer by Bj...
Datapolis Guest Expert Presentation: Limitations of SharePoint Designer by Bj...
 
158 - Product Management for Enterprise-Grade platforms
158 - Product Management for Enterprise-Grade platforms 158 - Product Management for Enterprise-Grade platforms
158 - Product Management for Enterprise-Grade platforms
 
Project Sherpa: How RightScale Went All in on Docker
Project Sherpa: How RightScale Went All in on DockerProject Sherpa: How RightScale Went All in on Docker
Project Sherpa: How RightScale Went All in on Docker
 
Patching is Your Friend in the New World Order of EPM and ERP Cloud
Patching is Your Friend in the New World Order of EPM and ERP CloudPatching is Your Friend in the New World Order of EPM and ERP Cloud
Patching is Your Friend in the New World Order of EPM and ERP Cloud
 
Lifecycle Management with SharePoint Apps and Solutions
Lifecycle Management with SharePoint Apps and SolutionsLifecycle Management with SharePoint Apps and Solutions
Lifecycle Management with SharePoint Apps and Solutions
 
Building enterprise platforms - off the beaten path - SharePoint User Group U...
Building enterprise platforms - off the beaten path - SharePoint User Group U...Building enterprise platforms - off the beaten path - SharePoint User Group U...
Building enterprise platforms - off the beaten path - SharePoint User Group U...
 
Software development planning and essentials
Software development planning and essentialsSoftware development planning and essentials
Software development planning and essentials
 
Software development planning and essentials
Software development planning and essentialsSoftware development planning and essentials
Software development planning and essentials
 
Software development life cycle
Software development life cycleSoftware development life cycle
Software development life cycle
 
Change Management in Hybrid landscapes 2017
Change Management in Hybrid landscapes 2017Change Management in Hybrid landscapes 2017
Change Management in Hybrid landscapes 2017
 
ALM with TFS: From the Drawing Board to the Cloud
ALM with TFS: From the Drawing Board to the CloudALM with TFS: From the Drawing Board to the Cloud
ALM with TFS: From the Drawing Board to the Cloud
 
Why retail companies can't afford database downtime
Why retail companies can't afford database downtimeWhy retail companies can't afford database downtime
Why retail companies can't afford database downtime
 
Making the Transition from Suite to the Hub
Making the Transition from Suite to the HubMaking the Transition from Suite to the Hub
Making the Transition from Suite to the Hub
 
Keeping in Touch -- Collaborative Technologies
Keeping in Touch -- Collaborative TechnologiesKeeping in Touch -- Collaborative Technologies
Keeping in Touch -- Collaborative Technologies
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting example
 
Strategy vs. Tactical Testing: Actions for Today, Plans for Tomorrow​
Strategy vs. Tactical Testing: Actions for Today, Plans for Tomorrow​Strategy vs. Tactical Testing: Actions for Today, Plans for Tomorrow​
Strategy vs. Tactical Testing: Actions for Today, Plans for Tomorrow​
 
Building SharePoint Enterprise Platforms - Off the beaten path
Building SharePoint Enterprise Platforms - Off the beaten pathBuilding SharePoint Enterprise Platforms - Off the beaten path
Building SharePoint Enterprise Platforms - Off the beaten path
 
cloud session uklug
cloud session uklugcloud session uklug
cloud session uklug
 
Transforming Enterprise Teams to DevOps Workflows
Transforming Enterprise Teams to DevOps WorkflowsTransforming Enterprise Teams to DevOps Workflows
Transforming Enterprise Teams to DevOps Workflows
 

More from Mandi Walls

DOD Raleigh Gamedays with Chaos Engineering.pdf
DOD Raleigh Gamedays with Chaos Engineering.pdfDOD Raleigh Gamedays with Chaos Engineering.pdf
DOD Raleigh Gamedays with Chaos Engineering.pdfMandi Walls
 
Addo reducing trauma in organizations with SLOs and chaos engineering
Addo  reducing trauma in organizations with SLOs and chaos engineeringAddo  reducing trauma in organizations with SLOs and chaos engineering
Addo reducing trauma in organizations with SLOs and chaos engineeringMandi Walls
 
Full Service Ownership
Full Service OwnershipFull Service Ownership
Full Service OwnershipMandi Walls
 
PagerDuty: Best Practices for On Call Teams
PagerDuty: Best Practices for On Call TeamsPagerDuty: Best Practices for On Call Teams
PagerDuty: Best Practices for On Call TeamsMandi Walls
 
InSpec at DevOps ATL Meetup January 22, 2020
InSpec at DevOps ATL Meetup January 22, 2020InSpec at DevOps ATL Meetup January 22, 2020
InSpec at DevOps ATL Meetup January 22, 2020Mandi Walls
 
Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019Mandi Walls
 
Using Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure SecurityUsing Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure SecurityMandi Walls
 
Adding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17xAdding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17xMandi Walls
 
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Mandi Walls
 
BuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec WorkshopBuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec WorkshopMandi Walls
 
InSpec Workshop at Velocity London 2018
InSpec Workshop at Velocity London 2018InSpec Workshop at Velocity London 2018
InSpec Workshop at Velocity London 2018Mandi Walls
 
DevOpsDays InSpec Workshop
DevOpsDays InSpec WorkshopDevOpsDays InSpec Workshop
DevOpsDays InSpec WorkshopMandi Walls
 
Adding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpecAdding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpecMandi Walls
 
InSpec - June 2018 at Open28.be
InSpec - June 2018 at Open28.beInSpec - June 2018 at Open28.be
InSpec - June 2018 at Open28.beMandi Walls
 
habitat at docker bud
habitat at docker budhabitat at docker bud
habitat at docker budMandi Walls
 
Ingite Slides for InSpec
Ingite Slides for InSpecIngite Slides for InSpec
Ingite Slides for InSpecMandi Walls
 
Habitat at LinuxLab IT
Habitat at LinuxLab ITHabitat at LinuxLab IT
Habitat at LinuxLab ITMandi Walls
 
InSpec Workshop DevSecCon 2017
InSpec Workshop DevSecCon 2017InSpec Workshop DevSecCon 2017
InSpec Workshop DevSecCon 2017Mandi Walls
 
Habitat Workshop at Velocity London 2017
Habitat Workshop at Velocity London 2017Habitat Workshop at Velocity London 2017
Habitat Workshop at Velocity London 2017Mandi Walls
 
InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017Mandi Walls
 

More from Mandi Walls (20)

DOD Raleigh Gamedays with Chaos Engineering.pdf
DOD Raleigh Gamedays with Chaos Engineering.pdfDOD Raleigh Gamedays with Chaos Engineering.pdf
DOD Raleigh Gamedays with Chaos Engineering.pdf
 
Addo reducing trauma in organizations with SLOs and chaos engineering
Addo  reducing trauma in organizations with SLOs and chaos engineeringAddo  reducing trauma in organizations with SLOs and chaos engineering
Addo reducing trauma in organizations with SLOs and chaos engineering
 
Full Service Ownership
Full Service OwnershipFull Service Ownership
Full Service Ownership
 
PagerDuty: Best Practices for On Call Teams
PagerDuty: Best Practices for On Call TeamsPagerDuty: Best Practices for On Call Teams
PagerDuty: Best Practices for On Call Teams
 
InSpec at DevOps ATL Meetup January 22, 2020
InSpec at DevOps ATL Meetup January 22, 2020InSpec at DevOps ATL Meetup January 22, 2020
InSpec at DevOps ATL Meetup January 22, 2020
 
Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019
 
Using Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure SecurityUsing Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure Security
 
Adding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17xAdding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17x
 
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
 
BuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec WorkshopBuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec Workshop
 
InSpec Workshop at Velocity London 2018
InSpec Workshop at Velocity London 2018InSpec Workshop at Velocity London 2018
InSpec Workshop at Velocity London 2018
 
DevOpsDays InSpec Workshop
DevOpsDays InSpec WorkshopDevOpsDays InSpec Workshop
DevOpsDays InSpec Workshop
 
Adding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpecAdding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpec
 
InSpec - June 2018 at Open28.be
InSpec - June 2018 at Open28.beInSpec - June 2018 at Open28.be
InSpec - June 2018 at Open28.be
 
habitat at docker bud
habitat at docker budhabitat at docker bud
habitat at docker bud
 
Ingite Slides for InSpec
Ingite Slides for InSpecIngite Slides for InSpec
Ingite Slides for InSpec
 
Habitat at LinuxLab IT
Habitat at LinuxLab ITHabitat at LinuxLab IT
Habitat at LinuxLab IT
 
InSpec Workshop DevSecCon 2017
InSpec Workshop DevSecCon 2017InSpec Workshop DevSecCon 2017
InSpec Workshop DevSecCon 2017
 
Habitat Workshop at Velocity London 2017
Habitat Workshop at Velocity London 2017Habitat Workshop at Velocity London 2017
Habitat Workshop at Velocity London 2017
 
InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017
 

Recently uploaded

20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_planJamie (Taka) Wang
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?SANGHEE SHIN
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 

Recently uploaded (20)

20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_plan
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 

Design Reviews for Operations - Velocity Europe 2014

  • 1. Design Review Best Practices Velocity Barcelona 2014 Mandi Walls Consulting Director, EMEA, Chef mandi@getchef.com
  • 2. whoami • Mandi Walls • Professional Services for Chef • @lnxchk
  • 3. Why Design Reviews? • Your opportunity to get a look at an incoming project • Hopefully early enough to have influence! Talk to development before it’s too late to make changes to support Ops
  • 4. What You Want • Maximize the conversation • Get a good feel for important details • Don’t get too in the weeds, focus on what alleviates the most headaches
  • 5. Set Your Goals • Initial deployment of the application • Runtime management and day-to-day operations • Releases and upgrades • Handling Outages
  • 6. Have A Checklist • Combination of global requirements and needs specific to the project • Doesn’t have to be exhaustive, or hit everything the first time • Sample: http://bit.ly/lnxchk_reviewcklist • Sets goals by layer • You can also set goals by activity • It’s a living document • Make changes as you learn, launch more projects
  • 7. Some Application FE Web Tier LB App Tier External Services Data Tier BI
  • 8. Investigating Layer by Layer • Different components will require different focus! • Some topics will hit all layers
  • 9. Frontend Entry Point • To look at during deploy • Gathering all FQDNs • Determining SSL Requirements • Looking at Geographic Topologies
  • 10. FE Gotchas: Rewrite Rules, Divided Farms • Sophisticated load balancing equipment allow for lots of cleverness • Documentation of intent is key • Later migrations, requirements changes are more difficult /foo /bar
  • 11. Web Tier • Server type and version • Access methods to other tiers • Caching? • Locations • Modules and their configurations
  • 12. Web Insanity: Custom HTTP Servers https://www.flickr.com/photos/16210667@N02/13973604274
  • 13. Application Tier • Basics: server type and version, port requirements, protocols • Outbound connections to outside services • User affinity • Necessary libraries and their release cycles
  • 14. App Tier Gotchas • Things that affect scaling, replacement of hosts • Whitelisting! Why!? • What changes require restarts? • New FE connections? New BE connections to cache? To DB?
  • 15. Data Tier – Our Data • In multi-location topologies, where is the master? • Is there whitelisting or other app authorization? • What are compliance/reporting requirements? • SOX? HIPAA? PCI? • Will schemas be versioned?
  • 16. Doing Horrible Things to Data • Premature optimization can hamstring a project • Are the data tiers susceptible to queuing? • Management of data should be automated
  • 17. Data Tier – Imports and ETL • Schedules • Measuring successful load • Contact information for upstream provider • Storage locations and intended size
  • 18. Operationalizing ETL • Some of the fiddliest fiddly bits https://www.flickr.com/photos/joming/2182022195
  • 19. Data Tier – Exports, Reporting, BI • Packaging • Scheduling • Live replica versus full dump and export • Should have no impact on production operation of the datastore!
  • 20. External APIs - Consumer • Relying on services you don’t own is tricky business • Account name and owner • SLA of service provider • Contact info for outages
  • 21. Graceful Degradation • What does the App do if the external service is unavailable? • Shouldn’t hang • Should log some message • Should have some reasonable timeout, plus a hold down • Users shouldn’t notice
  • 22. Red Button • Can you completely disconnect the code from a broken service? • Who can make the call? https://www.flickr.com/photos/bluetsunami/535781178
  • 23. Gotchas for All Tiers • Host firewalls • Application user accounts – in the central service? • Logging policy • Metrics collection • Backups • OS Versions
  • 24. Alerts • A stacktrace is not an alertable event • Log hygiene is important – what’s in the prod logs needs to be useful • If it’s worth writing to disk, someone should know what to do with it 2014-06-01:16:44:00:UTC ERROR: $$$$$$$$ 2014-06-01:16:44:07:UTC ERROR: Host Not Found
  • 25. Releases • Schedule – Over night? Weekends? Rolling continuous deploy? • Getting operations requirements into Dev cycle • Security requirements • Upgrades and patches • Service restarts • Graceful horizontal restarts • Full-zone downtime
  • 26. Nightmare: Attaching Prod App to Dev DB • All artifacts should ship production ready • Localized configurations ready in advance • Lean on config management tools for other environments https://www.flickr.com/photos/garon/121923087
  • 27. Software Library • Set organizational standards • Rely on your OS vendors when possible • Pre-check that all dependencies are available in all repos for all environments • Suppress the urge to build special packages for absolutely everything • Streamlined, repeatable, reliable installs
  • 28. Spelunking Through History • Keeping up with OS releases makes later scaling, replacement of dead hosts smoother • Always be current or no more than one release back in all envs
  • 29. Performance and Tuning • Hard to be concise with new projects • Performance regression should be part of the testing process! • Know in advance what tunables exist, what their indicators are • Heap • GC • Network stacks • In-memory caching
  • 30. Outage Planning • Known failure patterns and indicators • Who is oncall from Dev? • Who is oncall from Product or other decision-making stakeholders?
  • 31. The Dark Art of Healthchecks • At some point, there should be a check that works back through the whole stack for status • These can be expensive • You have to know, when you hit the FE, that it is able to serve all the components of the app
  • 32. Static Failover Pages • Mechanism for not serving up 500s or blanks during downtime • Set to non-cachable, show a picture, give a better experience
  • 33. Other Team Info • Know who’s who • Dev teams should have some sort of oncall in case of problems Ops can’t fix • Intake process for bug reports, feedback, getting Ops issues fixed • Like those crazy log messages! • Get invited to team meetings! Go occasionally!
  • 34. Success Metrics • What will success actually look like after launch? • Is the product team watching a particular set of KPIs? • How will the data for success be collected • BI, beacons, third party collectors
  • 35. Things That are Challenging to Find in Review • Bottlenecks • That flat table-top graph isn’t a good thing • Impact of third-party inclusions • Rigor of the integration testing environment • What parts of the system users are really going to use • People might like something you thought was going to be minor
  • 36. Learn From Experience • Over time, the infrastructure build out will get easier • Operations teams develop their own heuristics for difficulty • Always be learning and improving • Give everyone a chance to learn and improve too • Create standards so you don’t have to investigate all dark corners • Update your checklist periodically

Editor's Notes

  1. There are a lot of tools that promise to help with ETL. Many of them produce some kind of job script or war file or similar that has no logging, or predictable error reporting, or clean stop/start. They may dissolve into chaos if the data is a little bit wonky. Complex pipelines of data transformations are very hard to test, replicate, and debug.
  2. Red buttoning is one of the easiest ways to control your application when an upstream service is misbehaving. The button is there to completely bypass any code reliant on the service, and continue the application flow. These buttons should be easy to manipulate, maybe in a CMS, or in a configuration file that is periodically re-read by the application. The intent is to not have to restart your service because someone else is having an outage. Red button should be a manual process; automatic graceful degradation with hold down will take care of transient errors. A semi-permanent use of a red button is when the upstream provider has changed their API, and your application has not been prepared; red button the feature until the app is fixed and updated.
  3. The last thing you want to do with a running application in distress is dig around hoping you have the software used to build the farm originally. Keep application platforms updated; don’t let any environments get behind.
  4. We’ve learned a lot about post mortems, and talking about failure in general, over the past couple of years. A design review gives you the opportunity to revisit lessons learned from your systems history and make sure the learnings from those activities get incorporated into new systems.