Agile Team Autonomy –
Don’t Just Give It Away,
Make Teams Earn It
©2019 CISQ
1
Dave Norton
Executive Director
Consortium for Information & Software Quality
david.norton@it-cisq.org
Two Basic Truths
©2019 CISQ 2
Things are more complex and the pace of change is relentless
Agenda
©2019 CISQ 3
• What are the drivers for automation
• How do we introduce more automation
• What role do standards play
Agenda
©2019 CISQ 4
• What are the drivers for automation
• How do we introduce more automation
• What role do standards play
Complex Technology Stack
©2019 CISQ 5
Multi-language,multi-layerArchitecture
EJB
PL/SQL
Oracle
SQL
Server
DB2
T/SQL
Hibernate
Spring
Struts
.NET
COBOL
IMS
Messaging
Sybase
• Code style & layout
• Expression complexity
• Code documentation
• Class or program design
• Basic coding standards
• Developer level
Unit Level1
Technology Stack
Java
Java
Java
Web
Services • Single language/technology layer
• Intra-technology architecture
• Intra-layer dependencies
• Inter-program invocation
• Security vulnerabilities
• Development team level
Technology Level2
 Integration quality
 Architectural compliance
 Risk propagation
 Application security
 Resiliency checks
 Transaction integrity
 Function point,
 Effort estimation
 Data access control
 SDK versioning
 Calibration across
technologies
 IT organization level
System Level3
JSP ASP.NETAPIs
Drive for Velocity
©2019 CISQ 6
Everyone wants faster time to market, but few
want to hear about the risks
Complex Toolchains
©2019 CISQ 7
• Production metrics, objects and feedback
• Requirements
• Business metrics
• Update release metrics
• Release plan, timing and business case
• Security policy and requirement
• Design of the software and
configuration
• Coding including code quality
and performance
• Software build and build
performance
• Release candidate
• Acceptance testing
• Regression testing
• Security and vulnerability analysis
• Performance
• Configuration testing
• Approval/preapprovals
• Package configuration
• Triggered releases
• Release staging and holding
• Infrastructure storage,
database and network
provisioning and
configuring
• Application provision
and configuration.
• Performance of IT infrastructure
• End-user response and experience
• Production metrics and statistics
• Application monitoring
Increasing Technical Debt
©2019 CISQ 8
Software Quality Iceberg (Code Complete, Steve McConnell)
Code complexity
Maintainability
Internal Coupling
Functional Size
Redundant code
Testability
External Coupling
Operating Cost
Maintenance Cost
Reliability
Performance
Business Value
Example After 120 Day Project
©2019 CISQ 9https://forio.com/simulate/dpnorton66/tech-debt-v3/simulation/#
Example After 120 Day Project
©2019 CISQ 10
Refactoring
FTE Tech
Debt Refactoring Cost
Team
Size
Inject
Rate Rate Days Left At $240 At $1040
5 5 - 15% 10% 16.3 $3,912 $16,952
10 5 - 15% 10% 32.7 $7,848 $34,008
20 5 - 15% 10% 65.3 $15,672 $67,912
Example After 120 Day Project
©2019 CISQ 11
Refactoring
FTE Tech
Debt Refactoring Cost
Team Size
Inject
Rate Rate Days Left At $240 At $1040
5 10 - 25% 10% 63.2 $15,168 $65,728
10 10 - 25% 10% 126.4 $30,336 $131,456
20 10 - 25% 10% 252.8 $60,672 $262,912
What about a poor team, what then
3.8 X the refactoring cost of a good team
Example After 120 Day Project
©2019 CISQ 12
But wait…..what if its another team doing the
refactoring and maintenance ?
Then assume for each hour of coding by the
original team allow between 2 to 8 hours by the
maintenance team to understand and refactor the
original code.
Questions on Productivity
13
Desire for Autonomy
©2019 CISQ 14Autonomy at Spotify —  by Henrik Kniberg
Quality Starts With The System Integrator, They Build The
Foundation Digital Business Is Based On
©2019 CISQ 15
Quality Starts With The System Integrator, They Build The
Foundation Digital Business Is Based On
©2019 CISQ 16
CEOs are Paying The Price For Poor IT Quality
©2019 CISQ 17
Let’s Learn From The Past
©2019 CISQ 18
As industries mature they automate, from robots to fly-by-wire
Agenda
©2019 CISQ 19
• What are the drivers for automation
• How do we introduce more automation
• What role do standards play
Focus on Culture and Behavior – Be Specific
©2019 CISQ 20
• Don’t expect everyone to like
automation, some people just like
doing it the hard way
• Incentivize the behavior you want for
the individual and team.
• Have agreed metrics and KPI linked
to automation.
• Show results
Develop The Correct Skills
©2019 CISQ 21
Process
Design
Scripting
Toolchain
Integration
Standards
Definition
Obtain Commitment From the Team
©2019 CISQ 22
Certify The Environment Regarding QA, Don’t Assume It
©2019 CISQ 23
Don’t Assume You Are OK if Each CI/CD Pipeline is OK
Tactical Enterprise
Complexity
Complexity is not
a constant
It is not a linear function
of the enterprise
It's a nonlinear function that
may level "S" or rise
exponentially
In a nonlinear system, 90% of the complexity is a result of less than 10% of the node connections.
Gamify - Link Automation & Consistency to Team Autonomy
Autonomy
Time of
Deployments
Intra-day
allowed
After hours and
on weekends
Frequency of
Deployments
No limits on
changes per
today
Few changes
per week
Change
Advisory
Board
CAB for
information
purposes only
CAB for all
changes
Freeze
Periods
Only exceptional
change freeze
periods apply
All freeze
periods apply
Continuous
Integration
Environments
Quality
Assurance
Incident
Management
Release
Management
Coding
Practices
Team
A
Level of Automation
Team
B
Stay in Control With Agile Governance
• Communities of
Practice
• Toolchain Consistency
• Tools Register
• Automation Best
Practice
Link Automation to KPI, and Set Targets For Tech Debt
Reduction
• Feature throughput
• Lead-time/Cycle-time
• IT Downtime
• Business Downtime
• Percentage of task
automated
• Refactoring rate and cost
Embed Automation With Suppliers
CISQ has been referenced by the U.S. General
Services Administration (GSA), formally citing CISQ
requirements in a Information Technology (IT)
statement of work from the Office of the CIO for the
Office of Public Buildings. GSA is an independent
agency of the U.S. government that supports general
services of Federal agencies.
See page 21, section 5.9 in GSA’s document,
Schedule 70 Blank Purchase Agreement for IT and
Development Services…
“PB-ITS (Project Based IT Services) is seeking to
establish code quality standards for its existing code
base, as well as new development tasks. As an
emerging standard, PB-ITS references the
Consortium for IT Software Quality (CISQ) for
guidance on how to measure, evaluate and improve
software.”
Focus on Outcomes
Agenda
• What are the drivers for automation
• How do we introduce more automation
• What role do standards play
We Need Standards We Can Implement With DevOps
We built this city, we built this city on rock an' roll
We Need Standards We Can Implement With DevOps
We built this city, we built this city on rock an' roll
ISO 25010 In Structural Code Analysis, Practical Examples
• OWASP Top 10 Vulnerabilities—most critical web application security risks –
CWEs & CVEs
• OWASP Application Security Verification Std v4.0 – 14 categories guide
automated unit & integration tests – most all verification checks have
corresponding CWEs
• SANS/CWE Top 25 — most commonly encountered common weakness
enumerators (CWEs)
• CISQ / Object Management Group (OMG) Automated Source Code
Measures for technical debt & structural quality (Security, Reliability,
Performance Efficiency & Maintainability) – all based on MITRE CWEs
CISQ Structural Quality Measures
Working With Suppliers
Scorecard
Measurement and discussion in
governance committees to help
set behavior
SLAs
 Treat software enhancements
and maintenance as a service;
track levels, penalties, credits
Recommendation email
 Email to vendor delivery leaders
that they should consider using
CISQ guidelines for all ADM
work
Acceptance criteria
 Measure and demand minimal
set of acceptance criteria for any
new development or release
RFP
 Initial statement of requirements
and project definition can set
the tone for quality of
deliverables
SOW
 Definition of specific project
scope and deliverable can
include definition of quality and
security
Six Levels of Engaging Vendors with CISQ Standards
CISQ Get The Standards – They Are Free
https://www.it-cisq.org/standards/
CISQ Work With Us

Agile Team Autonomy – Don’t Just Give It Away Make Teams Earn It

  • 1.
    Agile Team Autonomy– Don’t Just Give It Away, Make Teams Earn It ©2019 CISQ 1 Dave Norton Executive Director Consortium for Information & Software Quality david.norton@it-cisq.org
  • 2.
    Two Basic Truths ©2019CISQ 2 Things are more complex and the pace of change is relentless
  • 3.
    Agenda ©2019 CISQ 3 •What are the drivers for automation • How do we introduce more automation • What role do standards play
  • 4.
    Agenda ©2019 CISQ 4 •What are the drivers for automation • How do we introduce more automation • What role do standards play
  • 5.
    Complex Technology Stack ©2019CISQ 5 Multi-language,multi-layerArchitecture EJB PL/SQL Oracle SQL Server DB2 T/SQL Hibernate Spring Struts .NET COBOL IMS Messaging Sybase • Code style & layout • Expression complexity • Code documentation • Class or program design • Basic coding standards • Developer level Unit Level1 Technology Stack Java Java Java Web Services • Single language/technology layer • Intra-technology architecture • Intra-layer dependencies • Inter-program invocation • Security vulnerabilities • Development team level Technology Level2  Integration quality  Architectural compliance  Risk propagation  Application security  Resiliency checks  Transaction integrity  Function point,  Effort estimation  Data access control  SDK versioning  Calibration across technologies  IT organization level System Level3 JSP ASP.NETAPIs
  • 6.
    Drive for Velocity ©2019CISQ 6 Everyone wants faster time to market, but few want to hear about the risks
  • 7.
    Complex Toolchains ©2019 CISQ7 • Production metrics, objects and feedback • Requirements • Business metrics • Update release metrics • Release plan, timing and business case • Security policy and requirement • Design of the software and configuration • Coding including code quality and performance • Software build and build performance • Release candidate • Acceptance testing • Regression testing • Security and vulnerability analysis • Performance • Configuration testing • Approval/preapprovals • Package configuration • Triggered releases • Release staging and holding • Infrastructure storage, database and network provisioning and configuring • Application provision and configuration. • Performance of IT infrastructure • End-user response and experience • Production metrics and statistics • Application monitoring
  • 8.
    Increasing Technical Debt ©2019CISQ 8 Software Quality Iceberg (Code Complete, Steve McConnell) Code complexity Maintainability Internal Coupling Functional Size Redundant code Testability External Coupling Operating Cost Maintenance Cost Reliability Performance Business Value
  • 9.
    Example After 120Day Project ©2019 CISQ 9https://forio.com/simulate/dpnorton66/tech-debt-v3/simulation/#
  • 10.
    Example After 120Day Project ©2019 CISQ 10 Refactoring FTE Tech Debt Refactoring Cost Team Size Inject Rate Rate Days Left At $240 At $1040 5 5 - 15% 10% 16.3 $3,912 $16,952 10 5 - 15% 10% 32.7 $7,848 $34,008 20 5 - 15% 10% 65.3 $15,672 $67,912
  • 11.
    Example After 120Day Project ©2019 CISQ 11 Refactoring FTE Tech Debt Refactoring Cost Team Size Inject Rate Rate Days Left At $240 At $1040 5 10 - 25% 10% 63.2 $15,168 $65,728 10 10 - 25% 10% 126.4 $30,336 $131,456 20 10 - 25% 10% 252.8 $60,672 $262,912 What about a poor team, what then 3.8 X the refactoring cost of a good team
  • 12.
    Example After 120Day Project ©2019 CISQ 12 But wait…..what if its another team doing the refactoring and maintenance ? Then assume for each hour of coding by the original team allow between 2 to 8 hours by the maintenance team to understand and refactor the original code.
  • 13.
  • 14.
    Desire for Autonomy ©2019CISQ 14Autonomy at Spotify —  by Henrik Kniberg
  • 15.
    Quality Starts WithThe System Integrator, They Build The Foundation Digital Business Is Based On ©2019 CISQ 15
  • 16.
    Quality Starts WithThe System Integrator, They Build The Foundation Digital Business Is Based On ©2019 CISQ 16
  • 17.
    CEOs are PayingThe Price For Poor IT Quality ©2019 CISQ 17
  • 18.
    Let’s Learn FromThe Past ©2019 CISQ 18 As industries mature they automate, from robots to fly-by-wire
  • 19.
    Agenda ©2019 CISQ 19 •What are the drivers for automation • How do we introduce more automation • What role do standards play
  • 20.
    Focus on Cultureand Behavior – Be Specific ©2019 CISQ 20 • Don’t expect everyone to like automation, some people just like doing it the hard way • Incentivize the behavior you want for the individual and team. • Have agreed metrics and KPI linked to automation. • Show results
  • 21.
    Develop The CorrectSkills ©2019 CISQ 21 Process Design Scripting Toolchain Integration Standards Definition
  • 22.
    Obtain Commitment Fromthe Team ©2019 CISQ 22
  • 23.
    Certify The EnvironmentRegarding QA, Don’t Assume It ©2019 CISQ 23
  • 24.
    Don’t Assume YouAre OK if Each CI/CD Pipeline is OK Tactical Enterprise Complexity Complexity is not a constant It is not a linear function of the enterprise It's a nonlinear function that may level "S" or rise exponentially In a nonlinear system, 90% of the complexity is a result of less than 10% of the node connections.
  • 25.
    Gamify - LinkAutomation & Consistency to Team Autonomy Autonomy Time of Deployments Intra-day allowed After hours and on weekends Frequency of Deployments No limits on changes per today Few changes per week Change Advisory Board CAB for information purposes only CAB for all changes Freeze Periods Only exceptional change freeze periods apply All freeze periods apply Continuous Integration Environments Quality Assurance Incident Management Release Management Coding Practices Team A Level of Automation Team B
  • 26.
    Stay in ControlWith Agile Governance • Communities of Practice • Toolchain Consistency • Tools Register • Automation Best Practice
  • 27.
    Link Automation toKPI, and Set Targets For Tech Debt Reduction • Feature throughput • Lead-time/Cycle-time • IT Downtime • Business Downtime • Percentage of task automated • Refactoring rate and cost
  • 28.
    Embed Automation WithSuppliers CISQ has been referenced by the U.S. General Services Administration (GSA), formally citing CISQ requirements in a Information Technology (IT) statement of work from the Office of the CIO for the Office of Public Buildings. GSA is an independent agency of the U.S. government that supports general services of Federal agencies. See page 21, section 5.9 in GSA’s document, Schedule 70 Blank Purchase Agreement for IT and Development Services… “PB-ITS (Project Based IT Services) is seeking to establish code quality standards for its existing code base, as well as new development tasks. As an emerging standard, PB-ITS references the Consortium for IT Software Quality (CISQ) for guidance on how to measure, evaluate and improve software.”
  • 29.
  • 30.
    Agenda • What arethe drivers for automation • How do we introduce more automation • What role do standards play
  • 31.
    We Need StandardsWe Can Implement With DevOps We built this city, we built this city on rock an' roll
  • 32.
    We Need StandardsWe Can Implement With DevOps We built this city, we built this city on rock an' roll
  • 33.
    ISO 25010 InStructural Code Analysis, Practical Examples • OWASP Top 10 Vulnerabilities—most critical web application security risks – CWEs & CVEs • OWASP Application Security Verification Std v4.0 – 14 categories guide automated unit & integration tests – most all verification checks have corresponding CWEs • SANS/CWE Top 25 — most commonly encountered common weakness enumerators (CWEs) • CISQ / Object Management Group (OMG) Automated Source Code Measures for technical debt & structural quality (Security, Reliability, Performance Efficiency & Maintainability) – all based on MITRE CWEs
  • 34.
  • 35.
    Working With Suppliers Scorecard Measurementand discussion in governance committees to help set behavior SLAs  Treat software enhancements and maintenance as a service; track levels, penalties, credits Recommendation email  Email to vendor delivery leaders that they should consider using CISQ guidelines for all ADM work Acceptance criteria  Measure and demand minimal set of acceptance criteria for any new development or release RFP  Initial statement of requirements and project definition can set the tone for quality of deliverables SOW  Definition of specific project scope and deliverable can include definition of quality and security Six Levels of Engaging Vendors with CISQ Standards
  • 36.
    CISQ Get TheStandards – They Are Free https://www.it-cisq.org/standards/
  • 37.