Software metrics to manage vendor relationships​. How to measure the delivered source code.

1. CTTI
Government of Catalonia
Next gen sourcing for application services
Software metrics to manage vendor relationships

2. CTTI Introduction
2
Provides services to citizens
Manages citizens relationship
Plans, executes and evaluates
public policies
Translates government
requirements into ICT projects
Optimizes resources
Promotes innovation
Supplies ICT services
Promotes innovation within
solutions design
Catalan Government
CTTI
ICT Suppliers
Public enterprise that has capacity to design, plan, oversee, manage and
execute ICT projects
Telecommunic
ations
suppliers
Workplace
suppliers
Data centers
suppliers
Applications
suppliers
Justice Culture
Health
Governance &
Inst. Relations
Business &
Labor
Education
Internal
Affairs
Presidency
Agriculture
Territory &
Sustainability
Social Welfare
& Family
Economy &
Knowledge
Public
organiz
ations

3. Telecommunications and Information Technology Centre
The challenge
3
601 M€ ICT budget
200.000 Workplaces
5.300 Buildings
26 Call centers
1.800 Applications (71 M€)
475 Active projects
41 Data Centersand 71 Tech
rooms (<5 servers)
5.500 xDSL Lines
1.000 Fiber optics connections
3 Gbps InternetAccess
220 DTV & Radio (Infrastructure)
Volume EFFICENCY25%
TRANSFORMATION
COMPETITIVENESS
Improve efficiency and
savings in the provision of
ICT services and energy
spending
Consolidate infrastructure and
transversal applications, and
modernize desktop and
telecommunication services
Provide Catalonia with a high
capacity telecommunications
network. Promote regional
balance
Boost the Catalan
ICT sector and
attract investments
from multinationals
in Catalonia
ICT SECTOR

4. Applications context
4
Structured by
business
packages
11 packages, classified by business or technical domain. Some of the last
grouped with business affinity (Institutional & BI & Reporting)
One supplier for
each package
8 suppliers
(outsourcing of development and maintenance of applications)
Goals
• SaaS oriented, BO
• Demand rationalization
• Reduce application portfolio: consolidation and rationalization of applications,
reusability
• Standardization of hardware, software and development platforms
• Application modernization
• Improvement of maintenance process, methodologies
• Model that meets the functional priorities of the departments, institutions and
agencies of the Government
RFP 71M€ per year, for 6 years (since September2012)

5. ICT Governance
Governance model
5
Transició
i
Transforamció
Govern Propi
CTTI
Applications Data infrastructures
Desktop services
SAU (Users helpdesk service)
Communications
infrastructures
Strategic governance
Financial governance
Relationship
with
departments and
entities
governance
Demand governance
Project
governance
Billing governance
Supply
governance
Services
governance
Quality governance
Security governance
Transition
and
transformation
People, buildings, resources and tools governance
GUIDING PRINCIPLES FOR CHANGE
TRANSPARENCY AGILITY EFFICIENCY TRACEABILITY
Two externalized packages for governancesupport,one of them includes
Quality Office

6. Quality governance
Our quality policy
6
Target
Quality policy Attitude and approachtowards
quality
Improve customer
satisfaction with steady
progress in quality of our
solutions and services, at
the lowest cost
Reduce the cost of poor
quality
Risk management
Useful and efficient risk
management
Selection of acceptance
criteria and effort of
quality assurance and
quality control depending
on risk of applications
Tools
Requirements
management and testing
management
Quality source code
review
Performance Testing
Functional Testing
Automation
Checklists
Quality model
Definition of procedures,
instructions and rules to
achieve our goals
(standards-based)
Reviews of deliverables.
Checkpoints. Detect
defects early

7. Why measure code quality, Why CAST?
7
Why measure?
• Measuring software quality is a goal of
governance that cannot be fully transferred
to vendors
• Objectively measure transition (“I’m a vendor
who has received a bad quality code from
previous vendor, all changes will be more
expensive”)
• Identify sources of critical risk
• Verifying compliance with goals set in SLAs
• Establishing baselines and historical trends.
Agreement and monitoring improvement.
• Reduce maintenance spending
Why CAST?
• Technologies and platforms coverage
• Broad set (it covers more than 90%
portfolio)
• Database analysis
• Specific rules for best in class
frameworks (JSF, Spring, Struts,
Hibernate, …)
• Multi-tier analysis
• Function points measurement
• Action plan definition for improvement
• Justification of false positives
• Centralization of governance tool

8. 21
40
17
8
10
6 7
4 3
6 5
3 4
2 1 2 3
0
3
0 1
35
0
5
10
15
20
25
30
35
40
45
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
1900
2000
More
Prioritizing the application portfolio
8
Classify applications by
risk
High: 10% / Medium: 50%
/ Low: 40%
Do we need to review all our
applications?
Business impact, Visibility
(type, number of users,
frequency of use)
Strategic value
Complexity, Planned
evolution, Technology
maturity, Reutilization and
size, Interoperability
Scope More than 1800 apps.
Risk classification
Probability of failure
New applications
Target
Old applications
High and medium mandatory
High risk with more than 600 hours on maintenance effort
last year (named Grade A). Medium risk depending on
agreement with customer
Exceptions
Applications planned to be transformed
Applications based on products (content management, …)
25 applicationssum up 40% of
total effort in corrective
maintenance
72 applicationsrepresent95% of
maintenance effort(named Grade A)
Partitioning in effort of maintenance for high
risk applications
(21 apps with zero effort, 40 withless than 100 hours,…)
In scope of review
Yearly
effort (hrs)

9. Organizational structure of QA analysis & reporting
 Project governance
▪ Managers – similar to PMO
▪ They work with the vendors/providers to coordinate on behalf of customer
▪ Management of milestones, baselines and quality of deliverables
 Quality Office
▪ Prescriptive function, responsible for quality tools management
▪ Managing the quality scores, analyses results and make recommendations
▪ Support of project governance
 Supplier
▪ One quality manager assigned. He/she ensures the implementation of
quality model in all projects.
9

10. The QA data feeds into vendor management
 RFP included acceptance criteria and review cycle
10
 Acceptance criteria relaxed to minimum score of 3,00 for each
health factor
 Critical defects always zero
 Flat rate for corrective & perfective maintenance (service)
▪ Baseline review and maintenance reviews
 New projects
▪ Review at 20% of development (preventive)
▪ Review before production deployment

11. Operationalizing the analysis of the portfolio
11
Deliv ery of source
code
Rev iew at
owned CAST
license
3
T1
Send CAST
conf iguration and
summary report
Quality
Management
Quality
Office
Supplier
CTTI
Source code review process
Quality management
Each last day of month
Check what
needs to be
rev iewed
1
Notif y vendor
quality
manager
2
Has to be rev iewed?
Check acceptance
criteria
T1
Repeat
analy sis and
publish in
dashboard
7
T2
CAST AIP
T2 SVN
4
5
6 Is acomplished?
Send action plan
8
NO
Af ter max.1 month
Check and
monitoring with
managers
9
T1
Review cycle
• RFP stipulated every release, but
that’s too frequent (too many
releases with less than 50 hours of
development)
• Depending on degree
• 1st degree with more than 600
hours in maintenance during the
last 12 months, every 3 months,
some exceptions based on
release schedule (at least 4 per
year)
• 2nd degree, at customer
experience (at least 2 per year).
High cost of maintenance, too
many defects, …
• 3rd degree, no reviews
• All suppliers with CAST license;
Review their source code before
delivery; Send results to CTTI

12. Results over the last 6 months
12
Some big systems have been divided in several
applications in our portfolio. As the code is at
same location, some of them have been reviewed
completely
Avg TQI
Avg TQI
Avg TQI
Avg Maintainability
Avg Maintainability
Avg Maintainability
Avg Robustness
Avg Robustness
Avg Robustness
Avg Performance
Avg Performance
Avg Performance
Avg of Security
Avg of Security
Avg of Security
Avg Transferability
Avg Transferability
Avg Transferability
Avg Changeability
Avg Changeability
Avg Changeability
0,0 0,5 1,0 1,5 2,0 2,5 3,0 3,5 4,0
Low
Medium
High
The worstfactorsare transferability and performance High
47
Medium
42
Low
6
95
95 applications and 23M lines of
code analyzed
26 Action plans of Grade A defined (50% of expected)
All suppliers havealready reviewed
severalapplications,we are now
ready to begin the improvement

13. Future plans
 Realign and focus on the most risky applications, agreed with
managers. Prioritization on improvement versus analysis (after the first
round of reviews are finished).
 Measure lowering of tickets and corrective maintenance after
improvements on code
 Measure efficiency of our suppliers and use function points
 Increase acceptance criteria depending on risk of applications (above
3,25 for high degree, for example)
13

14. Q&A
 Questions and answers
14