4. CEO IT Services
Services are delivered through an Internal Service Fund
“Running a Service Delivery Business”
5. Pricing a Service
What does internet access cost?
Hardware: •Redundant Core Switches Maintenance Contract HW and SW)
•Redundant Firewalls •Content Filtering Maintenance Contract
•Redundant Routers •County ARIN IP and BGP AS Services
•Redundant Core Switches
•IPS / IDS Hardware Depreciation:
•Monitoring (Orion) Servers •Firewalls - Hardware and software Depreciation associated
•Content Filtering Appliances •Routers - Hardware and software Depreciation associated
•Content Filtering Report Servers •Core Switches - Hardware and software Depreciation associated
•DNS Servers •Monitoring - Hardware and software Depreciation associated
•Content Filtering - Hardware and software Depreciation associated
Software:
•Firewall Security Software licensing Other CEOIT Services:
•Content Filtering Software licensing & Report Services •Contracts & Procurement Management
•Orion Monitoring Software licensing •Resource Management
•DNS Software (Unix and Wintel) •Financial and Cost, Budget Analysis
•Billing Processing
Data Center Facilities
Labor:
Contracts: •Network Engineering Labor
•Redundant Internet Service Providers: •Security Engineering Labor
1.TW Telecom •Availability Management Analysis
2.VPLS (this is a fail-over service) •Capacity Management Analysis
•IPS / IDS monitoring Service •Incident Management (Service Desk)
•Redundant WAN Circuit Contracts •Change Management
•Redundant Firewall Maintenance Contract (HW and SW) •Management
•Redundant Routers Maintenance Contract HW and SW)
6. Pricing a Service
What does internet access cost?
Hardware: •Redundant Core Switches Maintenance Contract HW and SW)
•Redundant Firewalls •Content Filtering Maintenance Contract
•Redundant Routers •County ARIN IP and BGP AS Services
•Redundant Core Switches
•IPS / IDS Hardware Depreciation:
•Monitoring (Orion) Servers •Firewalls - Hardware and software Depreciation associated
•Content Filtering Appliances •Routers - Hardware and software Depreciation associated
•Content Filtering Report Servers •Core Switches - Hardware and software Depreciation associated
•DNS Servers •Monitoring - Hardware and software Depreciation associated
•Content Filtering - Hardware and software Depreciation associated
Software:
•Firewall Security Software licensing Other CEOIT Services:
•Content Filtering Software licensing & Report Services •Contracts & Procurement Management
•Orion Monitoring Software licensing •Resource Management
•DNS Software (Unix and Wintel) •Financial and Cost, Budget Analysis
•Billing Processing
Data Center Facilities
Labor:
Contracts: •Network Engineering Labor
•Redundant Internet Service Providers: •Security Engineering Labor
1.TW Telecom •Availability Management Analysis
2.VPLS (this is a fail-over service) •Capacity Management Analysis
•IPS / IDS monitoring Service •Incident Management (Service Desk)
•Redundant WAN Circuit Contracts •Change Management
•Redundant Firewall Maintenance Contract (HW and SW) •Management
•Redundant Routers Maintenance Contract HW and SW)
7. The Story…
Operational Transformation in the Face of Financial Distress
Environment and Results: How:
Organization: Combination:
1. A culture built on Heroics 1. Leadership
2. Bad relationship w/ vendor (expiring K) 2. Soft skills introduction
3. Siloed organization 3. Sustained “drumbeat”
3. Assigning blame 4. The persuader
4. Resistance to change 5. ITSM principles
5. No priorities
Method:
Results: 1. Plan
2. Implement
1. Moved to top 10% against benchmark 3. Absorb
2. Reduced Budget 27% 4. Measure
3. Increased customer satisfaction 16%
4. Reduced SLA exceptions by 77%
5. Reduced incidents by 38%
6. Improved incident resolution from 5 days
to 2 days “Be transparent, or be gone”
8. Phases of the Journey
Method - Plan, Implement, Absorb, Measure
Phase 1: ITSM Training April 2009
DISC (Soft Skills) Training May 2009
Capacity Management May 2009
Availability Management June 2009
Resource Planning June 2009
Project Prioritization June 2009
Phase 2: Network & Platform Services Reorganization October 2009
Service Design Package November 2009
Monthly MAP meetings March 2010
Phase 3: Marketing Brochure (Mini Service Catalog - BRAND) October 2010
Service Desk re-implementation December 2010
Phase 4: Network Operations Center January 2011
ITSM Full Suite (BMC – “AMP”) March 2011
Phase 5: Re-benchmark AIMM July 2011
Service Catalog – Version 1 October 2011
Customer Ranking of Services January 2012
Service Catalog – Version 2 February 2012
ITSM Reorganization Future
Service Level Management Future
Further Automate Capacity & Availability Future
9.
10. Phase 1 Driving Continuous Improvements Phase 3
2009 2010
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized:
Phase 2 -Plan & Design Customer Survey (Jul – Dec 09)
-Build & Transition
Training -Operate & Maintain
Incident Mgt Process ITSM Integrated Suite
Service Desk
SDP Process Customer Survey (Jan – Jun 10)
Capacity Reporting “A journey of a
Chaos CTO Portal
thousand miles begins
“One Team” with a single step” Certifications: 95
IT Service Management Path to Maturity
Marketing Brochure
“Agency Success is
Availability Reporting AIMM (HP) Report 46 Our Mission”
-Technology & Architecture AIMM (HP) Report 34
ACS Program -Mgt Tools & Process Maturity
-Culture & IT Staff Maturity
Executive hired Ticket Aging Process -Demand, Supply & IT Governance
Communication Team Formed
Resource Planning
Performance Audit Report
County Connection: OCDC
Project Prioritization Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Data Center Tour
(DISC) Not a single failed non-standard RFC
Tech Fest
Not a single Service Level Breach
11. Phase 4 Driving Continuous Improvements Phase 5
2011 Method
2012
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
ITSM based
ITSM Integrated Suite Launch Reorganization
Service Catalog v 1 Service Catalog v 2
– FULL SUITE (CMDB, Change,
Release…)
Application Monitoring
MSB Planning
Success Customer Survey (Jul – Dec 10)
IT Service Management Path to Maturity
AIMM (HP) Report 10 Service Level
Management
Network
Operations
Center Customer
Launched Grading by
Service
Virtual Server Farm Replacement
14. Management Action Plan
Results through Present
400
350
300
250
200
150
100
50
0
Nov Mar April May June Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jul Aug Sep
2010 2011
Cumulative Monthly Accomplishments
15. Capacity & Availability Reporting
Phase 1
Capacity Report
Capacity Report Availability Report
Table of Contents
Executive Summary............................................................................................................ 4
Dashboard Summary........................................................................................................... 4
Risks.................................................................................................................................... 4
Warnings ............................................................................................................................. 4
Scope................................................................................................................................... 5
Capacity Measurements ...................................................................................................... 5
Network .......................................................................................................................... 5
County Wide Area Network ....................................................................................... 5
Time Warner Internet.................................................................................................. 6
Security ........................................................................................................................... 7
Storage ............................................................................................................................ 8
Tier-1........................................................................................................................... 8
Tier-2........................................................................................................................... 8
E-Mail ........................................................................................................................... 10
Virtual Environment ..................................................................................................... 13
Analysis Additional Capacity ................................................................................... 14
Conclusion ................................................................................................................ 14
Recommendation ...................................................................................................... 14
Power5 Systems............................................................................................................ 15
Mainframe..................................................................................................................... 18
Power ............................................................................................................................ 19
Data Center Space......................................................................................................... 22
Floor Space ............................................................................................................... 22
Office Space.............................................................................................................. 24
Telephone Services ....................................................................................................... 25
APPENDIX – A – Report Data Source and Contributors ................................................ 26
APPENDIX – B – Virtual Environment Cluster Information .......................................... 27
18. Resource Planning
Key Metrics
3 Week Summary (by project):
Row Labels Sum of ETC
Disaster Recovery Implementation (FY11-12) 336.00
Enterprise Secure File Transfer Migration 33.20
ePages Replacement 22.00
Exchange 2010 Production Build out 112.00
HR & FS Access Definition Project Schedule 1.50
Secure FTP Infrastructure Upgrade 48.58
Grand Total 553.28
Projects with late tasks:
Sum of ETC
Row Labels Grand Total
ePages Replacement 10.00
Grand Total 10.00
20. Driving Continuous Improvements
2009 2010
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized:
Phase 2 -Plan & Design Customer Survey (Jul – Dec 09)
-Build & Transition
Training -Operate & Maintain
Incident Mgt Process ITSM Integrated Suite
Service Desk
SDP Process Customer Survey (Jan – Jun 10)
Capacity Reporting “A journey of a
Chaos CTO Portal
thousand miles begins
“One Team” with a single step” Certifications: 95
IT Service Management Path to Maturity
Marketing Brochure
“Agency Success is
Availability Reporting AIMM (HP) Report 46 Our Mission”
-Technology & Architecture AIMM (HP) Report 34
ACS Program -Mgt Tools & Process Maturity
-Culture & IT Staff Maturity
Executive hired Ticket Aging Process -Demand, Supply & IT Governance
Communication Team Formed
Resource Planning
Performance Audit Report
County Connection: OCDC
Project Prioritization Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Data Center Tour
(DISC) Not a single failed non-standard RFC
Tech Fest
Not a single Service Level Breach
21. ITSM Organizational Design
Phase 2: Reorganization of NPS
ASG IRM Process Security NPS OPS TSD PMO BA
Plan & Build & Operate &
Design Transition Maintain
22. ITSM Organizational Design
Plan and Design
Operate and Maintain
Build and Transition
Overall Architecture and
Technologies / Standards* Design Validation / Review and
System acceptance
Acceptance
Capacity Requirements Registering application in ops
Procurements
Availability Requirements (contact information, service levels,
Configuration and Installation monitoring and backup information,
Training Requirements new or changed procedures, Access
Update and Audit of Asset
POC requirements Lists/CMDB levels)
Design Diagrams and Requirements RFC Execution and Update procedures
MOUs/SLRs Coordination/Communication System lockdown
Success Criteria Rollout Plan QA and Validation of Monitoring,
TCO Calculations Backout Plan Backups and Access
Alternatives Risk Assessment for RFC Assist in Negotiating SLRs/SLAs
Backup requirements Ops Training and Handoff Freeze and Warranty Period
Management
Process / Procedure requirements QA and Validation of Systems
Monitoring requirements Feedback to Design Team
Access Requirements Approval of Rollout from Ops
Standard Requests Approval of Rollout from customer
Engineering and Ops review and Approval of Rollout from Design
acceptance group
Continuous improvement or Support and monitoring during cool
scalability plans down/warranty period
Approval of the Design from Peer
Groups
Approval of the design from Chief
Architect
Approval of the design from Customer
23. SDR SDP – Issues identified through the process
25. Driving Continuous Improvements Phase 3
2009 2010
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized:
-Plan & Design Customer Survey (Jul – Dec 09)
-Build & Transition
Training -Operate & Maintain
Incident Mgt Process ITSM Integrated Suite –
Service Desk
SDP Process Customer Survey (Jan – Jun 10)
Capacity Reporting “A journey of a
Chaos CTO Portal
thousand miles begins
“One Team” with a single step” Certifications: 95
IT Service Management Path to Maturity
Marketing Brochure
“Agency Success is
Availability Reporting AIMM (HP) Report 46 Our Mission”
-Technology & Architecture AIMM (HP) Report 34
ACS Program -Mgt Tools & Process Maturity
-Culture & IT Staff Maturity
Executive hired Ticket Aging Process -Demand, Supply & IT Governance
Communication Team Formed
Resource Planning
Performance Audit Report
County Connection: OCDC
Project Prioritization Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Data Center Tour
(DISC) Not a single failed non-standard RFC
Tech Fest
Not a single Service Level Breach
26. It’s working...
Phase 3: “Mojo”
November 2009
Not a single Service Level Breech
- Marshall Goldsmith Incidents – from over 5 days to 1
Not a single failed non-standard RFC
Service request turnaround improvements
(100%)
30. Network Operations Center
Phase 4
Network Monitoring Server Monitoring
Handoff/Review Logs Handoff/Review Logs
Orion Network Dashboard Orion Server Dashboard
Daily Walkthrough AM/PM VM Console
IP Monitor Int/Ext/DR IP Monitor Probation/Int
INMON HP SIM
38. Evidence of Improvements
Results: HP’s Converged Infrastructure Maturity Model
Orange County versus US Public Sector Benchmark
July 2011
4.00
3.50
3.00 Orange County
US Public Sector
2.50
2.00
1.50
1.00
Technology & Tools & Processes Culture & IT Staff Demand, Supply & IT
Architecture Maturity Governance
39. Evidence of Improvements
Results: HP’s Converged Infrastructure Maturity Model
Orange County Percent of Characteristics "At or Better"
July 2011
Head
100
count
Top 10%
90
80
70 - 27%
Better
60
Equal
50
40
30
20
10
0
Dec-09 Jul-10 Jul-11
42. Driving Continuous Improvements Phase 5
2011 Method
2012
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
ITSM based
ITSM Integrated Suite Launch Reorganization
Service Catalog v 1 Service Catalog v 2
– FULL SUITE (CMDB, Change,
Release…)
Application Monitoring
MSB Planning
Success Customer Survey (Jul – Dec 10)
IT Service Management Path to Maturity
AIMM (HP) Report 10 Service Level
Management
Network
Operations
Center Customer
Launched Grading by
Service
Virtual Server Farm Replacement
43. Sorted Top to Bottom (and vice-versa)
Sorted by: Customer Sorted by: Customer
Ave Ave
1 Enterprise Time Synchronization 96.50 1 Disaster Recovery 66.50
2 Midrange Platform (AIX Platform) 96.50 2 Enterprise Secure Web Filtering 82.50
3 Mainframe Platform 96.00 3 eGov Services 83.33
4 Enterprise Network Connectivity (WAN) 94.71 4 Service Desk 84.78
5 Enterprise Secure File Transfer Services 94.20 5 Project Management 85.00
6 Enterprise RightFax Services 94.00 6 OCTNET 86.39
7 SLA1 – Unmanaged Server Hosting 94.00 7 Enterprise Remote Access Infrastructure 86.86
8 Database Mgt – Information Resource Mgt 93.67 8 SLA3 Service 87.80
A 9 Enterprise Domain Name Resolution
10 Blackberry Services
11 OnBase
12 VTL Replication Service
93.20
92.67
92.60
91.67
9 DeltaCopy Data Protection Servce (DCDP)
10 Business Recovery Center
11 Enterprise Network Security
12 Enterprise Global User Directory Services
88.33
88.33
88.50
88.67
13 Standard Backup Service 91.50 13 Publishing Services 88.67
14 Enterprise Email Services 91.33 14 Desktop Support 90.00
15 Finance Services 91.33 15 Applications Services Group 90.50
16 Enterprise Storage Services (SAN) 91.25 16 Technical Consultation 90.80
17 Enterprise Internet Service Provider (ISP) 90.86 17 Enterprise Internet Service Provider (ISP) 90.86
18 Technical Consultation 90.80 18 Enterprise Storage Services (SAN) 91.25
19 Applications Services Group 90.50 19 Enterprise Email Services 91.33
20 Desktop Support 90.00 20 Finance Services 91.33
21 Enterprise Global User Directory Services 88.67 21 Standard Backup Service 91.50
22 Publishing Services 88.67 22 VTL Replication Service 91.67
23 Enterprise Network Security 88.50 23 OnBase 92.60
24 DeltaCopy Data Protection Servce (DCDP) 88.33 24 Blackberry Services 92.67
25 Business Recovery Center 88.33 25 Enterprise Domain Name Resolution 93.20
B 26 SLA3 Service
27 Enterprise Remote Access Infrastructure
28 OCTNET
29 Project Management
87.80
86.86
86.39
85.00
26 Database Mgt – Information Resource Mgt
27 Enterprise RightFax Services
28 SLA1 – Unmanaged Server Hosting
29 Enterprise Secure File Transfer Services
93.67
94.00
94.00
94.20
30 Service Desk 84.78 30 Enterprise Network Connectivity (WAN) 94.71
31 eGov Services 83.33 31 Mainframe Platform 96.00
32 Enterprise Secure Web Filtering 82.50 32 Enterprise Time Synchronization 96.50
33 Disaster Recovery 66.50 33 Midrange Platform (AIX Platform) 96.50
48. Conclusions & Lessons Learned
1. Financial distress clouds have their silver linings in IT environments.
a. Working together in the face of a single galvanizing event
b. Speak with “one voice”
2. It Takes an IT mojo-driven strategy to implement sustained continued improvements in
financial distress.
a. Courage and consistency; buy-in
b. Incremental changes (process, organizational), planning and soft skills provided results
3. New leadership can ignite IT Mojo.
a. Common goals
b. Common messages
4. Establish a drumbeat to pace the changes and measure visibly and consistently.
a. Messaging consistency
b. Allow the organization to digest changes
5. Establish key measurements of the categories shown in the customer surveys.
a. Report on them
b. Make sure everyone knows what’s being measured
6. Establish a consistent program of accountability .
a. Report on them
b. Make sure everyone knows what’s being measured
7. Benchmark the organization before commencing the journey.