The document summarizes the operational transformation of an IT organization facing financial distress over 5 phases from 2009 to 2011. Key steps included ITSM training, capacity and availability reporting, resource planning, reorganizing around ITSM principles, launching an integrated ITSM suite, and establishing a network operations center. Results included moving to the top 10% on benchmarks, increasing customer satisfaction by 16%, and reducing SLA exceptions by 77%.

1. Accelerating IT Mojo in the Face of Financial Distress
Joel A. Manfredo
Chief Technology Officer
October 12, 2011
1

2. Agenda
Background
Timeline
Phases 1 - 5
Results
Next Steps
Conclusions and Lessons Learned

3. County of Orange
$5.6 Billion Budget
7 Programs
32 Agencies / Departments

4. CEO IT Services
Services are delivered through an Internal Service Fund
“Running a Service Delivery Business”

5. Pricing a Service
What does internet access cost?
Hardware: •Redundant Core Switches Maintenance Contract HW and SW)
•Redundant Firewalls •Content Filtering Maintenance Contract
•Redundant Routers •County ARIN IP and BGP AS Services
•Redundant Core Switches
•IPS / IDS Hardware Depreciation:
•Monitoring (Orion) Servers •Firewalls - Hardware and software Depreciation associated
•Content Filtering Appliances •Routers - Hardware and software Depreciation associated
•Content Filtering Report Servers •Core Switches - Hardware and software Depreciation associated
•DNS Servers •Monitoring - Hardware and software Depreciation associated
•Content Filtering - Hardware and software Depreciation associated
Software:
•Firewall Security Software licensing Other CEOIT Services:
•Content Filtering Software licensing & Report Services •Contracts & Procurement Management
•Orion Monitoring Software licensing •Resource Management
•DNS Software (Unix and Wintel) •Financial and Cost, Budget Analysis
•Billing Processing
Data Center Facilities
Labor:
Contracts: •Network Engineering Labor
•Redundant Internet Service Providers: •Security Engineering Labor
1.TW Telecom •Availability Management Analysis
2.VPLS (this is a fail-over service) •Capacity Management Analysis
•IPS / IDS monitoring Service •Incident Management (Service Desk)
•Redundant WAN Circuit Contracts •Change Management
•Redundant Firewall Maintenance Contract (HW and SW) •Management
•Redundant Routers Maintenance Contract HW and SW)

6. The Story…
Operational Transformation in the Face of Financial Distress
Environment: How:
Organization: Combination:
1. A culture built on Heroics 1. Leadership
2. Siloed organization 2. Soft skills introduction
3. Assigning blame 3. Sustained “drumbeat”
4. Resistance to change 4. The persuader
5. No priorities 5. ITSM principles
Financial: Method:
1. Plan
1. 27% budget reduction (2 years) 2. Implement
2. Expiring staff augmentation contract 3. Absorb
4. Measure
Results:
1. Moved to top 10% against benchmark
2. Increased customer satisfaction 16%
3. Reduced SLA exceptions by 77%
“Be transparent, or be gone”

7. Phases of the Journey
Method - Plan, Implement, Absorb, Measure
Phase 1: ITSM Training April 2009
DISC (Soft Skills) Training May 2009
Capacity Management May 2009
Availability Management June 2009
Resource Planning June 2009
Project Prioritization June 2009
Phase 2: Network & Platform Services Reorganization October 2009
Service Design Package November 2009
Monthly MAP meetings March 2010
Phase 3: Marketing Brochure (Mini Service Catalog) October 2010
Service Desk re-implementation December 2010
Phase 4: Network Operations Center January 2011
ITSM Full Suite (BMC – “AMP”) March 2011
Phase 5: Re-benchmark AIMM July 2011 Today
Service Catalog October 2011
ITSM Reorganization Future
Service Level Management Future
Further Automate Capacity & Availability Future

8. Phase 1 Driving Continuous Improvements Phase 3
2009 2010
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized:
Phase 2 -Plan & Design Customer Survey (Jul – Dec 09)
-Build & Transition
Training -Operate & Maintain
Incident Mgt Process ITSM Integrated Suite
Service Desk
SDP Process Customer Survey (Jan – Jun 10)
Capacity Reporting “A journey of a
Chaos CTO Portal
thousand miles begins
“One Team” with a single step” Certifications: 95
IT Service Management Path to Maturity
Marketing Brochure
“Agency Success is
Availability Reporting AIMM (HP) Report 46 Our Mission”
-Technology & Architecture AIMM (HP) Report 34
ACS Program -Mgt Tools & Process Maturity
-Culture & IT Staff Maturity
Executive hired Ticket Aging Process -Demand, Supply & IT Governance
Communication Team Formed
Resource Planning
Performance Audit Report
County Connection: OCDC
Project Prioritization Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Data Center Tour
(DISC) Not a single failed non-standard RFC
Tech Fest
Not a single Service Level Breach

9. Phase 4 Driving Continuous Improvements Phase 5
2011 Method
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
ITSM based
Reorganization
ITSM Integrated Suite Launch
– FULL SUITE
(CMDB, Change, Release…)
MSB Planning
Service Catalog
Plan
Success Customer Survey (Jul – Dec 10)
Implement
IT Service Management Path to Maturity
Absorb
AIMM (HP) Report 10 Service Level
Management
Network
Operations
Center Measure
Launched Application Monitoring
Virtual Server Farm Replacement

10. Phase 1

11. DISC Assessment
Phase 1: Dominance, Influence, Steadiness, Compliance

12. Management Action Plan
Results through Present
400
350
300
250
200
150
100
50
0
Nov Mar April May June Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jul Aug Sep
2010 2011
Cumulative Monthly Accomplishments

13. Capacity & Availability Reporting
Phase 1
Capacity Report
Capacity Report Availability Report
Table of Contents
Executive Summary............................................................................................................ 4
Dashboard Summary........................................................................................................... 4
Risks.................................................................................................................................... 4
Warnings ............................................................................................................................. 4
Scope................................................................................................................................... 5
Capacity Measurements ...................................................................................................... 5
Network .......................................................................................................................... 5
County Wide Area Network ....................................................................................... 5
Time Warner Internet.................................................................................................. 6
Security ........................................................................................................................... 7
Storage ............................................................................................................................ 8
Tier-1........................................................................................................................... 8
Tier-2........................................................................................................................... 8
E-Mail ........................................................................................................................... 10
Virtual Environment ..................................................................................................... 13
Analysis Additional Capacity ................................................................................... 14
Conclusion ................................................................................................................ 14
Recommendation ...................................................................................................... 14
Power5 Systems............................................................................................................ 15
Mainframe..................................................................................................................... 18
Power ............................................................................................................................ 19
Data Center Space......................................................................................................... 22
Floor Space ............................................................................................................... 22
Office Space.............................................................................................................. 24
Telephone Services ....................................................................................................... 25
APPENDIX – A – Report Data Source and Contributors ................................................ 26
APPENDIX – B – Virtual Environment Cluster Information .......................................... 27

14. Capacity & Availability Reporting
Phase 1

15. Resource Planning
Key Metrics
3 Week Summary (by organization unit):
Values
Row Labels ETC Hours 3 Week FTE
Enterprise Infrastructure Srvcs 13.92 0.12
Networks and Platform Services 523.87 4.37
Security & Bus Continuity Oper 15.50 0.13
Grand Total 553.28 4.61

16. Resource Planning
Key Metrics
3 Week Summary (by project):
Row Labels Sum of ETC
Disaster Recovery Implementation (FY11-12) 336.00
Enterprise Secure File Transfer Migration 33.20
ePages Replacement 22.00
Exchange 2010 Production Build out 112.00
HR & FS Access Definition Project Schedule 1.50
Secure FTP Infrastructure Upgrade 48.58
Grand Total 553.28
Projects with late tasks:
Sum of ETC
Row Labels Grand Total
ePages Replacement 10.00
Grand Total 10.00

17. Phase 2

18. Driving Continuous Improvements
2009 2010
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized:
Phase 2 -Plan & Design Customer Survey (Jul – Dec 09)
-Build & Transition
Training -Operate & Maintain
Incident Mgt Process ITSM Integrated Suite
Service Desk
SDP Process Customer Survey (Jan – Jun 10)
Capacity Reporting “A journey of a
Chaos CTO Portal
thousand miles begins
“One Team” with a single step” Certifications: 95
IT Service Management Path to Maturity
Marketing Brochure
“Agency Success is
Availability Reporting AIMM (HP) Report 46 Our Mission”
-Technology & Architecture AIMM (HP) Report 34
ACS Program -Mgt Tools & Process Maturity
-Culture & IT Staff Maturity
Executive hired Ticket Aging Process -Demand, Supply & IT Governance
Communication Team Formed
Resource Planning
Performance Audit Report
County Connection: OCDC
Project Prioritization Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Data Center Tour
(DISC) Not a single failed non-standard RFC
Tech Fest
Not a single Service Level Breach

19. ITSM Organizational Design
Phase 2: Reorganization of NPS
ASG IRM Process Security NPS OPS TSD PMO BA
Plan & Build & Operate &
Design Transition Maintain

20. ITSM Organizational Design
Plan and Design
Operate and Maintain
Build and Transition
Overall Architecture and
Technologies / Standards* Design Validation / Review and
System acceptance
Acceptance
Capacity Requirements Registering application in ops
Procurements
Availability Requirements (contact information, service levels,
Configuration and Installation monitoring and backup information,
Training Requirements new or changed procedures, Access
Update and Audit of Asset
POC requirements Lists/CMDB levels)
Design Diagrams and Requirements RFC Execution and Update procedures
MOUs/SLRs Coordination/Communication System lockdown
Success Criteria Rollout Plan QA and Validation of Monitoring,
TCO Calculations Backout Plan Backups and Access
Alternatives Risk Assessment for RFC Assist in Negotiating SLRs/SLAs
Backup requirements Ops Training and Handoff Freeze and Warranty Period
Management
Process / Procedure requirements QA and Validation of Systems
Monitoring requirements Feedback to Design Team
Access Requirements Approval of Rollout from Ops
Standard Requests Approval of Rollout from customer
Engineering and Ops review and Approval of Rollout from Design
acceptance group
Continuous improvement or Support and monitoring during cool
scalability plans down/warranty period
Approval of the Design from Peer
Groups
Approval of the design from Chief
Architect
Approval of the design from Customer

21. SDR SDP – Issues identified through the process

22. Phase 3

23. Driving Continuous Improvements Phase 3
2009 2010
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized:
-Plan & Design Customer Survey (Jul – Dec 09)
-Build & Transition
Training -Operate & Maintain
Incident Mgt Process ITSM Integrated Suite –
Service Desk
SDP Process Customer Survey (Jan – Jun 10)
Capacity Reporting “A journey of a
Chaos CTO Portal
thousand miles begins
“One Team” with a single step” Certifications: 95
IT Service Management Path to Maturity
Marketing Brochure
“Agency Success is
Availability Reporting AIMM (HP) Report 46 Our Mission”
-Technology & Architecture AIMM (HP) Report 34
ACS Program -Mgt Tools & Process Maturity
-Culture & IT Staff Maturity
Executive hired Ticket Aging Process -Demand, Supply & IT Governance
Communication Team Formed
Resource Planning
Performance Audit Report
County Connection: OCDC
Project Prioritization Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Data Center Tour
(DISC) Not a single failed non-standard RFC
Tech Fest
Not a single Service Level Breach

24. It’s working...
Phase 3: “Mojo”
 November 2009
 Not a single Service Level Breech
- Marshall Goldsmith  Incidents – from over 5 days to 1
 Not a single failed non-standard RFC
 Service request turnaround improvements
(100%)

25. Marketing Brochure
Phase 3: “Mini Service Catalog”

26. Marketing Brochure
Phase 3: “Mini Service Catalog”

27. Phase 4

28. Phase 4 Driving Continuous Improvements
2011 Method
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
ITSM based
Reorganization
ITSM Integrated Suite Launch
– FULL SUITE (CMDB, Change,
Release…)
MSB Planning
Service Catalog
Plan
Success Customer Survey (Jul – Dec 10)
Implement
IT Service Management Path to Maturity
Absorb
AIMM (HP) Report 10 Service Level
Management
Network
Operations
Center Measure
Launched Application Monitoring
Virtual Server Farm Replacement

29. Network Operations Center
Phase 4
Network Monitoring Server Monitoring
Handoff/Review Logs Handoff/Review Logs
Orion Network Dashboard Orion Server Dashboard
Daily Walkthrough AM/PM VM Console
IP Monitor Int/Ext/DR IP Monitor Probation/Int
INMON HP SIM

30. Integrated ITSM Suite
Phase 4: BMC Suite through the Cloud
Results
Custom
Written

31. More Results

32. 100
200
300
400
500
600
700
800
0
100%
50%
70%
80%
90%
60%
Jan-09
Feb-09
Mar-09
Results
Apr-09
May-09
Jun-09
Jul-09
Aug-09
Sep-09
Oct-09
Nov-09
Dec-09
Jan-10
Feb-10
Mar-10
Apr-10
May-10
Jun-10
Incident Tickets
Percent Resolved
Jul-10
Aug-10
Sep-10
Oct-10
Nov-10
Evidence of Improvements
Dec-10
Jan-11
Feb-11
Mar-11
Apr-11
May-11
Jun-11
Jul-11
Aug-11
Sep-11

33. Evidence of Improvements
Results
SLA Exceptions Reduced by 77%
500
400
300
200
100
0
CY2009 CY2010

34. Evidence of Improvements
Results
Average Days to Resolve Tickets Reduced 75%
14
12
10
8
6
4
2
0
CY2009 CY2010

35. Evidence of Improvements
Results: HP’s Converged Infrastructure Maturity Model

36. Evidence of Improvements
Results: HP’s Converged Infrastructure Maturity Model

37. Evidence of Improvements
Results: HP’s Converged Infrastructure Maturity Model
Orange County versus US Public Sector Benchmark
July 2011
4.00
3.50
3.00 Orange County
US Public Sector
2.50
2.00
1.50
1.00
Technology & Tools & Processes Culture & IT Staff Demand, Supply & IT
Architecture Maturity Governance

38. Evidence of Improvements
Results: HP’s Converged Infrastructure Maturity Model
Orange County Percent of Characteristics "At or Better"
July 2011
Head
100
count
Top 10%
90
80
70 - 27%
Better
60
Equal
50
40
30
20
10
0
Dec-09 Jul-10 Jul-11

39. Evidence of Improvements
Customer Survey Scale: 1 = worst; 4 = best

40. Phase 5

41. Driving Continuous Improvements Phase 5
2011 Method
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
ITSM based
Reorganization
ITSM Integrated Suite Launch
– FULL SUITE
(CMDB, Change, Release…)
MSB Planning
Service Catalog
Plan
Success Customer Survey (Jul – Dec 10)
Implement
IT Service Management Path to Maturity
Absorb
AIMM (HP) Report 10 Service Level
Management
Network
Operations
Center Measure
Launched Application Monitoring
Virtual Server Farm Replacement

42. Service Catalog
Internet Service Provider
SERVICE DESCRIPTION
CEO/IT is the ISP for the County and its agencies. This service
includes reasonable and equitable Internet bandwidth and access HOW WE CHARGE
capability to and from the County backbone to the Internet. This service is included in the County Network Services
Rate. CEO/IT utilizes the total email address count from
WHAT IS INCLUDED each agency’s Active Directory (AD) as a divisor to calculate
This includes the installation, configuration, ongoing the cost distribution for CEO/IT Security Services,
maintenance and security of the core routers that are outward Enterprise Shared Services, and County Network
facing to the Internet Connectivity.
Provides redundant Internet providers configured in failover
configuration
64 Class C Networks that are available and managed by CEO/IT SERVICE TIPS
for existing outward facing applications Limit Internet usage to business related activities
Availability of an Enterprise Class, Tier 2 Data Center
Network Operations Center (NOC) (staffed hours: Mon-Fri,
8am-5pm)
Monitoring of Enterprise Components and Hosts on a 24x7x365 ADDITIONAL INFORMATION
basis For those who wish to filter their Internet browsing, we
Command Center staffed on a 24x7x365 basis offer a secure web filtering solution. See Secure Web
Enterprise Software Licensing and Contracts Management Filtering Services.
associated with this service
Enterprise Services

43. Extend ITSM Based Organization
Move the Plan & Design Group and the Build & Transition Group “up”

44. Conclusions and Lessons Learned

45. Conclusions & Lessons Learned
1. Financial distress clouds have their silver linings in IT environments.
a. Working together in the face of a single galvanizing event
b. Speak with “one voice”
2. It Takes an IT mojo-driven strategy to implement sustained continued improvements in
financial distress.
a. Courage and consistency; buy-in
b. Incremental changes (process, organizational), planning and soft skills provided results
3. New leadership can ignite IT Mojo.
a. Common goals
b. Common messages
4. Establish a drumbeat to pace the changes and measure visibly and consistently.
a. Messaging consistency
b. Allow the organization to digest changes
5. Establish key measurements of the categories shown in the customer surveys.
a. Report on them
b. Make sure everyone knows what’s being measured
6. Establish a consistent program of accountability .
a. Report on them
b. Make sure everyone knows what’s being measured
7. Benchmark the organization before commencing the journey.

46. Driving Continuous Improvements
2009 2010
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized:
-Plan & Design Customer Survey (Jul – Dec 09)
-Build & Transition
Training -Operate & Maintain
Incident Mgt Process ITSM Integrated Suite
Service Desk
SDP Process Customer Survey (Jan – Jun 10)
Capacity Reporting “A journey of a
Chaos CTO Portal
thousand miles begins
“One Team” with a single step” Certifications: 95
IT Service Management Path to Maturity
Marketing Brochure
“Agency Success is
Availability Reporting AIMM (HP) Report 46 Our Mission”
-Technology & Architecture AIMM (HP) Report 34
ACS Program -Mgt Tools & Process Maturity
-Culture & IT Staff Maturity
Executive hired Ticket Aging Process -Demand, Supply & IT Governance
Communication Team Formed
Resource Planning
Performance Audit Report
County Connection: OCDC
Project Prioritization Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Data Center Tour
(DISC) Not a single failed non-standard RFC
Tech Fest
Not a single Service Level Breach

47. Driving Continuous Improvements
2011 Method
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
ITSM based
Reorganization
ITSM Integrated Suite Launch
– FULL SUITE
(CMDB, Change, Release…)
MSB Planning
Service Catalog
Plan
Success Customer Survey (Jul – Dec 10)
Implement
IT Service Management Path to Maturity
Absorb
AIMM (HP) Report 10 Service Level
Management
Network
Operations
Center Measure
Launched Application Monitoring
Virtual Server Farm Replacement