More Related Content
Similar to Mortgage Stability 20120420
Similar to Mortgage Stability 20120420 (20)
More from Suzanne Goldsberry
More from Suzanne Goldsberry (12)
Mortgage Stability 20120420
- 2. 2 © 2010 PHH Corporation. All rights reserved.
Process Flow: Client Request to SOW (4/1 – 5/5)
Average Turn
Time: 8 Days
Average Turn
Time: 9 Days
Average Turn
Time: 2 Days
- 3. 3 © 2010 PHH Corporation. All rights reserved.
Client Funded Requests - KPIs
KPI Description YTD 3/31/11 Apr TD
Request Rating Turn time from client request to
rating (low, med, high). Rating
determines process deliverables.
Lowest Value = 1 day
Highest Value = 41 days
Average = 10 days
Sample Size = 58
Lowest Value = 5 day
Highest Value = 11 days
Average = 8 days
Sample Size = 3
Requirements
SOW
Turn time from Requirements SOW
requested to SOW prepared.
Involves getting work effort and
cost to gather and document
requirements and review with
internal PHH IT staff and client.
Lowest Value = 2 days
Highest Value = 60days
Average = 22 days
Sample Size = 8
Lowest Value = 2 days
Highest Value = 2 days
Average = 2 days
Sample Size = 1
Implementation
SOW
Turn time from Implementation
SOW requested to SOW
prepared. Involves getting work
effort and cost for analysis, design,
development, testing, management
and preparation of deliverables.
Lowest Value = 1 days
Highest Value = 49 days
Average = 14 days
Sample Size = 32
Lowest Value = 2 days
Highest Value = 26 days
Average = 9 days
Sample Size = 10
- 4. 4 © 2010 PHH Corporation. All rights reserved.
Client Funded Requests – KPIs - UBS
KPI Description YTD 3/31/11 Apr TD
Request Rating Turn time from client request to
rating (low, med, high). Rating
determines process deliverables.
Lowest Value = 2 day
Highest Value = 28 days
Average = 11 days
Sample Size = 6
Lowest Value = 5 day
Highest Value = 5 days
Average = 5 days
Sample Size = 1
Requirements
SOW
Turn time from Requirements SOW
requested to SOW prepared.
Involves getting work effort and
cost to gather and document
requirements and review with
internal PHH IT staff and client.
Lowest Value = 55 days
Highest Value = 57 days
Average = 56 days
Sample Size = 2
Lowest Value = ~
Highest Value = ~
Average = ~
Sample Size = 0
Implementation
SOW
Turn time from Implementation
SOW requested to SOW
prepared. Involves getting work
effort and cost for analysis, design,
development, testing, management
and preparation of deliverables.
Lowest Value = 1 days
Highest Value = 14 days
Average = 7 days
Sample Size = 6
Lowest Value = 6 days
Highest Value = 26 days
Average = 12 days
Sample Size = 4
- 5. 5 © 2010 PHH Corporation. All rights reserved.
99.20%
99.30%
99.40%
99.50%
99.60%
99.70%
99.80%
99.90%
100.00%
Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12
Percentage
Month
Client Availability
April Availability Summary
Key Takeaways
Critical Availability is 99.99%
M-F. 8:30am – 8:00pm
Planned Availability is 99.98%
21x7. Excluding 3am – 6am &
Thursday 10pm – 6am
95.00%
96.00%
97.00%
98.00%
99.00%
100.00%
Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12
Percentage
Month
Availability Trend (Critical)
Mortgage w/ Vendor Vendor Mortgage w/o Vendor
- 6. 6 © 2010 PHH Corporation. All rights reserved.
New Critical Incidents
Name Date Completed Next Steps
- -
No critical incidents
- 7. 7 © 2010 PHH Corporation. All rights reserved.
No Root Cause
Name Date Completed Next Steps
Fee Audit Calls 2/14 • Connection fails to Interthinx
• Transferring log files between both companies
• Met with Interthinxx management team
• Emergency procedures are in place and being executed in
under 5m
• Incident happens for 45m
• Ready to troubleshoot next time this occurs
Fee Audit Manual Failover 12/14 • Connection surge created performance bottleneck
• Secondary configuration went live on 2/3
• Expanded scope to include multiple applications
• Build database by 4/13
• Replicate failover 5/4
Net New Money & Verification of
Deposit Release
12/16 • Web server required a restart
• Submitted ticket to Oracle
• Updated operating procedure with server restar
• Replicated issue with Oracle.
• Working with Oracle to re-produce this problem in a non-prod
environment.
• We still can’t re-produce the problem in a non-prod
environment. Similar problem is re-produced in a non-prod
with a slight different configuration and Oracle support could
not confirm whether or it is bug. A temporary deployment
procedure is place to work around this problem.
- 8. 8 © 2010 PHH Corporation. All rights reserved.
Root Cause Determined
Name Date Completed Next Steps
General External System Slowness 4/5 • Determined BG1 data circuit over utilized
• Demonstrated control to Morgan Stanley
• Routed web traffic through Sparks
• Hourly internal updates
• Morgan Stanley two-a-days
• Complete two-a-days with MS on 4/16
• Acquire 300Mb circuit within 120 days
• Lessons learned with networking team by 4/27
Various PCs throughout BG1 – slow
access to network, PC appears to
hang.
3/28 • Caused by the Cluster Node FSBG101D hanging up
sporadically.
• Troubleshooting with Microsoft on 4/2
• Apply Microsoft recommended fixes to include increasing
worker threads and updating NIC driver – 4/20
REBO Domain delayed email
delivery
3/9 • Subsidiaries impacted
• Caused by off-shore developer sending a high per minute
volume of test emails.
• Advised to use non-journal service email accounts for
future testing
• Messaging to determine email volume requirements of
service accounts – 4/30
• Exchange monitoring – 5/18
• Implement email throttling to avoid email bottlenecks – 5/25
Identity Management Ongoing • LoanSpan Originator Failover Successful
• Cash & iDesk failover successful
• DocViewer failover successful
• “What Else?” meeting with IDM team by 4/20
Hyperion Outages Ongoing • Health Assessment Complete • Present options to MTSC on 4/20
• Apply minor release upgrade by 8/31
35k Servicing Emails & Servicing
Notifications Delayed
8/16 • Presented monitoring scenarios to IT SLT on 3/13 • Review TIBCO message types and current monitoring by 5/4
QRM • Implemented backup/redundant site to process scripts
• Performance improvement verified
• Updated SLAs with average processing time
• Insert ability to recover in SLAs
• Deploy monitoring by 5/31
IDM iManager sync issues 4/2 • Delays when resetting passwords
• Troubleshooting with Novell, they advised to increase
disk space in VM servers
• Determine IDM system administration ownership by 4/27
• Determine next steps by 4/20
Asset pull pub is missing form
production
3/28 • Archive process removed a pub file needed for printing • Update script to not archive pub file by 4/20
- 9. 9 © 2010 PHH Corporation. All rights reserved.
Critical Availability
Applications < 100%
Vendors < 100%
Vendor KPI Description Time Period Target Actual Tolerance Left Tolerance Used
SourceCorp Critical Availability M-F 8:30AM-8PM ≥ 99.00% 96.56% -522.1 736
Speedy Title and
Appraisal
Critical Availability M-F 8:30AM-8PM ≥ 99.00% 99.50% 107.9 106
Application KPI Description Time Period Target Actual Tolerance Left Tolerance Used
Rightfax Critical Availability M-F 8:30AM-8PM ≥ 99.00% 99.21% 31.8 120
Conditions
Management
Critical Availability M-F 8:30AM-8PM ≥ 99.00% 99.89% 134.8 17
- 10. 10 © 2010 PHH Corporation. All rights reserved.
Planned Availability
Applications < 100%
Application KPI Description Time Period Target Actual Tolerance Left Tolerance Used
iDesk Planned Availability 21x7 ≥ 99.00% 99.57% 86.8 65
DocViewer Planned Availability 21x7 ≥ 99.00% 99.57% 86.8 65
Rightfax Planned Availability 21x7 ≥ 99.00% 99.21% 31.8 120
FSMS Planned Availability 21x7 ≥ 99.00% 99.57% 86.8 65
Conditions Management Planned Availability 21x7 ≥ 99.00% 99.89% 134.8 17
- 11. 11 © 2010 PHH Corporation. All rights reserved.
Improve Availability, Resiliency & Performance
Improve Availability, Resiliency & Performance