SlideShare a Scribd company logo
1 of 11
Improve Availability,
Resiliency & Performance
April 20, 2012
2 © 2010 PHH Corporation. All rights reserved.
Process Flow: Client Request to SOW (4/1 – 5/5)
Average Turn
Time: 8 Days
Average Turn
Time: 9 Days
Average Turn
Time: 2 Days
3 © 2010 PHH Corporation. All rights reserved.
Client Funded Requests - KPIs
KPI Description YTD 3/31/11 Apr TD
Request Rating Turn time from client request to
rating (low, med, high). Rating
determines process deliverables.
Lowest Value = 1 day
Highest Value = 41 days
Average = 10 days
Sample Size = 58
Lowest Value = 5 day
Highest Value = 11 days
Average = 8 days
Sample Size = 3
Requirements
SOW
Turn time from Requirements SOW
requested to SOW prepared.
Involves getting work effort and
cost to gather and document
requirements and review with
internal PHH IT staff and client.
Lowest Value = 2 days
Highest Value = 60days
Average = 22 days
Sample Size = 8
Lowest Value = 2 days
Highest Value = 2 days
Average = 2 days
Sample Size = 1
Implementation
SOW
Turn time from Implementation
SOW requested to SOW
prepared. Involves getting work
effort and cost for analysis, design,
development, testing, management
and preparation of deliverables.
Lowest Value = 1 days
Highest Value = 49 days
Average = 14 days
Sample Size = 32
Lowest Value = 2 days
Highest Value = 26 days
Average = 9 days
Sample Size = 10
4 © 2010 PHH Corporation. All rights reserved.
Client Funded Requests – KPIs - UBS
KPI Description YTD 3/31/11 Apr TD
Request Rating Turn time from client request to
rating (low, med, high). Rating
determines process deliverables.
Lowest Value = 2 day
Highest Value = 28 days
Average = 11 days
Sample Size = 6
Lowest Value = 5 day
Highest Value = 5 days
Average = 5 days
Sample Size = 1
Requirements
SOW
Turn time from Requirements SOW
requested to SOW prepared.
Involves getting work effort and
cost to gather and document
requirements and review with
internal PHH IT staff and client.
Lowest Value = 55 days
Highest Value = 57 days
Average = 56 days
Sample Size = 2
Lowest Value = ~
Highest Value = ~
Average = ~
Sample Size = 0
Implementation
SOW
Turn time from Implementation
SOW requested to SOW
prepared. Involves getting work
effort and cost for analysis, design,
development, testing, management
and preparation of deliverables.
Lowest Value = 1 days
Highest Value = 14 days
Average = 7 days
Sample Size = 6
Lowest Value = 6 days
Highest Value = 26 days
Average = 12 days
Sample Size = 4
5 © 2010 PHH Corporation. All rights reserved.
99.20%
99.30%
99.40%
99.50%
99.60%
99.70%
99.80%
99.90%
100.00%
Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12
Percentage
Month
Client Availability
April Availability Summary
Key Takeaways
Critical Availability is 99.99%
M-F. 8:30am – 8:00pm
Planned Availability is 99.98%
21x7. Excluding 3am – 6am &
Thursday 10pm – 6am
95.00%
96.00%
97.00%
98.00%
99.00%
100.00%
Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12
Percentage
Month
Availability Trend (Critical)
Mortgage w/ Vendor Vendor Mortgage w/o Vendor
6 © 2010 PHH Corporation. All rights reserved.
New Critical Incidents
Name Date Completed Next Steps
- -
No critical incidents
7 © 2010 PHH Corporation. All rights reserved.
No Root Cause
Name Date Completed Next Steps
Fee Audit Calls 2/14 • Connection fails to Interthinx
• Transferring log files between both companies
• Met with Interthinxx management team
• Emergency procedures are in place and being executed in
under 5m
• Incident happens for 45m
• Ready to troubleshoot next time this occurs
Fee Audit Manual Failover 12/14 • Connection surge created performance bottleneck
• Secondary configuration went live on 2/3
• Expanded scope to include multiple applications
• Build database by 4/13
• Replicate failover 5/4
Net New Money & Verification of
Deposit Release
12/16 • Web server required a restart
• Submitted ticket to Oracle
• Updated operating procedure with server restar
• Replicated issue with Oracle.
• Working with Oracle to re-produce this problem in a non-prod
environment.
• We still can’t re-produce the problem in a non-prod
environment. Similar problem is re-produced in a non-prod
with a slight different configuration and Oracle support could
not confirm whether or it is bug. A temporary deployment
procedure is place to work around this problem.
8 © 2010 PHH Corporation. All rights reserved.
Root Cause Determined
Name Date Completed Next Steps
General External System Slowness 4/5 • Determined BG1 data circuit over utilized
• Demonstrated control to Morgan Stanley
• Routed web traffic through Sparks
• Hourly internal updates
• Morgan Stanley two-a-days
• Complete two-a-days with MS on 4/16
• Acquire 300Mb circuit within 120 days
• Lessons learned with networking team by 4/27
Various PCs throughout BG1 – slow
access to network, PC appears to
hang.
3/28 • Caused by the Cluster Node FSBG101D hanging up
sporadically.
• Troubleshooting with Microsoft on 4/2
• Apply Microsoft recommended fixes to include increasing
worker threads and updating NIC driver – 4/20
REBO Domain delayed email
delivery
3/9 • Subsidiaries impacted
• Caused by off-shore developer sending a high per minute
volume of test emails.
• Advised to use non-journal service email accounts for
future testing
• Messaging to determine email volume requirements of
service accounts – 4/30
• Exchange monitoring – 5/18
• Implement email throttling to avoid email bottlenecks – 5/25
Identity Management Ongoing • LoanSpan Originator Failover Successful
• Cash & iDesk failover successful
• DocViewer failover successful
• “What Else?” meeting with IDM team by 4/20
Hyperion Outages Ongoing • Health Assessment Complete • Present options to MTSC on 4/20
• Apply minor release upgrade by 8/31
35k Servicing Emails & Servicing
Notifications Delayed
8/16 • Presented monitoring scenarios to IT SLT on 3/13 • Review TIBCO message types and current monitoring by 5/4
QRM • Implemented backup/redundant site to process scripts
• Performance improvement verified
• Updated SLAs with average processing time
• Insert ability to recover in SLAs
• Deploy monitoring by 5/31
IDM iManager sync issues 4/2 • Delays when resetting passwords
• Troubleshooting with Novell, they advised to increase
disk space in VM servers
• Determine IDM system administration ownership by 4/27
• Determine next steps by 4/20
Asset pull pub is missing form
production
3/28 • Archive process removed a pub file needed for printing • Update script to not archive pub file by 4/20
9 © 2010 PHH Corporation. All rights reserved.
Critical Availability
Applications < 100%
Vendors < 100%
Vendor KPI Description Time Period Target Actual Tolerance Left Tolerance Used
SourceCorp Critical Availability M-F 8:30AM-8PM ≥ 99.00% 96.56% -522.1 736
Speedy Title and
Appraisal
Critical Availability M-F 8:30AM-8PM ≥ 99.00% 99.50% 107.9 106
Application KPI Description Time Period Target Actual Tolerance Left Tolerance Used
Rightfax Critical Availability M-F 8:30AM-8PM ≥ 99.00% 99.21% 31.8 120
Conditions
Management
Critical Availability M-F 8:30AM-8PM ≥ 99.00% 99.89% 134.8 17
10 © 2010 PHH Corporation. All rights reserved.
Planned Availability
Applications < 100%
Application KPI Description Time Period Target Actual Tolerance Left Tolerance Used
iDesk Planned Availability 21x7 ≥ 99.00% 99.57% 86.8 65
DocViewer Planned Availability 21x7 ≥ 99.00% 99.57% 86.8 65
Rightfax Planned Availability 21x7 ≥ 99.00% 99.21% 31.8 120
FSMS Planned Availability 21x7 ≥ 99.00% 99.57% 86.8 65
Conditions Management Planned Availability 21x7 ≥ 99.00% 99.89% 134.8 17
11 © 2010 PHH Corporation. All rights reserved.
Improve Availability, Resiliency & Performance
Improve Availability, Resiliency & Performance

More Related Content

What's hot

PROJECT STORYBOARD: Reducing Learning Curve Ramp for Temp Employees by 2 Weeks
PROJECT STORYBOARD: Reducing Learning Curve Ramp for Temp Employees by 2 WeeksPROJECT STORYBOARD: Reducing Learning Curve Ramp for Temp Employees by 2 Weeks
PROJECT STORYBOARD: Reducing Learning Curve Ramp for Temp Employees by 2 WeeksGoLeanSixSigma.com
 
Agile Aspects of Performance Testing
Agile Aspects of Performance TestingAgile Aspects of Performance Testing
Agile Aspects of Performance TestingAlexander Podelko
 
The changing role of a QA | QualiTest Group
The changing role of a QA | QualiTest GroupThe changing role of a QA | QualiTest Group
The changing role of a QA | QualiTest GroupQualitest
 
Performance On Demand
Performance On DemandPerformance On Demand
Performance On DemandFranky Smit
 
Testing as a Managed Service using SLAs and KPIs
Testing as a Managed Service using SLAs and KPIsTesting as a Managed Service using SLAs and KPIs
Testing as a Managed Service using SLAs and KPIsProlifics
 
Kent Dyer Resume
Kent Dyer ResumeKent Dyer Resume
Kent Dyer ResumeKent Dyer
 
SUCCESS STORY: Reducing Provider Payment Process Rework From 25% to 5%
SUCCESS STORY: Reducing Provider Payment Process Rework From 25% to 5%SUCCESS STORY: Reducing Provider Payment Process Rework From 25% to 5%
SUCCESS STORY: Reducing Provider Payment Process Rework From 25% to 5%GoLeanSixSigma.com
 
T19 performance testing effort - estimation or guesstimation revised
T19   performance testing effort - estimation or guesstimation revisedT19   performance testing effort - estimation or guesstimation revised
T19 performance testing effort - estimation or guesstimation revisedTEST Huddle
 
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference SheetBig Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference SheetJason Tice
 
Personnel Productivity System - Exec Pres
Personnel Productivity System - Exec PresPersonnel Productivity System - Exec Pres
Personnel Productivity System - Exec PresThomas Bronack
 
Utilizing Dashboards to improve efficiency
Utilizing Dashboards to improve efficiencyUtilizing Dashboards to improve efficiency
Utilizing Dashboards to improve efficiencyThomas Bronack
 
Dcag training on VMware DR Process
Dcag training on VMware DR ProcessDcag training on VMware DR Process
Dcag training on VMware DR ProcessThomas Bronack
 
Optimizing the it and business environment through dashboards
Optimizing the it and business environment through dashboardsOptimizing the it and business environment through dashboards
Optimizing the it and business environment through dashboardsThomas Bronack
 
Software QA Metrics Dashboard Benchmarking
Software QA Metrics Dashboard BenchmarkingSoftware QA Metrics Dashboard Benchmarking
Software QA Metrics Dashboard BenchmarkingJohn Carter
 
Puppet Camp San Francisco 2015: Puppet Adoption in a Mature Environment
Puppet Camp San Francisco 2015: Puppet Adoption in a Mature EnvironmentPuppet Camp San Francisco 2015: Puppet Adoption in a Mature Environment
Puppet Camp San Francisco 2015: Puppet Adoption in a Mature EnvironmentPuppet
 
Test case-point-analysis (whitepaper)
Test case-point-analysis (whitepaper)Test case-point-analysis (whitepaper)
Test case-point-analysis (whitepaper)KMS Technology
 

What's hot (20)

PROJECT STORYBOARD: Reducing Learning Curve Ramp for Temp Employees by 2 Weeks
PROJECT STORYBOARD: Reducing Learning Curve Ramp for Temp Employees by 2 WeeksPROJECT STORYBOARD: Reducing Learning Curve Ramp for Temp Employees by 2 Weeks
PROJECT STORYBOARD: Reducing Learning Curve Ramp for Temp Employees by 2 Weeks
 
Agile Aspects of Performance Testing
Agile Aspects of Performance TestingAgile Aspects of Performance Testing
Agile Aspects of Performance Testing
 
The changing role of a QA | QualiTest Group
The changing role of a QA | QualiTest GroupThe changing role of a QA | QualiTest Group
The changing role of a QA | QualiTest Group
 
Performance On Demand
Performance On DemandPerformance On Demand
Performance On Demand
 
Testing as a Managed Service using SLAs and KPIs
Testing as a Managed Service using SLAs and KPIsTesting as a Managed Service using SLAs and KPIs
Testing as a Managed Service using SLAs and KPIs
 
Kent Dyer Resume
Kent Dyer ResumeKent Dyer Resume
Kent Dyer Resume
 
SUCCESS STORY: Reducing Provider Payment Process Rework From 25% to 5%
SUCCESS STORY: Reducing Provider Payment Process Rework From 25% to 5%SUCCESS STORY: Reducing Provider Payment Process Rework From 25% to 5%
SUCCESS STORY: Reducing Provider Payment Process Rework From 25% to 5%
 
T19 performance testing effort - estimation or guesstimation revised
T19   performance testing effort - estimation or guesstimation revisedT19   performance testing effort - estimation or guesstimation revised
T19 performance testing effort - estimation or guesstimation revised
 
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference SheetBig Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
 
Personnel Productivity System - Exec Pres
Personnel Productivity System - Exec PresPersonnel Productivity System - Exec Pres
Personnel Productivity System - Exec Pres
 
PEnDAR webinar 2 with notes
PEnDAR webinar 2 with notesPEnDAR webinar 2 with notes
PEnDAR webinar 2 with notes
 
Utilizing Dashboards to improve efficiency
Utilizing Dashboards to improve efficiencyUtilizing Dashboards to improve efficiency
Utilizing Dashboards to improve efficiency
 
Dcag training on VMware DR Process
Dcag training on VMware DR ProcessDcag training on VMware DR Process
Dcag training on VMware DR Process
 
Optimizing the it and business environment through dashboards
Optimizing the it and business environment through dashboardsOptimizing the it and business environment through dashboards
Optimizing the it and business environment through dashboards
 
PEnDAR: software v&v for complex systems
PEnDAR: software v&v for complex systemsPEnDAR: software v&v for complex systems
PEnDAR: software v&v for complex systems
 
Software QA Metrics Dashboard Benchmarking
Software QA Metrics Dashboard BenchmarkingSoftware QA Metrics Dashboard Benchmarking
Software QA Metrics Dashboard Benchmarking
 
PEnDAR webinar 1 with notes
PEnDAR webinar 1 with notesPEnDAR webinar 1 with notes
PEnDAR webinar 1 with notes
 
Puppet Camp San Francisco 2015: Puppet Adoption in a Mature Environment
Puppet Camp San Francisco 2015: Puppet Adoption in a Mature EnvironmentPuppet Camp San Francisco 2015: Puppet Adoption in a Mature Environment
Puppet Camp San Francisco 2015: Puppet Adoption in a Mature Environment
 
Test case-point-analysis (whitepaper)
Test case-point-analysis (whitepaper)Test case-point-analysis (whitepaper)
Test case-point-analysis (whitepaper)
 
performance testing
performance testingperformance testing
performance testing
 

Similar to Mortgage Stability 20120420

Sivasubramani_N_Testing
Sivasubramani_N_TestingSivasubramani_N_Testing
Sivasubramani_N_Testingnsivamsit
 
ATAGTR2017 Batch Workload Modelling and Performance Optimization
ATAGTR2017 Batch Workload Modelling and Performance Optimization ATAGTR2017 Batch Workload Modelling and Performance Optimization
ATAGTR2017 Batch Workload Modelling and Performance Optimization Agile Testing Alliance
 
Ashok Poreddy_IBM_ Mainframes_8yrs
Ashok Poreddy_IBM_ Mainframes_8yrsAshok Poreddy_IBM_ Mainframes_8yrs
Ashok Poreddy_IBM_ Mainframes_8yrsAshok Reddy
 
Patrick O. Madden Resume 2
Patrick O. Madden Resume 2Patrick O. Madden Resume 2
Patrick O. Madden Resume 2Patrick Madden
 
Using Lean Thinking to identify and address Delivery Pipeline bottlenecks
Using Lean Thinking to identify and address Delivery Pipeline bottlenecksUsing Lean Thinking to identify and address Delivery Pipeline bottlenecks
Using Lean Thinking to identify and address Delivery Pipeline bottlenecksSanjeev Sharma
 
Archana_Yadav_Resume
Archana_Yadav_ResumeArchana_Yadav_Resume
Archana_Yadav_Resumearchana yadav
 
Archana_Yadav_Resume
Archana_Yadav_ResumeArchana_Yadav_Resume
Archana_Yadav_Resumearchana yadav
 
DevopsBusinessCaseTemplate
DevopsBusinessCaseTemplateDevopsBusinessCaseTemplate
DevopsBusinessCaseTemplatePeter Lamar
 
Discovering Lean at Hewlett Packard Laserjet Division
Discovering Lean at Hewlett Packard Laserjet DivisionDiscovering Lean at Hewlett Packard Laserjet Division
Discovering Lean at Hewlett Packard Laserjet DivisionIrina Dzhambazova
 
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...ssuser835d1a
 
What is Robotics Process Automation ?
What is Robotics Process Automation ?What is Robotics Process Automation ?
What is Robotics Process Automation ?Aditya Sharma
 
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...VMworld
 
Value Driven Development by Dave Thomas
Value Driven Development by Dave Thomas Value Driven Development by Dave Thomas
Value Driven Development by Dave Thomas Naresh Jain
 
Upgrade JDE Quicker, Faster, and More Predictable
Upgrade JDE Quicker, Faster, and More PredictableUpgrade JDE Quicker, Faster, and More Predictable
Upgrade JDE Quicker, Faster, and More PredictableTerillium
 
Resume of Mohamed Sajid Maqba_2016
Resume of Mohamed Sajid Maqba_2016Resume of Mohamed Sajid Maqba_2016
Resume of Mohamed Sajid Maqba_2016Mohamed Sajid Maqba
 

Similar to Mortgage Stability 20120420 (20)

TR.Bhavani_Inf
TR.Bhavani_InfTR.Bhavani_Inf
TR.Bhavani_Inf
 
Sivasubramani_N_Testing
Sivasubramani_N_TestingSivasubramani_N_Testing
Sivasubramani_N_Testing
 
ATAGTR2017 Batch Workload Modelling and Performance Optimization
ATAGTR2017 Batch Workload Modelling and Performance Optimization ATAGTR2017 Batch Workload Modelling and Performance Optimization
ATAGTR2017 Batch Workload Modelling and Performance Optimization
 
Ashok Poreddy_IBM_ Mainframes_8yrs
Ashok Poreddy_IBM_ Mainframes_8yrsAshok Poreddy_IBM_ Mainframes_8yrs
Ashok Poreddy_IBM_ Mainframes_8yrs
 
Patrick O. Madden Resume 2
Patrick O. Madden Resume 2Patrick O. Madden Resume 2
Patrick O. Madden Resume 2
 
Using Lean Thinking to identify and address Delivery Pipeline bottlenecks
Using Lean Thinking to identify and address Delivery Pipeline bottlenecksUsing Lean Thinking to identify and address Delivery Pipeline bottlenecks
Using Lean Thinking to identify and address Delivery Pipeline bottlenecks
 
Archana_Yadav_Resume
Archana_Yadav_ResumeArchana_Yadav_Resume
Archana_Yadav_Resume
 
Archana_Yadav_Resume
Archana_Yadav_ResumeArchana_Yadav_Resume
Archana_Yadav_Resume
 
nitesh_rajpurkar_2016
nitesh_rajpurkar_2016nitesh_rajpurkar_2016
nitesh_rajpurkar_2016
 
Rajesh Paleru
Rajesh PaleruRajesh Paleru
Rajesh Paleru
 
Dipti Rekha Mahanta
Dipti Rekha MahantaDipti Rekha Mahanta
Dipti Rekha Mahanta
 
DevopsBusinessCaseTemplate
DevopsBusinessCaseTemplateDevopsBusinessCaseTemplate
DevopsBusinessCaseTemplate
 
Discovering Lean at Hewlett Packard Laserjet Division
Discovering Lean at Hewlett Packard Laserjet DivisionDiscovering Lean at Hewlett Packard Laserjet Division
Discovering Lean at Hewlett Packard Laserjet Division
 
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
 
What is Robotics Process Automation ?
What is Robotics Process Automation ?What is Robotics Process Automation ?
What is Robotics Process Automation ?
 
Fundamentals Performance Testing
Fundamentals Performance TestingFundamentals Performance Testing
Fundamentals Performance Testing
 
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...
 
Value Driven Development by Dave Thomas
Value Driven Development by Dave Thomas Value Driven Development by Dave Thomas
Value Driven Development by Dave Thomas
 
Upgrade JDE Quicker, Faster, and More Predictable
Upgrade JDE Quicker, Faster, and More PredictableUpgrade JDE Quicker, Faster, and More Predictable
Upgrade JDE Quicker, Faster, and More Predictable
 
Resume of Mohamed Sajid Maqba_2016
Resume of Mohamed Sajid Maqba_2016Resume of Mohamed Sajid Maqba_2016
Resume of Mohamed Sajid Maqba_2016
 

More from Suzanne Goldsberry

More from Suzanne Goldsberry (12)

PROPERTY PROFILE TEMPLATE-JAX
PROPERTY PROFILE TEMPLATE-JAXPROPERTY PROFILE TEMPLATE-JAX
PROPERTY PROFILE TEMPLATE-JAX
 
List of duties at PHH
List of duties at PHHList of duties at PHH
List of duties at PHH
 
Most current Word test results
Most current Word test resultsMost current Word test results
Most current Word test results
 
Most recent Outlook testing results
Most recent Outlook testing resultsMost recent Outlook testing results
Most recent Outlook testing results
 
Most recent Excel test results
Most recent Excel test resultsMost recent Excel test results
Most recent Excel test results
 
THANK YOU FROM MARK SHAW
THANK YOU FROM MARK SHAWTHANK YOU FROM MARK SHAW
THANK YOU FROM MARK SHAW
 
LOOP PROPOSAL
LOOP PROPOSALLOOP PROPOSAL
LOOP PROPOSAL
 
UES LUMP SUM PROPOSAL-PDF
UES LUMP SUM PROPOSAL-PDFUES LUMP SUM PROPOSAL-PDF
UES LUMP SUM PROPOSAL-PDF
 
Jacksonville OpsReport 6-28-2014
Jacksonville OpsReport 6-28-2014Jacksonville OpsReport 6-28-2014
Jacksonville OpsReport 6-28-2014
 
Final PP 7-2014 - RP Meeting 7-23-14 final
Final PP 7-2014 - RP Meeting 7-23-14 finalFinal PP 7-2014 - RP Meeting 7-23-14 final
Final PP 7-2014 - RP Meeting 7-23-14 final
 
Letter of recommendation - 2
Letter of recommendation - 2Letter of recommendation - 2
Letter of recommendation - 2
 
S Goldsberry - 2013 Performance Assessment - IH
S Goldsberry - 2013 Performance Assessment - IHS Goldsberry - 2013 Performance Assessment - IH
S Goldsberry - 2013 Performance Assessment - IH
 

Mortgage Stability 20120420

  • 1. Improve Availability, Resiliency & Performance April 20, 2012
  • 2. 2 © 2010 PHH Corporation. All rights reserved. Process Flow: Client Request to SOW (4/1 – 5/5) Average Turn Time: 8 Days Average Turn Time: 9 Days Average Turn Time: 2 Days
  • 3. 3 © 2010 PHH Corporation. All rights reserved. Client Funded Requests - KPIs KPI Description YTD 3/31/11 Apr TD Request Rating Turn time from client request to rating (low, med, high). Rating determines process deliverables. Lowest Value = 1 day Highest Value = 41 days Average = 10 days Sample Size = 58 Lowest Value = 5 day Highest Value = 11 days Average = 8 days Sample Size = 3 Requirements SOW Turn time from Requirements SOW requested to SOW prepared. Involves getting work effort and cost to gather and document requirements and review with internal PHH IT staff and client. Lowest Value = 2 days Highest Value = 60days Average = 22 days Sample Size = 8 Lowest Value = 2 days Highest Value = 2 days Average = 2 days Sample Size = 1 Implementation SOW Turn time from Implementation SOW requested to SOW prepared. Involves getting work effort and cost for analysis, design, development, testing, management and preparation of deliverables. Lowest Value = 1 days Highest Value = 49 days Average = 14 days Sample Size = 32 Lowest Value = 2 days Highest Value = 26 days Average = 9 days Sample Size = 10
  • 4. 4 © 2010 PHH Corporation. All rights reserved. Client Funded Requests – KPIs - UBS KPI Description YTD 3/31/11 Apr TD Request Rating Turn time from client request to rating (low, med, high). Rating determines process deliverables. Lowest Value = 2 day Highest Value = 28 days Average = 11 days Sample Size = 6 Lowest Value = 5 day Highest Value = 5 days Average = 5 days Sample Size = 1 Requirements SOW Turn time from Requirements SOW requested to SOW prepared. Involves getting work effort and cost to gather and document requirements and review with internal PHH IT staff and client. Lowest Value = 55 days Highest Value = 57 days Average = 56 days Sample Size = 2 Lowest Value = ~ Highest Value = ~ Average = ~ Sample Size = 0 Implementation SOW Turn time from Implementation SOW requested to SOW prepared. Involves getting work effort and cost for analysis, design, development, testing, management and preparation of deliverables. Lowest Value = 1 days Highest Value = 14 days Average = 7 days Sample Size = 6 Lowest Value = 6 days Highest Value = 26 days Average = 12 days Sample Size = 4
  • 5. 5 © 2010 PHH Corporation. All rights reserved. 99.20% 99.30% 99.40% 99.50% 99.60% 99.70% 99.80% 99.90% 100.00% Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 Percentage Month Client Availability April Availability Summary Key Takeaways Critical Availability is 99.99% M-F. 8:30am – 8:00pm Planned Availability is 99.98% 21x7. Excluding 3am – 6am & Thursday 10pm – 6am 95.00% 96.00% 97.00% 98.00% 99.00% 100.00% Oct-11 Nov-11 Dec-11 Jan-12 Feb-12 Mar-12 Apr-12 Percentage Month Availability Trend (Critical) Mortgage w/ Vendor Vendor Mortgage w/o Vendor
  • 6. 6 © 2010 PHH Corporation. All rights reserved. New Critical Incidents Name Date Completed Next Steps - - No critical incidents
  • 7. 7 © 2010 PHH Corporation. All rights reserved. No Root Cause Name Date Completed Next Steps Fee Audit Calls 2/14 • Connection fails to Interthinx • Transferring log files between both companies • Met with Interthinxx management team • Emergency procedures are in place and being executed in under 5m • Incident happens for 45m • Ready to troubleshoot next time this occurs Fee Audit Manual Failover 12/14 • Connection surge created performance bottleneck • Secondary configuration went live on 2/3 • Expanded scope to include multiple applications • Build database by 4/13 • Replicate failover 5/4 Net New Money & Verification of Deposit Release 12/16 • Web server required a restart • Submitted ticket to Oracle • Updated operating procedure with server restar • Replicated issue with Oracle. • Working with Oracle to re-produce this problem in a non-prod environment. • We still can’t re-produce the problem in a non-prod environment. Similar problem is re-produced in a non-prod with a slight different configuration and Oracle support could not confirm whether or it is bug. A temporary deployment procedure is place to work around this problem.
  • 8. 8 © 2010 PHH Corporation. All rights reserved. Root Cause Determined Name Date Completed Next Steps General External System Slowness 4/5 • Determined BG1 data circuit over utilized • Demonstrated control to Morgan Stanley • Routed web traffic through Sparks • Hourly internal updates • Morgan Stanley two-a-days • Complete two-a-days with MS on 4/16 • Acquire 300Mb circuit within 120 days • Lessons learned with networking team by 4/27 Various PCs throughout BG1 – slow access to network, PC appears to hang. 3/28 • Caused by the Cluster Node FSBG101D hanging up sporadically. • Troubleshooting with Microsoft on 4/2 • Apply Microsoft recommended fixes to include increasing worker threads and updating NIC driver – 4/20 REBO Domain delayed email delivery 3/9 • Subsidiaries impacted • Caused by off-shore developer sending a high per minute volume of test emails. • Advised to use non-journal service email accounts for future testing • Messaging to determine email volume requirements of service accounts – 4/30 • Exchange monitoring – 5/18 • Implement email throttling to avoid email bottlenecks – 5/25 Identity Management Ongoing • LoanSpan Originator Failover Successful • Cash & iDesk failover successful • DocViewer failover successful • “What Else?” meeting with IDM team by 4/20 Hyperion Outages Ongoing • Health Assessment Complete • Present options to MTSC on 4/20 • Apply minor release upgrade by 8/31 35k Servicing Emails & Servicing Notifications Delayed 8/16 • Presented monitoring scenarios to IT SLT on 3/13 • Review TIBCO message types and current monitoring by 5/4 QRM • Implemented backup/redundant site to process scripts • Performance improvement verified • Updated SLAs with average processing time • Insert ability to recover in SLAs • Deploy monitoring by 5/31 IDM iManager sync issues 4/2 • Delays when resetting passwords • Troubleshooting with Novell, they advised to increase disk space in VM servers • Determine IDM system administration ownership by 4/27 • Determine next steps by 4/20 Asset pull pub is missing form production 3/28 • Archive process removed a pub file needed for printing • Update script to not archive pub file by 4/20
  • 9. 9 © 2010 PHH Corporation. All rights reserved. Critical Availability Applications < 100% Vendors < 100% Vendor KPI Description Time Period Target Actual Tolerance Left Tolerance Used SourceCorp Critical Availability M-F 8:30AM-8PM ≥ 99.00% 96.56% -522.1 736 Speedy Title and Appraisal Critical Availability M-F 8:30AM-8PM ≥ 99.00% 99.50% 107.9 106 Application KPI Description Time Period Target Actual Tolerance Left Tolerance Used Rightfax Critical Availability M-F 8:30AM-8PM ≥ 99.00% 99.21% 31.8 120 Conditions Management Critical Availability M-F 8:30AM-8PM ≥ 99.00% 99.89% 134.8 17
  • 10. 10 © 2010 PHH Corporation. All rights reserved. Planned Availability Applications < 100% Application KPI Description Time Period Target Actual Tolerance Left Tolerance Used iDesk Planned Availability 21x7 ≥ 99.00% 99.57% 86.8 65 DocViewer Planned Availability 21x7 ≥ 99.00% 99.57% 86.8 65 Rightfax Planned Availability 21x7 ≥ 99.00% 99.21% 31.8 120 FSMS Planned Availability 21x7 ≥ 99.00% 99.57% 86.8 65 Conditions Management Planned Availability 21x7 ≥ 99.00% 99.89% 134.8 17
  • 11. 11 © 2010 PHH Corporation. All rights reserved. Improve Availability, Resiliency & Performance Improve Availability, Resiliency & Performance