IT Outage & Continuity AlertingPowered by AtHocChris Petteruti, Director of Technology, Kaiser PermanenteDiablo Service Ar...
Kaiser Northern California Region 21 Hospitals 70 Medical Office Buildings 3.35 Million members, > 35%of Kaiser members...
3 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSystem...
4 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesPowered by AtHoc ...
5 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundPowered...
6 5/8/2013IT Outage & Continuity ofOperations Communication
7 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSystem...
8 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts System OverviewHQ2 Way Notificat...
6 Month Pilot Period New Install Packed Desktop Client AD Integration Lab Incident Joint Commission & other Surveyors...
10 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP AlertsOne System, Multiple Uses IT O...
USE CASE:Critical Outage Impacting all of NCAL Incident: Primary EHR system down–Date: 12/24/12–Start time: 06:30 AM–Reso...
Diablo Response using KP Alerts:– Sent 12,000 Desktop Alerts in our Diablo Service Area– All linked to ALTERNATE logon pag...
14 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSyste...
Unexpected Downtime – Policy & Procedure15 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
Unexpected Downtime – Policy & Procedure16 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
Unexpected Downtime – Policy & Procedure17 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
AtHoc – Day Light Savings Alert18 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013Scheduled ...
AtHoc – Planned Rilis Outage19 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013Multiple-choi...
KP Alerts System - Enterprise Approach Centralized enterprise deployment servicing all locations– Enterprise-wide system ...
21 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSyste...
Future Plans – Smart Phone Use Expand alerts to Mobile app Mobile Duress button withlocation information forHome Health ...
23 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013Future Plans - Additional UsesOne System...
24 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013Future Plans - Additional UsesOne System...
25 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSyste...
iHT2 Health IT Summit Boston 2013 – Chris Petteruti, Director of Technology Department, Kaiser Permanente – NCAL Case Stud...
Upcoming SlideShare
Loading in …5
×

iHT2 Health IT Summit Boston 2013 – Chris Petteruti, Director of Technology Department, Kaiser Permanente – NCAL Case Study "Strategies for Minimizing Uncertainty and Downtime”

375 views

Published on

iHT2 Health IT Summit Boston 2013 – Chris Petteruti, Director of Technology Department, Kaiser Permanente – NCAL Case Study "Strategies for Minimizing Uncertainty and Downtime”

Learning Objectives:

∙ Eliminating clinician frustration during IT outages
∙ Improving employee communication with desktop messages
∙ Strategies to get the most out of your communication platform investment

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
375
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Our Northern Cali region is composed of the following
  • Preceding the outage I received a call from our leadership team to meet the next day to have a post mortem to discuss the communication challenges during the outage and the impact to our staff and patient care“I was like, ut oh”. I was brought into a room with our hospital leadership, our CNO, COO and Physician IT Leadership The Problem Statement was this:We have tolerated chaos, confusion, and needlessly inconveniencing patients/doctors/staff and perhaps delayed important care for patients during outages that could have been avoided with better communication and information for far too long.  Our Problem: We needed a method to notify users when our computer systems go down to be able to communicate expected timelines and operational workaroundsThere were many issues, namely only being able to communicate with manager level users hoping that they could relay the message to their staff. It just sucked me in immediately, Before looking for solutions, I needed to understand our environment and the shortcomings associated with our communication process during outages. I learned that we had been doing the same thing for a very long time. The practices had been in place for a very long time. It wasn’t that the people or the systems in place were bad, everyone was trying to accomplish the same goal, which was to notify the end user community when our IT systems went down. Our IT folks were working in a flawed system that is setup the wrong way, its dependent on communicating directly to manager’s level personal, because those were the only ones that could receive our IT outage messages over pagers, and cell phones. We depended on them to relay our messages to the front end staff and physicians. It was a single point of failure, if that manager didn’t have their pager readily available and turned on, or if they were in a meeting, there was a good chance that the front line staff did not get the messages. The problem with this system is that its broken, its ineffective, its time consuming and ultimately its From the end users perspective they were left asking questions like :Is this happening to anyone else?Is it something I did?Should I call the help desk and report the problem?How long will it last?How will I know when its resolved?What do I do in the meantime? 
  • During our trial we had the company come out to do the install and quickly interfaced the system with our Active Directory using LDAP. This meant that all of our HR and contact information got sucked into the system immediately without having to obtain that information manually. From that data source we began creating dynamic distribution lists on the fly to target specific departments, roles within our organization, and workstations based on physical locationConcurrently we created a software package that we could remote distribute to our workstations and began pushing the desktop client to our small group of workstations. Then a crisis hit: We were smack dab in the middle of bringing up a new Lab Information System and I got a page at 430 in the morning that the entire lab system was down. Sent alert to leadership team with the phone option to dial 1 to join the callWithin a matter of seconds notifications were shooting out to multiple modalities. Our administrators were rolling out of bed and connecting to our call. After everyone learned about the severity of the issue and the impact to patient care, we all came up with a plan of action. Because our Lab serves both our Inpatient and outpatient members, the messages were entirely different based on whether you worked on the inpatient side or the outpatient clinic side.With the inpatient messaging we needed to make sure that our Nurses knew what to do operationally to ensure labs were drawn on time and delivered down to the lab as not to impact our morning discharges. On the outpatient clinical side, we need to let our staff and physicians know about the outage and to encourage members to avoid the lab during the outage to save them from standing in long lines and having a bad service experience.The beauty of this system was that we were able to target our messaging accordingly: Two distinct user groups, with 2 very different messagesEventually our system came back on-line and business returned to normal. In addition, it was a join commission survey year for us at one of our hospitals. We had the opportunity to alert specific individuals when the surveyors showed up onsite. The system delivered timely messages to our hospital administration on multiple modalities and provided the opportunity to get 2 way feedback from the field. We were able to see set response options in the messages to our targeted individuals and get instant results in realtime reports from our user console.  As the pilot concluded the Lab outage was the exact outage was written upAnd although the system worked as expected, it came with a price tag. When our business case was presented to our CFO, he had a hard time justifying spending dollars on such an application.  As you can imagine, I had spent months researching and investing time in looking for a system to solve our problem with communication for our area. In my mind, it was a done deal, how could we ask for anything better. Instead of giving up, I started looking more broadly and began kicking ideas around at what this specific application could be used for, then the lightbulb went off!
  • Our users were getting an error message when trying to launch KPHCSSS team were getting enamored with calls, no conceivable way to keep up and get everyone connectedWe were able to send a desktop pop up message with an alternative link to access KPHC to all our workstations in 60 secs via the AtHoc application.SSS sent an alternativeCritical Incident effecting all of NCALDate: 12/24/12 Start time: 06:30 AMResolution 07:42 AM ("Per Support on the Tech Bridge services have been restored and have confirmed with the Business ..Restored TIme 07:42am") Duration: 1 hour and 12 minutesRelated Incidents: 113 across all of NCAL Diablo Statistics:Sent to 931 Desktops in DRV and WCR, all linked to the RAP logon page and users were able to instantly access the system and get on with their day
  • Get data for similar issue – It was during the week day around 9am so the impact was much more significant
  • http://cl.kp.org/pkc/ncal/clib/Training/kphc_inpatient/docs/gen_dst_spr_fwrd_031013.pdf
  • iHT2 Health IT Summit Boston 2013 – Chris Petteruti, Director of Technology Department, Kaiser Permanente – NCAL Case Study "Strategies for Minimizing Uncertainty and Downtime”

    1. 1. IT Outage & Continuity AlertingPowered by AtHocChris Petteruti, Director of Technology, Kaiser PermanenteDiablo Service Area, Northern California
    2. 2. Kaiser Northern California Region 21 Hospitals 70 Medical Office Buildings 3.35 Million members, > 35%of Kaiser membership 7,000 physicians2 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
    3. 3. 3 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSystem OverviewAdditional Use CasesFuture PlansQuestionsPowered by AtHoc
    4. 4. 4 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesPowered by AtHoc Eliminating clinician frustration during IToutages Improving employee communication withdesktop messages Strategies to get the most out of yourcommunication platform investment
    5. 5. 5 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundPowered by AtHoc
    6. 6. 6 5/8/2013IT Outage & Continuity ofOperations Communication
    7. 7. 7 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSystem OverviewPowered by AtHoc
    8. 8. 8 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts System OverviewHQ2 Way Notification
    9. 9. 6 Month Pilot Period New Install Packed Desktop Client AD Integration Lab Incident Joint Commission & other Surveyors Conclusion9 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
    10. 10. 10 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP AlertsOne System, Multiple Uses IT Outage Communication Staffing Open Shifts Home Health Personnel Safety Staff Emergency NotificationPowered by AtHoc
    11. 11. USE CASE:Critical Outage Impacting all of NCAL Incident: Primary EHR system down–Date: 12/24/12–Start time: 06:30 AM–Resolution 07:42 AM–Duration: 1 hour and 12 minutes Related Incidents: 113 across all of NCAL11 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
    12. 12. Diablo Response using KP Alerts:– Sent 12,000 Desktop Alerts in our Diablo Service Area– All linked to ALTERNATE logon page– Our users were able to recover within < 3 minutes, access the system andget on with their dayOther Sites (not using KP Alerts):– Users were down for entire duration of outage (>1h)– Help desk flooded with incoming complaints– Help desk personnel rushed to location of critical users for assistance13 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013RESULT
    13. 13. 14 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSystem OverviewAdditional Use CasesPowered by AtHoc
    14. 14. Unexpected Downtime – Policy & Procedure15 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
    15. 15. Unexpected Downtime – Policy & Procedure16 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
    16. 16. Unexpected Downtime – Policy & Procedure17 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
    17. 17. AtHoc – Day Light Savings Alert18 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013Scheduled alert
    18. 18. AtHoc – Planned Rilis Outage19 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013Multiple-choice responseper recipient
    19. 19. KP Alerts System - Enterprise Approach Centralized enterprise deployment servicing all locations– Enterprise-wide system with localized accounts– Permission based access govern each location’s access rights and scope– Failover/alternate system available– Access to text and voice messaging via secure public cloud service yet with allpersonnel information secure behind the firewall Data security and Privacy via Private Cloud deployment– On-premise private-cloud installation ensures PII behind firewall– Integration with active directory for up-to-date contact data– Connected to networked computers behind firewall20 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
    20. 20. 21 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSystem OverviewAdditional Use CasesFuture PlansPowered by AtHoc
    21. 21. Future Plans – Smart Phone Use Expand alerts to Mobile app Mobile Duress button withlocation information forHome Health personnel andfor staff safety22 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013
    22. 22. 23 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013Future Plans - Additional UsesOne System, Multiple Uses IT Outage & COOP Communication Staffing Open Shifts Home Health Personnel Safety Staff Emergency NotificationPowered by AtHoc√√
    23. 23. 24 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013Future Plans - Additional UsesOne System, Multiple Uses IT Outage & COOP Communication Staffing Open Shifts Home Health Personnel Safety Staff Emergency NotificationPowered by AtHoc√√√
    24. 24. 25 | © 2012 Kaiser Foundation Health Plan, Inc. For internal use only.May 8, 2013KP Alerts3 Key MessagesBackgroundSystem OverviewAdditional Use CasesFuture PlansQuestionsPowered by AtHoc

    ×