Recommended
More Related Content
Similar to Comp8 unit8b lecture_slides
Similar to Comp8 unit8b lecture_slides (20)
More from CMDLMS
Recently uploaded
Recently uploaded (20)
Comp8 unit8b lecture_slides
- 1. Installation and Maintenance of Health IT Systems Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers and Users Lecture b This material (Comp 8 Unit 8) was developed by Johns Hopkins University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number IU24OC000024. This material was updated in 2016 by The University of Texas Health Science Center at Houston under Award Number 90WT0006. This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/.
- 2. Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users Learning Objectives 1. Identify and implement an effective troubleshooting procedure for reporting, evaluating, fixing, deploying, and follow-up of errors, problems, or limitations for the system (lectures a & b). 2. Integrate downtime schedule for OS, network, database, and client application maintenance and updates (lecture b). 3. Develop a process for communicating requirements and supplying updates between vendors/developer and users (lecture b). 4. Create a baseline for system performance measurement and comparison for troubleshooting (lecture b). 2
- 3. Troubleshooting: Diagnosing Complex EHR Infrastructure Issues • Identify the problem • Determine the scope of the issue • Start Simple • Seek Additional help during an Outage • Remember to communicate (Halamka) 3
- 4. Troubleshooting: Diagnosing Complex EHR Infrastructure Issues (cont’d) • Use the logs • Develop a timeline • Stick to the Plan…Set deadlines for escalation • Stay focused on resolving the problem. • Don’t be too quick to declare a problem resolved. (Halamka; AHRQ, 2011) 4
- 5. Troubleshooting: An Example • User contacts the EHR Helpdesk about slow EHR… – Start with the local machine o Recent Updates or Patches o System and Application Logs o Wireless/ Ethernet Connectivity 5
- 6. Troubleshooting: An Example (cont’d) • If the issue is more widespread… – Network or ISP Issues o Misconfigured switch, router, ISP Connectivity – Erroneous OS Upgrade/ Misconfigured Server – Server Hardware Error/ Upgrade Needed – Application Upgrade Error o EHR Design o EHR Software Code 6
- 7. Performance Baseline • Generated after completion of performance baseline testing. • Shows normal operating parameters of your system under normal load conditions. • Used to gauge overall system health & assist with isolating performance problems. (Wunder) 7
- 8. Performance Baseline: Testing • “Benchmark testing” • Compare performance of new server/system against standard (e.g., existing measurements or software specs). • Test at regular intervals, when problem-free: – System resources (server & hardware) – Network architecture – Operating system – Database applications – Client applications (Sqlserver, 2009) 8
- 9. Performance Baseline: Utilities Available • Work with vendor to determine best method for your EHR. • Many third-party utilities available. • Windows built-in utilities: – System Monitor (collect & view real-time data on usage of memory, disk, processor) – Performance logs & alerts – Task Manager – Event Tracing for Windows (ETW; trace & log events raised by user- mode applications & kernel-mode drivers; export most SQL Server events.) • Databases often require special utilities. • SQL Server built-in utilities: – SQL Trace – SQL Server Profiler – SQL Server Management Studio Activity Monitor – SQL Server Management Studio Graphical Showplan – Stored procedures – Database Console Commands (DBCC) – Built-in functions – Trace flags (Microsoft, 2011; Technical, 2005) 9
- 10. Performance Baseline: Measurements & Next Steps • Measurements – Peak vs. off-peak hours – Production-query, batch-command response times – Database backup and restore completion times • Next steps – Compare baseline statistics to current server/system performance. – Investigate if numbers far above or below baseline. (Wunder) 10
- 11. EHR Maintenance & Upgrades • Upgrades crucial for extending EHR functionality & overall system lifespan. • Improperly planned/managed upgrade procedures can severely damage system, reducing performance. • Use highly structured approach in upgrading any major, critical production system. 11
- 12. EHR Maintenance & Upgrades: Structured Approach • Upgrade team – Works with vendor. – Reads upgrade documentation. – Analyzes functional enhancements. – Works with vendor on unknown variables. – Makes plan to configure & test affected applications. • Technical team develops alternative testing environment (test bed). • Upgrade team compiles pre-installation & installation checklists, determines downtime. • After first testing, logs reviewed & errors/discrepancies reported to vendor. 12
- 13. A Structured Approach (cont’d) • After vendor resolves issues, upgrade process retested, ensuring data integrity. • Consider software to simulate workflow & users; track data for performance analysis. • Unified spreadsheet to track issues. • Once testing complete & upgrade approved, final documentation prepared & training scheduled. • Installation team finalized & assigned roles. • Upgrade scheduled during off-peak times. • EHR “shadow copy” (read-only, without real- time interfaced results) accessible during upgrade. 13
- 14. A Structured Approach (cont’d - 2) • Upgraded system tested immediately after completion. • Once testing complete, help desk notified & systems brought back online. • Dedicated team available for several business days to resolve issues. • (Follow similar processes for upgrades to server OS, workstation OS, databases.) 14
- 15. Interacting with Users • Stay up to date • Set realistic expectations • Value your user’s time • Be friendly, but stay on task. • Listen Closely, and acknowledge the user’s issue • Use the opportunity to educate • Be patient (Boyer, 2005) 15
- 16. Client-Vendor Relationship • Essential not only for successful implementation, but also for continued success of product. – Issues will arise beyond scope of your IT workforce. – Routine upgrades needed. – Vendor plays pivotal role in these & other events. • Successful partnerships built on mutual goals, trust. – “In this together”. – Success of vendor depends on success of product. • Long-term commitment between organization & vendor. – Relationship becomes complex once organization dependent on fully embedded EHR. (Boyer, 2005) 16
- 17. Client-Vendor Relationship: Before Purchase • Understand vendor company & culture. – Visit headquarters, talk with as many as you can (including development & support). – Follow the money. What drives revenue stream? – Dependent on support fees? Understand support-fee structure. – Financially solvent enough to provide long-term support? • Hosted and SaaS (Software as a Service) solutions: understand vendor’s upgrade strategy. – Will it conflict with your business practices? • Impressions of vendor’s support staff (Boyer, 2005) 17
- 18. Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users Summary • Performance baseline – Helps gauge system health & isolate problems. • Structured approach for upgrades – Separate environment for testing upgrades prior to rollout. Work closely with vendor to resolve issues. Rollout during non-peak hours. • Client-vendor relationship – EHR purchase brings long-term commitment. Complexity once dependent on EHR. Understand vendor & culture before purchase. 18
- 19. Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users References – Lecture b References Boyer, E. and M. Soback (2005). Production Support. Implementing an Electronic Health Record System. J. M. Walker, E. J. Bieber and F. Richards, Springer London: 95-100. http://www.springerlink.com/content/n520ghg078416463/ Event Tracing for Windows (2011). Microsoft.com. [Internet]]. http://msdn.microsoft.com/en- us/library/ff545699.aspx . AHRQ. (2011). Guide to Reducing Unintended Consequences of Electronic Health Records [Online]. https://healthit.ahrq.gov/sites/default/files/docs/publication/guide-to-reducing-unintended- consequences-of-electronic-health-records.pdf Halamka, John. 10 tips for troubleshooting complex EHR infrastructure problems. KevinMD.com. [Internet]. http://www.kevinmd.com/blog/2010/09/10-tips-troubleshooting-complex-ehr- infrastructure-problems.html. Technical Comparison of Oracle Database 10g and SQL Server 2005: Focus on Manageability, May 2005 (2005, May). Oracle [Online]. http://www.oracle.com/technetwork/database/manageability/ss-1.pdf 19
- 20. Installation and Maintenance of Health IT Systems Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users Lecture b This material was developed by Duke University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number IU24OC000024. This material was updated in 2016 by The University of Texas Health Science Center at Houston under Award Number 90WT0006. 20
Editor's Notes
- Welcome to Installation and Maintenance of Health IT Systems, Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users, This is lecture b. This component, Installation and Maintenance of Health IT Systems, covers fundamentals of selection, installation, and maintenance of typical Electronic Health Records (EHR) systems. This unit, Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users, will discuss ways you can implement an infrastructure for troubleshooting, and maintaining EHRs and their existing infrastructure.
- The objectives for this unit, Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users are to: Identify and implement an effective troubleshooting procedure for reporting, evaluating, fixing, deploying, and follow-up of errors, problems, or limitations for the system Integrate downtime schedule for OS, network, database, and client application maintenance and updates Develop a process for communicating requirements and supplying updates between vendors/developer and users. Creating a baseline for system performance measurement and comparison for troubleshooting. In the first part of our lecture, we discussed some IT support infrastructure strategies seen in many of today’s healthcare institutions. In the second part of our lecture we will discuss troubleshooting maintenance & upgrade strategies as well as the importance of creating a baseline for measuring system performance. Finally, we will talk about interactions between vendors, developers, and users, and the importance of developing a process for communicating requirements and supplying updates. We will briefly discuss the importance of client-vendor relationships and offer some suggestions for ensuring a healthy and long life together.
- Each EHR system…and networking environment for that matter, is different. However, there are some standard troubleshooting practices that ring true no matter what system you are troubleshooting. Let’s take a look at some tips for troubleshooting complex EHR problems. First…Identify the Problem. Be sure you contact the user personally and listen closely to the symptoms of the issue. Whenever possible test the application yourself, preferably from the user’s workstation and attempt to duplicate the problem. Next, you will need to determine the size and scope of the issue. Does the problem appear to be localized to one computer or user? If so, start your troubleshooting there. Ask the user if any changes were recently made to the system, their computer or EHR access privileges, or their profile and have them recreate the error again and watch for abnormalities. Next, move on to localized application and hardware checks before looking at the system as a whole. If you are receiving multiple complaints or if the issue appears to be related to database or server issues begin to expand your search. Ask yourself what has changed recently? Are there any new upgrades patches, or procedures implemented? Any new hardware added to the network, system or server? Are there any commonalities between the affected users? More often than not, the most complex problems are the result of a single failure. Start simple. If a light bulb is out, you wouldn’t immediately blame the circuit would you? No, you’d probably replace the bulb first. The same adage holds true in IT related issues. Don’t waste time when a significant outage occurs and you cannot determine its cause. Have a plan in place to address outage issues so you can bring in additional resources and minimize downtime. Remember, communication is important…both to your users and your stakeholders. Provide regular updates…perhaps hourly updates, describing what actions are being taken to find a resolution. However, it’s also important your key staff don't become overburdened with communication efforts. They need time to do their work.
- Using the EHR application, system, and security logs are an important step to diagnosing an issue. Understand where these logs are stored, as well how to access and decipher them. The logs can provide additional insight about where things went wrong and usually offer some sort of explanation as to the cause. After researching the issue develop a timeline of events leading up to the issue. This is particularly useful for issues resulting from a series of events occurring over a period of time. IT staff sometimes get tunnel vision…they can become so engrossed on the issue that they lose track of time. Have an escalation plan in place to bring in other resources and stick to it. It’s also important to stay focused on resolving the problem at hand. We all make mistakes. A major outage is not the time for finger pointing or for egos to get in the way of completing the task . Once the outage or problem has been resolved, you will have time to analyze the root cause and make procedural changes to help prevent similar issues from reoccurring. Another tendency is to immediately declare an issue resolved as soon as the issue returns to normal. Generally, it’s wise to monitor the system closely…perhaps 24-48 hours before you should feel truly comfortable that you have resolved the issue. Remember, problems and outages are just part of the job. You should gauge your reputation not on one single issue but how you and your team responds, evolves, and improves over time.
- Let’s take what we’ve learned so far and apply it to a common example: Susan, who works at the nearby clinic calls you at the helpdesk to complain about the EHR being abnormally sluggish this morning. “I could go get a cup a coffee while I wait for my query to return” are her exact words. You are currently unaware of any issues with the EHR which could be affecting performance. What would you do? Well, from what we’ve learned we know to start with the local machine…or the user and the machine. Lets look at some things we could check: Is the machine she is using today different in any way? Were any updates done on the machine? Did she have to contact the helpdesk recently to resolve an EHR or system issue? If she is on a different machine, does the machine meet the recommended EHR application requirements? You should also check: Whether the application and the system is updated and current. You should examine the system and application logs for abnormalities. Ensure the network card connected properly and is functioning within normal parameters. If she is connecting wirelessly, is the wireless connection operating at optimal performance levels? While you are on the phone with Susan, another co-worker from her clinic contacts the helpdesk with a similar issue. Now you may begin to broaden the search: Could there be an issue with the Local Area Network? Could a switch or router be configured incorrectly? Is there an abnormally high amount of traffic on the network eating bandwidth this morning? Could the ISP connecting the clinic to the EHR Server infrastructure be having issues?
- If the issue turns out to be more widespread in nature, you may need to begin looking at the EHR server components themselves. If the server infrastructure is in house you (with help from your vendor and/or server administrator) could determine: is the memory or hard drive space on the server due for an upgrade? Was the server recently upgraded, tweaked or configured differently? Is connectivity to the server itself operating optimally? If you are assisting with an EHR that is a SaaS (Software as a Service) or is hosted by a vendor, you would need to work with the vendor to determine: Is their data connection failing? Could an issue in the EHR design or code be the cause? Additional diagnosis options available to you will depend on the particular EHR system your organization uses.
- Since we are on the issue of troubleshooting, maintenance and upgrades, let’s spend a couple of moments discussing “performance baselines” and why they are important. A performance baseline is generated after completion of performance baseline testing and shows the normal operating parameters of your system under normal load conditions. It is used to gauge the overall health of the system and to assist with isolating performance issues. We use baselines to help us confirm normal system operation (say for instance, automated monitoring systems) when we think there may be an anomaly in the system and to assist us in pinpointing where to begin looking for a root cause.
- Performance baseline testing, also referred to as “benchmark testing”, is typically performed by the test engineer to compare the performance of a new server, application, or system to a known standard of reference, like existing measurements or software specifications. To take a performance baseline, you should begin by taking performance measurements at regular intervals when you believe the system is functioning within normal parameters, then begin comparing each set of measurements to the ones taken earlier. Over time you will get a gauge of normal system performance which will vary based on time of day, amount of system usage, etc. “The following areas affect the performance of [many systems and should be considered part of your baseline strategy]: System resources ([server and system] hardware) Network architecture The operating system Database applications Client applications”
- Databases often require special utilities. SQL Server built-in utilities: SQL Trace SQL Server Profiler SQL Server Management Studio Activity Monitor SQL Server Management Studio Graphical Showplan Stored procedures Database Console Commands (DBCC) Built-in functions Trace flags
- Baseline measurements should be used to determine at a minimum: When are your peak and off-peak hours of operation; What are your average production-query or batch-command response times; How long does it take to complete your database backup and restore jobs. After you have completed and established a server performance baseline, you may then use this baseline on a day to day basis to compare the baseline statistics to your current server or system performance. When performance numbers deviate too far above or below your baseline statistics, you have cause to investigate further. For example, if it takes longer now to execute a set of queries, then you may decide its time to determine if you can rewrite the queries or add new indices.
- Upgrading system components is crucial for extending EHR functionality and overall system lifespan. However, improperly planned or managed upgrade procedures can severely damage the system, reducing performance or even bring down the entire system. Most vendors publish at least one major upgrade a year and patches throughout the year may be needed to repair issues and improve performance glitches. Upgrading any major critical production system should take a highly structured approach.
- First, the, upgrade team reads over the upgrade documentation, and analyzes functional enhancements and configuration decisions, being sure to work with the vendor to address any unknown variables in documentation or due to system customization. If the upgrade involves more than one application, the upgrade team develops a plan for integrating configuration and testing the affected applications. There are some important questions that should be addressed before embarking on an upgrade project: Is the upgrade warranted or needed to conform to regulatory policies? What new functionality or benefits will be added as a result of the upgrade? Will additional hardware or software upgrades, if any, be needed to ensure the upgrade’s success? Is the recommended hardware or software compatible with existing components? How will the upgrade affect your users? What should you do to minimize downtime and educate your users? If a test bed is not already in place, the technical team develops an alternative environment closely mimicking the production environment for testing purposes. The upgrade team compiles a pre-installation and installation checklist and accompanying documentation and determines how much downtime will be needed for the upgrade process to complete. After the first testing is completed on the test bed, logs are reviewed and any errors or discrepancies are reported to the vendor for resolution. Working with the vendor is crucial for successful upgrade implementations.
- After the vendor completes any needed improvements, the upgrade team performs another install test, being careful to follow upgrade documentation and coordinating testing with the affected application teams. Additionally, special attention is given to ensuring preservation of data integrity. Specialized software, capable of simulating standard workflows and number of users and tracking data, helps paint a clearer picture of network performance during and after the upgrade. All of this data is sent to the vendor for analysis. A unified spreadsheet for recording, prioritizing, and documenting issues, along with their resolutions, can be very useful for managing problems as they arise and communicating with the vendor. Once testing has been completed and the upgrade process approved, final documentation is prepared and training is scheduled. The installation team is also finalized and assigned various roles. Upgrade windows should be scheduled during off-peak times to minimize issues. Additionally, a shadow copy of the EHR – in other words, a read-only copy without real-time interfaced results – would ideally remain available for access during the upgrade window.
- Testing of the upgrade should begin immediately after completion. Once testing is complete, the helpdesk is notified and systems are brought back online. After the upgrade has been completed, a dedicated team should be available over the next few business days to resolve any issues the upgrade could’ve created. Similar processes should be followed for upgrades to the server OS, workstation OS, & databases.
- Your ability to interact with users is as equally important as your technical skill set. Whether at the desk side, on the phone, or communicating through email, each interaction with a user is an opportunity to build good rapport between yourself, your IT department, and your end users. First and foremost, stay up to date. Being informed about what’s going on with the network, as well as any developments, patches, or upgrades taking place will help you serve your users better and can expedite diagnosis. Take advantage of any available training, particularly regarding new EHR upgrades or technology trends being adopted in the workplace. Set realistic expectations. If you work in a large, fully staffed, IT setting, particularly settings involving critical care patients, you may have little say in what expectations are imposed. However if you are one of a few…or the only IT personnel in a small clinic, You will eventually find that multiple users are constantly vying for your time. It maybe unrealistic to expect you to contact a user or resolve an non-emergent issue within 15 minutes of submitting a ticket. You need time to work and it’s only fair to attempt to resolve issues in the order they are submitted. Set up realistic expectations as to when a user should expect to hear from you and when you can resolve the issue. Whenever possible, centralize the first point of contact and have that person prioritize requests to minimize disruption of busy technicians. Decide what constitutes an “emergency” (such as an outage) and devise a method for contacting support technicians if circumstances warrant it. Publish these expectations and be consistent about meeting them. The name of the game here is maximizing reliability and productivity…for your users…the patients…yourself…and your team. Make every attempt to be punctual, particularly so when the user’s presence is a necessity, and communicate any status changes with your users promptly. Building good relationships is important for developing links, making allies, and building user confidence…and it can be fun as well. It’s important that both users and support teams understand they are interacting with people…with concerns, feelings, and educated opinions. Start the session off by being friendly and interested but respectful and letting them know their issue is important (after all they are professionals who've work hard to be where they are too). At the same time, don’t deviate too far. Stay focused on completing your task. Remember, when you first contact a user there’s a good chance he will already be frustrated…particularly in healthcare environments where medical professionals are often already working under stressful conditions and may also be sensitive about being placed in a subordinate role. Don’t make your users feel inferior if they make a mistake. Reassure them. This tactic can help disarm a tense situation. As I said before…communication is important…return calls promptly and take time when you contact the user to listen carefully to what she has to say. Probe for details. Once you believe you have a handle on what exactly the user is requesting take time to clarify the issue with them and let them know how you plan to address it before beginning work in earnest. Each opportunity you interact with a user is an opportunity to educate them. Explain to them what you are doing while resolving their issue. Also take a minute to remind them of any new procedural changes or updates which may be relevant. When you are done, ask them if there is anything else you can assist them with. Be as patient and as thorough as needed for your user to grasp the concepts. Reassure them. Remember, you too had to learn these techniques. Ultimately, you may save yourself another visit. Using some of these tips, you should be able to use your soft skills to better understand and build good working relationships with users while effectively addressing their technical needs.
- Building and maintaining a relationship with your EHR vendor is essential, not only for a successful implementation, but also for continued success throughout the product’s life cycle. Issues will arise that are beyond the scope of your IT workforce. Upgrades will need to be routinely applied. Your vendor will play a pivotal role in these and many more facets of your EHR management scheme. Successful business partnerships are built on mutual goals, trust, and the knowledge that both entities are “in this together” and that the success of the vendor goes hand-in-hand with the success of their product. Purchasing an EHR system brings about a long-term commitment between your organization and the vendor. These relationships become quite complex once the EHR system becomes fully embedded into the organization, which then becomes completely dependent on its capabilities. The type and nature of your relationship with the vendor will no doubt vary depending on your organization’s size and the scope of the implementation.
- Here are a few tips to consider before purchasing your EHR system to help ensure your vendor selection best suits your needs: Understand the company and its culture: Visit the company headquarters and talk with as many folks as you can, including the development and support teams. Develop some impressions about the company. Are staff engaged and enthusiastic? Does the company’s philosophy mesh well with your organization? Where is their focus: marketing, development, implementation? Follow the money. What drives the revenue stream for the company? Are they dependent more on support fees? Do you fully understand the support fee structure they are modeling for your company? Are they financially solvent enough to provide support throughout the lifecycle of the product? If you are planning to purchase a hosted or a Software as a Service (or S.a.a.S.) solution, do you fully understand their upgrade strategy? Will their upgrade strategy conflict with your business practices? Lastly, have your development staff interface with theirs. After all, they will be spending a great deal of time together. Does your staff feel like they are competent and enthusiastic about supporting their product?
- This concludes, Troubleshooting; Maintenance and Upgrades; Interaction with Vendors, Developers, and Users. Establishing a performance baseline helps gauge the overall health of the system and assists in isolating performance problems. Upgrading any major critical production system should take a highly structured approach including: Maintaining a test bed for testing upgrades prior to rollout. Working with the vendor to resolve issues and inconsistencies prior to rollout. Rolling out during non-peak hours. Finally, purchasing an EHR system brings about a long-term commitment between your organization and the vendor. These relationships become quite complex once the EHR system becomes fully embedded into the organization, which then becomes dependent on its capabilities. Learn about the vendor, its culture, its financial status, and its strategies before you make the purchase.
- No Audio. Ten seconds of silence.
- No Audio.