This document discusses troubleshooting performance and scalability issues with GE Centricity EMR/CPS. It outlines the process for identifying issues, including determining if the problem is reproducible or global. It also describes various application and platform tools that can help troubleshoot, such as SQL tracing, UI tracing, and performance counters. The document then explains GEHC's process for resolving issues, which typically involves database tuning, hardware optimizations, or client/software fixes in later releases. It provides examples of historically common issues and ways to enable instrumentation for capturing workflow timing data.

1. GE Centricity EMR / CPS
Performance and Scalability
Troubleshooting – Steven
Oubre & Chris Hays, GEHC

2. Centricity EMR/CPS - Performance &
Scalability Troubleshooting
Objectives:
• Understand the process needed for identifying
performance and scalability issues.
• Understand application / tools support for
troubleshooting issues.
• Understand the GEHC process for resolving
performance and scalability issues.

3. Process for identifying performance
and scalability (PNS) issues
Questions to ask yourself (or your staff):
• Can you identify the problematic workflow?
• Is it affecting more than one user (global vs. local
problem)?
• Is the problem reproducible?
– Can you reproduce the problem on a different client (Terminal
Server vs. fat client)?
• Do you have our support number handy?
– Call support before GE management, the support engineers are
trained troubleshooters! Support #: 888.436.8491
– Email:
– Centricity PM: CentricityPMServices@ge.com
– Centricity EMR: CentricityEMRServices@ge.com

4. Application / Tools Support for
Troubleshooting Issues
Application – based tools
• Trace capture
– SQL, GUI, Workflow (PNS instrumentation), Full, MEL
Platform – based tools
• OS (Windows, HPUX, AIX)
– Perfmon, glance, nmon, Topas, vmstat
• DBMS (SQL Server, Oracle)
– Oracle AWR/ASH, SQL Server Dashboard / DMV views

5. IP Performance Core Team Process

6. Application Tracing Methods
EMR / CPS-Chart Module hot-keys:
• SQL trace: ctrl-alt-s, logfile: c:tracesql.txt
– Used for tracking down long running sql queries, etc.
• Full DB trace: ctrl-alt-t, logfile: c:trace.txt
– SQL trace on steroids, records formulas firing, Causes client to slow
down.
• UI trace: ctrl-alt-u, logfile: c:traceui.txt
– Used for tracking UI clicks, can help in tracing a problematic workflow /
application crash
• MEL trace: ctrl-alt-m, logfile: c:tracemel.txt
– Used for troubleshooting complex forms
• For EMR 6.1 / 9.2: Go to Options menu and select
Tracing…, full diagnostic dialog for configuring
tracing options. Very useful in Citrix / WTS

7. Performance Counters
Common OS/Server performance
measurements
• CPU
– processor utilization (Is the server busy)
– CPU queue length (Ideally should = # processors or less on
average)
• Disk
– Queue length (looking for i/o bottlenecks)
– Disk utilization (looking for hot spots)
• Memory
– Used vs. Available (possible memory pressure)
– Pagefile usage (possible undersized RAM config)
• Network
– Retries
– Utilization

8. Resolution of PNS Issues
90% of all server-side PNS issues resolved
thru db tuning /hw optimizations
Issues are resolved through 1 of the following:
• Server side fix: Indexing (new/modified), PL/SQL
stored procedure, initialization parameters, splitting
of db files onto separate media
• Next service pack / software release
– In the interim, these are available through support
• Client side fix: Custom DLL / defs file change (if
PNS issue is client-side critical). Will usually get
rolled into next sp release.

9. How to run DB Performance Reports
Oracle
• AWR: In the [Oracle_Home]rdbmsadmin folder
– Cmd prompt: sqlplus ml/ml@(dbname) @awrrpt.sql
– Follow the prompts (select html format – easier to read/navigate)
– 1 hour snapshots are good for tracking down db performance issues
• ASH: similar to AWR but for individual session
– Sqlplus ml/ml@(dbname) @ashrpt.sql
– Follow the prompts (select html format – easier to read/navigate)
SQL Server
• Performance Dashboard reports
– Run out of SQL Server Mgmt Studio, select Reports from DB popup
menu
– can download current versions from MS website.
• Call CPS Support for custom script

10. PNS Instrumentation
Used to measure internally various workflow-
related timings at the client level.
Can be enabled system-wide or for individual
clients.
Instrumentation data is stored in the database.

11. PNS Instrumentation – How to
enableCPS (start a query window in SQL Server
Mgmt Studio):
• select * from medlists where tablename =
'PNS_Instrumentation‘ (Make sure the Code
column value is ‘On’)
• Create the Instrumentation.Config.xml file in client
installation folder or
• Update the Default Configuration in the
PNS_Configuration table
– select * from PNS_Configuration
– update PNS_Configuration set config=‘{put xml here}’ where
configurationid=1

12. PNS Instrumentation -
Instrumentation.Config.xml file Contents
<Instrumentation MasterSwitch ="On“ FlushInterval=“10”>
<!-- Measure workflow timing.-->
<WorkflowTiming OnOff="On" />
<ResourceSnapshot OnOff="On"/>
<PerformanceCounters OnOff=“Off" Interval="10">
<!-- Specify which performance counters to trace. Here is an example. Remember,
not all counters have an instance. -->
<PerformanceCounter Category ="Process" Name="% Processor Time"
Instance="CPOPM06"/>
</PerformanceCounters>
</Instrumentation>

13. PNS Instrumentation – How to
enable (continued)
EMR (using sqlplus/w or other Oracle db query
tool):
• All workstations:
– Update ml.symbol set value=1 where object=‘Clinic’ and
name=‘EnableWrkflwLog’;
– commit;
– restart client application(s)
• Single workstation:
– In client installation folder, set “_EnableWrkflowLog” to “1” in
mlsetvar.txt and restart application.

14. PNS Instrumentation – Workflow
events we currently capture - CPS
CPS/EMR:
Load EMR hub
Open Chart
Join Update
Open Document
End Update
Hold document
Sign Update
Sign document
Sign Order
Send Order
Rx Refill
Print
Send Flag
Route Document
New Medication
Change Medication
Save and Continue Medication
CPS/PM:
Login Time
Open Schedule
Create or Reschedule Appointment
Cancel Appointment
Review Patient Demographic Info
Review Patient Insurance Info
Create Patient
Save Patient
New visit
Save Visit
Post single charge
Post payment

15. PNS Instrumentation – Workflow
events we currently capture –
EMR/OracleChart Open
Login
New/Join Update
Open Document
End Update
Sign Order
Send Order
RxRefill
Print – this one logs a context field that
shows what type of object was printed
(48 types currently)
Send Flag
Route Document
Sign Document
Sign Update
Hold Update
Add new med
Change med
Save and Continue med

16. PNS Instrumentation –
Useful Reporting Queries
CPS:
select * from vTimingEvent where sessionid > ##;
-- grouped one:
select timingeventDescription,count(*) total ,max(waittime) theMax, min(waittime) theMin,
avg(waittime) theAvg, round(stdev(waittime),2) theStdev from vTimingevent where
sessionid > ## group by timingeventdescription;
select * from vresourceusage where sessionid > ##;
-- grouped one
select description,count(*) total, max(procgdihandlecount) maxgdi,max(procuserhandlecount)
maxusr from vresourceusage where sessionid > ## group by description;
EMR:
select * from vtimingevent where sessionid > ##;
-- grouped one:
select timingeventdescription,count(*) total ,max(waittime) theMax, min(waittime) theMin,
avg(waittime) theAvg, round(stddev(waittime),2) theStdev from vTimingevent where
sessionid > ## group by timingeventdescription;

17. Issues Historically affecting
Performance and Scalability
• Data set size
– databases keep growing, requiring client app / db tuning changes to
handle larger data sets (obs, large user tables, unsigned
documents, etc)
• Hardware Issues
– Older Cisco switches / routers had auto-negotiation issues, would
drop connections to 10Mb Half
– Undersized workstation clients w.r.t. 3rd
party add ons (CCC for
example)
– Undersized server hardware
– Current hardware specifications available at:
http://centricitypractice.gehealthcare.com (password-enabled)
• RDBMS issues
– Oracle: 8.0 -> 8i -> 9i -> 10gR1 –> 10gR2 Each version required
adjustments to OS / SW / HW configurations in order to make them
work

18. Questions?