This document discusses a "DIY" approach to rating EMR and physician practice systems without relying on third-party rating agencies. It involves directly contacting reference clients provided by vendor finalists to get feedback from end users. Key steps include issuing a request for information to identify leading vendors, having different end users conduct peer-to-peer reference calls using a checklist, scoring the results, and sharing findings to involve users in the selection process. The approach aims to provide more relevant data than ratings, heavily involve users who must live with the system, and teach them that no system is perfect.

1. Rating the Ratings
Part 8 – How to “DIY”
By Vince Ciotti & Elise Ames
HIS Professionals, LLC

2. Who Should You Believe?
• So who has the best large hospital EMR: Epic, Cerner or Allscripts?

3. © 2016 HIS Professionals, LLC
And the Best Physician Practice System?
• Should your practice select MediTouch, PracticeFusion or Epic?

4. © 2016 HIS Professionals, LLC
“Do It Yourself” (DIY) Rating
• It’s not that hard actually, and employs the very people most
impacted by the new system anyway: your end users.
• Just takes a few steps at minimal cost, possibly less than the
large fees some rating agencies charge for their reports.
• We developed this DIY approach during over 160 system
selections our firm has conducted in 25 years of consulting.
– Each search involved 5-6 vendors, so we vetted thousands
of HIS and MD systems using this “DIY” phone survey.
• Yeah, we’ve made a number of mistakes too, but learned
from them and thank our clients for helping us correct them.
• And there’s no ratings we can share (or sell!): the “right”
vendor varies with every unique hospital & practice…

5. © 2016 HIS Professionals, LLC
Step 1: Starts with an RFI
• First step is to start your selection process by issuing a “Request For
Information” (RFI) to the leading vendors being considered:
- That’s an area where these rating agencies might come in very
handy: identifying the 5-6 leaders for the apps you are buying –
not the “perfect” one to buy, but just the leading contenders.
- In the RFI, make vendors list 3-5 client references each for your:
• State – for EDI with your state’s Medicaid & Medicare Inter-
mediary, exchanging CCD with HIEs & RHIOs, PR taxes…
• Size - # of beds: CAH (<25), small (<100), mid-size (1-300),
Large (300+) or IDN (# of sites); and/or your # of physicians…
• Product – the release they are proposing: not some old version
barely supported, or ones they acquired and are sunsetting…
• Conversions - from your old system – if they ever did any…
• Recent Installs – over the past 1-3 years, not 20 years ago…

6. © 2016 HIS Professionals, LLC
Some Common Vendor Dodges
• We send out an Excel SS with 3-5 rows for each category on the
previous slide, and vertical columns for key info required for each:
- Hospital name, city & state, # of beds, & switchboard phone #.
- In the remaining columns, we list the key apps being considered
so the vendor can place an X under those the site has installed.
• Common tactics some vendor employ to dodge such requests are:
- Provide only their primary contact phone number, usually the CIO,
who they call in advance to make sure she’s happy (no sweat,
you can easily look up the switchboard # on their web site!)
- Stating their contract prevents any such reference calls – which
is bull-roar: nurses can’t talk about their EMR with other nurses?
- Vendors must join any reference calls – you can’t talk to their
clients without them being on the call. What are they afraid of?
- If the vendor persists with such doges – eliminate them. If you
can’t to talk with their clients, they’re hiding something!

7. © 2016 HIS Professionals, LLC
2. Peer to Peer Reference Calls
• Who should then call each client reference given? Not your:
– CIO & IT staff alone – who concentrate mostly on tech issues, or
– C-Suite – when is the last time your COO entered an order?
• But end users – the poor folks stuck with the system after go-live:
– Nurses – Informaticist(s) if you have any, shift supervisors and
any EMR “super users” for various floor and shifts.
– Physicians – CMIO if you have one, but the CMO and any super
users or IT champions on staff who build EMR screens & alerts.
– User Departments – not just the Directors/Managers, but the
super user or IT Coordinator for every department involved:
• Lab, Radiology RX, ED, HIM, Access, RCM, HIM, Finance…
• Why involve so many people in making phone reference calls?
– “Buy In” - if end users pick the system, they will make it work
despite the inevitable problems they’ll know about from the calls.
– If you pick the system, they’ll blame you for every problem…

8. © 2016 HIS Professionals, LLC
Step 3: Tracking the Calls
• It’s hard to get some end users to take time off their busy schedules
to call about a new system, so some techniques to get it done:
– Require users to fill out a checklist for each call and keep a SS
of how many each department turns in (samples follow).
– Share the results with your C-suite so they can harass those
user departments under them who aren’t cooperating.
• For departments who say
they can’t get through to
anyone, suggest they call
vendor users they might
know from state or
regional associations like
HIMSS, HFMA, ACHE,
AONE, etc.
• The more calls the merrier!

9. © 2016 HIS Professionals, LLC
4. Getting Through on the Phone
• Have your users call clients in each category of the RFI list:
– Your state, your size, your product, your apps, etc.
• Call the client’s switchboard and ask for their counterpart:
– Access (Admitting), RCM (Patient Accounting), ERP (finance),
Nursing (RN Informaticist, CNO, Nurse Supervisor…), etc.
• As busy as most healthcare employees are today, odds are they’ll
end up leaving a message, so call & leave many messages!
• Once you get through (or one calls you back), the conversation gets
very granular as users talk shop that only peers appreciate:
– RNs talk about OE & RR, eMAR, BMV, TPR, I&O, notes…
– Lab Directors chat about instrument interfaces, BB, Micro…
– MDs rant about the rotten CPOE, annoying alerts, scribes…
– CIOs & IT staff compare notes on DR, upgrades, interfaces…
• Miles deeper than any rating agency (or consultant) can ever dig!

10. © 2016 HIS Professionals, LLC
5. Sample Phone Reference Checklist
Hospital Caller: ____________________ Vendor: _____________________
Date of Call: __________ Hospital Called: ___________________________
Contact Name: _____________________ Title: ______________________
Answer the questions with 1 2 3 4 5
a according to this grid: Very Poor Ave. Good Very
Poor Good
1. What is your overall satisfaction? How do you rate separately their:
product (programs) vs. service (support)?
2. How would rate user friendliness:
b. Help key?
implement?
budget?
flexibility?
training?
hardware?
how would you rate ongoing support:
Phone support? On-Site Visits ?
User Groups? Periodic releases ?

11. © 2016 HIS Professionals, LLC
Step 6: Score the Results
• Next step is to
score the phone
checklists in a SS
like this one:
• Questions must be
answered in a
numeric scale, e.g.:
1-to-5 or 1-to-10.
• Have a separate
page for each user
dept. so they see
their results.
• Hand out the results
in a meeting where
each user relays
what they heard
about each vendor.

12. © 2016 HIS Professionals, LLC
The Net
• This DIY rating approach may give you far more relevant data on
vendors than just trusting a third-party “objective” rating agency:
- Our scores average about 70 points, with the best in the 80s & the
worst in the 60s, far more realistic numbers than the near-perfect
scores some rating agencies give to paying vendors.
- It heavily involves end users, and teaches them that they are
picking the “least of the evils,” rather than a perfect system.
- You can apply the numeric checklist & scoring to several other
steps in the selection process as well: demos, site visits, etc.
• Hope you enjoyed the series – we bet the raters give it a low score!
- Elise Ames
- 413.329.6925, eames@hispros.com
- Vince Ciotti
- 505.466.4958, vciotti@hispros.com