AACC e-letter survey data analysis (JA)

1 © 2014 Regional Health Education, The Permanente Medical Group, Inc.
Evaluating AACC E-Letters:
Member Survey Results
Jessica Alper, MPH, MSW

Background
 The AACC sends out e-letters to members on specific health topics
– The purpose of the e-letters is to help members prepare for a doctor visit or
learn to manage their symptoms at home (depending upon health topic).
– E-letters cover a wide range of topics in OB GYN, Pediatrics, and Medicine.
– The AACC currently offers 101 e-letters.
 Evaluation rationale
– Assessing member perceptions of e-letter quality, outcomes, and operations
will uncover opportunities for improvement.
© 2014 Regional Health Education, The Permanente Medical Group, Inc.

Evaluating E-Letters: Survey Goals
 Quality
– Identify member perceptions of e-letter clarity
– Assess member satisfaction with e-letter length
– Evaluate perceived value of resources links
 Outcomes
– Assess how well e-letters help members prepare for MD visit or manage
symptoms at home
– Determine whether e-letters contribute to perceived value of KP membership
 Operations
– Gauge how consistently AACC staff ask permission before sending e-letters

Research Methods
 The AACC selected 7 e-letters to evaluate
– Criteria: utilization rate and space constraints
 Methodology: 7-question online survey for each of the 7 e-letters;
members received survey via link embedded in e-letter
 Surveys administered via Survey Monkey from May 2014 —
September 2014
 Data analyzed using Excel

Survey Response Rates
 520 responses in total/61,964 sent (0.8% response rate)
– UTI: 260 responses/10,523 sent (2.5% response rate)
– Back pain: 71 responses/19,928 sent (0.7% response rate)
– Prenatal welcome: 70 responses/8,985 sent (0.8% response rate)
– Cough/Cold/Flu-Adults: 54 responses/6,195 sent (0.9% response rate)
– Pap smear: 52 responses/9,766 sent (0.5% response rate)
– Vaginitis: 37 responses/5,337 sent (0.7% response rate)
– Vomit/Diarrhea 6 mos-3 yrs: 22 responses/4,260 sent (0.5% response rate)
50% of all responses are
about UTI e-letter

Key Findings
*See Appendix for details regarding each e-letter’s length and use of resource links.
Quality
 Nearly all (86-100%) respondents indicated that the e-letters were clear.
 More than 87% of respondents indicated that the length of the e-letters was “just about right.”
 A high majority of members indicated that the resource links were “very helpful” or
“somewhat helpful.”
Outcomes
 70-100% of respondents affirmed that 6 out of the 7 e-letters surveyed helped them prepare
for a doctor’s visit or understand how to manage their symptoms at home.
– Vaginitis e-letter was less successful:
 Only 51% of respondents rated the vaginitis e-letter as helpful
 46% of vaginitis e-letter respondents chose a neutral response
 79-97% of respondents indicated that the e-letters made them feel that KP cares about their
health.
Operations
 Survey results indicated that the AACC staff are consistently asking permission before
sending an e-letter.

Interpretation of Results
 Applying Rules of Thumb:
– 65% or more positive is considered a strength
– 35% or more negative is considered a challenge
– 30% or more neutral suggests uncertainty, indicating possible results bias
– A difference of 5 percentage points or more is considered notable
 Positive ratings: the sum of two positive response categories (e.g.,
Strongly Agree/Agree)
 Negative ratings: The sum of two negative categories (e.g., Strongly
Disagree/Disagree)

Q1: I thought that this email was clear.
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
Strongly Agree/Agree Strongly
Disagree/Disagree
Neither agree nor disagree (blank)
Back Pain
Cough Cold Flu
Pap Smear
Prenatal Welcome Letter
UTI Treatment and Prevention
Vaginitis
Vomit Diarrhea 6mo to 3yrs
• 86—100% of all respondents indicated that the e-letter was clear.
• The back pain e-letter received the lowest proportion (86%) of positive responses and the
highest proportion (9%) of negative responses.

Q2: The length of this email was:
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
Just about right Too long Too short (blank)
Back Pain
Cough Cold Flu
Pap Smear
Vaginitis
• More than 87% of respondents indicated that the length of the e-letters was “just about
right.”
• Respondents were 2X more likely to rate e-letters as too short (34%) than too long (15%).
• Average length of surveyed e-letters: 262 words. (See appendix)
— Longest: 350 words.
— Shortest: 192 words.

Q3: This email
provided a few
resources about
[health topic].
How helpful did
you find the
links?
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
I didn't click
on the links.
Not helpful Somewhat
helpful
Very helpful (blank)
Back Pain
Cough Cold Flu
Pap Smear
Vaginitis
• A high majority of all respondents found the links to be “somewhat helpful” or “very helpful.”
• The pediatric vomit/diarrhea e-letter received the highest proportion (73%) of positive (“very
helpful”) responses.
— Notably, this e-letter doesn’t include any resource links (see appendix). This result
indicates member confusion, rendering results less meaningful.
• The back pain e-letter received the highest proportion (14%) of negative (“not helpful”)
member ratings.
— This e-letter included a single link, to a video (see appendix).

Q4: This email helped
me prepare for my
doctor’s visit (or
“helped me to
understand how to
manage my
symptoms at home,
depending on the e-
letter.)
• A high majority (70%-100%) of respondents affirmed that 6 out of 7 of the e-letters were
helpful.
• The back pain e-letter received the highest proportion (15%) of negative responses.
• The vaginitis e-letter received the lowest proportion (51%) of positive responses.
• 46% of vaginitis e-letter respondents indicated a neutral response.
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
Back Pain
Cough Cold Flu
Pap Smear
Vaginitis

Q5: This email made me feel that KP cares
about my health.
• A high majority (79—97%) of all respondents indicated that the e-letters made them feel that
KP cares about their health.
• The back pain e-letter received the highest percentage (9%) of negative responses.
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
Strongly Agree/Agree Strongly
Disagree/Disagree
Neither agree nor disagree (blank)
Back Pain
Cough Cold Flu
Pap Smear
Vaginitis

13
Q6: You recently
called the KP
AACC. During that
phone call, did you
agree to receive a
follow-up email
with additional
information?
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
I don't
remember
No Yes (blank)
Back Pain
Cough Cold Flu
Pap Smear
Vaginitis
• 89%-99% of all respondents indicated that they had agreed to receive an e-letter.
• This suggests that AACC staff are consistently asking permission before sending e-letters.

Member Comments: UTI E-Letter
“It was clear and concise info
that could be printed out
If needed.”
“The email came
quickly and the
information was
direct, specific, and
to the point. Great
customer service!”
“I've checked the
email twice so far,
the call gave too
much information at
once for me to
remember. It was
great not to have to
write everything
down.”
“Thank you for this
email,
it really shows that
Kaiser cares
about their patients.”
“I could probably find this info
elsewhere, but it’s so much easier
to have it all there in an email…
No one wants to do research
when dealing with a UTI!”
“It was very
helpful to
have
info on Azo
and other
medications
and how to
prevent
another UTI.”

Member Comments: Back Pain E-Letter
“I would prefer to read, not
watch a video, and a
transcript was not
available.”
“The video wouldn’t load on my
phone so I had to sit painfully to
watch it on my computer.”
“This is a very useful tool
and I'm very grateful for
Kaiser’s care and concern
regarding my back pain.”
“I asked about middle/upper back pain and they
sent me a video about lower back pain.”
“All of the links in the video
were very confusing. It was
annoying to click to see a video
and then have to fill out a whole
survey before watching the
video.”
“The email just
says to watch
the video, it
doesn’t
give any real
help for
back pain.”

Member Comments: Prenatal Welcome E-Letter
“It doesn’t actually
explain the process
of starting prenatal
care.”
“I was expecting a
link to general
advice but could
only see links to a
video on birth
defects and
information about
trial participation.”
“When I clicked to sign up for the newsletter
It sent me to a “find a doctor” page and I couldn’t
sign up for the newsletter…This was confusing.”
“Adding the phone number
for the advice line would
be helpful.”
“The person who sent me the email told me this was
an email on what I can expect to happen during my
first visit with the prenatal clerk, and there was no
information about this. Therefore I found this email
completely unhelpful.”
“I wish it had
info about what is
safe to eat.”

Member Comments: Adult Cough/Cold/Flu E-Letter
“While the information provided was detailed about how
to care for my cough, the major point of the phone call
was how to deal with my asthma symptoms. This
information was good general information but it did not
get to the core of my health issue.”
“I appreciated the
information and the option
of having a prescription or
trying home remedies.”
“I am gratified by
the extent to which
you can help me
get self-education
through these
follow-up emails.
Keep up the good
work!”
“The email, with the
link for ease of further
info if I want any, is a
good way to go, better
than just scribbling
notes while I'm on the
phone.”

Member Comments: Pap Smear E-Letter
“My Ob Gyn told me that I CAN use a tampon
until just before the visit and take it out right
before she comes in the room. That's contrary
to what this email says…I think that Kaiser
employees need to read their own
handouts/emails!”
“I feel more prepared for my
appointment and I know what to
expect after reading this email.”
“The most important item needed was
not in this email anywhere—the actual
appointment date and time.”
“Very helpful and effective information.”

Member Comments: Vaginitis E-Letter
“Thank you for sending this
information, I found it very
helpful and useful.”
“Thank you for providing this
service to your patients!”
“I received this
information right away
and it was clear and
concise.”

Member Comments: Pedi Vomit/Diarrhea E-Letter
“Thank you for creating emails like
this one — it makes it much easier to
care for my child at home when I
have clear notes and instructions.”
“This email really should include a general
timeframe. If diarrhea continues for more
than x days, contact Kaiser or seek medical
attention or?? Something to give parents a
general idea of when to be patient and when
to be concerned.”
“This email is so helpful — I
really appreciate it!

Recommendations
 Consider revising the back pain, prenatal welcome, and vaginitis e-letters to address member
concerns about lack of relevant content.
 Consider developing member service language for AACC staff to use when referencing e-
letters.
 Continue to promote health education content via links embedded in e-letters. Consider
incorporating multiple modalities (PDF, MDO, video), particularly with e-letters that currently
feature only video links.
 Consider reviewing e-letters on a regular basis to ensure up-to-date content and working
links.
 Consider adding phone number to address member frustration with generic messages such
as, “Call us if you experience side effects.”
 Consider incorporating messages of concern about member well-being into all e-letters (e.g.,
“Wishing you good health!”)
 Consider gathering TSR and RN feedback about e-letter challenges and opportunities.
 Consider developing e-letter template to standardize length, content guidelines, link
guidelines, and tone; assess with members.

Caveats and Considerations
 Only 7 of the 101 e-letters were assessed in this survey. Therefore, results may not be
generalizable to all e-letters.
 Due to the disproportionately high percentage of survey participants who responded to the
UTI e-letter survey, the results from the other 6 surveys should be considered less robust and
representative.
 The high volume of “I found the links very helpful” responses from members surveyed about
e-letters that did not include links suggests respondent confusion about the question and
renders survey findings less meaningful.
 The Pediatric Cough/Cold/Flu and Pediatric Fever e-letters were initially selected for this
survey, but removed from analysis because they were replaced by new e-letters that did not
have the space to accommodate survey links.

Appendix
 Average length of e-letter: 262 words
E-Letter Length
UTI Treatment and Prevention 265 words
Back Pain Online Video 192 words
Prenatal Welcome Letter 200 words
Cough/Cold/Flu-Adults 200 words
Pap Smear 200 words
Vaginitis 321 words
Vomit/Diarrhea 6 mos-3 yrs 350 words

Appendix (ctd.)
E-Letter Number
of
resource
links
Type of resource link
Placement of
resource link
MDO
home
page
Video PDF
hand
out
TPMG
Online
tool
Research
project
UTI Treatment
and Prevention
0 NA
Back Pain Online
Video
1  Upper third
Prenatal
Welcome Letter
4     2 in middle third
2 in bottom third
Cough/Cold/Flu-
Adults
1, listed
twice
 1 in upper third
1 in bottom third
Pap smear 1  Upper third
Vaginitis 1  Upper third
Vomit/Diarrhea 6
mos-3 yrs
0 NA

Appendix (ctd.)
Survey Comments Summary: UTI Treatment and Prevention E-Letter
 136 answered; 124 skipped
 107 positive responses
 8 negative responses
 14 neutral responses (e.g. “Thank you” without additional comments)
– 56 responses praising the AACC
– 21 responses praising e-letter
– 6 responses complaining that Rx was not ready for pharmacy pick up within the time frame specified by AACC
staff.
– 6 responses praising phone appointment option
– 5 requests for more detailed information
– 3 responses beyond scope of AACC (about member health status or experience with specific medication)
– 2 responses about lack of links
– 1 response stating frustration about speaking with Advice Nurse rather than doctor
– 1 response stating that AACC makes member feel KP tries to avoid seeing members in-person

Appendix (ctd.)
Survey Comments Summary: Back Pain Online Video E-Letter
 4 neutral responses
– 9 responses stating appreciation for helpful information
– 8 complaints about problems loading and/or watching video
– 4 responses expressing dislike of survey before video
– 3 comments that video was too specific to lower-back pain
– 2 responses stating that navigating the e-letter was confusing
– 2 comments stating that the e-letter did not match AACC description
– 2 requests for written information
– 1 response complaining there was no information provided on back pain treatment
– 1 response about member’s personal back pain experience

Appendix (ctd.)
Survey Comments Summary:
Prenatal Welcome E-Letter
– 3 requests for more detailed information, including
dietary advice
– 2 requests to include phone number
– 2 complaints about links
– 2 comments that info on birth defects was off-
putting
– 2 comments that e-letter did not give info on how to
enroll in KP Prenatal care
– 1 comment that receiving call from MD/RN 2 weeks
in was too late
Cough/Cold/Flu—Adults E-Letter
– 14 comments praising the AACC
– 12 comments stating the e-letter was helpful
– 2 comments that purpose of call was not
cough/cold/flu, and therefore e-letter was not helpful
– 1 complaint that the AACC made respondent feel
KP does not want to see members in-person
– 1 complaint that the e-letter did not provide new
information
– 1 request for an appointment confirmation in the e-
letter
– 1 request for more detailed information re:
medications

Appendix (ctd.)
Pap Smear E-Letter
– 5 comments that the e-letter was helpful
– 2 requests to add appointment date and time to e-
letter
– 2 complaints about problems with video
– 2 comments that the e-letter didn’t provide new
information
– 1 comment praising the AACC for excellent service
– 1 respondent called for an issue other than Pap
Smear so found e-letter unhelpful
– 1 complaint that e-letter advice conflicted with
doctor’s advice
– 1 complaint that link to schedule appointment didn’t
work
Vaginitis E-Letter
 2 negative responses were negative
 1 neutral response
– 3 comments praising the e-letter
– 1 complaint regarding Rx not being available for
pick up as AACC stated

Appendix (ctd.)
Comments Summary:
Vomit/Diarrhea 6 mos-3 yrs E-Letter
 1 negative response
 1 neutral response
– 1 comment praising AACC e-letter
– 1 complaint that no time frame was provided to let
parents know when to seek medical attention
Comments Summary:
Cough/Cold/Flu—Adults E-Letter
– 12 comments stating the e-letter was helpful
– 2 comments that purpose of call was not
cough/cold/flu, and therefore e-letter was not helpful
– 1 complaint that the AACC made respondent feel
KP does not want to see members in-person
– 1 complaint that the e-letter did not provide new
information
– 1 request for an appointment confirmation in the e-
letter
– 1 request for more detailed information re:
medications

AACC e-letter survey data analysis (JA)

Recommended

Recommended

More Related Content

What's hot

What's hot (12)

Viewers also liked

Viewers also liked (7)

Similar to AACC e-letter survey data analysis (JA)

Similar to AACC e-letter survey data analysis (JA) (20)

AACC e-letter survey data analysis (JA)