SlideShare a Scribd company logo
Will They Love You Tomorrow?
Conducting Effective Alumni Research




Renee Kart
Director of Project Strategy
SimpsonScarborough

June 9, 2011
What We Want to Know
• What type of relationship do alumni seek to have with our
  institution?
• How can we do a better job of engaging alumni through
  communications and activities?
• What services do alumni need and want from us?
• How do our alumni want us to communicate with them?
  How do they want to communicate with us?
• How effective are our institution’s print and electronic
  communication strategies?
• How do our alumni feel about their experience at our
  institution?
• What messages about our institution build a sense of
  pride for our alumni?
• Are there ways that we can model alumni engagement to
  make better decisions about resource allocation?
• What types of events will strengthen relationships with
  alumni?
• How should our institution segment our alumni to
  maximize the effectiveness of our efforts to engage them?
• Who are the leaders among our alumni?
                                                2
Establishing Guiding Questions
• “Guiding Questions” are the 5-8 broad questions
  you want your research to answer
    • If you have more than 8, your study probably      Many survey research studies try
      lacks focus                                       to gather information about too
    • Make the hard trade-offs (a “must know” vs.       many topics rather than delving
      “nice to know”); no laundry lists                 deeply into one or two pressing
• They are not actual questions that would be asked                  issues.
  during a focus group, in-depth interview, or survey
• Imagine they could be the section headers on your      After you develop your guiding
  PowerPoint of the findings                              questions, you will be able to
• Get buy-in from other stakeholders                         decide if qualitative or
                                                        quantitative research (or both) is
• List the guiding questions on all communications         best to address your needs.
  regarding the project; include them at the top of
  every draft of your moderator guide, in-depth
  interview script, or survey instruments


                                            3
Choosing Research Methods (1 of 3)

                        Research Methods
                                                  Focus Group
         Qualitative                         (In-Person or Online)
                                              In-Depth Interviews
                                                  Direct Mail
         Quantitative                           Phone Survey
                                                Online Survey
                                           Alumni Network Analysis
            Other                           Segmentation Analysis
                                           Discrete- Choice Modeling




                               4
Choosing Research Methods (2 of 3)

          Methods of Quantitative Research
Method            Advantages               Disadvantages
           •   Less bias              •   Significantly longer    There are pros and cons to every
           •   Respondents work           response time               form of data collection.
 Direct
               at own pace            •   No probing
  Mail
           •   Ensures anonymity      •   No control over
           •   Less expensive             response pool            The decision on which method is
           •   Strong response        •   No visual aids          best for your study requires you to
               rates                  •   Difficult to              accept the limitations of your
           •   Quicker than direct        establish rapport                chosen approach.
Phone
               mail                   •   Interviewer bias
           •   Sequence is flexible   •   Overused
           •   Allows for probing     •   Expensive
           •   Quickest response
               time                   •   Difficult to ensure
           •   Generally less             representation
Online
               expensive              •   Difficult to identify
           •   Allows for complex         usable sample
               branching

                                                  5
Choosing Research Methods (3 of 3)
            Example of an actual research plan for a project currently underway.

                                           Research Plan
  Target
                     Sampling Frame             Qualitative Research      Quantitative Research
 Audience

                                             4 online focus groups (2
                                                outside NYC-area
                                             undergraduate alumni/2          Online Survey
                                                outside NYC- area
                   All living alumni of
                                                graduate alumni)             (Discrete-choice
                  undergraduate and
 Alumni                                                                       modeling and
                graduate programs with                                   segmentation analysis
                                             2 in-person focus groups
                an email address on file                                will be included in online
                                                    (1 NYC-area
                                                                                  survey)
                                             undergraduate alumni/1
                                                NYC-area graduate
                                                      alumni)




                                                  6
Focus Group Basics
• Used to understand our audiences:
     • Feelings, perceptions, and needs
     • Motivations to give/not give                  Focus groups are generally more
• 8-12 members of target audience gathered for an   effective than surveys at exploring
  open-ended discussion about topic of interest              emotional drivers.

• One of the most frequently used market research
  techniques                                        Focus group generate qualitative
• Typically lasts 1 to 1 ½ hours; no longer          data which means the findings
                                                      are presented as key themes
• To ensure accuracy, conduct multiple groups        rather than through the use of
• Always audio or video tape                          statistics, as is the case with
                                                         quantitative research.
• Typical project has at least four groups
     • Same target audience, same guide, same
       moderator, etc.



                                              7
Focus Group Examples
• Fordham University
    • 4 online focus groups (2 with outside NYC undergraduate alumni; 2 with outside
      NYC graduate alumni)
    • 2 in-person focus groups (1 with NYC-area undergraduate alumni; 1 with NYC-area
      graduate alumni)
    • Used as precursor to quantitative analysis
• MIT Sloan School of Management
    • 4 online focus groups (1 with domestic undergraduate alumni outside Boston
      area, 1 with domestic graduate alumni, 1 with international undergraduate
      alumni, 1 with international graduate alumni)
    • 2 in-person focus groups (1 with undergraduate alumni in Boston area; 1 with
      graduate alumni in Boston area)
    • Used as precursor to quantitative analysis
• Alfred University
    • 4 online focus groups with alumni from the 1970’s and 1980’s (2 with alumni in
      western NY; 2 with alumni outside western NY)
                                           8
In-Person vs. Online Focus Groups
                 Advantages                    • Formal, traditional focus groups are
                                                 conducted in a focus group facility with
     In-Person                Online             two-way mirror
Ability to observe    Cheaper incentives       • Online focus groups are becoming an
body language         can be used                industry standard
Viewers can observe Save on cost of
                                               • Online focus groups typically involve:
using facility with focus group facility
two-way mirror                                     • The use of “webinar” software like
                                                     WebEx or Microsoft Live Meeting
Stronger “show”       Shorter recruiting
rate                  time                         • Participants dial into a toll-free
                                                     number and log onto a secure
Don’t have to deal    No geographic
                                                     server at the same time
with “technical       limitations
difficulties”                                      • The moderator can push
                                                     content/visuals via the computer
More flexibility to
                                                     screen and take polls
use exercises and
worksheets                                         • Audio/video recorded and accessed
                                                     online

                                           9
Developing the Moderator Guide (1 of 2)
• Must contain thoughtfully crafted questions to get
  the responses you need
     • Rushing through a long list of questions will    Most moderator guides are no
       not yield in-depth insights                       more than two pages long.
     • Developing these questions is a unique skill
       learned chiefly through experience
                                                        The moderator guide should be
• Open with simple, easy, fun, and compelling          filled with open-ended questions
  questions that get the group participating            or ones that begin with “how,”
• Address general questions toward the beginning,         “why,” “what,” “where,” and
  specific questions toward the end                                 “when.”
• Use exercises to break up “straight questioning”
  and avoid “group think” (polls, sticker book, role   A question that should be asked
  playing, visual aids/websites)                        frequently during the group is,
• Moderator’s role is to develop the right questions       “Why do you say that?”
  and facilitate the discussion to get at this
  information

                                             10
Developing the Moderator Guide (2 of 2)
              Excerpt from actual moderator guide.




                               11
Analyzing Results & Report Preparation
• Have the moderator and another reviewer listen
  to all tapes; compare notes
• Focus group findings are typically presented as      Clip from actual focus group.
  bullet points in a PowerPoint                       Question about decision to move
• Report key characteristics of participants to         printed Alumni Magazine to
  provide context                                                  online.
• Key findings are presented as themes; report only
  those themes that emerge very commonly across
  all groups
• Report all interesting non-themes as isolated
  opinion or “good ideas”
• Avoid temptation to present focus group findings
  using charts, graphs, and/or tables
• Use direct quotes and audio/video clips to
  reinforce key themes


                                             12
Survey Research Basics
• Close-ended questions
• Conducted over the phone, face-to-face, direct
  mail, or online                                      Quantitative research is about
• One of the most frequently used market research      getting the hard measures of a
  techniques                                                       market
• Typical instruments last 10-15 minutes, but alumni       • How important is…
  can withstand up to 20 minutes                       • What is your level of interest
• Good questionnaire design is extremely important;                  in…
  wording can dramatically change measurements          • What is the likelihood you
• Typical project has one online survey                             will…




                                           13
Stimulating Participation
• Sell value of research
• Guarantee confidentiality
• Avoid non-vital questions and long questionnaires    Incentives are a critical condition
• Avoid questions you already have the answers to in   when working to generate survey
  your own data records                                            response.

• Use an attractive design and layout
• Use preliminary notification                           Offer an incentive that your
                                                        target audience values and be
• Utilize reminders/call-backs                                     creative.
• Include return envelope with postage
• Use incentives; cash is the universal incentivizer




                                             14
Sampling
• Generally, seeking to secure a “representative
  sample” of our target population
• All else being equal, a larger sample size n        If you want 400 completed
  leads to increased precision in estimates of         surveys and expect a 40%
  various properties of the population             response rate, select a sample of
• But, larger samples are not necessarily better                1,000.
  because of the opportunities for non-sampling
  error
• To achieve desired sample size (n), make an
  educated guess as to what your response rate
  (r) will be; then select a sample that is n/r




                                            15
Response Rates
• Response rates directly linked to your target audience’s interest in you and your topic
• The higher their interest, the higher your response rate
• Typical response rate for alumni is between 15% to 25%
• Online tools such as Raosoft will generate recommended number of respondents
  determined by population size, desired margin of error (typically 5%), and desired
  confidence level (typically 95%)




                                             16
Designing Survey Instruments
• Exclude interesting, but not vital, questions
• Pay close attention to design, layout, and perceived ease of completion
• Avoid leading or ambiguous questions; maintain objectivity
• Open with simple, interesting questions
• Progressively narrow the scope of questions
• Place difficult, sensitive, or complex questions toward the end
• Include personal information/demographic questions last
• Plan time into the research process for extensive re-writing of the instrument
• Ask the HARD questions
• Typical instruments should not take longer than 10-15 minutes, but alumni can
  withstand up to 20 minutes (regardless of method of collection)




                                             17
Structuring Survey Questions (1 of 5)
WHAT’S WRONG HERE?                        SOLUTION
Q1. How interested are you in attending   Q1. Rate your level of interest in attending the following.
Reunion 2012?                                                        1 = Low
                                                                                2   3   4   5
                                                                                                6 = High
                                                                     interest                   interest
     •   1 = Low interest                   Reunion 2012
     •   2
     •   3                                  Homecoming 2012
     •   4
                                            Sports outing with
     •   5                                  fellow alumni
     •   6 = High interest
                                            Cultural outing with
                                            fellow alumni
Q2. How interested are you in attending
Homecoming 2012?
     •   1 = Low interest                            Group like questions together to
     •   2                                             reduce respondent burden.
     •   3
     •   4
     •   5                                Respondents can get
     •   6 = High interest                   “burned out.”

                                             18
Structuring Survey Questions (2 of 5)
WHAT’S WRONG HERE?
“Select your level of agreement with this statement. I am proud to be
an alumnus of SCHOOL because it provided me with job training,
allowed me to meet new people, and looks good on my resume.”            Three questions are being
                                                                               asked here.
     •   Strongly disagree
     •   Disagree
     •   Agree
     •   Strongly agree


SOLUTION
“Select your level of agreement with this statement. I am proud to be
an alumnus of SCHOOL because it provided me with job training.”          Be specific and isolate
     •   Strongly disagree                                                      issues.
     •   Disagree
     •   Agree
     •   Strongly agree




                                                 19
Structuring Survey Questions (3 of 5)
WHAT’S WRONG HERE?
“How far would you be willing to travel to an event sponsored by
SCHOOL?”
     •   Less than 50 miles
     •   50-100 miles                                              These response options are not
     •   100-150 miles                                                   mutually exclusive.
     •   150-200 miles
     •   More than 200 miles


SOLUTION
“How far would you be willing to travel to an event sponsored by
SCHOOL?”
     •   Less than 50 miles
     •   50-100 miles                                              Structure answer options so they
     •   101-150 miles                                             are independent of one another.
     •   151-200 miles
     •   More than 200 miles



                                                 20
Structuring Survey Questions (4 of 5)
WHAT’S WRONG HERE?
“What college/school within INSTITUION did you receive your degree
from?” _________________________
                                                                   This is too broad; respondents
                                                                   might use jargon or acronyms.
SOLUTION
“Select which college/school within INSTITUION you received your
degree from. ”
     •   College of Arts and Sciences
     •   Arthur Miller School of Business
     •   Nancy Peete School of Education                           This allows you to control the
     •   College of Engineering                                    responses, which will result in
                                                                        better data analysis.




                                                21
Structuring Survey Questions (5 of 5)
WHAT’S WRONG HERE?
“It is a good idea for SCHOOL to stop printing the Alumni Magazine.”
     •   Agree
                                                                 This question includes an applied
     •   Disagree
                                                                            assumption.

SOLUTION
“In your opinion, should SCHOOL transition the printed Alumni
Magazine to an online format?”
     • Yes                                                       This questions asks, rather than
     • No                                                          assumes, the opinion of the
                                                                respondent and is clear about the
                                                                implications of the resulting data.




                                                 22
Data Analysis
• Think about your data analysis as
                                              Question                 Data Analysis Plan
  you are preparing your instrument
                                       Indicate which information
• Create a “Data Analysis Plan” as      about SCHOOL you would
                                                                        Analyze results to build
                                                                         strategy for print and
  you finalize your instrument            be most interested in
                                                                      electronic communications.
                                             hearing about.
• Create “dummy tables” which
                                       Rate the likelihood that you
  mock up the tables and charts you                                     Analyze results to build
                                        will participate in each of
                                                                        strategy for on-and off-
  will prepare once your data is        the following activities in
                                                                            campus events.
  collected and analyzed                     the next 2 years.

• Include all information the reader
  needs to interpret the finding
  accurately when you are not
  around




                                       23
Alumni Network Analysis (1 of 3)
• The true leaders of your alumni pool may not necessarily be the ones who are current
  engaged with your institution
• Alumni network analysis reveals and nurtures the existing social networks among
  alumni
• Alumni are significantly influenced to attend or ignore events based on who is attending
  the program
• Alumni are asked to identify their most cherished relationship with peers as well as
  other at your institution
• Data is aggregated and the social network analysis, in addition to traditional statistical
  analyses, is conducted to reveal the maps of relationships among alumni
• Using the information revealed by this analysis the ones who are connected to the
  greatest number of fellow alumni can be tapped for event planning and other initiatives




                                              24
Alumni Network Analysis (2 of 3)




                           25
Alumni Network Analysis (3 of 3)




                                   Mr. Barry P is
                                    a “leader”
                                   because he is
                                   connected to
                                    many other
                                     alumni.




                           26
Segmentation Analysis
• Segmentation analysis takes the
  alumni’s attitudes about the
  institution (specifically their types
  of previous interaction and level of
  interest in future interactions) and
  seeks natural groupings of alumni
  based on those attitudes                       Biggest Fan                   Quiet Supporter
                                                                                                       Alumni Segments

                                                                                                                            Unengaged                                Critic
                                                   Supportive and                  Supportive but not                      Unengaged and not                 Unengaged and

• Those segments can be examined                     engaged                           engaged                              paying attention           intentionally not supportive



  with an eye towards identifying
  how they differ, both on the
  attitudes and all other measures of
                                                                                                  Segment Characteristics
  alumni captured in the research         •
                                          •
                                              3+ alumni events in last 2 yrs
                                              Member online alumni commy
                                                                               •
                                                                               •
                                                                                   Hired MIT Sloan alum
                                                                                   Lives in Northeast
                                                                                                                      •
                                                                                                                      •
                                                                                                                          MBA graduate
                                                                                                                          Lives in Midwest or abroad
                                                                                                                                                       •
                                                                                                                                                       •
                                                                                                                                                           MBA graduate
                                                                                                                                                           Lives in Midwest or abroad

  process                                 •
                                          •
                                          •
                                              Graduated 2000+
                                              Lives in Boston
                                              Email address on file
                                                                               •
                                                                               •
                                                                                   Graduated <2000
                                                                                   Email address on file
                                                                                                                      •
                                                                                                                      •
                                                                                                                          Earns <$150K annually
                                                                                                                          No email on file
                                                                                                                                                       •
                                                                                                                                                       •
                                                                                                                                                           Low student satisfaction rating
                                                                                                                                                           No email on file




• This helps you to understand the        •   18 contacts per year             •
                                                                                                    Communication Flow
                                                                                   16 contacts per year               • 12 contacts per year           • 5 contacts per year
                                          •   News updates                     •   News updates                       • News updates                   • Selected news updates

  groups based on their differences       •
                                          •
                                              Social and networking invites
                                              Invited to mentor students
                                                                               •
                                                                               •
                                                                                   Networking invites
                                                                                   Invited to meet with students in
                                                                                   small groups
                                                                                                                      • Invited to review resumes of
                                                                                                                        graduating students


  on a number of different
  attitudinal, behavioral, and
  demographic characteristics
                                                  27
Discrete-Choice Modeling (1 of 2)
• Discrete-choice modeling describes                                   In-Person Opportunity
   consumer decisions as they would be in               Factors                             Levels
   the “real world” where we always make                               For making professional connections
                                                   Why
                                                                       Primarily social in nature
   “trade-offs” in our decisions                   An opportunity is
                                                                       To increase my knowledge about topical
                                                   offered
                                                                       issues
• By observing respondent choices, you                                 Other SCHOOL alumni
   can determine:                                  Who
                                                                       Other SCHOOL alumni and faculty
                                                                       Other SCHOOL alumni and current students
     • What is important to them in the            attends the
                                                   opportunity
                                                                       SCHOOL President
                                                                       Other SCHOOL alumni and non-SCHOOL
        decision-making process regarding                              speakers
        attendance at in-person and online                             Monday through Thursday evening
                                                                       between 5 p.m.and 9 p.m.
        events and opportunities                                       Friday, Saturday, or Sunday evening
                                                   When
• Each respondent is exposed to four               the opportunity
                                                                       between 5 p.m. and 9 p.m.
                                                                       Saturday or Sunday between 12 p.m. and 5
  scenarios with two options each                  occurs
                                                                       p.m.
• They are forced to choose the activity in                            Monday through Friday morning between 8
                                                                       a.m. and 10 a.m.
  which they would be most likely to               Where               Less than 30 minutes from your home
  participate                                      an opportunity      30 minutes to an hour from your home
                                                   occurs              1 to 2 hours from your home
• One level within each of the factors                                 0-$10
  identified to the right were included in         How much
                                                                       $11-25
                                                                       $26-50
  each option                                      it costs
                                                                       $51-75
                                                                       Over $75
                                              28
Discrete-Choice Modeling (2 of 2)




                           29
How Much Will It Cost?
• Cost influenced by
    • Method of data collection
    • How much qualitative and quantitative      Alumni research can cost $35,000
      research is done                           to $150,000 for research strategy
    • Amount of work you are able to do                 and development.
      internally
• In-house vs. Outsourcing
    • What expertise exists internally?
    • Can internal provider be objective?
    • Can internal provider meet ambitious
      deadlines?




                                            30
Resources
• The Green Book – http://www.greenbook.org/
• Sample Research RFPs - http://simpsonscarborough.com/resources/sample-requests-
  for-proposals/
• Survey Monkey – http://www.surveymonkey.com/
• Question Pro – www.questionpro.com
• Inquisite Online Survey Software – http://www.inquisite.com/
• Qualtrics Online Survey Software - http://www.qualtrics.com/
• Marketing Research Association – http://www.mra-net.org/
• Survey Sampling – http://www.surveysampling.com/
• International Code of Marketing and Social Research Practice –
  http://www.esomar.org/uploads/pdf/ESOMAR_Codes&Guidelines_ICCCode.pdf
• Raosoft Sample Size Calculator - http://www.raosoft.com/samplesize.html




                                          31
Thank You!




    32

More Related Content

Similar to Will They Love You Tomorrow? Conducting Effective Alumni Research

Survey design-workshop-1234170716539145-3[1]
Survey design-workshop-1234170716539145-3[1]Survey design-workshop-1234170716539145-3[1]
Survey design-workshop-1234170716539145-3[1]
nemururati
 
SWK_340_Powerpoint_Chapter_11._Survey_research_rDeCarloTextbook.pptx
SWK_340_Powerpoint_Chapter_11._Survey_research_rDeCarloTextbook.pptxSWK_340_Powerpoint_Chapter_11._Survey_research_rDeCarloTextbook.pptx
SWK_340_Powerpoint_Chapter_11._Survey_research_rDeCarloTextbook.pptx
McPoolMac
 
Improving Library Resource Discovery
Improving Library Resource DiscoveryImproving Library Resource Discovery
Improving Library Resource Discovery
Danya Leebaw
 
Questionnaire design
Questionnaire designQuestionnaire design
Questionnaire design
chetan1923
 

Similar to Will They Love You Tomorrow? Conducting Effective Alumni Research (20)

A Framework for Health IT Evaluation
A Framework for Health IT EvaluationA Framework for Health IT Evaluation
A Framework for Health IT Evaluation
 
Introduction to UX Research: Fundamentals of Contextual Inquiry
Introduction to UX Research: Fundamentals of Contextual InquiryIntroduction to UX Research: Fundamentals of Contextual Inquiry
Introduction to UX Research: Fundamentals of Contextual Inquiry
 
User Research
User ResearchUser Research
User Research
 
Survey design-workshop-1234170716539145-3[1]
Survey design-workshop-1234170716539145-3[1]Survey design-workshop-1234170716539145-3[1]
Survey design-workshop-1234170716539145-3[1]
 
SWK_340_Powerpoint_Chapter_11._Survey_research_rDeCarloTextbook.pptx
SWK_340_Powerpoint_Chapter_11._Survey_research_rDeCarloTextbook.pptxSWK_340_Powerpoint_Chapter_11._Survey_research_rDeCarloTextbook.pptx
SWK_340_Powerpoint_Chapter_11._Survey_research_rDeCarloTextbook.pptx
 
Research process and sampling
Research process and samplingResearch process and sampling
Research process and sampling
 
Get the most out of getting out of the building
Get the most out of getting out of the buildingGet the most out of getting out of the building
Get the most out of getting out of the building
 
Session1 methods research_question
Session1 methods research_questionSession1 methods research_question
Session1 methods research_question
 
Improving Library Resource Discovery
Improving Library Resource DiscoveryImproving Library Resource Discovery
Improving Library Resource Discovery
 
eResearch activities brochure
eResearch activities brochureeResearch activities brochure
eResearch activities brochure
 
Putting Users in UX: Research Methods for Strategy
Putting Users in UX: Research Methods for StrategyPutting Users in UX: Research Methods for Strategy
Putting Users in UX: Research Methods for Strategy
 
User-centered UX: Bringing the User into the Design Process
User-centered UX: Bringing the User into the Design ProcessUser-centered UX: Bringing the User into the Design Process
User-centered UX: Bringing the User into the Design Process
 
UX Burlington 2017: Exploratory Research in UX Design
UX Burlington 2017: Exploratory Research in UX DesignUX Burlington 2017: Exploratory Research in UX Design
UX Burlington 2017: Exploratory Research in UX Design
 
Questionnaire design
Questionnaire designQuestionnaire design
Questionnaire design
 
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive WorkshopUsing Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
 
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive WorkshopUsing Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
 
Finalreview
FinalreviewFinalreview
Finalreview
 
eAssessment in practice
eAssessment in practiceeAssessment in practice
eAssessment in practice
 
Các phương pháp nghiên cứu thị trường - Market research methods
Các phương pháp nghiên cứu thị trường - Market research methodsCác phương pháp nghiên cứu thị trường - Market research methods
Các phương pháp nghiên cứu thị trường - Market research methods
 
Chapter 8.pptx
Chapter 8.pptxChapter 8.pptx
Chapter 8.pptx
 

Recently uploaded

Recently uploaded (20)

Morse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxMorse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptx
 
Advances in production technology of Grapes.pdf
Advances in production technology of Grapes.pdfAdvances in production technology of Grapes.pdf
Advances in production technology of Grapes.pdf
 
50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...
50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...
50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...
 
Pragya Champions Chalice 2024 Prelims & Finals Q/A set, General Quiz
Pragya Champions Chalice 2024 Prelims & Finals Q/A set, General QuizPragya Champions Chalice 2024 Prelims & Finals Q/A set, General Quiz
Pragya Champions Chalice 2024 Prelims & Finals Q/A set, General Quiz
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).
 
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
Operations Management - Book1.p  - Dr. Abdulfatah A. SalemOperations Management - Book1.p  - Dr. Abdulfatah A. Salem
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
 
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptxMatatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
 
An Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptxAn Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptx
 
....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf
 
Word Stress rules esl .pptx
Word Stress rules esl               .pptxWord Stress rules esl               .pptx
Word Stress rules esl .pptx
 
“O BEIJO” EM ARTE .
“O BEIJO” EM ARTE                       .“O BEIJO” EM ARTE                       .
“O BEIJO” EM ARTE .
 
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptxslides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
 
Benefits and Challenges of Using Open Educational Resources
Benefits and Challenges of Using Open Educational ResourcesBenefits and Challenges of Using Open Educational Resources
Benefits and Challenges of Using Open Educational Resources
 
How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17
 
The Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve ThomasonThe Art Pastor's Guide to Sabbath | Steve Thomason
The Art Pastor's Guide to Sabbath | Steve Thomason
 
[GDSC YCCE] Build with AI Online Presentation
[GDSC YCCE] Build with AI Online Presentation[GDSC YCCE] Build with AI Online Presentation
[GDSC YCCE] Build with AI Online Presentation
 
Open Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPointOpen Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPoint
 
Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17
 
B.ed spl. HI pdusu exam paper-2023-24.pdf
B.ed spl. HI pdusu exam paper-2023-24.pdfB.ed spl. HI pdusu exam paper-2023-24.pdf
B.ed spl. HI pdusu exam paper-2023-24.pdf
 
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdfINU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
INU_CAPSTONEDESIGN_비밀번호486_업로드용 발표자료.pdf
 

Will They Love You Tomorrow? Conducting Effective Alumni Research

  • 1. Will They Love You Tomorrow? Conducting Effective Alumni Research Renee Kart Director of Project Strategy SimpsonScarborough June 9, 2011
  • 2. What We Want to Know • What type of relationship do alumni seek to have with our institution? • How can we do a better job of engaging alumni through communications and activities? • What services do alumni need and want from us? • How do our alumni want us to communicate with them? How do they want to communicate with us? • How effective are our institution’s print and electronic communication strategies? • How do our alumni feel about their experience at our institution? • What messages about our institution build a sense of pride for our alumni? • Are there ways that we can model alumni engagement to make better decisions about resource allocation? • What types of events will strengthen relationships with alumni? • How should our institution segment our alumni to maximize the effectiveness of our efforts to engage them? • Who are the leaders among our alumni? 2
  • 3. Establishing Guiding Questions • “Guiding Questions” are the 5-8 broad questions you want your research to answer • If you have more than 8, your study probably Many survey research studies try lacks focus to gather information about too • Make the hard trade-offs (a “must know” vs. many topics rather than delving “nice to know”); no laundry lists deeply into one or two pressing • They are not actual questions that would be asked issues. during a focus group, in-depth interview, or survey • Imagine they could be the section headers on your After you develop your guiding PowerPoint of the findings questions, you will be able to • Get buy-in from other stakeholders decide if qualitative or quantitative research (or both) is • List the guiding questions on all communications best to address your needs. regarding the project; include them at the top of every draft of your moderator guide, in-depth interview script, or survey instruments 3
  • 4. Choosing Research Methods (1 of 3) Research Methods Focus Group Qualitative (In-Person or Online) In-Depth Interviews Direct Mail Quantitative Phone Survey Online Survey Alumni Network Analysis Other Segmentation Analysis Discrete- Choice Modeling 4
  • 5. Choosing Research Methods (2 of 3) Methods of Quantitative Research Method Advantages Disadvantages • Less bias • Significantly longer There are pros and cons to every • Respondents work response time form of data collection. Direct at own pace • No probing Mail • Ensures anonymity • No control over • Less expensive response pool The decision on which method is • Strong response • No visual aids best for your study requires you to rates • Difficult to accept the limitations of your • Quicker than direct establish rapport chosen approach. Phone mail • Interviewer bias • Sequence is flexible • Overused • Allows for probing • Expensive • Quickest response time • Difficult to ensure • Generally less representation Online expensive • Difficult to identify • Allows for complex usable sample branching 5
  • 6. Choosing Research Methods (3 of 3) Example of an actual research plan for a project currently underway. Research Plan Target Sampling Frame Qualitative Research Quantitative Research Audience 4 online focus groups (2 outside NYC-area undergraduate alumni/2 Online Survey outside NYC- area All living alumni of graduate alumni) (Discrete-choice undergraduate and Alumni modeling and graduate programs with segmentation analysis 2 in-person focus groups an email address on file will be included in online (1 NYC-area survey) undergraduate alumni/1 NYC-area graduate alumni) 6
  • 7. Focus Group Basics • Used to understand our audiences: • Feelings, perceptions, and needs • Motivations to give/not give Focus groups are generally more • 8-12 members of target audience gathered for an effective than surveys at exploring open-ended discussion about topic of interest emotional drivers. • One of the most frequently used market research techniques Focus group generate qualitative • Typically lasts 1 to 1 ½ hours; no longer data which means the findings are presented as key themes • To ensure accuracy, conduct multiple groups rather than through the use of • Always audio or video tape statistics, as is the case with quantitative research. • Typical project has at least four groups • Same target audience, same guide, same moderator, etc. 7
  • 8. Focus Group Examples • Fordham University • 4 online focus groups (2 with outside NYC undergraduate alumni; 2 with outside NYC graduate alumni) • 2 in-person focus groups (1 with NYC-area undergraduate alumni; 1 with NYC-area graduate alumni) • Used as precursor to quantitative analysis • MIT Sloan School of Management • 4 online focus groups (1 with domestic undergraduate alumni outside Boston area, 1 with domestic graduate alumni, 1 with international undergraduate alumni, 1 with international graduate alumni) • 2 in-person focus groups (1 with undergraduate alumni in Boston area; 1 with graduate alumni in Boston area) • Used as precursor to quantitative analysis • Alfred University • 4 online focus groups with alumni from the 1970’s and 1980’s (2 with alumni in western NY; 2 with alumni outside western NY) 8
  • 9. In-Person vs. Online Focus Groups Advantages • Formal, traditional focus groups are conducted in a focus group facility with In-Person Online two-way mirror Ability to observe Cheaper incentives • Online focus groups are becoming an body language can be used industry standard Viewers can observe Save on cost of • Online focus groups typically involve: using facility with focus group facility two-way mirror • The use of “webinar” software like WebEx or Microsoft Live Meeting Stronger “show” Shorter recruiting rate time • Participants dial into a toll-free number and log onto a secure Don’t have to deal No geographic server at the same time with “technical limitations difficulties” • The moderator can push content/visuals via the computer More flexibility to screen and take polls use exercises and worksheets • Audio/video recorded and accessed online 9
  • 10. Developing the Moderator Guide (1 of 2) • Must contain thoughtfully crafted questions to get the responses you need • Rushing through a long list of questions will Most moderator guides are no not yield in-depth insights more than two pages long. • Developing these questions is a unique skill learned chiefly through experience The moderator guide should be • Open with simple, easy, fun, and compelling filled with open-ended questions questions that get the group participating or ones that begin with “how,” • Address general questions toward the beginning, “why,” “what,” “where,” and specific questions toward the end “when.” • Use exercises to break up “straight questioning” and avoid “group think” (polls, sticker book, role A question that should be asked playing, visual aids/websites) frequently during the group is, • Moderator’s role is to develop the right questions “Why do you say that?” and facilitate the discussion to get at this information 10
  • 11. Developing the Moderator Guide (2 of 2) Excerpt from actual moderator guide. 11
  • 12. Analyzing Results & Report Preparation • Have the moderator and another reviewer listen to all tapes; compare notes • Focus group findings are typically presented as Clip from actual focus group. bullet points in a PowerPoint Question about decision to move • Report key characteristics of participants to printed Alumni Magazine to provide context online. • Key findings are presented as themes; report only those themes that emerge very commonly across all groups • Report all interesting non-themes as isolated opinion or “good ideas” • Avoid temptation to present focus group findings using charts, graphs, and/or tables • Use direct quotes and audio/video clips to reinforce key themes 12
  • 13. Survey Research Basics • Close-ended questions • Conducted over the phone, face-to-face, direct mail, or online Quantitative research is about • One of the most frequently used market research getting the hard measures of a techniques market • Typical instruments last 10-15 minutes, but alumni • How important is… can withstand up to 20 minutes • What is your level of interest • Good questionnaire design is extremely important; in… wording can dramatically change measurements • What is the likelihood you • Typical project has one online survey will… 13
  • 14. Stimulating Participation • Sell value of research • Guarantee confidentiality • Avoid non-vital questions and long questionnaires Incentives are a critical condition • Avoid questions you already have the answers to in when working to generate survey your own data records response. • Use an attractive design and layout • Use preliminary notification Offer an incentive that your target audience values and be • Utilize reminders/call-backs creative. • Include return envelope with postage • Use incentives; cash is the universal incentivizer 14
  • 15. Sampling • Generally, seeking to secure a “representative sample” of our target population • All else being equal, a larger sample size n If you want 400 completed leads to increased precision in estimates of surveys and expect a 40% various properties of the population response rate, select a sample of • But, larger samples are not necessarily better 1,000. because of the opportunities for non-sampling error • To achieve desired sample size (n), make an educated guess as to what your response rate (r) will be; then select a sample that is n/r 15
  • 16. Response Rates • Response rates directly linked to your target audience’s interest in you and your topic • The higher their interest, the higher your response rate • Typical response rate for alumni is between 15% to 25% • Online tools such as Raosoft will generate recommended number of respondents determined by population size, desired margin of error (typically 5%), and desired confidence level (typically 95%) 16
  • 17. Designing Survey Instruments • Exclude interesting, but not vital, questions • Pay close attention to design, layout, and perceived ease of completion • Avoid leading or ambiguous questions; maintain objectivity • Open with simple, interesting questions • Progressively narrow the scope of questions • Place difficult, sensitive, or complex questions toward the end • Include personal information/demographic questions last • Plan time into the research process for extensive re-writing of the instrument • Ask the HARD questions • Typical instruments should not take longer than 10-15 minutes, but alumni can withstand up to 20 minutes (regardless of method of collection) 17
  • 18. Structuring Survey Questions (1 of 5) WHAT’S WRONG HERE? SOLUTION Q1. How interested are you in attending Q1. Rate your level of interest in attending the following. Reunion 2012? 1 = Low 2 3 4 5 6 = High interest interest • 1 = Low interest Reunion 2012 • 2 • 3 Homecoming 2012 • 4 Sports outing with • 5 fellow alumni • 6 = High interest Cultural outing with fellow alumni Q2. How interested are you in attending Homecoming 2012? • 1 = Low interest Group like questions together to • 2 reduce respondent burden. • 3 • 4 • 5 Respondents can get • 6 = High interest “burned out.” 18
  • 19. Structuring Survey Questions (2 of 5) WHAT’S WRONG HERE? “Select your level of agreement with this statement. I am proud to be an alumnus of SCHOOL because it provided me with job training, allowed me to meet new people, and looks good on my resume.” Three questions are being asked here. • Strongly disagree • Disagree • Agree • Strongly agree SOLUTION “Select your level of agreement with this statement. I am proud to be an alumnus of SCHOOL because it provided me with job training.” Be specific and isolate • Strongly disagree issues. • Disagree • Agree • Strongly agree 19
  • 20. Structuring Survey Questions (3 of 5) WHAT’S WRONG HERE? “How far would you be willing to travel to an event sponsored by SCHOOL?” • Less than 50 miles • 50-100 miles These response options are not • 100-150 miles mutually exclusive. • 150-200 miles • More than 200 miles SOLUTION “How far would you be willing to travel to an event sponsored by SCHOOL?” • Less than 50 miles • 50-100 miles Structure answer options so they • 101-150 miles are independent of one another. • 151-200 miles • More than 200 miles 20
  • 21. Structuring Survey Questions (4 of 5) WHAT’S WRONG HERE? “What college/school within INSTITUION did you receive your degree from?” _________________________ This is too broad; respondents might use jargon or acronyms. SOLUTION “Select which college/school within INSTITUION you received your degree from. ” • College of Arts and Sciences • Arthur Miller School of Business • Nancy Peete School of Education This allows you to control the • College of Engineering responses, which will result in better data analysis. 21
  • 22. Structuring Survey Questions (5 of 5) WHAT’S WRONG HERE? “It is a good idea for SCHOOL to stop printing the Alumni Magazine.” • Agree This question includes an applied • Disagree assumption. SOLUTION “In your opinion, should SCHOOL transition the printed Alumni Magazine to an online format?” • Yes This questions asks, rather than • No assumes, the opinion of the respondent and is clear about the implications of the resulting data. 22
  • 23. Data Analysis • Think about your data analysis as Question Data Analysis Plan you are preparing your instrument Indicate which information • Create a “Data Analysis Plan” as about SCHOOL you would Analyze results to build strategy for print and you finalize your instrument be most interested in electronic communications. hearing about. • Create “dummy tables” which Rate the likelihood that you mock up the tables and charts you Analyze results to build will participate in each of strategy for on-and off- will prepare once your data is the following activities in campus events. collected and analyzed the next 2 years. • Include all information the reader needs to interpret the finding accurately when you are not around 23
  • 24. Alumni Network Analysis (1 of 3) • The true leaders of your alumni pool may not necessarily be the ones who are current engaged with your institution • Alumni network analysis reveals and nurtures the existing social networks among alumni • Alumni are significantly influenced to attend or ignore events based on who is attending the program • Alumni are asked to identify their most cherished relationship with peers as well as other at your institution • Data is aggregated and the social network analysis, in addition to traditional statistical analyses, is conducted to reveal the maps of relationships among alumni • Using the information revealed by this analysis the ones who are connected to the greatest number of fellow alumni can be tapped for event planning and other initiatives 24
  • 25. Alumni Network Analysis (2 of 3) 25
  • 26. Alumni Network Analysis (3 of 3) Mr. Barry P is a “leader” because he is connected to many other alumni. 26
  • 27. Segmentation Analysis • Segmentation analysis takes the alumni’s attitudes about the institution (specifically their types of previous interaction and level of interest in future interactions) and seeks natural groupings of alumni based on those attitudes Biggest Fan Quiet Supporter Alumni Segments Unengaged Critic Supportive and Supportive but not Unengaged and not Unengaged and • Those segments can be examined engaged engaged paying attention intentionally not supportive with an eye towards identifying how they differ, both on the attitudes and all other measures of Segment Characteristics alumni captured in the research • • 3+ alumni events in last 2 yrs Member online alumni commy • • Hired MIT Sloan alum Lives in Northeast • • MBA graduate Lives in Midwest or abroad • • MBA graduate Lives in Midwest or abroad process • • • Graduated 2000+ Lives in Boston Email address on file • • Graduated <2000 Email address on file • • Earns <$150K annually No email on file • • Low student satisfaction rating No email on file • This helps you to understand the • 18 contacts per year • Communication Flow 16 contacts per year • 12 contacts per year • 5 contacts per year • News updates • News updates • News updates • Selected news updates groups based on their differences • • Social and networking invites Invited to mentor students • • Networking invites Invited to meet with students in small groups • Invited to review resumes of graduating students on a number of different attitudinal, behavioral, and demographic characteristics 27
  • 28. Discrete-Choice Modeling (1 of 2) • Discrete-choice modeling describes In-Person Opportunity consumer decisions as they would be in Factors Levels the “real world” where we always make For making professional connections Why Primarily social in nature “trade-offs” in our decisions An opportunity is To increase my knowledge about topical offered issues • By observing respondent choices, you Other SCHOOL alumni can determine: Who Other SCHOOL alumni and faculty Other SCHOOL alumni and current students • What is important to them in the attends the opportunity SCHOOL President Other SCHOOL alumni and non-SCHOOL decision-making process regarding speakers attendance at in-person and online Monday through Thursday evening between 5 p.m.and 9 p.m. events and opportunities Friday, Saturday, or Sunday evening When • Each respondent is exposed to four the opportunity between 5 p.m. and 9 p.m. Saturday or Sunday between 12 p.m. and 5 scenarios with two options each occurs p.m. • They are forced to choose the activity in Monday through Friday morning between 8 a.m. and 10 a.m. which they would be most likely to Where Less than 30 minutes from your home participate an opportunity 30 minutes to an hour from your home occurs 1 to 2 hours from your home • One level within each of the factors 0-$10 identified to the right were included in How much $11-25 $26-50 each option it costs $51-75 Over $75 28
  • 30. How Much Will It Cost? • Cost influenced by • Method of data collection • How much qualitative and quantitative Alumni research can cost $35,000 research is done to $150,000 for research strategy • Amount of work you are able to do and development. internally • In-house vs. Outsourcing • What expertise exists internally? • Can internal provider be objective? • Can internal provider meet ambitious deadlines? 30
  • 31. Resources • The Green Book – http://www.greenbook.org/ • Sample Research RFPs - http://simpsonscarborough.com/resources/sample-requests- for-proposals/ • Survey Monkey – http://www.surveymonkey.com/ • Question Pro – www.questionpro.com • Inquisite Online Survey Software – http://www.inquisite.com/ • Qualtrics Online Survey Software - http://www.qualtrics.com/ • Marketing Research Association – http://www.mra-net.org/ • Survey Sampling – http://www.surveysampling.com/ • International Code of Marketing and Social Research Practice – http://www.esomar.org/uploads/pdf/ESOMAR_Codes&Guidelines_ICCCode.pdf • Raosoft Sample Size Calculator - http://www.raosoft.com/samplesize.html 31