Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Will They Love You Tomorrow? Conducting Effective Alumni Research


Published on

Presentation by Renee Kart, Director of Project Strategy at Simpson Scarborough, given at the 2011 SUNYCUAD Conference held in Saratoga Springs, NY on June 9.

Published in: Education, Business, Technology
  • Be the first to comment

Will They Love You Tomorrow? Conducting Effective Alumni Research

  1. 1. Will They Love You Tomorrow?Conducting Effective Alumni ResearchRenee KartDirector of Project StrategySimpsonScarboroughJune 9, 2011
  2. 2. What We Want to Know• What type of relationship do alumni seek to have with our institution?• How can we do a better job of engaging alumni through communications and activities?• What services do alumni need and want from us?• How do our alumni want us to communicate with them? How do they want to communicate with us?• How effective are our institution’s print and electronic communication strategies?• How do our alumni feel about their experience at our institution?• What messages about our institution build a sense of pride for our alumni?• Are there ways that we can model alumni engagement to make better decisions about resource allocation?• What types of events will strengthen relationships with alumni?• How should our institution segment our alumni to maximize the effectiveness of our efforts to engage them?• Who are the leaders among our alumni? 2
  3. 3. Establishing Guiding Questions• “Guiding Questions” are the 5-8 broad questions you want your research to answer • If you have more than 8, your study probably Many survey research studies try lacks focus to gather information about too • Make the hard trade-offs (a “must know” vs. many topics rather than delving “nice to know”); no laundry lists deeply into one or two pressing• They are not actual questions that would be asked issues. during a focus group, in-depth interview, or survey• Imagine they could be the section headers on your After you develop your guiding PowerPoint of the findings questions, you will be able to• Get buy-in from other stakeholders decide if qualitative or quantitative research (or both) is• List the guiding questions on all communications best to address your needs. regarding the project; include them at the top of every draft of your moderator guide, in-depth interview script, or survey instruments 3
  4. 4. Choosing Research Methods (1 of 3) Research Methods Focus Group Qualitative (In-Person or Online) In-Depth Interviews Direct Mail Quantitative Phone Survey Online Survey Alumni Network Analysis Other Segmentation Analysis Discrete- Choice Modeling 4
  5. 5. Choosing Research Methods (2 of 3) Methods of Quantitative ResearchMethod Advantages Disadvantages • Less bias • Significantly longer There are pros and cons to every • Respondents work response time form of data collection. Direct at own pace • No probing Mail • Ensures anonymity • No control over • Less expensive response pool The decision on which method is • Strong response • No visual aids best for your study requires you to rates • Difficult to accept the limitations of your • Quicker than direct establish rapport chosen approach.Phone mail • Interviewer bias • Sequence is flexible • Overused • Allows for probing • Expensive • Quickest response time • Difficult to ensure • Generally less representationOnline expensive • Difficult to identify • Allows for complex usable sample branching 5
  6. 6. Choosing Research Methods (3 of 3) Example of an actual research plan for a project currently underway. Research Plan Target Sampling Frame Qualitative Research Quantitative Research Audience 4 online focus groups (2 outside NYC-area undergraduate alumni/2 Online Survey outside NYC- area All living alumni of graduate alumni) (Discrete-choice undergraduate and Alumni modeling and graduate programs with segmentation analysis 2 in-person focus groups an email address on file will be included in online (1 NYC-area survey) undergraduate alumni/1 NYC-area graduate alumni) 6
  7. 7. Focus Group Basics• Used to understand our audiences: • Feelings, perceptions, and needs • Motivations to give/not give Focus groups are generally more• 8-12 members of target audience gathered for an effective than surveys at exploring open-ended discussion about topic of interest emotional drivers.• One of the most frequently used market research techniques Focus group generate qualitative• Typically lasts 1 to 1 ½ hours; no longer data which means the findings are presented as key themes• To ensure accuracy, conduct multiple groups rather than through the use of• Always audio or video tape statistics, as is the case with quantitative research.• Typical project has at least four groups • Same target audience, same guide, same moderator, etc. 7
  8. 8. Focus Group Examples• Fordham University • 4 online focus groups (2 with outside NYC undergraduate alumni; 2 with outside NYC graduate alumni) • 2 in-person focus groups (1 with NYC-area undergraduate alumni; 1 with NYC-area graduate alumni) • Used as precursor to quantitative analysis• MIT Sloan School of Management • 4 online focus groups (1 with domestic undergraduate alumni outside Boston area, 1 with domestic graduate alumni, 1 with international undergraduate alumni, 1 with international graduate alumni) • 2 in-person focus groups (1 with undergraduate alumni in Boston area; 1 with graduate alumni in Boston area) • Used as precursor to quantitative analysis• Alfred University • 4 online focus groups with alumni from the 1970’s and 1980’s (2 with alumni in western NY; 2 with alumni outside western NY) 8
  9. 9. In-Person vs. Online Focus Groups Advantages • Formal, traditional focus groups are conducted in a focus group facility with In-Person Online two-way mirrorAbility to observe Cheaper incentives • Online focus groups are becoming anbody language can be used industry standardViewers can observe Save on cost of • Online focus groups typically involve:using facility with focus group facilitytwo-way mirror • The use of “webinar” software like WebEx or Microsoft Live MeetingStronger “show” Shorter recruitingrate time • Participants dial into a toll-free number and log onto a secureDon’t have to deal No geographic server at the same timewith “technical limitationsdifficulties” • The moderator can push content/visuals via the computerMore flexibility to screen and take pollsuse exercises andworksheets • Audio/video recorded and accessed online 9
  10. 10. Developing the Moderator Guide (1 of 2)• Must contain thoughtfully crafted questions to get the responses you need • Rushing through a long list of questions will Most moderator guides are no not yield in-depth insights more than two pages long. • Developing these questions is a unique skill learned chiefly through experience The moderator guide should be• Open with simple, easy, fun, and compelling filled with open-ended questions questions that get the group participating or ones that begin with “how,”• Address general questions toward the beginning, “why,” “what,” “where,” and specific questions toward the end “when.”• Use exercises to break up “straight questioning” and avoid “group think” (polls, sticker book, role A question that should be asked playing, visual aids/websites) frequently during the group is,• Moderator’s role is to develop the right questions “Why do you say that?” and facilitate the discussion to get at this information 10
  11. 11. Developing the Moderator Guide (2 of 2) Excerpt from actual moderator guide. 11
  12. 12. Analyzing Results & Report Preparation• Have the moderator and another reviewer listen to all tapes; compare notes• Focus group findings are typically presented as Clip from actual focus group. bullet points in a PowerPoint Question about decision to move• Report key characteristics of participants to printed Alumni Magazine to provide context online.• Key findings are presented as themes; report only those themes that emerge very commonly across all groups• Report all interesting non-themes as isolated opinion or “good ideas”• Avoid temptation to present focus group findings using charts, graphs, and/or tables• Use direct quotes and audio/video clips to reinforce key themes 12
  13. 13. Survey Research Basics• Close-ended questions• Conducted over the phone, face-to-face, direct mail, or online Quantitative research is about• One of the most frequently used market research getting the hard measures of a techniques market• Typical instruments last 10-15 minutes, but alumni • How important is… can withstand up to 20 minutes • What is your level of interest• Good questionnaire design is extremely important; in… wording can dramatically change measurements • What is the likelihood you• Typical project has one online survey will… 13
  14. 14. Stimulating Participation• Sell value of research• Guarantee confidentiality• Avoid non-vital questions and long questionnaires Incentives are a critical condition• Avoid questions you already have the answers to in when working to generate survey your own data records response.• Use an attractive design and layout• Use preliminary notification Offer an incentive that your target audience values and be• Utilize reminders/call-backs creative.• Include return envelope with postage• Use incentives; cash is the universal incentivizer 14
  15. 15. Sampling• Generally, seeking to secure a “representative sample” of our target population• All else being equal, a larger sample size n If you want 400 completed leads to increased precision in estimates of surveys and expect a 40% various properties of the population response rate, select a sample of• But, larger samples are not necessarily better 1,000. because of the opportunities for non-sampling error• To achieve desired sample size (n), make an educated guess as to what your response rate (r) will be; then select a sample that is n/r 15
  16. 16. Response Rates• Response rates directly linked to your target audience’s interest in you and your topic• The higher their interest, the higher your response rate• Typical response rate for alumni is between 15% to 25%• Online tools such as Raosoft will generate recommended number of respondents determined by population size, desired margin of error (typically 5%), and desired confidence level (typically 95%) 16
  17. 17. Designing Survey Instruments• Exclude interesting, but not vital, questions• Pay close attention to design, layout, and perceived ease of completion• Avoid leading or ambiguous questions; maintain objectivity• Open with simple, interesting questions• Progressively narrow the scope of questions• Place difficult, sensitive, or complex questions toward the end• Include personal information/demographic questions last• Plan time into the research process for extensive re-writing of the instrument• Ask the HARD questions• Typical instruments should not take longer than 10-15 minutes, but alumni can withstand up to 20 minutes (regardless of method of collection) 17
  18. 18. Structuring Survey Questions (1 of 5)WHAT’S WRONG HERE? SOLUTIONQ1. How interested are you in attending Q1. Rate your level of interest in attending the following.Reunion 2012? 1 = Low 2 3 4 5 6 = High interest interest • 1 = Low interest Reunion 2012 • 2 • 3 Homecoming 2012 • 4 Sports outing with • 5 fellow alumni • 6 = High interest Cultural outing with fellow alumniQ2. How interested are you in attendingHomecoming 2012? • 1 = Low interest Group like questions together to • 2 reduce respondent burden. • 3 • 4 • 5 Respondents can get • 6 = High interest “burned out.” 18
  19. 19. Structuring Survey Questions (2 of 5)WHAT’S WRONG HERE?“Select your level of agreement with this statement. I am proud to bean alumnus of SCHOOL because it provided me with job training,allowed me to meet new people, and looks good on my resume.” Three questions are being asked here. • Strongly disagree • Disagree • Agree • Strongly agreeSOLUTION“Select your level of agreement with this statement. I am proud to bean alumnus of SCHOOL because it provided me with job training.” Be specific and isolate • Strongly disagree issues. • Disagree • Agree • Strongly agree 19
  20. 20. Structuring Survey Questions (3 of 5)WHAT’S WRONG HERE?“How far would you be willing to travel to an event sponsored bySCHOOL?” • Less than 50 miles • 50-100 miles These response options are not • 100-150 miles mutually exclusive. • 150-200 miles • More than 200 milesSOLUTION“How far would you be willing to travel to an event sponsored bySCHOOL?” • Less than 50 miles • 50-100 miles Structure answer options so they • 101-150 miles are independent of one another. • 151-200 miles • More than 200 miles 20
  21. 21. Structuring Survey Questions (4 of 5)WHAT’S WRONG HERE?“What college/school within INSTITUION did you receive your degreefrom?” _________________________ This is too broad; respondents might use jargon or acronyms.SOLUTION“Select which college/school within INSTITUION you received yourdegree from. ” • College of Arts and Sciences • Arthur Miller School of Business • Nancy Peete School of Education This allows you to control the • College of Engineering responses, which will result in better data analysis. 21
  22. 22. Structuring Survey Questions (5 of 5)WHAT’S WRONG HERE?“It is a good idea for SCHOOL to stop printing the Alumni Magazine.” • Agree This question includes an applied • Disagree assumption.SOLUTION“In your opinion, should SCHOOL transition the printed AlumniMagazine to an online format?” • Yes This questions asks, rather than • No assumes, the opinion of the respondent and is clear about the implications of the resulting data. 22
  23. 23. Data Analysis• Think about your data analysis as Question Data Analysis Plan you are preparing your instrument Indicate which information• Create a “Data Analysis Plan” as about SCHOOL you would Analyze results to build strategy for print and you finalize your instrument be most interested in electronic communications. hearing about.• Create “dummy tables” which Rate the likelihood that you mock up the tables and charts you Analyze results to build will participate in each of strategy for on-and off- will prepare once your data is the following activities in campus events. collected and analyzed the next 2 years.• Include all information the reader needs to interpret the finding accurately when you are not around 23
  24. 24. Alumni Network Analysis (1 of 3)• The true leaders of your alumni pool may not necessarily be the ones who are current engaged with your institution• Alumni network analysis reveals and nurtures the existing social networks among alumni• Alumni are significantly influenced to attend or ignore events based on who is attending the program• Alumni are asked to identify their most cherished relationship with peers as well as other at your institution• Data is aggregated and the social network analysis, in addition to traditional statistical analyses, is conducted to reveal the maps of relationships among alumni• Using the information revealed by this analysis the ones who are connected to the greatest number of fellow alumni can be tapped for event planning and other initiatives 24
  25. 25. Alumni Network Analysis (2 of 3) 25
  26. 26. Alumni Network Analysis (3 of 3) Mr. Barry P is a “leader” because he is connected to many other alumni. 26
  27. 27. Segmentation Analysis• Segmentation analysis takes the alumni’s attitudes about the institution (specifically their types of previous interaction and level of interest in future interactions) and seeks natural groupings of alumni based on those attitudes Biggest Fan Quiet Supporter Alumni Segments Unengaged Critic Supportive and Supportive but not Unengaged and not Unengaged and• Those segments can be examined engaged engaged paying attention intentionally not supportive with an eye towards identifying how they differ, both on the attitudes and all other measures of Segment Characteristics alumni captured in the research • • 3+ alumni events in last 2 yrs Member online alumni commy • • Hired MIT Sloan alum Lives in Northeast • • MBA graduate Lives in Midwest or abroad • • MBA graduate Lives in Midwest or abroad process • • • Graduated 2000+ Lives in Boston Email address on file • • Graduated <2000 Email address on file • • Earns <$150K annually No email on file • • Low student satisfaction rating No email on file• This helps you to understand the • 18 contacts per year • Communication Flow 16 contacts per year • 12 contacts per year • 5 contacts per year • News updates • News updates • News updates • Selected news updates groups based on their differences • • Social and networking invites Invited to mentor students • • Networking invites Invited to meet with students in small groups • Invited to review resumes of graduating students on a number of different attitudinal, behavioral, and demographic characteristics 27
  28. 28. Discrete-Choice Modeling (1 of 2)• Discrete-choice modeling describes In-Person Opportunity consumer decisions as they would be in Factors Levels the “real world” where we always make For making professional connections Why Primarily social in nature “trade-offs” in our decisions An opportunity is To increase my knowledge about topical offered issues• By observing respondent choices, you Other SCHOOL alumni can determine: Who Other SCHOOL alumni and faculty Other SCHOOL alumni and current students • What is important to them in the attends the opportunity SCHOOL President Other SCHOOL alumni and non-SCHOOL decision-making process regarding speakers attendance at in-person and online Monday through Thursday evening between 5 p.m.and 9 p.m. events and opportunities Friday, Saturday, or Sunday evening When• Each respondent is exposed to four the opportunity between 5 p.m. and 9 p.m. Saturday or Sunday between 12 p.m. and 5 scenarios with two options each occurs p.m.• They are forced to choose the activity in Monday through Friday morning between 8 a.m. and 10 a.m. which they would be most likely to Where Less than 30 minutes from your home participate an opportunity 30 minutes to an hour from your home occurs 1 to 2 hours from your home• One level within each of the factors 0-$10 identified to the right were included in How much $11-25 $26-50 each option it costs $51-75 Over $75 28
  29. 29. Discrete-Choice Modeling (2 of 2) 29
  30. 30. How Much Will It Cost?• Cost influenced by • Method of data collection • How much qualitative and quantitative Alumni research can cost $35,000 research is done to $150,000 for research strategy • Amount of work you are able to do and development. internally• In-house vs. Outsourcing • What expertise exists internally? • Can internal provider be objective? • Can internal provider meet ambitious deadlines? 30
  31. 31. Resources• The Green Book –• Sample Research RFPs - for-proposals/• Survey Monkey –• Question Pro –• Inquisite Online Survey Software –• Qualtrics Online Survey Software -• Marketing Research Association –• Survey Sampling –• International Code of Marketing and Social Research Practice –• Raosoft Sample Size Calculator - 31
  32. 32. Thank You! 32