AaE Final Findings And Decisions


Published on

Findings of the eXtension Ask an Expert Task Force, June 2009.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

AaE Final Findings And Decisions

  1. 1. eXtension Ask an Expert Task Force: Findings and Decisions Final Report June 3, 2009 Commitee Membership: Kevin Gamble, Chair Rick Durham Mike Lambur Larry Lippke Ben MacNeill Terry Meisenbach Beth Raney Craig Wood
  2. 2. Ask an Expert Task Force Report Introduction Since its beginnings, eXtension has included an Ask an Expert (AaE) function as part of its portfolio of tools available for public information gathering. Concern has been expressed as to the success of AaE over time and its sustainability for the future. eXtension has focused resources on AaE and as a part of that focus has determined a need to study where AaE is today and where it can go in the future. With the addition of the Ask eXtension widget as a supplement to AaE, especially at the local/county level, AaE from the national perspective has garnered even greater attention. Staffing of the AaE system at the national level has been formally limited to launched Communities of Practice (CoP). At the same time, however, questions requiring expertise outside the launched CoPs also come to eXtension. The public knows that the Cooperative Extension System (CES) has expertise that extends far beyond that of the publicly launched CoPs. The growth in questions outside of staffed areas of expertise has created further questions regarding the sustainability of AaE in the long term. This Task Force was charged with reviewing the current status of eXtension's AaE system, and to provide direction for its future. Objectives of project • Assess the Cooperative Extension System's ability to operate an AaE system. • Assess what CES faculty and educators think are acceptable levels of performance: time to respond, depth of answers, etc. • Assess the current performance (human) to date: timeliness, capacity, correctness, coherence, completeness, etc. • Determine clientele expectations for an AaE system: timeliness, capacity, correctness, coherence, completeness. • Determine the level of staffing and resources required to operate a fully functioning AaE system. • Prepare report to the Governing Committee for consideration.
  3. 3. How is the public best served? Cooperative Extension's "stock and trade" for nearly a century has been via a community presence; Extension educators and agents located within a local community as an educational source for new practices in agriculture, family sciences, youth development, and community development. As more and more Americans turn to the Internet for information, they also find communities... people with similar interests who formally and informally band together in a social context. It stands to reason that Cooperative Extension should consider this online community as an equally important and relevant public with as much or more value as a public constituent group. In the traditional model, clients come to the Extension office, ask a question, and are answered by an expert supplemented with publications, fact sheets, and relevant additional educational resources. The new model via eXtension can and should operate with the same robust service and educational model. Americans have grown more and more sophisticated in their Internet use habits and are also more demanding as they seek information from myriad sources online. Previous research and metrics from eXtension in general show that 83% of clients come to eXtension from search engines (Google in particular), and that their visits are to specific pages deep within the site. They look at an average 2.1 pages and spend an average of 1:26 minutes per visit. General data on Internet use show that the public is interacting with the eXtension Web site much as they do other "destination" sites that are content centric. Increasingly they are visiting a few large sites, and staying longer. The total number of sites they are visiting continues to steadly decrease. According to Steve Reubel citing March 2009 data from Nielsen Online 1: After years of erosion it now it appears the destination web era is drawing to a close. This is a trend that digital thinkers like Om Malik have long noted. In fact, the numbers prove it. In March the average American visited a mere 111 domains and 2,500 web pages, according to Nielsen Online. What’s worse, our attention across these pages is highly fragmented. The average time spent per page is a mere 56 seconds. Portals and search engines dominate, capturing approximately 12 of the 75 hours spent online in March. However, people-powered sites like Wikipedia, Facebook and YouTube are not far behind, snagging nearly 4.5 hours of our monthly attention.
  4. 4. Current Conditions: a Snapshot Numbers served: Since its inception, the AaE system has received 12,205 questions. In May of 2009 it received 1,435. So far, in 2009 the number of questions received has been increasing at 29% per month. Figure 1 depicts the number of questions received and answered over the past year. It also shows the pattern as to when experts answer questions, with a significant drop-off in weekend activity and a flurry of answering on Mondays. The average time to respond to a question is 70 hours (3 days) with a median response time of 39 hours (1.6 days). Questions are "retired" after being in the AaE system without being answered after 10 days. Recipients are sent an apology and asked to contact their local Cooperative Extension offices. On average, 3.5% of the questions received are going unanswered either for non- performance, or because there are no experts in a subject matter available or willing to answer. In addition, 11.5% of the questions received are for subject matter areas where no current CoP exists to generate responses. Figure 1: Questions asked and answered How the public comes to Ask an Expert: Overall 83% of the people visiting www.extension.org enter directly from one of the major search engines. This is reflected in where people come from when coming to the Ask an Expert form on the eXtension web site. Eighty-six percent of the people coming to the AaE form on the eXtension site come from a content page deep within the site. They initially arrive via search, don't find the specific information they are seeking, and redirect to the AaE to begin asking their questions.
  5. 5. Who is being served: The audience being served is not particularly diverse or representative of the general population (Figure 2.) A quick glance tells us that the majority of users are female (63%), white (92%), well educated (36% with bachelors, and another 39% with advanced degrees), older (66% > than age 50), and financially better-off (72% reporting incomes > than the median family income). Figure 2. Demographics of users of the AaE system
  6. 6. Nonetheless, the AaE system is reaching new clientele (Figure 3). Twenty-nine percent of the people asking questions had never heard of Cooperative Extension prior to using the AaE system. Another 27% were aware of Cooperative Extension but had never accessed it previously. Figure 3: Previous Awareness/Use of Cooperative Extension Who is Answering Questions: The number of Extension faculty and educators answering questions is what you would expect from this sort of system. It is a classic Pareto Distribution (Figure 4) where 20% of the individuals in the AaE system are responding to 80% of the questions. Of the 656 Extension professionals who have answered questions, 131 of those individuals are providing the bulk of the effort. Less than 1% of the estimated 15,000 Extension professionals are currently active in the AaE system. Figure 4: Who's answering questions?
  7. 7. What Was Studied Four studies were conducted as a part of this review: Directors and Administrators: A sample of 15 Directors and Administrators were interviewed in face-to- face meetings and by phone. The questions used in the interviews were validated and reviewed at a meeting of the eXtension Governing Committee in March of 2009. Public survey: A complexity-based approach was used to examine the various factors that affect users’ perceptions of the Ask An Expert service 2. A survey was conducted of the 6,557 unique users of the AaE system. The Cognitive Edge SenseMaker™ software was used for the analysis: SenseMaker™ contains a range of analytical and interrogation tools that allow both recall and interpretation of sense-making items. 3 It makes extensive use of visualizations to allow complex patterns and exceptions to be discovered. The entire population of question askers were sent an email survey. Subsequent follow-up surveys were sent to non- responders. Five hundred and four individuals completed the survey for a response rate of 7.7%. This was a qualitative research design, and 200 responses were needed at the lower range to begin to see meaningful patterns in the data4. If sampling were to be done with this population a sample size of 376 would be required for a probability of .05.5 The research methodology used was a pre-hypothesis narrative-based approach6. Faculty/Educator Focus Groups: Five focus groups were conducted using the Adobe Connect Pro web conferencing system with selected CoPs, highly active AaE responders, and a group of less-than-enthusiastic CoP members, to gain insight to each of the studied questions (see What Was Observed). The polling features of Connect were used to determine the extent to which the questions identified the significance of the issues in the minds of the participants, and to collect qualitative information through discussion. Log Analysis: A review of logs was conducted, and other data analyzed to get a clear picture of how the public was interacting with the www.extension.org site. This was to answer the questions as to whether the public was able to find the page and easily enter their questions. Additionally, an analysis of the internal logs was conducted to determine the responders' use of the AaE system as they interacted with it to provide answers. What Was Observed Following are the major findings from the four studies: The public believes that the answers provided by the AaE system are science-based (figure 5). This is a remarkable finding when you consider that 56% of the respondents had no previous experience working with Cooperative Extension. In addition, they believed the answers were from a trusted and unbiased source (figure 6). These two strong preferences would indicate that the Cooperative Extension System is perceived as being true to its mission even with people who had little or no previous familiarity with the system.
  8. 8. Figure 5: Public perception as to the science-based nature of AaE answers. In addition, they trusted the answers that were provided and believed them to be from an unbiased source. Figure 6: Public perception of Answers from eXtension as Not trusted to Trusted What the clientele said: “I am glad to have my other research confirmed however by a trustworthy source.” “I got a reply that, although not what I wanted to hear, I trusted” “I had a very prompt reply, with very accurate information plus direction to a paper.” "A Google search provided an answer but I needed someone with authority and experience to corroborate/validate that answer so I could do my calculations." Clientele are primarily asking questions that they consider to be of personal and economic importance. The AaE system is not being used to ask trivial questions or to inquire about things considered a curiosity (in stark contract to Yahoo! Answers: answers.yahoo.com). The questions were also skewed toward being of economic importance as well. When the responses were filtered to eliminate consumer horticulture, then the economic importance of the questions moved even further towards being of significant economic need.
  9. 9. Figure 7: Public description of their question as to importance. Figure 8: Public description of their question as to economic need. What the clientele said: “I was worried about food safety for my family if I planted my vegetable garden as planned.” “I wanted to know about safety concerns when cleaning up a rat/mouse infested house as it related to hanta virus.” "Squirrels have been eating my house. We tried several suggestions, and the situation is improving.” “He gave practical information about how to decide what is best for my family, as opposed to necessarily doing what's best for the tree. Very helpful." The one variable that impacts the utility of the answers provided is the rapidity with which an answer is provided. After two days, people indicated that the usefulness of the answer in helping them to solve their problem was diminished (Figures 9 and 10). If the Cooperative Extension System desires to make a difference in people's lives, to provide answers that are timely, and also of practical use, then it simply must respond more quickly.
  10. 10. Figures 9 and 10: Public's Perception of Utility by Response Time (What role did the answer play in solving the problem?) What the clientele said: “We were supplied with the information we wanted in an expedient, competent and professional manner (which is so unbelievably rare these days)." “I do know it took four days for me to get an answer and I thought at the time if the answer was needed was urgent, I would have been in trouble.” “To be frank, it’s kinda hit or miss and I’m not sold on the 'quick' response promised.” Timeliness of the response comes into play once again when measuring the public's satisfaction with the service. Satisfaction was at 80% when they received an answer within a day. Their satisfaction begins to decrease after two days, and drops to just 68% if the answer isn't provided within a week. Satisfaction falls dramatically when it takes longer than a week for the experts to respond.
  11. 11. Figure 11: Clientele satisfaction and timeliness of response What the clientele said: “I didn’t expect an answer so quickly. In the matter of an hour or so..the response was very helpful. I am thrilled with this service” “I will not keep a client waiting a week for a response. My suggestion would be that if you are going to provide this question and answer service, you need to respond in a timely fashion (within 2 days)” These findings are in stark contrast to what faculty and educators determined were realistic response times given other demands of their appointments. From the focus group findings: Experts should not be expected to respond to questions during weekends and holidays. Such time should be subtracted from the time triggering escalation reports. "Office hours" should be published so questioners do not have false expectations on when to expect a response. But, during such office hours, 48 hours seems an appropriate amount of time. Exacerbating this is the public's desire for answers that are deeper and more detailed than the experts are providing. They believed that the answers provided leaned toward being too simple, lacked context, and specificity. Providing more detailed answers is likely to lead to a larger commitment of time in providing more in-depth and researched answers. Figure 12 shows the tendency for the responses to be considered too simplistic with the preponderance of responses falling below the mid-point on the Z scale.
  12. 12. Figure 12: Visualization of quality of answer. Red: science-based; Green: trusted; Black: simple/complex What the clientele said: “I did not really get an answer...I did ask a fairly long, drawn out and complicated question. I was rather disappointed in the short answer.” "The answer was well-written and included even more information that I had expected.” “The ‘Expert’ forwarded an article but it did not address the specifics of my question. I felt rather disappointed and ended up doing more research on my own... even re-emailed the ‘Expert’ a 2nd time to clarify but she did not return my email.” “As a person who has been involved in Dairy business, I knew what we should do but I wanted to get a confirmation on what we are going to do, but unfortunately what I got was very simple concepts." Faculty and Staff Focus Group: Tenure track faculty observed that answering the public's questions is not considered scholarly work, and is not rewarded. They acknowledged that it is an important component of the Extension mission, but that it is best valued internally when carried out by individuals in non-tenure leading positions. One strategy is to provide the opportunity for responding to AaE questions at an appropriate organizational level where it will be adequately acknowledged and rewarded. This will differ by state Cooperative Extension Systems.
  13. 13. Faculty, educators and staff also indicated that they needed additional training opportunities to improve awareness and to enhance performance of the AaE. Administrative Interviews Administrators' perception of faculty and staff involvement in working with AaE was very positive. There were few concerns voiced regarding faculty and staff working or serving clientele outside of their assigned geographic responsibilities. It was often cited as a positive aspect of eXtension, and an area that was more critical than ever given the current budget situation. Virtually every administrator mentioned eXtension and AaE as a key strategy for dealing with expertise gaps and personnel shortages. Administrators observed that AaE is not considered scholarly work, but saw value in reaching clientele through this technology. There was a general acknowledgment that involvement of faculty and staff in AaE varies widely between states. Perception and awareness seem to depend on the state's administrative views and promotion of AaE and eXtension in general. In states where the administration encourages and supports involvement, AaE is perceived as positive. In those states where encouragement does not occur, there is little engagement and faculty and staff are reluctant to participate. What Needs to Happen As noted earlier, Americans want engagement with online communities. They also want a more interactive experience in that engagement process. To be a viable and valuable resource, eXtension must pay attention to the needs of clientele and work with Cooperative Extension leaders to prepare for the current and future demands of AaE. There is a need for more people at the appropriate level within and outside Cooperative Extension to answer questions. CoP's primary focus needs to be on building their communities, recruiting new people to answer questions, and providing professional development opportunities to better prepare experts to answer questions in an appropriate and timely manner. With the potential to receive questions from many more people (orders of magnitude larger), Cooperative Extension is going to need to grow its capacity to serve ever increasing audience numbers. Far more than the current group of 131 experts will be needed if the CES is to even begin to service the public's demand for this type of system. Indeed, the system is struggling to perform given the moderate number of questions received currently, and this will only be alleviated by growing the number of active experts responding to questions. Regardless of the information and educational content provided on the public website from launched CoPs, when engaging with the site the public asks questions that fall outside the expertise of the current CoPs. Clientele know that the CES has expertise outside of the areas of the launched CoPs and will ask their questions. Indeed, 11.5% of the questions received are in areas where eXtension is not currently staffed to respond. It is not acceptable to not respond to these questions. Each state needs at least one designated Question Wrangler to handle the assignment of questions emanating from their state and to assist in locating experts to answer incoming questions in areas where there are no current CoPs. Additional experts in each state need to be identified. Working through Institutional Teams each state should develop a
  14. 14. strategy to have their staff flesh-out their profiles in the eXtension directory application found at people.extension.org (People) so that they can more easily be identified to assist in responding to questions. Performance metrics must match clientele expectations. The results are clear; customer satisfaction and the utility of the answers provided drop significantly when it takes longer than two days to receive a response. To speed responses, the timing of non-performance escalation reports to CoP leaders needs to be reduced from the current 48 hours to 24. Unanswered questions need to be reassigned after 24 hours. Multiple people need to be alerted to questions in their areas of expertise to mitigate the problems caused by the current system of assigning each question to an individual creating a single-point-of-failure. The soon to be released version of the AaE system that allows for multiple responses to a question should help to alleviate some of the issues of experts not always being available. Create AaE only CoPs. Using the profiling capabilities of for CES staff found in People, work to proactively form new CoPs with an initial single focus of answering clientele questions. Provide tailored professional development opportunities to assist them in getting organized, and in the mechanics of using the AaE system. Assist in helping to recruiting new members. Continue to advance the use of widgets to involve more local staff with AaE. It has been noticed that many local educators who get involved with answering questions through their own county widgets also get involved in answering questions coming in from the www.extension.org site. Provide additional professional development opportunities on using the widgets. Work with state level IT staff on the use of the AaE system, and on installation and maintenance of widgets. Provide additional AaE training and professional development to raise expertise across the entire CES. This needs to include not only the mechanics of using the AaE system, but also proper etiquette, answering strategies, and awareness concerning clientele expectations. Continue to focus on online engagement. To engage or not to engage with online audiences is not a choice. We must engage-- our clientele expect it. AaE is the place within eXtension where this is done, and it needs to be done well. It can no longer be viewed as an add-on, but needs to be a top priority. Even though faculty get less credit…this is an opportunity for the CES to be relevant in ways heretofore unimagined. Eighty-three percent of the people coming to the AaE system are arriving from an Internet search engine. It is the content of the www.extension.org site which brings them. One concern is the lack of diversity in the clientele using AaE. Consideration needs to be given to creating CoPs that will appeal to a more diverse and younger demographic. Recognize that AaE is reaching new, online audiences; and this needs to be promoted. The AaE represents a strategic opportunity for the CES that should not be squandered. Vision From the beginning, one of eXtension's promises has been to reach new audiences and to bring new people to the CES. Through the eXtension brand study, through the ECOP Marketing Task Force brand development process, and through this AaE task force, it has been learned that Americans who don't know
  15. 15. Cooperative Extension but who ultimately interact with it, are pleased with their experience. They believe Cooperative Extension is reliable and trustworthy. They believe the land-grant university connection adds credibility and trustworthiness to the information they find. And, in the case of the studies done with this effort, many of those, over half, who are currently using the AaE system are unfamiliar with the Cooperative Extension System. So, where does Cooperative Extension go from here? From these studies it is known that people value and demand Ask an Expert type services. Therefore it is important to meet this challenge as an opportunity for further engagement with current audiences, and new engagement with people who have yet to discover Cooperative Extension. There is an opportunity to offer a service on the Internet that is both unique and in demand: scientifically-based knowledge from trusted sources. This is a unique niche, and and opportunity to be seized. As evidenced in the data above, demand both in volume and in time, can quickly dwarf Cooperative Extension's ability to respond to Ask an Expert questions. Cooperative Extension must look to alternative strategies including broadening the expertise base to answer questions. It must look at the use of trained volunteers; incorporating eXtension's outstanding professional development capabilities and subject matter specialists to provide training. It must take advantage of social networking strategies available now with Web 2.0 to bring experts to a more granular level engaging at the community level where Cooperative Extension has done its best work for nearly a century. It must also "grow" its expertise bank through a mentor/protege approach where current experts share their knowledge one-on-one to bring new and younger faculty and staff into this effort. The "graying" of America will not stop at the doors to America's land-grant universities, and thus Cooperative Extension must look toward empowering and enabling new faculty and staff to build these experts as soon as possible. It has long been the goal of eXtension to put Cooperative Extension into the homes of more and more Americans. The AeE system truly meets that goal. It is an interactive approach; an engaging approach; a value-added approach. It takes full advantage of CES' reputation for reliability, credibility, and for providing research-based expertise in a customer-focused manner. That trust, coupled with a responsive, well-trained bank of experts will serve CES well into the future. References 1. Reubel, Steve. "The End of the Destination Web Era." Micropersuasion, http://www.micropersuasion.com/2009/05/the-end-of-the-destination-web-era.html, May 3, 2009. 2. Elliott, Jane. Using Narrative in Social Research: Qualitative and Quantitative Approaches. Thousand Oaks, California: Sage Publications, 2005. 3. Sense-Maker Suite version 3, http://www.sensemaker-suite.com 4. Lorenz, M. O. "Methods of measuring the concentration of wealth." Publications of the American Statistical Association. 9 (1905) : 209–219. 5. Chelveldave, Michael. (March 24, 2009). Personal conversation. 6. Israel, Glenn. Determining Sample Size: PEOD6. IFAS, University of Florida, 1992.
  16. 16. 7. Snowden, David and Mary Boone. Leader's Framework for Decision Making. Harvard Business Review (November 2007): 10pp.