1. Using Internet Research Methods
In a Needs Assessment
Framework
Donna Smith-Moncrieffe
Dr. Brian Cugelman
Canadian Evaluation Society
Conference
Edmonton, May 1-4, 2011

2. Overview
● 1. Evaluation and Internet research
­ What is a needs assessment framework
­ Increased use in the social sciences
● Example 1: Internet data reflects actual data
● Example 2: Potential Internet usage in crime e­scan
­ Advantages and disadvantages
● 2. Case study:
­ Using web­based research to determine the need for evaluation resources
­ Methods and tools used:
● Assessment of site traffic
● Word search index
● Times Series Analysis
● Cross­Sectional Analysis
­ Findings and implications
● 3. Conclusions
2

3. 1. Evaluation and Internet
research
3

4. Context: Needs Assessment in the
Evaluation Cycle
● 1. Formative
­ Needs Assessment
­ Evaluability assessment
­ Process evaluation
● 2. Monitoring
­ Developmental
­ Correction/Modifications
● 3. Summative
­ Impact evaluation
­ Cost benefit or cost Effectiveness
analysis
­ Meta­Analysis
4

5. Needs Assessment Framework in Evaluation
Utility of a Needs Assessment Framework in Evaluation?
● Determines needs for the program
● Assess the level of needs
­ Measures the number of individuals in a community or population that
require the service
● Assesses what might work to meet the need
● Identifies the type of services needed
­ Assesses the capacity of existing services to meet identified needs
● Measures the relative need for services across different communities, federal,
provinces and territories (FPT’s)
● Co­ordination of beneficiaries/organizations
5

6. Internet Research:
Utility in the Private versus Social Sciences
Since 1996, there has been an increased use of web-based tools in the
social sciences (Mike Thelwall, 2004)
Private sector Social Sciences
● Attain consumer insight ● Assess changes in public behaviours
over time
● Asses trends in buying
­ Crime, Health and
behaviour Socioeconomic trends
● Assess charitable giving trends
● Interest in issues over time
● Brand impact and related
metrics ­ Internet can reflect social
priorities that provide indicators
for planning
6

7. Example: United States Flu Activity and the
Correlation between Online and Offline Data
● Strong correlations between “real data” and date sourced from
the internet have been identified (Ginsberg et al, 2009)
● Studies have found that Internet data strongly mirrors reality
http://www.google.org/flutrends/about/how.html
Google Flu Trends estimate
United States data
7

8. Needs Assessment: Crime Example
In 2007, a City of Toronto Gang project used traditional methods to identify a
Youth Crime Index. Complementary web­based methods can be considered
for future needs assessments or environmental scans
Traditional Methods Potential Web-based Methods
● Geographical Information ● Time series analysis of a keyword
System (GIS) Proximity Analysis search index
● Secondary data provided by
● Geographic analysis of a keyword
­ Toronto Anti­Violence
Prevention Strategy search index
­ 2006 Census
● Content analysis of social media
­ Toronto School Board
discussions: blogs, Twitter, YouTube
­ 211Ontario.ca
● Indicator ratings and weighting
● Relationships between online and
offline social networks
8

9. Internet Research used in Planning Phase:
Advantages and Disadvantages
Consider pros and cons of using internet research methods during the
evaluation planning stage. Utilize complementary methods where there
may be considerable disadvantages
Advantages Disadvantages
● Free access to data ● Demographical information can
● Easy to obtain once the user sometimes be challenging to obtain
has access to computer and ● Data may be limited to the ‘computer
basic web applications (i.e. savvy’ sector of the population
Google insights) ● Purpose of the research may not be
● Access to an international congruent with the data provided
sample ­ (i.e. Internet data on gang trends
● Data can be both quantitative could have multiple
(numeric data is collected) and interpretations)
qualitative (social media)
9

10. Needs Assessment Framework :
Traditional and Internet Research Techniques
Traditional Research Activities Alternative/Complementary Internet
● Review of social service directories Research Techniques
(offline) ● Time­series analysis of search
engine terms
● Surveys sent to specified
organizations
● Key informant interviews (identified ● Time­series analysis of keywords
from blogs, micro­blogs
through purposive sampling)
● Focus Groups with community
representatives ● Website analytics and usage trends
● Document Reviews of:
● Qualitative content analysis
­ Government development plans
­ Project or annual reports
● Network and link relationships
● Observations
­ Directly assess client needs at ● Web surveys and polls
site visits
10

11. 2. Case study
11

12. Case Study:
What is the National and International Need for
Evaluation over the last decade?
Case Study: What is the National and International Need for evaluation over the last decade?
Overall Goal: Demonstrate need and market value of a user­friendly evaluation resource to a
potential Publisher
Evaluation Questions:
­ What is the level of Need for an evaluation research book that would appeal to
laypersons?
­ Who are the potential users of this resource?
­ How can this resource best address this need?
Objectives
­ To quantify the potential readers in the international Community
­ To quantify the type of users
­ To qualify how the resource can address these needs
Methods
­ Assessment of site traffic
­ Word Search Index
­ Time Series Analysis
­ Cross­Sectional Analysis
12

13. Internet Research Methods and Tools:
Assessment of Site Traffic
Case Study Methods and Tools: Assessment of Site Traffic
Why
­ Assessment of site traffic among global evaluation websites provides an overall
assessment of the importance of evaluation in an international context
What
­ Assessing “in­link” site hits and “unique visits” to a number of evaluation societies
can provides a quantitative figure on the frequency of visits and minimum time spent
on each website
How
­ Statistical relationship between “in­links” and site traffic
­ Data for the estimation phase is sourced through compete.com and Google Ad
Planner (Cugelman B. (2008), Cyber metrics Research Group)
13

14. Internet Research Methods and Tools:
Assessment of Global Evaluation Society Websites
Findings
There were 25 evaluation societies that had accessible websites
● There are roughly 39,179 unique monthly visits to the global network of
evaluation websites. Per year this is 470, 131 unique visits to
evaluation websites
● Using the application Google Ad Planner, this study found that
­ The average visitor frequented the site at least 14 times
­ Average time spent was 20 minutes on the site
● Based on as assessment of monthly unique visits and hours spent on
each site), the top most frequently used evaluation websites include:
­ American Evaluation Society (AEA)
­ Canadian Evaluation Society (CES)
­ German Evaluation Society (DeGEval)
­ European Evaluation Society (EES)

15. Internet Research Methods and Tools:
Word Search Index
Case Study Methods and Tools: Building a Word Search Index
Why
­ Searching a few evaluation related terms can significantly produce inaccurate
estimates
What
­ Build a table of all possible evaluation related terms that could possibly be searched
with a view to developing a scale/index of only relevant terms. In this case, 27
evaluation related words were assessed prior to developing the index.
How
­ Set up a word search index to ensure that you can identify all possible evaluation
related terms
­ Asses each word on 6 key criteria and choose words > than the self­identified cut off
score (i.e. All words that score >8 will be included in the index)
15

16. Internet Research Methods and Tools:
Word Search Index (Sample)
Key-word Global Google Trend Relevance Usage Competition Estimate
Monthly Search Average
Search Volume CPC
Analytics 5,000,000.0 52 Massive Low Closely 0.15 CA $
0 Increase linked 8.59
with
marketing
Impact 135,000.00 46 Even High Mostly 0.08 C.A $
Evaluation related to 1.29
health
evaluation
Evaluation 80,000.00 53 Declining High Varies ­ ­
Global Monthly Searches: This is the number of times people enter Google searches with these search terms
Google Search Volume: This number represents the total number of searches over time scaled to a score from
0­100. A high number indicates a term has been commonly searched since 2004.
Competition: This is a measure of competition on Google for advertising. This shows the commercial interest in
products and services related to these terms
Estimated Average CPC: This is the average cost per click paid for each ad that a person clicks on.
16

17. Internet Research Methods:
Time Series Analysis
Case Study Methods and Tools: Time-series analysis
Why
­ To understand individual interest in topics over time
­ To compare interest between topics
­ Assess correlations between trends and other events
What
­ Time­series analysis shows public searches for key terms over time
­ The data can be correlated with time­based events
How
­ Using Google search databases
­ Use blog and micro blogging data to correlate trends
17

18. Time Series Analysis 2004-2010:
Evaluation Word Search Index
18

19. Time Series Analysis 2008-2010:
Key Word Search Index (Top 4 Evaluation Terms)
Monthly averages
19

20. Internet Research Methods and Tools:
Cross-Sectional Analysis
Case Study Methods and Tools: Cross-Section Analysis
Why
­ To conduct an audience analysis
­ What type of organizations are conducting these internet searches
What
­ A cross­sectional analysis was used to identify the correlation between “unique page
views” and the type of organizations conducting the evaluation related searches
How
­ Domains of 801 collected URLs were analyzed according to their secondary top
level domain
­ Develop a table that provides an agreed upon description of the social sector
indicator (i.e. Agreed upon definitions and criteria for “.org” for example)
20

21. Internet Research Methods and Tools:
Cross-Sectional Analysis
Social Description of domain Reliability Sum of % Estimated Estimated
sector Domains searches searches per
indicator per month year
.org Org was intended to mean Somewhat 371 46% 28,022 336,262
organization, and is primarily used
by non-commercial organizations,
though there is no control, making
it a moderate indicator.
Mixed This includes all other domains N/A 173 22% 13,067 156,801
domains and cannot be generalized.
.com Com was intended to mean Somewhat 97 12% 7,326 87,918
commercial, though there is no
control, making it a poor indicator.
.gov Gov domain names and sub- Very strong 96 12% 7,251 87,011
domains are restricted making
them a very strong indicator of
government.
.edu Edu domain names and sub- Very strong 64 8% 4,834 58,007
domains are restricted making
them a very strong indicator of
educational institutes.
801 100% 60,500 726,000
21

22. Evaluation Case Study:
Key Findings
Levels of Interest in Evaluation in 2010
● There were 39, 178 unique visits to 25 evaluation societies per year. Just under
half a million (470,148,00.00) unique viewers used evaluation websites per year
Use of Evaluation by Specific Groups
● Just under 50% of Non­Governmental organizations (46%) are showing
interest in evaluation related issues based on the “evaluation word search
index”
● Between 10­12% of commercial/private, government and the education sector
are showing online interest in these areas
­ Complementary qualitative follow­up might indicate that NGO’s are more
likely to search for online user­friendly tools as traditional evaluation texts
books are designed for trained practitioners
22

23. Case Study Findings Cont’d
Building a Word Search Index
● Using only a few evaluation terms during an internet search can be misleading
­ The trend analysis of the single term evaluation showed a decline since
2007
­ After 27 related evaluation terms were assessed and reduced to 9 terms
that were more relevant and valid to the study, it was determined that the
index of terms needed to include the following:
● Various types of evaluation approaches (i.e. Impact Evaluation)
● Various types of techniques (i.e. cost benefit analysis)
● Utilize concepts from other disciplines (i.e. Social Rate of Investment
is very similar concept to cost­benefit analysis so both terms should
be considered)
23

24. Case Study: Implications
What does this Mean for Evaluation Practitioners?
● To maintain or increase professional demand, evaluators need to
ensure that they promote themselves by using a variety of evaluation
terms and approaches used by various disciplines
● The following are some evaluation approaches conducted by personnel
that are not necessarily trained evaluators:
­ Quality Assurance
­ Performance Measurement
­ Social Rates on Investment
­ Balanced Scored Board (Including Six Sigma)
● The cross­sectional analysis demonstrated the need to provide more
user­friendly evaluation resources and tools for the non­profit sector
24

25. 3. Conclusions
25

26. Conclusions
 Consider internet research approaches during the planning phases of an evaluation
 Thoroughly weigh the pros and cons of conducting internet research
 Use internet research to complement traditional evaluation planning approaches
and ensure that triangulation is used to verify web­based results
 Consider the following techniques:
Web-based Techniques Purpose
● Assessment of site traffic ● Quantifies the need and level of interest
● Composite word search index ● Provides an exhaustive list to describe the
construct of interest
●The final index increases the relevance of the
search
● Time Series Analysis ● Provides a context and considers temporal
changes over a specified period of time
● Cross­Sectional Analysis ● Identifies the audience or recipient searching
for the information
26

27. Thank You
Donna Smith-Moncrieffe Dr. Brian Cugelman
Public Safety Canada AlterSpark Consulting
www.publicsafety.gc.ca
info@alterspark.com
www.alterspark.com
27