Report Evaluation Model for Analysts in Mutual Fund Companies.doc
Report Evaluation Model for Analysts in Mutual Fund
Insurance and Financial Management Department,
Takming University of Science and Technology, 56, Sec.1, Huanshan Rd., Nei Hu,
Taipei 11451, Taiwan, R.O.C.
Yu Ching Ho
Logistics Management Department, Dahan Institute of Technology, No.1, Shjen
Street, Dahan ,Sincheng Hualien 971 Taiwan , R.O.C.
This study develops an evaluation and selection model of brokerage firm’s
research reports by modified Delphi method and AHP. This model provides a distinct
approach to examine brokerage firm’s research reports. We establish four main-
criteria and nineteen sub-criteria to evaluate brokerage firm’s research reports. The
primary criterion is the objective of brokerage firm’s research reports. No bias
statement, no serious omission and detail description of information source are key
sub-criteria to reach objective. Consistent evaluation of target firm’s industry is also
important evaluation sub-criterion.
Modified Delphi method, AHP, Brokerage firm’s research reports, Objective
The major function of analysts in mutual fund companies is to provide investment
target selection suggestions to portfolio managers. With the boarder range of
investment area, to visit every company directly in the universe becomes a mission
impossible. In order to increase their understanding of the investment targets, analysts
must rely on research reports from all kinds of sources.
Although these reports play an important role in making investment decisions, the
purposes of issuing reports may differ from each other. For example, industrial
statistical data published by government is aim for increasing the transparency of
markets, but investment banks issue research papers may be in a willing to promote
their IPO stocks. Academic research shows that research reports from brokerage
firms are often optimistic or biased due to the brokerage firm analysts’ career
concerns and conflicts of interest inside the brokerage firm (e.g. Elton et al. ,1986;
Womack, 1996; Barber et al., 2001, 2003; Hong & Kubik, 2003; Azzi, Bird,
Ghiringhelli & Rossi, 2006; Cowen, Groysberg & Healy, 2006; Jacob, Rock & Weber,
2008). Therefore, carefully evaluation and selection between reports become a key
successful factor of creating superior investment performance.
Most related literatures focus on analyzing the information content of the brokerage
firm’s research reports or why and how those reports are biased. We lack an approach
to identify a most valuable brokerage firm’s research reports. To establish an
evaluation and selection model of brokerage firm’s research reports is the main
purpose of our study.
Analysts must evaluate the value of each report and decide a valuable report by
various selection criteria such as assumption of the research report, timely published,
conscientious, objective, etc., simultaneously. In this study, a model incorporated with
the Modified Delphi method and the Analytical Hierarchy Processing (AHP) method
to select the most valuable brokerage firm’s research reports is proposed. The
Modified Delphi method is adopted to extract the criteria from asking the opinions of
a panel of experts, and the AHP method is used to decide the priority of those criteria
gathered from the Modified Delphi method and compute the relative weight of each
alternative research reports.
To enhance investment performance of mutual funds, predictive suggestions of
investment targets from analysts are essential. Research reports with good quality help
analysts provide precise suggestions. Facing variety reports, this evaluation and
selection model supports analysts identify useful reports easily. This study attempts to
make a contribution to a better appreciation of brokerage firm’s research reports
through the viewpoint of analysts.
The remainder of this paper is organized as follows: Section 2 describes the
research methodology we use in this study. Section 3 develops the model. Section 4
discusses and concludes.
2. Research methodology
The purpose of this study is to establish a model to help investors to evaluate and
select the valuable brokerage firm’s research reports by the Modified Delphi method
and the AHP method. Before developing the model, we will introduce the Modified
Delphi method and AHP in general.
Modified Delphi method
The Delphi method is an approach to elicit experts’ opinion by an iterative process
without face to face grouping discussion. It is a systematic process that attempts to
obtain group consensus in much more open and in-depth research ( MacCarthy &
Atthirawong, 2003). Series stages of questionnaires are designed to elicit and refine
common opinions within a pre-selected panel of experts via mail. Murry and
Hammons (1995) proposed the Modified Delphi method which enables researchers to
shorten the determination process. The difference between these two methods is that
the Modified Delphi method needs to develop a structured questionnaire by literature
review or expert interview instead of open-ended questionnaire in the first stage. By
using structured questionnaire, the research horizon will be shortened and objective-
related criteria could be determined faster. Because the research burden of
ANALYSTS is heavy, that would be a tough work to have them discuss together by
face to face meeting. Therefore using the Modified Delphi method is an appropriate
approach to collect opinions of ANALYSTS about the criteria decisions they used to
evaluate and select brokerage firm’s research reports without disturbance.
Analytical hierarchy process
The Analytic Hierarchy Process developed by Satty is a kind of multi-criteria
decision making (MCDM) techniques and enables decision makers to represent the
simultaneous interaction of many factors in complex and unstructured situations. It
helps them to identify and set priorities on the basis of their objectives and their
knowledge and experience of each problem and provide a structured approach to
decision making (Saaty, 1999). The AHP is performed well to solve complex
decision-making problems in different areas, such as planning (Kwak & Lee, 2002;
Radash & Kwak, 1998), resources evaluation and allocation (Alphonce, 1997; Jaber
& Mohsen, 2001; Hsu, Wu & Li, 2008), measuring performance (Frei & Harker,
1999; Ahsan & Bartlema, 2004), choosing the best policy after finding a set of
alternatives (Poh and Ang, 1999; Chang et al., 2007), setting priorities (Schniederjans
and Wilson, 1991). The first step is to decompose a complex situation into relevant
main criteria and sub-criteria, then using these criteria to establish a hierarchy
structure. A basic hierarchy model of AHP including four levels (Figure 1). The top
level is the goal we want to achieve. The second and third levels are criteria and sub-
criteria respectively. Since human being could not compare too many elements
simultaneously, the elements in each main criteria and sub-criteria should not exceed
seven. Under this limitation, it may carry on the reasonable comparison and easier
ensure the consistency (Satty, 1980). The bottom level is the replacement alternatives.
Criteria B1 B2 B3
Sub-criteria C1 C2 C3 C4 C5 C6 C7 C8 C9 C10
Alternatives D1 D2 D3
Figure 1. A basic hierarchy model of AHP
To derive the priorities of main criteria and sub-criteria within the hierarchy
structure, AHP incorporates the comparison judgments of all decision makers into a
final decision, without having to elicit their utility functions on subjective and
objective criteria, by pair-wise comparisons of the alternatives and check their
consistency (Saaty, 1990). The various hierarchies’ elements weight computation are
shown as follow:
(1). Establish the pair-wise comparison matrix A
If there are n elements, it must carry out n(n-1)/2 elements pair-wise comparisons.
Let C1, C2, L , Cn denote the set of elements, while aij represents a quantified
judgment on a pair of elements Ci, Cj. The relative importance of two elements is
rated using a scale with the values 1, 3, 5, 7, and 9, where 1 refers to “equally
important”, 3 denotes “slightly more important”, 5 equals “strongly more important”,
7 represents “demonstrably more important” and 9 denotes “absolutely more
important”. This yields an n-by-n matrix A as follows:
C1 C2 K Cn
C1 1 a12 L a1n
C 1/ a 1 L a2 n
A = aij = 2 12
M O M
Cn 1/ a1n 1/ a2 n L 1
Where aij = 1 and aij = 1/ aij , i, j = 1, 2, ..., n. In matrix A, the problem turn
into assign the n elements C1, C2, …, Cn a set of numerical weights W1, W2, …, Wn that
reflects the recorded judgments. If A is a consistency matrix, the relations between
weights Wi and judgments aij are simply given by Wi / W j = aij (for i, j = 1, 2, ..., n. )
and matrix A as follows:
C1 C2 K Cn
C1 w1 / w1 w1 / w2 L w1 / wn
C w / w w2 / w2 L w2 / wn
A= 2 2 1 , (2)
M M M O M
Cn wn / w1 wn / w2 L wn / wn
(2) Eigenvalue and Eigenvector calculation
Matrix A multiply the elements weight vector ( x ) equal to nx , that is ( A - nI )
x =0, the x is the Eigenvalue ( n ) of Eigenvector. Due to aij is the decision makers’
subjective judgment comparison and appraisal, the true value ( Wi / W j ) may be a little
different, so that Ax = n.x is unable to set up. Satty (1990) suggested that the largest
eigenvalue λ max be:
λmax = ∑ aij , (3)
j =1 Wi
If A is a consistency matrix, eigenvector X can be calculated by
( A − λmax I ) X = 0 , (4)
(3) Consistency test
The essential idea of AHP is that a matrix A of rank n is only consistent if it has
one positive eigenvalue n = λmax while all other eigenvalues are zero. Further, Saaty
developed the consistency index (CI) to measure the deviation from a consistent
CI = (λmax − n) /(n − 1) , (5)
The consistency ratio (CR) is introduced to aid the decision on revising the
matrix or not. It is defined as the ratio of the CI to the so-called random index (RI)
which is a CI of randomly generated matrices:
CR = CI / RI . (6)
For n = 3 the required consistency ratio (CR Goal ) should be less than 0.05, for n
= 4 it should be less than 0.08 and for n ≥ 5 it should be less than 0.10 to get a
sufficient consistent matrix. Otherwise the matrix should be revised (Saaty, 1994).
Once the priorities weights of each main criteria is established, then the relative
performance measures of the alternatives can be calculated in terms of each main
criteria and the best alternative is decided by relative performance scores.
3. Model development
A series steps to perform the AHP analysis are described as follows:
Step 1: Defining the evaluation criteria and sub-criteria used to select the valuable
brokerage firm’s research reports and establishing an AHP-based hierarchical
The goal of our study is to select a valuable brokerage firm’s research reports. This
is the top level of the AHP-based model. Then we must break down our goal into
several elements as main criteria and sub-criteria, and arranges them hierarchically
through the Modified Delphi method. Although there are many literature discussed
brokerage firm analysts’ research reports, but there is few researches focused on the
topic we discuss in this study. By literature review, expert interview and some Taiwan
government regulations, the modified Delphi structure questionnaires are developed.
These regulations stipulate some norms that researchers of investment company or
securities company must obey before they publish research reports. After referring to
these literatures and in-depth expert interview, we select 28 indicators and organize
them into questionnaire to perform the Modified Delphi method. We sent the
Modified Delphi questionnaire to thirteen ANALYSTS, then analyze the opinion
feedback and extract decision elements, including four main criteria and nineteen sub-
criteria. All the criteria and denotation are summarized in Table 1. A hierarchy
structure shown as figure 2 is arranged by deep interview with three experts to ensure
the rationality of the hierarchy structure.
Table 1. The criteria extract by the Modified Delphi method
Main criteria 1.
assumption of research
Sub-criteria for C1
value investing (SC1) (SC1)Ways to search for undervalued stocks
financial model (SC2) (SC2)Stock evaluation using common models
consistent evaluation (SC3)consistent evaluation principals for all stocks
of target firm’s in the same industry
industry (SC3) (SC4)coherent evaluation process for the whole
coherent evaluation industry
of target firm’s
Main criteria 2.
timely research (C2)
popular issues (SC5)
market trend (SC6) (SC5)Address relevant opinions about popular
predictions before issues in today’s market
market shock (SC7) (SC6)Address relevant opinions about future
(SC7)Address relevant opinions before market
Main criteria 3
Honest stock (SC8)The recommendation are based on unbiased
recommendation and reasonable judgements
(SC9)Work experience of brokerage firm’s
work experience of research team in the investment related industries
brokerage firm’s (SC10)Work experience of brokerage firm’s
research team (SC9) research team in the industries which they covered
industry experience (SC11)Choose suitable models and make sure the
of brokerage firm’s methodology of the model using is correct
research team (SC10) (SC12)Evaluate the inputs of models carefully
using financial model (SC13)Confirm the correctness of all the data
of every numeral and
Applying (SC14) Refer to opinions of professionals or
authoritative information from authorities
(SC15)Confirm and compare data/ information
establishing multi- from different resources
information resources (SC16)Clear description of each data/ information
detail description of (SC17)Data is not fake or been changed
information source (SC18)Statements are unbiased and not related to
(SC16) self interests
true record (SC17) (SC19) no serious omission
no bias statement
no serious omission
Consistent evaluation of target
Coherent evaluation of target
Address relevant opinions about
popular issues in today ’s market
Address relevant opinions about
future market trend
Address relevant opinions before
Honest stock recommendation
Work experience of brokerage
Valuable firm’s research team
firm ’s Industry experience of brokerage
research firm’s research team
Using financial model correctly
Evaluating financial accounting
Careful examination of every
numeral and statement published
Establishing multi -information
Detail description of information
No bias statement
No serious omission
Figure 2. The hierarchy structure of AHP-based model
Step 2: Establishing pair-wise comparison matrix of each factor
Based on the hierarchy structure, an AHP questionnaire is developed to make a
pair-wise comparisons in order to determine the relative priorities of each criteria. The
pair-wise comparisons are based on the scale of relative importance that assumes
values between 1 and 9. This scale can be applied with ease to criteria that can be
defined numerically as well as to those cannot be defined numerically. Relative
importance scale is presented. ANALYSTS is supposed to specify their judgments of
the relative importance of each contribution of every criterion towards achieving the
In this study, a purposive expert sampling is applied to sample ten respondents
from various ANALYSTS. The weights of level 2 criteria and level 3 sub-criteria are
then determined for a sample group of ten individuals matching the above
characteristics with each respondent making a pair-wise comparison of the decision
elements and assigning them relative scores. The relative scores provided by ten
experts are aggregated using the arithmetic mean method. Each decision maker in the
fund company makes a pair-wise comparison of the report evaluation under nineteen
subjective sub-criteria and, then, assigns those relative scores. We using the Eq. (1)
and (2) to calculated the aggregate pair-wise comparison matrix.
The results of the pair-wise comparison matrices about main-criteria and sub-
criteria are shown as table 2 and 3.
Step 3: Calculating the eigenvalue and eigenvector
The comparison in Tables 2 to 3 are used to calculate the eigenvectors using Eq.
(3) and (4). Table 4 summarizes the results of eigenvectors and weights for the main-
criteria and sub-criteria.
Step 4: Consistency test
According to Eq. (5) and (6), the consistency test of each criteria level is
calculated and the results are shown as table 2 and 3. The CR. of each comparison
matrices are all < 0.1, indicating “consistency”.
Step 5: Computing relative weight of each levels’ elements
Aggregate the related scores provided by all experts using simple additive
weighting and the results for each levels relative weight of the elements are shown as
Table 4. After sorting the four main-criteria by relative weights, the most important
main-criteria is objective(0.421), next are conscientious(0.237), assumption of
research(0.227) and timely research(0.116) separately.
The sub-criteria are sorted and analyzed based on relative weights under each
main-criteria as shown in Table 4. The results are summarized as follows:
(1) There are six sub-criteria under the most important main-criteria – objective. The
highest relative weight sub-criterion is no bias statement (0.225). We observe
that the relative weights of no serious omission (0.190), detail description of
information source (0.183) and true record (0.172) are also important criteria for
ANALYSTS to screen brokerage firm’s research reports. Applying authoritative
information (0.091) seems to be less important criterion under objective.
(2) With regard to the sub-criteria under conscientious, the most important criterion
is industry experience (0.296). The remainder criteria sorted by relative weights
are evaluating financial accounting data correctly (0.208), careful examination of
every numeral and statement published (0.169), Honest stock recommendation
(0.128), using financial model correctly (0.124) and work experience (0.075).
(3) According to the priority of relative weight, the sub-criteria under assumptions of
research reports ranked are consistent evaluation of target firm’s industry
(0.351), coherent evaluation target firm’s industry (0.282), value investing
(0.217) and financial model (0.149).
(4) For ANALYSTS, timely research reports mean address relevant opinions about
future market trend (0.470) and address relevant opinions before market shock
(0.416), addressing relevant opinions about popular issues in today’s market
(0.170) is not an important consideration for ANALYSTS.
Step 6. Computing global priority of each sub-criterion
Global priority of each sub-criterion is gathered by multiplying its relative-weight
by corresponding main-criterion’s relative-weight. The results are arranged in Table 5.
The top five relative-weight sub-criterion are no bias statement (0.095), no serious
omission (0.080), consistent evaluation of target firm’s industry (0.080), detail
description of information source (0.077) and true record (0.072) respectively. The
bottom five relative-weight sub-criterion are address relevant opinions about popular
issues in today’s market (0.013), work experience of brokerage firm’s research team
(0.018), using financial model correctly (0.029), honest stock recommendation
(0.030) and financial model (0.034) respectively.
As shown in Table 5, we discover that four of the top-five global priorities of sub-
criteria are of objective. This result is responding to the duty of ANALYSTS,
generating objective and valuable investment suggestions for mutual fund managers.
Brokerage firm’s analysts usually publish over-optimistic statements to lead investors
based on their own career concern or under top-management pressures. Investors like
ANALYSTS may suffer serious loss for adopting an over-optimistic opinion.
Therefore, they have to examine if research reports existing bias statement, serious
omission or false record and describing information source particularly before
accepting the investment recommendations and target price to make investment
decisions. One of top-five sub-criteria is of assumption of research reports. This sub-
criterion is consistent evaluation of target firm’s industry. Consistent evaluation
method makes valid and meaningful comparison with similar stocks for ANALYSTS.
The sixth global priority of sub-criterion is the industries’ domain knowledge or
experience of the researches’ target in the firm’s research team. But the importance of
work experience of brokerage firm’s research team is very low, the priority is 18. In
the viewpoint of ANALYSTS, the contribution to conscientious of industry
experience is greater than work experience. Consistent with Mikhail, Walther and
Willis (2003), they prove that analysts become more accurate with firm-specific
forecasting experience. Brokerage firm’s research team concentrates their attentions in
single industry will produce more valuable research reports.
Addressing relevant opinions about popular issues in today’s market is not
important considerations for ANALYSTS since stock market price has reflects those
popular issues; ANALYSTS is unable to acquire returns through such information.
The most important task of ANALYSTS is to find out under-valued stocks and invest
them now. Under-valued stocks mean their market price is lower than their real value
now and will go up in the future. Therefore ANALYSTS need an objective research
reports which can indicate what will happen in the future and how is the market trend.
Except future market trend, addressing opinions about possible market shock could
assist ANALYSTS to avoid loss. As for whether applying authoritative information or
not is not an important consideration for ANALYSTS.
Brokerage firm’s research reports provide investment information for institution
and individual investors to make investment decisions. Prior researches focus on
analyzing brokerage firm’s analyst or the relation between research reports and stock
market price, and so on. This study is aimed to provide a different approach to
evaluate brokerage firm’s research reports. A model with 4 main-criteria and 19 sub-
criteria is developed to assist investors to sieve out the most valuable brokerage firm’s
research reports by modified Delphi method and AHP. Investors, new employee of
securities investment trust company and brokerage firm’s research department can
benefit by our model.
Investors can judge which report is more valuable and worthy to refer through the
criteria in our model. In the case of new employee of securities investment trust
company, our model can help them to familiarize with research practices quickly. As
for brokerage firms research department, users of their research reports are existing
and potential clients. If those clients do not trust brokerage firm’s research reports,
then they will leave and brokerage firms will lose revenue. Our model is an important
impetus for brokerage firms when producing research reports.
We find the primary evaluation criterion of brokerage firm’s research reports is
objective. Brokerage firm’s analysts should make sure that there are no biases and
serious omissions in research reports published and declare information source to
ensure the objectivity. In the meanwhile, they must evaluate target firm’s industry
consistently. ANALYSTS do not care about the work experience of brokerage firm’s
research team when they evaluate the usefulness of research reports. They do care
about how brokerage firm’s analysts are familiar with the industries and the accuracy
of analysis process since these would affect the conscientious of brokerage firm’s
Ahsan, M. K. and Bartlema, J. (2004), Monitoring healthcare performance by analytic
hierarchy process: a developing-country perspective, International Transactions
In Operational Research, 11(4), 465-478.
Azzi, Sarah and Bird, Ron (2005), Prophets during gloom and doom downunder,
Global Finance Journal, 15(3), 337–67.
Azzi, Sarah; Bird, Ron; Ghiringhelli, Paolo; Rossi, Emanuele (2006), Biases and
information in analysts' recommendations: The European experience, Journal of
Asset Management, 6(5), 345-380.
Barber, Brad; Lehavy, Reuven; McNichols, Maureen and Trueman, Brett (2001), Can
investors profit from the prophets? Security analyst recommendations and stock
returns, Journal of Finance, 56(2), 531-563.
Barber, B.M.; Lehavy, R.; McNichols, M. and Trueman, Brett (2003), Reassessing the
returns to analysts’ stock recommendations, Financial Analysts Journal, 59(2),
Bjerring, J.H.; Lakonishok, J. and Vermaelan, T. (1983), Stock prices and financial
analysts’ recommendations, Journal of Finance, 38(1), 187–204.
Bradshaw, Mark t. (2002), The use of target prices to justify sell-side analysts’ stock
recommendations, Accounting Horizons, 16(1), 27-41.
Brav, Alon and Lehavy, Reuven (2003), An empirical analysis of analysts’ target
prices: Short-term informativeness and long-term dynamics, Journal of Finance,
Chang, Che-Wei; Wu, Cheng-Ru; Lin, Chin-Tsai and Chen, Huang-Chu (2007), An
application of AHP and sensitivity analysis for selecting the best slicing machine,
Computer & Industrial Engineering, 52, 296-307.
Chang, Che-Wei; Wu, Cheng-Ru; Lin and Chen, Huang-Chu (2008), Using expert
technology to select unstable slicing machine to control wafer slicing quality via
fuzzy AHP, Expert Systems with Applications, 34, 2210-2220.
Cowen, A.; Groysberg, B. and Healy, P.M. (2006), Which types of analyst firms are
more optimistic?, Journal of Accounting and Economics, 41(1/2), 146–199.
Elton, J. Edwin, Martin J. Gruber, and Seth Grossman (1986), Discrete expectational
data and portfolio performance, Journal of Finance, 41(3), 699-714.
Hong, Harrison and Kubik, Jeffrey D. (2003), Analyzing the analysts: Career
concerns and Biased Earnings forecasts, Journal of Finance, 58(1), 313-351.
Hsu, Pi-Fang; Wu, Cheng-Ru and Li, Zhao-Rong (2008), Optimizing resource-based
allocation for senior citizen housing to ensure a competitive advantage using the
analytic hierarchy process, Building and Environment, 43, 90-97.
Jacob, John; Rock, Steve and Weber, David P. (2008), Do non-investment bank
analysts make better earnings forecasts?, Journal of Accounting, Auditing &
Finance, 23(1), 23-61.
Jegadeesh, Narasimhan; Kim, Joonghyuk; Krische, Susan D. and Lee, Charles M. C.
(2004), Analyzing the analysts: When do recommendations add value?, Journal of
Finance, 59(3), 1083-1124.
MacCarthy, B. and Atthirawong, W. (2003), Factors affecting location decisions in
international operations: a Delphi study, International Journal of Operations and
production Management, 23(7), 794-818.
Michaely, Roni and Womack, Kent L. (1999), Conflict of interest and the credibility
of underwriter analyst recommendations, The Review of Financial Studies, 12(4),
Mikhail, Michael B., Walther, Beverly R. and Willis, Richard H. (2003), The effect of
experience on security analyst underreaction, The Journal of Accounting and
Economics, 35, 101–116
Ryan, Paul and Taffler, Richard (2006), Do brokerage houses add value? The market
impact of UK sell-side analyst recommendation changes, The British Accounting
Review, 38, 371–386.
Saaty, T. (1999), Decision making for leaders: The analytic hierarchy process for
decisions in a complex world, RWS Publications, Pittsburgh.
Stickel, S.E. (1995), The anatomy of the performance of buy and sell
recommendations, Financial Analysts Journal, 51(5), 25–39.
Womack, Kent L. (1996), Do brokerage analysts’ recommendations have investment
value?, Journal of Finance, 51(1), 137-167.