Download to read offline
![2
estimation has become the center of the battlegrounds in most, if not all, patent infringement cases.6
Second, reasonable royalties are the predominant measure of damages, accounting for about eighty
percent of the total damages awarded throughout the last decade.7
Methodologies for calculating reasonable royalties that the Federal Circuit8
has adopted largely
fall in two categories.9
First, the analytical approach focuses on infringer’s projection of profit for the
infringing product.10
Reasonable royalties under this approach are estimated from infringer’s extra
profit realized from sales of infringing devices.11
The second, more common approach is the so-called “hypothetical negotiation” or “willing
licensor-willing licensee” approach.12
This approach presumes that the patent claims at issue are valid
and infringed. It then attempts to ascertain the royalty a patentee and an infringer would have agreed to
just before the infringement began. This is an “ex ante licensing negotiation scenario” that willing
parties would have executed a negotiated royalty payment scheme.13
Georgia-Pacific Corp. v. U.S. Plywood Corp. established the legal framework for assessing
reasonable royalties and enumerated fifteen relevant factors to consider.14
The Federal Circuit has
5
The standard for awarding lost profits is established by Panduit Corp v. Stahlin Bros. Fibre Works, Inc., 575 F.2d 1152
(6th Cir. 1978), in which the Federal Circuit has adopted and applied in numerous patent infringement cases. Panduit
provides that the patentee must demonstrate with reasonable probability that bur-for the infringement, it would not have lost
profits. Thus, the patentee must show: (1) demand for the patented product; (2) absence of acceptable non-infringing
substitutes; (3) manufacturing capability to exploit the demand; and (4) the amount of profit the patentee would have made.
Id. at 1156.
6
E.g., Rembrandt Social Media, LP v. Facebook, Inc., 22 F. Supp. 3d 585 (E.D. Va. 2013).
7
BARRY ET AL., supra note 4, at 11. Reasonable royalties constitute eighty-one percent of whole damages awards in
between 2007 and 2012 and seventy-nine percent in between 2001-2006. Id. at 11. During that period, the second most
frequently awarded damages, lost profits, constituted only about thirty percent. Because each form of damages may be
awarded on a non-exclusive base, the totals exceed one hundred percent. Damages awards based on price erosion have
become miniscule.
8
The United States Court of Appeals for the Federal Circuit has exclusive jurisdiction over patent infringement actions
appealed from the district courts. 28 U.S.C. § 1295(a)(1).
9
Lucent Techs. Inc. v. Gateway, Inc., 580 F.3d 1301, 1324–25 (Fed. Cir. 2009).
10
Id. at 1324; TWM Mfg. Co. v. Dura Corp., 789 F.2d 895, 898–900 (Fed. Cir. 1986).
11
TWM Mfg., 789 F.2d at 899 (“[the expert] subtracted the infringer’s usual or acceptable net profit from its anticipated net
profit realized from sales of infringing devices”).
12
Lucent, 580 F.3d at 1324–25.
13
Id. at 1325.
14
318 F. Supp. 1116, 1120 (S.D.N.Y. 1970). Fifteen factors are: (1) royalties the patentee has received for the licensing of
the patent in suit; (2) royalty rates the licensee has paid for the use of other patents comparable to the patent in suit; (3) the
exclusivity and restrictiveness of the license; (4) patentee’s licensing policy to maintain or control patent monopoly by not
licensing the invention to others; (5) the commercial relationship between parties; (6) the effect of selling the patented
specialty in promoting sales of other products; (7) the duration of the patent and the term of the license; (8) the established
profitability of the products made under the patent; (9) advantages of the patented product over the old products; (10) the
commercial aspects of the patented invention; (11) the extent to which the infringer has used the invention; (12) the portion
of the profit customarily allowed for the use of the patent; (13) the portion of the realized profit attributable to the patent; (14)
qualified experts’ testimony; and (15) the outcome from a hypothetical arm’s-length negotiation between parties. Id.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-2-2048.jpg)
![3
adopted Georgia-Pacific analysis as the “legally-required framework.”15
The test is a flexible one: not
all factors are necessarily relevant to the case at bar, nor is any particular factor dispositive.16
A royalty – whether reasonable or unreasonable, or whether structured as a running royalty or a
lump-sum payment – is a product of two distinct quantities: a royalty base and a royalty rate.17
For
example, assume that a patent at issue claims a method of manufacturing a special polymer material.
The patentee/licensor may license the patented technology to a golf ball manufacturer who uses the
method to manufacture and sell golf balls to retailers at a wholesale price of $500 for a box. The
patentee and the manufacturer may enter into a license agreement, which specifies the royalty base to be
the wholesale price and the royalty rate to be 20%. Assuming further that the manufacturer sold 1,000
boxes, the agreement requires the manufacturer to pay 000,100$1000%20500$ =×× as a royalty to the
patentee. The patentee and another manufacturer may agree to use the royalty base as the
manufacturer’s profit margin and apply the royalty rate of 30%. If this manufacturer’s profit margin is
$150 for a box and it sold 2,000 boxes, the patentee is expected to receive royalty payment of
000,90$2000%30150$ =×× .
Although these examples assumed a voluntary, willing licensor-willing licensee scenario, the
same principle applies to post-infringement reasonable royalty calculations. The royalty base represents
“the revenue pool implicated by the infringement,” and the royalty rate accounts the percentage of the
revenue pool “adequate to compensate” the patentee for the infringement.18
Both the base and the rate,
as distinct quantities, must be reasonable to reach the end result of a reasonable royalty.19
A. Royalty Rate
The first two Georgia-Pacific factors examine “royalties the patentee receives for licensing the
patent in suit” and “[t]he rates paid by the licensee for the use of other patents comparable to the patent
15
Lucent, 580 F.3d at 1335.
16
Id.
17
Cornell Univ. Hewlett-Packard Co., 609 F. Supp. 2d 279, 285 (N.D.N.Y. 2009).
18
Id.
19
Id. (“An over-inclusive royalty base including revenues from the sale of non-infringing components is not permissible
simply because the royalty rate is adjustable.” (citing Rite-Hite Corp. v. Kelley Co., 56 F.3d 1538, 1549 n.9 (Fed. Cir. 1995)));
see Virnetx, Inc. v. Cisco Sys., 767 F.2d 1308, 1328–30 (Fed. Cir. 2014) (holding that the expert testimony on royalty base
was inadmissible and should have been excluded, whereas finding that the district court did not abuse its discretion in
permitting the testimony on royalty rate estimation). Mathematically, infinite combinations of the royalty base and rate are
possible to produce the same royalty payment. For example, 1% of $100 and 10% of $10 both result in $1 royalty. It
illustrates dangers of how easily one can manipulate the base and rate to reach the value that one conclusively has presumed
to be reasonable.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-3-2048.jpg)
![4
in suit.”20
Until recently, however, the so-called “25% rule of thumb” has been widely used by the
parties and accepted (or at least passively tolerated) by the court. It assumed, as a rule of thumb, that
“25% of the value of the product would go to the patent owner and the other 75% would remain with
[the infringer/licensee],” because the infringer/licensee also has substantially contributed to the
development or realization of the commercialization of the product embodying the patented
technology.21
At first glance, automatically allocating a quarter of the product’s value to the patentee is
methodologically unsound, even considering that 25% is but a temporary, adjustable baseline rate.
Simply stated, the 25% rule of thumb’s rationale (or lack thereof) does not reach the Daubert standard.22
The Federal Circuit in Uniloc conceded that it has tolerated the use of the 25% rule.23
The
Circuit court noted that lower courts have invariably admitted royalty evidence based on the 25% rule,
because of its widespread acceptance or because its admissibility was uncontested.24
Experts, in turn,
justified the use of the rule because courts have accepted the rule as an appropriate methodology in
determining damages.25
At last, Uniloc concluded that mere widespread use of the 25% rule is not
sufficient to pass the muster under Daubert and the Federal Rules of Evidence, and consequently held
that testimony based on the 25% rule inadmissible, because “it fails to tie a reasonable royalty base to
the facts of the case at issue.”26
B. Royalty Base
Likewise, the Federal Circuit has reviewed the reasonable royalty base calculation with
heightened scrutiny, demanding a scientific methodology that reflects case-specific, real-world data.
What have particularly troubled the court are the patentee’s attempts to inflate the royalty base.
Intuitively, the maximum royalty base that the patentee could seek is the entire value of the product that
embodies the patented feature. This so-called “entire market value rule” (EMVR) permits recovery of
damages based on the value of the entire product even though such product embodies many features not
encompassed by the patent at issue.
20
Georgia-Pacific Corp. v. U.S. Plywood Corp., 318 F. Supp. 1116, 1120 (S.D.N.Y. 1970).
21
Uniloc USA Inc. v. Microsoft Corp., 632 F.3d 1292, 1318 (Fed. Cir. 2011).
22
It is quite surprising that the Federal Circuit abolished the 25% rule of thumb in 2011 in Uniloc, about two decades after
the Supreme Court decided Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579 (1993). Measuring from Kumho Tire Co.,
Ltd. v. Carmichael, 526 U.S. 137 (1999), more than a decade has passed.
23
Uniloc, 632 F.3d at 1314 (“court has passively tolerated [25% rule’s] use where its acceptability has not been the focus of
the case . . . or where the parties disputed only the percentage to be applied (i.e., one-quarter to one-third), but agreed as to
the rule’s appropriateness”) (internal citations omitted).
24
Id.
25
Id. at 1311.
26
Id. at 1315.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-4-2048.jpg)
![5
The EMVR is permissible under certain circumstances. The value of the entire product may
reasonably represent the value of the patented feature if the product contains only one patented feature
or the patented feature is the single most important feature that draws a substantial consumer demand.
Assume that a product contains ten claimed inventions, one patentee holds all of them, and the accused
is found to be infringing all ten valid patents. In such a case, while it may be difficult to isolate the
estimated value of each patented feature, the patentee may be allowed to base her loyalty calculation on
the value of the entire product.
However, such a hypothetical situation is an exception rather than the norm in most of today’s
patent infringement suits where the accused products are multi-component, multi-functional systems.
For example, a smartphone contains about 250,000 patented technologies.27
Of course, many of the
quarter million patents are licensed (e.g., under reasonable and non-discriminatory licensing agreements
or by cross-licenses), have been expired or invalidated, probably should not have been issued at all, or
will likely be invalidated. The point is that most of today’s products or services consist of several, if not
hundreds or thousands of, patented invention.
As early as late nineteenth century, courts recognized the reality and risk of over-inclusiveness of
using the EMVR.28
The use of the infringing product’s entire market value as a royalty base is allowed
“only where the patented feature creates the ‘basis for customer demand’ or ‘substantially create[s] the
value of the component parts.’”29
Specifically, using the entire market value as a royalty base requires
adequate proof of following conditions: (1) the infringing components must be the basis for consumer
demand for the entire product, beyond the claimed invention; (2) the individual infringing and non-
infringing components must be sold together; and (3) the individual infringing and non-infringing
components must be analogous to a single functioning unit.30
Where the EMVR is inappropriate and exaggerates the royalty base, the royalty base must be tied
to the patented feature at issue. As required under Daubert and Federal Rules of Evidence, after all,
27
Innovation, TECHDIRT, There Are 250,000 Active Patents That Impact Smartphones; Representing One In Six Active
Patents Today, https://www.techdirt.com/blog/innovation/articles/20121017/10480520734/there-are-250000-active-patents-
that-impact-smartphones-representing-one-six-active-patents-today.shtml (last visited Apr. 2, 2015).
28
Lucent Techs. Inc. v. Gateway, Inc., 580 F.3d 1301, 1336–37 (Fed. Cir. 2009) (tracing the origins of the entire market
value rule to several Supreme Court cases).
29
Uniloc, 632 F.3d at 1318 (quoting Lucent, 580 F.3d at 1336; Rite-Hite Corp. v. Kelley Co., 56 F.3d 1538, 1549–50 (Fed.
Cir. 1995)) (emphasis added).
30
Cornell Univ. v. Hewlett-Packard Co., 609 F. Supp. 2d 279, 286–87 (N.D.N.Y. 2009).](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-5-2048.jpg)
![10
B. Bundle of Attributes
Conjoint analysis conceptualizes products or services as bundles of attributes. Each attribute can
have one or more levels. The end product is assumed to be characterized solely by the set of attributes
with designated levels embodied in that product.
For example, a consumer who considers buying a laptop may consider following four attributes:
brand, display size, price, and storage type, to name but a few. Each attribute can have one or more
levels, and the levels can be either quantitative or qualitative. A brand may have four qualitative levels,
e.g., Apple, Dell, Lenovo, and Toshiba. Similarly, a storage type may consist of two qualitative levels,
e.g., Hard Disk Drive (HDD) and Solid State Drive (SSD). Three quantitative levels may comprise an
attribute display size, e.g., less than 13-inch, 14-to-16 inch, and 17-inch or larger. A price range may be
divided into three levels, e.g., less than $1,000, $1,000 to $1,200, and over $1,200.
A combination of attributes and corresponding levels, which referred to as a profile,
characterizes each laptop. One laptop may be an Apple laptop with 13-inch display and SSD storage,
sold at $1,299 (“Profile 00”). Another laptop may be a Lenovo laptop with 14-inch display and HDD
storage, sold at $1,099 (“Profile 01”). In theory, this example can generate up to 72 ( 2334 ××× )
profiles.
Conjoint analysis assumes that each consumer has a set of weights (or values) in units of utility
associated with each level of each attribute. They are referred to as partworth.51
Each partworth
contributes to the total value (or utility) of a product.52
Consumers are assumed to behave rationally in a
sense that they would choose the product with the maximum utility.53
Put differently, what an
individual selects is deemed to have the maximum utility to that individual among the available
alternatives. Thus, conjoint analysis is “consistent with economic and consumer-behavior theories of
approximate utility maximization.”54
Using the same laptop example, if a consumer was asked to choose one model from the 72
alternative laptops, and if Profile 00 was chosen, it is assumed that her choice has revealed or stated that
the Profile 00 gives her the maximum utility among 72 available alternatives. Comparing just two
51
Partworth and utility are used interchangeably. Strictly speaking, however, partworths are measured in units of utility.
For example, the partworth of the HDD is ten (units of utility) and that of the SDD is fifteen (units of utility).
52
The way partworth contributes to the total utility differs depending on how one models the utility function. See infra Part
III.
53
MOSHE BEN-AKIVA & STEVEN R. LERMAN, DISCRETE CHOICE ANALYSIS: THEORY AND APPLICATION TO TRAVEL DEMAND
38 (Marvin L. Manheim ed., 1985).
54
Expert Report of John R. Hauser at 11, Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK (N.D. Cal.
July 26, 2012), ECF No. 1363–1 [hereinafter Hauser Report].](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-10-2048.jpg)
![12
Figure 1. Three types of the utility function.60
Before jumping to Part II, it is worth mentioning briefly why marketing professionals prefer a
seemingly complex conjoint survey to a simple, direct questioning to isolate and assess the value of a
particular feature from a multi-component system.
D. Advantage & Validity
Academics recognize the unreliability of direct questioning to price a subject matter. One study
in particular shows that what a respondent says how she would value differs from how she actually
reacts. A classic example is a survey conducted on MBA students regarding their career choice.61
When asked directly prior to making decisions, they ranked salary as the sixth most important factor in
their career choice.62
However, salary was the most important factor influencing their choice as the
conjoint analysis analyzed after they actually had accepted a position.63
This study has made researchers
doubt the reliability of the data obtained from direct questioning. Academics also recognize that focus
bias (or hypothetical bias, which is similar to a leading question),64
upward bias for socially sensitive
60
Green et al., supra note 45, at S60 (originally from Green & Srinivasan, supra note 58, at 106).
61
Expert Report of Dr. V. Srinivasan (Blue-ray Players) at 7, TV Interactive Data Corp. v. Sony Corp., No. 4:10–cv–00475–
PJH (N.D. Cal. Jan. 21, 2013), ECF No. 580–1 [hereinafter Srinivasan Report] (citing David B. Montgomery, Conjoint
Calibration of the Customer/Competitor Interface in Industrial Markets, in INDUSTRIAL MARKETING: A GERMAN-AMERICAN
PERSPECTIVE 297–319 (Klaus Backhaus & David T. Wilson eds., 1985)).
62
Id.
63
Id.
64
Sentius Int’l, LLC v. Microsoft Corp., No. 5:13–cv–00825–PSG, 2015 U.S. Dist. LEXIS 8782, at *19–20 (N.D. Cal. Jan.
23, 2015). In Sentius, the survey expert recognized that the respondents might have overestimated their willingness-to-pay
for a particular feature of a product when answering direct open-ended questions. Id. at * 20. Therefore, the expert used a
calibration factor of 1.46 to adjust the respondents’ overestimation. Id.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-12-2048.jpg)
![15
A. Hypo74
A car manufacturer, MANU, is planning to launch a new model that uses light-emitting diodes
(LEDs) on its headlight. Before investing a chunk of money on R&D and implementation, MANU
wants to know how consumers would value its new model with LED headlights. It determines that only
three features would influence consumers’ decision-making. The three attributes MANU chooses are: (1)
whether the headlights are LEDs; (2) fuel efficiency as measured by mile-per-gallon (mpg); and (3)
price. It decides to test for two levels of the fuel efficiency: 30 mpg and 35 mpg. Price levels are set at
$30,000, $32,000, and $34,000, as MANU’s new model targets in that price range.
Table 1 summarizes three attributes and each level of the corresponding attributes. The first
column indicates that, for convenience, each level of an attribute will be coded with numbers 0, 1, or 2.
Accordingly, each mode/profile can be described by a set of codes. For example, a model that does not
have LEDs (“0”), with fuel efficiency of 35 mpg (“1”), and with a price tag of $32,000 (“1”)
corresponds to (0, 1, 1). Similarly, a model with LEDs and 30 mpg, and sold at $34,000 is represented
by (1, 0, 2).
Table 1. Attributes and levels for MANU cars.
Attributes/Levels LEDs Fuel Efficiency [mpg] Price [$]
0 No 30 30,000
1 Yes 35 32,000
2 34,000
B. Full-Profile Rating Model
The combination of the attributes and levels can generate up to 12 ( 322 ×× ) profiles. In a full-
profile survey, respondents are shown a set of cards that fully describes the profile. Here, each card
describes: whether the headlights are made of LEDs; whether the fuel efficiency is 30 mpg or 35 mpg;
and the price of the car as one of the three amounts – $30,000, $32,000, or $34,000. The respondent in
74
The same fact pattern can be used in the context of a patent infringement suit, where the patentee attempts to evaluate the
value of the LED headlights. For example, an inventor INVE has a patent that claims a method of manufacturing LEDs.
MANU sells a model that INVE thinks incorporates the LEDs manufactured by the method claimed in his patent. While
MANU offers sub-models with a range of prices and other features, INVE cannot be certain about the value of the LED
headlights as all MANU models use the same LED headlights INVE thinks infringing. INVE conducts a conjoint survey
assuming that only the same three attributes characterize MANU cars and affect consumers’ purchasing decision.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-15-2048.jpg)
![20
coefficient. For example, the 95% confidence interval for LEDβ is (1.03577, 2.23090) and icePrβ is (-
3.52654, -2.77346). Both intervals do not include zero.
In addition to the estimate of the reliability of the parameter estimates, the regression result also
provides a measure of the overall goodness of fit, i.e., how well the regression equation fits the data. R-
squared (or 2
R ) is a statistic that “measures the percentage of variation in the dependent variable that is
accounted for by all the explanatory variables.”86
Its value ranges from 0 (the explanatory variables
explain none of the variation of the dependent variable) to 1 (the explanatory variables explain all of the
variation of the dependent variable). While there is a no clear-cut level that makes the model
satisfactory, a high 2
R is favored.87
In this example, 2
R is very high (0.99393), which means that the
three attributes explain more than 99% of the variation in this particular respondent’s rating (and total
utility).
From Table 3 and equation (5), this respondent’s willingness-to-pay for the LED headlights in
MANU’s car can be derived. 15.3Pr −=iceβ means that the increase in price from $30,000 to $32,000
(or from $32,000 to $34,000) decreases the relative worth of the car at the rate of 3.15 units. That is,
$2,000 corresponds to the partworth of 3.15 units, and one unit of partworth represents
util
util
15.3
000,2$]/[$635 = . As the respondent’s utility with the LED headlights increase by 1.63
units, this adds 035,1$]/[$635][63.1 =× utilutil worth of value to the respondent. Thus, this
respondent’s willingness-to-pay for the LED headlights is $1,035 (when the price of the car ranges in
between $30,000 and $34,000).
A different approach would lead to the same result. Here, the task is to estimate the price
premium the respondent would be willing-to-pay for the LED headlights. Assume that the base model
has 35 mpg with a price tag at $32,000. The one model that does not have the LED headlight feature is
described by using equation (5) as
02.1315.397.52.10 =−+=NoY (6)
86
Id. at 345.
87
Id.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-20-2048.jpg)
![27
)(
)()|(
)|(
DP
HPHDP
DHP
×
= . (21)
Here, the probability of the hypothesis given the data, )|( DHP , is known as its posterior
probability.105
This is the probability of the hypothesis that reflects the data upon which the hypothesis
is based. The probability of the hypothesis, )(HP , is known as its prior probability. It describes the
analyst’s belief about the hypothesis before she saw the data. The probability of the data given the
hypothesis, )|( HDP , is known as the likelihood of the data. It is the probability of seeing that
particular collection of data, conditioned on that hypothesis about the data. Thus, equation (21) tells that
the posterior probability of the hypothesis is proportional to the product of the likelihood of the data
under that hypothesis and the prior probability of that hypothesis. The Bayesian framework provides a
formula to update the prior estimate of the probability.
The HB method analyzes the model in a hierarchical form.106
That is, the model parameters in
one level (or hierarchy) are explained in subsequent levels. Re-writing equation (10) in a simple form,
we obtain
),()( ,,,, ijVVPiP njnjnini ≠∀+>+= εε . (22)
Re-writing equation (12) in a generalized form with more than three attributes, we obtain.
nnini xV β,,
ʹ= . (23)
nix ,
ʹ , attributes of the alternative i for the n respondent is expressed in a vector form. The HB method
further assumes that individual partworth, }{ nβ , has the multivariate normal distribution, i.e.,
),(~ ∑β
ββ Normaln (24)
105
The ACA/HB Module for Hierarchical Bayes Estimation v3, SAWTOOTH SOFTWARE INC., 6 (July 2006) [hereinafter
ACA/HB Module], https://www.sawtoothsoftware.com/download/techpap/acahbtec.pdf.
106
Greg M. Allenby & Peter E. Rossi, Perspectives Based on 10 Years of HB in Marketing Research, SAWTOOTH SOFTWARE
INC., 3 (2003), http://www.sawtoothsoftware.com/download/techpap/allenby.pdf.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-27-2048.jpg)
![28
where β is the mean of the distribution of individuals’ partworth and ∑β
denotes (a matrix of)
covariances of the distribution of the partworths across individuals.
Equation (22) describes the lowest level of the hierarchy. The same interpretation given in
equations (8)-(11) applies to equation (24). At the highest level, equation (24) allows for heterogeneity
among the respondents (i.e., variance within the sample). The individual partworth estimate, }{ nβ , are
linked by a common distribution, { ∑β
β , }, which represents the sample. (Note that { ∑β
β , } do
not contain individual-specific subscript, such as n.) { ∑β
β , } are known as hyper-parameters of the
model.107
Thus, in theory, estimates of individual level parameters give data, )|( DP nβ , can be obtained
by first obtaining the joint probability of all model parameters given the data,
),|()|( | ∑∏ × β
βββ n
n
nn PDP . And,
)|,},({)|( DPDP nn ∑= β
βββ
)(
),()],|()|([
)(
)()|( |
DP
PPDP
DP
PDP n
n
nn
nn
∑∑∏ ××
=
×
=
ββ
ββββ
ββ
. (25)108
Integrating equation (25) results in
∑∫ ∑ −= ββ
βββββ dddDPDP knn )|,},({)|( . (26)109
“-k” in k−β denotes “except k.” Equations (25) and (26), thus, provide an operational procedure for
estimating a specific individual’s set of partworths, }{ nβ , given all the data in the sample, D , not
merely her data, nD . As such, the Bayes theorem provides a method of bridging the analysis across the
respondents, which conjoint analysis essentially attempts to achieve.
107
Id. at 4.
108
Id.
109
Id.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-28-2048.jpg)
![30
So far, underlying principles of two major types of conjoint analysis rest on sound mathematical
theorems. The admissibility of the expert testimony based on conjoint analysis largely depends on the
minor premise. Expert’s major battleground is on how the survey gathers and supplies the data and the
validity of the statistical analysis rendered on the data. Part III identifies eight key areas where the
admissibility and credibility of conjoint survey evidence can be challenged.
III. CONJOINT ANALYSIS AS SURVEY EVIDENCE
Since conjoint analysis is a survey-based research tool, issues pertinent to admissibility and
credibility of the survey evidence apply to expert opinions based on conjoint analysis. Over the last half
century, the survey method has been proved to be an economical and systematic way to obtain data and
draw inferences about a large number of individuals or other units.113
A complete census of the universe
can be expensive, time-consuming, and sometimes impossible. With the increasing uses of the surveys
by academic researches, businesses, and the government, both federal and state courts have admitted
survey evidence on a variety of contexts such as: discrimination in jury panel composition; employment
discrimination; class certification; community standards for obscenity; antitrust; mass torts; consumer
perception and memory; trademark infringement; and patent infringement cases.114
One federal court in
a trademark infringement case treated the absence of a survey as the plaintiff’s failure to establish actual
consumer confusion.115
While survey evidence has been attacked as inadmissible on the theory that it is inadmissible
hearsay, the contemporary view is that the hearsay objection is unsound.116
The respondents’ answers
are either nonhearsay or admissible as exceptions to the hearsay rule as declarations of present state of
mind or under the residual exception.117
Under the Federal Rule of Evidence and Daubert, the
admissibility of survey result centers on the “validity of the techniques employed rather than [on]
relatively fruitless inquires whether hearsay is involved.”118
The key is on the quality of survey. To be admissible, surveys should generally satisfy following
foundational requirements: (1) a relevant population (or universe) was properly defined; (2) a
113
DIAMOND, supra note 79, at 367.
114
Id. at 364–67; see, e.g., Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK, 2012 WL 2571332, at *9–
10 (N.D. Cal. June 30, 2012).
115
Morrison Entm’t Grp. v. Nintendo of Am., 56 F. App’x 782, 785 (9th Cir. 2003); DIAMOND, supra note 79, at 372
(“[S]everal courts have drawn negative inference from the absence of a survey, taking the position that failure to undertake a
survey may strongly suggest that a properly done survey would not support the plaintiff’s position.”).
116
1 PAUL C. GIANNELLI ET AL., SCIENTIFIC EVIDENCE, § 15.04[b], at 851–52 (5th ed., 2012).
117
Id.
118
FED. R. EVID. 703 advisory committee’s note.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-30-2048.jpg)
![31
representative sample of that population was selected; (3) the questions were presented in a clear and
non-leading manner; (4) interview procedures were sound and unbiased; (5) the data was accurately
gathered and reported; (6) the data was analyzed in accordance with accepted statistical principles; and
(7) objectivity of the process was assured.119
This Part examines each factor in turn. The third factor
will be addressed in two parts. Issues relating to the determination of survey attributes and levels will be
examined first. Then, the focus will move to the phrasing and/or presentation of the questionnaires.
A. Relevant Population
A population or universe is a complete set or all the units of interest to the researcher. The target
population must be relevant to the questions of the survey.120
The starting point in the development of a
methodologically sound survey is identification of the appropriate population. Thus, courts have
considered the selection of proper population “as one of the most important factors in assessing the
validity of a survey as well as the weight that it should receive.”121
Leelanau Wine Cellar v. Black & Red is a trademark infringement case between competing
wineries in Michigan.122
Leelanau’s primary theory of the case was based on the secondary meaning of
the disputed mark and the resulting consumer confusion. Specifically, Leelanau alleged that Black &
Red’s “Chateau de Leelanau” mark caused consumers of the defendant’s products to mistakenly believe
that the defendant’s products were from the same source as, or were connected with, the plaintiff’s
products.123
Leelanau retained an expert to conduct a consumer survey to measure the extent to which
consumers who encounter defendant’s Chateau de Leelanau wines believed them to be the same as or
related to Leelanau Wine Cellars.124
Leelanau’s expert defined the universe as “Michigan consumers
over 21 years of age who had either purchased a bottle of wine in the $ 5 to $ 14 price range in the last
119
Leelanau Wine Cellars, Ltd. v. Black & Red, Inc., 452 F. Supp. 2d 772, 778 (W.D. Mich. 2006) (citations omitted). The
court added that, “[b]ecause almost all surveys are subject to some sort of criticism, courts generally hold that flaws in survey
methodology go to the evidentiary weight of the survey rather than its admissibility.” Id. Manual for Complex Litigation
suggests the same seven factors in assessing the admissibility of a survey. MANUAL FOR COMPLEX LITIGATION (Fourth) §
11.493 (2004). See also DIAMOND, supra note 79, at 367 (“Several critical factors have emerged that have limited the value
of some of [the] surveys: problems in defining the relevant target population and identifying an appropriate sampling frame, a
response rates that raise questions about the representativeness of the results, and a failure to ask questions that assess
opinions on the relevant issue.”).
120
DIAMOND, supra note 79, at 377 (“A survey that provides information about a wholly irrelevant population is itself
irrelevant.”).
121
Leelanau Wine Cellars, 452 F. Supp. 2d at 781.
122
Id. at 772.
123
Id. at 779.
124
Id.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-31-2048.jpg)
![32
three months or who expected to purchase a bottle of wine in that price range in the three months
following the interview.”125
The court found this universe was flawed because it was significantly overbroad.126
The court
first concluded that when the dispute centers on secondary meaning of the mark, the proper universe is
the potential purchasers of defendant’s products.127
The court noted that Black & Red’s wines are
available only through their local tasting rooms and websites.128
That finding was crucial because, while
Leelanau’s universe would certainly include purchasers of the defendant’s wine, only a tiny percentage
of the respondents in Leelanau’s universe would probably purchase Black & Red’s wines in connection
with actual visits to its tasting rooms or websites.129
In another case, the court was more receptive to survey evidence. In a patent infringement suit
between competing smartphone manufacturers, Apple’s damages expert designed and conducted a
conjoint survey to determine price premium, if any, Samsung consumers would be willing to pay for the
touchscreen features associated with the Apple’s patents at issue.130
One of the arguments for
Samsung’s unsuccessful Daubert challenge was that Apple’s expert surveyed improper recent Samsung
purchasers, rather than potential Samsung purchasers.131
Rejecting Samsung’s motion to exclude the
expert opinion, the court first noted that Samsung failed to explain why recent Samsung purchasers are
not the proper universe for Apple’s survey.132
The court found that even if the category of recent
Samsung purchasers was underinclusive, they were at least members of the relevant universe of survey
participants.133
Concluding that the underinclusive population was still probative, the court stated,
“[g]enerally, underinclusiveness of a survey goes to weight, not admissibility.”134
B. Representative Sample
A sample is a subset of the population. The sample is drawn from the population for a particular
purpose of conducting the survey. Conjoint analysis quantifies sample’s preferences and estimates
sample’s partworth and willingness-to-pay. Thus, in addition to the appropriate identification of the
125
Id. at 782.
126
Id. at 782–83.
127
Id. at 782.
128
Id.
129
Id.
130
Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK, 2012 WL 2571332, at *9–10 (N.D. Cal. June 30,
2012).
131
Id. at *9.
132
Id. at *10.
133
Id.
134
Id.; see also Microsoft Corp. v. Motorola, Inc., 905 F. Supp. 2d 1109, 1120 (W.D. Wash. 2012).](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-32-2048.jpg)
![33
relevant population, it is essential to select a sample that properly represents the relevant characteristics
of the population. The ultimate goal of the sample survey is “to provide information on the relevant
population from which the sample was drawn,” even though the data are incomplete as not obtained
from the population.135
Largely, there are two types of concerns – quantitative and qualitative – about the sample.
Quantitatively, the statistician wants a sample large enough to generate reliable statistics. Courts have
barred samples in some cases when the sample was too small to yield reliable statistics.136
The rule of
thumb is that a sample size exceeding thirty may provide stable statistics.137
More is better in general,
but not always so. When the sample is systematically skewed or biased, a larger sample size would
aggravate, not reduce, the systematic error (even though the standard error is inversely proportional to
the square root of the sample size).138
Qualitatively, the sample must be unbiased.139
Most survey researchers employ probabilistic
approaches in sampling.140
Probability sampling is known to maximize both the representativeness of
the survey results and the reliability of estimates obtained from the survey.141
Probability sampling
methods range from a simple random sampling method to complex multistage sampling schemes.142
In
a basic simple random sampling method, every element in the population has equal non-zero probability
135
DIAMOND, supra note 79 at 361.
136
See 1 GIANNELLI ET AL., supra note 116, at § 15.04[b], 858.
137
Id. at 845.
138
Id. n.98. The classic example of the large sample with the systematic bias is the Literary Digest’s 1936 presidential
election poll. The Literary Digest was one of the most popular magazines of that era and had a history of accurately
predicting the winners of presidential elections since 1916. It sent out 10 million straw ballots asking people who they
planned to vote for the 1936 presidential election. The magazine received 2.4 million ballots. Based on the responses, it
predicted Alfred Landon would beat Franklin D. Roosevelt 57% to 43%. As it turned out, Roosevelt won, with a whopping
margin of 62% to 37%. There were two huge problems with the poll. First, the initial sample (initial 10 million recipients)
did not correctly represent the population, i.e., voters. Literary Digest used lists of phone numbers, drivers’ registration, and
country club memberships to identify the sample. However, at the time, these luxuries were more often available to the
middle- and upper-class voters, which tended to exclude lower-income voters. On the other side, Roosevelt’s campaign
centered on reviving the economy at the height of the depression, which appealed to the majority of the lower income people.
Second, the sample chosen suffered from voluntary response (or nonresponse) bias, with a huge nonresponse rate of more
than 75%. A high nonresponse rate suggests that the voters who supported Roosevelt were less inclined to respond to the
survey. Famous Statistical Blunders in History, THE OXFORD MATH CTR.,
http://www.oxfordmathcenter.com/drupal7/node/251 (last visited Apr. 14, 2015); Case Study I: The 1936 Literary Digest
Poll, UNIV. PA. DEP’T OF MATHEMATICS, http://www.math.upenn.edu/~deturck/m170/wk4/lecture/case1.html (last visited
Apr. 14, 2015).
139
1 GIANNELLI ET AL., supra note 116, at § 15.04[a], 846–47.
140
See DIAMOND, supra note 79, at 382.
141
Id. at 380 (“Probability sampling offers two important advantages over other types of sampling. First, the sample can
provide an unbiased estimate that summarizes the responses of all persons in the population from which the sample was
drawn; that is, the expected value of the sample estimate is the population value being estimated. Second, the researcher can
calculate a confidence interval that describes explicitly how reliable the sample estimate of the population is likely to be.”).
142
Id.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-33-2048.jpg)
![34
of being selected in the sample.143
In stratified random sampling, the population is first divided into
mutually exclusive and exhaustive strata, and samples are selected from within these strata by basic
random sampling.144
Courts have also admitted into evidence survey results drawn from non-
probabilistic sampling.145
However, the proponent should be prepared to justify why she took a specific
non-probabilistic method to select the sample respondents in the instant case.146
With recent technological innovations, businesses and academics have increasingly used Internet
surveys for a variety of purposes. Internet survey can reduce substantially the cost of reaching potential
respondents and, at the same time, can improve the way survey is designed and presented to the
respondents. It also eliminates risks involving interviewer biases and reduces inaccuracies of data-
recording/saving as the computer program presents survey questions and collects answers automatically.
The threshold issue in evaluating an Internet survey is that the web-surfers may not fairly
represent the relevant population whose responses the survey was designed to measure.147
For example,
some Internet market research service companies maintain a huge panel of volunteers (the “panel
population”) consisting of multi-million consumers.148
Although a subset of the panel may be randomly
sampled from the panel population for conducting a specific survey, the panel population itself may not
be the product of a random selection process. The panel population likely over-represents a group of
relatively active and informative market participants.149
They may have a particular viewpoint on
subject matters and/or a motive that might bias the survey results.
Other issues concern respondent qualification and duplication. As the Internet survey is
conducted over the online, security measures must be taken to confirm that the selected sample
conforms to the purpose of the study. The procedures that Apple’s expert took in Apple v. Samsung
illustrate the concerns.150
The expert and a market research company hired by him first checked the
143
Id.
144
Id.
145
Id. at 382.
146
Id.
147
Id. at 406.
148
E.g., Hauser Report, supra note 54, at 25 (“Research now . . . maintains an invitation-only panel over 3.6 million
consumers in the United States and over 6 million panelists worldwide.”); Srinivasan Report, supra note 61, at 12 (“Optimal
Strategix Group utilized Authentic Response’s US Adults panel of approximately 3 million respondents.”); see also
DIAMOND, supra note 79, at 382 n.102.
149
One expert report notes that the panel population held by a market survey company does not accurately represent the
whole population. Srinivasan Report, supra note 61, at 12 (“This [3 million] panel is maintained to be reflective of the U.S.
Census (adults), but it is not exactly balanced with respect to the U.S. Census (adults).”). Therefore, the survey company
weighted the sample population to be a more balanced representative of the U.S. Census of adults than the un-weighted
sample. Id.
150
Hauser Report, supra note 54, at 26.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-34-2048.jpg)
![35
identity of panelists by reviewing registered email addresses and their basic demographic information.151
Based on the information, they invited people who indicated that they own smartphones.152
The
invitation email included a link to actual survey, hosted on a website maintained by the market research
company affiliated with the expert.153
The email link contained an embedded identification number to
assure that only invited respondents could answer the survey and could do so only once.154
Before
starting the survey, respondents were prompted to a CAPTCHA challenge to ensure that responses were
not computer-generated.155
In addition, a high nonresponse rate can significantly bias the survey results and can be a ground
for excluding survey evidence.156, 157
Nonresponses aggravate systematic error more seriously when the
nonresponse is not random.158
The key in understanding the pattern and effect of the nonresponse in a
survey is to determine the extent to which nonrespondents differ from respondents with respect to the
dividing characteristics of these groups.159
The proponent must review the underlying raw data to investigate whether there exists
significant nonresponses and whether the nonresponses are systematic. However, mere low response
rate may not be damming.160
“Contrary to earlier assumptions, surprisingly comparable results have
151
Id.
152
Id. This initial criterion screens the pool of potential respondents by determining whether they belong to the target
population of the survey. As the target population in the survey was smartphone owners, online survey in itself does not pose
a serious underinclusive problem with respect to the availability of Internet access. “The screening questions must be drafted
so that they do not appeal to or deter specific groups within the target population, or convey information that will influence
the respondent’s answers on the main survey.” DIAMOND, supra note 79, at 385–86.
153
Hauser Report, supra note 54, at 26.
154
Id.
155
Id. at 27. CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart. “A
CAPTCHA challenge refers to a program that protects websites against bots (i.e., computer-generated responses) by
generating and grading tests that humans can pass, but current computer programs cannot.” Id. n.32.
156
Methods of computing response/nonresponse rates vary. “[A]lthough response rate can be generally defined as the
number of complete interviews with reporting units divided by the number of eligible reporting units in the sample, decisions
on how to treat partial completions and how to estimate the eligibility of nonrespondents can produce differences in measures
of response rate.” DIAMOND, supra note 79, at 384 n.109.
157
See id. at 383–85 (suggesting 80% or higher response rates are desirable); 1 GIANNELLI ET AL., supra note 116, at §
15.04[b], 860–63.
158
DIAMOND, supra note 79, at 383 (“for example, persons who are single typically have three times the ‘not at home’ rate in
U.S. Census Bureau surveys as do family members”). See supra text accompanying note 138 (regarding Literary Digest’s
1936 presidential election poll).
159
See DIAMOND, supra note 79, at 383–84 (“The key is whether nonresponse is associated with systematic differences in
response that cannot be adequately modeled or assessed.”).
160
For example, Apple’s expert reported that for his conjoint survey, only 22.8% of the invitees responded. Hauser Report,
supra note 54, at 37 n.47 (“Out of the total 38,795 participants invited for the smartphone survey, 8,844 began the survey,
resulting in a response rate of 22.8%. . . . Out of the total 8,844 participants who started the smartphone survey, 604 passed
through the screening questions and qualified for the choice tasks, resulting in an incidence rate of 6.8%.”). The survey result
was not excluded.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-35-2048.jpg)
![36
been obtained in many surveys with varying response rates, suggesting that surveys may achieve
reasonable estimates even with relatively low response rates.”161
Where a notable pattern is found
between the respondents and nonrespondents groups, the proponent must be prepared to explain the
cause and prove that either the high nonresponse rate does not bias the survey results or the effects are
insignificant.
The opponent must attack a survey’s high nonresponse rate. Even if the survey evidence with
high response rate might be admitted to the trier of fact, unexplained high nonresponse rate reduces its
probative value. On the other hand, the opponent should take a survey with a high response rate with no
less scrutiny. What matters are the determinants of the nonresponse.162
Even if the survey has a high
response rate, it still can underrepresent some portions of the relevant population.163
Therefore, the
opponent should examine the underlying raw data from which the results were drawn.
The following case illustrates the importance of recording and reporting nonresponse rates in a
survey. In a class action lawsuit, defendants-employers moved to exclude plaintiffs-employees’ expert
report by asserting that it failed to meet the standard for expert testimony outlined by Federal Rule of
Evidence 702 and Daubert.164
One of the defendants’ arguments was that the expert’s study suffered
from a substantial undocumented nonresponse bias, undermining the reliability of the opinion.165
The
plaintiffs responded that the nonresponse rate was low and that any bias was minimal.166
Noting that
“[t]he U.S. Office of Statistical Standards suggest that because nonresponse is often not random, a
survey should maintain a response rate of 75% or above[,]” the court sided with the defendants.167
The
court pointed out that the plaintiffs’ expert admitted that while the nonresponse rate was small, it was
not documented.168
The expert’s careless failure to record nonresponse rate was the ground that the
court found the expert testimony unreliable and inadmissible under Rule 702.169
161
DIAMOND, supra note 79, at 384.
162
Id. at 385.
163
Id. (“If a general population sample is used to chart changes in the proportion of the population that knows someone with
HIV, the survey would underestimate the population value if some groups more likely to know someone with HIV (e.g.,
intravenous drug users) are underrepresented in the sample.”).
164
Chavez v. IBP, Inc., No. CV–01–5093–RHW, 2004 U.S. Dist. LEXIS 28838, at *1–2 (E.D. Wash. Dec. 8, 2004).
165
Id. at *28.
166
Id.
167
Id.
168
Id.
169
Id. at *29–34. In this regard, Apple’s expert recorded and reported the initial (non)response rate and the end incidence
rate, although response and incidence rates were very low. See supra text accompanying note 160.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-36-2048.jpg)
![40
The court continued to criticize the survey’s methodology. The fact that the survey did not
include important product features such as battery life and WiFi capability, but instead selected an
arguably unimportant feature, voice dialing, exacerbated the focus bias problem.192
The expert, the
court found, failed to provide any reasonable criteria for choosing non-patented features in the survey
and instead forced the respondents to focus on the patented functionalities.193
In TV Interactive Data v. Sony, plaintiff’s damages expert designed the conjoint survey to
estimate consumers’ willingness-to-pay for the incremental benefit from using TV Interactive Data’s
patented technology, a so-called “Autoplay” function, in Sony’s Blue-ray players.194
The expert decided
to limit the number of attributes to five – the attribute representing the Autoplay function, price discount,
and three non-patented attributes.195
The expert attempted to select non-patented attributes that have
relatively similar values to that of the Autoplay function.196
He adopted a two-phase approach.
In Phase I of the study, the expert focused on generating enough attributes to be able to identify
six attributes for Phase II.197
(The partworth or the willingness-to-pay for the patented technology at
issue, Autoplay, was measured in the Phase II of the study.) The expert needed to identify six attributes
because he decided to conduct Phase II in two versions, using three non-patented attributes (with price
discount and the Autoplay function) in each version of the study. The expert, with the help of a hired
market research company, first collected information on Blu-ray players and their attributes from the
manufacturers’ and well-known retailers’ websites.198
The expert then sorted out sixteen attributes
based on a set of general considerations.199
To determine how respondents would value eighteen attributes, in Phase I the expert essentially
conducted a mini conjoint survey by using a method called Adaptive Self-Explication of Multi-Attribute
192
Id.
193
Id.
194
Srinivasan Report, supra note 61, at 3.
195
Id. at 5.
196
Id. at 10–12. Thus, the expert’s main concern in determination of non-patented attributes was slight different from that of
the Oracle court. In TV Interactive Data, the expert selected attributes that have “similar values” to the infringing feature. In
contrast, the court in Oracle criticized the expert’s failure to include “important” features. 2012 WL 850705, at * 10.
197
Id. at 9.
198
Id. at 8–9.
199
Id. at 9 (“(a) Avoid attributes that are nearly universally present (> 95% of the [Stock Keeping Units] have this attribute).
Such attributes are unlikely to be a basis for customer choice; (b) Avoid attributes that are nearly universally absent (<5% of
the [Stock Keeping Units] have this attribute). Such attributes are unlikely to be a basis for customer choice. (c) Avoid
attributes that have a large number of values (e.g., playable disc types, the physical measurements of the Blu-ray player). A
phenomenon called ‘number of levels effect’ in conjoint analysis indicates that the value for an attribute with a large number
of levels would be overstated. (d) Avoid attributes that include adjacent product categories (e.g., combinations of Blu-ray
players with Recorders, HD-DVD players) because such product categories have additional sets of attributes as well; (e)
Avoid attributes that were too highly technical to translate to customer language.”)](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-40-2048.jpg)
![41
Preferences (ASEMAP).200
As the number of attributes was rather large, he first categorized the
eighteen attributes into two buckets – the more important nine and the less important nine.201
Respondents were then asked to rank attributes within each bucket.202
This created a rank order of the
importance of all eighteen attributes. Based on the list, respondents were asked to compare a pair of two
attributes in a relative scale.203
For example, a respondent was asked to compare Autoplay and price discount in the fixed scale
of 100 points. If the respondent valued Autoplay more and assigned 60 points on it, it was deemed that
the price discount received the remaining 40 points. The expert designed the pairs such that thirteen
strategically paired comparisons were sufficient to get accurate results with less than 5% average
error.204
Log-linear multiple regression was applied to analyze the data and estimate relative importance
of the eighteen attributes.205
Comparing the relative importance of all attributes, except price discount,
the expert could identify six attributes – three for each version – thought to have comparable values to
that of the patented feature.206
iv. Include non-infringing alternative levels207
The availability of non-infringing alternatives directly influences consumer’s willingness-to-pay
for the infringing attribute. The importance of the availability of the non-infringing alternatives has
been emphasized by the court, particularly when the calculation of the reasonable royalty base was
disputed.208
When the non-infringing alternatives exist, the proponent must assess the degree to which
consumers prefer the patented feature to then available non-infringing alternatives.209
Thus, the
200
Id. at 6.
201
Id. at 11.
202
Id.
203
Id.
204
Id.
205
Id. at 12; see also Oded Netzer & V. Srinivasan, Adaptive Self-Explication of Multi-Attribute Preferences, 48 J.
MARKETING RES. 140, 140–56 (2011).
206
The expert compared “the average importance from the weighted sample for [Autoplay]” as well. Srinivasan Report,
supra note 61, at 13.
207
Although sounds similar, this topic differs from the subject discussed in the preceding section. Here, the focus is on the
level, not on the attribute. For example, where the patented technology at issue is about an “X method” of accomplishing a
touchscreen functionality, the “touchscreen capability” attribute should include non-infringing “Y method” and “Z method,”
if such methods were available at the time of the infringement. On the other hand, non-patented attributes in the preceding
section concerned about including other important or comparable attributes such as battery life or WiFi capability, which are
not the subset of the touchscreen capability.
208
Cornell Univ. Hewlett-Packard Co., 609 F. Supp. 2d 279, 286–87 (N.D.N.Y. 2009).
209
Cameron et al., supra note 47.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-41-2048.jpg)
![43
the court excluded expert’s survey results finding that the survey did not properly capture the claimed
invention because survey question was overly broad and confused the issue.216
ii. Questions should be clear and unambiguous
Second, questions should be clear to the respondents when presented. Even questions that
appear clear to the surveyor can convey unexpected meanings and ambiguities to potential
respondents.217
The wording of such questions would naturally raise doubts about the validity of the
survey by potentially distorting response data. What the questions intended to ask simply may not
match what the answers deliver if the question is subject to multiple interpretations.
Precise questions without legal or technical jargons are preferred. If using jargon cannot be
avoided, these terms must be fully explained to the respondents before asking them to answer the
questions. Still, determining whether the question was clear enough to be accurately understood would
depend on the respondents’ identity and may go to weight rather than admissibility.218
Microsoft and Motorola disputed over the licensing of patented technology used in Microsoft’s
Xbox 360 console.219
Microsoft moved to exclude Motorola’s survey evidence by arguing that the
survey was unreliable.220
Microsoft pointed that the survey asked respondents to “select the types of
video content [he or she] has viewed on his or her Xbox console and providing options of ‘MBAFF
(Macroblock Adaptive Frame/Field),’ ‘Progressive,’ and ‘not sure.’” Finding the argument
unpersuasive, the court noted that the survey defined the disputed terms. The court added that since the
questions were given to respondents who owned an Xbox, the respondents were more likely to
understand the technical terms.221
The court held that “the clarity of the survey questionnaire is a
question of fact, best resolved at trial through cross examination.”222
216
Fractus, S.A. v. Samsung, No. 6:09–cv–203–LED–JDL, 2011 U.S. Dist. LEXIS 154697, at *13 (E.D. Tex. Apr. 29, 2011).
217
DIAMOND, supra note 79, at 387.
218
Microsoft Corp. v. Motorola, Inc., 905 F. Supp. 2d 1109, 1119–20 (2012). But see Firestone v. Crown Ctr.
Redevelopment Corp., 693 S.W.2d 99, 102–03 (Mo. 1985) (en banc). The court in Firestone concluded that the question
“Based on what you have heard, read or seen, do you believe that in the current compensatory damage trials, the defendants,
such as the contractors, designers, owners, and operators of the Hyatt Hotel, should be punished?” could neither be correctly
understood nor easily answered. Id. at 103. The court noted that the phrase “compensatory damage” was unlikely to be
meaningful for laypersons. Id.
219
Microsoft, 905 F. Supp. 2d at 1119–20.
220
Id.
221
Id. at 1120.
222
Id.](https://image.slidesharecdn.com/ee567b2f-99e5-47e9-9a98-9181450641ed-151116043456-lva1-app6891/75/Conjoint-survey-paper-43-2048.jpg)
Uploaded byJaeWon Lee
Conjoint survey paper
1) Reasonable royalty calculations are important in patent infringement cases, as they represent the minimum damages a patent holder is entitled to and are the most common form of damages sought, particularly by non-practicing entities. 2) There are two main approaches to calculating reasonable royalties - the analytical approach based on infringer's profits, and the hypothetical negotiation approach which aims to determine the royalty the patent holder and infringer would have agreed to. 3) The Georgia-Pacific factors provide a legal framework for assessing reasonable royalties, considering factors relevant to a hypothetical negotiation between willing parties such as existing royalty agreements, industry standards, and the merits of the invention.
Conjoint survey paper
- 1. 1 PRICING A PATENTBY SURVEYING JaeWon Lee BACKGROUND: REASONABLE ROYALTY Reasonable royalty calculation in patent infringement litigation is critical in, at least, two aspects. First, the federal statute specifies a reasonable royalty as the minimum that the patentee is entitled to be compensated.1 Calculating reasonable royalty damages is essential in cases where non-practicing entities (NPEs) assert patent infringement claims.2 Because NPEs, by definition, neither manufacture nor sell products or services embodying the claimed invention, they mostly fail to receive reliefs based on lost profits or price erosions (or, needless to say, injunction). Simply put, NPEs do not suffer those types of injuries; they neither charge a price nor realize a profit from manufacturing or selling the products embodying the claimed invention. As a result, NPEs are usually constrained to seek damages based solely on reasonable royalties.3 Considering the increasing percentage of patent infringement suits brought by the NPEs4 and difficulties of proving other type of damages,5 reasonable royalty 1 “Upon finding for the claimant the court shall award the claimant damages adequate to compensate for the infringement, but in no event less than a reasonable royalty.” 35 U.S.C. § 284 (emphasis added). At the same time, section 284 does not set the ceiling. Reasonable royalty can be higher than the price of infringing unit or the infringer’s net profit margin. Rite- Hite Corp. v. Kelley Co., 56 F.3d 1538, 1555 (Fed. Cir. 1995). 2 Unlike the entities that use the patented technology in their products or services and exclude others from using the same, the NPEs enforce patent rights against accused infringers for profit, most commonly in the form of licensing fees, but do not manufacture products or supply services using the patented claim at issues. The NPE here refers to a broad class of entities including individual inventors, research firms, and universities. Some NPEs are known as patent monetizing entities (PMEs) as their business model relies on actively buying patents from others for the purpose of asserting the claims for profit. U.S. GOV’T ACCOUNTABILITY OFFICE, INTELLECTUAL PROPERTY: ASSESSING FACTORS THAT AFFECT PATENT INFRINGEMENT LITIGATION COULD HELP IMPROVE PATENT QUALITY 2–3 (2013), available at http://www.gao.gov/assets/660/657103.pdf. 3 Damages awards based on the lost profit and price erosion and reasonable royalty damages are not mutually exclusive. A patentee may seek all types of damages from the defendant who has infringed the same patent at bar. Also, NPEs may seek injunction as a remedy. However, the recourse in equity is much challenging, and the four-prong test under eBay Inc. v. MercExchange, L.L.C., 547 U.S. 388 (2006), significantly disfavors granting injunctive awards in the NPEs’ favor. 4 A recent study reveals that in between 2007 and 2011 suits brought by the NPEs – including PMEs, likely PMEs, individuals, research firms, or universities – have significantly increased. U.S. GOV’T ACCOUNTABILITY OFFICE, supra note 2, at 17. While the study conceded that the increase attributed to the PMEs and likely PMEs is not statistically significant, it concluded that the decrease in suits brought by operating companies and related entities (thus, non-NPEs) is statistically significant. Id. In total, the notable trend is that the volume of the suits brought by the NPEs is surging. See id. at 18. Moreover, one study revealed that although the overall success rate of the patent infringement litigation is lower for the NPEs (24.3%) than the practicing entities (34.5%), the median damages awarded to the NPEs (about $8.88 million) was significantly higher than that for the practicing entities (about $5.35 million). CHRIS BARRY ET AL., PRICEWATERHOUSECOOPERS LLP, 2013 PATENT LITIGATION STUDY: BIG CASES MAKE HEADLINES, WHILE PATENT CASES PROLIFERATE 25 (2013), available at http://www.pwc.com/en_US/us/forensic-services/publications/assets/2013-patent- litigation-study.pdf. This study analyzed 1,856 district court patent decisions issued in between 1995 and 2012.
- 2. 2 estimation has becomethe center of the battlegrounds in most, if not all, patent infringement cases.6 Second, reasonable royalties are the predominant measure of damages, accounting for about eighty percent of the total damages awarded throughout the last decade.7 Methodologies for calculating reasonable royalties that the Federal Circuit8 has adopted largely fall in two categories.9 First, the analytical approach focuses on infringer’s projection of profit for the infringing product.10 Reasonable royalties under this approach are estimated from infringer’s extra profit realized from sales of infringing devices.11 The second, more common approach is the so-called “hypothetical negotiation” or “willing licensor-willing licensee” approach.12 This approach presumes that the patent claims at issue are valid and infringed. It then attempts to ascertain the royalty a patentee and an infringer would have agreed to just before the infringement began. This is an “ex ante licensing negotiation scenario” that willing parties would have executed a negotiated royalty payment scheme.13 Georgia-Pacific Corp. v. U.S. Plywood Corp. established the legal framework for assessing reasonable royalties and enumerated fifteen relevant factors to consider.14 The Federal Circuit has 5 The standard for awarding lost profits is established by Panduit Corp v. Stahlin Bros. Fibre Works, Inc., 575 F.2d 1152 (6th Cir. 1978), in which the Federal Circuit has adopted and applied in numerous patent infringement cases. Panduit provides that the patentee must demonstrate with reasonable probability that bur-for the infringement, it would not have lost profits. Thus, the patentee must show: (1) demand for the patented product; (2) absence of acceptable non-infringing substitutes; (3) manufacturing capability to exploit the demand; and (4) the amount of profit the patentee would have made. Id. at 1156. 6 E.g., Rembrandt Social Media, LP v. Facebook, Inc., 22 F. Supp. 3d 585 (E.D. Va. 2013). 7 BARRY ET AL., supra note 4, at 11. Reasonable royalties constitute eighty-one percent of whole damages awards in between 2007 and 2012 and seventy-nine percent in between 2001-2006. Id. at 11. During that period, the second most frequently awarded damages, lost profits, constituted only about thirty percent. Because each form of damages may be awarded on a non-exclusive base, the totals exceed one hundred percent. Damages awards based on price erosion have become miniscule. 8 The United States Court of Appeals for the Federal Circuit has exclusive jurisdiction over patent infringement actions appealed from the district courts. 28 U.S.C. § 1295(a)(1). 9 Lucent Techs. Inc. v. Gateway, Inc., 580 F.3d 1301, 1324–25 (Fed. Cir. 2009). 10 Id. at 1324; TWM Mfg. Co. v. Dura Corp., 789 F.2d 895, 898–900 (Fed. Cir. 1986). 11 TWM Mfg., 789 F.2d at 899 (“[the expert] subtracted the infringer’s usual or acceptable net profit from its anticipated net profit realized from sales of infringing devices”). 12 Lucent, 580 F.3d at 1324–25. 13 Id. at 1325. 14 318 F. Supp. 1116, 1120 (S.D.N.Y. 1970). Fifteen factors are: (1) royalties the patentee has received for the licensing of the patent in suit; (2) royalty rates the licensee has paid for the use of other patents comparable to the patent in suit; (3) the exclusivity and restrictiveness of the license; (4) patentee’s licensing policy to maintain or control patent monopoly by not licensing the invention to others; (5) the commercial relationship between parties; (6) the effect of selling the patented specialty in promoting sales of other products; (7) the duration of the patent and the term of the license; (8) the established profitability of the products made under the patent; (9) advantages of the patented product over the old products; (10) the commercial aspects of the patented invention; (11) the extent to which the infringer has used the invention; (12) the portion of the profit customarily allowed for the use of the patent; (13) the portion of the realized profit attributable to the patent; (14) qualified experts’ testimony; and (15) the outcome from a hypothetical arm’s-length negotiation between parties. Id.
- 3. 3 adopted Georgia-Pacific analysisas the “legally-required framework.”15 The test is a flexible one: not all factors are necessarily relevant to the case at bar, nor is any particular factor dispositive.16 A royalty – whether reasonable or unreasonable, or whether structured as a running royalty or a lump-sum payment – is a product of two distinct quantities: a royalty base and a royalty rate.17 For example, assume that a patent at issue claims a method of manufacturing a special polymer material. The patentee/licensor may license the patented technology to a golf ball manufacturer who uses the method to manufacture and sell golf balls to retailers at a wholesale price of $500 for a box. The patentee and the manufacturer may enter into a license agreement, which specifies the royalty base to be the wholesale price and the royalty rate to be 20%. Assuming further that the manufacturer sold 1,000 boxes, the agreement requires the manufacturer to pay 000,100$1000%20500$ =×× as a royalty to the patentee. The patentee and another manufacturer may agree to use the royalty base as the manufacturer’s profit margin and apply the royalty rate of 30%. If this manufacturer’s profit margin is $150 for a box and it sold 2,000 boxes, the patentee is expected to receive royalty payment of 000,90$2000%30150$ =×× . Although these examples assumed a voluntary, willing licensor-willing licensee scenario, the same principle applies to post-infringement reasonable royalty calculations. The royalty base represents “the revenue pool implicated by the infringement,” and the royalty rate accounts the percentage of the revenue pool “adequate to compensate” the patentee for the infringement.18 Both the base and the rate, as distinct quantities, must be reasonable to reach the end result of a reasonable royalty.19 A. Royalty Rate The first two Georgia-Pacific factors examine “royalties the patentee receives for licensing the patent in suit” and “[t]he rates paid by the licensee for the use of other patents comparable to the patent 15 Lucent, 580 F.3d at 1335. 16 Id. 17 Cornell Univ. Hewlett-Packard Co., 609 F. Supp. 2d 279, 285 (N.D.N.Y. 2009). 18 Id. 19 Id. (“An over-inclusive royalty base including revenues from the sale of non-infringing components is not permissible simply because the royalty rate is adjustable.” (citing Rite-Hite Corp. v. Kelley Co., 56 F.3d 1538, 1549 n.9 (Fed. Cir. 1995))); see Virnetx, Inc. v. Cisco Sys., 767 F.2d 1308, 1328–30 (Fed. Cir. 2014) (holding that the expert testimony on royalty base was inadmissible and should have been excluded, whereas finding that the district court did not abuse its discretion in permitting the testimony on royalty rate estimation). Mathematically, infinite combinations of the royalty base and rate are possible to produce the same royalty payment. For example, 1% of $100 and 10% of $10 both result in $1 royalty. It illustrates dangers of how easily one can manipulate the base and rate to reach the value that one conclusively has presumed to be reasonable.
- 4. 4 in suit.”20 Until recently,however, the so-called “25% rule of thumb” has been widely used by the parties and accepted (or at least passively tolerated) by the court. It assumed, as a rule of thumb, that “25% of the value of the product would go to the patent owner and the other 75% would remain with [the infringer/licensee],” because the infringer/licensee also has substantially contributed to the development or realization of the commercialization of the product embodying the patented technology.21 At first glance, automatically allocating a quarter of the product’s value to the patentee is methodologically unsound, even considering that 25% is but a temporary, adjustable baseline rate. Simply stated, the 25% rule of thumb’s rationale (or lack thereof) does not reach the Daubert standard.22 The Federal Circuit in Uniloc conceded that it has tolerated the use of the 25% rule.23 The Circuit court noted that lower courts have invariably admitted royalty evidence based on the 25% rule, because of its widespread acceptance or because its admissibility was uncontested.24 Experts, in turn, justified the use of the rule because courts have accepted the rule as an appropriate methodology in determining damages.25 At last, Uniloc concluded that mere widespread use of the 25% rule is not sufficient to pass the muster under Daubert and the Federal Rules of Evidence, and consequently held that testimony based on the 25% rule inadmissible, because “it fails to tie a reasonable royalty base to the facts of the case at issue.”26 B. Royalty Base Likewise, the Federal Circuit has reviewed the reasonable royalty base calculation with heightened scrutiny, demanding a scientific methodology that reflects case-specific, real-world data. What have particularly troubled the court are the patentee’s attempts to inflate the royalty base. Intuitively, the maximum royalty base that the patentee could seek is the entire value of the product that embodies the patented feature. This so-called “entire market value rule” (EMVR) permits recovery of damages based on the value of the entire product even though such product embodies many features not encompassed by the patent at issue. 20 Georgia-Pacific Corp. v. U.S. Plywood Corp., 318 F. Supp. 1116, 1120 (S.D.N.Y. 1970). 21 Uniloc USA Inc. v. Microsoft Corp., 632 F.3d 1292, 1318 (Fed. Cir. 2011). 22 It is quite surprising that the Federal Circuit abolished the 25% rule of thumb in 2011 in Uniloc, about two decades after the Supreme Court decided Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579 (1993). Measuring from Kumho Tire Co., Ltd. v. Carmichael, 526 U.S. 137 (1999), more than a decade has passed. 23 Uniloc, 632 F.3d at 1314 (“court has passively tolerated [25% rule’s] use where its acceptability has not been the focus of the case . . . or where the parties disputed only the percentage to be applied (i.e., one-quarter to one-third), but agreed as to the rule’s appropriateness”) (internal citations omitted). 24 Id. 25 Id. at 1311. 26 Id. at 1315.
- 5. 5 The EMVR ispermissible under certain circumstances. The value of the entire product may reasonably represent the value of the patented feature if the product contains only one patented feature or the patented feature is the single most important feature that draws a substantial consumer demand. Assume that a product contains ten claimed inventions, one patentee holds all of them, and the accused is found to be infringing all ten valid patents. In such a case, while it may be difficult to isolate the estimated value of each patented feature, the patentee may be allowed to base her loyalty calculation on the value of the entire product. However, such a hypothetical situation is an exception rather than the norm in most of today’s patent infringement suits where the accused products are multi-component, multi-functional systems. For example, a smartphone contains about 250,000 patented technologies.27 Of course, many of the quarter million patents are licensed (e.g., under reasonable and non-discriminatory licensing agreements or by cross-licenses), have been expired or invalidated, probably should not have been issued at all, or will likely be invalidated. The point is that most of today’s products or services consist of several, if not hundreds or thousands of, patented invention. As early as late nineteenth century, courts recognized the reality and risk of over-inclusiveness of using the EMVR.28 The use of the infringing product’s entire market value as a royalty base is allowed “only where the patented feature creates the ‘basis for customer demand’ or ‘substantially create[s] the value of the component parts.’”29 Specifically, using the entire market value as a royalty base requires adequate proof of following conditions: (1) the infringing components must be the basis for consumer demand for the entire product, beyond the claimed invention; (2) the individual infringing and non- infringing components must be sold together; and (3) the individual infringing and non-infringing components must be analogous to a single functioning unit.30 Where the EMVR is inappropriate and exaggerates the royalty base, the royalty base must be tied to the patented feature at issue. As required under Daubert and Federal Rules of Evidence, after all, 27 Innovation, TECHDIRT, There Are 250,000 Active Patents That Impact Smartphones; Representing One In Six Active Patents Today, https://www.techdirt.com/blog/innovation/articles/20121017/10480520734/there-are-250000-active-patents- that-impact-smartphones-representing-one-six-active-patents-today.shtml (last visited Apr. 2, 2015). 28 Lucent Techs. Inc. v. Gateway, Inc., 580 F.3d 1301, 1336–37 (Fed. Cir. 2009) (tracing the origins of the entire market value rule to several Supreme Court cases). 29 Uniloc, 632 F.3d at 1318 (quoting Lucent, 580 F.3d at 1336; Rite-Hite Corp. v. Kelley Co., 56 F.3d 1538, 1549–50 (Fed. Cir. 1995)) (emphasis added). 30 Cornell Univ. v. Hewlett-Packard Co., 609 F. Supp. 2d 279, 286–87 (N.D.N.Y. 2009).
- 6. 6 expert opinions onthe royalty base must be sufficiently tied to the facts of the case.31 In Daubert’s words, it must “fit.”32 Otherwise, the expert opinion is irrelevant (or marginally relevant at best) and likely will confuse the jury. The court in Cornell Univ. v. Hewlett-Packard Co. first apportioned the royalty base to the “smallest salable infringing unit with close relation to the claimed invention.”33 The Federal Circuit, in LaserDynamics, subsequently adopted the smallest salable patent-practicing unit formulation from Cornell.34 As the two cases demonstrate, the idea is that the royalty base for the infringed feature must be reduced, at least, to the smallest unit embodying that feature available in the market. For example, if a claimed invention is a method for identifying the type of optical disc inserted into an optical disc drive (ODD), the royalty base calculation must start from, at most, the ODD, not from the entire computer that incorporates the ODD.35 Although useful in some cases, the smallest salable patent-practicing unit approach has not provided a complete solution for the fundamental issue in determining the reasonable royalty base. That is because even the smallest unit can still encompass a number of non-infringing components or functions. The ODD example illustrates this point. A typical ODD certainly is manufactured based on numerous patented technologies and incorporates more than one patented feature claimed in LaserDynamics.36 Reading Cornell and LaserDynamics together with Lucent and Uniloc, courts require an apportionment whenever the accused product is a multi-component product encompassing non- infringing features.37 C. Market’s Willingness-To-Pay & Survey While the Federal Circuit has not developed a Daubert motion-proof method of estimating reasonable royalties, it has underscored several times that the estimation must be “tied to the facts of the 31 Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 591 (1993) (“An additional consideration under Rule 702 – and another aspect of relevancy – is whether expert testimony proffered in the case is sufficiently tied to the facts of the case that it will aid the jury in resolving a factual dispute.”). 32 Id. 33 Cornell, 609 F. Supp. 2d at 288. 34 LaserDynamics, Inc. v. Quanta Computer, Inc., 694 F.3d 51, 67 (Fed. Cir. 2012). 35 Id. at 56–57, 67. 36 The patent at issue in LaserDynamics, U.S. Patent No. 5,587,981 (the “’981 Patent”), cites four patents directed at optical disk systems. More than forty U.S. patents cite the ’981 Patent. 37 Dynetix Design Solutions, Inc. v. Synopsys, Inc., No. C 11–05973 PSG, 2013 WL 4538210, at *3 (N.D. Cal. Aug. 22, 2013); see Virnetx, Inc. v. Cisco Sys., 767 F.2d 1308, 1327 (Fed. Cir. 2014) (“Where the smallest salable unit is, in fact, a multi-component product containing several non-infringing features with no relation to the patented feature . . ., the patentee must do more to estimate what portion of the value of that product is attributable to the patented technology.”).
- 7. 7 case” and reflect“footprint in the market place.”38 After Lucent and Uniloc and their progenies, parties have increasingly sought a research tool capable of analyzing consumer behaviors in the market and quantifying the part effect a particular feature, in isolation, can contribute to the consumer’s decision- making process. Naturally, a market data-driven approach has emerged as an attractive tool to quantify consumers’ willingness-to-pay a price premium for a specific feature of a multi-component/function system.39 The underlying idea is that by aggregating individual consumer’s willingness-to-pay, the market’s willingness-to-pay the premium can be used as a reliable estimate that captures the additional value a particular feature contributes to the infringing product’s total value.40 The price premium that the particular feature contributes to the infringing product becomes the royalty base (or at least a sound starting point) for reasonable royalty calculations.41 The challenge still remains: how to isolate the specific feature from the product and find consumer’s willingness-to-pay just for that feature – especially where such market does not exist in reality. A type of consumer surveys called the conjoint survey or conjoint analysis can accomplish the task. Part I initially explains conjoint analysis in a conceptual level. Then, based on a hypo, two types of widely applied conjoint analyses will be described in Part II. Part III identifies eight key areas where the admissibility and credibility of survey evidence can be challenged and addresses critical legal and practical issues relevant to conjoint analysis. Before closing the paper, Part IV briefly discusses the interpretation of the conjoint survey results. Although this paper starts with and focuses on the calculation of reasonable royalty damages in a patent infringement suit, most issues and points addressed in Parts III and IV apply beyond the patent 38 Uniloc USA Inc. v. Microsoft Corp., 632 F.3d 1292, 1317 (Fed. Cir. 2011) (“To be admissible, expert testimony opinion on a reasonable royalty must ‘carefully tie proof of damages to the claimed invention’s footprint in the market place.’” (quoting ResQNet.com, Inc. v. Lansa, Inc., 594 F.3d 860, 869 (Fed. Cir. 2010))). 39 Shankar Iyer, IP Expert On The Push For Market-Based Evidence In High-End Patent Litigation, METRO. CORP. COUNSEL (Oct. 23, 2014), http://www.veooz.com/news/vHdZq0R.html. Strictly speaking, the conjoint analysis does not always rely on market data (revealed data in the market), but instead relies on survey data known as stated-preferences. See infra Part I.A. The survey, however, has been recognized as an important and reliable tool to understand and, more importantly, predict the market, if carefully designed and conducted accordingly. 40 See S. Christian Platt & Bob Chen, Recent Trends and Approaches in Calculating Patent Damages: Nash Bargaining Solution and Conjoint Surveys, 86 PATENT, TRADEMARK & COPYRIGHT J. 1, 4 (2013), available at http://www.paulhastings.com/docs/default-source/PDFs/platt_chen_insight_final.pdf; Shankar Iyer, Consumer Surveys and Other Market-Based Methodologies in Patent Damages, A.B.A. SEC. INTELL. PROP. (Oct. 16, 2014), http://apps.americanbar.org/litigation/committees/intellectual/articles/fall2014-0914-consumer-surveys.html. 41 This paper focuses on deriving reasonable royalty bases using conjoint analysis. As explained, reasonably royalty calculations also require a reasonable royalty rate. The rate can be estimated, for example, by evaluating rates the licensee has paid for the use of other patents comparable to the patent in suit and/or rates the patentee has accepted from other licensees for the infringed patent. See Georgia-Pacific Corp. v. U.S. Plywood Corp., 318 F. Supp. 1116, 1120 (S.D.N.Y.).
- 8. 8 infringement litigation. Thosediscussions equally apply to survey evidence in general beyond the context of the conjoint survey. It should be emphasized that using conjoint analysis, or surveys, to quantify consumer’s willingness-to-pay for an isolated, particular feature is only the first step.42 Reasonable royalty calculations require determination of at least one other distinct term, a reasonable royalty rate.43 I. WHAT IS CONJOINT ANALYSIS? Conjoint analysis concerns day-to-day decisions of consumers: how and why consumers choose one product over another. It is a survey-based market research tool that quantitatively measures trade- offs made by consumers. The approach is particularly useful when a multitude of factors potentially influencing consumers complicate the analysis of identifying a causal relationship between a specific feature and consumer preferences. On one end, several features are joined together in a product, and consumers consider the product jointly on the other end.44 Since its introduction to the field of marketing research in 1971, academics and industry practitioners have made conjoint analysis the most widely studied and applied form of quantitative consumer preference measurements.45 A. Survey & Stated Preference In the free market system, when needs arise, consumers gather information about products or services, compares any available alternatives, consider constraints, and decide which one to choose by making trade-offs among competing alternatives.46 In the market, consumer preference is revealed as a 42 “The first step in a damages study is the translation of the legal theory of the harmful event into an analysis of the economic impact of that event.” Comcast Corp. v. Behrend, 133 S. Ct. 1426, 1435 (2013) (citing FEDERAL JUDICIAL CENTER, REFERENCE MANUAL ON SCIENTIFIC EVIDENCE 432 (3d ed. 2011)) (emphasis from the Court). 43 Royalty payment can be structured either as a running royalty or a lump-sum royalty. The first method would require determination of the numbers of units sold or used as a multiplying factor to reach the end amount the patentee is entitled. Significantly distinguishable risks and analyses involve in two types of the royalty calculations. See Lucent, 580 F.3d at 1326–32. 44 See BRYAN K. OMRE, GETTING STARTED WITH CONJOINT ANALYSIS: STRATEGIES FOR PRODUCT DESIGN AND PRICING RESEARCH 29 (2d. 2010), available at http://www.sawtoothsoftware.com/download/techpap/cahistory.pdf (“Marketers sometimes have thought (or been taught) that the word conjoint refers to respondents evaluating features of products or services CONsidered JOINTly. In reality, the adjective conjoint derives from the verb to conjoin, meaning joined together.” (internal quotation marks omitted)). 45 Paul E. Green et al., Thirty Years of Conjoint Analysis: Reflections and Prospects, 31 INTERFACES S56, S57 (2001) (“Conjoint analysis is, by far, the most used marketing research method for analyzing consumer trade-offs.”); see also ANDERS GUSTAFSSON ET AL., CONJOINT MEASUREMENT – METHODS AND APPLICATIONS 3 (Anders Gustafsson et al. eds., 4th ed. 2007) (“Based on a 2004 Sawtooth Software customer survey, the leading company in Conjoint Software, between 5,000 and 8,000 conjoint analysis projects were conducted by Sawtooth Software users during 2003.”). 46 There may be a situation where no alternative exists or the product is a necessity. Consumer behaviors, such as demand elasticity, become markedly different under such circumstances.
- 9. 9 form of actualpurchases, e.g., which product consumers bought out of available alternatives and/or how much they paid for it. This is a type of information called revealed preferences or revealed data – they are revealed in the market. Marketing professionals use these data to answer why consumers picked one over the rest. When available, revealed preferences provide the most reliable data to study consumer behaviors because they reflect actual decisions consumers made, as compared to those not yet been realized in the market.47 However, benefits and availability of the revealed data are not without limit. There are practical difficulties in gathering and using market data, including privacy issues.48 Controlled group analysis for comparison may not be available because market data are typically obtained under uncontrolled circumstances. Most notably, by definition, revealed data do not exist for a product that has not been introduced to the market yet. As a survey-based tool, conjoint analysis relies on stated preferences or stated data.49 Rather than observing the market itself ex post, conjoint analysis explores surveys to conduct controlled experiments with the sampled participants sampled from the target population.50 Respondents participating in the survey usually agree to provide their demographic and product-related information and make them available for the survey analysis. The surveyor constructs a hypothetical market and simulates consumer preferences. There, the surveyor designs and controls experimental parameters. Revealed preferences and stated preferences supplement, not conflict with, each other. Revealed preferences provide practical information to the surveyors to start with. The analyst may compare the stated data with the revealed data for the verification purposes. 47 Lisa Cameron et al., The Role of Conjoint Surveys in Reasonable Royalty Cases, THE BRATTLE GRP. (Oct. 16, 2013, 6:37 PM ET), http://www.brattle.com/system/publications/pdfs/000/004/948/original/The_Role_Of_Conjoint_Surveys_In_Reasonable_Roy alty_Cases.pdf?1382111156. 48 See FED. TRADE COMM’N, PROTECTING CONSUMER PRIVACY IN AN ERA OF RAPID CHANGE: RECOMMENDATIONS FOR BUSINESSES AND POLICYMAKERS 1 (Mar. 2012), available at https://www.ftc.gov/sites/default/files/documents/reports/federal-trade-commission-report-protecting-consumer-privacy-era- rapid-change-recommendations/120326privacyreport.pdf. The Federal Trade Commission’s privacy framework applies to all commercial entities that collect or use consumer data (both offline and online) that can be reasonably linked to a specific consumer. Id. at 15–22. The Commission suggests the companies collect only the data they need to accomplish a specific business purpose. Id. at 26. The Commission also proposes that the “data brokers,” who collect consumers’ personal information for the purpose of reselling such information to their customers, improve transparency and give consumers control over the company’s data practices. Id. at 68–70. 49 The power and beauty of consumer surveys in marketing field is that the researcher can design and test the market with a new product that has not yet been introduced in the market. Modifying certain features of an existing product in a survey is undoubtedly much cost-less and risk-less than introducing such alternatives in the market and testing market response. Moreover, using hypothetical products in surveys allows the researchers to predict the market and suggest the roadmap for research and development and/or investment plans. 50 See infra Part III for the discussion on the issues regarding conjoint analysis as survey evidence.
- 10. 10 B. Bundle ofAttributes Conjoint analysis conceptualizes products or services as bundles of attributes. Each attribute can have one or more levels. The end product is assumed to be characterized solely by the set of attributes with designated levels embodied in that product. For example, a consumer who considers buying a laptop may consider following four attributes: brand, display size, price, and storage type, to name but a few. Each attribute can have one or more levels, and the levels can be either quantitative or qualitative. A brand may have four qualitative levels, e.g., Apple, Dell, Lenovo, and Toshiba. Similarly, a storage type may consist of two qualitative levels, e.g., Hard Disk Drive (HDD) and Solid State Drive (SSD). Three quantitative levels may comprise an attribute display size, e.g., less than 13-inch, 14-to-16 inch, and 17-inch or larger. A price range may be divided into three levels, e.g., less than $1,000, $1,000 to $1,200, and over $1,200. A combination of attributes and corresponding levels, which referred to as a profile, characterizes each laptop. One laptop may be an Apple laptop with 13-inch display and SSD storage, sold at $1,299 (“Profile 00”). Another laptop may be a Lenovo laptop with 14-inch display and HDD storage, sold at $1,099 (“Profile 01”). In theory, this example can generate up to 72 ( 2334 ××× ) profiles. Conjoint analysis assumes that each consumer has a set of weights (or values) in units of utility associated with each level of each attribute. They are referred to as partworth.51 Each partworth contributes to the total value (or utility) of a product.52 Consumers are assumed to behave rationally in a sense that they would choose the product with the maximum utility.53 Put differently, what an individual selects is deemed to have the maximum utility to that individual among the available alternatives. Thus, conjoint analysis is “consistent with economic and consumer-behavior theories of approximate utility maximization.”54 Using the same laptop example, if a consumer was asked to choose one model from the 72 alternative laptops, and if Profile 00 was chosen, it is assumed that her choice has revealed or stated that the Profile 00 gives her the maximum utility among 72 available alternatives. Comparing just two 51 Partworth and utility are used interchangeably. Strictly speaking, however, partworths are measured in units of utility. For example, the partworth of the HDD is ten (units of utility) and that of the SDD is fifteen (units of utility). 52 The way partworth contributes to the total utility differs depending on how one models the utility function. See infra Part III. 53 MOSHE BEN-AKIVA & STEVEN R. LERMAN, DISCRETE CHOICE ANALYSIS: THEORY AND APPLICATION TO TRAVEL DEMAND 38 (Marvin L. Manheim ed., 1985). 54 Expert Report of John R. Hauser at 11, Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK (N.D. Cal. July 26, 2012), ECF No. 1363–1 [hereinafter Hauser Report].
- 11. 11 profiles (00 and01), the consumer perceives a higher (or at least an equal) utility from having a specific brand name and a SSD (that the Profile 00 provides) over receiving $200 discount and an extra 1-inch display size (that the Profile 01 adds). A caveat is that her preference should be interpreted to reflect the closed universe of choice sets – she compared only 72 profiles as represented by only four attributes.55 C. Utility Function Most critically, conjoint analysis assumes that utility function can model consumers’ decision- making and that the consumers behave as though having the utility function.56 This is especially so because utility function is closely interrelated with the design of the conjoint survey and the choice of estimation methods, which will be applied in estimating the partworth of the level of the attribute at issue.57 Figure 1 illustrates three forms of the utility function: (1) vector model; (2) ideal point model; and (3) partworth function model. In practice, partworth utility function is adopted most widely by the researchers to model consumer preferences.58 This model represents the attribute utilities by a piecewise linear curve. It is particularly useful when dealing with qualitative attributes, such as brands or discrete functionalities because the values of each level of the qualitative attributes do not vary linearly. Conjoint analysis is a generic term referring to a collection of methods. More than one form of the utility function can model consumer behaviors. There are many ways to phrase questions and to estimate parameters of the modelled utility function by applying different statistical approaches. While many models share basic properties such as additivity and linearity, such are not universally required.59 Furthermore, no single model can describe or fit to every situation equally. A model that works well in a certain set of data may not do so for another set of data. 55 Thus, although consumers typically associate a smaller display size with an enhanced portability and/or an increased battery life, the survey respondent is instructed not to consider such associated effects, however closely interrelated they are in real life. The conjoint analysis, and the survey in general, controls the risk of secondary implication by expressly instructing the respondents to assume that all conditions other than the stated features are identical. See infra Part III.E.i. Still, it is difficult to prove that the instructions eliminate a possibility that the participants might unconsciously have associated the stated-features with those features unstated, but are closely interrelated. 56 The form of which the utility function takes is also the most difficult assumption to make. BEN-AKIVA & LERMAN, supra note 53, at 45. 57 See JOHN R. HAUSER & VITHALA R. RAO, CONJOINT ANALYSIS, RELATED MODELING, AND APPLICATIONS 1, 4–16 (2002), available at http://www.mit.edu/~hauser/Papers/GreenTributeConjoint092302.pdf . 58 Green et al., supra note 45, at S59–S61; Paul E. Green & V. Srinivasan, Conjoint Analysis in Consumer Research: Issues and Outlook, 5 J. CONSUMER RES. 103, 105–06 (1978). 59 BEN-AKIVA & LERMAN, supra note 53, at 40–41.
- 12. 12 Figure 1. Threetypes of the utility function.60 Before jumping to Part II, it is worth mentioning briefly why marketing professionals prefer a seemingly complex conjoint survey to a simple, direct questioning to isolate and assess the value of a particular feature from a multi-component system. D. Advantage & Validity Academics recognize the unreliability of direct questioning to price a subject matter. One study in particular shows that what a respondent says how she would value differs from how she actually reacts. A classic example is a survey conducted on MBA students regarding their career choice.61 When asked directly prior to making decisions, they ranked salary as the sixth most important factor in their career choice.62 However, salary was the most important factor influencing their choice as the conjoint analysis analyzed after they actually had accepted a position.63 This study has made researchers doubt the reliability of the data obtained from direct questioning. Academics also recognize that focus bias (or hypothetical bias, which is similar to a leading question),64 upward bias for socially sensitive 60 Green et al., supra note 45, at S60 (originally from Green & Srinivasan, supra note 58, at 106). 61 Expert Report of Dr. V. Srinivasan (Blue-ray Players) at 7, TV Interactive Data Corp. v. Sony Corp., No. 4:10–cv–00475– PJH (N.D. Cal. Jan. 21, 2013), ECF No. 580–1 [hereinafter Srinivasan Report] (citing David B. Montgomery, Conjoint Calibration of the Customer/Competitor Interface in Industrial Markets, in INDUSTRIAL MARKETING: A GERMAN-AMERICAN PERSPECTIVE 297–319 (Klaus Backhaus & David T. Wilson eds., 1985)). 62 Id. 63 Id. 64 Sentius Int’l, LLC v. Microsoft Corp., No. 5:13–cv–00825–PSG, 2015 U.S. Dist. LEXIS 8782, at *19–20 (N.D. Cal. Jan. 23, 2015). In Sentius, the survey expert recognized that the respondents might have overestimated their willingness-to-pay for a particular feature of a product when answering direct open-ended questions. Id. at * 20. Therefore, the expert used a calibration factor of 1.46 to adjust the respondents’ overestimation. Id.
- 13. 13 issues,65 and/or ambiguity66 may taintthe reliability of the responses obtained from direct (and open- ended) questioning. Conjoint analysis handles these concerns by asking respondents to compare several alternatives and evaluate the alternatives with respect to each other. The bundles-of-attributes description works in two ways. On one end, respondent’s choice does not expose her preference on a particular feature. The choice reveals (before the data is processed and analyzed) her preference only on the set of features, i.e., the product. On the other end, the choice set does not lead respondent’s focus to a specific feature. Thus, the question does not unduly influence her choice. Providing comparable alternatives also makes respondents’ decision-making process more realistic because it resembles our day-to-day decision-making process. The bottom line is that pricing an isolated feature from a multi-component/function system is not the way ordinary consumers make decisions in real life. Conjoint analysis creates a hypothetical market place. If carefully designed and executed, it can approximate the real world realistically. The benefits conjoint analysis has provided to marketing researchers and professionals are much more than a mere theoretical plausibility. The validity of the major assumptions underlying the theory of conjoint analysis has been empirically tested and updated during the last several decades.67 Although not a perfect tool, conjoint analysis has been proven to work well in practice in a wide range of industries. Academics have published hundreds of peer-reviewed papers on the theory and practice of conjoint analysis.68 Product developers and marketers have applied conjoint analysis in almost every commercial area where the consumption of goods and services occurs including healthcare and 65 Upward bias might be created because “respondents do not wish to come across as ‘cheap’” when direct questions were asked. Srinivasan Report, supra note 61, at 7. 66 Paul E. Green & V. Srinivasan, Conjoint Analysis in Marketing: New Developments with Implications for Research and Practice, 54 J. MARKETING 3, 9 (1990) (“‘How important is attribute X?’ is highly ambiguous because the respondent may answer on the basis of his or her own range of experience over existing products rather than on the experimentally defined range of the attribute levels.”). 67 An early reliability study on conjoint analysis compared its different methodological variants. David Reibstein et al., Conjoint Analysis Reliability: Empirical Findings, 7 MARKETING SCI. 271 (1988). Authors reached the conclusion that “the conjoint method under a variety of methods of data collection and across a number of product categories appears to be reliable in an absolute sense.” Id. at 284. More recently, a group of academics developed a new survey method called “polyhedral method” for the choice-based conjoint survey. (Part II.C. explains choice-based conjoint analysis in detail.) This method was successfully applied to support “the design of new executive education programs for a business school at a major university.” Oliver Toubia et al., Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis, 41 J. MARKETING RES. 116, 126–29 (2004). 68 HAUSER & RAO, supra note 57, at 2 (noting in 2002 that Professor Paul Green, a leading scholar in marketing research, himself has contributed almost 100 articles and books on conjoint analysis).
- 14. 14 education.69 Conjoint analysis hasbeen conducted in the design or development of: AT&T’s first cellular telephone; IBM’s workstation; FedEx’s service; MasterCard features; Monsanto’s herbicide packaging; Polaroid’s instant camera design; Marriott’s time-share units; and Ritz Carton’s hotel décor and services.70 And the list goes on. Government also has applied conjoint analysis: it was used successfully in New Jersey’s and New York’s EZ-Pass toll collection project, and in the design of U.S. Navy’s benefit packages for reenlistment.71 II. TWO TYPES OF CONJOINT ANALYSIS This Part explains two types of the conjoint analysis based on a hypothetical situation. First, a traditional full-profile rating (or scoring) method will be described. While this method has limitations, it remains the most common form of conjoint analysis.72 The obvious weakness is that the respondent’s burden grows dramatically as the number of attributes and levels increases. (The laptop example in Part I.B, even with its rather simple configurations, could generate 72 profiles.) Next, choice-based (or discrete choice) conjoint analysis will be discussed. This form of the conjoint analysis is more realistic and natural for the consumer’s decision-making behavior. Consumers consider a bundle of features and make trade-offs, but they usually do not rate or rank a series of products prior to the purchase. In addition, discrete choice analysis offers powerful benefits, including the ability to do a better job of modeling interactions between attributes and the flexibility to incorporate alternative-specific attributes.73 The downside is that the choice-based method generally requires more data to estimate the model parameters. Because the analyst needs to infer the partworth based on the information obtained from the responses of choice-based questionnaires, each response provides substantially limited information. 69 See GUSTAFSSON ET AL., supra note 45, at 3 (“Based on a 2004 Sawtooth Software customer survey, the leading company in Conjoint Software, between 5,000 and 8,000 conjoint analysis projects were conducted by Sawtooth Software users during 2003.”); John F. P. Bridges et al., Conjoint Analysis Applications in Health – a Checklist: A Report of the ISPOR Good Research Practices for Conjoint Analysis Task Force, 14 VALUE IN HEALTH 403 (2011); Green et al., supra note 45, at S67; Toubia et al., supra note 67, at 126–29. 70 See e.g., Green et al., supra note 45, at S67. 71 Id. at S68. 72 HAUSER & RAO, supra note 57, at 8. 73 OMRE, supra note 44, at 33.
- 15. 15 A. Hypo74 A carmanufacturer, MANU, is planning to launch a new model that uses light-emitting diodes (LEDs) on its headlight. Before investing a chunk of money on R&D and implementation, MANU wants to know how consumers would value its new model with LED headlights. It determines that only three features would influence consumers’ decision-making. The three attributes MANU chooses are: (1) whether the headlights are LEDs; (2) fuel efficiency as measured by mile-per-gallon (mpg); and (3) price. It decides to test for two levels of the fuel efficiency: 30 mpg and 35 mpg. Price levels are set at $30,000, $32,000, and $34,000, as MANU’s new model targets in that price range. Table 1 summarizes three attributes and each level of the corresponding attributes. The first column indicates that, for convenience, each level of an attribute will be coded with numbers 0, 1, or 2. Accordingly, each mode/profile can be described by a set of codes. For example, a model that does not have LEDs (“0”), with fuel efficiency of 35 mpg (“1”), and with a price tag of $32,000 (“1”) corresponds to (0, 1, 1). Similarly, a model with LEDs and 30 mpg, and sold at $34,000 is represented by (1, 0, 2). Table 1. Attributes and levels for MANU cars. Attributes/Levels LEDs Fuel Efficiency [mpg] Price [$] 0 No 30 30,000 1 Yes 35 32,000 2 34,000 B. Full-Profile Rating Model The combination of the attributes and levels can generate up to 12 ( 322 ×× ) profiles. In a full- profile survey, respondents are shown a set of cards that fully describes the profile. Here, each card describes: whether the headlights are made of LEDs; whether the fuel efficiency is 30 mpg or 35 mpg; and the price of the car as one of the three amounts – $30,000, $32,000, or $34,000. The respondent in 74 The same fact pattern can be used in the context of a patent infringement suit, where the patentee attempts to evaluate the value of the LED headlights. For example, an inventor INVE has a patent that claims a method of manufacturing LEDs. MANU sells a model that INVE thinks incorporates the LEDs manufactured by the method claimed in his patent. While MANU offers sub-models with a range of prices and other features, INVE cannot be certain about the value of the LED headlights as all MANU models use the same LED headlights INVE thinks infringing. INVE conducts a conjoint survey assuming that only the same three attributes characterize MANU cars and affect consumers’ purchasing decision.
- 16. 16 this survey isgiven 12 cards, each representing a profile, and asked to rate each profile in between one and twenty with a higher-rated profile indicating a more attractive option. While the respondent saw a full description of each profile, the data gathered is expressed as codes for the analysis. Table 2 summarizes a list of profiles and the scores the respondent rated. Stimuli column simply identifies the profile. Table 2. Full-profile rating result.75 Stimuli Profile Score/Rate 1 0, 0, 0 9 2 0, 0, 1 7 3 0, 0, 2 4 4 0, 1, 0 16 5 0, 1, 1 13 6 0, 1, 2 10 7 1, 0, 0 12 8 1, 0, 1 8 9 1, 0, 2 6 10 1, 1, 0 18 11 1, 1, 1 15 12 1, 1, 2 11 The respondent assigned 18 points to (1, 1, 0) and 11 points to (1, 1, 2). Points represent the relative partworths of each profile. So, relatively speaking, to this respondent, profile (1, 1, 0) provides her with 7 more utility than profile (1, 1, 2) does. The comparison between these two profiles makes sense, as all conditions being equal, a more expensive car is less attractive than the cheaper one. Comparing profiles (1, 1, 0) and (1, 0, 0), the respondent favors the former with 6 extra points. Again, the result makes sense because, assuming other two attributes remain identical, improvement on fuel efficiency would add some value. 75 The data set here is generated assuming a certain set of the partworth for each attribute only to illustrate how this method works. Because the score of each profile is calculated systematically with the predetermined partworth in mind, the parameter estimates obtained by the least squares regression fit very well, and, thus, are reliable, as the statistics in Table 3 show.
- 17. 17 Equation (1) takesa simple linear, additive form. It describes the relationship between the total utility of a hypothetical vehicle model and three attributes of interest. The utility of each profile ( ofilenU Pr, ) is the sum of the partworth of all attributes constructing the profile. Unknown parameters (or dummy variables) { Attributesn,β } are weights of independent (or explanatory) variables { AttributesX }. The goal of the conjoint analysis is to find a set of { Attributesn,β } using given data set { ofilenU Pr, , AttributesX } with the assumption that the data set follows equation (1). iceicenEfficiencyEfficiencynLEDLEDnjn XXXU PrPr,,,, βββ ++= . (1) Subscript n denotes that { ofilenU Pr, } and { Attributesn,β } are individual’s utility and weights for the attributes. Each respondent may assign different weights on each attribute. That means each respondent would rate differently, creating a new table of data set. Still, equation (1) can be expanded to represent the market’s view on the partworth of each attribute in two ways. Assume that there are 100=N respondents. One, { Attributesn,β } for each individual can be first estimated. Then, 100=N set of the estimates can be averaged in a certain way. Two, { ofilenU Pr, , AttributesX } for all 100=N respondents can be first aggregated and averaged. Then, { AttributesN ,β } can be estimated, which represent the market’s (here that of the sample size of 100=N ) partworth. Ordinary linear squares (OLS) regression is a natural and efficient means with which to estimate { Attributesn,β }, in this model.76 Regression analysis is a statistical process for estimating relationship among variables. 77 First, the linear regression model takes the form similar to equation (1): jiceiceEfficiencyEfficiencyLEDLEDj XXXY εββββ ++++= PrPr0 . (2)78 jY is a dependent variable. Total utility of the profile is represented by the points assigned to { jY }. 0β is an intercept, i.e., a constant value that adjusts the level of the points. jε is the random error term that 76 HAUSER & RAO, supra note 57, at 12; see Paul E. Green & Yoram Wind, New way to measure consumers’ judgments, 53 HARV. BUS. REV. 107 (1975). 77 WIKIPEDIA, http://en.wikipedia.org/wiki/Regression_analysis (last visited Apr. 15, 2015). 78 Subscript n is taken out to simplify the equation.
- 18. 18 accounts for thecollective unobservable influence of any omitted explanatory variables. Typically, although not necessary for least squares to be used, this term is assumed to follow normal distribution. In OLS regression analysis, { Estimatesβ } are estimated by fitting the equation to the data so that the sum of the squared deviation of the data from the line are minimized (thus, least squares).79 The best set of { Estimatesβ } with the minimum deviation from the model approximates the data set most accurately. Each response generates a data point. For example, for the 9th stimuli with a profile (1, 0, 2) and the 11th stimuli with a profile (1, 1, 1), equation (2) becomes 9Pr0 26 εβββ +++= iceLED (3) 11Pr015 εββββ ++++= iceEfficiencyLED (4) respectively. An individual thus has generated total 12 data points (or equations) that can be used to estimate four unknown parameters { 0β , LEDβ , Efficiencyβ , icePrβ }. A simple regression analysis was applied to the data set in Table 2 using a Regression Data Analysis tool in Microsoft Excel. Table 3 summarizes the result. The Coefficients column corresponds to the estimates of four { Estimatesβ }. Inserting the estimates, equation (2) becomes iceEfficiencyLED XXXY Pr15.397.563.12.10ˆ −++= . (5) Hat on Y ⌢ is used to indicate that Y ⌢ is an estimate, not a data point. The total utility of a car can be predicted by using equation (5) (with a limitation80 ). Because a higher price diminishes the total utility, the weight of the price term has a negative sign. In addition to parameter estimates (under Coefficients column), OLS regression results provide an estimate of the reliability of the parameter estimates and a measure of the overall goodness of fit of the regression model.81 The standard error of each estimate (listed in Standard Error column) shows the magnitude of variations in the data set. The greater the variation in the data, the larger the standard error 79 SHARI SEIDMAN DIAMOND, Reference Guide on Survey Research, in REFERENCE MANUAL ON SCIENTIFIC EVIDENCE 333– 51 (The Federal Judicial Center ed., 3d ed. 2011), available at http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$file/SciMan3D01.pdf. 80 For example, extrapolation typically generates an unreliable prediction. 81 DIAMOND, supra note 79, at 340.
- 19. 19 and the lessreliable the regression results become.82 It should be noted that the absolute value of standard errors might mislead the interpretation. For example, the standard error value of 1 is much greater than that of 0.01. However, if the coefficient estimated in the first case is in the order of 100 and the latter is in the range of 0.001, the reliability of the estimate is much higher in the first case. Table 3. Regression Result Regression Statistics Coefficients Standard Error t-Stat R-squared ( 2 R ) 0.99393 0β 10.20000 0.31848 32.02728 Standard Error 0.41115 LEDβ 1.63333 0.25271 6.46327 Efficiencyβ 5.96667 0.25271 23.61071 icePrβ -3.15000 0.15924 -19.78155 T-statistics might be used to cure the potential misinterpretation. The t-statistic is defined as the ratio of the parameter estimate to its standard error. Thus, the t-statistics for the previous example are 1 100100 = and 01.0 001.01.0 = , respectively. T-statistics can be interpreted in terms of confidence intervals. In general, a confidence interval around any parameter estimate can be constructed such that a 95% confident interval is a range of values that one is 95% confident that the true value of parameter is within that confident interval.83 Accordingly, when the t-statistic is less than 1.96 in magnitude, the 95% confidence interval around the parameter must include zero, and the estimate is said to be not statistically significant.84 Conversely, if the t-statistic is greater than 1.96 in absolute value, it can be concluded that the true value of the estimate is unlikely to be zero, and the estimate is statistically significant.85 The t-Stat column in Table 3 demonstrates that the estimated { Estimatesβ } in this hypo are statistically significant. The absolute values are significantly greater than 1.96 for each and every 82 Id. at 341. 83 Id. at 342. Unlike Gaussian distribution, the 95% confidence interval of the t-distribution is not fixed to 1.96. Rather, it varies by the degree of freedom of the t-distribution, which is equal to the difference between the number of samples and the number of parameters to be estimated. WIKIPEDIA, http://en.wikipedia.org/wiki/Student%27s_t- distribution#Table_of_selected_values (last visited Apr. 17, 2015). While validity of the Gaussian distribution is predicated on a large sample size, the t-statistic applies to any sample size. See DIAMOND, supra note 79, at 343 n.82. The t-distribution approximates the normal distribution as the sample gets large. Id. 84 Id. at 342–43. 85 Id. at 343.
- 20. 20 coefficient. For example,the 95% confidence interval for LEDβ is (1.03577, 2.23090) and icePrβ is (- 3.52654, -2.77346). Both intervals do not include zero. In addition to the estimate of the reliability of the parameter estimates, the regression result also provides a measure of the overall goodness of fit, i.e., how well the regression equation fits the data. R- squared (or 2 R ) is a statistic that “measures the percentage of variation in the dependent variable that is accounted for by all the explanatory variables.”86 Its value ranges from 0 (the explanatory variables explain none of the variation of the dependent variable) to 1 (the explanatory variables explain all of the variation of the dependent variable). While there is a no clear-cut level that makes the model satisfactory, a high 2 R is favored.87 In this example, 2 R is very high (0.99393), which means that the three attributes explain more than 99% of the variation in this particular respondent’s rating (and total utility). From Table 3 and equation (5), this respondent’s willingness-to-pay for the LED headlights in MANU’s car can be derived. 15.3Pr −=iceβ means that the increase in price from $30,000 to $32,000 (or from $32,000 to $34,000) decreases the relative worth of the car at the rate of 3.15 units. That is, $2,000 corresponds to the partworth of 3.15 units, and one unit of partworth represents util util 15.3 000,2$]/[$635 = . As the respondent’s utility with the LED headlights increase by 1.63 units, this adds 035,1$]/[$635][63.1 =× utilutil worth of value to the respondent. Thus, this respondent’s willingness-to-pay for the LED headlights is $1,035 (when the price of the car ranges in between $30,000 and $34,000). A different approach would lead to the same result. Here, the task is to estimate the price premium the respondent would be willing-to-pay for the LED headlights. Assume that the base model has 35 mpg with a price tag at $32,000. The one model that does not have the LED headlight feature is described by using equation (5) as 02.1315.397.52.10 =−+=NoY (6) 86 Id. at 345. 87 Id.
- 21. 21 because 0=LEDX ,1=EfficiencyX , and 1Pr =iceX for this alternative. Conversely, the alternative with the LED headlight features is iceiceYes XXY PrPr 15.38.1715.397.563.12.10 −=−++= (7) as now 1=LEDX and 1=EfficiencyX . Consumer’s willingness-to-pay for the particular feature is drawn by solving equations (6) and (7) when LEDNo YY = . At that price, the consumer does not favor one alternative over the other. Solving the equation results in 517.1Pr =iceX . Assuming that the utility for the price varies linearly in between $32,000 ( 1Pr =iceX ) and $34,000 ( 2Pr =iceX ), 517.1Pr =iceX corresponds to $33,035. In short, the respondent’s willingness-to-pay a price premium for the LED headlight is, again, $1,035 (over $32,000). Assuming that the data in Table 2 represents the market data, the same result is interpreted to represent market’s willingness-to-pay for this particular feature of the product. C. Choice-Based or Discrete Choice Model The choice-based model differs from the previous model because the dependent variable in choice-based model is discrete. For example, when the question is the choice between only two alternatives, the answer is either yes (prefer or purchase) or no. Similarly, the output for the choice task where more than two alternatives are available is still binary – yes or no for each alternative. The utility function in the choice-based conjoint analysis can be modeled by using a similar approach as in the rating-based approach. (See equation (12) below.) However, as opposed to a traditional form consisting of unknown but certain parameters, the choice-based model assumes that the respondent’s utility is represented by random variables having a probabilistic distribution.88 The model, thus, is called a random utility model (RUM).89 88 JORDAN J. LOUVIERE ET AL., STATED CHOICE METHODS: ANALYSIS AND APPLICATION 38 (Louviere et al. eds., 2000). 89 HAUSER & RAO, supra note 57, at 10.
- 22. 22 The probability ofany alternative i being selected by a person n from a choice set nC depends on the probability that the utility of the alternative i ( niU , ) is higher than any other available alternatives from the set.90 It is expressed that: ),()|( ,, nnjnin CijUUPCiP ∈≠∀>= . (8) Equation (8) reads: when the alternatives j is not identical to i, and both are from the choice set ( nC ), the probability of a person n choosing the alternative i is equal to the probability that the utility of the profile i is higher than any other available alternatives in the set nC . The distinctive characteristic of RUM is that it models the utility function as a combination of two parts: the systematic (or representative) component ( niV , ) and the random (or error) component ( ni,ε ) such that ninini VU ,,, ε+= . (9) The random component makes the utility a random variable. Inserting equation (9) into equation (8) and rearranging it yields equation (10). Equation (11) is a simplified form of equation (10). ),()|( ,,,, nnjnininjn CijVVPCiP ∈≠∀−<−= εε (10) )()( ,,,, njnininjn VVPiP −<−= εε (11) Note that the probability of choosing alternative i is determined as a function of the relative values of random components ( ninj ,, εε − ) and systematic components ( njni VV ,, − ). The systematic component represents the levels and attributes of the alternatives with the assumption that they are linear and additive: iceicenEfficiencyEfficiencynLEDLEDnni XXXV PrPr,,,, βββ ++= . (12) 90 LOUVIERE ET AL., supra note 88, at 59.
- 23. 23 Now, the discretechoice model relies on what distributional forms or assumptions the error terms take. For example, if the error terms are multivariate normal, the probit model is obtained.91 The prevalent form of utility function in discrete choice model is called multinomial logit.92 Multinomial logit model presumes that ni,ε and nj,ε are independent and identically distributed.93 Under the assumption, ε εεε − − =−−=< e j eP )expexp()( . (13)94 Incorporating equations (11) and (12) to equation (13), and arranging it to a simple form, equation (11) becomes ∑∈ = n nj ni Cj V V n e e iP , , )( . (14)95 The way the survey question is presented in the choice-based conjoint survey differs from that of the full-profile rating survey described in the preceding section. Here, a respondent n is given a set of options (choice set, nC ) and asked to pick one profile. For example, assume that the choice set consists of profiles 5, 6, 7, and 12, and further assume that the respondent n selects profile 5, in consistent with the rating results shown in Table 2. This decision generates three inequality constraints (or data points). By choosing profile 5 when three other alternatives were available, the respondent n stated her preference of 5 over 6, 7, and 12.96 91 HAUSER & RAO, supra note 57, at 10; see BEN-AKIVA & LERMAN, supra note 53, at 69–70 (deriving binary probit model). 92 See Green et al., supra note 45, at S64 (“In choice-based conjoint analysis, analysts typically employ multinomial logit models, although occasionally they use probit-based models.”). 93 LOUVIERE ET AL., supra note 88, at 44–47. This is often known as the Independence-from-Irrelevant Alternatives (IIA) axiom. It states that “the ratio of the probabilities of choosing one alternative over another (given that both alternatives have a non-zero probability of choice) is unaffected by the presence or absence of any additional alternatives in the choice set.” Id. at 44. “Satisfaction of the IIA condition, however, should not be of general concern because the independence assumption is a priori neither desirable nor undesirable, but should be accepted or rejected on empirical grounds depending on the circumstances.” Id. at 45. 94 Id. 95 BEN-AKIVA & LERMAN, supra note 53, at 71; LOUVIERE ET AL., supra note 88, at 47. 96 Three inequality constraints are: nn UU ,6,5 > ; nn UU ,7,5 > ; and nn UU ,12,5 > .
- 24. 24 The probability ofselecting profile 5 from this choice set is obtained by combining equation (14) and equation (12): nnnn n VVVV V eeee e P ,12,7,65 ,5 )5( +++ = icenEfficiencynLEDnLEDnicenEfficiencynicenEfficiencyn icenEfficiencyn eeee e Pr,,,,Pr,,Pr,, Pr,, 22 ββββββββ ββ ++++ + +++ = . (15) In the choice-based conjoint analysis, each respondent is typically provided with about fifteen choice sets, each consisting of four profiles.97 Thus, one respondent supplies 153× data points. Typically, many more data points are required to estimate model parameters reliably in discrete choice model as both the choice and the utility take the probabilistic forms. While a set of data for an individual respondent can be estimated first and then aggregated with that of the others in the rating study, the choice-based model prefers aggregating the data prior to estimating model parameters. The natural result is that under the choice-based model, the market’s utility and willingness-to-pay, rather than those of individuals, are obtained. As the utility is considered as a random variable, and each choice decision is expressed as a probability, the likelihood of the entire set of data (e.g., 153× data points for the respondent n) is the product of the likelihoods of the individual observation as expressed in equation (16).98 Therefore, the estimation method seeks to find the set of model parameters { nβ } that maximizes the likelihood of generating the entire set of data. The estimation method is referred to as the maximum likelihood estimation (MLE). ∏∏= = = N n I i f n ni ipL 1 1 * , )( . (16) Equation (16) indicates that the likelihood of each observation is multiplied across the sample: that is, n ranges from 1 to N (the total number of the survey respondents). All answers from each respondent are also multiplied: that is, i ranges from 1 to I (the total set of stimuli, fifteen in the hypo). 97 See Hauser Report, supra note 54, at 35–40 (sixteen choice tasks with four alternatives); Srinivasan Report, supra note 61, at 14–15 (four options presented in each of 15 choice sets). 98 BEN-AKIVA & LERMAN, supra note 53, at 80–81, 118–20; LOUVIERE ET AL., supra note 88, at 47–50.
- 25. 25 nif , isa dummy variable such that 1, =nif if respondent n chose the alternative i and 0, =nif if otherwise. Note that in equation (16), when the alternative is not chosen, that does not affect the likelihood function because 10, == pp nif . From equation (16), log likelihood function L can be written as ∑ ∑= = == N n I i nni ipfLL 1 1 , * )(lnln . (17) As the set of { nβ } that maximizes L also maximizes * L , finding the set of solutions that maximize equation (17) best estimates the discrete choice problem.99 Substituting )(iPn with equation (14) gives, ∑ ∑ ∑∑ ∑ ∑ = = ∈ = = ∈ −== N n I i Cj V nini N n I i Cj V V ni n nj n nj ni eVf e e fL 1 1 ,,1 1 , )ln(ln , , , . (18) Note also that L is a function of { niV , }, which in turn are functions of the utility parameters, { nβ }. Furthermore, it is known that L is a convex function of { nβ }.100 Thus, equation (18) can be maximized with respect to { nβ } by using some non-linear maximization algorithms.101 A good property of convex optimization problem is that we can always achieve globally optimal value no matter what initial conditions are assumed, e.g., 0}{ =nβ . Such algorithms are usually iterative.102 Typically, initial values of the { nβ } are guessed and used in equation (12) to calculate { niV , }. The initial set of { niV , } are then used in equation (14) to calculate { )(iPn }. These values are used in equation (17) to calculate a starting value of L . The procedures are repeated by changing { nβ } systematically until the increase in L reaches a predetermined level of tolerance. 99 LOUVIERE ET AL., supra note 88, at 50, 66–71. 100 Id. at 66–67. 101 Id. at 50–51, 70–71. 102 See infra Part II.D.
- 26. 26 The choice-based modelsignificantly differs from the rating-based model in two aspects. First, from the modeling perspective, the dependent variable, which represents consumer preferences, is discrete, not continuous. Due to this discrete nature, the choice and the utility are defined in terms of probability distributions. Second, the MLE method is applied to estimate model parameters. The choice of estimation method also relates to the discrete and probabilistic nature of the choice problem. As the objective of the parameter estimation is to approximate the entire data set, and the data consists of individual probability distribution, the optimal set of parameter estimates is the one that maximizes the likelihood of reproducing the entire data set as closely as possible. Next section briefly describes more advanced estimation method called Hierarchical Bayes method.103 While this method “transformed the way discrete choice studies were analyzed,” it also applies to rating-based conjoint analysis with incremental benefits.104 D. Hierarchical Bayes (HB) Estimation Method The approaches taken so far are examples of conventional (non-Bayesian) statistical analyses. Under the non-Bayesian approach, the probability distribution of the data is investigated, conditioned on the assumptions embodied in the model (or hypothesis) and its parameters. In Bayesian analysis, the probability distribution of the parameter, given the data, is investigated. Under the Bayes theorem, the probability of a particular hypothesis ( H ) given the data ( D ) is )( )( )|( DP DHP DHP ∩ = . (19) Similarly, because )( )( )|( HP HDP HDP ∩ = (20) and )()( HDPDHP ∩∩ = , rearranging equations (19) and (20) yields equation (21). 103 Experts in two recent patent infringement cases used this method to estimate market’s willingness-to-pay in choice-based conjoint analyses. Hauser Report, supra note 54; Srinivasan Report, supra note 61. 104 OMRE, supra note 44, at 34.
- 27. 27 )( )()|( )|( DP HPHDP DHP × = . (21) Here,the probability of the hypothesis given the data, )|( DHP , is known as its posterior probability.105 This is the probability of the hypothesis that reflects the data upon which the hypothesis is based. The probability of the hypothesis, )(HP , is known as its prior probability. It describes the analyst’s belief about the hypothesis before she saw the data. The probability of the data given the hypothesis, )|( HDP , is known as the likelihood of the data. It is the probability of seeing that particular collection of data, conditioned on that hypothesis about the data. Thus, equation (21) tells that the posterior probability of the hypothesis is proportional to the product of the likelihood of the data under that hypothesis and the prior probability of that hypothesis. The Bayesian framework provides a formula to update the prior estimate of the probability. The HB method analyzes the model in a hierarchical form.106 That is, the model parameters in one level (or hierarchy) are explained in subsequent levels. Re-writing equation (10) in a simple form, we obtain ),()( ,,,, ijVVPiP njnjnini ≠∀+>+= εε . (22) Re-writing equation (12) in a generalized form with more than three attributes, we obtain. nnini xV β,, ʹ= . (23) nix , ʹ , attributes of the alternative i for the n respondent is expressed in a vector form. The HB method further assumes that individual partworth, }{ nβ , has the multivariate normal distribution, i.e., ),(~ ∑β ββ Normaln (24) 105 The ACA/HB Module for Hierarchical Bayes Estimation v3, SAWTOOTH SOFTWARE INC., 6 (July 2006) [hereinafter ACA/HB Module], https://www.sawtoothsoftware.com/download/techpap/acahbtec.pdf. 106 Greg M. Allenby & Peter E. Rossi, Perspectives Based on 10 Years of HB in Marketing Research, SAWTOOTH SOFTWARE INC., 3 (2003), http://www.sawtoothsoftware.com/download/techpap/allenby.pdf.
- 28. 28 where β isthe mean of the distribution of individuals’ partworth and ∑β denotes (a matrix of) covariances of the distribution of the partworths across individuals. Equation (22) describes the lowest level of the hierarchy. The same interpretation given in equations (8)-(11) applies to equation (24). At the highest level, equation (24) allows for heterogeneity among the respondents (i.e., variance within the sample). The individual partworth estimate, }{ nβ , are linked by a common distribution, { ∑β β , }, which represents the sample. (Note that { ∑β β , } do not contain individual-specific subscript, such as n.) { ∑β β , } are known as hyper-parameters of the model.107 Thus, in theory, estimates of individual level parameters give data, )|( DP nβ , can be obtained by first obtaining the joint probability of all model parameters given the data, ),|()|( | ∑∏ × β βββ n n nn PDP . And, )|,},({)|( DPDP nn ∑= β βββ )( ),()],|()|([ )( )()|( | DP PPDP DP PDP n n nn nn ∑∑∏ ×× = × = ββ ββββ ββ . (25)108 Integrating equation (25) results in ∑∫ ∑ −= ββ βββββ dddDPDP knn )|,},({)|( . (26)109 “-k” in k−β denotes “except k.” Equations (25) and (26), thus, provide an operational procedure for estimating a specific individual’s set of partworths, }{ nβ , given all the data in the sample, D , not merely her data, nD . As such, the Bayes theorem provides a method of bridging the analysis across the respondents, which conjoint analysis essentially attempts to achieve. 107 Id. at 4. 108 Id. 109 Id.
- 29. 29 Because usually equation(26) is impossible to be solved analytically, an iterative process is used to estimate the model parameters. The so-called “Monte Carlo Markov Chain” method provides an efficient algorithm to HB problems. Starting with a set of initial values { mβ , mβ , ∑ mβ , mσ }, with 0=m , each iteration consists of following four steps to generate a set of updated values, { 1+mβ , 1+mβ , ∑ +1mβ , 1+mσ }. 1) Using present estimates of mβ , ∑ mβ , and mσ , generate new estimates of 1+mβ . 2) Using present estimates of mβ and ∑ mβ , generate a new estimate of 1+mβ . 3) Using present estimates of mβ and mβ , draw a new estimate of ∑ +1mβ . 4) Using present estimates of mβ , ∑ mβ , and mβ , generate a new estimate of 1+mσ .110 As the four steps show, one set of parameters is re-estimated conditionally in each step, given current values for the other three. Compared to its traditional counterparts, the HB estimation method provides two significant benefits. First, as Bayesian statistical analysis, HB has the advantage of using prior data to update next parameter estimation. Using available data (e.g., utility decreases as price increases, or utility increases as the processor gets faster and/or the storage capacity becomes larger) enhances the reliability of conjoint analysis.111 This aspect of the HB method also makes it easier to update the next question based on respondent’s previous answers, making the survey more efficient.112 Second, its hierarchical form provides a more realistic estimation platform when the heterogeneity (i.e., variance within the sample) of the data is high. The HB approach accounts for the heterogeneity present in the data set and bridges the individuals’ responses across the sample. 110 ACA/HB Module, supra note 105, at 6. 111 Greg M. Allenby et al., Incorporating Prior Knowledge into the Analysis of Conjoint Studies, 32 J. MARKETING RES. 152 (1995); see also Hauser Report, supra note 54, at 41. 112 The underlying mechanism is similar to that of the general computer-based tests (CBTs). The next question (or choice set) is selected from the pool of questions (or redesigned) in response to exam-takers’ (or survey respondents’) previous answers. See generally Oliver Toubia et al., Probabilistic Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis: Theory and Application, 26 MARKETING SCI. 596 (2007). However, adding prior information is not limited to the HB estimation method. Traditional regression methods can also incorporate prior information for data analysis. HAUSER & RAO, supra note 57, at 12.
- 30. 30 So far, underlyingprinciples of two major types of conjoint analysis rest on sound mathematical theorems. The admissibility of the expert testimony based on conjoint analysis largely depends on the minor premise. Expert’s major battleground is on how the survey gathers and supplies the data and the validity of the statistical analysis rendered on the data. Part III identifies eight key areas where the admissibility and credibility of conjoint survey evidence can be challenged. III. CONJOINT ANALYSIS AS SURVEY EVIDENCE Since conjoint analysis is a survey-based research tool, issues pertinent to admissibility and credibility of the survey evidence apply to expert opinions based on conjoint analysis. Over the last half century, the survey method has been proved to be an economical and systematic way to obtain data and draw inferences about a large number of individuals or other units.113 A complete census of the universe can be expensive, time-consuming, and sometimes impossible. With the increasing uses of the surveys by academic researches, businesses, and the government, both federal and state courts have admitted survey evidence on a variety of contexts such as: discrimination in jury panel composition; employment discrimination; class certification; community standards for obscenity; antitrust; mass torts; consumer perception and memory; trademark infringement; and patent infringement cases.114 One federal court in a trademark infringement case treated the absence of a survey as the plaintiff’s failure to establish actual consumer confusion.115 While survey evidence has been attacked as inadmissible on the theory that it is inadmissible hearsay, the contemporary view is that the hearsay objection is unsound.116 The respondents’ answers are either nonhearsay or admissible as exceptions to the hearsay rule as declarations of present state of mind or under the residual exception.117 Under the Federal Rule of Evidence and Daubert, the admissibility of survey result centers on the “validity of the techniques employed rather than [on] relatively fruitless inquires whether hearsay is involved.”118 The key is on the quality of survey. To be admissible, surveys should generally satisfy following foundational requirements: (1) a relevant population (or universe) was properly defined; (2) a 113 DIAMOND, supra note 79, at 367. 114 Id. at 364–67; see, e.g., Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK, 2012 WL 2571332, at *9– 10 (N.D. Cal. June 30, 2012). 115 Morrison Entm’t Grp. v. Nintendo of Am., 56 F. App’x 782, 785 (9th Cir. 2003); DIAMOND, supra note 79, at 372 (“[S]everal courts have drawn negative inference from the absence of a survey, taking the position that failure to undertake a survey may strongly suggest that a properly done survey would not support the plaintiff’s position.”). 116 1 PAUL C. GIANNELLI ET AL., SCIENTIFIC EVIDENCE, § 15.04[b], at 851–52 (5th ed., 2012). 117 Id. 118 FED. R. EVID. 703 advisory committee’s note.
- 31. 31 representative sample ofthat population was selected; (3) the questions were presented in a clear and non-leading manner; (4) interview procedures were sound and unbiased; (5) the data was accurately gathered and reported; (6) the data was analyzed in accordance with accepted statistical principles; and (7) objectivity of the process was assured.119 This Part examines each factor in turn. The third factor will be addressed in two parts. Issues relating to the determination of survey attributes and levels will be examined first. Then, the focus will move to the phrasing and/or presentation of the questionnaires. A. Relevant Population A population or universe is a complete set or all the units of interest to the researcher. The target population must be relevant to the questions of the survey.120 The starting point in the development of a methodologically sound survey is identification of the appropriate population. Thus, courts have considered the selection of proper population “as one of the most important factors in assessing the validity of a survey as well as the weight that it should receive.”121 Leelanau Wine Cellar v. Black & Red is a trademark infringement case between competing wineries in Michigan.122 Leelanau’s primary theory of the case was based on the secondary meaning of the disputed mark and the resulting consumer confusion. Specifically, Leelanau alleged that Black & Red’s “Chateau de Leelanau” mark caused consumers of the defendant’s products to mistakenly believe that the defendant’s products were from the same source as, or were connected with, the plaintiff’s products.123 Leelanau retained an expert to conduct a consumer survey to measure the extent to which consumers who encounter defendant’s Chateau de Leelanau wines believed them to be the same as or related to Leelanau Wine Cellars.124 Leelanau’s expert defined the universe as “Michigan consumers over 21 years of age who had either purchased a bottle of wine in the $ 5 to $ 14 price range in the last 119 Leelanau Wine Cellars, Ltd. v. Black & Red, Inc., 452 F. Supp. 2d 772, 778 (W.D. Mich. 2006) (citations omitted). The court added that, “[b]ecause almost all surveys are subject to some sort of criticism, courts generally hold that flaws in survey methodology go to the evidentiary weight of the survey rather than its admissibility.” Id. Manual for Complex Litigation suggests the same seven factors in assessing the admissibility of a survey. MANUAL FOR COMPLEX LITIGATION (Fourth) § 11.493 (2004). See also DIAMOND, supra note 79, at 367 (“Several critical factors have emerged that have limited the value of some of [the] surveys: problems in defining the relevant target population and identifying an appropriate sampling frame, a response rates that raise questions about the representativeness of the results, and a failure to ask questions that assess opinions on the relevant issue.”). 120 DIAMOND, supra note 79, at 377 (“A survey that provides information about a wholly irrelevant population is itself irrelevant.”). 121 Leelanau Wine Cellars, 452 F. Supp. 2d at 781. 122 Id. at 772. 123 Id. at 779. 124 Id.
- 32. 32 three months orwho expected to purchase a bottle of wine in that price range in the three months following the interview.”125 The court found this universe was flawed because it was significantly overbroad.126 The court first concluded that when the dispute centers on secondary meaning of the mark, the proper universe is the potential purchasers of defendant’s products.127 The court noted that Black & Red’s wines are available only through their local tasting rooms and websites.128 That finding was crucial because, while Leelanau’s universe would certainly include purchasers of the defendant’s wine, only a tiny percentage of the respondents in Leelanau’s universe would probably purchase Black & Red’s wines in connection with actual visits to its tasting rooms or websites.129 In another case, the court was more receptive to survey evidence. In a patent infringement suit between competing smartphone manufacturers, Apple’s damages expert designed and conducted a conjoint survey to determine price premium, if any, Samsung consumers would be willing to pay for the touchscreen features associated with the Apple’s patents at issue.130 One of the arguments for Samsung’s unsuccessful Daubert challenge was that Apple’s expert surveyed improper recent Samsung purchasers, rather than potential Samsung purchasers.131 Rejecting Samsung’s motion to exclude the expert opinion, the court first noted that Samsung failed to explain why recent Samsung purchasers are not the proper universe for Apple’s survey.132 The court found that even if the category of recent Samsung purchasers was underinclusive, they were at least members of the relevant universe of survey participants.133 Concluding that the underinclusive population was still probative, the court stated, “[g]enerally, underinclusiveness of a survey goes to weight, not admissibility.”134 B. Representative Sample A sample is a subset of the population. The sample is drawn from the population for a particular purpose of conducting the survey. Conjoint analysis quantifies sample’s preferences and estimates sample’s partworth and willingness-to-pay. Thus, in addition to the appropriate identification of the 125 Id. at 782. 126 Id. at 782–83. 127 Id. at 782. 128 Id. 129 Id. 130 Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK, 2012 WL 2571332, at *9–10 (N.D. Cal. June 30, 2012). 131 Id. at *9. 132 Id. at *10. 133 Id. 134 Id.; see also Microsoft Corp. v. Motorola, Inc., 905 F. Supp. 2d 1109, 1120 (W.D. Wash. 2012).
- 33. 33 relevant population, itis essential to select a sample that properly represents the relevant characteristics of the population. The ultimate goal of the sample survey is “to provide information on the relevant population from which the sample was drawn,” even though the data are incomplete as not obtained from the population.135 Largely, there are two types of concerns – quantitative and qualitative – about the sample. Quantitatively, the statistician wants a sample large enough to generate reliable statistics. Courts have barred samples in some cases when the sample was too small to yield reliable statistics.136 The rule of thumb is that a sample size exceeding thirty may provide stable statistics.137 More is better in general, but not always so. When the sample is systematically skewed or biased, a larger sample size would aggravate, not reduce, the systematic error (even though the standard error is inversely proportional to the square root of the sample size).138 Qualitatively, the sample must be unbiased.139 Most survey researchers employ probabilistic approaches in sampling.140 Probability sampling is known to maximize both the representativeness of the survey results and the reliability of estimates obtained from the survey.141 Probability sampling methods range from a simple random sampling method to complex multistage sampling schemes.142 In a basic simple random sampling method, every element in the population has equal non-zero probability 135 DIAMOND, supra note 79 at 361. 136 See 1 GIANNELLI ET AL., supra note 116, at § 15.04[b], 858. 137 Id. at 845. 138 Id. n.98. The classic example of the large sample with the systematic bias is the Literary Digest’s 1936 presidential election poll. The Literary Digest was one of the most popular magazines of that era and had a history of accurately predicting the winners of presidential elections since 1916. It sent out 10 million straw ballots asking people who they planned to vote for the 1936 presidential election. The magazine received 2.4 million ballots. Based on the responses, it predicted Alfred Landon would beat Franklin D. Roosevelt 57% to 43%. As it turned out, Roosevelt won, with a whopping margin of 62% to 37%. There were two huge problems with the poll. First, the initial sample (initial 10 million recipients) did not correctly represent the population, i.e., voters. Literary Digest used lists of phone numbers, drivers’ registration, and country club memberships to identify the sample. However, at the time, these luxuries were more often available to the middle- and upper-class voters, which tended to exclude lower-income voters. On the other side, Roosevelt’s campaign centered on reviving the economy at the height of the depression, which appealed to the majority of the lower income people. Second, the sample chosen suffered from voluntary response (or nonresponse) bias, with a huge nonresponse rate of more than 75%. A high nonresponse rate suggests that the voters who supported Roosevelt were less inclined to respond to the survey. Famous Statistical Blunders in History, THE OXFORD MATH CTR., http://www.oxfordmathcenter.com/drupal7/node/251 (last visited Apr. 14, 2015); Case Study I: The 1936 Literary Digest Poll, UNIV. PA. DEP’T OF MATHEMATICS, http://www.math.upenn.edu/~deturck/m170/wk4/lecture/case1.html (last visited Apr. 14, 2015). 139 1 GIANNELLI ET AL., supra note 116, at § 15.04[a], 846–47. 140 See DIAMOND, supra note 79, at 382. 141 Id. at 380 (“Probability sampling offers two important advantages over other types of sampling. First, the sample can provide an unbiased estimate that summarizes the responses of all persons in the population from which the sample was drawn; that is, the expected value of the sample estimate is the population value being estimated. Second, the researcher can calculate a confidence interval that describes explicitly how reliable the sample estimate of the population is likely to be.”). 142 Id.
- 34. 34 of being selectedin the sample.143 In stratified random sampling, the population is first divided into mutually exclusive and exhaustive strata, and samples are selected from within these strata by basic random sampling.144 Courts have also admitted into evidence survey results drawn from non- probabilistic sampling.145 However, the proponent should be prepared to justify why she took a specific non-probabilistic method to select the sample respondents in the instant case.146 With recent technological innovations, businesses and academics have increasingly used Internet surveys for a variety of purposes. Internet survey can reduce substantially the cost of reaching potential respondents and, at the same time, can improve the way survey is designed and presented to the respondents. It also eliminates risks involving interviewer biases and reduces inaccuracies of data- recording/saving as the computer program presents survey questions and collects answers automatically. The threshold issue in evaluating an Internet survey is that the web-surfers may not fairly represent the relevant population whose responses the survey was designed to measure.147 For example, some Internet market research service companies maintain a huge panel of volunteers (the “panel population”) consisting of multi-million consumers.148 Although a subset of the panel may be randomly sampled from the panel population for conducting a specific survey, the panel population itself may not be the product of a random selection process. The panel population likely over-represents a group of relatively active and informative market participants.149 They may have a particular viewpoint on subject matters and/or a motive that might bias the survey results. Other issues concern respondent qualification and duplication. As the Internet survey is conducted over the online, security measures must be taken to confirm that the selected sample conforms to the purpose of the study. The procedures that Apple’s expert took in Apple v. Samsung illustrate the concerns.150 The expert and a market research company hired by him first checked the 143 Id. 144 Id. 145 Id. at 382. 146 Id. 147 Id. at 406. 148 E.g., Hauser Report, supra note 54, at 25 (“Research now . . . maintains an invitation-only panel over 3.6 million consumers in the United States and over 6 million panelists worldwide.”); Srinivasan Report, supra note 61, at 12 (“Optimal Strategix Group utilized Authentic Response’s US Adults panel of approximately 3 million respondents.”); see also DIAMOND, supra note 79, at 382 n.102. 149 One expert report notes that the panel population held by a market survey company does not accurately represent the whole population. Srinivasan Report, supra note 61, at 12 (“This [3 million] panel is maintained to be reflective of the U.S. Census (adults), but it is not exactly balanced with respect to the U.S. Census (adults).”). Therefore, the survey company weighted the sample population to be a more balanced representative of the U.S. Census of adults than the un-weighted sample. Id. 150 Hauser Report, supra note 54, at 26.
- 35. 35 identity of panelistsby reviewing registered email addresses and their basic demographic information.151 Based on the information, they invited people who indicated that they own smartphones.152 The invitation email included a link to actual survey, hosted on a website maintained by the market research company affiliated with the expert.153 The email link contained an embedded identification number to assure that only invited respondents could answer the survey and could do so only once.154 Before starting the survey, respondents were prompted to a CAPTCHA challenge to ensure that responses were not computer-generated.155 In addition, a high nonresponse rate can significantly bias the survey results and can be a ground for excluding survey evidence.156, 157 Nonresponses aggravate systematic error more seriously when the nonresponse is not random.158 The key in understanding the pattern and effect of the nonresponse in a survey is to determine the extent to which nonrespondents differ from respondents with respect to the dividing characteristics of these groups.159 The proponent must review the underlying raw data to investigate whether there exists significant nonresponses and whether the nonresponses are systematic. However, mere low response rate may not be damming.160 “Contrary to earlier assumptions, surprisingly comparable results have 151 Id. 152 Id. This initial criterion screens the pool of potential respondents by determining whether they belong to the target population of the survey. As the target population in the survey was smartphone owners, online survey in itself does not pose a serious underinclusive problem with respect to the availability of Internet access. “The screening questions must be drafted so that they do not appeal to or deter specific groups within the target population, or convey information that will influence the respondent’s answers on the main survey.” DIAMOND, supra note 79, at 385–86. 153 Hauser Report, supra note 54, at 26. 154 Id. 155 Id. at 27. CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart. “A CAPTCHA challenge refers to a program that protects websites against bots (i.e., computer-generated responses) by generating and grading tests that humans can pass, but current computer programs cannot.” Id. n.32. 156 Methods of computing response/nonresponse rates vary. “[A]lthough response rate can be generally defined as the number of complete interviews with reporting units divided by the number of eligible reporting units in the sample, decisions on how to treat partial completions and how to estimate the eligibility of nonrespondents can produce differences in measures of response rate.” DIAMOND, supra note 79, at 384 n.109. 157 See id. at 383–85 (suggesting 80% or higher response rates are desirable); 1 GIANNELLI ET AL., supra note 116, at § 15.04[b], 860–63. 158 DIAMOND, supra note 79, at 383 (“for example, persons who are single typically have three times the ‘not at home’ rate in U.S. Census Bureau surveys as do family members”). See supra text accompanying note 138 (regarding Literary Digest’s 1936 presidential election poll). 159 See DIAMOND, supra note 79, at 383–84 (“The key is whether nonresponse is associated with systematic differences in response that cannot be adequately modeled or assessed.”). 160 For example, Apple’s expert reported that for his conjoint survey, only 22.8% of the invitees responded. Hauser Report, supra note 54, at 37 n.47 (“Out of the total 38,795 participants invited for the smartphone survey, 8,844 began the survey, resulting in a response rate of 22.8%. . . . Out of the total 8,844 participants who started the smartphone survey, 604 passed through the screening questions and qualified for the choice tasks, resulting in an incidence rate of 6.8%.”). The survey result was not excluded.
- 36. 36 been obtained inmany surveys with varying response rates, suggesting that surveys may achieve reasonable estimates even with relatively low response rates.”161 Where a notable pattern is found between the respondents and nonrespondents groups, the proponent must be prepared to explain the cause and prove that either the high nonresponse rate does not bias the survey results or the effects are insignificant. The opponent must attack a survey’s high nonresponse rate. Even if the survey evidence with high response rate might be admitted to the trier of fact, unexplained high nonresponse rate reduces its probative value. On the other hand, the opponent should take a survey with a high response rate with no less scrutiny. What matters are the determinants of the nonresponse.162 Even if the survey has a high response rate, it still can underrepresent some portions of the relevant population.163 Therefore, the opponent should examine the underlying raw data from which the results were drawn. The following case illustrates the importance of recording and reporting nonresponse rates in a survey. In a class action lawsuit, defendants-employers moved to exclude plaintiffs-employees’ expert report by asserting that it failed to meet the standard for expert testimony outlined by Federal Rule of Evidence 702 and Daubert.164 One of the defendants’ arguments was that the expert’s study suffered from a substantial undocumented nonresponse bias, undermining the reliability of the opinion.165 The plaintiffs responded that the nonresponse rate was low and that any bias was minimal.166 Noting that “[t]he U.S. Office of Statistical Standards suggest that because nonresponse is often not random, a survey should maintain a response rate of 75% or above[,]” the court sided with the defendants.167 The court pointed out that the plaintiffs’ expert admitted that while the nonresponse rate was small, it was not documented.168 The expert’s careless failure to record nonresponse rate was the ground that the court found the expert testimony unreliable and inadmissible under Rule 702.169 161 DIAMOND, supra note 79, at 384. 162 Id. at 385. 163 Id. (“If a general population sample is used to chart changes in the proportion of the population that knows someone with HIV, the survey would underestimate the population value if some groups more likely to know someone with HIV (e.g., intravenous drug users) are underrepresented in the sample.”). 164 Chavez v. IBP, Inc., No. CV–01–5093–RHW, 2004 U.S. Dist. LEXIS 28838, at *1–2 (E.D. Wash. Dec. 8, 2004). 165 Id. at *28. 166 Id. 167 Id. 168 Id. 169 Id. at *29–34. In this regard, Apple’s expert recorded and reported the initial (non)response rate and the end incidence rate, although response and incidence rates were very low. See supra text accompanying note 160.
- 37. 37 C. Attributes &Levels The premise of conjoint analysis is that a bundle of attributes having partworth represents the total utility of the product, which decision makers attempt to maximize. Identifying attributes and levels in the design phase of the conjoint survey is crucial to designing the survey. Whether the proponent properly designed the survey by including enough significant attributes has been contested in almost all cases involving conjoint analysis, with or without success.170 i. Number of attributes is limited As a preliminary matter, the number of attributes and levels a proponent can include in the survey is limited for practical reasons. Including many more attributes, and, thus more profiles and more questions, comes with a cost. While more attributes (and levels) could be thought to provide more detailed information on the analysis of consumers’ willingness-to-pay study, it makes the survey more complex and demanding. Respondents’ fatigue and resulting unreliability of responses have been the major concerns that the conjoint analysis has evolved to improve.171 Academics typically suggest using no more than seven attributes.172 Seven attributes with two levels for each generate 128 ( 7 2 ) profiles. Surveying on a complete list of such profiles is impractical. As the number of the profiles increases sharply with increasing numbers of attributes and levels, only the subset of the potential profiles is systematically selected and tested.173 Conjoint analyses conduced in recent patent infringement cases show that experts followed the guideline and designed their surveys by using no more than seven attributes.174 ii. Attributes must include one that describes the claimed invention Whether the attribute (and one of its levels) accurately captures the scope of the claimed invention is the critical question determining the admissibility of expert testimony based on conjoint 170 E.g., TV Interactive Data Corp. v. Sony Corp., 929 F. Supp. 2d 1006, 1025–26 (N.D. Cal. 2013) (admitting expert opinion over defendant’s challenge that the expert employed too few relevant features in his conjoint survey); Oracle America, Inc. v. Google Inc., No. C 10–03561 WHA, 2012 WL 850705, at * 9–11 (N.D. Cal. Mar. 13, 2012) (excluding market share calculation as unreliable because the survey does not cover important product features whereas arguably unimportant feature is used included instead). 171 See Allenby et al., supra note 111; Peter E. Rossi & Greg M. Allenby, A Bayesian Approach to Estimating Household Parameters, 30 J. MARKETING RES. 171 (1993); Toubia et al., supra note 112; see, e.g., Green & Srinivasan, supra note 66. 172 E.g., Green & Srinivasan, supra note 66, at 8–9. 173 See e.g., Sidney Addelman, Orthogonal Main-Effect Plans for Asymmetrical Factorial Experiments, 4 TECHNOMETRICS 21 (1962). 174 E.g., Hauser Report, supra note 54, at 38 (seven attributes); Srinivasan Report, supra note 61, at 9–10 (six attributes).
- 38. 38 analysis. The statisticianneeds input from a market expert to properly design the survey. Without an appropriate attribute and level representing the infringed feature, the survey is irrelevant to prove damages the patentee claims to have been suffered, and the result of the survey cannot be of any help for the trier of fact to decide the reasonable royalty at hand.175 The survey not only must include the infringed feature but also should capture the scope of the claimed invention accurately. However, as the following two cases illustrate, the standard is not high (or arguably is inconsistent). On one hand, in Fractus v. Samsung, the district court granted the accused infringer’s Daubert motion to exclude patentee’s survey expert testimony.176 There, the patentee commissioned two conjoint surveys for the purposes of estimating the value of incorporating internal antennas in cell phones in place of external antennas.177 However, the patented invention, as the court noted and the patentee conceded, was only one type of internal antenna that purportedly provided a certain advantages.178 Accordingly, the court found that the survey did not properly capture the value of the claimed invention, but rather merely measured the perceived consumer value of cell phones with any internal antennas. Since the survey employed an overbroad attribute, the survey failed to directly tie the attribute to the patented technology and, thus, confused the issues.179 More recently, in a patent infringement dispute centered on a check-spelling-as-you-type function, Microsoft moved to exclude Sentius expert’s survey by asserting that the survey questions were overly broad.180 The patented invention was, according to Microsoft, its “use of a look-up table and its performance of certain steps in the order required by the claims.”181 Microsoft argued, and the court agreed, that the survey questioned not about respondents’ preferences for Microsoft’s spell check features implemented in a specific way, but for Microsoft’s features in general.182 Despite its acknowledgement that the survey could have been more narrowly tailored, however, the court did not exclude the survey result. The court explained that this defect was insufficient to establish that the survey question’s description varied too much from what is claimed in the invention.183 175 See FED. R. EVID. 702(a). 176 Fractus, S.A. v. Samsung, No. 6:09–cv–203–LED–JDL, 2011 U.S. Dist. LEXIS 154697, at *10–13 (E.D. Tex. Apr. 29, 2011). 177 Id. at *11. 178 Id. at *12–13. 179 Id. at *13. 180 Sentius Int’l, LLC v. Microsoft Corp., No. 5:13–cv–00825–PSG, 2015 U.S. Dist. LEXIS 8782, at *7–11 (N.D. Cal. Jan. 23, 2015). 181 Id. at *11. 182 Id. 183 Id. at *12.
- 39. 39 iii. Be carefulin selecting non-patented attributes Conjoint survey usually includes an attribute representing price or price discount. The price attribute is necessary as the partworth of each level of each attribute represents only relative (and conceptual) value in the units of utility. To estimate willingness-to-pay and royalty base eventually, the relative utility must be calibrated against the absolute, concrete scale, i.e., price. In addition to at least two necessary attributes (one or more representing claimed invention and one for the price), to avoid focus bias, researchers typically include a number of non-patented features. These non-patented features are called distraction features.184 These features are unrelated to the claimed invention in suit by definition, but nevertheless important in making the survey realistic and fair.185 Without distraction features, the conjoint survey is nothing more than a direct, closed-end, questioning, in which focus biases can easily taint the survey results. Oracle v. Google illustrates how the court would address the point.186 There, Oracle’s expert conducted a conjoint survey to measure relative importance of seven smartphone attributes: application multitasking; application startup time; availability of third-party applications; mobile operating system brand; price; screen size; and voice command capabilities.187 The patent at issue covered the first three functionalities. The expert selected the other four attributes from the focus-group research, from which he discovered thirty-nine features that real-world consumers said they would have considered when purchasing a smartphone.188 Google argued, and the court agreed, that the conjoint survey results were unreliable because the survey selected only seven features but purposely omitted important features that would have played an important role in real-world consumer behaviors.189 The court questioned the reliability of the survey results. To the court’s eye, it was highly likely that the survey respondents would have placed greater importance on a selected feature because only seven features were selected as opposed to thirty-eight other features.190 The court reasoned that the respondents were artificially forced to focus on a selected feature like startup time even if in the real world, it was unimportant to them.191 184 Declaration of John R. Hauser at 3, Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK (N.D. Cal. July 26, 2012), ECF No. 1363. 185 Iyer, supra note 39. 186 Oracle America, Inc. v. Google Inc., No. C 10–03561 WHA, 2012 WL 850705, at * 9–11 (N.D. Cal. Mar. 13, 2012). 187 Id. at *9. 188 Id. at *10. 189 Id. 190 Id. 191 Id.
- 40. 40 The court continuedto criticize the survey’s methodology. The fact that the survey did not include important product features such as battery life and WiFi capability, but instead selected an arguably unimportant feature, voice dialing, exacerbated the focus bias problem.192 The expert, the court found, failed to provide any reasonable criteria for choosing non-patented features in the survey and instead forced the respondents to focus on the patented functionalities.193 In TV Interactive Data v. Sony, plaintiff’s damages expert designed the conjoint survey to estimate consumers’ willingness-to-pay for the incremental benefit from using TV Interactive Data’s patented technology, a so-called “Autoplay” function, in Sony’s Blue-ray players.194 The expert decided to limit the number of attributes to five – the attribute representing the Autoplay function, price discount, and three non-patented attributes.195 The expert attempted to select non-patented attributes that have relatively similar values to that of the Autoplay function.196 He adopted a two-phase approach. In Phase I of the study, the expert focused on generating enough attributes to be able to identify six attributes for Phase II.197 (The partworth or the willingness-to-pay for the patented technology at issue, Autoplay, was measured in the Phase II of the study.) The expert needed to identify six attributes because he decided to conduct Phase II in two versions, using three non-patented attributes (with price discount and the Autoplay function) in each version of the study. The expert, with the help of a hired market research company, first collected information on Blu-ray players and their attributes from the manufacturers’ and well-known retailers’ websites.198 The expert then sorted out sixteen attributes based on a set of general considerations.199 To determine how respondents would value eighteen attributes, in Phase I the expert essentially conducted a mini conjoint survey by using a method called Adaptive Self-Explication of Multi-Attribute 192 Id. 193 Id. 194 Srinivasan Report, supra note 61, at 3. 195 Id. at 5. 196 Id. at 10–12. Thus, the expert’s main concern in determination of non-patented attributes was slight different from that of the Oracle court. In TV Interactive Data, the expert selected attributes that have “similar values” to the infringing feature. In contrast, the court in Oracle criticized the expert’s failure to include “important” features. 2012 WL 850705, at * 10. 197 Id. at 9. 198 Id. at 8–9. 199 Id. at 9 (“(a) Avoid attributes that are nearly universally present (> 95% of the [Stock Keeping Units] have this attribute). Such attributes are unlikely to be a basis for customer choice; (b) Avoid attributes that are nearly universally absent (<5% of the [Stock Keeping Units] have this attribute). Such attributes are unlikely to be a basis for customer choice. (c) Avoid attributes that have a large number of values (e.g., playable disc types, the physical measurements of the Blu-ray player). A phenomenon called ‘number of levels effect’ in conjoint analysis indicates that the value for an attribute with a large number of levels would be overstated. (d) Avoid attributes that include adjacent product categories (e.g., combinations of Blu-ray players with Recorders, HD-DVD players) because such product categories have additional sets of attributes as well; (e) Avoid attributes that were too highly technical to translate to customer language.”)
- 41. 41 Preferences (ASEMAP).200 As thenumber of attributes was rather large, he first categorized the eighteen attributes into two buckets – the more important nine and the less important nine.201 Respondents were then asked to rank attributes within each bucket.202 This created a rank order of the importance of all eighteen attributes. Based on the list, respondents were asked to compare a pair of two attributes in a relative scale.203 For example, a respondent was asked to compare Autoplay and price discount in the fixed scale of 100 points. If the respondent valued Autoplay more and assigned 60 points on it, it was deemed that the price discount received the remaining 40 points. The expert designed the pairs such that thirteen strategically paired comparisons were sufficient to get accurate results with less than 5% average error.204 Log-linear multiple regression was applied to analyze the data and estimate relative importance of the eighteen attributes.205 Comparing the relative importance of all attributes, except price discount, the expert could identify six attributes – three for each version – thought to have comparable values to that of the patented feature.206 iv. Include non-infringing alternative levels207 The availability of non-infringing alternatives directly influences consumer’s willingness-to-pay for the infringing attribute. The importance of the availability of the non-infringing alternatives has been emphasized by the court, particularly when the calculation of the reasonable royalty base was disputed.208 When the non-infringing alternatives exist, the proponent must assess the degree to which consumers prefer the patented feature to then available non-infringing alternatives.209 Thus, the 200 Id. at 6. 201 Id. at 11. 202 Id. 203 Id. 204 Id. 205 Id. at 12; see also Oded Netzer & V. Srinivasan, Adaptive Self-Explication of Multi-Attribute Preferences, 48 J. MARKETING RES. 140, 140–56 (2011). 206 The expert compared “the average importance from the weighted sample for [Autoplay]” as well. Srinivasan Report, supra note 61, at 13. 207 Although sounds similar, this topic differs from the subject discussed in the preceding section. Here, the focus is on the level, not on the attribute. For example, where the patented technology at issue is about an “X method” of accomplishing a touchscreen functionality, the “touchscreen capability” attribute should include non-infringing “Y method” and “Z method,” if such methods were available at the time of the infringement. On the other hand, non-patented attributes in the preceding section concerned about including other important or comparable attributes such as battery life or WiFi capability, which are not the subset of the touchscreen capability. 208 Cornell Univ. Hewlett-Packard Co., 609 F. Supp. 2d 279, 286–87 (N.D.N.Y. 2009). 209 Cameron et al., supra note 47.
- 42. 42 damages expert shouldfirst consider whether non-infringing alternatives are available, and if the answer is yes, non-infringing levels of the relevant attributes should be included in the conjoint survey. v. Large number of levels can bias the survey result Not only is the absolute number of the attribute levels important in designing a realistic and reliable conjoint survey, but the relative numbers of the levels in different attributes are also important. A large number of levels in an attribute can overstate the value of that attribute particularly.210 This so- called “number of levels effect” in conjoint analysis implicates two points. First, attributes having a large number of levels should be avoided.211 Second, the number of levels across the attributes should be kept at the relatively similar level.212 D. Relevant, Clear & Unbiased Questions213 The experts should not only seriously consider issues addressed in the preceding section in designing survey questions. The questions must also be phrased and presented to the respondents in a proper manner. Only questions relevant to the purposes of the survey should be include, and they should be phrased clearly and fairly. These questions should be presented following fair and sound procedures. i. Questions should be relevant to the purposes of the survey Survey questions must be sufficiently tied to specific facts of the case.214 Still, the standard for relevancy is not set in stone. In patent infringement suits where the conjoint survey results were proffered, opponents have contended that the survey questions did not accurately capture the invention claimed by the patents. In one case, the court concluded that the opponent’s criticism of the description of the patented features in the survey goes to weight, not admissibility.215 In contrast, in another case, 210 Srinivasan Report, supra note 61, at 9. 211 Id. 212 See e.g., Hauser Report, supra note 54, Exhibit D, at 7–14 (using four levels for all seven attributes). 213 Questions include the presentation of choice sets when the survey relies on discrete, choice-based model. 214 See Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 591 (1993) (“An additional consideration under Rule 702 – and another aspect of relevancy – is whether expert testimony proffered in the case is sufficiently tied to the facts of the case that it will aid the jury in resolving a factual dispute.”) (emphasis added). 215 Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK, 2012 WL 2571332, at *10 (N.D. Cal. June 30, 2012).
- 43. 43 the court excludedexpert’s survey results finding that the survey did not properly capture the claimed invention because survey question was overly broad and confused the issue.216 ii. Questions should be clear and unambiguous Second, questions should be clear to the respondents when presented. Even questions that appear clear to the surveyor can convey unexpected meanings and ambiguities to potential respondents.217 The wording of such questions would naturally raise doubts about the validity of the survey by potentially distorting response data. What the questions intended to ask simply may not match what the answers deliver if the question is subject to multiple interpretations. Precise questions without legal or technical jargons are preferred. If using jargon cannot be avoided, these terms must be fully explained to the respondents before asking them to answer the questions. Still, determining whether the question was clear enough to be accurately understood would depend on the respondents’ identity and may go to weight rather than admissibility.218 Microsoft and Motorola disputed over the licensing of patented technology used in Microsoft’s Xbox 360 console.219 Microsoft moved to exclude Motorola’s survey evidence by arguing that the survey was unreliable.220 Microsoft pointed that the survey asked respondents to “select the types of video content [he or she] has viewed on his or her Xbox console and providing options of ‘MBAFF (Macroblock Adaptive Frame/Field),’ ‘Progressive,’ and ‘not sure.’” Finding the argument unpersuasive, the court noted that the survey defined the disputed terms. The court added that since the questions were given to respondents who owned an Xbox, the respondents were more likely to understand the technical terms.221 The court held that “the clarity of the survey questionnaire is a question of fact, best resolved at trial through cross examination.”222 216 Fractus, S.A. v. Samsung, No. 6:09–cv–203–LED–JDL, 2011 U.S. Dist. LEXIS 154697, at *13 (E.D. Tex. Apr. 29, 2011). 217 DIAMOND, supra note 79, at 387. 218 Microsoft Corp. v. Motorola, Inc., 905 F. Supp. 2d 1109, 1119–20 (2012). But see Firestone v. Crown Ctr. Redevelopment Corp., 693 S.W.2d 99, 102–03 (Mo. 1985) (en banc). The court in Firestone concluded that the question “Based on what you have heard, read or seen, do you believe that in the current compensatory damage trials, the defendants, such as the contractors, designers, owners, and operators of the Hyatt Hotel, should be punished?” could neither be correctly understood nor easily answered. Id. at 103. The court noted that the phrase “compensatory damage” was unlikely to be meaningful for laypersons. Id. 219 Microsoft, 905 F. Supp. 2d at 1119–20. 220 Id. 221 Id. at 1120. 222 Id.
- 44. 44 Researchers have suggesteda variety of ways to improve the clarity of communication with respondents. These methods include using pretests or pilot tests,223 which provide opportunities to receive respondent feedbacks and to improve questions before conducting a full-scale survey.224 Typically, interviewers conduct a pretest on a small sample of the same type of respondents eligible to participate in the full-scale survey.225 The interviewers then observe any difficulties pretested respondents may have with the questions and probe for the source of any such difficulties.226 Interviewers find a way to rephrase the questions to eliminate, or at the least minimize, any confusion or difficulty that arose. Pretesting may be designed in multi-phases so that the rephrased questions may be tested again, and the responses can be used to further refine the questions. As pretesting might be an attractive option to add before conducting the full-scale survey, it bears emphasis that the pretesting sample itself must comply with the guidelines discussed above. In short, the pretesting sample must represent the sample and the target population fairly. The representativeness of the pretesting sample is crucial as the questions are modified under the assumption that the pretesting sample accurately represents the sample. The pretesting sample can be randomly selected from the sample, which is already selected randomly from the target population. iii. Questions should be fair and unbiased Lastly, questions must be unbiased in the sense that they may not lead respondents. First, forms of the questions should be consistent. In this regard, an Internet survey has an advantage over the surveys in which interviewers directly interact with the respondents, because the program can ask exactly the same questions without adding any nuance. Highlights – by different color, font, size, underline, or other similar effects – can easily draw respondents’ attention. Unbalanced degree of details on a particular term or feature also can lead the respondents to overestimate the importance of the highlighted feature. Likewise, if the question incorporates completely different formats to describe an attribute or level, it may unfairly impress the respondents. For example, if only one of the attribute or its level appears as a graphic or animation, but all the other features are described in text, the interviewee easily would recognize where the focal point of the survey lies. 223 DIAMOND, supra note 79, at 388. Footnote 128 explains the difference between pretest and pilot test: “The terms pretest and pilot test are sometimes used interchangeably to describe pilot work done in the planning stages of research. When they are distinguished, the difference is that a pretest tests the questionnaire, whereas a pilot test generally tests proposed collection procedures as well.” Id. (italics original). 224 Id. at 388–89; Hauser Report, supra note 54, at 23–25. 225 DIAMOND, supra note 79, at 388–89. 226 Id. at 389.
- 45. 45 However, consistency informs does not mean that a set of questions must be asked in the same order. In fact, the order of questions can influence respondent behaviors.227 For example, when the alternatives are presented visually, as in mail surveys or in face-to-face interviews, they are more likely to select the first choice offered.228 In contrast, when the alternatives are orally presented, as in telephone surveys, they are more likely to choose the last choice offered.229 Although these effects are typically small in some cases, courts have criticized the survey because it failed to rotate the question order randomly.230 It is desirable for the proponent to randomize both the order of the questions and the order of choices within the questions.231 Of course, exceptions to the rotation rules exist: Certain options such as “Other,” “All of the above,” and “None of the above” always come last to preserve logical flow of the question set.232 With such limitations in mind, the opponent must carefully review the survey questions and investigate raw data to probe whether there is a pattern of responses. Even if the magnitude of the bias caused by nonrandomized ordering is insignificant, and it might not be the ground for excluding the survey results, the opponent must raise doubts on the reliability of the survey results in the jury’s mind. The opponent may point out that the Internet survey can be so easily adapted to administer questions randomly to the respondents, but the expert failed to randomize survey questions. E. Sound Procedures Surveys start way before questions are presented to respondents and end when the analysis is completed and reviewed. The validity of the survey result is predicated on the reliability of the subjects discussed in preceding sections – whether the target population is proper; whether the sample appropriately represents the population; and whether the questions are designed following sound principles and presented accurately without biases. Of course, respondents’ answers must be accurately recorded and maintained securely. In addition to certain measures that have been discussed already (such as including preliminary questions for respondent identification and qualification, conducting pretests or constructing the survey 227 Id. at 396. 228 Id. (“a primacy effect”). 229 Id. (“a recency effect”). 230 Id. 231 Apple’s damage expert “rotated answer options so that different respondents see the options in different orders and any possible order effects cancel out across respondents.” Hauser Report, supra note 54, at 31. TV Interactive Data’s damage expert also randomized the order of the presentation of attributes across respondents. Srinivasan Report, supra note 61, at 14. 232 Hauser Report, supra note 54, at 31.
- 46. 46 in multi-phases, andrandomizing the questions’ order), the proponent and her damages expert should consider to adopt following procedures to ensure overall soundness of the survey. i. Clear and conspicuous instructions must be given Instructions must explain clearly what the respondents are expected to do in the survey. As the conjoint survey compares only the attributes and levels purposely selected for the study, the instructions must direct the respondents to assume that all other features are the same or irrelevant.233 If any words or phrases in the survey convey nonconventional meanings, the instructions must make that point by providing a precise description for such terms. ii. Interviewer may consider including filters “Filters are questions and/or answer options that eliminate respondents who are not relevant or who do not have opinions.”234 By including filters, the survey can avoid asking irrelevant questions or coercing respondents to express a belief that they do not have.235 Interviewers (if they conduct the survey personally) should be instructed to terminate the survey, or the Internet survey could likewise be programmed to terminate the survey if respondents’ answer triggers any filter question. Similarly, quasi-filters offer answer options such as “don’t know” or “no opinion.”236 Quasi-filters avoid speculation and reduce the risk of respondents’ guessing especially when they do not have an opinion or preference on a given question.237 iii. Double-blind test is desirable When interviewers ask questions to respondents personally, it is desirable not to tell the interviewers who is sponsoring the survey and/or the purpose of the survey.238 For surveys conducted to support litigation, the standard interview practice is to do double-blind test whenever possible.239 Double-blind test means that both interviewers and respondents are blind to the survey’s sponsor and purpose. 233 See, e.g., id. at 36; Srinivasan Report, supra note 61, at 14. 234 Hauser Report, supra note 54, at 30–31. 235 Id. at 31. 236 Id. 237 DIAMOND, supra note 79, at 390 (“By signaling to the respondent that it is appropriate not to have an opinion, the question reduces the demand for an answer and, as a result, the inclination to hazard a guess just to comply. Respondents are more likely to choose a ‘no opinion’ option if it is mentioned explicitly by the interviewer than if it is merely accepted when the respondent spontaneously offers it as a response.”). 238 Id. at 410. 239 Id. at 410–11.
- 47. 47 When sponsor disclosureis unavoidable, it is important to evaluate whether the sponsor has apparent expectations and whether only interviewers or both interviewers and respondents are aware of the identity of the sponsor and/or certain perspectives of the sponsor.240 Trainings and instructions can reduce potential bias that might be induced by interviewers’ awareness.241 Respondents’ awareness is a serious threat and should be dealt with filtering questions, since training or instructing respondents not to be aware of the sponsor is generally ineffective. An Internet survey removes risks associated with inadvertent interviewer conducts as respondents are not exposed to human interviewers. As the damages experts typically hire a market survey company to work with the experts and to conduct the survey, the experts may consider not disclosing the sponsor’s identity and the survey’s specific purpose to the company, or disclosing information only to the extent necessary for conducting the survey. F. Accuracy of Data Even when survey questions were designed and presented fairly, the end analysis cannot be said to be reliable, if respondents’ answers are not recorded accurately or maintained securely. Internet surveys or any computerized methods may eliminate human errors involving the process of conducting surveys and recording and gathering data. Still, the safety measures should be carefully built in to protect the raw data from potential manipulation or contamination. G. Statistical Analysis Conjoint analysis, as a research tool relying on the survey data, produces estimates of population values by inferring from the sample survey results, not exact measures of those values.242 Consequently, conjoint analysis unavoidably relies on statistics for data interpretation. The gathered data must be analyzed and interpreted in accordance with accepted statistical principles. Estimation, standard errors (or standard deviations), confidence intervals, significance levels and hypothesis tests all are well-established statistical principles widely used in many areas of daily life.243 These statistics have been widely adopted in court proceedings.244 Likewise, multiple regression 240 Id. at 411. 241 Id. 242 Id. at 381. 243 See id. at 240–57. 244 See id.
- 48. 48 analyses have beenfrequently used in the courtroom.245 Although the use of the Bayes theorem has been objected, courts have become more receptive to the admission of the statistical evidence based on the theory.246 Mathematical and statistical foundations for logit estimation,247 which is the backbone of the choice-based conjoint analysis, have been well established by academics like econometrician Daniel McFadden, who shared the 2000 Nobel Prize in Economics “for his development of theory and methods for analyzing discrete choice.”248 H. Objectivity of the Process The objectivity of a survey must be evaluated by considering several procedural and substantive issues discussed here. The overall objectivity of the survey should be assured. Although the fact that the survey was not specifically prepared for, or in response to, litigation buttresses its objectivity, even such studies frequently ask irrelevant questions or select inappropriate samples.249 Still, surveys conducted for the purposes of anticipated and/or pending litigation may well support the proponent’s position. In practice, it is extremely unlikely that the survey conducted not in connection with the litigation could have properly captured the patented and non-infringing features. After all, the trial court construes the scope of the patent claim in so-called “Markman hearing,”250 and the Federal Circuit, until very recently, has reviewed de novo all aspects of the district court’s claim construction including its determination of subsidiary facts.251 Rather, Federal Rules of Evidence and Daubert require that the survey conform to sound scientific methodology and be tied to the facts of the case, to ensure that the survey evidence will help the trier of fact to find a fact in issue. In sum, Part III reviewed eight key areas of contention for survey evidence. A review of recent patent infringement cases reveals that the dispute often centers on the proper design of conjoint analysis – particularly on the selection of appropriate attributes.252 Even properly designed, conducted, and 245 Id. at 305–09; see also Franklin M. Fisher, Multiple Regression in Legal Proceedings, 80 COLUM. L. REV. 702 (1980). 246 1 GIANNELLI ET AL., supra note 116, at § 15.04, 823–29. 247 DANIEL MCFADDEN, Conditional logit analysis of qualitative choice behavior, in FRONTIERS IN ECONOMETRICS (ECONOMETRIC THEORY AND MATHEMATICAL ECONOMICS) 105–42 (Paul Zarembka ed. 1973), available at http://eml.berkeley.edu/reprints/mcfadden/zarembka.pdf. 248 The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2000, NOBELPRIZE.ORG, http://www.nobelprize.org/nobel_prizes/economic-sciences/laureates/2000/index.html (last visited Apr. 18, 2015). 249 DIAMOND, supra note 79, at 373. 250 Markman v. Westview Instruments, Inc., 517 U.S. 370 (1996). 251 See Teva Pharm. USA, Inc. v. Sandoz, Inc., 135 S. Ct. 831 (2015) (holding that the Federal Circuit must apply a clear error, not a de novo, standard of review to lower court’s factual findings in claim construction). 252 Compare Sentius Int’l, LLC v. Microsoft Corp., No. 5:13–cv–00825–PSG, 2015 U.S. Dist. LEXIS 8782, at *7–11 (N.D. Cal. Jan. 23, 2015) (admitting expert opinions based on conjoint analysis), and TV Interactive Data Corp. v. Sony Corp., 929 F. Supp. 2d 1006, 1025–26 (2013) (admitting), with Oracle America, Inc. v. Google Inc., No. C 10–03561 WHA, 2012 WL
- 49. 49 analyzed conjoint surveysstill require sound interpretation to help the trier of fact to find the reasonable base amount for royalty calculations. IV. INTERPRETING CONJOINT ANALYSIS Even with the appropriate survey design and data analysis, experts must interpret the survey results carefully. First, the expert must acknowledge the scope of the conjoint analysis – a boundary between what it tells and on which it keeps silent. Conjoint survey for one valid purpose may be inappropriate in another context.253 Second, as statistical evidence, the analyzed data should be reported in appropriate statistical expressions. A. Scope of Survey As Figure 1 illustrates, utility functions assume that the partworth of the levels residing somewhere between the designated levels in the survey can be interpolated. (In Part II, hypothetical respondent’s willingness-to-pay for the LED headlight was estimated based on this assumption.) This is a testable assumption especially when the attributes are quantitative and the utility function takes a vector model or an ideal point model. However, the proponent must be careful in extrapolating the data beyond the tested range of the attribute.254 Even a seemingly linear variable such as price has nonlinearity when the price increases or decreases beyond a certain level. Thus, a better approach is to set the levels of the disputed attribute reasonably but wide enough to cover the expected range of the effects on consumer preferences. An interpretation based on conjoint analysis must be limited to part of the reasonable royalty calculation problem. While a royalty base may be set by the market’s willingness-to-pay inferred from the survey data, to compute the royalty, a reasonable royalty rate must also be selected. As noted earlier, the base and the rate are distinct quantities. Even if conjoint analysis is found to have properly estimated one term, that does not guarantee that the same analysis estimates the other term properly. Thus, the proponent must explain two terms separately. 850705, at * 9–11 (N.D. Cal. Mar. 13, 2012) (excluding), and Fractus, S.A. v. Samsung, No. 6:09–cv–203–LED–JDL, 2011 U.S. Dist. LEXIS 154697, at *10–13 (E.D. Tex. Apr. 29, 2011) (excluding). 253 See Apple, Inc. v. Samsung Electronics Co., Ltd., No. 11–cv–01846–LHK, 2014 WL 9768989, at *10–16 (N.D. Cal. Mar. 6, 2014). The court there addressed the weights of the conjoint survey in response for Apple’s renewed motion for permanent injunction, not ruled on its admissibility for Daubert challenge. 254 See Srinivasan Report, supra note 61, at 17–18. The expert there truncated the price discount as not to extrapolate beyond the maximum price discount that he used in the conjoint surveys.
- 50. 50 B. As StatisticalEvidence The data must be presented in the form of statistics. Thus, the plaintiff’s experts relying on surveys inevitably describe the survey results in terms statistical outputs such as means, standard errors/deviations, 2 R , and/or confidence intervals. Such parameters account for error rates of survey evidence. For his or her part, the opponent must examine the raw data such as the sample size and nonresponse rates. Whenever the survey result is reported, such parameters must be accompanied with the end result to evaluate the scope of valid interpretation.255 CLOSING: SURVEY EVIDENCE AS A PROOF OF ECONOMIC DAMAGES The use of the survey and conjoint analysis in the courtroom is neither limited to proving a reasonable royalty base in patent infringement suits nor confined to damage calculations.256 When the survey is used to prove economic damages, as is mostly the case in patent infringement disputes, establishing a causal relationship between the harmful act and the plaintiff’s damages is critical.257 As the Supreme Court made clear in Kumho, Daubert applies to all types of expert testimony, including the expert testimony on the estimation of economic damages.258 The utility of conjoint analysis consists of its ability to provide a sound methodological approach to isolate the effect a particular feature brings on consumer behaviors when such feature coexists in aggregation with several non-separable features. Although conjoint analysis would meet Daubert’s flexible, but rigorous, standard,259 the proponent should remember that the financial expert testimony is excluded more often on the ground of the misuse of accepted methodologies than the introduction of novel analytical methods.260 255 See Hauser Report, supra note 54, at 48–49. 256 For example, in United States v. H&R Block, Inc., an antitrust action, defendant’s expert conducted conjoint analysis to “analyze the effect of a change in the price of each product on its own sales and the sales of the other tax preparation options.” 833 F. Supp. 2d 36, 67 (D.D.C. 2011). 257 DIAMOND, supra note 79, at 432. 258 Kumho Tire Co., Ltd. v. Carmichael, 526 U.S. 137 (1999); see also Sargon Enter., Inc. v. Univ. of S. Cal., 55 Cal.4th 747 (2012). 259 Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579 (1993). The Court set forth six factors for consideration: whether the theory or technique in question (1) is testable; (2) has been tested; (3) has been subject to peer review and publication; (4) has known or potential error rate; (5) has stands controlling the technique; and (6) has attracted widespread acceptance within a relevant scientific community. Id. at 581. 260 CHARLES REDDIN & DOUGLAS E. BRANCH, PRICEWATERHOUSECOOPERS LLP, DAUBERT CHALLENGES TO FINANCIAL EXPERTS: A YEARLY STUDY OF TRENDS AND OUTCOMES 2013 15, available at http://www.pwc.com/en_US/us/forensic- services/publications/assets/daubert-study-2013.pdf; LAWRENCE F. RANALLO & DOUGLAS E. BRANCH, PRICEWATERHOUSECOOPERS LLP, DAUBERT CHALLENGES TO FINANCIAL EXPERTS: AN 11-YEAR STUDY OF TRENDS AND
- 51. 51 The proponent, therefore,should consider all legal and practical issues examined above, and accordingly design, execute, and interpret conjoint surveys. For his or her part, the opponent should point to any unexplained weakness in any step of the survey. Although courts have generally been receptive to conjoint survey evidence, the opponent still may effectively attack the weight of the survey results. In addition, both sides should keep in mind that neither the survey method nor the hypothetical negotiation approach is the only accepted way of calculating reasonable royalties. When available, analytical approaches such as income-based or cost-based approaches should be used to double-check and confirm the soundness of market-based survey evidence. OUTCOMES 2000-2010 24, available at https://www.pwc.com/en_US/us/forensic-services/publications/assets/daubert-study- 2010.pdf.