Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013)

Intelligent Trademark
Analysis
Experiments in Large-Scale Evaluation
of Real-World Legal AI
Anna Ronkainen
Chief Scientist, Onomatics, Inc.
miscellaneous, University of Helsinki
anna.ronkainen@helsinki.fi
@ronkaine

―These papers had a low rate of
consideration of evaluation issues
reflecting common practice in research
biased development environments.
These results confirm that more attention
to evaluation is needed in the legal
knowledge based systems domain.‖
Hall & Zeleznikow (ICAIL 2001): Acknowledging Insufficiency in the Evaluation of
Legal Knowledge-based Systems: Strategies Towards a Broad-based Evaluation
Model
2013-06-12
Anna Ronkainen - @ronkaine
Intelligent Trademark Analysis
2

―Relevancy ranking is currently a foreign
concept in the trademark legal industry, as
assigning numeric weight to potentially
conflicting citations based on objective
computer analysis is currently avant-garde.‖
Anderson & Cary (2010): Navigating the Challenges of U.S. Pharmaceutical
Trademark Clearance Research
http://www.pharmpro.com/articles/2010/04/Navigating-the-Challenges-of-US-
Pharmaceutical-Trademark-Clearance-Research/
―Yet trademark law is supposed to be subjective
in the sense of being based on personal
opinions—it is just that the relevant opinion is
that of the average consumer [...]‖
Lisa Larrimore Ouellette (forthcoming): The Google Shortcut to Trademark law
http://ssrn.com/abstract=2195989
2013-06-12
3

TrademarkNow™
NameCheck™
• a system for trademark risk analysis
• relevancy-ranked trademark search
• based on a model of trademark (mark
and product) similarity, derived from (but
not directly based on) MOSONG
(Ronkainen 2010)
• additional features for absolute
grounds, word meanings etc.
• current name since March 2013
• originally launched as Onomatics Quick
Search (2012) without risk analysis
• web service, currently free preview but
otherwise by subscription only
2013-06-12
5

TrademarkNow™
NameRank™
• a tool for ranking 2–5 trademark
candidates to find the least risky
one
• same basic technology as
NameCheck™
• not actively offered at the moment
2013-06-12
9

Evaluation: MOSONG
First round:
30 most recent (2002) relevant cases:
• 20 from the Opposition Division and
• 10 from the Boards of Appeal
Result*: all cases predicted correctly
* when coded into the system by a domain expert
2013-06-12
11

Evaluation: MOSONG
Second round: Non-expert validation:
• done by non-law students taking a course in
intellectual property law (n=75)
• original validation set in two parts (15+15 cases)
• at the beginning and the end of the course
• completed non-interactively through a web form
• correct answer: 54.6±6.5%
• incorrect answer: 25.9±7.5%
• no answer: 19.5±5.2% (± = σ)
2013-06-12
12

Evaluation: MOSONG
% ±stderr before after total
group 1 (n=15) 41.3±1.7 65.8±2.8 53.5±1.7
group 2 (n=12) 46.1±2.0 65.0±3.0 55.6±1.9
group 3 (n=48) 43.3±1.3 65.9±1.3 54.7±0.9
total (n=75) 43.4±1.0 65.8±1.1 54.6±0.8
2013-06-12
13
Second round: Non-expert validation:

Testing NameCheck™
• system must work in real-world
conditions ⇒ has to be robust
• as wide a range of realistic test cases as
possible is desirable
• system is based on a predictive model
for trademark opposition cases so such
cases can be used (almost) directly for
testing
• specific service subcomponent for
evaluating cases in bulk: text file input
but otherwise same process
• inputs: mark and plain-text product(s)
2013-06-12
14

Test cases
• closed, unappealed first-instance
opposition cases
• only those where the prior right is found
in the database:
– same jurisdiction
– still valid
– word or combination mark
• typically multiple prior rights per case
• readily available as XML (but USPTO
TTAB XML is write-only)
• grand total: 55435 cases
2013-06-12
15

Test cases: EU
• source: OHIM (EU TM office)
Opposition Division
• for marks filed in 1996–2011
• opposition considered successful:
– opposition upheld (1563 cases)
– mark withdrawn (10516 cases)
• opposition unsuccessful:
– opposition rejected (4217 cases)
– opposition withdrawn (5368 cases)
2013-06-12
16

Test cases: US
• source: USPTO Trademark Trial and Appeal
Board (TTAB)
• for marks filed in 1980, 1985, 1990, 1995–
2011
• opposition successful
– opposition upheld (18431 cases)
– mark withdrawn (7329 cases)
• opposition unsuccessful:
– opposition dismissed (8897 cases)
– opposition withdrawn (8603 cases)
• categories and outcomes determined
heuristically so some overlap (duplicates
removed for the grand totals)
2013-06-12
17

Performance criteria
• result considered correct based
on the highest similarity score for
all prior rights iff
– opposition successful for at least
some of the goods and services and
score ≥ 0.5
– opposition unsuccessful and
score < 0.5
2013-06-12
18

Overall performance
• EU (21162 cases): precision
79.2%, recall 97.3%, F 87.3%
• US (33773 cases): precision
80.4%, recall 93.3%, F 86.4%
• total (55435 cases): precision
79.9%, recall 94.9%, F 86.7%
• low-risk cases hurt precision:
erring on the side of safety
2013-06-12
19

High-risk performance
Case type n listed in
top 10
listed in
top 100
correct
decision
EU:
opposition
upheld
1473 544
36.9%
921
62.5%
1425
96.7%
EU:
opposition
withdrawn
9515 4836
50.8%
6865
72.1%
9261
97.3%
US:
opposition
upheld
18409 5618
30.5%
9955
54.1%
17138
93.1%
US:
opposition
withdrawn
7278 2367
32.5%
4129
56.7%
6776
93.1%
2013-06-12
20

What is this good for
(absolutely nothing?)
• necessary but not sufficient
• inputs quite noisy: many cases decided on factors
other than likelihood of confusion: reviewing all of
them and removing the noise the only way to >99
% (not going to happen)
• mainly useful for regression testing (but too slow
for that, we mostly use two 1000-case samples)
• only measures one particular aspect of system
performance (and not the most relevant one)
• additional specific test case sets needed for
various elements in similarity ordering: developed
(and under development) from scratch, typically
based on actual marks
• to be continued...
2013-06-12
21

Thank you!
Further reading:
• @ronkaine on Twitter
• http://www.legalfuturology.com/ (my research blog)
• http://blog.onomatics.com/
• Ronkainen, Anna (2010): MOSONG, a Fuzzy Logic Model of Trade Mark Similarity
http://ssrn.com/abstract=1879399
• Ronkainen, Anna (2013): Redefining Trademark Clearance with Intelligent Legal
Technology. IPRinfo 1/2013. http://blog.onomatics.com/2013/02/redefining-
trademark-clearance-with.html
• Ronkainen, Anna (forthcoming): Scaling Intelligent Trademark Analysis from
Prototype to Production: From MOSONG to Onomatics Quick Search. Presented at
the 1st International Workshop on Artificial Intelligence and Intellectual Property.
222013-06-12

Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013)

Recommended

Recommended

More Related Content

Similar to Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013)

Similar to Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013) (20)

More from Anna Ronkainen

More from Anna Ronkainen (20)

Recently uploaded

Recently uploaded (20)

Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013)

Editor's Notes