SlideShare a Scribd company logo

Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013)

Presented at the International Conference on Artificial Intelligence and Law, Rome, June 2013. Abstract: The intelligent trademark analysis system developed by Onomatics is a trademark information system based on an AI model of trademark similarity (likelihood of confusion). The basic technology can be used as a general trademark search engine as well as for more specific purposes ranging from trademark candidate ranking (TrademarkNow NameRank) to comprehensive risk analysis (TrademarkNow NameCheck) and IPR enforcement. This paper presents a brief overview of the system and the concrete applications available as of April 2013. The system has been evaluated on a large number of actual trademark opposition cases, over 30 000 from the USPTO TTAB and over 20 000 from the OHIM Opposition Division, currently with a precision of 79.9% and a recall of 94.9%.

1 of 22
Intelligent Trademark
Analysis
Experiments in Large-Scale Evaluation
of Real-World Legal AI
Anna Ronkainen
Chief Scientist, Onomatics, Inc.
miscellaneous, University of Helsinki
anna.ronkainen@helsinki.fi
@ronkaine
―These papers had a low rate of
consideration of evaluation issues
reflecting common practice in research
biased development environments.
These results confirm that more attention
to evaluation is needed in the legal
knowledge based systems domain.‖
Hall & Zeleznikow (ICAIL 2001): Acknowledging Insufficiency in the Evaluation of
Legal Knowledge-based Systems: Strategies Towards a Broad-based Evaluation
Model
2013-06-12
Anna Ronkainen - @ronkaine
Intelligent Trademark Analysis
2
―Relevancy ranking is currently a foreign
concept in the trademark legal industry, as
assigning numeric weight to potentially
conflicting citations based on objective
computer analysis is currently avant-garde.‖
Anderson & Cary (2010): Navigating the Challenges of U.S. Pharmaceutical
Trademark Clearance Research
http://www.pharmpro.com/articles/2010/04/Navigating-the-Challenges-of-US-
Pharmaceutical-Trademark-Clearance-Research/
―Yet trademark law is supposed to be subjective
in the sense of being based on personal
opinions—it is just that the relevant opinion is
that of the average consumer [...]‖
Lisa Larrimore Ouellette (forthcoming): The Google Shortcut to Trademark law
http://ssrn.com/abstract=2195989
2013-06-12
Anna Ronkainen - @ronkaine
Intelligent Trademark Analysis
3
The application
TrademarkNow™
NameCheck™
• a system for trademark risk analysis
• relevancy-ranked trademark search
• based on a model of trademark (mark
and product) similarity, derived from (but
not directly based on) MOSONG
(Ronkainen 2010)
• additional features for absolute
grounds, word meanings etc.
• current name since March 2013
• originally launched as Onomatics Quick
Search (2012) without risk analysis
• web service, currently free preview but
otherwise by subscription only
2013-06-12
Anna Ronkainen - @ronkaine
Intelligent Trademark Analysis
5
Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013)

Recommended

Technology can transform the law – but only if done right (ReInvent Law Londo...
Technology can transform the law – but only if done right (ReInvent Law Londo...Technology can transform the law – but only if done right (ReInvent Law Londo...
Technology can transform the law – but only if done right (ReInvent Law Londo...Anna Ronkainen
 
Product management (at Boost Turku Startup Journey 2015)
Product management (at Boost Turku Startup Journey 2015)Product management (at Boost Turku Startup Journey 2015)
Product management (at Boost Turku Startup Journey 2015)Anna Ronkainen
 
Commercializing legal AI research: lessons learned
Commercializing legal AI research: lessons learnedCommercializing legal AI research: lessons learned
Commercializing legal AI research: lessons learnedAnna Ronkainen
 
Introduction to Legal Technology, lecture 3 (2015)
Introduction to Legal Technology, lecture 3 (2015)Introduction to Legal Technology, lecture 3 (2015)
Introduction to Legal Technology, lecture 3 (2015)Anna Ronkainen
 
From Research to Innovative Legal Tech Products
From Research to Innovative Legal Tech ProductsFrom Research to Innovative Legal Tech Products
From Research to Innovative Legal Tech ProductsAnna Ronkainen
 
Introduction to Legal Technology, lecture 7 (2015)
Introduction to Legal Technology, lecture 7 (2015)Introduction to Legal Technology, lecture 7 (2015)
Introduction to Legal Technology, lecture 7 (2015)Anna Ronkainen
 
Introduction to Legal Technology, lecture 4 (2015)
Introduction to Legal Technology, lecture 4 (2015)Introduction to Legal Technology, lecture 4 (2015)
Introduction to Legal Technology, lecture 4 (2015)Anna Ronkainen
 
Introduction to Legal Technology, lecture 1 (2015)
Introduction to Legal Technology, lecture 1 (2015)Introduction to Legal Technology, lecture 1 (2015)
Introduction to Legal Technology, lecture 1 (2015)Anna Ronkainen
 

More Related Content

Similar to Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013)

Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...InsightInnovation
 
Sbdc2018 master slidedeck-final
Sbdc2018 master slidedeck-finalSbdc2018 master slidedeck-final
Sbdc2018 master slidedeck-finalFreek Bomhof
 
[Jennifer vu huong]continuous improvement
[Jennifer vu huong]continuous improvement[Jennifer vu huong]continuous improvement
[Jennifer vu huong]continuous improvementJen Vuhuong
 
18 Jul 2018 - FMEA and Risk Management in Practice
18 Jul 2018 - FMEA and Risk Management in Practice 18 Jul 2018 - FMEA and Risk Management in Practice
18 Jul 2018 - FMEA and Risk Management in Practice Intland Software GmbH
 
Supply & Demand Chain Executive Magazine
Supply & Demand Chain Executive MagazineSupply & Demand Chain Executive Magazine
Supply & Demand Chain Executive MagazineKristal Snider
 
ICIC 2013 Conference Proceedings Sumair Riyaz Dolcera
ICIC 2013 Conference Proceedings Sumair Riyaz DolceraICIC 2013 Conference Proceedings Sumair Riyaz Dolcera
ICIC 2013 Conference Proceedings Sumair Riyaz DolceraDr. Haxel Consult
 
Setting the supply chain standard slide share
Setting the supply chain standard   slide shareSetting the supply chain standard   slide share
Setting the supply chain standard slide shareKristal Snider
 
Data Science Application in Business Portfolio & Risk Management
Data Science Application in Business Portfolio & Risk ManagementData Science Application in Business Portfolio & Risk Management
Data Science Application in Business Portfolio & Risk ManagementData Science Thailand
 
Veolia Open Innovation Challenge - Sulfide Sensors
Veolia Open Innovation Challenge - Sulfide SensorsVeolia Open Innovation Challenge - Sulfide Sensors
Veolia Open Innovation Challenge - Sulfide SensorsJean Grangier Sarfati
 
NI Automated Test Outlook 2016
NI Automated Test Outlook 2016NI Automated Test Outlook 2016
NI Automated Test Outlook 2016Hank Lydick
 
STS. Smarter devices. Smarter test systems.
STS. Smarter devices. Smarter test systems.STS. Smarter devices. Smarter test systems.
STS. Smarter devices. Smarter test systems.Hank Lydick
 
WhiteSource Webinar-New Research Reveals Key Strategy to Manage Open Source S...
WhiteSource Webinar-New Research Reveals Key Strategy to Manage Open Source S...WhiteSource Webinar-New Research Reveals Key Strategy to Manage Open Source S...
WhiteSource Webinar-New Research Reveals Key Strategy to Manage Open Source S...WhiteSource
 
GSIPA2M, Roundtable 4, Challenging unmerited patents - Georgina Gerde
GSIPA2M, Roundtable 4, Challenging unmerited patents - Georgina GerdeGSIPA2M, Roundtable 4, Challenging unmerited patents - Georgina Gerde
GSIPA2M, Roundtable 4, Challenging unmerited patents - Georgina GerdeMakeMedicinesAffordable
 
How to proactively manage your MEMS IP (or any IP for that matter), by Sergio...
How to proactively manage your MEMS IP (or any IP for that matter), by Sergio...How to proactively manage your MEMS IP (or any IP for that matter), by Sergio...
How to proactively manage your MEMS IP (or any IP for that matter), by Sergio...Fides Sales
 
Questel Halliburton Aug2 2016 RD (Webinar)
Questel Halliburton Aug2 2016 RD (Webinar)Questel Halliburton Aug2 2016 RD (Webinar)
Questel Halliburton Aug2 2016 RD (Webinar)Eric Moran
 
Spider Patent Search - New Tool for Mining Intelligence from Patent Information
Spider Patent Search - New Tool for Mining Intelligence from Patent InformationSpider Patent Search - New Tool for Mining Intelligence from Patent Information
Spider Patent Search - New Tool for Mining Intelligence from Patent InformationRuchica Kumar
 
OECD workshop on measuring the link between public procurement, R&D and innov...
OECD workshop on measuring the link between public procurement, R&D and innov...OECD workshop on measuring the link between public procurement, R&D and innov...
OECD workshop on measuring the link between public procurement, R&D and innov...STIEAS
 
Marketing Project Presentation
Marketing Project PresentationMarketing Project Presentation
Marketing Project PresentationAllison Caparros
 

Similar to Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013) (20)

Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
Data Quality Doesn’t Just Happen: And Here’s What Some of the Industry’s Most...
 
Sbdc2018 master slidedeck-final
Sbdc2018 master slidedeck-finalSbdc2018 master slidedeck-final
Sbdc2018 master slidedeck-final
 
[Jennifer vu huong]continuous improvement
[Jennifer vu huong]continuous improvement[Jennifer vu huong]continuous improvement
[Jennifer vu huong]continuous improvement
 
2016 virus bulletin
2016 virus bulletin2016 virus bulletin
2016 virus bulletin
 
18 Jul 2018 - FMEA and Risk Management in Practice
18 Jul 2018 - FMEA and Risk Management in Practice 18 Jul 2018 - FMEA and Risk Management in Practice
18 Jul 2018 - FMEA and Risk Management in Practice
 
Supply & Demand Chain Executive Magazine
Supply & Demand Chain Executive MagazineSupply & Demand Chain Executive Magazine
Supply & Demand Chain Executive Magazine
 
ICIC 2013 Conference Proceedings Sumair Riyaz Dolcera
ICIC 2013 Conference Proceedings Sumair Riyaz DolceraICIC 2013 Conference Proceedings Sumair Riyaz Dolcera
ICIC 2013 Conference Proceedings Sumair Riyaz Dolcera
 
Setting the supply chain standard slide share
Setting the supply chain standard   slide shareSetting the supply chain standard   slide share
Setting the supply chain standard slide share
 
Data Science Application in Business Portfolio & Risk Management
Data Science Application in Business Portfolio & Risk ManagementData Science Application in Business Portfolio & Risk Management
Data Science Application in Business Portfolio & Risk Management
 
Veolia Open Innovation Challenge - Sulfide Sensors
Veolia Open Innovation Challenge - Sulfide SensorsVeolia Open Innovation Challenge - Sulfide Sensors
Veolia Open Innovation Challenge - Sulfide Sensors
 
NI Automated Test Outlook 2016
NI Automated Test Outlook 2016NI Automated Test Outlook 2016
NI Automated Test Outlook 2016
 
STS. Smarter devices. Smarter test systems.
STS. Smarter devices. Smarter test systems.STS. Smarter devices. Smarter test systems.
STS. Smarter devices. Smarter test systems.
 
WhiteSource Webinar-New Research Reveals Key Strategy to Manage Open Source S...
WhiteSource Webinar-New Research Reveals Key Strategy to Manage Open Source S...WhiteSource Webinar-New Research Reveals Key Strategy to Manage Open Source S...
WhiteSource Webinar-New Research Reveals Key Strategy to Manage Open Source S...
 
GSIPA2M, Roundtable 4, Challenging unmerited patents - Georgina Gerde
GSIPA2M, Roundtable 4, Challenging unmerited patents - Georgina GerdeGSIPA2M, Roundtable 4, Challenging unmerited patents - Georgina Gerde
GSIPA2M, Roundtable 4, Challenging unmerited patents - Georgina Gerde
 
How to proactively manage your MEMS IP (or any IP for that matter), by Sergio...
How to proactively manage your MEMS IP (or any IP for that matter), by Sergio...How to proactively manage your MEMS IP (or any IP for that matter), by Sergio...
How to proactively manage your MEMS IP (or any IP for that matter), by Sergio...
 
[Paul Holland] Bad Metrics and What You Can Do About It
[Paul Holland] Bad Metrics and What You Can Do About It[Paul Holland] Bad Metrics and What You Can Do About It
[Paul Holland] Bad Metrics and What You Can Do About It
 
Questel Halliburton Aug2 2016 RD (Webinar)
Questel Halliburton Aug2 2016 RD (Webinar)Questel Halliburton Aug2 2016 RD (Webinar)
Questel Halliburton Aug2 2016 RD (Webinar)
 
Spider Patent Search - New Tool for Mining Intelligence from Patent Information
Spider Patent Search - New Tool for Mining Intelligence from Patent InformationSpider Patent Search - New Tool for Mining Intelligence from Patent Information
Spider Patent Search - New Tool for Mining Intelligence from Patent Information
 
OECD workshop on measuring the link between public procurement, R&D and innov...
OECD workshop on measuring the link between public procurement, R&D and innov...OECD workshop on measuring the link between public procurement, R&D and innov...
OECD workshop on measuring the link between public procurement, R&D and innov...
 
Marketing Project Presentation
Marketing Project PresentationMarketing Project Presentation
Marketing Project Presentation
 

More from Anna Ronkainen

Introduction to Legal Technology, lecture 9 (2018)
Introduction to Legal Technology, lecture 9 (2018)Introduction to Legal Technology, lecture 9 (2018)
Introduction to Legal Technology, lecture 9 (2018)Anna Ronkainen
 
Creating products that lawyers love (sic!) – design in legal technology
Creating products that lawyers love (sic!) – design in legal technologyCreating products that lawyers love (sic!) – design in legal technology
Creating products that lawyers love (sic!) – design in legal technologyAnna Ronkainen
 
AI in legal practice – the research perspective
AI in legal practice – the research perspectiveAI in legal practice – the research perspective
AI in legal practice – the research perspectiveAnna Ronkainen
 
Finnish Legal Tech Forum launch presentation
Finnish Legal Tech Forum launch presentationFinnish Legal Tech Forum launch presentation
Finnish Legal Tech Forum launch presentationAnna Ronkainen
 
Modeling meaning and knowledge: legal knowledge
Modeling meaning and knowledge: legal knowledgeModeling meaning and knowledge: legal knowledge
Modeling meaning and knowledge: legal knowledgeAnna Ronkainen
 
Ethical machines: data mining and fairness – the optimistic view
Ethical machines: data mining and fairness – the optimistic viewEthical machines: data mining and fairness – the optimistic view
Ethical machines: data mining and fairness – the optimistic viewAnna Ronkainen
 
Tavaramerkki. NYT! IPR-palveluihin uusi ulottuvuus
Tavaramerkki. NYT! IPR-palveluihin uusi ulottuvuusTavaramerkki. NYT! IPR-palveluihin uusi ulottuvuus
Tavaramerkki. NYT! IPR-palveluihin uusi ulottuvuusAnna Ronkainen
 
Helsinki Legal Tech Meetup: TrademarkNow demo/presentation
Helsinki Legal Tech Meetup: TrademarkNow demo/presentationHelsinki Legal Tech Meetup: TrademarkNow demo/presentation
Helsinki Legal Tech Meetup: TrademarkNow demo/presentationAnna Ronkainen
 
What is legal technology?
What is legal technology?What is legal technology?
What is legal technology?Anna Ronkainen
 
Helsinki Legal Tech Meetup introduction
Helsinki Legal Tech Meetup introductionHelsinki Legal Tech Meetup introduction
Helsinki Legal Tech Meetup introductionAnna Ronkainen
 
Tulevaisuus on jo täällä
Tulevaisuus on jo täälläTulevaisuus on jo täällä
Tulevaisuus on jo täälläAnna Ronkainen
 
Product management – what makes or breaks a startup
Product management – what makes or breaks a startupProduct management – what makes or breaks a startup
Product management – what makes or breaks a startupAnna Ronkainen
 
How to do things with AI & law research
How to do things with AI & law researchHow to do things with AI & law research
How to do things with AI & law researchAnna Ronkainen
 
Tietokone korvaa juristin – vai korvaako?
Tietokone korvaa juristin – vai korvaako?Tietokone korvaa juristin – vai korvaako?
Tietokone korvaa juristin – vai korvaako?Anna Ronkainen
 
TrademarkNow (and its research background)
TrademarkNow (and its research background)TrademarkNow (and its research background)
TrademarkNow (and its research background)Anna Ronkainen
 
Introduction to legal design: Product & project management
Introduction to legal design: Product & project managementIntroduction to legal design: Product & project management
Introduction to legal design: Product & project managementAnna Ronkainen
 
Introduction to Legal Technology, lecture 10 (2015)
Introduction to Legal Technology, lecture 10 (2015)Introduction to Legal Technology, lecture 10 (2015)
Introduction to Legal Technology, lecture 10 (2015)Anna Ronkainen
 
Introduction to Legal Technology, lecture 9 (2015)
Introduction to Legal Technology, lecture 9 (2015)Introduction to Legal Technology, lecture 9 (2015)
Introduction to Legal Technology, lecture 9 (2015)Anna Ronkainen
 
Introduction to Legal Technology, lecture 8 (2015)
Introduction to Legal Technology, lecture 8 (2015)Introduction to Legal Technology, lecture 8 (2015)
Introduction to Legal Technology, lecture 8 (2015)Anna Ronkainen
 
Introduction to Legal Technology, lecture 6 (2015)
Introduction to Legal Technology, lecture 6 (2015)Introduction to Legal Technology, lecture 6 (2015)
Introduction to Legal Technology, lecture 6 (2015)Anna Ronkainen
 

More from Anna Ronkainen (20)

Introduction to Legal Technology, lecture 9 (2018)
Introduction to Legal Technology, lecture 9 (2018)Introduction to Legal Technology, lecture 9 (2018)
Introduction to Legal Technology, lecture 9 (2018)
 
Creating products that lawyers love (sic!) – design in legal technology
Creating products that lawyers love (sic!) – design in legal technologyCreating products that lawyers love (sic!) – design in legal technology
Creating products that lawyers love (sic!) – design in legal technology
 
AI in legal practice – the research perspective
AI in legal practice – the research perspectiveAI in legal practice – the research perspective
AI in legal practice – the research perspective
 
Finnish Legal Tech Forum launch presentation
Finnish Legal Tech Forum launch presentationFinnish Legal Tech Forum launch presentation
Finnish Legal Tech Forum launch presentation
 
Modeling meaning and knowledge: legal knowledge
Modeling meaning and knowledge: legal knowledgeModeling meaning and knowledge: legal knowledge
Modeling meaning and knowledge: legal knowledge
 
Ethical machines: data mining and fairness – the optimistic view
Ethical machines: data mining and fairness – the optimistic viewEthical machines: data mining and fairness – the optimistic view
Ethical machines: data mining and fairness – the optimistic view
 
Tavaramerkki. NYT! IPR-palveluihin uusi ulottuvuus
Tavaramerkki. NYT! IPR-palveluihin uusi ulottuvuusTavaramerkki. NYT! IPR-palveluihin uusi ulottuvuus
Tavaramerkki. NYT! IPR-palveluihin uusi ulottuvuus
 
Helsinki Legal Tech Meetup: TrademarkNow demo/presentation
Helsinki Legal Tech Meetup: TrademarkNow demo/presentationHelsinki Legal Tech Meetup: TrademarkNow demo/presentation
Helsinki Legal Tech Meetup: TrademarkNow demo/presentation
 
What is legal technology?
What is legal technology?What is legal technology?
What is legal technology?
 
Helsinki Legal Tech Meetup introduction
Helsinki Legal Tech Meetup introductionHelsinki Legal Tech Meetup introduction
Helsinki Legal Tech Meetup introduction
 
Tulevaisuus on jo täällä
Tulevaisuus on jo täälläTulevaisuus on jo täällä
Tulevaisuus on jo täällä
 
Product management – what makes or breaks a startup
Product management – what makes or breaks a startupProduct management – what makes or breaks a startup
Product management – what makes or breaks a startup
 
How to do things with AI & law research
How to do things with AI & law researchHow to do things with AI & law research
How to do things with AI & law research
 
Tietokone korvaa juristin – vai korvaako?
Tietokone korvaa juristin – vai korvaako?Tietokone korvaa juristin – vai korvaako?
Tietokone korvaa juristin – vai korvaako?
 
TrademarkNow (and its research background)
TrademarkNow (and its research background)TrademarkNow (and its research background)
TrademarkNow (and its research background)
 
Introduction to legal design: Product & project management
Introduction to legal design: Product & project managementIntroduction to legal design: Product & project management
Introduction to legal design: Product & project management
 
Introduction to Legal Technology, lecture 10 (2015)
Introduction to Legal Technology, lecture 10 (2015)Introduction to Legal Technology, lecture 10 (2015)
Introduction to Legal Technology, lecture 10 (2015)
 
Introduction to Legal Technology, lecture 9 (2015)
Introduction to Legal Technology, lecture 9 (2015)Introduction to Legal Technology, lecture 9 (2015)
Introduction to Legal Technology, lecture 9 (2015)
 
Introduction to Legal Technology, lecture 8 (2015)
Introduction to Legal Technology, lecture 8 (2015)Introduction to Legal Technology, lecture 8 (2015)
Introduction to Legal Technology, lecture 8 (2015)
 
Introduction to Legal Technology, lecture 6 (2015)
Introduction to Legal Technology, lecture 6 (2015)Introduction to Legal Technology, lecture 6 (2015)
Introduction to Legal Technology, lecture 6 (2015)
 

Recently uploaded

Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31shyamraj55
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceVijayananda Mohire
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotelPhilippines
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanDatabarracks
 
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...htrindia
 
Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfMostafa Higazy
 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...Neo4j
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceSusan Ibach
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?MENGSAYLOEM1
 
How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxInfosec
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsInflectra
 
"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys Vasyliev"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys VasylievFwdays
 
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...DianaGray10
 
"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura RochniakFwdays
 
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...Product School
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxNeo4j
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...ISPMAIndia
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVARobert McDermott
 

Recently uploaded (20)

In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...
 
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial Intelligence
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company Profile
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response Plan
 
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
 
Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdf
 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data science
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?
 
How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptx
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
 
"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys Vasyliev"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys Vasyliev
 
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
 
"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak
 
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVA
 

Intelligent Trademark Analysis: Experiments in Large-Scale Evaluation of Real-World Legal AI (ICAIL 2013)

  • 1. Intelligent Trademark Analysis Experiments in Large-Scale Evaluation of Real-World Legal AI Anna Ronkainen Chief Scientist, Onomatics, Inc. miscellaneous, University of Helsinki anna.ronkainen@helsinki.fi @ronkaine
  • 2. ―These papers had a low rate of consideration of evaluation issues reflecting common practice in research biased development environments. These results confirm that more attention to evaluation is needed in the legal knowledge based systems domain.‖ Hall & Zeleznikow (ICAIL 2001): Acknowledging Insufficiency in the Evaluation of Legal Knowledge-based Systems: Strategies Towards a Broad-based Evaluation Model 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 2
  • 3. ―Relevancy ranking is currently a foreign concept in the trademark legal industry, as assigning numeric weight to potentially conflicting citations based on objective computer analysis is currently avant-garde.‖ Anderson & Cary (2010): Navigating the Challenges of U.S. Pharmaceutical Trademark Clearance Research http://www.pharmpro.com/articles/2010/04/Navigating-the-Challenges-of-US- Pharmaceutical-Trademark-Clearance-Research/ ―Yet trademark law is supposed to be subjective in the sense of being based on personal opinions—it is just that the relevant opinion is that of the average consumer [...]‖ Lisa Larrimore Ouellette (forthcoming): The Google Shortcut to Trademark law http://ssrn.com/abstract=2195989 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 3
  • 5. TrademarkNow™ NameCheck™ • a system for trademark risk analysis • relevancy-ranked trademark search • based on a model of trademark (mark and product) similarity, derived from (but not directly based on) MOSONG (Ronkainen 2010) • additional features for absolute grounds, word meanings etc. • current name since March 2013 • originally launched as Onomatics Quick Search (2012) without risk analysis • web service, currently free preview but otherwise by subscription only 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 5
  • 9. TrademarkNow™ NameRank™ • a tool for ranking 2–5 trademark candidates to find the least risky one • same basic technology as NameCheck™ • not actively offered at the moment 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 9
  • 11. Evaluation: MOSONG First round: 30 most recent (2002) relevant cases: • 20 from the Opposition Division and • 10 from the Boards of Appeal Result*: all cases predicted correctly * when coded into the system by a domain expert 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 11
  • 12. Evaluation: MOSONG Second round: Non-expert validation: • done by non-law students taking a course in intellectual property law (n=75) • original validation set in two parts (15+15 cases) • at the beginning and the end of the course • completed non-interactively through a web form • correct answer: 54.6±6.5% • incorrect answer: 25.9±7.5% • no answer: 19.5±5.2% (± = σ) 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 12
  • 13. Evaluation: MOSONG % ±stderr before after total group 1 (n=15) 41.3±1.7 65.8±2.8 53.5±1.7 group 2 (n=12) 46.1±2.0 65.0±3.0 55.6±1.9 group 3 (n=48) 43.3±1.3 65.9±1.3 54.7±0.9 total (n=75) 43.4±1.0 65.8±1.1 54.6±0.8 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 13 Second round: Non-expert validation:
  • 14. Testing NameCheck™ • system must work in real-world conditions ⇒ has to be robust • as wide a range of realistic test cases as possible is desirable • system is based on a predictive model for trademark opposition cases so such cases can be used (almost) directly for testing • specific service subcomponent for evaluating cases in bulk: text file input but otherwise same process • inputs: mark and plain-text product(s) 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 14
  • 15. Test cases • closed, unappealed first-instance opposition cases • only those where the prior right is found in the database: – same jurisdiction – still valid – word or combination mark • typically multiple prior rights per case • readily available as XML (but USPTO TTAB XML is write-only) • grand total: 55435 cases 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 15
  • 16. Test cases: EU • source: OHIM (EU TM office) Opposition Division • for marks filed in 1996–2011 • opposition considered successful: – opposition upheld (1563 cases) – mark withdrawn (10516 cases) • opposition unsuccessful: – opposition rejected (4217 cases) – opposition withdrawn (5368 cases) 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 16
  • 17. Test cases: US • source: USPTO Trademark Trial and Appeal Board (TTAB) • for marks filed in 1980, 1985, 1990, 1995– 2011 • opposition successful – opposition upheld (18431 cases) – mark withdrawn (7329 cases) • opposition unsuccessful: – opposition dismissed (8897 cases) – opposition withdrawn (8603 cases) • categories and outcomes determined heuristically so some overlap (duplicates removed for the grand totals) 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 17
  • 18. Performance criteria • result considered correct based on the highest similarity score for all prior rights iff – opposition successful for at least some of the goods and services and score ≥ 0.5 – opposition unsuccessful and score < 0.5 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 18
  • 19. Overall performance • EU (21162 cases): precision 79.2%, recall 97.3%, F 87.3% • US (33773 cases): precision 80.4%, recall 93.3%, F 86.4% • total (55435 cases): precision 79.9%, recall 94.9%, F 86.7% • low-risk cases hurt precision: erring on the side of safety 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 19
  • 20. High-risk performance Case type n listed in top 10 listed in top 100 correct decision EU: opposition upheld 1473 544 36.9% 921 62.5% 1425 96.7% EU: opposition withdrawn 9515 4836 50.8% 6865 72.1% 9261 97.3% US: opposition upheld 18409 5618 30.5% 9955 54.1% 17138 93.1% US: opposition withdrawn 7278 2367 32.5% 4129 56.7% 6776 93.1% 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 20
  • 21. What is this good for (absolutely nothing?) • necessary but not sufficient • inputs quite noisy: many cases decided on factors other than likelihood of confusion: reviewing all of them and removing the noise the only way to >99 % (not going to happen) • mainly useful for regression testing (but too slow for that, we mostly use two 1000-case samples) • only measures one particular aspect of system performance (and not the most relevant one) • additional specific test case sets needed for various elements in similarity ordering: developed (and under development) from scratch, typically based on actual marks • to be continued... 2013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis 21
  • 22. Thank you! Further reading: • @ronkaine on Twitter • http://www.legalfuturology.com/ (my research blog) • http://blog.onomatics.com/ • Ronkainen, Anna (2010): MOSONG, a Fuzzy Logic Model of Trade Mark Similarity http://ssrn.com/abstract=1879399 • Ronkainen, Anna (2013): Redefining Trademark Clearance with Intelligent Legal Technology. IPRinfo 1/2013. http://blog.onomatics.com/2013/02/redefining- trademark-clearance-with.html • Ronkainen, Anna (forthcoming): Scaling Intelligent Trademark Analysis from Prototype to Production: From MOSONG to Onomatics Quick Search. Presented at the 1st International Workshop on Artificial Intelligence and Intellectual Property. 222013-06-12 Anna Ronkainen - @ronkaine Intelligent Trademark Analysis

Editor's Notes

  1. product
  2. That’s it, thanks for listening!