This document discusses evaluating legal vocabularies using competency questions. It presents a model-driven architecture approach to represent legal knowledge using concepts, facts, and rules in a semi-formal language called SBVR. Competency questions are designed to verify the accuracy, completeness, and consistency of the legal vocabulary. The questions are translated into formal queries to check inferences from the ontology. Experimental results show how competency questions can identify incomplete or sensitive concepts. The legal rules represented in the vocabulary can then be used to generate arguments in a legal expert system.

1. 1
Legal Vocabulary and its Transformation Evaluation
using Competency Questions
15th International Conference on AI and Law- ICAIL 2015
11-June 2015
San Diego, CA, USA
Shashishekar Ramakrishna, Łukasz Górski and Adrian Paschke

2. 2
In-C Tests : Processing Model
2
Introduction
Legal
Expert
System

3. 3
In-C Tests : Processing Model
3
Model Driven Architecture approach to Legal KR
► OMG : Object Management Group (OMG)
► Founded in 1989 by 11 companies (IBM, Apple, Sun Microsystems etc.)
► Not-for-profit computer industry standards consortium
►Model Driven Architecture

4. 4
Informal to Semi-Formal via Legal Decision Models
35 U.S.C. 112, 1st Paragraph,- Specification
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear,
concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and
use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
[Ramakrishna , Gorski and Paschke: ESWC Workshop 2015]

5. 5
In-C Tests : Processing Model
5
Semi-Formal Legal KR via SBVR
SBVR: Semantics of Business Vocabulary and Business Rules
► Sits on CIM layer of OMG’s MDA (OMG’s standard since 2005, )
► To support computationalindependent modeling of business rules by
business people in business language
Two Meta-models defined in Form of vocabularies
► Vocabulary for Describing Business Vocabularies
► Vocabulary for Describing Business Rules

6. 6
In-C Tests : Processing Model
6
SBVR in Legal Domain:
► Legal Concept: (individual concept, object type )
► e.g. patent, claim, examiner
► Fact type: denoting some relationship b/w 2 or more noun concept
► e.g. patent includes claim
► Legal (procedural/substantive) rule: a rule under legal jurisdiction
► e.g. It is obligatory that a patent includes atleast 1 claim.

7. 7
Semi-Formal Representation
¶ 7.33.01 Rejection : Claim [1] rejected under 35 U.S.C. 112, first paragraph, as based on a disclosure which is not enabling. [2] critical or
essentialto the practice of the invention, but not included in the claim(s) is not enabled by the disclosure.
1. This rejection must be preceded by form paragraph 7.30.01 or 7.103.
2. In bracket 2, recite the subject matter omitted from the claims.
Legal Concepts
Claim, office_action, Paragraphs, essential_subject_matter_requirement, Paragarph_7_33_01
Enables identification of associated context information and relations between them.
Context information
for each legal concept
E.g.:
KR4IPLaw Legal Concept
Recommender System

8. 8
In-C Tests : Processing Model
8
Semi-Formal Representation contd…
Legal Facts (verb concepts are defined in blue)
office_action includes paragraphs
claim is_rejected_under essential_subject_matter_requirement
office_action include statement
applicant conceals effective_feature
effective_feature is_about the invention
examiner applies Paragraph_7_33_01
examiner rejects the claim
Legal (procedural) rules (for ¶ 7.33.01) in Structured English:
1. It is obligatory that examiner rejects the claim and office_action includes paragraphs
Paragraph_7_33_01 if claim is_rejected_under essential_subject_matter_requirement
2. It is obligatory that office_action include statement and argument and date and
drawing if claim is_rejected_under Paragraph_7_33_01
3. It is necessary that examiner applies Paragraph_7_33_01 if applicant conceals
effective_feature and effective_feature is_about the invention

9. 9
In-C Tests : Processing Model
9
Semi-Formal to Formal Representation
Legal Rules in SBVR-SELegal Facts
OWL2

10. 10
In-C Tests : Processing Model
10
Legal Vocabulary Evaluation ?
Increases confidence in using the inferred knowledge in the form of legal arguments.
Goal is to eliminate black boxes (from end-users perspective) in a (Semi-/) automated decision support
systems.

11. 11
Competency Question Design: Actors
Content Specialist
Design Specialist
Interviewer
Respondent
Legal practitioner
Knowledge Modeler
Ontology query engine
Ontology
Creates a set of competency questions
defined in semi-formal language.
Performs the translation into formal
query language

12. 12
Building the
system right
Accuracy
Adaptability
Clarity
Completeness
Computational
Efficiency
Conciseness
Consistency
Organizational
fitness
Legal Vocabulary Evaluation: Verification

13. 13
Verification
Accuracy
Adaptability
Clarity
Completeness
Computational
Efficiency
Conciseness
Consistency
Organizational
fitness
Verification contd…

14. 14
Building the
right system
(In-)
consistency
(In-)
completeness
(In-)
conciseness
(In-)
sensitiveness
Legal Vocabulary Evaluation :Validation

15. 15
Validation
(In-)
consistency
(In-)
completeness
(In-)
conciseness
(In-)
sensitiveness
Validation contd…

16. 16
CQ-Based Evaluation: The Process

17. 17
CQ based Evaluation: Formal Definition
Then:
Each CQ should address two generalized conditions:
Formal CQs can be defined formally as entailment or consistency problem wrt axioms in ontology
If
Tontology = a set of axioms
Tground = a set of ground literals
Q = a first-order legal sentence in the language of Tontology
a) Determine
if Tontology ∪ Tground ⊨ Q?
b) Determine
if Tontology ∪ Tground ⊭ ¬Q?
=> Transformation is correct and verifiable

18. 18
Sample CQs Pattern
Selection Question Module
What is a claim? Define a X
Where is the language of a claim W is/are the X of Y (W ∈ {Where, What, Who}
What is a relation between claim and a patent What is a relation between X and Y?
What does patent include What does X R?
Binary Question Module
Is the patent application filed in US? Is/Are X relation Y?
Does the patent application include a claim? Do/Does X relation Y?
Counting Question Module
What is the minimum number of claims in a patent
application?
The Q of X in Y? (Q∈{number, min, max}
CQs and Patterns

19. 19
Formal CQ Pattern
Selection Question Module
[skos:definition][CE]? SELECT ?LegalConcept ?definition
WHERE { ?LegalConcept a owl:Class;
Define a claim skos:definition ?definition . }
[OPE][CE1][CE2]? processQuery("PREFIX : <abc#> n" + "SELECT ?x ?y
WHERE {n" + PropertyValue(?x,:OPE, ?y)"+"}")
[CE1][OPE]=[CE2]? SELECT { ?y WHERE {n" + PropertyValue(CE1,:OPE,
?y)"+"};
Binary Question Module
[OPE][CE1][CE2]=Y/N? ASK {:OPE rdfs:domain :CE1 ; rdfs:range :CE2.}
Counting Question Module
[OPE@min/maxCardinality][CE1][CE2]? SELECT ?cardinality
WHERE { <http://abc> owl:equivalentClass ?c. ?c
owl:qualifiedCardinality ?cardinality }
Formal CQs mapped to DL based queries

20. 20
Experimental Results
Overall knowledge (in)completeness within the legal vocabulary for our consideredexample
Legal vocabulary may be regarded as incomplete, if the legal concepts do not contain of all
the necessary definitions, some defined formally and some via its annotation properties

21. 21
Experimental Results
Legal knowledge (in-)completeness for indvidual legal concepts

22. 22
Experimental Results
Permissible sensitivity inversely proportional to Threshold
E.g. : offce_action central legal concept
Modification to legal concept office_action → effects several other concepts

23. 23
Legal Rules: Platform Independent Layer: KR4IPLaw

24. 24
Legal Rules: Platform Specific Layer: Prova
Rule Engine
A rule is evaluated only if guard conditions are evaluated to be True
Verification and Validation of legal knowledge increases the clarity of legal arguments generated by a
Legal Expert System

25. 25
Further Reading:
• Ramakrishna, Gorski and Paschke, 2015. The Role of Pragmatics in Legal Norm Representation. International Workshop On
Legal Domain And Semantic Web Applications, ESWC 2015
• Ramakrishna, Paschke 2014 . A Process for Knowledge Transformation and Knowledge Representation of Patent Law.
RuleML – 2014
• Paschke, 2006 . Verification, validation and integrity of distributed and interchanged rule based policies and contracts
in the semantic web.
• Ramakrishna and Paschke. Semi-Automated Vocabulary Building for Structured Legal English. In RuleML 2014, volume
8620 of LNCS. Springer, 2014.

26. 26
Thank you
Shashishekar Ramakrishna
Free University of Berlin,
Dept of Mathematics and Computer Science
shashi792@gmail.com
Łukasz Górski
Nicolaus Copernicus University
lgorski@mat.umk.pl
Adrian Paschke
Free University of Berlin,
Dept of Mathematics and Computer Science
paschke@inf-fu-berlin.de