Talk on 14th June 2017 on the LegalHackers Copenhagen Meetup about the use of topic modeling as a tool to study and intrepret the case-law of International Courts

1. through Topic modeling
and other animals
Yannis Panagis,
LegalHackersCPH,
15 June 2017
Attempts to
study the case-
law of Intl Courts
ioannis.panagis@jur.ku.dk
ypanagis

2. International courts
18/06/2017 2
International courts are formed by treaties
between nations or under the authority of an
international organization such as the United
Nations and include ad hoc tribunals and
permanent institutions but exclude any courts
arising purely under national authority.

3. International courts (2)
• International Court of Justice
• European Court on Human Rights
• Court of Justice of the EU
• International Criminal Court
• International Criminal Tribunal for the former
Yugoslavia
• International Criminal Tribunal for Rwanda
• Interamerican court of Human rights
• East African court of justice
Etc.
18/06/2017 3

4. • Court of Justice of the EU
(CJEU):
• Founded 1952
• 28 (or rather 27) member
states
• ~11000 judgments
Court of Justice of the EU
18/06/2017 4

5. What are the options for the researchers?
18/06/2017 5
• Keep calm and study on
• Focus on selected cases or
“landmark cases”
• Select a manageable subset of case
law they can explore and study
• Get funding for human coders to
read and analyse everything
http://depts.washington.edu/echrdb

6. Machine Learning
• Classification:
• Classify data into pre-specified categories, e.g. Tweets into
positive negative
• Supervised
• Label a data subset manually – Training set
• Train a machine learning algorithm on the training set to
recognize unseen data
• Unsupervised
• Skip manual labelling.
• Define the number of classes and let the algorithm figure out
18/06/2017 6

7. Topic modeling
• Unsupervised machine learning for text
• Classify a text collection into a number of categories (topics)
• Each topic a collection of words that the algorithm believes it
best represents the topic.
• Each topic a probability that a word belongs to a topic
(Distribution)
• The documents are also assigned a probability to belong to
each of the topics
18/06/2017 7

8. First use-case
18/06/2017 8
Source: https://www.slideshare.net/anikobokornekovacs/eu-citizenship-53296485
Proceedings of Conf. on Legal Knowl.
& Inf. Systems, JURIX 2015

9. Fundamental questions
Courts are citing precedent cases:
24. The situations falling within the scope ratione materiae of
Community law include those involving the exercise of the
fundamental freedoms guaranteed by the Treaty, in particular
those involving the freedom to move and reside within the
territory of the Member States, as conferred by Article 18 EC
(Case C-274/96 Bickel and Franz [1998] ECR I-7637, paragraphs
15 and 16, Grzelczyk , cited above, paragraph 33, and D'Hoop ,
paragraph 29).
(Case C-148/02, Garcia Avello, par. 24)
• Why is the Court of Justice citing?
• What is the force of the citations?
18/06/2017 9

10. Data and methods
18/06/2017 10
• Isolated 120 cases from the CJEU about ”EU citizenship”, “Citizenship of
the Union”.
• Extracted the paragraphs containing citations
• Used GATE Developer to create rules that detect annotate the citations in
the text
• Processed citations and cited paragraphs and stored results in MySQL
• Visualized with Gephi

11. Use of GATE
• Constructed rules to match the paragraph citation
patterns of the court, e.g.
• […]in particular Case 66/85 Lawrie-Blum [1986] ECR 2121,
paragraphs 16 and 17
• Problem:
• How to detect this?
paragraphs 15 and 16, Grzelczyk , cited above, paragraph 33,
and D'Hoop , paragraph 29).
• Gazetteer with case names or pattern matching
18/06/2017 11

12. Outcome
18/06/2017 12
Case ids
helping us to
identify cited
cases

13. Network
• 2633 nodes –
paragraphs
• 2064 edges - citations
• Sparse
18/06/2017 13
• Idea:
• Use topic modelling
• Latent Dirichlet
Allocation (LDA)
• Check the topics of
most cited paragraphs
https://icourts.dk/networks/sigma/citizenship

14. Latent Dirichlet Allocation
• An algorithm to read through document collections
and generate the k topics that best describe the
collection
• Unsupervised (no manual training)
• k must be given by the user
• Each topic is a collection of words and probabilities
18/06/2017 14

15. LDA for our setting
• Used MALLET on the network’s paragraphs with
Collapsed Gibbs sampling
• Removed standard English stopwords and words like
“ECJ”, “TFEU”, “ECR” (court boilerplates)
• Chose k=50 topics after experimenting with various
values and choosing the value making sense
• There are methods to mathematically determine the
best value for k, esp. via topic-coherence (O’ Callaghan
et al., 2015)
18/06/2017 15

16. Our topic model
Topic Topic words Label
2 person worker employment remuneration meaning relationship
regarded
the concept of EU worker
4 law situations treaty union scope nationals treatment scope of EU law
7 movement freedom workers free community treaty the right to move /workers
12 objective national justified pursued proportionate legitimate justification of restrictions
18 discrimination treatment fact nationality forms result lead principle of nondiscrimination
23 directive residence article union permanent host preamble secondary legislation
24 article workers regulation employment worker social status regulation (social advantage)
27 case-law inter alia cited cases effect union settled case law
29 article move reside territory union freedom freely right to move and reside
37 family worker host members regulation article residence family members of workers
18/06/2017 16

17. Some examples
Grzelcyk, 31: […]Union citizenship is destined to be the
fundamental status of nationals of the Member States,
enabling those who find themselves in the same
situation to enjoy the same treatment in law irrespective
of their nationality, subject to such exceptions as are
expressly provided for.
Topic 4: law situations treaty union scope nationals
treatment
18/06/2017 17

18. Some examples
Pusa, 19: […] Those opportunities could not be fully
effective if a national of a Member State could be
deterred from availing himself of them by obstacles
raised to his residence in the host Member State by
legislation of his State of origin penalising the fact that
he has used them (see, by analogy, D’Hoop, paragraph
31)
Topic 18: discrimination treatment fact nationality forms
result lead
18/06/2017 18

19. Some examples
Garcia Avello, 22:
[…]As the Court has ruled on several occasions (see,
inter alia , Case C-413/99 Baumbast and R [2002] ECR I-
7091, paragraph 82), citizenship of the Union is destined
to be the fundamental status of nationals of the Member
States.
Topic 4: law situations treaty union scope nationals
treatment
Topic 18: case-law inter alia cited cases effect union
18/06/2017 19

20. Useful remarks
• Qualitative evidence that the topic model is not too
far off for the given texts
• Does not directly capture the Court’s formulations
(“fundamental status”), need to preprocess?
• Requires the user/researcher to go back to the
decisions and improvise the label
• Helps us to reaffirm that:
• The court cites to reproduce relevant paradigms and court-
made concepts (topics 2 & 7 – eu workers)
• The court uses their formulations together with the EU Treaty
as an effort to interpret the latter (topic 4)
• k=50 seems to be unnecessarily large
18/06/2017 20

21. Second use-case
18/06/2017 21
t
Proceedings of Conf. on Legal Knowl.
& Inf. Systems, JURIX 2016

22. Problem setting
• Apply topic modeling to the entire collections of
judgments of CJEU and ECHR and study their
evolution
• A “straightforward” method:
• Compute LDA and calculate the average topic probability per
year (caveat?)
• Dynamic LDA (Blei and Lafferty, 2006)
18/06/2017 22

23. Problem setting
Questions to address:
• How useful is topic modeling to identify topics and
topic change in a dynamic judgment collection?
• Can we put topic modeling the service of the
researchers of international courts?
We used Non-negative Matrix Factorisation and a
Dynamic variant (Greene & Cross, 2017)
18/06/2017 23

24. Background
• A text collection of n documents and a total of m
terms can be represented as an n x m matrix A.
• Each element aij is the TF-IDF score of tij.
• TF-IDF (Term Frequency x Inverse Document
Frequency):
• A heuristic to highlight which terms are important to describe
the document
• Frequency of term t in document d x
1/(% of documents term t appears in)
18/06/2017 24

25. Background (cont’d)
• Normally we should start clustering matrix A BUT…
• The number or terms is usually too big and thus
running a simple clustering algorithm (e.g. k-means) is
almost impossible.
• Non-negative Matrix Factorization (NMF)
A = W x H
• A: n x m
• W: n x k
• H: k x m
for some integer k s.t. k «m , e.g. 100 vs. 10.000
18/06/2017 25

26. Interpretation of NMF
• This already is a topic model!
• This transformation of the initial document-term matrix is said to
uncover the latent topic structure of the corpus
• Each document is represented by means of k-topics (but probably some
of them with very small weight)
• Each topic is also a vector of terms so we usually output the top scoring
terms
• k is something that has to be set apriori
18/06/2017 27
n x m n x k
k x m
A W H
= x
Docu-
ments
terms topics terms
topics

27. Dynamic NMF – Step 1
18/06/2017 28
H1
H2
H3 Hn-1
Hn
H1
H2
H3
Hn-1
Hn
…
• Greene, D. and J.P. Cross, Exploring
the Political Agenda of the European
Parliament Using a Dynamic Topic
Modeling Approach. Political Analysis
25:1 (2017)
• Topic models for each period
• For each period (window) keep only
the top t terms per topic.
• Concatenate all topic-term matrices
from all the periods ⇒ new matrix B
• Delete columns that have all zeros
B

28. Dynamic NMF – Step 2
• Reapply NMF
• Produce two matrices U
and V
• Each row of U is a window
topic represented as vector
of dynamic topics
18/06/2017 29
Hn
H1
H2
H3
Hn-1
…
n' x k’
k’ x m
U
V
= x
Dynamic topics
terms
B
Dynamic topics

29. Dynamic NMF – Step 2
Mapping of dynamic topics to window topics:
• Dynamic topic j is assigned to the window topic i1 if:
j has the maximum weight in row i1 (argmax(i1))
18/06/2017 30
H1
Window 1
Window
topic i1
Dynamic topic j
UB

30. Dataset construction and preprocessing
• Data collection: Texts and meta obtained from
crawling EUR-Lex
• English judgments
• Divided into time windows per decade until 2000, after
that per 5 years
18/06/2017 31
0
500
1000
1500
2000
2500
1950-60 1970 1980 1990 2000 2005 2010-14
Periods
Judgments per period

31. Dataset construction and preprocessing
(cont’d)
• Lowercasing
• Tokenization
• Stopword removal
• English stopwords plus court stopwords: echr, ecj, court,
article…
• Document pruning:
• CJEU: kept only grounds
• Tf-idf for all periods
• Kept terms appearing in at least 10 documents
18/06/2017 32

32. Picking the topics per period
• Constructed a word2vec embeddings model for all
documents
• Estimated the optimal number of topics per period by
selecting k that maximizes Topic Coherence by
word2vec (TC-W2V)
• Using the same technique we chose k’ = 29
18/06/2017 33
10
14
19
39
37
31
19
0
5
10
15
20
25
30
35
40
45
1950-60 1970 1980 1990 2000 2005 2010-14
No. of topics
No. of topics

33. Rank Overall 1950-60 70s 80s 90s 90s (alt) 2000-05
1 sugar scrap sugar dumping dumping tobacco sugar
2 price ferrous regulation anti anti prices oct
3 dumping equalization production price imports harvest octs
4 prices applicant levy regulation injury premium rice
5 market scheme quota prices price price imports
6 anti undertakings marketing duty duty producers price
7 tobacco consumption year margin sales varieties beet
8 imports high beet normal institutions premiums regulation
9 production undertaking quotas injury community intervention community
10 quota contributions intervention imports japan variety cumulation
Dynamic model – An example
18/06/2017 34
• We have a global model that describes the entire lifespan of the court
• And the period-wise topic models that are associated with dynamic
model
• Terms have a ranking in both global and period models
• And…
• We have to pick a label for the topic…
”Common commercial policy”

34. Top dynamic models - ECJ
• We rank dynamic topics by the number of periods
they include (frequency)
18/06/2017 35
Topic Topic words Freq
T12 pension, insurance, benefits, age, social, retirement, legislation, security, benefit, scheme 7
T3 regulation, article, refund, export, compensatory, council, refunds, community,
agricultural, amounts
6
T4 directive, article, provisions, states, directives, annex, comply, substances, laws, water 6
T5 tax, income, taxation, resident, dividends, state, member, taxpayer, paragraph, taxable 6
Τ13 heading, tariff, cn, subheading, classificationnomenclature, classified, headings,
explanatory, customs
6
T2 residence, family, state, member, turkish, worker, employment, nationals, child 5
Τ28 sugar, price, dumping, prices, market, anti, tobacco, imports, production, quota 5

35. Top dynamic models - ECJ
• T4: an overall topic discussing EU law.
• T12: social benefits a topic that is persistently
discussed by the court
• T28: Common commercial policy
18/06/2017 36
Topic Topic words Freq
T12 pension, insurance, benefits, age, social, retirement, legislation, security, benefit, scheme 7
T3 regulation, article, refund, export, compensatory, council, refunds, community,
agricultural, amounts
6
T4 directive, article, provisions, states, directives, annex, comply, substances, laws, water 6
T5 tax, income, taxation, resident, dividends, state, member, taxpayer, paragraph, taxable 6
Τ13 heading, tariff, cn, subheading, classificationnomenclature, classified, headings,
explanatory, customs
6
T2 residence, family, state, member, turkish, worker, employment, nationals, child 5
Τ28 sugar, price, dumping, prices, market, anti, tobacco, imports, production, quota 5

36. Top dynamic models - ECJ
Common commercial policy:
• A subject that started as ”scrap” and ”ferrous
products” and ”undertakings” (50-70)
• More subjects of the internal market ”anti-dumping”
18/06/2017 37
Rank Overall 1950-60 70s 80s 90s 90s (alt) 2000-05
1 sugar scrap sugar dumping dumping tobacco sugar
2 price ferrous regulation anti anti prices oct
3 dumping equalization production price imports harvest octs
4 prices applicant levy regulation injury premium rice
5 market scheme quota prices price price imports
6 anti undertakings marketing duty duty producers price
7 tobacco consumption year margin sales varieties beet
8 imports high beet normal institutions premiums regulation
9 production undertaking quotas injury community intervention community
10 quota contributions intervention imports japan variety cumulation

37. Top dynamic models - ECJ
Top judgments:
• 80s: Case C-240/84 - NTN Toyo vs. Council
“38 UNDER THOSE CIRCUMSTANCES THE IMPOSITION OF AN ANTI-DUMPING DUTY ON IMPORTS OF ALL
SMALL BALL-BEARINGS FROM JAPAN CANNOT BE REGARDED AS CONTRARY TO THE PRINCIPLE OF
PROPORTIONALITY .”
• 90s: Case C-174/87 – Ricoh vs. Council
“77 In view of the foregoing considerations, the plea in law alleging miscalculation of the anti-dumping
duty must be rejected, and consequently the application must be dismissed in its entirety.”
• 00s: Case C-101/99 – British Sugar
” 69 As regards the concept of force majeure used in Article 4(4) of Regulation No 3665/87,
it must be emphasised that it is to be understood to refer to unusual circumstances outside
the control of the exporter (see, to that effect, Case 11/70 Internationale
Handelsgesellschaft [1970] ECR 1125, paragraph 23). An exporter's mistake as to the
classification of sugar does not amount to such circumstances.”
18/06/2017 38

38. What happened to anti-dumping?
• The Court didn’t have many decisions in the 2000s but
then there is a resurge after 2010
• The topic is drifts towards dynamic topic 3
(regulations)
18/06/2017 39
0
2
4
6
8
10
12
2000 2001 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Anti-dumping cases per year
Total

39. Another dynamic topic
Rank Overall 1970 1980 1990 (alt) 1990 (alt) 1990 (alt) 2000 (alt) 2000 (alt) 2005 2010
1 directive directive directive directive directive substances directive bathing directive directive
2 article italian article commission article programmes commission water article article
3 provisions directives women comply states water comply waters states
environment
al
4 states member provisions obligations provisions pollution obligations directive member public
5 directives comply member fulfil national list reasoned values annex law
6 comply provisions employees directives member waters fulfil pollution protection national
7 member states states prescribed annex discharges regulations quality information member
8 substances republic men laws law directive prescribed annex
environment
al states
9 laws obligations employer regulations council quality laws nitrogen national copyright
10 water article council failing health values article article provisions project
18/06/2017 40
Seems to be about compliance of the member states to EU directives but
interestingly:
• 1970s-80s: provisions, employees
• 90s: provisions but also water pollution
• 2000s: similar picture
• 2005-2015: environment and later on copyright

40. Remarks
• Dynamic NMF:
• Window and dynamic topics seem to make sense (most of the
times)
• Groups together semantically related documents from
different periods
• Topics relatively hard for experts to interpret
• Doctrinal topics are missing
• Need to look at Dynamic LDA (slow)
• Text pre-processing: Multi-words expression, TF-IDF
on bi-/tri-grams
18/06/2017 43

41. Thank you
Questions?
ioannis.panagis@jur.ku.dk
18/06/2017 44