SlideShare a Scribd company logo
.lusoftware verification & validation
VVS
An AI-assisted Approach
for Checking the Completeness of
Privacy Policies Against GDPR
Zurich, September 3rd, 2020
Damiano Torre1, Sallam Abualhaija1,
Mehrdad Sabetzadeh2,1, Lionel Briand1,2
{damiano.torre, sallam.abualhaija}@uni.lu
{msabetza, lbriand}@uottawa.ca
1 University of Luxembourg, Luxembourg
2 University of Ottawa, Canada
Katrien Baetens, Peter Goes,
Sylvie Forastier
{katrien.baetens, peter.goes,
sylvie.forastier}@linklaters.com
Linklaters, Luxembourg
Problem Definition
General Data Protection Regulation
• A complex EU regulation to harmonize data protection and
privacy in the European Union and the European Economic Area
• Every organization, EU-based or not, must comply with GDPR as
long as it collects and processes personal data of EU citizens and
residents
• Failure to comply with GDPR may result in fines of up to €20m or
4% of an organization's global turnover for specific breaches
3
Industrial Motivation
• Linklaters is a global law firm,
headquartered in London with a
base in Luxembourg
• Most of GDPR compliance checking
procedures are carried out manually
• Manual checking is time consuming
• Effective automated compliance
checking is needed
4
https://www.linklaters.com
Privacy Policy
Document stating how an organization handles personal data
5
How many privacy policies have you
read so far before clicking on “I agree”?
Twitter @bertsesame
Example of Checking
To comply with the regulation,
the privacy policy should include:
6
Article 13.2. (b) the existence
of the right to request from
the controller access to and
rectification or erasure of
personal data or restriction of
processing concerning the
data subject […].
CompletenessCompliance
Research Questions
RQ1: What are the metadata types required for checking the completeness
of a privacy policy according to GDPR?
Ø Building a conceptual model of GDPR privacy-related requirements
RQ2: How can the metadata required for completeness checking of a
privacy policy be extracted automatically?
Ø Developing automated metadata identification approach using NLP & ML
RQ3: How accurately can we extract metadata from privacy policies?
Ø Evaluating our metadata identification approach
RQ4: How accurately can we check the completeness of privacy policies?
Ø Evaluating our completeness checking approach
7
Approach
(RQ1) Building a Conceptual Model
of Privacy Policies Metadata
Analyzing GDPR Privacy-Related
Articles
9
From GDPR to Metadata Model
10
(b) the existence of the right to request from the controller access to and rectification or
erasure of personal data or restriction of processing concerning the data subject […].
Article 13.2. The controller shall, at the time when personal data are obtained, provide the
data subject with the following further information to ensure fair and transparent processing:
DATA_SUBJECT_RIGHT
ACCESS RECTIFICATIONRESTRICTIONERASURE
DATA SUBJECT
RIGHT
11
RQ1: What are the metadata types required for checking
the completeness of a privacy policy according to GDPR?
RESTRICTION
COMPLAINT
ERASURE
OBJECT
PORTABILITY
WITHDRAW
CONSENT
ACCESS
RECTIFICATION
SA
DIRECT
INDIRECT PD ORIGIN
PUBLICLY
COOKIE
THIRD-PARTY
12
Glossary
Completeness
Criteria
PD ORIGIN
INDIRECT
not needed
INDIRECT
THIRD-PARTY
found
INDIRECT
PUBLICLY found
2
Is PD ORIGIN
INDIRECT
found?
PD ORIGIN
INDIRECT
not found
PD ORIGIN
INDIRECT
partially found
3
Is INDIRECT
THIRD-PARTY
found?
4
Is INDIRECT
PUBLICLY
found?
Start
Final
No
Yes
1
Will data be
collected
indirectly?
No Yes Yes
Yes No No
Approach
(RQ2) Identifying Metadata in
Privacy Policies
14
Keywords
Training Data
Pre-
processing
1
Pre-trained Classifiers
Identified
Metadata
Pre-trained
Word Embeddings
Prediction
7
Similarity-based
Classification
5
Vectorization
3
KW-based
Classification
6
ML-based
Classification
4
Pre-
processing
1
Post-
processing
8
Text
Generalization
2
Privacy Policy
RQ2: How can the metadata required for completeness
checking of a privacy policy be extracted automatically?
Applying the NLP pipeline
Unit of Analysis: Sentence
DATA PROTECTION NOTICE
The protection of your personal data is important to the BNP Paribas Group in France,
which has adopted strong principles in that respect for the entire Group.
This data protection notice provides you with detailed information relating to the
protection of your personal data by the BNP Paribas Asset Management entities listed
in the appendix, which are all subject to this data protection notice (hereunder referred
to as “we”). We are responsible, as a controller, for collecting and processing your
personal data in relation to our activities.
DATA PROTECTION NOTICE
The protection of your personal data is important to the BNP Paribas Group in France,
which has adopted strong principles in that respect for the entire Group.
This data protection notice provides you with detailed information relating to the
protection of your personal data by the BNP Paribas Asset Management entities listed
in the appendix, which are all subject to this data protection notice (hereunder referred
to as “we”). We are responsible, as a controller, for collecting and processing your
personal data in relation to our activities.
WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM?
In accordance with applicable regulations, you have the following rights:
• To access: you can obtain information relating to the processing of your personal
data, and a copy of such personal data.
• To rectify: where you consider that your personal data are inaccurate or incomplete,
you can require that such personal data be modified accordingly.
• To erase: you can require the deletion of your personal data, to the extent permitted
by law.
• To restrict: you can request the restriction of the processing of your personal data.
• To object: you can object to the processing of your personal data, on grounds
relating to your particular situation. You have the absolute right to object to the
processing of your personal data for direct marketing purposes, which includes
profiling related to such direct marketing.
• To withdraw your consent: where you have given your consent for the processing
of your personal data, you have the right to withdraw your consent at any time.
• To data portability: where legally applicable, you have the right to have the personal
data you have provided to us be returned to you or, where technically feasible,
transferred to a third party.
If you wish to exercise the rights listed above, please send a letter to the following
address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009
PARIS, FRANCE or an email using dataprotection@bnpparibas.com.
In accordance with applicable regulation, in addition to your rights above, you are also
entitled to lodge a complaint with the competent supervisory authority.
Pre-processing
15
Stopwords Removal
The protection of your personal data is important to the
BNP Paribas Group in France, which has adopted
strong principles in that respect for the entire Group.
protection personal data important BNP Paribas Group
France, adopt strong principle respect entire Group.
Lemmatization
Text Generalization
Replacing specific textual entities with more general ones
Ø Locations, organizations, websites, telephone numbers, e-mail and addresses
16
DATA PROTECTION NOTICE
The protection of your personal data is important to the BNP Paribas Group in France,
which has adopted strong principles in that respect for the entire Group.
This data protection notice provides you with detailed information relating to the
protection of your personal data by the BNP Paribas Asset Management entities listed
in the appendix, which are all subject to this data protection notice (hereunder referred
to as “we”). We are responsible, as a controller, for collecting and processing your
personal data in relation to our activities.
WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM?
In accordance with applicable regulations, you have the following rights:
• To access: you can obtain information relating to the processing of your personal
data, and a copy of such personal data.
• To rectify: where you consider that your personal data are inaccurate or incomplete,
you can require that such personal data be modified accordingly.
• To erase: you can require the deletion of your personal data, to the extent permitted
by law.
• To restrict: you can request the restriction of the processing of your personal data.
• To object: you can object to the processing of your personal data, on grounds
relating to your particular situation. You have the absolute right to object to the
processing of your personal data for direct marketing purposes, which includes
profiling related to such direct marketing.
• To withdraw your consent: where you have given your consent for the processing
of your personal data, you have the right to withdraw your consent at any time.
• To data portability: where legally applicable, you have the right to have the personal
data you have provided to us be returned to you or, where technically feasible,
transferred to a third party.
If you wish to exercise the rights listed above, please send a letter to the following
address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009
PARIS, FRANCE or an email using dataprotection@bnpparibas.com.
In accordance with applicable regulation, in addition to your rights above, you are also
entitled to lodge a complaint with the competent supervisory authority.
protection personal data important BNP Paribas Group
France adopt strong principle respect entire Group.
protection personal data important
ORGANIZATION LOCATION
adopt strong principle respect
entire Group.
exercise right list please send letter
following address data protection
officer ORGANIZATION ADDRESS
email use EMAIL.
Transforming the sentences into 100-dimentional vectors
Ø Using GloVe pre-trained models
Vectorization
17
DATA PROTECTION NOTICE
The protection of your personal data is important to the BNP Paribas Group in France,
which has adopted strong principles in that respect for the entire Group.
This data protection notice provides you with detailed information relating to the
protection of your personal data by the BNP Paribas Asset Management entities listed
in the appendix, which are all subject to this data protection notice (hereunder referred
to as “we”). We are responsible, as a controller, for collecting and processing your
personal data in relation to our activities.
WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM?
In accordance with applicable regulations, you have the following rights:
• To access: you can obtain information relating to the processing of your personal
data, and a copy of such personal data.
• To rectify: where you consider that your personal data are inaccurate or incomplete,
you can require that such personal data be modified accordingly.
• To erase: you can require the deletion of your personal data, to the extent permitted
by law.
• To restrict: you can request the restriction of the processing of your personal data.
• To object: you can object to the processing of your personal data, on grounds
relating to your particular situation. You have the absolute right to object to the
processing of your personal data for direct marketing purposes, which includes
profiling related to such direct marketing.
• To withdraw your consent: where you have given your consent for the processing
of your personal data, you have the right to withdraw your consent at any time.
• To data portability: where legally applicable, you have the right to have the personal
data you have provided to us be returned to you or, where technically feasible,
transferred to a third party.
If you wish to exercise the rights listed above, please send a letter to the following
address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009
PARIS, FRANCE or an email using dataprotection@bnpparibas.com.
In accordance with applicable regulation, in addition to your rights above, you are also
entitled to lodge a complaint with the competent supervisory authority.
data protection noticePre-processed sentence
protection
data
notice
[ -0.4710, 0.6158, 0.6897, -0.1815, 0.3078, -0.8415, -0.4187, -0.2001, 0.2818, -0.3401, 0.7729, -0.2277, 0.0599, -0.2414,
0.8778, 0.7204, 0.6429, 0.3625, 0.4162, 0.1300, -0.4707, -0.4466, 0.4736, 0.4076, -1.0341, -1.1422, 0.3744, 0.2463, -0.6729,
0.4918, 0.4651, 0.1361, -0.9380, 0.5189, 0.5155, -0.2651, -0.1455, 0.2252, 0.3524, -0.7965, -0.4225, -0.9059, -0.8400, 0.4536,
-0.7249, -0.1259, 0.4366, -0.5366, 0.0205, -0.7461, 1.1925, 0.1572, 0.2932, 0.9266, 0.4824, -1.8290, -0.0127, -0.3703, 2.3618,
0.3359, -0.1544, 0.1466, -0.1131, -0.0249, 0.3193, 0.2882, -0.2963, -0.3303, 1.4774, 0.2374, -0.2531, 0.6137, 0.5681, -0.5699,
0.4880, 0.0654, 0.2826, -0.1354, -1.1096, -0.3597, 0.8531, 0.4630, -1.1223, 0.0072, -1.7636, -0.4455, 1.2478, -0.3754, -0.2163,
0.4594, -0.1139, 0.7558, -0.2442, -0.0565, 0.5479, -0.3093, 0.2592, -0.5961, 0.2760, 0.0880]
[ 0.0330, -0.2653, 0.2386, -0.1449, 0.4572, -0.2129, -0.6080, -0.4135, 0.0960, -0.2114, -0.0625, -0.1854, 0.0909, -0.2509,
0.0931, -0.2927, 0.9347, -0.1482, -0.9635, 0.0151, -0.3910, -0.2225, -0.1056, 0.4497, 0.0575, 0.3878, -0.2755, -0.5580, -0.9241,
-0.4803, 0.3122, -0.0470, -0.0105, 0.4453, 0.3733, -0.2642, -0.0194, 0.0778, -0.0885, 0.0392, -0.1973, -0.3003, 0.2938, -0.2649,
0.5680, -0.2838, -0.4947, 0.3432, -0.4117, -0.9482, 0.5263, 0.1430, -0.7211, 1.2293, 0.4163, -0.9417, 0.8058, -1.0777, 2.4755,
0.3097, 0.1849, 0.3422, -0.2709, 0.2793, 1.0019, 0.0157, -0.4184, -0.1016, 0.8306, -0.3263, 0.1229, -0.4821, 0.3061, -0.1621,
0.7334, -0.1168, -0.3701, 0.1795, -1.1684, 0.1200, 0.8329, -0.1387, 0.0737, 0.5353, -1.1984, 0.7402, 0.1241, 0.0189, 0.2222,
-0.0001, 1.0163, -1.2003, 0.0576, 0.1005, 0.3886, 0.6566, -0.2975, -0.0867, 0.5531, -0.3809]
[ 0.1377, -0.5041, -0.1082, -0.2499, -0.2414, 0.3749, -0.5977, 0.4089, 0.3198, -0.4266, 0.2197, 0.0467, -0.0241, -0.3295,
0.1505, -0.0197, -0.0795, 0.3808, -0.0841, 0.0600, -0.2723, 0.3564, -0.5089, -0.5272, 0.1763, 0.0837, -0.0244, -0.0261, 0.2384,
-0.6271, 0.4758, 0.0080, 0.1979, -0.2095, -0.4243, 0.5012, -0.2161, -0.7097, -0.0681, -0.1287, -0.2156, 0.6505, 0.3222, -0.4843,
0.1309, -0.2888, -0.0246, -0.1711, 0.2656, -1.0098, 1.1561, -0.2706, -0.1719, 0.4489, 0.0501, -1.2639, -0.0260, -0.6504, 1.7581,
0.1059, -0.3542, 0.8539, -0.5211, 0.0547, 0.8483, -0.1820, -0.2561, 0.1676, -0.2452, -0.0495, -0.4471, -0.2160, 0.2036, -0.5234,
0.3364, 0.9539, -0.0427, -0.3200, -0.9549, -0.4273, 0.1453, -1.0337, -0.5458, -0.2693, -1.0983, -0.4366, 0.3313, -0.0407, 0.1030,
0.1724, 0.0570, -0.2919, 0.0124, -0.3216, 0.0375, 0.6194, -0.0108, 0.0652, -0.3073, 0.1408]
Corresponding
100-D vector
[ -0.1001, -0.0512, 0.2734, -0.1921, 0.1745, -0.2265, -0.5415, -0.0682, 0.2326, -0.3260, 0.3100,
-0.1221, 0.0422, -0.2739, 0.3738, 0.1360, 0.4994, 0.1984, -0.2105, 0.0684, -0.3780, -0.1042, -0.0470,
0.1100, -0.2668, -0.2236, 0.0248, -0.1126, -0.4529, -0.2052, 0.4177, 0.0324, -0.2502, 0.2516, 0.1548,
-0.0094, -0.1270, -0.1356, 0.0653, -0.2953, -0.2785, -0.1852, -0.0747, -0.0985, -0.0087, -0.2328, -0.0276,
-0.1215, -0.0419, -0.9014, 0.9583, 0.0099, -0.1999, 0.8683, 0.3162, -1.3449, 0.2557, -0.6995, 2.1985,
0.2505, -0.1079, 0.4475, -0.3017, 0.1030, 0.7232, 0.0406, -0.3236, -0.0881, 0.6876, -0.0461, -0.1925,
-0.0281, 0.3593, -0.4185, 0.5193, 0.3008, -0.0434, -0.0920, -1.0776, -0.2223, 0.6105, -0.2365, -0.5315,
0.0911, -1.3534, -0.0473, 0.5677, -0.1324, 0.0363, 0.2106, 0.3198, -0.2455, -0.0581, -0.0925, 0.3247,
0.3222, -0.0164, -0.2059, 0.1739, -0.0507]
Prediction
18
Keywords
Training Data
Pre-
processing
1
Pre-trained Classifiers
Identified
Metadata
Pre-trained
Word Embeddings
Prediction
7
Similarity-based
Classification
5
Vectorization
3
KW-based
Classification
6
ML-based
Classification
4
Pre-
processing
1
Post-
processing
8
Text
Generalization
2
Privacy Policy
Hierarchical Model
19
MLML + Sim + KWKW+NER
ML-based Classification
• A binary classifier for each metadata type (levels 1&2)
20
DATA SUBJECT RIGHT
Not DATA SUBJECT RIGHT
Not DATA SUBJECT RIGHT
Not DATA SUBJECT RIGHT
DATA PROTECTION NOTICE
The protection of your personal data is important to the BNP Paribas Group in France,
which has adopted strong principles in that respect for the entire Group.
This data protection notice enables you to obtain detailed information relating to the
protection of your personal data by the BNP Paribas Asset Management entities listed
in the appendix, which are all subject to this data protection notice (hereunder referred
to as “we”). We are responsible, as a controller, for collecting and processing your
personal data in relation to our activities.
Checking semantic similarity between a sentence and a set of
training examples of a metadata type (level 2)
-0.1983, 0.1698, 0.3051, -0.0343, 0.2145, 0.0277, -0.3181, -0.1116, 0.1986, -0.2342, 0.2577,
-0.1901, 0.3291, -0.1272, 0.1686, 0.0948, 0.4405, 0.2328, -0.2926, 0.1354, -0.3191, -0.1708, -0.0148,
0.0395, -0.2850, -0.3252, 0.0696, -0.1286, -0.2628, -0.0727, 0.2690, 0.0940, -0.2561, 0.0484, 0.0598,
0.1618, -0.0639, 0.0702, -0.1344, -0.0973, -0.1914, -0.0648, 0.0528, -0.1219, -0.1378, -0.1070, -0.0277,
-0.0734, -0.1583, -0.6287, 0.7184, 0.2003, 0.0483, 0.7144, 0.0752, -1.2867, 0.3194, -0.4712, 1.7194,
0.3681, -0.1071, 0.4806, -0.2039, -0.0508, 0.8415, -0.0838, -0.0044, 0.0302, 0.4905, -0.0457, -0.1050,
-0.1170, 0.2246, -0.2939, 0.3821, 0.1018, -0.1512, -0.1171, -0.8723, -0.2130, 0.4374, -0.0327, -0.2389,
0.0673, -1.2155, 0.0636, 0.2182, -0.1731, -0.0059, 0.0620, -0.0136, -0.0645, 0.0555, -0.1652, 0.1210,
0.1492, -0.1160, -0.3832, 0.2884, 0.0442"
Similarity-based Classification
21
[ -0.2282, -0.1558, 0.0652, 0.0991, 0.1608, -0.1088, -0.2718, -0.0173, 0.1660, 0.1387,
0.1073, -0.1066, 0.2735, -0.0381, 0.1411, 0.0162, 0.1138, 0.2237, 0.0039, 0.3900,
-0.2075, -0.3058, 0.1522, -0.1538, -0.3828, -0.2691, -0.0107, -0.0786, -0.0987, -0.0090,
0.1191, 0.3814, -0.2909, 0.0209, 0.0454, -0.1930, 0.0577, 0.0796, 0.1150, -0.2799,
-0.2460, -0.3236, 0.1911, -0.0752, -0.3893, -0.0191, -0.0130, -0.2376, -0.1571, -0.6903,
0.5475, 0.0387, 0.0706, 0.9536, 0.1957, -1.3846, 0.0126, -0.2413, 1.5954, 0.3453,
-0.0842, 0.3065, -0.0789, -0.0849, 0.6592, -0.0325, 0.0970, -0.0936, 0.7144, 0.0332,
-0.0842, -0.1491, 0.2932, -0.2050, 0.2586, -0.0586, -0.0516, -0.0666, -0.7936, 0.0595,
0.4763, 0.3987, -0.4171, 0.1743, -1.2655, 0.1015, 0.3702, -0.1000, -0.0591, -0.3456,
-0.2736, 0.2876, 0.0137, 0.0495, 0.3142, -0.0320, 0.0510, -0.5056, 0.5006, -0.1134]
[ -0.2111, -0.0454, 0.3794, 0.1048, -0.1989, -0.0568, -0.3195, -0.1061, 0.3210, 0.0961,
0.1302, -0.0328, 0.1682, -0.2576, 0.1328, 0.0617, 0.2283, 0.3493, -0.0362, 0.2115,
-0.1524, -0.3676, 0.2312, 0.1922, -0.4278, -0.5339, -0.2137, -0.0735, -0.2329, 0.1074,
0.3383, 0.2820, -0.5242, -0.0686, -0.0450, 0.1989, -0.1216, -0.1705, -0.2182, -0.1445,
-0.1320, -0.1095, -0.1330, -0.0862, -0.6284, -0.2005, 0.2613, -0.1753, -0.2242, -0.7376,
0.7149, 0.1868, 0.2578, 0.5627, 0.0887, -1.1176, 0.5187, -0.2217, 1.3552, 0.0823,
-0.2786, 0.2048, -0.1019, 0.1011, 0.6854, -0.2996, 0.1295, -0.0006, 0.2986, -0.2940,
-0.1595, 0.1385, 0.3246, -0.3247, 0.3632, 0.1793, -0.2596, 0.1123, -0.7059, -0.2170,
0.4344, 0.1247, -0.4316, -0.0446, -1.1833, -0.0789, 0.4744, -0.1370, -0.4004, -0.1094,
-0.2521, 0.2583, -0.0554, -0.2261, 0.1829, -0.1604, -0.0274, -0.4844, 0.3389, 0.0507]
0.46
0.93 Represented by an average vector
Training examples
on “ACCESS”
Looking for the keywords of a metadata type that are present in the
sentence
DATA PROTECTION NOTICE
The protection of your personal data is important to the BNP Paribas Group in France,
which has adopted strong principles in that respect for the entire Group.
This data protection notice enables you to obtain detailed information relating to the
protection of your personal data by the BNP Paribas Asset Management entities listed
in the appendix, which are all subject to this data protection notice (hereunder referred
to as “we”). We are responsible, as a controller, for collecting and processing your
personal data in relation to our activities.
WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM?
In accordance with applicable regulations, you have the following rights:
• To access: you can obtain information relating to the processing of your personal
data, and a copy of such personal data.
• To rectify: where you consider that your personal data are inaccurate or incomplete,
you can require that such personal data be modified accordingly.
• To erase: you can require the deletion of your personal data, to the extent permitted
by law.
• To restrict: you can request the restriction of the processing of your personal data.
• To object: you can object to the processing of your personal data, on grounds
relating to your particular situation. You have the absolute right to object to the
processing of your personal data for direct marketing purposes, which includes
profiling related to such direct marketing.
• To withdraw your consent: where you have given your consent for the processing
of your personal data, you have the right to withdraw your consent at any time.
• To data portability: where legally applicable, you have the right to have the personal
data you have provided to us be returned to you or, where technically feasible,
transferred to a third party.
If you wish to exercise the rights listed above, please send a letter to the following
address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009
PARIS, FRANCE or an email using dataprotection@bnpparibas.com.
In accordance with applicable regulation, in addition to your rights above, you are also
entitled to lodge a complaint with the competent supervisory authority.
Keyword-based Classification
22
data protection notice enable obtain
detail information relate protection
personal data ORGANIZATION entity
list appendix all subject data
protection notice hereunder refer
access obtain information
relate processing personal data
copy personal data
Keywords of “ACCESS”
Combining the results
DATA PROTECTION NOTICE
The protection of your personal data is important to the BNP Paribas Group in France,
which has adopted strong principles in that respect for the entire Group.
This data protection notice enables you to obtain detailed information relating to the
protection of your personal data by the BNP Paribas Asset Management entities listed
in the appendix, which are all subject to this data protection notice (hereunder referred
to as “we”). We are responsible, as a controller, for collecting and processing your
personal data in relation to our activities.
WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM?
In accordance with applicable regulations, you have the following rights:
• To access: you can obtain information relating to the processing of your personal
data, and a copy of such personal data.
• To rectify: where you consider that your personal data are inaccurate or incomplete,
you can require that such personal data be modified accordingly.
• To erase: you can require the deletion of your personal data, to the extent permitted
by law.
• To restrict: you can request the restriction of the processing of your personal data.
• To object: you can object to the processing of your personal data, on grounds
relating to your particular situation. You have the absolute right to object to the
processing of your personal data for direct marketing purposes, which includes
profiling related to such direct marketing.
• To withdraw your consent: where you have given your consent for the processing
of your personal data, you have the right to withdraw your consent at any time.
• To data portability: where legally applicable, you have the right to have the personal
data you have provided to us be returned to you or, where technically feasible,
transferred to a third party.
If you wish to exercise the rights listed above, please send a letter to the following
address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009
PARIS, FRANCE or an email using dataprotection@bnpparibas.com.
In accordance with applicable regulation, in addition to your rights above, you are also
entitled to lodge a complaint with the competent supervisory authority.
Prediction
23
Keywords
Training Data
Pre-
processing
1
Pre-trained Classifiers
Identified
Metadata
Pre-trained
Word Embeddings
Prediction
7
Similarity-based
Classification
5
Vectorization
3
KW-based
Classification
6
ML-based
Classification
4
Pre-
processing
1
Post-
processing
8
Text
Generalization
2
Privacy Policy
DATA SUBJECT
RIGHT
COMPLAINT SA
DATA SUBJECT
RIGHT
ACCESS
DATA PROTECTION NOTICE
The protection of your personal data is important to the BNP Paribas Group in France,
which has adopted strong principles in that respect for the entire Group.
This data protection notice enables you to obtain detailed information relating to the
protection of your personal data by the BNP Paribas Asset Management entities listed
in the appendix, which are all subject to this data protection notice (hereunder referred
to as “we”). We are responsible, as a controller, for collecting and processing your
personal data in relation to our activities.
Considering the contextual information
Post-processing
24
Not DATA SUBJECT RIGHT
Not DATA SUBJECT RIGHT
Not DATA SUBJECT RIGHT
DATA SUBJECT RIGHTDATA SUBJECT RIGHT
Not DATA SUBJECT RIGHT
Empirical Evaluation
Case Study
DATA SUBJECT RIGHT and LEGAL BASIS
26
TO ENTER
CONTRACT
CONTRACTUAL
STATUTORY
CONSENT
PUBLIC
FUNCTION
LEGITIMATE
INTEREST
VITAL INTEREST
CONTRACT
LEGAL
OBLIGATION
LEGAL BASIS
DATA SUBJECT
RIGHT
RESTRICTION
COMPLAINT
ERASURE
OBJECT
PORTABILITY
WITHDRAW
CONSENT
SA
ACCESS
RECTIFICATION
Document Collection
A total of 234 privacy policies, fully annotated
Ø 90% training-set and 10% testset
27
DATA SUBJECT RIGHT # of Sentences
ACCESS 228
COMPLAINT 196
SA 183
ERASURE 219
OBJECT 298
PORTABILITY 171
RECTIFICATION 215
RESTRICTION 179
WITHDRAW CONSENT 204
TOTAL 1710
LEGAL BASIS # of Sentences
CONSENT 272
CONTRACT 265
TO ENTER CONTRACT 69
CONTRACTUAL 164
STATUTORY 16
LEGAL OBLIGATION 351
LEGITIMATE INTEREST 507
PUBLIC FUNCTION 71
VITAL INTEREST 13
TOTAL 1479
Empirical Evaluation
(RQ3) Metadata Identification
Approach
Evaluation Metrics
• Precision (P): percentage of correctly identified metadata types
by the approach – TP/(TP+FP)
• Recall (R): percentage of correctly identified metadata types from
the ones actually present in the privacy policies – TP/(TP+FN)
29
True Positive
(TP)
Metadata type is present in the privacy policy and is
identified by the approach
False Positive
(FP)
Metadata type is NOT present in the privacy policy
but is identified by the approach
False Negative
(FN)
Metadata type is present in the privacy policy but is
NOT identified by the approach
RQ3: How accurately can we extract metadata from a
given privacy policy?
30
DATA SUBJECT RIGHT P(%) R(%)
ACCESS 100 91
COMPLAINT 100 100
SA 100 100
ERASURE 100 89
OBJECT 94 94
PORTABILITY 100 100
RECTIFICATION 100 95
RESTRICTION 100 94
WITHDRAW CONSENT 100 94
LEGAL BASIS P(%) R(%)
CONSENT 95 100
CONTRACT 90 95
TO ENTER CONTRACT 100 87
CONTRACTUAL 94 100
STATUTORY 83 100
LEGAL OBLIGATION 100 96
LEGITIMATE INTEREST 100 81
PUBLIC FUNCTION 75 60
VITAL INTEREST 100 80
Empirical Evaluation
(RQ4) Completeness Checking
Approach
Case Study Completeness Criteria
32
DATA SUBJECT
RIGHT
RESTRICTION
COMPLAINT
ACCESS
RECTIFICATION
C1: Must always be present C2:
C4:C3:
DATA SUBJECT
RIGHT
ERASURE
OBJECT
PORTABILITY
WITHDRAW
CONSENT
CONSENT
LEGAL BASISIF THEN
DATA SUBJECT
RIGHT
PORTABILITY
CONTRACT
LEGAL BASIS
IF THEN
DATA SUBJECT
RIGHT
OBJECT
PUBLIC
FUNCTION
LEGITIMATE
INTEREST
LEGAL BASIS
IF THEN
https://www.clipartkey.co
m/view/hJTTiw_muppet-
wiki-sesame-street-bert/
If you give your consent for sharing
your data, can you withdraw it?
Example of Incompleteness Issue
33
DATA SUBJECT
RIGHT
RESTRICTION
COMPLAINT
ACCESS
RECTIFICATION
C1: Must always be present
C3:
DATA SUBJECT
RIGHT
PORTABILITY
CONTRACT
LEGAL BASIS
IF THEN
DATA SUBJECT
RIGHT
RESTRICTION
COMPLAINT
ACCESS
RECTIFICATION
DATA SUBJECT
RIGHT
PORTABILITY
CONTRACT
LEGAL BASIS
IF THEN
DATA PROTECTION NOTICE
The protection of your personal data is important […]
WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM?
In accordance with applicable regulations, you have the following rights:
• To access: you can obtain information relating to the processing of your
personal data, and a copy of such personal data.
• To rectify: where you consider that your personal data are inaccurate or
incomplete, you can require that such personal data be modified
accordingly.
[…]
In accordance with applicable regulation, in addition to your rights above,
you are also entitled to lodge a complaint with the competent supervisory
authority.
WHY AND ON WHICH BASIS DO WE USE YOUR PERSONAL DATA?
a) To comply with our legal and regulatory obligations
We use your personal data to comply with various legal and regulatory obligations, […]
b) To perform a contract with you or to take steps at your request
before entering into a contract
We use your personal data to enter into and perform our contracts to provide you with
information regarding our products and services.
Evaluation Metrics
• Precision (P): percentage of correctly identified incompleteness
by the approach – TP/(TP+FP)
• Recall (R): percentage of correctly identified incompleteness from
what actually is in the privacy policies – TP/(TP+FN)
34
True Positive
(TP)
Incompleteness issue exists in the privacy policy and
is identified by the approach
False Positive
(FP)
Incompleteness issue does NOT exist in the privacy
policy but is identified by the approach
False Negative
(FN)
Incompleteness issue exists in the privacy policy but
is NOT identified by the approach
RQ4: How accurately can we check the completeness
of a given privacy policy?
On our test set (24 privacy policies):
Ø45 out of 47 incompleteness issues are correctly identified
ØFalse alarm on eight occasions
35
Criterion TPs FPs FNs P(%) R(%)
C1 16 4 0 80 100
C2 5 1 0 83 100
C3 3 0 2 100 60
C4 21 3 0 88 100
Summary 45 8 2 85 96
Summary
36
Thanks for listening! Time for questions.
Participate in BERT’s quiz at this link:
https://shorturl.at/ditES
37
https://tenor.com/search/bert-gifs

More Related Content

What's hot

Presentation on GDPR
Presentation on GDPRPresentation on GDPR
Presentation on GDPR
DipanjanDey12
 
Compatible use of personal data (개인정보 이용의 양립가능성)
Compatible use of personal data (개인정보 이용의 양립가능성)Compatible use of personal data (개인정보 이용의 양립가능성)
Compatible use of personal data (개인정보 이용의 양립가능성)
David Lee
 
General data protection regulation
General data protection regulationGeneral data protection regulation
General data protection regulation
Fahad Ameen
 
EU GDPR(general data protection regulation)
EU GDPR(general data protection regulation)EU GDPR(general data protection regulation)
EU GDPR(general data protection regulation)
RAKESH S
 
How to get your business GDPR ready
How to get your business GDPR readyHow to get your business GDPR ready
How to get your business GDPR ready
Premier EPOS
 
GDPR ASAP: A Seven-Step Guide to Prepare for the General Data Protection Regu...
GDPR ASAP: A Seven-Step Guide to Prepare for the General Data Protection Regu...GDPR ASAP: A Seven-Step Guide to Prepare for the General Data Protection Regu...
GDPR ASAP: A Seven-Step Guide to Prepare for the General Data Protection Regu...
ObservePoint
 
White-Paper_Security-DBSec_EU-GDPR_06-2016
White-Paper_Security-DBSec_EU-GDPR_06-2016White-Paper_Security-DBSec_EU-GDPR_06-2016
White-Paper_Security-DBSec_EU-GDPR_06-2016stefanjung
 
How to get started with being GDPR compliant
How to get started with being GDPR compliantHow to get started with being GDPR compliant
How to get started with being GDPR compliant
Siddharth Ram Dinesh
 
General data protection
General data protectionGeneral data protection
General data protection
BrijeshR3
 
GDPR - A practical guide
GDPR - A practical guideGDPR - A practical guide
GDPR - A practical guide
Angad Dayal
 
General Data Protection Regulation (GDPR)
General Data Protection Regulation (GDPR)General Data Protection Regulation (GDPR)
General Data Protection Regulation (GDPR)
Extentia Information Technology
 
The principles of the Data Protection Act in detail - uk
The principles of the Data Protection Act in detail - ukThe principles of the Data Protection Act in detail - uk
The principles of the Data Protection Act in detail - uk
- Mark - Fullbright
 
Covid 19 privacy hot topics and the evolving ccpa regulations
Covid 19 privacy hot topics and the evolving ccpa regulationsCovid 19 privacy hot topics and the evolving ccpa regulations
Covid 19 privacy hot topics and the evolving ccpa regulations
Knobbe Martens - Intellectual Property Law
 
General Data Protection Regulation for Ops
General Data Protection Regulation for OpsGeneral Data Protection Regulation for Ops
General Data Protection Regulation for Ops
Kamil Rextin
 
SCCE Processors and GDPR
SCCE Processors and GDPRSCCE Processors and GDPR
SCCE Processors and GDPRRobert Bond
 
A quick look at gdpr
A quick look at gdprA quick look at gdpr
A quick look at gdpr
CookieYes
 
The International Comparative Legal Guide to: Data Protection 2016
The International Comparative Legal Guide to: Data Protection 2016The International Comparative Legal Guide to: Data Protection 2016
The International Comparative Legal Guide to: Data Protection 2016
Matheson Law Firm
 
EU GDPR (training)
EU GDPR (training)  EU GDPR (training)
EU GDPR (training)
Elizabeth Baker, JD, CRCMP
 
UK GDPR: What New Direction?
UK GDPR:  What New Direction?UK GDPR:  What New Direction?
UK GDPR: What New Direction?
David Erdos
 

What's hot (20)

Presentation on GDPR
Presentation on GDPRPresentation on GDPR
Presentation on GDPR
 
Compatible use of personal data (개인정보 이용의 양립가능성)
Compatible use of personal data (개인정보 이용의 양립가능성)Compatible use of personal data (개인정보 이용의 양립가능성)
Compatible use of personal data (개인정보 이용의 양립가능성)
 
General data protection regulation
General data protection regulationGeneral data protection regulation
General data protection regulation
 
EU GDPR(general data protection regulation)
EU GDPR(general data protection regulation)EU GDPR(general data protection regulation)
EU GDPR(general data protection regulation)
 
How to get your business GDPR ready
How to get your business GDPR readyHow to get your business GDPR ready
How to get your business GDPR ready
 
GDPR ASAP: A Seven-Step Guide to Prepare for the General Data Protection Regu...
GDPR ASAP: A Seven-Step Guide to Prepare for the General Data Protection Regu...GDPR ASAP: A Seven-Step Guide to Prepare for the General Data Protection Regu...
GDPR ASAP: A Seven-Step Guide to Prepare for the General Data Protection Regu...
 
White-Paper_Security-DBSec_EU-GDPR_06-2016
White-Paper_Security-DBSec_EU-GDPR_06-2016White-Paper_Security-DBSec_EU-GDPR_06-2016
White-Paper_Security-DBSec_EU-GDPR_06-2016
 
How to get started with being GDPR compliant
How to get started with being GDPR compliantHow to get started with being GDPR compliant
How to get started with being GDPR compliant
 
General data protection
General data protectionGeneral data protection
General data protection
 
Privacy Access Letter I Feb 5 07
Privacy Access Letter I   Feb 5 07Privacy Access Letter I   Feb 5 07
Privacy Access Letter I Feb 5 07
 
GDPR - A practical guide
GDPR - A practical guideGDPR - A practical guide
GDPR - A practical guide
 
General Data Protection Regulation (GDPR)
General Data Protection Regulation (GDPR)General Data Protection Regulation (GDPR)
General Data Protection Regulation (GDPR)
 
The principles of the Data Protection Act in detail - uk
The principles of the Data Protection Act in detail - ukThe principles of the Data Protection Act in detail - uk
The principles of the Data Protection Act in detail - uk
 
Covid 19 privacy hot topics and the evolving ccpa regulations
Covid 19 privacy hot topics and the evolving ccpa regulationsCovid 19 privacy hot topics and the evolving ccpa regulations
Covid 19 privacy hot topics and the evolving ccpa regulations
 
General Data Protection Regulation for Ops
General Data Protection Regulation for OpsGeneral Data Protection Regulation for Ops
General Data Protection Regulation for Ops
 
SCCE Processors and GDPR
SCCE Processors and GDPRSCCE Processors and GDPR
SCCE Processors and GDPR
 
A quick look at gdpr
A quick look at gdprA quick look at gdpr
A quick look at gdpr
 
The International Comparative Legal Guide to: Data Protection 2016
The International Comparative Legal Guide to: Data Protection 2016The International Comparative Legal Guide to: Data Protection 2016
The International Comparative Legal Guide to: Data Protection 2016
 
EU GDPR (training)
EU GDPR (training)  EU GDPR (training)
EU GDPR (training)
 
UK GDPR: What New Direction?
UK GDPR:  What New Direction?UK GDPR:  What New Direction?
UK GDPR: What New Direction?
 

Similar to An AI-assisted Approach for Checking the Completeness of Privacy Policies Against GDPR

Are you GDPRed yet?
Are you GDPRed yet?Are you GDPRed yet?
An Overview of GDPR
An Overview of GDPR An Overview of GDPR
An Overview of GDPR
The Pathway Group
 
An Overview of GDPR by Pathway Group
An Overview of GDPR by Pathway GroupAn Overview of GDPR by Pathway Group
An Overview of GDPR by Pathway Group
The Pathway Group
 
GDPR - what you need to know
GDPR -  what you need to know GDPR -  what you need to know
GDPR - what you need to know
Maddie Malling-May
 
Gdpr zilla
Gdpr zillaGdpr zilla
Gdpr zilla
David Boswell
 
Gdpr in a nutshell
Gdpr in a nutshellGdpr in a nutshell
Gdpr in a nutshell
Matthew Butler
 
Reddico GDPR Presentation
Reddico GDPR PresentationReddico GDPR Presentation
Reddico GDPR Presentation
Luke Kyte
 
Understanding the EU's new General Data Protection Regulation (GDPR)
Understanding the EU's new General Data Protection Regulation (GDPR)Understanding the EU's new General Data Protection Regulation (GDPR)
Understanding the EU's new General Data Protection Regulation (GDPR)
Acquia
 
Opportunity or burden
Opportunity or burdenOpportunity or burden
Opportunity or burden
IRIS
 
GDPR: Training Materials by Qualsys
GDPR: Training Materials  by QualsysGDPR: Training Materials  by Qualsys
GDPR: Training Materials by Qualsys
Qualsys Ltd
 
GDPR Changing Mindset
GDPR Changing MindsetGDPR Changing Mindset
GDPR Changing Mindset
NetworkIQ
 
GDPR_Skillcast Presentation Template (1).pptx
GDPR_Skillcast Presentation Template (1).pptxGDPR_Skillcast Presentation Template (1).pptx
GDPR_Skillcast Presentation Template (1).pptx
kimonesinghunicomerc
 
skillcast-gdpr-training-presentation-q320.pptx
skillcast-gdpr-training-presentation-q320.pptxskillcast-gdpr-training-presentation-q320.pptx
skillcast-gdpr-training-presentation-q320.pptx
RahulGarg294918
 
GDPR: Are you EU Compliant?
GDPR: Are you EU Compliant? GDPR: Are you EU Compliant?
GDPR: Are you EU Compliant?
GreenRope
 
GDPR for Security Professionals
GDPR for Security ProfessionalsGDPR for Security Professionals
GDPR for Security Professionals
Saumya Vishnoi
 
GDPR changes affect direct marketing
GDPR changes affect direct marketingGDPR changes affect direct marketing
GDPR changes affect direct marketing
Spotler
 
How the EU-GDPR May Affect Your Website
How the EU-GDPR May Affect Your WebsiteHow the EU-GDPR May Affect Your Website
How the EU-GDPR May Affect Your Website
SilverTech
 
GDPR Data Subject Rights - What You Need to Know
GDPR Data Subject Rights - What You Need to KnowGDPR Data Subject Rights - What You Need to Know
GDPR Data Subject Rights - What You Need to Know
Piwik PRO
 
GDPR for Dummies
GDPR for DummiesGDPR for Dummies
GDPR for Dummies
Caroline Boscher
 
Things to know about GDPR in 2018
Things to know about GDPR in 2018Things to know about GDPR in 2018
Things to know about GDPR in 2018
Webkul Software Pvt. Ltd.
 

Similar to An AI-assisted Approach for Checking the Completeness of Privacy Policies Against GDPR (20)

Are you GDPRed yet?
Are you GDPRed yet?Are you GDPRed yet?
Are you GDPRed yet?
 
An Overview of GDPR
An Overview of GDPR An Overview of GDPR
An Overview of GDPR
 
An Overview of GDPR by Pathway Group
An Overview of GDPR by Pathway GroupAn Overview of GDPR by Pathway Group
An Overview of GDPR by Pathway Group
 
GDPR - what you need to know
GDPR -  what you need to know GDPR -  what you need to know
GDPR - what you need to know
 
Gdpr zilla
Gdpr zillaGdpr zilla
Gdpr zilla
 
Gdpr in a nutshell
Gdpr in a nutshellGdpr in a nutshell
Gdpr in a nutshell
 
Reddico GDPR Presentation
Reddico GDPR PresentationReddico GDPR Presentation
Reddico GDPR Presentation
 
Understanding the EU's new General Data Protection Regulation (GDPR)
Understanding the EU's new General Data Protection Regulation (GDPR)Understanding the EU's new General Data Protection Regulation (GDPR)
Understanding the EU's new General Data Protection Regulation (GDPR)
 
Opportunity or burden
Opportunity or burdenOpportunity or burden
Opportunity or burden
 
GDPR: Training Materials by Qualsys
GDPR: Training Materials  by QualsysGDPR: Training Materials  by Qualsys
GDPR: Training Materials by Qualsys
 
GDPR Changing Mindset
GDPR Changing MindsetGDPR Changing Mindset
GDPR Changing Mindset
 
GDPR_Skillcast Presentation Template (1).pptx
GDPR_Skillcast Presentation Template (1).pptxGDPR_Skillcast Presentation Template (1).pptx
GDPR_Skillcast Presentation Template (1).pptx
 
skillcast-gdpr-training-presentation-q320.pptx
skillcast-gdpr-training-presentation-q320.pptxskillcast-gdpr-training-presentation-q320.pptx
skillcast-gdpr-training-presentation-q320.pptx
 
GDPR: Are you EU Compliant?
GDPR: Are you EU Compliant? GDPR: Are you EU Compliant?
GDPR: Are you EU Compliant?
 
GDPR for Security Professionals
GDPR for Security ProfessionalsGDPR for Security Professionals
GDPR for Security Professionals
 
GDPR changes affect direct marketing
GDPR changes affect direct marketingGDPR changes affect direct marketing
GDPR changes affect direct marketing
 
How the EU-GDPR May Affect Your Website
How the EU-GDPR May Affect Your WebsiteHow the EU-GDPR May Affect Your Website
How the EU-GDPR May Affect Your Website
 
GDPR Data Subject Rights - What You Need to Know
GDPR Data Subject Rights - What You Need to KnowGDPR Data Subject Rights - What You Need to Know
GDPR Data Subject Rights - What You Need to Know
 
GDPR for Dummies
GDPR for DummiesGDPR for Dummies
GDPR for Dummies
 
Things to know about GDPR in 2018
Things to know about GDPR in 2018Things to know about GDPR in 2018
Things to know about GDPR in 2018
 

More from Lionel Briand

Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive Goal
Lionel Briand
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
Lionel Briand
 
Metamorphic Testing for Web System Security
Metamorphic Testing for Web System SecurityMetamorphic Testing for Web System Security
Metamorphic Testing for Web System Security
Lionel Briand
 
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Lionel Briand
 
Fuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingFuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation Testing
Lionel Briand
 
Data-driven Mutation Analysis for Cyber-Physical Systems
Data-driven Mutation Analysis for Cyber-Physical SystemsData-driven Mutation Analysis for Cyber-Physical Systems
Data-driven Mutation Analysis for Cyber-Physical Systems
Lionel Briand
 
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled SystemsMany-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Lionel Briand
 
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
Lionel Briand
 
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Lionel Briand
 
PRINS: Scalable Model Inference for Component-based System Logs
PRINS: Scalable Model Inference for Component-based System LogsPRINS: Scalable Model Inference for Component-based System Logs
PRINS: Scalable Model Inference for Component-based System Logs
Lionel Briand
 
Revisiting the Notion of Diversity in Software Testing
Revisiting the Notion of Diversity in Software TestingRevisiting the Notion of Diversity in Software Testing
Revisiting the Notion of Diversity in Software Testing
Lionel Briand
 
Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Applications of Search-based Software Testing to Trustworthy Artificial Intel...Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Lionel Briand
 
Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
Autonomous Systems: How to Address the Dilemma between Autonomy and SafetyAutonomous Systems: How to Address the Dilemma between Autonomy and Safety
Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
Lionel Briand
 
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Lionel Briand
 
Reinforcement Learning for Test Case Prioritization
Reinforcement Learning for Test Case PrioritizationReinforcement Learning for Test Case Prioritization
Reinforcement Learning for Test Case Prioritization
Lionel Briand
 
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
Lionel Briand
 
On Systematically Building a Controlled Natural Language for Functional Requi...
On Systematically Building a Controlled Natural Language for Functional Requi...On Systematically Building a Controlled Natural Language for Functional Requi...
On Systematically Building a Controlled Natural Language for Functional Requi...
Lionel Briand
 
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
Lionel Briand
 
Guidelines for Assessing the Accuracy of Log Message Template Identification ...
Guidelines for Assessing the Accuracy of Log Message Template Identification ...Guidelines for Assessing the Accuracy of Log Message Template Identification ...
Guidelines for Assessing the Accuracy of Log Message Template Identification ...
Lionel Briand
 
A Theoretical Framework for Understanding the Relationship between Log Parsin...
A Theoretical Framework for Understanding the Relationship between Log Parsin...A Theoretical Framework for Understanding the Relationship between Log Parsin...
A Theoretical Framework for Understanding the Relationship between Log Parsin...
Lionel Briand
 

More from Lionel Briand (20)

Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive Goal
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
 
Metamorphic Testing for Web System Security
Metamorphic Testing for Web System SecurityMetamorphic Testing for Web System Security
Metamorphic Testing for Web System Security
 
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
 
Fuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingFuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation Testing
 
Data-driven Mutation Analysis for Cyber-Physical Systems
Data-driven Mutation Analysis for Cyber-Physical SystemsData-driven Mutation Analysis for Cyber-Physical Systems
Data-driven Mutation Analysis for Cyber-Physical Systems
 
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled SystemsMany-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
 
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
 
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
 
PRINS: Scalable Model Inference for Component-based System Logs
PRINS: Scalable Model Inference for Component-based System LogsPRINS: Scalable Model Inference for Component-based System Logs
PRINS: Scalable Model Inference for Component-based System Logs
 
Revisiting the Notion of Diversity in Software Testing
Revisiting the Notion of Diversity in Software TestingRevisiting the Notion of Diversity in Software Testing
Revisiting the Notion of Diversity in Software Testing
 
Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Applications of Search-based Software Testing to Trustworthy Artificial Intel...Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Applications of Search-based Software Testing to Trustworthy Artificial Intel...
 
Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
Autonomous Systems: How to Address the Dilemma between Autonomy and SafetyAutonomous Systems: How to Address the Dilemma between Autonomy and Safety
Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
 
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
 
Reinforcement Learning for Test Case Prioritization
Reinforcement Learning for Test Case PrioritizationReinforcement Learning for Test Case Prioritization
Reinforcement Learning for Test Case Prioritization
 
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
 
On Systematically Building a Controlled Natural Language for Functional Requi...
On Systematically Building a Controlled Natural Language for Functional Requi...On Systematically Building a Controlled Natural Language for Functional Requi...
On Systematically Building a Controlled Natural Language for Functional Requi...
 
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
 
Guidelines for Assessing the Accuracy of Log Message Template Identification ...
Guidelines for Assessing the Accuracy of Log Message Template Identification ...Guidelines for Assessing the Accuracy of Log Message Template Identification ...
Guidelines for Assessing the Accuracy of Log Message Template Identification ...
 
A Theoretical Framework for Understanding the Relationship between Log Parsin...
A Theoretical Framework for Understanding the Relationship between Log Parsin...A Theoretical Framework for Understanding the Relationship between Log Parsin...
A Theoretical Framework for Understanding the Relationship between Log Parsin...
 

Recently uploaded

TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
Tier1 app
 
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Anthony Dahanne
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
AMB-Review
 
Graphic Design Crash Course for beginners
Graphic Design Crash Course for beginnersGraphic Design Crash Course for beginners
Graphic Design Crash Course for beginners
e20449
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Globus
 
Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604
Fermin Galan
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
wottaspaceseo
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
Globus
 
Enterprise Resource Planning System in Telangana
Enterprise Resource Planning System in TelanganaEnterprise Resource Planning System in Telangana
Enterprise Resource Planning System in Telangana
NYGGS Automation Suite
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Globus
 
Lecture 1 Introduction to games development
Lecture 1 Introduction to games developmentLecture 1 Introduction to games development
Lecture 1 Introduction to games development
abdulrafaychaudhry
 
Cyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdfCyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdf
Cyanic lab
 
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
informapgpstrackings
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
Globus
 
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume MontevideoVitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
takuyayamamoto1800
 
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxTop Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
rickgrimesss22
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Globus
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
Adele Miller
 
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of PassageA Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
Philip Schwarz
 

Recently uploaded (20)

TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
 
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
 
Graphic Design Crash Course for beginners
Graphic Design Crash Course for beginnersGraphic Design Crash Course for beginners
Graphic Design Crash Course for beginners
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
 
Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604Orion Context Broker introduction 20240604
Orion Context Broker introduction 20240604
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
Enterprise Resource Planning System in Telangana
Enterprise Resource Planning System in TelanganaEnterprise Resource Planning System in Telangana
Enterprise Resource Planning System in Telangana
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
 
Lecture 1 Introduction to games development
Lecture 1 Introduction to games developmentLecture 1 Introduction to games development
Lecture 1 Introduction to games development
 
Cyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdfCyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdf
 
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume MontevideoVitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume Montevideo
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
 
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxTop Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
 
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of PassageA Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
 

An AI-assisted Approach for Checking the Completeness of Privacy Policies Against GDPR

  • 1. .lusoftware verification & validation VVS An AI-assisted Approach for Checking the Completeness of Privacy Policies Against GDPR Zurich, September 3rd, 2020 Damiano Torre1, Sallam Abualhaija1, Mehrdad Sabetzadeh2,1, Lionel Briand1,2 {damiano.torre, sallam.abualhaija}@uni.lu {msabetza, lbriand}@uottawa.ca 1 University of Luxembourg, Luxembourg 2 University of Ottawa, Canada Katrien Baetens, Peter Goes, Sylvie Forastier {katrien.baetens, peter.goes, sylvie.forastier}@linklaters.com Linklaters, Luxembourg
  • 3. General Data Protection Regulation • A complex EU regulation to harmonize data protection and privacy in the European Union and the European Economic Area • Every organization, EU-based or not, must comply with GDPR as long as it collects and processes personal data of EU citizens and residents • Failure to comply with GDPR may result in fines of up to €20m or 4% of an organization's global turnover for specific breaches 3
  • 4. Industrial Motivation • Linklaters is a global law firm, headquartered in London with a base in Luxembourg • Most of GDPR compliance checking procedures are carried out manually • Manual checking is time consuming • Effective automated compliance checking is needed 4 https://www.linklaters.com
  • 5. Privacy Policy Document stating how an organization handles personal data 5 How many privacy policies have you read so far before clicking on “I agree”? Twitter @bertsesame
  • 6. Example of Checking To comply with the regulation, the privacy policy should include: 6 Article 13.2. (b) the existence of the right to request from the controller access to and rectification or erasure of personal data or restriction of processing concerning the data subject […]. CompletenessCompliance
  • 7. Research Questions RQ1: What are the metadata types required for checking the completeness of a privacy policy according to GDPR? Ø Building a conceptual model of GDPR privacy-related requirements RQ2: How can the metadata required for completeness checking of a privacy policy be extracted automatically? Ø Developing automated metadata identification approach using NLP & ML RQ3: How accurately can we extract metadata from privacy policies? Ø Evaluating our metadata identification approach RQ4: How accurately can we check the completeness of privacy policies? Ø Evaluating our completeness checking approach 7
  • 8. Approach (RQ1) Building a Conceptual Model of Privacy Policies Metadata
  • 10. From GDPR to Metadata Model 10 (b) the existence of the right to request from the controller access to and rectification or erasure of personal data or restriction of processing concerning the data subject […]. Article 13.2. The controller shall, at the time when personal data are obtained, provide the data subject with the following further information to ensure fair and transparent processing: DATA_SUBJECT_RIGHT ACCESS RECTIFICATIONRESTRICTIONERASURE
  • 11. DATA SUBJECT RIGHT 11 RQ1: What are the metadata types required for checking the completeness of a privacy policy according to GDPR? RESTRICTION COMPLAINT ERASURE OBJECT PORTABILITY WITHDRAW CONSENT ACCESS RECTIFICATION SA DIRECT INDIRECT PD ORIGIN PUBLICLY COOKIE THIRD-PARTY
  • 12. 12 Glossary Completeness Criteria PD ORIGIN INDIRECT not needed INDIRECT THIRD-PARTY found INDIRECT PUBLICLY found 2 Is PD ORIGIN INDIRECT found? PD ORIGIN INDIRECT not found PD ORIGIN INDIRECT partially found 3 Is INDIRECT THIRD-PARTY found? 4 Is INDIRECT PUBLICLY found? Start Final No Yes 1 Will data be collected indirectly? No Yes Yes Yes No No
  • 13. Approach (RQ2) Identifying Metadata in Privacy Policies
  • 14. 14 Keywords Training Data Pre- processing 1 Pre-trained Classifiers Identified Metadata Pre-trained Word Embeddings Prediction 7 Similarity-based Classification 5 Vectorization 3 KW-based Classification 6 ML-based Classification 4 Pre- processing 1 Post- processing 8 Text Generalization 2 Privacy Policy RQ2: How can the metadata required for completeness checking of a privacy policy be extracted automatically?
  • 15. Applying the NLP pipeline Unit of Analysis: Sentence DATA PROTECTION NOTICE The protection of your personal data is important to the BNP Paribas Group in France, which has adopted strong principles in that respect for the entire Group. This data protection notice provides you with detailed information relating to the protection of your personal data by the BNP Paribas Asset Management entities listed in the appendix, which are all subject to this data protection notice (hereunder referred to as “we”). We are responsible, as a controller, for collecting and processing your personal data in relation to our activities. DATA PROTECTION NOTICE The protection of your personal data is important to the BNP Paribas Group in France, which has adopted strong principles in that respect for the entire Group. This data protection notice provides you with detailed information relating to the protection of your personal data by the BNP Paribas Asset Management entities listed in the appendix, which are all subject to this data protection notice (hereunder referred to as “we”). We are responsible, as a controller, for collecting and processing your personal data in relation to our activities. WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM? In accordance with applicable regulations, you have the following rights: • To access: you can obtain information relating to the processing of your personal data, and a copy of such personal data. • To rectify: where you consider that your personal data are inaccurate or incomplete, you can require that such personal data be modified accordingly. • To erase: you can require the deletion of your personal data, to the extent permitted by law. • To restrict: you can request the restriction of the processing of your personal data. • To object: you can object to the processing of your personal data, on grounds relating to your particular situation. You have the absolute right to object to the processing of your personal data for direct marketing purposes, which includes profiling related to such direct marketing. • To withdraw your consent: where you have given your consent for the processing of your personal data, you have the right to withdraw your consent at any time. • To data portability: where legally applicable, you have the right to have the personal data you have provided to us be returned to you or, where technically feasible, transferred to a third party. If you wish to exercise the rights listed above, please send a letter to the following address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009 PARIS, FRANCE or an email using dataprotection@bnpparibas.com. In accordance with applicable regulation, in addition to your rights above, you are also entitled to lodge a complaint with the competent supervisory authority. Pre-processing 15 Stopwords Removal The protection of your personal data is important to the BNP Paribas Group in France, which has adopted strong principles in that respect for the entire Group. protection personal data important BNP Paribas Group France, adopt strong principle respect entire Group. Lemmatization
  • 16. Text Generalization Replacing specific textual entities with more general ones Ø Locations, organizations, websites, telephone numbers, e-mail and addresses 16 DATA PROTECTION NOTICE The protection of your personal data is important to the BNP Paribas Group in France, which has adopted strong principles in that respect for the entire Group. This data protection notice provides you with detailed information relating to the protection of your personal data by the BNP Paribas Asset Management entities listed in the appendix, which are all subject to this data protection notice (hereunder referred to as “we”). We are responsible, as a controller, for collecting and processing your personal data in relation to our activities. WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM? In accordance with applicable regulations, you have the following rights: • To access: you can obtain information relating to the processing of your personal data, and a copy of such personal data. • To rectify: where you consider that your personal data are inaccurate or incomplete, you can require that such personal data be modified accordingly. • To erase: you can require the deletion of your personal data, to the extent permitted by law. • To restrict: you can request the restriction of the processing of your personal data. • To object: you can object to the processing of your personal data, on grounds relating to your particular situation. You have the absolute right to object to the processing of your personal data for direct marketing purposes, which includes profiling related to such direct marketing. • To withdraw your consent: where you have given your consent for the processing of your personal data, you have the right to withdraw your consent at any time. • To data portability: where legally applicable, you have the right to have the personal data you have provided to us be returned to you or, where technically feasible, transferred to a third party. If you wish to exercise the rights listed above, please send a letter to the following address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009 PARIS, FRANCE or an email using dataprotection@bnpparibas.com. In accordance with applicable regulation, in addition to your rights above, you are also entitled to lodge a complaint with the competent supervisory authority. protection personal data important BNP Paribas Group France adopt strong principle respect entire Group. protection personal data important ORGANIZATION LOCATION adopt strong principle respect entire Group. exercise right list please send letter following address data protection officer ORGANIZATION ADDRESS email use EMAIL.
  • 17. Transforming the sentences into 100-dimentional vectors Ø Using GloVe pre-trained models Vectorization 17 DATA PROTECTION NOTICE The protection of your personal data is important to the BNP Paribas Group in France, which has adopted strong principles in that respect for the entire Group. This data protection notice provides you with detailed information relating to the protection of your personal data by the BNP Paribas Asset Management entities listed in the appendix, which are all subject to this data protection notice (hereunder referred to as “we”). We are responsible, as a controller, for collecting and processing your personal data in relation to our activities. WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM? In accordance with applicable regulations, you have the following rights: • To access: you can obtain information relating to the processing of your personal data, and a copy of such personal data. • To rectify: where you consider that your personal data are inaccurate or incomplete, you can require that such personal data be modified accordingly. • To erase: you can require the deletion of your personal data, to the extent permitted by law. • To restrict: you can request the restriction of the processing of your personal data. • To object: you can object to the processing of your personal data, on grounds relating to your particular situation. You have the absolute right to object to the processing of your personal data for direct marketing purposes, which includes profiling related to such direct marketing. • To withdraw your consent: where you have given your consent for the processing of your personal data, you have the right to withdraw your consent at any time. • To data portability: where legally applicable, you have the right to have the personal data you have provided to us be returned to you or, where technically feasible, transferred to a third party. If you wish to exercise the rights listed above, please send a letter to the following address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009 PARIS, FRANCE or an email using dataprotection@bnpparibas.com. In accordance with applicable regulation, in addition to your rights above, you are also entitled to lodge a complaint with the competent supervisory authority. data protection noticePre-processed sentence protection data notice [ -0.4710, 0.6158, 0.6897, -0.1815, 0.3078, -0.8415, -0.4187, -0.2001, 0.2818, -0.3401, 0.7729, -0.2277, 0.0599, -0.2414, 0.8778, 0.7204, 0.6429, 0.3625, 0.4162, 0.1300, -0.4707, -0.4466, 0.4736, 0.4076, -1.0341, -1.1422, 0.3744, 0.2463, -0.6729, 0.4918, 0.4651, 0.1361, -0.9380, 0.5189, 0.5155, -0.2651, -0.1455, 0.2252, 0.3524, -0.7965, -0.4225, -0.9059, -0.8400, 0.4536, -0.7249, -0.1259, 0.4366, -0.5366, 0.0205, -0.7461, 1.1925, 0.1572, 0.2932, 0.9266, 0.4824, -1.8290, -0.0127, -0.3703, 2.3618, 0.3359, -0.1544, 0.1466, -0.1131, -0.0249, 0.3193, 0.2882, -0.2963, -0.3303, 1.4774, 0.2374, -0.2531, 0.6137, 0.5681, -0.5699, 0.4880, 0.0654, 0.2826, -0.1354, -1.1096, -0.3597, 0.8531, 0.4630, -1.1223, 0.0072, -1.7636, -0.4455, 1.2478, -0.3754, -0.2163, 0.4594, -0.1139, 0.7558, -0.2442, -0.0565, 0.5479, -0.3093, 0.2592, -0.5961, 0.2760, 0.0880] [ 0.0330, -0.2653, 0.2386, -0.1449, 0.4572, -0.2129, -0.6080, -0.4135, 0.0960, -0.2114, -0.0625, -0.1854, 0.0909, -0.2509, 0.0931, -0.2927, 0.9347, -0.1482, -0.9635, 0.0151, -0.3910, -0.2225, -0.1056, 0.4497, 0.0575, 0.3878, -0.2755, -0.5580, -0.9241, -0.4803, 0.3122, -0.0470, -0.0105, 0.4453, 0.3733, -0.2642, -0.0194, 0.0778, -0.0885, 0.0392, -0.1973, -0.3003, 0.2938, -0.2649, 0.5680, -0.2838, -0.4947, 0.3432, -0.4117, -0.9482, 0.5263, 0.1430, -0.7211, 1.2293, 0.4163, -0.9417, 0.8058, -1.0777, 2.4755, 0.3097, 0.1849, 0.3422, -0.2709, 0.2793, 1.0019, 0.0157, -0.4184, -0.1016, 0.8306, -0.3263, 0.1229, -0.4821, 0.3061, -0.1621, 0.7334, -0.1168, -0.3701, 0.1795, -1.1684, 0.1200, 0.8329, -0.1387, 0.0737, 0.5353, -1.1984, 0.7402, 0.1241, 0.0189, 0.2222, -0.0001, 1.0163, -1.2003, 0.0576, 0.1005, 0.3886, 0.6566, -0.2975, -0.0867, 0.5531, -0.3809] [ 0.1377, -0.5041, -0.1082, -0.2499, -0.2414, 0.3749, -0.5977, 0.4089, 0.3198, -0.4266, 0.2197, 0.0467, -0.0241, -0.3295, 0.1505, -0.0197, -0.0795, 0.3808, -0.0841, 0.0600, -0.2723, 0.3564, -0.5089, -0.5272, 0.1763, 0.0837, -0.0244, -0.0261, 0.2384, -0.6271, 0.4758, 0.0080, 0.1979, -0.2095, -0.4243, 0.5012, -0.2161, -0.7097, -0.0681, -0.1287, -0.2156, 0.6505, 0.3222, -0.4843, 0.1309, -0.2888, -0.0246, -0.1711, 0.2656, -1.0098, 1.1561, -0.2706, -0.1719, 0.4489, 0.0501, -1.2639, -0.0260, -0.6504, 1.7581, 0.1059, -0.3542, 0.8539, -0.5211, 0.0547, 0.8483, -0.1820, -0.2561, 0.1676, -0.2452, -0.0495, -0.4471, -0.2160, 0.2036, -0.5234, 0.3364, 0.9539, -0.0427, -0.3200, -0.9549, -0.4273, 0.1453, -1.0337, -0.5458, -0.2693, -1.0983, -0.4366, 0.3313, -0.0407, 0.1030, 0.1724, 0.0570, -0.2919, 0.0124, -0.3216, 0.0375, 0.6194, -0.0108, 0.0652, -0.3073, 0.1408] Corresponding 100-D vector [ -0.1001, -0.0512, 0.2734, -0.1921, 0.1745, -0.2265, -0.5415, -0.0682, 0.2326, -0.3260, 0.3100, -0.1221, 0.0422, -0.2739, 0.3738, 0.1360, 0.4994, 0.1984, -0.2105, 0.0684, -0.3780, -0.1042, -0.0470, 0.1100, -0.2668, -0.2236, 0.0248, -0.1126, -0.4529, -0.2052, 0.4177, 0.0324, -0.2502, 0.2516, 0.1548, -0.0094, -0.1270, -0.1356, 0.0653, -0.2953, -0.2785, -0.1852, -0.0747, -0.0985, -0.0087, -0.2328, -0.0276, -0.1215, -0.0419, -0.9014, 0.9583, 0.0099, -0.1999, 0.8683, 0.3162, -1.3449, 0.2557, -0.6995, 2.1985, 0.2505, -0.1079, 0.4475, -0.3017, 0.1030, 0.7232, 0.0406, -0.3236, -0.0881, 0.6876, -0.0461, -0.1925, -0.0281, 0.3593, -0.4185, 0.5193, 0.3008, -0.0434, -0.0920, -1.0776, -0.2223, 0.6105, -0.2365, -0.5315, 0.0911, -1.3534, -0.0473, 0.5677, -0.1324, 0.0363, 0.2106, 0.3198, -0.2455, -0.0581, -0.0925, 0.3247, 0.3222, -0.0164, -0.2059, 0.1739, -0.0507]
  • 18. Prediction 18 Keywords Training Data Pre- processing 1 Pre-trained Classifiers Identified Metadata Pre-trained Word Embeddings Prediction 7 Similarity-based Classification 5 Vectorization 3 KW-based Classification 6 ML-based Classification 4 Pre- processing 1 Post- processing 8 Text Generalization 2 Privacy Policy
  • 20. ML-based Classification • A binary classifier for each metadata type (levels 1&2) 20 DATA SUBJECT RIGHT Not DATA SUBJECT RIGHT Not DATA SUBJECT RIGHT Not DATA SUBJECT RIGHT DATA PROTECTION NOTICE The protection of your personal data is important to the BNP Paribas Group in France, which has adopted strong principles in that respect for the entire Group. This data protection notice enables you to obtain detailed information relating to the protection of your personal data by the BNP Paribas Asset Management entities listed in the appendix, which are all subject to this data protection notice (hereunder referred to as “we”). We are responsible, as a controller, for collecting and processing your personal data in relation to our activities.
  • 21. Checking semantic similarity between a sentence and a set of training examples of a metadata type (level 2) -0.1983, 0.1698, 0.3051, -0.0343, 0.2145, 0.0277, -0.3181, -0.1116, 0.1986, -0.2342, 0.2577, -0.1901, 0.3291, -0.1272, 0.1686, 0.0948, 0.4405, 0.2328, -0.2926, 0.1354, -0.3191, -0.1708, -0.0148, 0.0395, -0.2850, -0.3252, 0.0696, -0.1286, -0.2628, -0.0727, 0.2690, 0.0940, -0.2561, 0.0484, 0.0598, 0.1618, -0.0639, 0.0702, -0.1344, -0.0973, -0.1914, -0.0648, 0.0528, -0.1219, -0.1378, -0.1070, -0.0277, -0.0734, -0.1583, -0.6287, 0.7184, 0.2003, 0.0483, 0.7144, 0.0752, -1.2867, 0.3194, -0.4712, 1.7194, 0.3681, -0.1071, 0.4806, -0.2039, -0.0508, 0.8415, -0.0838, -0.0044, 0.0302, 0.4905, -0.0457, -0.1050, -0.1170, 0.2246, -0.2939, 0.3821, 0.1018, -0.1512, -0.1171, -0.8723, -0.2130, 0.4374, -0.0327, -0.2389, 0.0673, -1.2155, 0.0636, 0.2182, -0.1731, -0.0059, 0.0620, -0.0136, -0.0645, 0.0555, -0.1652, 0.1210, 0.1492, -0.1160, -0.3832, 0.2884, 0.0442" Similarity-based Classification 21 [ -0.2282, -0.1558, 0.0652, 0.0991, 0.1608, -0.1088, -0.2718, -0.0173, 0.1660, 0.1387, 0.1073, -0.1066, 0.2735, -0.0381, 0.1411, 0.0162, 0.1138, 0.2237, 0.0039, 0.3900, -0.2075, -0.3058, 0.1522, -0.1538, -0.3828, -0.2691, -0.0107, -0.0786, -0.0987, -0.0090, 0.1191, 0.3814, -0.2909, 0.0209, 0.0454, -0.1930, 0.0577, 0.0796, 0.1150, -0.2799, -0.2460, -0.3236, 0.1911, -0.0752, -0.3893, -0.0191, -0.0130, -0.2376, -0.1571, -0.6903, 0.5475, 0.0387, 0.0706, 0.9536, 0.1957, -1.3846, 0.0126, -0.2413, 1.5954, 0.3453, -0.0842, 0.3065, -0.0789, -0.0849, 0.6592, -0.0325, 0.0970, -0.0936, 0.7144, 0.0332, -0.0842, -0.1491, 0.2932, -0.2050, 0.2586, -0.0586, -0.0516, -0.0666, -0.7936, 0.0595, 0.4763, 0.3987, -0.4171, 0.1743, -1.2655, 0.1015, 0.3702, -0.1000, -0.0591, -0.3456, -0.2736, 0.2876, 0.0137, 0.0495, 0.3142, -0.0320, 0.0510, -0.5056, 0.5006, -0.1134] [ -0.2111, -0.0454, 0.3794, 0.1048, -0.1989, -0.0568, -0.3195, -0.1061, 0.3210, 0.0961, 0.1302, -0.0328, 0.1682, -0.2576, 0.1328, 0.0617, 0.2283, 0.3493, -0.0362, 0.2115, -0.1524, -0.3676, 0.2312, 0.1922, -0.4278, -0.5339, -0.2137, -0.0735, -0.2329, 0.1074, 0.3383, 0.2820, -0.5242, -0.0686, -0.0450, 0.1989, -0.1216, -0.1705, -0.2182, -0.1445, -0.1320, -0.1095, -0.1330, -0.0862, -0.6284, -0.2005, 0.2613, -0.1753, -0.2242, -0.7376, 0.7149, 0.1868, 0.2578, 0.5627, 0.0887, -1.1176, 0.5187, -0.2217, 1.3552, 0.0823, -0.2786, 0.2048, -0.1019, 0.1011, 0.6854, -0.2996, 0.1295, -0.0006, 0.2986, -0.2940, -0.1595, 0.1385, 0.3246, -0.3247, 0.3632, 0.1793, -0.2596, 0.1123, -0.7059, -0.2170, 0.4344, 0.1247, -0.4316, -0.0446, -1.1833, -0.0789, 0.4744, -0.1370, -0.4004, -0.1094, -0.2521, 0.2583, -0.0554, -0.2261, 0.1829, -0.1604, -0.0274, -0.4844, 0.3389, 0.0507] 0.46 0.93 Represented by an average vector Training examples on “ACCESS”
  • 22. Looking for the keywords of a metadata type that are present in the sentence DATA PROTECTION NOTICE The protection of your personal data is important to the BNP Paribas Group in France, which has adopted strong principles in that respect for the entire Group. This data protection notice enables you to obtain detailed information relating to the protection of your personal data by the BNP Paribas Asset Management entities listed in the appendix, which are all subject to this data protection notice (hereunder referred to as “we”). We are responsible, as a controller, for collecting and processing your personal data in relation to our activities. WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM? In accordance with applicable regulations, you have the following rights: • To access: you can obtain information relating to the processing of your personal data, and a copy of such personal data. • To rectify: where you consider that your personal data are inaccurate or incomplete, you can require that such personal data be modified accordingly. • To erase: you can require the deletion of your personal data, to the extent permitted by law. • To restrict: you can request the restriction of the processing of your personal data. • To object: you can object to the processing of your personal data, on grounds relating to your particular situation. You have the absolute right to object to the processing of your personal data for direct marketing purposes, which includes profiling related to such direct marketing. • To withdraw your consent: where you have given your consent for the processing of your personal data, you have the right to withdraw your consent at any time. • To data portability: where legally applicable, you have the right to have the personal data you have provided to us be returned to you or, where technically feasible, transferred to a third party. If you wish to exercise the rights listed above, please send a letter to the following address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009 PARIS, FRANCE or an email using dataprotection@bnpparibas.com. In accordance with applicable regulation, in addition to your rights above, you are also entitled to lodge a complaint with the competent supervisory authority. Keyword-based Classification 22 data protection notice enable obtain detail information relate protection personal data ORGANIZATION entity list appendix all subject data protection notice hereunder refer access obtain information relate processing personal data copy personal data Keywords of “ACCESS”
  • 23. Combining the results DATA PROTECTION NOTICE The protection of your personal data is important to the BNP Paribas Group in France, which has adopted strong principles in that respect for the entire Group. This data protection notice enables you to obtain detailed information relating to the protection of your personal data by the BNP Paribas Asset Management entities listed in the appendix, which are all subject to this data protection notice (hereunder referred to as “we”). We are responsible, as a controller, for collecting and processing your personal data in relation to our activities. WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM? In accordance with applicable regulations, you have the following rights: • To access: you can obtain information relating to the processing of your personal data, and a copy of such personal data. • To rectify: where you consider that your personal data are inaccurate or incomplete, you can require that such personal data be modified accordingly. • To erase: you can require the deletion of your personal data, to the extent permitted by law. • To restrict: you can request the restriction of the processing of your personal data. • To object: you can object to the processing of your personal data, on grounds relating to your particular situation. You have the absolute right to object to the processing of your personal data for direct marketing purposes, which includes profiling related to such direct marketing. • To withdraw your consent: where you have given your consent for the processing of your personal data, you have the right to withdraw your consent at any time. • To data portability: where legally applicable, you have the right to have the personal data you have provided to us be returned to you or, where technically feasible, transferred to a third party. If you wish to exercise the rights listed above, please send a letter to the following address data protection officer BNP Paribas Asset Management, 14 rue Bergere 75009 PARIS, FRANCE or an email using dataprotection@bnpparibas.com. In accordance with applicable regulation, in addition to your rights above, you are also entitled to lodge a complaint with the competent supervisory authority. Prediction 23 Keywords Training Data Pre- processing 1 Pre-trained Classifiers Identified Metadata Pre-trained Word Embeddings Prediction 7 Similarity-based Classification 5 Vectorization 3 KW-based Classification 6 ML-based Classification 4 Pre- processing 1 Post- processing 8 Text Generalization 2 Privacy Policy DATA SUBJECT RIGHT COMPLAINT SA DATA SUBJECT RIGHT ACCESS
  • 24. DATA PROTECTION NOTICE The protection of your personal data is important to the BNP Paribas Group in France, which has adopted strong principles in that respect for the entire Group. This data protection notice enables you to obtain detailed information relating to the protection of your personal data by the BNP Paribas Asset Management entities listed in the appendix, which are all subject to this data protection notice (hereunder referred to as “we”). We are responsible, as a controller, for collecting and processing your personal data in relation to our activities. Considering the contextual information Post-processing 24 Not DATA SUBJECT RIGHT Not DATA SUBJECT RIGHT Not DATA SUBJECT RIGHT DATA SUBJECT RIGHTDATA SUBJECT RIGHT Not DATA SUBJECT RIGHT
  • 26. Case Study DATA SUBJECT RIGHT and LEGAL BASIS 26 TO ENTER CONTRACT CONTRACTUAL STATUTORY CONSENT PUBLIC FUNCTION LEGITIMATE INTEREST VITAL INTEREST CONTRACT LEGAL OBLIGATION LEGAL BASIS DATA SUBJECT RIGHT RESTRICTION COMPLAINT ERASURE OBJECT PORTABILITY WITHDRAW CONSENT SA ACCESS RECTIFICATION
  • 27. Document Collection A total of 234 privacy policies, fully annotated Ø 90% training-set and 10% testset 27 DATA SUBJECT RIGHT # of Sentences ACCESS 228 COMPLAINT 196 SA 183 ERASURE 219 OBJECT 298 PORTABILITY 171 RECTIFICATION 215 RESTRICTION 179 WITHDRAW CONSENT 204 TOTAL 1710 LEGAL BASIS # of Sentences CONSENT 272 CONTRACT 265 TO ENTER CONTRACT 69 CONTRACTUAL 164 STATUTORY 16 LEGAL OBLIGATION 351 LEGITIMATE INTEREST 507 PUBLIC FUNCTION 71 VITAL INTEREST 13 TOTAL 1479
  • 28. Empirical Evaluation (RQ3) Metadata Identification Approach
  • 29. Evaluation Metrics • Precision (P): percentage of correctly identified metadata types by the approach – TP/(TP+FP) • Recall (R): percentage of correctly identified metadata types from the ones actually present in the privacy policies – TP/(TP+FN) 29 True Positive (TP) Metadata type is present in the privacy policy and is identified by the approach False Positive (FP) Metadata type is NOT present in the privacy policy but is identified by the approach False Negative (FN) Metadata type is present in the privacy policy but is NOT identified by the approach
  • 30. RQ3: How accurately can we extract metadata from a given privacy policy? 30 DATA SUBJECT RIGHT P(%) R(%) ACCESS 100 91 COMPLAINT 100 100 SA 100 100 ERASURE 100 89 OBJECT 94 94 PORTABILITY 100 100 RECTIFICATION 100 95 RESTRICTION 100 94 WITHDRAW CONSENT 100 94 LEGAL BASIS P(%) R(%) CONSENT 95 100 CONTRACT 90 95 TO ENTER CONTRACT 100 87 CONTRACTUAL 94 100 STATUTORY 83 100 LEGAL OBLIGATION 100 96 LEGITIMATE INTEREST 100 81 PUBLIC FUNCTION 75 60 VITAL INTEREST 100 80
  • 32. Case Study Completeness Criteria 32 DATA SUBJECT RIGHT RESTRICTION COMPLAINT ACCESS RECTIFICATION C1: Must always be present C2: C4:C3: DATA SUBJECT RIGHT ERASURE OBJECT PORTABILITY WITHDRAW CONSENT CONSENT LEGAL BASISIF THEN DATA SUBJECT RIGHT PORTABILITY CONTRACT LEGAL BASIS IF THEN DATA SUBJECT RIGHT OBJECT PUBLIC FUNCTION LEGITIMATE INTEREST LEGAL BASIS IF THEN https://www.clipartkey.co m/view/hJTTiw_muppet- wiki-sesame-street-bert/ If you give your consent for sharing your data, can you withdraw it?
  • 33. Example of Incompleteness Issue 33 DATA SUBJECT RIGHT RESTRICTION COMPLAINT ACCESS RECTIFICATION C1: Must always be present C3: DATA SUBJECT RIGHT PORTABILITY CONTRACT LEGAL BASIS IF THEN DATA SUBJECT RIGHT RESTRICTION COMPLAINT ACCESS RECTIFICATION DATA SUBJECT RIGHT PORTABILITY CONTRACT LEGAL BASIS IF THEN DATA PROTECTION NOTICE The protection of your personal data is important […] WHAT ARE YOUR RIGHTS AND HOW CAN YOU EXERCISE THEM? In accordance with applicable regulations, you have the following rights: • To access: you can obtain information relating to the processing of your personal data, and a copy of such personal data. • To rectify: where you consider that your personal data are inaccurate or incomplete, you can require that such personal data be modified accordingly. […] In accordance with applicable regulation, in addition to your rights above, you are also entitled to lodge a complaint with the competent supervisory authority. WHY AND ON WHICH BASIS DO WE USE YOUR PERSONAL DATA? a) To comply with our legal and regulatory obligations We use your personal data to comply with various legal and regulatory obligations, […] b) To perform a contract with you or to take steps at your request before entering into a contract We use your personal data to enter into and perform our contracts to provide you with information regarding our products and services.
  • 34. Evaluation Metrics • Precision (P): percentage of correctly identified incompleteness by the approach – TP/(TP+FP) • Recall (R): percentage of correctly identified incompleteness from what actually is in the privacy policies – TP/(TP+FN) 34 True Positive (TP) Incompleteness issue exists in the privacy policy and is identified by the approach False Positive (FP) Incompleteness issue does NOT exist in the privacy policy but is identified by the approach False Negative (FN) Incompleteness issue exists in the privacy policy but is NOT identified by the approach
  • 35. RQ4: How accurately can we check the completeness of a given privacy policy? On our test set (24 privacy policies): Ø45 out of 47 incompleteness issues are correctly identified ØFalse alarm on eight occasions 35 Criterion TPs FPs FNs P(%) R(%) C1 16 4 0 80 100 C2 5 1 0 83 100 C3 3 0 2 100 60 C4 21 3 0 88 100 Summary 45 8 2 85 96
  • 37. Thanks for listening! Time for questions. Participate in BERT’s quiz at this link: https://shorturl.at/ditES 37 https://tenor.com/search/bert-gifs