This document presents a study on automatically detecting protection-impacting changes (PICs) in WordPress and MediaWiki source code versions. The study involved analyzing 211 WordPress and 193 MediaWiki release pairs to identify PICs using an existing tool. Characteristics of identified PIC commits were analyzed and machine learning models were developed that achieved high accuracy in predicting PICs. Future work could involve introducing new security metrics to improve models and expanding the research to other systems beyond PHP.

1. Just-in-time Detection of Protection-Impacting
Changes on WordPress and MediaWiki
SWAT
Département de Génie Logiciel
École Polytechnique de Montréal
Novembre 2018
Mohamed Amine Barrak
Director : Foutse Khomh
Co-Director : Guiliano Antoniol

2. Outline
• Problem statement
• Related studies
• Introduction and
background
• Lack of existing work
• Case study design
• Case study results
• Summary and future works
2

3. 3
Problem Statement

4. 4
Problem Statement
Source: Info Security Magazine

5. 5
Problem Statement
Source: Open Web Application Security Project (OWASP)

6. 6
Problem Statement
Source: Open Web Application Security Project (OWASP)

7. 7
What is Broken Access Control Risk ?
Source: Direct Line Development Blog

8. Previous Studies
• MA Laverdière and E. Merlo (2018). “Detection of
protection-impacting changesduring software evolution”.
SANER 2018
• S. Georgios and A. Lefteris and T. Dimitrios (2017). “
Assessment of vul-nerability severity using text mining”.
ACM 2017.
• An, Le and Khomh, Foutse (2015a). “An empirical study of
crash-inducing commits in mozilla“, PROMISE 2015
8

9. Introduction and Background
9

10. Role-Based Access Control (RBAC)
10
Source: https://www.idenhaus.com/how-integrate-role-based-access-control-group-environment/

11. Role-Based Access Control (RBAC)
11
HR Manager Developer
Commit-
Code View-
Sprint
Hire
Approve-
Budget
Create-Sprint
View-CVs
View-Performance
=
Divides who can do what using groups
and privileges
Product Manager
Commercialise
product
Access to sales
revenue

12. • Access control is implemented in code source
• Code changes may affect privilege protection
Code change causing protection-
impacting change
VersionA VersionB
Codechange
f unction f o o ( ) {
/ / assumes pr i v i l ege p1 he ld
}
fo o ( );
if (!current_user_can(’p1’)) die();
foo();
Protected for p1
Application-specificstereotypicalAPI
}
if (!current_user_can(’p1’)) die();
foo();
Protected for p1
f unction f o o ( ) {
/ / assumes pr i v i l ege p1 he ld
10

13. How to detect Lines of Protection
Impacting Changes (Existing tool)?
13

14. • Compose source code to Abstract Syntax Tree (AST)
• Convert AST to inter-procedural Control Flow Graph (CFGs)
• Link CFGs to be intra-procedural CFGs (called PTFA)
Pattern Traversal Flow Analysis (PTFA)
14
Determine privilege protection on all execution paths

15. • Occurs when definite privilege protection for a statement s
that is common to Version A and Version B but differs
between versions.
Definite Protection Difference (DPD)
15
VersionA VersionB
Codechange
f unction f o o ( ) {
/ / assumes pr i v i l ege p1 he ld
}
+ fo o ( );
if (!current_user_can(’p1’)) die();
foo();
Protected for p1
Application-specificstereotypicalAPI
}
if (!current_user_can(’p1’)) die();
foo();
Protected for p1
f unction f o o ( ) {
/ / assumes pr i v i l ege p1 he ld

16. Protection-Impacting Changes (Oracle)
16
PICAnalyzer
Code
Differences
Definite
Protection
Differences
Protection
Impacting
Changes
(PIC)
PTFA
Models
Output
Input
Protection
Impacting
Change Oracle
dataset
V1 V2 gain/ loss filename line_number
Output format:
Source
Code
Version
Pairs

17. Lack of Existing Work
and Our Goal
17

18. 18
Cost of Security Defect related to
development process?
• Cost of a Data Breach $7.2M
from law suits, loss of customer
trust, damage to brand
Find during Development
$80 / defect
Find during Build
$240 / defect
Find during QA/Test
$960 / defect
Find in Production
$7,600 / defect
Source: Protecting Mission-Critical Source Code from Application Security Vulnerabilities (IBM study)
Source: PonemonInstitute
Source: National Institute of Standards and Technology

19. • Delay of reporting PICs lines, take almost 10 minutes to
generate results of PICs per release pair
• Cost of executing the tool at each modification in the
source code
• Applied after the vulnerability occurrence and already
affected a large population of users
• Average Cost of a Data Breach $7.2M from law suits,
loss of customer trust, damage to brand
Lack of Existing Works
19

20. • Automatically identifying protection impacting change vulnerability before
introducing it in the system
• Implement an automated and just-in-time tool in the coding phase
• Usage of existing work to create machine learning algorithm
Our Goal: Prevention is better
than a cure
20
Source: http://ehealth.eletsonline.com/2016/08/preventive-cure-prevention-is-better-than-cure/
Create automated and Predictive
Models of Protection Impacting
Change

21. Case Study Design
21

22. 22
Choice of Projects for PIC analysis
(CVE : Common Vulnerabilities and
Exposures)
WordPress Vulnerabilities MediaWiki Vulnerabilities
High
percentage
High
percentage

23. Software Configuration Management
23
Branch-by release model

24. Software Configuration Management
24
Branch-by purpose model.

25. • Manual analysis of Releases tree
– Verify the adapted software configuration
management
– Choose a consecutive versions to minimize the
complexity of the analysis
– Make sure that the selected pair versions are not in
different branches
Attention: Several studies are comparing code source in
a system without paying attention to the unrelated
brunches.
How we choose pair releases for PIC
detection?
25
3.7.1
3.7.2 3.8.0
3.7.3
3.8.1
3.7.4 3.8.2
3.7.5 3.8.3
3.7.6 3.8.4 3.9.0
3.7.7 3.8.5 3.9.1
WordPress Release Tree

26. • Analysis include
– 211 releases pairs of WordPress from (2.0 to 4.7.3)
– 193 releases pairs of MediaWiki from (1.5.0 to 1.29.2)
• Mature open source systems (WordPress and
MediaWiki)
– Long release history and available archive
– Implement RBAC approach
– Important LOC
Data Collection
26

27. • Run Protection Impacting Changes tool to report
Protection-Impacting Lines
• Readable results for end user
– V1 V2 gain/ loss filename line_number
• Consider vulnerable lines as an Oracle
Protection-Impacting Change Lines
27

28. From PIC lines to PIC commits
28
• From protection impacting line to PIC commit
– B : Bifurcation point between V1 and V2
– C’ : Last commit that modified line_number
between V1 and V2
• Commit that modified privilege protection
– “git blame -L line_number B..C’ -- filename “
• Commits that removed privilege protection
– “git blame --reverse -L line_number B..C’ --
filename ”

29. From PIC lines to PIC commits
29
• Example:
– V1: 2.0.2 V2: 2.0.3
– B : ba8bb5af9e
– C’ : e1172126a1
– modified line is: 11
– Deleted line: 213
– Modified file : user-edit.php
• Commit that modified privilege protection
“git blame -L 11 ba8bb5af9e..e1172126a1 -- user-
edit.php“
• Commits that removed privilege protection
“git blame --reverse -L 11
ba8bb5af9e..e1172126a1 -- user-edit.php”
Bifurcation
V2
V1

30. 30

31. • Date (week day, month day, Month)
• Author experience
• Commit size:
– Number of changed files
– Number of added and deleted lines
• Message size
• Is Bug fix ?
Commit Log metrics (structure of
commit code)
31

32. • Undertand tool (Source Code Analysis & Metrics)
– LOC
– Cyclomatic complexity McCabe
– Maximum nesting
– Number of declarative statements
– Number of functions
– Number of blank lines
– Ratio of comment lines over all lines in a file
Code Complexity metrics (code quality)
32

33. Research Questions
33
RQ1) What is the proportion of protection-impacting changes in
Wordpress and MediaWiki?
RQ2) What are the characteristics of protection-impacting changes?
RQ3) To which extent can we predict protection-impacting changes?
RQ4) Why do automatic machine learning models misclassify some
protection-impacting changes?

34. Case Study Results
34

35. RQ1:What is the proportion of
protection-impacting changes in
Wordpress and MediaWiki?
35

36. Proportion of Protection-Impacting
changes commits in WordPress and
MediaWiki
36
PIC commit does not mean that
all its changed lines are
vulnerable
Software developers should strive to catch PIC commits as soon as
possible

37. RQ2: What are the characteristics
of protection-impacting changes?
37

38. Characteristics of PIC commits
38
Metrics PIC Non-PIC PIC Non-PIC
Message size 19 15 27 18
Author experience 1531 1343 1378 1012
Cyclomatic Complexity 9.9 7.9 2.4 2.2
Max Nesting 3.6 3.4 3.3 1.6
Declarative statement 42.1 37.7 57.3 27.8
Is bug fix 20% 21.7% 29.4% 23.4%
PN: More results are in the
paper

39. RQ3: To which extent can we predict
protection-impacting changes?
39

40. Steps of Building the Machine Learning
Infrastructure
40
Source: https://www.7wdata.be/big-data/building-the-machine-learning-infrastructure/
• Eliminate Collinearity between
variables using Variance Inflation
Factor (VIF)
• Divide your dataset into training set
and test set with 10-cross
validation technique
• Apply Machine Learning Algorithms
to predict the PIC variable.
• Evaluate models by reporting
(precision, recall, accuracy, etc)

41. Machine Learning Algorithm Results
41
Random Forest algorithm achieve best prediction performance in both
projects

42. RQ4: Why do automatic machine
learning models misclassify some
protection-impacting changes?
42

43. 43
Qualitative Observations Of Wrongly
Classified Commits
Adding more information to the model will increase the prediction
performance

44. Conclusion
There is a high proportion of PIC commits that engineers
should take care of
PICs commits are submitted by experienced developers.
They contain longer commit messages and make
complex changes in files
High accuracy of predicting PIC commits using ML
algorithms, then improving research by applying a
qualitative analysis to know causes of misclassification
44

45. Future Work
Introduce new security metrics to improve our models
such as Developer Activity Metrics
Improve our actual models by ignoring commits of
pushing versions or changing JS, HTML or CSS files.
Expand our research not only on PHP software systems,
and with more Oracle systems
45

46. 46
Thank you for your attention

47. Related Papers
• Paper 1: Searching for a Needle in a Haystack:
Predicting Security Vulnerabilities for Windows Vista
• empirically evaluate the efficacy of classical metrics
like complexity, churn, coverage, dependency
measures, and organizational structure of the
company to predict vulnerabilities and assess how
well these software measures correlate with
vulnerabilities
47

48. Paper2: Evaluating Complexity, Code Churn,
and Developer Activity Metrics as Indicators
of Software Vulnerabilities
48

49. 49
CVE Tickets in WordPress about
Access Control
WordPress through 4.9.6 allows Author users to execute arbitrary code by leveraging directory
traversal in the wp-admin/post.php thumb parameter, which is passed to the PHP unlink
function and can delete the wp-config.php file. This is related to missing filename validation in
the wp-includes/post.php wp_delete_attachment function. The attacker must have capabilities
for files and posts that are normally available only to the Author, Editor, and Administrator roles.
The attack methodology is to delete wp-config.php and then launch a new installation process
to increase the attacker's privileges.