The document discusses a system called linter-maintainer that customizes static analysis tools to align with the coding rules actually followed by developers, aiming to reduce false positives and negatives in code analysis. It details a study involving four JavaScript projects, revealing that a significant percentage of rules were not default or were ignored by developers. The preliminary evaluations show that the tool effectively reduces false positives by over 49% while tracing the latest followed coding rules.

1. Automatically Customizing Static
Analysis Tools to Coding Rules
Really Followed by Developers
Yuki Ueda, Takashi Ishio and Kenichi Matsumoto
Nara Institute of Science and Technology
SANER2021

2. Static Analysis Tools or Linters
2
Static analysis tools or linters can detect:
1. Security Vulnerabilities,
2. Performance Issues,
3. Bad Programming Practices, Code Smells
These violations are defined by Coding Rules

3. Example of coding rule violations
3
if (a=b) {
return True
} else {
threw new
Error();
}
return False
JavaScript
Rule 1:
Limit the assignment in
conditional statement
a=b a==b

4. Example of coding rule violations
4
if (a=b) {
return True
} else {
threw new
Error();
}
return False
JavaScript
Rule 2:
Limit the unreachable code
return False

5. Linters in SE Research
5
Code Review [1]
[1] Carvalho, A., Luz, W., Marc´ılio, D., Bonif´acio, R.,Pinto, G. and Canedo, E. D.: C-3PR: A Bot for Fixing Static Analysis Violations via Pull Requests,
SANER’20
[2] Bavishi, Rohan, Hiroaki Yoshida, and Mukul R. Prasad. "Phoenix: Automated data-driven synthesis of repairs for static analysis violations." ESEC/FSE’19
[3] Trautsch, Alexander, Steffen Herbold, and Jens Grabowski. "Static source code metrics and static analysis warnings for fine-grained just-in-time defect
prediction." ICSME’20
Defect Prediction [3]
Bug Repair [2]

6. Linters in SE Research
6
Code Review [1]
These Studies Assume the Linters Are Correct
[1] Carvalho, A., Luz, W., Marc´ılio, D., Bonif´acio, R.,Pinto, G. and Canedo, E. D.: C-3PR: A Bot for Fixing Static Analysis Violations via Pull Requests,
SANER’20
[2] Bavishi, Rohan, Hiroaki Yoshida, and Mukul R. Prasad. "Phoenix: Automated data-driven synthesis of repairs for static analysis violations." ESEC/FSE’19
[3] Trautsch, Alexander, Steffen Herbold, and Jens Grabowski. "Static source code metrics and static analysis warnings for fine-grained just-in-time defect
prediction." ICSME’20
Defect Prediction [3]
Bug Repair [2]

7. Default Rules != Followed Rules
7
Dataset: 4 JavaScript Projects
• Used in BugsJS [4]
• Using ESLint [5]
[4] Gyimesi, Péter, et al. "BUGSJS: a benchmark and taxonomy of JavaScript
bugs." Software Testing, Verification and Reliability (2019): e1751.
[5] N. C. Zakas, B. Mills, T. Nagashima, and M. Djermanovic, “ES-Lint -
Pluggable JavaScript linter,” https://eslint.org/, 2020
N = 241 Default Rules: 57 Non-default Rules: 184
Followed by All Projects: 64 True Positives: 38 False Negatives: 26
Unfollowed by 1+ Projects: 177 False: Positives: 19 True Negatives: 158
1. 40% (26 / 64) of followed rules are not default

8. Default Rules != Followed Rules
8
N = 241 Default Rules: 57 Non-default Rules: 184
Followed by All Projects: 64 True Positives: 38 False Negatives: 26
Unfollowed by 1+ Projects: 177 False: Positives: 19 True Negatives: 158
1. 40% (26 / 64) of followed rules are not default
2. 33% (19 / 57) of default rules are ignored
Dataset: 4 JavaScript Projects
• Used in BugsJS [4]
• Using ESLint [5]
[4] Gyimesi, Péter, et al. "BUGSJS: a benchmark and taxonomy of JavaScript
bugs." Software Testing, Verification and Reliability (2019): e1751.
[5] N. C. Zakas, B. Mills, T. Nagashima, and M. Djermanovic, “ES-Lint -
Pluggable JavaScript linter,” https://eslint.org/, 2020

9. Default Rules != Followed Rules
9
N = 241 Default Rules: 57 Non-default Rules: 184
Followed by All Projects: 64 True Positives: 38 False Negatives: 26
Unfollowed by 1+ Projects: 177 False: Positives: 19 True Negatives: 158
1. 40% (26 / 64) of followed rules are not default
2. 33% (19 / 57) of default rules are ignored
Dataset: 4 JavaScript Projects
• Used in BugsJS [4]
• Using ESLint [5]
[4] Gyimesi, Péter, et al. "BUGSJS: a benchmark and taxonomy of JavaScript
bugs." Software Testing, Verification and Reliability (2019): e1751.
[5] N. C. Zakas, B. Mills, T. Nagashima, and M. Djermanovic, “ES-Lint -
Pluggable JavaScript linter,” https://eslint.org/, 2020
Linter Rules should be customized

10. Goal: Optimizing the Coding Rules
1. Reduce False Negative (non-default but followed) rules
Enabling the rules that are followed by the all of project files
2. Reduce False Positive (default but unfollowed) rules
Disabling the rules that are violated by the one or more files
10

11. Linter-Maintainer:
Tracing Followed Coding Rules
11
Project Code Files Coding Rule
Configuration File
Followed Rule List
When developer installed linter, execute
Input Output
$linter-maintainer --generate --eslint-js ./your/project/path > .eslintrc.json

12. Evaluation Process 1 / 4
12
Generate
Version 1.0.0
Configuration File 1.0.0

13. Evaluation Process 2 / 4
13
Apply and
Calc False Positive/Negative
Version 1.0.1
Version 1.0.0
Configuration File 1.0.0

14. Evaluation Process 3 / 4
14
Generate
Version 1.0.1 Version 1.0.2
Configuration File 1.0.1

15. Evaluation Process 4 / 4
15
Version 1.0.2
Apply both of
Major and Maintainance Rules
Version 1.0.1
Version 1.0.0
Configuration File 1.0.0 Configuration File 1.0.1

16. Rule Updates Reduce Median False Positives
by >49%
16
0.00-0.89
Precision 1.00
0.70-0.99
Maintainance
Major
0.00-0.39
Recall 1.00
0.88-0.99
Original
1.0.0 -> 1.0.2 1.0.1 -> 1.0.2

17. Rule Updates Reduce Median False Positives
by >49%
17
0.00-0.89
Precision 1.00
0.70-0.99
Maintainance
Major
0.00-0.39
Recall 1.00
0.88-0.99
Original
Rule updates for each major version are effective enough
1.0.0 -> 1.0.2 1.0.1 -> 1.0.2

18. Conclusion
• Linter-Maintainer traces the latest followed rules
• Linter-maintainer reduced false negatives by >49%
Future Plan
• Increasing evaluated languages and projects
• Improving rule selecting approach by using Git history
18
Install
Source
Unsupported Supported Evaluated
C/C++ Java JavaScript
Ruby Python
TypeScript