Static analyzers help us find problems in our code. They are like compilers in terms of complexity; they need to know the syntax of target languages. Is it possible to build such tools without comprehensive upfront design? He will share his experience in making such tool in Python over the past 8 years.

1. Test-Driven
Development of a Static
Code Analyzer in Python
terry@odd-e.com

2. 2
3 Steps
Make it work
Make it right
Make it fast
This formulation of this statement has been attributed to KentBeck;
it has existed as part of the UnixWay for a long time.

3. 3
Story of Lizard
HFCCA
https://github.com/terryyin/lizard

4. Cyclomatic Complexity
The cyclomatic complexity of a section of source code
(functions in C) is the count of the number of linearly
independent paths through the source code. For
instance,
• if the source code contained no decision points such
as IF statements or FOR loops, the complexity would
be 1
• If the code had a single IF statement then it’s 2
• For every IF/FOR or OR/AND in the code +1
Most studies find a strong positive correlation between cyclomatic complexity and
defects: modules that have the highest complexity tend to also contain the most
defects. We use 15 as the threshold for ‘complex function’, which is already a VERY
BIG number.

5. It also has extensions …
5
lizard -EwordCount

6. Pycodestyle in 8 years
lizard -Ewordcount ../pycodestyle/

7. 7
A list of supported languages:
• C/C++ (works with C++14)
• Java
• C# (C Sharp)
• JavaScript
• Objective C
• Swift
• Python
• Ruby
• TTCN-3
• PHP
• Scala
• GDScript

8. It was driven by tests
8
Spec Code
Refactoring

9. 9
4 Years ago at PyconSG …

10. 10
Finally, a month ago…

11. How would you start to build a code
clone detector?
11

12. I just start with some simple
speculation:
Perhaps I will need to do some
hashing …
12

13. 13

14. 14
Refactoring
I did some small refactoring to get my test helper ready.

15. 15
lizard_ext/lizardduplicate.py
test/test_extensions/testDuplicate.py

16. 16
Refactoring
Yeah! We have our first failing test!!!
Spec

17. 17
lizard_ext/lizardduplicate.py

18. 18
Refactoring
I made my test pass by just returning an empty list…
And I’m not kidding.
Spec Code

19. Refactor
19
Code
Refactoring

20. 20
test/test_extensions/testDuplicate.py
Add a new test

21. 21
Refactoring
Nothing to refactor, so I add a new test…
Spec Code
Spec

22. 22
lizard_ext/lizardduplicate.py

23. 23
Refactoring
As long as there’s code (token), it will return 1
Spec Code
Spec Code

24. Refactor
24
Code
Refactoring

25. 25
test/test_extensions/testDuplicate.py

26. 26
Refactoring
Nothing to refactor in production code,
But there’s “duplication” in test code
Spec Code
Spec Code
Refactoring

27. 27
test/test_extensions/testDuplicate.py

28. 28
Refactoring
New test for the repeating times.
Now you should know how to make it pass …
Spec Code
Spec Code
Refactoring
Spec

29. 29

30. 30
Refactoring
Fake it, when you can…
Implement it when you have to.
Spec Code
Spec Code
Refactoring
Spec Code

31. 31
3 more cycles later…

32. 32
Refactoring
It now looks like …
Spec Code
Spec Code
Refactoring
Spec Code

33. 33

34. 34
Refactoring
We refactor to make the code “right” again
Spec Code
Spec Code
Refactoring
Spec Code
Refactoring

35. 35
5 more cycles later…

36. 36

37. 37

38. 38
Refactoring
At some point, our code becomes like this
Spec Code
Spec Code
Refactoring
Spec Code
Refactoring
Refactoring
Spec Code
Spec Code
Refactoring
Spec Code

39. 39
Code, listen you shall.
Let the design be the
way it wants to be!
Data Do Not Speak, Code Sometimes Does

40. 40
Refactoring
Let the design emerge from working software
Spec Code
Spec Code
Refactoring
Spec Code
Refactoring
Refactoring
Spec Code
Spec Code
Refactoring
Spec Code
Refactoring

41. 41
Basic Duplicate Detection
def test_empty_file(self):
def test_two_functions_that_are_exactly_the_same(self):
def test_two_functions_that_are_exactly_the_same_with_start_end_line_number(self):
def test_two_5_line_functions_that_are_exactly_the_same_detail(self):
def test_two_functions_that_are_not_the_same(self):
def test_2_duplicates(self):
def test_1_duplicate_3_times(self):
Detection Ignoring const, variable names similar operators
def test_duplicate_with_different_integer_value(self):
def test_duplicate_with_different_string_value(self):
def test_duplicate_with_different_variable_name(self):
def test_duplicate_with_different_operator(self):
def test_duplicate_with_value_dense_block__value_should_be_kept(self):
def test_duplicate_with_value_dense_block_in_brackets(self):
Merging big duplicate blocks
def test_duplicate_with_2_big_blocks(self):
def test_no_duplicate_with_1_big_blocks_of_the_same_number(self):
def test_many_identifiers_together(self):
def test_repeating_patterns(self):
def test_repeating_patterns_multiple_matches(self):
Duplicate threshold
def test_threshold(self):
def test_threshold_with_too_many_repeatings(self):
def test_threshold_exceeded(self):
Multiple Files
def test_2_files_basic_duplicate(self):
def test_multiple_files_more_duplicate(self):

42. Hash?
42
Yah, that was my initial speculation. Eventually,
my implementation didn’t use hash.
Which is fine.

43. What about performance?
43
Not an excuse to
write smelly code.

44. 44
Rules Of Optimization
•FirstRuleOfOptimization - Don't.
•SecondRuleOfOptimization - Don't... yet.
•ProfileBeforeOptimizing
Source: Michael Jackson

45. 45
Without much optimisation
•The lizard duplicate detector can process the entire
ansible codebase (700K+ NLOC) in less than 2
minutes.

46. 46
Duplication may be
the root of all evil in
software.
Clean Code, Robert C. Martin

47. 47
Duplicate block:
--------------------------
../django/django/conf/locale/en_AU/formats.py:17 ~ 38
../django/django/conf/locale/en_GB/formats.py:17 ~ 38
../django/django/conf/locale/cy/formats.py:17 ~ 34
^^^^^^^^^^^^^^^^^^^^^^^^^
Duplicate block:
--------------------------
../django/django/conf/locale/en_GB/formats.py:8 ~ 39
../django/django/conf/locale/cy/formats.py:8 ~ 35
^^^^^^^^^^^^^^^^^^^^^^^^^^
Duplicate block:
--------------------------
../django/django/shortcuts.py:74 ~ 92
../django/django/shortcuts.py:98 ~ 113
^^^^^^^^^^^^^^^^^^^^^^^^^^
Total duplicate rate: 23.71%
Using 50 tokens as the minimum duplicated block size.
The duplicate rate is high than actual.

48. 48
Minimum Token 50
Files Avg CCN Duplicate rate
django 1933 1.8 23.7%
Flask 69 1.5 6.41%
Tornado 118 1.8 5.25%
Youtube-dl 845 4.3 17.43%
Ansible 3237 4.0 117.30%
Puppet (Ruby) 2419 1.5 23.93%

49. 49
— Kent Beck, inventor of XP