The document discusses the crange project which focuses on cross-referencing source code using the Clang compiler. It covers tools considered for indexing, challenges faced during implementation, and provides examples of source code manipulation with Abstract Syntax Trees (AST). Additionally, it highlights the performance of crange in generating a tag database and mentions various functionalities and features of the system.

1. CRANGE
CROSS-REFERENCING SOURCE CODE WITH CLANG
— —AnuragPatel RedHat github.com/crange/crange
0

2. TABLE OF CONTENTS
1. Tools considered
2. Clangand AST
3. Indexingwith libclang
4. Crange
5. Challenges
6. Summary

3. SOURCE CODE
#include<stdio.h>
intFibonacci(int);
intFibonacci(intn){
if(n==0)
return0;
elseif(n==1)
return1;
else
return(Fibonacci(n-1)+Fibonacci(n-2));
}
intmain(){
intn=12,i=0,c;
for(c=1;c<=n;c++){
printf("%dn",Fibonacci(i));i++;
}
return0;
}
Fibonacciseries

4. TOOLS CONSIDERED

5. CTAGS
+ Fast, comprehensive language support
-Definitions only

6. DOXYGEN
+ X-Ref, XML output
-Slow

7. GNU GLOBAL
+ Fast, definitions, X-Ref

8. ANTLR
+ Seems powerful, LL(*) parser
-Java, grammar files, complicated

9. FLEX+BISON
+ Is powerful
-CS 143 -Compilers

10. CLANG
+ Declarations, definitions, X-Ref, range, statements, operators,
expressions, literals

11. CLANG

12. CLANG
Compiler frontend and static analyzer for C, C++, Objective-C.
Converts programs to ASTand allows ASTmanipulation.

13. AST
[snip]
|-FunctionDecl0x610a090<fibonacci.c:2:1,col:18>Fibonacci'int(int)'
|`-ParmVarDecl0x6109fd0<col:15>'int'
|-FunctionDecl0x610a1d0prev0x610a090<line:3:1,line:10:1>Fibonacci'int(int)'
||-ParmVarDecl0x610a150<line:3:15,col:19>n'int'
|`-CompoundStmt0x6153a70<col:22,line:10:1>
| `-IfStmt0x6153a40<line:4:3,line:9:44>
| |-<<<NULL>>>
| |-BinaryOperator0x610a2d8<line:4:7,col:12>'int''=='
| ||-ImplicitCastExpr0x610a2c0<col:7>'int'<LValueToRValue>
| ||`-DeclRefExpr0x610a278<col:7>'int'lvalueParmVar0x610a150'n''int'
| |`-IntegerLiteral0x610a2a0<col:12>'int'0
| |-ReturnStmt0x610a320<line:5:5,col:12>
| |`-IntegerLiteral0x610a300<col:12>'int'0
| `-IfStmt0x6153a10<line:6:8,line:9:44>
[snip]
$ clang-cc1 -ast-dump fibonacci.c

14. LIBCLANG

15. LIBCLANG
Libraryfor processingsource code.
Translate source to AST
Resolve identifiers and symbols
Expand macros
Syntax highlighting
Code completion

16. INDEXING AND X-REF (0/6)
#include<stdio.h>
intFibonacci(int);
intFibonacci(intn){
if(n==0)
return0;
elseif(n==1)
return1;
else
return(Fibonacci(n-1)+Fibonacci(n-2));
}
intmain(){
intn=12,i=0,c;
for(c=1;c<=n;c++){
printf("%dn",Fibonacci(i));i++;
}
return0;
}
Walk the AST

17. INDEXING AND X-REF (1/6)
intFibonacci(int);
intFibonacci(intn){
if(n==0)
return0;
elseif(n==1)
return1;
else
return(Fibonacci(n-1)+Fibonacci(n-2));
}
intmain(){
intn=12,i=0,c;
for(c=1;c<=n;c++){
printf("%dn",Fibonacci(i));i++;
}
return0;
}
#include<stdio.h>
Macros

18. INDEXING AND X-REF (2/6)
#include<stdio.h>
int (int);
int (int ){
if(n==0)
return0;
elseif(n==1)
return1;
else
return(Fibonacci(n-1)+Fibonacci(n-2));
}
int (){
int =12, =0, ;
for(c=1;c<=n;c++){
printf("%dn",Fibonacci(i));i++;
}
return0;
}
Fibonacci
Fibonacci n
main
n i c
Declarations

19. INDEXING AND X-REF (3/6)
#include<stdio.h>
intFibonacci(int);
intFibonacci(intn){
if( ==0)
return0;
elseif( ==1)
return1;
else
return( (-1)+ (-2));
}
intmain(){
intn=12,i=0,c;
for( =1; <= ; ++){
printf("%dn", ()); ++;
}
return0;
}
n
n
Fibonaccin Fibonaccin
c c n c
Fibonaccii i
References

20. INDEXING AND X-REF (4/6)
#include<stdio.h>
intFibonacci(int);
intFibonacci(intn){
if(n==0)
;
elseif(n==1)
;
else
;
}
intmain(){
intn=12,i=0,c;
;
}
return0
return1
return(Fibonacci(n-1)+Fibonacci(n-2))
for(c=1;c<=n;c++){
printf("%dn",Fibonacci(i));i++;
}
return0
Statements

21. INDEXING AND X-REF (5/6)
#include<stdio.h>
intFibonacci(int);
intFibonacci(intn){
if(n==0)
return0;
elseif(n==1)
return1;
else
return( + );
}
intmain(){
intn=12,i=0,c;
for(c=1;c<=n;c++){
("%dn", );i++;
}
return0;
}
Fibonacci(n-1) Fibonacci(n-2)
printf Fibonacci(i)
Expressions

22. INDEXING AND X-REF (6/6)
#include<stdio.h>
intFibonacci(int);
intFibonacci(intn){
if(n 0)
return0;
elseif(n 1)
return1;
else
return(Fibonacci(n1) Fibonacci(n2));
}
intmain(){
intn=12,i=0,c;
for(c 1;c n;c ){
printf("%dn",Fibonacci(i));i ;
}
return0;
}
==
==
- + -
= <= ++
++
Operators

23. CRANGE

24. FRONTEND
C++
libclang+ Python
libclang+ Python + multiprocessing
crtags: generatingtagdb
crange: query

25. DATASTORE
JSON
HDF5
BerkeleyDB
SQLite

26. MULTIPROCESSING MODEL
1 DBworker
1 multiprocessing.Queue
n-1 ASTworkers*

27. MULTIPROCESSING.POOL()
1 async DBworker, n-1 sync ASTworkers

28. CRTAGS
$cdlinux-3.13.5
$timecrtags-v-j32.
Parsingfs/xfs/xfs_bmap_btree.c(count:1)
Indexingfs/xfs/xfs_bmap_btree.c(nodes:379,qsize:0)
...
Parsingsound/soc/codecs/ak4641.h(count:34348)
Generatingindexes
real 415m10.974s
user 8108m52.241s
sys 72m51.554s
$
Generates 22GBtags file in ~7 hours
16GBRAM &32 CPUs

29. CURL DEMO
$ time crtags -v-j 32 curl-7.35.0/

30. CHALLENGES

31. INCLUDE PATHS
$ clang-I/usr/include -Iinclude -I`llvm-config--cflags`

32. MACROS
-D_DEBUG -D_GNU_SOURCE -D_CONSTANT_MACROS

33. MAKEFILE/CMAKE
[
{"directory":"/home/anurag/llvm/build",
"command":"/usr/bin/clang++-Irelative-DSOMEDEF="Withspaces,quotesand-es."
"file":"MachineVerifier.cpp"},
...
]

34. PYTHON VS C++

35. SUMMARY

36. CRANGE CAN PROVIDE
Definitions
Declarations
References
Expressions
Operators
Symbols
Source range
Fancysearch
Code folding*
Find similar code*

37. FURTHER READING
https://github.com/crange/crange
http://clang.llvm.org/get_started.html
github.com/trolldbois/python-clang/tree/master/tests/cindex
https://docs.python.org/2/library/multiprocessing.html
http://clang.llvm.org/doxygen/group__CINDEX.html

38. END