Code formatting is an opinionated beast. It always has been a matter of taste, and it always will be a matter of taste. This is the reason, why professional formatting tools, such as Eclipse JDT, offer a gazillion number of options. Which is still not sufficient enough. After all, you can override them inline with tag-comments to make the formatter shut up. Can't we do better than that? What if we could use machine learning techniques to detect the preferred code style that was use in a codebase so far? Turns out, we can. The Antlr Codebuff project (https://github.com/antlr/codebuff) offers a generic formatter for pretty much any given language. As long as a grammar file exists, existing source can be analyzed to learn about the rules that have been applied while writing the code. Those can than be used to pretty print newly written code. No configuration required. And existing sources will stay as nicely formatted as they are. In the end, the primary purpose of code formatting is not to re-arrange all the keywords, but to make the source layout consistent. In this talk, we will demonstrate the usage of the codebuff project and how it can be used to format the sources of your repo in a consistent way. We'll also show some other gems that have been revealed when toying around with the technology.

1. EDCM_2017_{
↦ Formattingn
↦ ↦ forn
↦ ↦ then
↦ Massesn
}

2. HolgerSebastian

3. Formatting

4. Beautiful Code!

5. grammar org.eclipse.xtext.example.domainmodel.Domainmodel with
org.eclipse.xtext.xbase.Xbase generate domainmodel "http://www.xtext.org/
example/Domainmodel" DomainModel: importSection=XImportSection?
elements+=AbstractElement*; AbstractElement: PackageDeclaration|
Entity;PackageDeclaration:'package'
name=QualifiedName'{'elements+=AbstractElement*'}';Entity:'entity'name=ValidID('
extends'superType=JvmParameterizedTypeReference)?'{'features+=Feature*'}';Featur
e:Property|
Operation;Property:name=ValidID':'type=JvmTypeReference;Operation:'op'name=Valid
ID'('(params+=FullJvmFormalParameter(','params+=FullJvmFormalParameter)*)?')'(':
' type=JvmTypeReference)?body=XBlockExpression;
Valid, but not Readable!

6. grammar org.eclipse.xtext.example.domainmodel.Domainmodel with org.eclipse.xtext.xbase.Xbase
generate domainmodel "http://www.xtext.org/example/Domainmodel"
DomainModel:
importSection=XImportSection? elements+=AbstractElement*;
AbstractElement:
PackageDeclaration
| Entity;
PackageDeclaration:
'package' name=QualifiedName '{'
elements+=AbstractElement*
'}';
Entity:
'entity' name=ValidID ('extends' superType=JvmParameterizedTypeReference)? '{' features+=Feature* '}';
Feature:
Property
| Operation;
Property:
name=ValidID ':' type=JvmTypeReference;
Operation:
'op' name=ValidID '(' (params+=FullJvmFormalParameter (',' params+=FullJvmFormalParameter)*)? ')'
(':' type=JvmTypeReference)? body=XBlockExpression;
Same Code, but Readable!

7. grammar org.eclipse.xtext.example.domainmodel.Domainmodel with org.eclipse.xtext.xbase.Xbase
generate domainmodel "http://www.xtext.org/example/Domainmodel"
DomainModel:
importSection=XImportSection? elements+=AbstractElement*;
AbstractElement:
PackageDeclaration
| Entity;
PackageDeclaration:
'package' name=QualifiedName '{'
elements+=AbstractElement*
'}';
Entity:
'entity' name=ValidID ('extends' superType=JvmParameterizedTypeReference)? '{' features+=Feature* '}';
Feature:
Property
| Operation;
Property:
name=ValidID ':' type=JvmTypeReference;
Operation:
'op' name=ValidID '(' (params+=FullJvmFormalParameter (',' params+=FullJvmFormalParameter)*)? ')'
(':' type=JvmTypeReference)? body=XBlockExpression;
Same Code, but Readable!
Spaces

8. grammar org.eclipse.xtext.example.domainmodel.Domainmodel with org.eclipse.xtext.xbase.Xbase
generate domainmodel "http://www.xtext.org/example/Domainmodel"
DomainModel:
importSection=XImportSection? elements+=AbstractElement*;
AbstractElement:
PackageDeclaration
| Entity;
PackageDeclaration:
'package' name=QualifiedName '{'
elements+=AbstractElement*
'}';
Entity:
'entity' name=ValidID ('extends' superType=JvmParameterizedTypeReference)? '{' features+=Feature* '}';
Feature:
Property
| Operation;
Property:
name=ValidID ':' type=JvmTypeReference;
Operation:
'op' name=ValidID '(' (params+=FullJvmFormalParameter (',' params+=FullJvmFormalParameter)*)? ')'
(':' type=JvmTypeReference)? body=XBlockExpression;
Linebreak
Same Code, but Readable!

9. grammar org.eclipse.xtext.example.domainmodel.Domainmodel with org.eclipse.xtext.xbase.Xbase
generate domainmodel "http://www.xtext.org/example/Domainmodel"
DomainModel:
importSection=XImportSection? elements+=AbstractElement*;
AbstractElement:
PackageDeclaration
| Entity;
PackageDeclaration:
'package' name=QualifiedName '{'
elements+=AbstractElement*
'}';
Entity:
'entity' name=ValidID ('extends' superType=JvmParameterizedTypeReference)? '{' features+=Feature* '}';
Feature:
Property
| Operation;
Property:
name=ValidID ':' type=JvmTypeReference;
Operation:
'op' name=ValidID '(' (params+=FullJvmFormalParameter (',' params+=FullJvmFormalParameter)*)? ')'
(':' type=JvmTypeReference)? body=XBlockExpression;
Indentation
Same Code, but Readable!

10. grammar org.eclipse.xtext.example.domainmodel.Domainmodel with org.eclipse.xtext.xbase.Xbase
generate domainmodel "http://www.xtext.org/example/Domainmodel"
DomainModel:
importSection=XImportSection? elements+=AbstractElement*;
AbstractElement:
PackageDeclaration
| Entity;
PackageDeclaration:
'package' name=QualifiedName '{'
elements+=AbstractElement*
'}';
Entity:
'entity' name=ValidID ('extends' superType=JvmParameterizedTypeReference)? '{' features+=Feature* '}';
Feature:
Property
| Operation;
Property:
name=ValidID ':' type=JvmTypeReference;
Operation:
'op' name=ValidID '(' (params+=FullJvmFormalParameter (',' params+=FullJvmFormalParameter)*)? ')'
(':' type=JvmTypeReference)? body=XBlockExpression;
Alignment
Same Code, but Readable!

11. But there are already formatters
out there!

12. But there are already formatters
out there!

13. But there are already formatters
out there!

14. What’s the Point?

15. Formatting is a Matter of Taste!

16. Natural Formatting
public NotificationChain myMethod(EObject otherEnd, int featureID)
{
switch (featureID)
{
….
}
return super.eInverseRemove(otherEnd, featureID);
}
public NotificationChain myMethod(EObject otherEnd, int featureID) {
switch (featureID) {
….
}
return super.eInverseRemove(otherEnd, featureID);
}
EMF

17. Eclipse JDT

18. What, if there is no Option?

19. Manual Formatting FTW
/**
* @formatter:off
*/
public void foo() {};
/**
* @formatter:on
*/

20. What, if there is no Formatter?

21. Domain Specific Languages

22. Bob Nystrom
Author of dartfmt
http://journal.stuffwithstuff.com/2015/09/08/the-hardest-program-ive-ever-written/
The Hardest Program
I've Ever Written

23. Rich API to Define Formatters

24. Options are not coming for free!

25. What if formatting would work by
using an example?

26. Towards a
Universal Code Formatter
through
Machine Learning
Terence Parr, Jurgin Vinju
06/2016
https://arxiv.org/abs/1606.08866

27. CodeBuff

28. Mimic developer during formatting

29. (Invalid) Simplification of ML / AI
()
n1
n2
n3
..
nk
( )↦ m1
..

30. CodeBuff
⊕

31. grammar org.eclipse.xtext.example.domainmodel.Domainmodel with org.eclipse.xtext.xbase.Xbase
generate domainmodel "http://www.xtext.org/example/Domainmodel"
DomainModel:
importSection=XImportSection? elements+=AbstractElement*;
AbstractElement:
PackageDeclaration
| Entity;
PackageDeclaration:
'package' name=QualifiedName '{'
elements+=AbstractElement*
'}';
Entity:
'entity' name=ValidID ('extends' superType=JvmParameterizedTypeReference)? '{' features+=Feature* '}';
Feature:
Property
| Operation;
Property:
name=ValidID ':' type=JvmTypeReference;
Operation:
'op' name=ValidID '(' (params+=FullJvmFormalParameter (',' params+=FullJvmFormalParameter)*)? ')'
(':' type=JvmTypeReference)? body=XBlockExpression;
Xtext Grammar Language

32. PackageDeclaration:
'package' name=QualifiedName '{'
elements+=AbstractElement*
'}';

33. PackageDeclaration:
'package' name=QualifiedName '{'
elements+=AbstractElement*
'}';
split(Document)↦Seq<Token>

34. PackageDeclaration:
'package' name=QualifiedName '{'
elements+=AbstractElement*
'}';
format (Token)↦TextChange

35. format (Token)↦TextChange
PackageDeclaration:
'package' name=QualifiedName '{'n
.... ....elements+=AbstractElement*
'}';

36. match(TokenProp)↦WS
apply(WS)↦NewText
prop(Token)↦TokenProp
format (Token)↦TextChange

37. 1. sp
2. nl
3. indent(Token)
4. align(Token)
5. nothing )
WS:=(

38. WS:=(sp, nl)

39. WS:=(8, 1)
PackageDeclaration:
'package' name=QualifiedName '{'n
.... ....elements+=AbstractElement*
'}';

40. match(TokenProp)↦ (sp, nl)
apply(sp, nl)↦NewText
prop(Token)↦TokenProp

41. prop(Token)↦TokenProp

42. TokenProp:=Vector<..>

43. 1. Token Type (String, Identifier, …)
2. Prev Token, Next Token
3. Paired Token ( { + }, : + ;, [ + ] )
4. Parent Node Type
5. List Index and List Size
TokenProp:=Vector<..>

44. TokenProp:=V<21>

45. prop(Token)↦V<21>
match (V<21>)↦ (sp, nl)
apply(sp, nl)↦NewText

46. 1. Train Corpus Documents
a. Compute prop(Token)↦tp
b. Store (tp, sp, nl)

47. ( )
tpt1
1
tpt1
2
tpt1
3
..
tpt1
x
spt1
1
nlt1
2
..
tpt2
1
tpt2
2
tpt2
3
..
tpt2
x
spt2
1
nlt2
2
tptc
1
tptc
2
tptc
3
..
tptc
x
sptc
1
nltc
2

48. 1. Train Corpus Documents
a. Compute prop(Token)↦tp
b. Store (tp, sp, nl)
2. Format Unknown Document

49. ()
tpu
1
tpu
2
tpu
3
..
tpu
x
?
?

50. 1. Train Corpus Documents
a. Compute prop(Token)↦tp
b. Store (tp, sp, nl)
2. Format Unknown Document
a. Compute prop(Token)↦tpu
b. Find match(tpu)↦ (spu, nlu)
c. Apply WS to Document

51. ()
tpu
1
tpu
2
tpu
3
..
tpu
x
( )spu
1
nlu
2
( )
tpt1
1
tpt1
2
tpt1
3
..
tpt1
x
spt1
1
nlt1
2
..
tpt2
1
tpt2
2
tpt2
3
..
tpt2
x
spt2
1
nlt2
2
tptc
1
tptc
2
tptc
3
..
tptc
x
sptc
1
nltc
2
↦⊕

52. match(tpu
)↦(spu
, nlu
)
K Nearest Neighbour

53. KNN
1. Select k most similar V<21> as vs
2. Select best (spu
, nlu
)b from vs
by weigh function b
3. k = 11

54. ( )
tpk
1
tpk
2
tpk
3
..
tpk
x
spk
1
nlk
2
()
tpu
1
tpu
2
tpu
3
..
tpu
x
( )spk’
1
nlk’
2
tpt1
1
tpt1
2
tpt1
3
..
tpt1
x
spt1
1
nlt1
2
..
tptc
1
tptc
2
tptc
3
..
tptc
x
sptc
1
nltc
2
↦⊕..

55. Demo

58. Download it now!
0 €
Not ready to use!