7. Lexer
Splits document text into tokens
Works before and independent of the parser
Token matching
First match wins
Precedence
All keywords
Local terminal rules
Terminal rules from super grammar
As last resort you can define a custom lexer
8. Lexing
// Examplen
’Lamp’
n
events button_pressed
n
state on
nt
button_pressed => off
n
state off
nt
button_pressed => light
SL_COMMENT
STRING
WS
events WS ID
WS
state ID
WS
ID WS => WS ID
WS
state ID
WS
ID WS => WS ID
events
state
=>
ID
STRING
ML_COMMENT
SL_COMMENT
WS
ANY_OTHER
Tokens
Token StreamChar Stream
11. Hidden Terminals
Are ignored by the parser inside the respective scope
Can be defined on grammar or rule level (inherited)
grammar org.eclipse.xtext.common.Terminals
hidden(WS, ML_COMMENT, SL_COMMENT)
or on parser rules
QualifiedName hidden(): // strictly no whitespace
ID ('.' ID)*;
12. Datatype Rules
Return a value (instead of an EObject)
Are processed by the parser
Superior alternative to terminal rules
Examples
Double returns ecore::EDouble hidden():
'-'? INT '.' INT ('e'|'E' '-'? INT)?;
QualifiedName: // returns ecore::EString (default)
ID ('.' ID)*;
ValidID:
ID | 'keyword0' | 'keyword1';
13. Transforms textual representation to value and back
For terminal and datatype rules
e.g. strips quotes (STRINGValueConverter)
remove leading ^ (IDValueConverter)
EMF defaults are often sufficient (EFactory)
Value Converter
class MyValueConverterService extends AbstractDeclarativeValueConverterService {
@Inject MyValueConverter myValueConverter;
!
@ValueConverter(rule = "MY_TERMINAL") def MY_TERMINAL() {
return myValueConverter
}
}
15. Example
Expression:
If;
If:
Variable |
'if' condition=Expression
'then' then=Expression
('else' else=Expression)?;
Variable:
name=ID;
warning(200): ...: Decision can match input such as "'else'" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
warning(200): ...: Decision can match input such as "'else'" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
if cond1 then
if cond2 then
foo
else // dangling
bar
16. Ambiguity Analysis
In the workflow, add
!
AntlrWorks www.antlr3.org/works/
fragment = DebugAntlrGeneratorFragment {
options = auto-inject {}
}
17. Ambiguity Resolution
Add keywords
Use syntactic predicates:
“If the predicated element matches in the current
decision, decide for it.”
plain
(=>'else' else=Expression)?;
=>('else' else=Expression)?; // not so good
first-set
->('else' else=Expression)?;
19. grammar org.xtextcon.Statemachine
with org.eclipse.xtext.common.Terminals
!
generate statemachine
"http://www.xtextcon.org/Statemachine"
!
Statemachine returns Statemachine:
name=STRING
('events' events+=Event+)?
states+=State*;
!
Event returns Event:
name=ID description=STRING?;
!
State returns State:
'state' name=ID
transitions+=Transition*;
!
Transition returns Transition:
event=[Event|ID] '=>' state=[State|ID];
statemachine
name : String
Statemachine
name : String
Transition
name : String
State
name : String
description: String
Event
*
* *states events
transitions
event1
1
state
20. Supertypes
...
!
Element:
Event | State;
!
Event returns Event:
name=ID description=STRING?;
!
State returns State:
'state' name=ID
transitions+=Transition*;
!
...
description: String
EventState
name : String
Element
name : String
Transition
Common features are promoted to the super type
Dispatcher rule Element needs not to be called
A rule can return a subtype of the specified return type
23. EObject Creation
The current pointer
Points to the EObject returned by a parser rule call
Is set to null when entering the rule
On the first assignment the EObject is created and
assigned to current
Further assignments go to current
24. EObject Creation
Type :
(Class | DataType) 'is' visibility=('public' |'private');
DataType :
'datatype' name=ID;
Class :
'class' name=ID;
datatype A is public
Type current = null;
current = new DataType();
current.setName(“A”);
current.setVisibility(“public”);
return current;
25. Actions
Alternative without assignment -> no returned object
Actions make sure an EObject is created
Specified type must be compatible with return type
Created element is assigned to current
BooleanLiteral:
value?='true' | 'false';
BooleanLiteral returns Expression:
{BooleanLiteral} (value?=‘true’ | ‘false');
The rule 'BooleanLiteral' may be consumed without object instantiation.
Add an action to ensure object creation, e.g. '{BooleanLiteral}'.
28. Assigned Action
Addition returns Expression:
Number ({Sum.left = current} '+' right=Number)*;
Number:
value = INT;
1
Expression current = null;
current = new Number();
current.setValue(1);
return current;
// model
Number {
value = 1
}
29. Assigned Action II
Addition returns Expression:
Number ({Sum.left = current} '+' right=Number)*;
Number:
value = INT;
1 + 2
Expression current = null;
current = new Number();
current.setValue(1);
Expression temp = new Sum();
temp.setLeft(current);
current = temp;
…
current.right = <second Number>;
return current;
// model
Addition {
left = Number {
value = 1
}
right = Number {
value = 2
}
}
30. Assigned Action III
Creates a new element
makes it the parent of current
and sets current to it
Needed
normalizing left-factored grammars
infix operators
32. Best Practices
Use nsURIs for imported EPackages
Prefer datatype rules to terminal rules
Use syntactic predicates instead of backtracking
Be sloppy in the grammar but strict in validation