The document discusses the internals of Twig, a templating language for PHP. It provides an overview of key Twig concepts like the lexer, parser, abstract syntax tree, template compilation process, and extensions. The lexer tokenizes template code into a stream of tokens. The parser processes these tokens to build an abstract syntax tree, representing the template's structure. Twig then compiles templates by transforming the abstract syntax tree into PHP code.

1. DIVING DEEP
INTO TWIG
or
TWIG INTERNALS

2. MATTHIAS NOBACK
Zeist, the Netherlands
Feature addition in March 2014
Started as a web designer (2003)
Was employed for 6 years at
several companies...

3. Now self employed: Noback's Office
Blog: php-and-symfony.matthiasnoback.nl
Twitter: @matthiasnoback

4. Tony Piper
Follow
@tonypiper
@matthiasnoback enjoyed reading your book ­ should be
mandatory reading for any Symfony developer.
6:30 PM ­ 7 Sep 2013
Damon Jones
Follow
@damon__jones
@matthiasnoback An excellent job, good sir. Your blog posts
and now your book are really showing us the way to be better
developers.
10:47 PM ­ 5 Sep 2013
leanpub.com/a-yearwith-symfony

5. ARMIN RONACHER
Developed Jinja for Python (2006)
Ported Jinja to PHP, called it Twig
(2008)

6. FABIEN POTENCIER
Lead developer of the Symfony project
Was looking for a Django-like templating
engine for Symfony2
Found Twig and started "hacking" on it

7. OVERVIEW
Sample
Extensions
Lexer
Parser
Token parsers
Node visitors
Compiler

8. SAMPLES

9. BLOCKS AND VARIABLES
{ frue i ues%
% o sr n sr }
<i{ ue.ae}<l>
l>{ srnm }/i
{ edo %
% nfr }

10. FUNCTIONS
Ia { rno('ap' 'a')}
m { adm[hpy, sd] }
{ stsesrne1 2)%
% e tp=ag(, 0 }
<icas"{cce[od,'vn] i }"../i
l ls={ yl('d' ee', ) }>.<l>

11. FILTERS
H,{ ue.aecptlz }!
i { srnm|aiaie }
{ ue.hnNmesfrt}
{ srpoeubr|is }
{ ps.asji(,' }
{ ottg|on' ) }

12. TESTS
{ i eprDt i dfnd%..
% f xiyae s eie }.
{ i uesi epy%..
% f sr s mt }.
{ i ii od%
% f
s d }

13. TAGS
{ i ue.nbe %..
% f sreald }.
{ boksdbr%..
% lc iea }.
{ icue'poiehm.wg %..
% nld _rfl.tlti' }.

14. EXTENDING
TWIG

15. EXTENSIONS
itraeTi_xesoItrae
nefc wgEtninnefc
{
pbi fnto gtucin(;
ulc ucin eFntos)
pbi fnto gtitr(;
ulc ucin eFles)
pbi fnto gtet(;
ulc ucin eTss)
}
..
.
$n-adxeso(etnin / i tewy
ev>dEtnin$xeso) / s h a
(Symfony2: create a service with a t i . x e s o tag)
wgetnin

16. FUNCTIONS
casMEtninetnswgEtnin
ls yxeso xed Ti_xeso
{
pbi fnto gtucin(
ulc ucin eFntos)
{
rtr ary
eun ra(
nwwgSmlFnto(
e Ti_ipeucin
'yucin,
mFnto'
fnto (tig {
ucin $hn)
rtr srnf'hsi <>y/>%.,$hn)
eun pit(Ti s bm<b s' tig;
}
)
)
;
}
}
{ mFnto(cmue" }
{ yucin"optr) }

17. FILTERS
casMEtninetnswgEtnin
ls yxeso xed Ti_xeso
{
pbi fnto gtitr(
ulc ucin eFles)
{
rtr ary
eun ra(
nwwgSmlFle(
e Ti_ipeitr
'ie,
mn'
fnto (wa,$ie=tu){
ucin $ht mn
re
rtr srnf
eun pit(
's(hc % mn),
% wih s ie'
$ht
wa,
$ie?'s:i nt
mn
i''s o'
)
;
}
)
)
;
}
}
{ tigmn(as)}
{ hn|iefle }

18. TESTS
casMEtninetnswgEtnin
ls yxeso xed Ti_xeso
{
pbi fnto gtet(
ulc ucin eTss)
{
rtr ary
eun ra(
nwwgSmlTs(
e Ti_ipeet
'_ofrne,
acneec'
fnto(nm){
ucin$ae
rtr $ae=='yfnCn;
eun nm = Smoyo'
}
)
)
;
}
}
{ i "yfnCn i acneec %Itl yus{ edf%
% f Smoyo" s _ofrne } od o o% ni }

19. TAGS
{ cneec %
% ofrne }
Needs some explaining...

20. LOADING A
TEMPLATE

21. TWIG ENVIRONMENT
$n =nwwgEvrnet)
ev
e Ti_niomn(;
$n-stodrnwwgLae_ieytm_DR_'tmlts);
ev>eLae(e Ti_odrFlsse(_I_./epae')
$otx =ary
cnet
ra(
..
.
)
;
eh $n-rne(idxhm.wg,$otx)
co ev>edr'ne.tlti' cnet;

22. WHAT IS A TEMPLATE?
casTi_niomn
ls wgEvrnet
{
..
.
}
pbi fnto rne(nm,ary$otx =ary)
ulc ucin edr$ae ra cnet
ra()
{
rtr $hs>odepae$ae-rne(cnet;
eun ti-laTmlt(nm)>edr$otx)
}
A template is a class that implements
wgTmltItrae
Ti_epaenefc
l a T m l t ( returns an instance of such a class
odepae)

23. COMPILED TEMPLATE CLASS
/ idxhm.wg*
* ne.tlti /
cas_TiTmlt_12098fe1d89ef1e6etnsTi_epae
ls _wgepaedd753ba3f93c0e59 xed wgTmlt
{
poetdfnto dDslyary$otx,ary$lcs=ary)
rtce ucin oipa(ra cnet ra bok
ra()
{
/ ln 1
/ ie
i (se(cnet"ae]){$nm_=$otx[nm";}es {$nm_=nl
f ist$otx[nm")
_ae
cnet"ae]
le
_ae
u
eh ti_saefle(ti-ev $nm_ "tl,nl,tu)
co wgecp_itr$hs>n, _ae, hm" ul re;
eh "i i afl"
co
s n
ie;
}
}
pbi fnto gtepaeae)
ulc ucin eTmltNm(
{
rtr "ne.tlti"
eun idxhm.wg;
}

24. BEFORE
<>{nm }<p
p{ ae }/>
AFTER
eh 'p'
co <>;
i (se(cnet"ae]){
f ist$otx[nm")
$nm_=$otx[nm";
_ae
cnet"ae]
}es {
le
$nm_=nl;
_ae
ul
}
eh ti_saefle(ti-ev $nm_ "tl,nl,tu)
co wgecp_itr$hs>n, _ae, hm" ul re;
eh '/>;
co <p'

25. HOW TWIG CREATES A
TEMPLATE CLASS
1. Retrieve the source (written in "Twig") from the loader(s)
2. Compile the source to a PHP class

26. COMPILING A TEMPLATE
THE LEXER

27. LEXER
1. Matches the input string against known patterns
("lexemes")
2. Determines token types for these matches
3. Returns a stream of tokens

28. THE LEXER IN YOUR MIND

29. FINDING TWIG BLOCKS
The lexer first checks for the position of the main markers:
Start of block:
{
%
Start of variable: {
{
Start of comment: {
#
Then the lexer
1. iterates over the resulting positions, while
2. checking some basic syntax rules, and
3. collecting tokens on its way to E F
O

30. TOKEN TYPES
Tokens have:
a type
a value (optional)
a line number
BOKSAT
LC_TR
BOKED
LC_N
VRSAT
A_TR
VRED
A_N
TX
ET
NM
AE
NME
UBR
SRN
TIG
OEAO
PRTR
PNTAIN
UCUTO
..
.
{
%
%
}
{
{
}
}
raw template data
f r i , etc.
o, f
a number
" . "or ' . '
..
..
+ * ~ etc.
, , ,
| [ { etc.
, , ,
..
.

31. Take this template:
{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }
<u>
/l
$ee =$n-gtee(;
lxr
ev>eLxr)
$epae=..
tmlt
.;
$oeSra =$ee-tknz(tmlt)
tkntem
lxr>oeie$epae;

32. TOKEN STREAM
eh $oeSra;
co tkntem
BOKSAT
LC_TR
NM(ni)
AEedf
BOKED
LC_N
TX(u>
ET<l)
BOKSAT
LC_TR
NM(o)
AEfr
NM(tm
AEie)
OEAO(n
PRTRi)
NM(tm)
AEies
BOKED
LC_N
..
.
EF
O
{
%
edf
ni
%
}
raw template
data
{
%
fr
o
ie
tm
i
n
ies
tm
%
}
..
.
end of input
{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }
<u>
/l

33. STATES
To keep track of what the lexer is doing.
DATA
lexing raw template data (start state)
BLOCK lexing a block
VAR
lexing a variable
STRING lexing a string

34. CONSECUTIVE STATES
DT
AA
BOK
LC
BOK
LC
DT
AA
BOK
LC
BOK
LC
BOK
LC
BOK
LC
BOK
LC
DT
AA
VR
A
..
.
DT
AA
template data
block e d f
ni
starts
block e d f
ni
ends
<l
u>
block f rstarts
o
name: i e
tm
name: i
n
name: i e s
tm
block f rends
o
<i
l>
variable starts,
name: i e
tm
...
<u>
/l
{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }
<u>
/l

35. SYNTAX VALIDATION
Each block and variable should be closed
{ fr{ i
% o % f
Brackets ( [should be closed symmetrically
{
{ [a }
{ '' }
Closing brackets ] )can not occur first
}
{ ]}
{
}

36. SYNTAX VALIDATION
(CONTINUED)
Expressions may not contain unexpected characters
{ }
{
}
Comments should be closed
{ cmet
# omn

37. FROM SYNTAX TO
SEMANTICS
The resulting list of tokens may be semantically incorrect.
In the Twig language, that is...
{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }
<u>
/l

38. COMPILING A TEMPLATE
THE PARSER

39. PARSING THE TOKEN
STREAM
The parser
Processes the token stream
Builds an Abstract Syntax Tree for the template

40. CREATING THE ABSTRACT
SYNTAX TREE
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }
<u>
/l
$epae=..
tmlt
.;
$oeSra =$ee-tknz(tmlt)
tkntem
lxr>oeie$epae;
$asr=$n-gtasr)
pre
ev>ePre(;
$oere=$asr>as(tkntem;
ndTe
pre-pre$oeSra)
eh $oere
co ndTe;

41. EXCERPT OF THE ABSTRACT
SYNTAX TREE
Ti_oeMdl(
wgNd_oue
bd:Ti_oeBd(
oy wgNd_oy
0 Ti_oe
: wgNd(
0 Ti_oeTx(aa 'u>)
: wgNd_etdt: <l'
1 Ti_oe
: wgNd(
0 Ti_oeStep
: wgNd_eTm(
nm:'tm'
ae ies
)
1 Ti_oeFr
: wgNd_o(
vletre:Ti_oeEpeso_sinae
au_agt wgNd_xrsinAsgNm(
nm:'tm
ae ie'
)
sq Ti_oeEpeso_epae
e: wgNd_xrsinTmNm(
nm:'tm'
ae ies
)
bd:Ti_oe
oy wgNd(
0 Ti_oe
: wgNd(
0 Ti_oeTx(
: wgNd_et
dt:'l>
aa <i'
)
1 Ti_oe
: wgNd(
0 Ti_oeStep
: wgNd_eTm(
nm:'tm
ae ie'
)
1 Ti_oePit
: wgNd_rn(
ep:Ti_oeEpeso_itr
xr wgNd_xrsinFle(
nd:Ti_oeEpeso_itr
oe wgNd_xrsinFle(

42. THE ROOT NODE
The parsing process results in a root node containing
the body of the template,
or a collection of blocks,
the link to a parent template,
...

43. THE MAIN TOKEN TYPES
The parser collects nodes based on the token at the current
position in the token stream.
TX
ET
template text
create a T x node with the value of the
et
token
VRSAT
A_TR
variable
parse the expression that follows and
expect V R E D
A_N
B O K S A T block with a tag expect a name, which is the name of the
LC_TR
tag (i.e. for, if, etc.) and call a subparser

44. SUBPARSER == TOKEN
PARSER
Each token parser defines its own rules for the tokens that
should follow the tag:
{ frie i ies%
% o tm n tm }
{ icue'epaehm'wt {fo:'a' %
% nld tmlt.tl ih 'o' br} }
{ stfo br='o' 'a'%
% e o, a
fo, br }

45. FROM TOKENS TO NODES
The token parser returns nodes based on the tokens it finds.
Returned nodes are inserted in the Abstract Syntax Tree

46. A CUSTOM TOKEN PARSER
casCneecTknasretnswgTknasr
ls ofrneoePre xed Ti_oePre
{
pbi fnto prewgTkn$oe)
ulc ucin as(Ti_oe tkn
{
$hs>asr>eSra(-epc(Ti_oe:BOKED;
ti-pre-gttem)>xetwgTkn:LC_N)
$xr=nwwgNd_xrsinCntn(Smoyo'
ep
e Ti_oeEpeso_osat'yfnCn,
$oe-gtie);
tkn>eLn()
}
}
rtr nwwgNd_rn(ep,$oe-gtie) $hs>eTg);
eun e Ti_oePit$xr tkn>eLn(, ti-gta()
pbi fnto gta(
ulc ucin eTg)
{
rtr 'ofrne;
eun cneec'
}
{ cneec %
% ofrne }

47. Teew g,{ cneec %!
hr e o % ofrne }
$epae=..
tmlt
.;
$n =nwwgEvrnet)
ev
e Ti_niomn(;
$n-adoePre(e CneecTknasr);
ev>dTknasrnw ofrneoePre()
eh $n-pre$n-tknz(tmlt);
co ev>as(ev>oeie$epae)
(Better: register them using the g t o e P r e s )of
eTknasr(
your Twig extension class)

48. EXCERPT OF THE ABSTRACT
SYNTAX TREE
Ti_oeMdl(
wgNd_oue
bd:Ti_oeBd(
oy wgNd_oy
0 Ti_oe
: wgNd(
0 Ti_oeTx(
: wgNd_et
dt:'hr w g,'
aa Tee e o
)
1 Ti_oePit
: wgNd_rn(
ep:Ti_oeEpeso_osat
xr wgNd_xrsinCntn(
vle 'yfnCn
au: Smoyo'
)
)
2 Ti_oeTx(
: wgNd_et
dt:''
aa !
)
)
)
)

49. EXPRESSIONS
{ 5+ae*4}
{
g
}
$epae=..
tmlt
.;
$oueoe=$n-pre$n-tknz(tmlt);
mdlNd
ev>as(ev>oeie$epae)
eh $oueoe
co mdlNd;
Expressions are parsed by a specialized expression parser.

50. EXPRESSIONS
..
.
ep:Ti_oeEpeso_iayAd
xr wgNd_xrsinBnr_d(
lf:Ti_oeEpeso_osatvle 5
et wgNd_xrsinCntn(au: )
rgt Ti_oeEpeso_iayMl
ih: wgNd_xrsinBnr_u(
lf:Ti_oeEpeso_epaenm:'g'
et wgNd_xrsinTmNm(ae ae)
rgt Ti_oeEpeso_osatvle 4
ih: wgNd_xrsinCntn(au: )
)
)

51. ASSOCIATIVITY
Most operators are left associative,
which means that
a+b+c
is to be read as
(a+b +c
(
)
)
and not as
( +( +c)
a
b
)

52. PRECEDENCE
Operators have a number indicating
their precedence, so
a+b*c
will always be interpreted as
( +( *c)
a
b
)
instead of
( +b *c
a
)
)

53. ALSO: NODE VISITORS
Allowed to revisit the entire node tree and change anything.
E.g. auto-escaping

54. COMPILING A TEMPLATE
THE COMPILER

55. THE ROOT NODE
The node tree contains nodes generated by:
the parser
the expression parser
token parsers
node visitors

56. THE COMPILE STEP
{ frie i ies%
% o tm n tm }
{ ie|aiaie}<r
{ tmcptlz }b>
{ edo %
% nfr }
$epae=..
tmlt
.;
eh $n-cmieore$epae;
co ev>oplSuc(tmlt)

57. EXCERPT OF THE COMPILED
TEMPLATE
cas_TiTmlt_4dc9f0249098c82eetnsTi_epae
ls _wgepaed18d80b0e809ef47 xed wgTmlt
{
poetdfnto dDslyary$otx,ary$lcs=ary)
rtce ucin oipa(ra cnet ra bok
ra()
{
/ ln 1
/ ie
i (se(cnet"tm") {$ies =$otx[ies] }es {$ies =
f ist$otx[ies])
_tm_
cnet"tm";
le
_tm_
$otx[_aet]=(ra)$otx;
cnet'prn'
ary cnet
$otx[_e' =ti_nuetaesbe$ies)
cnet'sq]
wgesr_rvral(_tm_;
frah(cnet'sq]a $otx[_e" = $otx[ie"){
oec $otx[_e' s cnet"ky] > cnet"tm]
/ ln 2
/ ie
eh ""
co
;
i (se(cnet"tm]){$ie_=$otx[ie";}es {$ie_=
f ist$otx[ie")
_tm
cnet"tm]
le
_tm
eh ti_saefle(ti-ev ti_aiaiesrn_itr$hs>n,
co wgecp_itr$hs>n, wgcptlz_tigfle(ti-ev
eh "b>;
co <r"
}
$prn =$otx[_aet]
_aet
cnet'prn';
ust$otx[_e',$otx[_trtd] $otx[_e',$otx[ie'
ne(cnet'sq] cnet'ieae', cnet'ky] cnet'tm]
$otx =arymre$prn,aryitretky$otx,$prn);
cnet
ra_eg(_aet ra_nesc_e(cnet _aet)
}
}

58. RECURSIVE COMPILING
The compiler just calls the c m i e )
opl(
method of the root node.
Which calls the c m i e )of child
opl(
nodes, etc, etc.

59. CUSTOM TOKEN PARSER
REVISITED
casCneecTknasretnswgTknasr
ls ofrneoePre xed Ti_oePre
{
pbi fnto preTi_oe $oe)
ulc ucin as(wgTkn tkn
{
..
.
$xr=nwwgNd_xrsinCntn(Smoyo'
ep
e Ti_oeEpeso_osat'yfnCn,
$oe-gtie);
tkn>eLn()
}
}
rtr nwwgNd_rn(ep,$oe-gtie) $hs>eTg);
eun e Ti_oePit$xr tkn>eLn(, ti-gta()

60. COMPILING A PRINT NODE
casTi_oePit
ls wgNd_rn
{
pbi fnto cmieTi_oplr$oplr
ulc ucin opl(wgCmie cmie)
{
$oplr
cmie
-adeuIf(ti)
>dDbgno$hs
-wie'co'
>rt(eh )
-sbopl(ti-gtoe'xr)
>ucmie$hs>eNd(ep')
-rw""
>a(;n)
;
}
}
The result can be found in the compiled template:
poetdfnto dDslyary$otx,ary$lcs=ary)
rtce ucin oipa(ra cnet ra bok
ra()
{
/ ln 1
/ ie
..
.
eh "yfnCn;
co Smoyo"
..
.
}

61. SOME REFLECTIONS
1. You can put any PHP code you want inside a template
2. You can do (heavy) calculations at compile time (just once)
You only have to create your own node type and
implement its compile method.

62. TESTING AND DEBUGGING
Writing parsers and nodes can be quite difficult, so
1. read the compiled templates in your cache directory and
2. write unit tests for your custom parser and node type
(extend from w g T s _ o e e t a e
Ti_etNdTsCs)

63. THAT'S ALL

64. QUESTIONS?

65. THANK YOU
AND GOOD BYE
joind.in/10368
nobacksoffice.nl
php-and-symfony.matthiasnoback.nl
@matthiasnoback

66. REFERENCES
Armin's blog
Jinja documentation
Templating engines in PHP
Templating engines in PHP - Follow-up
Article about node visitors
IMAGES
http://www.stockfreeimages.com/
http://twig.sensiolabs.org/