How Xslate Works

How Xslate works The next generation's template engine YAPC::Asia Tokyo 2010, Day 2, 2010/10/16 10:10-10:50 Fuji, Goro (gfx) [email_address] http://github.com/gfx /

Who am I?
Fuji, Goro ( 藤吾郎 )
id:gfx (hatena)
@__gfx__ (twitter)
I love...
Perl Internals
Perl/XS
Moose/Mouse (as Any::Moose)

Agenda
What is Xslate
How Xslate works
Execution process
Why so fast
Futures (TODOs)
Information

What is Xslate?
A template engine for Perl5
Written in XS
But also available in pure Perl (thanks to @maka2_donzoko)
Fast, safe, easy to enhance
Multi-syntaxes (esp. TT-like syntax)
Kind error messages

What are template engines?
Text processor vital for web applicatins
my $var = sprintf 'Hello, %s world', $foo; # <- this
$template = 'Hello, %foo% world?'; # <- this
$templat =~ s/ %(w+)% / $vars->{$1 }/xmsge;

So why to use Xslate?
There are too many template engines on CPAN
Template::Toolkit 2 (and ::Tiny)
HTML::Template (and ::Pro, ::Compiled, ::JIT)
Text::MicroTemplate (and ::Extended)
Mojo::Template
Text::ClearSilver (a perl binding to clearsilver)
HTML::Mason, Text::MicroMason
Tenjin, Template::Alloy, Template::Sandbox, etc...
There is no perfect module, though

Pick up the well-known
Template::Toolkit 2
+: Rich functionality, easy to use, easy to enhance
-: Very slow, too large spec, vulnerable to XSS
HTML::Template::Pro
+: Fast, ristricted
-: Hard to use, poor functionality, vulnerable to XSS
Text::MocroTemplate
+: Fast enough, easy to install, tolerant to XSS
-: Too free, only usaful to Perl programmers

Comparison with Xslate
v.s. Template::Toolkit 2
Much faster (100+ times faster)
Tolerant to XSS
v.s. HTML::Template::Pro
Even faster
Allows complex expressions, and tolerant to XSS
v.s. Text::MocroTemplate
Slightly faster
Ristricted (you cannot call system() nor fork())

Text::ClearSilver
A Perl binding to clearsilver (C library)
ClearSilver is a good template engine,
but the official ClearSilver.pm sucks!
-> Text::ClearSilver (requested by id:craftworks)
however, ClearSilver is not perfect

Thus, use Text::Xslate!
There is no the best, but is a better, Xslate
Xslate is at least the fastest, the most safe in the template engines on CPAN

Tutorial
Basic usage: Text::Xslate->new()->render()
new() accepts: syntax (template syntax) , module (function-based modules), function (additional functions), path (include path), cache (caching level), cache_dir (used for caches)
render() returns a rendered text; cannot output to filehandles directly, nor set calbacks (unlike TT)
See also Text::Xslate and Text::Xslate::Manual

Simplest example #!perl -w use 5.10.0; use strict; use Text::Xslate; my $tx = Text::Xslate-> new (); my %vars = ( lang => 'Xslate', ); # or $tx-> render ($file, vars); say $tx-> render_string (<<'EOT', vars); Hello, <: $lang :> world! EOT # -> Hello, Xslate world!

Xslate templates <ul> :# This is a template with the default syntax Kolon :# an one-line code starts from a colon ":" :# comments start from a hash symbol "#" :# an in-line code is surrounded by ”<:” and ":>" : for $list -> $item { <li> <: $item.name :> = <: $item.value :> </li> : } </ul> : include 'foo.tx' { bar => 'overrided' } :# See also ` perldoc Text::Xslate::Syntax::Kolon `

Demo
cpanm Text::Xslate
requires Any::Moose (Mouse), Data::MessagePack, etc.
xslate -e 'Hello, <: $ARGV[0] :>' Xslate
xslate -s TTerse -e '[% ARGV.0 %]' 'Hi, TTerse'
See also example files in Text-Xslate/example/

Details
Performance
Safe (XSS tolerance and ristriction)
Multi-syntaxes
Kolon
TTerse
Clevery
Enhancement
Template cascading

Performance (1) Run Text-Xslate/benchmark/x-rich-env.pl!

Performance (2)
Sam Graham's report ”Template roundup”
http://www.illusori.co.uk/projects/Template-Roundup/
based on benchmarks with Template::Benchmark
Accoding to this report, Xslate is fastest on the 'instance_reuse' condition
i.e. Persistent PSGI applications will take the best performance

Safe (1)
Risks caused by Template Engines
XSS: Cross Site Scripting
a type of computer security vulnerability typically found in web applications that enables malicious attackers to inject client-side script into web pages viewed by other users. (by Wikipedia)

Safe (2)
Typically caused by missing HTML escaping
e.g. [% foo | html %] (TT2)
Problem: TT2 requires explicit escaping
Explicity sucks!

Safe (3)
Solution: Template engines must apply HTML-escaping automatically
-> Text::MicroTemplate, Text::Xslate
This is XSS tolerance
Modern template engines (not only in Perl) should have XSS tolerance, and in fact do so.

How to write safe templates
Don't use the 'raw' filter in templates
Don't do that: <: $foo | raw :>
Escape $foo in perl code, because whether $foo is escaped or not is obscure from templates
The 'raw' filter is provided only for string literals

Multi-syntaxes
There are multiple template syntaxes
Kolon – the default, fully-featured
TTerse – TT2 compatible
Clevery – Smarty (in PHP) compatible

Kolon
The default template syntax
All the features are supported
Most optimized

TTerse
TT2 compatible (urged by tokuhirom++)
Available by default
More ristricted than the original TT2
No plugins, exceptions, nor mysterious features
As good as Kolon to run-time performance
Highly compatible, but diffrent to the basics of escaping mechanism

Clevery
Smarty compatible template syntax
Need to install Text::Clevery
With some overhead
Clevery Clevery

Enhancement
Add TT2-like methods to Xslate
Text::Xslate::Bridge::TT2Like
Function-based modules
Available via the "module" option
No plugin namespaces are required

Template cascading
Inspired by template inheritance of T::MicroTemplate::Extended (typester++)
Originated from Django, a framework for Python
More powerful than the 'include' command
Only available in Kolon
See also Text-Xslate/example/{cascade.pl,cascade,tx,base.tx}

How Xslate works
Execution process
Preprocessing
Parsing
Compiling
Saveing/loading bytecode
Executing
Why so fast

Execution Process <: $foo :> <: $foo :> print $foo; <: $foo :> fetch_s "foo" print end print $foo Virtual Machine { foo => 'bar' } ”bar"

Preprocessing
" Hello, <: $lang :> world! " is difficult to parse :(
So convert to that first:
" print_raw 'Hello, '; print $lang; print_raw ' world!' "
See &T::X::Parser::preprocess

Parsing
Parse " print_raw 'Hello, '; print $lang; print_raw ' world!' " and build syntax tree
Using Top Down Operator Precendence method
Explained in ” Beautiful Code ” (O'REILLY)
" It is easy to use. It feels a lot like Recursive Descent, but with the need for less code and with significantly better performance. ", Douglas Crockford introduced there.
See also http://javascript.crockford.com/tdop/

Top Down Operator Precedence
A top down parsering method
Templates -> Tokens (string) -> Symbols (T::X::Symbol) -> Nodes (T::X::Symbol)
Symbols (e.g. '+' for infix:<+>) know what they do
One easily can extend parsers by adding symbols
The parser entry point is &T::X::Parser::statements

The heart of TDOP
A symbol might have denotations:
std : statement denotation (for statements)
nud : null denotation (for objects and prefixes)
led : left denotation (for infixes and postfixes)
A "denotation" is a method called by statements()

Statement Denotation
e.g. 'if', 'for', 'include'
sub std_while { my($parser, $symbol) = @_; # $symbol represents 'while' my $proc = $symbol->clone( arity => 'while' # node type ); $proc->first( # set first child $parser->expression(0) ); $parser->pointy($proc); # parse return $proc; }

Null Denotation
e.g. variables, literals, parens
# '(' expr ')' sub nud_paren { my($parser, $symbol) = @_; # $symbol is '(' my $expr = $parser->expression(0); # $symbol->conterpart is ')' $parser->advance( $symbol->counterpart ); return $expr; }

Left Denotation
e.g. 'if', 'for', 'include'
sub led_infix { my($parser, $symbol, $left) = @_; return $parser->binary( $symbol, $left, $parser->expression( $symbol->lbp) ); }

Compiling
Convert the abstract syntax tree into bytecode
Bytecode is a serialized sequence of opcodes
# Hello, <: $lang :> world!
print_raw_s "Hello, "
fetch_s "lang"
print
print_raw_s " world!"
end
Also perform optimization

Saveing/Loading Bytecode
Precompiling is vital for template engines
e.g. ClearSilver compiles templates, but it cannot save compiled ones (maybe because it targets CGI)
Using Data::MessagePack for serialization
MessagePack: spec for binary data serialization
Like JSON, but fast and small
See also http://msgpack.org/

Assembling
Convert the name of opcode to the address of it
Using the direct threaded code paradigm

Executing
Execute by the virtual machine
An opcode (e.g. 'fetch_s') is a block of code with an argument (a sv, int, or address of opcode)
e.g. <: $foo :> ($a is a register, $out is the reuslt)
fetch_s "foo" # $a = $vars->{foo}
print # $out .= $a

Execution Process <: $foo :> <: $foo :> print $foo; <: $foo :> fetch_s "foo" print end print $foo Virtual Machine { foo => 'bar' } ” bar"

Why so fast
Precompiling & optimizing
Virtual Machine paradigm
Direct threaded code (but gcc only)
HTML escaping
Preallocation of the output buffer

Optimizing bytecode
Constant folding
<: 1 + 2 :> (->compile->) literal_i 3
<: if(0) { ... } :> (->compile->) noop
Injection of cascading templates
Cascading templates are statically concatinated
Thus, you cannot pass variables to 'cascade'
$ ack -i 'optimize' lib/Text/Xslate/Compiler.pm

Virtual Machine
Virtual machine is fast!
Register machine (two registers + one stack)

Direct Threaded Code
Using direct pointers of labels, not pointers of functions nor indexes of function table

HTML Escaping
Important for all the template engines
Using highly optimized routines (kazuho++)
As fast as memcpy(3)
tx_sv_cat_with_html_escape_force() at Text-Xslate/xs/Text-Xslate.xs
Experimental hacks for Text::MicroTemplate:
Rewriteing HTML escaping routines makes TMT faster than HTML::Template::Pro

Preallocation
Assumption: a template produces a string, and the size of the string is always the same
Then
First: Save the size of a template
Later: Preallocate the buffer with the saved size

Futures
Loop controls
Context controls
Augoment block modifiers
More template syntaxes
Xslate in WAF

Loop controls
last, next (redo?)
<: last if $a > 10 :> in Kolon
[% LAST IF a > 10 %] in TTerse

Context controls
Xslate calls functions/methods in scalar context, but there are other cases...
postfix:<context>?
<: [ $o->foo() list ] :>
postfix:<@context>?
<: [ $o->foo() @list ] :>
Any ideas?

Augment Block Modifiers
Xslate already supports 'around', 'before', 'after'
'augment' like block modifiers are requested

More templat syntaxes
HTML::Template (if someone wants)
Django (in Python)
Jinja (in Python)
Any ideas?

Xslate in WAF
There are some bridges
Catalyst::View::Xslate for Catalyst
MojoX::Renderer::Xslate for Mojo
Dancer::Template::Xslate for Dancer
Amon uses Xslate via Tiffany
More examples are required!
Help me, hackers!

Information
http://xslate.org
Web+DB Press Vol.59

Vote me! Mac Book Proほしいけどお金がありません！(>_<) もしよかったら投票お願いします！ http://yapcasia.org/2010/vote

Questions
Q. Perlコードに落とし込むという方法もあるが、なぜわざわざVMを？
A. テンプレートのレンダリングに特化しているので速い。特にエスケープしながらの連結が効果的
Q. XslateのVMの上でPerl6は動かせる？
A. 普通の言語ではないので無理
Q. 他の言語のバインディングは可能？
A. Perlと密結合しているので無理。書き直すしか

That's all Thanks for your attention!

How Xslate Works

by Goro Fuji

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 260

Statistics

How Xslate Works Presentation Transcript

http://d.hatena.ne.jp	259
http://sayperl.org	1