Download to read offline
Parsing and Rewriting Ruby Templates
- 1. Parsing and Rewriting RubyTemplates
- 2. Parsing and RewritingRuby Templates John Hawthorn @jhawthorn Rails committer Ruby Architecture @ GitHub
- 3.
- 4. Rails 6.0 -ActionView • Template de-duplication • Faster dev-mode • Faster template resolution
- 5.
- 6.
- 7.
- 8.
- 9. Ruby Templates Never declared localor method? locals hash
- 10. Plan Scan all viewsfor render calls
- 11. Parsing render calls Railsalready does this!
- 12.
- 13. Running Ruby Parse CompileRun Ruby AST YARV Bytecode Process
- 14. Running Ruby Parse CompileRun Ruby AST YARV Bytecode RubyVM::AbstractSyntaxTree (new in Ruby 2.6) Process
- 15. Ruby Parsers • RubyVM::AbstractSyntaxTree(Ruby 2.6+) • Ripper • jruby • parser (gem) • ruby_parser (gem) ❤️
- 16.
- 17.
- 18.
- 19.
- 20.
- 21. Parsing render calls FCALL :renderARRAY STR "hello"
- 22.
- 23. Parsing render calls FCALL :renderARRAY STR "hello" HASH LIT STR :name "..."
- 24. Parsing render calls FCALL :renderARRAY STR "hello" HASH LIT STR :name "..."
- 25. Parsing render calls keysmust be literals!
- 26.
- 27. Testing a realapplication
- 30.
- 32.
- 33. Initial request timing Measuredover 100 requests 40% faster!
- 34. • 💎 actionview_precompiler •✅ It works • ✅ Any templating language (ERB/HAML/slim) • ✅ Fast! • ⚠️ RubyVM::AST is Ruby 2.6+
- 35.
- 36. Local assignment Can weavoid this?
- 37.
- 38.
- 39. What if wecould rewrite local variable usage?
- 40. (known local) (unknown iflocal or method) Local assignment
- 41. A ruby toruby transpiler with dynamic locals github.com/jhawthorn/dynamic_locals
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48. Rewriting locals!!! • 💎dynamic_locals • ✅ It works! • ✅ Passes Rails ActionView test suite 100% • ⚠️ Slower? • ⚠️ Errors and warnings might be different • ⚠️ It's complicated!
- 49.
Editor's Notes
- #3 Hi I'm John Hawthorn. I'm a Rails committer and I work on the Ruby Architecture team at GitHub.
- #4 This talk is about Ruby templates. Here is an example. We have a file named "_hello.html.erb", with the contents Buenos dias, and then we an ERB tag with with ruby code inside, which will interpolate a name. We have another file named "home.html.erb". This renders the original file Most of the techniques in this talk would apply to all Ruby code, but I'm interested in templates.
- #5 Among the many improvements in Rails 6.0 were several performance optimizations in ActionView. We now allocate fewer template objects, templates in development mode should now be as fast as production, and template resolution (which is how we map template names to files) is now much faster.
- #6 Another performance improvement we've wanted to be able to make for a long time is being able to load the templates at boot. So why do we want this...
- #7 Unicorn, puma, and similar web servers for MRI Ruby, spawn workers using forking. After fully loading the application they make a fork: a full copy of the process. This is relatively fast, we only need to load the application once, and usually the workers can share most of the memory from the parent process. This also means we want to boot the application fully and load as much as possible in the parent process.
- #8 With the current system where we lazy load views, we have to load them once for each worker. We end up repeating work, and worst of all that work is done while a user is waiting for a response.
- #9 If we could instead load views in the parent we wouldn't need to repeat work in the worker and we could use less memory. In general, for production, we always want to load as much as possible in the parent process. This way we do no extra work, use no extra memory, and avoid wasting time when serving user requests. So why has it been hard for us to introduce this change...
- #11 The big challenge we've always had is when looking at this template, we don't know if "name" is a local variable passed in, or a method. Unlike a normal ruby method, name is never declared. It's actually pretty unusual to reference a variable which is never declared, you can't write a method that works this way. We only know what locals are defined when we look at the keys of the hash passed to render.
- #12 I want to share two ways I've tried to solve this. The first is to scan all views, to find all the render calls, and to find what keys the view is being passed for locals.
- #13 I was surprised to learn that Rails already does this! Kind of. We have a feature called "Russian Doll Caching" which invalidates cache keys when a template's content changes. To do this it has a dependency tracker which determines which other templates a view might render. Unfortunately it does it with this very large regular expression. Regular expressions aren't great for parsing code, it does work well for caching, but for our needs it isn't able to parse out the locals hash, and I think it would be too challenging to do so. We need a different solution.
- #14 When we run Ruby we normally only interact with it as a file, like we would with our text editor, or a process as we would with a console session or running web server.
- #15 To run code Ruby actually goes through a series of steps, which are usually hidden from us.
- #16 When we run Ruby code we're actually going through a series of steps, which are usually hidden from us. Most of the time when writing ruby we're only looking at the first and last of these steps. The first is where I open my editor and the last is where I have pry.
- #17 We don't want to run this code. We're probably not able to.
- #18 This isn't the first time we've had a parser available in Ruby and it's not the only option. I've tried all of these and they're all good. I really like the Ruby 2.6 parser because it's built-in, and we can expect it to keep up with any new ruby syntaxes. One problem is that it's only in Ruby 2.6+ and it's marked experimental.
- #19 It's likely you already use some sort of static analysis, like RuboCop in your projects.
- #20 What we get from the parser is an abstract syntax tree.
- #21 So we know we can parse ruby, what about templates? There are many templating languages and they all compile into ruby. Let's take a look
- #22 Ruby templates _are_ ruby. This is most apparent when looking at a template language like jbuilder
- #23 When you use jbuilder inside Rails, the template handler really just takes the template and adds one line of setup, and one line at the end to make sure we return the output.
- #24 To "compile" jbuilder, we add a little bit of code around the handler to initialize the json local variable, and to get the output at the end of the template.
- #25 ERB templates are ruby code. I think we recognize that from looking at them that they have ruby in them, but when we run them Rails will first turn them into Ruby.
- #26 ERB templates are ruby code. I think we recognize that from looking at them that they have ruby in them, but when we run them Rails will first turn them into Ruby. We can do this manually with ERB.new(src).src, this is essentially what Rails action_view will do internally. It's not pretty ruby, but it is plain Ruby.
- #27 And this is what it would look like as Rails finally renders the template. It's compiled into a method and this is what Rails will call. The local variable name is assigned here, we need to compile a different method depending on while locals are passed in.
- #28 So our plan is to take our template language, compile it into ruby, and then parse it into an AST. Because we can compile ERB to ruby and then parse that, we should be able to use any static analysis tool with ruby views. The Brakeman security scanner does this already, and there are tools which will let you use rubocop this way. We're still looking for render calls. Let's try it out...
- #29 ...
- #30 Let's look at just that render call to keep it simple. We have our source code as a string. We pass it to the parser, and we get this node object out. This object is a lisp-like definition, so I can be helpful to look at it as a tree.
- #31 To parse the render we need to check every aspect of the call. We check that it's a function, that the function name is render, that it has arguments, and that the first argument is a string. Then after that we can get the name of the template we're rendering. Around here I gain more appreciation for the large regular expression solution. This is kind of like writing a method_missing, but a little more detailed. This code isn't hard, it's just tedious. I find it necessary to write a lot of tests.
- #32 So we can parse the basic render call, same as the regex did, now we need to pick up on the keys of the hash.
- #33 So we can parse the basic render call, same as the regex did, now we need to pick up on the keys of the hash.
- #34 We need to make sure the keys are literals: either strings or symbols. If they're dynamic or use a variable we just have to skip this render. That's fine. Ruby is a very dynamic language, any time we are doing static analysis we need to accept that it might not work.
- #35 This technique is now a gem: actionview_precompiler. If we ran it on the example we'd been using it it does find the render of hello with name. For the top level "home" template it assumes it gets rendered with no arguments.
- #36 That's great for our small example. I wanted to verify this technique on a Real application. I used exercism.io, a very cool code practice tool. It's a real application with a few hundred views.
- #37 This is the tree of all the views in exercism and how they render each other.
- #39 To test this I booted the web server and then measured the very first request it served. This is without precompilation. This is a flamegraph which shows where the application spent time in the first request. The horizontal x axis is time. I've highlighted where the view compilation occurs. There are 8 views which are compiled and rendered in this request.
- #40 The only modification made to exercism.io is to add one initializer, which runs Actionview::Precompiler on boot.
- #41 And this is with eager loaded views. It was able to eager load all but one template render, which was inside a helper.
- #42 Profiling isn't benchmarking, so I re-ran the test 100 times with and without precompilation and found again that we are went from an average 100ms to 60ms.
- #43 So this worked great. I'm hoping someday soon we can add this to Rails, and we can give every Rails app this benefit. For today it's a gem actionview_precompiler, it's available on my github.
- #44 So we've seen the benefits from parsing. We can analyze code and use those results at runtime. We can do more by actually rewriting the code we've parsed.
- #45 Both of the existing popular parsing gems allow modifying the AST and then have a matching library to allow converting it back to ruby.
- #46 Unfortunately the new ruby parser doesn't support converting back to Ruby. Existing tools had ruby2ruby and unparser.
- #47 Each node comes with a start and end line and column.
- #48 So that's I think the the good solution, from what I learned implementing that I also came up with a strange solution.
- #49 I learned a lot about parsing from the previous work and began to wonder if there was another solution. The fundamental problem seemed to be that we needed to compile assignment of the keys passed in as locals.
- #50 It would be nice if there was another way to do this, but we can't do it dynamically. You can try with eval or binding.local_variable_set, it won't work.
- #51 The reason we can't do this is that name is compiled differently depending on if there has been an assignment before or not. This is done at parse time. If we know it's a local it gets compiled to LVAR otherwise it gets compiled to VCALL. I think VCALL means variable call but I'm not sure.
- #52 If we can parse this code, maybe we can parse the code and then rewrite it so that it does support dynamically assigning locals.
- #53 We can tell these are parsed differently because
- #54 I think VCALL means variable-call.
- #55 Here's the idea: ... Every time we assign to a local variable, we can instead write to a locals hash. Every time we read from a local, we can read from a locals hash. No big change here. Here's the big one, we can change those ambiguous calls or variables accesses to check if the key is in the hash and otherwise call the method.
- #56 What this is is a ruby-to-ruby transpiler with dynamic locals
- #58 We take our original source code.
- #59 Compile the ERB into plain Ruby.
- #60 Then we can run dynamic locals translate to convert it to use the locals hash.
- #61 And finally we can put it inside our method. We don't need to know what locals this template is rendered with. There are a few problems with this...
- #62 The defined keyword can be used to check if something is a local or not. If we didn't change it it would no longer work. We can use the same trick as before, return "local-variable" if it's in our fake locals hash, otherwise we can call the original.
- #63 binding has a similar problem. If a user calls binding.pry they probably want their locals to be available. Any time we see `binding`, we can add our dynamic variables onto it and then start pry on that.
- #64 Somehow this actually works, and it passes Rails' test suite 100%. I don't think that means this is going to production or should ship with rails. It's still very strange and probably has some issues, but I found this a very fun experiment. These techniques don't have to be about views. I hope this might make you think of some interesting ways you might use parsing or static analysis to find ways to use Ruby in completely new ways.