Metaprogramming in Scala 2.10        Eugene Burmako          EPFL, LAMP         28 April 2012
Agenda  Intro  Reflection  Universes  Demo  Macros  Summary
Metaprogramming     Metaprogramming is the writing of computer programs     that write or manipulate other programs or the...
Compiler   Q: How to enable metaprogramming?   A: Who has more data about a program than a compiler?   Let’s expose the co...
Reflection   In 2.10 we expose data about programs via reflection API.   The API is spread between scala-library.jar (interf...
Martin knows better   Design of the reflection API is explained in great detail in a recent   talk by Martin Odersky:   cha...
Hands on  Today we will learn the fundamentals of reflection API and learn to  learn more about it via a series of hands-on...
Macros  Q: Hey! What about macros?  A: Reflection is at the core of macros, reflection provides macros  with an API, reflecti...
Agenda  Intro  Reflection  Universes  Demo  Macros  Summary
Core data structures       Trees       Symbols       Types   C:ProjectsKepler>scalac -Xshow-phases   phase name id descrip...
Trees   Short-lived, mostly immutable, mostly plain case classes.   Apply(Ident("println"), List(Literal(Constant("hi!")))...
Learn to learn       -Xprint:parser (for naked trees)       -Xprint:typer (for typechecked trees)       -Yshow-trees and i...
-Yshow-trees   // also try -Yshow-trees-stringified   // and -Yshow-trees-compact (or both simultaneously)   >scalac -Xpri...
showRaw  scala> scala.reflect.mirror.reify{object Test {    println("Hello World!")  }}  res0: reflect.mirror.Expr[Unit] =...
Symbols  Link definitions and references to definitions. Long-lived, mutable.  Declared in scala/reflect/api/Symbols.scala (v...
Learn to learn       -Xprint:namer or -Xprint:typer       -uniqid       symbol.kind and -Yshow-symkinds       :type -v    ...
-uniqid and -Yshow-symkinds   >cat Foo.scala   def foo[T: TypeTag](x: Any) = x.asInstanceOf[T]   foo[Long](42)   // there ...
:type -v   We have just seen how to discover symbols used in trees.   However, symbols are also used in types.   Thanks to...
Types  Immutable, long-lived, sometimes cached case classes declared in  scala/reflect/api/Types.scala.  Store the informat...
Learn to learn       -Xprint:typer       -Xprint-types       :type -v       -explaintypes
-Xprint-types   -Xprint-types is yet another option that modifies tree printing.   Nothing very fancy, let’s move on to som...
:type -v   scala> :type -v def impl[T: c.TypeTag](c: Context) = ???   // Type signature   [T](c: scala.reflect.makro.Conte...
-explaintypes   >cat Test.scala   class Foo { class Bar; def bar(x: Bar) = ??? }   object Test extends App {     val foo1 ...
Big picture       Trees are created naked by Parser.       Both definitions and references (expressed as ASTs) get their   ...
Agenda  Intro  Reflection  Universes  Demo  Macros  Summary
Universes   Universes are environments that pack together trees, symbols and   their types.       Compiler (scala.tools.ns...
Symbol tables   Universes abstract population of symbol tables.       Compiler loads symbols from pickles using its own *....
Entry points   Using a universe depends on your scenario.       You can play with compiler’s universe (aka global) in REPL...
Path dependency  An important quirk is that all universe artifacts are path-dependent  on their universe. Note the ”reflect...
Agenda  Intro  Reflection  Universes  Demo  Macros  Summary
Inspect members   scala> import scala.reflect.mirror   import scala.reflect.mirror   scala> trait X { def foo: String }   ...
Analyze, typecheck and invoke members   val d = new DynamicProxy{ val dynamicProxyTarget = x }   d.noargs   d.noargs()   d...
Defeat erasure   scala> def foo[T](x: T) = x.getClass   foo: [T](x: T)Class[_ <: T]   scala> foo(List(1, 2, 3))   res0: Cl...
Compile at runtime   import scala.reflect.mirror._   val tree = Apply(Select(Ident("Macros"),       newTermName("foo")), L...
Agenda  Intro  Reflection  Universes  Demo  Macros  Summary
Macros  In our hackings above, we used the runtime mirror  (scala.reflect.mirror) to reflect against program structure.  We ...
No, really   That’s it.
Agenda  Intro  Reflection  Universes  Demo  Macros  Summary
Summary     In 2.10 you can have all the information about your program     that the compiler has (well, almost).     This...
Status   We’re already there.   Recently released 2.10.0-M3 includes reflection and macros.
Thanks!          eugene.burmako@epfl.ch
Upcoming SlideShare
Loading in...5
×

Metaprogramming in Scala 2.10, Eugene Burmako,

6,203

Published on

Talk by Eugene Burnako at the 9th meetup of Belarusian Scala Enthusiasts (scala.by) on metaprogramming capabilities in Scala 2.10

Published in: Technology, Business

Metaprogramming in Scala 2.10, Eugene Burmako,

  1. 1. Metaprogramming in Scala 2.10 Eugene Burmako EPFL, LAMP 28 April 2012
  2. 2. Agenda Intro Reflection Universes Demo Macros Summary
  3. 3. Metaprogramming Metaprogramming is the writing of computer programs that write or manipulate other programs or themselves as their data. —Wikipedia
  4. 4. Compiler Q: How to enable metaprogramming? A: Who has more data about a program than a compiler? Let’s expose the compiler to the programmer.
  5. 5. Reflection In 2.10 we expose data about programs via reflection API. The API is spread between scala-library.jar (interfaces), scala-reflect.jar (implementations) and scala-compiler.jar (runtime compilation).
  6. 6. Martin knows better Design of the reflection API is explained in great detail in a recent talk by Martin Odersky: channel9.msdn.com/Lang-NEXT-2012/Reflection-and-Compilers
  7. 7. Hands on Today we will learn the fundamentals of reflection API and learn to learn more about it via a series of hands-on examples.
  8. 8. Macros Q: Hey! What about macros? A: Reflection is at the core of macros, reflection provides macros with an API, reflection enables macros. Our focus today is understanding reflection, macros are just a tiny bolt-on. For more information about macros, their philosophy and applications, take a look at my Scala Days talk: http://scalamacros.org/talks/2012-04-18-ScalaDays2012.pdf.
  9. 9. Agenda Intro Reflection Universes Demo Macros Summary
  10. 10. Core data structures Trees Symbols Types C:ProjectsKepler>scalac -Xshow-phases phase name id description ---------- -- ----------- parser 1 parse source into ASTs, simple desugaring namer 2 resolve names, attach symbols to named trees typer 4 the meat and potatoes: type the trees pickler 7 serialize symbol tables I’ll do my best to explain these concepts, but it’s barely possible to do it better than Paul Phillips. Be absolutely sure to watch the Inside the Sausage Factory talk (when the video is up).
  11. 11. Trees Short-lived, mostly immutable, mostly plain case classes. Apply(Ident("println"), List(Literal(Constant("hi!")))) Comprehensive list of public trees can be found here: scala/reflect/api/Trees.scala (be sure to take a look at the ”standard pattern match” section in the end of the file).
  12. 12. Learn to learn -Xprint:parser (for naked trees) -Xprint:typer (for typechecked trees) -Yshow-trees and its cousins scala.reflect.mirror.showRaw(scala.reflect.mirror.reify(...)) Q: Where do I pull these compiler flags from? A: scala/tools/nsc/settings/ScalaSettings.scala
  13. 13. -Yshow-trees // also try -Yshow-trees-stringified // and -Yshow-trees-compact (or both simultaneously) >scalac -Xprint:parser -Yshow-trees HelloWorld.scala [[syntax trees at end of parser]]// Scala source: HelloWorld.scala PackageDef( "<empty>" ModuleDef( 0 "Test" Template( "App" // parents ValDef( private "_" <tpt> <empty> ) ...
  14. 14. showRaw scala> scala.reflect.mirror.reify{object Test { println("Hello World!") }} res0: reflect.mirror.Expr[Unit] = ... scala> scala.reflect.mirror.showRaw(res0.tree) res1: String = Block(List(ModuleDef( Modifiers(), newTermName("Test"), Template(List(Ident(newTypeName("Object"))), List( DefDef(Modifiers(), newTermName("<init>"), List(), List(List()), TypeTree(), Block(List(Apply(Select(Super(This(newTypeName("")), newTypeName("")), newTermName("<init>")), List())), Literal(Constant(())))), Apply(Select(Select(This(newTypeName("scala")), newTermName("Predef")), newTermName("println")), List(Literal(Constant("Hello World!")))))))), Literal(Constant(())))
  15. 15. Symbols Link definitions and references to definitions. Long-lived, mutable. Declared in scala/reflect/api/Symbols.scala (very much in flux right now!) def foo[T: TypeTag](x: Any) = x.asInstanceOf[T] foo[Long](42) foo, T, x introduce symbols (T actually produces two different symbols). DefDef, TypeDef, ValDef - all of those subtype DefTree. TypeTag, x, T, foo, Long refer to symbols. They are all represented by Idents, which subtype RefTree. Symbols are long-lived. This means that any reference to Int (from a tree or from a type) will point to the same symbol instance.
  16. 16. Learn to learn -Xprint:namer or -Xprint:typer -uniqid symbol.kind and -Yshow-symkinds :type -v Don’t create them by yourself. Just don’t, leave it to Namer. In macros most of the time you create naked trees, and Typer will take care of the rest.
  17. 17. -uniqid and -Yshow-symkinds >cat Foo.scala def foo[T: TypeTag](x: Any) = x.asInstanceOf[T] foo[Long](42) // there is a mysterious factoid hidden in this printout! >scalac -Xprint:typer -uniqid -Yshow-symkinds Foo.scala [[syntax trees at end of typer]]// Scala source: Foo.scala def foo#8339#METH [T#8340#TPE >: Nothing#4658#CLS <: Any#4657#CLS] (x#9529#VAL: Any#4657#CLS) (implicit evidence$1#9530#VAL: TypeTag#7861#TPE[T#8341#TPE#SKO]) : T#8340#TPE = x#9529#VAL.asInstanceOf#6023#METH[T#8341#TPE#SKO]; Test#14#MODC.this.foo#8339#METH[Long#1641#CLS](42) (scala#29#PK.reflect#2514#PK.‘package‘#3414#PKO .mirror#3463#GET.TypeTag#10351#MOD.Long#10361#GET)
  18. 18. :type -v We have just seen how to discover symbols used in trees. However, symbols are also used in types. Thanks to Paul (who hacked this during one of Scala Nights) there’s an easy way to inspect types as well. Corresponding REPL incantation is shown on one of the next slides.
  19. 19. Types Immutable, long-lived, sometimes cached case classes declared in scala/reflect/api/Types.scala. Store the information about the full wealth of the Scala type system: members, type arguments, higher kinds, path dependencies, erasures, etc.
  20. 20. Learn to learn -Xprint:typer -Xprint-types :type -v -explaintypes
  21. 21. -Xprint-types -Xprint-types is yet another option that modifies tree printing. Nothing very fancy, let’s move on to something really cool.
  22. 22. :type -v scala> :type -v def impl[T: c.TypeTag](c: Context) = ??? // Type signature [T](c: scala.reflect.makro.Context)(implicit evidence$1: c.TypeTag[T])Nothing // Internal Type structure PolyType( typeParams = List(TypeParam(T)) resultType = MethodType( params = List(TermSymbol(c: ...)) resultType = MethodType( params = List(TermSymbol(implicit evidence$1: ...)) resultType = TypeRef( TypeSymbol(final abstract class Nothing) ) ) ) )
  23. 23. -explaintypes >cat Test.scala class Foo { class Bar; def bar(x: Bar) = ??? } object Test extends App { val foo1 = new Foo val foo2 = new Foo foo2.bar(new foo1.Bar) } // prints explanations of type mismatches >scalac -explaintypes Test.scala Test.foo1.Bar <: Test.foo2.Bar? Test.foo1.type <: Test.foo2.type? Test.foo1.type = Test.foo2.type? false false false Test.scala:6: error: type mismatch; ...
  24. 24. Big picture Trees are created naked by Parser. Both definitions and references (expressed as ASTs) get their symbols filled in by Namer (tree.symbol). When creating symbols, Namer also creates their completers, lazy thunks that know how to populate symbol types (symbol.info). Typer inspects trees, uses their symbols to transform trees and assign types to them (tree.tpe). Shortly afterwards Pickler kicks in and serializes reachable symbols along with their types into ScalaSignature annotations.
  25. 25. Agenda Intro Reflection Universes Demo Macros Summary
  26. 26. Universes Universes are environments that pack together trees, symbols and their types. Compiler (scala.tools.nsc.Global) is a universe. Reflective mirror (scala.reflect.mirror) is a universe too. Macro context (scala.reflect.makro.Context) holds a reference to a universe.
  27. 27. Symbol tables Universes abstract population of symbol tables. Compiler loads symbols from pickles using its own *.class parser. Reflective mirror uses Java reflection to load and parse ScalaSignatures. Macro context refers to the compiler’s symbol table.
  28. 28. Entry points Using a universe depends on your scenario. You can play with compiler’s universe (aka global) in REPL’s :power mode. With mirrors you go through the Mirror interface, e.g. symbolOfInstance or classToType. In a macro context, you import c.mirror. and can use imported factories to create trees and types (don’t create symbols manually, remember?).
  29. 29. Path dependency An important quirk is that all universe artifacts are path-dependent on their universe. Note the ”reflect.mirror” prefix in the type of the result printed below. scala> scala.reflect.mirror.reify(2.toString) res0: reflect.mirror.Expr[String] = Expr[String](2.toString()) When you deal with runtime reflection, you simply import scala.reflect.mirror. , and enjoy, because typically there is only one runtime mirror. However with macros it’s more complicated. To pass artifacts around (e.g. into helper functions), you need to also carry the universe with you. Or you can employ the technique outlined in a discussion at scala-internals.
  30. 30. Agenda Intro Reflection Universes Demo Macros Summary
  31. 31. Inspect members scala> import scala.reflect.mirror import scala.reflect.mirror scala> trait X { def foo: String } defined trait X scala> typeTag[X] res1: TypeTag[X] = ConcreteTypeTag[X] scala> res1.tpe.members res2: Iterable[reflect.mirror.Symbol] = List(method $asInstanceOf, method $isInstanceOf, method sync hronized, method ##, method !=, method ==, method ne, method eq, constructor Object, method notifyAl l, method notify, method clone, method getClass, method hashCode, method toString, method equals, me thod wait, method wait, method wait, method finalize, method asInstanceOf, method isInstanceOf, meth od !=, method ==, method foo)
  32. 32. Analyze, typecheck and invoke members val d = new DynamicProxy{ val dynamicProxyTarget = x } d.noargs d.noargs() d.nullary d.value d.- d.$("x") d.overloaded("overloaded with object") d.overloaded(1) val car = new Car d.typeArgs(car) // inferred d.default(1, 3) d.default(1) d.named(1, c=3, b=2) // works, wow The proxy uses reflection that is capable of resolving all these calls. More examples at test/files/run/dynamic-proxy.scala. Implementation is at scala/reflect/DynamicProxy.scala.
  33. 33. Defeat erasure scala> def foo[T](x: T) = x.getClass foo: [T](x: T)Class[_ <: T] scala> foo(List(1, 2, 3)) res0: Class[_ <: List[Int]] = class scala.collection.immutable.$colon$colon scala> def foo[T: TypeTag](x: T) = typeTag[T] foo: [T](x: T)(implicit evidence$1: TypeTag[T])TypeTag[T] scala> foo(List(1, 2, 3)) res1: TypeTag[List[Int]] = ConcreteTypeTag[List[Int]] scala> res1.tpe res2: reflect.mirror.Type = List[Int]
  34. 34. Compile at runtime import scala.reflect.mirror._ val tree = Apply(Select(Ident("Macros"), newTermName("foo")), List(Literal(Constant(42)))) println(Expr(tree).eval) Eval creates a toolbox under the covers (universe.mkToolBox), which is a full-fledged compiler. Toolbox wraps the input AST, sets its phase to Namer (skipping Parser) and performs the compilation into an in-memory directory. After the compilation is finished, toolbox fires up a classloader that loads and lauches the code.
  35. 35. Agenda Intro Reflection Universes Demo Macros Summary
  36. 36. Macros In our hackings above, we used the runtime mirror (scala.reflect.mirror) to reflect against program structure. We can do absolutely the same during the compile time. The universe is already there (the compiler itself), the API is there as well (scala.reflect.api.Universe inside the macro context). We only need to ask the compiler to call ourselves during the compilation (currently, our trigger is macro application and the hook is the macro keyword). The end.
  37. 37. No, really That’s it.
  38. 38. Agenda Intro Reflection Universes Demo Macros Summary
  39. 39. Summary In 2.10 you can have all the information about your program that the compiler has (well, almost). This information includes trees, symbols and types. And annotations. And positions. And more. You can reflect at runtime (scala.reflect.mirror) or at compile-time (macros).
  40. 40. Status We’re already there. Recently released 2.10.0-M3 includes reflection and macros.
  41. 41. Thanks! eugene.burmako@epfl.ch
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×