In-depth look at the Flex compiler and HFCD

2,079 views

Published on

Published in: Technology
1 Comment
4 Likes
Statistics
Notes
  • HFCD is freaking incredible. For starters it dropped my compile times for 2-3 minutes to 1.6 - 2.8 seconds on Dedoose, a huge Flex 4 project. Additionally by seperating the compiling processes outside of the JVM running flash builder, Flash Builder's speed and responsiveness has sky-rocketed as well. Awesome work guys!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
2,079
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
0
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide


















































  • In-depth look at the Flex compiler and HFCD

    1. 1. IN-DEPTH LOOK AT THE FLEX COMPILER AND HFCD Clement Wong clement@stopcoding.org http://stopcoding.wordpress.com
    2. 2. AGENDA • Basic Architecture of Flex Compiler • Compiler Extensibility • Overview of Flex Compiler API • HellFire Compiler Daemon (HFCD) • Demo, Q/A
    3. 3. SPEAKER BIO • Founder of Bytecode Workshop; Creator of HFCD • Former Flex Compiler/Profiler Architect & Engineering Lead • Former ColdFusion MX Architect & Engineering Lead • Licensed Professional Engineer in the Province of Ontario • Studied CS and Mech. Eng. at the University of Waterloo
    4. 4. FLEX COMPILER TIMELINE
    5. 5. FLEX COMPILER 1.0-1.5 • Supports MXML and AS2 (not AS3). • Based on the AS2 compiler from Flash Authoring. • Ported from C++ to Java for J2EE. • Invoked by a MXML servlet.
    6. 6. FLEX COMPILER 1.0-1.5 • Major challenges from the AS2 compiler. • Notarchitected or partitioned for Flex - no clear compilation phases defined - at least not clear enough for MXML. • Difficult to extend; not MXML friendly. • Pre AS3/AVM+: classes and assets were both first-class citizens - very difficult to implement a good linker.
    7. 7. FLEX COMPILER 2.0 - 4.0 • Itis a multi-language compiler. It supports mxml, as, css, properties and now fxg. • There is a “mini compiler” for each language. Each “mini compiler” implements some well-defined compilation phases. • The “top level” Flex compiler becomes a coordinator - responsible for looking up source and SWC files; determines when and which “mini compiler” to call; etc...
    8. 8. “MINI COMPILER” EXAMPLES Package Class InterfaceCompiler, MXML flex2.compiler.mxml ImplementationCompiler AS3 flex2.compiler.as3 As3Compiler CSS flex2.compiler.css CssCompiler ABC flex2.compiler.abc AbcCompiler Properties flex2.compiler.i18n I18nCompiler
    9. 9. SPECIAL CASE: MXML • Why are there 2 “mini compilers” for MXML? • InterfaceCompiler for MXML produces API signature. Does not produce abc bytecode in generate(). • ImplementationCompiler for MXML generates “full AS3 source code” only after knowing dependent type info. Generates abc bytecode in generate(). • But why?
    10. 10. SPECIAL CASE: MXML • Consider this: <c:MyComp ...><c:prop .../></c:MyComp> • Need to know the meaning of every tag in MXML before accurate MXML-to-AS3 conversion could happen. • Is “prop” a component or is it a property of “MyComp”? • We must know the API signature of “MyComp”!
    11. 11. COMPILATION PHASES • preprocess() • analyze4() • parse1() • generate() • parse2() • postprocess() • analyze1() • analyze2() • analyze3()
    12. 12. COMPILATION PHASES preprocess() preparation works before parsing. generates syntax tree; identifies top-level parse1() definitions, superclasses and interfaces. rarely used. used only by the MXML “mini parse2() compiler” for data binding. performs flow analysis. can’t proceed without analyze1() superclass and interface info. checks syntax tree and identifies namespaces.
    13. 13. COMPILATION PHASES flow analysis continued. identifies unknown analyze2() references. performs constant and metadata evaluation. analyze3() resolve unknown references in syntax trees. finalizes constant evaluation. all symbols fully analyze4() resolved. also performs “lint” evaluation. performs code generation. produces abc generate() bytecode.
    14. 14. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    15. 15. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    16. 16. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    17. 17. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    18. 18. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    19. 19. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    20. 20. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    21. 21. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    22. 22. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    23. 23. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    24. 24. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    25. 25. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    26. 26. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    27. 27. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    28. 28. A SIMPLE ILLUSTRATION Assume ‘A’ is a subclass of ‘B’. ‘A’ and ‘B’ reference ‘C’.
    29. 29. COMPILER EXTENSIONS • The “mini compiler” implementations are “shared” code among SDK tools. • mxmlc, compc and asdoc all use the “mini compilers” but they serve different purposes. They need to do different things at the end of each compilation phase. • Solution: “mini compiler” supports “compiler extension”. • Implement the “flex2.compiler.as3.Extension” interface.
    30. 30. EXTENSION EXAMPLES • ASDocExtension • DataBindingExtension • EmbedExtension • SignatureExtension • StyleExtension
    31. 31. FLEX 4: YAFCEM • YAFCEM?? Yet Another Flex Compiler Extension Mechanism • Introducedin Flex 4. Flex developers could insert Java code into the compiler. The compiler would execute the Java code at a number of locations defined by the extension interfaces. • e.g. IApplicationExtension, ILibraryExtension. • Note: Theyare not executed at the end of the mini compiler compilation phases. e.g. IApplicationExtension executed at the end of Application.build().
    32. 32. TRANSCODERS • When there is an [Embed] in your code, how does the compiler process it? • Answer: EmbedExtension. EmbedExtension uses transcoders. • Typically generates an AS3 class and a SWF asset tag. • The generated classes are handed to the “top-level” compiler “coordinator”. The classes are then scheduled to be compiled and linked.
    33. 33. TRANSCODERS • Transcoders implement the “flex2.compiler.Transcoder” interface. • The interface looks simple. However, implementation could be tricky because: • one must be familiar with the SWF file format. • one must know what to generate for the asset classes. • use -keep-generated-actionscript to learn about codegen.
    34. 34. TRANSCODER EXAMPLES • DataTranscoder • SVGTranscoder • FontTranscoder • morein the “flex2.compiler.media” • JPEGTranscoder package. • LosslessImageTranscoder • MovieTranscoder • SoundTranscoder
    35. 35. LINKER • The Flex compiler not just compiles. It also links. • Application SWFs and library (SWC) SWFs are different. • Application SWFs typically have 2 frames: Preloader and essential Flex framework classes in the 1st frame. The rest of the application and framework classes in the 2nd frame. • LibrarySWFs are simpler: everything in one frame. Debug info is always embedded. No SWF optimizations.
    36. 36. LINKER • The SWF structure of these 2 SWF types are encapsulated in 2 Flex classes. • Application: flex2.linker.FlexMovie • Library: flex2.compiler.swc.SwcMovie • Both classes (the generate() method) take a list of compilation units; run dependency analysis; determine export order and generate SWF.
    37. 37. LINKER • Theexport order of application SWFs is important because the ActionScript VM in the Player demands the export order be based on type dependencies. It is a reasonable demand because that simplifies VM class loading and verification. • Use topological sort to determine export order. In fact, topological sort is used in the compile phase as well (for processing AS3 inheritance chains in the AS3 compiler and for circular dependency detection).
    38. 38. TOPOLOGICAL SORT •A vertex represents a definition (e.g. a class or a function). •A directed edge between two vertices represents the dependency between the two definitions. For example, when vertex B points to vertex A, that means A depends on B. • Ifa vertex has no arrow pointing towards it, it goes to the export order. The vertex is then removed. Its edges (all pointing outwards) are also removed. This step continues.
    39. 39. TOPOLOGICAL SORT • The sorting ends when the in-degrees of all the vertices in the dependency graph are not zero. • Ifthere are still vertices in the graph, those definitions are in some circular dependency loops.
    40. 40. POST-LINK • Foreach file compiled, the compiler generates a set of ActionScript bytecode. It is self-contained, completed with a constant pool, a set of API signatures and a set of function bodies. • Anapplication SWF, if un-optimized, can have a lot of constant pools. They can be quite overlapping. • One of the key steps in the post-link phase is “abc merging”.
    41. 41. FLEX COMPILER API • Flex toolchains need a simple way to invoke the compiler. • Users of the Flex Compiler API: • Flex Builder • SAP • Flex Data Services
    42. 42. FLEX COMPILER API • Recently, one of the users of the Flex Compiler API is HellFire Compiler Daemon (HFCD). • The APIclasses are in the flex2.tools.oem and flex2.tools.flexbuilder packages. • Use flex2.tools.oem.Application to build applications and modules. • Use flex2.tools.oem.Library to build libraries.
    43. 43. FLEX COMPILER API Application app = new Application(new File(“HelloWorld.mxml”)); app.setOutput(new File(“HelloWorld.swf ”)); Configuration config = app.getDefaultConfiguration(); SWF config.setXXX(...); app.setConfiguration(config); app.build(true); Library lib = new Library(); lib.addComponent(...); lib.setOutput(...); Configuration config = app.getDefaultConfiguration(); SWC config.setSourcePath(...); config.setXXX(...); lib.setConfiguration(config); lib.build(true);
    44. 44. FLEX COMPILER API • The API usage is simple, but toolchains usually do more... They implement some of the following compiler interfaces. • Logger • ProgressMeter • PathResolver • Report
    45. 45. HELLFIRE COMPILER DAEMON • What is HFCD? It is an out-of-process Flex compiler. • Socket, TCP/IP based. Implemented a RPC version of the Flex Compiler API. • Usehardware, multicore technologies to speed up build performance. • Eclipse based. Flex Builder compatible. FDT support coming soon.
    46. 46. HFCD ARCHITECTURE
    47. 47. HFCD ADVANTAGES • Socket based. Could run HFCD on a second machine (i.e. access to more CPU and memory resources). • Use Java server VM to run the compiler. Consistently 30% faster than Java client VM. • Withmultiple CPU cores, HFCD builds applications/libraries concurrently if possible. Build speedup factor could reach the number of available CPUs.
    48. 48. HFCD ADVANTAGES • Use HFCD ant tasks, the ant <parallel> task and multiple HFCD servers to build. • For more information, please visit: • http://bytecode-workshop.com • http://stopcoding.wordpress.com
    49. 49. HFCD DEMO
    50. 50. Q/A

    ×