Gregor modules

498 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
498
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Gregor modules

  1. 1. Modules Doug Gregor, AppleWednesday, November 7, 12
  2. 2. Roadmap • The fundamental brokenness of headers • A module system for the C family • Building better toolsWednesday, November 7, 12
  3. 3. On the Fundamental Brokenness of HeadersWednesday, November 7, 12
  4. 4. C Preprocessing Model #include <stdio.h> int main() { printf(“Hello, world!n”); }Wednesday, November 7, 12
  5. 5. C Preprocessing Model #include <stdio.h> // stdio.h int main() { typedef struct { printf(“Hello, world!n”); ... } } FILE; int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on...Wednesday, November 7, 12
  6. 6. C Preprocessing Model #include <stdio.h> // stdio.h int main() { typedef struct { printf(“Hello, world!n”); ... } } FILE; int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on...Wednesday, November 7, 12
  7. 7. C Preprocessing Model // from stdio.h typedef struct { ... } FILE; int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on... int main() { printf(“Hello, world!n”); }Wednesday, November 7, 12
  8. 8. Problems with the Model // from stdio.h typedef struct { ... } FILE; int printf(const char*, ...); int fprintf(FILE *, const char*, ...); • Fragility int remove(const char*); // on and on... • Performance int main() { printf(“Hello, world!n”); }Wednesday, November 7, 12
  9. 9. Header Fragility #define FILE “MyFile.txt” #include <stdio.h> int main() { printf(“Hello, world!n”); }Wednesday, November 7, 12
  10. 10. Header Fragility #define FILE “MyFile.txt” // stdio.h #include <stdio.h> typedef struct { int main() { ... printf(“Hello, world!n”); } FILE; } int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on...Wednesday, November 7, 12
  11. 11. Header Fragility #define FILE “MyFile.txt” // stdio.h #include <stdio.h> typedef struct { int main() { ... printf(“Hello, world!n”); } FILE; } int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on...Wednesday, November 7, 12
  12. 12. Header Fragility // from stdio.h typedef struct { ... } “MyFile.txt”; int printf(const char*, ...); int fprintf(“MyFile.txt” *, const char*, ...); int remove(const char*); // on and on... int main() { printf(“Hello, world!n”); }Wednesday, November 7, 12
  13. 13. Conventional WorkaroundsWednesday, November 7, 12
  14. 14. Conventional Workarounds • LLVM_WHY_PREFIX_UPPER_MACROSWednesday, November 7, 12
  15. 15. Conventional Workarounds • LLVM_WHY_PREFIX_UPPER_MACROS • LLVM_CLANG_INCLUDE_GUARD_HWednesday, November 7, 12
  16. 16. Conventional Workarounds • LLVM_WHY_PREFIX_UPPER_MACROS • LLVM_CLANG_INCLUDE_GUARD_H • template<class _Tp> const _Tp& min(const _Tp &__a, const _Tp &__b);Wednesday, November 7, 12
  17. 17. Conventional Workarounds • LLVM_WHY_PREFIX_UPPER_MACROS • LLVM_CLANG_INCLUDE_GUARD_H • template<class _Tp> const _Tp& min(const _Tp &__a, const _Tp &__b); • #include <windows.h> #undef min // because #define NOMINMAX #undef max // doesn’t always workWednesday, November 7, 12
  18. 18. How Big is a Source File?Wednesday, November 7, 12
  19. 19. How Big is a Source File? #include <stdio.h> int main() { printf(“Hello, world!n”); }Wednesday, November 7, 12
  20. 20. How Big is a Source File? #include <stdio.h> int main() { printf(“Hello, world!n”); } C Hello C++ Hello SemaOverload Source 64 81 469,939 Headers 11,072 1,161,033 3,824,521Wednesday, November 7, 12
  21. 21. How Big is a Source File? #include <stdio.h> #include <iostream> int main() { int main() { printf(“Hello, world!n”); std::cout << “Hello, world!” } << std::endl; } C Hello C++ Hello SemaOverload Source 64 81 469,939 Headers 11,072 1,161,033 3,824,521Wednesday, November 7, 12
  22. 22. How Big is a Source File? #include <stdio.h> #include <iostream> int main() { int main() { printf(“Hello, world!n”); std::cout << “Hello, world!” } << std::endl; } C Hello C++ Hello SemaOverload Source 64 81 469,939 Headers 11,072 1,161,033 3,824,521Wednesday, November 7, 12
  23. 23. How Big is a Source File? #include <stdio.h> #include <iostream> int main() { int main() { printf(“Hello, world!n”); std::cout << “Hello, world!” } << std::endl; } C Hello C++ Hello SemaOverload Source 64 81 469,939 Headers 11,072 1,161,033 3,824,521Wednesday, November 7, 12
  24. 24. Inherently Non-Scalable • M headers with N source files • M x N build cost • C++ templates exacerbate the problem • Precompiled headers are a terrible solutionWednesday, November 7, 12
  25. 25. A Module System for the C FamilyWednesday, November 7, 12
  26. 26. What Is a Module? • A module is a package describing a library • Interface of the library (API) • Implementation of the libraryWednesday, November 7, 12
  27. 27. Module Imports import std; int main() { printf(“Hello, World!n”); } • ‘import’ makes the API of the named module availableWednesday, November 7, 12
  28. 28. Module Imports // std module (includes stdio) import std; typedef struct { int main() { ... printf(“Hello, World!n”); } FILE; } int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on... • ‘import’ makes the API of the named module availableWednesday, November 7, 12
  29. 29. Module Imports // std module (includes stdio) import std; typedef struct { int main() { ... printf(“Hello, World!n”); } FILE; } int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on... • ‘import’ makes the API of the named module availableWednesday, November 7, 12
  30. 30. Import Resilience • ‘import’ ignores preprocessor state within the source fileWednesday, November 7, 12
  31. 31. Import Resilience #define FILE “MyFile.txt” import std; int main() { printf(“Hello, World!n”); } • ‘import’ ignores preprocessor state within the source fileWednesday, November 7, 12
  32. 32. Import Resilience // std module (includes stdio) #define FILE “MyFile.txt” import std; typedef struct { ... int main() { } FILE; printf(“Hello, World!n”); } int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on... • ‘import’ ignores preprocessor state within the source fileWednesday, November 7, 12
  33. 33. Import Resilience // std module (includes stdio) #define FILE “MyFile.txt” import std; typedef struct { ... int main() { } FILE; printf(“Hello, World!n”); } int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on... • ‘import’ ignores preprocessor state within the source fileWednesday, November 7, 12
  34. 34. Selective Import // std module // stdio submodule typedef struct { ... } FILE; int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on... // stdlib submodule void abort(void); int rand(void); // on and on...Wednesday, November 7, 12
  35. 35. Selective Import // std module import std.stdio; // stdio submodule int main() { typedef struct { printf(“Hello, World!n”); ... } } FILE; int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on... // stdlib submodule void abort(void); int rand(void); // on and on...Wednesday, November 7, 12
  36. 36. Selective Import // std module import std.stdio; // stdio submodule int main() { typedef struct { printf(“Hello, World!n”); ... } } FILE; int printf(const char*, ...); int fprintf(FILE *, const char*, ...); int remove(const char*); // on and on... // stdlib submodule void abort(void); int rand(void); // on and on...Wednesday, November 7, 12
  37. 37. What Does import Import? • Functions, variables, types, templates, macros, etc. • Only public API -- everything else can be hidden. • No special namespace mechanism.Wednesday, November 7, 12
  38. 38. Writing a Module Futuristic VersionWednesday, November 7, 12
  39. 39. Writing a Module // stdio.c export std.stdio: public: typedef struct { ... } FILE; int printf(const char*, ...) { // ... } int fprintf(FILE *, const char*, ...) { // ... } int remove(const char*) { // ... }Wednesday, November 7, 12
  40. 40. Writing a Module // stdio.c export std.stdio: public: typedef struct { ... } FILE; • Specify module name in source file int printf(const char*, ...) { // ... } int fprintf(FILE *, const char*, ...) { // ... } int remove(const char*) { // ... }Wednesday, November 7, 12
  41. 41. Writing a Module // stdio.c export std.stdio: public: typedef struct { ... } FILE; • Specify module name in source file int printf(const char*, ...) { } // ... • Public access int fprintf(FILE *, describes API const char*, ...) { // ... } int remove(const char*) { // ... }Wednesday, November 7, 12
  42. 42. Writing a Module // stdio.c export std.stdio: public: typedef struct { ... } FILE; • Specify module name in source file int printf(const char*, ...) { } // ... • Public access int fprintf(FILE *, describes API const char*, ...) { } // ... • No headers! int remove(const char*) { // ... }Wednesday, November 7, 12
  43. 43. Problems With This Future • Transitioning existing header-based libraries • Interoperability with compilers that don’t implement modules • Requires tools that understand modulesWednesday, November 7, 12
  44. 44. Writing a Module Transitional VersionWednesday, November 7, 12
  45. 45. Embracing Headers • Build modules directly from the headers • Headers remain “the truth” • Good for interoperability • Doesn’t change the programmer modelWednesday, November 7, 12
  46. 46. Module Maps // /usr/include/module.map module std { module stdio { header “stdio.h” } module stdlib { header “stdlib.h” } module math { header “math.h” } } • module defines a named (sub)module • header includes the contents of the named header in the current (sub)moduleWednesday, November 7, 12
  47. 47. Umbrella Headers // clang/include/clang/module.map module ClangAST { umbrella header “AST/AST.h” module * { } } • An umbrella header includes all of the headers in its directoryWednesday, November 7, 12
  48. 48. Umbrella Headers // clang/include/clang/module.map module ClangAST { umbrella header “AST/AST.h” module * { } } • An umbrella header includes all of the headers in its directory • Wildcard submodules (module *) create a submodule for each included header • AST/Decl.h -> ClangAST.Decl • AST/Expr.h -> ClangAST.ExprWednesday, November 7, 12
  49. 49. Module Map FeaturesWednesday, November 7, 12
  50. 50. Module Map Features module LLVMADT { • Umbrella directories } umbrella “llvm/ADT” module * { export * }Wednesday, November 7, 12
  51. 51. Module Map Features module LLVMADT { • Umbrella directories } umbrella “llvm/ADT” module * { export * } module _Builtin { module avx { • Submodule requirements } requires avx header “avxintrin.h” }Wednesday, November 7, 12
  52. 52. Module Map Features module LLVMADT { • Umbrella directories } umbrella “llvm/ADT” module * { export * } module _Builtin { module avx { • Submodule requirements } requires avx header “avxintrin.h” } • Excluded headers module std { exclude header “assert.h” }Wednesday, November 7, 12
  53. 53. Compilation Model import std.stdio; int main() { printf(“Hello, World!n”); }Wednesday, November 7, 12
  54. 54. Compilation Model import std.stdio; int main() { printf(“Hello, World!n”); } 1. Find a module map for the named moduleWednesday, November 7, 12
  55. 55. Compilation Model import std.stdio; int main() { printf(“Hello, World!n”); } 1. Find a module map for the named module 2. Spawn a separate instance of the compiler: • Parse the headers in the module map • Write the module fileWednesday, November 7, 12
  56. 56. Compilation Model import std.stdio; int main() { printf(“Hello, World!n”); } 1. Find a module map for the named module 2. Spawn a separate instance of the compiler: • Parse the headers in the module map • Write the module file 3. Load the module file at the ‘import’ declarationWednesday, November 7, 12
  57. 57. Compilation Model import std.stdio; int main() { printf(“Hello, World!n”); } 1. Find a module map for the named module 2. Spawn a separate instance of the compiler: • Parse the headers in the module map • Write the module file 3. Load the module file at the ‘import’ declaration 4. Cache module file for later re-useWednesday, November 7, 12
  58. 58. Adopting Modules: Libraries • Eliminate non-modular behavior: • Multiply-defined structs, functions, macros, must be consolidated • Headers should import what they depend on • Write module maps covering the libraryWednesday, November 7, 12
  59. 59. Adopting Modules: Users #include <stdio.h> int main() { printf(“Hello, World!n”); }Wednesday, November 7, 12
  60. 60. Adopting Modules: Users #include <stdio.h> import std.stdio; int main() { int main() { printf(“Hello, World!n”); printf(“Hello, World!n”); } } • “Simply” rewrite each #include as an importWednesday, November 7, 12
  61. 61. Translating #include to import #include <stdio.h> import std.stdio; int main() { int main() { printf(“Hello, World!n”); printf(“Hello, World!n”); } }Wednesday, November 7, 12
  62. 62. Translating #include to import #include <stdio.h> import std.stdio; int main() { int main() { printf(“Hello, World!n”); printf(“Hello, World!n”); } } • Use module maps to determine (sub)module corresponding to an #include’d headerWednesday, November 7, 12
  63. 63. Translating #include to import #include <stdio.h> import std.stdio; int main() { int main() { printf(“Hello, World!n”); printf(“Hello, World!n”); } } • Use module maps to determine (sub)module corresponding to an #include’d header • Optional Fix-Its, tooling to finalize the rewriteWednesday, November 7, 12
  64. 64. Translating #include to import #include <stdio.h> import std.stdio; int main() { int main() { printf(“Hello, World!n”); printf(“Hello, World!n”); } } • Use module maps to determine (sub)module corresponding to an #include’d header • Optional Fix-Its, tooling to finalize the rewrite • Enabling modules is transparent to the userWednesday, November 7, 12
  65. 65. Building Better Tools (For the Future)Wednesday, November 7, 12
  66. 66. Compilation Performance • Algorithmic improvement for parsing time • Module headers parsed once, cached • MxN M+N • Benefits for all source-based toolsWednesday, November 7, 12
  67. 67. Automatic Linking // clang/include/clang/module.map module ClangAST { umbrella header “AST/AST.h” module * { } link “-lclangAST” } • Drastically simplifies use of a libraryWednesday, November 7, 12
  68. 68. Automatic Import int main() { std::vector<int> v; } • Forgotten #include terrible diagnosticWednesday, November 7, 12
  69. 69. Automatic Import int main() { std::vector<int> v; } • Forgotten #include terrible diagnostic • vector.cpp:2:6: error: ‘vector‘ template is not available std::vector<int> v; ^ note: import ‘std.vector‘ to use ‘std::vector’Wednesday, November 7, 12
  70. 70. Debugging FlowWednesday, November 7, 12
  71. 71. Debugging Flow <iostream> foo.cppWednesday, November 7, 12
  72. 72. Debugging Flow <iostream> foo.cpp Clang ASTsWednesday, November 7, 12
  73. 73. Debugging Flow foo.o iostream DWARF <iostream> foo.cpp Clang ASTs foo DWARF object codeWednesday, November 7, 12
  74. 74. Debugging Flow foo.o iostream DWARF <iostream> foo.cpp Clang ASTs foo LLDB DWARF object codeWednesday, November 7, 12
  75. 75. Debugging Flow foo.o iostream DWARF <iostream> foo.cpp Clang ASTs foo LLDB Clang ASTs DWARF object codeWednesday, November 7, 12
  76. 76. Debugging Flow foo.o iostream DWARF <iostream> foo.cpp Clang ASTs foo LLDB Clang ASTs DWARF object code • Round-trip through DWARF is lossy • Only ‘used’ types, functions available • Inline functions, template definitions lostWednesday, November 7, 12
  77. 77. Redundant Debug Info foo.o iostream DWARF <iostream> foo.cpp Clang ASTs foo LLDB Clang ASTs DWARF object codeWednesday, November 7, 12
  78. 78. Redundant Debug Info foo.o iostream DWARF <iostream> foo.cpp Clang ASTs foo LLDB Clang ASTs DWARF object code bar.o iostream DWARF <iostream> bar.cpp Clang ASTs foo DWARF object codeWednesday, November 7, 12
  79. 79. Redundant Debug Info foo.o iostream DWARF <iostream> foo.cpp Clang ASTs foo LLDB Clang ASTs DWARF object code bar.o iostream DWARF <iostream> bar.cpp Clang ASTs foo DWARF object codeWednesday, November 7, 12
  80. 80. Debugging with Modules foo.o <iostream> foo.cpp Clang ASTs foo DWARF object code LLDB Clang ASTs bar.o <iostream> bar.cpp Clang ASTs foo DWARF object codeWednesday, November 7, 12
  81. 81. Debugging with Modules std.iostream module foo.o <iostream> foo.cpp Clang ASTs foo DWARF object code LLDB Clang ASTs bar.o <iostream> bar.cpp Clang ASTs foo DWARF object codeWednesday, November 7, 12
  82. 82. Debugging with Modules • Improved build performance • Compiler emits less DWARF • Linker de-duplicates less DWARFWednesday, November 7, 12
  83. 83. Debugging with Modules • Improved build performance • Compiler emits less DWARF • Linker de-duplicates less DWARF • Improved debugging experience • Perfect AST fidelity in debugger • Debugger doesn’t need to search DWARFWednesday, November 7, 12
  84. 84. Modules SummaryWednesday, November 7, 12
  85. 85. Modules Summary • Modules are a huge potential win for C(++) • Compile/build time improvements • Fix various preprocessor problems • Far better tool experienceWednesday, November 7, 12
  86. 86. Modules Summary • Modules are a huge potential win for C(++) • Compile/build time improvements • Fix various preprocessor problems • Far better tool experience • Design enables smooth transition pathWednesday, November 7, 12
  87. 87. Modules Summary • Modules are a huge potential win for C(++) • Compile/build time improvements • Fix various preprocessor problems • Far better tool experience • Design enables smooth transition path • Clang implementation underwayWednesday, November 7, 12

×