Avoiding Developer Taxes(Making development more robustwith introspection tools. )Stephen.Kennedy @ havok.comPrincipal Eng...
Awesome CodingAwesome “Other”     Collecting Awards                                        2
Awesome                      Meetings Coding             Refactoring          Debugging              Awards               ...
4
The Classic Game TaxesSerialization    Memory ReportingScript Binding     Versioning                                    5
Memory Lane10 Years of taxesWhy we changedAutomateCorrectnessSpread the workload                      6
StructureSerializationReflectionMemory ReportingScript BindingVersioning                   7
Manual serialization    xtream.TokenMustBe("POINT_A"); xtream>>point_A;    bool has_two_bodies = true;    if (xtream.GetVe...
Reflection Recap (1)struct   A0 { float x; byte y; }struct   A1 { float z; byte p; byte q };struct   A2 { byte i[3]; }…str...
Reflection Recap (2)struct A0 { float x; byte y; }char A0_type[]={ “FB” };void save_any( void* obj, char* type )  while(*t...
Serialization with Reflection Straightforward. (mostly) The problem now is …                                11
Get ReflectedHow to reflect our data?How to keep it up to date?Robustness                             12
Manual Reflection    class Foo {       public:                           RTTI_DESCRIBE_CLASS( Foo, (         enum Flags {…...
Parsing Headers    class Foo {                                Foo.h       public:         int i;         char* pc;        ...
We Went Full Auto   C++               Master.h       GCC-XML          DBheaders     Script1        Script2       Generate ...
Clang AST Consumerclass RawDumpASTConsumer : public ASTConsumer{    virtual void Initialize(ASTContext& Context);    virtu...
Clang Custom OutputFile( id=20270, location=Base/Types/Geometry/hkGeometry.h )RecordType( id=20271, name=hkGeometry, polym...
Build IntegrationPrebuild step runs Clang if necessaryPlugins run on the databaseThat’s it               Generate    Refle...
Runtime Introspectionstruct X{                         float   float time;     bool   bool used;             float   float...
R   20
Reflection ConclusionLLVM Clang pass    •   Robust    •   Eliminates out-of-sync errorsDB Consumer pass    •   Pre-compile...
Language BindingExpose C++ to scriptData: NaturalCallables: Harder                       22
Sample Interface    class Timeline    {         /// Adds a label at the given time and         /// returns an id for that ...
What is a Binding?     Lua State                     int addLabel(      Lua_String        float time,         “GDC”       ...
Three Parts of a Binding    Lua State             float time = lua_...                          char* label = lua_...     ...
Manual Bindings    int wrap_timeLine_addLabel(lua_state* s) {       TimeLine* tline = lua_checkuserdata(s,1);       int ar...
Normal “Fat” Bindings    timeLine:addLabel(1,”x”)                               1:1 wrapper:native        wrap_addLabel() ...
Slimmer Bindings?    wrap_x()   wrap_z()                           wrap_any()           wrap_y()                          ...
What is a Trampoline?typedef void (*ErrorFunc)(const char* msg, void* context);void set_error_function( ErrorFunc func, vo...
Reflected Functionstruct FunctionReflection {   const char* name;   Callable callable; // “function pointer”   TypeReflect...
Slimmer Bindingint wrap_anything(lua_state* s) {   FunctionReflection* fr = func_get(s);   Buffer buf;   unpack_args(s, fr...
Function Trampolinetypedef int (*Func_int__int_charp)(int i, char* c);void trampoline(void** buf, Func_int__int_charp func...
Trampolines                call_in_lua()                 lua_bridge()    trampoline_int()    trampoline_int_charp()f0(int ...
Fat vs Slim Memory Cost            N functions            T distinct trampolines (T≤N)    N * wrap_func()          1 lua_b...
Sharing the Trampolinestl:addLabel(1,”x”)           tl.addLabel(1,’x’)    lua_bridge()                  python_bridge()   ...
Bindings ConclusionGenerated “Slim” Bindings     Crossplatform & Multilanguage!Marginally slower     Extra indirectionCo...
Memory Reporting Goals    More than just block sizes    Account for every byte    Low maintenance burden    Customizable o...
Memory ReportingManual getMemoryStatistics()           void hkpMeshShape::calcContentStatistics                      ( hkS...
Automatic ReportsAim for 100% automatic coverageProvide debugging for missingLeapfrog Technique:      Remember types of (...
Raw BlocksRaw pairs of(address,size) M               40
Type Roots               ?          BvTreeHooked classoperator       Mesh   ?       ?new/delete               ? M         ...
Reflection WalkMesh {         ?            BvTree Section [];};               Mesh   Section   voidSection { Vector4[]; In...
Reflection WalkFinds all pointer-to                     vector4[]         BvTreerelationshipsDebug – Time &      Mesh     ...
Memory Implementation DetailsCustom Type Handlers        slack space – false positivesObfuscated Pointers        ptr ^= ...
Annotated Memory DumpD   1287169594M   Win32 “demos.exe"                        #L(ocation) <address> <string name>?T   0 ...
M   46
Memory Report ConclusionLabel root blocks & grow with reflectionWe found offline analysis usefulLow maintenanceAccounts fo...
Versioning    Change Happens    Hitting a moving target.    Not much in literature.V                              48
Manual Conditionals                            xtream.TokenMustBe("POINT_A"); xtream>>point_A;                            ...
Ideal Versioning System    Don’t slow me down    Don’t constrain my changes    Help me fix it up    Don’t make me think   ...
Snapshot VersioningCompare all previous vs current metadataFunction updates changed membersCheck up-to-date with CRC of re...
Snapshot exampleA { int x; int y; }     A { int y; }B { A* a; }             B { A* a; int x;}void Update_B( void* oldB, vo...
Version FunctionA { int x; int y; }     A { int y; }B { A* a; }             B { A* a; int x;}void Update_B( Object& oldB, ...
Chaining Snapshot Updates    V1 → V2                                      V2 → Current    A { int x; int y; }      A { int...
What Have We Gained?    Manual                                                                                            ...
Finer Grained ChangesWe hired artistsFull snapshots too heavyNew products, lots of branches     •   No global timeline V  ...
Patching (1)A0 { int x; }            A0 { int x; }         A1 { int x; int y; }B0 { A* a; }             B1 { A* a; int y; ...
Patching (2)A0 { int x; }            A0 { int x; }          A1 { int x; int y; }B0 { A* a; }             B1 { A* a; int y;...
Sample Patch// This block is generated by the metadata comparison.// It is pasted into the source and expands out to a few...
Verification Previous       Apply        Patched Reflection    Patches      Reflection     Patches                Compare ...
Versioning WorkflowMake your changesRun a reflection diff utility    Copy & paste the suggested patch    Edit (add&remove ...
Versioning ConclusionMassively reduced taxQuick to add versioningCheap to testNon-prescriptive workflow                   ...
Data, Not Functions       Generating code?            Stop, generate data       Functions:            big and single pur...
Conclusion             github:clang-extract Reflection   Serialization   Memory         Binding       Versioning          ...
Upcoming SlideShare
Loading in...5
×

개발 과정 최적화 하기 내부툴로 더욱 강력한 개발하기 Stephen kennedy _(11시40분_103호)

411

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
411
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

개발 과정 최적화 하기 내부툴로 더욱 강력한 개발하기 Stephen kennedy _(11시40분_103호)

  1. 1. Avoiding Developer Taxes(Making development more robustwith introspection tools. )Stephen.Kennedy @ havok.comPrincipal Engineergithub:clang-extract
  2. 2. Awesome CodingAwesome “Other” Collecting Awards 2
  3. 3. Awesome Meetings Coding Refactoring Debugging Awards 3
  4. 4. 4
  5. 5. The Classic Game TaxesSerialization Memory ReportingScript Binding Versioning 5
  6. 6. Memory Lane10 Years of taxesWhy we changedAutomateCorrectnessSpread the workload 6
  7. 7. StructureSerializationReflectionMemory ReportingScript BindingVersioning 7
  8. 8. Manual serialization xtream.TokenMustBe("POINT_A"); xtream>>point_A; bool has_two_bodies = true; if (xtream.GetVersion() >= 1350) { xtream.TokenMustBe("HAS_TWO_BODIES"); xtream>>has_two_bodies; } if (has_two_bodies) { xtream.TokenMustBe("BODY_B"); xtream>>prot_buffer; priv_rigid_body_B = prot_subspace->GetRigidBody(prot_buffer); if (!priv_rigid_body_B) throw_exception("Rigidbody unknown in Spring"); } xtream.TokenMustBe("POINT_B"); xtream>>point_B; //…S 8
  9. 9. Reflection Recap (1)struct A0 { float x; byte y; }struct A1 { float z; byte p; byte q };struct A2 { byte i[3]; }…struct A1023 { };S 9
  10. 10. Reflection Recap (2)struct A0 { float x; byte y; }char A0_type[]={ “FB” };void save_any( void* obj, char* type ) while(*type) { switch( *type++ ) { case ‘F’:// write(); obj += sizeof(float); case ‘B’:// write(); obj += sizeof(bool); case 0: return; }S 10
  11. 11. Serialization with Reflection Straightforward. (mostly) The problem now is … 11
  12. 12. Get ReflectedHow to reflect our data?How to keep it up to date?Robustness 12
  13. 13. Manual Reflection class Foo { public: RTTI_DESCRIBE_CLASS( Foo, ( enum Flags {…}; RTTI_FIELD(i, RTTI_FLD_PUBLIC), int i; RTTI_PTR(pc, RTTI_FLD_PUBLIC), char* pc; RTTI_FIELD(d, RTTI_FLD_PUBLIC), double d; RTTI_FIELD(f, RTTI_FLD_PUBLIC), Flags flags; RTTI_ARRAY(larr, RTTI_FLD_PROTECTED), RTTI_PTR(pa, RTTI_FLD_PROTECTED) protected: ) ); long larr[10]; A* pa; };R 13
  14. 14. Parsing Headers class Foo { Foo.h public: int i; char* pc; double d; enum Flags flags; protected: Dirty long larr[10]; Regexes class B* pb; #ifdef PLATFORM_Y special* s; #endif FooReflect.cpp };R 14
  15. 15. We Went Full Auto C++ Master.h GCC-XML DBheaders Script1 Script2 Generate Reflection R 15
  16. 16. Clang AST Consumerclass RawDumpASTConsumer : public ASTConsumer{ virtual void Initialize(ASTContext& Context); virtual void HandleTopLevelDecl(DeclGroupRef DG) { // ... if( const FieldDecl* fd = dyn_cast<FieldDecl>(declIn) ) // ... else if( const CXXMethodDecl* md = dyn_cast<CXXMethodDecl>(declIn) ) // ... else if( const EnumConstantDecl* ed = dyn_cast<EnumConstantDecl>(declIn) ) // ... }}; R 16
  17. 17. Clang Custom OutputFile( id=20270, location=Base/Types/Geometry/hkGeometry.h )RecordType( id=20271, name=hkGeometry, polymorphic=False,abstract=False, scopeid=20270 )Method( id=20317, recordid=20271, typeid=20316,name=getTriangle )Field( id=20320, recordid=20271, typeid=9089,name=m_vertices ) R 17
  18. 18. Build IntegrationPrebuild step runs Clang if necessaryPlugins run on the databaseThat’s it Generate Reflect.cpp ReflectionClangAST DB Static Analysis R 18
  19. 19. Runtime Introspectionstruct X{ float float time; bool bool used; float float pos; bool canmove; bool short float short flags; short bool bool}; float R 19
  20. 20. R 20
  21. 21. Reflection ConclusionLLVM Clang pass • Robust • Eliminates out-of-sync errorsDB Consumer pass • Pre-compile logic checks • Runtime errors → compile errorsUnit Tests • Examine reflection data 21
  22. 22. Language BindingExpose C++ to scriptData: NaturalCallables: Harder 22
  23. 23. Sample Interface class Timeline { /// Adds a label at the given time and /// returns an id for that label int addLabel( float time, const char* label ); };B 23
  24. 24. What is a Binding? Lua State int addLabel( Lua_String float time, “GDC” const char* label) { Lua_Num // ... 1.4 }B 24
  25. 25. Three Parts of a Binding Lua State float time = lua_... char* label = lua_... Lua_String “GDC” push time Lua_Num push label 1.4 call addLabel pop id Lua_Int ret = 64 lua_set_return( id )B 25
  26. 26. Manual Bindings int wrap_timeLine_addLabel(lua_state* s) { TimeLine* tline = lua_checkuserdata(s,1); int arg0 = lua_checkint(s,2); const char* arg1 = lua_checkstring(s,3); int id = tline->addLabel( arg0, arg1 ); lua_pushint(id); return 1; } timeLine:addLabel(1,”x”)B 26
  27. 27. Normal “Fat” Bindings timeLine:addLabel(1,”x”) 1:1 wrapper:native wrap_addLabel() Manual or Generated ~400b per wrapper TimeLine::addLabel()B 27
  28. 28. Slimmer Bindings? wrap_x() wrap_z() wrap_any() wrap_y() data_x data_y wrap_z() data_z data_wB 28
  29. 29. What is a Trampoline?typedef void (*ErrorFunc)(const char* msg, void* context);void set_error_function( ErrorFunc func, void* context );class MyHandler { void on_error(const char* msg) = 0; };void trampoline(const...;MyHandler* handler = char* msg, void* context) {set_error_function( &MyHandler::on_error, handler } ((MyErrorHandler*)context)->on_error(msg); );MyErrorHandler* handler = ...;set_error_function( &trampoline, handler );B 29
  30. 30. Reflected Functionstruct FunctionReflection { const char* name; Callable callable; // “function pointer” TypeReflection* argTypes; int numArgs; // Including return type};B 30
  31. 31. Slimmer Bindingint wrap_anything(lua_state* s) { FunctionReflection* fr = func_get(s); Buffer buf; unpack_args(s, fr->argTypes, buf); (*fr->callable)(buf); return pack_args(s, fr->argTypes, buf);}B 31
  32. 32. Function Trampolinetypedef int (*Func_int__int_charp)(int i, char* c);void trampoline(void** buf, Func_int__int_charp funcptr){ int& ret = *(int*)buf[0]; int& a0 = *(int*)buf[1]; char* a1 = *(char**)buf[2]; ret = (*funcptr)(a0,a1);}B 32
  33. 33. Trampolines call_in_lua() lua_bridge() trampoline_int() trampoline_int_charp()f0(int x) f1(int x) native(int,char*)B 33
  34. 34. Fat vs Slim Memory Cost N functions T distinct trampolines (T≤N) N * wrap_func() 1 lua_bridge() N * Reflection T * trampoline() N * func() N * func() ~400*N ~40*N + ~64*TB 34
  35. 35. Sharing the Trampolinestl:addLabel(1,”x”) tl.addLabel(1,’x’) lua_bridge() python_bridge() trampoline() the_actual_native_function()B 35
  36. 36. Bindings ConclusionGenerated “Slim” Bindings  Crossplatform & Multilanguage!Marginally slower  Extra indirectionConsiderably smaller 36
  37. 37. Memory Reporting Goals More than just block sizes Account for every byte Low maintenance burden Customizable outputM 37
  38. 38. Memory ReportingManual getMemoryStatistics() void hkpMeshShape::calcContentStatistics ( hkStatisticsCollector* collector ) constBuggy { collector->addArray( "SubParts", this->m_subparts ); for( int i = 0; i < this->m_childInfo.getSize(); i++ ) { collector->addReferencedObject( "Child", m_childInfo[i].m_shape );Tedious } hkpShapeCollection::calcContentStatistics(collector); }M 38
  39. 39. Automatic ReportsAim for 100% automatic coverageProvide debugging for missingLeapfrog Technique:  Remember types of (some) allocations  Know offsets of pointers in each type M 39
  40. 40. Raw BlocksRaw pairs of(address,size) M 40
  41. 41. Type Roots ? BvTreeHooked classoperator Mesh ? ?new/delete ? M 41
  42. 42. Reflection WalkMesh { ? BvTree Section [];}; Mesh Section voidSection { Vector4[]; Indices[]; ?}; M 42
  43. 43. Reflection WalkFinds all pointer-to vector4[] BvTreerelationshipsDebug – Time & Mesh Section voidStacktraceVerify everything int[3][]reached M 43
  44. 44. Memory Implementation DetailsCustom Type Handlers  slack space – false positivesObfuscated Pointers  ptr ^= 1;Untyped Data  void* M 44
  45. 45. Annotated Memory DumpD 1287169594M Win32 “demos.exe" #L(ocation) <address> <string name>?T 0 UNKNOWN #C(allstack) <callstack id> <address>+T 1 hkNativeFileSystem #T(ype) <type id> <type name>C 13 0x29656826 0x29828312 … #a(llocation) <alloc id> <size>a 0 8 #t(ype instance) <alloc id> <type id>t 0 1 #c(allstack instance) <alloc id> <callstack id>c 0 13 #o(wns) <alloc id> <owned alloc id>+… #M(odule information) <platform-dependent>o 1 113 9 #D(ate of capture) <timestamp>o 2 193…L 0x29656826 sourcecommonbasesystemiofilesystemhknativefilesystem.h(90):hkNativeFileSystem::operatornew M 45
  46. 46. M 46
  47. 47. Memory Report ConclusionLabel root blocks & grow with reflectionWe found offline analysis usefulLow maintenanceAccounts for all reachable memory  Or tells you where it came from 47
  48. 48. Versioning Change Happens Hitting a moving target. Not much in literature.V 48
  49. 49. Manual Conditionals xtream.TokenMustBe("POINT_A"); xtream>>point_A; bool has_two_bodies = true; Version number if (xtream.GetVersion()>=1350) { Conditionals xtream.TokenMustBe("HAS_TWO_BODIES"); xtream>>has_two_bodies; Inter-object changes? } if (has_two_bodies) { Large refactor? xtream.TokenMustBe("BODY_B"); xtream>>prot_buffer; priv_rigid_body_B = s->GetRigidBody(prot_buffer); if (!priv_rigid_body_B) throw_exception("Rigidbody unknown in Spring"); } xtream.TokenMustBe("POINT_B"); xtream>>point_B;V //… 49
  50. 50. Ideal Versioning System Don’t slow me down Don’t constrain my changes Help me fix it up Don’t make me think Don’t make me revisitV 50
  51. 51. Snapshot VersioningCompare all previous vs current metadataFunction updates changed membersCheck up-to-date with CRC of reflectionV 51
  52. 52. Snapshot exampleA { int x; int y; } A { int y; }B { A* a; } B { A* a; int x;}void Update_B( void* oldB, void* newB ) { // newB->x = oldB->a->x; }{ “A”, 0x1234, 0x8989, NULL }{ “B”, 0xabcd, 0xfed4, Update_B }V 52
  53. 53. Version FunctionA { int x; int y; } A { int y; }B { A* a; } B { A* a; int x;}void Update_B( Object& oldB, Object& newB ){ newB[“x”] = oldB[“a”].asObject()[“x”].asInt();}V 53
  54. 54. Chaining Snapshot Updates V1 → V2 V2 → Current A { int x; int y; } A { int y; } A { int y; } A { int y; int z; } B { A* a; } B { A* a; int x;} B { A* a; int x; } B { A* a; int x; } Current void Update_B( void* oldB, void* newB ) { void Update_A( void* oldA, void* newA ) { // newB->x = oldB->a->x; } // newA->z = 1000; } { “A”, 0x1234, 0x8989, NULL } { “A”, 0x8989, 0x52c1, Update_A } { “B”, 0xabcd, 0xfed4, Update_B } { “B”, 0xfed4, 0xf115, NULL }V 54
  55. 55. What Have We Gained? Manual if version<1 if version<1 if version<2 if version<1 if version<2 if version<3 v0 v1 v2 Now Snapshots A { int x; int y; } A { int y; } A { int x; int y; } A { int y; } A { int x; int y; } A { int y; } B { A* a; } B { A* a; int x;} B { A* a; } B { A* a; int x;} B { A* a; } B { A* a; int x;} void Update_B( void* oldB, void* newB ) { void Update_B( void* oldB, void* newB ) { void Update_B( void* oldB, void* newB ) { // newB->x = oldB->a->x; } // newB->x = oldB->a->x; } // newB->x = oldB->a->x; } { “A”, 0x1234, 0x8989, NULL } { “A”, 0x1234, 0x8989, NULL } { “A”, 0x1234, 0x8989, NULL } { “B”, 0xabcd, 0xfed4, Update_B } { “B”, 0xabcd, 0xfed4, Update_B } { “B”, 0xabcd, 0xfed4, Update_B } v0 v1 v2 NowV 55
  56. 56. Finer Grained ChangesWe hired artistsFull snapshots too heavyNew products, lots of branches • No global timeline V 56
  57. 57. Patching (1)A0 { int x; } A0 { int x; } A1 { int x; int y; }B0 { A* a; } B1 { A* a; int y; } B2 { A* a; } B0→B1 B.add(int,“y”) V 57
  58. 58. Patching (2)A0 { int x; } A0 { int x; } A1 { int x; int y; }B0 { A* a; } B1 { A* a; int y; } B2 { A* a; } A0→A1 B1→B2 B0→B1 A.add(int,“y”) B.add(int,“y”) A.y = B.y B.rem(“y”) V 58
  59. 59. Sample Patch// This block is generated by the metadata comparison.// It is pasted into the source and expands out to a few constant structures.PATCH_BEGIN("hkbLookAtModifier", 2, "hkbLookAtModifier", 3) // Require hkbEventBase version 3 before running this patch PATCH_DEPENDS("hkbEventBase", 3) PATCH_MEMBER_ADDED_VEC_4("neckForwardLS", 0.0f,1.0f,0.0f,0.0f) // user edit: add & remove changed to rename PATCH_MEMBER_RENAMED("headForwardHS", "headForwardLS") // user edit: call a C function here PATCH_FUNCTION( hkbLookAtModifier_2_to_3 )PATCH_END() V 59
  60. 60. Verification Previous Apply Patched Reflection Patches Reflection Patches Compare Patches Wrong/ against current reflection OK Missing V 60
  61. 61. Versioning WorkflowMake your changesRun a reflection diff utility Copy & paste the suggested patch Edit (add&remove → rename) Write version function if necessaryDone V 61
  62. 62. Versioning ConclusionMassively reduced taxQuick to add versioningCheap to testNon-prescriptive workflow 62
  63. 63. Data, Not Functions Generating code?  Stop, generate data Functions:  big and single purpose. Data:  small and multi-purpose 63
  64. 64. Conclusion github:clang-extract Reflection Serialization Memory Binding Versioning 64
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×