개발 과정 최적화 하기 내부툴로 더욱 강력한 개발하기 Stephen kennedy _(11시40분_103호)
Upcoming SlideShare
Loading in...5
×
 

개발 과정 최적화 하기 내부툴로 더욱 강력한 개발하기 Stephen kennedy _(11시40분_103호)

on

  • 635 views

 

Statistics

Views

Total Views
635
Views on SlideShare
635
Embed Views
0

Actions

Likes
2
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    개발 과정 최적화 하기 내부툴로 더욱 강력한 개발하기 Stephen kennedy _(11시40분_103호) 개발 과정 최적화 하기 내부툴로 더욱 강력한 개발하기 Stephen kennedy _(11시40분_103호) Presentation Transcript

    • Avoiding Developer Taxes(Making development more robustwith introspection tools. )Stephen.Kennedy @ havok.comPrincipal Engineergithub:clang-extract
    • Awesome CodingAwesome “Other” Collecting Awards 2
    • Awesome Meetings Coding Refactoring Debugging Awards 3
    • 4
    • The Classic Game TaxesSerialization Memory ReportingScript Binding Versioning 5
    • Memory Lane10 Years of taxesWhy we changedAutomateCorrectnessSpread the workload 6
    • StructureSerializationReflectionMemory ReportingScript BindingVersioning 7
    • Manual serialization xtream.TokenMustBe("POINT_A"); xtream>>point_A; bool has_two_bodies = true; if (xtream.GetVersion() >= 1350) { xtream.TokenMustBe("HAS_TWO_BODIES"); xtream>>has_two_bodies; } if (has_two_bodies) { xtream.TokenMustBe("BODY_B"); xtream>>prot_buffer; priv_rigid_body_B = prot_subspace->GetRigidBody(prot_buffer); if (!priv_rigid_body_B) throw_exception("Rigidbody unknown in Spring"); } xtream.TokenMustBe("POINT_B"); xtream>>point_B; //…S 8
    • Reflection Recap (1)struct A0 { float x; byte y; }struct A1 { float z; byte p; byte q };struct A2 { byte i[3]; }…struct A1023 { };S 9
    • Reflection Recap (2)struct A0 { float x; byte y; }char A0_type[]={ “FB” };void save_any( void* obj, char* type ) while(*type) { switch( *type++ ) { case ‘F’:// write(); obj += sizeof(float); case ‘B’:// write(); obj += sizeof(bool); case 0: return; }S 10
    • Serialization with Reflection Straightforward. (mostly) The problem now is … 11
    • Get ReflectedHow to reflect our data?How to keep it up to date?Robustness 12
    • Manual Reflection class Foo { public: RTTI_DESCRIBE_CLASS( Foo, ( enum Flags {…}; RTTI_FIELD(i, RTTI_FLD_PUBLIC), int i; RTTI_PTR(pc, RTTI_FLD_PUBLIC), char* pc; RTTI_FIELD(d, RTTI_FLD_PUBLIC), double d; RTTI_FIELD(f, RTTI_FLD_PUBLIC), Flags flags; RTTI_ARRAY(larr, RTTI_FLD_PROTECTED), RTTI_PTR(pa, RTTI_FLD_PROTECTED) protected: ) ); long larr[10]; A* pa; };R 13
    • Parsing Headers class Foo { Foo.h public: int i; char* pc; double d; enum Flags flags; protected: Dirty long larr[10]; Regexes class B* pb; #ifdef PLATFORM_Y special* s; #endif FooReflect.cpp };R 14
    • We Went Full Auto C++ Master.h GCC-XML DBheaders Script1 Script2 Generate Reflection R 15
    • Clang AST Consumerclass RawDumpASTConsumer : public ASTConsumer{ virtual void Initialize(ASTContext& Context); virtual void HandleTopLevelDecl(DeclGroupRef DG) { // ... if( const FieldDecl* fd = dyn_cast<FieldDecl>(declIn) ) // ... else if( const CXXMethodDecl* md = dyn_cast<CXXMethodDecl>(declIn) ) // ... else if( const EnumConstantDecl* ed = dyn_cast<EnumConstantDecl>(declIn) ) // ... }}; R 16
    • Clang Custom OutputFile( id=20270, location=Base/Types/Geometry/hkGeometry.h )RecordType( id=20271, name=hkGeometry, polymorphic=False,abstract=False, scopeid=20270 )Method( id=20317, recordid=20271, typeid=20316,name=getTriangle )Field( id=20320, recordid=20271, typeid=9089,name=m_vertices ) R 17
    • Build IntegrationPrebuild step runs Clang if necessaryPlugins run on the databaseThat’s it Generate Reflect.cpp ReflectionClangAST DB Static Analysis R 18
    • Runtime Introspectionstruct X{ float float time; bool bool used; float float pos; bool canmove; bool short float short flags; short bool bool}; float R 19
    • R 20
    • Reflection ConclusionLLVM Clang pass • Robust • Eliminates out-of-sync errorsDB Consumer pass • Pre-compile logic checks • Runtime errors → compile errorsUnit Tests • Examine reflection data 21
    • Language BindingExpose C++ to scriptData: NaturalCallables: Harder 22
    • Sample Interface class Timeline { /// Adds a label at the given time and /// returns an id for that label int addLabel( float time, const char* label ); };B 23
    • What is a Binding? Lua State int addLabel( Lua_String float time, “GDC” const char* label) { Lua_Num // ... 1.4 }B 24
    • Three Parts of a Binding Lua State float time = lua_... char* label = lua_... Lua_String “GDC” push time Lua_Num push label 1.4 call addLabel pop id Lua_Int ret = 64 lua_set_return( id )B 25
    • Manual Bindings int wrap_timeLine_addLabel(lua_state* s) { TimeLine* tline = lua_checkuserdata(s,1); int arg0 = lua_checkint(s,2); const char* arg1 = lua_checkstring(s,3); int id = tline->addLabel( arg0, arg1 ); lua_pushint(id); return 1; } timeLine:addLabel(1,”x”)B 26
    • Normal “Fat” Bindings timeLine:addLabel(1,”x”) 1:1 wrapper:native wrap_addLabel() Manual or Generated ~400b per wrapper TimeLine::addLabel()B 27
    • Slimmer Bindings? wrap_x() wrap_z() wrap_any() wrap_y() data_x data_y wrap_z() data_z data_wB 28
    • What is a Trampoline?typedef void (*ErrorFunc)(const char* msg, void* context);void set_error_function( ErrorFunc func, void* context );class MyHandler { void on_error(const char* msg) = 0; };void trampoline(const...;MyHandler* handler = char* msg, void* context) {set_error_function( &MyHandler::on_error, handler } ((MyErrorHandler*)context)->on_error(msg); );MyErrorHandler* handler = ...;set_error_function( &trampoline, handler );B 29
    • Reflected Functionstruct FunctionReflection { const char* name; Callable callable; // “function pointer” TypeReflection* argTypes; int numArgs; // Including return type};B 30
    • Slimmer Bindingint wrap_anything(lua_state* s) { FunctionReflection* fr = func_get(s); Buffer buf; unpack_args(s, fr->argTypes, buf); (*fr->callable)(buf); return pack_args(s, fr->argTypes, buf);}B 31
    • Function Trampolinetypedef int (*Func_int__int_charp)(int i, char* c);void trampoline(void** buf, Func_int__int_charp funcptr){ int& ret = *(int*)buf[0]; int& a0 = *(int*)buf[1]; char* a1 = *(char**)buf[2]; ret = (*funcptr)(a0,a1);}B 32
    • Trampolines call_in_lua() lua_bridge() trampoline_int() trampoline_int_charp()f0(int x) f1(int x) native(int,char*)B 33
    • Fat vs Slim Memory Cost N functions T distinct trampolines (T≤N) N * wrap_func() 1 lua_bridge() N * Reflection T * trampoline() N * func() N * func() ~400*N ~40*N + ~64*TB 34
    • Sharing the Trampolinestl:addLabel(1,”x”) tl.addLabel(1,’x’) lua_bridge() python_bridge() trampoline() the_actual_native_function()B 35
    • Bindings ConclusionGenerated “Slim” Bindings  Crossplatform & Multilanguage!Marginally slower  Extra indirectionConsiderably smaller 36
    • Memory Reporting Goals More than just block sizes Account for every byte Low maintenance burden Customizable outputM 37
    • Memory ReportingManual getMemoryStatistics() void hkpMeshShape::calcContentStatistics ( hkStatisticsCollector* collector ) constBuggy { collector->addArray( "SubParts", this->m_subparts ); for( int i = 0; i < this->m_childInfo.getSize(); i++ ) { collector->addReferencedObject( "Child", m_childInfo[i].m_shape );Tedious } hkpShapeCollection::calcContentStatistics(collector); }M 38
    • Automatic ReportsAim for 100% automatic coverageProvide debugging for missingLeapfrog Technique:  Remember types of (some) allocations  Know offsets of pointers in each type M 39
    • Raw BlocksRaw pairs of(address,size) M 40
    • Type Roots ? BvTreeHooked classoperator Mesh ? ?new/delete ? M 41
    • Reflection WalkMesh { ? BvTree Section [];}; Mesh Section voidSection { Vector4[]; Indices[]; ?}; M 42
    • Reflection WalkFinds all pointer-to vector4[] BvTreerelationshipsDebug – Time & Mesh Section voidStacktraceVerify everything int[3][]reached M 43
    • Memory Implementation DetailsCustom Type Handlers  slack space – false positivesObfuscated Pointers  ptr ^= 1;Untyped Data  void* M 44
    • Annotated Memory DumpD 1287169594M Win32 “demos.exe" #L(ocation) <address> <string name>?T 0 UNKNOWN #C(allstack) <callstack id> <address>+T 1 hkNativeFileSystem #T(ype) <type id> <type name>C 13 0x29656826 0x29828312 … #a(llocation) <alloc id> <size>a 0 8 #t(ype instance) <alloc id> <type id>t 0 1 #c(allstack instance) <alloc id> <callstack id>c 0 13 #o(wns) <alloc id> <owned alloc id>+… #M(odule information) <platform-dependent>o 1 113 9 #D(ate of capture) <timestamp>o 2 193…L 0x29656826 sourcecommonbasesystemiofilesystemhknativefilesystem.h(90):hkNativeFileSystem::operatornew M 45
    • M 46
    • Memory Report ConclusionLabel root blocks & grow with reflectionWe found offline analysis usefulLow maintenanceAccounts for all reachable memory  Or tells you where it came from 47
    • Versioning Change Happens Hitting a moving target. Not much in literature.V 48
    • Manual Conditionals xtream.TokenMustBe("POINT_A"); xtream>>point_A; bool has_two_bodies = true; Version number if (xtream.GetVersion()>=1350) { Conditionals xtream.TokenMustBe("HAS_TWO_BODIES"); xtream>>has_two_bodies; Inter-object changes? } if (has_two_bodies) { Large refactor? xtream.TokenMustBe("BODY_B"); xtream>>prot_buffer; priv_rigid_body_B = s->GetRigidBody(prot_buffer); if (!priv_rigid_body_B) throw_exception("Rigidbody unknown in Spring"); } xtream.TokenMustBe("POINT_B"); xtream>>point_B;V //… 49
    • Ideal Versioning System Don’t slow me down Don’t constrain my changes Help me fix it up Don’t make me think Don’t make me revisitV 50
    • Snapshot VersioningCompare all previous vs current metadataFunction updates changed membersCheck up-to-date with CRC of reflectionV 51
    • Snapshot exampleA { int x; int y; } A { int y; }B { A* a; } B { A* a; int x;}void Update_B( void* oldB, void* newB ) { // newB->x = oldB->a->x; }{ “A”, 0x1234, 0x8989, NULL }{ “B”, 0xabcd, 0xfed4, Update_B }V 52
    • Version FunctionA { int x; int y; } A { int y; }B { A* a; } B { A* a; int x;}void Update_B( Object& oldB, Object& newB ){ newB[“x”] = oldB[“a”].asObject()[“x”].asInt();}V 53
    • Chaining Snapshot Updates V1 → V2 V2 → Current A { int x; int y; } A { int y; } A { int y; } A { int y; int z; } B { A* a; } B { A* a; int x;} B { A* a; int x; } B { A* a; int x; } Current void Update_B( void* oldB, void* newB ) { void Update_A( void* oldA, void* newA ) { // newB->x = oldB->a->x; } // newA->z = 1000; } { “A”, 0x1234, 0x8989, NULL } { “A”, 0x8989, 0x52c1, Update_A } { “B”, 0xabcd, 0xfed4, Update_B } { “B”, 0xfed4, 0xf115, NULL }V 54
    • What Have We Gained? Manual if version<1 if version<1 if version<2 if version<1 if version<2 if version<3 v0 v1 v2 Now Snapshots A { int x; int y; } A { int y; } A { int x; int y; } A { int y; } A { int x; int y; } A { int y; } B { A* a; } B { A* a; int x;} B { A* a; } B { A* a; int x;} B { A* a; } B { A* a; int x;} void Update_B( void* oldB, void* newB ) { void Update_B( void* oldB, void* newB ) { void Update_B( void* oldB, void* newB ) { // newB->x = oldB->a->x; } // newB->x = oldB->a->x; } // newB->x = oldB->a->x; } { “A”, 0x1234, 0x8989, NULL } { “A”, 0x1234, 0x8989, NULL } { “A”, 0x1234, 0x8989, NULL } { “B”, 0xabcd, 0xfed4, Update_B } { “B”, 0xabcd, 0xfed4, Update_B } { “B”, 0xabcd, 0xfed4, Update_B } v0 v1 v2 NowV 55
    • Finer Grained ChangesWe hired artistsFull snapshots too heavyNew products, lots of branches • No global timeline V 56
    • Patching (1)A0 { int x; } A0 { int x; } A1 { int x; int y; }B0 { A* a; } B1 { A* a; int y; } B2 { A* a; } B0→B1 B.add(int,“y”) V 57
    • Patching (2)A0 { int x; } A0 { int x; } A1 { int x; int y; }B0 { A* a; } B1 { A* a; int y; } B2 { A* a; } A0→A1 B1→B2 B0→B1 A.add(int,“y”) B.add(int,“y”) A.y = B.y B.rem(“y”) V 58
    • Sample Patch// This block is generated by the metadata comparison.// It is pasted into the source and expands out to a few constant structures.PATCH_BEGIN("hkbLookAtModifier", 2, "hkbLookAtModifier", 3) // Require hkbEventBase version 3 before running this patch PATCH_DEPENDS("hkbEventBase", 3) PATCH_MEMBER_ADDED_VEC_4("neckForwardLS", 0.0f,1.0f,0.0f,0.0f) // user edit: add & remove changed to rename PATCH_MEMBER_RENAMED("headForwardHS", "headForwardLS") // user edit: call a C function here PATCH_FUNCTION( hkbLookAtModifier_2_to_3 )PATCH_END() V 59
    • Verification Previous Apply Patched Reflection Patches Reflection Patches Compare Patches Wrong/ against current reflection OK Missing V 60
    • Versioning WorkflowMake your changesRun a reflection diff utility Copy & paste the suggested patch Edit (add&remove → rename) Write version function if necessaryDone V 61
    • Versioning ConclusionMassively reduced taxQuick to add versioningCheap to testNon-prescriptive workflow 62
    • Data, Not Functions Generating code?  Stop, generate data Functions:  big and single purpose. Data:  small and multi-purpose 63
    • Conclusion github:clang-extract Reflection Serialization Memory Binding Versioning 64