a JIT Smalltalk VM
        written in itself




 Javier Burroni        gera
                      (richie)
ESUG 2009 – JIT Security


                                       @1: 7F647D:   push EBP
test: arg                        ...
Closure Bug (6-Aug-2010)
testThomasMuhrNestedBlocks
  | abcdef |
  abcdef := [| var1 |
    var1 := 'AB'.
    [| var2 |    ...
Closure Bug (6-Aug-2010)
testThomasMuhrNestedBlocks
  | abcdef |
  abcdef := [| var1 |
    var1 := 'AB'.
    [| var2 |    ...
Closure Bug (6-Aug-2010)
testThomasMuhrNestedBlocks
  | abcdef |
  abcdef := [| var1 |
    var1 := 'AB'.
    [| var2 |    ...
Smalltalks 2009 – code freezer
GenerationalGCBuilder >> #populateInternals
(nativizer := BigNativizer new)
    addAllMetho...
Frozen Code: “Lookup”




Object >> #isNil                      UndefinedObject >> #isNil
    self class == Object        ...
Frozen Code: “Lookup”
nativizer
    at: #isNil addAll: (Array with: Object with: UndefinedObject)




  Object >> #isNil  ...
Frozen Code: “Lookup”
nativizer
    at: #isNil addAll: (Array with: Object with: UndefinedObject);
    at: #isNil default:...
Frozen Code: “Lookup”
    nativizer
        at: #isNil addAll: (Array with: Object with: UndefinedObject);
        at: #is...
New Goal


Smalltalk VM
New Goal


Smalltalk VM
  Written in Smalltalk
New Goal


Smalltalk VM
  Written in Smalltalk
     Compatible with Digitalk's (VS)
New Goal


Smalltalk VM
  Written in Smalltalk
     Compatible with Digitalk's (VS)
         Leveraging the environment
Smalltalk VM

Written in Smalltalk
 .exe generation
Method lookup
Object format
Memory management
 Object creation / copy
...
PE (.exe) generation
PEFile
                                 _commit: newSize
                                  assembler
...
GCSpace
Object subclass: #GCSpace             old
 InstanceVariableNames: '
   contents
   base
   nextFree               ...
GCSpace
                                              old includes: anObject
Object subclass: #GCSpace             old
 In...
GenerationalGC
Object subclass: #GenerationalGC   old
 InstanceVariableNames: '
   old from to
   roots literalsReferences...
GenerationalGC
                                          flipper purgeRoots
Object subclass: #GenerationalGC   old
 Instan...
GenerationalGC
                                        collect
GenerationalGC old     from    to         self
            ...
GenerationalGC
                                        collect
GenerationalGC old     from    to         self
            ...
GenerationalGC
                                        collect
GenerationalGC old     from    to         self
            ...
GenerationalGC
                                        collect
GenerationalGC old     from    to         self
            ...
GenerationalGC
                                     collect
GenerationalGC old   from    to        self
                  ...
GenerationalGC
                                    collect
GenerationalGC old     from   to      self
                    ...
GenerationalGC
                                     collect
GenerationalGC old    from    to       self
                  ...
GenerationalGC
                                 collect
GenerationalGC old   from   to     self
                          ...
GenerationalGC
                                  collect
GenerationalGC old   from   to      self
                        ...
GenerationalGC: testing
                                               collect
GenerationalGC old        from      to     ...
GenerationalGC: testing
testFollowObject
  | flipper from array string byteArray root flipSize currentSize |
  flipper := ...
a demo




VMBuilder installGenerational
What now?

Release OpenSource
 Education!!!

Closure for JIT

Memory Management
 Mark & Sweep
 new
 become:

Method Lookup...
¡gracias!
Upcoming SlideShare
Loading in...5
×

A JIT Smalltalk VM written in itself

1,063

Published on

A JIT Smalltalk VM written in itself. ESUG 2010, Barcelona, Spain

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,063
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

A JIT Smalltalk VM written in itself

  1. 1. a JIT Smalltalk VM written in itself Javier Burroni gera (richie)
  2. 2. ESUG 2009 – JIT Security @1: 7F647D: push EBP test: arg 7F647E: mov EBP, ESP ^self with: 0 with: inst with: arg 7F6480: push EAX 7F6481: mov ESI, EAX ... @2:7F6492: push 1 ; PushSmallInteger0 1 <13> PushSmallInteger0 7F6497: push [ESI] ; PushInstance1 7F6499: push [EBP+8] ; PushArgument1 2 <8C> PushInstance1 7F649C: mov EAX, ESI ; LoadSelf 3 <55> PushArgument1 7F649E: call 740207 ; SendSelector1 4 <0A> LoadSelf 7F64A3: mov ESP, EBP ; Return 5 <E2> SendSelector1 7F64A5: pop EBP 7F64A6: mov ESI, [EBP-4] 6 <48> Return 7F64A9: ret 4 NEAR BytecodeNativizerPushR Assembler >> #pushR assembler pushR self assembleByte: 16r50 " push eax "
  3. 3. Closure Bug (6-Aug-2010) testThomasMuhrNestedBlocks | abcdef | abcdef := [| var1 | var1 := 'AB'. [| var2 | BOOM! var2 := 'CD'. [| var3 | var3 := 'EF'. [var1 , var2 , var3] value] value] value] value. self assert: abcdef = 'ABCDEF' Fix #1 (7-Aug-2010) IndirectEscapeBytecode >> #assemble type := self nextByte. argument := type 4 = 0 ifTrue: [self nextIndex nextIndex] nextIndex + 1 ifFalse: [type 4 + 3]. type < 5 ifTrue: [^self loadEnvironmentTemporaryIndirect]. type < 9 ifTrue: [^self pushEnvironmentTemporaryIndirect]. ^self storeEnvironmentTemporaryIndirect
  4. 4. Closure Bug (6-Aug-2010) testThomasMuhrNestedBlocks | abcdef | abcdef := [| var1 | var1 := 'AB'. [| var2 | BOOM! var2 := 'CD'. [| var3 | var3 := 'EF'. [var1 , var2 , var3] value] value] value] value. self assert: abcdef = 'ABCDEF' Fix #3 (9-Aug-2010) loc_1000ACFA: push esi call JIT_getNextIndex mov edi, [esp+10h+aBytecodeNativizer] add esp, 4 dec eax nop mov [edi+4], eax jmp short loc_1000AD29
  5. 5. Closure Bug (6-Aug-2010) testThomasMuhrNestedBlocks | abcdef | abcdef := [| var1 | var1 := 'AB'. [| var2 | BOOM! var2 := 'CD'. [| var3 | var3 := 'EF'. [var1 , var2 , var3] value] value] value] value. self assert: abcdef = 'ABCDEF' Fix #2 (9-Aug-2010) putCode: code min: min max: max range: idx (idx >= min and: [idx <= max]) ifTrue: [self putNext: code - max + idx - 1] ifFalse: [ self putNext: code. self putIndex: idx idx] Idx + 1
  6. 6. Smalltalks 2009 – code freezer GenerationalGCBuilder >> #populateInternals (nativizer := BigNativizer new) addAllMethodsFor: GCSpace; addAllMethodsFor: ObjectGenerationalGC; at: #isSmallInteger default: Object >> #isSmallInteger; at: #isSmallInteger add: SmallInteger; at: #bitXor: add: SmallInteger; at: #between:and: add: SmallInteger; at: #_generation default: ExtensionBytecode class >> #_generation; at: #_commit: _primitive: ExtensionBytecode >> #_commit; ...
  7. 7. Frozen Code: “Lookup” Object >> #isNil UndefinedObject >> #isNil self class == Object self class == UndefinedObject IfFalse: [ IfFalse: [ ^self lookupAndSend] ^self lookupAndSend] ^false ^true
  8. 8. Frozen Code: “Lookup” nativizer at: #isNil addAll: (Array with: Object with: UndefinedObject) Object >> #isNil UndefinedObject >> #isNil self class == Object self class == UndefinedObject IfFalse: [ ^self lookupAndSend] ^false ^true
  9. 9. Frozen Code: “Lookup” nativizer at: #isNil addAll: (Array with: Object with: UndefinedObject); at: #isNil default: Object >> #isNil Object >> #isNil UndefinedObject >> #isNil self class == Object self class == UndefinedObject ^false ^false ^true
  10. 10. Frozen Code: “Lookup” nativizer at: #isNil addAll: (Array with: Object with: UndefinedObject); at: #isNil default: Object >> #isNil; at: #isNil add: SmallInteger self _isSmallInteger ^false Object >> #isNil UndefinedObject >> #isNil self class == Object self class == UndefinedObject ^false ^false ^true
  11. 11. New Goal Smalltalk VM
  12. 12. New Goal Smalltalk VM Written in Smalltalk
  13. 13. New Goal Smalltalk VM Written in Smalltalk Compatible with Digitalk's (VS)
  14. 14. New Goal Smalltalk VM Written in Smalltalk Compatible with Digitalk's (VS) Leveraging the environment
  15. 15. Smalltalk VM Written in Smalltalk .exe generation Method lookup Object format Memory management Object creation / copy GC Become Primitives FFI Processes Callbacks etc
  16. 16. PE (.exe) generation PEFile _commit: newSize assembler directories imports loadArgFromContext: index; exports convertArgToNative; pushArg; relocations convertToNative; pushR; code pushConstant: 4; pushConstant: 4096; sections exports pushArg; pushR; imports callTo: 'VirtualAlloc' from: 'KERNEL32.DLL'; convertToSmalltalk code nativizer addAllMethodsFor: (Array with: GenerationalGC with: GenerationalGC class with: GCSpace); at: #_commit: _primitive: ExtensionBytecode >> #_commit
  17. 17. GCSpace Object subclass: #GCSpace old InstanceVariableNames: ' contents base nextFree A commitLimit flip spaces reservedLimit ' classVariableNames: '' B poolDictionaries: '' includes: anObject ^anObject isSmallInteger not and: [anObject _oop between: self base and: self commitedLimit] reset self nextFree: self base
  18. 18. GCSpace old includes: anObject Object subclass: #GCSpace old InstanceVariableNames: ' to reset contents base from isReferredBy: object nextFree A commitLimit old growTo: newLimit reservedLimit ' classVariableNames: '' B moved := to shallowCopy: object poolDictionaries: '' from flipWith: to includes: anObject ^anObject isSmallInteger not and: [anObject _oop between: self base and: self commitedLimit] reset self nextFree: self base
  19. 19. GenerationalGC Object subclass: #GenerationalGC old InstanceVariableNames: ' old from to roots literalsReferences from weakContainers ephemerons rescuedEphemerons ' classVariableNames: '' poolDictionaries: '' to collect self followRoots; walkStack; rescueEphemerons; traverseWeakContainers; flipSpaces
  20. 20. GenerationalGC flipper purgeRoots Object subclass: #GenerationalGC old InstanceVariableNames: ' flipper follow: object old from to roots literalsReferences flipper moveToOldOrTo: object from weakContainers ephemerons rescuedEphemerons ' flipper classVariableNames: '' fixReferencesOrSetTombstone: poolDictionaries: '' to weakContainer flipper addInterrupt collect self followRoots; walkStack; rescueEphemerons; traverseWeakContainers; flipSpaces
  21. 21. GenerationalGC collect GenerationalGC old from to self followRoots; walkStack; follow: object | size | size := 0. object _isBytes ifFalse: [ object _hasWeaks ifTrue: [self addWeakOrEphemeron: object] ifFalse: [size := object _size]]. self follow: object count: 1 + size startingAt: 0
  22. 22. GenerationalGC collect GenerationalGC old from to self followRoots; walkStack; follow: objects count: size startingAt: base | index | index := base - 1. size timesRepeat: [| object | index := index + 1. object := objects _basicAt: index. (fromSpace includes: object) ifTrue: [ object _isProxy ifTrue: [objects _basicAt: index put: object _proxee] ifFalse: [| moved | moved := self moveToOldOrTo: object. objects _basicAt: index put: moved. self follow: moved]]
  23. 23. GenerationalGC collect GenerationalGC old from to self followRoots; walkStack; follow: objects count: size startingAt: base | index | index := base - 1. size timesRepeat: [| object | index := index + 1. object := objects _basicAt: index. (fromSpace includes: object) ifTrue: [ object _isProxy ifTrue: [objects _basicAt: index put: object _proxee] ifFalse: [| moved | moved := self moveToOldOrTo: object. objects _basicAt: index put: moved. self follow: moved]]]
  24. 24. GenerationalGC collect GenerationalGC old from to self traverseWeakContainers traverseWeakContainers weakContainers do: [:weakContainer | self fixReferencesOrSetTombstone: weakContainer] self reset: weakContainers fixReferencesOrSetTombstone: weakContainer | size | size := weakContainer _size. 1 to: size do: [:index | | instance | instance := weakContainer _basicAt: index. (instance isSmallInteger not and: [fromSpace includes: instance]) ifTrue: [| referenceOrThombstone | referenceOrThombstone := instance _isProxy ifTrue: [instance _proxee] ifFalse: [residueObject]. weakContainer _basicAt: index put: referenceOrThombstone]]
  25. 25. GenerationalGC collect GenerationalGC old from to self rescueEphemerons rescueEphemerons | unknowns rescan | unknowns := self localStack. rescan := false. [ephemerons isEmpty] whileFalse: [ rescan := self followEphemeronsCollectingIn: unknowns. rescan ifTrue: [ephemerons addAll: unknowns] IfFalse: [ unknowns do: [:ephemeron | self rescueEphemeron: ephemeron]]. self reset: unknowns] [Hayes1997] Barry Hayes, Ephemerons: a new finalization mechanism
  26. 26. GenerationalGC collect GenerationalGC old from to self flipSpaces flipSpaces fromSpace flipWith: toSpace. toSpace reset. GCSpace >> flipWith: space | aux | aux := space base. space base: self base. self base: aux. ...
  27. 27. GenerationalGC collect GenerationalGC old from to self flipSpaces A B nextFree := base + size nextFree := base copy contents A contents B
  28. 28. GenerationalGC collect GenerationalGC old from to self flipSpaces A B contents A contents B
  29. 29. GenerationalGC collect GenerationalGC old from to self flipSpaces A B contents A contents B VM assumption: from.commitLimit < to.base
  30. 30. GenerationalGC: testing collect GenerationalGC old from to self followRoots testFollowObject | flipper from array string byteArray root flipSize currentSize | flipper := self defaultFlipper. from := flipper fromSpace. array := from shallowCopy: (Array new: 3). string := from shallowCopy: 'a String' copy. from shallowCopy: 'leaked' copy. byteArray := from shallowCopy: #[1 2 3] copy. array at: 1 put: 1; at: 2 put: string; at: 3 put: byteArray. root := Array with: array. string := byteArray := array := nil. flipper addRoot: root; followRoots. flipSize := flipper toSpace nextFree - flipper toSpace base. currentSize := flipper fromSpace nextFree - flipper fromSpace base. self assert: flipSize < currentSize; assert: currentSize - flipSize = (8 + ('leaked' size roundTo: 4))
  31. 31. GenerationalGC: testing testFollowObject | flipper from array string byteArray root flipSize currentSize | flipper := self defaultFlipper. from := flipper fromSpace. array := from shallowCopy: (Array new: 3). string := from shallowCopy: 'a String' copy. from shallowCopy: 'leaked' copy. byteArray := from shallowCopy: #[1 2 3] copy. testFollowObject | flipper from array string byteArray root flipSize currentSize | flipper := self defaultFlipper. self execute: [:from | array := from shallowCopy: (Array new: 3). string := from shallowCopy: 'a String' copy. from shallowCopy: 'leaked' copy. byteArray := from shallowCopy: #[1 2 3] copy] proxying: flipper fromSpace.
  32. 32. a demo VMBuilder installGenerational
  33. 33. What now? Release OpenSource Education!!! Closure for JIT Memory Management Mark & Sweep new become: Method Lookup (strategies) etc.
  34. 34. ¡gracias!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×