This document discusses the challenges and pitfalls of implementing instrumentation using low-level hooks like run:with:in: in Smalltalk. It shows that such approaches lead to inefficient and complex instrumentation code that must handle issues like meta-recursion and concurrency. Better support is needed from the language, with safe, stratified proxies and efficient abstractions to make instrumentation easier to implement correctly. Examples demonstrate the performance overhead of different instrumentation techniques and how it can be reduced with optimizations like delayed analysis and customized inlined handlers.

1. Instrumentation &
the Pitfalls of
Abstraction
ESUG - 2023 - Lyon
* Supported by AlaMVic Action Exploratoire INRIA
guillermo.polito@inria.fr
@guillep
Evref
fervE
1
Generated by DALL-E

2. First: About Me
2
Evref
fervE
• Keywords: compilers, testing, test generation
• Ph.D.: Re
fl
ection, debloating, dynamic updates
• Interests: tooling, benchmarking, 日本語, board games, concurrency
Talk to me!
Or: guillermo.polito@inria.fr
guillermo.polito@inria.fr
@guillep

3. Building Dynamic Analyses
• Dynamic call graphs
• Code coverage
• Pro
fi
lers
• Time
• Number of calls
3

4. Method Wrappers, Objects as Methods
4

5. Remember Method Lookup
sender receiver
class
s1
s2
…
s4
…
m1
m2
m3
s2
superclass
s1 …
s5 …
inherits from
instance of
reference
message
5

6. sender receiver
class
s1
s2
…
s4
…
m1
m3
s2
superclass
s1 …
s5 …
Objects as methods
o1
inherits from
instance of
reference
message
6

7. sender receiver
class
s1
s2
…
s4
…
m1
m3
s2
superclass
s1 …
s5 …
Objects as methods + run:with:in:
o1
run:with:in:
inherits from
instance of
reference
message
7

8. How far can we get with
run:with:in: ?
8

9. A First Method Proxy
run: aSelector with: anArrayOfObjects in: aReceiver
| result |
self logBefore: aSelector.
result := self
forwardMethod: originalMethod
withReceiver: aReceiver
withArguments: anArrayOfObjects.
self logAfter: aSelector.
^ result
9

10. Let’s instrument factorial
doit 17
px
log:
fwd…
factorial
run:with:in:
10

11. doit 17
Let’s instrument factorial
px
log:
fwd…
run:with:in:
factorial
11

12. Let’s instrument factorial
doit 17
px
log:
fwd…
run:with:in:
factorial
16
factorial
…
12

13. Let’s instrument factorial
doit 17
px
log:
fwd…
run:with:in:
factorial
16
px
log:
fwd…
run:with:in:
factorial
…
13

14. Let’s get a bit more hardcore
14

15. doit
aSet
add:
px
run:with:in: log:
Instrumenting Set>>#add:
15

16. doit
aSet
add:
px
run:with:in: log:
otherSet
add:
Instrumenting Set>>#add:
16

17. Instrumenting Set>>#add:
doit
aSet
add:
px
run:with:in: log:
otherSet
add:
run:with:in:
17

18. Meta-Recursions
doit
aSet
add:
px
run:with:in: log:
otherSet
add:
run:with:in:
• The instrumentation gets instrumented!
• And, with more complex
instrumentation, more di
ffi
cult to debug
• The burden: on the developer
18

19. Solving Meta-Recursions
doit
aSet
add:
px
run:with:in: log:
otherSet
add:
19

20. Solving Meta-Recursions
doit
aSet
add:
px
run:with:in: log:
otherSet
add:
Instrumentation zone
20

21. Solving Meta-Recursions
doit
aSet
add:
px
run:with:in: log:
otherSet
add:
Instrumentation zone
21

22. And That’s not All
• Stack unwind (non-local returns, exceptions) pass around the logAfter:
• Concurrent access to our instrumentation zone?
• lose logs
• break the instrumentation
• Maybe we can do some concessions: e.g., do not proxy the proxy…
This burden, is on the developer
22

23. The Cost of Missing Abstraction
• The language gives us only low-level instrumentation hooks
• #run:with:in:
• #doesNotUnderstand:
• #cannotInterpret:
• i.e., they are at the wrong level of abstraction for proper instrumentation
Covering the GAP, is on the developer
23

24. The Proxy We Have
doit
aSet
add:
px
run:with:in: log:
otherSet
add:
Instrumentation zone
24

25. The Stratified Proxy We Want
doit
aSet
add:
px
run:with:in: log:
otherSet
add:
Instrumentation zone
handler
before:
after:
Infrastructure
• Meta-recursion
• Concurrency
User concern
• logging?
• analysis?
25

26. Stratified Proxies
26

27. Safe Method Proxies + Exact Method Profiler
• Method Proxies: https://github.com/pharo-contributions/MethodProxies
• Method Pro
fi
ler: https://github.com/pharo-contributions/MethodPro
fi
ler
• + common instrumentation layer between proxies and meta-links !
27

28. Let’s get a bit more hardcore
again
28

29. Let’s Instrument the Compiler
prf := PrfMethodProfiler new.
prf addPackage: OpalCompiler package.
prf addPackage: RBParser package.
prf profile: [ Integer recompile ].
29

30. prf := PrfMethodProfiler new.

32. ^ self < 2
ifTrue: [1]
ifFalse: [
(self-1) benchFib + (self-2) benchFib + 1]

33. Let’s Benchmark with Fibonacci
• Best case for proxy infrastructure
• no exceptions
• no non-local return
• no meta-recursion
• no concurrent usages by default
fi
b n - 1
fi
b n - 2
fi
b n - 3
fi
b n - 4
fi
b n - 4
fi
b n - 5
fi
b n - 3
fi
b n - 4
fi
b n - 5

34. Let’s Benchmark with Fibonacci (II)
• Good case to measure pro
fi
ler/proxy overhead
• Simulate a big call-tree
• Leaves are fast paths (early exits)
• => high overhead expected
• fib(n) ~~ number of messages
fi
b n - 1
fi
b n - 2
fi
b n - 3
fi
b n - 4
fi
b n - 4
fi
b n - 5
fi
b n - 3
fi
b n - 4
fi
b n - 5

35. Our Lower Bound is run:with:in:
35
run: aSelector with: anArrayOfObjects in: aReceiver
^ self
forwardMethod: originalMethod
withReceiver: aReceiver
withArguments: anArrayOfObjects

36. run:with:in: Performance vs fib(x)
• ~25x slower !
• Seems faster for lower args
• Noise due to µs measures?
36
0 10 20 30
0
5
10
15
20
25
Time
(relative
to
no
proxy)
Fibonacci Argument
* Averages of 100 runs in µs. X = 1 to: 28

37. run:with:in: Performance vs Messages
• Consistent ~25x overhead
• Cries for language
implementation
improvement (!!)
37 * Averages of 100 runs in µs.
0 300000 600000 900000 1200000
0
5
10
15
20
25
Time
(relative
to
no
proxy)
Messages

38. The Cost of Safety
• Safe method proxies are ~3000x worse
• Non-clean closures
• allocation
• thisContext rei
fi
cation
• More messages (!)
• #ensure:
• meta-recursion control
• #before, #after hooks
38
0 300000 600000 900000 1200000
0
1000
2000
3000
4000
Time
(relative
to
no
proxy)
Messages
* Averages of 100 runs in µs.

39. Can we get better?
• Down to ~400x just removing abstraction
• Inlinings (!!)
• to remove messages
• to avoid blocks
• di
ff
erentiate fast vs slow path
• concurrent, meta-recursive
39
0 300000 600000 900000 1200000
0
500
1000
1500
2000
2500
3000
3500
Time
(relative
to
no
proxy)
Messages
* Averages of 100 runs in µs.

40. Overhead of Call-Tree Construction
• 2 proxy variants
• Generic: handler object
• Customized: inlined handler
• 2 instrumentation variants
• Online: build the call tree while executing
• Delayed: trace the minimum to build it in a post-process
• Comparison Baseline: safe+opt proxy with no instrumentation
40 * Averages of 100 runs in µs.

41. Call-tree construction
• Generic + Online was o
ff
the charts :)
• => o
ff
the presentation too
• Delaying the analysis is the best
• Customization gets only slightly better
• removes 4 messages per call
41 * Averages of 100 runs in µs.
0 300000 600000 900000 1200000
0
2
4
6
8
10
12
Time
(relative
to
Generic
Online
Analysis)
Messages
Generic - No Instrumentation
Customized Proxy - Online Analysis
Generic Proxy - Delayed Analysis
Customized Proxy - Delayed Analysis

42. Call-tree construction
• Generic + Online was o
ff
the charts :)
• => o
ff
the presentation too
• Delaying the analysis is the best
• Customization gets only slightly better
• removes 4 messages per call
42 * Averages of 100 runs in µs.
0 300000 600000 900000 1200000
0
2
4
6
8
10
12
Time
(relative
to
Generic
Online
Analysis)
Messages
Generic - No Instrumentation
Customized Proxy - Online Analysis
Generic Proxy - Delayed Analysis
Customized Proxy - Delayed Analysis

43. Zooming in
• Delayed is ~1.25x proxy alone
~1.25 * 400x (safety) ~= ~500x overhead
(over no instrumentation)
43 * Averages of 100 runs in µs.
0 300000 600000 900000 1200000
1.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
Time
(relative
to
Generic
Online
Analysis) Messages
Generic - No Instrumentation
Generic Proxy - Delayed Analysis
Customized Proxy - Delayed Analysis

44. Profiling the Compiler — again
• Partial Instrumentation
• Down from ~110x to ~12x
44
Baseline
Generic+Online
Customized+Delayed
0.0
20.0
40.0
60.0
80.0
100.0
120.0
140.0
Time
(relative
to
no
instrumentation)
prf := PrfMethodProfiler new.
prf addPackage: OpalCompiler package.
prf addPackage: RBParser package.
prf profile: [ Integer recompile ].

45. Takeaways
• Users need native language support for instrumentation
• Safe, strati
fi
ed and **e
ffi
cient**
• Low-level hooks are not enough: they miss abstractions
• Think twice when writing your own proxy implementation!
• Think concurrency, think stack unwind, thing meta-recursions
45 * Supported by AlaMVic Action Exploratoire INRIA
Evref
fervE
https://github.com/pharo-contributions/MethodProxies
https://github.com/pharo-contributions/MethodPro
fi
ler