2011 ecoop

750 views
709 views

Published on

Code profilers are used to identify execution bottlenecks and understand the cause of a slowdown. Execution sampling is a monitoring technique commonly employed by code profilers because of its low impact on execution. Regularly sampling the execution of an application estimates the amount of time the interpreter, hardware or software, spent in each method execution time. Nevertheless, this execution time estimation is highly sensitive to the execution environment, making it non reproductive, non-deterministic and not comparable across platforms.
On our platform, we have observed that the number of messages sent per second remains within tight (±7%) bounds across a basket of 16 applications. Using principally the Pharo platform for experimentation, we show that such a proxy is stable, reproducible over multiple executions, profiles are comparable, even when obtained in different execution contexts. We have produced Compteur, a new code profiler that does not suffer from execution sampling limitations and have used it to extend the SUnit testing framework for execution comparison.

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
750
On SlideShare
0
From Embeds
0
Number of Embeds
87
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

2011 ecoop

  1. 1. Counting Messages as a Proxy for AverageExecution Time in Pharo ECOOP 2011 - Lancaster Alexandre Bergel Pleiad lab, DCC, University of Chile http://bergel.eu
  2. 2. The Mondrian Visualization Engine www.pharo-project.org 2
  3. 3. “I like the cool new features of Mondrian, but in my setting,drawing a canvas takes 10 seconds, whereas it took only 7yesterday. Please do something!” -- A Mondrian user, 2009 -- 3
  4. 4. “I like the cool new features of Mondrian, but in my setting,drawing my visualization takes 10 seconds, whereas it tookonly 7 yesterday. Please do something!” -- A Mondrian user, 2009 -- 4
  5. 5. Result of Pharo profiler54.8% {11501ms} MOCanvas>>drawOn: 54.8% {11501ms} MORoot(MONode)>>displayOn: 30.9% {6485ms} MONode>>displayOn: | 18.1% {3799ms} MOEdge>>displayOn: ... | 8.4% {1763ms} MOEdge>>displayOn: | | 8.0% {1679ms} MOStraightLineShape>>display:on: | | 2.6% {546ms} FormCanvas>>line:to:width:color: ... 23.4% {4911ms} MOEdge>>displayOn: ... 5
  6. 6. Yesterday version32.9% {6303ms} MOCanvas>>drawOn: 32.9% {6303ms} MORoot(MONode)>>displayOn: 24.4% {4485ms} MONode>>displayOn: | 12.5% {1899ms} MOEdge>>displayOn: ... | 4.2% {1033ms} MOEdge>>displayOn: | | 6.0% {1679ms} MOStraightLineShape>>display:on: | | 2.4% {546ms} FormCanvas>>line:to:width:color: ... 8.5% {2112ms} MOEdge>>displayOn: ... 6
  7. 7. “I like the cool new features of Mondrian, but in my setting,drawing my visualization takes 10 seconds, whereas it tookonly 7 yesterday. Please do something!” -- A Mondrian user, 2009 --On my machine I find 11 and 6 seconds. What’s going on? 7
  8. 8. How profilers workSampling the method call stack every 10 msA counter is associated to each frameEach counter is incremented when being sampled 8
  9. 9. How profilers work Sampling the method call stack every 10 ms A counter is associated to each frame Each counter is incremented when being sampled MONode displayOn: (1)method call MORoot displayOn: (1) stack Canvas drawOn: (1) Time = t 9
  10. 10. How profilers work Sampling the method call stack every 10 ms A counter is associated to each frame Each counter is incremented when being sampled MOEdge displayOn: (1) MONode displayOn: (2)method call MORoot displayOn: (2) stack Canvas drawOn: (2) Time = t + 10 ms 10
  11. 11. How profilers work Sampling the method call stack every 10 ms A counter is associated to each frame Each counter is incremented when being sampled MONode setCache (1)method call MORoot displayOn: (3) stack Canvas drawOn: (3) Time = t + 20 ms 11
  12. 12. How profilers work The counter is used to estimate the amount of timespent MONode setCache (1) MOEdge displayOn: (1) MONode displayOn: (2) MORoot displayOn: (3) Canvas drawOn: (3) 12
  13. 13. How profilers work The counter is used to estimate the amount of timespent MONode setCache (1) => 10 ms MOEdge displayOn: (1) => 10 ms MONode displayOn: (2) => 20 ms MORoot displayOn: (3) => 30 ms Canvas drawOn: (3) => 30 ms 13
  14. 14. Problem with execution sampling #1Strongly dependent on the executing environment CPU, memory management, threads, virtual machine, processesListening at a mp3 may perturb your profile 14
  15. 15. Problem with execution sampling #2Non-determinism Even using the same environment does not help“30000 factorial” takes between 3 803 and 3 869 ms 15
  16. 16. Problem with execution sampling #3 Lack of portability Profiles are not reusable across platform Buying a new laptop will invalidate the profile youmade yesterday 16
  17. 17. Counting messages to the rescue Pharo is a Smalltalk dialect Intensively based on sending message Almost “Optimization-free compiler” Why not to count messages instead of executiontime? 17
  18. 18. Counting messagesWallet >> increaseByOne money := money + 1Wallet >> addBonus self increaseByOne; increaseByOne; increaseByOne.aWallet addBonus=> 6 messages sent 18
  19. 19. Does this really work? What about the program?MyClass >> main self waitForUserClick We took scenarios from unit tests, which do not relyon user input 19
  20. 20. Experiment A 6 400 x 10 message sends 400000000 6 application 300 x 10 300000000 6 200 x 10 200000000 6 100 x 10 100000000 0 0 10000 20000 30000 40000 times (ms) The number of sent messages related to the averageexecution time over multiple executions 20
  21. 21. Experiment B Application time taken (ms) # sent messages ctime % cmessages % Collections 32 317 334 359 691 16.67 1.05 Mondrian 33 719 292 140 717 5.54 1.44 Nile 29 264 236 817 521 7.24 0.22 Moose 25 021 210 384 157 24.56 2.47 SmallDude 13 942 150 301 007 23.93 0.99 Glamour 10 216 94 604 363 3.77 0.14 Magritte 2 485 37 979 149 2.08 0.85 PetitParser 1 642 31 574 383 46.99 0.52 Famix 1 014 6 385 091 18.30 0.06 DSM 4 012 5 954 759 25.71 0.17 ProfStef 247 3 381 429 0.77 0.10 Network 128 2 340 805 6.06 0.44 AST 37 677 439 1.26 0.46 XMLParser 36 675 205 32.94 0.46 Arki 30 609 633 1.44 0.35 ShoutTests 19 282 313 5.98 0.11 Average 13.95 0.61Table 2.number of sent messages more stable third columns The Applications considered in our experiment (second and than theare average overtime over multiple executions execution 10 runs) 21
  22. 22. Experiment C 6 number of method 10000000 10.0 x 10 invocations 6 method 7500000 7.5 x 10 6 5.0 x 10 5000000 6 2.5 x 10 2500000 0 0 75 150 225 300 time (ms) The number of sent messages as useful as theexecution time to identify an execution bottleneck 22
  23. 23. CompteurCompteurMethod>> run: methodName with: args in: receiver | oldNumberOfCalls v | oldNumberOfCalls := self getNumberOfCalls. v := originalMethod valueWithReceiver: receiver arguments: args. numberOfCalls := (self getNumberOfCalls - oldNumberOfCalls) + numberOfCalls - 5. ˆv 23
  24. 24. New primitive in the VMCompteurMethod>> run: methodName with: args in: receiver | oldNumberOfCalls v | oldNumberOfCalls := self getNumberOfCalls. v := originalMethod valueWithReceiver: receiver arguments: args. numberOfCalls := (self getNumberOfCalls - oldNumberOfCalls) + numberOfCalls - 5. ˆv 24
  25. 25. Cost of the instrumentationOverhead (%) Overhead (%) 3000 10000 2250 1000 1500 100 750 10 0 1 0 10000 20000 30000 40000 0 10000 20000 30000 40000 Execution time (ms) Execution time (ms) (a) Linear scale (b) Logarithmic scale 25
  26. 26. Contrasting Execution Sampling with Message CountingNo need for samplingIndependent from the execution environmentStable measurements 26
  27. 27. Application #1 Counting messages in unit testingCollectionTest>>testInsertion self assert: [ Set new add: 1] fasterThan: [Set new add: 1; add: 2] 27
  28. 28. Application #1 Counting messages in unit testingMondrianSpeedTest>> testLayout2 | view1 view2 | view1 := MOViewRenderer new. view1 nodes: (Collection allSubclasses). view1 edgesFrom: #superclass. view1 treeLayout. view2 := MOViewRenderer new. view2 nodes: (Collection withAllSubclasses). view2 edgesFrom: #superclass. view2 treeLayout. self assertIs: [ view1 root applyLayout ] fasterThan: [ view2 root applyLayout ] 28
  29. 29. Application #2 Differencing profiling Comparison of two successive versions of a software(not in the paper) 29
  30. 30. Application #2 Differencing profiling Comparison of two successive versions of Mondrian(not in the paper) 30
  31. 31. More in the paper Linear regression model We replay some optimizations we had in our previouswork A methodology to evaluate profiler stability overmultiple run All the material to reproduce the experiments 31
  32. 32. Summary Counting method invocation is a more advantageousprofiling technique, in Pharo Stable correlation between message sending andaverage execution time 32
  33. 33. Closing words The same abstractions are used to profileapplications written in C and in Java Which objects is responsible of a slowdown? Which arguments make a method call slow? ... 33
  34. 34. 6 number of method 6 10000000 10.0 x 10 400 x 10 invocations message sends 400000000 6 6 7500000 7.5 x 10 300 x 10 300000000 6 6 200 x 10 200000000 5.0 x 10 5000000 6 6 100 x 10 100000000 2.5 x 10 2500000 0 0 10000 20000 30000 40000 0 0 75 150 225 300 times (ms) time (ms) Counting message as a proxy for average execution time Alexandre Bergel http://bergel.eu Overhead (%) Overhead (%) 3000 10000CollectionTest>>testInsertion 2250 1000self assert: [Set new 1500 1] add: 100 fasterThan: [Set new add: 1; add: 2] 750 10 0 1 0 10000 20000 30000 34 40000 0 10000 20000 30000 40000 Execution time (ms) Execution time (ms)

×