Q2.12: Benchmarking Techniques
Upcoming SlideShare
Loading in...5
×
 

Q2.12: Benchmarking Techniques

on

  • 187 views

Resource: Q2.12

Resource: Q2.12
Name: Benchmarking Techniques
Date: 30-05-2012
Speaker: Michael Hope

Statistics

Views

Total Views
187
Views on SlideShare
187
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Q2.12: Benchmarking Techniques Q2.12: Benchmarking Techniques Presentation Transcript

  • Benchmarking Techniques Michael Hope <michael.hope@linaro.org> bzr branch lp:~michaelh1/+junk/benchmarking-techniques r6
  • 2 Why benchmark?
  • 3 Issues are Relevance Accuracy Repeatability
  • 4 Picking relevant benchmarks: Profile Workload Features We use SPEC 2000 and EEMBC We'd like shareable benchmarks
  • 5 Test Platform Build / test / benchmark via Linux / web / commodity hardwar
  • 6 Measuring
  • 7 Realtime timers See 'man clock_gettime'
  • 8 <comparison of different timers> See https://wiki.linaro.org/WorkingGroups/ToolChain/Benchmarks/TimerAc
  • 9
  • 10
  • 11 External influences
  • 12 Other features to be wary of scheduler governor cpuidle, power management, thermal limiting SMP bugs, like core lockdown NEON startup See “Understanding the Linux Kernel” ch10: http://oreilly.com/catalog/linuxkernel/chapter/ch10.html
  • 13 Putting things together and running We use: timer built into the app run five times collect everything post process
  • 14 Statistics
  • 15
  • 16
  • 17 Standard deviation Dispersion / coefficient of variance t-scores t-test
  • 18 t= ̄X 1 − ̄X 2 √ s1 2 N 1 + s2 2 N 2 See http://en.wikipedia.org/wiki/Welch%27s_t_
  • 19 Compiler Mean Std CV gcc-4.6.2 1.00 134u 134u gcc-linaro-4.6-2011.11 4.25 5035u 1178u t = 30,300
  • 21 Variant Mean Std CV Plain 1.00 320u 320u With SMS 1.01 430u 426u t = 118
  • 22 Variant Mean Std CV Plain 1.00 320u 320u With vectoriser 1.001 404u 404u t = 8.27 - significant
  • 23 Our tools perf difftest betters tabulate Python LibreOffice! Other statistical tools like scipy.stat, R, Judge, and ministat
  • 24 http://help.libreoffice.org/Calc/Applying_AutoFilter
  • 25 www.linaro.org / wiki.linaro.org people.linaro.org/~michaelh/presentations