Benchmarking

1,354 views
1,310 views

Published on

Bandwidth
History of the Internet
Midterm Questions
How to tell if its a hardware or software problem?
What makes the fan go?
Benchmarking
Cheating on Benchmarks
Web Benchmarks
Apache is very good!

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,354
On SlideShare
0
From Embeds
0
Number of Embeds
971
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Benchmarking

  1. 1. Plan for Today • Finish up Internet (from last class) • Midterm Discussion • Benchmarking 17 October 2013 University of Virginia cs4414 1
  2. 2. Measuring Networks Latency Time from sending a bit until it arrives seconds (or seconds per geographic distance) Bandwidth Rate at which can you transmit bits per second 17 October 2013 University of Virginia cs4414 2
  3. 3. $ traceroute -q 1 -w 30 www.busselton.wa.gov.au traceroute to busselton.wa.gov.au (203.41.180.233), 64 hops max, 52 byte packets 1 dd-wrt (192.168.1.1) 11.156 ms 2 c-24-127-51-1.hsd1.va.comcast.net (24.127.51.1) 32.497 ms 3 te-2-3-ur01.charlville.va.richmond.comcast.net (68.85.226.149) 13.971 ms 4 xe-11-2-0-0-sur01.charlville.va.richmond.comcast.net (69.139.165.221) 12.312 ms 5 xe-4-1-2-0-ar02.charlvilleco.va.richmond.comcast.net (69.139.165.65) 12.395 ms 6 pos-1-2-0-0-cr01.ashburn.va.ibone.comcast.net (68.86.91.53) 25.624 ms 7 pos-3-10-0-0-cr01.56marietta.ga.ibone.comcast.net (68.86.86.221) 31.483 ms 8 pos-1-9-0-0-cr01.dallas.tx.ibone.comcast.net (68.86.87.233) 52.515 ms 9 he-0-12-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.117) 83.242 ms 10 as4637-cr01.losangeles.ca.ibone.comcast.net (75.149.228.222) 78.134 ms 11 i-0-2-0-11.tlot-core01.bi.telstraglobal.net (202.40.149.185) 86.131 ms 12 i-0-0-0-0.sydo-core01.bx.telstraglobal.net (202.84.140.5) 287.302 ms 13 tengige0-1-0-14.oxf-gw2.sydney.telstra.net (203.50.13.133) 300.060 ms 14 bundle-ether2.oxf-gw1.sydney.telstra.net (203.50.6.85) 274.270 ms Does traceroute tell us anything 270.694 the 15 bundle-ether1.ken-core4.sydney.telstra.net (203.50.6.5)about ms 16 bundle-ether10.win-core1.melbourne.telstra.net (203.50.11.13) 275.252 ms bandwidth between here an Busselton, Austrailia? 17 bundle-ether6.fli-core1.adelaide.telstra.net (203.50.11.90) 405.600 ms 18 bundle-ether5.wel-core3.perth.telstra.net (203.50.11.19) 411.510 ms 19 gigabitethernet0-1.wel13.perth.telstra.net (203.50.115.151) 406.044 ms 20 * 17 October 2013 University of Virginia cs4414 3
  4. 4. What Impacts Bandwidth? IPv4: up to 99.96% efficient LAN: Ethernet (97.6% efficient for 12Kb packets) WAN: PPP (99.9% efficient – only 1-2 bytes overhead) 17 October 2013 University of Virginia cs4414 4
  5. 5. Below all this: Physical Layer 17 October 2013 University of Virginia cs4414 5
  6. 6. Bandwidth How much data can you transfer in a given amount of time? 17 October 2013 University of Virginia cs4414 6
  7. 7. Improving Bandwidth • Faster transmission – Train signalers to move semaphore flags faster – Use something less physically demanding to transmit • Bigger pipes – Have multiple signalers transmit every other letter at the same time • Better encoding – Figure out how to code more than 98 symbols with semaphore signal – Morse code (1840s) 17 October 2013 University of Virginia cs4414 7
  8. 8. Morse Code Represent letters with series of short and long electrical pulses 17 October 2013 University of Virginia cs4414 8
  9. 9. Circuit Switching • Reserve a whole path through the network for the whole message transmission Paris Bourges Nantes 17 October 2013 Lyon Toulon Once you start a transmission, know you will have use of the network until it is finished. But, wastes network resources. University of Virginia cs4414 9
  10. 10. Packet Switching • Use one link at a time Paris Lyon Bourges Toulon Interleave messages – send whenever the next link is free. Nantes 17 October 2013 University of Virginia cs4414 10
  11. 11. internetwork A collection of multiple networks connected together, so messages can be transmitted between nodes on different networks. 17 October 2013 University of Virginia cs4414 11
  12. 12. The First (international) internet 1800: Edelcrantz links Sweden and Denmark telegraph networks to coordinate defense 17 October 2013 University of Virginia cs4414 12
  13. 13. End of First (international) internet 1801: British attach Copenhagen; Sweden doesn’t help Denmark; network disconnected 17 October 2013 University of Virginia cs4414 13
  14. 14. The (capital-I) Internet Packet Switching: Leonard Kleinrock (UCLA) thinks he did, Donald Davies and Paul Baran, Edelcrantz’s signalling network (1809) Internet Protocol: Vint Cerf, Bob Kahn Vision, Funding (DARPA): J.C.R. Licklider, Bob Taylor Government: Al Gore First politician to promote Internet, 1986; act to connect government networks to form “Interagency Network” 17 October 2013 Vint Cerf (in the Rotunda, 2010) University of Virginia cs4414 14
  15. 15. First Use of the Internet October 1969: First packets on the ARPANet from UCLA to Stanford. Starts to send "LOGIN", but it crashes on the G. How impressive is this compared to communications event 3 months earlier? 17 October 2013 University of Virginia cs4414 15
  16. 16. 20 July 1969: Live b/w video from the moon, transmitted live to millions of televisions worldwide 17 October 2013 University of Virginia cs4414 16
  17. 17. Today’s Internet: Bandwidth Ethernet: up to 100 Mbits/sec My office: ~50 M bits / sec UVa Wireless: 57 M bits / sec Cable modem at home: ~ 30 M bits / sec 17 October 2013 University of Virginia cs4414 17
  18. 18. 1 Petabyte = 1015 bytes 1 Million petabytes = 1 zettabyte (NSA’s Utah facility is storing “Zbytes”?) projections 17 October 2013 University of Virginia cs4414 18
  19. 19. Midterm Discussion 17 October 2013 University of Virginia cs4414 19
  20. 20. Easiest Way to Ace Midterm 4 people selected #9, but none were bold enough to leave all the other answers blank. (You should only be “especially proud of” your answer to #9 if you are confident enough to leave all other answers blank.) 17 October 2013 University of Virginia cs4414 20
  21. 21. How many processes should a web browser create? University of Virginia cs4414 several per “tab” one per “tab” several process 1 process 17 October 2013 21
  22. 22. How many threads should a web browser create? University of Virginia cs4414 several per “tab” one per “tab” several process 1 process 17 October 2013 22
  23. 23. How many processes should a web browser developed in Rust create? University of Virginia cs4414 several per “tab” one per “tab” several process 1 process 17 October 2013 23
  24. 24. How many tasks should a web browser developed in Rust create? University of Virginia cs4414 several per “tab” one per “tab” several process 1 process 17 October 2013 24
  25. 25. Your technically unsophisticated (but smart, rich, and curious) uncle complains that his laptop is too noisy since the fan is running all the time. He expects that as someone taking upperlevel computing courses at an elite, public university, you should be able to help him. What do you do? 17 October 2013 University of Virginia cs4414 25
  26. 26. Hardware problem or software problem? 17 October 2013 University of Virginia cs4414 26
  27. 27. Hardware problem or software problem? 17 October 2013 University of Virginia cs4414 27
  28. 28. SOSP 2009 17 October 2013 University of Virginia cs4414 28
  29. 29. Uncle’s Fan Problems Could it be caused by deadlock? 17 October 2013 University of Virginia cs4414 29
  30. 30. Fan Removes Heat • CPU generates heat when it is executing lots of instructions • At least with my MBP, one thread is not enough: 100% CPU usage – no fan 200% CPU usage – light fan 800% CPU usage – max fan 17 October 2013 University of Virginia cs4414 30
  31. 31. Performance Benchmarking 17 October 2013 University of Virginia cs4414 31
  32. 32. Why Benchmark? 17 October 2013 University of Virginia cs4414 32
  33. 33. Developer Benchmarks • Find bottlenecks: know what to spend time optimizing • Measure impact of changes to optimize performance • Predict what resources you will need to scale service (“healthcare.gov”) Goal is a benchmark that represents well the (anticipated) actual usage 17 October 2013 University of Virginia cs4414 33
  34. 34. Micro vs. Macro Benchmarks • Micro Benchmarks – Test one specific operation 17 October 2013 University of Virginia cs4414 34
  35. 35. “Customer” Benchmarks • Allow fair and accurate comparisons between different solutions • Standard accepted by all/many vendors Goal is a benchmark that represents well the (anticipated) actual usage for “typical” users 17 October 2013 University of Virginia cs4414 35
  36. 36. “Cheating” on Benchmarks How could you “cheat” on a benchmark? 17 October 2013 University of Virginia cs4414 36
  37. 37. 1. On the Exynos 5410, Samsung was detecting the presence of certain benchmarks and raising thermal limits (and thus max GPU frequency) in order to gain an edge on those benchmarks, and 2. On both Snapdragon 600 and Exynos 5410 SGS4 platforms, Samsung was detecting the presence of certain benchmarks and automatically driving CPU voltage/frequency to their highest state right away. Also on Snapdragon platforms, all cores are plugged in immediately upon benchmark detect. 17 October 2013 University of Virginia cs4414 37
  38. 38. Everyone (except Google and NVidia) is cheating! Samsung is just better at it than anyone else. 17 October 2013 University of Virginia cs4414 38
  39. 39. Is It Possible to Prevent Benchmark “Cheating”? 17 October 2013 University of Virginia cs4414 39
  40. 40. Benchmarking Zhtta 17 October 2013 University of Virginia cs4414 40
  41. 41. http://www.comlore.com/redist_files/httperf-www2007-slides.pdf 17 October 2013 University of Virginia cs4414 41
  42. 42. Example httperf --hog --server=localhost --port=4414 --num-conns=1000 --rate=10 10 requests per second Should you run httperf on the machine running your server? 17 October 2013 University of Virginia cs4414 42
  43. 43. Maximum connect burst length: 1 Total: connections 1000 requests 0 replies 0 test-duration 99.901 s Connection rate: 10.0 conn/s (99.9 ms/conn, <=1 concurrent connections) Connection time [ms]: min 0.0 avg 0.0 max 0.0 median 0.0 stddev 0.0 Connection time [ms]: connect 0.1 Connection length [replies/conn]: 0.000 Request rate: 0.0 req/s (0.0 ms/req) Request size [B]: 0.0 Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (19 samples) Reply time [ms]: response 0.0 transfer 0.0 Reply size [B]: header 0.0 content 0.0 footer 0.0 (total 0.0) Reply status: 1xx=0 2xx=0 3xx=0 4xx=0 5xx=0 CPU time [s]: user 27.52 system 72.40 (user 27.6% system 72.5% total 100.0%) Net I/O: 0.0 KB/s (0.0*10^6 bps) Errors: total 1000 client-timo 0 socket-timo 0 connrefused 1000 connreset 0 Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0 17 October 2013 University of Virginia cs4414 43
  44. 44. Macrobenchmarks Need a workload that represents typical load 17 October 2013 University of Virginia cs4414 44
  45. 45. http://ita.ee.lbl.gov/html/traces.html 17 October 2013 University of Virginia cs4414 45
  46. 46. http://www.hpl.hp.com/techreports/1999/HPL-1999-35R1.pdf 17 October 2013 University of Virginia cs4414 46
  47. 47. 17 October 2013 University of Virginia cs4414 47
  48. 48. 1998 17 October 2013 University of Virginia cs4414 Image from http://www.hpl.hp.com/techreports/1999/HPL-1999-35R1.pdf 48
  49. 49. Macrobenchmarks Need a workload that represents typical load Synthetic Workload 17 October 2013 Replayed Workload University of Virginia cs4414 49
  50. 50. Weilin’s Test Workload > dd if=/dev/urandom of=5K.bin bs=5K count=1 > dd if=/dev/urandom of=5M.bin bs=5M count=1 > dd if=/dev/urandom of=10M.bin bs=10M /5K.bin count=1 > dd if=/dev/urandom of=20M.bin bs=20M /index.htm count=1 > dd if=/dev/urandom of=40M.bin bs=40M /5K.bin count=1 /5K.bin > dd if=/dev/urandom of=80M.bin bs=80M /80M.bin count=1 > dd if=/dev/urandom of=512M.bin bs=512M count=1 /5K.bin http://www.cs.virginia.edu/~wx4ed/cs4414/ ps3/zhtta-test-urls.txt 17 October 2013 University of Virginia cs4414 /5K.bin /5K.bin /5K.bin /index.htm /5K.bin … 50
  51. 51. Testing Zhtta > httperf --server localhost --port 4414 --rate 60 --num-conns 60 --wlog=y,./zhtta-test-urls.httperf Total: connections 60 requests 60 replies 60 test-duration 32.588 s Connection rate: 1.8 conn/s (543.1 ms/conn, <=46 concurrent connections) Connection time [ms]: min 0.4 avg 1237.8 max 31954.9 median 563.5 stddev 4132.5 Connection time [ms]: connect 0.0 Connection length [replies/conn]: 1.000 Request rate: 1.8 req/s (543.1 ms/req) Request size [B]: 67.0 Reply rate [replies/s]: min 0.0 avg 2.0 max 11.6 stddev 4.7 (6 samples) Reply time [ms]: response 1215.7 transfer 22.1 Reply size [B]: header 71.0 content 16640936.0 footer 0.0 (total 16641007.0) Reply status: 1xx=0 2xx=60 3xx=0 4xx=0 5xx=0 CPU time [s]: user 7.40 system 25.12 (user 22.7% system 77.1% total 99.8%) Net I/O: 29920.6 KB/s (245.1*10^6 bps) Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0 Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0 17 October 2013 University of Virginia cs4414 51
  52. 52. Test Total Duration (seconds) Zhtta with SRPT and caching Zhtta with SRPT Zhtta with FIFO Zhttpto Single-task server 0 10 20 30 40 Is our memory caching helping? 17 October 2013 University of Virginia cs4414 52
  53. 53. Test Duration (seconds) Zhtta with SRPT and caching Zhtta with SRPT Zhtta with FIFO Average Response Time (ms) Zhttpto Single-task server Zhtta with SRPT and caching 0 10 20 30 40 Zhtta with SRPT Zhtta with FIFO How good is this? Zhttpto Single-task server 0 17 October 2013 5000 10000 University of Virginia cs4414 15000 53
  54. 54. Possible to do Much Better! Test Duration (seconds) Apache2 Zhtta with SRPT and caching Zhtta with SRPT Zhtta with FIFO Zhttpto Single-task server 0 17 October 2013 10 20 University of Virginia cs4414 30 40 54
  55. 55. Average Response Time (ms) 0.2 ms Apache2 Zhtta with SRPT and caching Zhtta with SRPT Zhtta with FIFO Zhttpto Single-task server 0 5000 10000 15000 httperf is measuring “Response Time” as latency to receive first byte! 17 October 2013 University of Virginia cs4414 55
  56. 56. Charge PS3 is due less than one week from today! Still, no one has claimed OpenPiazza! • You should definitely be able to show improvements using Weilin’s benchmark (posted with today’s notes) • To prevent overly-targeted optimizing, we will also test your servers with a different workload • Demo will separately check correctness – e.g., if a served file changes, should serve new file not cached old one (within 10 seconds) – To prevent “cheating”, we may look at your code 17 October 2013 University of Virginia cs4414 56

×