Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Cisco Public 1© 2013 Cisco and/or its affiliates. All rights reserved.Open Source for CiscoHigh PerformanceComputingDr. Je...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 21. Who am I?2. Cisco and Open Source3. My Open Sour...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 4MeTechnical Lead atCisco SystemsServer division, VI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 5I am not inmarketing
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 6I cannot fixyour Linksysrouter for you(perhaps you ...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 7I write codeLots of codeAll dayEvery day
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 10Open MPIHardware Locality (hwloc)OpenFabricsLinux ...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 12Undergrad, grad Post doc
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 13LAM/MPII inherited thisI founded this
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 14PACX-MPILAM/MPILA-MPIFT-MPISun CT 6Project founded...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 15PACX-MPILAM/MPILA-MPIFT-MPISun CT 6Me
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 16PACX-MPILAM/MPILA-MPIFT-MPISun CT 6Me
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 17PACX-MPILAM/MPILA-MPIFT-MPISun CT 6
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 18Us
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 19Differences=Good
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 20You writeto your ownlevel ofexpectations
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 21You writeto theirlevel ofexpectations
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 22You writeyour best code
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 24More than just the 4 projectsI participate in
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 25More than just the 4 projectsI participate in
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 26After the clouds part,you are left with…
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 27Major contributionto
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 29Why does Ciscodo Open Source?
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 30Why does Ciscodo Open Source?• Stand on the should...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 32WeareelevatedWeelevateothers
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 33Circle oftrustyou
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 34Circle oftrustThis is whereyou needto be
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 35Insert FOSSprojectname hereThis is whereyou needto...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 36Pretend this isa really grosspicture of a leech
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 37Pretend this isa really grosspicture of a leech
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 38Pretend this isa really grosspicture of a leechJus...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 39Let’s be clear here…EVERYONE(that’s kinda the poin...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 40Let’s be clear here…EVERYONE(that’s kinda the poin...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 41I don’t contribute to everypiece of FOSS I useDo y...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 42Those who can,should
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 43Big companies can contribute
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 44Individuals can contribute
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 45Small organizations can contribute
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 46Those who can,should
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 47Those who can,should
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 48A giant
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 49You, standingon the giant’sshoulders…
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 50…in thecircleof trust
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 51…contributingto the community
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 52Cisco UCSblade serverCisco Nexus7000 router
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 55Using supercomputers to solvereal world problems t...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 56Supercomputer=(Many) Racks of (commodity)high-end ...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 57Rack of36 1Uservers
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 58Computational problemInput OutputTake your computa...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 59…and split it up!Computational problemInput Output
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 60Computational problemInput OutputDistribute the in...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 61Input OutputUse the network between serversto comm...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 62Input OutputUse the network between serversto comm...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 63Message Passing Interface (MPI)middleware is used ...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 64Computational problemOneprocessorhour1 processor =...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 65Computational problemOneprocessorhourOneprocessorh...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 66This communicationmay happen a LOTIt therefore nee...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 67SourceserverDestinationserver
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 68HPC applicationMPI middlewareTCP stackNIC driverNI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 69HPC applicationMPI middlewareTCP stackNIC driverNI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 70YES
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 71• Intel Core i7 E5-2690 with turbo boost (3.5-3.8G...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 72HPC apps can do a LOT of computationduring network...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 73Hardware is faster than software.The sooner softwa...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 74HPC applicationMPI middlewareTCP stackNIC driverNI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 75HPC applicationMPI middlewareTCP stackNIC driverNI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 76HPC applicationMPI middlewareTCP stackNIC driverNI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 77HPC applicationMPI middlewareTCP stackNIC driverNI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 78HPC applicationMPI middlewareTCP stackNIC driverNI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 79HPC applicationMPI middlewareTCP stackNIC driverNI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 80Linux userspaceapplicationLinux kernelCisco VIChar...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 81Linux userspaceapplicationLinux kernelCisco VIChar...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 82Linux userspaceapplicationLinux kernelCisco VIChar...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 83Linux userspaceapplicationLinux kernelCisco VIChar...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 84Linux userspaceapplicationLinux kernelCisco VIChar...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 85• Coalition of network vendors• Successfully upstr...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 86HPC applicationMPI middlewareVIC driverCisco VICOu...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 87KernelCisco VIC hardwareTCP / IP stackCisco VIC dr...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 88KernelUserspace verbs libraryCisco VIC hardwareMPI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 89Hardware Locality (hwloc)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 90• Query your server’stopology• NUMA nodesIncluding...
Machine (128GB)NUMANode P#0 (64GB)Socket P#0L3 (20MB)L2 (256KB)L1d (32KB)L1i (32KB)Core P#0PU P#0PU P#16L2 (256KB)L1d (32K...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 92• Output formats supported:PDF, JPG, PNG, TIFF, FI...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 93• hwloc-bind socket:0.core:2 commandBind command t...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 94• Get a tree data structure representing the topol...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 95• Verify the internal topology of your serverHow m...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 96• Bind services to specific coresEnsure related se...
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 97Cisco Public 9797© 2013 Cisco and/or its affiliate...
Thank you.
MOSSCon 2013, Cisco Open Source talk
MOSSCon 2013, Cisco Open Source talk
MOSSCon 2013, Cisco Open Source talk
Upcoming SlideShare
Loading in …5
×

MOSSCon 2013, Cisco Open Source talk

1,656 views

Published on

These are the slides that I presented at MOSSCon 2013 (slightly edited, because the original slides contained some animations that I morphed to look ok on Slideshare).

The general talk is about two things:

1. General philosophy of open source at Cisco.
2. My specific open source work at Cisco.

Enjoy!

Published in: Technology, Education
  • Be the first to comment

MOSSCon 2013, Cisco Open Source talk

  1. 1. Cisco Public 1© 2013 Cisco and/or its affiliates. All rights reserved.Open Source for CiscoHigh PerformanceComputingDr. Jeffrey M. Squyres
  2. 2. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 21. Who am I?2. Cisco and Open Source3. My Open Source work at Cisco
  3. 3. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 4MeTechnical Lead atCisco SystemsServer division, VIC group
  4. 4. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 5I am not inmarketing
  5. 5. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 6I cannot fixyour Linksysrouter for you(perhaps you should tryDD-WRT)
  6. 6. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 7I write codeLots of codeAll dayEvery day
  7. 7. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
  8. 8. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
  9. 9. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 10Open MPIHardware Locality (hwloc)OpenFabricsLinux kernelVast majorityof my workis hereI’ve mademinor contributionsto these other 3
  10. 10. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
  11. 11. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 12Undergrad, grad Post doc
  12. 12. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 13LAM/MPII inherited thisI founded this
  13. 13. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 14PACX-MPILAM/MPILA-MPIFT-MPISun CT 6Project foundedin 2003,merging multipleopen sourceMPI projects
  14. 14. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 15PACX-MPILAM/MPILA-MPIFT-MPISun CT 6Me
  15. 15. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 16PACX-MPILAM/MPILA-MPIFT-MPISun CT 6Me
  16. 16. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 17PACX-MPILAM/MPILA-MPIFT-MPISun CT 6
  17. 17. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 18Us
  18. 18. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 19Differences=Good
  19. 19. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 20You writeto your ownlevel ofexpectations
  20. 20. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 21You writeto theirlevel ofexpectations
  21. 21. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 22You writeyour best code
  22. 22. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 24More than just the 4 projectsI participate in
  23. 23. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 25More than just the 4 projectsI participate in
  24. 24. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 26After the clouds part,you are left with…
  25. 25. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 27Major contributionto
  26. 26. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
  27. 27. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 29Why does Ciscodo Open Source?
  28. 28. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 30Why does Ciscodo Open Source?• Stand on the shoulders of giants• Become part of the community• Contribute to tools / ecosystemthat we all use• Sell more products
  29. 29. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
  30. 30. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 32WeareelevatedWeelevateothers
  31. 31. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 33Circle oftrustyou
  32. 32. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 34Circle oftrustThis is whereyou needto be
  33. 33. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 35Insert FOSSprojectname hereThis is whereyou needto be
  34. 34. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 36Pretend this isa really grosspicture of a leech
  35. 35. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 37Pretend this isa really grosspicture of a leech
  36. 36. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 38Pretend this isa really grosspicture of a leechJust say noto leeches!
  37. 37. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 39Let’s be clear here…EVERYONE(that’s kinda the point, right?)
  38. 38. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 40Let’s be clear here…EVERYONE(that’s kinda the point, right?)
  39. 39. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 41I don’t contribute to everypiece of FOSS I useDo you?
  40. 40. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 42Those who can,should
  41. 41. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 43Big companies can contribute
  42. 42. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 44Individuals can contribute
  43. 43. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 45Small organizations can contribute
  44. 44. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 46Those who can,should
  45. 45. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 47Those who can,should
  46. 46. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 48A giant
  47. 47. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 49You, standingon the giant’sshoulders…
  48. 48. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 50…in thecircleof trust
  49. 49. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 51…contributingto the community
  50. 50. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 52Cisco UCSblade serverCisco Nexus7000 router
  51. 51. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
  52. 52. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 55Using supercomputers to solvereal world problems that areTOO BIGfor laptops, desktops,or individuals servers
  53. 53. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 56Supercomputer=(Many) Racks of (commodity)high-end servers(this is one definition; there are others)
  54. 54. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 57Rack of36 1Uservers
  55. 55. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 58Computational problemInput OutputTake your computational problem…
  56. 56. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 59…and split it up!Computational problemInput Output
  57. 57. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 60Computational problemInput OutputDistribute the input dataacross a bunch of servers
  58. 58. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 61Input OutputUse the network between serversto communicate / coordinate
  59. 59. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 62Input OutputUse the network between serversto communicate / coordinate
  60. 60. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 63Message Passing Interface (MPI)middleware is used for this communicationInput Output
  61. 61. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 64Computational problemOneprocessorhour1 processor = …a long time…
  62. 62. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 65Computational problemOneprocessorhourOneprocessorhourOneprocessorhour21 processors = ~1 hour (!)Disclaimer: scaling is rarely perfectOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhourOneprocessorhour
  63. 63. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 66This communicationmay happen a LOTIt therefore needsto be FAST
  64. 64. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 67SourceserverDestinationserver
  65. 65. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 68HPC applicationMPI middlewareTCP stackNIC driverNIC hardwareHPC applicationMPI middlewareTCP stackNIC driverNIC hardwarePortAPortB
  66. 66. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 69HPC applicationMPI middlewareTCP stackNIC driverNIC hardwareHPC applicationMPI middlewareTCP stackNIC driverNIC hardwarePortAPortB200 nanoseconds299,792,458 m/s c~8 microseconds(modern hardware) ~8-40 microsecondsTotal:~17 – 81microseconds~40 microseconds(older hardware)
  67. 67. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 70YES
  68. 68. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 71• Intel Core i7 E5-2690 with turbo boost (3.5-3.8Ghz)“Sandy Bridge” 22nm processor• LinX v0.6.4 (Linpack v10.3.4.007) benchmarkMeasures floating point operations per second• 81.34 GflopsThat’s 81,340,000,000 floating point operations per second17μs = 137,757,800 floating point operations81μs = 656,375,400 floating point operationsConclusion: yes, we absolutely care about 17-81μs!Source: http://www.anandtech.com/show/4503/sandy-bridge-memory-scaling-choosing-the-best-ddr3
  69. 69. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 72HPC apps can do a LOT of computationduring network communicationLatency
  70. 70. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 73Hardware is faster than software.The sooner software canhand off to hardware, the better.
  71. 71. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 74HPC applicationMPI middlewareTCP stackNIC driverNIC hardwareHPC applicationMPI middlewareTCP stackNIC driverNIC hardwarePortAPortB200 nanoseconds299,792,458 m/s 299,792,458 m/s~8 microseconds(modern hardware) ~8-40 microseconds~40 microseconds(older hardware)
  72. 72. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 75HPC applicationMPI middlewareTCP stackNIC driverNIC hardwareHPC applicationMPI middlewareTCP stackNIC driverNIC hardwarePortAPortB200 nanoseconds299,792,458 m/s 299,792,458 m/s~8 microseconds(modern hardware) ~8-40 microseconds~40 microseconds(older hardware)Can’t do much about the speed of light 
  73. 73. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 76HPC applicationMPI middlewareTCP stackNIC driverNIC hardwareHPC applicationMPI middlewareTCP stackNIC driverNIC hardwarePortAPortB200 nanoseconds299,792,458 m/s 299,792,458 m/s~8 microseconds(modern hardware) ~8-40 microseconds~40 microseconds(older hardware)Can’t do much about the speed of light Fastest Ethernet switches today are about 200ns(they’ll probably get a little faster over time)
  74. 74. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 77HPC applicationMPI middlewareTCP stackNIC driverNIC hardwareHPC applicationMPI middlewareTCP stackNIC driverNIC hardwarePortAPortB200 nanoseconds299,792,458 m/s 299,792,458 m/s~8 microseconds(modern hardware) ~8-40 microseconds~40 microseconds(older hardware)Can’t do much about the speed of light Fastest Ethernet switches today are about 200ns(they’ll probably get a little faster over time)8-40us is, by far, the biggest chunk of timeReduce this!
  75. 75. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 78HPC applicationMPI middlewareTCP stackNIC driverNIC hardwareWhat if we can skip some of these layers?Who needs TCP? Raw L2 Ethernet frames, baby!Who needs the operating system driver?
  76. 76. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 79HPC applicationMPI middlewareTCP stackNIC driverNIC hardwareWhat if we can skip some of these layers?Who needs TCP? Raw L2 Ethernet frames, baby!Who needs the operating system driver?Let MPI talk directly to the NIC hardware
  77. 77. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 80Linux userspaceapplicationLinux kernelCisco VIChardwareCan I seethe hardware?Please?
  78. 78. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 81Linux userspaceapplicationLinux kernelCisco VIChardwareNo.
  79. 79. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 82Linux userspaceapplicationLinux kernelCisco VIChardwareCan I seethe OpenFabricshardware?
  80. 80. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 83Linux userspaceapplicationLinux kernelCisco VIChardwareSure!
  81. 81. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 84Linux userspaceapplicationLinux kernelCisco VIChardwareCan I seethe OpenFabricshardware?Yay!
  82. 82. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 85• Coalition of network vendors• Successfully upstreamed “OS bypass for networking” into Linux• http://www.openfabrics.org
  83. 83. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 86HPC applicationMPI middlewareVIC driverCisco VICOur project: enabling this MPI direct-to-hardware communication on Cisco serverswith the Cisco Virtual Interface Card (VIC)in Linux.Everything above the firmware will beopen source.
  84. 84. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 87KernelCisco VIC hardwareTCP / IP stackCisco VIC driverUserspaceUserspace sockets libraryMPI libraryApplication
  85. 85. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 88KernelUserspace verbs libraryCisco VIC hardwareMPI libraryUserspaceVerbs IB coreCisco USNIC driverBootstrappingand setupSend and receivefast pathApplicationCisco codehere
  86. 86. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 89Hardware Locality (hwloc)
  87. 87. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 90• Query your server’stopology• NUMA nodesIncluding memory• Processor sockets• L3, L2, L1 cachesInstruction and data• Cores• Hyperthreads• PCI devicesMachine (128GB)NUMANode P#0 (64GB)Socket P#0L3 (20MB)L2 (256KB)L1d (32KB)L1i (32KB)Core P#0PU P#0PU P#16L2 (256KB)L1d (32KB)L1i (32KB)Core P#1PU P#1PU P#17L2 (256KB)L1d (32KB)L1i (32KB)Core P#2PU P#2PU P#18L2 (256KB)L1d (32KB)L1i (32KB)Core P#3PU P#3PU P#19L2 (256KB)L1d (32KB)L1i (32KB)Core P#4PU P#4PU P#20L2 (256KB)L1d (32KB)L1i (32KB)Core P#5PU P#5PU P#21L2 (256KB)L1d (32KB)L1i (32KB)Core P#6PU P#6PU P#22L2 (256KB)L1d (32KB)L1i (32KB)Core P#7PU P#7PU P#23PCI 8086:1521eth0PCI 8086:1521eth1PCI 8086:1521eth2PCI 8086:1521eth3PCI 1137:0043eth4PCI 1137:0043eth5PCI 102b:0522NUMANode P#1 (64GB)Socket P#1L3 (20MB)L2 (256KB)L1d (32KB)L1i (32KB)Core P#0PU P#8PU P#24L2 (256KB)L1d (32KB)L1i (32KB)Core P#1PU P#9PU P#25L2 (256KB)L1d (32KB)L1i (32KB)Core P#2PU P#10PU P#26L2 (256KB)L1d (32KB)L1i (32KB)Core P#3PU P#11PU P#27L2 (256KB)L1d (32KB)L1i (32KB)Core P#4PU P#12PU P#28L2 (256KB)L1d (32KB)L1i (32KB)Core P#5PU P#13PU P#29L2 (256KB)L1d (32KB)L1i (32KB)Core P#6PU P#14PU P#30L2 (256KB)L1d (32KB)L1i (32KB)Core P#7PU P#15PU P#31PCI 1000:005bsda sdbPCI 1137:0043eth6PCI 1137:0043eth7Indexes: physicalDate: Mon Jan 28 10:51:26 2013
  88. 88. Machine (128GB)NUMANode P#0 (64GB)Socket P#0L3 (20MB)L2 (256KB)L1d (32KB)L1i (32KB)Core P#0PU P#0PU P#16L2 (256KB)L1d (32KB)L1i (32KB)Core P#1PU P#1PU P#17L2 (256KB)L1d (32KB)L1i (32KB)Core P#2PU P#2PU P#18L2 (256KB)L1d (32KB)L1i (32KB)Core P#3PU P#3PU P#19L2 (256KB)L1d (32KB)L1i (32KB)Core P#4PU P#4PU P#20L2 (256KB)L1d (32KB)L1i (32KB)Core P#5PU P#5PU P#21L2 (256KB)L1d (32KB)L1i (32KB)Core P#6PU P#6PU P#22L2 (256KB)L1d (32KB)L1i (32KB)Core P#7PU P#7PU P#23PCI 8086:1521eth0PCI 8086:1521eth1PCI 8086:1521eth2PCI 8086:1521eth3PCI 1137:0043eth4PCI 1137:0043eth5PCI 102b:0522NUMANode P#1 (64GB)Socket P#1L3 (20MB)L2 (256KB)L1d (32KB)L1i (32KB)Core P#0PU P#8PU P#24L2 (256KB)L1d (32KB)L1i (32KB)Core P#1PU P#9PU P#25L2 (256KB)L1d (32KB)L1i (32KB)Core P#2PU P#10PU P#26L2 (256KB)L1d (32KB)L1i (32KB)Core P#3PU P#11PU P#27L2 (256KB)L1d (32KB)L1i (32KB)Core P#4PU P#12PU P#28L2 (256KB)L1d (32KB)L1i (32KB)Core P#5PU P#13PU P#29L2 (256KB)L1d (32KB)L1i (32KB)Core P#6PU P#14PU P#30L2 (256KB)L1d (32KB)L1i (32KB)Core P#7PU P#15PU P#31PCI 1000:005bsda sdbPCI 1137:0043eth6PCI 1137:0043eth7Indexes: physicalDate: Mon Jan 28 10:51:26 2013
  89. 89. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 92• Output formats supported:PDF, JPG, PNG, TIFF, FIG, …Text (for console windows)CursesXML• Great for feeding into scripts!
  90. 90. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 93• hwloc-bind socket:0.core:2 commandBind command to core 2 on socket 0• hwloc-bind –getPrint a bitmap of your current bindings• hwloc-bind --get | hwloc-calc -p -H socket.corePrint something more readable than a bitmap• hwloc-bind --get | hwloc-calc -p -H socket.core.puEven show the hardware threads• hwloc-ps [-a]Show where processes are bound
  91. 91. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 94• Get a tree data structure representing the topology• Many API calls for manipulating / traversing the tree• Typical actions:Get, set processor and memory bindingsReact to cache sizes• …everything you can do in the CLI, and more
  92. 92. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 95• Verify the internal topology of your serverHow much memory do you have?Where is that memory?What processor(s) are local to that memory?How big are your L1, L2, L3 caches?• Verify your internal PCI devicesDistinguish ethX devices from each other
  93. 93. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 96• Bind services to specific coresEnsure related services are on thesame NUMA nodePut non-essential services on core 0(e.g., NTP)• Bind server-related servicesApache, Bind, NFS, …etc.Increase performance by not lettingthem migrateKeeps memory local, less inter-NUMA-node trafficNTPetc.ApacheNFS
  94. 94. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 97Cisco Public 9797© 2013 Cisco and/or its affiliates. All rights reserved.“Open source is good.Open source works.
  95. 95. Thank you.

×