1. The BSU Xseed:
Experiences Building a Top 500 Supercomputer
Mark Matties Bowie State University
June 22, 2009

2. Bowie State University

Founded in 1865
− Oldest HBCU in Maryland
− 5th oldest in US

Part of University System of Maryland

Regional Comprehensive University
− Officially, primary mission is teaching,
not research

Enrollment ~5500 (Fall 2008)

3. BSU Dept. of Computer Science

Undergraduate Programs
− B.S. In Computer Science
− B.S. In Computer Technology

Graduate Programs
− M.S. In Computer Science
− D.Sci. In Applied Computer Science (est. 2007)

4. The BSU Xseed – How It All Began

Small grant from FDA to test BOINC on a
couple of machines
– Just enough for 1 UG for 1 semester

On advise of dept chair, wrote a 2 page brief “to
let people know what you're doing”

BSU President Calvin Lowe did read it
– Was already thinking of creating cluster at BSU

When the President calls, you answer

5. The BSU Xseed in the Top 500

Placed 166 in Nov 2005
− Top 25 of Universities

Dropped to 308 in Nov 2005

Dropped to 468 in Jun 2006

6. The BSU Xseed – Tech Specs

Speed > 2.1 Tflops on LinPACK

Compute Nodes 224 x Apple Xserve G5

Head Nodes 2 x Apple Xserve G5

Network Myrinet-2000 (single chassis)

Storage 2 x 3.5 TB Apple Xraid

7. Apple Xserve G5

CPU is 2 x 2.0 GHz

Memory is 2 GB

Local disk is 80 GB

Network is 2 x 1 Gbps Ethernet

OS is OS X 10.4

1 U rack mount case (driven by VT)

Added Myrinet card

8. Myrinet-2000

Connects up to 256 hosts in one chassis

Speed is 2+2 Gbps (full duplex)

MPI Latency is 2.6 – 3.2 s

TCP/IP data rate is 1.98 Gbps

9. What it Cost

Total cost ~ $1.1M
− Hardware > $800k
Computers, Myrinet, SAN
− Facilities ~ $250k
Air handlers (20 T of cooling)
New electrical circuits
Space renovation

10. Who Paid?

BSU Title I money ~ $900k

PG County Small Business Incubator ~ $200k

BSU Dept. of Computer Science ~ $20k

11. The BSU Xseed – Preconstruction

Late Spring 2005

Received 226 nodes in 2 shipments

Received Myrinet switch chassis + 224 cards

Received 9 racks

Be kind to your Shipping/Receiving people!

12. The BSU Xseed – Construction v1.0

Late Spring 2005

Conversion/renovation not finished, but we
wanted to construct and run for Jun 2005 Top
500 list

Built up in lab in CS building

Rented 3 small air handlers

Ran ~ 20 extension cords

To no avail – could not support such a strain

13. The BSU Xseed – Construction v2.0

Summer / Fall 2005

Renovation finished, time to move

Tore down 9 racks, 226 nodes

Moved across campus by moving company

Rebuilt (for the last time!)

Started LinPACK runs
– Thanks, Apple!

14. Applications

LAM / MPI + MPICH

|Open source – almost no money for software
– Intel compiler, OS X upgrade

Home grown code
– Expected experienced users first

Globus? Sun Grid Engine? Apple?
– Settled on distributed ssh

15. Top User

Elephants Dream (2005 - 2006)

Rendering for an open source movie using
Blender

Over ~ 125 days

Thousands of hours of CPU time

Spare time donated in exchange for publicity

16. Spin Off Benefits

Co-located Projects
− BSU VIN (Virtual Incubator Network)

Granville Sawyer, virtual business incubator
− Streaming Media for BIOL 203 Oceanus

Bill Lawrence, online course

Research Network
− Physically separated from campus network
− 1000 Mpbs between BSU Xseed, CS bldg

No cost 100 Mbps ISP connection (INet fiber)

17. What's Next?

I am currently working on a project in
augmented reality with Jayfus Doswell, et al.
− Lots of image processing

Approach using GPU/CUDA, so we'll apply for
a grant to build nVidia GPU based cluster

The nVidia Tesla S1070 will deliver
~4 Tflops + 16 GB of Memory in 1U case

“Personal Supercomputers”

18. Concluding Thoughts

If you build it, they will come.
Don't count on it!

Get all your administrative ducks lined up
− Where will it live? Who controls it?
• Get commitment for local support
− As much as you can upfront
• Thank your support people in advance!
− You're asking much of them.

19. Thanks

Calvin Lowe for his vision

Sadanand Srivastava for his many faceted
support

Scott McDowell for Apple's support

All the BSU students who helped build it

You – for listening