3. 1 Abstract
Having different architecture processors, cre-
ating a cluster for all types takes the right
software. Using Debian Linux operating sys-
tem, it is possible to create a fast computer
cluster of any architecture. Installing free
software to minimize cost and tweaking set-
tings so that computers can communicate
freely. Once everything is working, make it
more user friendly so that the cluster has
some use to people just learning about com-
puter cluster. Then testing the clusters dif-
ferent processors to see which one is best for
certain types of programs. The main test was
to see how a cluster of Raspberry Pis would
compare to a cluster of towers(desktop com-
puters). A Raspberry Pi is a credit card sized
computer that was made to keep costs as low
as possible.
2 Introduction
A computer cluster can help any scientist who
needs to run a program that takes a long time
to compute. Instead of using one computer,
using many that split the task and commu-
nicate with each other can cut the time to
complete the task.
Creating a computer cluster is one chal-
lenge. Building a computer cluster using
different architectures creates new obstacles.
Taking into account that the binary files for
one computer may not be able to run on an-
other computer because of the different ar-
chitecture. This creates a problem when all
the computers need to communicate with out
any error.
Being able to create a computer clus-
ter of different architecture allows people to
cheaply create a computer cluster and the op-
tion to choose which architecture is best for
a program. Certain architectures can do a
task faster than another and the other ar-
chitecture might be able to run a different
task faster. This creates a computer cluster
that can adapt to many different program-
ming techniques.
Creating a computer cluster can be expen-
sive if you want the fastest computers. Us-
ing Raspberry Pis that cost thirty-five dol-
lars each and use less power then towers can
cut the cost of creating a cluster. But just
how does a cluster of 700Mhz processor of a
Raspberry Pi compare to a cluster of towers
today?
3 Materials and Methods
The next sections are used to explain how
to repeat what has been done. Getting the
different architectures to communicate with
one another has not been accomplished yet.
Getting them to communicate will be done at
a future date in time.
3.1 Materials Needed
Then main component of a computer cluster
are the computers. The faster the comput-
ers you have the faster the cluster will be
overall. Using three different architectures
was enough to see results that were needed.
The three architecture I used were Intel,
1
4. Broadcom, and PowerPC. For towers I used
one dual-core hyper-threaded 2.66Ghz Intel
Xenon, four single-core 2.4Ghz Intel Pen-
tium, one single core 2.4Ghz AMD Opteron,
and two quad-core hyper-threaded 3.5Ghz In-
tel I7-3770k. I had five old iBook G4 that
are single-core 500Mhz PowerPc and thirty
Raspberry Pis with 700Mhz BCM2835.
Also needed are a mean of communication
between the computers so a network for all
of these computers is required. One network
cable for each computer and switches with
enough ports to plug up each computer to
it plus one to make installing packages from
the internet faster. Buying more thaen one
network switch may be necessary. An inter-
net connection is required to install packages
onto each computer and so other people can
use the cluster once it is complete. One of
the faster computers needs to have two Eth-
ernet ports on it. One is to go to the switch
and the other is to connect to the internet.
Battery backups and surge protectors are not
necessary but are recommended to protect
the computers. One SD card for each Rasp-
berry Pi is needed because the SD card is
the hard drive for the Raspberry Pi. A cou-
ple monitors and a keyboard are a must and
micro USB to standard USB and USB wall
chargers to power the Raspberry Pis. Refer
to Figure 1 for an image of a micro USB. The
keyboard should be a USB one to be able to
control the Raspberry Pis and a monitor that
can support AV jacks or HDMI to view the
Raspberry Pis output.
usb.jpg
Figure 1: From left to right: Micro USB, Mini
USB, Standard USB
3.2 Hardware Setup
With all the wires it is recommended to stay
organized and use twist ties to keep wires or-
ganized. The towers will be the easiest to
start with. Plug the power supply into a
surge protector strip then the surge protec-
tor to the battery backup. And of course,
the battery backup to the wall. Just plug all
of the towers power supply to the power surge
and make sure that the battery backup is
not over used. About seventy percent power
usage should be used while the computers
are idle. When programs start to run on all
the computers they draw more power. Once
plugged up plug one end of a network cable
to a tower and the other to a switch.
Figure 2 shows how each computer(nodes)
connect to a switch. If more then one switch
is used, connecting the two switches works
just as well as one big one. The head is the
2
5. setup.png
Figure 2: The Networking Setup
fastest computer in the cluster and will be
the computer that will control the rest of the
cluster. The head and a switch is connected
to the internet during setup. But when the
cluster is finished the switch is removed from
the internet connection so that the head is
the only connection to the internet.
The iBooks are setup the same way and
so are the Raspberry Pis. The only main
difference is the power connection. For the
iBooks use the chargers they came with and
keep them plugged in. For the Raspberry
Pis connect the micro USB to the Raspberry
Pi and the standard USB to the USB wall
charger. Once Raspberry Pis are connected
to power, they turn on. Recommended is to
use a surge protector that can be turned on
and off and connect just the Raspberry Pis to
those. That way you can turn off the cluster
when ever needed.
As for monitors, one is needed if the towers
have HDMI but if not then two. Also the key-
board needs to be USB and if the towers can-
not use a USB keyboard then two keyboards
are needed. Connect the monitor and key-
board, throughout the setup, to which ever
computer you are currently working on. The
Raspberry Pis can use HDMI to display in-
formation and also AV jack is supported. To
connect a keyboard just plug the keyboard
into one of the two USB ports on the Rasp-
berry Pi.
3.3 Software Setup
What makes a computer cluster is how they
communicate. The most important way for
them to communicate is the proper software.
3.3.1 Operating System
What makes the computer work is an operat-
ing system. A light weight free operating sys-
tem that could be installed on just about any
architecture is what is needed. Debian Linux
is just what is needed to create a computer
cluster. It allows to just download the shell
and not the graphical user interface. Consid-
ering the only way to control the cluster is
through commands there is not a need for a
graphical user interface.
Installing Debian on the towers is easy.
Download a copy of Debian with the correct
architecture(i386 for the towers) and burn it
to a cd(Debian). Once burnt, insert the CD
into the drive and boot into it. Most comput-
ers automatically try to boot into a CD if one
is present. Keeping most settings their de-
fault there are a few key point during instal-
3
6. lation. It will ask for a hostname and server
for the computer. The hostname should be
easy to remember. Using ”tower1”, ”tower2”,
”mac1”, ”mac2”, etc. keeps things organized.
Another key point is to remember the root
password and the username and password it
asks for. Recommended is to use the user-
name ”mpich.” Keep the same passwords and
username for all computer to stay organized
and uniform. Another option ask how you
want to install it. Installing all files in one
partition and using the entire hard drive is
the best option for what we are doing. You do
not want a graphical interface so deselect that
option when it comes up, but a SSH server is
required, so make sure that is selected.
The process is similar on the iBooks. Using
the same CD will not work so download the
same version just a different architecture im-
age(power pc for the macs) and burn it to a
CD. Insert the CD into the iBook then while
the iBook is booting up hold down the C key
which will boot into the CD. Once booted
just follow the same options as for the tow-
ers.
The Raspberry Pis are different. Go to
Raspberry Pis home page and go to the down-
loads page and download the newest version
of Raspian. This is Debian for the Raspberry
Pi. To install this operating system onto a
SD card download win32 diskimager. This
will take the iso file and extract and install it
onto the SD card. An iso file is a CD image
file. This file is burned to a CD. The win32
diskimager takes the iso and copies it to a
SD card instead of a CD. After inserting the
SD card into the Raspberry Pi and turning it
on by plugging power to it, you will be taken
to a setup screen. Check for updates, change
the keyboard settings so that it matches your
keyboard, enable SSH server, expand to fill
the SD card, and change the user pi’s pass-
word. This will complete the setup once you
finish and let it reset completely.
3.3.2 MPICH
There is a distributed package that has ev-
erything needed to make a cluster called
Bootable Cluster CD(BCCD). The only
problem with this distribution is that it only
allowed Intel architecture which would not
work on the iBooks and also needed a CD
drive to install and the Raspberry Pis do not
have a CD drive.
Searching through the documentation, I
found that it uses a package called MPICH
that is a free package that can be installed
on any architecture. This package is easy to
install. Find the path to the exact position
of its tarball(a file type similar to Windows
zip files) by going to MPICH’s homepage and
going to the downloads page(MPICH). Right
click on the link of the current MPICH re-
lease and say open in new tab. The URL in
the web browser is the exact path to the tar-
ball. On the computer login as the root user
and run the following command:
wget <URL to File>
Replace <URL to File>with the file
destination. Expect something like
http://www.mpich.org/static/downloads/1.5rc3
/mpich2-1.5rc3.tar.gz. This downloads it
onto the computer. To extract the file into
its folders run:
4
7. tar -xvzf mpich2-1.5rc3.tar.gz
Before installing a MPICH a few packages
are needed so that MPICH can run correctly.
Install gcc, g++, gfortran, and make. The
packages with the g’s are compilers to compile
your program and make is a program that
will install MPICH for us. To install these
packages run:
apt-get install g++ gcc gfortran make
It will ask if you are sure you want to down-
load these packages. Agree to it by pressing
the y key. After it is installed navigate in-
side the folder the tar function created. To
see the current folders use the ”ls” command.
To navigate inside of a certain folder use ”cd
<folder path>.” Once inside run:
./configure --prefix=/mpich2
This sets up MPICH to be ready to in-
stall. The parameter ”prefix” sets the desti-
nation of the install of MPCIH. The previous
command will install MPICH at ”/mpich2/.”
This process might take a while but it only
has to be done once per architecture. Once
it is complete run:
make;make install
The above command is actually two com-
mands. The command before the semi-colon
is ran first then the one after. The ”make”
command compiles everything MPICH uses
and the ”make install” command actually in-
stalls it where you specified in the prefix pa-
rameter.
This process only has to be done once per
architecture. Once done with one go to an-
other computer of a different architecture and
repeat the process.
3.3.3 Setting Up SSH
MPICH uses a process manager called HY-
DRA to carry out its processing among the
other computers. HYDRA uses a secure
shell(SSH) to send and receive messages from
all of the computers in the cluster. We have
already setup each operating system with
SSH but now they need to communicate with-
out a password and using hostnames.
To setup up hostnames you must be in
the root user of a computer. Editing the
”/etc/hosts” file will accomplish this task. To
edit a file in Debian run:
nano <path to file>
Of course replace the <path to file >with
the file. In this case ”/etc/hosts.” We need to
add lines to the very end of this document.
On one line we need specify the ip address
of another computer and the hostname asso-
ciated with that computer. It will look like
this:
17.117.17.2 tower1.fmarion.edu tower1
There should be a tab after the ip address
and the hostname/servername. The mid-
dle column specifies the hostname and server
name. You also need a line including the
current computers ip address and hostname.
Add a new line containing this detail for each
of the computers of your cluster. Once you
are done with that computers hosts file press
”ctrl X” then ”y” then ”enter.” This saves
the document and hostnames are now setup
on that computer. This has to be done on all
the other computers before you move on.
5
8. Setting up a password less ssh can be easy
for two computers but pain staking on a clus-
ter. On the same computer under the user
you created(mpich for example) run:
ssh-keygen -t rsa
To logout of the root run ”exit.” Then type
in the username and password. After enter-
ing in the above command keep all the de-
faults by pressing enter to all the questions.
After this the computer is ready to send files
to other computers. Only the password less
ssh setup process needs to be done on all the
computers in the cluster. The hosts file can
be done later and faster.
Since all the computers now have
the files we need, copy the file
”/home/mpich/.ssh/id rsa.pub” to all
the other computers. Do this by using
the ”scp” command. Use the following
command:
scp /home/mpich/.ssh/id_rsa.pub
tower1:/home/mpich/.ssh/id_rsa.pub.1
This command copies the file from the cur-
rent computer to the computer with the host-
name tower1. It also changes the file name
from id rsa.pub to id rsa.pub.1 which will
make sure that no files are overwritten. Each
time you do this it will ask you for a pass-
word. On that one computer do this to copy
that one file to all the other computers by
changing the hostname appropriately. Once
done, log into another computer and do the
same thing but change the id rsa.pub.1 to
id rsa.pub.2 and so forth to make sure no
files are over written. Keep doing this until
all computers have all the other computers
id rsa.pub file.
Once all the computers have the files run
the following command:
cat /home/mpich/.ssh/id_rsa.pub.1 >>
/home/mpich/.ssh/authorized_keys
This takes the contents of the id rsa.pub.1
and attaches it to the end of the autho-
rized keys file. Repeat this until all the
id rsa.pub files are appended to the autho-
rized keys file. Again, this process needs to
be repeated on all the other computers. The
computers should now be able to communi-
cate with eachother without the need of a
password.
3.3.4 Setting Up MPICH
This next part is really easy once the com-
puters can communicate without a password.
Under the root user of a computer that has
MPICH installed on it use the following com-
mand:
scp -r /mpich2 tower1:/mpich2
Copy this same folder to all the other com-
puters of the same architecture. Repeat this
process for all the other architectures you
may have in the cluster. Once everything
is copied over, log into the computer that
has the two Ethernet ports. This will be the
head computer, the one that will control all
the others. Make a file called hostfile in the
/mpich2/ folder and add the hostnames of
all the computers in the cluster on a separate
line. Ex.
6
9. tower1
tower2
This tells MPICH what computers to use
for the cluster. For the computer to recog-
nize commands and the file as the host file
add the following to lines to the end of the
/home/mpich/.bashrc file:
export PATH=$PATH;/mpich2/bin
export HYDRA-HOST-FILE=/mpich2/hostfile
Only the first line has to be copied
to all the other computers to their
/home/mpich/.bashrc file. This com-
pletes the cluster and should now run
correctly.
3.4 Running Programs
This will not teach you how to write an MPI
program but will teach you how to compile
and run a program.
Once a code has been written in C, C++ or
Fortran(java is compatable but did not down-
load a java compiler) run the following com-
mand:
mpicc -o <exec> <file>
The mpicc is used to compile C and C++
programs. Use the mpifortran instead if the
program is written in Fortran. The ”o” pa-
rameter is used to specify the name of the
output file(<exec>). The <file>is the name
of the file you want to compile. To run the
program we need to copy the file to all the
other computers first using the scp command.
Then run the following command on the head
computer:
mpiexec ./<exec>
This runs the program through MPICH
and does whatever you told the program to
do. MPICH and the cluster is now set up to
use.
4 Troubleshooting
While I was building the cluster a few ob-
stacles came in the way. The first was that
the binary files compiled on one architecture
will not run on a different architecture com-
puter. To fix this I created three separate
clusters that used the same architecture com-
puters. They are all still networked together
but changed the /mpich2/hostfile to only use
the computers of the same architecture. Then
on a different computer of another architec-
ture add the /mpich2/hostfile file with the
appropriate computers.
The other obstacle encountered was when I
used the command MPI Broad in a program.
I got an error saying that the connection with
a certain process was denied. To fix this error
I added the parameter to the mpiexec com-
mand, -disable-hostname-propagation which
stops HYDRA from sending the hostnames
of the process to MPICH.
The last obstacle was that the cluster was
only using one core and not all of them on
a single computer. Fixing this I edited the
/mpich2/hostfile. I added a semicolon to all
the computers plus how many cores that com-
puter had. It looked like this when I was
done:
tower1:2
7
10. tower2:8
5 Results
I made a program that clocked how fast each
node can run a certain process. This process
calculated pi. I clocked how fast the actual
computation was and how fast the compu-
tation and the communication took. I com-
pared a currently working cluster at Clem-
son University called the Palmetto Cluster
to the one I created. One of our quad-core
hyper-threaded computer was faster then one
of theirs. This was understandable consider-
ing we just built ours and theirs are a few
years old.
I then compared all of the separate clus-
ters to the others I created. The tower clus-
ter I created was faster then the other clus-
ters when I used all the cores. When I did
not it was slower but still faster but not by
much. The Raspberry Pis were the second
fastest then the iBook G4s. This was un-
derstandable considering the Raspberry Pi’s
clock speed is faster than the iBook’s.
After comparing results I concluded that
for heavier processes the tower cluster would
be the best choice. For processes that need to
be split up among a lot of processors than the
Raspberry Pis will be the best choice. The
iBooks are only good for debugging a pro-
gram.
6 Acknowledgments
I would like to say thanks to Talon Brown,
who helped me tremendously on deciding on
how to set things up. Doctor Moldenhaure
made it all possible to build the cluster. I
would like to thank Francis Marion Univer-
sity for supporting me in my studies on the
cluster.
7 Citations
BCCD Documentation, BCCD, [online] 2013,
http://bccd.net/wiki/index.php/Main Page
(Accessed April 29, 2013)
Downloading Debian CD images with
BitTorrent, Debian, [online] 2013,
http://www.debian.org/CD/torrent-cd/
(Accessed April 29, 2013)
Downloads, MPICH, [online] 2013,
http://www.mpich.org/downloads/(Accessed
April 29, 2013)
8
