Mangling your .bashrc

There was a time when the department's installation on MPI was broken, and I gave directions to use an installation under the cs448b account instead. The department installation seems to work now, and I'm not maintaining the old install under cs448b. So, there's nothing that you need to do to your .bashrc.

If you did change your .bashrc according to the previous instructions, you can change it back. The old instructions had you set your PATH and LD_LIBRARY_PATH variables. If you were using the department defaults, then you shouldn't need to set these variables. If you've customized them for other reasons, just remove the entries for cs448b stuff.

Setting up SSH keys

To run MPI processes on multiple linux boxes, you will use the program mpiexec which needs to be able to start processes on the other machines. It uses ssh to do this, and must be able to make these connections without a password. If you don't have ssh keys set up on your account already, you should do so by following the directions below:

Given the command (from the unix/linux shell prompt): ssh-keygen -t dsa
ssh-keygen will ask you for a key phrase -- just press the <enter> key. It will ask you to enter the key phrase again, and you should just press the <enter> key again. This allows machines that know the RSA keys that ssh-keygen creates to connect to each other without needing to enter a password.
The -t rsa flag told ssh-keygen to generate a private/public key pair RSA public-key cryptography. You can also use DSA. If you care, give the command man ssh-keygen to read all about it.
Go to the directory ~/.ssh. You should see that ssh-keygen has generated files id_rsa (your newly minted private key) and id_rsa.pub (the corresponding public key).
Create a new file, authorized_keys in ~/.ssh.
Make 25 copies of the file ~/.ssh/id_rsa.pub in it, and modify them, so they authorize

you@lin01.ugrad.cs.ubc.ca

you@lin02.ugrad.cs.ubc.ca

you@lin03.ugrad.cs.ubc.ca

...

you@lin25.ugrad.cs.ubc.ca

to connect, where you replace ``you'' with your username on these machines. Note that the machine name is the last entry on the line that you got from ~/.ssh/id_ras.pub
You can probably get away with only authorizing a few machines. It should be enough just to authorize the machines that you will run mpi from; I don't think you need to list every machine that mpiexec will be connecting to. Also, there might be a way to wildcard machine names, but I don't know how to do that.
To test this, try ssh lin02.ugrad.cs.ubc.ca when you're logged onto lin01.ugrad.cs.ubc.ca. It should let you log-on without a password.
Note, that at any given moment, any of the lin?? machines can be down. This is because they all live in a graphics lab in the basement of the ICICS building. Some student turn-off the machine they are using when they leave (even though it's obviously not a windoze box). So, if you can't connect to lin01 (or lin02), try a different machine. Of course, if you find you can't log on to any of the lin?? machines, ask for help.

Setting up your MPI secret word

Create the file ~/.mpd.conf and put the single line in it:

MPD_SECRETWORD=MY_DARKEST_SECRET

where you can (and should) replace MY_DARKEST_SECRET with some ``secret word'' of your own choosing.
Your ~/.mpd.conf file must be readable by you and only you. Give the command

chmod 600 ~/.mpd.conf

and the permissions will be the ones MPI requires.

Setting up your hosts file

It would be nice if you could just have a hosts file that lists all of the ugrad linux boxes and be done with it. However, I've found that on different days, different subsets of these boxes are available.

Get a copy of the file findhosts, and give the command

chmod 755 ~/findhosts

to make it executable.
Now, run the command

findhosts

and it will create a hosts file for you. This will take a few seconds to a minute depending on how many of the linux boxes are down (it took 15 seconds when I just ran it). You should probably run this command each time you start using MPI to get today's list of working linux boxes.
Check to see how many running linux boxes you have available by giving the command

wc hosts

The first number printed by wc is the number of lines in hosts. When I just tried it, I found that there are 20 hosts available.

Running your first MPI program

Get the file c3s_mpi.c. This is an MPI implementation of the count-3s program.
Compile the count-3s program:

mpicc -std=c99 c3s_mpi.c -o c3s_mpi
Here's a data file for the count-3s program: randomData

Run the count-3s program:

mpiexec -n 1 -hostfile hosts `pwd`/c3s_mpi `pwd`/randomData

The `pwd`/ in front of c3s_mpi and randomData is because of how mpiexec handles the idea of the current directoy. When MPI runs a process remotely, it apparently doesn't to a cd to the director from which you invoked mpiexec. So, I put the `pwd/ in front of c3s_mpi so the remote processes can find the executable. To make matters more confusing, this wasn't needed on the MPI install we had last year -- ouch!

I also put a `pwd`/ in front of randomData. I tried it with and without the `pwd`/ and it works either way. I'm guessing that this is because the count 3's program has the process of rank 0 read the file, and I'll guess that this is the process that is started from the command line (and therefore already has its directory set to the current one). I left the `pwd`/ in front of randomData for good measure -- it doesn't break anything, and it may prevent problems with the next version of MPI.

When I run the program with the command lne above, I get:

Hi! My name is 0, and I'm running on lin01
My name is 0, and I counted 10133 3's
Number of 3's: 10133
Try with progressively larger numbers for the -n flag. I found that the first time I used MPI, I got a bunch of messages prompts from ssh asking me to confirm that I believe the identities of the various machines. Unfortunately, when it prompts for several machines at the same time, none of them seem to get my affirmative reply. The work-around that I found was to run mpiexec adding one machine at a time:

    % mpiexec -n 1 -hostfile hosts c3s_mpi randomData
    % mpiexec -n 2 -hostfile hosts c3s_mpi randomData
    % mpiexec -n 3 -hostfile hosts c3s_mpi randomData
    % ...

Once you've said yes for a machine, you shouldn't have to do it again; so, this is a one-time hassle. Now, that you've got access to a bunch of hosts, try

mpiexec -n 4 -hostfile hosts c3s_mpi randomData

I get the output

	Hi! My name is 0, and I'm running on lin01
	Hi! My name is 1, and I'm running on lin05
	Hi! My name is 2, and I'm running on lin04
	Hi! My name is 3, and I'm running on lin03
	My name is 1, and I counted 2490 3's
	My name is 2, and I counted 2553 3's
	My name is 3, and I counted 2526 3's
	My name is 0, and I counted 2559 3's
	Number of 3's: 10128

If you make it this far, you should be ready to try use MPI on real examples.

Running your second MPI program

Get the file mpi_timing.c. I wrote this program to measure MPI communication latency and bandwidth. Feel free to read the code, figure out what it does, and try it.

	Hi! My name is 0, and I'm running on lin01
	My name is 0, and I counted 10133 3's
	Number of 3's: 10133