513 lines
21 KiB
HTML
513 lines
21 KiB
HTML
<HTML>
|
|
<HEAD>
|
|
|
|
<TITLE>
|
|
Linux Cluster HOWTO </title>
|
|
</TITLE>
|
|
</HEAD>
|
|
<BODY BGCOLOR=white>
|
|
|
|
<HR>
|
|
<H1> Linux Cluster HOWTO </H1>
|
|
|
|
<H2>Ram Samudrala <CODE>(me@ram.org)</CODE> </H2> v1.5, September 5, 2005
|
|
<HR>
|
|
<EM>How to set up high-performance Linux computing clusters.</EM>
|
|
<HR>
|
|
<P>
|
|
<H2><A NAME="toc1">1.</A> <A HREF="#s1">Introduction</A></H2>
|
|
|
|
<P>
|
|
<H2><A NAME="toc2">2.</A> <A HREF="#s2">Hardware</A></H2>
|
|
|
|
<UL>
|
|
<LI><A NAME="toc2.1">2.1</A> <A HREF="#ss2.1">Node hardware</A>
|
|
<LI><A NAME="toc2.2">2.2</A> <A HREF="#ss2.2">Server hardware</A>
|
|
<LI><A NAME="toc2.3">2.3</A> <A HREF="#ss2.3">Desktop and terminal hardware</A>
|
|
<LI><A NAME="toc2.4">2.4</A> <A HREF="#ss2.4">Miscellaneous/accessory hardware</A>
|
|
<LI><A NAME="toc2.5">2.5</A> <A HREF="#ss2.5">Putting-it-all-together hardware</A>
|
|
<LI><A NAME="toc2.6">2.6</A> <A HREF="#ss2.6">Costs</A>
|
|
</UL>
|
|
<P>
|
|
<H2><A NAME="toc3">3.</A> <A HREF="#s3">Software</A></H2>
|
|
|
|
<UL>
|
|
<LI><A NAME="toc3.1">3.1</A> <A HREF="#ss3.1">Operating system: Linux, of course!</A>
|
|
<LI><A NAME="toc3.2">3.2</A> <A HREF="#ss3.2">Networking software</A>
|
|
<LI><A NAME="toc3.3">3.3</A> <A HREF="#ss3.3">Parallel processing software</A>
|
|
<LI><A NAME="toc3.4">3.4</A> <A HREF="#ss3.4">Costs</A>
|
|
</UL>
|
|
<P>
|
|
<H2><A NAME="toc4">4.</A> <A HREF="#s4">Set up, configuration, and maintenance</A></H2>
|
|
|
|
<UL>
|
|
<LI><A NAME="toc4.1">4.1</A> <A HREF="#ss4.1">Disk configuration</A>
|
|
<LI><A NAME="toc4.2">4.2</A> <A HREF="#ss4.2">Package configuration </A>
|
|
<LI><A NAME="toc4.3">4.3</A> <A HREF="#ss4.3">Operating system installation and maintenance</A>
|
|
<LI><A NAME="toc4.4">4.4</A> <A HREF="#ss4.4">Known hardware issues </A>
|
|
<LI><A NAME="toc4.5">4.5</A> <A HREF="#ss4.5">Known software issues </A>
|
|
</UL>
|
|
<P>
|
|
<H2><A NAME="toc5">5.</A> <A HREF="#s5">Performing tasks on the cluster</A></H2>
|
|
|
|
<UL>
|
|
<LI><A NAME="toc5.1">5.1</A> <A HREF="#ss5.1">Rough benchmarks</A>
|
|
<LI><A NAME="toc5.2">5.2</A> <A HREF="#ss5.2">Uptimes</A>
|
|
</UL>
|
|
<P>
|
|
<H2><A NAME="toc6">6.</A> <A HREF="#s6">Acknowledgements </A></H2>
|
|
|
|
<P>
|
|
<H2><A NAME="toc7">7.</A> <A HREF="#s7">Bibliography </A></H2>
|
|
|
|
<HR>
|
|
<H2><A NAME="s1">1.</A> <A HREF="Cluster-HOWTO.html#toc1">Introduction</A></H2>
|
|
|
|
<P> This document describes how we set up our Linux computing clusters
|
|
for high-performance computing which we need for
|
|
<A HREF="http://compbio.washington.edu">our research</A>. </P>
|
|
<P> Use the information below at your own risk. I disclaim all
|
|
responsibility for anything you may do after reading this HOWTO. The
|
|
latest version of this HOWTO will always be available at
|
|
<A HREF="http://www.ram.org/computing/linux/linux_cluster.html">http://www.ram.org/computing/linux/linux_cluster.html</A>. </P>
|
|
<P> Unlike other documentation that talks about setting up clusters in
|
|
a general way, this is a specific description of how our lab is setup
|
|
and includes not only details the compute aspects, but also the
|
|
desktop, laptop, and public server aspects. This is done mainly for
|
|
local use, but I put it up on the web since I received several e-mail
|
|
messages based on my newsgroup query requesting the same information.
|
|
Even today, as I plan another 64-node cluster, I find that there is a
|
|
dearth of information about exactly how to assemble components to form
|
|
a node that works reliably under Linux that includes information not
|
|
only about the compute nodes, but about hardware that needs to work
|
|
well with the nodes for productive research to happen. The main use
|
|
of this HOWTO as it stands is that it's a report on what kind of
|
|
hardware works well with Linux and what kind of hardware doesn't. </P>
|
|
<HR>
|
|
<H2><A NAME="s2">2.</A> <A HREF="Cluster-HOWTO.html#toc2">Hardware</A></H2>
|
|
|
|
<P> This section covers the hardware choices I've made. Unless noted
|
|
in the
|
|
<A HREF="#known_hardware_issues">known hardware issues</A>
|
|
section, assume that everything works <I>really</I> well. </P>
|
|
<P> Hardware installation is also fairly straight-forward unless
|
|
otherwise noted, with most of the details covered by the manuals. For
|
|
each section, the hardware is listed in the order of purchase (most
|
|
recent is listed first). </P>
|
|
<H2><A NAME="ss2.1">2.1</A> <A HREF="Cluster-HOWTO.html#toc2.1">Node hardware</A>
|
|
</H2>
|
|
|
|
<P> 32 machines have the following setup each: </P>
|
|
<P>
|
|
<UL>
|
|
<LI> 2 XEON 2.66GHZ 533FSB CPUs </LI>
|
|
<LI> Supermicro 6013A-T 1u case and motherboard </LI>
|
|
<LI> 2 512MB PC2100 DDR REG ECC RAM </LI>
|
|
<LI> 1 80GB SEA 7200 SATA HD </LI>
|
|
<LI> 1 250GB SEA 7200 SATA HD </LI>
|
|
</UL>
|
|
</P>
|
|
<P> 32 machines have the following setup each: </P>
|
|
<P>
|
|
<UL>
|
|
<LI> 2 XEON 2.4GHZ 533FSB CPUs </LI>
|
|
<LI> Supermicro X5DPR-1G2 motherboard </LI>
|
|
<LI> 2 512MB PC2100 DDR REG ECC RAM </LI>
|
|
<LI> 1 40GB SEA 7200 HD </LI>
|
|
<LI> 1 120GB SEA 7200 HD </LI>
|
|
<LI> Supermicro Slim 24X CDROM </LI>
|
|
<LI> CSE-812 400 C/B 1U case </LI>
|
|
</UL>
|
|
</P>
|
|
<P> 32 machines have the following setup each: </P>
|
|
<P>
|
|
<UL>
|
|
<LI> 2 AMD Palamino MP XP 2000+ 1.67 GHz CPUs </LI>
|
|
<LI> Asus A7M266-D w/LAN Dual DDR motherboard </LI>
|
|
<LI> 2 Kingston 512mb PC2100 DDR-266MHz REG ECC RAM </LI>
|
|
<LI> 1 41 GB Maxtor 7200rpm ATA100 HD </LI>
|
|
<LI> 1 120 GB Maxtor 5400rpm ATA100 HD </LI>
|
|
<LI> Asus CD-A520 52x CDROM </LI>
|
|
<LI> 1.44mb floppy drive </LI>
|
|
<LI> ATI Expert 2000 Rage 128 32mb </LI>
|
|
<LI> IN-WIN P4 300ATX Mid Tower case </LI>
|
|
<LI> Enermax P4-430ATX power supply </LI>
|
|
</UL>
|
|
</P>
|
|
<P> 32 machines have the following setup each: </P>
|
|
<P>
|
|
<UL>
|
|
<LI> 2 AMD Palamino MP XP 1800+ 1.53 GHz CPUs </LI>
|
|
<LI> Tyan S2460 Dual Socket-A/MP motherboard </LI>
|
|
<LI> Kingston 512mb PC2100 DDR-266MHz REG ECC RAM </LI>
|
|
<LI> 1 20 GB Maxtor UDMA/100 7200rpm HD </LI>
|
|
<LI> 1 120 GB Maxtor 5400rpm ATA100 HD </LI>
|
|
<LI> Asus CD-A520 52x CDROM </LI>
|
|
<LI> 1.44mb floppy drive </LI>
|
|
<LI> ATI Expert 98 8mb AGP video card </LI>
|
|
<LI> IN-WIN P4 300ATX Mid Tower case </LI>
|
|
<LI> Intel PCI PRO-100 10/100Mbps network card </LI>
|
|
<LI> Enermax P4-430ATX power supply </LI>
|
|
</UL>
|
|
</P>
|
|
<P> 32 machines have the following setup each: </P>
|
|
<P>
|
|
<UL>
|
|
<LI> 2 Pentium III 1 GHz Intel CPUs </LI>
|
|
<LI> Supermicro 370 DLE Dual PIII-FCPGA motherboard </LI>
|
|
<LI> 2 256 MB 168-pin PC133 Registered ECC Micron RAM </LI>
|
|
<LI> 1 20 GB Maxtor ATA/66 5400 RPM HD </LI>
|
|
<LI> 1 40 GB Maxtor UDMA/100 7200 RPM HD </LI>
|
|
<LI> Asus CD-S500 50x CDROM </LI>
|
|
<LI> 1.4 MB floppy drive </LI>
|
|
<LI> ATI Expert 98 8 MB PCI video card </LI>
|
|
<LI> IN-WIN P4 300ATX Mid Tower case </LI>
|
|
</UL>
|
|
</P>
|
|
<H2><A NAME="ss2.2">2.2</A> <A HREF="Cluster-HOWTO.html#toc2.2">Server hardware</A>
|
|
</H2>
|
|
|
|
<P> Two servers for external use (dissemination of information) with
|
|
the following setups:</P>
|
|
<P>
|
|
<UL>
|
|
<LI> 2 AMD Opteron 240 1.4 GHz CPUs </LI>
|
|
<LI> RIOWORKS HDAMB DUAL OPTERON motherboard </LI>
|
|
<LI> 4 KINGSTON 512MB PC3200 REG ECC RAM </LI>
|
|
<LI> 80GB MAX 7200 UDMA 133 HD </LI>
|
|
<LI> 6 200GB WD 7200 8MB HD </LI>
|
|
<LI> ASUS 52X CD-A520 CDROM </LI>
|
|
<LI> 1.44mb floppy drive </LI>
|
|
<LI> Antec 4U22ATX550EPS 4u case </LI>
|
|
</UL>
|
|
</P>
|
|
<P>
|
|
<UL>
|
|
<LI> 2 AMD Palamino MP XP 2000+ 1.67 GHz CPUs </LI>
|
|
<LI> Asus A7M266-D w/LAN Dual DDR </LI>
|
|
<LI> 4 Kingston 512mb PC2100 DDR-266MHz REG ECC RAM </LI>
|
|
<LI> Asus CD-A520 52x CDROM </LI>
|
|
<LI> 1 41 GB Maxtor 7200rpm ATA100 HD </LI>
|
|
<LI> 6 120 GB Maxtor 5400rpm ATA100 HD </LI>
|
|
<LI> 1.44mb floppy drive </LI>
|
|
<LI> ATI Expert 2000 Rage 128 32mb </LI>
|
|
<LI> IN-WIN P4 300ATX mid tower case </LI>
|
|
<LI> Enermax P4-430ATX power supply </LI>
|
|
</UL>
|
|
</P>
|
|
<H2><A NAME="ss2.3">2.3</A> <A HREF="Cluster-HOWTO.html#toc2.3">Desktop and terminal hardware</A>
|
|
</H2>
|
|
|
|
<P> We have identified at least two kinds of users of our cluster:
|
|
those that need (i.e., take advantage of) permanent local processing
|
|
power and disk space in conjunction with the cluster to speed up
|
|
processing, and those that just need only the cluster processing
|
|
power. The former are assigned "desktops" which are essentially
|
|
high-performance machines, and the latter are assigned dumb
|
|
"terminals". Our desktops are usually dual or quad processor machines
|
|
with the current high-end CPU being a 1.6 GHz Opteron, having as much
|
|
as 10 GB of RAM, and over 1 TB of local disk space. Our terminals are
|
|
essentially machines where a user can log in and then run jobs on our
|
|
farm. In this setup, people may also use laptops as dumb terminals. </P>
|
|
<H2><A NAME="ss2.4">2.4</A> <A HREF="Cluster-HOWTO.html#toc2.4">Miscellaneous/accessory hardware</A>
|
|
</H2>
|
|
|
|
<P> We generally use/prefer Viewsonic monitors, Microsoft Intellimouse
|
|
mice, and Microsoft Natural keyboards. These generally have worked
|
|
quite reliably for us. </P>
|
|
<H2><A NAME="ss2.5">2.5</A> <A HREF="Cluster-HOWTO.html#toc2.5">Putting-it-all-together hardware</A>
|
|
</H2>
|
|
|
|
<P> For visual access to the nodes, we initially used to use KVM
|
|
switches with a cheap monitor to connect up and "look" at all the
|
|
machines. While this was a nice solution, it did not scale. We
|
|
currently wheel a small monitor around and hook up cables as needed.
|
|
What we need is a small hand held monitor that can plug into the back
|
|
of the PC (operated with a stylus, like the Palm). </P>
|
|
<P> For networking, we generally use Netgear and Cisco switches. </P>
|
|
<H2><A NAME="ss2.6">2.6</A> <A HREF="Cluster-HOWTO.html#toc2.6">Costs</A>
|
|
</H2>
|
|
|
|
<P> Our vendor is Hard Drives Northwest (
|
|
<A HREF="http://www.hdnw.com">http://www.hdnw.com</A>). For each
|
|
compute node in our cluster (containing two processors), we paid about
|
|
$1500-$2000, including taxes. Generally, our goal is to keep the cost
|
|
of each processor to below $1000 (including housing it). </P>
|
|
<HR>
|
|
<H2><A NAME="s3">3.</A> <A HREF="Cluster-HOWTO.html#toc3">Software</A></H2>
|
|
|
|
<H2><A NAME="ss3.1">3.1</A> <A HREF="Cluster-HOWTO.html#toc3.1">Operating system: Linux, of course!</A>
|
|
</H2>
|
|
|
|
<P> The following kernels and distributions are what are being used:</P>
|
|
<P>
|
|
<UL>
|
|
<LI> Kernel 2.2.16-22, distribution KRUD 7.0</LI>
|
|
<LI> Kernel 2.4.9-7, distribution KRUD 7.2</LI>
|
|
<LI> Kernel 2.4.18-10, distribution KRUD 7.3</LI>
|
|
<LI> Kernel 2.4.20-13.9, distribution KRUD 9.0</LI>
|
|
<LI> Kernel 2.4.22-1.2188, distribution KRUD 2004-05</LI>
|
|
</UL>
|
|
</P>
|
|
<P>These distributions work very well for us since updates are sent to us
|
|
on CD and there's no reliance on an external network connection for
|
|
updates. They also seem "cleaner" than the regular Red Hat
|
|
distributions, and the setup is extremely stable. </P>
|
|
<H2><A NAME="ss3.2">3.2</A> <A HREF="Cluster-HOWTO.html#toc3.2">Networking software</A>
|
|
</H2>
|
|
|
|
<P> We use Shorewall 1.3.14a ((
|
|
<A HREF="http://www.shorewall.net">http://www.shorewall.net</A>) for the firewall. </P>
|
|
<H2><A NAME="ss3.3">3.3</A> <A HREF="Cluster-HOWTO.html#toc3.3">Parallel processing software</A>
|
|
</H2>
|
|
|
|
<P> We use our own software for parallelising applications but have
|
|
experimented with
|
|
<A HREF="http://www.csm.ornl.gov/pvm/pvm_home.html">PVM</A> and
|
|
<A HREF="http://www-unix.mcs.anl.gov/mpi/mpich/">MPI</A>. In my view, the overhead for these pre-packaged programs
|
|
is too high. I recommend writing application-specific code for the
|
|
tasks you perform (that's one person's view). </P>
|
|
<H2><A NAME="ss3.4">3.4</A> <A HREF="Cluster-HOWTO.html#toc3.4">Costs</A>
|
|
</H2>
|
|
|
|
<P> Linux and most software that run on Linux are freely
|
|
copiable. </P>
|
|
<HR>
|
|
<H2><A NAME="s4">4.</A> <A HREF="Cluster-HOWTO.html#toc4">Set up, configuration, and maintenance</A></H2>
|
|
|
|
<H2><A NAME="ss4.1">4.1</A> <A HREF="Cluster-HOWTO.html#toc4.1">Disk configuration</A>
|
|
</H2>
|
|
|
|
<P> This section describes disk partitioning strategies. Our goal is
|
|
to keep the virtual structures of the machines organised such that
|
|
they are all logical. We're finding that the physical mappings to the
|
|
logical structures are not sustainable as hardware and software
|
|
(operating system) change. Currently, our strategy is as follows: </P>
|
|
<P>
|
|
<BLOCKQUOTE><CODE>
|
|
<PRE>
|
|
farm/cluster machines:
|
|
|
|
partition 1 on system disk - swap (2 * RAM)
|
|
partition 2 on system disk - / (remaining disk space)
|
|
partition 1 on additional disk - /maxa (total disk)
|
|
|
|
servers:
|
|
|
|
partition 1 on system disk - swap (2 * RAM)
|
|
partition 2 on system disk - / (4-8 GB)
|
|
partition 3 on system disk - /home (remaining disk space)
|
|
partition 1 on additional disk 1 - /maxa (total disk)
|
|
partition 1 on additional disk 2 - /maxb (total disk)
|
|
partition 1 on additional disk 3 - /maxc (total disk)
|
|
partition 1 on additional disk 4 - /maxd (total disk)
|
|
partition 1 on additional disk 5 - /maxe (total disk)
|
|
partition 1 on additional disk 6 - /maxf (total disk)
|
|
partition 1 on additional disk(s) - /maxg (total disk space)
|
|
|
|
desktops:
|
|
|
|
partition 1 on system disk - swap (2 * RAM)
|
|
partition 2 on system disk - / (4-8 GB)
|
|
partition 3 on system disk - /spare (remaining disk space)
|
|
partition 1 on additional disk 1 - /maxa (total disk)
|
|
partition 1 on additional disk(s) - /maxb (total disk space)
|
|
</PRE>
|
|
</CODE></BLOCKQUOTE>
|
|
</P>
|
|
<P> Note that in the case of servers and desktops, maxg and maxb can
|
|
be a single disk or a conglomeration of disks. </P>
|
|
<H2><A NAME="ss4.2">4.2</A> <A HREF="Cluster-HOWTO.html#toc4.2">Package configuration </A>
|
|
</H2>
|
|
|
|
<P> Install a minimal set of packages for the farm. Users are allowed
|
|
to configure desktops as they wish, provided the virtual structure is
|
|
kept the same described above is kept the same. </P>
|
|
<H2><A NAME="ss4.3">4.3</A> <A HREF="Cluster-HOWTO.html#toc4.3">Operating system installation and maintenance</A>
|
|
</H2>
|
|
|
|
<H3>Personal cloning strategy</H3>
|
|
|
|
<P> I believe in having a completely distributed system. This means
|
|
each machine contains a copy of the operating system. Installing the
|
|
OS on each machine manually is cumbersome. To optimise this process,
|
|
what I do is first set up and install one machine exactly the way I
|
|
want to. I then create a tar and gzipped file of the entire system
|
|
and place it on a bootable CD-ROM which I then clone on each machine
|
|
in my cluster. </P>
|
|
<P> The commands I use to create the tar file are as follows: </P>
|
|
<P>
|
|
<BLOCKQUOTE><CODE>
|
|
<PRE>
|
|
tar -czvlps --same-owner --atime-preserve -f /maxa/slash.tgz /
|
|
</PRE>
|
|
</CODE></BLOCKQUOTE>
|
|
</P>
|
|
<P> I use a script called <CODE>go</CODE> that takes a machine
|
|
number as its argument and untars the <CODE>slash.tgz</CODE> file on the
|
|
CD-ROM and replaces the hostname and IP address in the appropriate
|
|
locations. A version of the <CODE>go</CODE> script and the input files for
|
|
it can be accessed at:
|
|
<A HREF="http://www.ram.org/computing/linux/cluster/">http://www.ram.org/computing/linux/linux/cluster/</A>. This script
|
|
will have to be edited based on your cluster design. </P>
|
|
<P> To make this work, I use Martin Purschke's Custom Rescue Disk
|
|
(
|
|
<A HREF="http://www.phenix.bnl.gov/~purschke/RescueCD/">http://www.phenix.bnl.gov/~purschke/RescueCD/</A>) to create a
|
|
bootable CD image containing the .tgz file representing the cloned
|
|
system, as well as the <CODE>go</CODE> script and other associated files.
|
|
This is burned onto a CD-ROM. </P>
|
|
<P> There are several documents that describe how to create your own
|
|
custom bootable CD, including the Linux Bootdisk HOWTO (
|
|
<A HREF="http://www.linuxdoc.org/HOWTO/Bootdisk-HOWTO/">http://www.linuxdoc.org/HOWTO/Bootdisk-HOWTO/</A>), which also
|
|
contains links to other pre-made boot/root disks. </P>
|
|
<P> Thus you have a system where all you have to do is insert a CDROM,
|
|
turn on the machine, have a cup of coffee (or a can of coke) and come
|
|
back to see a full clone. You then repeat this process for as many
|
|
machines as you have. This procedure has worked extremely well for me
|
|
and if you have someone else actually doing the work (of inserting and
|
|
removing CD-ROMs) then it's ideal. In my system, I specify the IP
|
|
address by specifying the number of the machine, but this could be
|
|
completely automated through the use of DHCP. </P>
|
|
<P> Rob Fantini (
|
|
<A HREF="mailto:rob@fantinibakery.com">rob@fantinibakery.com</A>) has contributed modifications of the
|
|
scripts above that he used for cloning a Mandrake 8.2 system
|
|
accessible at
|
|
<A HREF="http://www.ram.org/computing/linux/cluster/fantini_contribution.tgz">http://www.ram.org/computing/linux/cluster/fantini_contribution.tgz</A>.</P>
|
|
<H3>Cloning and maintenance packages</H3>
|
|
|
|
<H3>FAI </H3>
|
|
|
|
<P> FAI (
|
|
<A HREF="http://www.informatik.uni-koeln.de/fai/">http://www.informatik.uni-koeln.de/fai/</A>) is an automated
|
|
system to install a Debian GNU/Linux operating system on a PC
|
|
cluster. You can take one or more virgin PCs, turn on the power and
|
|
after a few minutes Linux is installed, configured and running on the
|
|
whole cluster, without any interaction necessary.</P>
|
|
|
|
<H3>SystemImager </H3>
|
|
|
|
<P> SystemImager (
|
|
<A HREF="http://systemimager.org">http://systemimager.org</A>) is software that automates Linux
|
|
installs, software distribution, and production deployment. </P>
|
|
<H3>DHCP vs. hard-coded IP addresses</H3>
|
|
|
|
<P> If you have DHCP set up, then you don't need to reset the IP
|
|
address and that part of it can be removed from the <CODE>go</CODE>
|
|
script. </P>
|
|
<P> DHCP has the advantage that you don't muck around with IP
|
|
addresses at all provided the DHCP server is configured
|
|
appropriately. It has the disadvantage that it relies on a centralised
|
|
server (and like I said, I tend to distribute things as much as
|
|
possible). Also, linking hardware ethernet addresses to IP addresses can
|
|
make it inconvenient if you wish to replace machines or change
|
|
hostnames routinely. </P>
|
|
<H2><A NAME="known_hardware_issues"></A> <A NAME="ss4.4">4.4</A> <A HREF="Cluster-HOWTO.html#toc4.4">Known hardware issues </A>
|
|
</H2>
|
|
|
|
<P> The hardware in general has worked really well for us. Specific
|
|
issues are listed below: </P>
|
|
<P> The AMD dual 1.2 GHz machines run really hot. Two of them in a
|
|
room increase the temperature significantly. Thus while they might be
|
|
okay as desktops, the cooling and power consumption when using them as
|
|
part of a large cluster is a consideration. The AMD Palmino
|
|
configuration described previously seems to work really well, but I
|
|
definitely recommend getting two fans in the case--this solved all our
|
|
instability problems. </P>
|
|
<H2><A NAME="known_software_issues"></A> <A NAME="ss4.5">4.5</A> <A HREF="Cluster-HOWTO.html#toc4.5">Known software issues </A>
|
|
</H2>
|
|
|
|
<P> Some tar executables apparently don't create a tar file the nice
|
|
way they're supposed to (especially in terms of referencing and
|
|
de-referencing symbolic links). The solution to this I've found is to
|
|
use a tar executable that does, like the one from RedHat 7.0. </P>
|
|
<HR>
|
|
<H2><A NAME="s5">5.</A> <A HREF="Cluster-HOWTO.html#toc5">Performing tasks on the cluster</A></H2>
|
|
|
|
<P> This section is still being developed as the usage on my cluster
|
|
evolves, but so far we tend to write our own sets of message passing
|
|
routines to communicate between processes on different machines. </P>
|
|
<P> Many applications, particularly in the computational genomics
|
|
areas, are massively and trivially parallelisable, meaning that
|
|
perfect distribution can be achieved by spreading tasks equally across
|
|
the machines (for example, when analysing a whole genome using a
|
|
technique that operates on a single gene/protein, each processor can
|
|
work on one gene/protein at a time independent of all the other
|
|
processors). </P>
|
|
<P> So far we have not found the need to use a professional queueing
|
|
system, but obviously that is highly dependent on the type of
|
|
applications you wish to run. </P>
|
|
<H2><A NAME="ss5.1">5.1</A> <A HREF="Cluster-HOWTO.html#toc5.1">Rough benchmarks</A>
|
|
</H2>
|
|
|
|
<P> For the single most important program we run (our <I>ab
|
|
initio</I> protein folding simulation program), using the Pentium 3 1
|
|
GHz processor machine as a frame of reference, on average: </P>
|
|
<P>
|
|
<BLOCKQUOTE><CODE>
|
|
<PRE>
|
|
Xeon 1.7 GHz processor is about 22% slower
|
|
Athlon 1.2 GHz processor is about 36% faster
|
|
Athlon 1.5 GHz processor is about 50% faster
|
|
Athlon 1.7 GHz processor is about 63% faster
|
|
Xeon 2.4 GHz processor is about 45% faster
|
|
Xeon 2.7 GHz processor is about 80% faster
|
|
Opteron 1.4 GHz processor is about 70% faster
|
|
Opteron 1.6 GHz processor is about 88% faster
|
|
</PRE>
|
|
</CODE></BLOCKQUOTE>
|
|
</P>
|
|
<P> Yes, the Athlon 1.5 GHz is faster than the Xeon 1.7 GHz since the
|
|
Xeon executes only six instructions per clock (IPC) whereas the Athlon
|
|
executes nine IPC (you do the math!). This is however an highly
|
|
non-rigourous comparison since the executables were each compiled on
|
|
the machines (so the quality of the math libraries for example will
|
|
have an impact) and the supporting hardware is different. </P>
|
|
<H2><A NAME="ss5.2">5.2</A> <A HREF="Cluster-HOWTO.html#toc5.2">Uptimes</A>
|
|
</H2>
|
|
|
|
<P> These machines are incredibly stable both in terms of hardware and
|
|
software once they have been debugged (usually some in a new batch of
|
|
machines have hardware problems), running constantly under very heavy
|
|
loads. One common example is given below. Reboots have generally
|
|
occurred when a circuit breaker is tripped.</P>
|
|
<P>
|
|
<BLOCKQUOTE><CODE>
|
|
<PRE>
|
|
2:29pm up 495 days, 1:04, 2 users, load average: 4.85, 7.15, 7.72
|
|
</PRE>
|
|
</CODE></BLOCKQUOTE>
|
|
</P>
|
|
<HR>
|
|
<H2><A NAME="s6">6.</A> <A HREF="Cluster-HOWTO.html#toc6">Acknowledgements </A></H2>
|
|
|
|
<P> The following people have been helpful in getting this HOWTO
|
|
done: </P>
|
|
<P>
|
|
<UL>
|
|
<LI> Michal Guerquin (
|
|
<A HREF="mailto:mikeg@u.washington.edu">Michal Guerquin</A>)</LI>
|
|
<LI> Michael Levitt (
|
|
<A HREF="mailto:michael.levitt@stanford.edu">Michael Levitt</A>)</LI>
|
|
</UL>
|
|
</P>
|
|
<HR>
|
|
<H2><A NAME="references"></A> <A NAME="s7">7.</A> <A HREF="Cluster-HOWTO.html#toc7">Bibliography </A></H2>
|
|
|
|
<P> The following documents may prove useful to you---they are links
|
|
to sources that make use of high-performance computing clusters:</P>
|
|
<P>
|
|
<UL>
|
|
<LI>
|
|
<A HREF="http://www.ram.org/research/research.html">Ram Samudrala's research page (which describes the kind of research done with these clusters)</A>
|
|
</LI>
|
|
<LI>
|
|
<A HREF="http://www.ram.org/computing/ramp/ramp.html">RAMP web page</A>
|
|
</LI>
|
|
<LI>
|
|
<A HREF="http://www.ram.org/computing/rambin/rambin.html">RAMBIN web page</A></LI>
|
|
</UL>
|
|
</P>
|
|
<HR>
|
|
</BODY>
|
|
</HTML>
|