From ea177ceb5488167d1db58f021d93adc62ef83f28 Mon Sep 17 00:00:00 2001 From: gferg <> Date: Tue, 9 Apr 2002 14:00:22 +0000 Subject: [PATCH] updated --- LDP/howto/linuxdoc/Cluster-HOWTO.sgml | 296 +++++++++++++++++++------- 1 file changed, 216 insertions(+), 80 deletions(-) diff --git a/LDP/howto/linuxdoc/Cluster-HOWTO.sgml b/LDP/howto/linuxdoc/Cluster-HOWTO.sgml index a98ba733..0084a9e5 100644 --- a/LDP/howto/linuxdoc/Cluster-HOWTO.sgml +++ b/LDP/howto/linuxdoc/Cluster-HOWTO.sgml @@ -5,7 +5,7 @@ Linux Cluster HOWTO Ram Samudrala (me@ram.org) - v0.3, August 21, 2001 + v0.92, April 8, 2002 How to set up high-performance Linux computing clusters. @@ -33,8 +33,11 @@ and includes not only details the compute aspects, but also the desktop, laptop, and public server aspects. This is done mainly for local use, but I put it up on the web since I received several e-mail messages based on my newsgroup query requesting the same information. -The main use as it stands is that it's a report on what kind of -hardware works well with Linux and what kind of hardware doesn't.

+Even today, as I plan another 64-node cluster, I find there is a +dearth of information about exactly how to assemble components to form +a node that works reliably under Linux. The main use of this HOWTO as +it stands is that it's a report on what kind of hardware works well +with Linux and what kind of hardware doesn't.

@@ -44,11 +47,14 @@ hardware works well with Linux and what kind of hardware doesn't.

Hardware -

This section covers the hardware choices I've made. Unless noted, -assume that everything works really well.

+

This section covers the hardware choices I've made. Unless noted +in the +section, assume that everything works really well.

Hardware installation is also fairly straight-forward unless -otherwise noted, with most of the details covered by the manuals.

+otherwise noted, with most of the details covered by the manuals. For +each section, the hardware is listed in the order of purchase (most +recent is listed first).

@@ -57,15 +63,31 @@ otherwise noted, with most of the details covered by the manuals.

32 machines have the following setup each: - 2 Pentium III 1 GHz Intel CPUs - Supermicro 370 DLE Dual PIII-FCPGA motherboard - 2 256 MB 168-pin PC133 Registered ECC Micron RAM - 1 20 GB Maxtor ATA/66 5400 RPM HD - 1 40 GB Maxtor UDMA/100 7200 RPM HD - Asus CD-S500 50x CDROM - 1.4 MB floppy drive - ATI Expert 98 8 MB PCI video card - Mid-tower case + 2 AMD Palamino MP XP 1800+ 1.53 GHz CPUs + Tyan S2460 Dual Socket-A/MP motherboard + Kingston 512mb PC2100 DDR-266MHz REG ECC RAM + 1 20 GB Maxtor UDMA/100 7200rpm HD + 1 120 GB Maxtor 5400rpm ATA100 HD + Asus CD-A520 52x CDROM + 1.44mb floppy drive + ATI Expert 98 8mb AGP video card + IN-WIN P4 300ATX Mid Tower case + Intel PCI PRO-100 10/100Mbps network card + +

+ +

32 machines have the following setup each: + + + 2 Pentium III 1 GHz Intel CPUs + Supermicro 370 DLE Dual PIII-FCPGA motherboard + 2 256 MB 168-pin PC133 Registered ECC Micron RAM + 1 20 GB Maxtor ATA/66 5400 RPM HD + 1 40 GB Maxtor UDMA/100 7200 RPM HD + Asus CD-S500 50x CDROM + 1.4 MB floppy drive + ATI Expert 98 8 MB PCI video card + IN-WIN P4 300ATX Mid Tower case

@@ -75,18 +97,19 @@ otherwise noted, with most of the details covered by the manuals.

Server hardware -

1 external server with the following setup: +

1 server for external use (dissemination of information) with the +following setup: - 2 Pentium III 1 GHz Intel CPUs - Supermicro 370 DLE Dual PIII-FCPGA motherboard - 2 256 MB 168-pin PC133 Registered ECC Micron RAM - 1 20 GB Maxtor ATA/66 5400 RPM HD - 2 40 GB Maxtor UDMA/100 7200 RPM HD - Asus CD-S500 50x CDROM - 1.4 MB floppy drive - ATI Expert 98 8 MB PCI video card - Full-tower case + 2 Pentium III 1 GHz Intel CPUs + Supermicro 370 DLE Dual PIII-FCPGA motherboard + 2 256 MB 168-pin PC133 Registered ECC Micron RAM + 1 20 GB Maxtor ATA/66 5400 RPM HD + 2 40 GB Maxtor UDMA/100 7200 RPM HD + Asus CD-S500 50x CDROM + 1.4 MB floppy drive + ATI Expert 98 8 MB PCI video card + Full-tower case with 300W PS

@@ -96,69 +119,112 @@ otherwise noted, with most of the details covered by the manuals.

Desktop hardware +

1 desktop with the following setup: + + + 2 Intel Xeon 1.7 GHz 256K 400FS + Supermicro P4DCE Dual Xeon motherboard + 4 256mb RAMBUS 184-Pin 800 MHz memory + 2 120 GB Maxtor ATA/100 5400 RPM HD + 1 60 GB Maxtor ATA/100 7200 RPM HD + 52X Asus CD-A520 INT IDE CDROM + 1.4 MB floppy drive + Leadtex 64 MB GF2 MX400 AGP + Creative SB LIVE Value PCI 5.1 + Microsoft Natural Keyboard + Microsoft Intellimouse Explorer + Supermicro SC760 full-tower case with 400W PS + + +

2 desktops with the following setup: + + + 2 AMD K7 1.2g/266 MP Socket A CPU + Tyan S2462NG Dual Socket A motherboard + 4 256mb PC2100 REG ECC DDR-266Mhz + 3 40 GB Maxtor UDMA/100 7200 RPM HD + 50X Asus CD-A520 INT IDE CDROM + 1.4 MB floppy drive + Chaintech Geforce2 MX200 32mg AGP + Creative SB LIVE Value PCI + Microsoft Natural Keyboard + Microsoft Intellimouse Explorer + Full-tower case with 300W PS + +

+ +

2 desktops with the following setup: + + + 2 Pentium III 1 GHz Intel CPUs + Supermicro 370 DLE Dual PIII-FCPGA motherboard + 4 256 MB 168-pin PC133 Registered ECC Micron RAM + 3 40 GB Maxtor UDMA/100 7200 RPM HD + Asus CD-S500 50x CDROM + 1.4 MB floppy drive + Jaton Nvidia TNT2 32mb PCI + Creative SB LIVE Value PCI + Microsoft Natural Keyboard + Microsoft Intellimouse Explorer + Full-tower case with 300W PS + +

+ +

2 desktops with the following setup: + + + 2 Pentium III 1 GHz Intel CPUs + Supermicro 370 DLE Dual PIII-FCPGA motherboard + 4 256 MB 168-pin PC133 Registered ECC Micron RAM + 3 40 GB Maxtor UDMA/100 7200 RPM HD + Mitsumi 8x/4x/32x CDRW + 1.4 MB floppy drive + Jaton Nvidia TNT2 32mb PCI + Creative SB LIVE Value PCI + Microsoft Natural Keyboard + Microsoft Intellimouse Explorer + Full-tower case with 300W PS + +

+

4 desktops with the following setup: - 2 Pentium III 1 GHz Intel CPUs - Supermicro 370 DE6 Dual PIII-FCPGA motherboard - 4 256 MB 168-pin PC133 Registered ECC Micron RAM - 3 40 GB Maxtor UDMA/100 7200 RPM HD - Ricoh 32x12x10 CDRW/DVD Combo EIDE - 1.4 MB floppy drive - Asus V7700 64mb GeForce2-GTS AGP video card - Creative SB Live Platinum 5.1 sound card - Microsoft Natural Keyboard - Microsoft Intellimouse Explorer - Full-tower case + 2 Pentium III 1 GHz Intel CPUs + Supermicro 370 DE6 Dual PIII-FCPGA motherboard + 4 256 MB 168-pin PC133 Registered ECC Micron RAM + 3 40 GB Maxtor UDMA/100 7200 RPM HD + Ricoh 32x12x10 CDRW/DVD Combo EIDE + 1.4 MB floppy drive + Asus V7700 64mb GeForce2-GTS AGP video card + Creative SB Live Platinum 5.1 sound card + Microsoft Natural Keyboard + Microsoft Intellimouse Explorer + Full-tower case with 300W PS +

-

2 desktops with the following setup: + - - 2 Pentium III 1 GHz Intel CPUs - Supermicro 370 DLE Dual PIII-FCPGA motherboard - 4 256 MB 168-pin PC133 Registered ECC Micron RAM - 3 40 GB Maxtor UDMA/100 7200 RPM HD - Mitsumi 8x/4x/32x CDRW - 1.4 MB floppy drive - Jaton Nvidia TNT2 32mb PCI - Creative SB LIVE Value PCI - Microsoft Natural Keyboard - Microsoft Intellimouse Explorer - Full-tower case - -

+ -

2 desktops with the following setup: - - - 2 Pentium III 1 GHz Intel CPUs - Supermicro 370 DLE Dual PIII-FCPGA motherboard - 4 256 MB 168-pin PC133 Registered ECC Micron RAM - 3 40 GB Maxtor UDMA/100 7200 RPM HD - Asus CD-S500 50x CDROM - 1.4 MB floppy drive - Jaton Nvidia TNT2 32mb PCI - Creative SB LIVE Value PCI - Microsoft Natural Keyboard - Microsoft Intellimouse Explorer - Full-tower case - -

+ Miscellaneous/accessory hardware

Backup: - 2 Sony 20/40 GB DSS4 SE LVD DAT + 2 Sony 20/40 GB DSS4 SE LVD DAT

Monitors: - 4 21" Sony CPD-G500 .24mm monitor - 2 18" Viewsonic VP-181 TFT-LCD monitor + 1 22" Viewsonic P220F 0.25-0.27m monitor + 4 21" Sony CPD-G500 .24mm monitor + 2 18" Viewsonic VP181 LCD monitor + 1 17" Viewsonic VE170 LCD monitor

@@ -172,9 +238,9 @@ otherwise noted, with most of the details covered by the manuals.

at all the machines: - 15" .28dp XLN CTL Monitor - 3 Belkin Omniview 16-Port Pro Switches - 40 KVM cables + 15" .28dp XLN CTL Monitor + 3 Belkin Omniview 16-Port Pro Switches + 40 KVM cables

@@ -186,7 +252,8 @@ more monitor switches/KVM cables.

Networking is important: - 1 Cisco Catalyst 3448 XL Enterprise Edition network switch. + 1 Cisco Catalyst 3448 XL Enterprise Edition 48 port network switch. + 1 Netgear FS524 24 port network switch

@@ -199,7 +266,7 @@ more monitor switches/KVM cables.

Our vendor is Hard Drives Northwest (). For each compute node in our cluster (containing two processors), we paid about -$1500, including taxes. Generally, our goal is to keep each node to +$1500-$2000, including taxes. Generally, our goal is to keep each node to below $2000.00 (which is what our desktop machines cost).

@@ -216,10 +283,18 @@ below $2000.00 (which is what our desktop machines cost).

Linux, of course! -

Specfically we use 2.2.17-14 kernel based on the KRUD 7.0 -distribution. We use our own software for parallising applications but -have experimented with PVM and MPI. In my view, the overhead for these -pre-packaged programs is too high.

+

We use Linux systems with a 2.4.9-7 kernel based on the KRUD 7.2 +distribution, and 2.2.17-14 kernel based on the KRUD 7.0 +distribution. These distributions work very well for us since updates +are sent to us on CD and there's no reliance on an external network +connection for updates. They also seem "cleaner" than the regular Red +Hat distributions.

+ +

We use our own software for parallelising applications +but have experimented with PVM and MPI. In my view, the overhead for +these pre-packaged programs is too high. I recommend writing +application-specific code for the tasks you perform (that's one +person's view).

@@ -342,6 +417,13 @@ many machines as you have. This procedure has worked extremely well for me and if you have someone else actually doing the work (of inserting and removing CD-ROMs) then it's ideal.

+

+has contributed modifications of the scripts above that he used for +cloning a Mandrake 8.2 system accessible at . +

+ @@ -368,12 +450,29 @@ hostnames routinely.

+ Known hardware issues + + + Performing tasks on the cluster + +

This section is still being developed as the usage on my cluster evolves, but so far we tend to write our own sets of message passing routines to communicate between processes on different machines.

@@ -385,10 +484,47 @@ the machines (for example, when analysing a whole genome using a single gene technique, each processor can work on one gene at a time independent of all the other processors).

-

So far we have not found the need to use a professional queing +

So far we have not found the need to use a professional queueing system, but obviously that is highly dependent on the type of applications you wish to run.

+ + + Rough benchmarks + +

For the single most important program we run (our ab initio +protein folding simulation program), using the Pentium 3 1 GHz +processor machine as a reference frame, the Athlon 1.2 GHz processor +machine is about 16% faster on average, the Pentium 4 1.7 GHz machine +is about 25-32% faster on average, and the Athlon 1.5 GHz processor is +about 80% faster on average (yes, the Athlon 1.5 GHz is faster than +the Xeon 1.7 GHz since the Xeon executes only six instructions per +clock (IPC) whereas the Athlon executes nine IPC (you do the math!)). +

+ + + + + + Uptimes + +

These machines are incredibly stable both in terms of hardware and +software once they have been debugged (usually some in a new batch of +machines have hardware problems). Reboots have generally occurred +when a circuit breaker is tripped. The first machine I installed has +been up since its birth! + + +~ ram@fp1 % uptime + 4:49am up 374 days, 2:47, 1 user, load average: 2.08, 2.02, 2.01 + + +

+ + + + +