old-www/HOWTO/Parallel-Processing-HOWTO-6...

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
 <META NAME="GENERATOR" CONTENT="SGML-Tools 1.0.9">
 <TITLE>Linux Parallel Processing HOWTO: Of General Interest</TITLE>
 <LINK HREF="Parallel-Processing-HOWTO-5.html" REL=previous>
 <LINK HREF="Parallel-Processing-HOWTO.html#toc6" REL=contents>
</HEAD>
<BODY>
Next
<A HREF="Parallel-Processing-HOWTO-5.html">Previous</A>
<A HREF="Parallel-Processing-HOWTO.html#toc6">Contents</A>
<HR>
<H2><A NAME="s6">6. Of General Interest</A></H2>

<P>
<P>The material covered in this section applies to all four parallel
processing models for Linux.
<P>
<H2><A NAME="ss6.1">6.1 Programming Languages And Compilers</A>
</H2>

<P>
<P>I am primarily known as a compiler researcher, so I'd like to be able
to say that there are lots of really great compilers automatically
generating efficient parallel code for Linux systems.  Unfortunately,
the truth is that it is hard to beat the performance obtained by
expressing your parallel program using various explicit communication
and other parallel operations within C code that is compiled by GCC.
<P>The following language/compiler projects represent some of the best
efforts toward producing reasonably efficient code from high-level
languages.  Generally, each is reasonably effective for the kinds of
programming tasks it targets, but none is the powerful general-purpose
language and compiler system that will make you forever stop writing C
programs to compile with GCC...  which is fine.  Use these languages
and compilers as they were intended, and you'll be rewarded with
shorter development times, easier debugging and maintenance, etc.
<P>There are plenty of languages and compilers beyond those listed here
(in alphabetical order).  A list of freely available compilers (most
of which have nothing to do with Linux parallel processing) is at
<A HREF="http://www.idiom.com/free-compilers/">http://www.idiom.com/free-compilers/</A>.
<P>
<H3>Fortran 66/77/PCF/90/HPF/95</H3>

<P>
<P>At least in the scientific computing community, there will always be
Fortran.  Of course, now Fortran doesn't mean the same thing it did in
the 1966 ANSI standard.  Basically, Fortran 66 was pretty simple stuff.
Fortran 77 added tons of features, the most noticeable of which were the
improved support for character data and the change of <CODE>DO</CODE> loop
semantics.  PCF (Parallel Computing Forum) Fortran attempted to add a
variety of parallel processing support features to 77.  Fortran 90 is
a fully-featured modern language, essentially adding C++-like
object-oriented programming features and parallel array syntax to the
77 language.  HPF (High-Performance Fortran,
<A HREF="http://www.crpc.rice.edu/HPFF/home.html">http://www.crpc.rice.edu/HPFF/home.html</A>), which has itself gone
through two versions (HPF-1 and HPF-2), is essentially the enhanced,
standardized, version of what many of us used to know as CM Fortran,
MasPar Fortran, or Fortran D; it extends Fortran 90 with a variety of
parallel processing enhancements, largely focussed on specifying data
layouts.  Finally, Fortran 95 represents a relatively minor
enhancement and refinement of 90.
<P>What works with C generally can also work with <CODE>f2c</CODE>,
<CODE>g77</CODE> (a nice Linux-specific overview is at
<A HREF="http://linux.uni-regensburg.de/psi_linux/gcc/html_g77/g77_91.html">http://linux.uni-regensburg.de/psi_linux/gcc/html_g77/g77_91.html</A>),
or the commercial Fortran 90/95 products from
<A HREF="http://extweb.nag.co.uk/nagware/NCNJNKNM.html">http://extweb.nag.co.uk/nagware/NCNJNKNM.html</A>.  This is because
all of these compilers eventually come down to the same code-generation
used in the back-end of GCC.
<P>Commercial Fortran parallelizers that can generate code for SMPs are
available from
<A HREF="http://www.kai.com/">http://www.kai.com/</A> and
<A HREF="http://www.psrv.com/vast/vast_parallel.html">http://www.psrv.com/vast/vast_parallel.html</A>.  It is not
clear if these compilers will work for SMP Linux, but it should be
possible given that the standard POSIX threads (i.e., LinuxThreads)
work under SMP Linux.
<P>The Portland Group,
<A HREF="http://www.pgroup.com/">http://www.pgroup.com/</A>, has commercial
parallelizing HPF Fortran (and C, C++) compilers that generate code for
SMP Linux; they also have a version targeting clusters using MPI or
PVM.  FORGE/spf/xHPF products at
<A HREF=" http://www.apri.com/"> http://www.apri.com/</A>
might also be useful for SMPs or clusters.
<P>Freely available parallelizing Fortrans that might be made to work
with parallel Linux systems include:
<P>
<UL>
<LI>ADAPTOR (Automatic DAta Parallelism TranslaTOR,
<A HREF="http://www.gmd.de/SCAI/lab/adaptor/adaptor_home.html">http://www.gmd.de/SCAI/lab/adaptor/adaptor_home.html</A>),
which can translate HPF into Fortran 77/90 code with MPI or PVM calls,
but does not mention Linux.
</LI>
<LI>Fx
<A HREF="http://www.cs.cmu.edu/~fx/Fx">http://www.cs.cmu.edu/~fx/Fx</A> at Carnegie Mellon
targets some workstation clusters, but Linux?
</LI>
<LI>HPFC (prototype HPF Compiler,
<A HREF="http://www.cri.ensmp.fr/~coelho/hpfc.html">http://www.cri.ensmp.fr/~coelho/hpfc.html</A>) generates Fortran 77
code with PVM calls.  Is it usable on a Linux cluster?
</LI>
<LI>Can PARADIGM (PARAllelizing compiler for DIstributed-memory
General-purpose Multicomputers,
<A HREF="http://www.crhc.uiuc.edu/Paradigm/">http://www.crhc.uiuc.edu/Paradigm/</A>) be used with Linux?
</LI>
<LI>The Polaris compiler,
<A HREF="http://ece.www.ecn.purdue.edu/~eigenman/polaris/">http://ece.www.ecn.purdue.edu/~eigenman/polaris/</A>, generates
Fortran code for shared memory multiprocessors, and may soon be
retargeted to PAPERS Linux clusters.
</LI>
<LI>PREPARE,
<A HREF="http://www.irisa.fr/EXTERNE/projet/pampa/PREPARE/prepare.html">http://www.irisa.fr/EXTERNE/projet/pampa/PREPARE/prepare.html</A>,
targets MPI clusters...  it is not clear if it can generate code to
run on IA32 processors.
</LI>
<LI>Combining ADAPT and ADLIB, shpf (Subset High Performance Fortran
compilation system,
<A HREF="http://www.ccg.ecs.soton.ac.uk/Projects/shpf/shpf.html">http://www.ccg.ecs.soton.ac.uk/Projects/shpf/shpf.html</A>) is
public domain and generates Fortran 90 with MPI calls...  so, if you
have a Fortran 90 compiler under Linux....
</LI>
<LI>SUIF (Stanford University Intermediate Form, see
<A HREF="http://suif.stanford.edu/">http://suif.stanford.edu/</A>) has parallelizing compilers for both
C and Fortran.  This is also the focus of the National Compiler
Infrastructure Project...  so, is anybody targeting parallel Linux
systems?</LI>
</UL>
<P>I'm sure that I have omitted many potentially useful compilers for
various dialects of Fortran, but there are so many that it is difficult
to keep track.  In the future, I would prefer to list only those
compilers known to work with Linux.  Please email comments and/or
corrections to
<A HREF="hankd@engr.uky.edu">hankd@engr.uky.edu</A>.
<P>
<H3>GLU (Granular Lucid)</H3>

<P>
<P>GLU (Granular Lucid) is a very high-level programming system based on
a hybrid programming model that combines intensional (Lucid) and
imperative models.  It supports both PVM and TCP sockets.  Does it run
under Linux?  More information is available at
<A HREF="http://www.csl.sri.com/GLU.html">http://www.csl.sri.com/GLU.html</A>.
<P>
<H3>Jade And SAM</H3>

<P>
<P>Jade is a parallel programming language that extends C to exploit
coarse-grain concurrency in sequential, imperative programs.  It
assumes a distributed shared memory model, which is implemented by SAM
for workstation clusters using PVM.  More information is available at
<A HREF="http://suif.stanford.edu/~scales/sam.html">http://suif.stanford.edu/~scales/sam.html</A>.
<P>
<H3>Mentat And Legion</H3>

<P>
<P>Mentat is an object-oriented parallel processing system that works
with workstation clusters and has been ported to Linux.  Mentat
Programming Language (MPL) is an object-oriented programming language
based on C++.  The Mentat run-time system uses something vaguely
resembling non-blocking remote procedure calls.  More information is
available at
<A HREF="http://www.cs.virginia.edu/~mentat/">http://www.cs.virginia.edu/~mentat/</A>.
<P>Legion
<A HREF="http://www.cs.virginia.edu/~legion/">http://www.cs.virginia.edu/~legion/</A> is built on top
on Mentat, providing the appearance of a single virtual machine across
wide-area networked machines.
<P>
<H3>MPL (MasPar Programming Language)</H3>

<P>
<P>Not to be confussed with Mentat's MPL, this language was originally
developed as the native parallel C dialect for the MasPar SIMD
supercomputers.  Well, MasPar isn't really in that business any more
(they are now NeoVista Solutions,
<A HREF="http://www.neovista.com">http://www.neovista.com</A>,
a data mining company), but their MPL compiler was built using GCC, so
it is still freely available.  In a joint effort between the
University of Alabama at Huntsville and Purdue University, MasPar's
MPL has been retargeted to generate C code with AFAPI calls (see
section 3.6), and thus runs on both Linux SMPs and clusters.  The
compiler is, however, somewhat buggy...  see
<A HREF="http://www.math.luc.edu/~laufer/mspls/papers/cohen.ps">http://www.math.luc.edu/~laufer/mspls/papers/cohen.ps</A>.
<P>
<H3>PAMS (Parallel Application Management System)</H3>

<P>
<P>Myrias is a company selling a software product called PAMS (Parallel
Application Management System).  PAMS provides very simple directives
for virtual shared memory parallel processing.  Networks of Linux
machines are not yet supported.  See
<A HREF="http://www.myrias.com/">http://www.myrias.com/</A> for more information.
<P>
<H3>Parallaxis-III</H3>

<P>
<P>Parallaxis-III is a structured programming language that extends
Modula-2 with "virtual processors and connections" for data
parallelism (a SIMD model).  The Parallaxis software comprises
compilers for sequential and parallel computer systems, a debugger
(extensions to the gdb and xgbd debugger), and a large variety of
sample algorithms from different areas, especially image processing.
This runs on sequential Linux systems...  an old version supported
various parallel targets, and the new version also will (e.g.,
targeting a PVM cluster).  More information is available at
<A HREF="http://www.informatik.uni-stuttgart.de/ipvr/bv/p3/p3.html">http://www.informatik.uni-stuttgart.de/ipvr/bv/p3/p3.html</A>.
<P>
<H3>pC++/Sage++</H3>

<P>
<P>pC++/Sage++ is a language extension to C++ that permits data-parallel
style operations using "collections of objects" from some base
"element" class.  It is a preprocessor generating C++ code that can
run under PVM.  Does it run under Linux?  More information is
available at
<A HREF="http://www.extreme.indiana.edu/sage/">http://www.extreme.indiana.edu/sage/</A>.
<P>
<H3>SR (Synchronizing Resources)</H3>

<P>
<P>SR (Synchronizing Resources) is a concurrent programming language in
which resources encapsulate processes and the variables they share;
operations provide the primary mechanism for process interaction. SR
provides a novel integration of the mechanisms for invoking and
servicing operations. Consequently, all of local and remote procedure
call, rendezvous, message passing, dynamic process creation,
multicast, and semaphores are supported. SR also supports shared
global variables and operations.
<P>It has been ported to Linux, but it isn't clear what parallelism it
can execute with.  More information is available at
<A HREF="http://www.cs.arizona.edu/sr/www/index.html">http://www.cs.arizona.edu/sr/www/index.html</A>.
<P>
<H3>ZPL And IronMan</H3>

<P>
<P>ZPL is an array-based programming language intended to support
engineering and scientific applications.  It generates calls to a
simple message-passing interface called IronMan, and the few functions
which constitute this interface can be easily implemented using nearly
any message-passing system.  However, it is primarily targeted to PVM
and MPI on workstation clusters, and Linux is supported.  More
information is available at
<A HREF="http://www.cs.washington.edu/research/projects/orca3/zpl/www/">http://www.cs.washington.edu/research/projects/orca3/zpl/www/</A>.
<P>
<H2><A NAME="ss6.2">6.2 Performance Issues</A>
</H2>

<P>
<P>There are a lot of people who spend a lot of time benchmarking
particular motherboards, network cards, etc., trying to determine
which is the best.  The problem with that approach is that by the time
you've been able to benchmark something, it is no longer the best
available; it even may have been taken off the market and replaced by
a revised model with entirely different properties.
<P>Buying PC hardware is like buying orange juice.  Usually, it is made
with pretty good stuff no matter what company name is on the label.
Few people know, or care, where the components (or orange juice
concentrate) came from.  That said, there are some hardware
differences that you should pay attention to.  My advice is simply
that you be aware of what you can expect from the hardware under
Linux, and then focus your attention on getting rapid delivery, a good
price, and a reasonable policy for returns.
<P>An excellent overview of the different PC processors is given in
<A HREF="http://www.pcguide.com/ref/cpu/fam/">http://www.pcguide.com/ref/cpu/fam/</A>; in fact, the whole WWW
site
<A HREF="http://www.pcguide.com/">http://www.pcguide.com/</A> is full of good technical
overviews of PC hardware.  It is also useful to know a bit about
performance of specific hardware configurations, and the Linux
Benchmarking HOWTO
<A HREF="http://sunsite.unc.edu/LDP/HOWTO/Benchmarking-HOWTO.html">http://sunsite.unc.edu/LDP/HOWTO/Benchmarking-HOWTO.html</A> is a
good place to start.
<P>The Intel IA32 processors have many special registers that can be used
to measure the performance of a running system in exquisite detail.
Intel VTune,
<A HREF="http://developer.intel.com/design/perftool/vtune/">http://developer.intel.com/design/perftool/vtune/</A>, uses the
performance registers extensively in a very complete code-tuning
system...  that unfortunately doesn't run under Linux.  A loadable
module device driver, and library routines, for accessing the Pentium
performance registers is available from
<A HREF="http://www.cs.umd.edu/users/akinlar/driver.html">http://www.cs.umd.edu/users/akinlar/driver.html</A>.  Keep in mind
that these performance registers are different on different IA32
processors; this code works only with Pentium, not with 486, Pentium
Pro, Pentium II, K6, etc.
<P>Another comment on performance is appropriate, especially for those
of you who want to build big clusters and put them in small spaces.
At least some modern processors incorporate thermal sensors and
circuits that are used to slow the internal clock rate if operating
temperature gets too high (an attempt to reduce heat output and
improve reliability).  I'm not suggesting that everyone should go buy
a peltier device (heat pump) to cool each CPU, but you should be aware
that high operating temperature does not just shorten component life -
it also can directly reduce system performance.  Do not arrange your
computers in physical configurations that block airflow, trap heat
within confined areas, etc.
<P>Finally, performance isn't just speed, but also reliability and
availability.  High reliability means that your system almost never
crashes, even when components fail...  which generally requires
special features like redundant power supplies and hot-swap
motherboards.  That usually isn't cheap.  High availability refers to
the concept that your system is available for use nearly all the
time...  the system may crash when components fail, but the system is
quickly repaired and rebooted.  There is a High-Availability HOWTO
that discusses many of the basic issues.  However, especially for
clusters, high availablity can be achieved simply by having a few
spares.  I recommend at least one spare, and prefer to have at least
one spare for every 16 machines in a large cluster.  Discarding faulty
hardware and replacing it with a spare can yield both higher
availability and lower cost than a maintenance contract.
<P>
<H2><A NAME="ss6.3">6.3 Conclusion - It's Out There</A>
</H2>

<P>
<P>So, is anybody doing parallel processing using Linux?  Yes!
<P>It wasn't very long ago that a lot of people were wondering if the
death of many parallel-processing supercomputer companies meant that
parallel processing was on its way out.  I didn't think it was dead
then (see
<A HREF="http://dynamo.ecn.purdue.edu/~hankd/Opinions/pardead.html">http://dynamo.ecn.purdue.edu/~hankd/Opinions/pardead.html</A> for a
fun overview of what I think really happened), and it seems quite
clear now that parallel processing is again on the rise.  Even Intel,
which just recently stopped making parallel supercomputers, is proud
of the parallel processing support in things like MMX and the upcoming
IA64 EPIC (Explicitly Parallel Instruction Computer).
<P>If you search for "Linux" and "parallel" with your favorite search
engine, you'll find quite a few places are involved in parallel
processing using Linux.  In particular, Linux PC clusters seem to be
popping-up everywhere.  The appropriateness of Linux, combined with
the low cost and high performance of PC hardware, have made parallel
processing using Linux a popular approach to supercomputing for both
small, budget-constrained, groups and large, well-funded, national
research laboratories.
<P>Various projects listed elsewhere in this document maintain lists of
"kindred" research sites that have similar parallel Linux
configurations.  However, at
<A HREF="http://yara.ecn.purdue.edu/~pplinux/Sites/">http://yara.ecn.purdue.edu/~pplinux/Sites/</A>, there is a
hypertext document intended to provide photographs, descriptions, and
contact information for all the various sites using Linux systems for
parallel processing.  To have information about your site posted there:
<P>
<UL>
<LI>You must have a "permanent" parallel Linux site:  an SMP,
cluster of machines, SWAR system, or PC with attached processor, which
is configured to allow users to <EM>execute parallel programs under
Linux</EM>.  A Linux-based software environment (e.g., PVM, MPI,
AFAPI) that directly supports parallel processing must be installed on
the system.  However, the hardware need not be dedicated to parallel
processing under Linux, and may be used for completely different
purposes when parallel programs are not being run.
</LI>
<LI>Request that your site be listed.  Send your site information to
<A HREF="mailto:hankd@engr.uky.edu">hankd@engr.uky.edu</A>.  Please follow the format used in other
entries for your site information.  <EM>No site will be listed without
an explicit request from the contact person for that site.</EM></LI>
</UL>
<P>There are 14 clusters in the current listing, but we are aware of at
least several dozen Linux clusters world-wide.  Of course, listing
does not imply any endorsement, etc.; our hope is simply to increase
awareness, research, and collaboration involving parallel processing
using Linux.
<P>
<HR>
Next
<A HREF="Parallel-Processing-HOWTO-5.html">Previous</A>
<A HREF="Parallel-Processing-HOWTO.html#toc6">Contents</A>
</BODY>
</HTML>