914 lines
52 KiB
HTML
914 lines
52 KiB
HTML
<!--startcut ==========================================================-->
|
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
|
|
<HTML>
|
|
<HEAD>
|
|
<title>Thoughts about Linux LG #33</title>
|
|
</HEAD>
|
|
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#A000A0"
|
|
ALINK="#FF0000">
|
|
<!--endcut ============================================================-->
|
|
|
|
<H4>
|
|
"Linux Gazette...<I>making Linux just a little more fun!</I>"
|
|
</H4>
|
|
|
|
<P> <HR> <P>
|
|
<!--===================================================================-->
|
|
|
|
<center>
|
|
<H1><font color="maroon">Thoughts about Linux</font></H1>
|
|
<H4>By <a href="mailto:jurgen.defurne@scc.be">Jurgen Defurne</a></H4>
|
|
</center>
|
|
<P> <HR> <P>
|
|
|
|
<H1>Introduction</H1>
|
|
<P>First, I want to give a small explanation on the backgrounds of this
|
|
document. There are several parts which lead to my advocating of Linux
|
|
in the corporate environment.
|
|
<P>First of all, it is already four years since I discovered Linux. It is only
|
|
recently however that I really started using Linux itself. I used some
|
|
GNU tools on the DOS and OS/2 platform, but only through recently
|
|
expanding my storage could I install Linux. I printed some manuals,
|
|
subscribed myself to the Linux Journal, I try to read the Linux Gazette
|
|
frequently. Well, I consider myself almost a fan of the first hour of
|
|
Linux.
|
|
<P>Secondly, since the beginning of 1997 I have worked in a traditional
|
|
mini/COBOL/database environment and I have noticed that the people
|
|
who use these systems, find a lot in such an environment : easy to
|
|
control and operate, you need only one person to program, background
|
|
operation etc. The other side of the coin is that these proprietary
|
|
systems are expensive. You pay every year an expensive maintenance
|
|
contract or you pay an expensive price for reparation and upgrading.
|
|
<P>My third reason, last but not least, is that I have never liked
|
|
Windows in any of its incarnations since 1990. It generated GPF's for
|
|
unknown reasons in 1990 and eight years later, it still does. It forces
|
|
people in buying expensive hardware, which then cannot be utilised
|
|
efficiently (if you don't want to crash).
|
|
<P>These three reasons have lead me to the writing of three document,
|
|
which I want to be published via the Linux Gazette. The reasons for
|
|
this is that I found that the Linux Gazette is also read by people who
|
|
have other system backgrounds than only DOS or Linux, and this is
|
|
crucial for the objective that I want to reach.
|
|
<P>This objective is in its essence the same as Linus Torvalds says,
|
|
and it is :<CITE>World domination</CITE>. However, I have my own
|
|
reasons to believe that world domination will not be attained only
|
|
through the PC, workstation and Internet applications market. <EM>I
|
|
believe that Linux has the potential to compete in the corporate
|
|
marketplace.</EM> Alas, there are still a lot of holes to be filled in
|
|
before this will come true. However, I also think there is enough
|
|
potential among the Linux enthusiasts to make this dream come true.
|
|
<P>The following text consists of three parts in which I was trying to
|
|
order my ideas about what Linux further needs to attain the stated
|
|
goal.
|
|
<P>In the first part that I wrote, I am trying to compare Linux systems with mini-
|
|
and mainframe-computers that I know and their architectures and I want to make
|
|
an appeal to people who might be interested in developing Linux for large
|
|
systems. I posted it on several c.o.l.* newsgroups, but I did not
|
|
receive much response (only 1 person seemed interested).
|
|
|
|
<H2>Mainframe Linux/Linux Mainframes</H2>
|
|
<H3>0. To do</H3>
|
|
<P>This document should be thoroughly cleaned up and restructured. The main
|
|
reason that I send over the Internet as is, is that I want to know the amount
|
|
of response it generates. If there is no interest whatsoever, then the
|
|
project will be cancelled. If you made it through here, please read on.
|
|
Any ideas to have a good working title or something like that, are always
|
|
welcome.
|
|
<P>This document doesn't have the status of HOWTO. If I would assign it a status,
|
|
then it would be something like an RFC, although not that official.
|
|
<P>I apologise if things are not always clear. I need to document some parts
|
|
with graphics to provide a clearer understanding. It should probably
|
|
also be created as an sgml-file, to have more processing power.
|
|
<P>Although this paper is sent to different linux newsgroups, it should be best
|
|
to try to pick just one newsgroup to communicate about this document.
|
|
<H3>1.</H3>
|
|
<P>This document is by no means complete. It attempts to define a framework to
|
|
develop and deploy Linux as a mainframe operating system. If any idea's in
|
|
this document have duplicates somewhere else in the Linux development community,
|
|
I would be glad to know of them, so that
|
|
<OL>
|
|
<LI>They can be used with much less development effort
|
|
<LI>They can be referenced to in (hopefully) further editions of this
|
|
document
|
|
</OL>
|
|
<H3>2. Terms and conditions</H3>
|
|
<P>This document is for the moment completely my own responsibility and my own
|
|
copyright. It may be distributed everywhere, but I am the only one who may
|
|
change it. Please, send questions and suggested changes to my email-address
|
|
jurgen.defurne at scc.be.
|
|
All trademarks acknowledged.
|
|
<P>I intend to put much time into this project. I have a fine, regular job working
|
|
daytime as a COBOL programmer, so time should be not really a concern.
|
|
<H3>3. Rationale</H3>
|
|
<P>The ideas in this document are a reflection of my own experiences in working
|
|
with computers and things that I have read about in a whole bestiary of
|
|
publications (magazines, books, RFC's, HOWTO's, The Web, Symposium records,
|
|
etc...). The basis is this : Linux is highly scalable. For me, it has proven
|
|
to be far more scalable than any MS product. I run Linux on the following
|
|
systems :
|
|
<UL>
|
|
<LI>Toshiba portable computer 386sx/16MHz/3Mb/200 Mb
|
|
<LI>386sx/16MHz/16Mb/Diskless
|
|
<LI>386/33MHz/4Mb/200Mb
|
|
<LI>486/50MHz/32Mb/100Mb
|
|
<LI>6x86/P150+/16Mb/500Mb
|
|
</UL>
|
|
<P>Some of these systems are interconnected, others not (yet). With the use of
|
|
telnet, X and TCP/IP it is possible to use these systems together, to run
|
|
tasks on different systems etc. But I want more. What I would really like is
|
|
that these interconnected systems can be viewed as one single system, with a
|
|
common address space, and where their individual resources are added together
|
|
to form a more powerful computer. The main target would be to make it possible
|
|
to introduce Linux in environments where traditional minicomputers are used
|
|
for data-entry and data-processing. This may sound like pretty ambitious goal.
|
|
I don't know if it is. What I do know is that these are environments where
|
|
high availability is a top priority (<A HREF="#note_1">Note 1</A>).
|
|
<P>Another reason to do this project is the fact that in the beginning of the year
|
|
Tandem has built a mainframe computer using 64 4-way SMP systems, NT,
|
|
their own interconnection software and Oracle Parallel Server. Why shouldn't we
|
|
be able to do something alike ?
|
|
<H3>5. Goals</H3>
|
|
<P>This document must describe not only software, but also hardware and system
|
|
procedures. I hope to revise it very regularly. I would like it to contain
|
|
links to used source code, schematics, construction plans, all used sources
|
|
and a history and possible planning of the project. It should also give people
|
|
who want to make money from Linux the possibility to do this on a professional
|
|
level. That is, they should be able to help companies with processing
|
|
requirements to assess their needs, give advice on required hardware, install
|
|
and implement the system and provide service, maintenance and education.
|
|
<H3>6. What makes a good mini/mainframe environment ?</H3>
|
|
<P>I haven't had a regular programming education. I am an electronics engineer.
|
|
After school I got into microcomputers and programming and I broadened my
|
|
education with courses on business organisation and industrial informatics. My
|
|
experiences in the mini/mainframe world date back from as recently as januari
|
|
1997. At first I got to work in WANG VS (<A HREF="#note_2">Note 2</A>)
|
|
environment, now I am still working as WANG programmer, but the WANG's
|
|
have a duty as front-end input processors to the mainframe (Bull
|
|
DPS8000/DPS9000) and as legal document processing systems. In my first
|
|
job, the WANG VS minicomputer was used more as production mainframe system.
|
|
<P>Now, what do these systems have in common ?
|
|
<UL>
|
|
<LI>a fault-tolerant, high performance file system
|
|
<LI>database products with transactional capabilities
|
|
<LI>a single compiler for development, which supports the file system and
|
|
the database management system
|
|
<LI>a versatile scripting or job control language
|
|
<LI>an interactive development system
|
|
<LI>easy access to operating system functions
|
|
<LI>easy, but powerful access to operate the system
|
|
<LI>optional : powerful tools for non-programmers to access, extract and
|
|
process data from the database (Reports, Queries, ...)
|
|
</UL>
|
|
<P>The main difference between the mini and the mainframe is in the operation of
|
|
the system. The four main tasks that have to be done on a computer system are
|
|
administration, exploitation, production maintenance and development. On a
|
|
mainframe these tasks are done by different people, on a mini these tasks can
|
|
be done by one person, or shared, but you don't need full time personnel for
|
|
the different tasks (except for programming, that is). The system running on a
|
|
mainframe can be sufficiently complicated that some tasks or operations may only
|
|
be done by some trusted personnel.
|
|
<P>Operating the system comprises the following tasks :
|
|
<UL>
|
|
<LI>Managing printers and print-queues
|
|
<LI>Managing jobs and job-queues
|
|
<LI>Managing communications and communications devices
|
|
<LI>Managing disks, tapes, workstations and system options
|
|
</UL>
|
|
<H3>7. What makes a good mini/mainframe computer ?</H3>
|
|
<P>Basically, the ability to handle tasks efficient and fast. If you want to know
|
|
more about the chores of operating systems, there is enough literature available
|
|
(see literature list). The basic problem in running a large computer system is
|
|
the difference between batch-operations and interactive or real-time operations.
|
|
You want batch programs as fast possible to be executed and you want for the
|
|
other kind a fast response time. The basic problem with PC's versus
|
|
mini/mainframe computers is that the IO structure of the PC is very primitive.
|
|
This is starting to change, first with VESA, now with PCI, but it still comes
|
|
nowhere in the neighbourhood of a minicomputer. Basically, these systems always
|
|
have a separate internal processor (or more than one) on the IO bus to handle
|
|
data transport between devices and the memory. With I2O, this should become
|
|
available to the PC world, but it is still proprietary and not available to
|
|
Linux and/or Open Source developers.
|
|
<P>Tasks compiled for x86-architectures tend also to use more memory. Let's take
|
|
some examples from minicomputers and mainframes I know about and have access
|
|
to documentation.
|
|
<TABLE>
|
|
<CAPTION>Overview of some corporate systems</CAPTION>
|
|
<TR><TH>System <TH>Main memory <TH>Clock <TH>Bus size <TH>Users supported
|
|
<TR><TD>WANG VS 6 <TD>4 Mb <TD>16 MHz <TD>16 bit <TD>32
|
|
<TR><TD>WANG VS 6120<TD>16 Mb <TD>20 MHz <TD>32 bit <TD>253
|
|
<TR><TD>WANG VS 6250<TD>64 Mb <TD>50 MHz <TD>32/64 bit <TD>253
|
|
<TR><TD>WANG VS 8000<TD>32 Mb <TD>N.A. <TD>32/64 bit <TD>253
|
|
<TR><TD>BULL DPS9000<TD>2 x 64 Mb <TD>N.A. <TD>N.A. <TD>N.A.
|
|
<TR><TD>BULL DPS8000<TD>2 x 32 Mb <TD>N.A. <TD>N.A. <TD>N.A.
|
|
</TABLE>
|
|
<P>These systems are smaller than PC's in terms of memory, yet they support more
|
|
users and tasks than a PC would do. I wouldn't use my Toshiba portable to
|
|
support ten users on a database. Yet, that is what the WANG VS 6 is (was)
|
|
capable of, with the same characteristics.
|
|
<P>This is for the moment my main criticism of standard PC's and their software :
|
|
they are extremely inefficient. The first inefficiency comes from the methods
|
|
used to lower the price of a PC : the CPU is responsible for data transport
|
|
between devices and memory. You have DMA available, but it isn't very efficient.
|
|
The second inefficiency comes from the software mostly used on PC's : it takes
|
|
up much space on disk and in memory.
|
|
<P>A third inefficiency is in the software itself : it has so many features, but
|
|
these aren't used much. The more features in the software, the less efficient
|
|
it becomes (<A HREF="#note_3">Note 3</A>).
|
|
<P>There is another thing to be learned from mini/mainframe environments : keep
|
|
things simple. I don't think the current desktop/GUI environment is simple. It
|
|
doesn't have a steep learning curve, but basically what you have are super
|
|
souped up versions of what are basically simple programs. When writing
|
|
programs or designing systems, one should always keep in that after a certain
|
|
point it costs more effort to add more functionality to a program, while this
|
|
functionality decreases efficiency.
|
|
<H3>8. Existing functionality</H3>
|
|
<P>In the area of parallel computers there is the Beowulf system and associated
|
|
libraries. Their basic target is parallel processing for scientific purposes,
|
|
while my purpose is business data processing. As I see it, some of their goals
|
|
walk parallel with mine, especially in the areas of existing bottlenecks : the
|
|
network, distributed file access, load balancing etc. However, the way business
|
|
programs are run differs from scientific computing. MPP is also more in the way
|
|
of creating a computer to run really big tasks, while on a business machine
|
|
you have logins from users for data querying, transactional processing, batch
|
|
processing of incoming data, preparing outgoing data, establishing communication
|
|
with other systems. In this sense, what we are looking for is not to distribute
|
|
one task over several computers to speedup processing, but to serve up adequate
|
|
processing power, data manipulation facilities and information bandwidth for a
|
|
large number of users. These goals need different OS support than MPP.
|
|
<P>I have studied the Beowulf structure (a Beowulf HOWTO is available
|
|
on the Internet). The Beowulf structure works is a MPP system in which
|
|
only one computer effectively runs the application. All other nodes in
|
|
the system are slaves to this one CPU. This is why the Beowulf system
|
|
is only partially suited to attain my goal.
|
|
<H3>9. Where do we start ?</H3>
|
|
<P>We need to start with a set of completely defined Linux operated computers,
|
|
from now on called CPU's, which are somehow connected to each other by means
|
|
of an abstract communications layer or CL. This CL can be implemented using
|
|
serial connections, Ethernet, SCSI or anything else that we can devise to
|
|
make CPU's talk to each other. A CPU may be a single-way computer or a
|
|
multi-way SMP computer.
|
|
<H3>10. Where do we want to end ?</H3>
|
|
<P>I think the end point should be to view the system as one single entity. To
|
|
do this, the following requirements should be met :
|
|
<UL>
|
|
<LI>Every process should see the same file system
|
|
<LI>Resources (via /dev files) should be shareable accross CPU's
|
|
<LI>Every CPU should have the same view of OS and memory
|
|
<LI>Process information should be shared accross CPU's
|
|
</UL>
|
|
<P>One of the fundamental changes in the OS should be the way exec() operates.
|
|
When exec() starts a new process, this could be on any CPU. The original
|
|
links need to be preserved and processes should end in the same way as
|
|
always.
|
|
<P>Interprocess communication is straightforward I think. What I would like to
|
|
know is if it is worthwile to strive for a system view in which all memory
|
|
is mapped into one address space ? (Idea behind it : provide every CPU with
|
|
the same view of the system : it's OS, followed by the memory pools of all
|
|
other CPU's mapped into the same address space). This is what NUMA
|
|
(non-uniform memory access) is about. Can the Linux community attain
|
|
this subgoal, or does it need to much specialised resources ?
|
|
<H3>11. High Availability</H3>
|
|
<P>Some key parts of Linux should be redesigned or replaced by fault-tolerant
|
|
parts. The largest part which comes to mind is the file-system. A few months
|
|
ago I had a nasty experience. A connector on the cable of my SCSI subsystem
|
|
had a defect, with the consequence that the system of a sudden completely froze
|
|
while I was busy using X-Windows. The trouble with e2fs is that on these
|
|
occassions the whole filesystem gets corrupted. This should be made more sturdy.
|
|
<P>The other part is that the system may not freeze on these occasions. It should
|
|
be possible to provide a bare minimum of functionality, eg. that the kernel
|
|
takes completely over and switches to text mode to provide diagnostic
|
|
information or tries to create a core dump.
|
|
<P>Another problem that I have encountered is the lack of reliability when a
|
|
harddisk drive gives trouble. What happened to me whas that on using an old
|
|
SCSI drive the kernel and/or e2fs started to write strange messages when I
|
|
tried to use the disk. When the system encounters problems with devices, the
|
|
problems should be logged, the operation should be stopped and informative
|
|
messages should be displayed.
|
|
<P>Other key features in the area of HA should be the tolerance of the complete
|
|
system when a CPU is missing. A CPU may only be added when it passes the self
|
|
test completely and finds out that everything is working fine. When a CPU
|
|
quits while being in the system, there should be possibilities to restart
|
|
processes which have been interrupted. For this one should provide the
|
|
programmer with features to help with this problems : a transactional file
|
|
system, checkpoint functions (other ?).
|
|
<P>The last idea I think of is maybe the possibility of swapping a complete task
|
|
between two CPU's. A task consists of CODE and DATA. You don't need to save
|
|
CODE. DATA can be completely swapped to harddisk. If you have a way to transfer
|
|
the process information from one CPU to another, then it should be possible to
|
|
reload CODE and DATA and restart the process on another system.
|
|
<H3>12. Summary</H3>
|
|
<P>There are two targets. The first is the creation of an extension which combines
|
|
several Linux PC into one system. Users and processes should get a same view of
|
|
of the complete system as one system. This should also mean that certain
|
|
administrative chores should depend only on centrally stored and shared
|
|
information.
|
|
<P>The second one is to add more and better managed fault tolerance, preferably
|
|
more interactively managed.
|
|
<P>Well, this is it. I hope that people ask sane questions, that I don't get
|
|
flamed and that it raises enough interest to advance Linux to a higher level.
|
|
<H3>References.</H3>
|
|
<P><STRONG>Ths reference list is clearly not finished. I need to obtain
|
|
more details about some works.</STRONG><BR>
|
|
The Linux High Availability White Paper.<BR>
|
|
The Beowulf HOWTO<BR>
|
|
The Parallel Processing HOWTO.<BR>
|
|
Andrew S. Tanenbaum, Design and Implementation of Operating Systems<BR>
|
|
<H3>Notes.</H3>
|
|
<P><A NAME="note_1">Note 1.</A><NOTE>I worked for 16 months in a small
|
|
transport company. The core of the business was contained in a WANG VS
|
|
minicomputer. If the system was offline, then nobody could do his work
|
|
properly. The system was basically a database to store dispatching
|
|
operations, the revenues of all operations, the cost control and the
|
|
accountancy. I think there are many small firms, who can't afford
|
|
mainframes, but who need more processing power than the average PC can
|
|
handle, and where many people need different views of the same data.</NOTE>
|
|
<P><A NAME="note_2">Note 2.</A><NOTE>The WANG VS is a particular good
|
|
example of proprietary solution which does an excellent job, but with a
|
|
very steep price. They are very expensive for the initial buy, the
|
|
expansion of the system and the maintenance. I think this is one of the
|
|
main reason's why people want to get rid of their WANG systems. You can
|
|
buy, expand and maintain an HP system for one tenth of the price the
|
|
WANG VS costs.</NOTE>
|
|
<P><A NAME="note_3">Note 3.</A><NOTE>If you wonder why I emphasize
|
|
efficiency : I became interested in microprocessors in 1980 when you
|
|
hadn't much microprocessor and memory. My first computer was a Sinclair
|
|
ZX Spectrum with a> whopping 48 kB RAM. I am still astonished what some
|
|
programmers could do with that tiny amount of memory. There are other
|
|
points besides this : what processing power could be freed up if you
|
|
were able to use all those wasted processor cycles in the common
|
|
desktop PC's ? For small companies, a PC is still rather expensive.
|
|
Combining the power of their PC's could maybe give them an extra edge
|
|
in their operations.</NOTE>
|
|
<HR>
|
|
<P>In the second part I am trying to develop an architecture to extend Linux into a
|
|
parallel processing system, not for numerical processing like Beowulf, but for
|
|
administrative dataprocessing.
|
|
<H2>Description of the booting sequence of the multi-processing
|
|
architecture</H2>
|
|
<P>The goal of this document is to establish the components which should
|
|
comprise the project which was mentioned in the previous document (Linux
|
|
mainframes). To do this, a description of the boot sequence will be
|
|
given, together with the possible failures and the solutions.
|
|
<P>Before attempting this, however, I want to give a short summary of the
|
|
guidelines which should lead us toward the goal of Linux systems which
|
|
can be deployed in corporate environments.
|
|
<P>Minicomputers and mainframes provide reliability and high processing
|
|
power. The reliability is largely obtained in two ways. The first one is
|
|
in the design of the system, the second one is the existence of a
|
|
thorough support department with online help and specialised
|
|
technicians. The emphasize in this document is on the hardware side of
|
|
the system.
|
|
<P>High processing power is obtained in several ways. They involve the use
|
|
of cache-memory, wider data-paths, increasing clock frequency,
|
|
pipelining processing and efficient data-transfer between memory and IO.
|
|
|
|
<H3>What can we do about reliability ?</H3>
|
|
|
|
<P>On the reliability side the system is dependent on hard- and software.
|
|
If we are to use currently available parts (motherboards and cards) then
|
|
the only thing we can influence is the way systems are assembled. Care
|
|
should be taken to avoid static discharges, by using anti-static mats
|
|
and bracelets.
|
|
<P>On the software side we have the Linux operating system which is very
|
|
reliable, with reports of systems running for months without erroneous
|
|
reboots.
|
|
<P>However, hardware can fail and in this respect I think that there still
|
|
needs work done on Linux. If the error is not in the processor or
|
|
the system memory, then a running system should be able to intercept
|
|
hardware errors and handle them gracefully. If at all possible, system
|
|
utilities should be available to test the CPU, the system memory, the
|
|
cache and the address translation system.
|
|
<P>The Linux High Availability White Paper documents clustering of small
|
|
systems. Later on in this document, some other techniques will be
|
|
proposed.
|
|
|
|
<H3>What can we do about the processing power ?</H3>
|
|
|
|
<P>Processing power comes on several levels. On the first level, that of
|
|
the CPU and the main memory we can't do much. With current motherboards
|
|
with bus speeds of 66, 75 and 100 Mhz, we get data transfer speeds
|
|
between memory and CPU of 264 MB/s, 300 MB/s and 400 MB/s. These should
|
|
be sufficient for most applications. Memory is cheap, sizes of 64 to 128
|
|
MB should also give headroom for large applications.
|
|
<P>The largest problem with standard motherboards is that all IO needs to
|
|
be handled by the CPU or else by a slow DMA system. This means that a
|
|
large part of the operating system is being used by device driver code.
|
|
In mini/mainframe systems this is not the case. All IO is handled by
|
|
separate IO-processors. These IO-processors implement the device drivers
|
|
and as such free a large part of the central processor.
|
|
<P>To relieve the central processor of this burden, there are three
|
|
solutions. The first one is being implemented by the I2O consortium. It
|
|
defines standards for intelligent IO-boards on the PCI bus. These boards
|
|
can transfer the requested data themselves to the main memory of the
|
|
CPU. The only problem is that as far as Linux is concerned, I2O is
|
|
proprietary.
|
|
<P>I think that two other solutions should be possible. The first, and
|
|
probably easiest, is to use an SMP motherboard and program the operating
|
|
system so that one processor is completely responsible for all IO, and
|
|
the rest of the CPU's do the real work. Another idea is in the absence
|
|
of SMP use two motherboards, run one with an adapted version of Linux to
|
|
handle all IO and use the other one to run only applications. The only
|
|
trouble here is which system will be used to interconnect the
|
|
motherboards. Especially in the case of mass storage devices, you want
|
|
to stream the data from the device as fast as possible into the memory
|
|
of the application. Currently, this means using the PCI bus in one way
|
|
or another.
|
|
|
|
<H3>Summary</H3>
|
|
|
|
<P>Since we, as Linux users, have no sight on the design process of
|
|
motherboards, reliability should be obtained through good standards of
|
|
assembly and by implementing redundancy.
|
|
<P>To obtain more processing power, the main CPU should be relieved as much
|
|
as possible from IO. This could be implemented by using SMP or by
|
|
interconnecting motherboards.
|
|
|
|
<H3>A proposal for an architecture for Linux mini/mainframes</H3>
|
|
|
|
<P>Based on the previous ideas, using several motherboards interconnected
|
|
by a high-speed network could give us the following benefits :
|
|
<UL>
|
|
<LI>Redundancy to increase reliability
|
|
<LI>Offloading IO tasks to one or more specially appointed nodes
|
|
<LI>Increased processing power
|
|
</UL>
|
|
<P>To obtain these benefits when the system is assembled, some operating
|
|
system changes need to be provided. It is possible to interconnect
|
|
computers and make these work in parallel, but all administration must
|
|
be manually accounted for. So, what we need when the system is booted,
|
|
is not a vision of several separate systems, but only one system.
|
|
|
|
<H3>Description of the boot sequence</H3>
|
|
|
|
<P>When booting the system, all nodes start in the usual way : installed
|
|
hardware is identified, necessary drivers are run, a connection to the
|
|
network should be made, NFS drives should be mounted, local file systems
|
|
should be checked and mounted.
|
|
<P>In the case of a normal system, all background processes would be
|
|
started and users should be able to log in on the system.
|
|
<P>When the system should be seen as one complete system, the boot sequence
|
|
should be modified at this point. Resources which are normally only
|
|
accessible on one node, should be shareable throughout the system. To
|
|
build a common view, every node should have access to a common file
|
|
system. In this file system the directories /dev, /etc and /proc should
|
|
be accessible by every node.
|
|
<P>The directory /dev contains all shared devices. The directory /proc
|
|
provides access to system structures which should be shared by every
|
|
node. The directory /etc contains the necessary files to control the
|
|
system :
|
|
<UL>
|
|
<LI>users
|
|
<LI>groups
|
|
<LI>fstab
|
|
<LI>inittab
|
|
<LI>...
|
|
</UL>
|
|
<P>Every operating system on every node must be adapted to work via these shared
|
|
directories.
|
|
<P>To control the creation of this shared system, one node will have to be
|
|
designated 'master'. After the initial boot sequence, every node will
|
|
have to wait for the master to initialize the network. This
|
|
initialization can proceed in the following way :
|
|
<UL>
|
|
<LI>create /proc
|
|
<LI>create a system process table (accessible via proc)
|
|
<LI>create /dev
|
|
<LI>gather all shared devices on the network
|
|
<LI>execute fstab, inittab and other scripts to initialize the
|
|
complete system
|
|
</UL>
|
|
<P>Started processes fall apart in two categories. Local processes run on
|
|
the nodes which contain the resources that the process needs access to
|
|
eg. getty, fax drivers, etc. Global process are independent of hardware
|
|
and should be able to run on any node in the system.
|
|
<P>Any node should also be able to start a new process on the system. By
|
|
using a load balancing system, all started processes must be evenly
|
|
divided over all nodes.
|
|
|
|
<H3>Providing reliability in the system</H3>
|
|
|
|
The system as proposed above could present some problems. The first one
|
|
is its dependency on a single master computer. If this master fails,
|
|
then the whole system fails. To alleviate this, it should be possible to
|
|
define several masters. If the power is applied and the master nodes
|
|
boot, then the first one to get hold of the interconnection network will
|
|
act as coordinator. If one master then fails, the only implication would
|
|
be that his shared resources are not available in the system.
|
|
<P>If a master fails while the system is up and running, then the basic
|
|
coordination of the system is gone. To overcome this problem, a backup
|
|
master must be defined. This backup master needs to keep an updated copy
|
|
of all master system information. If the real master should fail then
|
|
all nodes in the network should block themselves until the backup master
|
|
has come up.
|
|
The system should provide dynamic management of nodes. This means that
|
|
nodes must be attachable by using system calls. This goes via the
|
|
master, which then adds the system on the network. If a node must be
|
|
detached, then none of its resources should be in use, otherwise the
|
|
call fails.
|
|
<P>If a node fails when in use then this surely will pose problems. A
|
|
failure can show itself on the network (network interface problem,
|
|
processor error) or local. If a process uses a remote device, it will do
|
|
this by means of messages which are sent over the interconnection
|
|
network. In the case of malfunction, the addressed node won't (can't)
|
|
answer anymore. The OS must block the process until the malfunction is
|
|
removed.
|
|
<P>If there are problems in critical parts of the system, device drivers or
|
|
system processes should not blow-up the system or interfere with user
|
|
processes, but they should have the means to correctly report the
|
|
problem and block the processes which are using the particular resource.
|
|
If the malfunction is on a local level (device) then the device driver
|
|
can return a message stating the error.
|
|
<P>The most critical part in the system is the interconnection network.
|
|
This should be tested and tuned according to system demands. If
|
|
possible, a fast protocol should be used instead of TCP/IP.
|
|
|
|
<H3>Summary</H3>
|
|
|
|
<P>The view every node has of the system should be the same.
|
|
Devices must be shareable accross the interconnection network.
|
|
The OS should be extended so that the exec() function, which is basic
|
|
for starting processes, executes on a global level.
|
|
<P>Reliability should be built-in and configurable on several levels.
|
|
A message-based protocol is needed to share devices across the
|
|
interconnection network.
|
|
|
|
<H3>Proposals for interconnection</H3>
|
|
|
|
<P>Basically, there are for the moment two interconnection systems which
|
|
can be used of the shelf.
|
|
<P>The first is Ethernet. Based on the money to spend, you can assemble
|
|
systems with 10 Mbit, 100 Mbit or 1 Gbit networks. Increasing bandwith
|
|
means increasing processing power. To obtain the maximum of your
|
|
bandwidth, the ideal is using an SMP motherboard in which one CPU takes
|
|
care of all network-to-memory data transport.
|
|
<P>The second one which attracts interest in the Linux community, is the
|
|
SCSI interface. Using modern SCSI cards, up to 16 motherboards could be
|
|
connected together to provide for parallel processing.
|
|
<HR>
|
|
<P>This is the third part. I have compiled some cases where I have
|
|
participated to highlight some points that need more support in Linux.
|
|
<H2>Cases where Linux might be employed, but where it isn't</H2>
|
|
<P>Through several enhancements (Beowulf, Coda FS, Andrew FS) Linux gets more and
|
|
more powerful. But how powerful is powerful really ? Linux is announced and used
|
|
in more and more places, but there is a serious lack of numbers on the capacity
|
|
of Linux in different environments and configurations.
|
|
<P>This is however a crucial point. In many environments, Linux gets introduced
|
|
through the reuse of PC's (which is in itself a good point). There are however
|
|
other environments where the introduction of new hard- and software depends on
|
|
the provision of hard numbers for acquisition, deployment, education,
|
|
maintenance, infrastructure and depreciation of systems. This can range from a
|
|
small office which only needs to cough up the required cash up to a financial
|
|
institute which has large dataprocessing and communication needs.
|
|
<P>In some of these areas Linux hasn't probably even touched anything because those
|
|
people use computers as a means to an end. The computer itself does not stir
|
|
their imagination. They have tasks to be done and the computer is their
|
|
instrument to complete those tasks faster and more precise. These are the
|
|
environments which are lured into buying MS products. I know however several
|
|
people which work in various different Wintel environments and none of them are
|
|
satisfied.
|
|
|
|
Some complaints :
|
|
|
|
<P>Lock up of course : power users lock up more easily their PC, because they use
|
|
a lot of applications next to each other.
|
|
|
|
<P>Unexplainable configuration changes : you enter your office and your application
|
|
does not start. Reason : some ASCII text file has reverted to a
|
|
previous state (I had this one several times with the TCP/.IP
|
|
'services' file).
|
|
|
|
<P>MS Office for Windows 95 : You can not seem to use Word for large
|
|
documents (this is a complaint from a user in a large company).
|
|
|
|
<P>Windows NT : can not be deployed in situations where older applications need
|
|
access to older and/or proprietary hardware.
|
|
|
|
<P>I am sure anyone who has ever used the system, knows other bugs.
|
|
|
|
<P>I think that one of the reasons why Linux isn't more employed in these
|
|
environments is that it is mostly deployed using a single type of configuration
|
|
existing of an IA32 CPU, a PC AT architecture, IDE/SCSI disk subsystem, an
|
|
Ethernet NIC and standard serial devices. This makes it very easy to use Linux
|
|
in the following places :
|
|
<UL>
|
|
<LI>e-mail
|
|
<LI>nntp
|
|
<LI>http
|
|
<LI>file systems
|
|
<LI>printing
|
|
<LI>MPP (Beowulf)
|
|
<LI>embedded systems
|
|
<LI>workstations
|
|
<LI>telecommunications
|
|
<LI>networked, distributed systems
|
|
</UL>
|
|
<P>These are technical solutions for technical problems, implemented by technical
|
|
people. However, for some places, some pieces are still missing and there are
|
|
places where Linux could be used, but where it is not. The usability of Linux
|
|
still depends too much on the technical skill level of the user. This should not
|
|
be necessary. Companies should be able to deploy Linux quick, efficient and
|
|
flawless. Introductory courses should be provided. This will mostly mean
|
|
migrating from Windows knowledge to Linux knowledge. People should be made to
|
|
understand that there are three pillars in the usage of a computer system and/or
|
|
program :
|
|
<UL>
|
|
<LI>operations
|
|
<LI>administration
|
|
<LI>maintenance
|
|
</UL>
|
|
<P>On the system level these should be integrated transparently and tightly. A user
|
|
shouldn't need to go through heaps of paper and manuals to find something quick,
|
|
so menu driven is probably the best answer for this, with good context sensitive
|
|
help. I even think that from the point of view of the user, things should be
|
|
accessible under a heading 'Applications' where all his production programs
|
|
should reside, and a heading 'Maintenance' where operational, administrative,
|
|
system maintenance and diagnostic programs are located.
|
|
<P>If we want Linux systems to be used more in environments where people are not
|
|
concerned with their computer per se, but as a means to do their job, then
|
|
support will have to grow on several levels. To project these levels, I will
|
|
present some cases more or less detailed. These cases present environments where
|
|
I have worked, customers which needed support, people I know.
|
|
|
|
<H3>Case 1 : The SOHO environment</H3>
|
|
|
|
<P>With this I mean the family sized company which provides some basic services
|
|
(grocer, plummer, carpenter, etc...). At most two persons are responsible for
|
|
handling all administration. This consists mostly of two parts : accounting and
|
|
handling of incoming/outgoing messages. The first part of the problem is
|
|
providing this environment with a suitable accounting package which is
|
|
applicable for the country where the company resides.
|
|
<P>The second part of the problem is handling all incoming and outgoing messages.
|
|
This requires access to three channels : phone, fax and e-mail (if there are any
|
|
other options then these are probably too expensive for this environment).
|
|
Depending on the situation, there could be constraints on the usage of the
|
|
channels (eg. no channel should block another channel, when answering the phone,
|
|
the fax and e-mail should not be prohibited and/or prohibit each other). The
|
|
configuration could probably be extended using a PABX card in the system, to
|
|
provide extended telephony services via Linux.
|
|
<P>Like it or not, but these people have become accustomed to using WYSIWYG word
|
|
processors and spread sheets, so the least that must be done is provide them
|
|
with this functionality. There are at least two good packages available for
|
|
Linux in this respect. Another thing that should be provided is a customer
|
|
database which is closely linked to the former package. Creating new documents
|
|
and using fill in documents from a user entry should be a must. Creation and
|
|
insertion of simple graphics should be an available option too.
|
|
<P>If we consider at most two people then the system could be configured using two
|
|
workstations of the same capacity, where some tasks are shared between each
|
|
other, or it could be done using one more powerful system, which provides all
|
|
services, and one cheap PC workstation, configured as an X-server.
|
|
|
|
<H3>Case 2 : A medium sized company I (10 users)</H3>
|
|
<P><B>File- and print-services, bookkeeping, inventory control</B>
|
|
<P>The company where I first worked from 1990 to 1991 had a Novell Netware system
|
|
installed. We used the system to provide printservices for Mac- and PC-systems,
|
|
as a repository for all kinds of drivers and diagnostic software and as a shared
|
|
database via the bookkeeping and inventory control program. Everybody who needed
|
|
access to the network had his or her own PC or Mac. We mostly used DOS back
|
|
then, although with the introduction of Win 3.0 some people migrated to it.
|
|
Everybody had access to a phone and there was one central fax in the
|
|
administrative department. We installed and maintained PC's and Mac's for
|
|
graphical applications. These applications provided output for typesetting
|
|
printers (mostly via Postscript) or plotters. The supported applications where
|
|
Adobe Photoshop, Aldus Pagemaker and AutoCad. We were also a reseller of the
|
|
bookkeeping package that was used on the network.
|
|
<P>The printing could be spooled to several large laserprinter, a high-speed
|
|
dot-matrix printer and a photographic typesetter.
|
|
<P>File services under Linux are probably the easiest of problems. I networked,
|
|
recompiled, linked and started a small TCP/IP network using two computers in
|
|
less than an hour. NFS is very comprehensive, as are telnet and other TCP/IP
|
|
services. If you need to provide only a central server, then the following
|
|
things need to be done :
|
|
<UL>
|
|
<LI>assign separate network numbers to your NICs
|
|
<LI>configure server and WSs for NFS
|
|
<LI>configure the exports file
|
|
</UL>
|
|
<P>For the workstations the following needs to be done
|
|
<UL>
|
|
<LI>assign separate node numbers
|
|
<LI>configure NFS
|
|
<LI>add your network directory to fstab
|
|
</UL>
|
|
<P>The main difference between Novell and NFS is in the administration. On a
|
|
Netware server, all administration is kept central to the server. The only thing
|
|
which needs to be done on a workstation is load an IPX driver at boot time. On a
|
|
TCP/IP workstation, some administration is kept centrally and some
|
|
administration is kept locally. This makes the process of maintaining and
|
|
updating the network more laborious.
|
|
<P>Installing print services under Linux is generally much harder than under
|
|
Netware. This is because all settings are to be added manually using a text
|
|
editor in the file printcap. But, since this is a very structured file, with a
|
|
rather small set of commands, why hasn't any body ever written a dialog system
|
|
to scan printcap and present the user with an overview of available printers and
|
|
the possibilities of adding and modifying printers and their settings ? This
|
|
would be a great step forward in installing printers. Filters for different
|
|
types of printers could be presented, so that the configuration on the network
|
|
could be simplified (as an aside, RedHat provides such a system).
|
|
<P>The other part of printing is the operation of the queues. The lpd
|
|
system provides only command line control. But since this system is
|
|
also understood very fine, why haven't there been any attempts to
|
|
rewrite the lpd system for menu-driven operation ? After all, entering
|
|
a command or pressing a function key can invoke the same behaviour. All
|
|
queues and printers can be presented to the user, with the possibility
|
|
of providing more details.
|
|
<P>The accounting program was written in Clipper and did not use Btrieve.
|
|
This means that all access to the data in the files generated a lot of
|
|
traffic over the network. This was alleviated by segmenting the network
|
|
in three parts so that the accounting department didn't interfere with
|
|
the other departments. The whole package ran under DOS. In the course
|
|
of years, the company which programmed the package made in 1994 the
|
|
transition from Clipper to FoxPro, and only as recent as 1997 they made
|
|
the transition from DOS to Windows (with the DOS version still being
|
|
sold and supported).
|
|
<P>This presents us with a case of providing support for migration of
|
|
xBase dialects to Linux, while adding value to these languages through
|
|
transparent client/server computing. There should also be support for
|
|
people migrating from these DOS-based systems to Linux. There are a
|
|
whole lot of programmers who work alone and who make a living by
|
|
writing and maintaining small database applications for SOHO users
|
|
(using xBase and several 4GL tools which run under DOS). Providing
|
|
incentives and support for these people to migrate and to help their
|
|
customers migrate could give a double benefit to Linux. The key lays of
|
|
course in the way that support for these tools becomes available under
|
|
Linux or that conversion tools become available under Linux.
|
|
<P>Printing support under Un*x and hence Linux has always strongly been oriented at
|
|
typesetting. Providing support for Postscript should not be a problem under
|
|
Linux. Adding a typesetter should be as easy as installing a printer on a server
|
|
or on the network via a print server. There are already some strong graphical
|
|
packages available for Linux. In this case, migration is a question of importing
|
|
and/or converting graphical files and showing the user how to do the tasks he
|
|
does normally with the new application.
|
|
<P>Plotting and/or cutting should be the same as printing. The application program
|
|
is responsible for translating it's own internal drawing database into a format
|
|
that can be used by the addressed peripheral.
|
|
|
|
<H3>Case 3 : The drafting department</H3>
|
|
<B>Drawing workstations, central database, drawing lock, usage
|
|
statistics</B>
|
|
<P>Drafting departments are a case where networking and central storage are really
|
|
put to the test. It consists of a drawing database, which is a front-end to the
|
|
drafting programs. User should be able to look at drawings, create, edit, delete
|
|
and print drawings and collect usage statistics about drawings. In addition,
|
|
only one user should be able to edit a drawing or part of a drawing at one time,
|
|
and it should be possible to see who is editing what. If this all sounds like
|
|
using a file system, then you are right. The difference is that you only use one
|
|
type of file. I worked on one system in the previous case. It was written using
|
|
Clipper as a front-end. I know of other environments where Autocad is used, but
|
|
under a WinNT network, and there are some companies who deliver complete turnkey
|
|
solutions consisting of powerful minicomputers and proprietary workstations for
|
|
real high-end drafting work.
|
|
<P>Providing the incentive to migrate to Linux consists in providing a powerful
|
|
server with large storage to accomodate all the drawings and a fast network to
|
|
deliver them to the workstations. All workstations should be tuned to the max to
|
|
deliver the utmost in graphic display and manipulation. Of course, utilities are
|
|
necessary to convert the original drawing database and all the drawings.
|
|
Networking should be flawlessly, and the program which uploads the drawing
|
|
should provide an indication of the time necessary to get the file and where it
|
|
is in the process.
|
|
|
|
<H3>Case 4 : A medium sized company II (20 users)</H3>
|
|
<B>Mini computer system, data entry and retrieval, commercial
|
|
department</B>
|
|
|
|
<P>This pertains to my previous job : a small transport company, which had ten
|
|
years ago decided to implement a computer system to automate several tasks and
|
|
to keep a database of all done transportations. They had taken WANG VS, which
|
|
was back then a successfull system, with many advanced features. Custom software
|
|
had been developed by an outside company first, by an in-house programmer later.
|
|
The system contains a very comprehensible fax package, which can be used by
|
|
anyone, but with strong security features. All outgoing messages are put in one
|
|
queue, where the operator can change their times and/or priorities. All
|
|
communication with the minicomputer is via terminals or via emulation cards on
|
|
PC's. Accounting is also done on the minicomputer, but the two systems are not
|
|
linked. The system is also equipped with a background task which controls batch
|
|
tasks in a queue.
|
|
<P>There are many medium-sized companies which still use minicomputers and who have
|
|
a problem shedding them, due to their highly specialized software. Migration
|
|
from a Un*x system to a Linux system should not pose as much problems as
|
|
migrating from a completely proprietary system to Linux.
|
|
<P>The main problem with these mini-computer systems is their high maintenance
|
|
cost. That should be the most pressing reason to migrate, although Y2K could
|
|
also be an incentive (not so with WANG VS, which is fully Y2K compliant).
|
|
<P>To provide the same functionality a DBMS package should be available which
|
|
provides a data dictionary, a screen design package and a COBOL74 compiler with
|
|
preprocessor to translate simple SQL SELECT statements. There are several
|
|
packages available. One package aids in the migration from WANG PACE (the WANG
|
|
DBMS) to Oracle (at the moment Oracle has only announced porting Oracle to
|
|
Linux), while Software AG has tools to port WANG PACE applications and screens
|
|
to ADABAS. On part of the compiler, where I work currently the porting is done
|
|
from WANG to HP-UX using Microfocus Cobol. The security features of the database
|
|
package should at least contain rollback recovery. The provided file-system
|
|
should absolutely not be e2fs. Reliability should be favored over speed. When
|
|
the power fails the file-system it self may be damaged, but these damages should
|
|
be simple to clean-up. Damages in transactional files are to be repaired with
|
|
the rollback option.
|
|
<P>On the hardware side, I noted that SCSI II provided enough speed to handle some
|
|
20 users, but ... this was a system with a specialized IO-processor to handle
|
|
all data transfers between main memory and all peripherals. To know how Linux
|
|
fares in this, benchmarks should be run and numbers should be provided. In our
|
|
last configuration (a 50 MHz CPU with 64 Mb), under a heavy load, our response
|
|
time was under 10 s.
|
|
<P>Fax support must be provided to interactive applications, but also to batch
|
|
applications.
|
|
<P>Batch processing of all tasks should be supported. Some programs can be started,
|
|
used to enter selection data and then launched at will in the background or in
|
|
the foreground at a time and day the user can enter. cron is fine for highly
|
|
skilled people, but not for your data-entry clerk, so you need a front end which
|
|
asks the date, time and repetition rate of your job. The application itself
|
|
should be able to provide the required parameters.
|
|
|
|
<H3>Case 5 : OEM</H3>
|
|
<B>Cash registers, inventory control, proprietary hardware</B>
|
|
|
|
<P>This company builds cash register systems using mostly common PC hardware and
|
|
one piece of proprietary hardware which interfaces to a magnetic card reader, a
|
|
bar code reader, a money drawer and a keyboard/display/pricing printer. The cash
|
|
register is connected via a network to a server which provides an inventory and
|
|
a price list. Upon booting, the cash register connects to the network and loads
|
|
its OS from the server. Every server has the possibility to connect at night to
|
|
a central database to update its pricelists and to order items which are getting
|
|
out of stock.
|
|
<P>For the cash register, a multi-user, multi-tasking OS is clearly overkill, while
|
|
in the case of the server, multiple cash-registers could connect via the network
|
|
to the server. The cash register would benefit, though, from multi-threading.
|
|
<P>Software development for servers and departmental systems is usually done with a
|
|
4GL tool, with a higher-level language only for those parts which 4GL does not
|
|
support.
|
|
|
|
<H3>Case 6 : Financial company (appr. 1000 users, agencies)</H3>
|
|
<B>Minicomputers, mainframe computers, terminals, workstations,
|
|
TCP/IP</B>
|
|
|
|
<P>The production environment of this company consists of 5 WANG VS minicomputers,
|
|
used for data-entry, data-preprocessing and to connect agencies remotely through
|
|
a telephone line. It consists also of a Bull mainframe system with two CPU's,
|
|
128 Mb memory, 240 Gb of on-line storage capacity, a transaction processing
|
|
system consisting of a network database and a screen editing and runtime
|
|
program. All this is controlled using JCL and COBOL-74. TCP/IP is implemented
|
|
between all systems.
|
|
<P>Replacing the minicomputers with Linux systems should be relatively straight
|
|
forward. Since no WANG PACE is implemented on these, only migration of the
|
|
COBOL-74 programma's needs to be done. Data entry and remote connection could be
|
|
done using telnet and/or serial connections. Transferring data between mainframe
|
|
and other systems is no problems. All this happens using FTP.
|
|
<P>Now, let us think really BIG! Could a case be made to build a system using
|
|
Linux, which can replace a mainframe computer, given the specs above ? As said
|
|
above, more numbers and benchmarks are needed on Linux and its implementations
|
|
to know how powerful Linux can be.
|
|
|
|
<H3>Case 7 : Software for highly skilled, non-technical people</H3>
|
|
<B>Doctors, dentists, lawyers, chemists, ...</B>
|
|
|
|
<P>These cases resemble the SOHO, but additionally need very specialized software
|
|
to support their job. This software is mostly written by very specialised
|
|
companies (niche software). What would they need in terms of software and
|
|
maintenance to be convinced to migrate to Linux ?
|
|
<P>One of the answers is surely that they can migrate their existing applications
|
|
easily and that conversion of their source code is supported by tools and API's
|
|
which provide the same (or better) functionality than their old tools.
|
|
<P>Configuration of these systems may be more specialized. Normally the user would
|
|
only use his system (enter customers, query the system). All administrative and
|
|
configuration chores could be left to the implementor. The applications
|
|
themselves are already as user-friendly as they can be, due to their specialised
|
|
nature.
|
|
|
|
<H3>Conclusion</H3>
|
|
|
|
<P>I have presented several real-world cases, where Linux IMHO could be used. In
|
|
most cases there are two recurring themes.
|
|
<P>The first is the need for migration support from other platforms to Linux. This
|
|
support spans a whole range, varying from multi-platform compilers over database
|
|
migration, up to replacement user applications.
|
|
<P>The second is the need to provide more user-friendly administration and
|
|
operation. This may be as well through character-based dialog boxes as through
|
|
GUI systems. In any case their access should be more centralised.
|
|
<P>Other themes which pop up are the following :
|
|
<UL>
|
|
<LI>Enhanced telecommunications support through more comprehensible fax
|
|
packages and a PABX support
|
|
<LI>Enhanced reliabality
|
|
<LI>Numbers and benchmarks on Linux applications and configurations
|
|
<LI>Internationalised accounting packages
|
|
<LI>A customer database system which integrates with other apps
|
|
</UL>
|
|
|
|
<!--===================================================================-->
|
|
<P> <hr> <P>
|
|
<center><H5>Copyright © 1998, Jurgen Defurne <BR>
|
|
Published in Issue 33 of <i>Linux Gazette</i>, October 1998</H5></center>
|
|
|
|
<!--===================================================================-->
|
|
<P> <hr> <P>
|
|
<A HREF="./index.html"><IMG ALIGN=BOTTOM SRC="../gx/indexnew.gif"
|
|
ALT="[ TABLE OF CONTENTS ]"></A>
|
|
<A HREF="../index.html"><IMG ALIGN=BOTTOM SRC="../gx/homenew.gif"
|
|
ALT="[ FRONT PAGE ]"></A>
|
|
<A HREF="./wilson.html"><IMG SRC="../gx/back2.gif"
|
|
ALT=" Back "></A>
|
|
<A HREF="./kunkel.html"><IMG SRC="../gx/fwd.gif" ALT=" Next "></A>
|
|
<P> <hr> <P>
|
|
<!--startcut ==========================================================-->
|
|
</BODY>
|
|
</HTML>
|
|
<!--endcut ============================================================-->
|