old-www/LDP/LG/issue59/nazario.html

708 lines
22 KiB
HTML

<!--startcut ==============================================-->
<!-- *** BEGIN HTML header *** -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML><HEAD>
<title>dmesg explained LG #59</title>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#0000AF"
ALINK="#FF0000">
<!-- *** END HTML header *** -->
<CENTER>
<A HREF="http://www.linuxgazette.com/">
<H1><IMG ALT="LINUX GAZETTE" SRC="../gx/lglogo.jpg"
WIDTH="600" HEIGHT="124" border="0"></H1></A>
<!-- *** BEGIN navbar *** -->
<IMG ALT="" SRC="../gx/navbar/left.jpg" WIDTH="14" HEIGHT="45" BORDER="0" ALIGN="bottom"><A HREF="lambert.html"><IMG ALT="[ Prev ]" SRC="../gx/navbar/prev.jpg" WIDTH="16" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="index.html"><IMG ALT="[ Table of Contents ]" SRC="../gx/navbar/toc.jpg" WIDTH="220" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><A HREF="../index.html"><IMG ALT="[ Front Page ]" SRC="../gx/navbar/frontpage.jpg" WIDTH="137" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="http://www.linuxgazette.com/cgi-bin/talkback/all.py?site=LG&article=http://www.linuxgazette.com/issue59/nazario.html"><IMG ALT="[ Talkback ]" SRC="../gx/navbar/talkback.jpg" WIDTH="121" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><A HREF="../faq/index.html"><IMG ALT="[ FAQ ]" SRC="./../gx/navbar/faq.jpg"WIDTH="62" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="nielsen.html"><IMG ALT="[ Next ]" SRC="../gx/navbar/next.jpg" WIDTH="15" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><IMG ALT="" SRC="../gx/navbar/right.jpg" WIDTH="15" HEIGHT="45" ALIGN="bottom">
<!-- *** END navbar *** -->
<P>
</CENTER>
<!--endcut ============================================================-->
<H4 ALIGN="center">
"Linux Gazette...<I>making Linux just a little more fun!</I>"
</H4>
<P> <HR> <P>
<!--===================================================================-->
<center>
<H1><font color="maroon">dmesg explained</font></H1>
<H4>By <a href="mailto:jose@biochemistry.cwru.edu">Jos&eacute; Nazario</a> and <a href="mailto:nxk3@po.cwru.edu">Natarajan Krishnaswami</A></H4>
</center>
<P> <HR> <P>
<!-- END header -->
<h2>
Abstract
</h2>
<p>
Often someone will write to a Linux help list asking for help with a
particular device they want to get working under Linux, and a standard
reply is "check the output of the dmesg command". This leaves a lot of new
users befuddled, and this document is here to hopefully help them navigate
this powerful debugging tool. Two sets of kernel boot messages are
presented and annotated, from an i386 system and a Linux-Pmac system.
<p>
<h2>
Introduction
</h2>
<p>
The Linux kernel is the central interface between the user and the
hardware. As such, it has to incorporate support for hardware if you are
to use it. Often, though, cryptic device names are used by the system,
making it difficult at first inspection to determine if some particular
hardware is supported. The command 'dmesg', which is used to print kernel
messages, is very useful in determining if a piece of hardware has been
found, and if so, what the system is referring to it as.
<p>
This artcle, including the title and format of the dmesg comments, were
directly inspired and copied from the OpenBSD Explained article by the
same name. I felt one on Linux would be useful for people.
<p>
The manpage for dmesg is quite simple:
<pre>
DMESG(8) DMESG(8)
NAME
dmesg - print or control the kernel ring buffer
SYNOPSIS
dmesg [ -c ] [ -n level ] [ -s bufsize ]
DESCRIPTION
dmesg is used to examine or control the kernel ring
buffer.
<BR></pre>
Upon boot, the dmesg output is from the kernel booting, showing the
devices it has found and if it has been able to configure them at all
(aside from userland configuration). This log is also available in the
file <tt>/var/log/dmesg</tt>.
<p>
<h2>
Kernel output on an i386 system
</h2>
<p>
Shown below is a dmesg from an x86 system immediately after boot. The
output is indented by several space, and comments and descriptions are
left justified.
</p>
<ul>
<tt>
Linux version 2.2.14-5.0 (root@porky.devel.redhat.com) (gcc version egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)) #1 Tue Mar 7 20:53:41 EST 2000
</ul>
</tt>
<p>
First up is the kernel version (2.2.14) and build (5), along with who built it, with what compile it was built, and when it wass built. This can be some inportant information, as some kernel versions and the GCC project don't interact correctly.
<p>
<ul>
<tt>
Detected 300683434 Hz processor.
<p>
</ul>
</tt>
My K6/2-300 processor running at 300 MHz.
<p>
<ul>
<tt>
Console: colour VGA+ 80x25
<p>
</ul>
</tt>
A standard PC console screen (15 inch monitor).
<p>
<ul>
<tt>
Calibrating delay loop... 599.65 BogoMIPS
<p>
</ul>
</tt>
The useless benchmark of BogoMIPS. They're bogus (hence the name), but are often used as a relative processor speed indicator.
<p>
<ul>
<tt>
Memory: 63008k/65536k available (1084k kernel code, 412k reserved, 968k data, 64k init, 0k bigmem)
<p>
</ul>
</tt>
My memory statistics. My machine has 64MB of real memory.
<p>
<ul>
<tt>
Dentry hash table entries: 262144 (order 9, 2048k)
<p>
</ul>
</tt>
The dentry cache (dcache) represents the kernel's view of the namespace of
mounted filesystems. There's pretty good documentation of it in
Documentation/filesystems/vfs.txt in the kernel source tree.
<p>
<ul>
<tt>
Buffer cache hash table entries: 65536 (order 6, 256k)
<p>
</ul>
</tt>
In 2.2, the buffer cache is used for caching and aggregating data for
writes to block devices. After 2.3.6, it is used for caching fs metadata,
such as inode information.
<p>
<ul>
<tt>
Page cache hash table entries: 16384 (order 4, 64k)
<p>
</ul>
</tt>
In 2.2, the page (VM) cache is used for caching swap, read and mmap data
(which was bad, because shared writable mappings were ugly). After 2.3.6,
it also is used for write data (i.e., the buffer and page caches are
mostly unified), and all became happiness and light (sorta like BSD).
<p>
<ul>
<tt>
VFS: Diskquotas version dquot_6.4.0 initialized
<p>
</ul>
</tt>
My kernel support quotas (though I'm not using them).
<p>
<ul>
<tt>
CPU: AMD AMD-K6(tm) 3D processor stepping 00
<p>
</ul>
</tt>
A quick identification of the processor.
<p>
<ul>
<tt>
Checking 386/387 coupling... OK, FPU using exception 16 error reporting.<br>
Checking 'hlt' instruction... OK.
<p>
</ul>
</tt>
I seem to recall there being some Intel processor issues, which the kernel has to know about if it's to invoke corrections.
<p>
<ul>
<tt>
POSIX conformance testing by UNIFIX<br>
PCI: PCI BIOS revision 2.10 entry at 0xfb490
<p>
</ul>
</tt>
And we start the probing of the PCI bus for peripherals.
<p>
<ul>
<tt>
PCI: Using configuration type 1<br>
PCI: Probing PCI hardware<br>
PCI: 00:38 [1106/0586]: Work around ISA DMA hangs (00)<br>
Activating ISA DMA hang workarounds.<br>
Linux NET4.0 for Linux 2.2
<p>
</ul>
</tt>
This kernel supports the Net4 networking codebase, which has a lot of features yet to be fully utilized.
<p>
<ul>
<tt>
Based upon Swansea University Computer Society NET3.039<br>
NET4: Unix domain sockets 1.0 for Linux NET4.0.<br>
NET4: Linux TCP/IP 1.0 for NET4.0<br>
IP Protocols: ICMP, UDP, TCP, IGMP
<p>
</ul>
</tt>
My core IP protocols supported. While not needed, IGMP can be fun. Note that some networks do not support muticasting.
<p>
<ul>
<tt>
TCP: Hash tables configured (ehash 65536 bhash 65536)<br>
Initializing RT netlink socket<br>
Starting kswapd v 1.5<br>
Detected PS/2 Mouse Port.
<p>
</ul>
</tt>
Should be quite obvious...
<p>
<ul>
<tt>
Serial driver version 4.27 with MANY_PORTS MULTIPORT SHARE_IRQ enabled<br>
ttyS00 at 0x03f8 (irq = 4) is a 16550A<br>
ttyS01 at 0x02f8 (irq = 3) is a 16550A
<p>
</ul>
</tt>
The information about my serial ports.
<p>
<ul>
<tt>
pty: 256 Unix98 ptys configured<br>
apm: BIOS version 1.2 Flags 0x07 (Driver version 1.9)
<p>
</ul>
</tt>
My motherboard supports the APM standard for sleeping.
<p>
<ul>
<tt>
Real Time Clock Driver v1.09<br>
RAM disk driver initialized: 16 RAM disks of 4096K size
<p>
</ul>
</tt>
My kernel supports RAM disks. While I'm not using any most days, sometimes I do use them; if you have the memory, they make a real fast filesystem (like /tmp or, for a webserver, the main pages loaded).
<p>
<ul>
<tt>
VP_IDE: IDE controller on PCI bus 00 dev 39<br>
VP_IDE: not 100% native mode: will probe irqs later<br>
ide0: BM-DMA at 0xe000-0xe007, BIOS settings: hda:DMA, hdb:DMA<br>
ide1: BM-DMA at 0xe008-0xe00f, BIOS settings: hdc:DMA, hdd:DMA
<p>
</ul>
</tt>
My IDE controllers.
<p>
<ul>
<tt>
hda: Maxtor 51369U3, ATA DISK drive
<p>
</ul>
</tt>
My hard drive in the machine.
<p>
<ul>
<tt>
hdb: IDE/ATAPI CD-ROM 32X, ATAPI CDROM drive
<p>
</ul>
</tt>
My CDROM drive.
<p>
<ul>
<tt>
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14<br>
hda: Maxtor 51369U3, 12949MB w/2048kB Cache, CHS=6577/64/63<br>
hdb: ATAPI 16X CD-ROM drive, 128kB Cache
<p>
</ul>
</tt>
Disk information.
<p>
<ul>
<tt>
Uniform CDROM driver Revision: 2.56<br>
Floppy drive(s): fd0 is 1.44M<br>
FDC 0 is a post-1991 82077
<p>
</ul>
</tt>
Floppy drive information.
<p>
<ul>
<tt>
md driver 0.90.0 MAX_MD_DEVS=256, MAX_REAL=12<br>
raid5: measuring checksumming speed<br>
raid5: MMX detected, trying high-speed MMX checksum routines<br>
pII_mmx : 761.238 MB/sec<br>
p5_mmx : 726.567 MB/sec<br>
8regs : 447.675 MB/sec<br>
32regs : 308.610 MB/sec<br>
using fastest function: pII_mmx (761.238 MB/sec)
<p>
</ul>
</tt>
A bunch of RAID and MD (used in multiple device devices, like disk arrays) information, again not used.
<p>
<ul>
<tt>
scsi : 0 hosts.<br>
scsi : detected total.
<p>
</ul>
</tt>
While the kernel supports SCSI, I'm not using any on this host.
<p>
<ul>
<tt>
md.c: sizeof(mdp_super_t) = 4096<br>
Partition check:<br>
hda: hda1 hda2 &lt; hda5 hda6 &gt;
<p>
</ul>
</tt>
My disk partition information. The brackets indicate extended partitions.
<p>
<ul>
<tt>
autodetecting RAID arrays<br>
autorun ...<br>
... autorun DONE.
<p>
</ul>
</tt>
Like I said above, I'm using not using any RAID arrays.
<p>
<ul>
<tt>
VFS: Mounted root (ext2 filesystem) readonly.
<p>
</ul>
</tt>
At this point we're almost done with the kernel and ready to start the system.
<p>
<ul>
<tt>
Freeing unused kernel memory: 64k freed<br>
Adding Swap: 66488k swap-space (priority -1)<br>
ne2k-pci.c:vpre-1.00e 5/27/99 D. Becker/P. Gortmaker http://cesdis.gsfc.nasa.gov/linux/drivers/ne2k-pci.html<br>
ne2k-pci.c: PCI NE2000 clone 'RealTek RTL-8029' at I/O 0xe800, IRQ 11.<br>
eth0: RealTek RTL-8029 found at 0xe800, IRQ 11, 00:80:AD:41:22:10.
<p>
</ul>
</tt>
My ethernet device is a PCI NE2000 based device. (A real cheap NIC, but almost every OS supports it.)
<p>
<ul>
<tt>
VFS: Disk change detected on device fd(2,0)
<p>
</ul>
</tt>
At this point, the kernel is done booting and we're ready to start /sbin/init (unless we supplied some information about init upon boot). The system then starts rc.sysinit and begins normal boot operations. The kernel has finished booting.
<p>
<h2>Kernel output on a Linux-Pmac system
</h2>
<p>
And, for comparison's sake, this is the output of Linux 2.2 on a PowerPC
system. Again, the dmesg output is indented and the description and
comments are left justified. For this system I was using BootX, which loads
the kernel into memory from within the MacOS, then completes bootstrapping
it after ditching the MacOS. Options can be passed to the kernel, as you
would with LILO on an Intel based PC, from within the app.
<p>
<ul>
<tt>
device tree used 17860 bytes
<p>
</ul>
</tt>
PowerPC systems use what is known as OpenFirmware, rather than a PC like
BIOS, and it has a 'device tree', which is arranged a bit like a UNIX
filesystem. This one uses about 16kb.
<p>
<ul>
<tt>
Total memory = 72MB; using 512kB for hash table (at c0280000)
<p>
</ul>
</tt>
Again, total physical RAM available.
<p>
<ul>
<tt>
Linux version 2.2.6-15apmac (root@video.linuxppc.org) (gcc version
egcs-2.91.66 19990314 (egcs-1.1.2 release)) #1 Mon May 31 03:54:09
EDT 1999
<p>
</ul>
</tt>
Kernel version (2.2.6 build 15 on a Pmac), who built it
(root@video.linuxppc.org), what version of gcc (or egcs), and when it was
built. This can be diagnostic as some versions of the Linux kernel don't
play well with some versions of GCC.
<p>
<ul>
<tt>
PCI bus 0 controlled by bandit at f2000000<br>
PCI bus 1 controlled by chaos at f0000000
<p>
</ul>
</tt>
Two PCI busses. OpenFirmware (OF) calls their controllers bandit and
chaos.
<p>
<ul>
<tt>
System has 32 possible interrupts<br>
via_calibrate_decr: decrementer_count = 100001 (600010 ticks)<br>
Console: colour dummy device 80x25<br>
Calibrating delay loop... 239.21 BogoMIPS
<p>
</ul>
</tt>
Ahh... sweet BogoMIPS, which mean pretty much nothing.
<p>
<ul>
<tt>
Memory: 69900k available (1532k kernel code, 2184k data, 112k init)
[c0000000,c4800000]
<p>
</ul>
</tt>
After the initial bootstrap, about 68 MB of memory is left, having
reserved some for the kernel and core system.
<p>
<ul>
<tt>
VFS: Diskquotas version dquot_6.4.0 initialized<br>
POSIX conformance testing by UNIFIX<br>
PCI: Probing PCI hardware
<p>
</ul>
</tt>
Let the PCI probing begin!
<p>
<ul>
<tt>
USB: Universal USB Driver v$Revision: 1.2 $<br>
USB-OHCI: USB Open Host Controller Interface Driver<br>
USB-HUBD: UUSBD Hub Driver v$Revision: 1.2 $<br>
USB-HIDD: USB Human Interface Devices Driver v$Revision: 1.2 $<br>
USB-HIDBP: USB HID Boot Protocol Driver v$Revision: 1.2 $<br>
<p>
</ul>
</tt>
USB support is in the kernel, though I have no USB devices on the system.
<p>
<ul>
<tt>
adb devices: [2]: 2 2 [3]: 3 1
<p>
</ul>
</tt>
ADB, or Apple Desktop Bus, has two devices, the keyboard and the mouse.
We'll see them detected below.
<p>
<ul>
<tt>
Linux NET4.0 for Linux 2.2<br>
Based upon Swansea University Computer Society NET3.039<br>
NET4: Unix domain sockets 1.0 for Linux NET4.0.<br>
NET4: Linux TCP/IP 1.0 for NET4.0<br>
IP Protocols: ICMP, UDP, TCP, IGMP
<p>
</ul>
</tt>
Core networking protocols (NET4) built into the kernel. The IGMP comes
from multicast support (see Stevens for more info).
<p>
<ul>
<tt>
Starting kswapd v 1.5<br>
MacOS display is /bandit/IMS,tt128mb8
<p>
</ul>
</tt>
Recall that bandit is the PCI controller (bus0). I use an IMS Twin Turbo 8
MB video chipset.
<p>
<ul>
<tt>
Console: switching to colour frame buffer device 80x30<br>
fb0: IMS TT (TVP) frame buffer; 8MB vram; chip version 2<br>
Monitor sense value = 0x73f, using video mode 6 and color mode 0.<br>
fb1: control display adapter
<p>
</ul>
</tt>
Some video settings. On PowerMac hardware, sometimes this can be important
if you've loaded a kernel level video driver, which can cause havoc on
some systems. This is useful stuff to check on a PPC that has some video
problems (ie in X).
<p>
<ul>
<tt>
ADB mouse at 3, handler set to 4<br>
ADB keyboard at 2, handler set to 5<br>
PowerMac Z8530 serial driver version 1.01<br>
tty00 at 0xc900e020 (irq = 15) is a Z8530 ESCC, port = modem<br>
tty01 at 0xc9011000 (irq = 16) is a Z8530 ESCC, port = printer
<p>
</ul>
</tt>
It found the ADB devices and also the two serial ports. It's useful to know
which ones are which. Recall that Macintosh machines have one labeled modem
and one labeled printer, so this is useful info to know.
<p>
<ul>
<tt>
pty: 256 Unix98 ptys configured<br>
Macintosh ADB mouse driver installed.<br>
DMA sound driver installed, using 4 buffers of 32k.<br>
RAM disk driver initialized: 16 RAM disks of 4096K size<br>
loop: registered device at major 7<br>
fd0: SWIM3 floppy controller
<p>
</ul>
</tt>
fd0, or the floppy drive 0, on PowerMacs uses a controller called 'SWIM3'.
Unfortunately, in this instance of the kernel, it's broken.
<p>
<ul>
<tt>
md driver 0.90.0 MAX_MD_DEVS=256, MAX_REAL=12<br>
USB-HUBM: Starting kusbdd (pid 2)<br>
USBD: No USB hosts found<br>
linear personality registered<br>
raid0 personality registered<br>
raid1 personality registered<br>
raid5: measuring checksumming speed
<p>
</ul>
</tt>
If I had RAID controllers, this would be neat. As such, it's just sucking
up space in my kernel.
<p>
<ul>
<tt>
8regs : 140.970 MB/sec<br>
32regs : 122.301 MB/sec<br>
using fastest function: 8regs (140.970 MB/sec)<br>
scsi0 : MESH<br>
scsi1 : 53C94<br>
scsi : 2 hosts.<br>
<p>
</ul>
</tt>
I have two SCSI controllers, both onboard, with one being internal (the
MESH one) and one being external (the 53C94 controller).
<p>
<ul>
<tt>
mesh: target 0 synchronous at 10.0 MB/s<br>
Vendor: IBM Model: DPES-31080 Rev: S31S<br>
Type: Direct-Access ANSI SCSI revision: 01 CCS<br>
Detected scsi disk sda at scsi0, channel 0, id 0, lun 0<br>
Vendor: QUANTUM Model: CTS80S Rev: 4.05<br>
Type: Direct-Access ANSI SCSI revision: 02<br>
Detected scsi disk sdb at scsi0, channel 0, id 1, lun 0<br>
mesh: target 3 synchronous at 5.0 MB/s<br>
Vendor: MATSHITA Model: CD-ROM CR-8005A Rev: 4.0i<br>
Type: CD-ROM ANSI SCSI revision: 02<br>
Detected scsi CD-ROM sr0 at scsi0, channel 0, id 3, lun 0<br>
mesh: target 4 synchronous at 10.0 MB/s<br>
Vendor: SEAGATE Model: ST31200N Rev: 8648<br>
Type: Direct-Access ANSI SCSI revision: 02<br>
Detected scsi disk sdc at scsi0, channel 0, id 4, lun 0<br>
Vendor: QUANTUM Model: FIREBALL_TM2110S Rev: 300X<br>
Type: Direct-Access ANSI SCSI revision: 02<br>
Detected scsi disk sdd at scsi1, channel 0, id 2, lun 0<br>
scsi : detected 1 SCSI cdrom 4 SCSI disks total.<br>
Uniform CDROM driver Revision: 2.54<br>
SCSI device sda: hdwr sector= 512 bytes. Sectors= 2118144 [1034 MB]
[1.0 GB]<br>
SCSI device sdb: hdwr sector= 512 bytes. Sectors= 166200 [81 MB]
[0.1 GB]<br>
SCSI device sdc: hdwr sector= 512 bytes. Sectors= 2061108 [1006 MB]
[1.0 GB]<br>
SCSI device sdd: hdwr sector= 512 bytes. Sectors= 4124736 [2014 MB]
[2.0 GB]
<p>
</ul>
</tt>
Disk detection. I'm not a big fan of how Linux names its disks (in the
order it finds them), but it found all of my devices.
<p>
<ul>
<tt>
PPP: version 2.3.3 (demand dialling)<br>
TCP compression code copyright 1989 Regents of the University of
California<br>
PPP line discipline registered.
<p>
</ul>
</tt>
Kernel PPP support. Compressed TCP headers in PPP is a great feature, by the way, in keeping message overhead low.
<p>
<ul>
<tt>
eth0: MACE at 00:05:02:10:e6:6d, chip revision 9.64
<p>
</ul>
</tt>
Good old eth0, the ethernet device. It's MACE on a PowerMac. If this had
been a modular driver, it wouldn't have shown up here but only after the
module had been inserted.
<p>
<ul>
<tt>
Partition check:<br>
sda: sda1 sda2 sda3 sda4 sda5 sda6<br>
sdb: sdb1<br>
sdc: sdc1 sdc2 sdc3 sdc4<br>
sdd: sdd1 sdd2 sdd3 sdd4<br>
<p>
</ul>
</tt>
The partition check is useful for knowing what the disks are laid out as.
<p>
<ul>
<tt>
md.c: sizeof(mdp_super_t) = 4104<br>
autodetecting RAID arrays<br>
autorun ...<br>
... autorun DONE.<br>
VFS: Mounted root (ext2 filesystem) readonly.<br>
Freeing unused kernel memory: 112k init 32k prep<br>
Adding Swap: 131428k swap-space (priority -1)
<p>
</ul>
</tt>
And we're done booting.
<p>
<h2>Concluding remarks
</h2>
<p>
Like I noted above at the end of the i386 dmesg output, the kernel, once
finished, then moves on to <tt>/sbin/init</tt> unless an argument poiting it elsewhere
has been passed to the kernel at boot time. An example would be telling the
kernel "<tt>init=/bin/sh</tt>", such that it would execute a shell upon boot, rather
than <tt>/sbin/init</tt> (and what follows). Note that the kernel only mounts the
root filesystem read-only, so if all you do is boot the kernel you have
to mount your disks read-write in order to affect changes on them (ie
editing <tt>/etc/passwd</tt> to rescue root's password).
<p>
While this isn't the most thorough of jobs, I hope that this little tour
has been enjoyable for everyone, and educational.
<!-- *** BEGIN copyright *** -->
<P> <hr> <!-- P -->
<H5 ALIGN=center>
Copyright &copy; 2000, Jose Nazario and Natarajan Krishnaswami.
Copying license <A HREF="../copying.html">http://www.linuxgazette.com/copying.html</A><BR>
Published in Issue 59 of <i>Linux Gazette</i>, November 2000</H5>
<!-- *** END copyright *** -->
<!--startcut ==========================================================-->
<HR><P>
<CENTER>
<!-- *** BEGIN navbar *** -->
<IMG ALT="" SRC="../gx/navbar/left.jpg" WIDTH="14" HEIGHT="45" BORDER="0" ALIGN="bottom"><A HREF="lambert.html"><IMG ALT="[ Prev ]" SRC="../gx/navbar/prev.jpg" WIDTH="16" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="index.html"><IMG ALT="[ Table of Contents ]" SRC="../gx/navbar/toc.jpg" WIDTH="220" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><A HREF="../index.html"><IMG ALT="[ Front Page ]" SRC="../gx/navbar/frontpage.jpg" WIDTH="137" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="http://www.linuxgazette.com/cgi-bin/talkback/all.py?site=LG&article=http://www.linuxgazette.com/issue59/nazario.html"><IMG ALT="[ Talkback ]" SRC="../gx/navbar/talkback.jpg" WIDTH="121" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><A HREF="../faq/index.html"><IMG ALT="[ FAQ ]" SRC="./../gx/navbar/faq.jpg"WIDTH="62" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="nielsen.html"><IMG ALT="[ Next ]" SRC="../gx/navbar/next.jpg" WIDTH="15" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><IMG ALT="" SRC="../gx/navbar/right.jpg" WIDTH="15" HEIGHT="45" ALIGN="bottom">
<!-- *** END navbar *** -->
</CENTER>
</BODY></HTML>
<!--endcut ============================================================-->