This commit is contained in:
gferg 2002-07-12 13:47:39 +00:00
parent 68fe133eb5
commit 290f18d0b4
3 changed files with 1904 additions and 1736 deletions

View File

@ -2525,7 +2525,7 @@ using Samba. </Para>
SMP-HOWTO</ULink>,
<CiteTitle>Linux SMP HOWTO</CiteTitle>
</Para><Para>
<CiteTitle>Updated: October 2000</CiteTitle>.
<CiteTitle>Updated: July 2002</CiteTitle>.
Reviews main issues related to SMP configuration under Linux. </Para>
</ListItem>

View File

@ -746,7 +746,7 @@ system. </Para>
SMP-HOWTO</ULink>,
<CiteTitle>Linux SMP HOWTO</CiteTitle>
</Para><Para>
<CiteTitle>Updated: October 2000</CiteTitle>.
<CiteTitle>Updated: July 2002</CiteTitle>.
Reviews main issues related to SMP configuration under Linux. </Para>
</ListItem>

View File

@ -3,8 +3,9 @@
<article>
<title>Linux SMP HOWTO
<author>David Mentré, <tt/David.Mentre@irisa.fr/
<date>v1.12.1, 25 october 2000
<author>Enkh Tumenbayar, <tt/etumenba@ouray.cudenver.edu/
<date>v1.4, 9 july 2002
<abstract>
This HOWTO reviews main issues (and I hope solutions) related to SMP
@ -24,27 +25,20 @@ http://www.fsf.org/licenses/fdl.html.
<p>
Linux works on SMP (Symmetric Multi-Processors) machines. SMP support
was introduced with kernel version 2.0, and has improved steadily ever
since. The kernel locking granularity is much finer in 2.2.x than in
2.0.x, which enables better performance when processes are accessing the
kernel!
since.
<p>
HOWTO maintained by David Mentré (<htmlurl name="David.Mentre@irisa.fr"
url="mailto:David.Mentre@irisa.fr">). The latest
HOWTO maintained by Enkh Tumenbayar (<htmlurl name="etumenba@ouray.cudenver.edu"
url="mailto:etumenba@ouray.cudenver.edu">). The latest
edition of this HOWTO can be found at
<itemize>
<item> <htmlurl url="http://www.irisa.fr/prive/mentre/smp-howto/"
name="http://www.irisa.fr/prive/mentre/smp-howto/"> (France)
<item> <htmlurl url="http://www.phy.duke.edu/brahma/smp-faq/"
name="http://www.phy.duke.edu/brahma/smp-faq/"> (USA)
<item> <htmlurl url="http://ouray.cudenver.edu/~etumenba/smp-howto/"
name="http://ouray.cudenver.edu/~etumenba/smp-howto/"> (USA)
</itemize>
<p>
If you want to contribute to this HOWTO, I would prefer a diff against the
<url name="SGML version"
url="http://www.irisa.fr/prive/mentre/smp-howto/smp-howto.sgml"> of this
document, but any remarks (in plain text) will be greatly
appreciated. If you send me an email about this HOWTO, please include a
If you want to contribute to this HOWTO, I would prefer a diff against the SGML version.
If you send me an email about this HOWTO, please include a
tag like <tt>[Linux SMP HOWTO]</> in the <tt>Subject:</> field of your
e-mail. It helps me to automatically sort mails (and you will have a faster
reply <tt>;)</>).
@ -52,7 +46,7 @@ reply <tt>;)</>).
<p>
This HOWTO is an improvement of a <url name="first draft"
url="http://www.ihoc.net/linux-smp-faq-draft.html"> made by <bf>Chris
Pirih</bf>.
Pirih</bf> and maintained by <bf>David Mentre</bf>.
<p>
All information contained in this HOWTO is provided "as is." All
@ -97,6 +91,11 @@ support SMP; the latter two probly won't ever.
That is, I'm going to hack on MIPS-SMP as soon as I get a SMP box ...
</descrip>
<item> <bf>Does SMP distribute the threads among the processors or is the library the one in charge of it?</bf>
<p>
(<bf>Matti Aarnio</bf>) The way Linux implements threads is to treat them at scheduling the same way as any process - thread just happens to share several resources of the originating process; memory space, file descriptors. See clone(2) for part of explanation.
<item> <bf>How do I make a Linux SMP kernel?</bf>
<p>
Most Linux distributions don't provide a ready-made SMP-aware kernel,
@ -105,6 +104,34 @@ your own kernel yet, this is a great reason to learn how. Explaining
how to make a new kernel is beyond the scope of this document; refer to
the Linux Kernel Howto for more information. (<bf>C. Polisher</bf>)
<p>
Configure the kernel and answer Y to CONFIG_SMP.
<p>
If you are using LILO, it is handy to have both SMP and non-SMP
kernel images on hand. Edit /etc/lilo.conf to create an entry
for another kernel image called "linux-smp" or something.
<p>
The next time you compile the kernel, when running a SMP kernel,
edit linux/Makefile and change "MAKE=make" to "MAKE=make -jN"
(where N = number of CPU + 1, or if you have tons of memory/swap
you can just use "-j" without a number). Feel free to experiment
with this one.
<p>
Of course you should time how long each build takes :-)
Example:
<code>
make config
time -v sh -c 'make dep ; make clean install modules modules_install'
</code>
<p>
If you are using some Compaq MP compliant machines you will need to set
the operating system in the BIOS settings to "Unix
<p>
In kernel series 2.0 up to but not including 2.1.132, uncomment the
<tt/SMP=1/ line in the main Makefile (<tt>/usr/src/linux/Makefile</>).
<p>
@ -140,7 +167,6 @@ processor. The MTRR support contains code that solves such processor
misconfiguration.
<p>
You must rebuild all your kernel and kernel modules when changing to and
from SMP mode. Remember to <tt>make modules</> and <tt>make
modules_install</> (from <bf>Alan Cox</bf>).
@ -221,6 +247,13 @@ Linux kernel subsystems are fully threaded: networking, VFS, VM, IO,
block/page caches, scheduling, interrupts, signals, etc. (<bf>Ingo
Molnar</bf>)
<item> <bf>What has changed between 2.2.x and 2.4.x kernels?</bf>
<p>
<bf>(Mark Hahn)</bf> In many parts of the kernel, there's little relation between 2.2 and 2.4. One of the biggest changes is SMP - not just the evolutionary fine-graining of locks, but the radically revamped VM, memory management, interrupt handling that's basically unrelated to 2.2, fairly revolutionary net changes (thread and zero-copy), etc.
<p>
In short, 2.2 doesn't use the hardware like 2.4 does.
<item> <bf>Does Linux SMP support processor affinity?</bf>
<p>
<descrip>
@ -252,7 +285,7 @@ include:
<item> forcing a cpu to run _only_ one process (and its children)
<item> getting information about a CPU's state
<item> creating/destroying sets of processors, to which processes may be
bound
bound
</itemize>
</descrip>
@ -570,7 +603,60 @@ info.
</descrip>
</enum>
<sect1>MultiProcessor Specification Support (MPS)
<p>
(<bf>Randy Dunlap</bf>) Linux supports MPS (MP spec.) version 1.1 and 1.4.
<p>
Linux doesn't have full support for all of MPS version 1.4.
<p>
Experience has shown that Linux usually works best when the BIOS is configure for MP Spec. version 1.1 if that is an option in your system's BIOS. I don't see why the MP Spec. version should matter to Linux, but it would be an interesting exercise to find out the differences as presented by BIOS tables, to determine why Linux fails with MP Spec. version 1.4 in some cases, and to fix Linux so that this wouldn't matter.
<p>
This document summarizes the major changes in MP spec. version 1.4 and their support status in Linux.
<p>
<sect2> Symmetric I/O Mode
<p>
The hardware must support a mode of operation in which the system can switch easily to Symmetric I/O mode from PIC or Virtual Wire mode. When the operating system is ready to swtich to MP operation, it writes a 01H to the IMCR register, if that register is implemented, and enables I/O APIC Redirection Table entries. The hardware must not require any other action on the part of software to make the transition to Symmetric I/O mode.
<p>
Linux recognizes and supports this MP configuration mode.
<p>
<sect2> Floating Point Exception Interrupt
<p>
For PC/AT compatibility, the bootstrap processor must support DOS-compatible FPU execution and exception handling while running in either of the PC/AT-compatible modes. This means that floating point error signals from the BSP must be routed to the interrupt request 13 signal, IRQ13, when the system is in PIC or virtual wire mode. While floating point error signals from an application processor need not be routed to IRQ13, platform designers may choose to connect the two. For example, connecting the floating point error signal from application processors to IRQ13 can be useful in the case of a platform that supports dynamic choice of BSP during boot.
<p>
In symmetric mode, a compliant system supports only on-chip floating point units, with error signaling via interrupt vector 16. Operating systems must use interrupt vector 16 to manage floating point exceptions when the system is in symmetric mode.
<p>
Linux does not use the floating point interrupt at all except in genuine i386 processor systems which are not SMP-capable. [In these systems, if they wire the FPU exception line in the PC/AT-compatible way, a run-time check for #MF exception availability is performed. If the #MF exception is available, then Linux handles this interrupt if it happens. (<bf>Maciej W. Rozycki</bf>)
<p>
<sect2> Multiple I/O APIC Configurations
<p>
Multiple I/O APICs are supported in Linux.
<p>
<sect2> MP Configuration Table
<p>
This table was made optional in MPS version 1.4. If the table isn't present, one of the default configurations should be used. An extended section was also added to it for new table entry types.
<p>
Linux supports the optional MP Configuration Table and uses a default configuration if the MP Config. Table is not present.
<p>
Linux tolerates extended section table entries by skipping over them if they are found. Data in the extended table entries is not used.
<p>
<sect2> MP Configuration Table Header Fields
<p>
New or changed fields for MP Spec. version 1.4:
<p>
<itemize>
<item> OEM Table Pointer: supported in Linux
<item> Extended Table Length: supported (tolerated, skipped) in Linux
<item> Extended Table Checksum: supported (tolerated, skipped) in Linux
</itemize>
<p>
<sect2> Extended MP Configuration Table Entries
<p>
Entry types for System Address Space Mapping, Bus Hierarchy Descriptor, and Compatibility Bus Address Space Modifier are defined.
<p>
Linux skips over (does not use) these extended MP Configuration table entries. Apparently this isn't critical to any shipping systems.
<sect>x86 architecture specific questions
@ -583,13 +669,7 @@ info.
<item> <bf>Can I use my Cyrix/AMD/non-Intel CPU in SMP?</bf>
<p>
<bf>Short answer:</bf> no.
<p>
<bf>Long answer:</bf> Intel claims ownership to the APIC SMP scheme, and
unless a company licenses it from Intel they may not use it. There are
currently no companies that have done so. (This of course can change in
the future) FYI - Both Cyrix and AMD support the non-proprietary OpenPIC
SMP standard but currently there are no motherboards that use it.
Yes. Current AMD Athlon MP processors support SMP with the AMD 760MP chipset. There are several boards available featuring this chipset, e.g. from Tyan, ASUS, etc. Athlon/SMP is supported by recent 2.4.x kernels and also by the latest 2.2.x kernels. (<bf>David Haring</bf>)
<item> <bf>Why doesn't my old Compaq work?</bf>
<p>
@ -612,16 +692,20 @@ recommandation. (<bf>Daniel Roesen</bf>)
<item> Save changes and reboot
</enum>
<item> <bf>I can't get my Compaq SystemPro work in SMP mode.</bf>
<p>
(<bf>Maciej W. Rozycki</bf>) Chances are that your Compaq do not make use of 82489DX APICs as they were introduced quite late -- in late 1992 or early 1993. There used to be i486 machines that implemented the APIC architecture. 82489DX is the chip that was used for them and it contained a local APIC unit and an I/O APIC unit.
<item> <bf>Why doesnt my ALR work?</bf>
<p>
>From <bf>Robert Hyatt</bf> : ALR Revolution quad-6 seems quite safe,
From <bf>Robert Hyatt</bf> : ALR Revolution quad-6 seems quite safe,
while some older revolution quad machines without P6 processors seem
"iffy"...
<item> <bf>Why does SMP go so slowly?</bf> or <bf>Why does one CPU show
a very low bogomips value while the first one is normal?</bf>
<p>
>From <bf>Alan Cox</bf>: If one of your CPU's is reporting a very low
From <bf>Alan Cox</bf>: If one of your CPU's is reporting a very low
bogomips value the cache is not enabled on it. Your vendor probably
provides a buggy BIOS. Get the patch to work around this or better yet
send it back and buy a board from a competent supplier.
@ -644,13 +728,15 @@ FPU support.
<item> <bf>Is there any advantage of Intel MP 1.4 over 1.1 specification?</bf>
<p>
Nope (according to Alan <tt/:)/ ), 1.4 is just a stricker specs of 1.1.
<p>
Please see the <url name="Useful Pointers" url="SMP-HOWTO-8.html"> for comparison between MP 1.4 and 1.1.
<item> <bf>Why does the clock drift so rapidly when I run linux SMP?</bf>
<p>
This is known problem with IRQ handling and long kernel locks in
the 2.0 series kernels. Consider upgrading to a later 2.2 kernel.
<p>
>From <bf>Jakob Oestergaard</bf>: Or, consider running xntpd. That should
From <bf>Jakob Oestergaard</bf>: Or, consider running xntpd. That should
keep your clock right on time. (I think that I've heard that enabling
RTC in the kernel also fixes the clock drift. It works for me! but I'm
not sure whether that's general or I'm just being lucky)
@ -681,15 +767,53 @@ kernel loads the correct root filesystem at boot.
(<bf>Robert M. Hyatt</bf>) With 3 cpus, do you have a terminator in the
4th slot?
<item> <bf>During boot machine hang signaling an IOAPIC problem</bf>
<item> <bf>During boot machine hang signaling an "unexpected IO-APIC" warning</bf>
<p>
Try boot options "noapic" (<bf>John Aldrich</bf>) and/or "reboot=bios" (<bf>Terry Shull</bf>).
<item> <bf>My system locks up during heavy NFS traffic</bf>
<bf>Short Answer:</bf> Change your MP setting from 1.4 to 1.1 (BIOS option), and boot with "noapic" option at boot prompt.
<bf>Long Answer:</bf> This message has nothing to do with your performance problems or why all interrupts go to one CPU. This message is for the ACPI(IO-APIC) maintainers to keep an eye on when there is new hardware. (<bf>Earle Nietzel</bf>)
<p>
To summarize the article found in official kernel documentation:
<enum>
<item> The "unexpected IO-APIC" is just an indicator that your motherboard is not on the whitelist.
<item> Cat your /proc/interrupts and if you see any line with IO-APIC then everything is fine because IO-APIC IRQ's are enabled.
</enum>
<item><bf>Do I need to do change MP from 1.4 to 1.1 and boot with (<bf>noapic</bf>) at the same time?</bf>
<p>
It depends.
<p>
I found that I do not need to turn off IO-APIC if I backed down from MP 1.4 and 1.1. Apparently some Xeon-based boards need to do both, but ASUS CUV4X boards do not. Turning off IO-APIC support needlessly imposes a probably small performance penalty on ASUS owners. (<bf>Vladimir G. Ivanovic</bf>)
<p>
Some IBM Netfinity machines will have problems initializing the onboard SCSI controller if MPS 1.1 is selected. Each possible LUB of each possible device on each possible bus will be queried with a timeout. Booting takes a uselessly long time. (<bf>E. Robert Bogusta</bf>)
<p>
There are reports that system with ASUS4X-DLS motherboard ran fine with IO-APIC enabled with MP 1.4.
<p>
For CUV4X-D motherboard, disabling the IDE controllers you probably can boot with MP 1.4 and APIC enabled.
<item><bf>Is there performance loss by running "noapic"?</bf>
<p>
(<bf>David Mentre</bf>) It has minor impact, except if you have high interrupt load (i.e., nearly nobody).
<item> <bf>My motherboard is an ASUS-CUV4X-DLS with the VIA 694XDP chipset. If I boot with the noapic flag, the machine boots fine and /proc/cpuinfo show sboth processors. However, /proc interrupts does not show any sharing of the interrupts.</bf>
<p>
Probably you need to upgrade your BIOS version to 1010.
<item> <bf>What are pros and cons of Xeons vs. Athlons?</bf>
<p>
Xeon's chipset (440GX) and accompanying motherboard (supermicro S2DGE) I'd be using is probably (much?) more reliable and well-supported under Linux SMP than Athlons' (AMD 760/760MP) simply because they've been around longer and through many more iterations.
<p>
Xeon's larger cache (1mb on the dual 400's I'm considering) might give performance enhancement (and given that I don't have only a single scientific code I'm planning to run on this, it's probably not helpful to test benchmark specifically for my code).
<p>
Athlon's significiantly has faster clock rate (along with full-speed L2 cache in Thunderbirds, although at only 384kb) and much higher memory bandwidth with PC2100 DDR memory could help a lot.
<p>
Cost is unclear until 760MP boards and PC2100 memory are released, but it will probably be ~$950 to get two 1GHz 385km L2 Thunderbirds, dual motherboard and 512mb of ECC PC2100 vs ~$750 to get two 400MHz 1mb L2 Xeons, dual motherboard and 512mb of ECC PC100. (<bf>Daniel Freedman</bf>)
<item><bf>My system locks up during heavy NFS traffic</bf>
<p>
Try the later 2.2.x kernels and the knfsd patches. This is
currently under investigation. (<bf>Wade Hampton</bf>)
<p>
<item> <bf>My system locks up with no oops messages</bf>
<p>
@ -993,6 +1117,30 @@ UltraLinux has ran on a 14 CPUs machine (see the <url name="dmesg
output" url="http://lwn.net/1998/1210/a/dm-sparc.html">) and on a
Starfire E10000 with 24 CPUs (see the <url name="dmesg
output" url="http://linuxcare.com.au/anton/e10000/dmesg_24.shtml">).
<p>
The SparcStation 10 and SparcStations 20 are SMP capable machine and according to the <url name="FAQABOSS" url="http://fagaboss.sunhelp.org"> the following combinations are known to work:
<itemize>
<item> 2xSM40 ( model 402 )
<item> 2xSM41 ( model 412 )
<item> 2xSM51 ( model 512 )
<item> 2xSM512 ( model 514 )
<item> 2xSM61 ( model 612 )
<item> 2xSM71 ( model 712 )
<item> 2xSM81 ( model 812 )
</itemize>
And, as stated earlier, CPU modules in SparcStations 10 and can run a different clock speeds, the following ones _SHOULD_ work:
<itemize>
<item> 2xSM50
<item> SM41, SM51
<item> SM41, SM61
<item> SM51, SM61
<item> SM71, SM81
</itemize>
How does it performs? Well, it is fast, really fast. Some of the java Demos can run faster on a dual HyperSparc 125Mhz 128MB ( ywing ) than on a dual celeron BP6 433@433Mhz 192MB ( calimero ). The same applies for the Gimp. When it comes to compiling calimero runs faster than ywing. Both computers running 2.2.16 kernel and calimero's hard disk subsystem is full SCSI.
<p>
One important detail when you plan to have different CPU modules in your computer is to have the same kind of modules, you cannot mix SuperSparc and HyperSparc for example, but you can have an odd number of CPUs, for example 3. They are said to be able to run modules at different clock speed as written in this article form AcesHardware , but I have not witnessed it. (<bf>Lionel, trollhunter Bouchpan-Lerus-Juery</bf>)
<sect1>Specific problem related to Sparc SMP support
<p>
@ -1003,12 +1151,6 @@ The only known problem, and one we don't intend to fix, is that if you
build an SMP kernel for 32-bit (ie. non-ultrasparc) systems, this kernel
will not work on sun4c systems.
<sect1>SMP specific limit with current kernel (2.2)
<p>
(<bf>David Miller</bf>) There is a bug in the include/linux/tasks.h
header file, it needs to define NR_CPUS to 64 on UltraSparc as this is
the upper limit for the hardware we support :-)
<sect>PowerPC architecture specific questions
<p>
@ -1081,9 +1223,6 @@ url="http://yara.ecn.purdue.edu/~pplinux/">
<item> <url name="Linux Parallel Processing HOWTO"
url="http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html">
<item> <bf>(outdated)</bf> <url name="Linux SMP home page"
url="http://www.uk.linux.org/SMP/title.html">
<item> linux-smp mailing list
<p>
To <bf>subscribe</bf>, send <tt/subscribe linux-smp/ in the message body
@ -1127,6 +1266,12 @@ url="http://cesdis.gsfc.nasa.gov/linux/drivers/">
<item> <url name="Dual Celeron System"
url="http://kikumaru.w-w.ne.jp/pc/celeron/index_e.html">
<item> <url name="LaTeX document describing implementation of MultiProcessor Linux"
url="http://www.linuxhq.com/kernel/v2.4/doc/smp.tex">
<item> <url name="IRQ affinity"
url="http://www.linuxhq.com/kernel/v2.4/doc/IRQ-affinity.txt.html">
</itemize>
<p>
@ -1236,7 +1381,9 @@ co-processor.
<item> <bf>IRQ</bf> Interrupt ReQuest.
<item> <bf>EBDA</bf> ??
<item> <bf>EBDA</bf> Extended BIOS Data Area.
<item> <bf>ACPI</bf> Advanced Configuration and Power Interface.
<item> <bf>oops</bf> Internal kernel error.
@ -1326,6 +1473,18 @@ of locks. :-)
<p>
<descrip>
<tag/v1.14, 9 july 2002
<itemize>
<item> First release since in almost two years
<item> Added article Multiprocessor Specification Support (<bf>Randy Dunlap</bf>)
<item> Added explanation of "unexpected IO-APIC" problem
<item> Added a note on changes between 2.2.x and 2.4.x kernels
<item> Added new section update on SPARC (<bf>Lionel, trollhunter Bouchpan-Lerust-Juery</bf>)
<item> Added various questions and answers
<item> Deleted "SMP specific limit with current kernel (2.2)": outdated
<item> Changed word "current" from kernel 2.2 documents
</itemize>
<tag/v1.12.1, 25 october 2000
<itemize>
<item> Put all authors in Bryant, Hartner, Qi and Venkitachalam paper
@ -1355,7 +1514,7 @@ by <bf>Ludovic Danigo</bf>)
<item> Deleted wrong links (<bf>Johan Ekenberg</bf>)
</itemize>
<tag/v1.9.1, 28 September 2000
<tag/v1.9.1, 28 september 2000
<itemize>
<item> updated with a submission from <bf>Stig Telfer</bf> detailing SMP support
on API Alpha systems
@ -1424,18 +1583,18 @@ vs. kernel threads (hww)
<item> new item about floppy access and sound (hww)
</itemize>
<tag/v1.2, 27 septembre 1999
<tag/v1.2, 27 september 1999
<itemize>
<item> name change: this document is now a HOWTO. TWD, and fast! (<bf>Guylhem Aznar</bf>)
</itemize>
<tag/v1.1, 26 septembre 1999
<tag/v1.1, 26 september 1999
<itemize>
<item> added a link to first Chris Pirih FAQ draft
<item> expanted an IRQ related problems
</itemize>
<tag/v1.00, 25 septembre 1999
<tag/v1.00, 25 september 1999
<itemize>
<item> first upgrade in a long long time!
<item> reprocessed the whole FAQ: 2.2 is here and 2.4 soon
@ -1729,6 +1888,15 @@ Many thanks to those who help me to maintain this HOWTO:
<item> Gero Wedemann
<item> Christopher Allen Wing
<item> Leonard N. Zubkoff
<item> Mark Hahn
<item> David Haring
<item> David Mentre
<item> Earle Nietzel
<item> Rick Lindsley
<item> Vladimir G. Ivanovic
<item> Daniel Freedman
<item> Matti Aarnio
<item> Maciej W. Rozycki
</enum>
</article>