mirror of https://github.com/tLDP/LDP
274 lines
16 KiB
XML
274 lines
16 KiB
XML
<?xml version="1.0"?>
|
|
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
|
|
"http://docbook.org/xml/4.1.2/docbookx.dtd" [
|
|
<!ENTITY ver3 "18b.bz2">
|
|
<!ENTITY ver4 "2.4.17.patch">
|
|
]>
|
|
<article id="IO-Perf-HOWTO">
|
|
|
|
<articleinfo>
|
|
|
|
<title>I/O Performance HOWTO</title>
|
|
|
|
<author>
|
|
<firstname>Sharon</firstname>
|
|
<surname>Snider</surname>
|
|
<authorblurb><para><ulink url="mailto:snidersd@us.ibm.com">snidersd@us.ibm.com</ulink></para></authorblurb>
|
|
</author>
|
|
|
|
<legalnotice><para>Linux is a trademark of Linus Torvalds. Other company, products, and service names may be trademarks or service marks of others.</para></legalnotice>
|
|
<pubdate>v1.1, 05/2002</pubdate>
|
|
|
|
|
|
<abstract><para>This HOWTO covers information on available patches for the 2.4 kernel that can improve the I/O performance of your Linux™ operating system. </para></abstract>
|
|
|
|
<revhistory id="revhistory">
|
|
<revision>
|
|
<revnumber>v1.1</revnumber>
|
|
<date>2002-05-01</date>
|
|
<authorinitials>sds</authorinitials>
|
|
<revremark>Updated technical information and links.</revremark>
|
|
</revision>
|
|
<revision>
|
|
<revnumber>v1.0</revnumber>
|
|
<date>2002-04-01</date>
|
|
<authorinitials>sds</authorinitials>
|
|
<revremark>Wrote and converted to DocBook XML.</revremark>
|
|
</revision>
|
|
</revhistory>
|
|
</articleinfo>
|
|
|
|
|
|
<sect1>
|
|
<title>Distribution Policy</title>
|
|
|
|
|
|
<para>The I/O Performance-HOWTO is copyrighted © 2002, by IBM Corporation </para>
|
|
|
|
<para>Permission is granted to copy, distribute, and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation with no Invariant Sections, no Front-Cover text, and no Back-Cover text. A copy of the license can be found at <ulink url="http://www.gnu.org/licenses/fdl.txt"></ulink>.</para>
|
|
</sect1>
|
|
|
|
<sect1 id="INTRODUCTION">
|
|
<title>Introduction</title>
|
|
|
|
<para>This HOWTO provides information on improving the input/output (I/O) performance of the Linux operating system for the 2.4 kernel. Additional patches will be added as they become available.</para>
|
|
|
|
<para>Please send any comments, or contributions via e-mail to <ulink url="mailto:snidersd@us.ibm.com"> Sharon Snider</ulink>. </para>
|
|
</sect1>
|
|
|
|
<sect1 id="OVERVIEW">
|
|
<title>Avoiding Bounce Buffers</title>
|
|
|
|
<para>This section provides information on applying and using the bounce buffer patch on the Linux 2.4 kernel. The bounce buffer patch, written by Jens Axboe, enables device drivers that support direct memory access (DMA) I/O to high-address physical memory to avoid bounce buffers.</para>
|
|
|
|
<para>This document provides a brief overview on memory and addressing in the Linux kernel, followed by information on why and how to make use of the bounce buffer patch.</para>
|
|
|
|
<sect2>
|
|
<title>Memory and Addressing in the Linux 2.4 Kernel</title>
|
|
|
|
<para>The Linux 2.4 kernel includes configuration options for specifying the amount of physical memory in the target computer. By default, the configuration is limited to the amount of memory that can be directly mapped into the kernel's virtual address space starting at PAGE_OFFSET. On i386 systems the default mapping scheme limits kernel-mode addressability to the first gigabyte (GB) of physical memory, also known as low memory. Conversely, high memory is normally the memory above 1 GB. High memory is not directly accessible or permanently mapped by the kernel. Support for high memory is an option that is enabled during <link linkend="config">configuration of the Linux kernel</link>.</para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>The Problem with Bounce Buffers</title>
|
|
|
|
<para>When DMA I/O is performed to or from high memory, an area is allocated in low memory known as a bounce buffer. When data travels between a device and high memory, it is first copied through the bounce buffer.</para>
|
|
|
|
<para>Systems with a large amount of high memory and intense I/O activity can create a large number of bounce buffers that can cause memory shortage problems. In addition, the excessive number of bounce buffer data copies can lead to performance degradation.</para>
|
|
|
|
<para>Peripheral component interface (PCI) devices normally address up to 4 GB of physical memory. When a bounce buffer is used for high memory that is below 4 GB, time and memory are wasted because the peripheral has the ability to address that memory directly. Using the bounce buffer patch can decrease, and possibly eliminate, the use of bounce buffers.</para>
|
|
</sect2>
|
|
|
|
<sect2 id="config">
|
|
<title>Locating the Patch</title>
|
|
|
|
<para> The latest version of the bounce buffer patch is <emphasis>block-highmem-all-&ver3;</emphasis>, and it is available from Andrea Arcangeli's -aa series kernels at
|
|
<ulink url="http://kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/"></ulink>.</para>
|
|
|
|
<sect3>
|
|
<title>Configuring the Linux Kernel to Avoid Bounce Buffers</title>
|
|
|
|
<para>This section includes information on configuring the Linux kernel to avoid bounce buffers. The Linux Kernel-HOWTO at <ulink url="http://www.linuxdoc.org/HOWTO/Kernel-HOWTO.html"></ulink> explains the process of re-compiling the Linux kernel.</para>
|
|
|
|
<para>The following kernel configuration options are required to enable the bounce buffer patch:</para>
|
|
|
|
|
|
<para><emphasis role="bold">Development Code</emphasis> - To enable the configurator to display the <option>High I/O Support</option> option, select the <option>Code maturity level options</option> category and specify "y" to <option>Prompt for development and/or incomplete code/drivers</option>.</para>
|
|
|
|
<para><emphasis role="bold">High Memory Support</emphasis> - To enable support for physical memory that is greater than 1 GB, select the <option>Processor type and features</option> category, and select a value from the <option>High Memory Support</option> option.</para>
|
|
|
|
<para><emphasis role="bold">High Memory I/O Support</emphasis> - To enable DMA I/O to physical addresses greater than 1 GB, select the <option>Processor type and features</option> category, and enter "y" to the <option>HIGHMEM I/O support</option> option. This configuration option is a new option introduced by the bounce buffer patch.</para>
|
|
</sect3>
|
|
|
|
|
|
|
|
<sect3 id="enabled">
|
|
<title>Enabled Device Drivers</title>
|
|
|
|
<para>The bounce buffer patch provides the kernel infrastructure, as well as the SCSI and IDE mid-level driver modifications to support DMA I/O to high memory. Updates for several device drivers to make use of the added support are also included with the patch.</para>
|
|
|
|
<para>If the bounce buffer patch is applied and you configure the kernel to support high memory I/O, many IDE configurations and the device drivers listed below perform DMA I/O without the use of bounce buffers:</para>
|
|
|
|
<para><simplelist columns="1" type="vert">
|
|
<member>aic7xxx_drv.o</member>
|
|
<member>aic7xxx_old.o</member>
|
|
<member>cciss.o</member>
|
|
<member>cpqarray.o</member>
|
|
<member>megaraid.o</member>
|
|
<member>qlogicfc.o</member>
|
|
<member>sym53c8xx.o</member>
|
|
</simplelist></para>
|
|
|
|
</sect3>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Modifying Your Device Driver to Avoid Bounce Buffers</title>
|
|
|
|
<para>If your device drivers are not listed above in the
|
|
<link linkend="enabled">Enabled Device Drivers</link> section, and the device is capable of high-memory DMA I/O, you can modify your device driver to make use of the bounce buffer patch as follows. More information on rebuilding a Linux device driver is available at <ulink url="http://www.xml.com/ldd/chapter/book/index.html"></ulink>.
|
|
</para>
|
|
<orderedlist>
|
|
<listitem>
|
|
<para>A.) For SCSI Adapter Drivers: set the <structfield>highmem_io</structfield> bit in the <structfield>Scsi_Host_Template</structfield> structure. </para>
|
|
|
|
<para>B.) For IDE Adapter Drivers: set the <structfield>highmem</structfield>bit in the <structfield>ide_hwif_t</structfield> structure.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Call <structfield>pci_set_dma_mask(struct pci_dev *pdev, dma_addr_t mask)</structfield> to specify the address bits that the device can successfully use on DMA operations. </para>
|
|
|
|
<para>If DMA I/O can be supported with the specified mask, <structfield>pci_set_dma_mask()</structfield> will set <structfield>pdev->dma_mask</structfield> and return 0. For SCSI or IDE, the mask value will also be passed by the mid-level drivers to <structfield>blk_queue_bounce_limit(request_queue_t *q, u64 dma_addr)</structfield> so that bounce buffers are not created for memory directly addressable by the device. Drivers other than SCSI or IDE must call <structfield>blk_queue_bounce_limit()</structfield> directly. </para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Use <structfield>pci_map_page(dev, page, offset, size, direction)</structfield>, instead of <structfield>pci_map_single(dev, address, size, direction)</structfield> to map a memory region so that it is accessible by the peripheral device. <structfield>pci_map_page() </structfield> supports both high and low memory.</para>
|
|
|
|
|
|
<para>The <structfield>address </structfield> parameter for <structfield>pci_map_single()</structfield> correlates to the <structfield>page</structfield> and<structfield> offset </structfield> parameters for <structfield>pci_map_page()</structfield>. Use the <structfield>virt_to_page()</structfield> macro to convert an <structfield>address</structfield> to a <structfield>page </structfield> and <structfield>offset</structfield>. The <structfield>virt_to_page()</structfield> macro is defined by including pci.h. For example:</para>
|
|
|
|
<para><screen><structfield>void *address;</structfield></screen>
|
|
|
|
<screen><structfield>struct page *page;</structfield></screen>
|
|
|
|
<screen><structfield>unsigned long offset;</structfield></screen>
|
|
|
|
<screen><structfield>page = virt_to_page(address);</structfield></screen>
|
|
|
|
<screen><structfield>offset = (unsigned long) address & ~PAGE_MASK;</structfield></screen></para>
|
|
|
|
|
|
<para>Call <structfield>pci_unmap_page()</structfield> after the DMA I/O transfer is complete to remove the mapping established by <structfield>pci_map_page()</structfield>.</para>
|
|
|
|
<note>
|
|
<para><structfield>pci_map_single()</structfield> is implemented using <structfield>virt_to_bus()</structfield>. <structfield>virt_to_bus()</structfield> handles low memory addresses only. Drivers supporting high memory should no longer call <structfield>virt_to_bus()</structfield> or <structfield>bus_to_virt()</structfield>.</para></note></listitem>
|
|
|
|
|
|
<listitem><para>If your driver calls <structfield>pci_map_sg()</structfield> to map a scatter-gather DMA operation, your driver should set the <structfield>page</structfield> and <structfield>offset</structfield> fields instead of the <structfield>address</structfield> field of the <structfield>scatterlist</structfield> structure. Refer to step 3 for converting an <structfield>address</structfield> to a <structfield>page</structfield> and <structfield>offset</structfield>.</para>
|
|
|
|
<note><para>If your driver is already using the PCI DMA API, continue to use <structfield>pci_map_page() </structfield> or <structfield>pci_map_sg()</structfield> as appropriate. However, do not use the <structfield>address</structfield> field of the <structfield>scatterlist</structfield> structure.</para></note>
|
|
|
|
</listitem>
|
|
</orderedlist></sect2>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>Raw I/O Variable-Size Optimization Patch</title>
|
|
|
|
<para>This section provides information on the raw I/O variable-size optimization patch for the Linux 2.4 kernel written by Badari Pulavarty. This patch is also known as the RAW VARY or PAGESIZE_io patch. </para>
|
|
|
|
<para>The raw I/O variable-size patch changes the block size used for raw I/O from <structfield>hardsect_size</structfield> (normally 512 bytes) to 4 kilobytes (K). The patch improves I/O throughput and CPU utilization by reducing the number of buffer heads needed for raw I/O operations.</para>
|
|
|
|
<sect2>
|
|
<title>Locating the Patch</title>
|
|
|
|
<para>You can download the patch from one of the following locations:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem><para>Andrea Arcangeli has made the patch available at
|
|
<ulink url="http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.18pre7aa2/"></ulink>.
|
|
The name of the file is <emphasis>10_rawio-vary-io-1</emphasis>.</para></listitem>
|
|
|
|
<listitem><para>Alan Cox has included the patch in the <emphasis>2.4.18pre9-ac2</emphasis> kernel patch. The patch is available at <ulink url="http://www.kernel.org/pub/linux/kernel/people/alan/linux-2.4/2.4.18/"></ulink>. </para></listitem>
|
|
|
|
<listitem><para>The patch is available from SourceForge at <ulink url="http://sourceforge.net/projects/lse/io"></ulink>. The latest version is <emphasis>PAGESIZE_io-&ver4;</emphasis>.</para> </listitem>
|
|
|
|
</itemizedlist>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Modifying Your Driver for the Raw I/O Variable-Size Optimization Patch</title>
|
|
|
|
<para>In previous versions of this patch, changes were enabled for all drivers. However, the 2.4.17 and later versions of the patch enable the changes only for the Adaptec, Qlogic ISP1020, and IBM ServerRAID drivers. All other drivers for version 2.4.17 and later must be modified to make use of the patch by setting the <structfield>can_do_varyio</structfield> bit in the <structfield>Scsi_Host_Template</structfield> structure.</para>
|
|
|
|
|
|
|
|
<note><para>Drivers that have the raw I/O patch enabled must support buffer heads of variable sizes <structfield>(b_size)</structfield> in a single I/O request because <structfield>hardsect_size</structfield> is used until the data buffer is aligned on a 4 K boundary.</para>
|
|
<para>Additional information is available on rebuilding Linux device drivers at <ulink url="http://www.xml.com/ldd/chapter/book/index.html"></ulink>.</para></note>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>I/O Request Lock Patch</title>
|
|
|
|
<para>This section provides information on the I/O request lock patch, also known as the scsi concurrent queuing patch (sior1), written by Johnathan Lahr. </para>
|
|
|
|
<para>The I/O request lock patch improves SCSI I/O performance on Linux 2.4 multi-processor systems by providing concurrent I/O request queuing. There are significant I/O performance and CPU utilization improvements possible by enabling multi-processors to concurrently drive multiple block devices.</para>
|
|
|
|
<para>Before the patch is applied block I/O requests are queued one at a time holding the global spin lock, <structfield> io_request_lock</structfield>. Once the patch is applied, SCSI requests are queued while holding the lock specific to the queue associated with the request. Requests that are made to different devices are queued concurrently, and requests that are made to the same device are queued serially.</para>
|
|
|
|
<sect2>
|
|
<title>Locating the Patch</title>
|
|
|
|
<para>You can download the I/O request patch from Sourceforge at <ulink url="http://sourceforge.net/projects/lse/io"></ulink>. The latest version is <emphasis>sior1-v1.2416</emphasis>. Patches that enable concurrent queuing for specific drivers are also available at SourceForge. The patch for the Emulex SCSI/FC is <emphasis>lpfc_sior1-v0.249</emphasis> and the patch for Adaptec SCSI is <emphasis>aic_sior1-v0.249</emphasis>.</para>
|
|
|
|
<para></para>
|
|
</sect2>
|
|
|
|
<sect2>
|
|
<title>Modifying Your Driver for the I/O Request Lock Patch</title>
|
|
|
|
<para>The I/O request lock patch installs concurrent queuing capability into the SCSI midlayer. Concurrent queuing is
|
|
activated for each SCSI adapter device driver. To activate the driver, the <structfield>concurrent_queue</structfield> field in the <structfield>Scsi_Host_Template</structfield> structure must be set when the driver is registered.</para>
|
|
|
|
<note><para>Drivers that activate concurrent queuing must ensure that any access of the <structfield>request_queue</structfield> by the driver is protected by the <structfield>request_queue.queue_lock</structfield>.</para>
|
|
|
|
<para>Additional information is available on rebuilding device drivers at <ulink url="http://www.xml.com/ldd/chapter/book/index.html"></ulink>.</para></note>
|
|
</sect2>
|
|
</sect1>
|
|
|
|
<sect1>
|
|
<title>Additional Resources</title>
|
|
|
|
<para>The following list of Web sites provides additional information on modifying device drivers and configuring the Linux kernel.</para>
|
|
|
|
<itemizedlist>
|
|
<listitem><para>Information on Dynamic DMA mapping is available at
|
|
<ulink url="http://lwn.net/2001/0712/a/dma-interface.php3"></ulink>.</para></listitem>
|
|
|
|
<listitem><para>Kernel-HOWTO is available from the Linux Documentation Project at
|
|
<ulink url="http://www.linuxdoc.org/HOWTO/Kernel-HOWTO.html"></ulink>.</para></listitem>
|
|
|
|
<listitem><para>Linux Device Drivers, 2nd Edition published by O'Reilly is available online at
|
|
<ulink url="http://www.xml.com/ldd/chapter/book/index.html"></ulink>.</para></listitem>
|
|
</itemizedlist>
|
|
</sect1>
|
|
</article>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|