LDP
/
old-www
1
1
Fork 0
Code Issues Pull Requests Releases Wiki Activity
old-www/LDP/LG/issue67/tag/13.html

355 lines
14 KiB
HTML
Raw Permalink Blame History

<!--startcut ======================================================= -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<META NAME="generator" CONTENT="lgazmail v1.3E.u">
<TITLE>The Answer Gang 67: LFS: Large File Summit/Support</TITLE>
</HEAD><BODY BGCOLOR="#FFFFFF" TEXT="#000000"
LINK="#3366FF" VLINK="#A000A0">
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
<P> <hr>
<CENTER>
<!-- *** BEGIN navbar *** -->
<!-- *** END navbar *** -->
</CENTER>
</p>
<P> <hr> <P>
<!-- begin tagnav ::::::::::::::::::::::::::::::::::::::::::::::::::-->
<p align="center">
<table width="100%" border="0"><tr>
<td align="right" valign="center"
><IMG ALT="" SRC="../../gx/navbar/left.jpg"
WIDTH="14" HEIGHT="45" BORDER="0" ALIGN="middle" border="0"
><A HREF="..//"
><IMG SRC="../../gx/navbar/toc.jpg" align="middle"
ALT="[ Table Of Contents ]" border="0"></A
><A HREF="../lg_answer67.html"
><IMG SRC="../../gx/dennis/answertoc.jpg" align="middle"
ALT="[ Answer Guy Current Index ]" border="0"></A></td>
<td align="center" valign="center"><A HREF="../lg_answer67.html#greeting"><img align="middle"
src="../../gx/dennis/smily.gif" alt="greetings" border="0"></A> &nbsp;
<A HREF="bios.html">bios</A> &nbsp;
<A HREF="1.html">1</A> &nbsp;
<A HREF="2.html">2</A> &nbsp;
<A HREF="3.html">3</A> &nbsp;
<A HREF="4.html">4</A> &nbsp;
<A HREF="5.html">5</A> &nbsp;
<A HREF="6.html">6</A> &nbsp;
<A HREF="7.html">7</A> &nbsp;
<A HREF="8.html">8</A> &nbsp;
<A HREF="9.html">9</A> &nbsp;
<A HREF="10.html">10</A> &nbsp;
<A HREF="12.html">12</A> &nbsp;
<A HREF="13.html">13</A> &nbsp;
<A HREF="14.html">14</A> &nbsp;
<A HREF="15.html">15</A> &nbsp;
<A HREF="16.html">16</A> &nbsp;
<A HREF="17.html">17</A> &nbsp;
<A HREF="18.html">18</A> &nbsp;
<A HREF="19.html">19</A> &nbsp;
<A HREF="20.html">20</A>
</td>
<td align="left" valign="center"><A HREF="../../tag/kb.html"
><IMG SRC="../../gx/dennis/answerpast.jpg" align="middle"
ALT="[ Index of Past Answers ]" border="0"></A
><IMG ALT="" SRC="../../gx/navbar/right.jpg" align="middle"
WIDTH="14" HEIGHT="45" BORDER="0"></td></tr></table>
</p>
<!-- end tagnav ::::::::::::::::::::::::::::::::::::::::::::::::::::-->
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
<center>
<H1><A NAME="answer">
<img src="../../gx/dennis/qbubble.gif" alt="(?)"
border="0" align="middle">
<font color="#B03060">The Answer Gang</font>
<img src="../../gx/dennis/bbubble.gif" alt="(!)"
border="0" align="middle">
</A></H1>
<BR>
<H4>By Jim Dennis, Ben Okopnik, Dan Wilder, Breen, Chris, and the Gang,
the Editors of Linux Gazette...
and You!
<br>Send questions (or interesting answers) to
<a href="mailto:linux-questions-only@ssc.com">linux-questions-only@ssc.com</a>
</H4>
<p><em><font color="#990000">There is no guarantee that your questions
here will <b>ever</b> be answered. You can be published anonymously
- just let us know!
</font></em></p>
</center>
<p><hr><p>
<!-- endcut ======================================================= -->
<!-- begin 13 -->
<H3 align="left"><img src="../../gx/dennis/qbubble.gif"
height="50" width="60" alt="(?) " border="0"
>LFS: Large File Summit/Support</H3>
<p><strong>From Albert
</strong></p>
<p align="right"><strong>Answered By Jim Dennis
<br></strong></p>
<!-- sig -->
<!-- ::
LFS: Large File Summit/Support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:: -->
<P><STRONG>
Hi,
</STRONG></P>
<P><STRONG>
I have an Intel-based box running RedHat 7.x, 2.4.x kernel and I'm
trying to write code to support large file (&gt;4GB) writes and seeks.
According to the manual pages, the <TT> llseek()</TT> would handle 64-bit
seeks if the kernel supported. However, I can't get my compiler to
recognize the <TT> llseek()</TT> call, perhaps an indication that the 2.4
kernel still doesn't support large files. Do you know of anything
else I could try? Is there any other way of manipulating large
files on 32-bit Linux? Is there going to be a 64-bit Linux version
anytime soon? Please help. Thanks!
</STRONG></P>
<P><STRONG>
-Albert
</STRONG></P>
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
HEIGHT="28" WIDTH="50" BORDER="0"
> [JimD]
I have to say that I'm surprised that this question hasn't come
up before and more often.
</BLOCKQUOTE>
<BLOCKQUOTE>
As you are aware Linux on 32-bit platforms (x86, SPARC/classic,
PowerPC, MIPS, etc) using a signed 32 bit value for off_t (the type
for expressing and return offsets for the <TT> lseek()</TT>, <TT> ftell()</TT>, and related
system calls and library functions). You may know be aware that the
off_t on 64 bit platforms (Alpha, UltraSPARC, IA64/Merced) is already
set to 64 bits.
</BLOCKQUOTE>
<BLOCKQUOTE>
Clearly a signed 32 value can only express an offset up to about 2Gb
(the negative offsets seek from backwards, either from the end of the
file or the current file offset back towards the beginning of the file).
This has led to Linux historical 2Gb file size limit on the most
common platforms.
</BLOCKQUOTE>
<BLOCKQUOTE>
This 2Gb limit was common for UNIX on 32-bit. At some point a number
of UNIX vendors (well, some engineers from the major UNIX vendors and
some major database and other applications vendors) got together and
held a "summit" to discuss some way to overcome this limitation and to
agree on a reasonably portable interface so that the ISV (the independent
software vendors) could write reasonably portable code to cope with this
change. So the specification that they agreed upon has been called the
LFS ("large file summit" or "large file support").
</BLOCKQUOTE>
<BLOCKQUOTE>
Linus used to say that anyone who needed to work with larger files
really should migrate to Alpha or to Merced or some other 64 bit
system. This was around the time that someone had submitted LFS
patches to him. However, somewhere over the years since then he
changed his mind.
</BLOCKQUOTE>
<BLOCKQUOTE>
I suspect that his change had a couple of elements (though I hate to
second guess him; but I'd hate to waste his time asking about it, even
worse). First, I think it became apparent that the need for large
file support was growing much faster than the market for 64 bit systems.
The 64-bit platforms haven't seen nearly the growth that Linux has;
and the cheap availability of very large hard drives and RAID arrays
as exacerbated that need (numbers and sizes of files send to grow larger
as disk capacity make room for them; demand grows to exceed supply).
The increasing use of Linux in imaging compute farms (Hollywood
animation production) and for scientific clustering (Beowulf) --- and
the continued preference for commodity PC/x86 hardware for those
applications has also underscored the need for Linux to support LFS.
</BLOCKQUOTE>
<BLOCKQUOTE>
I suspect that another thing that helped influence Linus opinion on
this is that I think someone submitted a different or cleaned up version
of the LFS patches. I seem to recall that Linus didn't like the
implementation of one of the early submissions --- so his rejection was
on both grounds (implementation, the surmountable one, and perceived
need/elegance --- a design judgement call).
</BLOCKQUOTE>
<BLOCKQUOTE>
Anyway, the 2.4 kernels do support LFS. Now you need to be able to
actually compile software to use this support.
</BLOCKQUOTE>
<BLOCKQUOTE>
What you need to do is sit down and read the libc TexInfo pages
(from a shell prompt issue the command 'info libc' or just 'info'
or from within EMACS or Xemacs use the M-x info function; usually
bound to [F1],[i] or C-h,i )
</BLOCKQUOTE>
<BLOCKQUOTE>
Here's an excerpt:
</BLOCKQUOTE>
<blockquote><pre> - Macro: _LARGEFILE_SOURCE
If this macro is defined some extra functions are available which
rectify a few shortcomings in all previous standards. More
concrete the functions `fseeko' and `ftello' are available.
Without these functions the difference between the ISO C interface
(`fseek', `ftell') and the low-level POSIX interface (`lseek')
would lead to problems.
This macro was introduced as part of the Large File Support
extension (LFS).
- Macro: _LARGEFILE64_SOURCE
If you define this macro an additional set of function gets
available which enables to use on 32 bit systems to use files of
sizes beyond the usual limit of 2GB. This interface is not
available if the system does not support files that large. On
systems where the natural file size limit is greater than 2GB
(i.e., on 64 bit systems) the new functions are identical to the
replaced functions.
The new functionality is made available by a new set of types and
functions which replace existing. The names of these new objects
contain `64' to indicate the intention, e.g., `off_t' vs.
`off64_t' and `fseeko' vs. `fseeko64'.
This macro was introduced as part of the Large File Support
extension (LFS). It is a transition interface for the time 64 bit
offsets are not generally used (see `_FILE_OFFSET_BITS').
- Macro: _FILE_OFFSET_BITS
This macro lets decide which file system interface shall be used,
one replacing the other. While `_LARGEFILE64_SOURCE' makes the
64 bit interface available as an additional interface
`_FILE_OFFSET_BITS' allows to use the 64 bit interface to replace
the old interface.
If `_FILE_OFFSET_BITS' is undefined or if it is defined to the
value `32' nothing changes. The 32 bit interface is used and
types like `off_t' have a size of 32 bits on 32 bit systems.
If the macro is defined to the value `64' the large file interface
replaces the old interface. I.e., the functions are not made
available under different names as `_LARGEFILE64_SOURCE' does.
Instead the old function names now reference the new functions,
e.g., a call to `fseeko' now indeed calls `fseeko64'.
This macro should only be selected if the system provides
mechanisms for handling large files. On 64 bit systems this macro
has no effect since the `*64' functions are identical to the
normal functions.
</pre></blockquote>
<BLOCKQUOTE>
... this is in a discussion about "feature test macros" (allowing
you to code up your #ifdef blocks). You may also need to define
some macros to include support for the LFS functions and APIs.
</BLOCKQUOTE>
<BLOCKQUOTE>
You see in these excerpts hints about the FSF/Glibc maintainers
view of LFS. They consider the adoption of LFS to be a three
stage process; before and old/legacy code, transitional code that
explicity calls the *64 functions, and finally a future where
LFS is the default (controlled by a #define?) and there is optional
support for the older interfaces.
</BLOCKQUOTE>
<BLOCKQUOTE>
Further evidence of this is seen in the following:
</BLOCKQUOTE>
<blockquote><pre> When the sources are compiling with `_FILE_OFFSET_BITS == 64' on a
32 bits machine this function is in fact `fopen64' since the LFS
interface replaces transparently the old interface.
</pre></blockquote>
<BLOCKQUOTE>
(in a discussion on "Opening Streams" and the <TT> fopen()</TT> function).
</BLOCKQUOTE>
<BLOCKQUOTE><BLOCKQUOTE><CODE>
There is a subtle gotchya in using the LFS support with some
<BR>of the f* functions, especially fgetpos for example. Many people
<BR>would use off_t (or even long int!) for storing the return values
<BR>from fgetpos()</TT>. That would be a bug. You should explicitly define
<BR>your variables for storing file positions as fpos_t (which is
<BR>defined as off_t or off64_t as appropriate to your system and the
<BR>#define settings in your sources.
</CODE></BLOCKQUOTE></BLOCKQUOTE>
<BLOCKQUOTE>
That's why I say you should <EM>read</EM> the libc info pages. Be meticulous
in following the prototypes that they offer for these functions.
</BLOCKQUOTE>
<BLOCKQUOTE>
There is a portion of these info pages which describes some of these
problems and recommends that you use the <TT> fgetpos()</TT> and <TT> fsetpos()</TT>
functions in preference to the <TT> ftell()</TT> and <TT> fseek()</TT> functions.
</BLOCKQUOTE>
<!-- end 13 -->
<!--startcut ======================================================= -->
<P> <hr> </p>
<!-- *** BEGIN copyright *** -->
<H5 align="center">This page edited and maintained by the Editors
of <I>Linux Gazette</I>
<a href="http://www.linuxgazette.com/copying.html"
>Copyright &copy;</a> 2001
<BR>Published in issue 67 of <I>Linux Gazette</I> June 2001</H5>
<H6 ALIGN="center">HTML script maintained by
<A HREF="mailto:star@starshine.org">Heather Stern</a> of
Starshine Technical Services,
<A HREF="http://www.starshine.org/">http://www.starshine.org/</A>
</H6>
<!-- *** END copyright *** -->
<P> <hr>
<!-- begin tagnav ::::::::::::::::::::::::::::::::::::::::::::::::::-->
<p align="center">
<table width="100%" border="0"><tr>
<td align="right" valign="center"
><IMG ALT="" SRC="../../gx/navbar/left.jpg"
WIDTH="14" HEIGHT="45" BORDER="0" ALIGN="middle" border="0"
><A HREF="..//"
><IMG SRC="../../gx/navbar/toc.jpg" align="middle"
ALT="[ Table Of Contents ]" border="0"></A
><A HREF="../lg_answer67.html"
><IMG SRC="../../gx/dennis/answertoc.jpg" align="middle"
ALT="[ Answer Guy Current Index ]" border="0"></A></td>
<td align="center" valign="center"><A HREF="../lg_answer67.html#greeting"><img align="middle"
src="../../gx/dennis/smily.gif" alt="greetings" border="0"></A> &nbsp;
<A HREF="bios.html">bios</A> &nbsp;
<A HREF="1.html">1</A> &nbsp;
<A HREF="2.html">2</A> &nbsp;
<A HREF="3.html">3</A> &nbsp;
<A HREF="4.html">4</A> &nbsp;
<A HREF="5.html">5</A> &nbsp;
<A HREF="6.html">6</A> &nbsp;
<A HREF="7.html">7</A> &nbsp;
<A HREF="8.html">8</A> &nbsp;
<A HREF="9.html">9</A> &nbsp;
<A HREF="10.html">10</A> &nbsp;
<A HREF="12.html">12</A> &nbsp;
<A HREF="13.html">13</A> &nbsp;
<A HREF="14.html">14</A> &nbsp;
<A HREF="15.html">15</A> &nbsp;
<A HREF="16.html">16</A> &nbsp;
<A HREF="17.html">17</A> &nbsp;
<A HREF="18.html">18</A> &nbsp;
<A HREF="19.html">19</A> &nbsp;
<A HREF="20.html">20</A>
</td>
<td align="left" valign="center"><A HREF="../../tag/kb.html"
><IMG SRC="../../gx/dennis/answerpast.jpg" align="middle"
ALT="[ Index of Past Answers ]" border="0"></A
><IMG ALT="" SRC="../../gx/navbar/right.jpg" align="middle"
WIDTH="14" HEIGHT="45" BORDER="0"></td></tr></table>
</p>
<!-- end tagnav ::::::::::::::::::::::::::::::::::::::::::::::::::::-->
<P> <hr>
<CENTER>
<!-- *** BEGIN navbar *** -->
<!-- *** END navbar *** -->
</CENTER>
</p>
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
</BODY></HTML>
<!--endcut ========================================================= -->
Reference in New Issue View Git Blame Copy Permalink