346 lines
13 KiB
HTML
346 lines
13 KiB
HTML
<!--startcut ======================================================= -->
|
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
|
|
<html>
|
|
<head>
|
|
<META NAME="generator" CONTENT="lgazmail v1.2M.b">
|
|
<TITLE>The Answer Guy 43: Out of Space....or Inodes? All Sparsity Lost?</TITLE>
|
|
</HEAD><BODY BGCOLOR="#FFFFFF" TEXT="#000000"
|
|
LINK="#3366FF" VLINK="#A000A0">
|
|
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
|
|
<H4>"The Linux Gazette...<I>making Linux just a little more fun!</I>"</H4>
|
|
<P> <hr> <P>
|
|
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
|
|
<center>
|
|
<H1><A NAME="answer">
|
|
<img src="../../gx/dennis/qbubble.gif" alt="(?)"
|
|
border="0" align="middle">
|
|
<font color="#B03060">The Answer Guy</font>
|
|
<img src="../../gx/dennis/bbubble.gif" alt="(!)"
|
|
border="0" align="middle">
|
|
</A></H1>
|
|
<BR>
|
|
<H4>By James T. Dennis,
|
|
<a href="mailto:linux-questions-only@ssc.com">linux-questions-only@ssc.com</a><BR>
|
|
LinuxCare,
|
|
<A HREF="http://www.linuxcare.com/">http://www.linuxcare.com/</A>
|
|
</H4>
|
|
</center>
|
|
|
|
<p><hr><p>
|
|
<!-- endcut ======================================================= -->
|
|
<!-- begin 9 -->
|
|
<H3 align="left"><img src="../../gx/dennis/qbubble.gif"
|
|
height="50" width="60" alt="(?) " border="0"
|
|
>Out of Space....or Inodes? All Sparsity Lost?</H3>
|
|
|
|
|
|
<p><strong>From Derek Wyatt on Fri, 11 Jun 1999
|
|
</strong></p>
|
|
<!-- ::
|
|
Out of Space....or Inodes? All Sparsity Lost?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
:: -->
|
|
<P><STRONG>
|
|
Hi James,
|
|
</STRONG></P>
|
|
<P><STRONG>
|
|
I know this question has been asked before (i'v read the 'stuff' in the
|
|
previous columns) but this one has an interesting wrinkle which i can't
|
|
answer. I hope you can
|
|
<IMG SRC="../../gx/dennis/smily.gif" ALT=":)"
|
|
height="24" width="20" align="middle">
|
|
</STRONG></P>
|
|
<P><STRONG>
|
|
I was copying a new slackware 4.0 installation from one disk to another.
|
|
Incidently, i used two methods, using <tt>tar</tt> and
|
|
<tt>find | afio</tt>, etc... It was the right way. I've done it
|
|
many many many times before.
|
|
</STRONG></P>
|
|
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
|
|
HEIGHT="28" WIDTH="50" BORDER="0"
|
|
>
|
|
You might not have preserved allocation "holes" (the
|
|
"sparsity") of the files as you transferred them.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
When a program opens a file in write mode and does a <tt>seek()</tt>
|
|
or <tt>lseek()</tt> to some point that is more than a block past the
|
|
end of the file, the Linux native filesystems (<tt>ext2</tt>, <tt>minix</tt>,
|
|
etc) will leave the unnecessary blocks unallocated. This
|
|
is possible in inode based filesystems (not in FAT/MS-DOS
|
|
formatted filesystems).
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
These filesystems treat reads into such unallocated regions
|
|
of a file as blocks of NULs (ASCII zero characters).
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
So, you use normal read and write commands in sequence
|
|
(like '<tt>cp</tt>' and '<tt>cat</tt>' to to copy files) then you'll expand
|
|
any such "holes" in the allocation map (the inode's list
|
|
of clusters) into blocks of NULs and the file will
|
|
take more space than it used to.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
One possibility is that you used to have such "sparse"
|
|
files and that your method of copying them failed to
|
|
preserve those "holes." You could use the GNU '<tt>cp
|
|
--sparse=always</tt>' option to restore the "holes" in
|
|
selected files (or create new ones wherever there are
|
|
blocks of NULs in the data).
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
Most files are not sparse --- in fact there are only a
|
|
couple of old dbm style libraries that used to create
|
|
them in normal system use (the sendmail <tt>newaliases</tt>
|
|
command used to be a prime example).
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
I don't think this accounts for your whole problem (i.e.
|
|
it's not wholly a "holey" problem).
|
|
</BLOCKQUOTE>
|
|
<P><STRONG><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)"
|
|
HEIGHT="28" WIDTH="50" BORDER="0"
|
|
>
|
|
Now, the problem is this: after the copy was complete, i used the
|
|
slackware bootdisk and rootdisk to reboot things nice and clean to test
|
|
the disk, and every copy i tried to do (including running lilo)
|
|
resulted in a "file too large" error message. A '<tt>df</tt>' reported that the
|
|
disk had lots of space on it, as did '<tt>du</tt>' (as did basic common sense
|
|
<IMG SRC="../../gx/dennis/smily.gif" ALT=":)"
|
|
height="24" width="20" align="middle"> ).
|
|
The disk became completely unusable until i destroyed it and reinstalled
|
|
slackware from scratch.
|
|
</STRONG></P>
|
|
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
|
|
HEIGHT="28" WIDTH="50" BORDER="0"
|
|
>
|
|
Perhaps you should look at the output of the '<TT>df -i</TT>' command.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
Your Linux filesystems actually have couple of resources that
|
|
are depleted at different rates from one another. If you
|
|
have lots of small files than you are using up inodes faster
|
|
than data blocks. The normal '<tt>df</tt>' command output reports
|
|
on your free <EM>data</EM> space. '<tt>df -i</tt>' reports on the
|
|
inode utilization.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
So, its possible that you ran out of inodes even if you have
|
|
plenty of disk space.
|
|
</BLOCKQUOTE>
|
|
<P><STRONG><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)"
|
|
HEIGHT="28" WIDTH="50" BORDER="0"
|
|
>
|
|
Now, considering that the disk was just a 'raw' disk with data on it (ie.
|
|
it wasn't the root partition at this point) I have no idea why it would
|
|
behave like this. I tried eliminating <TT>/proc/*</TT> just for the heck of it,
|
|
but to no avail.
|
|
</STRONG></P>
|
|
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
|
|
HEIGHT="28" WIDTH="50" BORDER="0"
|
|
>
|
|
It is very easy to accidentally copy/archive your <TT>/proc</TT>
|
|
(which is generally no harm). The problem is that you
|
|
can easily restore that to your new root fs and
|
|
mount a real <TT>/proc</TT> back over the restored <em>copy/snapshot</em>
|
|
of the state of your proc fs back when you did the backup.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
I recommend that you use the <tt>find -xdev</tt> or <tt>-mount</tt> options
|
|
to prevent your find from crossing filesystem boundaries.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
Let's say you have <TT>/</TT>, <TT>/usr/</TT>, <TT>/var</TT>,
|
|
<TT>/usr/local</TT>, and <TT>/home</TT> as local filesystems. To use
|
|
'<tt>cpio</tt>' to back them up you could use a command like
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE><BLOCKQUOTE><CODE>
|
|
find / /usr/ /var /usr/local /home -mount -depth | cpio ....
|
|
</CODE></BLOCKQUOTE></BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
... to feed only the file names that are on that list
|
|
of filesystem to <tt>cpio</tt>.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
When using '<tt>cpio</tt>' you can preserve sparsity while
|
|
COPYING IN your data using the <tt>--sparse</tt> option.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
Of course '<tt>tar</tt>' works differently from '<tt>cpio</tt>' in just
|
|
about everyway that you could think of. You have to
|
|
use something a bit more like:
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE><BlockQuote><Code>
|
|
tar cSlf /dev/st0 / /usr/ /var /usr/local /home
|
|
</Code></BlockQuote></BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
... where the <tt>-S</tt> preserves sparsity (during archive
|
|
creation; and apparently NOT during restoration
|
|
unless the archive was correctly created). Personally
|
|
I think that this is a bug in GNU tar.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE><em>[ I suppose fortcing someone to use -S (or --sparse)
|
|
when restoring offers the </em>ability<em> to desparsify the
|
|
file, on a new filesystem which has room for it. Why it
|
|
should be the default to not come out as it went in, though,
|
|
I've no idea. -- Heather ]
|
|
</em></BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
The <tt>tar -l</tt> option instructs '<tt>tar</tt>' not to cross fs boundaries.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
The key general point here is that you might have
|
|
mounted <TT>/proc</TT> or any other filesystem over a non-empty
|
|
mount point. I personally think that the distribution
|
|
maintainers should modify the default <tt>rc*</tt> scripts
|
|
so that they won't mount filesystems over non-empty
|
|
directories. I'd modify them to uniquely rename and
|
|
remake any non-empty directory before mounting something
|
|
over it (and post a nastygram to syslog, of course).
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE><em>[ I disagree; I often touch a file named
|
|
<code>THIS_IS_THE_UNDERLYING_MOUNTPOINT</code> for mount points,
|
|
and I've actually had occasional administrative use for a few
|
|
files to sit in the underlying drive in case that fs doesn't
|
|
mount. Usually, notes about what should have been there,
|
|
although I suppose that could be the content of the
|
|
commentary filename above. -- Heather ]
|
|
</em></BLOCKQUOTE>
|
|
<P><STRONG><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)"
|
|
HEIGHT="28" WIDTH="50" BORDER="0"
|
|
>
|
|
I hope i've given you enough information here. I've been using linux for
|
|
years and have never come across something like this.
|
|
</STRONG></P>
|
|
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
|
|
HEIGHT="28" WIDTH="50" BORDER="0"
|
|
>
|
|
I really don't know if I've given you the answer that will
|
|
help for your situation. I've just tried to explain the
|
|
most common couple of "out of space" situations that I've
|
|
seen and heard about --- with the hopes that you're
|
|
situation isn't more bizarre.
|
|
</BLOCKQUOTE>
|
|
<BLOCKQUOTE>
|
|
If your space problems persist through a reboot then
|
|
you don't have the old "open anonymous file" problem
|
|
(which I've described on other occasions). It's also a
|
|
very good idea to run <tt>fsck</tt> (do a filesystem check)
|
|
when you can't account for your missing space.
|
|
</BLOCKQUOTE>
|
|
<P><STRONG><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)"
|
|
HEIGHT="28" WIDTH="50" BORDER="0"
|
|
>
|
|
Thanks alot! And keep up the good work
|
|
<IMG SRC="../../gx/dennis/smily.gif" ALT=":)"
|
|
height="24" width="20" align="middle">
|
|
</STRONG></P>
|
|
<P><STRONG>
|
|
Sincerely,
|
|
<br>Derek Quinn Wyatt
|
|
</STRONG></P>
|
|
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
|
|
HEIGHT="28" WIDTH="50" BORDER="0"
|
|
>
|
|
I hope it helps.
|
|
</BLOCKQUOTE>
|
|
<!-- sig -->
|
|
|
|
<!-- end 9 -->
|
|
<hr width="40%" align="center">
|
|
|
|
<!-- begin 8 -->
|
|
<H3 align="left"><img src="../../gx/dennis/qbubble.gif"
|
|
height="50" width="60" alt="(?) " border="0"
|
|
>Need to learn details. Any suggestions?</H3>
|
|
|
|
<p><strong>From Derek Wyatt on Fri, 11 Jun 1999
|
|
</strong></p>
|
|
<p><strong>
|
|
Jim, thanks a lot for your quick reply.
|
|
</strong></p>
|
|
<p><strong>
|
|
I don't think this applies in my situation but there are a few things here
|
|
that are news to me. It's good to know. If you were here, i'm sure you
|
|
could figure it out
|
|
<IMG SRC="../../gx/dennis/smily.gif" ALT=":)"
|
|
height="24" width="20" align="middle">
|
|
But you're not. I simply need to learn more to
|
|
solve something like this myself.
|
|
</strong></p>
|
|
<p><strong>
|
|
My knowledge of linux, how to use it and administrate it is in the upper
|
|
intermediate level, i think. In order to get it higher, i need to learn
|
|
about the details of the filesystem, the kernel, processes, etc etc...
|
|
How would you recommend going about this sort of thing? Are there some
|
|
online documents, or some books you would recommend? How about some
|
|
source code to pour over?
|
|
</strong></p>
|
|
<p><strong>
|
|
Thanks again
|
|
<br><IMG SRC="../../gx/dennis/smily.gif" ALT=":)"
|
|
height="24" width="20" align="middle">
|
|
<br>Derek
|
|
</strong></p>
|
|
|
|
<blockquote><img src="../../gx/dennis/bbubble.gif"
|
|
height="50" width="60" alt="(!) " border="0">
|
|
Most Linux distributions, certainly all the large ones,
|
|
contain the option to install the source code.
|
|
</blockquote>
|
|
|
|
<blockquote>The Linux-Kernel mailing list
|
|
(<a href="http://www.tux.org/lkml/">http://www,tux.org/lkml/</a>)
|
|
has archives mirrored in a few places, and several of the documents
|
|
in the Linux Documentation Project
|
|
(<a href="http://metalab.unc.edu/LDP/">http://metalab,unc.edu/LDP/</a>)
|
|
are more rather than less technical.
|
|
</blockquote>
|
|
|
|
<!-- end 8 -->
|
|
<!--startcut ======================================================= -->
|
|
<P> <hr> <P>
|
|
<H5 align="center"><a href="http://www.linuxgazette.com/copying.html"
|
|
>Copyright ©</a> 1999, James T. Dennis
|
|
<BR>Published in <I>The Linux Gazette</I> Issue 43 July 1999</H5>
|
|
<H6 ALIGN="center">HTML transformation by
|
|
<A HREF="mailto:star@starshine.org">Heather Stern</a> of
|
|
Starshine Techinical Services,
|
|
<A HREF="http://www.starshine.org/">http://www.starshine.org/</A>
|
|
</H6>
|
|
<P> <hr> <P>
|
|
<!-- begin tagnav ::::::::::::::::::::::::::::::::::::::::::::::::::-->
|
|
<TABLE WIDTH="97%"><TR VALIGN="center" ALIGN="center">
|
|
<TD ROWSPAN="3" COLSPAN="1" WIDTH="40%"><A
|
|
HREF="../lg_answer43.html"
|
|
><IMG SRC="../../gx/dennis/answernew.gif"
|
|
ALT="[ Answer Guy Index ]"></A></td>
|
|
<TD WIDTH="19%"><A HREF="1.html">1</A></TD>
|
|
<TD WIDTH="19%"><A HREF="2.html">2</A></TD>
|
|
<TD WIDTH="19%"><A HREF="3.html">3</A></TD>
|
|
</TR><TR VALIGN="center" ALIGN="center">
|
|
<TD><A HREF="4.html">4</A></TD>
|
|
<TD><A HREF="5.html">5</A></TD>
|
|
<TD><A HREF="6.html">6</A></TD>
|
|
</TR><TR VALIGN="center" ALIGN="center">
|
|
<TD><A HREF="7.html">7</A></TD>
|
|
<TD><A HREF="8.html">8</A></TD>
|
|
<TD><A HREF="9.html">9</A></TD>
|
|
</TR></TABLE>
|
|
<!-- end tagnav ::::::::::::::::::::::::::::::::::::::::::::::::::::-->
|
|
<P> <hr> <P>
|
|
<!-- begin lgnav ::::::::::::::::::::::::::::::::::::::::::::::::::: -->
|
|
<A HREF="../index.html"
|
|
><IMG SRC="../../gx/indexnew.gif" ALT="[ Table Of Contents ]"></A>
|
|
<A HREF="/index.html"
|
|
><IMG SRC="../../gx/homenew.gif" ALT="[ Front Page ]"></A>
|
|
<A HREF="../lg_bytes43.html"
|
|
><IMG SRC="../../gx/back2.gif" ALT="[ Previous Section ]"></A>
|
|
<A HREF="../lg_tips43.html"
|
|
><IMG SRC="../../gx/fwd.gif" ALT="[ Next Section ]"></A>
|
|
<!-- end lgnav ::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
|
|
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
|
|
</BODY></HTML>
|
|
<!--endcut ========================================================= -->
|