old-www/LDP/LG/issue67/bennet.html

364 lines
22 KiB
HTML

<!--startcut ==============================================-->
<!-- *** BEGIN HTML header *** -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML><HEAD>
<title>Getting Debian: rsync Examples LG #67</title>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#0000AF"
ALINK="#FF0000">
<!-- *** END HTML header *** -->
<CENTER>
<A HREF="http://www.linuxgazette.com/">
<IMG ALT="LINUX GAZETTE" SRC="../gx/lglogo.png"
WIDTH="600" HEIGHT="124" border="0"></A>
<BR>
<!-- *** BEGIN navbar *** -->
<IMG ALT="" SRC="../gx/navbar/left.jpg" WIDTH="14" HEIGHT="45" BORDER="0" ALIGN="bottom"><A HREF="arndt.html"><IMG ALT="[ Prev ]" SRC="../gx/navbar/prev.jpg" WIDTH="16" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="index.html"><IMG ALT="[ Table of Contents ]" SRC="../gx/navbar/toc.jpg" WIDTH="220" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><A HREF="../index.html"><IMG ALT="[ Front Page ]" SRC="../gx/navbar/frontpage.jpg" WIDTH="137" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="http://www.linuxgazette.com/cgi-bin/talkback/all.py?site=LG&article=http://www.linuxgazette.com/issue67/bennet.html"><IMG ALT="[ Talkback ]" SRC="../gx/navbar/talkback.jpg" WIDTH="121" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><A HREF="../faq/index.html"><IMG ALT="[ FAQ ]" SRC="./../gx/navbar/faq.jpg"WIDTH="62" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="caruso.html"><IMG ALT="[ Next ]" SRC="../gx/navbar/next.jpg" WIDTH="15" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><IMG ALT="" SRC="../gx/navbar/right.jpg" WIDTH="15" HEIGHT="45" ALIGN="bottom">
<!-- *** END navbar *** -->
<P>
</CENTER>
<!--endcut ============================================================-->
<H4 ALIGN="center">
"Linux Gazette...<I>making Linux just a little more fun!</I>"
</H4>
<P> <HR> <P>
<!--===================================================================-->
<center>
<H1><font color="maroon">Getting Debian: rsync Examples</font></H1>
<H4>By <a href="mailto:chguy@chguy.net">Bill Bennet</a></H4>
</center>
<P> <HR> <P>
<!-- END header -->
The instructions for using your computer do not always include how to get a free operating system. <EM>[That's an understatement! -Ed.]</EM> <a href="http://www.debian.org">Debian</a> GNU/Linux is a free network operating system. You can get a Debian CD image file (".iso" or ".raw") for free from a download from the Internet. Then you can make a bootable install CD from that ".iso" file. You will need to use the ftp and rsync programs.
<P>When you post a question about how to get a Debian CD image file, you can run into a heavy work day. A heavy work day can often frustrate you because you may get the "speedy fast answer". On those "speeding fast answer days" you would be asked to <a href="#rea">read</a> the page at <a href="http://cdimage.debian.org/rsync-mirrors.html">cdimage.debian.org</a>. However, you may get stuck. Welcome to you new folks; you may get especially stuck if all of this Open Source freedom of speech and freedom of source code is new to you.
<P>Some "speedy fast answers" may ask you this question: "Do you want to know how to do this?". To the potential new Debian-folks out in cyberspace: read between the lines. You are being offered some clues. Just say yes, my friend. So, for all of you folks who want to know the straight and direct commands, this Bud's for you. This is how you get Debian with rsync:
<pre>
rsync -rv trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.iso /right/here
</pre>
<STRONG>To save trumpetti's bandwidth, use a mirror in or near your
own country instead. The list of mirrors is at
<A HREF="http://cdimage.debian.org/rsync-mirrors.html">http://cdimage.debian.org/rsync-mirrors.html</A>.
</STRONG>
<h3><font color="blue">Want to go faster?</font></h3>
<p>You can go blazing fast with costar's <a href="#co">pseudo-image kit</a> for making a Debian CD.
<ol>
<li><a href="#ge">*</a>Get a list for your CD
<li><a href="#ma">*</a>Make a note of a fast ftp server or even your local mirror of Debian
<li><a href="#gi">*</a>Give the list to the pseudo-image kit to make a pseudo-image file
<li><a href="#re">*</a>Rename the pseudo-image file to an official CD image (match your list)
<li><a href="#us">*</a>Use rsync to make it an exact official CD
<li><a href="#ch">*</a>Check your CD with md5sum
</ol>
<ol><li><a name="ge"></a>Get a list for your CD
<pre>
rsync -av trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.list /right/here
http://cdimage.debian.org/cd-images/2.2_rev3/i386/binary-i386-1_NONUS.list
</pre>
<li><a name="ma"></a>Make a note of a fast ftp server or even your local mirror of Debian.
<p>You can <a href="#part">make a mirror</a> of the Debian archive at home. When you want to access the mirror with the pseudo-image kit, you tell it the path to your mirror and also include the "debian" directory so that there is a "dists" directory waiting underneath.
<pre>
/mnt/mirror/debian
ftp://ftp.us.debian.org/debian
</pre>
<li><a name="gi"></a>Give the list to the pseudo-image kit to make a pseudo-image file
<pre>
./make-pseudo-image binary-i386-1_NONUS.list /mnt/mirror/debian
--or--
sh make-pseudo-image binary-i386-1_NONUS.list ftp://ftp.us.debian.org/debian
<a href="#j">**</a>
</pre>
<p><a name="j"></a>**There must be a dists directory directly under the mirror directory.
<li><a name="re"></a>Rename the pseudo-image file to an official CD image (match your list)
<pre>
mv pseudo-image binary-i386-1_NONUS.iso
</pre>
<li><a name="us"></a>Use rsync to make it an exact official CD
<pre>
rsync -av trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.iso /right/here
</pre>
<li><a name="ch"></a>Check your CD with md5sum
<p>There are some crippled MD5SUMS files that are next to useless. Here is a crippled one:
<p>MD5 (my-fine-new-iso-file.iso) = 9ce8f9fd8f5f1f47efe3eb77d5aae96b
<p>They have the filename and the sum but the "md5sum -c" option can not be used on these files. Here is the way it should look:
<p>9ce8f9fd8f5f1f47efe3eb77d5aae96b my-fine-new-iso-file.iso
<p>There is a space or two between the sum and the filename. Do not put three spaces or a space will be part of the filename.
<pre>
md5sum -c MD5SUMS
</pre>
</ol>
<h2><font color="blue">Why did I do all of this?</font></h2>
<p>To get it all faster. Like this afternoon.
<p>Elapsed time of a download on the release date from one of few CD iso image sites: Lots of hours and possible interrupted transmission.
<p>Elapsed time of a download on the release date from one of many Debian package mirror sites: Lots fewer hours and less possible interrupted transmissions.
<p>Your CD image will be made via ftp or even direct from your local mirror. Instead of wandering around in ftp-land, you got a list and nabbed only the files you need to make a CD. Smart.
<h2><font color="blue">Update your existing CD</font></h2>
<p>To update your CD image, rsync needs the same filename on each end.
<p>Let us say that you have made a dd of your CD.
<pre>
dd if=/dev/cdrom of=/here/is/mynewfile.iso
</pre>
<p>You can keep it around and update it with rsync.
<p>You need to know the name of the same CD file on the rsync server.
<p>Then you can rename this new iso to the right name.
<h2><font color="blue">Filenames to find</font></h2>
<p>To find the filenames:
<p>Find an <a href="http://cdimage.debian.org/rsync-mirrors.html">rsync server</a>.
<pre>
rsync -avn the.server.goes.here::
</pre>
<p>and then its modules will appear.
<h3><font color="green">example:</font></h3>
<pre>
rsync -avn ftp.fifi.org::
</pre>
<p>You will see a module named debian or debian-cd and you can infer that debian-cd probably has the iso images you want.
<p>Now get the list inside that cdimages module:
<pre>
rsync -avn the.server.goes.here::cdimages/
rsync -avn ftp.fifi.org::debian-cd/
</pre>
<p>will show you every entry in that module. A name and a directory location path for your CD image file will become apparent.
<h2><font color="blue">Encryption to get, security it is</font></h2>
<p>The image you want is the non-US version because it comes with security pgp and encryption software. You are not selling it from inside the USA to non-approved entities so you go for it. You will need to find a non-US image.
<pre>
rsync -rvn the.server.goes.here::cdimages/ | less
rsync -rvn trumpetti.atm.tut.fi::debian-cd/ | less
</pre>
<p>will get the list and it will be piped into the less pager.
<p>The pager less can go blank while you wait on a delayed loading of a huge list, and you can output to a file:
<pre>
rsync -avn the.server.goes.here::cdimages/ | less -o my-list-file
rsync -avn ftp.fifi.org::debian-cd/ &gt; my-list-file
</pre>
<h3><font color="green">example:</font></h3>
<p>The accidental big-big list:
<pre>
rsync -avn ftp.fifi.org::debian/ &gt; my-new-list-file
</pre>
<p>If you try to get a whole Debian mirror list of files it can take 7 megabytes to make your list file. Really. There is a whole world of software in Debian.
<p>The fifi site has a module called debian and if you try for a list of files from there you will get a big file indeed.
<h2><font color="blue">Rename to match the CD</font></h2>
<p>To get an rsync update, your file has to be named the same.
<pre>
mv mynewfile.iso /right/here/potato-i386-1_NONUS.iso
</pre>
<h3><font color="green">example:</font></h3>
<p>The command line length begins to wrap around on you and it can be shortened with the backslash newline character.
<pre>
rsync -rv --block-size 8192 the.server.goes.here::cdimages/2.2_rev3/i386/potato-i386-1_NONUS.iso /mnt/here/is/where/it/goes
</pre>
<p>There is a space between "iso" and "/mnt".
<pre>
rsync -rv --block-size 8192 \
the.server.goes.here::cdimages/2.2_rev3/i386/pot\
ato-i386-1_NONUS.iso /mnt/here/is/where/it/goes
</pre>
<p>You can chop it up with a backslash to compact the line.
<h2><font color="blue">Updating you are, rsync you do</font></h2>
<pre>
rsync -rv trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.iso /right/here
</pre>
<p>The above command will put the potato-i386-1_NONUS.iso in the /right/here/ directory.
<p>Only the differences between the two files will be transmitted. You save a lot of bandwidth.
<h2><font color="blue">The update for you, a checklist it is</font></h2>
<ol>
<li>Your file is named the same as the one you want.
<li>You want rsync of that remote file to synchronise with yours.
<li>rsync makes a shadow file of the same size in the rsync temporary directory. Then it moves it to your new filename.
</ol>
<p>Making a 670 MB file requires 670 MB free space!
<p>For an update that is 670 MB in addition to the CD image file already on your disk.
<pre>
rsync -av --timeout 999999 --block-size 8192 the.server.goes.here::cdimages/2.2_rev3/i386/potato-i386-1_NONUS.iso /right/here
rsync -av --timeout 999999 --block-size 8192 \
the.server.goes.here::cdimages/2.2_rev3/i386/po\
tato-i386-1_NONUS.iso /right/here
</pre>
<p>There is a space between "iso" and "/right".
<h2><font color="blue">The space has run out, a temp dir we need</font></h2>
<p>You can assign a temporary directory with -T if there is no room on the working directory partition. You need room for the huge shadow file which is the same size as that CD image file you are getting.
<pre>
type
df
to see your space
rsync -av --timeout 999999 --block-size 8192 -T /my/temp/dir \
the.server.goes.here::cdimages/2.2_rev3/i386/pota\
to-i386-1_NONUS.iso /right/here
</pre>
<h2><font color="red">Caution:</font></h2>
<p>There is a little catastrophe that you can do to yourself:
<pre>
rsync -av the.server.goes.here::cdimages
</pre>
<p>will try to dump all of the contents of cdimages onto you!
<p>put the / at the end of the line.
<p>The "-n" option is the "--dry-run" and will show you what it would have done.
("Let's not and say we did.")
<pre>
rsync -avn the.server.goes.here::cdimages/
</pre>
<p>will only show the list.
<h3><font color="green">real life example:</font></h3>
<pre>
rsync -av trumpetti.atm.tut.fi::debian-cd
</pre>
<p>will get you 16 gigabytes of CD image goodies in all architectures. Enjoy!
<pre>
rsync -av trumpetti.atm.tut.fi::debian-cd/ | less
</pre>
<p>shows a big list. Maybe use the "-n" to be safe when you are looking for files.
<h2><font color="blue">Big update right now</font></h2>
<pre>
rsync -av --timeout 999999 --block-size 8192 -T /my/temp/dir \
trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/potato-i386-1_NONUS.iso \
/mnt/here/is/where/it/goes
</pre>
<p>will dump the new CD image file into /mnt/here/is/where/it/goes.
<p>The manual does not have the examples shown here.
<h2><font color="blue">The rsync program is ready</font></h2>
<P>You can make a backup, you can use ssh, you can execute commands on the remote to make the file list, you can exclude files, you can include files, you can move the temporary directory, etc. It is a fine copy program and only the differences between files of the same name are transmitted across the bandwidth.
<P>You have got to have the same names of files locally. This rsync does a dandy job of mirroring, but if there are always a ton of new filenames then you are just doing ftp.
<P>Here is a first timer's primer:
<ol>
<li>First, get to typing. Limber up your fingers, <a href="#plu">plural</a>. Now we will start with a small exercise. On the <a href="#cs">console</a> or in an xterm please type the two letters df and then press Enter.
<li>Next, make note of how much free space it tells you about. Each block counted is 1024 bytes. 356784 output from df means three hundred and fifty six megabytes plus seven hundred and eighty-four blocks of 1024 bytes (356784 * 1024 bytes). It is 357 MB in shorthand.
<li>If you have a 670 MB CD file and the extra 670 MB of room for the working shadow file then go ahead and use rsync to update a CD file.
</ol>
<P>A lot of you are going to use rsync to grab a new CD file. Do us all a favour and keep that original CD file. You can update it to the new version by only getting the differences via rsync.
<h2><font color="blue">use rsync to update and backup your sites</font></h2>
<P>And now I am going to criticize everybody and his dog. You folks are wasting bandwidth. Are you always editing and slightly changing your web site? Then use rsync to mirror your web site. Want to add 50k of scripting everywhere on each of your pages? Then keep the same filenames and mirror it with rsync. Did you stay on the cutting edge and download that new iso file and make a brand new CD? Then rename it and update it with rsync for the newest version. What am I getting at?
<h1><font color="blue">rsync saves bandwidth</font></h1>
<P>When you try to get a Debian CD the huge list of files will be loaded into memory and that is what is taking so long. It is not stalled. Wait. Have patience. I quote Mr. Miyagi in the Karate Kid: "Wax on. Wax off." Each of those many files is being opened for a checksum to see if it needs to be transmitted. Wait.
<P>Get the md5sum.txt file. When it is in your directory after the rsync, type
<pre>
md5sum -c md5sum.txt
</pre>
<p>and the file will be checked for accuracy.
<h2><font color="blue"><a name="co"></a>costar's pseudo-image kit</font></h2>
<P>Get the pseudo-image kit <a href="http://cdimage.debian.org/~costar/pseudo-image-kit/">here</a>.
<P>Get the list for the pseudo-image kit.
<pre>
rsync -rv --timeout 999999 --block-size 8192 -T /my/temp/dir \
trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.list \
/mnt/here/is/where/it/goes
</pre>
<p>will put the list file in /mnt/here/is/where/it/goes. It is called binary-i386-1_NONUS.list.
<P>That binary-i386-1_NONUS.list will be used to build your CD image. If you have a partial mirror of Debian (oh sure, 14 gigabytes of stuff) then the pseudo-image kit will make a CD for you from your local mirror. It is very blazing fast for all you speed freaks. I must admit with disk drive prices being reasonable it is time for all of us to buy up the remaining stock of usable drives. You may not know about the <a href="#blarg">effort</a> to lock you down, but the age of freedom of fair use is ending.
<h2><font color="blue">Why not use ftp?</font></h2>
<p>Why not slam your ftp server with 670 more megabytes and several hours of work?
<p>To quote the clear coder J.A. Bezemer from a thread on debian-cd:
<pre>
"
&gt; Most people get bits from the local hard disk a lot faster than
&gt; they get bits over the network. The pseudo-image kit finishes in
&gt; minutes, instead of hours, on most "fast" connections.
That's one part of the story; the other is that we have only a few CD image
mirrors and about 250 packages mirrors, and the Kit does a great job to
distribute the used bandwidth more evenly between all mirrors. Remember that
every single bit flowing from any of the Debian mirrors is sponsored by
someone, and this way the people who invested in big disks for the CD images
don't have to pay that much for their net connection.
"
</pre>
'Nuff said.
<hr NOSHADE>
<a name="plu"></a>Using one finger over and over on the clicker button will give you a pain.
<hr NOSHADE>
<a name="rea"></a>Read means look at every word and try to make sense of it. It does not mean look for an easy link where someone does it for you and then give up. Capital Letters count. Punctuation counts. Poor writing can confuse, so have fun by pointing out the error to someone and then we can all fall all over ourselves to get the answer posted. It is a weird world. Use it to your advantage.
<hr NOSHADE>
<a name="cs"></a>You get to the virtual console from X windows by pressing Control + Alt + F2. It takes two hands and you may be surprised to learn that you can use Linux without X. You can even install it without a graphical user interface. You can even use Linux with 16 megabytes of RAM on these modern distributions. But not Mandrake; it will complain of low resource unless you have 64 MB RAM and a Pentium.
<hr NOSHADE>
<a name="part"></a>Make a partial mirror of Debian by using one of the contributed <a href="http://www.chguy.net/news/msg01827.html">scripts</a> from the Debian-folk. They make use of rsync and ftp for cutting down the size of the archive to suit your particular interest. The idea is to match the archive to the architecture of your machine.
<hr NOSHADE>
<a name="blarg"></a>Your next "advance" in hard disk storage is the copy-protected drive. It means you will not be able to use everything in the world any more. You lose each freedom by degrees. At some point you must revolt or become revolting.
<p>The copyrighted material on a music CD is definitely covered with legal rights and permissions. With the genetically engineered mutation of Napster, we can clearly see the lock-down <a href="http://www.chguy.net/news/feb01/gilmore-copy.html">coming</a>. New hard disk drives and other storage mediums are in danger of being copy-protected. You can use software, but it has to be approved software. Sounds all warm and fuzzy, eh? Put it this way: Thou shalt not reproduce.
<hr NOSHADE>
<a name="ab"></a>To quote the clear coder J. A. Bezemer from a thread on debian-cd:
<pre>
"Using --partial will kill your valuable pseudo-image if rsync fails after 1
byte has been transferred. So either back up your pseudo-image or do NOT use
--partial."
</pre>
So do not use partial, eh?
<hr NOSHADE>
<h2><a name="xfree"><font color="blue">Reference reading:</font></a></h2>
<P><a href="http://rsync.samba.org/">rsync.samba.org</a> -- the home of rsync and its Faq-O-Matic
<P><a href="http://www1.bell-labs.com/project/nsbd/">NSBD</a> -- Not-So-Bad-Distribution at Bell Labs uses rsync to securely update you
<P><a href="http://systemimager.org/">rsync resources</a> -- Multiple system install and updates with rsync
<P><a href="http://sunsite.dk/SunSITE/guides/rsync/rsync-mirroring.html">rsync resources</a> -- Setup anonymous rsync servers
<P><a href= "http://optics.ph.unimelb.edu.au/help/rsync/rsync_user.html">Help File for Captives</a> -- if you are seeking freedom
<hr><br><P>
<center><font color="blue"><strong>made with mcedit on an i486 with GNU/Linux 2.0.35</strong></font><P><font color="blue"><strong>There were no freeze-ups or crashes during testing of these procedures. We suspect GNU/Linux software is to blame.</strong></font></center>
<!-- *** BEGIN bio *** -->
<SPACER TYPE="vertical" SIZE="30">
<P>
<H4><IMG ALIGN=BOTTOM ALT="" SRC="../gx/note.gif">Bill Bennet</H4>
<P> <CITE>Bill, the ComputerHelperGuy, lives in Selkirk, Manitoba, Canada; the
"Catfish Capitol of North America" if not the world. He is on the Internet at
<A HREF="http://www.chguy.net">www.chguy.net</A>. He tells us "I have been a PC
user since 1983 when I got my start as a Radio Shack manager. After five years
in the trenches, I went into business for myself. Now happily divorced from
reality, I live next to my Linux box and sell and support GPL distributions of
all major Linux flavours. I was a beta tester for the PC version of Playmaker
Football and I play `pentium-required' games on the i486. I want to help Linux
become a great success in the gaming world, since that will be how Linux will
take over the desktop from DOS." It's hard to believe that his five years of
university was only good for fostering creative writing skills.</CITE>
<!-- *** END bio *** -->
<!-- *** BEGIN copyright *** -->
<P> <hr> <!-- P -->
<H5 ALIGN=center>
Copyright &copy; 2001, Bill Bennet.<BR>
Copying license <A HREF="../copying.html">http://www.linuxgazette.com/copying.html</A><BR>
Published in Issue 67 of <i>Linux Gazette</i>, June 2001</H5>
<!-- *** END copyright *** -->
<!--startcut ==========================================================-->
<HR><P>
<CENTER>
<!-- *** BEGIN navbar *** -->
<IMG ALT="" SRC="../gx/navbar/left.jpg" WIDTH="14" HEIGHT="45" BORDER="0" ALIGN="bottom"><A HREF="arndt.html"><IMG ALT="[ Prev ]" SRC="../gx/navbar/prev.jpg" WIDTH="16" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="index.html"><IMG ALT="[ Table of Contents ]" SRC="../gx/navbar/toc.jpg" WIDTH="220" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><A HREF="../index.html"><IMG ALT="[ Front Page ]" SRC="../gx/navbar/frontpage.jpg" WIDTH="137" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="http://www.linuxgazette.com/cgi-bin/talkback/all.py?site=LG&article=http://www.linuxgazette.com/issue67/bennet.html"><IMG ALT="[ Talkback ]" SRC="../gx/navbar/talkback.jpg" WIDTH="121" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><A HREF="../faq/index.html"><IMG ALT="[ FAQ ]" SRC="./../gx/navbar/faq.jpg"WIDTH="62" HEIGHT="45" BORDER="0" ALIGN="bottom"></A><A HREF="caruso.html"><IMG ALT="[ Next ]" SRC="../gx/navbar/next.jpg" WIDTH="15" HEIGHT="45" BORDER="0" ALIGN="bottom" ></A><IMG ALT="" SRC="../gx/navbar/right.jpg" WIDTH="15" HEIGHT="45" ALIGN="bottom">
<!-- *** END navbar *** -->
</CENTER>
</BODY></HTML>
<!--endcut ============================================================-->