From 546ae4e89e056911a5b4f0efab8d5aad729fa26a Mon Sep 17 00:00:00 2001 From: "Jason S. Evans" Date: Wed, 24 Feb 2016 09:53:06 +0000 Subject: [PATCH] add Linux-Complete-Backup-and-Recovery-HOWTO.adoc --- ...ux-Complete-Backup-and-Recovery-HOWTO.adoc | 2107 +++++++++++++++++ 1 file changed, 2107 insertions(+) create mode 100644 LDP/users/Jason-Evans/Linux-Complete-Backup-and-Recovery-HOWTO.adoc diff --git a/LDP/users/Jason-Evans/Linux-Complete-Backup-and-Recovery-HOWTO.adoc b/LDP/users/Jason-Evans/Linux-Complete-Backup-and-Recovery-HOWTO.adoc new file mode 100644 index 00000000..65cb4226 --- /dev/null +++ b/LDP/users/Jason-Evans/Linux-Complete-Backup-and-Recovery-HOWTO.adoc @@ -0,0 +1,2107 @@ +[[intro]] +Introduction +------------ + +The normal bare metal restoration process is: install the operating +system from the product disks. Install the backup software, so you can +restore your data. Restore your data. Then you get to restore +functionality by verifying your configuration files, permissions, etc. + +The process and scripts explained in this HOWTO will save re-installing +the operating system. The process explained here will restore only files +that were backed up from the production computer. Your configuration +will be intact when you restore the system, which should save you hours +of verifying configurations and data. + +[[copyright]] +Copyright Information +~~~~~~~~~~~~~~~~~~~~~ + +Copyright © 2001 through last date of modification Charles Curley and +distributed under the terms of the GNU Free Documentation License (GFDL) +license, stated below. Permission is granted to copy, distribute and/or +modify this document under the terms of the GNU Free Documentation +License, Version 1.1 or any later version published by the Free Software +Foundation; with no Invariant Sections, with no Front-Cover Texts, and +with no Back-Cover Texts. A copy of the license is included in the +section entitled link:#appendix1gfdl[``GNU Free Documentation +License'']. + +If you have any questions, please contact +mailto:linux-howto at metalab.unc.edu[linux-howto at metalab.unc.edu]. + +[[disclaimers]] +Disclaimers +~~~~~~~~~~~ + +No liability for the contents of this documents can be accepted by the +author, the http://www.tldp.org/[Linux Documentation Project] or anyone +else. Use the concepts, examples and other content at your own risk. +There may be errors and inaccuracies that may damage your system. +Proceed with caution, and, although errors are unlikely, the author take +no responsibility for them. + +All copyrights are held by their by their respective owners, unless +specifically noted otherwise. Use of a term in this document should not +be regarded as affecting the validity of any trademark or service mark. + +Naming of particular products or brands should not be seen as +endorsements. + +You are strongly recommended to take a backup of your system before +major installation and backups at regular intervals. In addition, you +are strongly recommended to use a sacrificial experimental computer when +mucking with the material, especially the scripts, in this HOWTO. + +[[newversions]] +New Versions +~~~~~~~~~~~~ + +You can find this document at its +link:&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO.html[home page] or +at the http://www.tldp.org/[Linux Documentation Project] web site in +many formats. Please comment to + +Depending on your browser, you may have to hold down the shift button +while you click on these in order to get them to download. + +* link:&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO/Linux-Complete-Backup-and-Recovery-HOWTO.chunky.html.tar.bz2[bzip2 +compressed chunky (lots of small pages. Faster reading.) HTML]. +* link:&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO/Linux-Complete-Backup-and-Recovery-HOWTO.smooth.html.tar.bz2[bzip2 +compressed smooth (one monster page -- no chunks. Easier to search.) +HTML]. +* link:&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO/Linux-Complete-Backup-and-Recovery-HOWTO.ps.bz2[bzip2 +compressed postscript (US letter format)]. +* link:&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO/Linux-Complete-Backup-and-Recovery-HOWTO.pdf.bz2[bzip2 +compressed PDF (US letter format)]. +* link:&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO/Linux-Complete-Backup-and-Recovery-HOWTO.txt.bz2[bzip2 +compressed raw ASCII text]. +* Use the +link:&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO/Linux-Complete-Backup-and-Recovery-HOWTO.tar.bz2[source], +Luke. +* link:&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO/md5sums[MD5] and +link:&myurl;/Linux-Complete-Backup-and-Recovery-HOWTO/sha1sums[SHA1] +sums. + +To ensure that you got a good download, validate the files against the +checksums above. The easiest way to do this is to pull in the sha1sum or +md5sum files (or both), and run the appropriate program against it: + +.... +$ sha1sum -c sha1sums +.... + +or/and + +.... +$ md5sum -c md5sums +.... + +[[credits]] +Credits +~~~~~~~ + +This document is derived from two articles originally published in +http://www.linuxjournal.com/[Linux Journal]. My thanks to Linux Journal +for reverting the rights to those articles, thereby helping make this +HOWTO possible. + +Thanks to Joy Y. Goodreau for excellent HOWTO editing, and to David +Palomares for correcting the spelling of Salvador DalITILDE's name. + +Also, thanks to mailto:pon at iki dot fi[Pasi Oja-Nisula] for a bug fix +and information on http://www.knoppix.org/[Knoppix]. + +[[feedback]] +Feedback +~~~~~~~~ + +Feedback is most certainly welcome for this document. Without your +corrections, suggestions and other input, this document wouldn't exist. +Please send your additions, comments and criticisms to me at: . + +[[translations]] +Translations +~~~~~~~~~~~~ + +Not everyone speaks English. Volunteers are welcome. + +[[Overview]] +Overview +-------- + +The process shown below is not easy, and can be hazardous to your data. +Practice it before you need it! Do as I did, and __practice on a +sacrificial computer__! + +The original target computer for this HOWTO was a Pentium computer. +Originally, it had a http://www.redhat.com[Red Hat] 7.1 Linux server or +workstation installation on one IDE hard drive. Since then, I have used +a number of computers, and they have been ugraded to Red Hat 8.0 and +http://fedora.redhat.com/[Fedora Cores 1, 3 and 4.]. The target computer +does not have vast amounts of data because the computer was set up as a +``sacrificial'' test bed. That is, I did not want to test this process +with a production computer and production data. Also, I did a fresh +installation before I started the testing so that I could always +re-install if I needed to revert to a known configuration. + +___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +*Note* + +The sample commands will show, in most cases, what I had to type to +recover the target system. You may have to use similar commands, but +with different parameters. It is up to you to be sure you duplicate your +setup, and not the test computer's setup. +___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +The basic procedure is set out in W. Curtis Preston, +http://www.oreilly.com/catalog/unixbr/[Unix Backup & Recovery], O'Reilly +& Associates, 1999, which I have favorably reviewed in +http://www2.linuxjournal.com/lj-issues/issue78/3839.html[Linux Journal]. +However, the book is a bit thin on specific, real-time questions. For +example, exactly which files do you back up? What metadata should you +preserve, and how? This document explores those questions. + +Before beginning the process set forth in this HOWTO you will need to +back up your system with a typical backup tool such as Amanda, BRU, tar, +Arkeia or cpio. The question, then, is how to get from toasted hardware +to the point where you can run the restoration tool that will restore +your data. + +Users of Red Hat Package Manager (RPM) based Linux distributions should +also save RPM metadata as part of their normal backups. The following is +in one of the scripts in this HOWTO: + +.... +bash# rpm -Va | sort +2 -t ' ' | uniq > /etc/rpmVa.txt +.... + +It provides a basis for comparison after a bare metal restoration. + +To get to this point, you must have: + +* Your hardware up and running again, with replacement components as +needed. The BIOS should be correctly configured, including time and +date, and hard drive parameters. At the moment, there is no provision +for using a different hard drive. +* When I started this project, I used a +http://www.iomega.com/zip/products/par100_250.html[ZIP drive]. Now, they +are rather cramped for space and can be inconvenient. You can substitute +a USB flash disk, NFS mount, CD-RW or other medium. Just be sure that +the Linux distribution you use for first stage restore supports your +medium. For historical reasons, this document will refer to the +http://www.iomega.com/zip/products/par100_250.html[ZIP drive]; please +substitute the medium of your choice. There is more discussion of +alternatives below in the section on link:#themeandvariations[Theme And +Variations]. +* Your normal backup media: tape hard drive, etc. +* A minimal Linux system that will allow you to run the restoration +software, which we will call the restoration Linux. + +To get there, you need at least two stages of backup, and possibly +three. Exactly what you back up and in which stage you back it up is +determined by your restoration process. For example, if you are +restoring a tape server, you may not need networking during the +restoration process. So only back up networking in your regular backups. + +You will restore in stages as well. In stage one, we build partitions, +file systems, etc. and restore a minimum of files from the ZIP disk. The +goal of stage one is to be able to boot to a running computer with a +network connection, tape drives, restoration software, or whatever we +need for stage two. + +The second stage, if it is necessary, consists of restoring backup +software and any relevant databases. For example, suppose you use Arkeia +and you are building a bare metal recovery ZIP disk for your backup +server. Arkeia keeps a huge database on the server's hard drives. You +can recover the database from the tapes, if you want. Instead, why not +tar and gzip the whole arkeia directory (at /usr/knox), and save that to +another computer over NFS or SSH? Stage one, as we have defined it +below, does not include X, so you will have some experimenting to do if +you wish to back up X as well as your backup program. Some restore +programs require X. + +Of course, if you are using some other backup program, you may have some +detective work to do to. You will have to find out the directories and +files it needs to run. If you use tar, gzip, cpio, mt or dd for your +backup and recovery tools, they will be saved to and restored from our +ZIP disk as part of the stage one process describe below. + +The last stage is a total restoration from tape or other media. After +you have done that last stage, you should be able to boot to a fully +restored and operational system. + +[[limitations]] +Limitations +~~~~~~~~~~~ + +This HOWTO is restricted to making a minimal backup such that, having +then restored that backup to new hardware (``bare metal''), you can then +use your regular backups to restore a completely working system. This +HOWTO does not deal with your regular backups at all. + +Even within that narrow brief, this HOWTO is not exhaustive. You still +have some research, script editing, and testing to do. + +The scripts here restore the partition data exactly as found on the +source hard drive. This is nice if you are restoring on an identical +computer or at least an identical hard drive, but that is often not the +case. For now, there are two remedies (which will make more sense after +you've read the rest of the HOWTO): + +* Edit the partition table input file. I've done that a few times. You +can also do this to add new partitions or delete existing ones (but edit +the scripts that use the partition table input file as well). +* Hand build a new partition table and go from there. That is one reason +why link:#restore.metadata[`restore.metadata`] does not call the hard +drive rebuilding script. Use the link:#make.dev.hda[rebuilding script]. + +The scripts shown here only handle ext2fs, FAT12, FAT16 and FAT32. Until +some eager volunteer supplies code for doing so in these scripts, you +will need other tools for backing up and restoring file systems we +haven't covered. However, see the note below on link:#ntfs[NTFS]. +http://www.partimage.org/[Partition Image] looks like a useful candidate +here. + +[[Preparation]] +Preparation +----------- + +_____________________________________________________________________________________________ +*Note* + +Do your normal backups on their regular schedule. This HOWTO is useless +if you don't do that. +_____________________________________________________________________________________________ + +Build yourself a restoration Linux disk. I have used +http://www.knoppix.org/[Knoppix]. See the notes on +link:#knoppix[Knoppix] below. However, everything here is command line. +We don't need a GUI. A GUI-less distribution will boot faster and can +load itself into memory (so you can use the CD drive) even on a minimal +machine. For this I now use http://www.finnix.org/[Finnix]. + +In the past, I have used http://www.toms.net/rb[tomsrtbt]. It is well +documented and packs a lot of useful tools onto one floppy diskette. +Unfortunately, the changes I've had to make in the scripts to handle +more recent Linux systems cause problems for +http://www.toms.net/rb[tomsrtbt]. The http://www.toms.net/rb[tomsrtbt] +2.0.103 tar is based on http://www.busybox.net/[busybox], so remarks +about it may apply to other Linux disties which use busybox. + +Next, figure out how to do the operating system backup you will need so +that you can restore your normal backup. I used to follow Preston's +advice and use an Iomega parallel port ZIP drive. The drives get +approximately 90 MB of useful storage to a disk. I need about 85 MB to +back up my desktop, so a 100MB ZIP drive may be pushing your luck. These +days I use CD-RWs or NFS. For more on those, see the sections on using +link:#cd-rom[CD-ROM]s and link:#nfs[NFS]. + +[[installingzipdrive]] +Installing the ZIP Drive +~~~~~~~~~~~~~~~~~~~~~~~~ + +Installing the ZIP drive is covered in the +http://www.tldp.org/HOWTO/mini/ZIP-Drive.html[ZIP Drive HOWTO], +available at http://www.tldp.org/[the Linux Documentation Project] and +at its home page, http://www.njtcom.com/dansie/zip-drive.html. + +[[backup.server]] +Backup Server +~~~~~~~~~~~~~ + +You can set up a backup server for this process. Scripts on the backup +server interact with the target machines (including itself) via SSH. +They assume that your backup server user (root here, for simplicity) can +log in with no password to the targets. This is necessary for unattended +backups. + +First, create a suitable directory to keep all the backups in. We'll +call it `backs`. In backs, create a directory for each target computer. +The first field in the directory should be the host name. Subsequent +fields can be other useful information. For example, to preserve the +last backup of a target before an installation of a new version of the +distribution, I use an abbreviation for the distribution, e.g. ``fc5''. +Fields are separated with periods (``.''). So, for example, `tester.f7`. +The host name is required because the scripts use that to determine +which host to back up. + +Copy the scripts `get` and `restore` into each target's directory. Then +customize them for each host as needed. + +Also create in `backs` a directory called `scripts` and put in it the +script `get.target`. This is a library for the backup and restore +scripts. It performs actions common to all backups and restorations. + +[[CreatingtheStage1BackUp]] +Creating the Stage 1 Back Up +---------------------------- + +Having made your production backups, you need to preserve your partition +information so that you can rebuild your partitions. + +The script link:#make.fdisk[`make.fdisk`] scans a hard drive for +partition information, and saves it in four files. The first is an +executable script, called link:#make.dev.hda[`make.dev.x`] (where ``x'' +is the name of the device file, e.g. hda). Second is +link:#mount.dev.hda[`mount.dev.x`], which creates mount points and +mounts the newly created partitions on them. The next, +link:#dev.hda[`dev.x`], is the commands necessary for `fdisk` to build +the partitions. Last is an input file for `sfdisk` to create partions. +(`sfdisk` is preferable and used if found.) You specify which hard drive +you want to build scripts for (and thus the file names) by naming the +associated device file as the argument to +link:#make.fdisk[`make.fdisk`]. For example, on a typical IDE system, + +.... +bash# make.fdisk /dev/hda +.... + +spits out the scripts link:#make.dev.hda[`make.dev.hda`], +link:#mount.dev.hda[`mount.dev.hda`] and the input files for `fdisk` and +`sfdisk`, link:#dev.hda[`dev.hda`] and link:#dev.hda.sfd[`dev.hda.sfd`], +respectively. + +In addition, if link:#make.fdisk[`make.fdisk`] encounters a FAT +partition, it preserves the partition's boot sector in a file named +`dev.xy`, where x is the drive's device name (e.g. sdc, hda) and y is +the partition number. The boot sector is the first sector, 512 bytes, of +the partition. This sector is restored at the same time the partitions +are rebuilt, in the script link:#make.dev.hda[`make.dev.hda`]. + +Fortunately, the price of hard drives is plummeting almost as fast as +the public's trust in politicians after an election. So it is good that +the output files are text, and allow hand editing. That's the most +difficult but most flexible way to rebuild on a larger replacement +drive. (See the link:#todo[To Do list].) + +Other metadata are preserved by the script +link:#save.metadata[`save.metadata`]. The script saves the partition +information in the file `fdisk.hda` in the root of the ZIP disk. It is a +good idea to print this file and your `/etc/fstab` so that you have hard +copy should you ever have to restore the partition data manually. You +can save a tree by toggling between two virtual consoles, running +`fdisk` in one and catting `/etc/fstab` or `/fdisk.hda` as needed. +However, doing so is error prone. + +You will also want to preserve files relevant to your restoration +method. For example, if you use NFS to save your data, you will need to +preserve hosts.allow, hosts.deny, exports, etc. Also, if you are using +any network-backed restoration process, such as Amanda or Quick Restore, +you will need to preserve networking files like HOSTNAME, hosts, etc. +and the relevant software tree. + +The simplest way to handle these and similar questions is to preserve +the entire `/etc` directory. + +There is no way a 100 MB ZIP drive is going to hold a server +installation of a modern distribution of Linux. We have to be much more +selective than simply preserving the whole kazoo. What files do we need? + +* The boot directory. +* The /etc directory and subdirectories. +* Directories needed at boot time. +* Device files in /dev. + +To determine the directories needed at boot, we look at the boot +initialization file `/etc/rc.sysinit`. It sets its own path like so: + +.... +.... + +Trial and error indicated that we needed some other directories as well, +such as `/dev`. In Linux, you can't do much without device files. + +In reading the script link:#save.metadata[`save.metadata`], note that we +aren't necessarily saving files that are called with absolute paths. + +We may require several iterations of back up, test the bare metal +restore, re-install from CD and try again, before we have a working +backup script. While I worked on this HOWTO, I made five such iterations +before I had a successful restoration. That is one reason why it is +essential to use scripts whenever possible. Test thoroughly! + +One thing you can do on an RPM based system is use the `rpm` program to +determine which files are where. For example, to get a complete list of +the files used by the openssh package, run: + +.... +bash# rpm -ql openssh +.... + +There are some things you don't need, like the man pages. You can +inspect each one and decide whether to back it up or not. + +____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +*Note* + +The second stage of restoration is run without overwriting previously +restored files. This means that the files restored in the first stage +are the ones that will be used after full restoration. So update your +bare metal backups whenever you update files in these directories! +____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +*Note* + +Recent kernels have incorporated a new ATA (IDE) hard drive driver, +libata. Because of this, parallel ATA drives (PATA) now show up as SCSI +drives, as serial ATA (SATA) have always done. However, not all rescue +distributions (e.g. finix) use this new driver. There is a line toward +the bottom of link:#save.metadata[`save.metadata`] wich very carefully +replaces "/dev/sda" with "/dev/hda". Use this as a template if you have +multiple IDE hard drives. Comment it out or delete it if this is not an +issue for you. + +Note that there is no guaranteed mapping! Systems with multiple hard +drives may have confusing mappings. Be sure to edit this line carefully. +Check it if you add or remove a hard drive of any interface type to or +from your system! + +N.B: if you have libata IDE drive issues, the grub-install line at the +end of link:#restore.metadata[`restore.metadata`] won't work. If it +doesn't, use your rescue disk to do the same. Or burn and boot to the +boot image that is made as part of the first stage backup. Boot to it +and do the second state restore as usual. The second state restore +should re-run `grub-install` or you can run it manually. +_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +*Note* + +The version of `tar` included in http://www.toms.net/rb[tomsrtbt] does +not preserve ownership when it restores. This may cause problems for +applications like link:#amanda[Amanda]. A backup and restoration tool, +Amanda has several directories owned by its own eponymous user. The +solution is: + +* Note which directories and files are not owned by root. +* Note their owners. +* Arrange to set the ownership correctly as part of the restoration +process. E.g: ++ +.... +bash# chown -R amanda:disk /var/lib/amanda +.... ++ +You can also add that line to your scripts for second state restoration, +such as link:#restore[`restore`]. +___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +*Note* + +http://www.toms.net/rb[tomsrtbt] does not support restoring owners by +UID/GID. To make backups suitable for restoring with +http://www.toms.net/rb[tomsrtbt], remove the tar command line option +``--numeric-owner'' from the command line options for tar in the +function crunch in the script link:#save.metadata[`save.metadata`]. +____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +The Archive +~~~~~~~~~~~ + +All of this gets stored into an archive under +`/var/lib/bare.metal.recovery`. Each day a first stage backup is made a +new directory is prepared, with the date encoded as YYYYMMDD, and the +day's archive deposited therein. It is up to you to prune obsolete +archives. It is a good idea to keep at least one old archive around in +case the computer crashes while you are making an archive. If a second +archive is made in a day, the earlier one for that day is replaced. + +The files in the archive directory include a `README.txt`, which has +information about the backup and the computer the backup was made on. +Other files are there in case hand intervention is required. + +Below the daily archive directory are several text files and three +directories. The scripts reside in `bin`, the tarballs in `data`, and +information about the system such as partitions and LVM volume backups +are in `metadata`. + +To create a CD, simply use a script or graphical tool to create a CD +starting at the daily archive directory. It is up to you to be sure your +archive will fit onto your medium, or to make other arrangements. + +[[ThemeAndVariations]] +Theme And Variations +~~~~~~~~~~~~~~~~~~~~ + +No ZIP drive +^^^^^^^^^^^^ + +This backup process used to require you to have the ZIP disk drive +present at each backup. It now creates the archive in a directory, which +you can back up over the net. Then you only need to build a ZIP disk +(with `cp -rp`) on the backup server when you need to restore. + +The backup process will be faster than directly writing to the ZIP +drive, but you should check that the resulting directory will fit on +your ZIP disk (with the output of `du -hs $target.zip` in the script +link:#save.metadata[`save.metadata`])! See the definition of the +variable `zip` in that script. + +One of my laptops has problems running both a network card and a ZIP +drive, so this is the process I use to back it up. I keep a backup image +as well as the current one, so that I have a fallback in case the +computer crashes during a backup. + +CD-ROM +^^^^^^ + +This is similar to the no ZIP drive option above. Save your backups to a +directory on your hard drive, as noted. Then use `mkisofs` to create an +ISO 9660 image from that directory, and burn it. This does not work with +some CD-ROM based restoration Linuxes, like +http://www.knoppix.org/[Knoppix], because the Linux has to have the +CD-ROM drive. Unless you have two CD-ROM drives, say one in a USB +clamshell. I have a DVD burner set up this way with exactly this in +mind. Or have http://www.finnix.org/[Finnix] load itself into memory on +boot and then use the CD-ROM drive from which you booted. + +These remarks should also apply to DVDs. + +Also, look at +http://www.knoppix.net/wiki/Knoppix_Remastering_Howto[remastering] +Knoppix with your first and second stage backups on the CD-ROM. You +should also be able to http://www.finnix.org/Remastering_Finnix[remaster +Finnix]. + +These days many computers come with a CD-ROM drive but no floppy +diskette. And floppy drives do fail. So it's a good idea to burn your +CD-ROM with a bootable image on it. The bad news is that the ``El +Torito'' format supports 1.2 MB, 1.44 MB and 2.88 MB floppy images, and +http://www.toms.net/rb[tomsrtbt] uses a 1.7 MB floppy. The good news is +that you can get a 2.88 MB version, `tomsrtbt-2.0.103.ElTorito.288.img`, +from the same mirrors where you get the floppy image. Place a _copy_ +footnote:[I emphasize copy because `mkisofs` will mung the file in the +directory from which it makes the ISO image.] in the root directory of +the backup files. Then use the `mkisofs` command line option -b to +specify `tomsrtbt-2.0.103.ElTorito.288.img` as the boot image file. + +The only down side of this process is that many older BIOSes do not +support 2.88 MB floppy images on CD-ROMs. Most of those will boot to a +http://www.toms.net/rb[tomsrtbt] floppy. + +An alternative is to use http://syslinux.zytor.com/[Syslinux]. It is not +dependent on a floppy diskette image, and you can build your own CD with +a number of tools, such as http://www.toms.net/rb[tomsrtbt], on it. + +You may have to adjust the BIOS options to allow the computer to boot to +CD-ROM drive. If you can't do that, either because the BIOS won't +support booting to CD-ROM, or because you can't get into the BIOS, see +http://btmgr.webframe.org/[Smart Boot Manager (SBM)] as described in the +link:#resources[Resources]. + +One variant is to dispense with the tarballs in the first stage, and +create a tarball of the entire system. When you build your restoration +CD, put the monster tarball in the data directory of the CD. The scripts +will pick that up and restore for you, combining the first and second +stages. This eliminates a lot of the cruft related to permissions and +ownership in link:#restore.metadata[`restore.metadata`] and +link:#save.metadata[`save.metadata`] + +_Test_ your CDs on the drive you will use at restoration time. If you +find you need to hack the scripts, you can copy them to `/tmp`, usually +a RAM drive, and edit them there. The scripts will run there. As a RAM +disk is volatile, be sure to save your changes before you reboot! + +NFS +^^^ + +If you back up across your network to a backup server, you will have all +the files on it you need. Set up the directory where you keep all your +backups as an NFS export. + +Then, on http://www.finnix.org/[Finnix], do the following (tab +completion is very nice here): + +.... +# mkdir /mnt/nfs +# /etc/init.d/portmap start +# mount server:/path/of/exportedfs /mnt/nfs +# cd /mnt/nfs/.../bin +.... + +Now restore as usual. + +There are several advantages to NFS for this job: You don't have to +worry about space on a CD-ROM or +http://www.iomega.com/zip/products/par100_250.html[ZIP drive]. You can +edit scripts on the server and they are preserved when you reboot the +target. + +Multiple ZIP disks +^^^^^^^^^^^^^^^^^^ + +By splitting up the two first stage scripts, +link:#restore.metadata[`restore.metadata`] and +link:#save.metadata[`save.metadata`], you could spread the first stage +metadata across multiple ZIP disks. + +Excluding From First Stage Saving +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There are time when you need to squeeze a few megabytes from the first +stage data, especially when you are pushing the limit of your ZIP disk. +The function `crunch` in the script link:#save.metadata[`save.metadata`] +takes multiple parameters to feed to `tar`. It can also take the +`--exclude` parameter. So, for example, you can exclude the `samba` and +`X11` directories under `/etc` like so: + +.... +.... + +Why those two? Because they're hard drive space hogs and we don't need +them when booting after the first stage. + +If you keep multiple kernels around, you can eliminate the modules for +all of the kernels you won't boot to. Check your `lilo.conf` or +`/boot/grub/menu.lst` to see which kernel you will use, and then check +`/lib/modules` for module directories you can exclude. + +How to find more good candidates for exclusion? List the target +directories with `ls -alSr` for individual files, and `du | sort -n` for +directories. + +Another (probably neater) way to exclude directories is to put a +complete list of directories into a file, then refer to it via the tar +option `--exclude-from=FILENAME`. + +[[initrd]] +Initrd +^^^^^^ + +If your system uses an initial RAM disk, or initrd, to boot, make sure +that link:#restore.metadata[`restore.metadata`] creates the directory +`/initrd`. The easiest way to do this is to ensure that it is included +in the list of directories used in the directory creating loop toward +the end. + +Your system will probably use an initrd if it boots from a SCSI drive or +has root on an ext3fs partition. Check `/etc/lilo.conf` or +`/boot/grub/menu.lst` to see if it calls for one. + +[[firststagerestore]] +First Stage Restore +------------------- + +[[Booting]] +Booting +~~~~~~~ + +The first thing to do is to verify that the hardware time is set +correctly. Use the BIOS setup for this. How close to exact you have to +set the time depends on your applications. For restoration, within a few +minutes of exact time should be accurate enough. This will allow +time-critical events to pick up where they left off when you finally +launch the restored system. + +[[Bootingtomsrtbt]] +tomsrtbt +^^^^^^^^ + +Before booting http://www.toms.net/rb[tomsrtbt], make sure your ZIP +drive is installed on a parallel port, either `/dev/lp0` or `/dev/lp1`. +The start-up software will load the parallel port ZIP drive driver for +you. + +The next step is to set the video mode. I usually like to see as much on +the screen as I can. When the option to select a video mode comes, I use +mode 6, 80 columns by 60 lines. Your hardware may or may not be able to +handle high resolutions like that, so experiment with it. + +[[Bootingknoppix]] +Knoppix +^^^^^^^ + +These instructions will probably work with other CD-ROM or USB pen +Linuxes, but you may have to vary them to suit. + +Before booting http://www.knoppix.org/[Knoppix], make sure your ZIP +drive (or substitute) is installed on a parallel port, either `/dev/lp0` +or `/dev/lp1`. Knoppix does not load the parallel port ZIP drive driver +for you. Instead, use the command `modprobe ppa` (as root) to install +it. + +Boot http://www.knoppix.org/[Knoppix] as usual. I find it faster and +more useful to boot to a console. At the boot menu, use the command +``knoppix 2''. Then become the root user, with `su -`. For the password, +just hit return. + +[[bootingfinnix]] +Finnix +^^^^^^ + +One option for booting http://www.finnix.org/[Finnix] is the "toram" +option, which lets you move the whole kazoo into RAM. That in turn +should let you load another CD, with your first stage data, into the CD +drive. + +[[restoration]] +Restoration +~~~~~~~~~~~ + +These instructions assume you are running +http://www.toms.net/rb[tomsrtbt]. If you are using a different Linux for +your restore system, you may have to adjust these instructions a bit. +For example, you should always run these scripts as root even if some +other user gives you the requisite privileges. + +Once the restoration Linux has booted and you have a console, mount the +ZIP drive. It is probably a good idea to mount it read only. On +http://www.toms.net/rb[tomsrtbt]: + +.... +# mount /dev/sda1 /mnt -o ro +.... + +Check to be sure it is there: + +.... +# ls -l /mnt +.... + +On http://www.knoppix.org/[Knoppix] or http://www.finnix.org/[Finnix], +you may want to make a directory under `/mnt` and mount it there, like +so: + +.... +# mkdir /mnt/zip +# mount /dev/sda1 /mnt/zip -o ro +.... + +Now cd into the mounted device, and into the `bin` directory below it. +On http://www.finnix.org/[Finnix], for example: + +.... +# cd /mnt/zip/bin +.... + +The scripts assume you are in this directory, and call data files +relative to it. At this point, you can run the restoration automatically +or manually. Use the automated restore if you don't need to make any +changes as you go along. + +One consideration here is whether you have multiple hard drives. If your +Linux installation mounts partitions on multiple hard drives, you must +mount the root partition first. This is to ensure that mount point +directories are created on the partition where they belong. The script +link:#first.stage[first.stage] will run the scripts to mount the drives +in the order in which they are created. If you have created them (in the +script `save.metadata`) in the order in which they cascade from root, +the mounting process should work just fine. + +If you have multiple hard drives, and they cross-mount, you are on your +own. Either combine and edit the scripts to mount them in the correct +order, or do it manually. + +Automated +^^^^^^^^^ + +The automatic process calls each of the manual scripts in proper order. +It does not allow for manual intervention, say for creating file systems +that this HOWTO does not support. To run the first stage restore +automatically, enter the command: + +.... +# first.stage +.... + +If you want to check for back blocks, add the `-c` option. + +Manually +^^^^^^^^ + +Run the script(s) that will restore the partition information and create +file systems. You may run them in any order, so long as they build +dependencies in the correct order. You can read the script +link:#first.stage[first.stage] to get an idea of the order. e.g.: + +.... +# ./make.dev.hda +.... + +If you want to check for back blocks, add the `-c` option. + +This script will: + +* Clean out the first 1024 bytes of the hard drive, killing off any +existing partition table and master boot record (MBR). +* Recreate the non-LVM partitions from the information gathered when you +ran link:#make.fdisk[`make.fdisk`]. +* Make ext2 and ext3 file systems on non-LVM partitions and Linux swap +partitions as appropriate. If you provide the `-c` option to the script, +it will also check for bad blocks. +* Make some types of FAT partitions. + +Now is a good time to check the geometry of the drive. Sometimes +different versions of Linux pick up different geometries, so the +geometry implicit in the file `dev.hdX` is incorrect. To force it to be +correct on http://www.knoppix.org/[Knoppix], edit +link:#make.dev.hda[`make.dev.x`]. Use the -C, -H and -S options to fdisk +to specify the cylnders, heads and sectors, respectively. Those you can +get from the file `fdisk.hdX` in the root directory of the ZIP drive. +Then re-run it. + +_______________________________________________________________________________________________________________________________________________________________________________________________ +*Note* + +If you have other operating systems or file systems to restore, now is a +good time to do so. When you've done that, reboot to your restoration +Linux and continue your first stage restoration. +_______________________________________________________________________________________________________________________________________________________________________________________________ + +If you have LVM volumes to restore, now is the time to run `make.lvs` +and `mount.lvs`. + +Now run the script(s) that create mount points and mount the partitions +to them. + +.... +# ./mount.dev.hda +.... + +Once you have created all your directories and mounted partitions to +them, you can run the script link:#restore.metadata[`restore.metadata`]. + +.... +# ./restore.metadata +.... + +This will restore the contents of the ZIP drive to the hard drive to +give you a minimal bootable system. + +You should see a directory of the ZIP disk's root directory, then a list +of the archive files as they are restored. Tar on +http://www.toms.net/rb[tomsrtbt] will tell you that tar's block size is +20, and that's fine. You can ignore it. Be sure that lilo prints out its +results: + +.... +.... + +That will be followed by the output from a ```df -m`'' command. + +Finishing Touches +^^^^^^^^^^^^^^^^^ + +If you normally boot directly to X, you could have some problems. To be +safe, the first stage script changes the run level in +`/target/etc/inittab` to 3. Note: different distributions use different +run level schemes. 3 works on Red Hat derived distributions; it may not +on others. + +You can now gracefully reboot. Remove the medium from your boot drive if +you haven't already done so, and give the computer the three fingered +salute, or its equivalent: + +.... +# shutdown -r now +.... + +or + +.... +# reboot +.... + +The computer will shut down and reboot. + +[[SecondStageRestoration]] +Second Stage Restoration +------------------------ + +As the computer reboots, go back to the BIOS and verify that the clock +is more or less correct. + +Once you have verified the clock is correct, exit the BIOS and reboot to +the hard drive. You can simply let the computer boot in its normal +sequence. You will see a lot of error messages, mostly along the lines +of ``I can't find blah! Waahhh!'' If you have done your homework +correctly up until now, those error messages won't matter. You don't +need linuxconf or apache to do what you need to do. + +___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +*Note* + +As an alternative, you can boot to single user mode (at the lilo prompt, +enter `linux single`), but you will have to configure your network +manually and fire up sshd or whatever daemons you need to restore your +system. How you do those things is very system specific. +___________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +You should be able to log into a root console (no X, no users, sorry). +You should now be able to use the network, for example to NFS mount the +backup of your system. + +If you did the two stage backup I suggested for Arkeia, you can now +restore Arkeia's database and executables. You should be able to run + +.... +/etc/rc.d/init.d/arkeia start +.... + +and start the server. If you have the GUI installed on another computer +with X installed, you should now be able to log in to Arkeia on your +tape server, and prepare your restoration. + +_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +*Note* + +When you restore, read the documentation for your restoration programs +carefully. For example, tar does not normally restore certain +characteristics of files, like suid bits. File permissions are set by +the user's umask. To restore your files exactly as you saved them, use +tar's p option. Similarly, make sure your restoration software will +restore everything exactly as you saved it. +_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +To restore the test computer: + +.... +bash# restore.all +.... + +If you used tar for your backup and restoration, and used the -k (keep +old files, don't overwrite) option, you will see a lot of this: + +.... +.... + +This is normal, as tar is refusing to overwrite files you restored +during the first stage of restoration. + +Then reboot. On the way down, you will see a lot of error messages, such +as ``no such pid.'' This is a normal part of the process. The shutdown +code is using the pid files from daemons that were running when the +backup was made to shut down daemons that were not started on the last +boot. Of course there's no such pid. + +Your system should come up normally, with a lot fewer errors than it had +before; ideally no errors. The acid test of how well your restore works +on an RPM based system is to verify all packages. During the first stage +backup, a verification was performed on the system, producing the file +`rpmVa.txt`. Verify your system again, and compare the results to the +one made earlier. E.g.: + +.... +bash# rpm -Va | sort +2 -t ' ' | uniq > ~/foo.txt +diff /mnt/zip/metadata/rpmVa.txt ~/foo.txt +.... + +Prelinking error messages are normal and you can ignore them. Do not +first run the command `/etc/cron.daily/prelink` to remove them. Doing so +may introduce new errors in the verification results that will skew your +results. + +Some files, such as configuration and log files, will have changed in +the normal course of things, and you should be able to mentally filter +those out of the report. Emacs users should check out its ediff +facilities. + +Now you should be up and running. It is time to test your applications, +especially those that run as daemons. The more sophisticated the +application, the more testing you may need to do. If you have remote +users, disable them from using the system, or make it ``read only'' +while you test it. This is especially important for databases, to +prevent making any corruption or data loss worse than it already might +be. + +If you normally boot to X, it was disabled as part of the first stage +restoration. Test X before you re-enable it. Re-enable it by changing +that one line in `/etc/inittab`. Find the line that looks like this: + +.... +.... + +and change it to this: + +.... +.... + +Or just run this on the target to change it back. Note: different +distributions use different run level schemes. These values work on Red +Hat derived distributions; they may not on others. + +.... +sed -i s/id:.:initdefault:/id:5:initdefault:/g /etc/inittab +.... + +You should now be ready for rock and roll -- and some aspirin and a +couch. + +[[DistributionSpecificNotes]] +Distribution Specific Notes +--------------------------- + +Below are distribution notes from past experiences. If you have +additional notes that you would like to add for other distributions, +please forward them to me. + +[[fedora3]] +Fedora +~~~~~~ + +The scripts now reflect Fedora 7, so you should not have to make any +changes to these link:#thescripts[scripts]. + +________________________________________________________________________________________________________________________________________________________________ +*Note* + +I tested the above on a fresh installation of FC3. I had problems with +devices after booting when I worked with a system that had been upgraded +from FC2 to FC3. +________________________________________________________________________________________________________________________________________________________________ + +[[knoppix]] +Knoppix +~~~~~~~ + +I used to use http://www.knoppix.org/[Knoppix]. +mailto:pon at iki dot fi[Pasi Oja-Nisula] reports: + +______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +For me the best thing about using Knoppix is that I don't need a +specific boot medium for each machine, but I can use the same tools all +the time. And hardware support in Knoppix is really great. I don't have +that much experience with different platforms, but all the machines I've +tried have worked fine, scsi drivers are found and so on. + +I'm doing this recovery thing by copying the backups over the network to +other machine. The restore involves booting the Knoppix cd, fetching the +metadata.tar.gz from the network machine. Then make.dev, mount.dev, +fetching the other tar.gz files, grub and reboot. Some typing involved +but thanks to your scripts it's quite straighforward. Unless changing +from ide to scsi or something, but even then it's not that difficult, +since Linux is easy to restore to different hardware. +______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +Let me add to that that http://www.knoppix.org/[Knoppix] detects USB +devices for you, which is really nice. They make excellent (and roomier) +substitutes for the ZIP drive. + +Also see +http://www-106.ibm.com/developerworks/linux/library/l-knopx.html?ca=dgr-lnxw04Knoppix[``System +recovery with Knoppix'']. + +Do your restore as user ``root'' rather than as user ``knoppix''. +Otherwise you may get some directories and files owned by an oddball +user or group. Also, for http://www.knoppix.org/[Knoppix], we tar the +first stage stuff saving numeric user & group values instead of by name. +The names may point to different numbers on knoppix, so we would be +restoring the files with incorrect user and group IDs. + +Finnix +~~~~~~ + +http://www.finnix.org/[Finnix] has some of the same advantages of +Knoppix. In addition, it runs in command line mode with mouse support, +which is great for the task at hand. It's small, under 100 MB as of this +writing, so you can remaster it with your first stage data on it. It +boots quickly. And it has LVM support. And Zile, a subset of Emacs. I am +pleased with http://www.finnix.org/[Finnix] for this use, and it is now +my standard first stage restoration Linux. + +[[ApplicationSpecificNotes]] +Application Specific Notes +-------------------------- + +Here are some notes about backing up particular applications. + +[[lvm]] +Logical Volume Manager +~~~~~~~~~~~~~~~~~~~~~~ + +Handling logical volumes turns out to be a bit of a trick: use the +http://www.finnix.org/[Finnix] distribution's startup code to turn LVM +on and off. This results in distribution specific code for the first +stage of restoration. It is generated in link:#make.fdisk[`make.fdisk`]. +To edit it, search link:#make.fdisk[`make.fdisk`] for ``Hideous''. + +LVM required the addition of two new LVM specific scripts, +link:#make.lvs[`make.lvs`] and link:#mount.lvs[`mount.lvs`]. They are +only generated and used if there are logical volumes present. + +[[selinux]] +Selinux +~~~~~~~ + +Selinux is disabled on the test machines. `/selinux` is not backed up in +any of these scripts. At a guess, you should probably disable selinux +after the first stage restoration, and you will probably have some +selinux specific tasks to perform before turning it back on. + +[[grub]] +GRUB +~~~~ + +The default bootloader in link:#fedora3[Fedora] is the +http://www.gnu.org/software/grub/[Grand Unified Bootloader (GRUB)]. It +has to run at the end of the first stage, or you won't be able to boot +thereafter. To preserve it for first stage restoration, make the +following changes: + +* Edit the penultimate stanza of +link:#restore.metadata[`restore.metadata`]: ++ +.... +.... +* Add the following stanza to link:#save.metadata[`save.metadata`]: ++ +.... +.... + +[[tripwire]] +Tripwire +~~~~~~~~ + +If you run Tripwire or any other application that maintains a database +of file metadata, rebuild that database immediately after restoring. + +[[Squid]] +Squid +~~~~~ + +Squid is a HTTP proxy and cache. As such it keeps a lot of temporary +data on the hard drive. There is no point in backing that up. Insert +``--exclude /var/spool/squid'' into the appropriate tar command in your +second stage backup script. Then, get squid to rebuild its directory +structure for you. Tack onto the tail end of the second stage restore +script a command for squid to initialize itself. Here is how I did it +over SSH in link:#restore[`restore`]: + +.... +.... + +The last command creates a file of length 0 called .OPB_NOBACKUP. This +is for the benefit of link:#arkeia[Arkeia], and tells Arkeia not to back +up below this directory + +[[Arkeia]] +Arkeia +~~~~~~ + +These notes are based on testing with Arkeia 4.2. + +http://www.arkeia.com/[Arkeia] is a backup and restore program that runs +on a wide variety of platforms. You can use Arkeia as part of a bare +metal restoration scheme, but there are two caveats. + +The first is probably the most problematic, as absent any more elegant +solution you have to hand select the directories to restore in the +navigator at restoration time. The reason is that, apparently, Arkeia +has no mechanism for not restoring files already present on the disk, +nothing analogous to `tar`'s -p option. If you simply allow a full +restore, the restore will crash as Arkeia over-writes a library which is +in use at restore time, e.g. `lib/libc-2.1.1.so`. Hand selection of +directories to restore is at best dicey, so I recommend against it. + +The second caveat is that you have to back up the Arkeia data dictionary +and/or programs. To do that, modify the `save.metatdata` script by +adding Arkeia to the list of directories to save: + +.... + $zip/arkeia.tar.gz]]> +.... + +You _must_ back up the data dictionary this way because Arkeia does not +back up the data dictionary. This is one of my complaints about Arkeia, +and I have solved it in the past by saving the data dictionary to tape +with http://www.estinc.com/[The TOLIS Group's BRU]. + +The data dictionary will be restored in the script `restore.metadata` +automatically. + +[[amanda]] +Amanda +~~~~~~ + +http://www.amanda.org/[Amanda] (The Advanced Maryland Automatic Network +Disk Archiver) works quite well with this set of scripts. Use the normal +Amanda back-up process, and build your first stage data as usual. Amanda +stores the data on tape in GNU tar or cpio format, and you can recover +from individual files to entire backup images. The nice thing about +recovering entire images is that you can then use variants on the +scripts in this HOWTO to restore from the images, or direct from tape. I +was able to restore my test machine with the directions from W. Curtis +Preston's http://www.oreilly.com/catalog/unixbr/[Unix Backup & +Recovery]. For more information on it, see the +link:#resources[Resources]. The Amanda chapter from the book is +http://www.backupcentral.com/amanda.html[on line]. + +I made two changes to the script link:#restore[`restore`]. First, I +changed it to accept a file name as an argument. Then, since Amanda's +`amrestore` decompresses the data as it restores it, I rewrote it to cat +the file into the pipe instead of decompressing it. + +The resulting line looks like this: + +.... +cat $file | ssh $target "umask 000 ; cd / ; tar -xpkf - " +.... + +where `$file` is the script's argument, the image recovered from the +tape by `amrestore`. + +Since the command line arguments to `tar` prohibit over-writing, restore +from images in the _reverse_ of the order in which they were made. +Restore most recent first. + +Amanda does require setting ownership by hand if you back up the amanda +data directory with link:#save.metadata[`save.metadata`]. Something +like: + +.... +bash# chown -R amanda:disk /var/lib/amanda +.... + +You can also add that line to your scripts for second state restoration, +such as link:#restore[`restore`]. + +[[ntfs]] +NTFS +~~~~ + +OK, NTFS isn't an application. It is a file system used by Microsoft +operating system Windows NT and its descendents, including Windows 2000 +and Windows XP. You can back it up and restore to it from Linux with +`ntfsclone`, one of the NTFS utilities in the ntfsprogs suite, available +from http://www.linux-ntfs.org/. + +These scripts will create NTFS partitions, but will not put a file +system on them. It is not clear from the docs whether `ntfsclone` will +lay down a file system on a virgin partition or not. + +[[SomeAdviceforDisasterRecovery]] +Some Advice for Disaster Recovery +--------------------------------- + +You should take your ZIP disk for each computer and the printouts you +made, and place them in a secure location in your shop. You should store +copies of these in your off-site backup storage location. The major +purpose of off-site backup storage is to enable disaster recovery, and +restoring each host onto replacement hardware is a part of disaster +recovery. + +You should also have several restoration Linux floppies or CD-ROMS, and +possibly some ZIP drives in your off-site storage as well. Also, have +copies of the rescue linux distribution on several of your computers so +that they back each other up. + +You should probably have copies of this HOWTO, with your site-specific +annotations on it, with your backups and in your off-site backup +storage. + +[[WhatNow]] +What Now? +--------- + +This HOWTO results from experiments on one computer. No doubt you will +find some directories or files you need to back up in your first stage +backup. I have not dealt with saving and restoring X on the first stage, +nor have I touched at all on processors other than AMD or Intel. + +I would appreciate your feedback as you test and improve these scripts +on your own computers. I also encourage vendors of backup software to +document how to do a minimal backup of their products. I'd like to see +the whole Linux community sleep just a little better at night. + +[[todo]] +To Do +~~~~~ + +Volunteers are most welcome. Check with me before you start on one of +these in case someone else is working on it already. + +* We have no way to determine the label of a swap partition. This means +that there is no way to provide the swap partition's label when +restoring. We could assume that a system with a single swap partition +(as indicated by fdisk) has the label used in the swap partition line in +`/etc/fstab`, but that only works on single hard drive systems, and +could produce subtle errors in systems with multiple swap partitions. ++ +The work-around is to add the label by hand by re-running `mkswap` with +the -L option on it. Sigh. +* A partition editor to adjust partition boundaries in the `dev.hdx` +file. This will let users adjust partitions for a different hard drive, +or the same one with different geometry, or to adjust partition sizes +within the same hard drive. A GUI would probably be a good idea here. On +the other tentacle, the FSF's +http://www.gnu.org/software/parted[`parted`] looks like it will fill +part of the bill. It does re-size existing partitions, but with +restrictions. +* link:#make.fdisk[`make.fdisk`] currently only recognizes some FAT +partitions, not all. Add code to link:#make.fdisk[`make.fdisk`] to +recognize others and make appropriate instructions to rebuild them in +the output files. +* For FAT12 or FAT16 partitions we do not format, write zeros into the +partition so that Mess-DOS 6.x does not get confused. See the notes on +`fdisk` for an explanation of the problem. +* Translations into other (human) languages. +* I've referred to Red Hat Package Manager (rpm) from time to time. What +are the equivalent deb commands? +* Modify the first stage backup code to only save the current kernel. + +[[TheScripts]] +The Scripts +----------- + +See the notes in the beginning of each script for a summary of what it +does. + +[[FirstStage]] +First Stage +~~~~~~~~~~~ + +[[make.fdisk]] +`make.fdisk` +^^^^^^^^^^^^ + +This script, run at backup time, creates scripts similar to +link:#make.dev.hda[`make.dev.hda`] and +link:#mount.dev.hda[`mount.dev.x`], below, for you to run at restore +time. It also produces data files similar to link:#dev.hda[`dev.hda`] +and link:#dev.hda.sfd[`dev.hda.sfd`], below. The names of the scripts +and data files produced depend on the device given this script as a a +parameter. Those script, run at restore time, build and mount the +partitions on the hard drive. `make.fdisk` is called from +link:#save.metadata[`save.metadata`], below. + +.... +.... + +[[make.dev.hda]] +`make.dev.hda` +^^^^^^^^^^^^^^ + +This script is a sample of the sort produced by +link:#make.fdisk[`make.fdisk`], above. It uses data files like +link:#dev.hda[`dev.hda`], below. It builds partitions and puts file +systems on some of them. This is the first script run at restore time. + +If you are brave enough to edit link:#dev.hda[`dev.hda`] or +link:#dev.hda.sfd[`dev.hda.sfd`] (q.v.), say, to add a new partition, +you may need to edit this script as well. + +If you want make.dev.hda to check for bad blocks when it puts a file +system on the partitions, use a "-c" command line option. + +.... +.... + +[[make.lvs]] +`make.lvs` +^^^^^^^^^^ + +`make.lvs` is generated by link:#make.fdisk[`make.fdisk`], but only if +logical volumes are present. As the name suggests, it builds the logical +volumes and makes file systems on them. + +.... +.... + +[[mount.dev.hda]] +`mount.dev.hda` +^^^^^^^^^^^^^^^ + +This script is a sample of the sort produced by +link:#make.fdisk[`make.fdisk`], above. It builds mount points and mounts +partitions on them, making the target file system ready for restoring +files. This is the second script run at restore time. + +If you are brave enough to edit link:#dev.hda[`dev.hda`] (q.v.), say, to +add a new partition, you may need to edit this script as well. + +.... +.... + +[[mount.lvs]] +`mount.lvs` +^^^^^^^^^^^ + +`mount.lvs` is generated by link:#make.fdisk[`make.fdisk`], but only if +logical volumes are present. As the name suggests, it mounts the logical +volumes ready for restoration. + +.... +.... + +[[dev.hda]] +`dev.hda` +^^^^^^^^^ + +This data file is used at restore time if `sfdisk` is not present on the +restoration Linux. It is fed to `fdisk` by the script +link:#make.dev.hda[`make.dev.hda`]. It is produced at backup time by +link:#make.fdisk[`make.fdisk`]. Those familiar with `fdisk` will +recognize that each line is an `fdisk` command or value, such as a +cylinder number. Thus, it is possible to change the partition sizes and +add new partitions by editing this file. That's why the penultimate +command is `v`, to verify the partition table before it is written. + +.... +.... + +[[dev.hda.sfd]] +`dev.hda.sfd` +^^^^^^^^^^^^^ + +This data file is used at restore time if `sfdisk` is present on the +restoration Linux system. It is fed to `sfdisk` by the script +link:#make.dev.hda[`make.dev.hda`]. It is produced at backup time by +link:#make.fdisk[`make.fdisk`]. Each line represents a partition. Thus, +it is possible to change the partition sizes and add new partitions by +editing this file. + +.... +.... + +[[save.metadata]] +`save.metadata` +^^^^^^^^^^^^^^^ + +This is the first script to run as part of the backup process. It calls +link:#make.fdisk[`make.fdisk`], above. If you have a SCSI hard drive or +multiple hard drives to back up, edit the call to +link:#make.fdisk[`make.fdisk`] appropriately. + +________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +*Note* + +Recent kernels have incorporated a new ATA (IDE) hard drive driver, +libata. Because of this, parallel ATA (PATA) drives now show up as SCSI +drives, as serial ATA (SATA) have always done. However, not all rescue +distributions (e.g. Finnix) use this new driver. There is a line toward +the bottom of link:#save.metadata[`save.metadata`] wich very carefully +replaces "/dev/sda" with "/dev/hda". Use this as a template if you have +multiple IDE hard drives. Comment it out or delete it if this is not an +issue for you. + +Note that there is no guaranteed mapping! Systems with multiple hard +drives may have confusing mappings. Be sure to edit this line carefully. +Check it if you add or remove a hard drive of any interface type to or +from your system! + +N.B: if you have libata IDE drive issues, the grub-install line at the +end of link:#restore.metadata[`restore.metadata`] won't work. If it +doesn't, use your rescue disk to do the same. Or burn and boot to the +boot image that is made as part of this script. Boot to it and do the +second state restore as usual. The second state restore should re-run +`grub-install`. +________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +.... +.... + +[[restore.metadata]] +`restore.metadata` +^^^^^^^^^^^^^^^^^^ + +This script restores metadata from the ZIP disk as a first stage +restore. + +N.B: if you have libata IDE drive issues, the grub-install line at the +end of this script won't work. If it doesn't, use your rescue disk to do +the same. + +.... +.... + +[[first.stage]] +`first.stage` +^^^^^^^^^^^^^ + +This script runs the entire first stage restore with no operator +intervention. + +If you want to check for bad blocks when it puts a file system on the +partitions, use a "-c" command line option. + +.... +.... + +[[SecondStage]] +Second Stage +~~~~~~~~~~~~ + +These scripts run on the computer being backed up or restored. + +[[back.up.all]] +`back.up.all` +^^^^^^^^^^^^^ + +This script saves to another computer via an NFS mount. You can adapt it +to save to tape drives or other media. + +.... +.... + +[[back.up.all.ssh]] +`back.up.all.ssh` +^^^^^^^^^^^^^^^^^ + +This script does exactly what link:#back.up.all[`back.up.all`] does, but +it uses SSH instead of NFS. + +.... +.... + +[[restore.all]] +`restore.all` +^^^^^^^^^^^^^ + +This is the restore script to use if you backed up using +link:#back.up.all[`back.up.all`]. + +.... +.... + +[[restore.all.ssh]] +`restore.all.ssh` +^^^^^^^^^^^^^^^^^ + +This is the restoration script to use if you used +link:#back.up.all.ssh[`back.up.all.ssh`] to back up. + +.... +.... + +[[BackupServerScripts]] +Backup Server Scripts +~~~~~~~~~~~~~~~~~~~~~ + +The SSH scripts above have a possible security problem. If you run them +on a firewall, the firewall has to have access via SSH to the backup +server. In that case, a clever cracker might also be able to crack the +backup server. It would be more secure to run backup and restore scripts +on the backup server, and let the backup server have access to the +firewall. That is what these scripts are for. + +These scripts backup and restore the target completely, not just the +stage one backup and restore. `get` backs up the bare metal archive +separately so that you can make a CD-ROM ir NFS mount from it. + +I use these scripts routinely. + +[[get]] +`get` +^^^^^ + +.... +.... + +[[restore]] +`restore` +^^^^^^^^^ + +.... +.... + +[[get.target]] +`get.target` +^^^^^^^^^^^^ + +.... +.... + +[[misc.files]] +Miscellaneous Files +~~~~~~~~~~~~~~~~~~~ + +[[install]] +`install` +^^^^^^^^^ + +This little script just installs things and sets up a few directories. +It would be a useful basis for an RPM or deb package. The placement of +files is based on the __http://www.pathname.com/fhs/[Filesystem +Hierarchy Standard]__, version 2.3, announced on January 29, 2004. + +.... +.... + +[[Resources]] +Resources +--------- + +In no particular order. These are things you might want to investigate +for yourself. A listing here should not be taken as an endorsement. In +fact, in many case I have not used the product and cannot comment on it. + +* http://osdev.berlios.de/netboot.html[Network-booting Your Operating +System] describes several techniques for booting across a network, using +http://www.gnu.org/software/grub/[grub] and some other tricks. I haven't +tried it, but I have a sneaky suspicion that with an especially trained +floppy diskette, you could get your entire first stage image onto the +computer to be restored. +* ``http://btmgr.webframe.org/[Smart Boot Manager (SBM)] is an OS +independent and full-featured boot manager with an easy-to-use user +interface. There are some screen shots available.'' It is essential if +your BIOS will not allow you to boot to CD-ROM and you want to use a +CD-ROM based Linux for Stage 1 recovery. +* http://www.oreilly.com/catalog/unixbr/author.html[W. Curtis Preston]'s +excellent http://www.oreilly.com/catalog/unixbr/[Unix Backup & +Recovery]. This is the book that got me started on this bare metal +recovery stuff. I highly recommend it; +http://www2.linuxjournal.com/lj-issues/issue78/3839.html[read my +review]. +* An old (2000) list of +http://www.fokus.gmd.de/linux/linux-distrib-small.html[small Linux +disties.] +* http://www.toms.net/rb[tomsrtbt], ``The most Linux on 1 floppy disk.'' +Tom also has links to other small disties. +* The http://www.tldp.org/[Linux Documentation Project]. See +particularly the ``LILO, Linux Crash Rescue HOW-TO.'' +* The Free Software Foundation's +http://www.gnu.org/software/parted[`parted`] for editing (enlarging, +shrinking, moving) partitions. +* http://qtparted.sourceforge.net/[QtParted] looks to do the same thing +with a GUI front end. +* http://www.partimage.org/[Partition Image] for backing up partitions. ++ +From the web page: ``Partition Image is a Linux/UNIX utility which saves +partitions in many formats (see below) to an image file. The image file +can be compressed in the GZIP/BZIP2 formats to save disk space, and +split into multiple files to be copied on removable floppies (ZIP for +example), .... The partition can be saved across the network since +version 0.6.0.'' +* http://sourceforge.net/projects/bacula[Bacula] is a GLPled backup +product which has bare metal recovery code inspired in part by this +HOWTO. +* ``http://www.feyrer.de/g4u/[g4u ('ghost for unix')] is a NetBSD-based +bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a +common setup on a number of PCs using FTP. The floppy/CD offers two +functions. First is to upload the compressed image of a local harddisk +to a FTP server. Other is to restore that image via FTP, uncompress it +and write it back to disk; network configuration is fetched via DHCP. As +the harddisk is processed as a image, any filesystem and operating +system can be deployed using g4u.'' +* ``We present +http://www.cs.utah.edu/flux/papers/frisbee-usenix03-base.html[Frisbee], +a system for saving, transferring, and installing entire disk images, +whose goals are speed and scalability in a LAN environment. Among the +techniques Frisbee uses are an appropriately-adapted method of +filesystem-aware compression, a custom application-level reliable +multicast protocol, and flexible application-level framing. This design +results in a system which can rapidly and reliably distribute a disk +image to many clients simultaneously. For example, Frisbee can write a +total of 50 gigabytes of data to 80 disks in 34 seconds on commodity PC +hardware. We describe Frisbee's design and implementation, review +important design decisions, and evaluate its performance.'' +* There are a number of USB key disties available. Check +http://www.distrowatch.com/[DistroWatch] for details. +* CD-ROM based rescue kits. This is not intended to be an exhaustive +list. If you know of one (or even something that pretends to be one), +please link:&myemail;[let me know]. You may find more recent information +at http://www.distrowatch.com/[DistroWatch]. +** Hugo Rabson's http://www.microwerks.net/~hugo/[Mondo] ``... creates +one or more bootable Rescue CD's (or tape+floppies) containing some or +all of your filesystem. In the event of catastrophic data loss, you will +be able to restore from bare metal.'' +** The http://crashrecovery.org/[Crash Recovery Kit for Linux] +** http://www-106.ibm.com/developerworks/linux/library/l-knopx.html?ca=dgr-lnxw04Knoppix[``System +recovery with Knoppix''] is a good introduction to system recovery in +general, and has some useful http://www.knoppix.org/[Knoppix] links. +** ``http://emergencycd2.sourceforge.net/[Cool Linux CD] is live CD with +Linux system. This used 2.4 kernel and some free and demo soft.'' +** http://www.sysresccd.org/index.en.php[SystemRescueCd]``is a linux +system on a bootable cdrom for repairing your system and your data after +a crash. It also aims to provide an easy way to carry out admin tasks on +your computer, such as creating and editing the partitions of the hard +disk. It contains a lot of system utilities (parted, partimage, fstools, +...) and basic ones (editors, midnight commander, network tools). It +aims to be very easy to use: just boot from the cdrom, and you can do +everything. The kernel of the system supports most important file +systems (ext2/ext3, reiserfs, xfs, jfs, vfat, ntfs, iso9660), and +network ones (samba and NFS).'' +** http://syslinux.zytor.com/[Syslinux] builds boot code for floppy +diskettes, CD-ROMs and Intel PXE (Pre-Execution Environment) images. It +is not dependent on a floppy diskette image. You can build your own CDs +with a number of tools, such as http://www.toms.net/rb[tomsrtbt], on it. +** In case you'd like to roll your own: +``http://www.linux-live.org/[Linux Live] is a set of bash scripts which +allows you to create [your] own LiveCD from every Linux distribution. +Just install your favourite distro, remove all unnecessary files (for +example man pages and all other files which are not important for you) +and then download and run these scripts.'' +** ``The http://www.linbox.com/en/ppart.html[PPART CD] allows you to +generate system recovery bootable CD of previously saved hard disks.'' +** http://rescuecd.sourceforge.net/[Timo's Rescue CD Set]: ``This set is +my approach for an easy way to generate a rescue system on a bootable +cd, which can easily be adapted to the own needs. The project evolves +more and more into a 'debian on cd' project, so it's not only possible +to use the system as a rescuecd, it is also possible to install a whole +debian system on cd.'' +** The http://www.frozentech.com/content/livecd.php[List of Live CDs] +has more CD based disties. + +[[appendix1gfdl]] +GNU Free Documentation License +------------------------------ + +Version 1.1, March 2000 + +____________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +Copyright (C) 2000 Free Software Foundation, Inc. 51 Franklin Street, +Fifth Floor, Boston, MA 02110-1301, USA Everyone is permitted to copy +and distribute verbatim copies of this license document, but changing it +is not allowed. +____________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +[[gfdl02]] +PREAMBLE +-------- + +The purpose of this License is to make a manual, textbook, or other +written document "free" in the sense of freedom: to assure everyone the +effective freedom to copy and redistribute it, with or without modifying +it, either commercially or noncommercially. Secondarily, this License +preserves for the author and publisher a way to get credit for their +work, while not being considered responsible for modifications made by +others. + +This License is a kind of "copyleft", which means that derivative works +of the document must themselves be free in the same sense. It +complements the GNU General Public License, which is a copyleft license +designed for free software. + +We have designed this License in order to use it for manuals for free +software, because free software needs free documentation: a free program +should come with manuals providing the same freedoms that the software +does. But this License is not limited to software manuals; it can be +used for any textual work, regardless of subject matter or whether it is +published as a printed book. We recommend this License principally for +works whose purpose is instruction or reference. + +[[gfdl03]] +APPLICABILITY AND DEFINITIONS +----------------------------- + +This License applies to any manual or other work that contains a notice +placed by the copyright holder saying it can be distributed under the +terms of this License. The "Document", below, refers to any such manual +or work. Any member of the public is a licensee, and is addressed as +"you". + +A "Modified Version" of the Document means any work containing the +Document or a portion of it, either copied verbatim, or with +modifications and/or translated into another language. + +A "Secondary Section" is a named appendix or a front-matter section of +the Document that deals exclusively with the relationship of the +publishers or authors of the Document to the Document's overall subject +(or to related matters) and contains nothing that could fall directly +within that overall subject. (For example, if the Document is in part a +textbook of mathematics, a Secondary Section may not explain any +mathematics.) The relationship could be a matter of historical +connection with the subject or with related matters, or of legal, +commercial, philosophical, ethical or political position regarding them. + +The "Invariant Sections" are certain Secondary Sections whose titles are +designated, as being those of Invariant Sections, in the notice that +says that the Document is released under this License. + +The "Cover Texts" are certain short passages of text that are listed, as +Front-Cover Texts or Back-Cover Texts, in the notice that says that the +Document is released under this License. + +A "Transparent" copy of the Document means a machine-readable copy, +represented in a format whose specification is available to the general +public, whose contents can be viewed and edited directly and +straightforwardly with generic text editors or (for images composed of +pixels) generic paint programs or (for drawings) some widely available +drawing editor, and that is suitable for input to text formatters or for +automatic translation to a variety of formats suitable for input to text +formatters. A copy made in an otherwise Transparent file format whose +markup has been designed to thwart or discourage subsequent modification +by readers is not Transparent. A copy that is not "Transparent" is +called "Opaque". + +Examples of suitable formats for Transparent copies include plain ASCII +without markup, Texinfo input format, LaTeX input format, SGML or XML +using a publicly available DTD, and standard-conforming simple HTML +designed for human modification. Opaque formats include PostScript, PDF, +proprietary formats that can be read and edited only by proprietary word +processors, SGML or XML for which the DTD and/or processing tools are +not generally available, and the machine-generated HTML produced by some +word processors for output purposes only. + +The "Title Page" means, for a printed book, the title page itself, plus +such following pages as are needed to hold, legibly, the material this +License requires to appear in the title page. For works in formats which +do not have any title page as such, "Title Page" means the text near the +most prominent appearance of the work's title, preceding the beginning +of the body of the text. + +[[gfdl04]] +VERBATIM COPYING +---------------- + +You may copy and distribute the Document in any medium, either +commercially or noncommercially, provided that this License, the +copyright notices, and the license notice saying this License applies to +the Document are reproduced in all copies, and that you add no other +conditions whatsoever to those of this License. You may not use +technical measures to obstruct or control the reading or further copying +of the copies you make or distribute. However, you may accept +compensation in exchange for copies. If you distribute a large enough +number of copies you must also follow the conditions in section 3. + +You may also lend copies, under the same conditions stated above, and +you may publicly display copies. + +[[gfdl05]] +COPYING IN QUANTITY +------------------- + +If you publish printed copies of the Document numbering more than 100, +and the Document's license notice requires Cover Texts, you must enclose +the copies in covers that carry, clearly and legibly, all these Cover +Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the +back cover. Both covers must also clearly and legibly identify you as +the publisher of these copies. The front cover must present the full +title with all words of the title equally prominent and visible. You may +add other material on the covers in addition. Copying with changes +limited to the covers, as long as they preserve the title of the +Document and satisfy these conditions, can be treated as verbatim +copying in other respects. + +If the required texts for either cover are too voluminous to fit +legibly, you should put the first ones listed (as many as fit +reasonably) on the actual cover, and continue the rest onto adjacent +pages. + +If you publish or distribute Opaque copies of the Document numbering +more than 100, you must either include a machine-readable Transparent +copy along with each Opaque copy, or state in or with each Opaque copy a +publicly-accessible computer-network location containing a complete +Transparent copy of the Document, free of added material, which the +general network-using public has access to download anonymously at no +charge using public-standard network protocols. If you use the latter +option, you must take reasonably prudent steps, when you begin +distribution of Opaque copies in quantity, to ensure that this +Transparent copy will remain thus accessible at the stated location +until at least one year after the last time you distribute an Opaque +copy (directly or through your agents or retailers) of that edition to +the public. + +It is requested, but not required, that you contact the authors of the +Document well before redistributing any large number of copies, to give +them a chance to provide you with an updated version of the Document. + +[[gfdl06]] +MODIFICATIONS +------------- + +You may copy and distribute a Modified Version of the Document under the +conditions of sections 2 and 3 above, provided that you release the +Modified Version under precisely this License, with the Modified Version +filling the role of the Document, thus licensing distribution and +modification of the Modified Version to whoever possesses a copy of it. +In addition, you must do these things in the Modified Version: + +A. Use in the Title Page (and on the covers, if any) a title distinct +from that of the Document, and from those of previous versions (which +should, if there were any, be listed in the History section of the +Document). You may use the same title as a previous version if the +original publisher of that version gives permission. +B. List on the Title Page, as authors, one or more persons or entities +responsible for authorship of the modifications in the Modified Version, +together with at least five of the principal authors of the Document +(all of its principal authors, if it has less than five). +C. State on the Title page the name of the publisher of the Modified +Version, as the publisher. +D. Preserve all the copyright notices of the Document. +E. Add an appropriate copyright notice for your modifications adjacent +to the other copyright notices. +F. Include, immediately after the copyright notices, a license notice +giving the public permission to use the Modified Version under the terms +of this License, in the form shown in the Addendum below. +G. Preserve in that license notice the full lists of Invariant Sections +and required Cover Texts given in the Document's license notice. +H. Include an unaltered copy of this License. +I. Preserve the section entitled "History", and its title, and add to +it an item stating at least the title, year, new authors, and publisher +of the Modified Version as given on the Title Page. If there is no +section entitled "History" in the Document, create one stating the +title, year, authors, and publisher of the Document as given on its +Title Page, then add an item describing the Modified Version as stated +in the previous sentence. +J. Preserve the network location, if any, given in the Document for +public access to a Transparent copy of the Document, and likewise the +network locations given in the Document for previous versions it was +based on. These may be placed in the "History" section. You may omit a +network location for a work that was published at least four years +before the Document itself, or if the original publisher of the version +it refers to gives permission. +K. In any section entitled "Acknowledgements" or "Dedications", +preserve the section's title, and preserve in the section all the +substance and tone of each of the contributor acknowledgements and/or +dedications given therein. +L. Preserve all the Invariant Sections of the Document, unaltered in +their text and in their titles. Section numbers or the equivalent are +not considered part of the section titles. +M. Delete any section entitled "Endorsements". Such a section may not +be included in the Modified Version. +N. Do not retitle any existing section as "Endorsements" or to conflict +in title with any Invariant Section. + +If the Modified Version includes new front-matter sections or appendices +that qualify as Secondary Sections and contain no material copied from +the Document, you may at your option designate some or all of these +sections as invariant. To do this, add their titles to the list of +Invariant Sections in the Modified Version's license notice. These +titles must be distinct from any other section titles. + +You may add a section entitled "Endorsements", provided it contains +nothing but endorsements of your Modified Version by various +parties--for example, statements of peer review or that the text has +been approved by an organization as the authoritative definition of a +standard. + +You may add a passage of up to five words as a Front-Cover Text, and a +passage of up to 25 words as a Back-Cover Text, to the end of the list +of Cover Texts in the Modified Version. Only one passage of Front-Cover +Text and one of Back-Cover Text may be added by (or through arrangements +made by) any one entity. If the Document already includes a cover text +for the same cover, previously added by you or by arrangement made by +the same entity you are acting on behalf of, you may not add another; +but you may replace the old one, on explicit permission from the +previous publisher that added the old one. + +The author(s) and publisher(s) of the Document do not by this License +give permission to use their names for publicity for or to assert or +imply endorsement of any Modified Version. + +[[gfdl07]] +COMBINING DOCUMENTS +------------------- + +You may combine the Document with other documents released under this +License, under the terms defined in section 4 above for modified +versions, provided that you include in the combination all of the +Invariant Sections of all of the original documents, unmodified, and +list them all as Invariant Sections of your combined work in its license +notice. + +The combined work need only contain one copy of this License, and +multiple identical Invariant Sections may be replaced with a single +copy. If there are multiple Invariant Sections with the same name but +different contents, make the title of each such section unique by adding +at the end of it, in parentheses, the name of the original author or +publisher of that section if known, or else a unique number. Make the +same adjustment to the section titles in the list of Invariant Sections +in the license notice of the combined work. + +In the combination, you must combine any sections entitled "History" in +the various original documents, forming one section entitled "History"; +likewise combine any sections entitled "Acknowledgements", and any +sections entitled "Dedications". You must delete all sections entitled +"Endorsements." + +[[gfdl08]] +COLLECTIONS OF DOCUMENTS +------------------------ + +You may make a collection consisting of the Document and other documents +released under this License, and replace the individual copies of this +License in the various documents with a single copy that is included in +the collection, provided that you follow the rules of this License for +verbatim copying of each of the documents in all other respects. + +You may extract a single document from such a collection, and distribute +it individually under this License, provided you insert a copy of this +License into the extracted document, and follow this License in all +other respects regarding verbatim copying of that document. + +[[gfdl09]] +AGGREGATION WITH INDEPENDENT WORKS +---------------------------------- + +A compilation of the Document or its derivatives with other separate and +independent documents or works, in or on a volume of a storage or +distribution medium, does not as a whole count as a Modified Version of +the Document, provided no compilation copyright is claimed for the +compilation. Such a compilation is called an "aggregate", and this +License does not apply to the other self-contained works thus compiled +with the Document, on account of their being thus compiled, if they are +not themselves derivative works of the Document. + +If the Cover Text requirement of section 3 is applicable to these copies +of the Document, then if the Document is less than one quarter of the +entire aggregate, the Document's Cover Texts may be placed on covers +that surround only the Document within the aggregate. Otherwise they +must appear on covers around the whole aggregate. + +[[gfdl10]] +TRANSLATION +----------- + +Translation is considered a kind of modification, so you may distribute +translations of the Document under the terms of section 4. Replacing +Invariant Sections with translations requires special permission from +their copyright holders, but you may include translations of some or all +Invariant Sections in addition to the original versions of these +Invariant Sections. You may include a translation of this License +provided that you also include the original English version of this +License. In case of a disagreement between the translation and the +original English version of this License, the original English version +will prevail. + +[[gfdl11]] +TERMINATION +----------- + +You may not copy, modify, sublicense, or distribute the Document except +as expressly provided for under this License. Any other attempt to copy, +modify, sublicense or distribute the Document is void, and will +automatically terminate your rights under this License. However, parties +who have received copies, or rights, from you under this License will +not have their licenses terminated so long as such parties remain in +full compliance. + +[[gfdl12]] +FUTURE REVISIONS OF THIS LICENSE +-------------------------------- + +The Free Software Foundation may publish new, revised versions of the +GNU Free Documentation License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. See http://www.gnu.org/copyleft/. + +Each version of the License is given a distinguishing version number. If +the Document specifies that a particular numbered version of this +License "or any later version" applies to it, you have the option of +following the terms and conditions either of that specified version or +of any later version that has been published (not as a draft) by the +Free Software Foundation. If the Document does not specify a version +number of this License, you may choose any version ever published (not +as a draft) by the Free Software Foundation. + +[[gfdl13]] +How to use this License for your documents +------------------------------------------ + +To use this License in a document you have written, include a copy of +the License in the document and put the following copyright and license +notices just after the title page: + +________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ +Copyright (c) YEAR YOUR NAME. Permission is granted to copy, distribute +and/or modify this document under the terms of the GNU Free +Documentation License, Version 1.1 or any later version published by the +Free Software Foundation; with the Invariant Sections being LIST THEIR +TITLES, with the Front-Cover Texts being LIST, and with the Back-Cover +Texts being LIST. A copy of the license is included in the section +entitled "GNU Free Documentation License". +________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ + +If you have no Invariant Sections, write "with no Invariant Sections" +instead of saying which ones are invariant. If you have no Front-Cover +Texts, write "no Front-Cover Texts" instead of "Front-Cover Texts being +LIST"; likewise for Back-Cover Texts. + +If your document contains nontrivial examples of program code, we +recommend releasing these examples in parallel under your choice of free +software license, such as the GNU General Public License, to permit +their use in free software.