225 lines
11 KiB
Plaintext
225 lines
11 KiB
Plaintext
Linux Crash HOWTO
|
|
|
|
Norman Patten
|
|
|
|
nepatten@us.ibm.com
|
|
|
|
2002-01-30
|
|
Revision History
|
|
Revision 1.0 2002-01-30 Revised by: NM
|
|
Initial release.
|
|
|
|
|
|
This document describes the installation and usage of the LKCD (Linux Kernel
|
|
Crash Dump) package.
|
|
|
|
-----------------------------------------------------------------------------
|
|
Table of Contents
|
|
1. Introduction
|
|
1.1. Copyright and License
|
|
|
|
|
|
2. How LKCD Works
|
|
2.1. What You Need
|
|
|
|
|
|
3. Installation of lkcd
|
|
3.1. Installing From Source Code
|
|
3.2. Building and Installing LKCD Utilities
|
|
3.3. What Gets Installed
|
|
3.4. Installing LKCD Utilities From RPM
|
|
3.5. Patching the Kernel
|
|
3.6. Build and Install the Kernel
|
|
|
|
|
|
4. Setup, Test, and Running crash
|
|
4.1. Setting up crash dump
|
|
4.2. Testing crash
|
|
4.3. Running crash
|
|
|
|
|
|
|
|
-----------------------------------------------------------------------------
|
|
1. Introduction
|
|
|
|
The LKCD (Linux Kernel Crash Dump) project is a set of kernel patches and
|
|
utilities to allow a copy of the kernel memory to be saved in the event of a
|
|
kernel panic. The saved kernel image makes forensics on the kernel panic
|
|
possible with utilities included in the package. Most commercial Unix
|
|
operating systems come with similar crash utilities, but this package is
|
|
fairly new to Linux and has to be added on manually. The LKCD utility is not
|
|
designed to gather helpful information in the case of a hardware caused panic
|
|
or a segment violation. The complete LKCD package is available for download
|
|
at [http://lkcd.sourceforge.net/] http://lkcd.sourceforge.net/.
|
|
-----------------------------------------------------------------------------
|
|
|
|
1.1. Copyright and License
|
|
|
|
This document is copyrighted (c) 2002 by Norman Patten. Permission is granted
|
|
to copy, distribute and/or modify this document under the terms of the GNU
|
|
Free Documentation License, Version 1.1 or any later version published by the
|
|
Free Software Foundation; with no Invariant Sections, with no Front-Cover
|
|
Texts, and with no Back-Cover Texts. A copy of the license is available at
|
|
[http://www.gnu.org/copyleft/fdl.html] http://www.gnu.org/copyleft/fdl.html.
|
|
|
|
Linux is a registered trademark of Linus Torvalds . lkcd is distributed under
|
|
the copyright of Silicon Graphics Inc.
|
|
|
|
Send feedback to nepatten@us.ibm.com.
|
|
-----------------------------------------------------------------------------
|
|
|
|
2. How LKCD Works
|
|
|
|
When a kernel encounters certain errors it calls the "panic" function which
|
|
results from a unrecoverable error. This panic results in LKCD initiating a
|
|
kernel dump where kernel memory is copied out to the pre-designated dump
|
|
area. The dump device is configured as primary swap by default. The kernel is
|
|
not completely functional at this point, but there is enough functionality to
|
|
copy memory to disk. After dump finishes copying memory to disk, the system
|
|
re-boots. When the system boots back up, it checks for a new crash dump. If a
|
|
new crash dump is found it is copied from the dump location to the file
|
|
system, "/var/log/dump" directory by default. After copying the image, the
|
|
system continues to boot normally and forensics can be performed at a later
|
|
date.
|
|
-----------------------------------------------------------------------------
|
|
|
|
2.1. What You Need
|
|
|
|
lkcd-kernelxxx.diff file for patching the kernel. The kernel version
|
|
supported will change routinely. lkcdutils-xx.src.rpm - this is the utilities
|
|
source and scripts you will need to setup and read a crash. At the time of
|
|
this writing there is a i386 binary rpm available from [http://
|
|
lkcd.sourceforge.net/] lkcd.sourceforge.net, but you will still need the
|
|
patches for the startup scripts from the source rpm.
|
|
-----------------------------------------------------------------------------
|
|
|
|
3. Installation of lkcd
|
|
|
|
3.1. Installing From Source Code
|
|
|
|
Get the lkcdutils-xxx.src.rpm and install it using rpm -i
|
|
kcdutils-xxx.src.rpm . This will place a file called lkcdutils-xxx.tar.gz in
|
|
the /usr/src/redhat/SOURCES directory. This file is a compressed tar image of
|
|
the lkcd source tree. Unwind the source in a directory of your choice like "/
|
|
usr/src" with tar -zxvf kcdutils-xxx.src.rpm . This will create a directory
|
|
called "kcdutils-xxx" which will contain the LKCD utilities source.
|
|
-----------------------------------------------------------------------------
|
|
|
|
3.2. Building and Installing LKCD Utilities
|
|
|
|
LKCD used the standard GCC compiler and make files. To build the suite, cd to
|
|
the LKCD src directory and run ./configure to build configuration files. The
|
|
next step is to run make to build the utilities, and finally run make install
|
|
to install the utilities and man pages.
|
|
-----------------------------------------------------------------------------
|
|
|
|
3.3. What Gets Installed
|
|
|
|
+-------------------------------------------------------------------------------+
|
|
| /etc/sysconfig/dump # Configuration file for dump |
|
|
| /sbin/lcrash # The crash utility |
|
|
| /sbin/lkcd # Script to configure and save a crash |
|
|
| /sbin/lkcd_config # Configuration utility for dump |
|
|
| /sbin/lkcd_ksyms # Utility for reconstructing kernel symbols |
|
|
| /usr/include/sial_api.h # Header file for the SIAL API |
|
|
| /usr/lib/libsial.a # Simple Image Access Language library |
|
|
| /usr/man/man1/lcrash.1 # man page for lcrash |
|
|
| /usr/man/man1/lkcd_config.1 # man page for lkcd_config |
|
|
| /usr/man/man1/lkcd_ksyms.1 # man page for lkcd_ksyms |
|
|
| /usr/share/sial/lcrash/ps.sial # ps command implementation of SIAL |
|
|
| |
|
|
+-------------------------------------------------------------------------------+
|
|
-----------------------------------------------------------------------------
|
|
|
|
3.4. Installing LKCD Utilities From RPM
|
|
|
|
You can install the pre-built utilities from rpm by running rpm -i
|
|
kcdutils-xxx.rpm . You will still need to patch the kernel and install the
|
|
startup script patches. However you can bypass building the utilities step.
|
|
-----------------------------------------------------------------------------
|
|
|
|
3.5. Patching the Kernel
|
|
|
|
The next step is patching and rebuilding the kernel. You will need to patch
|
|
the kernel source with the lkcd-xxx.diff file you downloaded from [http://
|
|
lkcd.sourceforge.net/] http://lkcd.sourceforge.net/. Copy the patch into the
|
|
same directory as your kernel and run patch -p0 < lkcd-kernelxxx.diff . Make
|
|
sure the patch you use is the same version as the kernel you are patching.
|
|
Next you will need to configure the kernel to enable crash dump support. By
|
|
default crash support is turned off after applying the patch. If you use make
|
|
menuconfig or make xconfig, the "LKCD support" option is under kernel
|
|
hacking. You will also need to enable other kernel features you might need.
|
|
See the The Linux Kernel HOWTO for further details.
|
|
-----------------------------------------------------------------------------
|
|
|
|
3.6. Build and Install the Kernel
|
|
|
|
The next is building and installing the crash enabled kernel. In the kernel
|
|
source directory run the following commands in order.
|
|
+---------------------------------------------------------------------------+
|
|
| |
|
|
| make depend |
|
|
| make install |
|
|
| make modules |
|
|
| make modules_install |
|
|
| |
|
|
+---------------------------------------------------------------------------+
|
|
|
|
This will build and installing the new kernel, you will also need to copy the
|
|
Kerntypes file from the kernel source to the /boot directory. You may also
|
|
need to edit the lilo.conf file to point to you new kernel. See [http://
|
|
linuxdoc.org/HOWTO/Kernel-HOWTO.html] http://www.linuxdoc.org/HOWTO/
|
|
Kernel-HOWTO.html for more information on building and installing a kernel.
|
|
-----------------------------------------------------------------------------
|
|
|
|
4. Setup, Test, and Running crash
|
|
|
|
4.1. Setting up crash dump
|
|
|
|
In order to save a core image that has been written into swap, the image must
|
|
be saved prior to swap being re-mounted during boot. To accomplish this, the
|
|
sysinit startup file needs to be changed. The lkcd source includes a scripts
|
|
directory which contains patches for various sysinit startup scripts. These
|
|
patches add the lkcd config and lkcd save commands to enable crash dumps and
|
|
to save any existing crash dumps upon startup.
|
|
-----------------------------------------------------------------------------
|
|
|
|
4.2. Testing crash
|
|
|
|
To force a panic to test you new crash setup, compile the following code with
|
|
cc -c -I/usr/src/linux/include panic.c . After building the panic.o module
|
|
just insmod panic.o to panic the kernel.
|
|
+---------------------------------------------------------------------------+
|
|
| ### panic.c ########################### |
|
|
| |
|
|
| #define __KERNEL__ |
|
|
| # MODULE |
|
|
| |
|
|
| # include init_module(void) |
|
|
| |
|
|
| int init_module (void) |
|
|
| { |
|
|
| panic(" panic has been called"); |
|
|
| return 0; |
|
|
| } |
|
|
| |
|
|
+---------------------------------------------------------------------------+
|
|
-----------------------------------------------------------------------------
|
|
|
|
4.3. Running crash
|
|
|
|
To view your kernel core file lcrash needs to be invoked with a couple of
|
|
parameters:
|
|
+---------------------------------------------------------------------------+
|
|
| lcrash [ System.map file ] [ dump image ] [ Kerntypes ] |
|
|
| |
|
|
| Example: |
|
|
| |
|
|
| lcrash /boot/System.map ./dump.1 /boot/Kerntypes |
|
|
| |
|
|
+---------------------------------------------------------------------------+
|
|
|
|
It will take a minute to load the kernel image into memory and drop you into
|
|
the crash shell. At the crash shell prompt you can type a ? to see the
|
|
available commands.
|