LDP/LDP/guide/docbook/sag/ch07.sgml

506 lines
20 KiB
Plaintext

<chapter id="system-monitoring">
<title>System Monitoring</title>
<blockquote><para><quote>That's Hall Monitor to you!</quote>Spongebob
Squarepants</para></blockquote>
<para>One of the most important responsibilities a system administrator
has, is monitoring their systems. As a system administrator you'll need
the ability to find out what is happening on your system at any given
time. Whether it's the percentage of system's resources currently used,
what commands are being run, or who is logged on. This chapter will cover
how to monitor your system, and in some cases, how to resolve problems
that may arise.</para>
<para>When a performance issue arises, there are 4 main areas to consider:
CPU, Memory, Disk I/O, and Network. The ability to determine where the
bottleneck is can save you a lot of time.</para>
<sect1 id="system-resources">
<title>System Resources</title>
<para>Being able to monitor the performance of your system
is essential. If system resources become too low it can cause a lot of
problems. System resources can be taken up by individual users, or by
services your system may host such as email or web pages. The ability to
know what is happening can help determine whether system upgrades are needed,
or if some services need to be moved to another machine.</para>
<sect2 id="top">
<title>The <command>top</command> command.</title>
<para>The most common of these commands is <command>top</command>.
The <command>top</command> will display a continually updating report
of system resource usage.
<screen>
<prompt>#</prompt> <userinput>top</userinput>
<computeroutput> 12:10:49 up 1 day, 3:47, 7 users, load average: 0.23, 0.19, 0.10
125 processes: 105 sleeping, 2 running, 18 zombie, 0 stopped
CPU states: 5.1% user 1.1% system 0.0% nice 0.0% iowait 93.6% idle
Mem: 512716k av, 506176k used, 6540k free, 0k shrd, 21888k buff
Swap: 1044216k av, 161672k used, 882544k free 199388k cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
2330 root 15 0 161M 70M 2132 S 4.9 14.0 1000m 0 X
2605 weeksa 15 0 8240 6340 3804 S 0.3 1.2 1:12 0 kdeinit
3413 weeksa 15 0 6668 5324 3216 R 0.3 1.0 0:20 0 kdeinit
18734 root 15 0 1192 1192 868 R 0.3 0.2 0:00 0 top
1619 root 15 0 776 608 504 S 0.1 0.1 0:53 0 dhclient
1 root 15 0 480 448 424 S 0.0 0.0 0:03 0 init
2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kapmd
4 root 35 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd_CPU0
9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
5 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kswapd
10 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated
11 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
15 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
81 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
1188 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
1675 root 15 0 604 572 520 S 0.0 0.1 0:00 0 syslogd
1679 root 15 0 428 376 372 S 0.0 0.0 0:00 0 klogd
1707 rpc 15 0 516 440 436 S 0.0 0.0 0:00 0 portmap
1776 root 25 0 476 428 424 S 0.0 0.0 0:00 0 apmd
1813 root 25 0 752 528 524 S 0.0 0.1 0:00 0 sshd
1828 root 25 0 704 548 544 S 0.0 0.1 0:00 0 xinetd
1847 ntp 15 0 2396 2396 2160 S 0.0 0.4 0:00 0 ntpd
1930 root 24 0 76 4 0 S 0.0 0.0 0:00 0 rpc.rquotad
</computeroutput>
</screen></para>
<para>The top portion of the report lists information such as
the system time, uptime, CPU usage, physical and swap memory usage,
and number of processes. Below that is a list of the processes sorted
by CPU utilization.</para>
<para>You can modify the output of <command>top</command> while
it is running. If you hit an <option>i</option>, top will no longer
display idle processes. Hit <option>i</option> again to see them
again. Hitting <option>M</option> will sort by memory usage,
<option>S</option> will sort by how long they processes have been
running, and <option>P</option> will sort by CPU usage again.</para>
<para>In addition to viewing options, you can also modify processes
from within the <command>top</command> command. You can use
<option>u</option> to view processes owned by a specific user,
<option>k</option> to kill processes, and <option>r</option> to
renice them.</para>
<para>For more in-depth information about processes you can look in
the <filename>/proc</filename> filesystem. In the <filename>/proc</filename>
filesystem you will find a series of sub-directories with numeric names.
These directories are associated with the processes ids of currently
running processes. In each directory you will find a series of files
containing information about the process.</para>
<para>YOU MUST TAKE EXTREME CAUTION TO NOT MODIFY THESE FILES, DOING
SO MAY CAUSE SYSTEM PROBLEMS!</para>
</sect2>
<sect2 id="iostat">
<title>The <command>iostat</command> command.</title>
<para>The <command>iostat</command> will display the current CPU load
average and disk I/O information. This is a great command to monitor
your disk I/O usage.
<screen>
<prompt>#</prompt> <userinput>iostat</userinput>
<computeroutput>Linux 2.4.20-24.9 (myhost) 12/23/2003
avg-cpu: %user %nice %sys %idle
62.09 0.32 2.97 34.62
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev3-0 2.22 15.20 47.16 1546846 4799520
</computeroutput>
</screen>
For 2.4 kernels the devices is names using the device's major
and minor number. In this case the device listed is <filename>
/dev/hda</filename>. To have <command>iostat</command> print this
out for you, use the <option>-x</option>.
<screen>
<prompt>#</prompt> <userinput>iostat -x</userinput>
<computeroutput>Linux 2.4.20-24.9 (myhost) 12/23/2003
avg-cpu: %user %nice %sys %idle
62.01 0.32 2.97 34.71
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/hdc 0.00 0.00 .00 0.00 0.00 0.00 0.00 0.00 0.00 2.35 0.00 0.00 14.71
/dev/hda 1.13 4.50 .81 1.39 15.18 47.14 7.59 23.57 28.24 1.99 63.76 70.48 15.56
/dev/hda1 1.08 3.98 .73 1.27 14.49 42.05 7.25 21.02 28.22 0.44 21.82 4.97 1.00
/dev/hda2 0.00 0.51 .07 0.12 0.55 5.07 0.27 2.54 30.35 0.97 52.67 61.73 2.99
/dev/hda3 0.05 0.01 .02 0.00 0.14 0.02 0.07 0.01 8.51 0.00 12.55 2.95 0.01
</computeroutput>
</screen>
<para>The <command>iostat</command> man page contains a detailed
explanation of what each of these columns mean.</para>
</sect2>
<sect2 id="ps">
<title>The <command>ps</command> command</title>
<para>The <command>ps</command> will provide you a list of
processes currently running. There is a wide variety of options
that this command gives you.</para>
<para>A common use would be to list all processes currently running.
To do this you would use the <command>ps -ef</command> command.
(Screen output from this command is too large to include, the following
is only a partial output.)
<screen>
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Dec22 ? 00:00:03 init
root 2 1 0 Dec22 ? 00:00:00 [keventd]
root 3 1 0 Dec22 ? 00:00:00 [kapmd]
root 4 1 0 Dec22 ? 00:00:00 [ksoftirqd_CPU0]
root 9 1 0 Dec22 ? 00:00:00 [bdflush]
root 5 1 0 Dec22 ? 00:00:00 [kswapd]
root 6 1 0 Dec22 ? 00:00:00 [kscand/DMA]
root 7 1 0 Dec22 ? 00:01:28 [kscand/Normal]
root 8 1 0 Dec22 ? 00:00:00 [kscand/HighMem]
root 10 1 0 Dec22 ? 00:00:00 [kupdated]
root 11 1 0 Dec22 ? 00:00:00 [mdrecoveryd]
root 15 1 0 Dec22 ? 00:00:01 [kjournald]
root 81 1 0 Dec22 ? 00:00:00 [khubd]
root 1188 1 0 Dec22 ? 00:00:00 [kjournald]
root 1675 1 0 Dec22 ? 00:00:00 syslogd -m 0
root 1679 1 0 Dec22 ? 00:00:00 klogd -x
rpc 1707 1 0 Dec22 ? 00:00:00 portmap
root 1813 1 0 Dec22 ? 00:00:00 /usr/sbin/sshd
ntp 1847 1 0 Dec22 ? 00:00:00 ntpd -U ntp
root 1930 1 0 Dec22 ? 00:00:00 rpc.rquotad
root 1934 1 0 Dec22 ? 00:00:00 [nfsd]
root 1942 1 0 Dec22 ? 00:00:00 [lockd]
root 1943 1 0 Dec22 ? 00:00:00 [rpciod]
root 1949 1 0 Dec22 ? 00:00:00 rpc.mountd
root 1961 1 0 Dec22 ? 00:00:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
root 2057 1 0 Dec22 ? 00:00:00 /usr/bin/spamd -d -c -a
root 2066 1 0 Dec22 ? 00:00:00 gpm -t ps/2 -m /dev/psaux
bin 2076 1 0 Dec22 ? 00:00:00 /usr/sbin/cannaserver -syslog -u bin
root 2087 1 0 Dec22 ? 00:00:00 crond
daemon 2195 1 0 Dec22 ? 00:00:00 /usr/sbin/atd
root 2215 1 0 Dec22 ? 00:00:11 /usr/sbin/rcd
weeksa 3414 3413 0 Dec22 pts/1 00:00:00 /bin/bash
weeksa 4342 3413 0 Dec22 pts/2 00:00:00 /bin/bash
weeksa 19121 18668 0 12:58 pts/2 00:00:00 ps -ef
</screen>
</para>
<para>The first column shows who owns the process. The second
column is the process ID. The Third column is the parent process
ID. This is the process that generated, or started, the process.
The forth column is the CPU usage (in
percent). The fifth column is the start time, of date if the process
has been running long enough. The sixth column is the tty associated
with the process, if applicable. The seventh column is the cumulitive
CPU usage (total amount of CPU time is has used while running). The
eighth column is the command itself.</para>
<para>With this information you can see exacly what is running on
your system and kill run-away processes, or those that are causing
problems.</para>
</sect2>
<sect2 id="vmstat">
<title>The <command>vmstat</command> command</title>
<para>The <command>vmstat</command> command will provide a report
showing statistics for system processes, memory, swap,
I/O, and the CPU. These statistics are generated using data from the
last time the command was run to the present. In the case of the
command never being run, the data will be from the last reboot until
the present.<para>
<screen>
<prompt>#</prompt> <userinput>vmstat</userinput>
<computeroutput>
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 181604 17000 26296 201120 0 2 8 24 149 9 61 3 36
</computeroutput>
</screen></para>
<para>The following was taken from the
<command>vmstat</command> man page.</para>
<blockquote><para><literallayout>
FIELD DESCRIPTIONS
Procs
r: The number of processes waiting for run time.
b: The number of processes in uninterruptable sleep.
w: The number of processes swapped out but otherwise runnable. This
field is calculated, but Linux never desperation swaps.
Memory
swpd: the amount of virtual memory used (kB).
free: the amount of idle memory (kB).
buff: the amount of memory used as buffers (kB).
Swap
si: Amount of memory swapped in from disk (kB/s).
so: Amount of memory swapped to disk (kB/s).
IO
bi: Blocks sent to a block device (blocks/s).
bo: Blocks received from a block device (blocks/s).
System
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.
CPU
These are percentages of total CPU time.
us: user time
sy: system time
id: idle time</literallayout>
</para></blockquote>
</sect2>
<sect2 id="lsof">
<title>The <command>lsof</command> command</title>
<para>The <command>lsof</command> command will print out a list of
every file that is in use. Since Linux considers everythihng a file,
this list can be very long. However, this command
can be useful in diagnosing problems. An example of this is if you wish
to unmount a filesystem, but you are being told that it is in use. You
could use this command and <command>grep</command> for the name of the
filesystem to see who is using it.</para>
<para>Or suppose you want to see all files in use by a particular process.
To do this you would use <command>lsof -p -processid-</command>.</para>
</sect2>
<sect2 id="more-utils">
<title>Finding More Utilities</title>
<para>To learn more about what command line tools are available, Chris
Karakas has wrote a reference guide titled <ulink
url="http://www.tldp.org/LDP/GNU-Linux-Tools-Summary/html/GNU-Linux-Tools-Summary.html"> GNU/Linux
Command-Line Tools Summary</ulink>. It's a good resource for learning
what tools are out there and how to do a number of tasks.</para>
</sect2>
<sect1 id="fs-usage">
<title>Filesystem Usage</title>
<para>Many reports are currently talking about how cheap storage has
gotten, but if you're like most of us it isn't cheap enough. Most of
us have a limited amount of space, and need to be able to monitor it
and control how it's used.</para>
<sect2 id="df">
<title>The df command</title>
<para>The <command>df</command> is the simplest tool available to
view disk usage. Simply type in <command>df</command> and you'll
be shown disk usage for all your mounted filesystems in 1K blocks
<screen>
<prompt>user@server:~> <prompt><userinput>df</userinput>
<computeroutput>
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda3 5242904 759692 4483212 15% /
tmpfs 127876 8 127868 1% /dev/shm
/dev/hda1 127351 33047 87729 28% /boot
/dev/hda9 10485816 33508 10452308 1% /home
/dev/hda8 5242904 932468 4310436 18% /srv
/dev/hda7 3145816 32964 3112852 2% /tmp
/dev/hda5 5160416 474336 4423928 10% /usr
/dev/hda6 3145816 412132 2733684 14% /var
</computeroutput>
</screen></para>
<para>You can also use the <command>-h</command> to see the output in
"human-readable" format. This will be in K, Megs, or Gigs depending
on the size of the filesystem. Alternately, you can also use the
<command>-B</command> to specify block size.</para>
<para>In addition to space usage, you could use the
<command>-i</command> option to view the number of used and available
inodes.
<screen>
<prompt>user@server:~> </prompt><userinput>df -i</userinput>
<computeroutput>
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/hda3 0 0 0 - /
tmpfs 31969 5 31964 1% /dev/shm
/dev/hda1 32912 47 32865 1% /boot
/dev/hda9 0 0 0 - /home
/dev/hda8 0 0 0 - /srv
/dev/hda7 0 0 0 - /tmp
/dev/hda5 656640 26651 629989 5% /usr
/dev/hda6 0 0 0 - /var
</computeroutput>
</screen></para>
</sect2>
<sect2 id="du">
<title>The du command</title>
<para>Now that you know how much space has been used on a filesystem
how can you find out where that data is? To view usage by a directory
or file you can use <command>du</command>. Unless you specify a
filename <command>du</command> will act recursively. For example:
<screen>
<prompt>user@server:~> </prompt><userinput>du file.txt</userinput>
<computeroutput>
1300 file.txt
</computeroutput>
</screen>
Or like the <command>df</command> I can use the <command>-h</command>
and get the same output in "human-readable" form.
<screen>
<prompt>user@server:~> </prompt><userinput>du -h file.txt</userinput>
<computeroutput>
1.3M file.txt
</computeroutput>
</screen>
</para>
<para>Unless you specify a filename <command>du</command> will act
recursively.
<screen>
<prompt>user@server:~> </prompt><userinput>du -h /usr/local</userinput>
<computeroutput>
4.0K /usr/local/games
16K /usr/local/include/nessus/net
180K /usr/local/include/nessus
208K /usr/local/include
62M /usr/local/lib/nessus/plugins/.desc
97M /usr/local/lib/nessus/plugins
164K /usr/local/lib/nessus/plugins_factory
97M /usr/local/lib/nessus
12K /usr/local/lib/pkgconfig
2.7M /usr/local/lib/ladspa
104M /usr/local/lib
112K /usr/local/man/man1
4.0K /usr/local/man/man2
4.0K /usr/local/man/man3
4.0K /usr/local/man/man4
16K /usr/local/man/man5
4.0K /usr/local/man/man
</computeroutput>
</screen></para>
<para>If you just want a summary of that directory you can use the
<command>-s</command> option.
<screen>
<prompt>user@server:~> </prompt><userinput>du -hs /usr/local</userinput>
<computeroutput>
210M /usr/local
</computeroutput>
</screen></para>
</sect2>
<sect2 id="quotas">
<title>Quotas</title>
<para>For more information about quotas you can read
<ulink url="http://www.tldp.org/HOWTO/Quota.html">The Quota HOWTO
</ulink>.
</sect2>
</sect1>
<sect1 id="monitoring-users">
<title>Monitoring Users</title>
<blockquote><para>Just because you're paranoid doesn't mean they
AREN'T out to get you... Source Unknown</para></blockquote>
<para>From time to time there are going to be occasions where you will
want to know exactly what people are doing on your system. Maybe you
notice that a lot of RAM is being used, or a lot of CPU activity.
You are going to want to see who is on the system, what they are
running, and what kind of resources they are using.</para>
<sect2 id="who">
<title>The who command</title>
<para>The easiest way to see who is on the system is to do a
<command>who</command> or <command>w</command>. The -->
<command>who</command> is a simple tool that lists out who is logged -->
on the system and what port or terminal they are logged on at.
<screen>
<prompt>user@server:~></prompt> <userinput> who</userinput>
<computeroutput>
bjones pts/0 May 23 09:33
wally pts/3 May 20 11:35
aweeks pts/1 May 22 11:03
aweeks pts/2 May 23 15:04
</computeroutput>
</screen></para>
</sect2>
<sect2 id="ps-u">
<title>The ps command -again!</title>
<para>In the previous section we can see that user aweeks is logged
onto both <filename>pts/1</filename> and <filename>pts/2</filename>,
but what if we want to see what they are doing? We could to a
<command>ps -u aweeks</command> and get the following output
<screen>
<prompt>user@server:~> </prompt><userinput>ps -u aweeks</userinput>
<computeroutput>
20876 pts/1 00:00:00 bash
20904 pts/2 00:00:00 bash
20951 pts/2 00:00:00 ssh
21012 pts/1 00:00:00 ps
</computeroutput>
</screen>
From this we can see that the user is doing a <command>ps</command>
<command>ssh</command>.</para>
<para>This is a much more consolidated use of the
<command>ps</command> than discussed previously.<para>
</sect2>
<sect2 id="w">
<title>The w command</title>
<para>Even easier than using the <command>who</command> and
<command>ps -u</command> commands is to use the <command>w</command>.
<command>w</command> will print out not only who is on the system,
but also the commands they are running.
<screen>
<prompt>user@server:~> </prompt><userinput>w</userinput>
<computeroutput>
aweeks :0 09:32 ?xdm? 30:09 0.02s -:0
aweeks pts/0 09:33 5:49m 0.00s 0.82s kdeinit: kded
aweeks pts/2 09:35 8.00s 0.55s 0.36s vi sag-0.9.sgml
aweeks pts/1 15:03 59.00s 0.03s 0.03s /bin/bash
</computeroutput>
</screen></para>
<para>From this we can see that I have a <command>kde</command> session
running, I'm working in this document :-), and have another terminal
open sitting idle at a bash prompt.</para>
</sect2>
<sect2 id="skill">
<title>The skill command</title>
<para> To Be Added</para>
</sect2>
<sect2 id="nice">
<title>nice and renice</title>
<para>To Be Added</para>
</sect2>
</sect1>
</chapter>