749 lines
27 KiB
HTML
749 lines
27 KiB
HTML
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
|
|
<HTML>
|
|
<HEAD>
|
|
<TITLE>Big Brother Unix Network Monitoring System</TITLE>
|
|
<META NAME="GENERATOR"
|
|
CONTENT="Mozilla/3.01Gold (X11; I; Linux 2.0.29 i586) [Netscape]">
|
|
</HEAD>
|
|
<BODY TEXT="#FFFFFF" BGCOLOR="#000000" LINK="#00FFAA" VLINK="#FFFF44"
|
|
BACKGROUND="./gx/sittler/bkg-purple.gif">
|
|
|
|
<H4>
|
|
"Linux Gazette...<I>making Linux just a little more fun!</I>"
|
|
</H4>
|
|
|
|
<P> <HR> <P>
|
|
<!--===================================================================-->
|
|
|
|
<P><IMG SRC="./gx/sittler/bb.gif" HEIGHT=166 WIDTH=176 ALIGN=RIGHT> </P>
|
|
|
|
<H1>Big Brother Network Monitoring System</H1>
|
|
|
|
<H3>A Web-based Unix Network Monitoring <BR>
|
|
and Notification System</H3>
|
|
|
|
<H4>By Paul M. Sittler,
|
|
<a href="mailto:p-sittler@tamu.edu">p-sittler@tamu.edu</a></H4>
|
|
|
|
<P> <HR> <P>
|
|
|
|
<P>Big Brother is Watching. . . </P>
|
|
|
|
<P>I wasn't bored: I don't have time to be bored. Texas
|
|
Agricultural Extension Service operates a fairly large
|
|
enterprise-wide network that stretches across hell's half acre,
|
|
otherwise known as Texas. We have around 3,000 users in 249
|
|
counties and 12 district offices who expect to get their e-mail
|
|
and files across our Wide Area Network. Some users actually
|
|
expect the network to work most of the time. We use ethernet
|
|
networking with Novell servers at some 35 locations, 15 or so
|
|
whose routers are connected via a mixture of 56Kb circuits,
|
|
fractional T1, Frame-Relay, and radio links. We are not
|
|
currently using barbed wire fences for our network, regardless
|
|
of what you may have heard. . . </P>
|
|
|
|
<P>I am privileged to be part of the team that set up that
|
|
network and tries to keep it going. We do not live in a perfect
|
|
network world. Things happen. Scarcely a day goes by when we do
|
|
not have one or more WAN link outages, usually of short
|
|
duration. We sometimes have our hands full trying to keep all
|
|
the pieces connected. Did I mention that the users expect the
|
|
mail and other software to actually work? </P>
|
|
|
|
<P>Cruising the USENET newsgroups, I read a posting about
|
|
"Big Brother, a solution to the problem of Unix Systems
|
|
Monitoring" written by Sean MacGuire of Montreal, Canada. I
|
|
was intrigued to notice that Big Brother was a collection of
|
|
shell scripts and simple c programs designed to monitor a bunch
|
|
of Unix machines on a network. So what if most of our mission
|
|
critical servers were Novell-based? Who cares if some of our web
|
|
servers run on Macintosh, OS/2, Win'95 or NT? We use both Linux and
|
|
various flavours of Unix in a surprisingly large number of
|
|
places. </P>
|
|
|
|
<P>We had cooked up a number of homemade monitoring systems.
|
|
Pinging and tracerouting to all the servers can be very
|
|
informative. We looked at a bunch of proprietary (and expensive)
|
|
network monitoring systems. It is amazing how much money these
|
|
things can cost. System adminstrators often reported difficult
|
|
installations and software incompatibilities with the monitoring
|
|
software. Thus, frustrated users often gave us our first hint
|
|
that all was not well. </P>
|
|
|
|
<P>According to the blurb on Big Brother: </P>
|
|
|
|
<BLOCKQUOTE>
|
|
<P>"Big Brother is a loosely-coupled distributed set of tools for
|
|
monitoring and displaying the current status of an entire Unix
|
|
network and notifying the admin should need be. It came about as
|
|
the result of automating the day to day tasks encountered while
|
|
actively administering Unix systems."</P>
|
|
</BLOCKQUOTE>
|
|
|
|
<P>The USENET news article provided a URL
|
|
<a href="http://www.iti.qc.ca/iti/users/sean/bb-dnld/">
|
|
("http://www.iti.qc.ca/iti/users/sean/bb-dnld/")</a> to
|
|
the home site of Big Brother. I pointed my browser to it and was
|
|
rewarded with a purple-sided screen background and a blue image
|
|
of a sinister face peering out under the caption "big
|
|
brother is watching." After my initial shock, I learned
|
|
that Big Brother featured: </P>
|
|
|
|
<TABLE WIDTH=100% CELLPADDING=10>
|
|
<TR><TD ALIGN=MIDDLE> <H2><I>f e a t u r e s</I></H2></TD>
|
|
<TD><H4>
|
|
<IMG SRC="./gx/sittler/green.gif" ALT="*"> Web-based status display<BR>
|
|
<IMG SRC="./gx/sittler/green.gif" ALT="*"> Configurable warning and panic levels<BR>
|
|
<IMG SRC="./gx/sittler/green.gif" ALT="*"> Notification via Pager or email<BR>
|
|
<IMG SRC="./gx/sittler/green.gif" ALT="*"> Free and includes Source Code<BR>
|
|
</H4></TD></TR>
|
|
</TABLE>
|
|
|
|
<P>I was fascinated. Especially by the last item, that said it
|
|
was free with source code. (I often tell people that Linux isn't
|
|
free, but priceless. . .) So what could a priceless package do
|
|
for me? What on earth did Big Brother check? </P>
|
|
|
|
<TABLE WIDTH=100% CELLPADDING=10>
|
|
<TR><TD ALIGN=MIDDLE> <H2><I>m o n i t o r s</I></H2></TD>
|
|
<TD><H4>
|
|
<IMG SRC="./gx/sittler/yellow.gif" ALT="*"> connectivity via ping<BR>
|
|
<IMG SRC="./gx/sittler/yellow.gif" ALT="*"> http servers up and running<BR>
|
|
<IMG SRC="./gx/sittler/yellow.gif" ALT="*"> disk space usage<BR>
|
|
<IMG SRC="./gx/sittler/yellow.gif" ALT="*"> uptime and cpu usage<BR>
|
|
<IMG SRC="./gx/sittler/yellow.gif" ALT="*"> essential processes are still running<BR>
|
|
<IMG SRC="./gx/sittler/yellow.gif" ALT="*"> system-generated messages and warnings<BR>
|
|
</H4></TD></TR>
|
|
</TABLE>
|
|
|
|
<P>Overall, very sensible. Looking for some "gotchas,"
|
|
I found that I would need a Unix-based machine, and:</P>
|
|
|
|
<TABLE WIDTH=100% CELLPADDING=10>
|
|
<TR><TD ALIGN=MIDDLE>
|
|
<H2><I>y o u ' l l<BR>
|
|
n e e d</I></H2>
|
|
</TD>
|
|
<TD><H4>
|
|
<IMG SRC="./gx/sittler/purple.gif" ALT="*"> A Functioning Web server & Browser -
|
|
for the display<BR>
|
|
<IMG SRC="./gx/sittler/purple.gif" ALT="*"> C compiler<BR>
|
|
<IMG SRC="./gx/sittler/yellow.gif" ALT="*"> Kermit and a modem line - for the pager<BR>
|
|
</H4></TD></TR>
|
|
</TABLE>
|
|
|
|
<P>A web server was no problem, as we run many. A c compiler
|
|
came with Linux, and we use kermit on many machines with modems.
|
|
So far, so good. </P>
|
|
|
|
<P>The web site provided links to a few demonstration sites, and
|
|
a link to download it as well. I connected to a demonstration
|
|
site and was greeted with an amazing display: </P>
|
|
|
|
<CENTER><TABLE CELLPADDING=5 >
|
|
<TR>
|
|
<TD VALIGN="TOP">
|
|
<H3>Legend</H3>
|
|
|
|
<P><IMG SRC="./gx/sittler/green.gif" ALT="green" HEIGHT=12 WIDTH=12><B><I> System OK<BR>
|
|
<IMG SRC="./gx/sittler/yellow.gif" ALT="yellow" HEIGHT=12 WIDTH=12> Attention<BR>
|
|
<IMG SRC="./gx/sittler/red.gif" ALT="red" HEIGHT=12 WIDTH=12> Trouble<BR>
|
|
<IMG SRC="./gx/sittler/purple.gif" ALT="blue" HEIGHT=12 WIDTH=12> No report</I></B><BR>
|
|
<BR>
|
|
</P>
|
|
|
|
<H3>Updated<BR>
|
|
<I>@ 22:52</I></H3>
|
|
</TD>
|
|
|
|
<TD><A HREF="http://www.iti.qc.ca/iti/users/sean/bb-dnld/">
|
|
<IMG SRC="./gx/sittler/bb.gif" ALT="BIG BROTHER" BORDER=0 HEIGHT=166 WIDTH=176>
|
|
</A><BR>
|
|
<BR>
|
|
</TD>
|
|
|
|
<TD ALIGN="RIGHT" VALIGN="ABSMIDDLE"><A HREF="bb-help.html"><IMG SRC="./gx/sittler/buthelp.gif" ALT="help" VSPACE=7 BORDER=0 HEIGHT=29 WIDTH=74></A><BR>
|
|
<A HREF="bb-info.html"><IMG SRC="./gx/sittler/butinfo.gif" ALT="info" VSPACE=7 BORDER=0 HEIGHT=29 WIDTH=74></A><BR>
|
|
<A HREF="bb-page.html"><IMG SRC="./gx/sittler/butpage.gif" ALT="page" VSPACE=7 BORDER=0 HEIGHT=29 WIDTH=74></A><BR>
|
|
<A HREF="bb2.html"><IMG SRC="./gx/sittler/butview.gif" ALT="view" VSPACE=7 BORDER=0 HEIGHT=29 WIDTH=74></A>
|
|
</TD>
|
|
</TR>
|
|
</TABLE></CENTER>
|
|
|
|
<CENTER><TABLE BORDER=3>
|
|
<TR>
|
|
<TD></TD>
|
|
|
|
<TD ALIGN="CENTER" WIDTH="45">
|
|
<H4><I>conn</I></H4>
|
|
</TD>
|
|
|
|
<TD ALIGN="CENTER" WIDTH="45">
|
|
<H4><I>cpu</I></H4>
|
|
</TD>
|
|
|
|
<TD ALIGN="CENTER" WIDTH="45">
|
|
<H4><I>disk</I></H4>
|
|
</TD>
|
|
|
|
<TD ALIGN="CENTER" WIDTH="45">
|
|
<H4><I>http</I></H4>
|
|
</TD>
|
|
|
|
<TD ALIGN="CENTER" WIDTH="45">
|
|
<H4><I>msgs</I></H4>
|
|
</TD>
|
|
|
|
<TD ALIGN="CENTER" WIDTH="45">
|
|
<H4><I>procs</I></H4>
|
|
</TD>
|
|
</TR>
|
|
|
|
<TR>
|
|
<TD ALIGN="CENTER"><B><A HREF="iti-s01">iti-s01</A></B></TD>
|
|
|
|
<TD ALIGN="CENTER"><A HREF="iti-s01.conn"><IMG SRC="./gx/sittler/green.gif" ALT="green" BORDER=0 HEIGHT=12 WIDTH=12></A></TD>
|
|
|
|
<TD ALIGN="CENTER"><A HREF="iti-s01.cpu"><IMG SRC="./gx/sittler/green.gif" ALT="green" BORDER=0 HEIGHT=12 WIDTH=12></A></TD>
|
|
|
|
<TD ALIGN="CENTER"><A HREF="iti-s01.disk"><IMG SRC="./gx/sittler/green.gif" ALT="green" BORDER=0 HEIGHT=12 WIDTH=12></A></TD>
|
|
|
|
<TD ALIGN="CENTER"><A HREF="iti-s01.http"><IMG SRC="./gx/sittler/green.gif" ALT="green" BORDER=0 HEIGHT=12 WIDTH=12></A></TD>
|
|
|
|
<TD ALIGN="CENTER"><A HREF="iti-s01.msgs"><IMG SRC="./gx/sittler/yellow.gif" ALT="yellow" BORDER=0 HEIGHT=12 WIDTH=12></A></TD>
|
|
|
|
<TD ALIGN="CENTER"><A HREF="iti-s01.procs"><IMG SRC="./gx/sittler/green.gif" ALT="green" BORDER=0 HEIGHT=12 WIDTH=12></A></TD>
|
|
</TR>
|
|
|
|
<TR>
|
|
<TD ALIGN="CENTER"><B>router-000</B></TD>
|
|
|
|
<TD ALIGN="CENTER"><A HREF="router-000.conn"><IMG SRC="./gx/sittler/green.gif" ALT="green" BORDER=0 HEIGHT=12 WIDTH=12></A></TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
</TR>
|
|
|
|
<TR>
|
|
<TD ALIGN="CENTER"><B>inet-gw-0</B></TD>
|
|
|
|
<TD ALIGN="CENTER"><A HREF="inet-gw-0.conn"><IMG SRC="./gx/sittler/green.gif" ALT="green" BORDER=0 HEIGHT=12 WIDTH=12></A></TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
|
|
<TD ALIGN="CENTER">-</TD>
|
|
</TR></TABLE></CENTER>
|
|
|
|
<P>Big Brother is watching! As I endured the scrutiny of the
|
|
Orwellian face peering out at me, I examined the rest of the
|
|
display. The display was coded like a traffic signal
|
|
(green/yellow/red), and the update time was clearly displayed
|
|
beneath it. To the right of "Big Brother" were four
|
|
buttons, marked clearly "Help," "Info,"
|
|
"Page" and "View." Beneath the header area
|
|
was a table with six column headings and three rows, each neatly
|
|
labelled with a computer hostname. The boxes formed by the
|
|
intersection of the rows and columns contained attractive green
|
|
and yellow balls. The overall effect was like a decorated tree.
|
|
The left side of the screen had a yellow tint, gradually
|
|
becoming black at the center.</P>
|
|
|
|
<P>I selected the "Help" button and was rewarded with
|
|
a brief explanation of what Big Brother was all about. Choosing
|
|
the "Info" Button provided a much longer and more
|
|
detailed explanation of the system, including a graphic that
|
|
really was worth a thousand words. I tried the "Page"
|
|
button to discover that this was a way to send a signal to a
|
|
radio-linked pager. Not at all what I had expected! Finally, the
|
|
"View" selection provided a briefer but perhaps more
|
|
useful view of the information, isolating only the systems with
|
|
problems.</P>
|
|
|
|
<P>In this case, only the "iti-s01" system was
|
|
displayed. My browser cursor indicated a link as it passed over
|
|
each colored dot, so I clicked on the blinking yellow dot and
|
|
received a message that read:</P>
|
|
|
|
<BLOCKQUOTE>
|
|
"yellow Tue Feb 18 22:50:53 EST 1997 Feb 16 12:22:33
|
|
iti-s01 kernel: WARNING: / was not properly dismounted"
|
|
</BLOCKQUOTE>
|
|
|
|
<P>This puzzled me at first. How on earth could it know that? It
|
|
seems that BB (Big Brother) checks the system /var/log/messages
|
|
file periodically and alerts on any line that says either
|
|
"WARNING" or "NOTICE." As I am certain that
|
|
Sean MacGuire is very conscientious, I suspect that he adds that
|
|
line to his message file so that something will <I>appear</I> to be
|
|
wrong.</P>
|
|
|
|
<P>Suddenly, my screen spontaneously updated! The update time
|
|
had changed by five minutes, and a blinking yellow dot appeared
|
|
under the column labelled "procs." I clicked on the
|
|
blinking yellow dot and was informed that the sendmail process
|
|
was not running. This got me really interested! Apparently, Big
|
|
Brother could monitor whether selected processes were
|
|
running!</P>
|
|
|
|
<P>I was also a little puzzled about the screen being updated on
|
|
its own. I used my browser to view the document source and
|
|
discovered some html commands that were new to me:</P>
|
|
|
|
<PRE>
|
|
<META HTTP-EQUIV="REFRESH" CONTENT="120">
|
|
<META HTTP-EQUIV="EXPIRES" CONTENT="Tue Feb 18 23:22:07 CST 1997">
|
|
</PRE>
|
|
|
|
<P>
|
|
The first line instructs browsers to get an update every 120
|
|
seconds. The second line tells the browser that it should get a
|
|
new copy after the expiration time and date. Very clever!</P>
|
|
|
|
<P>I returned to the graphics window and discovered that the
|
|
yellow area on the left had changed to red! A new hostname row
|
|
appeared with a blinking red dot under the column labelled
|
|
"conn." I clicked on the blinking red dot and read a
|
|
message that said:</P>
|
|
|
|
<BLOCKQUOTE>
|
|
"red Tue Feb 18 22:59:11 CST 1997 bb-network.sh: Can't
|
|
connect to router-000... (paging)"
|
|
</BLOCKQUOTE>
|
|
|
|
<P>The connection to the machine called router-000 had been
|
|
interrupted and the administrator had been paged. Amazingly,
|
|
while in Texas, I had become aware of a network outage in
|
|
Montreal, Canada. This really had possibilities. Perhaps I
|
|
might someday be able to take a vacation!</P>
|
|
|
|
<H2>Big Brother Installation</H2>
|
|
|
|
<P>I was so impressed with Big Brother that I decided to try to
|
|
use it. Sean has thoughtfully made its acquisition easy, but
|
|
requests that you fill out an on-line registration form with
|
|
your name and e-mail address. He would also like to know where
|
|
you heard about Big Brother. I filled these out in early
|
|
November 1996, and received an e-mail survey form in late
|
|
December.</P>
|
|
|
|
<TABLE WIDTH=100% CELLPADDING=10>
|
|
<TR><TD ALIGN=MIDDLE WIDTH=33%>
|
|
<A HREF="http://www.iti.qc.ca/iti/users/sean/bb-dnld/bb-dnld.html">
|
|
<H2><I>d o w n l o a d</I></H2></A></TD>
|
|
<TD><H4>
|
|
Click the link at left to download Big Brother and to get technical
|
|
information about how the system works, and how to install and
|
|
configure the package.
|
|
</H4></TD></TR></TABLE>
|
|
|
|
<P>When I clicked on the link to download Big Brother, I ended
|
|
up with a file called "bb-src.tgz." I impetuously
|
|
gunzipped this to get "bb-src.tar." I then thought
|
|
better of the impending error of my ways and decided to download
|
|
and print the installation instructions.</P>
|
|
|
|
<TABLE WIDTH=100% CELLPADDING=10>
|
|
<TR><TD ALIGN=MIDDLE WIDTH=33%>
|
|
<A HREF="http://www.iti.qc.ca/iti/users/sean/bb-dnld/bb-install.html">
|
|
<H2><I>i n s t a l l</I></H2></A></TD>
|
|
<TD><H4>
|
|
Click the link at left to look at the install procedure for Big
|
|
Brother. More information about how to set the system up lives
|
|
here.
|
|
</H4></TD></TR></TABLE>
|
|
|
|
<P>Just in case, I also grabbed and printed the debugging
|
|
information so thoughtfully provided (as it turned out, I did
|
|
not need it):</P>
|
|
|
|
<TABLE WIDTH=100% CELLPADDING=10>
|
|
<TR><TD ALIGN=MIDDLE WIDTH=33%>
|
|
<A HREF="http://www.iti.qc.ca/iti/users/sean/bb-dnld/bb-debug.html">
|
|
<H2><I>d e b u g</I></H2></A></TD>
|
|
<TD><H4>
|
|
The link at left provides debugging information for different
|
|
problems that may be experienced during the Big Brother
|
|
installation process.
|
|
</H4></TD></TR></TABLE>
|
|
|
|
<P>I had no real problems following the installation
|
|
instructions. I decided to make the $BBHOME directory
|
|
"/usr/src/bb"; use whatever makes sense to you. The
|
|
automatic configuration routines are said to work for AIX,
|
|
FreeBSD, HPUX 10, Irix, Linux, NetBSD, OSF, RedHat Linux, SCO,
|
|
SCO 3/5, Solaris, SunOS4.1, and UnixWare. I can vouch for Linux,
|
|
RedHat Linux, Solaris, and SunOS 4.1. </P>
|
|
|
|
<P>The c programs compiled without incident, and the
|
|
installation went smoothly. As always, your mileage may vary. In
|
|
less than an hour, I was looking at Big Brother's display of
|
|
coloured lights!</P>
|
|
|
|
<P>At this point, you may wish to re-examine the documentation
|
|
and information files. Personalize your installation as desired.
|
|
Above all, have fun!
|
|
</P>
|
|
|
|
<H2>Hacking</H2>
|
|
|
|
<P>I admit it. I am a closet hacker. I saw many things about
|
|
the stock BB distribution that I wanted to improve. Big
|
|
Brother's modular and elegantly simple construction makes it a
|
|
joy to modify as desired. The shell scripts are portable,
|
|
simple, well documented, and easy to understand. The use of the
|
|
modified hosts file to determine which hosts to monitor was
|
|
gratifyingly familiar. The "bbclient" script made it
|
|
extremely easy to move the required components to another
|
|
similar Unix host. Sean has done a remarkable job in making
|
|
this package easy to install!</P>
|
|
|
|
<P>I got obsessive-compulsive about hacking BB and modified it
|
|
slightly, working from Sean MacGuire's v1.03 distribution as a
|
|
base. I forwarded my changes to him for possible inclusion in a
|
|
later distribution.</P>
|
|
|
|
|
|
<P>Features that I added to BB proper include (code added is
|
|
<B>bold</B>):</P>
|
|
|
|
<UL>
|
|
<LI>Links to the info files in the brief view (bb2.html).
|
|
That's when I <I>need</I> them the most.
|
|
|
|
<P>
|
|
|
|
<LI>Links to html info files for each column heading and the
|
|
column info files themselves. These are placed in the
|
|
html directory along with bb.html and bb2.html and have
|
|
boring names like conn.html, cpu.html, . . . smtp.html.
|
|
|
|
<P>
|
|
|
|
<LI>Checks to see if ftp servers, pop3 post offices, and SMTP
|
|
Mail Transfer Agents (MTA's) are accessible
|
|
($BBHOME/bin/bb-network.sh). These all simply use bbnet to
|
|
telnet to the respective ports. This followed Sean's
|
|
style of adding comments to the bb-hosts file as follows:
|
|
|
|
<PRE>
|
|
128.194.44.99 behemoth.tamu.edu # BBPAGER <B>smtp ftp pop3</B>
|
|
165.91.132.4 bryan-ctr.tamu.edu # <B>pop3 smtp</B>
|
|
128.194.147.128 csdl.tamu.edu # http://csdl.tamu.edu/ <B>ftp smtp</B>
|
|
</PRE>
|
|
|
|
<P>
|
|
|
|
<LI>I added some environment variables to $BBHOME/etc/bbdef.sh
|
|
for the added monitoring as follows:
|
|
|
|
<PRE>
|
|
#
|
|
# WARNING AND PANIC LEVELS FOR DIFFERENT THINGS
|
|
# SEASON TO TASTE
|
|
#
|
|
DFPAGE=Y # PAGE ON DISK FULL (Y/N)
|
|
CPUPAGE=Y # PAGE FOR CPU Y/N
|
|
TELNETPAGE=Y # PAGE ON TELNET FAILURE?
|
|
HTTPPAGE=Y # PAGE ON HTTP FAILURE?
|
|
<B>FTPPAGE=Y # PAGE ON FTPD FAILURE?
|
|
POP3PAGE=Y # PAGE ON POP3 PO FAILURE?
|
|
SMTPPAGE=Y # PAGE ON SMTP MTA FAILURE?</B>
|
|
export DFPAGE CPUPAGE TELNETPAGE HTTPPAGE FTPPAGE POP3PAGE SMTPPAGE
|
|
</PRE>
|
|
|
|
<P>
|
|
|
|
<LI>I updated the bb-info.html and bb-help.html pages to
|
|
reflect a version of 1.03a and a date of 10 February 1997.
|
|
I also modified them to add brief mention of the new ftp,
|
|
pop3, and smtp monitoring things. Specifically, I changed
|
|
the bb-help.html file to add new pager codes for them as
|
|
follows:
|
|
|
|
<PRE>
|
|
100 - Disk Error. Disk is over 95% full...
|
|
200 - CPU Error. CPU load average is unacceptably high.
|
|
300 - Process Error. An important process has died.
|
|
400 - Message file contains a serious error.
|
|
500 - Network error, can't connect to that IP address.
|
|
600 - Web server HTTP error - server is down.
|
|
<B>610 - Ftp server error - server is down.
|
|
620 - POP3 server error - PopMail Post Office is down.
|
|
630 - SMTP MTA error - SMTP Mail Host is down.</B>
|
|
911 - User Page. Message is phone number to call back.
|
|
</PRE>
|
|
|
|
<P>
|
|
|
|
<LI>I added sections to the bb-info.html file to explain the
|
|
added ftp, pop3, and smtp monitoring.
|
|
|
|
<P>
|
|
|
|
<LI>I use a standard tagline file on each html page that
|
|
identifies the author and location of the page. Thus,
|
|
mkbb.sh and mkbb2.sh now look for an optional tagline file
|
|
to incorporate into the html documents that they generate.
|
|
The optional files are named mkbb.tag (for mkbb.sh) and
|
|
mkbb2.tag (for mkbb2.sh). The shell scripts look for the
|
|
optional tagline files in the $BBHOME/web directory (which
|
|
is where the mkbb.sh and mkbb2.sh files reside).
|
|
|
|
<P>
|
|
|
|
<LI>I went through ALL of the html-generating scripts and
|
|
html files to ensure that they actually had <HEAD>
|
|
sections and properly placed double quotes around the
|
|
various arguments.
|
|
|
|
<P>
|
|
|
|
<LI>For the most part, I edited the files so that everything
|
|
would fit on an 80-column screen.
|
|
|
|
<P>
|
|
|
|
<LI> I modified $BBHOME/etc/bbsys.sh to make it easier to
|
|
ignore certain disk volumes as follows:
|
|
|
|
<PRE>
|
|
#
|
|
# DISK INFORMATION
|
|
#
|
|
DFSORT="4" # % COLUMN - 1
|
|
DFUSE="^/dev" # PATTERN FOR LINES TO INCLUDE
|
|
DFEXCLUDE="<B>-E dos|cdrom</B>" # PATTERN FOR LINES TO EXCLUDE
|
|
</PRE>
|
|
|
|
<P>
|
|
|
|
<LI> I modified $BBHOME/etc/bbsys.linux so that the ping
|
|
program is properly found as follows:
|
|
|
|
<PRE>
|
|
#
|
|
# bbsys.linux
|
|
#
|
|
# BIG BROTHER
|
|
# OPERATING SYSTEM DEPENDENT THINGS THAT ARE NEEDED
|
|
#
|
|
<B>PING="/bin/ping" # LINUX CONNECTIVITY TEST</B>
|
|
PS="/bin/ps -ax" # LINUX
|
|
DF="/bin/df -k"
|
|
MSGFILE="/var/adm/messages"
|
|
TOUCH="/bin/touch" # SPECIAL TO LINUX
|
|
</PRE>
|
|
|
|
<P>
|
|
|
|
<LI>I added the ability to dynamically traceroute and ping
|
|
each system being monitored. I spoke with Sean about it,
|
|
and, in keeping with the KISS (Keep It Simple, Stupid)
|
|
principle, we thought these features were best added in
|
|
the info files. The user portion is pretty obvious in the
|
|
source to the info file. The cgi scripts are very simple
|
|
shell scripts included below:
|
|
|
|
</UL>
|
|
|
|
<PRE>
|
|
# traceroute.cgi ===========================================
|
|
#!/bin/sh
|
|
|
|
TRACEROUTE=/usr/bin/traceroute
|
|
|
|
echo Content-type: text/html
|
|
echo
|
|
|
|
if [ -x $TRACEROUTE ]; then
|
|
if [ $# = 0 ]; then
|
|
cat << EOM
|
|
<TITLE>TraceRoute Gateway</TITLE>
|
|
<H1>TraceRoute Gateway</H1>
|
|
|
|
<ISINDEX>
|
|
|
|
This is a gateway to "traceroute." Type the desired hostname
|
|
(like hostname.domain.name, eg. net.tamu.edu) in your
|
|
browser's search dialog, and enter a return.<P>
|
|
|
|
EOM
|
|
else
|
|
echo \<PRE\>
|
|
$TRACEROUTE $*
|
|
fi
|
|
else
|
|
echo Cannot find traceroute on this system.
|
|
fi
|
|
# traceroute.cgi ===========================================
|
|
|
|
|
|
# ping.cgi ===========================================
|
|
#!/bin/sh
|
|
|
|
PING=/bin/ping
|
|
|
|
echo Content-type: text/html
|
|
echo
|
|
|
|
if [ -x $PING ]; then
|
|
if [ $# = 0 ]; then
|
|
cat << EOM
|
|
<TITLE>TraceRoute Gateway</TITLE>
|
|
<H1>TraceRoute Gateway</H1>
|
|
|
|
<ISINDEX>
|
|
|
|
This is a gateway to "ping." Type the desired hostname
|
|
(like hostname.domain.name, eg. "net.tamu.edu") in your
|
|
browser's search dialog, and enter a return.<P>
|
|
|
|
EOM
|
|
else
|
|
echo \<PRE\>
|
|
$PING -c5 $*
|
|
fi
|
|
else
|
|
echo Cannot find ping on this system.
|
|
fi
|
|
|
|
# ping.cgi ===========================================
|
|
</PRE>
|
|
|
|
|
|
<H2>Future Enhancements of Big Brother</H2>
|
|
|
|
<P>Sean MacGuire is the primary author of Big Brother. In the
|
|
finest InterNet tradition of decentralized shared software
|
|
development, Sean solicits improvements, suggestions, and
|
|
enhancements from all. He then skillfully incorporates them as
|
|
appropriate into the Big Brother distribution. Thus, like
|
|
Linux, Big Brother is in a dynamic state of positive evolution
|
|
with contributions from a cast of thousands (at least dozens).
|
|
This constrained anarchy can produce interesting results with an
|
|
international flavour.</P>
|
|
|
|
<P>Jacob Lundqvist of Sweden is actively improving the paging
|
|
interface. He has done a superb job of enhancing the paging
|
|
portion, adding support for alphanumeric and SMS pagers. Darren
|
|
Henderson (Maine, US) added AIX support. David Brandon (Texas,
|
|
US) added proper IRIX support, and Jeff Matson (Minnesota, US)
|
|
made some IRIX fixes. Richard Dansereau (Canada) ported Big
|
|
Brother to SCO3 and provided support for other df's. Doug White
|
|
(Oregon, US) made some paging script bug fixes. Ron Nelson
|
|
(Minnesota, US) adapted BB to RedHat Linux. Jac Kersing
|
|
(Netherlands) made some security enhancements to bbd.c. Alan Cox
|
|
(Wales) suggested some shell script security modifications.
|
|
Douwe Dijkstra (Netherlands) provided SCO 5 support. Erik
|
|
Johannessen (Minnesota, US) survived SunOS 4.1.4 installation.
|
|
Curtis Olson (Minnesota, US) survived IRIX, Linux, and SunOS
|
|
installations. Gunnar Helliesen (Norway) ported Big Brother to
|
|
Ultrix, OSF, and NetBSD. Josh Wilmes (Missouri, US) added
|
|
Solaris changes for new ping stuff.</P>
|
|
|
|
<P>Many other unsung heros around the world are undoubtedly
|
|
working to enhance BB at this very moment.</P>
|
|
|
|
<P>I am (ab)using Big Brother in ways not originally envisioned
|
|
by its creator, Sean MacGuire. Texas Agricultural Extension's
|
|
networks are wildly heterogeneous mixtures of different
|
|
operating systems and protocols, rather than a homogeneous
|
|
Unix-based network. I would like to see Big Brother learn about
|
|
IPX/SPX protocols for Novell connectivity monitoring. I would
|
|
also like to see Big Brother data collection modules for
|
|
Macintosh, Novell, OS/2, Windows 3.1x, Windows'95, and Windows
|
|
NT. Rewriting Big Brother into perl might better serve these
|
|
disparate platforms. If I could only find the time!</P>
|
|
|
|
|
|
<H2>Big Brother's Impact at Texas Agricultural Extension Service</H2>
|
|
|
|
<P>We are now monitoring around 122 hosts. Only 20 are actually
|
|
Unix-based hosts that run Big Brother's bb program internally.
|
|
Some 28 are Novell servers, 39 are routers, and the rest are a
|
|
mixture of Macintosh, OS/2, Windows 3.1x, Windows'95, and
|
|
Windows NT machines running one or more types of servers (34 ftp
|
|
or 26 http). We also find it useful to monitor our 31 popmail
|
|
post offices and 43 mail hosts and gateways. We are checking
|
|
connectivity on three DNS servers as well, as they are mission
|
|
critical.</P>
|
|
|
|
<P>Big Brother (or, as I now affectionately refer to it, "Big
|
|
Bother") is now alerting us to outages five or more times daily.
|
|
Typically, the system administrator receives a page. BB's
|
|
display is checked and the info file is used to traceroute and
|
|
ping the offending machine to validate the outage. Many
|
|
connection outages involve routers, DSU/CSUs and multiplexors as
|
|
well as the actual host. BB's display allows us to quickly see
|
|
a pattern that aids in diagnosis. The ability to dynamically
|
|
traceroute and ping the host from the html info page also helps
|
|
to rapidly determine the actual point of failure. If the
|
|
administrator paged cannot correct the problem, he relays it to
|
|
the responsible person or agency.</P>
|
|
|
|
<P>Before we installed Big Brother, we were frequently notified
|
|
of these failures by frustrated users telephoning us. Now, we
|
|
are often aware of what has failed before they call us. The
|
|
users are also becoming aware that they may monitor the network
|
|
through the WWW interface. In many instances, we are able to
|
|
actually correct the problem before it perturbs our users. It
|
|
is difficult to accurately measure the time saved, but we
|
|
estimate that Big Brother has had a net positive effect.</P>
|
|
|
|
<P>We have a machine in a publicly visible area displaying the
|
|
brief view of Big Brother. The green, yellow, red and blue
|
|
screen splashes are clearly visible far down the hall. This
|
|
helps our network team to be more aware of problems as they
|
|
occur. The accessibility of the WWW page has made Big Brother
|
|
useful even to people at the far ends of our network.
|
|
So far, we are not inclined to shut Big Brother down. It has
|
|
become a helpful member of our network team.</P>
|
|
|
|
<P>Maybe now I'll have time to be bored. . . </P>
|
|
|
|
<P>
|
|
<HR>
|
|
<ADDRESS>
|
|
Texas Agricultural Extension WWW Server
|
|
<A HREF="http://taex.tamu.edu/">http://taex.tamu.edu/</A><BR>
|
|
Extension Information Technology / Texas Agricultural Extension Service<BR>
|
|
The Texas A&M University System /
|
|
College Station, Texas 77843-2468<BR>
|
|
This page was last modified <b>Thu Feb 20 15:47:14 1997
|
|
</b>
|
|
by
|
|
<A HREF="mailto:p-sittler@tamu.edu">PMS</A>.
|
|
(URL=http://taex001.tamu.edu/bb/articles/bbartlg.html)
|
|
</ADDRESS>
|
|
|
|
<!--===================================================================-->
|
|
<P> <hr> <P>
|
|
<center><H5>Copyright © 1997, Paul M. Sittler <BR>
|
|
Published in Issue 15 of the Linux Gazette, March 1997</H5></center>
|
|
|
|
<!--===================================================================-->
|
|
<P> <hr> <P>
|
|
<A HREF="./index.html"><IMG ALIGN=BOTTOM SRC="../gx/indexnew.gif"
|
|
ALT="[ TABLE OF CONTENTS ]"></A>
|
|
<A HREF="../index.html"><IMG ALIGN=BOTTOM SRC="../gx/homenew.gif"
|
|
ALT="[ FRONT PAGE ]"></A>
|
|
<A HREF="./clueless.html"><IMG SRC="../gx/back2.gif"
|
|
ALT=" Back "></A>
|
|
<A HREF="./2cent.html"><IMG SRC="../gx/fwd.gif" ALT=" Next "></A>
|
|
<P> <hr> <P>
|
|
</BODY>
|
|
</HTML>
|