328 lines
13 KiB
HTML
328 lines
13 KiB
HTML
<!--startcut ======================================================= -->
|
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
|
|
<html><head>
|
|
<META NAME="generator" CONTENT="lgazmail v1.1D.l">
|
|
<TITLE>The Answer Guy 34:
|
|
Test Suites for GNU and other Open Source (TM) Software
|
|
</TITLE>
|
|
<!-- ORIGINAL SUBJECT:
|
|
Is there a testsuite for glibc v2.0.x?
|
|
JTD SUBTITLE:
|
|
|
|
-->
|
|
</head>
|
|
|
|
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#A000A0"
|
|
ALINK="#FF0000">
|
|
<H4>"Linux Gazette...<I>making Linux just a little more fun!</I>"
|
|
</H4>
|
|
<P> <hr> <P>
|
|
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
|
|
<H1 align="center"><A NAME="answer">
|
|
<img src="../../gx/dennis/qbubble.gif" alt="" border="0" align="middle">
|
|
<a href="../index.html">The Answer Guy</a>
|
|
<img src="../../gx/dennis/bbubble.gif" alt="" border="0" align="middle">
|
|
</A></H1>
|
|
<BR>
|
|
<H4 align="center">By James T. Dennis,
|
|
<a href="mailto:linux-questions-only@ssc.com">linux-questions-only@ssc.com</a>
|
|
<BR>Starshine Technical Services, <A HREF="http://www.starshine.org/">http://www.starshine.org/</A>
|
|
</H4>
|
|
<p><hr><p>
|
|
<!--endcut ========================================================= -->
|
|
<H3><img src="../../gx/dennis/qbub.gif" alt="(?)"
|
|
width="50" height="28" align="left" border="0"
|
|
>Test Suites for GNU and other Open Source (TM) Software</H3>
|
|
<p><strong>From Steve Snyder on 20 Sep 1998 </strong></p>
|
|
<!-- begin body -->
|
|
|
|
|
|
<p><strong>Is there a validation test suite for glibc v2.0.x? I mean a more
|
|
comprehensive set of tests than are run by "<tt>make check</tt>" after
|
|
building the runtime libraries.
|
|
</strong></p>
|
|
|
|
<blockquote><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)" height="28" width="50" align="bottom">Not that I know of. I guess the conventional wisdom
|
|
is that if I install glibc, and a bunch of sources and
|
|
I compile the sources against the libc and run all of them
|
|
--- that failures will somehow be "obvious."
|
|
</blockquote>
|
|
|
|
<blockquote>Personally I think that this is stupid. Obviously it mostly
|
|
works for most of us most of the time. However, it would be
|
|
nice to have full test and regression suites that exercise a
|
|
large range of functions for each package --- and to
|
|
include these (and the input and reference data sets) in the
|
|
sources.
|
|
</blockquote>
|
|
|
|
<blockquote>It would also be nice if every one of them included a
|
|
"fuzz" script (calling the program with random combinations
|
|
of available options, switches and inputs --- particularly
|
|
with a list of the available options). This could test
|
|
the programs for robustness in the face of errors and
|
|
might even find some buffer overflows other bugs.
|
|
</blockquote>
|
|
|
|
<blockquote>However, I'm not a programmer. I used to do some quality
|
|
assurance --- and that whole segment of the market seems
|
|
to be in a sad state. I learned (when I studied
|
|
programming) that the documentation and the test suite
|
|
should be developed as part of the project. User and
|
|
programmer documentation should lead the coding (with
|
|
QA cycles and user reviews of the <EM>proposed</EM> user
|
|
interfaces, command sets, etc, prior to coding.
|
|
</blockquote>
|
|
|
|
<blockquote>The "whitebox" test suites should be developed incrementally
|
|
as parts of the code are delivered (if I write a function
|
|
that will be used in the project, some QA whiteboxer should
|
|
write a small specialized program that calls this function
|
|
with a range of valid and invalid inputs and tests the
|
|
function's behaviour against a suite that just applies to
|
|
it).
|
|
</blockquote>
|
|
|
|
<blockquote>Recently I decided that <tt>md5sum</tt> really needs an option
|
|
to read filenames from stdin. I want do write some
|
|
scripts that essentially do:
|
|
</blockquote>
|
|
|
|
|
|
<blockquote><blockquote>
|
|
<code>'find .... -print0 | md5sum -0f '
|
|
</code></blockquote></blockquote>
|
|
|
|
|
|
<blockquote>... kind of like '<TT>cpio</TT>' Actually I really want to do:
|
|
</blockquote>
|
|
|
|
|
|
<blockquote><blockquote>
|
|
<code>'rpm -qal | regular.files | md5sum -f'
|
|
</code></blockquote></blockquote>
|
|
|
|
<blockquote>... to generate some relatively large checksum files
|
|
for later use with the '<tt>-c</tt>' option. This '<TT>rpm</TT>' command
|
|
will "Query All packages for a List of all files. The
|
|
<tt>regular.files</tt> filter is just a short shell script that
|
|
does:
|
|
</blockquote>
|
|
|
|
<pre>#!/bin/sh
|
|
## This uses the test command to filter out filenames that
|
|
## refer to anything other than regular files (directories,
|
|
## Unix domain sockets, device nodes, FIFO/named pipes, etc)
|
|
while read i ; do
|
|
[ -f "$i" ] && echo "$i"
|
|
done
|
|
</pre>
|
|
|
|
<blockquote>So I grabbed the textutils sources, created a few
|
|
sets of md5sum files from my local files (using
|
|
'<tt>xargs</tt>'). Those are my test data sets.
|
|
</blockquote>
|
|
|
|
<blockquote>Then I got into <tt>md5sum.c</tt>, added the command options,
|
|
cut and pasted some parts of the existing functions
|
|
into a new function, and what able to get it cleanly
|
|
compiling in a couple hours. I said I'm not a programmer
|
|
didn't I. I think a decent programmer could have done
|
|
this in about an hour.
|
|
</blockquote>
|
|
|
|
<blockquote>Then I ran several tests. I ran the "<tt>make check</tt>" tests,
|
|
and used the new version's <tt>-c</tt> to check my test sets.
|
|
I then used same script that generated those to generate
|
|
a new set using the new version/binary. I then compared
|
|
those (using '<TT>cmp</TT>' and '<tt>diff</tt>') and checked them with
|
|
the old version. Then I generated new ones (with the
|
|
new switch I'd added, and again with the old version)
|
|
and cross check them again.
|
|
</blockquote>
|
|
|
|
<blockquote>This new version allows you to to use stdin or pass
|
|
a filename which contains a list of files to checksum
|
|
--- it uses the <tt>--filelist</tt> long argument as well as
|
|
the <tt>-f</tt> short form; and you can use <tt>-f -</tt> or just
|
|
<tt>-f</tt> to
|
|
use stdin. I didn't implement the <tt>-0</tt> (<tt>--null</tt>)
|
|
option
|
|
--- but I did put in the placeholders in the code where
|
|
it could be done.
|
|
</blockquote>
|
|
|
|
<blockquote>The point here is that I had a test suite that was longer
|
|
than the code. I also spent more time testing and
|
|
documenting (writing a note to Ulrich Drepper, the
|
|
original author of this package to offer the patches to
|
|
him) than I did on coding.
|
|
</blockquote>
|
|
|
|
<p><strong><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)" height="28" width="50" align=""bottom">Though a benchmarking component would be nice, my main concern is
|
|
to verify that all (or at least the vast majority) of the library
|
|
function work correctly. What I want to know is, given a specific
|
|
compiler and a specific version of glibc source files, how can I
|
|
verify that the libraries built are reliable?
|
|
</strong></p>
|
|
|
|
<blockquote><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)" height="28" width="50" align="bottom">By testing them. Unfortunately, that may mean that
|
|
you'll have to write your own test suites. You may
|
|
have to start a GNU/new project to create test suites.
|
|
</blockquote>
|
|
|
|
<blockquote>It is likely that most of the developers and maintainers
|
|
of these packages have test suites that they run before
|
|
they post their new versions. It would be nice if they
|
|
posted the test suites <em>as part of the source package</em>
|
|
--- and opened the testing part of the project to the
|
|
open development model.
|
|
</blockquote>
|
|
|
|
<blockquote>In addition these test suites and harnesses (the scripts
|
|
to create isolated and sample directory structures, etc)
|
|
to run a program (or library) through its paces) would
|
|
serve as a great addition to the documentation.
|
|
</blockquote>
|
|
|
|
<blockquote>I find '<TT>man</TT>' pages to be incredibly dense. They are
|
|
find if you know enough about the package that you are
|
|
just looking for a specific feature, that you think
|
|
<em>might</em> be there, or one that you <EM>know</EM> is in there
|
|
somewhere --- but you don't remember the switch or the
|
|
syntax. However, a test harness, script, and set of
|
|
associated inputs, outputs, and configurations files would
|
|
give plenty of examples of how the bloody thing is
|
|
<em>supposed</em> to work. I often have to hunt for examples
|
|
--- this would help.
|
|
</blockquote>
|
|
|
|
<p><strong><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)" height="28" width="50" align=""bottom">The specific version I want to test is the glibc v2.0.7 that comes
|
|
with RH Linux v5.1 and updated after 5.1 release by package
|
|
glibc-2.0.7-19.src.rpm. I think that such a testsuite, though, if
|
|
it exists, would be applicable to any platform.
|
|
</strong></p>
|
|
|
|
<blockquote><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)" height="28" width="50" align="bottom">I agree. I just wish I could really co-ordinate such
|
|
a project. I think this is another example where our
|
|
academic communities could really help. Before I've
|
|
said that I would like to see the "adopt a '<TT>man</TT>' page
|
|
project" --- where college and university professors
|
|
even high school teachers from around the world assign
|
|
a job to their students:
|
|
</blockquote>
|
|
|
|
<blockquote>Find a command or package for Linux, <A HREF="http://www.freebsd.org/">FreeBSD</A>,
|
|
etc. Read the man pages and other docs.
|
|
Find one way that the command is used or useful
|
|
that is not listed the "examples" section of
|
|
that man page. Write a canonical example of
|
|
that command variant.
|
|
</blockquote>
|
|
|
|
<blockquote>... they would get graded on their work --- and any A's
|
|
would be encouraged (solely at their option) to submit
|
|
the recommended example as a patch to the maintainer of
|
|
the package.
|
|
</blockquote>
|
|
|
|
<blockquote>Similar assigments would be given for system calls, library
|
|
functions, etc (as appropriate to the various classes and
|
|
class segments).
|
|
</blockquote>
|
|
|
|
<blockquote>Along with this, we could have a process by which students
|
|
are encouraged to find bugs in existing real world software
|
|
--- write test suites and scripts to test for the recurrence
|
|
of these bugs in future versions (regressions), and submit
|
|
the tests to that package's maintainer.
|
|
</blockquote>
|
|
|
|
<blockquote>The problem with all of this is that testing is not
|
|
glamorous. It is boring for most people. Everyone
|
|
knows Richard M. Stallman's and Linus Torvalds' names
|
|
--- but fewer people remember the names of the other
|
|
programmers that they work with and no one know who
|
|
contributed "just the testing."
|
|
</blockquote>
|
|
|
|
<blockquote>There are methods that can be used to many detect bugs quicker
|
|
and more reliably than by waiting until users "bump into"
|
|
them. These won't be comprehensive. They won't catch
|
|
"all" of the bugs. However, people will "bump" into enough
|
|
bugs in normal usage, even if we employ the best principles
|
|
of QA practice across the board.
|
|
</blockquote>
|
|
|
|
<blockquote>Unfortunately I don't have the time to really devote to
|
|
such a project. I devote most of my "free" time to the
|
|
tech support department. I do have spare machine cycles.
|
|
could gladly devote time to running these tests
|
|
and reporting results. Obviously some tests require whole
|
|
networks, preferably disconnected ones, on which to
|
|
run safely. Setting up such test beds, and designing
|
|
tests that return meaningful results is difficult work.
|
|
</blockquote>
|
|
|
|
<blockquote>I personally think that good test harnesses are often
|
|
harder to design than the programs that they are designed
|
|
to test.
|
|
</blockquote>
|
|
|
|
<p><strong><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)" height="28" width="50" align=""bottom">Thank you.
|
|
<br>***** Steve Snyder *****
|
|
</strong></p>
|
|
<!-- end body -->
|
|
|
|
<!--startcut ======================================================= -->
|
|
<P> <hr> <P>
|
|
<H5 align="center"><a href="http://www.linuxgazette.com/copying.html"
|
|
>Copyright ©</a> 1998, James T. Dennis <BR>
|
|
Published in <I>Linux Gazette</I> Issue 34 November 1998</H5>
|
|
<P> <hr> <P>
|
|
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
|
|
<P align="center">
|
|
<table width="98%"><tr valign="center" align="center">
|
|
<td rowspan="3"><A HREF="../lg_answer34.html"><IMG
|
|
SRC="../../gx/dennis/answernew.gif"
|
|
ALT="[ Answer Guy Index ]"></A></td>
|
|
<td><A HREF="./apache.html">apache</a>
|
|
<td><A HREF="./current.html">current</a>
|
|
<td><A HREF="./digi.html">digi</a>
|
|
<td><A HREF="./ether.html">ether</a>
|
|
<td><A HREF="./goodtimes.html">goodtimes</a>
|
|
<td><A HREF="./intlX.html">intlX</a>
|
|
<td><A HREF="./largedisk.html">largedisk</a>
|
|
</tr><tr valign="center" align="center">
|
|
<td><A HREF="./maybe.html">maybe</a>
|
|
<td><A HREF="./numlock.html">numlock</a>
|
|
<td><A HREF="./quota.html">quota</a>
|
|
<td><A HREF="./recovery.html">recovery</a>
|
|
<td><A HREF="./script.html">script</a>
|
|
<td><A HREF="./serial.html">serial</a>
|
|
<td><A HREF="./session.html">session</a>
|
|
</tr><tr valign="center" align="center">
|
|
<td><A HREF="./sound.html">sound</a>
|
|
<td><A HREF="./tape.html">tape</a>
|
|
<td><A HREF="./testsuite.html">testsuite</a>
|
|
<td><A HREF="./w95ie.html">w95ie</a>
|
|
<td><A HREF="./w95ras.html">w95ras</a>
|
|
<td><A HREF="./w95virus.html">w95virus</a>
|
|
<td><A HREF="./xdm.html">xdm</a>
|
|
</tr></table>
|
|
</P>
|
|
<P> <hr> <P>
|
|
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
|
|
<A HREF="../index.html"><IMG SRC="../../gx/indexnew.gif"
|
|
ALT="[ Table Of Contents ]"></A>
|
|
<A HREF="../../index.html"><IMG SRC="../../gx/homenew.gif"
|
|
ALT="[ Front Page ]"></A>
|
|
<A HREF="../lg_bytes34.html"><IMG SRC="../../gx/back2.gif"
|
|
ALT="[ Previous Section ]"></A>
|
|
<A HREF="../izquierdo.html"><IMG SRC="../../gx/fwd.gif"
|
|
ALT="[ Next Section ]"></A>
|
|
</body>
|
|
</html>
|
|
<!--endcut ========================================================= -->
|
|
|