1067 lines
21 KiB
HTML
1067 lines
21 KiB
HTML
<HTML
|
|
><HEAD
|
|
><TITLE
|
|
>Components of a running system</TITLE
|
|
><META
|
|
NAME="GENERATOR"
|
|
CONTENT="Modular DocBook HTML Stylesheet Version 1.76b+
|
|
"><LINK
|
|
REL="HOME"
|
|
TITLE="Usenet News HOWTO "
|
|
HREF="index.html"><LINK
|
|
REL="PREVIOUS"
|
|
TITLE="Access control in NNTPd"
|
|
HREF="x818.html"><LINK
|
|
REL="NEXT"
|
|
TITLE="Monitoring and administration"
|
|
HREF="x1094.html"></HEAD
|
|
><BODY
|
|
CLASS="SECTION"
|
|
BGCOLOR="#FFFFFF"
|
|
TEXT="#000000"
|
|
LINK="#0000FF"
|
|
VLINK="#840084"
|
|
ALINK="#0000FF"
|
|
><DIV
|
|
CLASS="NAVHEADER"
|
|
><TABLE
|
|
SUMMARY="Header navigation table"
|
|
WIDTH="100%"
|
|
BORDER="0"
|
|
CELLPADDING="0"
|
|
CELLSPACING="0"
|
|
><TR
|
|
><TH
|
|
COLSPAN="3"
|
|
ALIGN="center"
|
|
>Usenet News HOWTO</TH
|
|
></TR
|
|
><TR
|
|
><TD
|
|
WIDTH="10%"
|
|
ALIGN="left"
|
|
VALIGN="bottom"
|
|
><A
|
|
HREF="x818.html"
|
|
ACCESSKEY="P"
|
|
>Prev</A
|
|
></TD
|
|
><TD
|
|
WIDTH="80%"
|
|
ALIGN="center"
|
|
VALIGN="bottom"
|
|
></TD
|
|
><TD
|
|
WIDTH="10%"
|
|
ALIGN="right"
|
|
VALIGN="bottom"
|
|
><A
|
|
HREF="x1094.html"
|
|
ACCESSKEY="N"
|
|
>Next</A
|
|
></TD
|
|
></TR
|
|
></TABLE
|
|
><HR
|
|
ALIGN="LEFT"
|
|
WIDTH="100%"></DIV
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H1
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="COMPONENT">9. Components of a running system</H1
|
|
><P
|
|
>This chapter reviews the components of a running CNews+NNTPd server.
|
|
Analogous components will be found in an INN-based system too. We invite
|
|
additions from readers familiar with INN to add their pieces to this
|
|
chapter.</P
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H2
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="AEN844">9.1. <TT
|
|
CLASS="LITERAL"
|
|
>/var/lib/news</TT
|
|
>: the CNews control area</H2
|
|
><P
|
|
>This directory is more popularly known as <TT
|
|
CLASS="LITERAL"
|
|
>$NEWSCTL</TT
|
|
>. It
|
|
contains configuration, log and status files. There are no
|
|
articles or binaries kept here. Let's see what some of the
|
|
files are meant for. Control files are dealt in slightly greater detail in
|
|
<SPAN
|
|
CLASS="QUOTE"
|
|
>"<A
|
|
HREF="settingup.html#CONFIGURESYSTEM"
|
|
>Section 4.3</A
|
|
>>"</SPAN
|
|
></P
|
|
><P
|
|
></P
|
|
><UL
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>sys</TT
|
|
>:
|
|
One line per system/NDN listing all the newsgroup
|
|
hierarchies each system subscribes to. Each line is prefixed with the system
|
|
name and the one beginning with ME: indicates what we are going to receive.
|
|
Look up manpage of <TT
|
|
CLASS="LITERAL"
|
|
>newssys</TT
|
|
>.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>explist</TT
|
|
>:
|
|
This file has entries indicating articles of which
|
|
newsgroup expire and when and if they have to be archived. The order in
|
|
which the newsgroups are listed is important. See manpage of
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>expire</TT
|
|
> for file format.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>batchparms</TT
|
|
>:
|
|
Details of how to feed other sites/NDN, like the size of
|
|
batches, the mode of transmission (UUCP/NNTP) are specified here.
|
|
manpage to refer: <TT
|
|
CLASS="LITERAL"
|
|
>newsbatch</TT
|
|
>.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>controlperm</TT
|
|
>:
|
|
If you wish to authenticate a control message before any
|
|
action is taken on it, you must enter authentication-related information
|
|
here. The <TT
|
|
CLASS="LITERAL"
|
|
>controlperm</TT
|
|
> manpage will list all the fields
|
|
in detail.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>mailpaths</TT
|
|
>:
|
|
It features the e-mail address of the moderator for each
|
|
newsgroup who is responsible for approving/disapproving
|
|
articles posted to moderated newsgroups. The sample
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>mailpaths</TT
|
|
> file in the <TT
|
|
CLASS="LITERAL"
|
|
>tar</TT
|
|
> will
|
|
give you an idea of how entries are made.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>nntp_access/user_access</TT
|
|
>:
|
|
These files contain entries of servernames
|
|
and usernames on whom restrictions will apply when accessing newsgroups.
|
|
Again, the sample file in the tarball shall explain the format of the file.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>log, errlog</TT
|
|
>:
|
|
These are log files that keep growing large with each batch
|
|
that is received. The <TT
|
|
CLASS="LITERAL"
|
|
>log</TT
|
|
> file has one entry per
|
|
article telling you if it
|
|
has been accepted by your news server or rejected. To understand the
|
|
format of this file, refer to Chapter 2.2 of the <TT
|
|
CLASS="LITERAL"
|
|
>CNews</TT
|
|
>
|
|
guide. Errors, if any, while digesting the articles are
|
|
logged in <TT
|
|
CLASS="LITERAL"
|
|
>errlog</TT
|
|
>. These
|
|
log files have to be rolled as the files hog a lot of disk space. </P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>nntplog</TT
|
|
>:
|
|
This file logs information of the <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> giving
|
|
details of when a connection was established/broken and what commands were
|
|
issued. This file needs to be configured in <TT
|
|
CLASS="LITERAL"
|
|
>syslog</TT
|
|
>
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>syslogd</TT
|
|
> should be running.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>active</TT
|
|
>:
|
|
This file has one line per newsgroup to be found in your news
|
|
server. Besides other things, it tells you how many articles are
|
|
currently present in each newsgroup. It is updated when each batch is
|
|
digested or when articles are expired. The <TT
|
|
CLASS="LITERAL"
|
|
>active</TT
|
|
>
|
|
manpage will furnish more details about other paramaters.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>history</TT
|
|
>:
|
|
This file, again, contains one line per article, mapping
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>message-id</TT
|
|
> to newsgroup name and also giving its
|
|
associated article number in that newsgroup. It is updated
|
|
each time a feed is digested
|
|
and when <TT
|
|
CLASS="LITERAL"
|
|
>doexpire</TT
|
|
> is run. Plays a key role in
|
|
loop-detection and serves as an article database. Read manpage of
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>newsdb</TT
|
|
>, <TT
|
|
CLASS="LITERAL"
|
|
>doexpire</TT
|
|
> for the file format </P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>newsgroups</TT
|
|
>:
|
|
It has a one-line description for each newsgroup explaining
|
|
what kind of posts go into each of them. Ideally speaking, it should cover
|
|
all the newsgroups found in the <TT
|
|
CLASS="LITERAL"
|
|
>active</TT
|
|
> file.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><EM
|
|
>Miscellaneous files</EM
|
|
>:
|
|
Files like <TT
|
|
CLASS="LITERAL"
|
|
>mailname</TT
|
|
>, <TT
|
|
CLASS="LITERAL"
|
|
>organisation</TT
|
|
>,
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>whoami</TT
|
|
> contain information required for forming some of
|
|
the headers of an article. The contents of
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>mailname</TT
|
|
> form the <TT
|
|
CLASS="LITERAL"
|
|
>From:</TT
|
|
> header and
|
|
that of <TT
|
|
CLASS="LITERAL"
|
|
>organisation</TT
|
|
> form the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>Organisation:</TT
|
|
> header. <TT
|
|
CLASS="LITERAL"
|
|
>whoami</TT
|
|
> contains
|
|
the name of the news system. Refer to chapter 2.1 of
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>guide.ps</TT
|
|
> for a detailed list of files in the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSCTL</TT
|
|
> area. Read <TT
|
|
CLASS="LITERAL"
|
|
>RFC 1036</TT
|
|
> for
|
|
description of article headers .</P
|
|
></LI
|
|
></UL
|
|
></DIV
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H2
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="AEN917">9.2. <TT
|
|
CLASS="LITERAL"
|
|
>/var/spool/news</TT
|
|
>: the article repository</H2
|
|
><P
|
|
>This is also known as the <TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS</TT
|
|
> or
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSSPOOL</TT
|
|
> directory. This is where the
|
|
articles reside on your disk. No binaries or control files
|
|
should belong here. Enough space should be allocated to this
|
|
directory as the number of articles keep increasing with each
|
|
batch that is digested. An explanation of the following sub-directories will
|
|
give you an overview of this directory:
|
|
<P
|
|
></P
|
|
><UL
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>in.coming</TT
|
|
>:
|
|
Feeds/batches/articles from NDNs on their arrival and
|
|
before being processed reside in this directory. After processing, they
|
|
appear in
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS</TT
|
|
> or in its <TT
|
|
CLASS="LITERAL"
|
|
>bad</TT
|
|
> sub-directory
|
|
if there were errors. </P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>out.going</TT
|
|
>:
|
|
This directory contains batches/feeds to be sent to your
|
|
NDNs <EM
|
|
>i.e.</EM
|
|
> feeds to be pushed to your neighbouring sites
|
|
reside here before they are transmitted. It contains one sub-directory per
|
|
NDN mentioned in the <TT
|
|
CLASS="LITERAL"
|
|
>sys</TT
|
|
> file. These sub-directories
|
|
contain files called <TT
|
|
CLASS="LITERAL"
|
|
>togo</TT
|
|
> which contain information about
|
|
the article like the <TT
|
|
CLASS="LITERAL"
|
|
>message-id</TT
|
|
> or the article number
|
|
that is queued for transmission. </P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><A
|
|
NAME="NEWSGROUPDIR"
|
|
></A
|
|
>><EM
|
|
>newsgroup directories</EM
|
|
>:
|
|
For each newsgroup hierarchy that the news server
|
|
has subscribed to, a directory is created under
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS</TT
|
|
>.
|
|
Further sub-directories are created under the parent to hold
|
|
articles of specific newsgroups. For instance, for a
|
|
newsgroup like <TT
|
|
CLASS="LITERAL"
|
|
>comp.music.compose</TT
|
|
>, the parent directory
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>comp</TT
|
|
> will appear in <TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS</TT
|
|
> and a
|
|
sub-directory called <TT
|
|
CLASS="LITERAL"
|
|
>music</TT
|
|
> will be created under
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>comp</TT
|
|
>. The <TT
|
|
CLASS="LITERAL"
|
|
>music</TT
|
|
> sub-directory
|
|
shall contain a further sub-directory called <TT
|
|
CLASS="LITERAL"
|
|
>compose</TT
|
|
> and
|
|
all articles of <TT
|
|
CLASS="LITERAL"
|
|
>comp.music.compose</TT
|
|
>
|
|
shall reside here. In effect, article 242 of newsgroup
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>comp.music.compose</TT
|
|
> shall map to file
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS/comp/music/compose/242</TT
|
|
>.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>control</TT
|
|
>:
|
|
The control directory houses only the control messages that
|
|
have been received by this site. The control messages could be any of the
|
|
following: <TT
|
|
CLASS="LITERAL"
|
|
>newgroup, rmgroup, checkgroup</TT
|
|
> and
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>cancel</TT
|
|
> appearing in the subject line of the article.
|
|
More information to be found in <SPAN
|
|
CLASS="QUOTE"
|
|
>"<A
|
|
HREF="x64.html#CONTROLMSG"
|
|
>Section 2.4</A
|
|
>>"</SPAN
|
|
></P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>junk</TT
|
|
>:
|
|
The <TT
|
|
CLASS="LITERAL"
|
|
>junk</TT
|
|
> directory contains all
|
|
articles that the news
|
|
server has received and has decided, after processing, that it does not
|
|
belong to any of the hierarchies it has subscribed to. The news server
|
|
transfers/passes all articles in this directory to NDNs
|
|
that have subscribed to the <TT
|
|
CLASS="LITERAL"
|
|
>junk</TT
|
|
> hierarchy.</P
|
|
></LI
|
|
></UL
|
|
></P
|
|
></DIV
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H2
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="AEN963">9.3. <TT
|
|
CLASS="LITERAL"
|
|
>/usr/lib/newsbin</TT
|
|
>: the executables</H2
|
|
><P
|
|
>TO BE ADDED LATER</P
|
|
></DIV
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H2
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="CRONJOBS">9.4. <TT
|
|
CLASS="LITERAL"
|
|
>crontab and cron jobs </TT
|
|
></H2
|
|
><P
|
|
>The heart of the Usenet news server is the various scripts that run at regular
|
|
intervals processing articles, digesting/rejecting them and
|
|
transmitting them to NDNs. I shall try to enumerate the ones that are important
|
|
enough to be cronned. :)</P
|
|
><P
|
|
></P
|
|
><UL
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>newsrun</TT
|
|
>:
|
|
The key script. This script picks the batches in the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>in.coming</TT
|
|
> directory, uncompresses them if necessary and
|
|
feeds it to <TT
|
|
CLASS="LITERAL"
|
|
>relaynews</TT
|
|
> which then processes each
|
|
article digesting and batching them and logging any errors. This script
|
|
needs to run through <TT
|
|
CLASS="LITERAL"
|
|
>cron</TT
|
|
>
|
|
as frequently as you want the feeds to be digested. Every half hour should
|
|
suffice for a non-critical requirement.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>sendbatches</TT
|
|
>:
|
|
This script is run to transmit the <TT
|
|
CLASS="LITERAL"
|
|
>togo</TT
|
|
> files formed in
|
|
the <TT
|
|
CLASS="LITERAL"
|
|
>out.going</TT
|
|
> directory to your NDNs. It reads the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>batchparms</TT
|
|
> file to know
|
|
exactly how and to whom the batches need to be transmitted. The frequency,
|
|
again, can be set according to your requirements. Once an hour should be
|
|
sufficient.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>newsdaily</TT
|
|
>:
|
|
This script does maintenance chores like rolling logs and
|
|
saving them, reporting errors/anomalies and doing cleanup jobs.
|
|
It should typically run once a day.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>newswatch</TT
|
|
>:
|
|
This looks for news problems at a more detailed level than
|
|
newsdaily like looking for persistent lock files or unattended batches,
|
|
determining space shortage issues, and the likes. This should typically run
|
|
once every hour. For more on this and the above, read the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>newsmaint</TT
|
|
> manpage.</P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>doexpire</TT
|
|
>:
|
|
This script expires old articles as determined by the
|
|
control file <TT
|
|
CLASS="LITERAL"
|
|
>explist</TT
|
|
> and updates the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>active</TT
|
|
> file. This is necessary if you do not
|
|
want unnecessary/unwanted articels hogging up your disk space. Run it once
|
|
a day. Manpage: <TT
|
|
CLASS="LITERAL"
|
|
>expire</TT
|
|
></P
|
|
></LI
|
|
><LI
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>newsrunning off/on</TT
|
|
>:
|
|
This script shuts/starts off the news server for you.
|
|
You could choose to add this in your cron job if you think the news server
|
|
takes up lots of CPU time during peak hours and you wish to keep a check on
|
|
it. </P
|
|
></LI
|
|
></UL
|
|
></DIV
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H2
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="AEN1000">9.5. <TT
|
|
CLASS="LITERAL"
|
|
>newsrun</TT
|
|
> and <TT
|
|
CLASS="LITERAL"
|
|
>relaynews</TT
|
|
>: digesting received articles</H2
|
|
><P
|
|
>The heart and soul of the Usenet News system, <TT
|
|
CLASS="LITERAL"
|
|
>newsrun</TT
|
|
> just picks up the batches/
|
|
articles in the <TT
|
|
CLASS="LITERAL"
|
|
>in.coming</TT
|
|
> directory of
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS</TT
|
|
> and uncompresses them (if required) and calls
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>relaynews</TT
|
|
>. It should run from cron.</P
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>relaynews</TT
|
|
> picks up each article one by one through
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>stdin</TT
|
|
>, determines if it belongs to a subscribed group
|
|
by looking up
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>sys</TT
|
|
> file, looks in the <TT
|
|
CLASS="LITERAL"
|
|
>history</TT
|
|
> file
|
|
to determine that it does not already exist locally, digests it updating the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>active</TT
|
|
> and <TT
|
|
CLASS="LITERAL"
|
|
>history</TT
|
|
> file and batches it
|
|
for neighbouring sites. Logs errors on encountering problems while processing
|
|
the article and takes appropriate action if it happens to be
|
|
a control message. More info in manpage of <TT
|
|
CLASS="LITERAL"
|
|
>relaynews</TT
|
|
>.</P
|
|
></DIV
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H2
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="AEN1017">9.6. <TT
|
|
CLASS="LITERAL"
|
|
>doexpire</TT
|
|
> and <TT
|
|
CLASS="LITERAL"
|
|
>expire</TT
|
|
>: removing old articles</H2
|
|
><P
|
|
>A good way to get rid of unwanted/old articles from the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS</TT
|
|
> area is to run <TT
|
|
CLASS="LITERAL"
|
|
>doexpire</TT
|
|
> once a
|
|
day. It reads the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>explist</TT
|
|
> file from the <TT
|
|
CLASS="LITERAL"
|
|
>$NEWSCTL</TT
|
|
> directory
|
|
to determine what articles expire today. It can archive the
|
|
said article if so configured. It then updates the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>active</TT
|
|
> and the <TT
|
|
CLASS="LITERAL"
|
|
>history</TT
|
|
> file accordingly.
|
|
If you wish to retain the article entry in the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>history</TT
|
|
> file to avoid re-digesting it as a new
|
|
article after having expired it, add a special <EM
|
|
>/expired/;</EM
|
|
>
|
|
line in the control file. More on the options and functioning in the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>expire </TT
|
|
> manpage.</P
|
|
></DIV
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H2
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="AEN1031">9.7. <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> and <TT
|
|
CLASS="LITERAL"
|
|
>msgidd</TT
|
|
>: managing the NNTP interface</H2
|
|
><P
|
|
>As has already been discussed in the chapter on setting up the software,
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> is a TCP-based server daemon which runs under
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>inetd</TT
|
|
>. It is fired by <TT
|
|
CLASS="LITERAL"
|
|
>inetd</TT
|
|
>
|
|
whenever there's an incoming connection on the NNTP port, and it takes
|
|
over the dialogue from there. It reads the C-News configuration and data
|
|
files in <TT
|
|
CLASS="LITERAL"
|
|
>$NEWSCTL</TT
|
|
>, article files from
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS></TT
|
|
>, and receives incoming posts and
|
|
transfers. These it dutifully queues in
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS/in.coming</TT
|
|
>, either as batch files or single
|
|
article files.</P
|
|
><P
|
|
>It is important that <TT
|
|
CLASS="LITERAL"
|
|
>inetd</TT
|
|
> be configured to
|
|
fire <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> as user <TT
|
|
CLASS="LITERAL"
|
|
>news</TT
|
|
>, not as
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>root</TT
|
|
> like it does for other daemons like
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>telnetd</TT
|
|
> or <TT
|
|
CLASS="LITERAL"
|
|
>ftpd</TT
|
|
>. If this is not
|
|
done correctly, a lot of problems can be caused in the functioning of
|
|
the C-News system later.</P
|
|
><P
|
|
><TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> is fired each time a new NNTP connection
|
|
is received, and dies once the NNTP client closes its connection. Thus,
|
|
if one <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> receives a few articles by an incoming
|
|
batch feed (not a <TT
|
|
CLASS="LITERAL"
|
|
>POST</TT
|
|
> but an <TT
|
|
CLASS="LITERAL"
|
|
>XFER</TT
|
|
>),
|
|
then another <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> will not know about the receipt of
|
|
these articles till the batches are digested. This will hamper
|
|
duplicate newsfeed detection if there are multiple upstream NDNs feeding
|
|
our server with the same set of articles over NNTP. To fix this,
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> uses an ally: <TT
|
|
CLASS="LITERAL"
|
|
>msgidd</TT
|
|
>, the
|
|
message ID daemon. This
|
|
daemon is fired once at server bootup time through
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>newsboot</TT
|
|
>, and keeps running quietly in the
|
|
background, listening on a named Unix socket in the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSCTL</TT
|
|
> area. It keeps in its memory a list of all
|
|
message IDs which various incarnations of <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> have
|
|
asked it to remember.</P
|
|
><P
|
|
>Thus, when one copy of <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> receives an
|
|
incoming feed of news articles, it updates <TT
|
|
CLASS="LITERAL"
|
|
>msgidd</TT
|
|
>
|
|
with the message IDs of these messages through the Unix socket. When
|
|
another copy of <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> is fired later and the NNTP
|
|
client tries to feed it some more articles, the <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
>
|
|
checks each message ID against <TT
|
|
CLASS="LITERAL"
|
|
>msgidd</TT
|
|
>. Since
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>msgidd</TT
|
|
> stores all these IDs in memory, the lookup is
|
|
very fast, and duplicate articles are blocked at the NNTP interface
|
|
itself.</P
|
|
><P
|
|
>On a running system, expect to see one instance of
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> for each active NNTP connection, and just one
|
|
instance of <TT
|
|
CLASS="LITERAL"
|
|
>msgidd</TT
|
|
> running quietly in the background,
|
|
hardly consuming any CPU resources. Our <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> is
|
|
configured to die if the NNTP connection is more than a few minutes
|
|
idle, thus conserving server resources. This does not inconvenience the
|
|
user because modern NNTP clients simply re-connect. If an
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> instance is found to be running for days, it is
|
|
either hung due to a network error, or is receiving a very long incoming
|
|
NNTP feed from your upstream server. We used to receive our primary
|
|
incoming feed from our service provider through NNTP sessions lasting 18
|
|
to 20 hours without a break, every day.</P
|
|
></DIV
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H2
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="AEN1072">9.8. <TT
|
|
CLASS="LITERAL"
|
|
>nov</TT
|
|
>, the News Overview system</H2
|
|
><P
|
|
>NOV, the News Overview System is a recent augmentation to the
|
|
C-News and NNTP systems and to the NNTP protocol. This subsystem
|
|
maintains a file for each active newsgroup, in which it maintains one
|
|
line per current article. This line of text contains some key meta-data
|
|
about the article, <EM
|
|
>e.g.</EM
|
|
> the contents of the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>From</TT
|
|
>, <TT
|
|
CLASS="LITERAL"
|
|
>Subject</TT
|
|
>,
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>Date</TT
|
|
> and the article size and message ID. This speeds
|
|
up NNTP response enormously. The <TT
|
|
CLASS="LITERAL"
|
|
>nov</TT
|
|
> library has been
|
|
integrated into the <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
> code, and into key binaries
|
|
of C-News, thus providing seamless maintenance of the News Overview
|
|
database when articles are added or deleted from the repository.</P
|
|
><P
|
|
>When <TT
|
|
CLASS="LITERAL"
|
|
>newsrun</TT
|
|
> adds an article into
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>starcom.test</TT
|
|
>, it also updates
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>$NEWSARTS/starcom/test/.overview</TT
|
|
> and adds a line with
|
|
the relevant data, tab-separated, into it. When <TT
|
|
CLASS="LITERAL"
|
|
>nntpd</TT
|
|
>
|
|
comes to life with an NNTP client, and it sees the
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>XOVER</TT
|
|
> NNTP command, it reads this
|
|
<TT
|
|
CLASS="LITERAL"
|
|
>.overview</TT
|
|
> file, and returns the relevant lines to the
|
|
NNTP client. When <TT
|
|
CLASS="LITERAL"
|
|
>expire</TT
|
|
> deletes an article, it also
|
|
removes the corresponding line from the <TT
|
|
CLASS="LITERAL"
|
|
>.overview</TT
|
|
>
|
|
file. Thus, the maintenance of the NOV database is seamless.</P
|
|
></DIV
|
|
><DIV
|
|
CLASS="SECTION"
|
|
><H2
|
|
CLASS="SECTION"
|
|
><A
|
|
NAME="AEN1091">9.9. Batching feeds with UUCP and NNTP</H2
|
|
><P
|
|
>Some information about batching feeds has been provided in earlier
|
|
sections. More will be added later here in this document.</P
|
|
></DIV
|
|
></DIV
|
|
><DIV
|
|
CLASS="NAVFOOTER"
|
|
><HR
|
|
ALIGN="LEFT"
|
|
WIDTH="100%"><TABLE
|
|
SUMMARY="Footer navigation table"
|
|
WIDTH="100%"
|
|
BORDER="0"
|
|
CELLPADDING="0"
|
|
CELLSPACING="0"
|
|
><TR
|
|
><TD
|
|
WIDTH="33%"
|
|
ALIGN="left"
|
|
VALIGN="top"
|
|
><A
|
|
HREF="x818.html"
|
|
ACCESSKEY="P"
|
|
>Prev</A
|
|
></TD
|
|
><TD
|
|
WIDTH="34%"
|
|
ALIGN="center"
|
|
VALIGN="top"
|
|
><A
|
|
HREF="index.html"
|
|
ACCESSKEY="H"
|
|
>Home</A
|
|
></TD
|
|
><TD
|
|
WIDTH="33%"
|
|
ALIGN="right"
|
|
VALIGN="top"
|
|
><A
|
|
HREF="x1094.html"
|
|
ACCESSKEY="N"
|
|
>Next</A
|
|
></TD
|
|
></TR
|
|
><TR
|
|
><TD
|
|
WIDTH="33%"
|
|
ALIGN="left"
|
|
VALIGN="top"
|
|
>Access control in NNTPd</TD
|
|
><TD
|
|
WIDTH="34%"
|
|
ALIGN="center"
|
|
VALIGN="top"
|
|
> </TD
|
|
><TD
|
|
WIDTH="33%"
|
|
ALIGN="right"
|
|
VALIGN="top"
|
|
>Monitoring and administration</TD
|
|
></TR
|
|
></TABLE
|
|
></DIV
|
|
></BODY
|
|
></HTML
|
|
> |