LDP
/
LDP
mirror of https://github.com/tLDP/LDP
0
1
Fork 0
Code Releases Activity

New chapter added on security issues in Usenet

This commit is contained in:
starcom 2002-08-21 08:17:28 +00:00
parent 9ea21f44e1
commit f05766c9a0
12 changed files with 951 additions and 433 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
LDP/howto/docbook/Usenet-News-HOWTO/accesscontrol.sgml
View File

@ -26,26 +26,25 @@ messages, and discussion archives. Details are given below.
</para>
<section><title>Host-based access control</title>
<para></para>
<para>TO BE ADDED LATER</para>
</section>
<section><title>User authentication and authorisation</title>
<para></para>
<section><title>The NNTPd password file</title>
<para></para>
<para>TO BE ADDED LATER</para>
</section>
<section><title>Mapping users to newsgroups</title>
<para></para>
<para>TO BE ADDED LATER</para>
</section>
<section><title>The <literal>X-Authenticated-Author</literal> article header</title>
<para></para>
<para>TO BE ADDED LATER</para>
</section>
<section><title>Other article header additions</title>
<para></para>
<para>TO BE ADDED LATER</para>
</section>
</section>

42
LDP/howto/docbook/Usenet-News-HOWTO/clients.sgml
View File

@ -17,10 +17,11 @@ Usenet User Agents, along the lines of MUA for Mail User Agents.
</para>
<para>
There are other special-purpose clients, which either pull out articles
to copy or transfer somewhere else, or for analysis, <emphasis>e.g.</emphasis> a
search engine which allows you to search a Usenet article archive, like Google
(<literal>www.google.com</literal>) does.
There are other special-purpose clients, which either pull out
articles to copy or transfer somewhere else, or for analysis,
<emphasis>e.g.</emphasis> a search engine which allows you to search a
Usenet article archive, like Google (<literal>www.google.com</literal>)
does.
</para>
<para>
@ -39,15 +40,15 @@ transfer articles or do other special-purpose things with them.
<section><title>Usenet User Agents</title>
<section><title>Accessing articles: NNTP or spool area?</title>
<para></para>
<para>TO BE ADDED LATER</para>
</section>
<section><title>Threading</title>
<para></para>
<para>TO BE ADDED LATER</para>
</section>
<section><title>Quick reading features</title>
<para></para>
<para>TO BE ADDED LATER</para>
</section>
</section>
@ -62,6 +63,31 @@ to take contributed additions that discuss other client software.
</section>
<section><title>Special clients</title>
<para></para>
<section><title>NNTPCache</title>
<para>NNTPCache is an interesting transparent cacheing proxy for
news articles. News articles are read-only by definition,
<emphasis>i.e.</emphasis> they do not change once they are posted;
they can only be deleted. NNTPCache uses this feature to build a
local cache of news articles.</para>
<para>You set up NNTPCache to listen on the NNTP port of your local
Unix server, and act like an NNTP daemon. You configure it to
connect back-to-back to another NNTP daemon, further away, which has
all the interesting stuff the users want to read. When a user
connects to the local NNTPCache, it connects to the remote NNTP
server and acts as a relay for the NNTP connection, ferrying
commands and responses back and forth. What the user sees therefore
comes from the remote server, the first time. However, all news
articles fetched by NNTPCache are also stored in a local cache, thus
allowing the next user to browse the same set of articles faster.
Like all demand-driven caches, the advantage here is that the local
NNTPCache does not need (much) administering, and will automatically
delete all articles from its cache once they've been lying unread
long enough.</para>
<para>We list it here as an NNTP client because every proxy server
is a server on one side and a client on the other.</para>
</section>
</section>
</section>

91
LDP/howto/docbook/Usenet-News-HOWTO/components.sgml
View File

@ -1,4 +1,5 @@
<section><title>Components of a running system</title>
<section id="component" xreflabel="Components of a running system">
<title>Components of a running system</title>
<para>
This chapter reviews the components of a running CNews+NNTPd server.
Analogous components will be found in an INN-based system too. We invite
@ -11,7 +12,8 @@ chapter.
This directory is more popularly known as <literal>$NEWSCTL</literal>. It
contains configuration, log and status files. There are no
articles or binaries kept here. Let's see what some of the
files are meant for.
files are meant for. Control files are dealt in slightly greater detail in
<quote><xref linkend="configuresystem"/></quote>
</para>
<itemizedlist>
@ -47,7 +49,7 @@ files are meant for.
newsgroup who is responsible for approving/disapproving
articles posted to moderated newsgroups. The sample
<literal>mailpaths</literal> file in the <literal>tar</literal> will
you give an idea of how entries are made.
give you an idea of how entries are made.
</para></listitem>
<listitem><para><literal>nntp_access/user_access</literal>:
@ -59,7 +61,7 @@ files are meant for.
<listitem><para><literal>log, errlog</literal>:
These are log files that keep growing large with each batch
that is received. The <literal>log</literal> file has one entry per
<literal>article</literal> telling you if it
article telling you if it
has been accepted by your news server or rejected. To understand the
format of this file, refer to Chapter 2.2 of the <literal>CNews</literal>
guide. Errors, if any, while digesting the articles are
@ -68,10 +70,10 @@ files are meant for.
</para></listitem>
<listitem><para><literal>nntplog</literal>:
This file logs information of the <literal>nntp daemon</literal> giving
This file logs information of the <literal>nntpd</literal> giving
details of when a connection was established/broken and what commands were
issued. This file needs to be configured in syslog and syslog
<literal>daemon</literal> should be running.
issued. This file needs to be configured in <literal>syslog</literal>
<literal>syslogd</literal> should be running.
</para></listitem>
<listitem><para><literal>active</literal>:
@ -83,25 +85,25 @@ files are meant for.
</para></listitem>
<listitem><para><literal>history</literal>:
This file, again, contains one line per <literal>article</literal>, mapping
This file, again, contains one line per article, mapping
<literal>message-id</literal> to newsgroup name and also giving its
associated <literal>article</literal> no. in that newsgroup. It is updated
associated article number in that newsgroup. It is updated
each time a feed is digested
and when <literal>doexpire</literal> is run. Plays a key role in
loop-detection and serves as an article database. Read manpage of
<literal>newsdb</literal>, <literal>doexpire</literal> for the file format
</para></listitem>
<listitem><para>newsgroups:
It has a one line description for each newsgroup explaining
<listitem><para><literal>newsgroups</literal>:
It has a one-line description for each newsgroup explaining
what kind of posts go into each of them. Ideally speaking, it should cover
all the newsgroups found in the <literal>active</literal> file.
</para></listitem>
<listitem><para>Miscellaneous files:
<listitem><para><emphasis>Miscellaneous files</emphasis>:
Files like <literal>mailname</literal>, <literal>organisation</literal>,
<literal>whoami</literal> contain information required for forming some of
the headers of an <literal>article</literal>. The contents of
the headers of an article. The contents of
<literal>mailname</literal> form the <literal>From:</literal> header and
that of <literal>organisation</literal> form the
<literal>Organisation:</literal> header. <literal>whoami</literal> contains
@ -133,16 +135,15 @@ give you an overview of this directory:
<listitem><para><literal>out.going</literal>:
This directory contains batches/feeds to be sent to your
NDNs i.e. feeds to be pushed to your neighbouring sites reside here
before they are transmitted. It contains one sub-directory per NDN mentioned
in the <literal>sys</literal> file. These sub-directories contain files
called <literal>togo</literal>
which contain information about the <literal>article</literal> like the
<literal>message-id</literal>
or the article no. that is queued for transmission.
NDNs <emphasis>i.e.</emphasis> feeds to be pushed to your neighbouring sites
reside here before they are transmitted. It contains one sub-directory per
NDN mentioned in the <literal>sys</literal> file. These sub-directories
contain files called <literal>togo</literal> which contain information about
the article like the <literal>message-id</literal> or the article number
that is queued for transmission.
</para></listitem>
<listitem><para><anchor id="newsgroupdir"/>newsgroup directories:
<listitem><para><anchor id="newsgroupdir"/><emphasis>newsgroup directories</emphasis>:
For each newsgroup hierarchy that the news server
has subscribed to, a directory is created under
<literal>$NEWSARTS</literal>.
@ -154,18 +155,17 @@ give you an overview of this directory:
<literal>comp</literal>. The <literal>music</literal> sub-directory
shall contain a further sub-directory called <literal>compose</literal> and
all articles of <literal>comp.music.compose</literal>
shall reside here. In effect,
<literal>article</literal> 242 of newsgroup
shall reside here. In effect, article 242 of newsgroup
<literal>comp.music.compose</literal> shall map to file
<literal>$NEWSARTS/comp/music/compose/242</literal>.
</para></listitem>
<listitem><para>control:
<listitem><para><literal>control</literal>:
The control directory houses only the control messages that
have been received by this site. The control messages could be any of the
following: <literal>newgroup, rmgroup, checkgroup</literal> and
<literal>cancel</literal>
appearing in the subject line of the <literal>article</literal>.
<literal>cancel</literal> appearing in the subject line of the article.
More information to be found in <quote><xref linkend="controlmsg"/></quote>
</para></listitem>
<listitem><para><literal>junk</literal>:
@ -173,14 +173,15 @@ give you an overview of this directory:
articles that the news
server has received and has decided, after processing, that it does not
belong to any of the hierarchies it has subscribed to. The news server
transfers/passes all <literal>articles</literal> in this directory to NDNs
transfers/passes all articles in this directory to NDNs
that have subscribed to the <literal>junk</literal> hierarchy.
</para></listitem>
</itemizedlist>
</para>
</section>
<section><title><literal>/usr/lib/newsbin</literal>: the executables</title>
<para></para>
<para>TO BE ADDED LATER</para>
</section>
<section id="cronjobs"><title><literal>crontab and cron jobs </literal></title>
@ -196,14 +197,14 @@ enough to be cronned. :)
The key script. This script picks the batches in the
<literal>in.coming</literal> directory, uncompresses them if necessary and
feeds it to <literal>relaynews</literal> which then processes each
<literal>article</literal> digesting and
batching them and logging any errors. This script needs to run through cron
article digesting and batching them and logging any errors. This script
needs to run through <literal>cron</literal>
as frequently as you want the feeds to be digested. Every half hour should
suffice for a non-critical requirement.
</para></listitem>
<listitem><para><literal>sendbatches</literal>:
This script is run to transmit the togo files formed in
This script is run to transmit the <literal>togo</literal> files formed in
the <literal>out.going</literal> directory to your NDNs. It reads the
<literal>batchparms</literal> file to know
exactly how and to whom the batches need to be transmitted. The frequency,
@ -219,11 +220,10 @@ enough to be cronned. :)
<listitem><para><literal>newswatch</literal>:
This looks for news problems at a more detailed level than
newsdaily like looking for persistent lock files, determining if there is
enough space for a minimum no. of files, if there is a huge queue of
unattended batches and the likes. This should typically run once every hour.
For more on this and the above, read the <literal>newsmaint</literal>
manpage.
newsdaily like looking for persistent lock files or unattended batches,
determining space shortage issues, and the likes. This should typically run
once every hour. For more on this and the above, read the
<literal>newsmaint</literal> manpage.
</para></listitem>
<listitem><para><literal>doexpire</literal>:
@ -252,13 +252,14 @@ articles in the <literal>in.coming</literal> directory of
</para>
<para>
<literal>relaynews</literal> picks up each <literal>article</literal> one by one
through stdin, determines if it belongs to a subscribed group by looking up
<literal>relaynews</literal> picks up each article one by one through
<literal>stdin</literal>, determines if it belongs to a subscribed group
by looking up
<literal>sys</literal> file, looks in the <literal>history</literal> file
to determine that it does not already exist locally, digests it updating the
<literal>active</literal> and <literal>history</literal> file and batches it
for neighbouring sites. Logs errors on encountering problems while processing
the <literal>article</literal> and takes appropriate action if it happens to be
the article and takes appropriate action if it happens to be
a control message. More info in manpage of <literal>relaynews</literal>.
</para>
</section>
@ -266,15 +267,17 @@ a control message. More info in manpage of <literal>relaynews</literal>.
<section><title><literal>doexpire</literal> and <literal>expire</literal>: removing old articles </title>
<para>
A good way to get rid of unwanted/old articles from the
<literal>$NEWSARTS</literal> area is to run doexpire once a day. It reads the
<literal>$NEWSARTS</literal> area is to run <literal>doexpire</literal> once a
day. It reads the
<literal>explist</literal> file from the <literal>$NEWSCTL</literal> directory
to determine what articles expire today. It can archive the
said <literal>article</literal> if so configured. It then updates the
said article if so configured. It then updates the
<literal>active</literal> and the <literal>history</literal> file accordingly.
If you wish to retain the <literal>article</literal> entry in the
If you wish to retain the article entry in the
<literal>history</literal> file to avoid re-digesting it as a new
article after having expired it add a special /expired/; line
in the control file. More on the options and functioning in the expire manpage.
article after having expired it, add a special <emphasis>/expired/;</emphasis>
line in the control file. More on the options and functioning in the
<literal>expire </literal> manpage.
</para>
</section>

58
LDP/howto/docbook/Usenet-News-HOWTO/conclusion.sgml
View File

@ -5,26 +5,27 @@
This HOWTO is a by-product of many years of experience setting up and
managing Usenet news servers. We have learned a lot from those who have
trod the path ahead of us. Some of them include the team of the ERNET
Project, which brought the Internet technology to India's academic
institutions in the early nineties. We specially remember what we have
learned from the SIGSys Group of the Department of Computer Science of
the Indian Institute of Technology, Mumbai. We have also benefited
Project (Educational and Research Network), which brought the Internet
technology to India's academic institutions in the early
nineties. We specially remember what we have learned from the
<emphasis>SIGSys</emphasis> Group of the Department of Computer Science
of the Indian Institute of Technology, Mumbai. We have also benefited
enormously from the guidance we received from the Networking Group at
the NCST in Mumbai, specially from Geetanjali Sampemane.
</para>
the NCST (National Centre for Software Technology) in Mumbai, specially
from Geetanjali Sampemane. </para>
<para>On a wider scale, our learning along the path of systems and
networks started with Unix, without which our appreciation of computer
systems would have remained very fragmented and superficial. Our
insight into Unix came from our village elders at the Department
of Computer Science of the IIT at Mumbai, specially from ``Hattu,''
``Sathe,'' and ``Sapre,'' none of which are with the IIT today, and from
Professor D.B.Phatak and others, many of whom, luckily are still with
the IIT.</para>
systems would have remained very fragmented and superficial. Our insight
into Unix came from our ``Village Elders'' in the Department of Computer
Science of the IIT (Indian Institute of Technology) at Mumbai, specially
from ``Hattu,'' ``Sathe,'' and ``Sapre,'' none of whom are with the IIT
today, and from Professor D.B.Phatak and others, many of whom, luckily
are still with the Institute.</para>
<para>Coming down to specifics, all the members of Starcom Software who
<para>Coming to Starcom, all the members of Starcom Software who
have worked on various problems with networking, Linux, and Usenet news
installations, have helped the authors in understanding what works and
installations have helped the authors in understanding what works and
what doesn't. Without their work, this HOWTO would have been a dry text
book.</para>
</section>
@ -39,7 +40,8 @@ acknowledged in the HOWTO.</para>
<section><title>Copyright</title>
<para>
Copyright (c) 2002 by Starcom Software Private Limited, India
<emphasis role=bold>Copyright (c) 2002 Starcom Software Private Limited,
India</emphasis>
</para>
<para>Please freely copy and distribute (sell or give away) this
@ -59,17 +61,18 @@ you:</para>
<listitem><para>
Send your derivative work (in the most suitable format such as SGML) to the
LDP (Linux Documentation Project) or the like for posting on the Internet.
If not the LDP, then let the LDP know where it is available.
If not the LDP, then let the LDP know where it is available.
</para></listitem>
<listitem><para>
License the derivative work with this same license or use GPL. Include a
copyright notice and at least a pointer to the license used.
License the derivative work with this same license or use GPL.
Include a copyright notice and at least a pointer to the licence
used.
</para></listitem>
<listitem><para>
Give due credit to previous authors and major contributors.
If you're considering making a derived work other than a
Give due credit to previous authors and major contributors.
If you are considering making a derived work other than a
translation, it is requested that you discuss your plans with the
current maintainer.
</para></listitem>
@ -84,13 +87,14 @@ products and solutions using Linux and Web technology since 1996. Our
entire office runs on Linux, and we have built mission-critical
solutions for some of the top corporate entities in India and abroad.
Our client list includes arguably the world's largest securities
depository (The National Securities Depository of India Limited), one of
the world's top five stock exchanges in terms of trading volumes (The
National Stock Exchange of India Limited), and one of India's premier
financial institutions, which is listed on the NYSE. In all these cases,
we have introduced them to Linux, and in many cases, we have built them
their first mission-critical business applications on Linux. Contact the
authors or check the Website for more information about the work we have done.
depository (The National Securities Depository Limited, India,
<literal>www.nsdl.com</literal>), one of the world's top five stock
exchanges in terms of trading volumes (The National Stock Exchange of
India Limited, <literal>www.nseindia.com</literal>), and one of India's
premier financial institutions listed on the NYSE. In all these cases, we
have introduced them to Linux, and in many cases, we have built them their
first mission-critical business applications on Linux. Contact the authors
or check the Starcom Website for more information about the work we have done.
</para>
</section>
</section>

112
LDP/howto/docbook/Usenet-News-HOWTO/doc.sgml
View File

@ -1,4 +1,8 @@
<section><title>Documentation and information</title>
<section><title>Documentation, information and further reading</title>
<para>This section fills in gaps which were hard to classify under any
of the previous chapters.</para>
<section><title>The manpages</title>
<para>The following manpages are installed automatically when our
@ -125,15 +129,72 @@ The NNTP batch transmit program for outgoing push feeds
</section>
<section><title>The C-News guide</title>
<section><title>Papers, documents, articles</title>
<para>There are certain documents and published conference papers which
are a must-read for Usenet server administrators, both for their
historical value and for the insight they give into Usenet server
architecture in general. We list our chart-toppers here.
</para>
<section><title>The Usenix paper on C News</title>
<para>This very interesting paper has been mentioned in the section titled
<quote><xref linkend="softwarehistory"/></quote>. It is titled ``News
Need Not Be Slow'', and is available from
<literal>ftp://ftp.cs.toronto.edu/doc/programming/c-news.*</literal> or
from our Website
(<literal>http://www.starcomsoftware.com/proj/usenet/doc/c-news.{ps,pdf}</literal>).
</para>
<para>It focuses on B News, analyses it for performance, and
demonstrates how specific changes in design and implementation can speed
things up. It is well-written, and is educative in many areas
independent of Usenet news.</para>
</section>
<section><title>The Usenix paper on INN</title>
<para>This paper talks about the things that C News didn't address,
and takes Usenet news processing into the world of pure Internet
connectivity. Its author is Rich Salz, the author of INN, and the paper
is titled ``InterNetNews: Usenet Transport for Internet Sites.'' This
can be picked up from
<literal>ftp://ftp.uu.net/networking/news/nntp/inn/inn.usenix.ps.Z</literal>
or from our Website
(<literal>http://www.starcomsoftware.com/proj/usenet/doc/inn.usenix.{ps,pdf}</literal>),
uncompressed.
Be warned: this PostScript file is probably missing some mandatory
first-line tag like <literal>%!PS-Adobe-1.0</literal> and some
PostScript processors can have problems with it. For instance, on our
Linux boxes, <literal>ghostview</literal> can display it, but
<literal>kghostview</literal> can't, which is very strange.
</para>
<para>This paper analyses the world of Usenet servers with C News and
NNTPd, in the presence of multiple parallel feeds, and proceeds to build
a case for a powerful NNTP-optimised software architecture which will
handle multiple parallel incoming NNTP feeds efficiently. What later INN
users appear to miss sometimes when comparing C-News+NNTPd with INN, is
that INN's strengths are <emphasis>only</emphasis> in situations which
its author had specifically targeted, <emphasis>i.e.</emphasis> multiple
parallel incoming NNTP feeds. There is no clear superiority of one
system over the other in any other situation.
</para>
</section>
<section><title>The C News guide</title>
<para>This document is part of the C-News source, and is available in
the <literal>c-news/doc</literal> directory of the source tree. The
<literal>makefile</literal> here uses <literal>troff</literal> and the
source files to generate <literal>guide.ps</literal>. This C-News Guide
is a very well-written document to provide an introduction to the
functioning of C-News.</para>
source files to generate <literal>guide.ps</literal>. This C News Guide
is a very well-written document and provides an introduction to the
functioning of C News.</para>
</section>
</section>
<section><title>O'Reilly's books on Usenet news</title>
@ -151,9 +212,9 @@ C-News and INN.</para>
<para>We have a distinct preference for books published by O'Reilly; we
usually find them the best books on their subjects. We make no attempts
to hide this bias. We recommend both books. We believe that there is
very little in this HOWTO of value to someone who studies one of these
books and then peruses information on the Internet.</para>
to hide this bias. We recommend both books. In fact, we believe that
there is very little of value in this HOWTO for someone who studies one
of these books and then peruses information on the Internet.</para>
</section>
@ -197,6 +258,32 @@ way to C-News) source, configuration and bugs (if any)
team to provide assistance to you by email, as a service to the Linux
and Usenet administrator community, on a best effort basis.</para>
<para>We also offer you an integrated source distribution
of C News, NNTPd, as discussed earlier in the section titled
<quote><xref linkend="settingup"/></quote>. This integrated
source distribution fixes some bugs in the component packages it
includes, and it comes pre-configured with ready made configuration
files which allow all components to be compiled and installed
on a Linux server in a manner by which they can work together
(<emphasis>e.g.</emphasis> key directory paths are specified consistently
across all components, <emphasis>etc.</emphasis>) This is available at
<literal>http://www.starcomsoftware.com/proj/usenet/src/</literal></para>
<para>The URL
<literal>http://www.starcomsoftware.com/proj/usenet/src/archives/</literal>
holds the original sources of some of the software components we base our
distribution on. These include C News (<literal>c-news.tar.Z</literal>),
NNTPd (<literal>nntp.1.5.12.1.tar.Z</literal>), and Nestor
(<literal>nestor.tar.Z</literal>). Other components, like
<literal>pgpverify</literal> are maintained by their current maintainers
and can be obtained from their respective sites. Therefore, they are not
included in our archives.</para>
<para>The URL
<literal>http://www.starcomsoftware.com/proj/usenet/doc/</literal>
carries copies of some of the important technical articles and Usenix
papers on the subject of the Usenet.</para>
<para>We will endeavour to answer all queries sent to
<literal>usenet@starcomsoftware.com</literal>, pertaining to the source
distribution we have put together and its configuration and maintenance,
@ -209,10 +296,11 @@ we do not have access to, <emphasis>e.g.</emphasis> SGI IRIX. Intel
Linux will be supported as long as our group is alive; our entire office
runs on Linux servers and diskless Linux desktops.</para>
<para>You do not need to be dependent on us, because neither do we have
proprietary knowledge nor proprietary closed-source software. All the
extensions we are currently involved in with C-News and NNTPd will
immediately be made available to the Internet.</para>
<para>You are not forced to be dependent on us, because neither do we
have proprietary knowledge nor proprietary closed-source software. All
the extensions we are currently involved in with C-News and NNTPd will
immediately be made available to the Internet in freely redistributable
source form.</para>
</section>
</section>

2
LDP/howto/docbook/Usenet-News-HOWTO/inn.sgml
View File

@ -12,7 +12,7 @@ released 7 May 2002. The full sources can be downloaded from
<section><title>Compiling and installing</title>
<para>TO BE ADDED LATER.</para>
<para>TO BE EXTENDED LATER.</para>
</section>

1
LDP/howto/docbook/Usenet-News-HOWTO/mail2news.sgml
View File

@ -28,7 +28,6 @@ archived till expired.
</para>
<section><title>Feeding Usenet news to email</title>
<para>
In CNews, this is trivially done by adding one line to the
<literal>sys</literal> file, defining a new outgoing feed listing all the

140
LDP/howto/docbook/Usenet-News-HOWTO/monitoring.sgml
View File

@ -1,44 +1,49 @@
<section><title>Monitoring and administration</title>
<para>
Once the Usenet News system is in place and running, the news administrator
is then aided in monitoring the system by various reports generated by the
system. Also, he needs to make regular checks in specific directories and
file to ascertain the smooth working of the system.
is then aided in monitoring the system by various reports generated by it.
Also, he needs to make regular checks in specific directories and
files to ascertain the smooth working of the system.
</para>
<section><title>The <literal>newsdaily</literal> report</title>
<para>
This report is generated by newsdaily which is typically run through cron. I
shall enumerate some of the problems reported based on what I have seen.
</para>
This report is generated by the script <literal>newsdaily</literal> which is
typically run through <literal>cron</literal>. I shall enumerate some of the
problems that are reported by it, based on my observations .</para>
<itemizedlist>
<listitem><para>bad input batches:
This reports articles that have been processed and
declared bad and hence not digested. The reason for it is not mentioned. You
are expected to check the article and determine the cause.
<listitem><para><emphasis>bad input batches</emphasis>:
This gives a list of articles that have been declared bad after processing
and hence have not been digested. The reason for this is not given. You
are expected to check each article and determine the cause.
</para></listitem>
<listitem><para>leading unknown newsgroups by articles:
This gives a list of newsgroups
whose hierarchy has been subscribed to, but the specific newsgroup does not
appear in the active file. You could add the newsgroup in the active file if
you think it is important enough.
<listitem><para><emphasis>leading unknown newsgroups by articles</emphasis>:
Newsgroup names that do not appear in the <literal>active</literal> file
but their hierarchy has been subscribed to, would find their names mentioned
under this heading. Choose to add the name in the active file if you think
it is important. For <emphasis>e.g.</emphasis>, you would see this happen
if you have subscribed to the hierarchy <literal>comp</literal> but the
<literal>active</literal> does not contain the newsgroup name
<literal>comp.lang.java.3d</literal>. You could deny subscription to this
particular newsgroup by specifying so in the <literal>sys</literal> file.
</para></listitem>
<listitem><para>leading unsubscribed newsgroups:
This gives a list of newsgroups
that have not been subscribed to, of which the news server receives a
maximum no. of articles. You really cannot do much about this except to
<listitem><para><emphasis>leading unsubscribed newsgroups</emphasis>:
If the news server receives maximum articles of a particular newsgroup
hierarchy to which you haven't subscribed, it will appear under this
heading. You really cannot do much about this except to
subscribe to them if they are required.
</para></listitem>
<listitem><para>leading sites sending bad headers:
<listitem><para><emphasis>leading sites sending bad headers</emphasis>:
This will list your NDNs who
are sending articles with malformed/insufficient headers.
</para></listitem>
<listitem><para>leading sites sending stale/future/misdated news:
<listitem><para><emphasis>leading sites sending stale/future/misdated news</emphasis>:
This will list your NDNs who are sending you articles that are older than
the date you have specified for accepting feeds.
</para></listitem>
@ -46,36 +51,39 @@ shall enumerate some of the problems reported based on what I have seen.
<listitem><para>Some of the reports generated by us:
We have modified the newsdaily script to include some more statistics.
<itemizedlist>
<listitem><para>disk usage:
<listitem><para><emphasis>disk usage</emphasis>:
This reports the size in bytes of the <literal>$NEWSARTS</literal>
area. If you are receiving feeds regularly, you should see this figure
increasing.
</para></listitem>
<listitem><para>incoming feed statistics:
This reports the no. of articles and total bytes recevied from each of
your NDNs.
<listitem><para><emphasis>incoming feed statistics</emphasis>:
This reports the number of articles and total bytes recevied from each
of your NDNs.
</para></listitem>
<listitem><para>NNTP traffic report:
The output of nestor has also been included in
this report which gives details of each nntp connection and the overall
performance of the network connection read from the newslog file.
To understand the format, read manpage of nestor.
<listitem><para><emphasis>NNTP traffic report</emphasis>:
The output of nestor has also been included in this report which gives
details of each <literal>nntp</literal> connection and the overall
performance of the network connection read from the <literal>newslog
</literal> file. To understand the format, read the manpage of
<literal>nestor</literal>.
</para></listitem>
</itemizedlist>
</para></listitem>
<listitem><para>Error reporting from the errorlog file:
Reports errors logged in the errorlog file. Usually these are file
ownership or file missing problems which can be easily handled.
<listitem><para><emphasis>Error reporting from the <literal>errorlog</literal>
file</emphasis>:
Reports errors logged in the <literal>errorlog</literal> file. Usually
these are file ownership or file missing problems which can be easily
handled.
</para></listitem>
</itemizedlist>
</section>
<section><title>Crisis reports from <literal>newswatch</literal></title>
<para>
Most of the problems reported to me are ones with either space shortage or
Most of the problems reported to me are those with either space shortage or
persistent locks. There are instances when the scripts have created locks files
and have aborted/terminated without removing them. Sometimes they are
innocuous enough to be deleted but this should be determined after a careful
@ -87,9 +95,9 @@ to determine why sendbatches was failing this often.
<para>
The space shortage issue has to be addressed immediately. You could
delete unwanted articles by running doexpire or add more disk space at the OS
level. The latter seems a better option.
<para>
delete unwanted articles by running <literal>doexpire</literal> or add more disk
space at the OS level.
</para>
</section>
<section><title>Disk space</title>
@ -98,18 +106,20 @@ The <literal>$NEWSBIN</literal> area occupies space that is fixed. Since the
binaries do not grow once installed, you do not have to worry about disk
shortage here. The areas that take up more space as feeds come in are
<literal>$NEWSCTL</literal> and <literal>$NEWSARTS</literal>. The
<literal>$NEWSCTL</literal> has log files that keep growing with each feed and
as the articles are digested in huge numbers the <literal>$NEWSARTS</literal>
continues to grow. Also, if articles are being archived on expiry you will need
space. Allocate a few GB of disk space for <literal>$NEWSARTS</literal>
depending on the no. of hierarchies you are subscribing and the feeds that come
in everyday. <literal>$NEWSCTL</literal> grows to a lesser proportion as
compared to <literal>$NEWSARTS</literal>. Allocate space for this accordingly.
<literal>$NEWSCTL</literal> has log files that keep growing with each feed.
As the articles are digested in huge numbers, the <literal>$NEWSARTS</literal>
area continues to grow. Also, you will need space if you have chosen to archive
articles on expiry. Allocate a few GB of disk space for
<literal>$NEWSARTS</literal> depending on the number of hierarchies you are
subscribing and the feeds that come in everyday. <literal>$NEWSCTL</literal>
grows to a lesser proportion as compared to <literal>$NEWSARTS</literal>.
Allocate space for this accordingly.
</para>
</section>
<section><title>CPU load and RAM usage</title>
<para>With modern C-News and NNTPd, there is very little usage of these
<para>
With modern C-News and NNTPd, there is very little usage of these
system resources for processing news article flow. Key components like
<literal>newsrun</literal> or <literal>sendbatches</literal> do not load
the system much, except for cases where you have a very heavy flow of
@ -118,9 +128,11 @@ compressed outgoing batches and the compression utility is run by
amazingly efficient in the current C-News release. Even when it takes
half an hour to digest a large consignment of batches, it hardly loads the
CPU of a slow Pentium 200 MHz CPU or consumes much RAM in a 64 MB
system.</para>
system.
</para>
<para>One thing which does slow down a system is a large bunch of
<para>
One thing which does slow down a system is a large bunch of
users connecting using NNTP to browse newsgroups. We do not have
heuristic based figures off-hand to provide a guidance figure for
resource consumption for this, but we have found that the load on the
@ -132,7 +144,8 @@ a dual-P-III Intel Linux server, for instance. This loading has no
bearing on whether you are using INN or <literal>nntpd</literal>;
both have practically identical implementations for NNTP
<emphasis>reading</emphasis> and differ only in their handling of
feeds.</para>
feeds.
</para>
<para>Another situation which will slow down your Usenet news server is
when downstream servers connect to you for pulling out NNTP feeds using
@ -143,19 +156,20 @@ your server's I/O system and CPU.</para>
<section><title>The <literal>in.coming/bad</literal> directory</title>
<para>
The in.coming directory is where the batches/articles reside when you have
The <literal>in.coming</literal> directory is where the batches/articles reside when you have
received feeds from your NDN and before processing happens. Checking this
directory regularly to see if there are batches is a good way of determining
that feeds are coming in. The batches and articles have different nomenclature.
Names like nntp.GxhsDj are indicative of batches and individual
articles are named beginning with digits like <literal>0.10022643380.t</literal>
Batches, typically, have names like <emphasis>nntp.GxhsDj</emphasis> and
individual articles are named beginning with digits like <emphasis>0.10022643380.t</emphasis>
</para>
<para>
The bad sub-directory under in.coming holds batches/articles that have
encountered errors when they were being processed by relaynews. You will have
to look into the directory for the cause of it. Ideally speaking, this
directory should be empty.
The <literal>bad</literal> sub-directory under <literal>in.coming</literal>
holds batches/articles that have encountered errors when they were being
processed by <literal>relaynews</literal>. You will have to look at the
individual files in this directory to determine the cause . Ideally speaking,
this directory should be empty.
</para>
</section>
@ -173,17 +187,19 @@ directory should be empty.
<section><title>The <literal>junk</literal> and <literal>control</literal> groups</title>
<para>
Control messages are those that have a newgroup/rmgroup/cancel/checkgroup in
their subject line. Such messages result in relaynews calling the appropriate
script and on execution a message is mailed to the admin about the action
taken. These control messages are stored in the control directory of
<literal>$NEWSARTS</literal>. For the propogation of such messages, one must
subscribe to the control hierarchy.
Control messages are those that have a <literal>newgroup/rmgroup/cancel/checkgroup</literal> in
their subject line. Such messages result in <literal>relaynews</literal> calling
the appropriate script and on execution a message is mailed to the admin about
the action taken. These control messages are stored in the
<literal>control</literal> directory of <literal>$NEWSARTS</literal>. For the
propogation of such messages, one must subscribe to the
<literal>control</literal> hierarchy.
</para>
<para>
When your news system determines that a certain article has not been subscribed
by you, it is 'junked' i.e. such articles appear in the junk directory. This
by you, it is <quote>junked</quote> i.e. such articles appear in the junk
directory. This
directory plays a key role in transferring articles to your NDNs as they would
subscribe to the junk hierarchy to receive feeds. If you are a leaf node, there
is no reason why articles should pile here. Keep deleting them on a daily

196
LDP/howto/docbook/Usenet-News-HOWTO/pop.sgml
View File

@ -504,12 +504,194 @@ is discussed in quite a bit of detail in Section XXX titled
</section>
</section>
<section id="controlmsg"> <title>Control messages</title>
<para>
(Discuss control messages. Show examples of actual control messages
if possible. Discuss security issues in the form of control message
storms, and how digital signatures are being used to tackle it. This
sets the ground for <literal>pgpverify</literal> later on.)
</para>
<section id="controlmsg"><title>Control messages</title>
<para>The Usenet is a massive dispersed collection of servers which
operate almost without any supervision, provided they have adequate disk
space and do not suffer disk corruption due to power failures,
<emphasis>etc.</emphasis> (It is indeed surprising how self-managing a
good Usenet server is, provided these two pre-requisites are met.) These
servers are each under the control of human administrators, but it is
preferable that certain routine actions be performed across all these
servers remotely from one location, without the manual intervention of
these humans.</para>
<para>One common need for centralised operations is the creation of new
groups in the standard eight hierarchies. The Usenet follows a fairly
formal process which asks for votes from readers worldwide before
deciding on the restructuring of its newsgroups list, including merging of
low-volume groups, splitting of high-volume groups into many specialised
groups, creating new groups, and even deleting groups. Once the voting
process for a change concludes and the change action is to be carried
out, it would be extremely tedious to send email to the hundreds of
thousands of Usenet administrators and hope that they make the changes
right, and answer their doubts if they get confused. It would be much
better to have an <emphasis>automatic</emphasis> way to make the
changes across all servers, of course with proper authorisation.</para>
<para>The solution to this does not lie in giving some central authority
the ability to run an OS-level command of his choice on all the world's
Usenet servers, because OS commands differ from OS to OS, and because
few Usenet administrators would trust a stranger from another part of
the world with OS level access. Therefore, the solution lay in defining
a small set of common Usenet maintenance actions, and permitting only
these actions to be triggered on all servers through the passing of
special command messages, called <emphasis role=bold>control
messages</emphasis>.</para>
<para>Control messages look like ordinary Usenet articles, more or less.
They have an extra header line, with its value in a specific format,
but they usually carry body text which looks like a normal human-written
article. Here is a control message (a spurious one at that, but it'll
do for now):</para>
<programlisting>
<![CDATA[
Xref: news.starcomsoftware.com control:814217
Path: news.starcomsoftware.com!linux594.dn.net!news.dn.hoopoo.com!
feed-out.newsfeeds.com!newsfeeds.com!feed.newsfeeds.com!
newsfeeds.com!news-spur1.maxwell.syr.edu!news.maxwell.syr.edu!
newsfeed.icl.net!newsfeed.skycache.com!Cidera!newsfeed.gamma.ru!
Gamma.RU!carrier.kiev.ua!goblin.nadrabank.kiev.ua!not-for-mail
From: tale@uunet.uu.net (David C Lawrence)
Newsgroups: news.groups,humanities.hipcrime
Subject: cmsg newgroup humanities.hipcrime
Control: newgroup humanities.hipcrime
Date: Sun, 18 Feb 2001 11:50:28 GMT
Organization: The Cabal
Lines: 20
Approved: tale@uunet.uu.net
Message-ID: <3afWYZTIR.G5YOC2@uunet.uu.net>
NNTP-Posting-Host: 203.145.147.67
X-Trace: goblin.nadrabank.kiev.ua 982528840 21455 203.145.147.67
(18 Feb 2001 20:40:40 GMT)
X-Complaints-To: usenet@nadrabank.kiev.ua
NNTP-Posting-Date: 18 Feb 2001 20:40:40 GMT
X-No-Archive: Yes
humanities.hipcrime is an unmoderated newsgroup which passed its
vote for creation by 326:10 as reported in news.announce.newgroups
on 18 Feb 2001.
For your newsgroups file:
humanities.hipcrime HipCrime for Humanity - you committed one now!
Anyone can create a newsgroup in the alt, biz, comp, earth,
humanities, misc, news, meow, rec, sci, soc, talk, us, or
any other Usenet hierarchy. New newsgroup proposals may be
optionally discussed in news.groups. Please be sure that your
/usr/lib/news/control.ctl is configured correctly:
## NEWGROUP MESSAGES
## honor them all and log in \${LOG}/newgroup.log
newgroup:*:alt.*|biz.*|comp.*|earth.*|humanities.*|misc.*|news.*|\
meaw.*|rec.*|sci.*|soc.*|talk.*|us.*:doit=newgroup
## RMGROUP MESSAGES
## drop them all and don't log
rmgroup:*:*:drop
Meow!
David C Lawrence
]]>
</programlisting>
<para>A control message must have a <literal>Control</literal>
header. Besides, all control messages <emphasis>will</emphasis>
have an <literal>Approved</literal> header, like messages posted
to moderated newsgroups. The <literal>Control</literal> header
actually specifies a command to run on the local server, and the
parameter(s) to supply to it. The local Usenet server software is
supposed to figure out its own way to get the task done. In this
example, the command in the <literal>Control</literal> header is
<literal>newgroup</literal>, which creates a new newsgroup. And its
parameter is <literal>humanities.hipcrime</literal>, which gives the
name of the newsgroup to create.</para>
<para>In C-News, the control message implementation works through
separate shellscripts kept in a fixed directory,
<literal>$NEWSBIN/ctl/</literal>, as a security measure; if the
executable script isn't present there, the control message command will
be ignored. The control message types supported are:</para>
<itemizedlist>
<listitem><para><literal>checkgroups</literal>: control message to
check whether the list of newsgroups in your active file are all correct
as per a master list of newsgroups sent in the control message
</para></listitem>
<listitem><para><literal>newgroup</literal>: control message to create a
new newsgroup
</para></listitem>
<listitem><para><literal>rmgroup</literal>: control message to delete a
newsgroup and all articles in it
</para></listitem>
<listitem><para><literal>sendsys</literal>: control message to cause an
email response to be sent to the author with the <literal>sys</literal>
file of your server in it. This results in a response storm of
emails from all the Usenet servers in the world to the author. These
responses allow the sender of the control message to analyse all the
<literal>sys</literal> files of the world's Usenet servers and create the
directed graph of Usenet newsfeeds. Why someone would want to do this is
hard to guess, but the result is surely an awesome picture of one facet
of networked human civilisation, like looking at a giant world map.
</para><para>Incidentally, there is no invasion of privacy here, because
your server's <literal>sys</literal> file is supposed to be public
information, if you take feeds from the public Usenet.
</para></listitem>
<listitem><para><literal>version</literal>: control message which results
in your Usenet software sending an email to the author of the message,
containing the type and version of the Usenet news software you are
using. This too is not an invasion of privacy, because this information
is supposed to be public knowledge.
</para></listitem>
<listitem><para>The cancel message: the most frequently occurring type of
control messages. They specify the message ID of an article, and result
in the cancellation (deletion) of that article. If you post an article
and regret it a moment later, your Usenet newsreader software usually
allows you to ``cancel'' it by generating a cancel message.
</para></listitem>
</itemizedlist>
<para>The Usenet news software maintains a pseudo-newsgroup called
<literal>control</literal>, where it files all control messages it
receives. If you have an incoming newsfeed from the public Usenet, your
server's <literal>control</literal> group will usually be full with
thousands of cancel messages from trigger-happy fingers all over the
world. Usenet news server software like C-News allows you to filter the
incoming feed based on newsgroups, and will discard articles for groups
they do not subscribe to. But since all servers have to receive and
process control messages, they will all accept these cancel messages,
though many of them may apply to articles which are not part of your
highly-pruned subset of groups. <literal>C'est la vie</literal>.</para>
<para>Remember to set expiry for the <literal>control</literal> group to
one day or even shorter, so that the junk can be cleaned out as rapidly as
possible, just like the <literal>junk</literal> newsgroup.</para>
<para>The beauty of the control message architecture is that it
integrates seamlessly into the newsfeed mechanism for automatic control
of the network of servers. No separate channel of connection is needed
for the control actions. And article replication automatically
propagates control messages with human-readable articles, thus
guaranteeing reach across heterogenous networks technologies.</para>
<para>What your Usenet server does on receiving a
control message is governed by an authorisation file:
<literal>$NEWSCTL/controlperms</literal> in the case of C-News
and <literal>control.ctl</literal> in the case of INN, for
instance. The security measures implemented by this module are
further enhanced by the <literal>pgpcontrol</literal> package with its
<literal>pgpverify</literal> script. Using <literal>pgpverify</literal>,
your server can check that all control messages (except for article
cancellation messages) are digitally signed by a trusted party
using military-spec public key cryptography. Our integrated
Usenet news software distribution includes integration with
<literal>pgpverify</literal>.</para>
</section>
</section>

218
LDP/howto/docbook/Usenet-News-HOWTO/security.sgml Normal file
View File

@ -0,0 +1,218 @@
<section><title>Security issues</title>
<para>It almost seems strange that we are discussing security issues in
the context of Usenet news servers. Usenet news has been one of the most
open and free-for-all network services traditionally. However, with the
exponential growth of the Internet, all services are becoming aware of
potential threats. The community of Internet intruders too has acquired
new profiles: a lot of Internet intrusion attempts are program-driven,
and exploit a set of ``well known'' vulnerabilities,
<emphasis>i.e.</emphasis> vulnerabilities which have been identified by
the computer security and intrusion community and published in their
reports and advisories. Thus, the question of ``Why will someone attack
my harmless Usenet server?'' is no longer valid. It will be attacked if
it can be attacked, merely because its IP address falls in a range of
addresses being targeted, perhaps.</para>
<para>Security issues for Usenet news servers fall into two categories.
First come vulnerabilities which will allow an attacker to bring down
your server or run code of his choice on it. Second come vulnerabilities
which can distort or corrupt your Usenet article hierarchy, either by
junk postings, unsolicited commercial messages, or forged control
messages. The second category of threats is specific to Usenet news and
needs Usenet-specific protection mechanisms, some of which require
tapping into defence mechanisms designed by the Usenet administrator
community.</para>
<section><title>Intrusion threats</title>
<para>Here we discuss the vulnerabilities which will allow an intruder
to ``gain control'' of your Usenet server, or ``bring it down,'' either
of which may be irritating, embarassing, or downright disastrous for your
business or occupation.</para>
<section><title>Generic server vulnerabilities</title>
<para>Foremost among these vulnerabilities are those which render
<emphasis>any</emphasis>server vulnerable to intrusion attempts.
Most of these vulnerabilities are unrelated to Usenet news itself. For
instance, if you have the Telnet service active on a server exposed to
the Internet, then it is likely that systematic attempts by intruders
to acquire usernames and passwords will bear fruit, using methods we
will best leave to specialised texts on the subject. Once this is done,
the intruder will merely ``walk into'' your server by Telnetting into
it.</para>
<para>We will not discuss this class of vulnerabilities here any further;
they belong in documents dedicated to general security issues. For
further reading, check the ``Security HOWTO'', the ``Security Quickstart
HOWTO'', the ``User Authentication HOWTO'', the ``VPN HOWTO'', and
the ``VPN Masquerade HOWTO'' ... and that's just from the Linux HOWTO
collection. As one can see, there is, if anything, a surfeit of material
on this and related subjects.</para>
<para>There are vulnerabilities which allow an intruder to mount the
so-called DoS attacks, which make your service inaccessible to
legitimate users, even though it does not let the intruder in. The most
publicised of these attacks were the SYNFlood and the Ping of Death
attacks, both quite old and well-understood by now. A Linux server
running a recent version of the kernel and properly configured, should
be immune to both these attack methods. But network protocols being
what they are, there are always new DoS methods being thought up, which
can temporarily overload or slow down a server. Once again, the texts
discussing generic security issues are the best place to study these
vulnerabilities.</para>
</section>
<section><title>Vulnerabilities in Usenet software</title>
<para>Then come server vulnerabilities, if any, which are caused
specifically by Usenet news software. For instance, if it was possible
for an intruder to issue some string of bytes to your server's NNTP
server and cause it to execute a command of the intruder's choice, then
this vulnerability would be in this category.</para>
<para>Any server which accepts a text string as input from a
client is open to the buffer overrun class of attacks, if the
<literal>gets()</literal> C library function has been used in its code
instead of the <literal>fgets()</literal> with a buffer size limit. This
was a vulnerability made famous by the 1988 Morris Internet Worm,
discussions on which can be found elsewhere. (Go Google for it if you're
keen.) As far as we know, the INN NNTP server and the
<literal>nntpd</literal> which forms part of the NNTP Reference
Implementation both have no known buffer overrun vulnerabilities. This
class of vulnerabilities is less significant in the case of NNTPd or
INN because these daemons do not run as <literal>root</literal>. In
fact, they would begin to cause malfunctioning of the underlying Usenet
software if they ran as <literal>root</literal>. Therefore, even if an
intrepid intruder could find some way of gaining control of these
daemons, she would only be able to get into the server as user
<literal>news</literal>, which means that she can play havoc with the
Usenet installation, but no further. A daemon which runs as
<literal>root</literal>, if compromised, can allow an intruder to take
control of the operating system itself.</para>
<para>UUCP is generally believed to be insecure. We believe a careful
configuration of Taylor UUCP plugs a lot of these vulnerabilities. One
vulnerability with UUCP over TCP is that the username and password travel
in plaintext form in TCP data streams, much like with Telnet or FTP. We
therefore do not advise using UUCP over TCP in this manner if security
is a concern at all. We recommend the use of UUCP through a SSH tunnel,
with the SSH setup working only with a pre-installed public key. This way,
there is no need for usernames and passwords for the SSH tunnel setup,
and passwords cannot be leaked even intentionally. And the UUCP username
and password then passes through this encrypted tunnel and is therefore
totally superfluous for security; the preceding SSH tunnel provides a much
stronger connection authentication than the UUCP username and password.
And since we set up our SSH tunnels to demand key-based authentication
only, it rejects any attempt to connect using usernames and passwords
when the tunnel is being set up.</para>
<para>A third possible vulnerability is related to the back-end software
which processes incoming Usenet articles. It is conceivable that an
NNTP server will receive an incoming <literal>POST</literal> command,
receive an article, and queue it for processing on the local spool;
the NNTP server often does not perform any real-time processing on the
incoming post. The post-processing software which periodically processes
the incoming spool (the <literal>in.coming</literal> directory in C-News)
will read this article and somehow be forced to run a command of the
intruder's choice, either by buffer overrun vulnerabilities or any
other means.</para>
<para>While this possibility exists, it appears that neither the C-News
<literal>newsrun</literal> and family nor INN are vulnerable to this class
of attempts. We base our comment on the solid evidence that both these
systems have been around in an intrusion-prone world of public Usenet
servers for more than a decade. INN, the newer of the two, completed
one decade of life on 20 August 2002. And both these software systems
had their source freely available to all, including intruders. We can be
fairly certain that if vulnerabilities of this class have not been seen,
it not for want of intrusion attempts.</para>
</section>
</section>
<section><title>Vulnerabilities unique to the Usenet service</title>
<para>There are certain security precautions that a Usenet server
administrator has to take to ensure that her servers are not swamped by
irritating junk or configured out of shape by spurious control messages.
These vulnerabilities do not allow an intruder to run her software on
your servers, but allows her to mess up your server, causing you to lose
a precious weekend (or week) straightening out the mess.</para>
<section><title>Unsolicited commercial messages</title>
<para>Unsolicited commercial messages are called SPAM. There is a
war against SPAM being fought in the Internet community. The biggest
battlefront is in the world of email. Second to that is Usenet
newsgroups.</para>
<para>There are many tools that Usenet administrators use in their
battle against SPAM. The most important of these is the NoCeM suite. See
<literal>http://www.cm.org/</literal> for details of NoCeM, and the
newsgroup <literal>alt.nocem.misc</literal> for the SPAM cancel messages
which NoCeM reads to identify which articles to discard. Your server
will need a feed of <literal>alt.nocem.misc</literal> to use the NoCeM
facility. These special messages are signed by NoCeM volunteers whose
job is to identify SPAM articles, list their message-IDs, and then
issue these deletion instruction, digitally signed with special private
keys, which tell all Usenet servers to delete the SPAM messages. Your
server's NoCeM software will need public key software (typically PGP)
and a keyring with the public key of each NoCeM volunteer you want to
accept instructions from.</para>
<para>Other anti-spam tools for Usenet services are listed in the
Anti-SPAM Software Web page
(<literal>http://www.exit109.com/~jeremy/news/antispam.html</literal>).
The <literal>Cleanfeed</literal> software will clean out articles
identified as SPAM. There are many others.</para>
<para>SPAM is such a nuisance and a drain on organisational expense
pockets (by wasting bandwidth you pay for) that it is almost imperative
today that every Usenet server protects itself against it. We will
integrate some selected anti-SPAM measures into our integrated source
distribution soon.</para>
</section>
<section><title>Spurious control messages</title>
<para>Control messages, discussed in detail earlier in <xref
linkend="controlmsg"/>, instruct a Usenet server to take certain
actions, like delete a message or create a newsgroup. If this facility
is ``open to the public'', anyone with half a brain can forge control
messages to create twenty new newsgroups, and then post thousands of
articles into those groups. In the mid-nineties, we were hit by a storm
of over 2,000 (two thousand) <literal>newgroup</literal> control
messages, which rapidly taught us the danger of unprotected control
messages and the protection against them.</para>
<para>The standard protection mechanism against this
vulnerability is <literal>pgpverify</literal>, which can be
downloaded from multiple Websites and FTP mirror sites by
searching for <literal>pgpverify</literal> (the program) or
<literal>pgpcontrol</literal> (the total software package). We have
integrated this into our source distribution, so that our C-News works
in a tightly coupled manner with <literal>pgpverify</literal>.</para>
<para><literal>pgpverify</literal> works using public key cryptography,
much like NoCeM, and all the official maintainers of respective Usenet
group hierarchies sign control messages using their private keys. Your
server will carry their public keys, and <literal>pgpverify</literal>
will check the sign on each control message to ensure that it's from the
official maintainer of the hierarchy. It will then act upon legit
control messages and discard the spurious ones.</para>
<para>In today's nuisance-ridden Usenet environment, no sane Usenet
server administrator receiving a feed of ``public'' hierarchies and
control messages will even dream of running her server without
<literal>pgpverify</literal> protection.</para>
</section>
</section>
</section>

332
LDP/howto/docbook/Usenet-News-HOWTO/settingup.sgml
View File

@ -1,4 +1,4 @@
<section><title>Setting up CNews + NNTPd</title>
<section id="settingup" xreflabel="Setting up C News + NNTPd"><title>Setting up CNews + NNTPd</title>
<section><title>Getting the sources and stuff</title>
@ -13,7 +13,7 @@ configuration and installation processes, which are very well
documented. The version that is available is Cleanup Release revision G,
on which our own version is based.</para>
<para>NNTPd is available from
<para>NNTPd (the NNTP Reference Implementation) is available from
<literal>ftp://ftp.uu.net/networking/news/nntp/nntp.1.5.12.1.tar.Z</literal>.
It has no automatic scripts and processes to configure itself. After
fetching the sources, you will have to follow a set of directions given
@ -32,7 +32,7 @@ periodically to process the logs of <literal>nntpd</literal>, the NNTP
server which is part of NNTPd, and report usage statistics to the
administrator. We have integrated Nestor into our source base.</para>
<para>The fourth piece of the puzzle, without which no Usenet server
<para>The fourth piece of the system, without which no Usenet server
administrator dares venture out into the wild world of public Internet
newsfeeds, is <literal>pgpverify</literal>.</para>
@ -41,7 +41,7 @@ have fixed a few bugs in both packages. We have also integrated the four
software systems listed above, and added a few features here and there to
make things work more smoothly. We offer our entire source base to
anyone for free download from
<literal>http://www.starcomsoftware.com/proj/news/src/news.tar.gz</literal>.
<literal>http://www.starcomsoftware.com/proj/usenet/src/news.tar.gz</literal>.
There are no licensing restrictions on our sources; they are as freely
redistributable as the original components we started with.</para>
@ -66,14 +66,6 @@ to find a directory tree with the following subdirectories and files:</para>
sources and Makefile.
</para></listitem>
<listitem><para><literal>archives</literal>: a directory containing the
tarballs of the original C-News, NNTPd, Nestor and
<literal>pgpverify</literal> source distributions, in case you want
them. Strictly speaking, the <literal>archive</literal> directory is
not necessary unless you want to study what changes we have made,
what files we have added, to the original sources.
</para></listitem>
<listitem><para><literal>build.sh</literal>: a shellscript you can run
to compile the entire combined source tree and install binaries in the
right places, if you are lucky and all goes well.
@ -118,17 +110,15 @@ original by just mirroring them.</para>
<section><title>Compiling and installing</title>
<para>
For installing, first make sure you have an entry for a user called
<literal>news</literal> in your <literal>/etc/password</literal> file. Add one if not present. This
<literal>news</literal> in your <literal>/etc/password</literal> file. This
is setting the news-database owner to <literal>news</literal>. Now download
the source from us and untar it in the home directory of news. This creates
two main directories <emphasis>viz.</emphasis> <literal>c-news</literal>
and <literal>nntp</literal>.
To install and compile, run the script <literal>build.sh</literal> as root
in the
directory that contains the script. It is important that the script run as
<literal>root</literal> as it sets ownerships and installs and compiles as
<literal>news</literal> and hence should have adequate permissions to do
this. This
in the directory that contains the script. It is important that the script
run as <literal>root</literal> as it sets ownerships, installs and
compiles the source as user <literal>news</literal>. This
is a one-step process that puts in place both the C-News and the
NNTP software, setting correct permissions and paths.
Following
@ -151,11 +141,11 @@ original by just mirroring them.</para>
</para></listitem>
<listitem><para>
Compiles <literal>C-News</literal> and exits on error. This builds
all the software. Writes the error into a file called <literal>make.out</literal>. Read it to
determine the cause. Also, performs regression tests if the
compilation was successfull and does not exit on error. Sends out a
warning to read the error file <literal>make.out.r</literal> and fix 'em.
Compiles <literal>C-News</literal> and performs regression tests if the
compilation was successfull. Sends out a warning to read the error file
<literal>make.out.r</literal> and to fix 'em.
Compilation erros are written to a file called <literal>make.out</literal>.
</para></listitem>
<listitem><para>
@ -163,11 +153,11 @@ original by just mirroring them.</para>
</para></listitem>
<listitem><para>
Checks the presence of the three key directories:
Checks for the presence of the three key directories:
<literal>$NEWSARTS - (/var/spool/news)</literal> that houses the artciles,
<literal>$NEWSCTL -(/var/lib/news)</literal> that contain
the configuration, log and status files and <literal>$NEWSBIN -
(/usr/lib/newsbin)</literal> that contain the binaries and
configuration, log and status files and <literal>$NEWSBIN -
(/usr/lib/newsbin)</literal> that contain binaries and
executables for
the working of the Usenet News system. Tries to create them if non-existent
and exits if it results in failure.
@ -181,18 +171,20 @@ original by just mirroring them.</para>
<listitem><para>
Then starts the installation process of C News. It runs
<literal>make install </literal>to install binaries at the right locations; <literal>make setup </literal>to set
the correct paths, umask, create directories for newsgroups, determine who
will receive reports; make ui to set up inews and injnews and
make readpostcheck to use readnews, postnews and checknews provided by
C News. The errors, if any are to be found in the respective make.out
files. e.g. make.setup will write errors to make.out.setup
<literal>make install </literal>to install binaries at the right locations;
<literal>make setup </literal>to set the correct paths and umask, create
directories for newsgroups, determine who will receive reports;
<literal>make ui</literal> to set up inews and injnews and
<literal>make readpostcheck </literal>to use readnews, postnews and
checknews scripts provided by C News. The errors, if any are to be found in
the respective <literal>make.out</literal> files. e.g. make.setup will write
errors to <literal>make.out.setup</literal>
</para></listitem>
<listitem><para>
<literal>Newsspool</literal> which queues incoming
<literal>Newsspool</literal>, which queues incoming
batches in <literal>$NEWSARTS/in.coming</literal> directory should run as
set-userid and set-groupid This is done.
set-userid and set-groupid. This is done.
</para></listitem>
<listitem><para>
@ -207,101 +199,94 @@ original by just mirroring them.</para>
<listitem><para>
Sets up the manpages for C News and makes it world
readable. The NNTP manpages get installed when the software is installed.
Compiles the C News documentation guide.ps and makes it readable and
available in <literal>/usr/doc/packages/news</literal> or
Compiles the C News documentation <literal>guide.ps</literal> and makes it
readable and available in <literal>/usr/doc/packages/news</literal> or
<literal>/usr/doc/news</literal>.
</para></listitem>
<listitem><para>
Checks for the PGP binary and asks the administrator to get
it if not found.
it, if not found.
</para></listitem>
</itemizedlist>
</section>
<section><title>Configuring the system: What and how to configure files?</title>
<section id="configuresystem"><title>Configuring the system: What and how to configure files?</title>
<para>Once installed, you have to now configure the system to accept feeds and
batch them for neighbours. You will have to do the following:</para>
batch them for your neighbours. You will have to do the following:</para>
<itemizedlist>
<listitem><para><literal>nntpd</literal>:
Copy the compiled nntpd into a directory where
Copy the compiled <literal>nntpd</literal> into a directory where
executables are kept and activate it. It runs on port 119 as a daemon
through inetd unless you have compiled it as stand-alone.
through <literal>inetd</literal> unless you have compiled it as stand-alone.
An entry in the <literal>/etc/services</literal> file for nntp would look
like this:
<screen>nntp 119/tcp \# Network News Transfer Protocol</screen>
An entry in the <literal>inetd.conf </literal>file will be:
<screen> nntp stream tcp nowait news path-to-tcpd path-to-nntpd </screen>
An entry in the services file for nntp would look like this:
<programlisting>nntp 119/tcp \# Network News Transfer Protocol</programlisting>
An entry in the inetd.conf file will be:
<programlisting>nntp stream tcp nowait news path-to-tcpd path-to-nntpd</programlisting>
The last two fields in the inetd.conf file are the paths to binaries of the
tcp daemon and the nntpd daemon respectively.
The last two fields in the <literal>inetd.conf</literal> file are paths to
binaries of the <literal>tcp</literal> and the <literal>nntp </literal>
daemon respectively.
</para></listitem>
<listitem><para><emphasis role=bold>Configuring control files:</emphasis>
There are plenty of control files in <literal>$NEWSCTL</literal> that will
need to be configured before you can
start using the news system. The files mentioned here are explained in some
detail in chapter 8, section 8.1. The files to be configured are dealt in
detail in the following section.
</para>
start using the news system. The files mentioned here are also discussed
in the first section of the section titled
<quote><xref linkend="component"/></quote>. These control files are
dealt in detail in the following below.</para>
<itemizedlist>
<listitem><para><literal>sys</literal>:
One line per system/NDN listing all the
newsgroup hierarchies each system subscribes to. Each line is prefixed
with the system name and the one beginning with ME: indicates what we
are going to receive. Following are typical entries that go into this
file:
<programlisting>ME:comp,news,misc,netscape</programlisting>
This line indicates what newsgroups your server, as determined by the
whoami file have subscribed to and will receive.
<programlisting>server/server.starcomsoftware.com:all,!general/all:f</programlisting>
This is a list of newsgroups this site will pass on to its NDN.
The newsgroups specified should be a comma separated list and no spaces
should be inserted in the whole line. The f flag indicates that the
newsgroup name and article no. alongwith its size will be one entry in
the togo file in the $NEWSARTS/out.going directory.
with the system name and the one beginning with
<screen>ME:</screen> indicates what your
server is willing to receive. Following are typical entries that go into
this file: <screen>ME:comp,news,misc,netscape</screen>
This line indicates what newsgroups your server has subscribed to.
<screen>server/server.starcomsoftware.com:all,!general/all:f</screen>
This is a list of newsgroups your server will pass on to your NDN.
The newsgroups specified should be a comma separated list and the entire
line should contain no spaces. The <emphasis>f</emphasis> flag indicates
that the newsgroup name and the article number alongwith its size will
make up one entry in the <literal>togo</literal> file in the
<literal>$NEWSARTS/out.going</literal> directory.
</para></listitem>
<listitem><para><literal>explist</literal>:
This file has entries indicating
which <literal>articles</literal> expire and when and if they have to be
archived The order in which the newsgroups are listed is important. An
example follows:
<programlisting>comp.lang.java.3d x 60 /var/spool/news/Archive</programlisting>
This file has entries indicating which articles expire and when and
whether they have to be archived. The order in which the newsgroups are
listed is important. An example follows:
<screen>comp.lang.java.3d x 60 /var/spool/news/Archive</screen>
This means that the articles of comp.lang.java expire after 60 days and
shall be archived in the directory mentioned as the fourth field.
shall be archived in the directory mentioned in the fourth field.
Archiving is an option. The second field indicates that this line
applies to both moderated and unmoderted newsgroups.
<emphasis>m</emphasis> would
specify moderated and <emphasis>u</emphasis> would specify unmoderated
groups. If you want to specify an extremely large no. as the expiry
period you can use the word 'never'.
period you can use the keyword <quote>never</quote>.
</para></listitem>
<listitem><para><literal>batchparms</literal>:
Sendbatches is a program that
adminsters batched transmission of news to other sites. To do this it
consults the batchparms file. Each line in the file specifies the
behaviour for each site. There are five fields for each site to be
<literal>sendbatches</literal> is a program that
administers batched transmission of news articles to other sites. To do
this it consults the <literal>batchparms</literal> file. Each line in
the file specifies the behaviour for each of your NDN mentioned in the
<literal>sys</literal> file. There are five fields for each site to be
specified.</para>
<screen>server u 100000 100 batcher | gzip -9 | viauux -d gunzip</screen>
<screen format="linespecific"> server u 100000 100 batcher | gzip -9 | viauux -d gunzip </screen>
<para>
The first field is the site name which matches the entry in the sys
file and has a corresponding directory in $NEWSARTS/out.going by that
name.
The first field is the site name which matches the entry in the
<literal>sys</literal> file and has a corresponding directory in
<literal>$NEWSARTS/out.going </literal>by that name.
</para>
<para>
The second field is the class of the site, 'u' for UUCP and 'n' for
NNTP feeds. A '!' in this field means that batching for this site has
been disabled.
The second field is the class of the site,<emphasis>u</emphasis> for
UUCP and <emphasis>n </emphasis>for NNTP feeds. A <quote>!</quote> in
this field means that batching for this site has been disabled.
</para>
<para>
The third field is the size of batches to be prepared in bytes.
@ -312,37 +297,39 @@ batch them for neighbours. You will have to do the following:</para>
</para>
<para>
The fifth field is the command line to be used to build, compress and
transmit batches to that site. It receives the contents of the togo file
on standard input.
transmit batches to that site. The contents of the
<literal>togo </literal> file are made available on standard input.
</para>
</listitem>
<listitem><para><literal>controlperm</literal>:
This file controls how the news
system responds to control messages. Each line consists of 4-5 fields
separated by white space.</para>
separated by white space. Control messages has been discussed in
<quote><xref linkend="controlmsg"/></quote>.
</para>
<programlisting>comp,sci tale@uunet.uu.net nrc pv news.announce.newsgroups</programlisting>
<screen>comp,sci tale@uunet.uu.net nrc pv news.announce.newsgroups</screen>
<para>
The first field is a newsgroup pattern to which the line applies.
</para>
<para>
The second field is either 'any' or an e-mail address. The latter
specifies that the line applies only to control messages from that
author.
The second field is either the keyword <quote>any</quote> or an e-mail
address. The latter specifies that the line applies to control messages
from only that author.
</para>
<para>
The third field is a set of opcode letters indicating what control
operations need to be performed on messages emanating from the e-mail
address mentioned in the second field. 'n' stands for creating a
newgroup, 'r' stands for deleting a newsgroup and 'c' stands for
checkgroup.
address mentioned in the second field. <emphasis>n</emphasis> stands for
creating a newgroup, <emphasis>r</emphasis> stands for deleting a
newsgroup and <emphasis>c</emphasis> stands for checkgroup.
</para>
<para>
The fourth field is a set of flag letters indicating how to respond to
a control message that meets all the applicability tests:
<screen>
<screen format="linespecific">
y Do it.
n Don't do it.
v Report it and include the entire control
@ -354,14 +341,13 @@ batch them for neighbours. You will have to do the following:</para>
</para>
<para>
The fifth field, which is optional, will be used if the fourth field
contains a 'p'. It must contain the PGP key ID of the public key to be
used for signature verification.
</para>
</listitem>
contains a <emphasis>p</emphasis>. It must contain the PGP key ID of the
public key to be used for signature verification.
</para> </listitem>
<listitem><para><literal>mailpaths</literal>:
This file describes how to reach
the moderators of various heirarchies of news groups by mail. Each line
the moderators of various hierarchies of newsgroups by mail. Each line
consists of two fields: a news group pattern and an e-mail address. The
first line whose group pattern matches the newsgroup is used. As an
example:
@ -371,8 +357,9 @@ batch them for neighbours. You will have to do the following:</para>
all %s@moderators.uu.net
</screen>
In the second example, the %s gets replaced with the groupname and all
dots appearing in the name are substituted with dashes.
In the second example, the <literal>%s</literal> gets replaced with the
groupname and all dots appearing in the newsgroup name are substituted
with dashes.
</para></listitem>
<listitem><para><emphasis role=bold>Miscellaneous files:</emphasis>
@ -384,14 +371,14 @@ batch them for neighbours. You will have to do the following:</para>
</para></listitem>
<listitem><para><literal>organization:</literal>
Contains the default value for the
Organization: header for postings originating locally.
Contains the default value for the <literal>Organization:</literal>
header for postings originating locally.
</para></listitem>
<listitem><para><literal>whoami:</literal>
Contains the name of the news system. This
is the site name used in the Path: headers and hence should concur
with the names your neighbours use in their sys files.
Contains the name of the news system. This is the site name used in
the <literal>Path:</literal> headers and hence should concur with
the names your neighbours use in their <literal>sys</literal> files.
</para></listitem>
</itemizedlist>
</para></listitem>
@ -401,10 +388,10 @@ batch them for neighbours. You will have to do the following:</para>
newsgroup (not just the hierarchy) to be found on your news system. You
will have to get the most recent copy of the active file from
<literal>ftp://ftp.isc.org/usenet/CONFIG/active</literal> and prune it
to delete
newsgroups that you have not subscribed to. Run the script "addgroup"
for each newsgroup in this file which will create relevant directories
in the <literal>$NEWSARTS</literal> area. The "addgroup" script takes
to delete newsgroups that you have not subscribed to. Run the script
<literal>addgroup</literal> for each newsgroup in this file which will
create relevant directories in the <literal>$NEWSARTS</literal> area.
The <literal>addgroup</literal> script takes
two paramters: the newsgroup name being created and a flag. The flag can
be any one of the following:
<screen>
@ -420,41 +407,42 @@ batch them for neighbours. You will have to do the following:</para>
An entry in this file looks like this:
<programlisting>comp.lang.java.3d 0000003716 01346 m </programlisting>
<screen>comp.lang.java.3d 0000003716 01346 m </screen>
The first field is the name of the newsgroup. The second field is the
highest article number that has been used in that newsgroup. The
third field is the lowest article number in the group. The fourth
field is flag as explained above.
field is a flag as explained above.
</para></listitem>
<listitem><para><literal>newsgroups </literal>file:
This contains a one line description
This contains a one-line description
of each newsgroup to be found in the active file. You will have to
get the most recent file from
<literal>ftp://ftp.isc.org/usenet/CONFIG/newsgroups</literal>
and prune it to remove unwanted information. As an example:
<programlisting>comp.lang.java.3d 3D Grphics APIs for the Java language</programlisting>
<screen>comp.lang.java.3d 3D Graphics APIs for the Java language</screen>
</para></listitem>
<listitem><para><emphasis role=bold>Create aliases: </emphasis>
<listitem><para><emphasis role=bold>Aliases: </emphasis>
These aliases are required for trouble reporting.
Once the system is in place and scripts are run, anomalies/problems
are reported to addresses in the /etc/aliases file. These entries
include email addresses for <literal>newsmaster, newscrisis, news,
usenet, newsmap</literal>
are reported to addresses in the <literal>/etc/aliases</literal> file.
These entries include email addresses for <literal>newsmaster,
newscrisis, news, usenet, newsmap</literal>.
They should ideally point to an email address that will be
looked at regularly. Arrange the emails for "newsmap" to be
discarded to minimize the effect of "sendsys bombing" by practical
jokers.
accessed at regularly. Arrange the emails for
<literal>newsmap</literal> to be discarded to minimize the effect of
<literal>sendsys bombing</literal> by practical jokers.
</para></listitem>
<listitem><para><emphasis role=bold>Cron jobs:</emphasis>
Certain scripts like newsrun that picks up incoming
Certain scripts like <literal>newsrun</literal> that picks up incoming
batches and maintenance scripts, should run through news-database
owner's cron. The cron entries ideally will be for the following: A more
detailed report can be found in <xref linkend="cronjobs"/>
owner's cron which is <literal>news</literal>. The cron entries ideally
will be for the following: A more detailed report can be found in
<quote><xref linkend="cronjobs"/></quote>
<orderedlist>
<listitem><para><literal>newsrun: </literal>
This script processes incoming batches of
@ -483,32 +471,33 @@ batch them for neighbours. You will have to do the following:</para>
</orderedlist>
</para></listitem>
<listitem><para>newslog:
Make an entry in the system's syslog.conf
file for logging messages spewed out by the nntp daemon in "newslog".
It should be located in <literal>$NEWSCTL</literal>. The entry will
look like this:
<listitem><para><literal>newslog: </literal>
Make an entry in the system's <literal>syslog.conf</literal>
file for logging messages spewed out by <literal>nntpd</literal> in
<literal>newslog </literal>. It should be located in
<literal>$NEWSCTL</literal>. The entry will look like this:
<programlisting>news.debug -/var/lib/news/newslog</programlisting>
<screen>news.debug -/var/lib/news/newslog</screen>
</para></listitem>
<listitem><para>Newsboot:
Have newsboot run (as "news", the
<listitem><para><literal>Newsboot: </literal>
Have this run (as <literal>news</literal> the
news-database owner) when the system boots to clear out debris left
around by crashes.
</para></listitem>
<listitem><para>Add a Usenet mailer in sendmail:
The mail2news program provided as
The <literal>mail2news</literal> program provided as
part of the source code is a handy tool to send an e-mail to a newsgroup
which gets digested as an article. You will have to add the following
ruleset and mailer definition in your sendmail.cf file:</para>
ruleset and mailer definition in your <literal>sendmail.cf </literal>
file:</para>
<itemizedlist>
<listitem><para>Under SParse1, add the following:
<programlisting>
<screen>
R$+ . USENET < @ $=w . > $#usenet $: $1
</programlisting>
</screen>
</para></listitem>
<listitem><para>Under mailer definitions, define the mailer Usenet as:
@ -520,27 +509,23 @@ batch them for neighbours. You will have to do the following:</para>
</itemizedlist>
<para>In order to send a mail to a newsgroup you will now have to suffix
the
newsgroup name with usenet <emphasis>i.e.</emphasis> your To: header
will look like this:
the newsgroup name with usenet <emphasis>i.e.</emphasis> your
<literal>To:</literal> header will look like this:
<screen>To: misc.test.usenet@yourdomain.</screen>
The mailer definition of usenet will intercept this mail and post it to
the respective newsgroup, in this case, misc.test</para>
the respective newsgroup, in this case, <literal>misc.test</literal>
</para>
</listitem>
</itemizedlist>
</itemizedlist>
<para>
This, more or less, completes the configuration part.
</para>
<para>This, more or less, completes the configuration part.</para>
</listitem>
</listitem>
</itemizedlist>
</section>
<section><title>Testing the system</title>
<para>
To locally test the system, follow the steps given below:
</para>
<para>To locally test the system, follow the steps given below:</para>
<itemizedlist>
<listitem><para>post an article:
Create a local newsgroup
@ -550,22 +535,23 @@ To locally test the system, follow the steps given below:
and using <literal>postnews </literal>post an article to it.
</para></listitem>
<listitem><para>Has it arrived in <literal>$NEWSARTS</literal>/in.coming?:
<listitem><para>Has it arrived in <literal>$NEWSARTS/in.coming</literal>?:
The article should show up in the directory mentioned. Note the nomenclature
of the article.
</para></listitem>
<listitem><para>When newsrun runs:
When newsrun runs through cron, the article disappears from in.coming
directory and appears in <literal>$NEWSARTS</literal>/mysite/test. Look how
the newsgroup, active, log and history (not the errorlog) files and
<literal> .overview </literal>file in
When newsrun runs from <literal>cron </literal>, the article disappears from
<literal>in.coming</literal> directory and appears in
<literal>$NEWSARTS/mysite/test</literal>. Look how
the <literal>newsgroup, active, log and history </literal>(not the errorlog)
files and <literal>.overview </literal>file in
<literal>$NEWSARTS/mysite/test</literal> reflect the digestion of the file
into the news system.
</para></listitem>
<listitem><para>reading the article:
Try to read the article through readnews or any
Try to read the article through <literal>readnews</literal> or any
news client. If you are able to, then you have set most everything right.
</para></listitem>
</itemizedlist>
@ -573,12 +559,13 @@ To locally test the system, follow the steps given below:
<section><title><literal>pgpverify</literal> and <literal>controlperms</literal></title>
<para>
As mentioned in <xref linkend="controlmsg"/>, it becomes necessary to
authenticate control messages to protect yourself from being attacked by
pranksters. For this, you will have to configure the
<literal>$NEWSCTL</literal>/controlperm file to declare whose control
As mentioned in <quote><xref linkend="controlmsg"/></quote>, it becomes
necessary to authenticate control messages to protect yourself from being
attacked by pranksters. For this, you will have to configure the
<literal>$NEWSCTL/controlperm </literal>file to declare whose control
messages you are willing to honour and for what newsgroups alongwith their
public key ID. The controlperm manpage shall give you details on the format.
public key ID. The <literal>controlperm</literal> manpage shall give you
details on the format.
</para>
<para>
@ -586,13 +573,13 @@ To locally test the system, follow the steps given below:
verifies the Usenet control messages that have been signed using the
<literal>signcontrol</literal> process. The script can be found at
<literal>ftp://ftp.isc.org/pub/pgpcontrol/pgpverify</literal>.
<literal> pgpverify </literal>pgpverify internally uses the PGP binary which
<literal> pgpverify </literal>internally uses the PGP binary which
will have to be made available in the default executables directory. If you
wish to send control messages for your local news system, you will have to
digitally sign them using the above mentioned "signcontrol" program which is
available at
digitally sign them using the above mentioned <literal>signcontrol</literal>
program which is available at
<literal>ftp://ftp.isc.org/pub/pgpcontrol/signcontrol</literal>. You will
also have to configure the signcontrol program accordingly.
also have to configure the <literal>signcontrol</literal> program accordingly.
</para>
</section>
@ -637,7 +624,8 @@ To locally test the system, follow the steps given below:
sites, you will have to configure <literal>sys</literal> and
<literal>batchparms</literal> with more entries. The number of directories
in <literal>$NEWSARTS/out.going</literal> shall increase, too. Refer
to chapter 8, section 8.1 and 8.2 for a better understanding of
to first two sections of the chapter titled
<quote><xref linkend="component"/></quote>for a better understanding of
outgoing feeds. Again, you will have to determine how you wish to
transmit the feed: UUCP or NNTP.
</para>
@ -650,7 +638,7 @@ To locally test the system, follow the steps given below:
<para>A full treatment of UUCP configuration is beyond the scope of
this document. However, the basic steps will be as follows. First,
you will have to define a ``system'' in your Usenet server for the
you will have to define a <quote>system</quote> in your Usenet server for the
NDN (next door neighbour) host. This definition will include various
parameters, including the manner in which your server will call the
remote server, the protocol it will use, <emphasis>etc.</emphasis>

181
LDP/howto/docbook/Usenet-News-HOWTO/software.sgml
View File

@ -1,40 +1,39 @@
<section><title>Usenet news software</title>
<section><title>CNews and NNTPd</title>
<para>
Once upon a time, when Usenet news was a term not yet invented, the
first recorded attempt to use a UUCP-based email backbone to maintain a
replicated message repository, was called A-News. It connected four
servers in four universities, and was written as Unix shell
scripts.</para>
<section><title>A brief history of Usenet systems</title>
<para>The designers of A-News had not anticipated how much load users
would put on their simplistic system. A far superior, more sophisticated,
and faster implementation of Usenet news was written later, called
B-News. This was a mix of C and shell scripts, and was designed
much better than A-News, to allow handling of much larger volumes of
messages. B-News v2.x was the current version in around 1990. By 1992 or
so, it had been surpassed by C-News.</para>
<para>Towards the end of this HOWTO, we have added some information
about the history of Usenet server software by quoting sections from an
earlier Usenet Periodic Posting. We consider this historical
perspective, and the Usenix papers and other documents referred to in
it, essential reading for any Usenet server administrator. Please see
the section titled <quote><xref linkend="softwarehistory"/></quote>.
</para>
</section>
<section><title>C-News and NNTPd</title>
<para>C-News was written by Henry Spencer and Geoff Collyer of the
Department of Zoology, University of Toronto, almost entirely in shell
and <literal>awk</literal>, as a replacement for B-News. Once again, the
focus was on adding some extra features and a lot of performance. The
first release was called Shellscript Release, which was deployed by a very
large number of servers worldwide, as a natural upgrade to B-News. This
version of C-News even had upward compatibility with B-News meta-data,
<emphasis>e.g.</emphasis> history files. This was the version of C-News
which was initially rolled out in 1992 or so at the National Centre for
Software Technology (NCST, <literal>http://www.ncst.ernet.in</literal>)
and the Indian Institute of Technologies in India as part of the Indian
ERNET network.</para>
and <literal>awk</literal>, as a replacement for an earlier system called
B-News. The focus was on adding some extra features and a
lot of performance. The first release was called Shellscript Release,
which was deployed by a very large number of servers worldwide, as
a natural upgrade to B-News. This version of C-News had upward
compatibility with B-News meta-data, <emphasis>e.g.</emphasis> history
files. This was the version of C-News which was initially rolled out
in 1991 or so at the National Centre for Software Technology (NCST,
<literal>http://www.ncst.ernet.in</literal>) and the Indian Institutes
of Technology in India as part of the Indian educational and research
network (ERNET). We received guidance from the NCST about Usenet news
installation and management.</para>
<para>The Shellscript Release was soon followed by a re-write with a lot
more C code, called Performance Release, and then a set of cleanup and
component integration steps leading to the last release called the
Cleanup Release. This Cleanup Release was revised many times, and the
last one was CR.G (Cleanup Release revision G). The version of C-News
discussed in this HOWTO is a set of bug fixes on CR.G.</para>
component integration steps leading to the last release called the Cleanup
Release. This Cleanup Release was patched many times by the authors,
and the last one was CR.G (Cleanup Release revision G). The version of
C-News discussed in this HOWTO is a set of small bug fixes on CR.G.</para>
<para>Since C-News came from shellscript-based antecedents, its
architecture followed the set-of-programs style so typical of Unix,
@ -51,7 +50,7 @@ command-line interfaces, called from scripts.</para>
<para>C-News was born in a world with widely varying network line speeds,
where bandwidth utilisation was a big issue and dialup links with UUCP
file transfers was common. Therefore, it has very strong support for
file transfers was common. Therefore, it has strong support for
batched feeds, specially with a variety of compression techniques and
over a variety of fast and slow transport channels. And C-News virtually
does not know the existence of TCP/IP, other than one or two tiny batch
@ -59,7 +58,7 @@ transport programs like <literal>viarsh</literal>. However, its design
was so modular that there was absolutely no problem in plugging in NNTP
functionality using a separate set of C programs without modifying
a single line of C-News. This was done by a program suite called
NNTPd.</para>
NNTP Reference Implementation, which we call NNTPd.</para>
<para>This software suite could work with B-News and C-News article
repositories, and provided the full NNTP functionality. Since B-News
@ -88,22 +87,20 @@ HTTP servers are run in most cases, and allows INN to cache a lot of
things in its memory, including message-IDs of recently posted messages,
<emphasis>etc.</emphasis> This interesting architecture has been discussed
in an interesting paper by the author, where he explains the problems
of the older BNews and CNews systems that he tried to address. Anyone
of the older B-News and C-News systems that he tried to address. Anyone
interested in Usenet software in general and INN in particular should
study this paper.</para>
study this paper.
</para>
<para>
INN addresses a Usenet news world which revolves around NNTP, though it
has support for UUCP batches --- a fact that not many INN administrators
seem to talk about. The primary situations where INN works at higher
efficiency over the CNews-NNTPd combination are in processing incoming
NNTP feeds when there are multiple incoming NNTP feeds. For multiple
readers reading and posting news over NNTP, there is no difference
between the efficiency of INN and NNTPd. <xref linkend="innefficiency"/>
discusses the efficiency issues of INN over the earlier CNews
architecture, based on Rich Salz' paper and our analyses of usage
patterns.
</para>
seem to talk about. INN works faster than the CNews-NNTPd combination when
processing multiple parallel incoming NNTP feeds. For multiple readers
reading and posting news over NNTP, there is no difference between the
efficiency of INN and NNTPd. <xref linkend="innefficiency"/> discusses
the efficiency issues of INN over the earlier C-News architecture, based
on Rich Salz' paper and our analyses of usage patterns. </para>
<para>
INN's architecture has inspired a lot of high-performance Usenet news
@ -119,7 +116,10 @@ This is an interesting software system, to set up a ``small'' Usenet
news server on one computer which only receives newsfeeds but does not
have the headache of sending out bulk feeds to other sites,
<emphasis>i.e.</emphasis> it is a ``leaf node'' in the newsfeed flow
diagram.</para>
diagram. According to its homepage (<literal>www.leafnode.org</literal>),
``Leafnode is a USENET software package designed for small sites running
any flavour of Unix, with a few tens of readers and only a slow link
to the net. [...] The current version is 1.9.24.''</para>
<para>This software is a sort of combination of article repository and
NNTP news server, and receives articles, digests and stores them on the
@ -139,6 +139,22 @@ space. Of course, ease of configuration and management is dependent on
familiarity, and we are more familiar with C-News than with Leafnode. We
hope this HOWTO will help you in that direction.</para>
<para>There <emphasis>is</emphasis>, however, one area in which Leafnode
is far easier to administer than INN or C-News. Leafnode constantly
monitors the actual usage of the newsgroups it carries, based on
readership statistics of its NNTP readers. If a particular newsgroup
is not read at all by any user for a week, then Leafnode will delete
all articles in that newsgroup, free up disk space, and stop fetching
new articles for it. If it finds that a previously abandoned newsgroup
is now again receiving attention, even from one user, then it'll fetch
all articles for that group from its upstream server the next time it
connects. This self-tuning feature of Leafnode is really an excellent
advantage which makes a Leafnode site easier to manage, specially for
small setups with bandwidth and disk space constraints.</para>
<para>The Leafnode Website gives a lot of details in an easily
understood format.</para>
<para>TO BE EXTENDED AND CORRECTED.</para>
</section>
@ -176,12 +192,6 @@ away.</para>
<section><title>Carrier class software</title>
<para>We have touched upon the characteristics of carrier-class Usenet
software in the section where we discuss NNTP efficiency issues. As that
bit shows, the requirements of carrier-class Usenet servers is very
different from those run within organisations and institutes for
providing internal service to their members.</para>
<para>Carrier-class servers are expected to handle a complete feed of all
articles in all newsgroups, including a lot of groups which have what we
call a ``high noise-to-signal ratio.'' They do not have the luxury of
@ -189,56 +199,41 @@ choosing a ``useful'' subset like administrators of internal corporate
Usenet servers do. Secondly, carrier-class servers are expected to turn
articles around very fast, <emphasis>i.e.</emphasis> they are expected to
have very low latency from the moment they receive an article to the
time they retransmit it by NNTP to downstream servers. Third, they are
supposed to provide very high availability, <emphasis>i.e.</emphasis>
they are supposed to be like other carrier class services. This usually
means that they have parallel arrays of computers in load sharing
configurations. And fourth, they usually do not cater to retail
connections for reading and posting articles by human users. Usenet news
carriers usually reserve separate computers to handle retail
connections.</para>
time they retransmit it by NNTP to downstream servers. Third, they
are supposed to provide very high availability, like other ``carrier
class'' services. This usually means that they have parallel arrays of
computers in load sharing configurations. And fourth, they usually do
not cater to retail connections for reading and posting articles by human
users. Usenet news carriers usually reserve separate computers to handle
retail connections.</para>
<para>Thus, carrier-class servers do not need to maintain a repository
of articles with the usual residence times of days or weeks, and expire
articles after they age. They only need to focus on super-efficient
re-transmission. These highly specialised servers have software
which receive an article over NNTP, parse it, and immediately re-queue
it for outward transmission to dozens or hundreds of other servers. And
since they work at these high throughputs, their downstream servers
are also expected to be live on the Internet round the clock to receive
incoming NNTP connections from the carrier servers. Therefore, there's
no batching or long queueing needed, and batching cannot be used. In
fact, some carrier class servers state that if you wish to receive feeds
from them, then your servers need to be available round the clock and
connected with lines fast enough to take the blast of a full feed. If
you do not fulfil these conditions, your servers will lose articles,
and the carrier is not responsible for the loss.</para>
of articles; they only need to focus on super-efficient real-time
re-transmission. These highly specialised servers have software which
receive an article over NNTP, parse it, and immediately re-queue it for
outward transmission to dozens or hundreds of other servers. And since
they work at these high throughputs, their downstream servers are also
expected to be live on the Internet round the clock to receive incoming
NNTP connections, or be prepared to lose articles. Therefore, there's
no batching or long queueing needed, and C-News-style batching in fact
is totally inapplicable.</para>
<para>Therefore, one can almost say that carrier-class servers have
neither article repositories nor queues other than the current message(s)
being re-transmitted. If they fail to connect to five of their fifty
downstream neighbours, or fail to push an article through due to
a transmit error, those five neighbours will never receive that
article later from this server; the article will be dropped from their
queues. Retries are not part of the game. Therefore, carrier-class
Usenet servers are more like packet routers than servers with
repositories.</para>
<para>Therefore, these carrier-class Usenet servers are more like packet
routers than servers with repositories. They are referred to nowadays as
NNTP routers or news routers.</para>
<para>It can be seen why carrier-class software cannot hope to do its
job using batch-oriented repository management software like C-News and
why it needs a totally NNTP-oriented implementation. Therefore, the INN
antecedents of some of these systems is to be expected. We would
<emphasis>love</emphasis> to hear from any Linux HOWTO reader whose
Usenet server needs include carrier-class behaviour.</para>
<para>It can be seen why batch-oriented repository management
software like C-News is a total anachronism here, and why they need an
NNTP-oriented, online, real-time design. The INN antecedents of some
of these systems is therefore natural. We would love to hear from any
Linux HOWTO reader whose Usenet server requirements include carrier-class
behaviour.</para>
<para>As far as we know, there is no freely redistributable software
implementation of carrier-class Usenet news servers. There is no reason
why such services cannot be offered on Linux, even Intel Linux, provided
you have fast network links and arrays of servers. Linux as an OS platform
is not an issue here, but free software has not yet been made available
for this niche. Presumably it is because the users of such software are
service providers who earn money using it, and therefore are expected
to be willing to pay for it.</para>
<para>We are aware of only one freely redistributable NNTP router:
NNTPRelay (see <literal>http://nntprelay.maxwell.syr.edu/</literal>); this
software runs on NT. There is no reason why such services cannot run off
Linux servers, even Intel Linux, provided you have fast network links and
arrays of servers. Linux as an OS platform is not an issue here.</para>
<para>TO BE EXTENDED AND CORRECTED.</para>