From f05766c9a0e2b9b1d4eb083e9da338e396577fe9 Mon Sep 17 00:00:00 2001 From: starcom <> Date: Wed, 21 Aug 2002 08:17:28 +0000 Subject: [PATCH] New chapter added on security issues in Usenet --- .../Usenet-News-HOWTO/accesscontrol.sgml | 11 +- .../docbook/Usenet-News-HOWTO/clients.sgml | 42 ++- .../docbook/Usenet-News-HOWTO/components.sgml | 91 ++--- .../docbook/Usenet-News-HOWTO/conclusion.sgml | 58 +-- LDP/howto/docbook/Usenet-News-HOWTO/doc.sgml | 112 +++++- LDP/howto/docbook/Usenet-News-HOWTO/inn.sgml | 2 +- .../docbook/Usenet-News-HOWTO/mail2news.sgml | 1 - .../docbook/Usenet-News-HOWTO/monitoring.sgml | 140 ++++---- LDP/howto/docbook/Usenet-News-HOWTO/pop.sgml | 196 ++++++++++- .../docbook/Usenet-News-HOWTO/security.sgml | 218 ++++++++++++ .../docbook/Usenet-News-HOWTO/settingup.sgml | 332 +++++++++--------- .../docbook/Usenet-News-HOWTO/software.sgml | 181 +++++----- 12 files changed, 951 insertions(+), 433 deletions(-) create mode 100644 LDP/howto/docbook/Usenet-News-HOWTO/security.sgml diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/accesscontrol.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/accesscontrol.sgml index 5f870435..f715f18d 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/accesscontrol.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/accesscontrol.sgml @@ -26,26 +26,25 @@ messages, and discussion archives. Details are given below.
Host-based access control - +TO BE ADDED LATER
User authentication and authorisation -
The NNTPd password file - + TO BE ADDED LATER
Mapping users to newsgroups - + TO BE ADDED LATER
The <literal>X-Authenticated-Author</literal> article header - + TO BE ADDED LATER
Other article header additions - + TO BE ADDED LATER
diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/clients.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/clients.sgml index 0a20c5af..10ce5c54 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/clients.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/clients.sgml @@ -17,10 +17,11 @@ Usenet User Agents, along the lines of MUA for Mail User Agents. -There are other special-purpose clients, which either pull out articles -to copy or transfer somewhere else, or for analysis, e.g. a -search engine which allows you to search a Usenet article archive, like Google -(www.google.com) does. +There are other special-purpose clients, which either pull out +articles to copy or transfer somewhere else, or for analysis, +e.g. a search engine which allows you to search a +Usenet article archive, like Google (www.google.com) +does. @@ -39,15 +40,15 @@ transfer articles or do other special-purpose things with them.
Usenet User Agents
Accessing articles: NNTP or spool area? - + TO BE ADDED LATER
Threading - + TO BE ADDED LATER
Quick reading features - + TO BE ADDED LATER
@@ -62,6 +63,31 @@ to take contributed additions that discuss other client software.
Special clients - +
NNTPCache + NNTPCache is an interesting transparent cacheing proxy for + news articles. News articles are read-only by definition, + i.e. they do not change once they are posted; + they can only be deleted. NNTPCache uses this feature to build a + local cache of news articles. + + You set up NNTPCache to listen on the NNTP port of your local + Unix server, and act like an NNTP daemon. You configure it to + connect back-to-back to another NNTP daemon, further away, which has + all the interesting stuff the users want to read. When a user + connects to the local NNTPCache, it connects to the remote NNTP + server and acts as a relay for the NNTP connection, ferrying + commands and responses back and forth. What the user sees therefore + comes from the remote server, the first time. However, all news + articles fetched by NNTPCache are also stored in a local cache, thus + allowing the next user to browse the same set of articles faster. + Like all demand-driven caches, the advantage here is that the local + NNTPCache does not need (much) administering, and will automatically + delete all articles from its cache once they've been lying unread + long enough. + + We list it here as an NNTP client because every proxy server + is a server on one side and a client on the other. +
+
diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/components.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/components.sgml index 12ceece6..6d074f33 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/components.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/components.sgml @@ -1,4 +1,5 @@ -
Components of a running system +
+Components of a running system This chapter reviews the components of a running CNews+NNTPd server. Analogous components will be found in an INN-based system too. We invite @@ -11,7 +12,8 @@ chapter. This directory is more popularly known as $NEWSCTL. It contains configuration, log and status files. There are no articles or binaries kept here. Let's see what some of the -files are meant for. +files are meant for. Control files are dealt in slightly greater detail in + @@ -47,7 +49,7 @@ files are meant for. newsgroup who is responsible for approving/disapproving articles posted to moderated newsgroups. The sample mailpaths file in the tar will - you give an idea of how entries are made. + give you an idea of how entries are made. nntp_access/user_access: @@ -59,7 +61,7 @@ files are meant for. log, errlog: These are log files that keep growing large with each batch that is received. The log file has one entry per - article telling you if it + article telling you if it has been accepted by your news server or rejected. To understand the format of this file, refer to Chapter 2.2 of the CNews guide. Errors, if any, while digesting the articles are @@ -68,10 +70,10 @@ files are meant for. nntplog: - This file logs information of the nntp daemon giving + This file logs information of the nntpd giving details of when a connection was established/broken and what commands were - issued. This file needs to be configured in syslog and syslog - daemon should be running. + issued. This file needs to be configured in syslog + syslogd should be running. active: @@ -83,25 +85,25 @@ files are meant for. history: - This file, again, contains one line per article, mapping + This file, again, contains one line per article, mapping message-id to newsgroup name and also giving its - associated article no. in that newsgroup. It is updated + associated article number in that newsgroup. It is updated each time a feed is digested and when doexpire is run. Plays a key role in loop-detection and serves as an article database. Read manpage of newsdb, doexpire for the file format -newsgroups: - It has a one line description for each newsgroup explaining +newsgroups: + It has a one-line description for each newsgroup explaining what kind of posts go into each of them. Ideally speaking, it should cover all the newsgroups found in the active file. -Miscellaneous files: +Miscellaneous files: Files like mailname, organisation, whoami contain information required for forming some of - the headers of an article. The contents of + the headers of an article. The contents of mailname form the From: header and that of organisation form the Organisation: header. whoami contains @@ -133,16 +135,15 @@ give you an overview of this directory: out.going: This directory contains batches/feeds to be sent to your - NDNs i.e. feeds to be pushed to your neighbouring sites reside here - before they are transmitted. It contains one sub-directory per NDN mentioned - in the sys file. These sub-directories contain files - called togo - which contain information about the article like the - message-id - or the article no. that is queued for transmission. + NDNs i.e. feeds to be pushed to your neighbouring sites + reside here before they are transmitted. It contains one sub-directory per + NDN mentioned in the sys file. These sub-directories + contain files called togo which contain information about + the article like the message-id or the article number + that is queued for transmission. -newsgroup directories: +newsgroup directories: For each newsgroup hierarchy that the news server has subscribed to, a directory is created under $NEWSARTS. @@ -154,18 +155,17 @@ give you an overview of this directory: comp. The music sub-directory shall contain a further sub-directory called compose and all articles of comp.music.compose - shall reside here. In effect, - article 242 of newsgroup + shall reside here. In effect, article 242 of newsgroup comp.music.compose shall map to file $NEWSARTS/comp/music/compose/242. -control: +control: The control directory houses only the control messages that have been received by this site. The control messages could be any of the following: newgroup, rmgroup, checkgroup and - cancel - appearing in the subject line of the article. + cancel appearing in the subject line of the article. + More information to be found in junk: @@ -173,14 +173,15 @@ give you an overview of this directory: articles that the news server has received and has decided, after processing, that it does not belong to any of the hierarchies it has subscribed to. The news server - transfers/passes all articles in this directory to NDNs + transfers/passes all articles in this directory to NDNs that have subscribed to the junk hierarchy. +
<literal>/usr/lib/newsbin</literal>: the executables - +TO BE ADDED LATER
<literal>crontab and cron jobs </literal> @@ -196,14 +197,14 @@ enough to be cronned. :) The key script. This script picks the batches in the in.coming directory, uncompresses them if necessary and feeds it to relaynews which then processes each - article digesting and - batching them and logging any errors. This script needs to run through cron + article digesting and batching them and logging any errors. This script + needs to run through cron as frequently as you want the feeds to be digested. Every half hour should suffice for a non-critical requirement. sendbatches: - This script is run to transmit the togo files formed in + This script is run to transmit the togo files formed in the out.going directory to your NDNs. It reads the batchparms file to know exactly how and to whom the batches need to be transmitted. The frequency, @@ -219,11 +220,10 @@ enough to be cronned. :) newswatch: This looks for news problems at a more detailed level than - newsdaily like looking for persistent lock files, determining if there is - enough space for a minimum no. of files, if there is a huge queue of - unattended batches and the likes. This should typically run once every hour. - For more on this and the above, read the newsmaint - manpage. + newsdaily like looking for persistent lock files or unattended batches, + determining space shortage issues, and the likes. This should typically run + once every hour. For more on this and the above, read the + newsmaint manpage. doexpire: @@ -252,13 +252,14 @@ articles in the in.coming directory of -relaynews picks up each article one by one -through stdin, determines if it belongs to a subscribed group by looking up +relaynews picks up each article one by one through +stdin, determines if it belongs to a subscribed group +by looking up sys file, looks in the history file to determine that it does not already exist locally, digests it updating the active and history file and batches it for neighbouring sites. Logs errors on encountering problems while processing -the article and takes appropriate action if it happens to be +the article and takes appropriate action if it happens to be a control message. More info in manpage of relaynews.
@@ -266,15 +267,17 @@ a control message. More info in manpage of relaynews.
<literal>doexpire</literal> and <literal>expire</literal>: removing old articles A good way to get rid of unwanted/old articles from the -$NEWSARTS area is to run doexpire once a day. It reads the +$NEWSARTS area is to run doexpire once a +day. It reads the explist file from the $NEWSCTL directory to determine what articles expire today. It can archive the -said article if so configured. It then updates the +said article if so configured. It then updates the active and the history file accordingly. -If you wish to retain the article entry in the +If you wish to retain the article entry in the history file to avoid re-digesting it as a new -article after having expired it add a special /expired/; line -in the control file. More on the options and functioning in the expire manpage. +article after having expired it, add a special /expired/; +line in the control file. More on the options and functioning in the +expire manpage.
diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/conclusion.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/conclusion.sgml index 48ea9265..7ced05b4 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/conclusion.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/conclusion.sgml @@ -5,26 +5,27 @@ This HOWTO is a by-product of many years of experience setting up and managing Usenet news servers. We have learned a lot from those who have trod the path ahead of us. Some of them include the team of the ERNET -Project, which brought the Internet technology to India's academic -institutions in the early nineties. We specially remember what we have -learned from the SIGSys Group of the Department of Computer Science of -the Indian Institute of Technology, Mumbai. We have also benefited +Project (Educational and Research Network), which brought the Internet +technology to India's academic institutions in the early +nineties. We specially remember what we have learned from the +SIGSys Group of the Department of Computer Science +of the Indian Institute of Technology, Mumbai. We have also benefited enormously from the guidance we received from the Networking Group at -the NCST in Mumbai, specially from Geetanjali Sampemane. - +the NCST (National Centre for Software Technology) in Mumbai, specially +from Geetanjali Sampemane. On a wider scale, our learning along the path of systems and networks started with Unix, without which our appreciation of computer -systems would have remained very fragmented and superficial. Our -insight into Unix came from our village elders at the Department -of Computer Science of the IIT at Mumbai, specially from ``Hattu,'' -``Sathe,'' and ``Sapre,'' none of which are with the IIT today, and from -Professor D.B.Phatak and others, many of whom, luckily are still with -the IIT. +systems would have remained very fragmented and superficial. Our insight +into Unix came from our ``Village Elders'' in the Department of Computer +Science of the IIT (Indian Institute of Technology) at Mumbai, specially +from ``Hattu,'' ``Sathe,'' and ``Sapre,'' none of whom are with the IIT +today, and from Professor D.B.Phatak and others, many of whom, luckily +are still with the Institute. -Coming down to specifics, all the members of Starcom Software who +Coming to Starcom, all the members of Starcom Software who have worked on various problems with networking, Linux, and Usenet news -installations, have helped the authors in understanding what works and +installations have helped the authors in understanding what works and what doesn't. Without their work, this HOWTO would have been a dry text book.
@@ -39,7 +40,8 @@ acknowledged in the HOWTO.
Copyright -Copyright (c) 2002 by Starcom Software Private Limited, India +Copyright (c) 2002 Starcom Software Private Limited, +India Please freely copy and distribute (sell or give away) this @@ -59,17 +61,18 @@ you: Send your derivative work (in the most suitable format such as SGML) to the LDP (Linux Documentation Project) or the like for posting on the Internet. - If not the LDP, then let the LDP know where it is available. + If not the LDP, then let the LDP know where it is available. - License the derivative work with this same license or use GPL. Include a - copyright notice and at least a pointer to the license used. + License the derivative work with this same license or use GPL. + Include a copyright notice and at least a pointer to the licence + used. - Give due credit to previous authors and major contributors. - If you're considering making a derived work other than a + Give due credit to previous authors and major contributors. + If you are considering making a derived work other than a translation, it is requested that you discuss your plans with the current maintainer. @@ -84,13 +87,14 @@ products and solutions using Linux and Web technology since 1996. Our entire office runs on Linux, and we have built mission-critical solutions for some of the top corporate entities in India and abroad. Our client list includes arguably the world's largest securities -depository (The National Securities Depository of India Limited), one of -the world's top five stock exchanges in terms of trading volumes (The -National Stock Exchange of India Limited), and one of India's premier -financial institutions, which is listed on the NYSE. In all these cases, -we have introduced them to Linux, and in many cases, we have built them -their first mission-critical business applications on Linux. Contact the -authors or check the Website for more information about the work we have done. +depository (The National Securities Depository Limited, India, +www.nsdl.com), one of the world's top five stock +exchanges in terms of trading volumes (The National Stock Exchange of +India Limited, www.nseindia.com), and one of India's +premier financial institutions listed on the NYSE. In all these cases, we +have introduced them to Linux, and in many cases, we have built them their +first mission-critical business applications on Linux. Contact the authors +or check the Starcom Website for more information about the work we have done.
diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/doc.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/doc.sgml index 7f199f23..1ee087c7 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/doc.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/doc.sgml @@ -1,4 +1,8 @@ -
Documentation and information +
Documentation, information and further reading + +This section fills in gaps which were hard to classify under any +of the previous chapters. +
The manpages The following manpages are installed automatically when our @@ -125,15 +129,72 @@ The NNTP batch transmit program for outgoing push feeds
-
The C-News guide +
Papers, documents, articles + +There are certain documents and published conference papers which +are a must-read for Usenet server administrators, both for their +historical value and for the insight they give into Usenet server +architecture in general. We list our chart-toppers here. + + +
The Usenix paper on C News + +This very interesting paper has been mentioned in the section titled +. It is titled ``News +Need Not Be Slow'', and is available from +ftp://ftp.cs.toronto.edu/doc/programming/c-news.* or +from our Website +(http://www.starcomsoftware.com/proj/usenet/doc/c-news.{ps,pdf}). + + +It focuses on B News, analyses it for performance, and +demonstrates how specific changes in design and implementation can speed +things up. It is well-written, and is educative in many areas +independent of Usenet news. + +
+ +
The Usenix paper on INN + +This paper talks about the things that C News didn't address, +and takes Usenet news processing into the world of pure Internet +connectivity. Its author is Rich Salz, the author of INN, and the paper +is titled ``InterNetNews: Usenet Transport for Internet Sites.'' This +can be picked up from +ftp://ftp.uu.net/networking/news/nntp/inn/inn.usenix.ps.Z +or from our Website +(http://www.starcomsoftware.com/proj/usenet/doc/inn.usenix.{ps,pdf}), +uncompressed. +Be warned: this PostScript file is probably missing some mandatory +first-line tag like %!PS-Adobe-1.0 and some +PostScript processors can have problems with it. For instance, on our +Linux boxes, ghostview can display it, but +kghostview can't, which is very strange. + + +This paper analyses the world of Usenet servers with C News and +NNTPd, in the presence of multiple parallel feeds, and proceeds to build +a case for a powerful NNTP-optimised software architecture which will +handle multiple parallel incoming NNTP feeds efficiently. What later INN +users appear to miss sometimes when comparing C-News+NNTPd with INN, is +that INN's strengths are only in situations which +its author had specifically targeted, i.e. multiple +parallel incoming NNTP feeds. There is no clear superiority of one +system over the other in any other situation. + + +
+ +
The C News guide This document is part of the C-News source, and is available in the c-news/doc directory of the source tree. The makefile here uses troff and the -source files to generate guide.ps. This C-News Guide -is a very well-written document to provide an introduction to the -functioning of C-News. +source files to generate guide.ps. This C News Guide +is a very well-written document and provides an introduction to the +functioning of C News. +
O'Reilly's books on Usenet news @@ -151,9 +212,9 @@ C-News and INN. We have a distinct preference for books published by O'Reilly; we usually find them the best books on their subjects. We make no attempts -to hide this bias. We recommend both books. We believe that there is -very little in this HOWTO of value to someone who studies one of these -books and then peruses information on the Internet. +to hide this bias. We recommend both books. In fact, we believe that +there is very little of value in this HOWTO for someone who studies one +of these books and then peruses information on the Internet.
@@ -197,6 +258,32 @@ way to C-News) source, configuration and bugs (if any) team to provide assistance to you by email, as a service to the Linux and Usenet administrator community, on a best effort basis. +We also offer you an integrated source distribution +of C News, NNTPd, as discussed earlier in the section titled +. This integrated +source distribution fixes some bugs in the component packages it +includes, and it comes pre-configured with ready made configuration +files which allow all components to be compiled and installed +on a Linux server in a manner by which they can work together +(e.g. key directory paths are specified consistently +across all components, etc.) This is available at +http://www.starcomsoftware.com/proj/usenet/src/ + +The URL +http://www.starcomsoftware.com/proj/usenet/src/archives/ +holds the original sources of some of the software components we base our +distribution on. These include C News (c-news.tar.Z), +NNTPd (nntp.1.5.12.1.tar.Z), and Nestor +(nestor.tar.Z). Other components, like +pgpverify are maintained by their current maintainers +and can be obtained from their respective sites. Therefore, they are not +included in our archives. + +The URL +http://www.starcomsoftware.com/proj/usenet/doc/ +carries copies of some of the important technical articles and Usenix +papers on the subject of the Usenet. + We will endeavour to answer all queries sent to usenet@starcomsoftware.com, pertaining to the source distribution we have put together and its configuration and maintenance, @@ -209,10 +296,11 @@ we do not have access to, e.g. SGI IRIX. Intel Linux will be supported as long as our group is alive; our entire office runs on Linux servers and diskless Linux desktops. -You do not need to be dependent on us, because neither do we have -proprietary knowledge nor proprietary closed-source software. All the -extensions we are currently involved in with C-News and NNTPd will -immediately be made available to the Internet. +You are not forced to be dependent on us, because neither do we +have proprietary knowledge nor proprietary closed-source software. All +the extensions we are currently involved in with C-News and NNTPd will +immediately be made available to the Internet in freely redistributable +source form.
diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/inn.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/inn.sgml index b67028a3..65026123 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/inn.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/inn.sgml @@ -12,7 +12,7 @@ released 7 May 2002. The full sources can be downloaded from
Compiling and installing -TO BE ADDED LATER. +TO BE EXTENDED LATER.
diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/mail2news.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/mail2news.sgml index 91ece49c..128e8b54 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/mail2news.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/mail2news.sgml @@ -28,7 +28,6 @@ archived till expired.
Feeding Usenet news to email - In CNews, this is trivially done by adding one line to the sys file, defining a new outgoing feed listing all the diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/monitoring.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/monitoring.sgml index f64bbc9d..45ef2f53 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/monitoring.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/monitoring.sgml @@ -1,44 +1,49 @@
Monitoring and administration Once the Usenet News system is in place and running, the news administrator -is then aided in monitoring the system by various reports generated by the -system. Also, he needs to make regular checks in specific directories and -file to ascertain the smooth working of the system. +is then aided in monitoring the system by various reports generated by it. +Also, he needs to make regular checks in specific directories and +files to ascertain the smooth working of the system.
The <literal>newsdaily</literal> report -This report is generated by newsdaily which is typically run through cron. I -shall enumerate some of the problems reported based on what I have seen. - +This report is generated by the script newsdaily which is +typically run through cron. I shall enumerate some of the +problems that are reported by it, based on my observations . -bad input batches: - This reports articles that have been processed and - declared bad and hence not digested. The reason for it is not mentioned. You - are expected to check the article and determine the cause. +bad input batches: + This gives a list of articles that have been declared bad after processing + and hence have not been digested. The reason for this is not given. You + are expected to check each article and determine the cause. -leading unknown newsgroups by articles: - This gives a list of newsgroups - whose hierarchy has been subscribed to, but the specific newsgroup does not - appear in the active file. You could add the newsgroup in the active file if - you think it is important enough. +leading unknown newsgroups by articles: + Newsgroup names that do not appear in the active file + but their hierarchy has been subscribed to, would find their names mentioned + under this heading. Choose to add the name in the active file if you think + it is important. For e.g., you would see this happen + if you have subscribed to the hierarchy comp but the + active does not contain the newsgroup name + comp.lang.java.3d. You could deny subscription to this + particular newsgroup by specifying so in the sys file. + -leading unsubscribed newsgroups: - This gives a list of newsgroups - that have not been subscribed to, of which the news server receives a - maximum no. of articles. You really cannot do much about this except to +leading unsubscribed newsgroups: + If the news server receives maximum articles of a particular newsgroup + hierarchy to which you haven't subscribed, it will appear under this + heading. You really cannot do much about this except to subscribe to them if they are required. -leading sites sending bad headers: +leading sites sending bad headers: This will list your NDNs who are sending articles with malformed/insufficient headers. -leading sites sending stale/future/misdated news: +leading sites sending stale/future/misdated news: This will list your NDNs who are sending you articles that are older than the date you have specified for accepting feeds. @@ -46,36 +51,39 @@ shall enumerate some of the problems reported based on what I have seen. Some of the reports generated by us: We have modified the newsdaily script to include some more statistics. - disk usage: + disk usage: This reports the size in bytes of the $NEWSARTS area. If you are receiving feeds regularly, you should see this figure increasing. - incoming feed statistics: - This reports the no. of articles and total bytes recevied from each of - your NDNs. + incoming feed statistics: + This reports the number of articles and total bytes recevied from each + of your NDNs. - NNTP traffic report: - The output of nestor has also been included in - this report which gives details of each nntp connection and the overall - performance of the network connection read from the newslog file. - To understand the format, read manpage of nestor. + NNTP traffic report: + The output of nestor has also been included in this report which gives + details of each nntp connection and the overall + performance of the network connection read from the newslog + file. To understand the format, read the manpage of + nestor. -Error reporting from the errorlog file: - Reports errors logged in the errorlog file. Usually these are file - ownership or file missing problems which can be easily handled. +Error reporting from the errorlog +file: + Reports errors logged in the errorlog file. Usually + these are file ownership or file missing problems which can be easily + handled.
Crisis reports from <literal>newswatch</literal> -Most of the problems reported to me are ones with either space shortage or +Most of the problems reported to me are those with either space shortage or persistent locks. There are instances when the scripts have created locks files and have aborted/terminated without removing them. Sometimes they are innocuous enough to be deleted but this should be determined after a careful @@ -87,9 +95,9 @@ to determine why sendbatches was failing this often. The space shortage issue has to be addressed immediately. You could -delete unwanted articles by running doexpire or add more disk space at the OS -level. The latter seems a better option. - +delete unwanted articles by running doexpire or add more disk +space at the OS level. +
Disk space @@ -98,18 +106,20 @@ The $NEWSBIN area occupies space that is fixed. Since the binaries do not grow once installed, you do not have to worry about disk shortage here. The areas that take up more space as feeds come in are $NEWSCTL and $NEWSARTS. The -$NEWSCTL has log files that keep growing with each feed and -as the articles are digested in huge numbers the $NEWSARTS -continues to grow. Also, if articles are being archived on expiry you will need -space. Allocate a few GB of disk space for $NEWSARTS -depending on the no. of hierarchies you are subscribing and the feeds that come -in everyday. $NEWSCTL grows to a lesser proportion as -compared to $NEWSARTS. Allocate space for this accordingly. +$NEWSCTL has log files that keep growing with each feed. +As the articles are digested in huge numbers, the $NEWSARTS +area continues to grow. Also, you will need space if you have chosen to archive +articles on expiry. Allocate a few GB of disk space for +$NEWSARTS depending on the number of hierarchies you are +subscribing and the feeds that come in everyday. $NEWSCTL +grows to a lesser proportion as compared to $NEWSARTS. +Allocate space for this accordingly.
CPU load and RAM usage -With modern C-News and NNTPd, there is very little usage of these + +With modern C-News and NNTPd, there is very little usage of these system resources for processing news article flow. Key components like newsrun or sendbatches do not load the system much, except for cases where you have a very heavy flow of @@ -118,9 +128,11 @@ compressed outgoing batches and the compression utility is run by amazingly efficient in the current C-News release. Even when it takes half an hour to digest a large consignment of batches, it hardly loads the CPU of a slow Pentium 200 MHz CPU or consumes much RAM in a 64 MB -system. +system. + -One thing which does slow down a system is a large bunch of + +One thing which does slow down a system is a large bunch of users connecting using NNTP to browse newsgroups. We do not have heuristic based figures off-hand to provide a guidance figure for resource consumption for this, but we have found that the load on the @@ -132,7 +144,8 @@ a dual-P-III Intel Linux server, for instance. This loading has no bearing on whether you are using INN or nntpd; both have practically identical implementations for NNTP reading and differ only in their handling of -feeds. +feeds. + Another situation which will slow down your Usenet news server is when downstream servers connect to you for pulling out NNTP feeds using @@ -143,19 +156,20 @@ your server's I/O system and CPU.
The <literal>in.coming/bad</literal> directory -The in.coming directory is where the batches/articles reside when you have +The in.coming directory is where the batches/articles reside when you have received feeds from your NDN and before processing happens. Checking this directory regularly to see if there are batches is a good way of determining that feeds are coming in. The batches and articles have different nomenclature. -Names like nntp.GxhsDj are indicative of batches and individual -articles are named beginning with digits like 0.10022643380.t +Batches, typically, have names like nntp.GxhsDj and +individual articles are named beginning with digits like 0.10022643380.t -The bad sub-directory under in.coming holds batches/articles that have -encountered errors when they were being processed by relaynews. You will have -to look into the directory for the cause of it. Ideally speaking, this -directory should be empty. +The bad sub-directory under in.coming +holds batches/articles that have encountered errors when they were being +processed by relaynews. You will have to look at the +individual files in this directory to determine the cause . Ideally speaking, +this directory should be empty.
@@ -173,17 +187,19 @@ directory should be empty.
The <literal>junk</literal> and <literal>control</literal> groups -Control messages are those that have a newgroup/rmgroup/cancel/checkgroup in -their subject line. Such messages result in relaynews calling the appropriate -script and on execution a message is mailed to the admin about the action -taken. These control messages are stored in the control directory of -$NEWSARTS. For the propogation of such messages, one must -subscribe to the control hierarchy. +Control messages are those that have a newgroup/rmgroup/cancel/checkgroup in +their subject line. Such messages result in relaynews calling +the appropriate script and on execution a message is mailed to the admin about +the action taken. These control messages are stored in the +control directory of $NEWSARTS. For the +propogation of such messages, one must subscribe to the +control hierarchy. When your news system determines that a certain article has not been subscribed -by you, it is 'junked' i.e. such articles appear in the junk directory. This +by you, it is junked i.e. such articles appear in the junk +directory. This directory plays a key role in transferring articles to your NDNs as they would subscribe to the junk hierarchy to receive feeds. If you are a leaf node, there is no reason why articles should pile here. Keep deleting them on a daily diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/pop.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/pop.sgml index bd3c314b..f1d4425b 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/pop.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/pop.sgml @@ -504,12 +504,194 @@ is discussed in quite a bit of detail in Section XXX titled
-
Control messages - - (Discuss control messages. Show examples of actual control messages - if possible. Discuss security issues in the form of control message - storms, and how digital signatures are being used to tackle it. This - sets the ground for pgpverify later on.) - +
Control messages + +The Usenet is a massive dispersed collection of servers which +operate almost without any supervision, provided they have adequate disk +space and do not suffer disk corruption due to power failures, +etc. (It is indeed surprising how self-managing a +good Usenet server is, provided these two pre-requisites are met.) These +servers are each under the control of human administrators, but it is +preferable that certain routine actions be performed across all these +servers remotely from one location, without the manual intervention of +these humans. + +One common need for centralised operations is the creation of new +groups in the standard eight hierarchies. The Usenet follows a fairly +formal process which asks for votes from readers worldwide before +deciding on the restructuring of its newsgroups list, including merging of +low-volume groups, splitting of high-volume groups into many specialised +groups, creating new groups, and even deleting groups. Once the voting +process for a change concludes and the change action is to be carried +out, it would be extremely tedious to send email to the hundreds of +thousands of Usenet administrators and hope that they make the changes +right, and answer their doubts if they get confused. It would be much +better to have an automatic way to make the +changes across all servers, of course with proper authorisation. + +The solution to this does not lie in giving some central authority +the ability to run an OS-level command of his choice on all the world's +Usenet servers, because OS commands differ from OS to OS, and because +few Usenet administrators would trust a stranger from another part of +the world with OS level access. Therefore, the solution lay in defining +a small set of common Usenet maintenance actions, and permitting only +these actions to be triggered on all servers through the passing of +special command messages, called control +messages. + +Control messages look like ordinary Usenet articles, more or less. +They have an extra header line, with its value in a specific format, +but they usually carry body text which looks like a normal human-written +article. Here is a control message (a spurious one at that, but it'll +do for now): + + + +NNTP-Posting-Host: 203.145.147.67 +X-Trace: goblin.nadrabank.kiev.ua 982528840 21455 203.145.147.67 + (18 Feb 2001 20:40:40 GMT) +X-Complaints-To: usenet@nadrabank.kiev.ua +NNTP-Posting-Date: 18 Feb 2001 20:40:40 GMT +X-No-Archive: Yes + +humanities.hipcrime is an unmoderated newsgroup which passed its +vote for creation by 326:10 as reported in news.announce.newgroups +on 18 Feb 2001. + +For your newsgroups file: +humanities.hipcrime HipCrime for Humanity - you committed one now! + +Anyone can create a newsgroup in the alt, biz, comp, earth, +humanities, misc, news, meow, rec, sci, soc, talk, us, or +any other Usenet hierarchy. New newsgroup proposals may be +optionally discussed in news.groups. Please be sure that your +/usr/lib/news/control.ctl is configured correctly: + +## NEWGROUP MESSAGES +## honor them all and log in \${LOG}/newgroup.log +newgroup:*:alt.*|biz.*|comp.*|earth.*|humanities.*|misc.*|news.*|\ + meaw.*|rec.*|sci.*|soc.*|talk.*|us.*:doit=newgroup + +## RMGROUP MESSAGES +## drop them all and don't log +rmgroup:*:*:drop + +Meow! +David C Lawrence +]]> + + +A control message must have a Control +header. Besides, all control messages will +have an Approved header, like messages posted +to moderated newsgroups. The Control header +actually specifies a command to run on the local server, and the +parameter(s) to supply to it. The local Usenet server software is +supposed to figure out its own way to get the task done. In this +example, the command in the Control header is +newgroup, which creates a new newsgroup. And its +parameter is humanities.hipcrime, which gives the +name of the newsgroup to create. + +In C-News, the control message implementation works through +separate shellscripts kept in a fixed directory, +$NEWSBIN/ctl/, as a security measure; if the +executable script isn't present there, the control message command will +be ignored. The control message types supported are: + + +checkgroups: control message to +check whether the list of newsgroups in your active file are all correct +as per a master list of newsgroups sent in the control message + + +newgroup: control message to create a +new newsgroup + + +rmgroup: control message to delete a +newsgroup and all articles in it + + +sendsys: control message to cause an +email response to be sent to the author with the sys +file of your server in it. This results in a response storm of +emails from all the Usenet servers in the world to the author. These +responses allow the sender of the control message to analyse all the +sys files of the world's Usenet servers and create the +directed graph of Usenet newsfeeds. Why someone would want to do this is +hard to guess, but the result is surely an awesome picture of one facet +of networked human civilisation, like looking at a giant world map. +Incidentally, there is no invasion of privacy here, because +your server's sys file is supposed to be public +information, if you take feeds from the public Usenet. + + +version: control message which results +in your Usenet software sending an email to the author of the message, +containing the type and version of the Usenet news software you are +using. This too is not an invasion of privacy, because this information +is supposed to be public knowledge. + + +The cancel message: the most frequently occurring type of +control messages. They specify the message ID of an article, and result +in the cancellation (deletion) of that article. If you post an article +and regret it a moment later, your Usenet newsreader software usually +allows you to ``cancel'' it by generating a cancel message. + + + +The Usenet news software maintains a pseudo-newsgroup called +control, where it files all control messages it +receives. If you have an incoming newsfeed from the public Usenet, your +server's control group will usually be full with +thousands of cancel messages from trigger-happy fingers all over the +world. Usenet news server software like C-News allows you to filter the +incoming feed based on newsgroups, and will discard articles for groups +they do not subscribe to. But since all servers have to receive and +process control messages, they will all accept these cancel messages, +though many of them may apply to articles which are not part of your +highly-pruned subset of groups. C'est la vie. + +Remember to set expiry for the control group to +one day or even shorter, so that the junk can be cleaned out as rapidly as +possible, just like the junk newsgroup. + +The beauty of the control message architecture is that it +integrates seamlessly into the newsfeed mechanism for automatic control +of the network of servers. No separate channel of connection is needed +for the control actions. And article replication automatically +propagates control messages with human-readable articles, thus +guaranteeing reach across heterogenous networks technologies. + +What your Usenet server does on receiving a +control message is governed by an authorisation file: +$NEWSCTL/controlperms in the case of C-News +and control.ctl in the case of INN, for +instance. The security measures implemented by this module are +further enhanced by the pgpcontrol package with its +pgpverify script. Using pgpverify, +your server can check that all control messages (except for article +cancellation messages) are digitally signed by a trusted party +using military-spec public key cryptography. Our integrated +Usenet news software distribution includes integration with +pgpverify. +
diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/security.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/security.sgml new file mode 100644 index 00000000..a872e882 --- /dev/null +++ b/LDP/howto/docbook/Usenet-News-HOWTO/security.sgml @@ -0,0 +1,218 @@ +
Security issues + +It almost seems strange that we are discussing security issues in +the context of Usenet news servers. Usenet news has been one of the most +open and free-for-all network services traditionally. However, with the +exponential growth of the Internet, all services are becoming aware of +potential threats. The community of Internet intruders too has acquired +new profiles: a lot of Internet intrusion attempts are program-driven, +and exploit a set of ``well known'' vulnerabilities, +i.e. vulnerabilities which have been identified by +the computer security and intrusion community and published in their +reports and advisories. Thus, the question of ``Why will someone attack +my harmless Usenet server?'' is no longer valid. It will be attacked if +it can be attacked, merely because its IP address falls in a range of +addresses being targeted, perhaps. + +Security issues for Usenet news servers fall into two categories. +First come vulnerabilities which will allow an attacker to bring down +your server or run code of his choice on it. Second come vulnerabilities +which can distort or corrupt your Usenet article hierarchy, either by +junk postings, unsolicited commercial messages, or forged control +messages. The second category of threats is specific to Usenet news and +needs Usenet-specific protection mechanisms, some of which require +tapping into defence mechanisms designed by the Usenet administrator +community. + +
Intrusion threats + +Here we discuss the vulnerabilities which will allow an intruder +to ``gain control'' of your Usenet server, or ``bring it down,'' either +of which may be irritating, embarassing, or downright disastrous for your +business or occupation. + +
Generic server vulnerabilities + +Foremost among these vulnerabilities are those which render +anyserver vulnerable to intrusion attempts. +Most of these vulnerabilities are unrelated to Usenet news itself. For +instance, if you have the Telnet service active on a server exposed to +the Internet, then it is likely that systematic attempts by intruders +to acquire usernames and passwords will bear fruit, using methods we +will best leave to specialised texts on the subject. Once this is done, +the intruder will merely ``walk into'' your server by Telnetting into +it. + +We will not discuss this class of vulnerabilities here any further; +they belong in documents dedicated to general security issues. For +further reading, check the ``Security HOWTO'', the ``Security Quickstart +HOWTO'', the ``User Authentication HOWTO'', the ``VPN HOWTO'', and +the ``VPN Masquerade HOWTO'' ... and that's just from the Linux HOWTO +collection. As one can see, there is, if anything, a surfeit of material +on this and related subjects. + +There are vulnerabilities which allow an intruder to mount the +so-called DoS attacks, which make your service inaccessible to +legitimate users, even though it does not let the intruder in. The most +publicised of these attacks were the SYNFlood and the Ping of Death +attacks, both quite old and well-understood by now. A Linux server +running a recent version of the kernel and properly configured, should +be immune to both these attack methods. But network protocols being +what they are, there are always new DoS methods being thought up, which +can temporarily overload or slow down a server. Once again, the texts +discussing generic security issues are the best place to study these +vulnerabilities. + +
+ +
Vulnerabilities in Usenet software + +Then come server vulnerabilities, if any, which are caused +specifically by Usenet news software. For instance, if it was possible +for an intruder to issue some string of bytes to your server's NNTP +server and cause it to execute a command of the intruder's choice, then +this vulnerability would be in this category. + +Any server which accepts a text string as input from a +client is open to the buffer overrun class of attacks, if the +gets() C library function has been used in its code +instead of the fgets() with a buffer size limit. This +was a vulnerability made famous by the 1988 Morris Internet Worm, +discussions on which can be found elsewhere. (Go Google for it if you're +keen.) As far as we know, the INN NNTP server and the +nntpd which forms part of the NNTP Reference +Implementation both have no known buffer overrun vulnerabilities. This +class of vulnerabilities is less significant in the case of NNTPd or +INN because these daemons do not run as root. In +fact, they would begin to cause malfunctioning of the underlying Usenet +software if they ran as root. Therefore, even if an +intrepid intruder could find some way of gaining control of these +daemons, she would only be able to get into the server as user +news, which means that she can play havoc with the +Usenet installation, but no further. A daemon which runs as +root, if compromised, can allow an intruder to take +control of the operating system itself. + +UUCP is generally believed to be insecure. We believe a careful +configuration of Taylor UUCP plugs a lot of these vulnerabilities. One +vulnerability with UUCP over TCP is that the username and password travel +in plaintext form in TCP data streams, much like with Telnet or FTP. We +therefore do not advise using UUCP over TCP in this manner if security +is a concern at all. We recommend the use of UUCP through a SSH tunnel, +with the SSH setup working only with a pre-installed public key. This way, +there is no need for usernames and passwords for the SSH tunnel setup, +and passwords cannot be leaked even intentionally. And the UUCP username +and password then passes through this encrypted tunnel and is therefore +totally superfluous for security; the preceding SSH tunnel provides a much +stronger connection authentication than the UUCP username and password. +And since we set up our SSH tunnels to demand key-based authentication +only, it rejects any attempt to connect using usernames and passwords +when the tunnel is being set up. + +A third possible vulnerability is related to the back-end software +which processes incoming Usenet articles. It is conceivable that an +NNTP server will receive an incoming POST command, +receive an article, and queue it for processing on the local spool; +the NNTP server often does not perform any real-time processing on the +incoming post. The post-processing software which periodically processes +the incoming spool (the in.coming directory in C-News) +will read this article and somehow be forced to run a command of the +intruder's choice, either by buffer overrun vulnerabilities or any +other means. + +While this possibility exists, it appears that neither the C-News +newsrun and family nor INN are vulnerable to this class +of attempts. We base our comment on the solid evidence that both these +systems have been around in an intrusion-prone world of public Usenet +servers for more than a decade. INN, the newer of the two, completed +one decade of life on 20 August 2002. And both these software systems +had their source freely available to all, including intruders. We can be +fairly certain that if vulnerabilities of this class have not been seen, +it not for want of intrusion attempts. + +
+ +
+ +
Vulnerabilities unique to the Usenet service + +There are certain security precautions that a Usenet server +administrator has to take to ensure that her servers are not swamped by +irritating junk or configured out of shape by spurious control messages. +These vulnerabilities do not allow an intruder to run her software on +your servers, but allows her to mess up your server, causing you to lose +a precious weekend (or week) straightening out the mess. + +
Unsolicited commercial messages + +Unsolicited commercial messages are called SPAM. There is a +war against SPAM being fought in the Internet community. The biggest +battlefront is in the world of email. Second to that is Usenet +newsgroups. + +There are many tools that Usenet administrators use in their +battle against SPAM. The most important of these is the NoCeM suite. See +http://www.cm.org/ for details of NoCeM, and the +newsgroup alt.nocem.misc for the SPAM cancel messages +which NoCeM reads to identify which articles to discard. Your server +will need a feed of alt.nocem.misc to use the NoCeM +facility. These special messages are signed by NoCeM volunteers whose +job is to identify SPAM articles, list their message-IDs, and then +issue these deletion instruction, digitally signed with special private +keys, which tell all Usenet servers to delete the SPAM messages. Your +server's NoCeM software will need public key software (typically PGP) +and a keyring with the public key of each NoCeM volunteer you want to +accept instructions from. + +Other anti-spam tools for Usenet services are listed in the +Anti-SPAM Software Web page +(http://www.exit109.com/~jeremy/news/antispam.html). +The Cleanfeed software will clean out articles +identified as SPAM. There are many others. + +SPAM is such a nuisance and a drain on organisational expense +pockets (by wasting bandwidth you pay for) that it is almost imperative +today that every Usenet server protects itself against it. We will +integrate some selected anti-SPAM measures into our integrated source +distribution soon. + +
+ +
Spurious control messages + +Control messages, discussed in detail earlier in , instruct a Usenet server to take certain +actions, like delete a message or create a newsgroup. If this facility +is ``open to the public'', anyone with half a brain can forge control +messages to create twenty new newsgroups, and then post thousands of +articles into those groups. In the mid-nineties, we were hit by a storm +of over 2,000 (two thousand) newgroup control +messages, which rapidly taught us the danger of unprotected control +messages and the protection against them. + +The standard protection mechanism against this +vulnerability is pgpverify, which can be +downloaded from multiple Websites and FTP mirror sites by +searching for pgpverify (the program) or +pgpcontrol (the total software package). We have +integrated this into our source distribution, so that our C-News works +in a tightly coupled manner with pgpverify. + +pgpverify works using public key cryptography, +much like NoCeM, and all the official maintainers of respective Usenet +group hierarchies sign control messages using their private keys. Your +server will carry their public keys, and pgpverify +will check the sign on each control message to ensure that it's from the +official maintainer of the hierarchy. It will then act upon legit +control messages and discard the spurious ones. + +In today's nuisance-ridden Usenet environment, no sane Usenet +server administrator receiving a feed of ``public'' hierarchies and +control messages will even dream of running her server without +pgpverify protection. + +
+ +
+ +
diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/settingup.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/settingup.sgml index 8ed4e007..dcaac5b1 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/settingup.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/settingup.sgml @@ -1,4 +1,4 @@ -
Setting up CNews + NNTPd +
Setting up CNews + NNTPd
Getting the sources and stuff @@ -13,7 +13,7 @@ configuration and installation processes, which are very well documented. The version that is available is Cleanup Release revision G, on which our own version is based. -NNTPd is available from +NNTPd (the NNTP Reference Implementation) is available from ftp://ftp.uu.net/networking/news/nntp/nntp.1.5.12.1.tar.Z. It has no automatic scripts and processes to configure itself. After fetching the sources, you will have to follow a set of directions given @@ -32,7 +32,7 @@ periodically to process the logs of nntpd, the NNTP server which is part of NNTPd, and report usage statistics to the administrator. We have integrated Nestor into our source base. -The fourth piece of the puzzle, without which no Usenet server +The fourth piece of the system, without which no Usenet server administrator dares venture out into the wild world of public Internet newsfeeds, is pgpverify. @@ -41,7 +41,7 @@ have fixed a few bugs in both packages. We have also integrated the four software systems listed above, and added a few features here and there to make things work more smoothly. We offer our entire source base to anyone for free download from -http://www.starcomsoftware.com/proj/news/src/news.tar.gz. +http://www.starcomsoftware.com/proj/usenet/src/news.tar.gz. There are no licensing restrictions on our sources; they are as freely redistributable as the original components we started with. @@ -66,14 +66,6 @@ to find a directory tree with the following subdirectories and files: sources and Makefile. -archives: a directory containing the - tarballs of the original C-News, NNTPd, Nestor and - pgpverify source distributions, in case you want - them. Strictly speaking, the archive directory is - not necessary unless you want to study what changes we have made, - what files we have added, to the original sources. - - build.sh: a shellscript you can run to compile the entire combined source tree and install binaries in the right places, if you are lucky and all goes well. @@ -118,17 +110,15 @@ original by just mirroring them.
Compiling and installing For installing, first make sure you have an entry for a user called - news in your /etc/password file. Add one if not present. This + news in your /etc/password file. This is setting the news-database owner to news. Now download the source from us and untar it in the home directory of news. This creates two main directories viz. c-news and nntp. To install and compile, run the script build.sh as root - in the - directory that contains the script. It is important that the script run as - root as it sets ownerships and installs and compiles as - news and hence should have adequate permissions to do - this. This + in the directory that contains the script. It is important that the script + run as root as it sets ownerships, installs and + compiles the source as user news. This is a one-step process that puts in place both the C-News and the NNTP software, setting correct permissions and paths. Following @@ -151,11 +141,11 @@ original by just mirroring them. - Compiles C-News and exits on error. This builds - all the software. Writes the error into a file called make.out. Read it to - determine the cause. Also, performs regression tests if the - compilation was successfull and does not exit on error. Sends out a - warning to read the error file make.out.r and fix 'em. + Compiles C-News and performs regression tests if the + compilation was successfull. Sends out a warning to read the error file + make.out.r and to fix 'em. + Compilation erros are written to a file called make.out. + @@ -163,11 +153,11 @@ original by just mirroring them. - Checks the presence of the three key directories: + Checks for the presence of the three key directories: $NEWSARTS - (/var/spool/news) that houses the artciles, $NEWSCTL -(/var/lib/news) that contain - the configuration, log and status files and $NEWSBIN - - (/usr/lib/newsbin) that contain the binaries and + configuration, log and status files and $NEWSBIN - + (/usr/lib/newsbin) that contain binaries and executables for the working of the Usenet News system. Tries to create them if non-existent and exits if it results in failure. @@ -181,18 +171,20 @@ original by just mirroring them. Then starts the installation process of C News. It runs - make install to install binaries at the right locations; make setup to set - the correct paths, umask, create directories for newsgroups, determine who - will receive reports; make ui to set up inews and injnews and - make readpostcheck to use readnews, postnews and checknews provided by - C News. The errors, if any are to be found in the respective make.out - files. e.g. make.setup will write errors to make.out.setup + make install to install binaries at the right locations; + make setup to set the correct paths and umask, create + directories for newsgroups, determine who will receive reports; + make ui to set up inews and injnews and + make readpostcheck to use readnews, postnews and + checknews scripts provided by C News. The errors, if any are to be found in + the respective make.out files. e.g. make.setup will write + errors to make.out.setup - Newsspool which queues incoming + Newsspool, which queues incoming batches in $NEWSARTS/in.coming directory should run as - set-userid and set-groupid This is done. + set-userid and set-groupid. This is done. @@ -207,101 +199,94 @@ original by just mirroring them. Sets up the manpages for C News and makes it world readable. The NNTP manpages get installed when the software is installed. - Compiles the C News documentation guide.ps and makes it readable and - available in /usr/doc/packages/news or + Compiles the C News documentation guide.ps and makes it + readable and available in /usr/doc/packages/news or /usr/doc/news. Checks for the PGP binary and asks the administrator to get - it if not found. + it, if not found.
-
Configuring the system: What and how to configure files? +
Configuring the system: What and how to configure files? Once installed, you have to now configure the system to accept feeds and -batch them for neighbours. You will have to do the following: +batch them for your neighbours. You will have to do the following: nntpd: - Copy the compiled nntpd into a directory where + Copy the compiled nntpd into a directory where executables are kept and activate it. It runs on port 119 as a daemon - through inetd unless you have compiled it as stand-alone. + through inetd unless you have compiled it as stand-alone. + An entry in the /etc/services file for nntp would look + like this: + nntp 119/tcp \# Network News Transfer Protocol + An entry in the inetd.conf file will be: + nntp stream tcp nowait news path-to-tcpd path-to-nntpd - An entry in the services file for nntp would look like this: - nntp 119/tcp \# Network News Transfer Protocol - - An entry in the inetd.conf file will be: - nntp stream tcp nowait news path-to-tcpd path-to-nntpd - - The last two fields in the inetd.conf file are the paths to binaries of the - tcp daemon and the nntpd daemon respectively. + The last two fields in the inetd.conf file are paths to + binaries of the tcp and the nntp + daemon respectively. Configuring control files: There are plenty of control files in $NEWSCTL that will need to be configured before you can - start using the news system. The files mentioned here are explained in some - detail in chapter 8, section 8.1. The files to be configured are dealt in - detail in the following section. - + start using the news system. The files mentioned here are also discussed + in the first section of the section titled + . These control files are + dealt in detail in the following below. sys: One line per system/NDN listing all the newsgroup hierarchies each system subscribes to. Each line is prefixed - with the system name and the one beginning with ME: indicates what we - are going to receive. Following are typical entries that go into this - file: - - ME:comp,news,misc,netscape - - This line indicates what newsgroups your server, as determined by the - whoami file have subscribed to and will receive. - - server/server.starcomsoftware.com:all,!general/all:f - - This is a list of newsgroups this site will pass on to its NDN. - The newsgroups specified should be a comma separated list and no spaces - should be inserted in the whole line. The f flag indicates that the - newsgroup name and article no. alongwith its size will be one entry in - the togo file in the $NEWSARTS/out.going directory. + with the system name and the one beginning with + ME: indicates what your + server is willing to receive. Following are typical entries that go into + this file: ME:comp,news,misc,netscape + This line indicates what newsgroups your server has subscribed to. + server/server.starcomsoftware.com:all,!general/all:f + This is a list of newsgroups your server will pass on to your NDN. + The newsgroups specified should be a comma separated list and the entire + line should contain no spaces. The f flag indicates + that the newsgroup name and the article number alongwith its size will + make up one entry in the togo file in the + $NEWSARTS/out.going directory. explist: - This file has entries indicating - which articles expire and when and if they have to be - archived The order in which the newsgroups are listed is important. An - example follows: - - comp.lang.java.3d x 60 /var/spool/news/Archive - + This file has entries indicating which articles expire and when and + whether they have to be archived. The order in which the newsgroups are + listed is important. An example follows: + comp.lang.java.3d x 60 /var/spool/news/Archive This means that the articles of comp.lang.java expire after 60 days and - shall be archived in the directory mentioned as the fourth field. + shall be archived in the directory mentioned in the fourth field. Archiving is an option. The second field indicates that this line applies to both moderated and unmoderted newsgroups. m would specify moderated and u would specify unmoderated groups. If you want to specify an extremely large no. as the expiry - period you can use the word 'never'. + period you can use the keyword never. batchparms: - Sendbatches is a program that - adminsters batched transmission of news to other sites. To do this it - consults the batchparms file. Each line in the file specifies the - behaviour for each site. There are five fields for each site to be + sendbatches is a program that + administers batched transmission of news articles to other sites. To do + this it consults the batchparms file. Each line in + the file specifies the behaviour for each of your NDN mentioned in the + sys file. There are five fields for each site to be specified. - - server u 100000 100 batcher | gzip -9 | viauux -d gunzip + server u 100000 100 batcher | gzip -9 | viauux -d gunzip - The first field is the site name which matches the entry in the sys - file and has a corresponding directory in $NEWSARTS/out.going by that - name. + The first field is the site name which matches the entry in the + sys file and has a corresponding directory in + $NEWSARTS/out.going by that name. - The second field is the class of the site, 'u' for UUCP and 'n' for - NNTP feeds. A '!' in this field means that batching for this site has - been disabled. + The second field is the class of the site,u for + UUCP and n for NNTP feeds. A ! in + this field means that batching for this site has been disabled. The third field is the size of batches to be prepared in bytes. @@ -312,37 +297,39 @@ batch them for neighbours. You will have to do the following: The fifth field is the command line to be used to build, compress and - transmit batches to that site. It receives the contents of the togo file - on standard input. + transmit batches to that site. The contents of the + togo file are made available on standard input. controlperm: This file controls how the news system responds to control messages. Each line consists of 4-5 fields - separated by white space. + separated by white space. Control messages has been discussed in + . + - comp,sci tale@uunet.uu.net nrc pv news.announce.newsgroups + comp,sci tale@uunet.uu.net nrc pv news.announce.newsgroups The first field is a newsgroup pattern to which the line applies. - The second field is either 'any' or an e-mail address. The latter - specifies that the line applies only to control messages from that - author. + The second field is either the keyword any or an e-mail + address. The latter specifies that the line applies to control messages + from only that author. The third field is a set of opcode letters indicating what control operations need to be performed on messages emanating from the e-mail - address mentioned in the second field. 'n' stands for creating a - newgroup, 'r' stands for deleting a newsgroup and 'c' stands for - checkgroup. + address mentioned in the second field. n stands for + creating a newgroup, r stands for deleting a + newsgroup and c stands for checkgroup. The fourth field is a set of flag letters indicating how to respond to a control message that meets all the applicability tests: - + y Do it. n Don't do it. v Report it and include the entire control @@ -354,14 +341,13 @@ batch them for neighbours. You will have to do the following: The fifth field, which is optional, will be used if the fourth field - contains a 'p'. It must contain the PGP key ID of the public key to be - used for signature verification. - - + contains a p. It must contain the PGP key ID of the + public key to be used for signature verification. + mailpaths: This file describes how to reach - the moderators of various heirarchies of news groups by mail. Each line + the moderators of various hierarchies of newsgroups by mail. Each line consists of two fields: a news group pattern and an e-mail address. The first line whose group pattern matches the newsgroup is used. As an example: @@ -371,8 +357,9 @@ batch them for neighbours. You will have to do the following: all %s@moderators.uu.net - In the second example, the %s gets replaced with the groupname and all - dots appearing in the name are substituted with dashes. + In the second example, the %s gets replaced with the + groupname and all dots appearing in the newsgroup name are substituted + with dashes. Miscellaneous files: @@ -384,14 +371,14 @@ batch them for neighbours. You will have to do the following: organization: - Contains the default value for the - Organization: header for postings originating locally. + Contains the default value for the Organization: + header for postings originating locally. whoami: - Contains the name of the news system. This - is the site name used in the Path: headers and hence should concur - with the names your neighbours use in their sys files. + Contains the name of the news system. This is the site name used in + the Path: headers and hence should concur with + the names your neighbours use in their sys files. @@ -401,10 +388,10 @@ batch them for neighbours. You will have to do the following: newsgroup (not just the hierarchy) to be found on your news system. You will have to get the most recent copy of the active file from ftp://ftp.isc.org/usenet/CONFIG/active and prune it - to delete - newsgroups that you have not subscribed to. Run the script "addgroup" - for each newsgroup in this file which will create relevant directories - in the $NEWSARTS area. The "addgroup" script takes + to delete newsgroups that you have not subscribed to. Run the script + addgroup for each newsgroup in this file which will + create relevant directories in the $NEWSARTS area. + The addgroup script takes two paramters: the newsgroup name being created and a flag. The flag can be any one of the following: @@ -420,41 +407,42 @@ batch them for neighbours. You will have to do the following: An entry in this file looks like this: - comp.lang.java.3d 0000003716 01346 m + comp.lang.java.3d 0000003716 01346 m The first field is the name of the newsgroup. The second field is the highest article number that has been used in that newsgroup. The third field is the lowest article number in the group. The fourth - field is flag as explained above. + field is a flag as explained above. newsgroups file: - This contains a one line description + This contains a one-line description of each newsgroup to be found in the active file. You will have to get the most recent file from ftp://ftp.isc.org/usenet/CONFIG/newsgroups and prune it to remove unwanted information. As an example: - comp.lang.java.3d 3D Grphics APIs for the Java language + comp.lang.java.3d 3D Graphics APIs for the Java language - Create aliases: + Aliases: These aliases are required for trouble reporting. Once the system is in place and scripts are run, anomalies/problems - are reported to addresses in the /etc/aliases file. These entries - include email addresses for newsmaster, newscrisis, news, - usenet, newsmap + are reported to addresses in the /etc/aliases file. + These entries include email addresses for newsmaster, + newscrisis, news, usenet, newsmap. They should ideally point to an email address that will be - looked at regularly. Arrange the emails for "newsmap" to be - discarded to minimize the effect of "sendsys bombing" by practical - jokers. + accessed at regularly. Arrange the emails for + newsmap to be discarded to minimize the effect of + sendsys bombing by practical jokers. Cron jobs: - Certain scripts like newsrun that picks up incoming + Certain scripts like newsrun that picks up incoming batches and maintenance scripts, should run through news-database - owner's cron. The cron entries ideally will be for the following: A more - detailed report can be found in + owner's cron which is news. The cron entries ideally + will be for the following: A more detailed report can be found in + newsrun: This script processes incoming batches of @@ -483,32 +471,33 @@ batch them for neighbours. You will have to do the following: - newslog: - Make an entry in the system's syslog.conf - file for logging messages spewed out by the nntp daemon in "newslog". - It should be located in $NEWSCTL. The entry will - look like this: + newslog: + Make an entry in the system's syslog.conf + file for logging messages spewed out by nntpd in + newslog . It should be located in + $NEWSCTL. The entry will look like this: - news.debug -/var/lib/news/newslog + news.debug -/var/lib/news/newslog - Newsboot: - Have newsboot run (as "news", the + Newsboot: + Have this run (as news the news-database owner) when the system boots to clear out debris left around by crashes. Add a Usenet mailer in sendmail: - The mail2news program provided as + The mail2news program provided as part of the source code is a handy tool to send an e-mail to a newsgroup which gets digested as an article. You will have to add the following - ruleset and mailer definition in your sendmail.cf file: + ruleset and mailer definition in your sendmail.cf + file: Under SParse1, add the following: - + R$+ . USENET < @ $=w . > $#usenet $: $1 - + Under mailer definitions, define the mailer Usenet as: @@ -520,27 +509,23 @@ batch them for neighbours. You will have to do the following: In order to send a mail to a newsgroup you will now have to suffix - the - newsgroup name with usenet i.e. your To: header - will look like this: + the newsgroup name with usenet i.e. your + To: header will look like this: To: misc.test.usenet@yourdomain. The mailer definition of usenet will intercept this mail and post it to - the respective newsgroup, in this case, misc.test + the respective newsgroup, in this case, misc.test + - + - -This, more or less, completes the configuration part. - +This, more or less, completes the configuration part. - +
Testing the system - -To locally test the system, follow the steps given below: - +To locally test the system, follow the steps given below: post an article: Create a local newsgroup @@ -550,22 +535,23 @@ To locally test the system, follow the steps given below: and using postnews post an article to it. -Has it arrived in $NEWSARTS/in.coming?: +Has it arrived in $NEWSARTS/in.coming?: The article should show up in the directory mentioned. Note the nomenclature of the article. When newsrun runs: - When newsrun runs through cron, the article disappears from in.coming - directory and appears in $NEWSARTS/mysite/test. Look how - the newsgroup, active, log and history (not the errorlog) files and - .overview file in + When newsrun runs from cron , the article disappears from + in.coming directory and appears in + $NEWSARTS/mysite/test. Look how + the newsgroup, active, log and history (not the errorlog) + files and .overview file in $NEWSARTS/mysite/test reflect the digestion of the file into the news system. reading the article: - Try to read the article through readnews or any + Try to read the article through readnews or any news client. If you are able to, then you have set most everything right. @@ -573,12 +559,13 @@ To locally test the system, follow the steps given below:
<literal>pgpverify</literal> and <literal>controlperms</literal> - As mentioned in , it becomes necessary to - authenticate control messages to protect yourself from being attacked by - pranksters. For this, you will have to configure the - $NEWSCTL/controlperm file to declare whose control + As mentioned in , it becomes + necessary to authenticate control messages to protect yourself from being + attacked by pranksters. For this, you will have to configure the + $NEWSCTL/controlperm file to declare whose control messages you are willing to honour and for what newsgroups alongwith their - public key ID. The controlperm manpage shall give you details on the format. + public key ID. The controlperm manpage shall give you + details on the format. @@ -586,13 +573,13 @@ To locally test the system, follow the steps given below: verifies the Usenet control messages that have been signed using the signcontrol process. The script can be found at ftp://ftp.isc.org/pub/pgpcontrol/pgpverify. - pgpverify pgpverify internally uses the PGP binary which + pgpverify internally uses the PGP binary which will have to be made available in the default executables directory. If you wish to send control messages for your local news system, you will have to - digitally sign them using the above mentioned "signcontrol" program which is - available at + digitally sign them using the above mentioned signcontrol + program which is available at ftp://ftp.isc.org/pub/pgpcontrol/signcontrol. You will - also have to configure the signcontrol program accordingly. + also have to configure the signcontrol program accordingly.
@@ -637,7 +624,8 @@ To locally test the system, follow the steps given below: sites, you will have to configure sys and batchparms with more entries. The number of directories in $NEWSARTS/out.going shall increase, too. Refer - to chapter 8, section 8.1 and 8.2 for a better understanding of + to first two sections of the chapter titled + for a better understanding of outgoing feeds. Again, you will have to determine how you wish to transmit the feed: UUCP or NNTP. @@ -650,7 +638,7 @@ To locally test the system, follow the steps given below: A full treatment of UUCP configuration is beyond the scope of this document. However, the basic steps will be as follows. First, - you will have to define a ``system'' in your Usenet server for the + you will have to define a system in your Usenet server for the NDN (next door neighbour) host. This definition will include various parameters, including the manner in which your server will call the remote server, the protocol it will use, etc. diff --git a/LDP/howto/docbook/Usenet-News-HOWTO/software.sgml b/LDP/howto/docbook/Usenet-News-HOWTO/software.sgml index 42a06d58..7978c9f2 100644 --- a/LDP/howto/docbook/Usenet-News-HOWTO/software.sgml +++ b/LDP/howto/docbook/Usenet-News-HOWTO/software.sgml @@ -1,40 +1,39 @@
Usenet news software -
CNews and NNTPd - -Once upon a time, when Usenet news was a term not yet invented, the -first recorded attempt to use a UUCP-based email backbone to maintain a -replicated message repository, was called A-News. It connected four -servers in four universities, and was written as Unix shell -scripts. +
A brief history of Usenet systems -The designers of A-News had not anticipated how much load users -would put on their simplistic system. A far superior, more sophisticated, -and faster implementation of Usenet news was written later, called -B-News. This was a mix of C and shell scripts, and was designed -much better than A-News, to allow handling of much larger volumes of -messages. B-News v2.x was the current version in around 1990. By 1992 or -so, it had been surpassed by C-News. +Towards the end of this HOWTO, we have added some information +about the history of Usenet server software by quoting sections from an +earlier Usenet Periodic Posting. We consider this historical +perspective, and the Usenix papers and other documents referred to in +it, essential reading for any Usenet server administrator. Please see +the section titled . + +
+ +
C-News and NNTPd C-News was written by Henry Spencer and Geoff Collyer of the Department of Zoology, University of Toronto, almost entirely in shell -and awk, as a replacement for B-News. Once again, the -focus was on adding some extra features and a lot of performance. The -first release was called Shellscript Release, which was deployed by a very -large number of servers worldwide, as a natural upgrade to B-News. This -version of C-News even had upward compatibility with B-News meta-data, -e.g. history files. This was the version of C-News -which was initially rolled out in 1992 or so at the National Centre for -Software Technology (NCST, http://www.ncst.ernet.in) -and the Indian Institute of Technologies in India as part of the Indian -ERNET network. +and awk, as a replacement for an earlier system called +B-News. The focus was on adding some extra features and a +lot of performance. The first release was called Shellscript Release, +which was deployed by a very large number of servers worldwide, as +a natural upgrade to B-News. This version of C-News had upward +compatibility with B-News meta-data, e.g. history +files. This was the version of C-News which was initially rolled out +in 1991 or so at the National Centre for Software Technology (NCST, +http://www.ncst.ernet.in) and the Indian Institutes +of Technology in India as part of the Indian educational and research +network (ERNET). We received guidance from the NCST about Usenet news +installation and management. The Shellscript Release was soon followed by a re-write with a lot more C code, called Performance Release, and then a set of cleanup and -component integration steps leading to the last release called the -Cleanup Release. This Cleanup Release was revised many times, and the -last one was CR.G (Cleanup Release revision G). The version of C-News -discussed in this HOWTO is a set of bug fixes on CR.G. +component integration steps leading to the last release called the Cleanup +Release. This Cleanup Release was patched many times by the authors, +and the last one was CR.G (Cleanup Release revision G). The version of +C-News discussed in this HOWTO is a set of small bug fixes on CR.G. Since C-News came from shellscript-based antecedents, its architecture followed the set-of-programs style so typical of Unix, @@ -51,7 +50,7 @@ command-line interfaces, called from scripts. C-News was born in a world with widely varying network line speeds, where bandwidth utilisation was a big issue and dialup links with UUCP -file transfers was common. Therefore, it has very strong support for +file transfers was common. Therefore, it has strong support for batched feeds, specially with a variety of compression techniques and over a variety of fast and slow transport channels. And C-News virtually does not know the existence of TCP/IP, other than one or two tiny batch @@ -59,7 +58,7 @@ transport programs like viarsh. However, its design was so modular that there was absolutely no problem in plugging in NNTP functionality using a separate set of C programs without modifying a single line of C-News. This was done by a program suite called -NNTPd. +NNTP Reference Implementation, which we call NNTPd. This software suite could work with B-News and C-News article repositories, and provided the full NNTP functionality. Since B-News @@ -88,22 +87,20 @@ HTTP servers are run in most cases, and allows INN to cache a lot of things in its memory, including message-IDs of recently posted messages, etc. This interesting architecture has been discussed in an interesting paper by the author, where he explains the problems -of the older BNews and CNews systems that he tried to address. Anyone +of the older B-News and C-News systems that he tried to address. Anyone interested in Usenet software in general and INN in particular should -study this paper. +study this paper. + INN addresses a Usenet news world which revolves around NNTP, though it has support for UUCP batches --- a fact that not many INN administrators -seem to talk about. The primary situations where INN works at higher -efficiency over the CNews-NNTPd combination are in processing incoming -NNTP feeds when there are multiple incoming NNTP feeds. For multiple -readers reading and posting news over NNTP, there is no difference -between the efficiency of INN and NNTPd. -discusses the efficiency issues of INN over the earlier CNews -architecture, based on Rich Salz' paper and our analyses of usage -patterns. - +seem to talk about. INN works faster than the CNews-NNTPd combination when +processing multiple parallel incoming NNTP feeds. For multiple readers +reading and posting news over NNTP, there is no difference between the +efficiency of INN and NNTPd. discusses +the efficiency issues of INN over the earlier C-News architecture, based +on Rich Salz' paper and our analyses of usage patterns. INN's architecture has inspired a lot of high-performance Usenet news @@ -119,7 +116,10 @@ This is an interesting software system, to set up a ``small'' Usenet news server on one computer which only receives newsfeeds but does not have the headache of sending out bulk feeds to other sites, i.e. it is a ``leaf node'' in the newsfeed flow -diagram. +diagram. According to its homepage (www.leafnode.org), +``Leafnode is a USENET software package designed for small sites running +any flavour of Unix, with a few tens of readers and only a slow link +to the net. [...] The current version is 1.9.24.'' This software is a sort of combination of article repository and NNTP news server, and receives articles, digests and stores them on the @@ -139,6 +139,22 @@ space. Of course, ease of configuration and management is dependent on familiarity, and we are more familiar with C-News than with Leafnode. We hope this HOWTO will help you in that direction. +There is, however, one area in which Leafnode +is far easier to administer than INN or C-News. Leafnode constantly +monitors the actual usage of the newsgroups it carries, based on +readership statistics of its NNTP readers. If a particular newsgroup +is not read at all by any user for a week, then Leafnode will delete +all articles in that newsgroup, free up disk space, and stop fetching +new articles for it. If it finds that a previously abandoned newsgroup +is now again receiving attention, even from one user, then it'll fetch +all articles for that group from its upstream server the next time it +connects. This self-tuning feature of Leafnode is really an excellent +advantage which makes a Leafnode site easier to manage, specially for +small setups with bandwidth and disk space constraints. + +The Leafnode Website gives a lot of details in an easily +understood format. + TO BE EXTENDED AND CORRECTED.
@@ -176,12 +192,6 @@ away.
Carrier class software -We have touched upon the characteristics of carrier-class Usenet -software in the section where we discuss NNTP efficiency issues. As that -bit shows, the requirements of carrier-class Usenet servers is very -different from those run within organisations and institutes for -providing internal service to their members. - Carrier-class servers are expected to handle a complete feed of all articles in all newsgroups, including a lot of groups which have what we call a ``high noise-to-signal ratio.'' They do not have the luxury of @@ -189,56 +199,41 @@ choosing a ``useful'' subset like administrators of internal corporate Usenet servers do. Secondly, carrier-class servers are expected to turn articles around very fast, i.e. they are expected to have very low latency from the moment they receive an article to the -time they retransmit it by NNTP to downstream servers. Third, they are -supposed to provide very high availability, i.e. -they are supposed to be like other carrier class services. This usually -means that they have parallel arrays of computers in load sharing -configurations. And fourth, they usually do not cater to retail -connections for reading and posting articles by human users. Usenet news -carriers usually reserve separate computers to handle retail -connections. +time they retransmit it by NNTP to downstream servers. Third, they +are supposed to provide very high availability, like other ``carrier +class'' services. This usually means that they have parallel arrays of +computers in load sharing configurations. And fourth, they usually do +not cater to retail connections for reading and posting articles by human +users. Usenet news carriers usually reserve separate computers to handle +retail connections. Thus, carrier-class servers do not need to maintain a repository -of articles with the usual residence times of days or weeks, and expire -articles after they age. They only need to focus on super-efficient -re-transmission. These highly specialised servers have software -which receive an article over NNTP, parse it, and immediately re-queue -it for outward transmission to dozens or hundreds of other servers. And -since they work at these high throughputs, their downstream servers -are also expected to be live on the Internet round the clock to receive -incoming NNTP connections from the carrier servers. Therefore, there's -no batching or long queueing needed, and batching cannot be used. In -fact, some carrier class servers state that if you wish to receive feeds -from them, then your servers need to be available round the clock and -connected with lines fast enough to take the blast of a full feed. If -you do not fulfil these conditions, your servers will lose articles, -and the carrier is not responsible for the loss. +of articles; they only need to focus on super-efficient real-time +re-transmission. These highly specialised servers have software which +receive an article over NNTP, parse it, and immediately re-queue it for +outward transmission to dozens or hundreds of other servers. And since +they work at these high throughputs, their downstream servers are also +expected to be live on the Internet round the clock to receive incoming +NNTP connections, or be prepared to lose articles. Therefore, there's +no batching or long queueing needed, and C-News-style batching in fact +is totally inapplicable. -Therefore, one can almost say that carrier-class servers have -neither article repositories nor queues other than the current message(s) -being re-transmitted. If they fail to connect to five of their fifty -downstream neighbours, or fail to push an article through due to -a transmit error, those five neighbours will never receive that -article later from this server; the article will be dropped from their -queues. Retries are not part of the game. Therefore, carrier-class -Usenet servers are more like packet routers than servers with -repositories. +Therefore, these carrier-class Usenet servers are more like packet +routers than servers with repositories. They are referred to nowadays as +NNTP routers or news routers. -It can be seen why carrier-class software cannot hope to do its -job using batch-oriented repository management software like C-News and -why it needs a totally NNTP-oriented implementation. Therefore, the INN -antecedents of some of these systems is to be expected. We would -love to hear from any Linux HOWTO reader whose -Usenet server needs include carrier-class behaviour. +It can be seen why batch-oriented repository management +software like C-News is a total anachronism here, and why they need an +NNTP-oriented, online, real-time design. The INN antecedents of some +of these systems is therefore natural. We would love to hear from any +Linux HOWTO reader whose Usenet server requirements include carrier-class +behaviour. -As far as we know, there is no freely redistributable software -implementation of carrier-class Usenet news servers. There is no reason -why such services cannot be offered on Linux, even Intel Linux, provided -you have fast network links and arrays of servers. Linux as an OS platform -is not an issue here, but free software has not yet been made available -for this niche. Presumably it is because the users of such software are -service providers who earn money using it, and therefore are expected -to be willing to pay for it. +We are aware of only one freely redistributable NNTP router: +NNTPRelay (see http://nntprelay.maxwell.syr.edu/); this +software runs on NT. There is no reason why such services cannot run off +Linux servers, even Intel Linux, provided you have fast network links and +arrays of servers. Linux as an OS platform is not an issue here. TO BE EXTENDED AND CORRECTED.