LDP/LDP/lampadas/TODO

	Add an item, or claim an item by adding your name...

-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-


PENDING
=======

Write a Makefile to install Lampadas onto a fresh system. Also let `make test`
run the unit tests, and `make clean` remove any installation. `make tar` and
other build targets would be nice, too.
	-- wrote Makefiles, but no test target yet
	-- Alex is working on a new installation method

Add data for synchronizing translations.
	-- Added sk_seriesid to interface 2002-07-29 DCM.


RELEASE 0.9 (BETA)
==================

For Gnome:

Display the mode information as a string so it's a bit more readable.

The last_update field should mean the date of the most recent file that was updated,
and add a new field for last_published.

Add ScrollKeeper categories, per Eric Baudais. We need to keep existing categories
so we can import LDPDB data. Maybe the way to go is to provide for more than
one categorization scheme? LDP, OMF, Trove?

Add Gnome apps list.

Update db2omf to support ScrollKeeper extensions and work with Gnome.


Nonspecific:

Run iconv on source sgml files to get them all into utf-8 before processing.
Ideally there would be meta-data on the source encoding so we can handle it
intelligently.

Allow running of file reports on nonlocal files by using the mirrored copy.

Use shared memory to cache Lampadas data so a) it's not reloaded by
every Apache process and b) everyone sees up to date data and
c) we avoid collisions, where one user saves data and another
changes it.

To completely avoid overwriting data, look at each record's timestamp
before saving it. If the timestamp in the database has changed since we loaded,
abort the save and inform the user. This will require timestamps on
every file, which is already elsewhere on this list.

Translation report. Show a list of documents that have translations.
Indicate the master and list the children.

Clean up menus. We no longer need strange constructs to test out functionality.
It's been working well for a long time now. Remove menu-level permissions,
and only apply permissions at the page level. If there are no accessible pages,
the menu won't display anyway, so it's redundant.

Add replaced_by_id to the doc edit form.

Add translation interface, allowing for tracking of translations,
keeping them in synch, etc.

Need option for whether any registered user can add a document,
or only editor and up. Gnome will probably want this, since they don't
get a lot of outside contributions like the LDP does.

Database code has to be able to log on over the network.


RELEASE 1.0
===========

Add tabrecent_updates -- table of recent document updates.
Implement generally, by providing for sort order in doctable,
as well as a rowlimit. The two in combination will do this and more
(pending documents sorted by date entered, docs sorted by tickle
date, etc.).

Add in reviewed_date for both types of review. Maybe review_version also.

Make sure that email addresses are spam-protected everywhere they display on
the website! Put this in the Developer's Guide; it's policy.


LDP-SPECIFIC ISSUES
===================

Add the rest of the replaced_by_id fields in ldpdp-import.

Need to talk to iBiblio and Gnome folks about webserver permissions,
or come up with a reliable way to reset the service without having
them. The admin has to have a way to reset the server if things lock up.


UNSORTED
========

Add SourceFiles to modules documentation.

All .add methods should return the new object, not an id or code.

Archived documents don't work -- they should be entered as non-managed
documents using file:// files.

Add test code to check that pages, formats, etc., etc. are complete
in any language for which they are started. Actually, start managing
these as text documents in Lampadas.

Rename pub_date to orig_pub_date, add back pub_date to mean the date
this version was published, and last_update should be the date of the
file in CVS.

I18n support for dtds, lowercase the codes.

Add language preferences for users.

Read browser language preferences.

Send meta-data so clients don't cache the pages.

Audit logging -- it's not really very organized, and the messages
don't match the alleged log_level.

Remove section from conf file for WEBSERVER, which is now defunct.
Remove those items from Config.py as well (and make sure they're
not being used anywhere).

Write the email message that will be send to new account holders.
Emails need to be i18n as well as web pages. Right now that means the
one which includes your email account, but there will be others.

Mark required fields using class="required", and identify in the stylesheet.
Use the class on the label, too, so either the label or the field itself
can be highlighted.

Complete the little things we have to do to add the OSI-Certified graphic
legally.

Make sure the licenses list contains all of the OSI certified licenses,
all of them listed on SourceForge, and all of them listed on Freshmeat.

Still need to do editing interface for a document.
Character encoding might become an issue here.

We need a cron job that runs nightly and stores statistics into the stats and
stats_cdf tables. This should be a Python module (Stats.py), which deletes
and then replaces any existing records for the current day. The cron job would
just be a call to that library. So put the Python module in pylib, but put the
wrapper script under /lampadas/cron (a new directory). See the old Perl code,
and just port it. Feel free to recommend additional metrics that would be
valuable to the management of the site. Also add user count, which didn't
apply to the LDPDB but does apply to Lampadas. And docs served, pages served.

Add timestamps to additional records where it is appropriate.
Maybe even to all of them. It's nice meta-data to have. And add creator_id
also, to record the user who created the record. Then it will have to
be passed in through all the Add methods.

Remember the number of times a user has logged on, and when they last logged
on, and maybe even their IP address in case of DoS or other attacks by a user.

IBM wants to write a CQS module for Lampadas, but we need to define a search
API implemented in a URL scheme for them to code against. This is pretty
high priority because it will give us a huge amount of visibility and help
build the project.

Update ldpwn.pl to use the new database structure.


RELEASE:

These are TODO items that need to be done before each release as part of
the stabilization process:

1. Database Tables

	1.1. Check all tables to ensure all proper referential integrity checks
	are in place.

	1.2 Try inserting huge amounts of data in every field and saving.
	The forms should never let you put in too much data for the field,
	nor should it restrict you from entering huge amounts for text fields.

2. Unit Tests

	2.1. Make sure all possible Lintadas errors are generated by the
	test data, and that all can be cleared up through the web interface.

3. Installation

	3.1. Do a test install on a clean machine and make sure the system
	works with only the default data.

4. Documentation

	4.1. Compare tokens with HTML.py to make sure they match precisely.

5. Rendering

	5.1. Make sure pages are usable in all browsers, with all stylesheets,
	or with no stylesheet.

	5.2. Audit code to make sure all strings are being loaded from strings
	table, never hardcoded.

6. Security

	6.1. Make sure unauthorized users can't get to restricted data.
	Try typing in URLs manually even when they aren't listed, to make
	sure the code always returns the "unauthorized" box on restricted
	pages.


DONE:

Write language codes into ldp-import correctly for Spanish HOWTO and other
documents not in English.
	-- Done 2002-07-21 DCM.

Prevent the same file being assigned to a doc twice, both in database
and in object layer.
	-- Done in database layer 2002-07-31 DCM.

Don't call sql for every count() method.
	-- Done 2002-07-30 DCM.

Add ability to refer to local documents using file:// urls. This
lets us publish documents outside the CVS tree.
	-- Done 2002-07-29 DCM.

Webserver is holding connection open. Possibly Content-length header is
incorrect? I think maybe some of the delay is the time Apache takes to load
mod_python.
	-- Fixed, I think. It seemed to be very poor memory usage and
	   string concatenation. Page serving is much faster now, by
	   a factor of 10 or more. 2002-07-28 DCM.

Run iconv on OMF data when importing to get it all into utf-8.
	-- Done 2002-07-31 DCM.

Get static site to generate relative links so it can be mirrored.
	-- Done 2002-07-31 DCM.

Make sure all routines that load rows are called load_row.
	-- Done 2002-07-25 DCM.

Make sure all codes are terminated with _code for consistency. It's getting
confusing to remember the few exceptions.
	-- Done 2002-07-25 DCM.

Import Gnome document meta-data from their omf files.
	-- Done 2002-07-25 DCM.

Lintadas must update format_code in document to match the top file.
	Done 2002-07-24 DCM.

Add search form using doctable API.
	-- Done 2002-07-16 DCM.
	   It could still be extended.

Remove url and ref_url fields from document table and from doc form.
They are totally replaced by the document_file table.
	-- Done 2002-07-16 DCM.

Update file records from file information. Add timestamp, filesize.
	-- Done 2002-07-16 DCM.

Load and display CVS info about each file in the document -- committer, date/time,
etc., per Eric Baudais.Display history and other logs from cvs.
	-- Done 2002-07-17 DCM. Added ability to script anything you want
	   by writing filename to a tmp file, letting you run an external
	   script that reads it.
	
Only flag errors for documents which are active or archived.
When displaying them in doctable, do not attempt to link to them.
	-- Done 2002-07-14 DCM.

Add one more "description" field for a doc, per Eric Baudais.
	-- Done 2002-07-13 DCM.

Add "short title" field.
	-- Done 2002-07-13 DCM.

Add skseries_id to the document edit form.
Combo box will use "short title above, if available.
	-- Done 2002-07-13 DCM.

Add license version field, and a field for who owns the rights to the license,
per email from Eric Baudais.
	-- Done 2002-07-13 DCM.

Update concatenation to use the WOStringIO routine
from Nicolas (in Globals.py).
	-- Done 2002-07-12 Nico.

Speed up initial loading; it's annoying and overhead for short running scripts.
Loading all tables in single SELECT statements would help a lot,
but would require top level objects to do the loading.
	-- done 2002-07-11 DCM.
	   I optimized along those lines, but performance still lags.

Flatten the topic/subtopic structures into a single table with topic_id as
the record identifier, plus topic_num and subtopic_num for display only.
This will let people reorder and renumber topics without having to alter any
referring records in document_topic.
	-- didn't flatten, but linked document to subtopic directly, and
	made renumbering easy.

Write manpage.
	-- done DCM.
	
Need combobox for selecting seriesid for a document.
	-- done DCM.

Guylhem wants to be able to flag a document, "maintainers needed", even if it
is maintained but the maintainer needs or wants help.
	-- done 2002-07-04 DCM.

Block table should not include language support -- use only for HTML blocks
with embedded tokens.
	-- done 2002-07-04 DCM.

Add "primary" field to document_file to know which is the topmost file.
	-- Done 2002-07-07 DCM.

Need to devise set of error messages and add them to Lintadas.
We need to file document errors and file errors separately,
as in document_error and _file_error.
	-- Done 2002-07-12 DCM.

Define all error codes as constants in Lintadas.py.
	-- Done 2002-07-12 DCM.


WISHLIST
========

These are pie-in-the-sky ideas, many won't pan out. They won't be in the
initial release.

Add a table of programs, maybe from the Debian repository and/or Red Hat
and/or the rpmfind.net database. Then let people record what applications
a document covers. So, a document could be listed as a "User's Guide", and
also tagged with the application "mutt". Then the system knows by meta-data
that the document is a user's guide to mutt. Maybe these can be downloaded
from Freshmeat, organized by Trove classification?

Allow users to record the distribution they are using, and let them browse
through only the documentation that applies to that distribution, as a kind
of a filter. Applying filters could be a very powerful way to search the
database, especially once it gets up into the thousands of documents.

Add the capability to merge in the entire contents of the server's
ScrollKeeper database, automatically. That would make it a piece of cake
to publish the Gnome and KDE documentation, which is already available
there, with a minimum of maintenance.

Add support for the W3C core CSS styles, if possible.

Look into using XForms for form storage instead of HTML.

Add a user Q&A section where users can post questions and answers.
Let them assign a topic/subtopic and/or application name to it,
so they can be viewed by topic.

Add Bugzilla integration. Ideally, list the bugs in the Bugzilla database
on the doc's edit page, and allow developers/authors to jump right to them.
Also allow readers to go to the document's Bugzilla page and add a bug there,
so we have round-trip support.

Add a database of resource links, categorized by topic and/or application.
Perhaps we can use one of the existing Linux link databases for this --
dmoz, Thousands of Linux Links (or whatever it's called), or the SuperLinux
Encyclopedia of Linux Links.

Add a database of "tips" similar to the LOST project at
http://lost.sourceforge.net. LOST's license won't work for Lampadas, but
they might make an exception. Worth asking.

Provide a set of entities that will be available to authors, including the text
of licenses in DocBook XML/SGML, etc. This will make it easy for an author to
make a doc under a certain license, automatically.
These entities could be stored in a table and inserted by Lampadas, or
stored on the filesystem and merged at publication time.
They should be available in various translations as well, although in
most cases the English is required in the actual document as it is the
normative legal language.

Where licenses have options that affect their Freedom, list them as separate
licenses (e.g., OPL w/ options A or B). But use another field to list the
"variant" of the license, so the statistics can still show the total OPL
licensed docs, etc.

Submit a document through the web interface using an upload button.
This would add it to the CVS tree and create a record for it.
This would probably just be an "upload" button on the current document
edit form, where you can add a form at the same time as you upload it.

Allow users to tag documents as their "Favorites", so they can come back
and find them again quickly and easily.

Allow users to "request to be notified" when a document is updated.
This will be particularly useful for translators, but others could
find it useful as well.

Have a weekly cron job generate a summary including new documents,
updated documents, CVS commits by user, like the LDPWN. Have it also
collate all actual news items for that week, and mail it to a
subscriber list. We can use the LDPWN mailing list for this, 
<news@en.tldp.org>.

We might use the Google translation engine to translate to languages for
which we don't have a real translation. It would be better than nothing.

Add TODO lists for each registered user.

Add ability to log into a table, which would allow online 
monitoring of the log. And put the log_level and such in there
so an admin can filter it.

Undo functionality? By using CVS tags intelligently, we could allow
rolling back changes.

Add topic editor role. A topic editor can edit any documents within
that topic.

Add a list of mirror sites, and build a table of them organized by
country, region/province/state, and city.
Eventually this should check on the mirrors and report when they fail
to update or when they are down. We might embed a timestamp in
the mirrored HTML somewhere, so Lampadas can check it.
Maybe a meta-tag, "generator".

Improve the security system so that instead of single boolean fields to get
permissions (e.g., the admin and sysadmin fields), each user can be assigned
a role which comes with a set of permissions. This will allow more fine-
grained control over permissions, so admins can customize them to fit their
particular organizational needs.

The various boxes really should be editable, as the pages are.
Then site administrators can remove fields they do not use as part of
their workflow practices. And it should be faster w/o concatenation.