documentation logo

[Dustin Kirkland]

On Fri, Sep 5, 2008 at 2:00 AM, Matthew East <mdke at ubuntu.com> wrote:
> In this case, there isn't any doubt that the utility is a useful one:
> it's a great idea as those who responded to your initial request for
> comments agreed. But there is one important question: why does this
> project warrant a separate website, as opposed to going directly into
> help.ubuntu.com?

The scripts generate 60,000+ unique HTML pages per distribution
(dapper, feisty, gusty, hardy, intrepid).  Note that I dropped Edgy
since its support is ended.

They also copy the raw, gzip'd manpages to a similar location, which
feeds into phase 2 of this project...  Patching man itself to
optionally download the manpage.N.gz from the manpages.ubuntu.com
site, if it's not found on the local system.  I have a trivial shell
script wrapper for man, but long-term, I've discussed this thoroughly
with cjwatson, and he's going to help me work it into man itself.

We're talking about 300,000 unique HTML pages, plus 300,000 unique .gz
files, comprising approximately 3.4GB of generated data.

"Generated" being the key word.

The scripts that generate this content are best run in a confined
sandbox.  For one thing, they need local access (NFS-mounted, or some
such) to a full Ubuntu archive mirror.  And secondly, when they run,
they can really impact the performance of the server.  It needs 8-12
hours to generate the initial repository from scratch, and 20-30
minutes per night for the incremental updates.  At 20-30 minutes per
run, we can afford to run them nightly, and it's really nice to pick
up daily changes to the distribution under development (Intrepid,
currently).  This process is essentially a no-op on the stable
distributions (see the code!).


> Starting a new website under the ubuntu.com domain has traditionally
> been a big deal in the Ubuntu community.

True, it certainly wasn't trivial!

I wrote most of the code for this in May, shortly after the mailing
list post and during UDS Prague.  I didn't follow up on the list after
my initial post since the reception was somewhat lukewarm, and I
rarely saw technical discussions taking place on ubuntu-doc at .

It was usable on my demo system by June.  Then it needed a security
audit, for which I thank Kees Cook.  (Big thanks to Kees and Jamie
Strandboge, by the way, for the audit, and some code, and generally,
some of the design ideas behind it.)  The security audit was done by
end of July, and it's taken Canonical IT a month or so to get the site
live (Big thanks to LaMont Jones for that!).  I'm working through the
remaining couple of minor bugs now, though it should be online for
production use by next week.  As I said, I'll blog about it and
announce here.


> I remember that when we
> wanted to create a wiki at help.ubuntu.com and move documentation off
> wiki.ubuntu.com, we spent several months working on the spec, took it
> to the Community Council, who rejected the idea. That was
> disappointing, but I ultimately understood that the reason was because
> we hadn't made a compelling enough case to do it, and they were
> rightly concerned about new websites confusing users and developers.

In the immediate term, there will be a redirect from
help.ubuntu.com/manpages that points to manpages.ubuntu.com.  I can
open a second IT ticket to get help.ubuntu.com/manpages to act as a
reverse proxy, such that all of manpages.ubuntu.com's content can be
transparently served directly through help.ubuntu.com/manpages (with
no redirect).  I'll need to dig through some sites I did years ago to
get the apache conf syntax right, it's very tricky, put perhaps can be
done ;-)


> I'm not saying you need to do something similar here, but I do think
> you need to make a case for a separate website. For me, the manpages
> are documentation, and they should go on the same website as other the
> rest of Ubuntu documentation: help.ubuntu.com. Creating a new website
> means that help.u.c will no longer be the unified documentation
> resource that it was intended to be, and users will need to keep two
> sites in mind.

The overwhelming majority of manpages is written by upstream authors,
independent of Ubuntu.  I have a mild concern that some people could
be misled that the Ubuntu Community "wrote" the 300K manpages served.
That's clearly not the case, and very much a distinction from the rest
of the content hosted on help.ubuntu.com and wiki.ubuntu.com, which is
overwhelmingly written by the Ubuntu Community.

The documentation at wiki.ubuntu.com is User-contributed.

The documentation at help.ubuntu.com is Documentation-Team-contributed.

This documentation at manpages.ubuntu.com is Generated from the
manpages already included in each and every .deb shipped in main,
universe, restricted, and multiverse.

To me, those are 3 very distinct forms of documentation.

Furthermore, I think there are far more forms of documentation spread
throughout the Ubuntu Community.  I've posted here before about the
Google Custom Search I've implemented for Ubuntu (yes, there are a
number of others out there):
 * http://people.ubuntu.com/~kirkland/search.html
 * https://launchpad.net/ubuntu-documentation-search

I consider a wider scope of Ubuntu Documentation to include:
 * Formal Docs
  * Official Docs (help.ubuntu.com)
  * Wiki Docs (wiki.ubuntu.com)
  * Man Pages (manpages.ubuntu.com)  <------- N E W
  * Blueprints (blueprints.launchpad.net)
 *  User-level Discussion
  * Answers (answers.launchpad.net)
  * Forums (ubuntuforums.org)
  * Mailing Lists (lists.ubuntu.com)
 * Developer-level Discussion
  * IRC Logs (irclogs.ubuntu.com)
  * Bugs (bugs.launchpad.net/ubuntu)
  * Code (code.launchpad.net)


> You've driven this project, so you have a much better understanding of
> its needs than I do. So it may be that there are genuine reasons for a
> separate website with a new subdomain, which we don't know about yet!

Well, this wasn't intended to fork the documentation community in any
way.  Far from it :-)  I think you've stated above that you appreciate
the contribution--and I'm happy to work with anyone who would like to
contribute to the project I established in Launchpad, and in
particular with anyone who wans to help with "Phase 2" (the man
integration).

I think there are decent reasons why the code that generates this
belongs on a separate server, which we called manpages.ubuntu.com out
of a lack of imagination ;-)  But I'd like to tie it to
help.ubuntu.com in any way that makes sense.

:-Dustin