Moving w3m out of standard

[Soren Hansen]

On Fri, Jun 20, 2008 at 09:16:11PM -0500, Michael Hipp wrote:
>> If we look solely at the packages I suggested, I don't think 5-6
>> packages constitutes a greatly expanded list of items to be
>> installed.
> Agreed. But I work under the assumption that the list over time will
> grow greatly and not over a very long time at that.

Well, I'm not going to promise anything about the amount of packages.
First of all, it's not all up to me, so I couldn't keep the promise even
if I made it, and second of all, it's not really a useful metric as e.g.
packages get split from time to time, so even if nothing new is
installed, the package count can increase. I think a better promise
would be that I'll do whatever I can to make the set of packages
reasonable and sound. I'm as much against bloat as the next guy.

> If I may expound further (and I think this will speak to several of
> your questions below) I fear this list would become truly ridiculous
> if we pursue this path with anything less than a very hard line
> approach. 

> By way of example: I find it truly amazing, yeah, incomprehensible
> that someone would suggest a text mode browser should be installed by
> default on every server. My mind is unable to grasp the concept.
> Meaning no offense to any who hold this opinion, I just do not know
> how to even think about it.

Look at it this way: I doubt anyone wants w3m on their servers so that
they can read Planet Ubuntu or update their Facebook profiles while
they're in the server room at the console.

To enable our users to read man pages, we install man-db and groff-base
by default. If you're old school enough to know how to use it, groff can
be used as a very full featured typesetting system, which is also not
something that really belongs on all of our servers by default, but
because it's a prerequisite for viewing man pages, we accept it.

Likewise, it's a simple fact of life that some pieces of software have
chosen to provide their documentation in html format. In order to enable
users to read this documentation, we need something that can transform
html into something legible. At the moment, this is w3m.

> >> The rational-defensible approach, IMHO, is to keep the base install
> >> as lean as possible but also make it as easy as possible to layer
> >> on top of that base.
> > What is the goal you are trying to achieve by doing this?
> Thank you for asking. I think I answer this in pieces below.
> 
> I is somewhat scary tho as I thought this was actually a design goal of sorts 
> of Ubuntu (somewhat) and ubuntu-server in particular. 

Up until now, Ubuntu Server has consisted of nothing but the packages in
the standard seed (+ a kernel, of course). Great care is taken to make
sure nothing unnecessary is added to standard, so up until now, you've
actually been right. What I'm suggesting is that we change this policy
slightly to provide more of an integrated platform, based on what we as
a team (which includes anyone who wishes to participate) find to be best
practices.

Package maintainers put great care into making their packages as good as
possible, e.g. adding all the right options when they're compiling
software, putting stuff in the right places, etc. If someone has input,
they can file a bug, discuss it, and if consensus is reached, the
changes can be implemented, so really all packages represent best
practic in terms of packaging and (some) configuration.

However, when it comes to choosing the right packages, and integrating
everything, making it all work together and such, every user has to
start from scratch.  Sure, this makes for a good learning experience,
but I don't think most users are here to learn, but to get stuff done.
Some people refer to this as "freedom of choice", but that's only half
the truth.  There's no reason we couldn't provide the alternatives, so
that people can make different choices than we have, while still giving
them the benefit of being able to simply use what we have determined to
be best practice. Between us, we have probably centuries of sysadmin
experience. We all have our set of packages we always install, a
specific setting we always change, etc., etc.

> The necessity to keep all images on one CD being an example.

This is indeed a goal of every Ubuntu version. At some point in most
release cycles, something is sacrifised, or packages are split, or
others consolidated, etc. to make sure they fit on a CD.  In the server
edition we just happen to have the luxury to be far from the limit. This
doesn't mean that we'll just nonchalantly toss a couple hundred
megabytes of cruft on there, I'm just saying.

>>> The tools that a LAMP stack admin should be using are probably quite
>>> different than what one of my very simple Samba boxes would require.
>>> I don't think either's list should dictate the other.
>> You seem to wilfully miss my point. I'm not trying to determine the
>> /union/ of useful tools for /every possible/ user of Ubuntu Server.
>> I'm trying to determine the /intersection/ of useful tools for /most/
>> users of Ubuntu Server.
> I don't believe I'm doing that, certainly not willfully. I suppose
> it's just that I'm comfortable that the intersection has already been
> reached.

That might be the case. I'd just like to have the discussion about it,
so that *I* can be sure, too :)

> Not to say there aren't things that should be added. But there may
> also be things that could be removed. 

And we should certainly look into that as well. No doubt.

> And, if I may... The seeming conclusion that w3m is a good candidate
> for being added looks to me like the beginnings of a "union" process,
> not an "intersection" process. But I'm probably guilty of making too
> much of one example.

I hope my explanation earlier can at least slightly change your view on
this subject?

>>> Take the 60% group that needs the package. The only pain they feel
>>> is the necessity, after install, to type a 30-character apt-get
>>> command. (I'm assuming it's on the CD or the net is available.)
>>> Their system is no worse for it.
>> Since they take their time to install it later, they find the package
>> to be useful. Hence, the lack of said package makes their system less
>> useful than it could potentially be. "less useful" is "worse" here,
>> IMO.
> But the "damage" is minor ... trivial. Given that it can be rectified
> in a few seconds by typing one short command. A data point certainly,
> but not a slam dunk.

Well, the "damage" of having a package installed that only takes up 100
kB and doesn't do anything until executed manually by the user (so no
CPU time or RAM is wasted) surely is trivial as well? To such an extent
even, that I can't imagine anyone would bother to remove it.

>> Some of the suggested packages are less than 100kB. I know very few
>> people who allocate partition space with smaller granularity that a
>> megabyte.  (The ones who do, I would certainly call "old school", but
>> that's a different point).  If you're doing so, how do you cope with
>> increasing size (added help messages, more code) of packages when you
>> upgrade, for instance? I'd say you're asking for problems.
> See "irrational fears" paragraph above. A meg here and a meg there and
> pretty soon you're talking about some real disk space to mangle a
> quote from another realm.

I'm not suggesting opening the gates for a flood of packages. I'm simply
suggesting that we consider if there's any additional things we want to
add to add value to our default install.

> If the discussion really comes down to adding 5 or 6 packages of 100kB each, 
> then I'm wasting way too much of your time talking about it. I just doubt 
> that's really the long term effect of this.

It's hard to make predictions, especially about the future, so I won't
bother. :) Instead, I'll deal with it when it seems to *actually* be
getting out of hand.

> I will also mention, in passing, the cases where I'm installing a very
> small server on a flash card where the difference between, say, a 1GB
> root partition and 2GB one could be very significant. This is, I
> think, somewhat of a special case where the use of 'apt-get remove
> --MANY' would be in order. So it's not the primary decision driver.
> But we also shouldn't lose sight of such things either.

Yeah, I've talked about something like this a few times, too. It really
is interesting that the server edition is what you use for really tiny
systems (like the one you speak of) and for really huge systems (big
iron stuff). This becomes especially interesting when we decide how to
configure the server kernel. 
 
>> It's funny that when I suggest installing a packages, I'm assuming
>> that we'll choose them carefully so as to avoid there being asked
>> additional questions on install, while you seem convinced that if we
>> add but a single package more to be installed by default, it will
>> surely be huge one which will ask tons of questions during its
>> installation, will ony be useful at all to at most 51% of our users,
>> and extremely offensive to rest.
>> 
>> If you have this little faith in us, I think you're using the wrong
>> distro.
> 
> No, definitely not. You guys have done way too much good work (both
> for Ubuntu's users and the good name of Linux) for that conclusion to
> be likely.

:)

> But I don't take it for granted that it must always be that way.

That's a reasonable point. However, I don't think we're at a point just
yet where this is likely to be a problem.

>>> c) remember to uninstall something that is likely
>>> out-of-sight-out-of-mind, 
>> 
>> Why would you bother?
> I probably wouldn't. But perhaps that makes my point. You offered this
> as a solution 

I wasn't so much suggesting it as a solution to anything as I was making
the point that packages can be both installed and removed. 

>> The idea -- which seems to be lost somewhere -- is to add additional
>> packages, which users will not want to remove (other than to be
>> pointlessly difficult).
> If that's really what we achieve, then certainly so. I just didn't see
> this heading that way. See the (tiresome) w3m example above.

:)
 
>>> And please note that I place great weight on item 'e' above,
>>> uninstalling is far worse than installing. 
>> Why do you bother? I'm genuinely curious.
> Well, I sorta thought I'd covered this already.
> 
> - Keep disk/memory footprint small.

The acceptance threshold for packages that include daemons or stuff that
runs from cron should of course be higher than packages that simply a
set of binaries. In most cases, it'll mostly be a matter of disk space.

> - Have no unnecessary security exposure.
> - Minimize risk due to unnecessary software updates.
> - Minimize work due to unnecessary software updates.

These are reasonable points.

> - Retain ability to run on low-end hardware.

I still intend to make sure that there'll be an option to only install
the bare minimum (what we currently call the "server install"), by the
way.

> I just really don't want to allow the idea that removing is just equal
> and opposite to installing. I regard one as far more damaging and
> risky than the other.

Do you have examples of things gone horribly wrong when removing
packages?

>>> I think that's a *good* thing because I consider a server to be a
>>> *platform* upon which I add the things (applications) that make it
>>> do what I need.
>> Does "do what you need" include keeping out crackers?
> Yes. Does w3m help with that? :-)

It might :)

> Would a server with lots of internet-facing ports (LAMP, DNS, email,
> whatever) need the same security tools as a samba-only server with no
> ports exposed to the Internet and no user accounts? 

Strictly speaking, any machine that's not physically disconnected from
any network, and physically unaccessible for anyone but its owner is at
some level of risk. Besides, with these kinds of things, I'll rather be
safe than sorry. I've thus far not heard any users complain that their
system was too secure. :)

> But I'm open to education in that sphere. There's been several
> mentions of "best practices" which is always a good thing to have
> available. But often in the Enterprise space, best practices is a code
> phrase for "most expensive" practices (mostly as a way of covering the
> backside). I'm just rambling to belabor the point that the spectrum
> here is incomprehensibly wide.
> 
> Finding an intersection, any intersection, may be difficult.

I don't think I ever said it was going to be easy :)

>> Can you perhaps take a minute to explain why you insist on assuming
>> that "lots of clutter" is what we'll add. All that is being suggested
>> is to add "a few tools that are useful to most people".
> If that's really the goal then I stand corrected. It just did not
> appear that was what we were doing. And we need to decide if
> "most"==95% or "most"==51% or whatever. I think 51% is insufficient
> given that a large base of users are being "harmed". 

51% was the amount I found that you seemed to think that it would only
be useful for. I hope that what we come up with will be useful for
perhaps 85%.

> If it's not obvious, I would be a proponent here of something
> approaching "minority rule". This, given that the "harm" to the
> majority in accommodating the minority is essentially nonexistent.

Certainly. However, the presence of the "bare minimum" install option
might have something to say here.

>>> The cli can only be made tolerable with tools like screen. 
>> ..yet you only want to share this useful tool with the few lucky
>> users who already know about it.
> I had it installed for years on servers and didn't know about it.
> (Someone on a Linux list educated me to it. Changed my life.) Having
> it installed bears no relationship to knowing about it. 

That certainly depends. Remember: This is free software. Ubuntu is a
complete operating system based on free software. This means that we
control *all* the software in Ubuntu, so if we want to do stuff that
involves screen, that doesn't just mean we can only change the screen
package to make it more useful. If we wanted to, we could make ssh
logins always log into a screen session, print a helpful message that
instead of logging out, you could detach the session, and if when you
connected again, it would simply resume the session.  We could also make
anything we know to be long running, interactive programmes (text-mode
bittorrent clients, for instance) run inside screen. 

In short, screen (and everything else we want to install by default) can
be just as discoverable and useful as we want it to.

> I've been installing Ubuntu servers now since fall of 2004. In all
> those many times of bringing up a server from scratch, it never
> occurred to me that typing 'apt-get install screen' was a hardship or
> something I should have been spared from.

It's not about saving people from having to type a certain command. It's
about making everything more useful by default.

> Here's a thought exercise: what about when the day comes that the
> server image is threatening to pass 700MB; which packages will be the
> first candidates to be removed? 

There are a number of things to be done at that point. We have plenty of
experience from the desktop edition to benefit from :) We can split
packages to remove the unneeded bits (header files, documentation,
etc.), we can identify packages that seem to ship much of the same stuff
and put that into seperate packages that they can then depend on.. Lots
of stuff.

> And if they're a candidate then, then why include them now? 

Well, because right now, we have plenty of elbow room to fit useful
stuff on there. Just because I know it's going to be dark tonight
doesn't mean that I can't enjoy the sunshine today.

> Just something to think about.

I see your point.

>> You might have noticed that at least some of the tools I suggested to
>> begin with were security tools. They're designed to solve security
>> problems. Their failure to be installed by default are this very
>> minute making scores of machines more vulnerable. (and it's all our
>> fault, because we failed to simply install those packages by
>> default).
> I have to disagree. I don't think this is your fault. (Which is not to
> say I do not take note of your willingness to take responsibility for
> providing the best product you can.) But I submit that merely having
> those tools installed automatically on my boxes will do little if
> anything to secure them. 

Again: These tools will be as useful as we want them to be. We could
tweak chkrootkit to run every 10 minutes and play loud, obnoxious music
at full volume, send you a Jabber message, an e-mail, a message on
Facebook, etc.  if it finds even the slightest hint of a rootkit. Heck,
we could even make it shut down entirely, just to contain the damage.
There are very few limits to what we can do.

> But I'm open to education.  Repeating myself, just having it installed
> is not the same as making use of it.

Correct. It isn't necessarily so, but it *can* be.

> Thanks for the discussion, sorry I'm taking so much of your time and
> being such a tiresome bother. I apologize for taking this so
> personally and taking it out on you.

I've had a few of these very lengthy discussions before, and I must say
that this one has proven to be one of the most useful ones, so no need
to apologise.  :)

-- 
Soren Hansen               | 
Virtualisation specialist  | Ubuntu Server Team
Canonical Ltd.             | http://www.ubuntu.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-server/attachments/20080623/9e4a6607/attachment.pgp>