brainstorming for UDS-N - Application Developers

[Krzysztof Klimonda]

Most of the following response makes sense only if we treat Ubuntu as a
Linux/Software distribution. At the bottom is a short comment about
transforming Ubuntu into a core system that is being shipped and
maintained by us and creating a set of tools and practices for ISVs to
create applications for Ubuntu OS (as opposed  to Ubuntu Distribution).

It's a joint response to both mpt's and jcastro's mail but I'm using a
single email to keep it focused.

On Mon, 2010-10-04 at 15:31 +0100, Matthew Paul Thomas wrote:
> Yes, that was Evan's point. The only way we would have time to do that
> well for every Ubuntu application, would be if Ubuntu's application
> selection was even tinier than it is.
> 
> The usual consequence of the current approach is that people have to
> suffer a bug for up to six months after the application developer has
> released a fixed version.

Yes, it is a problem but not that bad. If application is popular there
is a good chance that both upstream and ubuntu developers know about it
and keep it maintained. That leaves a lot of packages that don't have
any active Ubuntu maintainer - some of them are cared by upstream
developers who ask us -- either by sending mail to the u-d-d mailing
list or by asking on #ubuntu-devel/#ubuntu-motu IRC channel -- to upload
a fix. That of course, leaves quite a bit of applications but, at least
in my opinion it's not that usual to say that we have to revamp it.

> 
> There has never been a "big distribution". No distribution, Ubuntu
> included, has ever had more than 2 percent market share.

big in terms of developers, of processes used for development etc rather
than the market share (or even "with big market share compared to other
distributions")

> 
> Every general-purpose operating system has gone through a stage where
> the primary publisher of software for it was the OS vendor themselves.
> Every popular general-purpose operating system quickly outgrew that stage.

Both Windows and Mac OS X were created with a different mindset than
Linux Distributions. So maybe we should change our mindset.

> Yes, that's part of the problem. Packaging should not require that
> experience and understanding of the system. Thirteen-year-olds write and
> sell iPad applications.

No, they can't. You have to be of the legal age (and pay $99 a year) to
even get a chance to send your application for a review. Same with
android (but you have to pay less). And that's on platforms created from
the bottom up to ensure that single application can't interact with
neither system nor other applications, not to mention have a limited
access to users' data.

> >  Adding a detailed changelog makes it easier for our users
> > to make sure what can they do with their software. detailed
> > dependencies make it easy to make transitions etc.
> Do you think that more than one in a thousand users of the average
> Ubuntu application have ever looked at its changelog?

So? Are you suggesting that we should not care about those, more
technical and knowledgeable users? If we do that we may find ourselves
without a balanced community that is able to support itself (in form of
more experienced users helping less experienced ones) - I can already
see it problem in our LoCo forum.

> <https://wiki.ubuntu.com/PackagingGuide/Complete> is 16518 words long.
> Even the "Basic Packaging" section alone is longer than the US
> Declaration of Independence, and the original US Constitution, combined.

But that's a different problem - yes, we can work on getting our
documentation fixed. At the same time creating a packaging for a single
Desktop application takes few minutes assuming it uses autotools or
another build system supported by debhelper - e.g. debian/rules is 3-4
lines long, debian/changelog is created by dh_make, there is no need for
maintainer scripts other than those created by debhelper.

> 
> >> We need to make it so that developers can quickly deploy an
> >> application that then appears in Ubuntu Software Center for anyone on
> >> that release of Ubuntu, regardless of where we our in our own cycle.
> > 
> > The can add software to the current development release and request a
> > backport to previous releases.
> 
> Which, as Jorge said, they don't.

But is it because the process is wrong or because people are not aware
of it? Maybe promoting backports (and allocating more resources to make
it snappier) is a better way.

> 
> > (...)
> >> This application deployment system I describe should at no point block
> >> on a human being from the Ubuntu project like our existing systems,
> >> both the regular archive and -extras, do.  These quite simply cannot
> >> scale.  Instead, lets create an automatic system that takes an
> >> application submission, runs its test suite, does some static
> >> analysis, and conducts any other sanity checks we can think of before
> >> letting it through to the archive.  If any of these tests fail, it
> >> simply rejects the application back to the developer.  Let them worry
> >> about fixing it.  Let them take complete ownership of their work.
> > 
> > No, we create a distribution for our users not as a place for
> > developers to dump their software to.
> 
> In the past four years, not one but two operating systems have each,
> starting from zero, accumulated more than 20 *times* as many
> applications as Ubuntu has. What we are doing is not working. It's time
> to try something else.

Why? Most of application for both iPhone and Android are completely
useless. You can hear people complaining about it all the time - dozens
of applications doing the same thing, "fart apps", direct clones. Sure,
they inflate the number of total applications but I'd disagree that they
make platform more appealing to users.

> 
> >                            Unit tests, static analysis, any sanity
> > checks we can think of can be either bypassed or plain wrong.
> 
> Why would a developer want to bypass checks that would help save them
> from disappointing their users?
> 
> Do you have any examples of sanity checks that produce unfixably wrong
> results?

Yeah, If we make it easier and easier to get applications into our
software center it's going to get worthwhile for attackers to use
software center as a point of attack for users' computers. To either
tamper with their data or to just use new computers for a botnet of
sort.

> 
> >                                                               Your idea
> > could work if all applications were sandboxed (i.e. Android model) but
> > not in Ubuntu.
> 
> Ubuntu Software Center, for example, has unit tests without being sandboxed.

But without being sandbox and tight control over what application can
and can't do the risk is too high. USC has unit tests to ensure that it
works and that there are no regressions. But we know the code, we trust
the code and developers not to do something bad. But the did earn our
trust.

> 
> > There is a scaling issue and the burden on both maintainers and QA team
> > is big but removing this burden by removing them from the process is
> > not going to make Ubuntu a better distribution (for our users [1]).
> 
> Why not?

Because, if we ship everything under our name, we take responsibility
for it. And there is already a justified criticism that we don't do a
good enough job.

> 
> > [1] I use "for our users" quite a lot - that's because what is good for
> > developers (or even for us) is not always good for the users of our
> > distribution - for example developers want to push their new software
> > to as many people as possible in as little time as possible but, on the
> > other hand, not all users are interested in the newest and greatest
> > stuff.
> >...
> 
> That decision should be up to individual users and organizations.

But if we don't prepare small patches for applications the choice we
leave our users with is whether they should keep current software and
deal with bugs (and security patches) or should they update. Various
upstream projects use different development models and not all of them
release point releases with bug fixes.

> 
> >> This system wont catch everything and we shouldn't ever operate on the
> >> false hope that it will.  All it's meant to do is filter out the very
> >> worst that we can catch, and leave the rest up to the users to try and
> >> report back on.
> > 
> > When broken software reaches users it's already too late. By publishing
> > something in *.ubuntu.com "namespace" we take responsibility for it and
> > users are going to complain to us.
> 
> That's not credible. Neither Ubuntu Software Center nor Update Manager
> reveal the domain name of the software they install, even for people who
> know what a "namespace" is.

Yeah, that's why PPAs are broken - because everything can be shipped
through them be it an application or some core library.
But still people believe that by not using any third-party repositories
they are given a fairly stable and predictable system.

> 
> >> We should also give the developers flexibility in the way in which
> >> they stage out these deployments.  We should allow them to offer an
> >> application to say, 5% of the Ubuntu audience, wait for feedback, and
> >> if it doesn't have any grave bug reports coming in, push it to the
> >> remaining 95%.
> > 
> > Now, the idea sounds pretty nice but I'm not sure it would work well.
> > Leaving aside the question how to do that I'm concerned that it would
> > make community support harder and making people feel bad that they
> > "don't have an access" to the newer software while their friends do.
> > Things like this are done - beta tests etc. but not at this level. It
> > would feel like we, as Ubuntu, are limiting our users.
> 
> Things like this are often done at this level, and at much larger
> levels. Twitter is doing it right now with 105 million registered users.
> Microsoft does it for Office's half a billion users. And Facebook also
> often does it with half a billion active users. They would be
> incompetent if they didn't.

And neither of those companies and projects is based on free software
and a community of supporters, developers and advocates. It's one thing
for developers of closed applications or services to decide that they
want only a part of their users to have an access to new software
version (and we should provide tools for them to do that) and a
different one for free software projects to do that - if only because
it's not entirely free and open model.

> 
> >...
> >> We need to make it our goal that any software developer should be able
> >> to package their application without difficulty and without causing
> >> harm to the system.
> > 
> > Isn't it a goal for Quickly?
> 
> Only for the type of applications created with Quickly.

Right, but the current packaging process isn't hard as long as there is
a good incentive to learn doing this. There probably isn't one for
hobbyists but then Quickly is targeted exactly at them. Other developers
would obviously be interested in getting their software onto our
platform because it's the best one in town. What I don't see is how
making it easier to package new application (as opposed to publishing
them) is going to lead to creation of new "killer applications" written
only for Ubuntu (or Linux in general) that will make Ubuntu more
popular. 

> 
> >                              We can obviously work on making it easier
> > for developers to prepare a preliminary packaging but it still has to
> > be checked by someone familiar with Ubuntu and Debian.
> 
> Yes, that is part of the problem.
> 
> > Upstream developers shouldn't have to understand various distribution,
> > their packaging methods, policies they have in place.
> 
> That is true only for unpopular operating systems.

Not only popular but also a much better targets, financially speaking,
to justify learning of things that are not directly related to your
application. But, the question is, can we make Ubuntu more popular and
more.. worthwhile target for Developers by making packaging of
applications easier, keep the quality from falling and keep the current
package system?

All points I'm making here (and in the previous email) are probably only
valid if we keep the current development style. But the email I was
answering for was not about whether we should change Ubuntu from the
current model (where its, in fact, a software distribution) to the model
where we ship only a base system, a software store/center and a set of
libraries for developers to develop for Ubuntu. I actually believe that
it would be a really good idea to focus ourselves only on a core system,
but that requires quite a lot of discussion.

First question that comes to my mind is "Where should we put a line?"
i.e. what is a part of the "Ubuntu OS" - what should we keep maintaining
ourselves? But also, what about other projects depending on Ubuntu -
what will happen with Kubuntu, Xubuntu and other derivatives? How is
that going to affect our relationship with both Debian and communities
of our core applications? Actually, how is that going to affect our own
community.

-- 
Krzysztof Klimonda <kklimonda at ubuntu.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20101004/bb267016/attachment-0001.pgp