Proposing a New App Developer Upload Process

[Scott Kitterman]

On Friday, September 07, 2012 11:01:55 AM Matthew Paul Thomas wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Scott Kitterman wrote on 06/09/12 22:02:
> > On Thursday, September 06, 2012 09:36:05 PM Matthew Paul Thomas
> > wrote:
> > 
> > ...
> > 
> >> Scott Kitterman wrote on 06/09/12 14:22:
> > ...
> > 
> >>> To pick just one example, rolling delivery of applications and
> >>> offering multiple versions of the same package (which is what
> >>> the BSD Unixes do, although not in a way I've personally found
> >>> at all satisfying as a user) implies huge changes in package
> >>> management. Not the least of which is the ability to support
> >>> downgrades, including migrating per user settings back to
> >>> previous versions.
> >> 
> >> Windows, OS X, iOS, and Android all let an author issue a new or
> >> updated application within a week or so of wanting to, which I
> >> guess is what you mean by "rolling delivery". This whole
> >> discussion is predicated on that being a requirement for a
> >> mass-market OS.
> > 
> > On iOS and Android there's rather more extreme sandboxing than is
> > being proposed here that make potential interactions between
> > applications less of an issue.  The term DLL hell was invented for
> > Windows.  I don't think it's at all obvious that replicating the
> > existing systems is a path to success.
> 
> DLL hell or no, Windows and (lesserly) Mac OS were both popular for
> decades before they both introduced sandboxing this year. But let's
> leave aside the definition of "success".
> 
> What kind of sandboxing, specifically, do you think would be necessary
> for hundreds of thousands of Ubuntu applications not to interfere with
> each other? It seems to me there are four possible points of contention:
> 1.  package names (versus the OS archive, and versus each other)
> 2.  installed files
> 3.  saved documents and settings
> 4.  resource use (memory, CPU, network, peripherals) while running.
> 
> All four might have benefits, but for this particular goal I think
> it's sufficient to have technical measures for #1 and #2. Do you at
> least agree on that?

For this particular goal, #1 and #2 are essential.  Some elements of #3 and #4 
are important as well to meet the security goals associated with the proposal, 
which is a critical element of reducing review/allowing third parties more 
direct control, so I agree we have to have #1 and #2.  It may even be sufficient 
to start with #1 and #2 designed and work out the details on #3 and #4 as the 
project matures.

> > There is an assumption here that there's a vast user base that
> > awaits if only it weren't so hard to run the latest versions of
> > things.
> 
> That's a misunderstanding. There are, indeed, users who get annoyed
> when they can't play a game like Hedgewars online because the packaged
> version of the game is too old for the server. (Raises hand.)

We have mechanisms in place already to fix this problem, but that's a separate 
issue.

> But the main motivation is that one of the things we need for a vast
> user base is an order of magnitude more and better applications. And
> one of the things we need to attract those application developers is
> the ability for them to issue new and updated applications at will.
> It's not sufficient, but it is necessary.

OK.  I guess that's the crux of the argument.  We need better/more 
applications (personally, I think better is far more important than more, but 
I guess that's a detail) and you believe more users will attract the 
developers to achieve that.  I suspect that helps more with more than better, 
but who knows.  For commercial vendors, I can definitely see the more users == 
more software, but for FOSS developers is there a significant body of 
developers for whom Ubuntu isn't already big enough to be important?

> > ...
> > 
> >> None of those OSes offer application downgrades (though
> >> individual applications might). None of them migrate user
> >> settings to previous versions. ("The file “iTunes Library” cannot
> >> be read because it was created by a newer version of iTunes.
> >> Would you like to download iTunes now?") And absent that settings
> >> problem, OS X is the only one that makes multiple application
> >> versions moderately easy ("portable apps" on Windows being the
> >> exception rather than the rule). That tells me that while they
> >> might be desirable, none of those three things is a requirement
> >> for a mass-market OS.
> > 
> > It is also quite normal for things to get screwed up on other
> > operating systems and people do a full reinstall to fix it.  One
> > of the fundamental design principles behind package management in
> > Debian and by extension Ubuntu is that once installed, systems can
> > just be upgraded.  Reinstalls should not be needed.  I think this
> > is an important feature that should not be lightly abandoned.  I
> > think the implication of that is you've got to find a way to
> > support downgrades.
> 
> It is not an implication of that, because Debian and Ubuntu have never
> allowed downgrades. Upgradeability may be a design principle behind
> package management in Debian and Ubuntu, but downgradeability is not.
> 
> Lack of rollback in general is a big problem: woe betide you if your
> battery runs out during an update. But again, that doesn't mean that
> problem is best solved together with the app developer upload problem.
> If you think it should be, now is the time to explain why.

I don't think it needs to be solved now.  I think it does need to be solved 
though.  What we need to do now is think enough about how we would solve it 
when the time comes to make sure that what we're doing now doesn't preclude 
good solutions to this problem when the time comes to address it.  I don't yet 
have an opinion about this.

It's primarily meant to be exemplary of the other issues that will, in time, 
need to be dealt with that should be part of the overall architecture of 
change within which this particular change should be considered.  We don't 
want to preclude solving other important problems later or solve this one in a 
way that is limited such that we'll have to solve it again, differently, when 
some other issue is tackled.  

This will take a significant investment of resources and I believe it's 
important to consider how it fits in with the overall road map for getting to 
however many users the current goal is (having missed the last few UDS's, I've 
no idea what the current number is).

> > I think it's feasible, perhaps using something like etckeeper, but
> > it would take work.
> > 
> > Allowing areas where we beat the proprietary competition in
> > quality/ capability to degrade to their level is not a recipe for
> > success.
> 
> Beat them how? Windows Installer allows rollback to a known-good
> state. dpkg does not.

Obviously that's not an are we beat them, but I think we certainly do for long 
lived, upgradeable installations.  I've been using Ubuntu full time on 
servers, desktops, laptops, and netbooks, starting with Dapper in 2006.  In 
all that time, I've only ever had to reinstall due to a system problem once 
and that was a time when I was doing something I knew in advance was risky and 
unsupported.

It is not considered abnormal in the Windows world to decide you need to 
proactively reinstall the entire operating system on some periodic basis.  
I've known people to do it as often as every six months.  Back when I used 
Windows (and it may still be the case) the received wisdom was never to 
upgrade, always do a fresh install because upgrades (I'm talking about OS 
major release upgrades here, not individual application upgrades) were very 
problematic.

While we certainly have significant issues with upgrades (I'm not trying to 
minimize them), I believe that for upgradability and recovery from system 
problems without reinstall, Debian/Ubuntu already provide a better offering.

> > ...
> > 
> > The current ARB process and this new proposal are focused on a
> > specific class of applications that make use of some specific
> > restricted features of the operating system.  The current proposal
> > is geared towards that.  If the real goal though, is to deliver
> > all applications this way, I think this proposal is not very
> > useful because I don't think it's extensible in the ways it would
> > need to be to grow into that.
> > 
> > ...
> 
> What else would it need? Be specific.

The problem is that this process, in order to allow the automation, sets a 
number of very specific constraints about file installation locations, types of 
system interactions, and tools that do not fit most packages.  To be a general 
solution to achieving the goal of allowing application developers to upload 
packages directly without review, it either needs to be extensible to allow 
most all applications to work through it or most all applications need to be 
modified to meet it's requirements.  I don't think either is realistic.

The new process was proposed to deal with a specific class of add on 
applications, not as a generalized solution for all packages.  Modulo some of 
the details we are discussing, I think it's a decent solution for the class of 
applications it targets.

Not all applications are going to use quickly.  That's one specific example of 
the tool set constraints that limits the extensibility of this proposal.

Communications are (reasonably) limited.  I don't see (as an example) any 
mention of access to bluetooth.

It's a fundamental design principle of this approach that we're 
removing/reducing the need for trust of the uploader by removing/reducing 
various system resources.  Applications that need access to these resources 
won't work in this system even if they are re-engineered to use Ubuntu specific 
tools like quickly.

Scott K