Proposing a New App Developer Upload Process

[Emmet Hikory]

Didier Roche wrote:
> I completely concur with David here. Having implemented it in
> Karmic, the /opt hacks for our build system (touching a lot of build
> components, like cdbs, debhelper, distutils…) for extra support and
> that were reverted erroneously some times, I can attest that
> supporting /opt is not easy. In addition to that and as already
> mentioned, ones need to patch all services, desktop file support,
> lens, scope and having various hacks like the gtkbuilder ones to get
> it working properly. I infer those are only part of the iceberg we
> discovered trying to support /opt and they are many others.
> 
> Trying to workaround with /opt/extras.ubuntu.com/applications or
> /opt/extras.ubuntu.com/share/dbus-1/services seems to be just to
> move the issue one step aside until the next one will come and won't
> address the issue with all use cases we will have in the future.

    Regardless of our final decision with regard to the implementation
limiting the impact on users of potentially malicious automatically
reviewed applications, I think we ought continue to make efforts so
that our tools support /opt properly.  There are many existing popular
and successful free and commercial applications using /opt today for
other unices, and we can only benefit by reducing the porting effort
for this software to Ubuntu.

> I can't wait for
> the Quickly commit which will remove all the /opt hacks we had with
> Mike to do. :)

    I'm in full agreement with this: Quickly should certainly not have
any of the existing /opt hacks.  If our typical tools have good support
for /opt, the most Quickly should have is some option to allow the
package build to be placed in /opt or not, where that option is
simply passed to the underlying tools.  If our typical tools do not
have good support for /opt, we should be filing bugs against those
tools and resolving this, rather than attempting to work around it.

> I really like Daniel 's idea of a conflict-check-before-publish
> service. One of the case that was raised on the thread is about "you
> can't predict the future". What about that example taking back the
> "Mad feathered creatures" shipping /usr/bin/birds
> - in precise, only the apps from extras is there
> - in quantal, we sync from debian, or upload directly to universe
> "Jolly Flying Animals" which ships the same file (new package or
> update)
> -> nothing happens at this stage (remember that extras is not opened)
> - a little bit (like 3 weeks?) before opening extras, we run (and
> then continuously run) the conflict-check-before-publish service.
> This one will detect the new conflict between the two packages, and:
> 1. add a Conflicts: in "Mad feathered creatures" debian/control file
> in extras against the package in universe.
> 2. will send an email to the app developer telling "hey, maybe not
> all ubuntu user will be able to use your apps as there is this
> excellent new application <…> into the archive"
> 
> At the extreme, if the component in conflict is a core component, as
> the ubuntu archive have an higher priority than the extra one
> (right?), then the core component will be preferred on dist-upgrade.
> This has the advantage of:
> - pushing the burden to the app developer, not to ubuntu developers
> - avoid having to do conflicts/replaces on our side and so diverging
> from debian
> - by pushing the burden to the app developer, still having a
> automatic update solution integrated into myapps, but mailing them,
> we ensure to have people committed to their application in ubuntu
> 
> I think this solution would fit for what will really be and stay,
> IMHO, a corner case. I doubt with all the precautions taken into the
> naming and namespace that will happen with every cycles and the few
> developers in that case will be warned and have time to react before
> the extras opening. In case they don't react, we have the automatic
> metadata addition in conflicts: which enables apt to deal with it.

    I can think of two classes of potential conflict here, both of
which we likely want to have a general solution for prior to agreeing
on this implementation:

1)  Backports

    We end up with an emerging essential tool that we want to promote
as standard for the next release, is suitable for backporting, and for
which there is user demand for availability in the current release.
Unfortunately, there is a filename collision with something in extras,
and we believe that the final solution should be in favour of the new
package we wish to backport.

    One potential way to address this issue would be to notify developers
of extras packages that we may remove or modify their packages in extras
or include them as system packages without notice and at our discretion
to resolve potential future filesystem conflicts: this gives us a selection
of several technical means resolve potential conflicts.  I imagine we ought
to be able to grant PPU access to affected developers in these cases so as
to reduce the impact to their ability to continue to maintain their
software.  (Note that automatically adding Conflicts: and rebuilding is
insufficient in this case, due to upgrade/install ordering issues on
end-user systems).

    Obviously, the determination of the "correct" solution in these cases
quickly becomes more complicated depending on the perception of the
importance of the backported package and the extras package, but by
including the potential for direct inclusion in the distribution, we
reduce this to the existing potential for conflicts between any two new
packages in Ubuntu, for which we really ought have a set of sensible
solutions in any case.

2)  Local user files using filenames and upgrades

    Some user has decided that their favorite extras app should
start whenever they log in.  The executable for this app shares
a name with a program that opens all files under /var/www and
keeps them open as a means to maintain active cache of files
and reduce first-load-time for rarely used files on a static
webserver.  The user finds their machine strangely slow and
regularly swapping and complains that the upgrade was a bad
idea, and they want to use the previous release.

    This can happen for many sorts of commonly created local
files, although generally the results are less extreme.  We
have any number of tools in all of the desktop environments
that allow the creation of custom launchers, startup programs,
group launches, and more.  Assuming we have done a good job
with the migration of any conflicts, it seems unreasonable to
expect users to review or regenerate all of these on upgrade.

    One potential way to address this issue would be to have
our preferred upgrade mechanisms scan common locations in all
home directories for such files, and if they reference paths
that we know to have been conflicting, provide some notice that
these files are in need of close review: while we can't guarantee
the administrator will actually tell the concerned user, this would
cover the single-user-machine case, likely also cover the
responsible-administrator-of-multiuser-server case, and has some chance
of covering the two-to-three-user-shared-machine case.  Ideally, the
implementation would be something that would allow all

    Of course, as much as I harp on this, the underlying issue isn't
significantly different from any file namespace changes that we might
deploy as part of regular distribution development, but if we wish to scale
to an arbitrarily large number of available applications, I think we will
see significantly more of these conflicts, and should consequently take
greater care to protect out users from the potential adverse effects.

-- 
Emmet HIKORY