Proposing a New App Developer Upload Process

[Emmet Hikory]

David Planella wrote:
> I fully sympathize and understand the advocacy for the use of /opt at
> the packaging level and I would in principle support it: it is the
> standard FHS location for add-on software packages and it creates a
> separate installation namespace that prevents file collisions. In
> theory, it seems the best approach.
> 
> However, reality has shown that (a) it is a big hurdle for app
> developers (who are actually the people we're trying to make the process
> easier for) to follow this policy and (b) we're enforcing a policy not
> even our tools support.
> 
> For (b), what I believe is that it is also clear that no one is going to
> work to improve /opt support in tooling or in developer toolkits in the
> near or distant future. So for this alone I consider it to be a dead end.

    Given the work that has been ongoing on this for the past few years,
I don't believe that we should simply dismiss it, especially by saying that
nobody is doing it or will do it, so obviously it won't get done.  It may
be that our current tools are insufficient, and that we determine that the
use of /opt is not part of our insulation solution, but let's take that
decision based on the relative merits of different mechanisms and the
identified volume of work required for them, rather than on the basis of
either assumptions about the motivations of others or circular reasoning.

    Similarly, if we are not implementing this in a way that is completely
transparent to the application developers, we are doing them a disservice,
as we are suggesting their upstream software be written in a manner that
reduces the ability for it to be packaged in other ways.

> Just to illustrate the kind of issues we've bumped into in MyApps
> submissions, here's just an example [1]: GtkBuilder does not work with
> the gettext Python library if you specify an alternative location to
> load translations from

    Thank you for providing an example without a trivial rebuttal: this
significantly increases my confidence that it is not just intransigence
that keeps us from wholeheartedly adopting this.

    Do we have bugs for all of these?  Are they tracked with a common
tag?  Even if we don't use this as part of our implementation, it remains
something that we should fix, if only in terms of FHS compliance (or, if
there are enough bugs that are truly insoluable, we should provide this
as input for future revisions of the FHS).

> While I like Emmet's idea of delegating the changes required to work
> with /opt to MyApps, this would also mean that the complexity in the
> logic behind the server would greatly increase. And in some cases (e.g.
> hardcoded paths) we would also need to actually modify the sources to
> ensure an app runs.

    To be clear, my intent was to isolate the complexity in the tools
used to build packages, expecting it to be a matter of appropriate
debhelper plugins, potentially combined with a package mangler installed
in the extras build chroots: if there is logic to enforce /opt stored in
the MyApps server code beyond that required to use the debhelper plugins,
then I'd claim the implementation widely differed from the requirements.

    Essentially, anything that can't be trivially emulated by someone
with a local build environment becomes unsuitable, as it limits the
ability of the developer to understand what is happening as a result
of their submission.  While some developers may have little or no
interest, there are plenty who would want to perform some local testing
prior to upload to MyApps.

> Summarizing, I think these are the 3 main points of discussion right now:

    Unless everyone happens to be silently agreeing with me, I think there
is a fourth: that it makes sense for MyApps to be able to deliver
applications into a qualified human-based review process for inclusion
in the primary repositories in cases where the application wants or needs
closer integration to the system than is provided by the automatic
insulation mechanisms proposed, and that the result of such review should
include backporting the application to the current release so that the
application remains delivered to the target intended by the submitter.

> * Package name conflicts: it seems that we've agreed that a solution for
> package name conflicts is to rely on 'extras-' prefixing on the server
> side (i.e. MyApps). This seems to be easy enough to do, fit well with
> other parts of the spec and would be transparent to app developers.

    Rather than implementing entirely in MyApps code, could this be
implemented as some text-transformation library which can be called
identically from MyApps, some command-line tool, and any arbitrary
packaging helpers or IDEs that happen to want this support?

> * File name conflicts: there I would suggest exploring Daniel's proposal
> of relying on a conflict checker that works across all archives, so that
> before an upload is accepted this service checks for any potential
> clashes and informs the uploader to fix the package before they can do
> the next submission. The uploader would either be an Ubuntu developer
> (through the main archive) or an app developer (through extras, via
> MyApps). This would not only benefit the app developer process, but also
> fix the existing issue in the regular Ubuntu upload process.

    To avoid conflicts of interest, I suggest that agreeing that the
content of the repository to be populated through MyApps is to be treated
as part of Ubuntu is a prerequisite: otherwise there is too much potential
for those Ubuntu Developers who have decided to entirely ignore the presence
of a third-party external repository called "extras" to complain that their
upload is unreasonably blocked, and no sensible conflict resolution path
due to the lack of a common authority other than sabdfl.

> * Namespace ownership: even with conflict checking there is the issue of
> who gets to own a particular file name or namespace. E.g. would "Mad
> Feathered Creatures" (/usr/bin/birds, from Extras) have priority in
> owning the binary's name if it had been submitted before "Jolly Flying
> Animals" (also /usr/bin/birds, from Universe)? I think if we want to
> make apps first-class citizens, even if not part of the distro, a simple
> suggestion would just be to do it on a first-come-first-serve basis.

    Blindly allowing first-come-first-serve is likely to be fairly annoying
(see the discussion about node), rather we ought generally allow developers
to use any unclaimed name, and encourage collaborative dispute resolution
for conflicts, with a clearly identified authority enabled to take decision
in cases where such collaborative resolution is unable to reach a solution
acceptable to the parties concerned.


> Finally, I believe we do need to provision for those cases, but my
> understanding is that the potential amount of occurrences would be
> rather low. So do you think they justify additional Engineering work, or
> rather they could be dealt manually on a case-by-case basis?

    If we're talking about hundreds of applications, yes the potential
for conflict is low.  If we're talking about hundreds of thousands of
applications, we should expect lots of conflicts.  It seems better to
take the time now to determine sensible policy, and begin to develop tools
that implement that policy, rather than attempting to generate ad-hoc policy
based on exceptions as they are discovered: any specific case surely has
merits and demerits that may weigh the decision in such a way that we end
up creating precedent that makes it hard to decide the other way for the
next specific case that apparently would encounter the same policy.

    Of course, there are surely many sorts of conflict that we have yet
to understand, so we ought take care to ensure that we have exception
processing built into all the policies we create surrounding this, with
facility for the exceptions to be handled in a way that does not force
the creation of precedent that may harden into policy later, but rather
specifically endorses the discussion of the policy issue independently
of the processing of the package which demonstrated the exception.

-- 
Emmet HIKORY