Proposing a New App Developer Upload Process

[Didier Roche]

Le 05/09/2012 16:26, David Planella a écrit :
> Al 05/09/12 14:29, En/na Scott Kitterman ha escrit:
>>
>> David Planella <david.planella at ubuntu.com> wrote:
>>
>>> Al 05/09/12 05:18, En/na Emmet Hikory ha escrit:
>>>> Steve Langasek wrote:
>>>>> On Tue, Sep 04, 2012 at 05:20:47PM -0400, Michael Hall wrote:
>>>>>>> It's possible that namespacing within /usr - for instance,
>>>>>>> requiring each subdirectory and binary name to be prefixed with
>>>>>>> 'extras.' - would give better results with regards to the upstream
>>>>>>> build systems, I'm not sure. But if we are going to try to do
>>>>>>> namespacing of extras packages to avoid the need for coordination,
>>>>>>> we definitely would need improved package helpers that support
>>>>>>> namespacing of packages (i.e., debhelper support).
>>>>>> Would it be reasonable for MyApps to add a package name prefix to
>>> the
>>>>>> submitted package, rather than requiring the developer to do so?
>>>>>> MyApps will already be modifying the submitted package to include
>>> the
>>>>>> generator AppArmor profile.
>>>>>> So, for example, if I submit quickly-gtk as a source package to
>>>>>> MyApps, it would convert it to extras-quickly-gtk, add the AppArmor
>>>>>> profile, and re-build the source package before submitting it to
>>> the
>>>>>> PPA/buildds.
>>>>> For package renaming that seems easy enough to do, but if we're
>>> worried
>>>>> about file conflicts, dynamically prefixing filenames when building
>>> the
>>>>> package is going to have an even worse impact on the upstream code
>>> than
>>>>> changing directories does.
>>>>      Changing the base filename indeed generates more complicated
>>> issues,
>>>> not only in terms of collisions, but also in terms of coordination
>>> between
>>>> code and data (e.g. code expects to find ${CONFIG_DIR}/package and
>>> fails
>>>> to initialise properly with only ${CONFIG_DIR}/extras-package.
>>>>
>>>>      However, if we consider "renaming" to apply to the entire path,
>>> rather
>>>> than the bare filename, this becomes considerably less of an issue.
>>> If
>>>> MyApps were to call tooling that changed the default installation
>>> location
>>>> for files while preserving bare filenames, then there is less
>>> potential
>>>> for conflict.  The standard mechanism for this is to place the
>>> non-system
>>>> files in per-package namespaced directories in /opt.
>>>>
>>> I fully sympathize and understand the advocacy for the use of /opt at
>>> the packaging level and I would in principle support it: it is the
>>> standard FHS location for add-on software packages and it creates a
>>> separate installation namespace that prevents file collisions. In
>>> theory, it seems the best approach.
>>>
>>> However, reality has shown that (a) it is a big hurdle for app
>>> developers (who are actually the people we're trying to make the
>>> process
>>> easier for) to follow this policy and (b) we're enforcing a policy not
>>> even our tools support.
>>>
>>> For (b), what I believe is that it is also clear that no one is going
>>> to
>>> work to improve /opt support in tooling or in developer toolkits in the
>>> near or distant future. So for this alone I consider it to be a dead
>>> end.
>>>
>>> We are assuming that build systems and libraries are flexible enough to
>>> cater for an alternative installation prefix, and that it will all just
>>> work at runtime. Unfortunately, this has proven not to be the case. And
>>> I think the amount of coordination and work that'd be required to
>>> provide solid /opt support in Ubuntu would be best put in other parts
>>> of
>>> the spec, such as its central one: sandboxing.
>>>
>>> Just to illustrate the kind of issues we've bumped into in MyApps
>>> submissions, here's just an example [1]: GtkBuilder does not work with
>>> the gettext Python library if you specify an alternative location to
>>> load translations from (e.g.
>>> '/opt/extras.ubuntu.com/$APP/share/locale/'). So we had to either wait
>>> or contribute to fix the upstream bug (unlikely as per the complexity
>>> involved) or implement a workaround. The workaround was to get Quickly
>>> to modify the source code to use 'locale' instead of 'gettext': an
>>> effective but nasty solution.
>>>
>>> And that's been the journey with /opt so far: continually playing catch
>>> with the next exception we have to fix or work around. This does not
>>> sound like a very solid approach or a good experience to provide to app
>>> developers. And again, I think we should rather direct resources where
>>> higher priorities lie.
>>>
>>> While I like Emmet's idea of delegating the changes required to work
>>> with /opt to MyApps, this would also mean that the complexity in the
>>> logic behind the server would greatly increase. And in some cases (e.g.
>>> hardcoded paths) we would also need to actually modify the sources to
>>> ensure an app runs.
>> I understand it would be a lot of work and people aren't working on it.  What's your basis for believing there will be resources available to implement this alternate approach?
>>
> We want to make Ubuntu a platform for app developers. This process has a
> driver, if agreed upon, blueprints will be drafted and discussed at UDS,
> and resources assigned in the same way we do every cycle.
>
> As far as I know, there is no one driving, volunteering or particularly
> interested in doing the work to better support /opt across our main
> build systems and runtime or development libraries (and if anyone is,
> please speak up!).

I completely concur with David here. Having implemented it in Karmic, 
the /opt hacks for our build system (touching a lot of build components, 
like cdbs, debhelper, distutils…) for extra support and that were 
reverted erroneously some times, I can attest that supporting /opt is 
not easy. In addition to that and as already mentioned, ones need to 
patch all services, desktop file support, lens, scope and having various 
hacks like the gtkbuilder ones to get it working properly. I infer those 
are only part of the iceberg we discovered trying to support /opt and 
they are many others.

Trying to workaround with /opt/extras.ubuntu.com/applications or 
/opt/extras.ubuntu.com/share/dbus-1/services seems to be just to move 
the issue one step aside until the next one will come and won't address 
the issue with all use cases we will have in the future.

I'm really thrilled that we are taking the application developers as one 
the heart of our target focus, and I don't think we should ask them to 
do various hacks to support us otherwise the developer story won't be 
successful. In addition to that, we will have committed developers 
(people looking at myapps, new ubuntu releases and waiting their 
application integration to excel in ubuntu) only if we make their life 
as easy as possible. I guess all of that mean dropping /opt that we 
tried for almost 2 years now. I can't wait for the Quickly commit which 
will remove all the /opt hacks we had with Mike to do. :)

>>> * File name conflicts: there I would suggest exploring Daniel's
>>> proposal
>>> of relying on a conflict checker that works across all archives, so
>>> that
>>> before an upload is accepted this service checks for any potential
>>> clashes and informs the uploader to fix the package before they can do
>>> the next submission. The uploader would either be an Ubuntu developer
>>> (through the main archive) or an app developer (through extras, via
>>> MyApps). This would not only benefit the app developer process, but
>>> also
>>> fix the existing issue in the regular Ubuntu upload process.
>> This would be useful, but insufficient.
>>
> Could you elaborate on why you think it would be insufficient and what
> alternative you believe would be a solution for file name conflicts in
> this context?
>
>>> * Namespace ownership: even with conflict checking there is the issue
>>> of
>>> who gets to own a particular file name or namespace. E.g. would "Mad
>>> Feathered Creatures" (/usr/bin/birds, from Extras) have priority in
>>> owning the binary's name if it had been submitted before "Jolly Flying
>>> Animals" (also /usr/bin/birds, from Universe)? I think if we want to
>>> make apps first-class citizens, even if not part of the distro, a
>>> simple
>>> suggestion would just be to do it on a first-come-first-serve basis.
>> I think it is fundamentally incorrect to give something built on Ubuntu namespace priority over Ubuntu itself. Additionally, if this service proves popular, this approach would drive a permanent namespace wedge between Debian and Ubuntu that might, over time, significanly change the nature of the relationship between the two distributions.
>>
> I can see the point of the namespace wedge if we become immensely
> popular. What do you think the alternatives are?
>
>>> What are your thoughts on these?
>>>
>>> Finally, I believe we do need to provision for those cases, but my
>>> understanding is that the potential amount of occurrences would be
>>> rather low. So do you think they justify additional Engineering work,
>>> or
>>> rather they could be dealt manually on a case-by-case basis?
>>>
>> You've got to consider it now or it won't scale.  IIRC the original presentations on this topic at UDS Orlando(?), the intent was to be able to scale out to hit numbers of applications equal to or greater than the Apple Appstore/Google Play.  If you hit that, then MyApps ends up being several times the size of the Ubuntu archive.
>>
> And that's what we're doing right now. My only concern is not to block
> on a situation that will concern just a fraction of the uploads, even at
> a higher scale. That's what I'm trying to get a feel for.

For all those 3 namespacing/files issues, maybe we can think of a simple 
solution.

I really like Daniel 's idea of a conflict-check-before-publish service. 
One of the case that was raised on the thread is about "you can't 
predict the future". What about that example taking back the "Mad 
feathered creatures" shipping /usr/bin/birds
- in precise, only the apps from extras is there
- in quantal, we sync from debian, or upload directly to universe "Jolly 
Flying Animals" which ships the same file (new package or update)
-> nothing happens at this stage (remember that extras is not opened)
- a little bit (like 3 weeks?) before opening extras, we run (and then 
continuously run) the conflict-check-before-publish service. This one 
will detect the new conflict between the two packages, and:
1. add a Conflicts: in "Mad feathered creatures" debian/control file in 
extras against the package in universe.
2. will send an email to the app developer telling "hey, maybe not all 
ubuntu user will be able to use your apps as there is this excellent new 
application <…> into the archive"

At the extreme, if the component in conflict is a core component, as the 
ubuntu archive have an higher priority than the extra one (right?), then 
the core component will be preferred on dist-upgrade.
This has the advantage of:
- pushing the burden to the app developer, not to ubuntu developers
- avoid having to do conflicts/replaces on our side and so diverging 
from debian
- by pushing the burden to the app developer, still having a automatic 
update solution integrated into myapps, but mailing them, we ensure to 
have people committed to their application in ubuntu

I think this solution would fit for what will really be and stay, IMHO, 
a corner case. I doubt with all the precautions taken into the naming 
and namespace that will happen with every cycles and the few developers 
in that case will be warned and have time to react before the extras 
opening. In case they don't react, we have the automatic metadata 
addition in conflicts: which enables apt to deal with it.

What do you think?

Cheers,
Didier