Packaging large Java software stacks ?

[Thierry Carrez]

Martin Pitt wrote:

> You impressively described why packaging those "properly", i. e. with
> separate libraries and build dependencies, is not really sensible, if
> at all possible. It doesn't sound like we could support those kind of
> builds in our usual "backport patches" way of working anyway, if these
> components get constantly changed.
> 
> However, do we actually need to? It doesn't sound like we would ever
> put those into main, just making it available conveniently for Ubuntu
> users?

The top stacks (Geronimo) might become targets for main, based on our
support strategy. I agree with you it would be simpler if they weren't.

>> Potential solutions to solve (1) include using a parallel delivery
>> mechanism for JARs, separated from the usual Ubuntu packages, something
>> that could hook into Maven repositories... or get ready to create a
>> separate package for each version of each JAR.
> 
> So is Maven a system to pull Java libraries/projects from external
> well-known sources, much like Python eggs or Ruby gems? If those
> repositories already exist and are maintained, then we should rather
> think about integrating them better than trying to do the sysiphos
> work of packaging them again properly.

Maven is a project management tool that handles build-time dependencies,
rather than a more classic runtime library delivery mechanism. Remember
Java "binary releases" tarballs usually include all the needed external
JARs directly, so there is no need for a runtime dependency delivery
system. However Maven makes use of artifact repositories from well-known
sources so we could try to leverage those.

>> A solution to workaround both problems would be to avoid targeting our
>> built-from-source repositories for such Java software and pack them with
>> their binary dependencies (or as binary directly)...
> 
> If I remember correctly, e. g. our glassfish package is a giant bundle
> of jars, i. e. includes everything you need to build and run the
> package. At this level, packaging seems to be feasible, and working
> (however ugly it might be wrt. "proper" packaging).

Yes, that's how glassfishv2 was packaged (in multiverse). For
glassfishv3, an attempt to make a "proper" packaging is under way but
hits the two issues I described in this post.

>> Please let me know what you think of all this problem. I'm pretty sure
>> the Java world will one day gets its senses back and have a more
>> distribution-friendly dependency system
> 
> Stable APIs have worked very well in the C, perl, and Python world. I
> can't imagine why that should intrinsically not be possible in the
> Java world. 

There is no reason why it would be impossible in the Java world. It's
just that the Java platform doesn't help (yet) and the Java developers
have decided not to care about "common library dependencies" but to just
consider them as "pre-written bricks included in my code".

> But for projects where the changes are really as dramatic
> as you point out, I agree that we currently just have to do the "giant
> jar bundle" workaround, if we need to package it in the first place.
> However, we do have a fair bunch of Java libraries in the
> repositories, and even in main, and their API doesn't seem to change
> that often (if at all). So is this really just a problem of the two
> big projects you mentioned (glassfish/geronimo) or a wide-spread
> phenomenon, and the packages we have in main are just the "good"
> exceptions?

I think the Geronimo case is extreme, given the number of dependencies
and the brittle nature of JavaEE TCK compliance. But for all projects
built with Maven, you will need the precise artifact versions to build,
or force it into using a slightly different version, without any easy
way of telling if they are at least API-compatible. This may just fail
to compile, or fail in strange ways at runtime... This is definitely
something you can work on and test for small Maven-based projects, but
for medium-sized ones it will be much more difficult, and you can't
count on upstream support (they decided to use Maven to avoid caring
about those issues).

-- 
Thierry Carrez
Ubuntu server team