Packaging large Java software stacks ?

[Emmet Hikory]

Thierry Carrez wrote:
> Emmet Hikory wrote:
>>     In the case of libraries in other languages (most commonly C), we
>> regularly port applications to work with the preferred version of the
>> libraries we ship.  When preparing new versions of C libraries, the API
>> and ABI are checked, with the binary package name changed where they
>> differ, to better ensure compatibility.  Perhaps we could use the Java
>> Introspection methods to generate some API report, and version Java
>> libraries based on changing APIs?  
> 
> Yes, that would be an option. I know Sun wants to go in a more
> stable-API-oriented direction in the future, having acknowledged that
> the current JAR version numbers don't really mean anything useful.

    Well, there's two factors here: a version number used to track
upstream workflow, and an API version used for dependency consistency.
In C, these are typically differentiated, so that the version of a
library may have little direct relation to the soname (e.g. 3.7.1 could
provide libfoo.s0.1.0.8).

>> In the case where an incompatibility
>> is not an API change, what sort of differences are encountered?  Might
>> these be considered bugs?  Is there a case where two applications depend
>> on the same API, but would break if used with different versions
>> providing that API?
> 
> When some project requires a precise version of a JAR, you don't really
> know if that's :
> - because it relies on a specific API
> - because there is a bugfix they rely on
> - because newer features introduce a regression
> - because Maven uses =JAR_VERSION types of dependencies by default
> 
> There might exist some Java library projects with strict versioning
> rules, like x.y.z where x.y defines API and .z defines bugfixes inside a
> stable API. But the general case is that they are allowed to (and will)
> break API and behavior for every version (no matter how minor), since
> (1) Maven helps developers in depending on a specific JAR version and
> (2) runtime JARs are shipped within the binary release tarball.

    The issues I'm interested in are the second and third on your list,
as the first is soluable with an API checker, and the last is an
artifact of current social practice.  The second is something that can
simply be documentation, and we can easily represent in the archive
using the existing dependencies model.  The third is clearly a bug, and
ought be fixed, rather than ignored (which is significantly easier if
the package in question is available in a source form from which we can
create distributable binary artifacts in the regular manner).

> There is the middle alternative, where we would still build from source
> everything we *distribute*. There are a lot more Java build dependencies
> than runtime dependencies: most of the Java build dependencies are just
> used to check external method signatures during the bytecode
> compilation, much like an API description. So we could still produce the
> needed runtime JARs from their source by shipping those JAR build
> dependencies as part of the source package. That way we solve the source
> code providing hole, and we can have a clean defect management system,
> for a much smaller packaging cost.

    Well, I see two issues with this approach.  The first is that if the
package is built against some arbitrary API which may or may not match
the API provided by other system libraries, there may be runtime issues
unless the jars are also shipped as part of the binary package (thereby
becoming part of that which is distributed).  The second is that we
still have no way to address any issues discovered: if there is an
inconsistency of some sort, we are again restricted to patching binary
objects in the hopes of addressing it.

    I'm also in support of Martin's suggestion that we ensure that
end-users can use the distribution-unfriendly system if they wish (as we
do for cpan modules, eggs, gems, etc.), but don't use these on the
buildds, and just ship binary blobs until the software in question is
available in such a way that we can effectively maintain it.  While this
may relegate some packages to multiverse while this is addressed, it
does more accurately represent our ability to support the application
directly.

-- 
Emmet HIKORY