Packaging large Java software stacks ?

[Matthias Klose]

I don't think this is related to Java software in general, but is related to the
 practice of development and deployment. It is not unusual that the vendor of an
application server specifies requirements to the operating system down to single
patches, and a vendor using an application server certifying only one specific
version of this AS, plus specific versions of third party software. Plus there
is the hope that you are able to test one build and assume same test results for
all deployments. For this kind of software you usually see not more than one
upload per release cycle in Ubuntu. We certainly can package such software, but
this kind of software is better targeted to the partner archive. We will see
more requests to allow such "pre-tested" software into the archive, especially
if the upstreams are involved in the packaging of the software as well.

Looking at OOo: The source distribution made by upstream includes all system
libraries required by the build. With the time the community around OOo did add
configuration options to build with libraries found on the system instead while
upstream does build with the included libraries to provide their own
distribution independent build. At least here we do have the option to integrate
the build into our environment.

I don't know who did develop maven, but a comparable system is developed in the
python world: setuptools allows you to download things at build time, and set
requirements on version numbers for imports. Now, the download thing can be
turned off, and we do recommend not to use the version stuff for packaging
(using --single-version-externally packaged). Interestingly this system was
sponsored and developed by a company/companies shipping large python
applications in binary form. Again, if we do want to use the libraries found on
the system, we can do so.

In main we try not to duplicate stuff (one thing doing something), and we do
spend a lot of time to demote old libraries and older versions to the universe
section, and removing old stuff from universe. The same rules should apply Java
based software.

Thierry Carrez schrieb:
> Fellow developers,
> 
> It is difficult to integrate the large recent Java software stacks
> (Glassfish, Geronimo...) in a Linux distribution in general. The key
> reason is that most of those stacks require very precise versions of
> libraries (JARs) to run and to build. They won't work with the latest
> version of libraries as those might change APIs and/or key behavior.
> Java developers are used to pick specific JAR versions and assemble the
> exact needed stack, they don't want to care about sharing their
> dependencies with other packages, or about dependencies being upgraded.
> Tools like Maven help them in this endeavor, and they rely very heavily
> on this external code : dozens of those JARs are usually needed at
> runtime, hundreds of those at build-time.

So maven basically is a tool to handle badly designed software? I do see the use
case, and maybe it is necessary to do this if you only have the compiled classes
for dependant libs available, but what does hinder us to fix the bugs in the
package and patch the version requirement as well so that maven is happy?

> That makes most attempts to properly package large Java software in
> Ubuntu to fail for two reasons.
> 
> (1) Need precisely-versioned artifacts
> Those stacks need, as build dependencies and as runtime dependencies,
> very precise versions of JARs. They won't build or run with a different
> one. Using a more recent one might break functionality in a creative
> way. A maven-based build will sometimes require 6 different versions of
> the same JAR. In our packages, we usually offer only one version,
> in corner cases one minor version for each major version. In some cases
> the Java software will run/build with ours, in most cases it won't.
> 
> (2) Building entirely from source
> Java "compilation" (in fact bytecoding) requires lots of JARs to be
> present, because they are used to check all external method signatures.
> Geronimo and Glassfish builds will require hundreds of them. Packaging
> all those build dependencies from source is a huge work, and each of
> those build dependencies, in turn, will require more. Combined with the
> first problem, this makes packaging those stacks too much work for so
> little to gain (they are usually easily installable by
> downloading/unpacking the upstream tarball).

Is it possible to disable these checks during the build? Assuming that we
provide these and assure those by (build-)dependencies?

> I am looking at ways to solve these issues, and would very much like to
> hear ideas from our awesome community :)
> 
> Potential solutions to solve (1) include using a parallel delivery
> mechanism for JARs, separated from the usual Ubuntu packages, something
> that could hook into Maven repositories... or get ready to create a
> separate package for each version of each JAR.

I don't see how such mechanism would be maintained, and what would be different
than packaging different versions of a library. This looks like another
ia32-libs ...

> Potential solutions to solve (2) include evolving our build-from-source
> policy to accept that JARs that would only be used as method-checking
> media during Java compilation to be considered part of the source...

For another software I was told that the upstream doesn't care if the
documentation can be built, so the docs are pre-built, and included in the
binary package.  In this case we are not able to patch these docs to fix a bug
(or at least with a higher effort), but maybe the docs are not that important as
the code is. method-checking looks like the same, but how do you ensure (and the
people who review software for main) that these are just used for checking and
not for building the package?

> A solution to workaround both problems would be to avoid targeting our
> built-from-source repositories for such Java software and pack them with
> their binary dependencies (or as binary directly)...

yes, this seems to work well for these packages.

Maybe another way would be to start with a really big source package, including
all sources, together with the committment to integrate this better into the
distribution. Not sure who does want to commit to this approach. Plus it would
not help to increase the stack of available Java packages in the distribution.

  Matthias