continuous packaging? anyone?

[Philip Muskovac]

On Wednesday 06 August 2014 19:25:42 Harald Sitter wrote:
> aloha packagers
> 
> today the idea of continuous packaging popped up and I wanted to get
> some general thoughts on the matter.
> 
> # what is this?
> CP is essentially what project-neon does, build all packages off of
> upstream git. Conceptually this technically is supposed lto be like
> proper CI (i.e. build as often as possible with as small changes as
> possible and run tests). Since this is partially not very practical
> from a packaging perspective let's for now think of it as regular
> daily builds.
> 
> # what's wrong with neon? :O
> Neon is built in a very specific way. Namely:
> - completely different prefix
> - excessive envrionment manipulation to separate data lookup (as much
> as possible) as well as user specific data and settings storage
> - single binary units (i.e. each upstream git repo creates exactly one
> debian package)
> It is made this way because it creates next to no overhead in
> packaging, this is curcial to what neon is meant to achive: Provide
> the most bleeding edge of KDE software *next to* the ongoing efforts
> of high quality packaging we do manually. In other words neon is made
> in such a way that it does not take away developer time from the real
> meat, at the expense of Debian policy compliance (in some ways
> anyway).
> 
> # so how would CP be different from neon?
> The idea is instead of doing daily dummy packages with only one deb as
> described above we build our actual packages on a daily basis against
> an actual upstream git branch. Continuously adjusting our packaging to
> what is in git and will therefore eventually become a release we will
> pick up.
> 
> # why oh why?
> Right now we get a batch of a upstream tarballs, throw our *old*
> packaging at it, and then hammer at it until it builds. This is not
> only stressful it also scales really bad. The more changes are between
> old-release and new-release the more hammering is needed, the longer
> it takes, the more exhausting it is, the more bugs we introduce. Add
> to that the fact that we sometimes get a very excessive batch of
> partially even unrelated releases at pretty much the same time and
> you've occupied some 2 or 3 packagers for a week or two (think KDE SC,
> Plasma and KF5 in the same week for example).
> 
> # how does CP solve that?
> The idea is, every day you have changes trickle in, ever so slowly,
> built against our packaging HEAD. Instead of getting a crazy week at
> the beginning of each month we spend maybe 5 minutes a day adjusting
> the packages that need adjustment and once the actual release hits we
> take our packaging throw it at the release tarballs and everything
> builds without problems (in theory).
> 
> # why, there must be a downside to all this...
> Numerous tiny problems popping up all over the place that will need to
> have solutions invented. So it would be good if we could think of as
> many as possible before moving forward. I'll make a start:
> - unless we replicate tooling for l10n retrieval, there won't be
> translations and more importantly .install files that explicitly list
> /usr/share/locale/ will fail
> - doing CP makes packaging branches more confusing unless we actually
> land into archive (which would seem slightly terrible) as we'd have
> two HEAD branches... one for the actual packaging in the archive and
> one for the CP that is constantly moving, since keeping a handle on
> many branches in bzr is a jolly nightmare I am not too fond of this
> 
> # and what about the tech?
> Thanks to blue systems and yours truly most of the tech does already
> exist and drives neon5 ;)
> 
> So, what do you guys and gals reckon?
> 
> HS

I'll add to the downside list:
- some of the packaging is really version centered (e.g. most games have a >= 
source:Version dep on kdegames-data which is from libkdegames) which would 
need fixing or packages will just be uninstallable - unless you build the whole 
full package set each day and only keep the date in the version. You might 
want to talk to debian about that or we'll just increase our diff.
- neon intentionally doesn't have mixed arch:all/any packages in those that 
are really built daily as that usually doesn't work out too well with daily 
builds. Esp. with the point above you would get a lot of build failures thanks 
to version mismatches between the architectures.

I personally wouldn't mind CP, but I already gave up setting up CP for 
*stable* releases, forget the dev release, as the benefit was in no relation to 
the amount of maintenance it would've required.

Philip