continuous packaging? anyone?

[Harald Sitter]

On Thu, Aug 7, 2014 at 9:58 AM, Philip Muskovac <yofel at gmx.net> wrote:
> On Wednesday 06 August 2014 19:25:42 Harald Sitter wrote:
>> aloha packagers
>>
>> today the idea of continuous packaging popped up and I wanted to get
>> some general thoughts on the matter.
>>
>> # what is this?
>> CP is essentially what project-neon does, build all packages off of
>> upstream git. Conceptually this technically is supposed lto be like
>> proper CI (i.e. build as often as possible with as small changes as
>> possible and run tests). Since this is partially not very practical
>> from a packaging perspective let's for now think of it as regular
>> daily builds.
>>
>> # what's wrong with neon? :O
>> Neon is built in a very specific way. Namely:
>> - completely different prefix
>> - excessive envrionment manipulation to separate data lookup (as much
>> as possible) as well as user specific data and settings storage
>> - single binary units (i.e. each upstream git repo creates exactly one
>> debian package)
>> It is made this way because it creates next to no overhead in
>> packaging, this is curcial to what neon is meant to achive: Provide
>> the most bleeding edge of KDE software *next to* the ongoing efforts
>> of high quality packaging we do manually. In other words neon is made
>> in such a way that it does not take away developer time from the real
>> meat, at the expense of Debian policy compliance (in some ways
>> anyway).
>>
>> # so how would CP be different from neon?
>> The idea is instead of doing daily dummy packages with only one deb as
>> described above we build our actual packages on a daily basis against
>> an actual upstream git branch. Continuously adjusting our packaging to
>> what is in git and will therefore eventually become a release we will
>> pick up.
>>
>> # why oh why?
>> Right now we get a batch of a upstream tarballs, throw our *old*
>> packaging at it, and then hammer at it until it builds. This is not
>> only stressful it also scales really bad. The more changes are between
>> old-release and new-release the more hammering is needed, the longer
>> it takes, the more exhausting it is, the more bugs we introduce. Add
>> to that the fact that we sometimes get a very excessive batch of
>> partially even unrelated releases at pretty much the same time and
>> you've occupied some 2 or 3 packagers for a week or two (think KDE SC,
>> Plasma and KF5 in the same week for example).
>>
>> # how does CP solve that?
>> The idea is, every day you have changes trickle in, ever so slowly,
>> built against our packaging HEAD. Instead of getting a crazy week at
>> the beginning of each month we spend maybe 5 minutes a day adjusting
>> the packages that need adjustment and once the actual release hits we
>> take our packaging throw it at the release tarballs and everything
>> builds without problems (in theory).
>>
>> # why, there must be a downside to all this...
>> Numerous tiny problems popping up all over the place that will need to
>> have solutions invented. So it would be good if we could think of as
>> many as possible before moving forward. I'll make a start:
>> - unless we replicate tooling for l10n retrieval, there won't be
>> translations and more importantly .install files that explicitly list
>> /usr/share/locale/ will fail
>> - doing CP makes packaging branches more confusing unless we actually
>> land into archive (which would seem slightly terrible) as we'd have
>> two HEAD branches... one for the actual packaging in the archive and
>> one for the CP that is constantly moving, since keeping a handle on
>> many branches in bzr is a jolly nightmare I am not too fond of this
>>
>> # and what about the tech?
>> Thanks to blue systems and yours truly most of the tech does already
>> exist and drives neon5 ;)
>>
>> So, what do you guys and gals reckon?
>>
>> HS
>
> I'll add to the downside list:
> - some of the packaging is really version centered (e.g. most games have a >=
> source:Version dep on kdegames-data which is from libkdegames) which would
> need fixing or packages will just be uninstallable - unless you build the whole
> full package set each day and only keep the date in the version. You might
> want to talk to debian about that or we'll just increase our diff.

Sounds like we are abusing version alignment TBH.

> - neon intentionally doesn't have mixed arch:all/any packages in those that
> are really built daily as that usually doesn't work out too well with daily
> builds. Esp. with the point above you would get a lot of build failures thanks
> to version mismatches between the architectures.

Very true indeed.
That's very solvable through automation though, we even discussed the
algorithm to use recently on IRC :)

HS