Changes in the container stack

[Lucas Kanashiro]

Hi Seth,

Em 01/08/2023 23:02, Seth Arnold escreveu:
> On Tue, Aug 01, 2023 at 06:45:59PM -0300, Lucas Kanashiro wrote:
>> With that in mind, we, the Server team, identified the need to decouple the
>> application (what users really use) and the library (what is used by rdeps)
>> in a way that, on the one hand, we can keep following upstream projects
>> without worrying about breaking changes and on the other hand, we keep the
>> library (-dev package) stable to avoid breakages of packages sync'ed from
>> Debian or already in stable releases during SRUs.
> Hello Lucas, I'm very concerned about this plan, I hope you can assuage my
> fears.

Thanks for raising your concerns. I hope I can help you clear them out :)

> This sounds like we'll introduce a new problem, where third-party
> applications built against the libraries will become incompatible with
> the main application.

That _might_ happen (depending on how those apps are interacting) if 
they are using the -dev package in the archive to build their app. 
However, from what I have seen in the Go ecosystem, people will not be 
doing that, they will be using the vendor code from upstream, pinning 
the version they want.

For the packages in the archive depending on -dev packages, I have seen 
API changes in generic functions used by other packages (software that 
does not interact directly with the containers stack but re-use some 
exported functions). So making those rdeps use an old version of the 
library does not seem an issue most of the time, considering there is no 
interaction during runtime among those components.

> If the ecosystems are changing external library ABIs too often, I can
> only shudder in fear at how quickly internal library ABIs are changing.
>
> What gives us the confidence that the internal ABIs are actually more
> stable and reliable than the external ABIs?

All the breakages I have seen so far are related to API changes (TBH not 
too complex changes), the main problem is that those changes require 
backporting upstream patches which usually slow down our velocity of 
updates (in the end, we cannot follow upstream releases because SRU 
process takes too long). I haven't seen weird ABI changes.

For the container stack itself, I do some manual testing with all the 
packages and there are also DEP-8 tests which guarantee that those 3 
pieces of software work well together before I perform any upload to the 
archive. Thinking about third-party apps interacting with the stack, 
usually, those upstream projects follow the latest (or considerably new) 
version of docker/containerd/runc, so to avoid ABI breakages would be 
better to follow upstream closely. Keeping an old version of those 
software is more likely to cause ABI issues.

As I mentioned, this ecosystem moves fast, if we do not try to follow 
upstream we will not be able to provide value to those users. This is a 
problem I have seen, some users are preferring to use, for instance, the 
Docker debian package provided by the Docker upstream maintainers 
instead of ours, because we are sometimes way behind them. I have some 
plans in mind to change this scenario, the decoupling I presented here 
is just the first step to improve users' experience.

I hope my answer makes sense to you Seth. Let me know if it does not.

Cheers!

-- 
Lucas Kanashiro