NACK: [PULL] [L/unstable] merge configs into annotations
Andrea Righi
andrea.righi at canonical.com
Fri Nov 18 09:44:38 UTC 2022
Sending a v2 soon (just for the records), then I'm probably going to
apply this to lunar:linux-unstable, since v2 will allow to keep using
the old configs+annotations (even with the patch set applied) and when
we decide to migrate to annotations-only it can be done automatically
via `debian/rules migrateconfigs` (no manual steps involved).
-Andrea
On Wed, Nov 16, 2022 at 07:08:24PM +0100, Andrea Righi wrote:
> [Overview]
>
> Each Ubuntu kernel needs to maintain its own .config for each supported
> architecture and each flavour.
>
> Every time a new patch is applied or a kernel is rebased on top of a new
> one, we need to update the .config's accordingly (config options can be
> added, removed and also renamed).
>
> So, we need to make sure that some critical config options are always
> matching the desired value in order to have a functional kernel.
>
> [State of the art]
>
> At the moment configs are maintained as a set of Kconfig chunks (inside
> debian.<kernel>/config/): a global one, plus per-arch / per-flavour
> chunks.
>
> In addition to that, we need to maintain also a file called
> 'annotations'; the purpose of this file is to make sure that some
> critical config options are not silently removed or changed when the
> real .config is re-generated (for example after a rebase or after
> applying a new set of patches).
>
> The main problem with this approach is that, often, we have duplicate
> information that is stored both in the Kconfig chunks *and* in the
> annotations files and, at the same time, the whole .config's information
> is distributed between Kconfig chunks and annotations, making it hard to
> maintain, review and manage in general.
>
> [Proposed solution]
>
> The proposed solution is to store all the config information into the
> "annotations" format and get rid of the config chunks (basically the
> real .config's can be produced "compiling" annotations).
>
> [Implementation]
>
> To help the management of the annotations an helper script is provided
> (debian/scripts/misc/annotations):
> ```
> usage: annotations [-h] [--version] [--file FILE] [--arch ARCH] [--flavour FLAVOUR] [--config CONFIG] (--query | --export | --import FILE | --check FILE)
>
> Manage Ubuntu kernel .config and annotations
>
> options:
> -h, --help show this help message and exit
> --version, -v show program's version number and exit
> --file FILE, -f FILE Pass annotations or .config file to be parsed
> --arch ARCH, -a ARCH Select architecture
> --flavour FLAVOUR, -l FLAVOUR
> Select flavour (default is "generic")
> --config CONFIG, -c CONFIG
> Select a specific config option
>
> Action:
> --query, -q Query annotations
> --export, -e Convert annotations to .config format
> --import FILE, -i FILE
> Import a .config into annotations
> --check FILE, -k FILE
> Validate kernel .config with annotations
> ```
>
> This script allows to query config settings (per arch/flavour/config),
> export them into the Kconfig format (generating the real .config files)
> and check if the final .config matches the rules defined in the
> annotations.
>
> Examples (annotations is defined as an alias to
> debian/scripts/annotations):
>
> - Show settings for `CONFIG_DEBUG_INFO_BTF` for master kernel across
> all the supported architectures and flavours:
>
> $ annotations --query --config CONFIG_DEBUG_INFO_BTF
> {
> "policy": {
> "amd64": "y",
> "arm64": "y",
> "armhf": "n",
> "ppc64el": "y",
> "riscv64": "y",
> "s390x": "y"
> },
> "note": "'Needs newer pahole for armhf'"
> }
>
> - Dump kernel .config for arm64 and flavour generic-64k:
>
> $ annotations --arch arm64 --flavour generic-64k --export
> CONFIG_DEBUG_FS=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_COMPAT=y
> ...
>
> - Update annotations file with a new kernel .config for amd64 flavour
> generic:
>
> $ annotations --arch amd64 --flavour generic --import build/.config
>
> Moreover, two additional kernelconfig commands are provided
> (debian/rules targets):
> - listnewconfigs: allow to generate a list of new config options (e.g.,
> after a rebase) and store them in CONFIGS/new-<arch>-<flavour> for
> review
> - importconfigs: after new .config's are generated and reviewed (in
> CONFIGS/<arch>-<flavour>) we can use this command to automatically
> import all of them into the local annotations
>
> [Pros and Cons]
>
> Pros:
> - avoid duplicate information in .config's and annotations
> - allow to easily define groups of config settings (for a specific
> environment or feature, such as annotations.clouds, annotations.ubuntu,
> annotations.snapd, etc.)
> - config options are more accessible, easy to change and review
> - we can easily document how config options are managed (and external
> contributors won't be discouraged anymore when they need to to change a
> config option)
>
> Cons:
> - potential regressions: the new tool/scripts can have potential bugs,
> so we could experience regressions due to some missed config changes
> - kernel team need to understand the new process (even if everything
> is transparent, kernel cranking process is the same, there might be
> corner cases that need to be addressed and resolved manually)
>
> [TODO]
>
> - Automatically migrated old config chunks into annotations (this is
> easy, but we need to document the procedure or provide a script to do
> it automatically)
>
> - Migrate flavour and arch definitions into annotations (rather than
> having this information defined in multiple places inside
> debian/scripts); right now this information is "partially" migrated,
> meaning that we need to define arches and flavours in the headers
> section of annotations (so that the annotations tool can figure out
> the list of supported arches and flavours), but arches and flavours
> are still defined elsewhere, ideally we would like to have arches and
> flavours defined only in one place: annotations.
>
> ---
>
> The following changes since commit 1938fabc4cf0d4ed450f4cb2b02cfd48b5e4f957:
>
> UBUNTU: Ubuntu-unstable-6.1.0-5.5 (2022-11-07 16:44:48 +0100)
>
> are available in the Git repository at:
>
> git://git.launchpad.net/~arighi/+git/lunar annotations
>
> for you to fetch changes up to cc5ca9b85d730f01bdbde450cd0141405a2ee082:
>
> UBUNTU: [Config] drop configs and rely on annotations (2022-11-16 17:28:47 +0100)
>
> ----------------------------------------------------------------
> Andrea Righi (13):
> UBUNTU: [Packaging] config-check: ignore values that are not defined in annotations
> UBUNTU: [Packaging] config-check: do not strictly enforce CONFIG_CC_VERSION_TEXT
> UBUNTU: [Packaging] introduce annotations script
> UBUNTU: [Packaging] automatically generate configs from annotations
> UBUNTU: [Packaging] drop deprecated script splitconfig.pl
> UBUNTU: [Packaging] drop deprecated script tristate.sh
> UBUNTU: [Packaging] drop config-check and use annotations
> UBUNTU: [Packaging] simplify kernelconfig
> UBUNTU: [Packaging] kernelconfig: always keep configs
> UBUNTU: [Packaging] provide listnewconfigs
> UBUNTU: [Packaging] kernelconfig: introduce importconfigs
> UBUNTU: [Config] import all configs into annotations
> UBUNTU: [Config] drop configs and rely on annotations
>
> debian.master/config/amd64/config.common.amd64 | 701 -
> debian.master/config/amd64/config.flavour.generic | 3 -
> debian.master/config/annotations | 6808 +++++-----
> debian.master/config/arm64/config.common.arm64 | 731 -
> debian.master/config/arm64/config.flavour.generic | 14 -
> .../config/arm64/config.flavour.generic-64k | 14 -
> debian.master/config/armhf/config.common.armhf | 712 -
> debian.master/config/armhf/config.flavour.generic | 15 -
> .../config/armhf/config.flavour.generic-lpae | 15 -
> debian.master/config/config.common.ubuntu | 13418 -------------------
> debian.master/config/ppc64el/config.common.ppc64el | 702 -
> .../config/ppc64el/config.flavour.generic | 3 -
> debian.master/config/riscv64/config.common.riscv64 | 694 -
> .../config/riscv64/config.flavour.generic | 3 -
> debian.master/config/s390x/config.common.s390x | 627 -
> debian.master/config/s390x/config.flavour.generic | 3 -
> debian/rules.d/1-maintainer.mk | 13 +-
> debian/rules.d/2-binary-arch.mk | 6 +-
> debian/rules.d/4-checks.mk | 5 +-
> debian/scripts/config-check | 163 -
> debian/scripts/misc/annotations | 140 +
> debian/scripts/misc/kconfig/__init__.py | 0
> debian/scripts/misc/kconfig/annotations.py | 242 +
> debian/scripts/misc/kernelconfig | 167 +-
> debian/scripts/misc/splitconfig.pl | 107 -
> debian/scripts/misc/tristate.sh | 26 -
> 26 files changed, 4092 insertions(+), 21240 deletions(-)
> delete mode 100644 debian.master/config/amd64/config.common.amd64
> delete mode 100644 debian.master/config/amd64/config.flavour.generic
> delete mode 100644 debian.master/config/arm64/config.common.arm64
> delete mode 100644 debian.master/config/arm64/config.flavour.generic
> delete mode 100644 debian.master/config/arm64/config.flavour.generic-64k
> delete mode 100644 debian.master/config/armhf/config.common.armhf
> delete mode 100644 debian.master/config/armhf/config.flavour.generic
> delete mode 100644 debian.master/config/armhf/config.flavour.generic-lpae
> delete mode 100644 debian.master/config/config.common.ubuntu
> delete mode 100644 debian.master/config/ppc64el/config.common.ppc64el
> delete mode 100644 debian.master/config/ppc64el/config.flavour.generic
> delete mode 100644 debian.master/config/riscv64/config.common.riscv64
> delete mode 100644 debian.master/config/riscv64/config.flavour.generic
> delete mode 100644 debian.master/config/s390x/config.common.s390x
> delete mode 100644 debian.master/config/s390x/config.flavour.generic
> delete mode 100755 debian/scripts/config-check
> create mode 100755 debian/scripts/misc/annotations
> create mode 100644 debian/scripts/misc/kconfig/__init__.py
> create mode 100644 debian/scripts/misc/kconfig/annotations.py
> delete mode 100755 debian/scripts/misc/splitconfig.pl
> delete mode 100755 debian/scripts/misc/tristate.sh
More information about the kernel-team
mailing list