ACK/Cmnt: [PULL v4] [L/unstable] merge configs into annotations
Tim Gardner
tim.gardner at canonical.com
Tue Nov 29 19:00:58 UTC 2022
On 11/18/22 6:23 AM, Andrea Righi wrote:
> ChangeLog (v3 -> v4):
> - allow to use config+annotations or annotations-only when generating
> the initial configs and config checks (during the package build)
> - support configs+annotations and annotations-only for snapcraft
> builds
>
> ChangeLog (v2 -> v3):
> - include only the relevant patches in the pull request
>
> ChangeLog (v1 -> v2):
> - added --update option to annotations script to import a partial
> .config into annotations (not a full resync like --import)
> - added debian/rules migrateconfigs to automatically merge old
> configs+annotations into annotations
> - support both configs+annotations and annotations-only (but once
> everything is migrated to annotations-only via migrateconfigs we
> can't go back to configs+annotations)
>
> [Overview]
>
> Each Ubuntu kernel needs to maintain its own .config for each supported
> architecture and each flavour.
>
> Every time a new patch is applied or a kernel is rebased on top of a new
> one, we need to update the .config's accordingly (config options can be
> added, removed and also renamed).
>
> So, we need to make sure that some critical config options are always
> matching the desired value in order to have a functional kernel.
>
> [State of the art]
>
> At the moment configs are maintained as a set of Kconfig chunks (inside
> debian.<kernel>/config/): a global one, plus per-arch / per-flavour
> chunks.
>
> In addition to that, we need to maintain also a file called
> 'annotations'; the purpose of this file is to make sure that some
> critical config options are not silently removed or changed when the
> real .config is re-generated (for example after a rebase or after
> applying a new set of patches).
>
> The main problem with this approach is that, often, we have duplicate
> information that is stored both in the Kconfig chunks *and* in the
> annotations files and, at the same time, the whole .config's information
> is distributed between Kconfig chunks and annotations, making it hard to
> maintain, review and manage in general.
>
> [Proposed solution]
>
> The proposed solution is to store all the config information into the
> "annotations" format and get rid of the config chunks (basically the
> real .config's can be produced "compiling" annotations).
>
> [Implementation]
>
> To help the management of the annotations an helper script is provided
> (debian/scripts/misc/annotations):
> ```
> usage: annotations [-h] [--version] [--file FILE] [--arch ARCH] [--flavour FLAVOUR]
> +[--config CONFIG]
> (--query | --export | --import FILE | --update FILE | --check
> +FILE)
>
> Manage Ubuntu kernel .config and annotations
>
> options:
> -h, --help show this help message and exit
> --version, -v show program's version number and exit
> --file FILE, -f FILE Pass annotations or .config file to be parsed
> --arch ARCH, -a ARCH Select architecture
> --flavour FLAVOUR, -l FLAVOUR
> Select flavour (default is "generic")
> --config CONFIG, -c CONFIG
> Select a specific config option
>
> Action:
> --query, -q Query annotations
> --export, -e Convert annotations to .config format
> --import FILE, -i FILE
> Import a full .config for a specific arch and flavour into
> +annotations
> --update FILE, -u FILE
> Import a partial .config into annotations (only resync
> +configs specified in FILE)
> --check FILE, -k FILE
> Validate kernel .config with annotations
> ```
>
> This script allows to query config settings (per arch/flavour/config),
> export them into the Kconfig format (generating the real .config files)
> and check if the final .config matches the rules defined in the
> annotations.
> Examples (annotations is defined as an alias to
> debian/scripts/annotations):
>
> - Show settings for `CONFIG_DEBUG_INFO_BTF` for master kernel across
> all the supported architectures and flavours:
>
> $ annotations --query --config CONFIG_DEBUG_INFO_BTF
> {
> "policy": {
> "amd64": "y",
> "arm64": "y",
> "armhf": "n",
> "ppc64el": "y",
> "riscv64": "y",
> "s390x": "y"
> },
> "note": "'Needs newer pahole for armhf'"
> }
>
> - Dump kernel .config for arm64 and flavour generic-64k:
>
> $ annotations --arch arm64 --flavour generic-64k --export
> CONFIG_DEBUG_FS=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_COMPAT=y
> ...
>
> - Update annotations file with a new kernel .config for amd64 flavour
> generic:
>
> $ annotations --arch amd64 --flavour generic --import build/.config
>
> Moreover, two additional kernelconfig commands are provided
> (debian/rules targets):
> - listnewconfigs: allow to generate a list of new config options (e.g.,
> after a rebase) and store them in CONFIGS/new-<arch>-<flavour> for
> review
> - importconfigs: after new .config's are generated and reviewed (in
> CONFIGS/<arch>-<flavour>) we can use this command to automatically
> import all of them into the local annotations
>
> To facilitate the migration from the old configs+annotations method the
> new new annotations-only method an additional debian/rules helper is
> provided: migrateconfigs.
>
> Running 'debian/rules migrateconfigs' will automatically merge all the
> previous configs into annotations and drop the unused scripts (local
> changes still need to be committed).
>
> [Pros and Cons]
>
> Pros:
> - avoid duplicate information in .config's and annotations
> - allow to easily define groups of config settings (for a specific
> environment or feature, such as annotations.clouds, annotations.ubuntu,
> annotations.snapd, etc.)
> - config options are more accessible, easy to change and review
> - we can easily document how config options are managed (and external
> contributors won't be discouraged anymore when they need to to change a
> config option)
>
> Cons:
> - potential regressions: the new tool/scripts can have potential bugs,
> so we could experience regressions due to some missed config changes
> - kernel team need to understand the new process (even if everything
> is transparent, kernel cranking process is the same, there might be
> corner cases that need to be addressed and resolved manually)
>
> [TODO]
>
> - Migrate all flavour and arch definitions into annotations (rather
> than having this information defined in multiple places inside
> debian/scripts); right now this information is "partially" migrated,
> meaning that we need to define arches and flavours in the headers
> section of annotations (so that the annotations tool can figure out
> the list of supported arches and flavours), but arches and flavours
> are still defined elsewhere, ideally we would like to have arches and
> flavours defined only in one place: annotations.
>
> ---
> The following changes since commit 192488d98049f301363e9a1b6ede4da0eb7cc82c:
>
> net: wwan: t7xx: Add AP CLDMA (2022-11-17 17:33:40 +0100)
>
> are available in the Git repository at:
>
> git://git.launchpad.net/~arighi/+git/lunar annotations
>
> for you to fetch changes up to 75879256cad3177ce388dc6707e00c4a8c438ebf:
>
> UBUNTU: [Config] merge configs into annotations (2022-11-18 12:44:24 +0100)
>
> ----------------------------------------------------------------
> Andrea Righi (17):
> UBUNTU: SAUCE: add python cached bytecode to .gitignore
> UBUNTU: [Packaging] config-check: ignore values that are not defined in annotations
> UBUNTU: [Packaging] config-check: do not strictly enforce CONFIG_CC_VERSION_TEXT
> UBUNTU: [Packaging] introduce annotations script
> UBUNTU: [Packaging] automatically generate configs from annotations
> UBUNTU: [Packaging] drop deprecated script tristate.sh
> UBUNTU: [Packaging] use annotations to generate initial configs and perform config-check
> UBUNTU: [Packaging] simplify kernelconfig
> UBUNTU: [Packaging] kernelconfig: always keep configs
> UBUNTU: [Packaging] provide listnewconfigs
> UBUNTU: [Packaging] kernelconfig: introduce importconfigs
> UBUNTU: [Packaging] snap: support new annotations with snapcraft
> UBUNTU: [Packaging] final-checks: support new annotations
> UBUNTU: [Packaging] introduce migrate-annotations
> UBUNTU: [Packaging] re-introduce previous kernelconfig as old-kernelconfig
> UBUNTU: [Packaging] provide migrateconfigs debian/rules target
> UBUNTU: [Config] merge configs into annotations
>
> .gitignore | 3 +
> debian.master/config/amd64/config.common.amd64 | 701 -
> debian.master/config/amd64/config.flavour.generic | 3 -
> debian.master/config/annotations | 28312 ++++++++++---------
> debian.master/config/arm64/config.common.arm64 | 732 -
> debian.master/config/arm64/config.flavour.generic | 14 -
> .../config/arm64/config.flavour.generic-64k | 14 -
> debian.master/config/armhf/config.common.armhf | 713 -
> debian.master/config/armhf/config.flavour.generic | 15 -
> .../config/armhf/config.flavour.generic-lpae | 15 -
> debian.master/config/config.common.ubuntu | 13419 ---------
> debian.master/config/ppc64el/config.common.ppc64el | 703 -
> .../config/ppc64el/config.flavour.generic | 3 -
> debian.master/config/riscv64/config.common.riscv64 | 695 -
> .../config/riscv64/config.flavour.generic | 3 -
> debian.master/config/s390x/config.common.s390x | 627 -
> debian.master/config/s390x/config.flavour.generic | 3 -
> debian/rules.d/1-maintainer.mk | 33 +-
> debian/rules.d/2-binary-arch.mk | 9 +-
> debian/rules.d/4-checks.mk | 11 +-
> debian/scripts/config-check | 163 -
> debian/scripts/misc/annotations | 163 +
> debian/scripts/misc/final-checks | 28 +-
> debian/scripts/misc/kconfig/__init__.py | 0
> debian/scripts/misc/kconfig/annotations.py | 240 +
> debian/scripts/misc/kernelconfig | 167 +-
> debian/scripts/misc/migrate-annotations | 33 +
> debian/scripts/misc/splitconfig.pl | 107 -
> debian/scripts/misc/tristate.sh | 26 -
> debian/snapcraft.mk | 6 +-
> 30 files changed, 14958 insertions(+), 32003 deletions(-)
> delete mode 100644 debian.master/config/amd64/config.common.amd64
> delete mode 100644 debian.master/config/amd64/config.flavour.generic
> delete mode 100644 debian.master/config/arm64/config.common.arm64
> delete mode 100644 debian.master/config/arm64/config.flavour.generic
> delete mode 100644 debian.master/config/arm64/config.flavour.generic-64k
> delete mode 100644 debian.master/config/armhf/config.common.armhf
> delete mode 100644 debian.master/config/armhf/config.flavour.generic
> delete mode 100644 debian.master/config/armhf/config.flavour.generic-lpae
> delete mode 100644 debian.master/config/config.common.ubuntu
> delete mode 100644 debian.master/config/ppc64el/config.common.ppc64el
> delete mode 100644 debian.master/config/ppc64el/config.flavour.generic
> delete mode 100644 debian.master/config/riscv64/config.common.riscv64
> delete mode 100644 debian.master/config/riscv64/config.flavour.generic
> delete mode 100644 debian.master/config/s390x/config.common.s390x
> delete mode 100644 debian.master/config/s390x/config.flavour.generic
> delete mode 100755 debian/scripts/config-check
> create mode 100755 debian/scripts/misc/annotations
> create mode 100644 debian/scripts/misc/kconfig/__init__.py
> create mode 100644 debian/scripts/misc/kconfig/annotations.py
> create mode 100755 debian/scripts/misc/migrate-annotations
> delete mode 100755 debian/scripts/misc/splitconfig.pl
> delete mode 100755 debian/scripts/misc/tristate.sh
>
Acked-by: Tim Gardner <tim.gardner at canonical.com>
This looks vastly superior to the way we're currently managing configs.
I'm looking forward to having some distro wide config blocks that apply
to all kernels, e.g., Apparmor settings and the like.
--
-----------
Tim Gardner
Canonical, Inc
More information about the kernel-team
mailing list