NACK: [PULL v3] [L/unstable] merge configs into annotations
Andrea Righi
andrea.righi at canonical.com
Fri Nov 18 12:18:36 UTC 2022
...sending a v4 soon, because I forgot that we also have
debian/snapcraft.mk (that is expecting to find the config chunks).
-Andrea
On Fri, Nov 18, 2022 at 11:19:32AM +0100, Andrea Righi wrote:
> ChangeLog (v2 -> v3):
> - include only the relevant patches in the pull request
>
> ChangeLog (v1 -> v2):
> - added --update option to annotations script to import a partial
> .config into annotations (not a full resync like --import)
> - added debian/rules migrateconfigs to automatically merge old
> configs+annotations into annotations
> - support both configs+annotations and annotations-only (but once
> everything is migrated to annotations-only via migrateconfigs we
> can't go back to configs+annotations)
>
> [Overview]
>
> Each Ubuntu kernel needs to maintain its own .config for each supported
> architecture and each flavour.
>
> Every time a new patch is applied or a kernel is rebased on top of a new
> one, we need to update the .config's accordingly (config options can be
> added, removed and also renamed).
>
> So, we need to make sure that some critical config options are always
> matching the desired value in order to have a functional kernel.
>
> [State of the art]
>
> At the moment configs are maintained as a set of Kconfig chunks (inside
> debian.<kernel>/config/): a global one, plus per-arch / per-flavour
> chunks.
>
> In addition to that, we need to maintain also a file called
> 'annotations'; the purpose of this file is to make sure that some
> critical config options are not silently removed or changed when the
> real .config is re-generated (for example after a rebase or after
> applying a new set of patches).
>
> The main problem with this approach is that, often, we have duplicate
> information that is stored both in the Kconfig chunks *and* in the
> annotations files and, at the same time, the whole .config's information
> is distributed between Kconfig chunks and annotations, making it hard to
> maintain, review and manage in general.
>
> [Proposed solution]
>
> The proposed solution is to store all the config information into the
> "annotations" format and get rid of the config chunks (basically the
> real .config's can be produced "compiling" annotations).
>
> [Implementation]
>
> To help the management of the annotations an helper script is provided
> (debian/scripts/misc/annotations):
> ```
> usage: annotations [-h] [--version] [--file FILE] [--arch ARCH] [--flavour FLAVOUR] [--config CONFIG]
> (--query | --export | --import FILE | --update FILE | --check FILE)
>
> Manage Ubuntu kernel .config and annotations
>
> options:
> -h, --help show this help message and exit
> --version, -v show program's version number and exit
> --file FILE, -f FILE Pass annotations or .config file to be parsed
> --arch ARCH, -a ARCH Select architecture
> --flavour FLAVOUR, -l FLAVOUR
> Select flavour (default is "generic")
> --config CONFIG, -c CONFIG
> Select a specific config option
>
> Action:
> --query, -q Query annotations
> --export, -e Convert annotations to .config format
> --import FILE, -i FILE
> Import a full .config for a specific arch and flavour into annotations
> --update FILE, -u FILE
> Import a partial .config into annotations (only resync configs specified in FILE)
> --check FILE, -k FILE
> Validate kernel .config with annotations
> ```
>
> This script allows to query config settings (per arch/flavour/config),
> export them into the Kconfig format (generating the real .config files)
> and check if the final .config matches the rules defined in the
> annotations.
>
> Examples (annotations is defined as an alias to
> debian/scripts/annotations):
>
> - Show settings for `CONFIG_DEBUG_INFO_BTF` for master kernel across
> all the supported architectures and flavours:
>
> $ annotations --query --config CONFIG_DEBUG_INFO_BTF
> {
> "policy": {
> "amd64": "y",
> "arm64": "y",
> "armhf": "n",
> "ppc64el": "y",
> "riscv64": "y",
> "s390x": "y"
> },
> "note": "'Needs newer pahole for armhf'"
> }
>
> - Dump kernel .config for arm64 and flavour generic-64k:
>
> $ annotations --arch arm64 --flavour generic-64k --export
> CONFIG_DEBUG_FS=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_COMPAT=y
> ...
>
> - Update annotations file with a new kernel .config for amd64 flavour
> generic:
>
> $ annotations --arch amd64 --flavour generic --import build/.config
>
> Moreover, two additional kernelconfig commands are provided
> (debian/rules targets):
> - listnewconfigs: allow to generate a list of new config options (e.g.,
> after a rebase) and store them in CONFIGS/new-<arch>-<flavour> for
> review
> - importconfigs: after new .config's are generated and reviewed (in
> CONFIGS/<arch>-<flavour>) we can use this command to automatically
> import all of them into the local annotations
>
> [Pros and Cons]
>
> Pros:
> - avoid duplicate information in .config's and annotations
> - allow to easily define groups of config settings (for a specific
> environment or feature, such as annotations.clouds, annotations.ubuntu,
> annotations.snapd, etc.)
> - config options are more accessible, easy to change and review
> - we can easily document how config options are managed (and external
> contributors won't be discouraged anymore when they need to to change a
> config option)
>
> Cons:
> - potential regressions: the new tool/scripts can have potential bugs,
> so we could experience regressions due to some missed config changes
> - kernel team need to understand the new process (even if everything
> is transparent, kernel cranking process is the same, there might be
> corner cases that need to be addressed and resolved manually)
>
> [TODO]
>
> - Migrate flavour and arch definitions into annotations (rather than
> having this information defined in multiple places inside
> debian/scripts); right now this information is "partially" migrated,
> meaning that we need to define arches and flavours in the headers
> section of annotations (so that the annotations tool can figure out
> the list of supported arches and flavours), but arches and flavours
> are still defined elsewhere, ideally we would like to have arches and
> flavours defined only in one place: annotations.
>
> ---
> The following changes since commit 192488d98049f301363e9a1b6ede4da0eb7cc82c:
>
> net: wwan: t7xx: Add AP CLDMA (2022-11-17 17:33:40 +0100)
>
> are available in the Git repository at:
>
> git://git.launchpad.net/~arighi/+git/lunar annotations
>
> for you to fetch changes up to 4cf4489b73d5e311036c99061d9c7f13e77a1933:
>
> UBUNTU: [Config] merge configs into annotations (using migrateconfigs) (2022-11-18 10:54:19 +0100)
>
> ----------------------------------------------------------------
> Andrea Righi (15):
> UBUNTU: SAUCE: add python cached bytecode to .gitignore
> UBUNTU: [Packaging] config-check: ignore values that are not defined in annotations
> UBUNTU: [Packaging] config-check: do not strictly enforce CONFIG_CC_VERSION_TEXT
> UBUNTU: [Packaging] introduce annotations script
> UBUNTU: [Packaging] automatically generate configs from annotations
> UBUNTU: [Packaging] drop deprecated script tristate.sh
> UBUNTU: [Packaging] use annotations to perform config-check
> UBUNTU: [Packaging] simplify kernelconfig
> UBUNTU: [Packaging] kernelconfig: always keep configs
> UBUNTU: [Packaging] provide listnewconfigs
> UBUNTU: [Packaging] kernelconfig: introduce importconfigs
> UBUNTU: [Packaging] introduce migrate-annotations
> UBUNTU: [Packaging] re-introduce previous kernelconfig as old-kernelconfig
> UBUNTU: [Packaging] provide migrateconfigs debian/rules target
> UBUNTU: [Config] merge configs into annotations (using migrateconfigs)
>
> .gitignore | 3 +
> debian.master/config/amd64/config.common.amd64 | 701 -
> debian.master/config/amd64/config.flavour.generic | 3 -
> debian.master/config/annotations | 28312 ++++++++++---------
> debian.master/config/arm64/config.common.arm64 | 732 -
> debian.master/config/arm64/config.flavour.generic | 14 -
> .../config/arm64/config.flavour.generic-64k | 14 -
> debian.master/config/armhf/config.common.armhf | 713 -
> debian.master/config/armhf/config.flavour.generic | 15 -
> .../config/armhf/config.flavour.generic-lpae | 15 -
> debian.master/config/config.common.ubuntu | 13419 ---------
> debian.master/config/ppc64el/config.common.ppc64el | 703 -
> .../config/ppc64el/config.flavour.generic | 3 -
> debian.master/config/riscv64/config.common.riscv64 | 695 -
> .../config/riscv64/config.flavour.generic | 3 -
> debian.master/config/s390x/config.common.s390x | 627 -
> debian.master/config/s390x/config.flavour.generic | 3 -
> debian/rules.d/1-maintainer.mk | 33 +-
> debian/rules.d/4-checks.mk | 5 +-
> debian/scripts/config-check | 163 -
> debian/scripts/misc/annotations | 163 +
> debian/scripts/misc/kconfig/__init__.py | 0
> debian/scripts/misc/kconfig/annotations.py | 240 +
> debian/scripts/misc/kernelconfig | 167 +-
> debian/scripts/misc/migrate-annotations | 33 +
> debian/scripts/misc/tristate.sh | 26 -
> 26 files changed, 14919 insertions(+), 31886 deletions(-)
> delete mode 100644 debian.master/config/amd64/config.common.amd64
> delete mode 100644 debian.master/config/amd64/config.flavour.generic
> delete mode 100644 debian.master/config/arm64/config.common.arm64
> delete mode 100644 debian.master/config/arm64/config.flavour.generic
> delete mode 100644 debian.master/config/arm64/config.flavour.generic-64k
> delete mode 100644 debian.master/config/armhf/config.common.armhf
> delete mode 100644 debian.master/config/armhf/config.flavour.generic
> delete mode 100644 debian.master/config/armhf/config.flavour.generic-lpae
> delete mode 100644 debian.master/config/config.common.ubuntu
> delete mode 100644 debian.master/config/ppc64el/config.common.ppc64el
> delete mode 100644 debian.master/config/ppc64el/config.flavour.generic
> delete mode 100644 debian.master/config/riscv64/config.common.riscv64
> delete mode 100644 debian.master/config/riscv64/config.flavour.generic
> delete mode 100644 debian.master/config/s390x/config.common.s390x
> delete mode 100644 debian.master/config/s390x/config.flavour.generic
> delete mode 100755 debian/scripts/config-check
> create mode 100755 debian/scripts/misc/annotations
> create mode 100644 debian/scripts/misc/kconfig/__init__.py
> create mode 100644 debian/scripts/misc/kconfig/annotations.py
> create mode 100755 debian/scripts/misc/migrate-annotations
> delete mode 100755 debian/scripts/misc/tristate.sh
More information about the kernel-team
mailing list