[PULL v4] [L/unstable] merge configs into annotations
Andrea Righi
andrea.righi at canonical.com
Fri Nov 18 13:23:09 UTC 2022
ChangeLog (v3 -> v4):
- allow to use config+annotations or annotations-only when generating
the initial configs and config checks (during the package build)
- support configs+annotations and annotations-only for snapcraft
builds
ChangeLog (v2 -> v3):
- include only the relevant patches in the pull request
ChangeLog (v1 -> v2):
- added --update option to annotations script to import a partial
.config into annotations (not a full resync like --import)
- added debian/rules migrateconfigs to automatically merge old
configs+annotations into annotations
- support both configs+annotations and annotations-only (but once
everything is migrated to annotations-only via migrateconfigs we
can't go back to configs+annotations)
[Overview]
Each Ubuntu kernel needs to maintain its own .config for each supported
architecture and each flavour.
Every time a new patch is applied or a kernel is rebased on top of a new
one, we need to update the .config's accordingly (config options can be
added, removed and also renamed).
So, we need to make sure that some critical config options are always
matching the desired value in order to have a functional kernel.
[State of the art]
At the moment configs are maintained as a set of Kconfig chunks (inside
debian.<kernel>/config/): a global one, plus per-arch / per-flavour
chunks.
In addition to that, we need to maintain also a file called
'annotations'; the purpose of this file is to make sure that some
critical config options are not silently removed or changed when the
real .config is re-generated (for example after a rebase or after
applying a new set of patches).
The main problem with this approach is that, often, we have duplicate
information that is stored both in the Kconfig chunks *and* in the
annotations files and, at the same time, the whole .config's information
is distributed between Kconfig chunks and annotations, making it hard to
maintain, review and manage in general.
[Proposed solution]
The proposed solution is to store all the config information into the
"annotations" format and get rid of the config chunks (basically the
real .config's can be produced "compiling" annotations).
[Implementation]
To help the management of the annotations an helper script is provided
(debian/scripts/misc/annotations):
```
usage: annotations [-h] [--version] [--file FILE] [--arch ARCH] [--flavour FLAVOUR]
+[--config CONFIG]
(--query | --export | --import FILE | --update FILE | --check
+FILE)
Manage Ubuntu kernel .config and annotations
options:
-h, --help show this help message and exit
--version, -v show program's version number and exit
--file FILE, -f FILE Pass annotations or .config file to be parsed
--arch ARCH, -a ARCH Select architecture
--flavour FLAVOUR, -l FLAVOUR
Select flavour (default is "generic")
--config CONFIG, -c CONFIG
Select a specific config option
Action:
--query, -q Query annotations
--export, -e Convert annotations to .config format
--import FILE, -i FILE
Import a full .config for a specific arch and flavour into
+annotations
--update FILE, -u FILE
Import a partial .config into annotations (only resync
+configs specified in FILE)
--check FILE, -k FILE
Validate kernel .config with annotations
```
This script allows to query config settings (per arch/flavour/config),
export them into the Kconfig format (generating the real .config files)
and check if the final .config matches the rules defined in the
annotations.
Examples (annotations is defined as an alias to
debian/scripts/annotations):
- Show settings for `CONFIG_DEBUG_INFO_BTF` for master kernel across
all the supported architectures and flavours:
$ annotations --query --config CONFIG_DEBUG_INFO_BTF
{
"policy": {
"amd64": "y",
"arm64": "y",
"armhf": "n",
"ppc64el": "y",
"riscv64": "y",
"s390x": "y"
},
"note": "'Needs newer pahole for armhf'"
}
- Dump kernel .config for arm64 and flavour generic-64k:
$ annotations --arch arm64 --flavour generic-64k --export
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_COMPAT=y
...
- Update annotations file with a new kernel .config for amd64 flavour
generic:
$ annotations --arch amd64 --flavour generic --import build/.config
Moreover, two additional kernelconfig commands are provided
(debian/rules targets):
- listnewconfigs: allow to generate a list of new config options (e.g.,
after a rebase) and store them in CONFIGS/new-<arch>-<flavour> for
review
- importconfigs: after new .config's are generated and reviewed (in
CONFIGS/<arch>-<flavour>) we can use this command to automatically
import all of them into the local annotations
To facilitate the migration from the old configs+annotations method the
new new annotations-only method an additional debian/rules helper is
provided: migrateconfigs.
Running 'debian/rules migrateconfigs' will automatically merge all the
previous configs into annotations and drop the unused scripts (local
changes still need to be committed).
[Pros and Cons]
Pros:
- avoid duplicate information in .config's and annotations
- allow to easily define groups of config settings (for a specific
environment or feature, such as annotations.clouds, annotations.ubuntu,
annotations.snapd, etc.)
- config options are more accessible, easy to change and review
- we can easily document how config options are managed (and external
contributors won't be discouraged anymore when they need to to change a
config option)
Cons:
- potential regressions: the new tool/scripts can have potential bugs,
so we could experience regressions due to some missed config changes
- kernel team need to understand the new process (even if everything
is transparent, kernel cranking process is the same, there might be
corner cases that need to be addressed and resolved manually)
[TODO]
- Migrate all flavour and arch definitions into annotations (rather
than having this information defined in multiple places inside
debian/scripts); right now this information is "partially" migrated,
meaning that we need to define arches and flavours in the headers
section of annotations (so that the annotations tool can figure out
the list of supported arches and flavours), but arches and flavours
are still defined elsewhere, ideally we would like to have arches and
flavours defined only in one place: annotations.
---
The following changes since commit 192488d98049f301363e9a1b6ede4da0eb7cc82c:
net: wwan: t7xx: Add AP CLDMA (2022-11-17 17:33:40 +0100)
are available in the Git repository at:
git://git.launchpad.net/~arighi/+git/lunar annotations
for you to fetch changes up to 75879256cad3177ce388dc6707e00c4a8c438ebf:
UBUNTU: [Config] merge configs into annotations (2022-11-18 12:44:24 +0100)
----------------------------------------------------------------
Andrea Righi (17):
UBUNTU: SAUCE: add python cached bytecode to .gitignore
UBUNTU: [Packaging] config-check: ignore values that are not defined in annotations
UBUNTU: [Packaging] config-check: do not strictly enforce CONFIG_CC_VERSION_TEXT
UBUNTU: [Packaging] introduce annotations script
UBUNTU: [Packaging] automatically generate configs from annotations
UBUNTU: [Packaging] drop deprecated script tristate.sh
UBUNTU: [Packaging] use annotations to generate initial configs and perform config-check
UBUNTU: [Packaging] simplify kernelconfig
UBUNTU: [Packaging] kernelconfig: always keep configs
UBUNTU: [Packaging] provide listnewconfigs
UBUNTU: [Packaging] kernelconfig: introduce importconfigs
UBUNTU: [Packaging] snap: support new annotations with snapcraft
UBUNTU: [Packaging] final-checks: support new annotations
UBUNTU: [Packaging] introduce migrate-annotations
UBUNTU: [Packaging] re-introduce previous kernelconfig as old-kernelconfig
UBUNTU: [Packaging] provide migrateconfigs debian/rules target
UBUNTU: [Config] merge configs into annotations
.gitignore | 3 +
debian.master/config/amd64/config.common.amd64 | 701 -
debian.master/config/amd64/config.flavour.generic | 3 -
debian.master/config/annotations | 28312 ++++++++++---------
debian.master/config/arm64/config.common.arm64 | 732 -
debian.master/config/arm64/config.flavour.generic | 14 -
.../config/arm64/config.flavour.generic-64k | 14 -
debian.master/config/armhf/config.common.armhf | 713 -
debian.master/config/armhf/config.flavour.generic | 15 -
.../config/armhf/config.flavour.generic-lpae | 15 -
debian.master/config/config.common.ubuntu | 13419 ---------
debian.master/config/ppc64el/config.common.ppc64el | 703 -
.../config/ppc64el/config.flavour.generic | 3 -
debian.master/config/riscv64/config.common.riscv64 | 695 -
.../config/riscv64/config.flavour.generic | 3 -
debian.master/config/s390x/config.common.s390x | 627 -
debian.master/config/s390x/config.flavour.generic | 3 -
debian/rules.d/1-maintainer.mk | 33 +-
debian/rules.d/2-binary-arch.mk | 9 +-
debian/rules.d/4-checks.mk | 11 +-
debian/scripts/config-check | 163 -
debian/scripts/misc/annotations | 163 +
debian/scripts/misc/final-checks | 28 +-
debian/scripts/misc/kconfig/__init__.py | 0
debian/scripts/misc/kconfig/annotations.py | 240 +
debian/scripts/misc/kernelconfig | 167 +-
debian/scripts/misc/migrate-annotations | 33 +
debian/scripts/misc/splitconfig.pl | 107 -
debian/scripts/misc/tristate.sh | 26 -
debian/snapcraft.mk | 6 +-
30 files changed, 14958 insertions(+), 32003 deletions(-)
delete mode 100644 debian.master/config/amd64/config.common.amd64
delete mode 100644 debian.master/config/amd64/config.flavour.generic
delete mode 100644 debian.master/config/arm64/config.common.arm64
delete mode 100644 debian.master/config/arm64/config.flavour.generic
delete mode 100644 debian.master/config/arm64/config.flavour.generic-64k
delete mode 100644 debian.master/config/armhf/config.common.armhf
delete mode 100644 debian.master/config/armhf/config.flavour.generic
delete mode 100644 debian.master/config/armhf/config.flavour.generic-lpae
delete mode 100644 debian.master/config/config.common.ubuntu
delete mode 100644 debian.master/config/ppc64el/config.common.ppc64el
delete mode 100644 debian.master/config/ppc64el/config.flavour.generic
delete mode 100644 debian.master/config/riscv64/config.common.riscv64
delete mode 100644 debian.master/config/riscv64/config.flavour.generic
delete mode 100644 debian.master/config/s390x/config.common.s390x
delete mode 100644 debian.master/config/s390x/config.flavour.generic
delete mode 100755 debian/scripts/config-check
create mode 100755 debian/scripts/misc/annotations
create mode 100644 debian/scripts/misc/kconfig/__init__.py
create mode 100644 debian/scripts/misc/kconfig/annotations.py
create mode 100755 debian/scripts/misc/migrate-annotations
delete mode 100755 debian/scripts/misc/splitconfig.pl
delete mode 100755 debian/scripts/misc/tristate.sh
More information about the kernel-team
mailing list