[SRU][S][RFC][PATCH 1/2] UBUNTU: [Packaging] Add incremental rebuild support
Jose Ogando
jose.ogando at canonical.com
Wed Jul 1 18:12:52 UTC 2026
Add a rebuild-% developer target that preserves the flavour object tree while invalidating the build/install stamps needed for iterative package rebuilds. Add a do_skip_btf option for local no-BTF builds and an incremental rebuild test that records logs and metrics for optimization work.
Signed-off-by: Jose Ogando <jose.ogando at canonical.com>
Co-authored-by: Copilot <223556219+Copilot at users.noreply.github.com>
---
debian/README.incremental-rebuild.md | 156 ++++++++++++
debian/rules.d/0-common-vars.mk | 8 +
debian/rules.d/2-binary-arch.mk | 19 +-
debian/tests/control | 4 +
debian/tests/incremental-rebuild | 368 +++++++++++++++++++++++++++
5 files changed, 554 insertions(+), 1 deletion(-)
create mode 100644 debian/README.incremental-rebuild.md
create mode 100755 debian/tests/incremental-rebuild
diff --git a/debian/README.incremental-rebuild.md b/debian/README.incremental-rebuild.md
new file mode 100644
index 000000000000..8b49a1ec69f4
--- /dev/null
+++ b/debian/README.incremental-rebuild.md
@@ -0,0 +1,156 @@
+# Incremental kernel rebuilds
+
+This tree provides developer targets for iterating on kernel patches without
+discarding the existing object tree between builds.
+
+## Why use this
+
+Normal clean builds are useful for release validation, but they are slow for
+patch-test loops because they remove `debian/build/build-<flavour>` and force
+Kbuild to rebuild everything. The `rebuild-<flavour>` target preserves that
+object tree, invalidates only the packaging/build stamps for the selected
+flavour, and lets Kbuild rebuild only what changed.
+
+This is intended for local developer testing. Use normal clean builds for final
+validation.
+
+## Targets
+
+| Target | Purpose |
+| --- | --- |
+| `binary` | Full Debian packaging target: arch packages, indep packages, common headers/tools/source packages, and all enabled flavours. |
+| `binary-generic` | Build and package only the `generic` flavour. This still runs the full generic install/package path. |
+| `rebuild-generic` | Rebuild and repackage the `generic` flavour while preserving `debian/build/build-generic`. |
+| `rebuild-<flavour>` | Same as above for another flavour, for example `rebuild-generic-64k`. |
+
+## Basic usage
+
+Start with an initial flavour build:
+
+```bash
+DEB_BUILD_OPTIONS=parallel=$(nproc) fakeroot debian/rules binary-generic
+```
+
+After changing kernel source, run:
+
+```bash
+DEB_BUILD_OPTIONS=parallel=$(nproc) fakeroot debian/rules rebuild-generic
+```
+
+The rebuild target removes:
+
+```text
+debian/stamps/stamp-build-generic
+debian/stamps/stamp-install-generic
+debian/build/abi-generic
+```
+
+It preserves:
+
+```text
+debian/build/build-generic
+```
+
+That preserved directory contains the existing `.config`, object files, and
+Kbuild state used for incremental rebuilds.
+
+## Optional no-BTF developer mode
+
+For faster local iteration, the initial build can disable BTF generation:
+
+```bash
+DEB_BUILD_OPTIONS=parallel=$(nproc) fakeroot debian/rules binary-generic do_skip_btf=true
+```
+
+This changes the generated build config for that object tree:
+
+```text
+CONFIG_DEBUG_INFO_BTF=n
+CONFIG_DEBUG_INFO_BTF_MODULES=n
+```
+
+It also uses the existing `bpftool` stub path so packaging does not try to dump
+`vmlinux.h` from a no-BTF `vmlinux`.
+
+Subsequent rebuilds inherit the no-BTF config automatically because
+`rebuild-generic` preserves `debian/build/build-generic/.config`:
+
+```bash
+DEB_BUILD_OPTIONS=parallel=$(nproc) fakeroot debian/rules rebuild-generic
+```
+
+If you run `debian/rules clean`, pass `do_skip_btf=true` again on the next
+initial build. This mode skips the annotations config check because the local
+developer config intentionally differs from Ubuntu policy.
+
+## Validation test
+
+The test script exercises the incremental workflow and saves logs/metrics for
+later optimization work:
+
+```bash
+DEB_BUILD_OPTIONS=parallel=$(nproc) debian/tests/incremental-rebuild \
+ --clean \
+ --flavour generic \
+ --log-dir ../incremental-rebuild-generic
+```
+
+No-BTF test run:
+
+```bash
+DEB_BUILD_OPTIONS=parallel=$(nproc) debian/tests/incremental-rebuild \
+ --clean \
+ --skip-btf \
+ --flavour generic \
+ --log-dir ../incremental-rebuild-generic-skip-btf
+```
+
+The test:
+
+1. Optionally runs `debian/rules clean`.
+2. Builds a baseline `binary-<flavour>`.
+3. Temporarily patches `init/version.c` to exercise the image/vmlinux path.
+4. Runs `rebuild-<flavour>` and verifies newer image/modules packages.
+5. Temporarily patches `drivers/net/dummy.c` to exercise a module path.
+6. Runs `rebuild-<flavour>` again and verifies newer packages.
+7. Restores the modified source files on exit.
+
+Saved artifacts include:
+
+```text
+full.log
+metrics.tsv
+image-packages.txt
+module-packages.txt
+*-packages.before.tsv
+*-packages.after.tsv
+```
+
+## Measured results
+
+On this system with `DEB_BUILD_OPTIONS=parallel=16`, the latest full rerun
+measured:
+
+| Scenario | With BTF | Without BTF | Saving |
+| --- | ---: | ---: | ---: |
+| Initial clean `binary-generic` | 48m25s | 39m03s | 9m22s |
+| `init/version.c` / zImage rebuild | 11m46s | 7m48s | 3m58s |
+| `drivers/net/dummy.c` / module rebuild | 8m16s | 7m38s | 38s |
+
+The no-BTF run produced zero BTF-related build lines, while the BTF-enabled run
+produced 13,810 BTF-related lines across the test.
+
+## Remaining bottlenecks
+
+Even with BTF disabled, module rebuilds still spend time in global module
+finalization and packaging. A module edit still triggers `MODPOST
+Module.symvers`, then `modules_install`, strip, signing, and package generation
+for thousands of modules.
+
+Potential future optimizations:
+
+1. Add a compile-only target that stops after `stamp-build-<flavour>`.
+2. Add a developer-only module repack target that replaces only changed modules
+ in an existing staging tree and rebuilds only `linux-modules-*`.
+3. Add finer-grained timing markers around Kbuild, `modules_install`,
+ module-signature checks, ABI generation, and each package build step.
diff --git a/debian/rules.d/0-common-vars.mk b/debian/rules.d/0-common-vars.mk
index 66a486abf219..fd4428e5bb69 100644
--- a/debian/rules.d/0-common-vars.mk
+++ b/debian/rules.d/0-common-vars.mk
@@ -148,6 +148,14 @@ do_flavour_header_package=true
# DTBs
do_dtbs=false
+# Developer-only shortcut to disable BTF in the generated build config.
+# Set do_skip_btf=true on the initial build; incremental rebuilds reuse the
+# existing build tree and inherit the generated no-BTF .config.
+do_skip_btf=false
+ifeq ($(do_skip_btf),true)
+do_tools_bpftool_stub=true
+endif
+
# ZSTD compressed kernel modules
do_zstd_ko=true
diff --git a/debian/rules.d/2-binary-arch.mk b/debian/rules.d/2-binary-arch.mk
index 2ce38327e804..b236d2e52cea 100644
--- a/debian/rules.d/2-binary-arch.mk
+++ b/debian/rules.d/2-binary-arch.mk
@@ -39,12 +39,21 @@ $(stampdir)/stamp-prepare-%: debian/scripts/fix-filenames
touch $(build_dir)/ubuntu-build
python3 debian/scripts/misc/annotations --export --arch $(arch) --flavour $* > $(build_dir)/.config
sed -i 's/.*CONFIG_VERSION_SIGNATURE.*/CONFIG_VERSION_SIGNATURE="Ubuntu $(DEB_VERSION_UPSTREAM)-$(DEB_REVISION)-$* $(raw_kernelversion)"/' $(build_dir)/.config
+ifeq ($(do_skip_btf),true)
+ scripts/config --file $(build_dir)/.config \
+ --disable DEBUG_INFO_BTF \
+ --disable DEBUG_INFO_BTF_MODULES
+endif
find $(build_dir) -name "*.ko" | xargs rm -f
$(kmake) O=$(build_dir) $(conc_level) rustavailable || true
$(kmake) O=$(build_dir) $(conc_level) olddefconfig
ifneq ($(do_skip_checks),true)
+ ifneq ($(do_skip_btf),true)
python3 debian/scripts/misc/annotations -f $(CURDIR)/$(DEBIAN)/config/annotations \
--arch $(arch) --flavour $* --check $(build_dir)/.config
+ else
+ @echo "Skipping annotations check because do_skip_btf=true"
+ endif
endif
$(stamp)
@@ -55,6 +64,15 @@ prepare-%: $(stampdir)/stamp-prepare-%
build-%: $(stampdir)/stamp-install-%
@echo Debug: $@
+# Used by developers to rebuild debs incrementally after patching.
+# Only the packaging stamps and ABI output are reset; debian/build/build-$*
+# is preserved so Kbuild can reuse existing object files.
+rebuild-%: FORCE
+ @echo Debug: $@
+ rm -f $(stampdir)/stamp-build-$* $(stampdir)/stamp-install-$*
+ rm -rf $(abi_dir)
+ $(MAKE) -f debian/rules do_full_build=false binary-$*
+
# Do the actual build, including image and modules
$(stampdir)/stamp-build-%: bldimg = $(call custom_override,build_image,$*)
$(stampdir)/stamp-build-%: $(stampdir)/stamp-build-perarch $(stampdir)/stamp-prepare-%
@@ -758,4 +776,3 @@ endif
.PHONY: binary-arch
binary-arch: $(binary-arch-deps-true)
@echo Debug: $@
-
diff --git a/debian/tests/control b/debian/tests/control
index d083d6562261..b791966cafc5 100644
--- a/debian/tests/control
+++ b/debian/tests/control
@@ -2,6 +2,10 @@ Tests: rebuild
Depends: @builddeps@, fakeroot
Restrictions: allow-stderr, skippable
+Tests: incremental-rebuild
+Depends: @builddeps@, fakeroot, git, python3
+Restrictions: allow-stderr, skippable
+
Tests: ubuntu-regression-suite
Depends: build-essential, fakeroot, gcc-multilib [amd64 armhf i386], gdb, git, python3, snapd, fuse-overlayfs
Restrictions: allow-stderr, isolation-machine, breaks-testbed, skippable
diff --git a/debian/tests/incremental-rebuild b/debian/tests/incremental-rebuild
new file mode 100755
index 000000000000..100ebf6b4540
--- /dev/null
+++ b/debian/tests/incremental-rebuild
@@ -0,0 +1,368 @@
+#!/bin/bash
+#
+# Validate debian/rules rebuild-$flavour for iterative patch testing.
+
+set -euo pipefail
+
+clean=false
+flavour="${FLAVOUR:-generic}"
+skip_btf="${SKIP_BTF:-false}"
+repo_root="$(pwd)"
+tmpdir=""
+log_file="${LOG_FILE:-}"
+log_dir="${LOG_DIR:-}"
+metrics_file=""
+log_started=false
+
+usage() {
+ cat <<EOF
+Usage: $0 [--clean|--no-clean] [--skip-btf|--no-skip-btf] [--flavour FLAVOUR] [--log-dir DIR] [--log-file PATH]
+
+Environment:
+ FLAVOUR Kernel flavour to test, default: generic
+ SKIP_BTF Disable BTF in the generated build config, default: false
+ DEB_BUILD_OPTIONS Passed through to debian/rules
+ LOG_DIR Directory for saved logs and metrics
+ LOG_FILE Report file path
+
+The test assumes an initial build already exists. It applies two temporary
+source edits, rebuilds with debian/rules rebuild-\$flavour, and verifies that
+the relevant object files and .deb packages become newer after each scenario.
+All output is written to the artifact directory and echoed to the terminal.
+EOF
+}
+
+log() {
+ printf '[%s] %s\n' "$(date -Ins)" "$*"
+}
+
+skip() {
+ log "SKIP: $*"
+ exit 77
+}
+
+fail() {
+ log "FAIL: $*"
+ exit 1
+}
+
+while [ "$#" -gt 0 ]; do
+ case "$1" in
+ --clean)
+ clean=true
+ shift
+ ;;
+ --no-clean)
+ clean=false
+ shift
+ ;;
+ --skip-btf)
+ skip_btf=true
+ shift
+ ;;
+ --no-skip-btf)
+ skip_btf=false
+ shift
+ ;;
+ --flavour)
+ [ "$#" -ge 2 ] || fail "--flavour requires an argument"
+ flavour="$2"
+ shift 2
+ ;;
+ --log-dir)
+ [ "$#" -ge 2 ] || fail "--log-dir requires an argument"
+ log_dir="$2"
+ shift 2
+ ;;
+ --log-file)
+ [ "$#" -ge 2 ] || fail "--log-file requires an argument"
+ log_file="$2"
+ shift 2
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ fail "unknown argument: $1"
+ ;;
+ esac
+done
+
+start_report() {
+ local run_id
+
+ run_id="$(date +%Y%m%d-%H%M%S)"
+ if [ -z "$log_dir" ]; then
+ if [ -n "$log_file" ]; then
+ log_dir="$(dirname "$log_file")"
+ elif [ -n "${AUTOPKGTEST_ARTIFACTS:-}" ]; then
+ log_dir="$AUTOPKGTEST_ARTIFACTS/incremental-rebuild-$flavour-$run_id"
+ else
+ log_dir="../incremental-rebuild-$flavour-$run_id"
+ fi
+ fi
+
+ mkdir -p "$log_dir"
+ log_dir="$(cd "$log_dir" && pwd)"
+ if [ -z "$log_file" ]; then
+ log_file="$log_dir/full.log"
+ fi
+ touch "$log_file"
+ log_file="$(cd "$(dirname "$log_file")" && pwd)/$(basename "$log_file")"
+ metrics_file="$log_dir/metrics.tsv"
+ printf 'timestamp\tevent\tname\tvalue\n' >"$metrics_file"
+ log_started=true
+ exec > >(tee -a "$log_file") 2>&1
+ log "artifacts: $log_dir"
+ log "report: $log_file"
+ log "metrics: $metrics_file"
+}
+
+cleanup() {
+ if [ -n "$tmpdir" ] && [ -d "$tmpdir" ]; then
+ [ -f "$tmpdir/version.c" ] && cp "$tmpdir/version.c" init/version.c
+ [ -f "$tmpdir/dummy.c" ] && cp "$tmpdir/dummy.c" drivers/net/dummy.c
+ rm -rf "$tmpdir"
+ fi
+ if [ "$log_started" = true ]; then
+ log "report complete: $log_file"
+ log "metrics complete: $metrics_file"
+ fi
+}
+
+metric() {
+ printf '%s\t%s\t%s\t%s\n' "$(date -Ins)" "$1" "$2" "$3" >>"$metrics_file"
+}
+
+mtime_ns() {
+ python3 - "$1" <<'PY'
+import os
+import sys
+
+print(os.stat(sys.argv[1]).st_mtime_ns)
+PY
+}
+
+assert_newer() {
+ local path="$1"
+ local before="$2"
+ local after
+
+ [ -e "$path" ] || fail "expected $path to exist"
+ after="$(mtime_ns "$path")"
+ if [ "$after" -le "$before" ]; then
+ fail "$path was not regenerated: before=$before after=$after"
+ fi
+ log "newer: $path"
+}
+
+collect_packages() {
+ local kind="$1"
+ local pattern
+
+ case "$kind" in
+ image)
+ pattern="../linux-image-*-${flavour}_${deb_version}_${host_arch}.deb"
+ ;;
+ modules)
+ pattern="../linux-modules-*-${flavour}_${deb_version}_${host_arch}.deb"
+ ;;
+ *)
+ fail "unknown package kind: $kind"
+ ;;
+ esac
+
+ find .. -maxdepth 1 -type f -name "$(basename "$pattern")" | sort
+}
+
+snapshot_packages() {
+ local output="$1"
+ shift
+ local path
+
+ : >"$output"
+ for path in "$@"; do
+ [ -e "$path" ] || fail "expected package $path to exist"
+ printf '%s\t%s\n' "$path" "$(mtime_ns "$path")" >>"$output"
+ done
+}
+
+assert_packages_newer() {
+ local snapshot="$1"
+ local path before
+
+ while IFS=$'\t' read -r path before; do
+ assert_newer "$path" "$before"
+ done <"$snapshot"
+}
+
+patch_zimage_source() {
+ local stamp="$1"
+
+ log "patch: init/version.c internal image marker $stamp"
+ python3 - "$stamp" <<'PY'
+from pathlib import Path
+import sys
+
+stamp = sys.argv[1]
+path = Path("init/version.c")
+text = path.read_text()
+needle = "BUILD_LTO_INFO;\n"
+insert = f'const char ubuntu_incremental_rebuild_zimage_test[] = "{stamp}";\n'
+if insert in text:
+ raise SystemExit(0)
+if needle not in text:
+ raise SystemExit(f"missing insertion point in {path}")
+path.write_text(text.replace(needle, needle + insert, 1))
+PY
+}
+
+patch_module_source() {
+ local stamp="$1"
+
+ log "patch: drivers/net/dummy.c module modinfo marker $stamp"
+ python3 - "$stamp" <<'PY'
+from pathlib import Path
+import sys
+
+stamp = sys.argv[1]
+path = Path("drivers/net/dummy.c")
+text = path.read_text()
+needle = 'MODULE_DESCRIPTION("Dummy netdevice driver which discards all packets sent to it");\n'
+insert = f'MODULE_INFO(incremental_rebuild_test, "{stamp}");\n'
+if insert in text:
+ raise SystemExit(0)
+if needle not in text:
+ raise SystemExit(f"missing insertion point in {path}")
+path.write_text(text.replace(needle, needle + insert, 1))
+PY
+}
+
+run_rebuild() {
+ local scenario="$1"
+ local start end duration
+
+ log "run: fakeroot debian/rules rebuild-$flavour"
+ start="$(date +%s)"
+ if [ "$skip_btf" = true ]; then
+ fakeroot debian/rules "rebuild-$flavour" do_skip_btf=true
+ else
+ fakeroot debian/rules "rebuild-$flavour"
+ fi
+ end="$(date +%s)"
+ duration="$((end - start))"
+ metric "duration_seconds" "$scenario" "$duration"
+ log "duration: $scenario rebuild ${duration}s"
+}
+
+run_baseline_build() {
+ local start end duration
+
+ log "run: fakeroot debian/rules binary-$flavour"
+ start="$(date +%s)"
+ if [ "$skip_btf" = true ]; then
+ fakeroot debian/rules "binary-$flavour" do_skip_btf=true
+ else
+ fakeroot debian/rules "binary-$flavour"
+ fi
+ end="$(date +%s)"
+ duration="$((end - start))"
+ metric "duration_seconds" "baseline" "$duration"
+ log "duration: baseline build ${duration}s"
+}
+
+assert_btf_disabled() {
+ if [ "$skip_btf" != true ]; then
+ return 0
+ fi
+
+ grep -q '^# CONFIG_DEBUG_INFO_BTF is not set$' "$build_dir/.config" || \
+ fail "CONFIG_DEBUG_INFO_BTF was not disabled in $build_dir/.config"
+ ! grep -q '^CONFIG_DEBUG_INFO_BTF_MODULES=y$' "$build_dir/.config" || \
+ fail "CONFIG_DEBUG_INFO_BTF_MODULES was not disabled in $build_dir/.config"
+ metric "config" "btf_disabled" "true"
+ log "verified: BTF disabled in $build_dir/.config"
+}
+
+[ -f debian/debian.env ] || fail "run this from the kernel source root"
+[ -f init/version.c ] || fail "missing init/version.c"
+[ -f drivers/net/dummy.c ] || fail "missing drivers/net/dummy.c"
+
+start_report
+
+if ! git diff --quiet -- init/version.c drivers/net/dummy.c; then
+ fail "test source files are already modified; commit/stash them before running"
+fi
+
+trap cleanup EXIT
+tmpdir="$(mktemp -d)"
+cp init/version.c "$tmpdir/version.c"
+cp drivers/net/dummy.c "$tmpdir/dummy.c"
+
+debian_dir="$(awk -F= '($1 == "DEBIAN") { print $2 }' <debian/debian.env)"
+deb_version="$(dpkg-parsechangelog -l"$debian_dir/changelog" -S version)"
+host_arch="$(dpkg-architecture -qDEB_HOST_ARCH)"
+build_dir="$repo_root/debian/build/build-$flavour"
+zimage_object="$build_dir/init/version.o"
+module_object="$build_dir/drivers/net/dummy.ko"
+
+metric "config" "flavour" "$flavour"
+metric "config" "clean" "$clean"
+metric "config" "skip_btf" "$skip_btf"
+metric "config" "version" "$deb_version"
+metric "config" "arch" "$host_arch"
+metric "config" "build_dir" "$build_dir"
+metric "config" "deb_build_options" "${DEB_BUILD_OPTIONS:-}"
+
+log "start: incremental rebuild validation flavour=$flavour clean=$clean skip_btf=$skip_btf version=$deb_version arch=$host_arch"
+
+if [ "$clean" = true ]; then
+ log "clean: fakeroot debian/rules clean"
+ fakeroot debian/rules clean
+ run_baseline_build
+else
+ log "clean: skipped"
+fi
+
+mapfile -t image_packages < <(collect_packages image)
+mapfile -t module_packages < <(collect_packages modules)
+
+[ "${#image_packages[@]}" -gt 0 ] || skip "no existing image package found for flavour=$flavour version=$deb_version arch=$host_arch"
+[ "${#module_packages[@]}" -gt 0 ] || skip "no existing modules package found for flavour=$flavour version=$deb_version arch=$host_arch"
+[ -e "$zimage_object" ] || skip "no existing $zimage_object; run an initial build first"
+[ -e "$module_object" ] || skip "no existing $module_object; ensure CONFIG_DUMMY=m and run an initial build first"
+assert_btf_disabled
+
+printf '%s\n' "${image_packages[@]}" >"$log_dir/image-packages.txt"
+printf '%s\n' "${module_packages[@]}" >"$log_dir/module-packages.txt"
+metric "object_mtime_ns" "zimage_before" "$(mtime_ns "$zimage_object")"
+metric "object_mtime_ns" "module_before" "$(mtime_ns "$module_object")"
+
+snapshot_packages "$log_dir/zimage-image-packages.before.tsv" "${image_packages[@]}"
+snapshot_packages "$log_dir/zimage-module-packages.before.tsv" "${module_packages[@]}"
+zimage_object_before="$(mtime_ns "$zimage_object")"
+patch_zimage_source "zimage-$(date +%s)"
+run_rebuild "zimage"
+assert_newer "$zimage_object" "$zimage_object_before"
+metric "object_mtime_ns" "zimage_after" "$(mtime_ns "$zimage_object")"
+snapshot_packages "$log_dir/zimage-image-packages.after.tsv" "${image_packages[@]}"
+snapshot_packages "$log_dir/zimage-module-packages.after.tsv" "${module_packages[@]}"
+assert_packages_newer "$log_dir/zimage-image-packages.before.tsv"
+assert_packages_newer "$log_dir/zimage-module-packages.before.tsv"
+log "scenario complete: internal image source edit regenerated object and packages"
+
+snapshot_packages "$log_dir/module-image-packages.before.tsv" "${image_packages[@]}"
+snapshot_packages "$log_dir/module-module-packages.before.tsv" "${module_packages[@]}"
+module_object_before="$(mtime_ns "$module_object")"
+patch_module_source "module-$(date +%s)"
+run_rebuild "module"
+assert_newer "$module_object" "$module_object_before"
+metric "object_mtime_ns" "module_after" "$(mtime_ns "$module_object")"
+snapshot_packages "$log_dir/module-image-packages.after.tsv" "${image_packages[@]}"
+snapshot_packages "$log_dir/module-module-packages.after.tsv" "${module_packages[@]}"
+assert_packages_newer "$log_dir/module-image-packages.before.tsv"
+assert_packages_newer "$log_dir/module-module-packages.before.tsv"
+log "scenario complete: module source edit regenerated module and packages"
+
+log "PASS: rebuild-$flavour preserved the build tree while regenerating newer packages"
--
2.53.0
More information about the kernel-team
mailing list