APPLIED/cmnt: [SRU][PULL][linux-firmware][Lunar] UBUNTU: [Packaging] scripts: add back list-lts-update-files
Juerg Haefliger
juerg.haefliger at canonical.com
Fri Jan 20 15:54:53 UTC 2023
Applied to linux-firmware lunar branch.
Note that your submission email also contains the patch content inline.
Please don't do that.
I also just realized that the patch author email and SOB are different (gmail
vs. canonical). I accept it this time but only because I missed it in your
jammy submission. In the future, please make sure the SOB and author
name/email are identical for patches that you author yourself.
...Juerg
> BugLink: https://bugs.launchpad.net/bugs/1999406
>
> [Impact]
>
> When migrating to a new hwe kernel or introduced a oem kernel of a
> newer version, the firmware blobs might not match the expectation of
> the new kernel. For example, a hwe-5.15 on Focal may include new
> drivers that relies on firmware blobs that are only available in Jammy.
>
> [Fix]
>
> A previously obsoleted (and removed in Kinetic and Lunar) script
> `list-lts-update-files` is used to enumerate firmware blobs to be
> backported. It's retored and revised with additional support to
> pick the right blobs especially in regard to ath and iwlwifi drivers.
>
> On Focal, the commits in need are enumerated by:
>
> $ debian/scripts/list-lts-update-files focal jammy \
> ../jammy/debian.master/abi/fwinfo | \
> xargs git log --graph --oneline focal..jammy --
> * 36f2ea9f7 ath11k: WCN6855 hw2.0: add WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3
> * 05b5dc001 ath11k: WCN6855 hw2.0: add board-2.bin and regdb.bin
> * 7a3005059 brcm: Add 43455 based AP6255 NVRAM for the ACEPC T8 Mini PC
> * ad185afa1 cypress: update firmware for cyw4373 sdio
> * 77d3eb8ef cypress: update firmware for cyw43570 pcie
> * 92e9acdbd cypress: update firmware for cyw4356 sdio
> * 5f88084be cypress: update firmware for cyw4354 sdio
> * f97e31677 cypress: update firmware for cyw43455 sdio
> * 3df9ea0b9 cypress: update firmware for cyw43430 sdio
> * 6150015cf cypress: update firmware for cyw43012 sdio
> * 2548d065b brcm: Add nvram for the Chuwi Hi8 (CWI509) tablet
> * e45c137e7 brcm: Add nvram for the Predia Basic tablet
> * d52886242 brcm: Add NVRAM for Vamrs 96boards Rock960
> * b503c9660 Merge branch 'ath10k-20201023' of git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/linux-firmware into main
> |\
> | * 34cb5fce2 ath11k: IPQ8074 hw2.0: add to WLAN.HK.2.1.0.1-01238-QCAHKSWPL_SILICONZ-2
> | * c0a8efd24 ath11k: IPQ8074 hw2.0: add board-2.bin
> | * ac7f5e93f ath11k: IPQ6018 hw1.0: add to WLAN.HK.2.1.0.1-01238-QCAHKSWPL_SILICONZ-2
> | * 2594e510a ath11k: IPQ6018 hw1.0: add board-2.bin
> * 04f71fe56 cypress: add Cypress firmware and clm_blob files
>
> Commit 05b5dc001 and 36f2ea9f7 include updates to other existing files,
> so they are dropped from this SRU. An additional commit that modifies
> only WHENCE, commit 0b558e8a7, found when resolving conflicts, were also
> added.
>
> On Jammy,
>
> $ debian/scripts/list-lts-update-files jammy kinetic \
> ../kinetic/debian.master/abi/fwinfo | \
> xargs git log --graph --oneline jammy..kinetic --
> * 35f8de521 UBUNTU: Add pre-compiled echoaudio firmware
> * c954892f6 Add initial AzureWave AW-CM256SM NVRAM file
> * 86f0c5642 ath10k: WCN3990 hw1.0: add board-2.bin
> * c3624ebd6 Merge branch 'ath10k-20220423' of git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/linux-firmware into main
> |\
> | * 1962cbab3 ath10k: QCA99X0 hw2.0: add board-2.bin
> | * 0d5e9f7e0 ath11k: WCN6750 hw1.0: add to WLAN.MSL.1.0.1-00887-QCAMSLSWPLZ-1
> | * a50132f7f ath11k: WCN6750 hw1.0: add board-2.bin
> | * 97f8b7563 ath11k: QCN9074 hw1.0: add to WLAN.HK.2.5.0.1-01208-QCAHKSWPL_SILICONZ-1
> | * f56505fe2 ath11k: QCN9074 hw1.0: add board-2.bin
> * | dfa3c4c30 qcom: add firmware files for Adreno a420 & related generations
> * | 4a43f1a84 qcom: add firmware files for Adreno a330
> * | ac21ab5d1 Mellanox: Add lc_ini_bundle for xx.2010.1006
> |/
> * 4ffcf980a brcm: rename Rock960 NVRAM to AP6356S and link devices to it
>
> Commit 4ffcf980a involves rename existing files in WHENCE, so ignored.
>
> While Jammy has 3 additional oem kernels, they were also tested:
>
> $ debian/scripts/list-lts-update-files jammy kinetic \
> ../oem-5.17/debian.oem/abi/fwinfo | \
> xargs git log --graph --oneline jammy..kinetic --
> (no new commits)
>
> $ debian/scripts/list-lts-update-files jammy lunar \
> ../oem-6.0/debian.oem/abi/fwinfo | \
> xargs git log --graph --oneline jammy..lunar --
> * 06dbfbc74 iwlwifi: add new FWs from core69-81 release
>
> $ debian/scripts/list-lts-update-files jammy lunar \
> ../oem-6.1/debian.oem/abi/fwinfo | \
> xargs git log --graph --oneline jammy..lunar --
> * 51fff4e69 i915: Add versionless HuC files for current platforms
>
> On Lunar, while it's not an LTS series and there will not be any hwe/oem
> kernel backported, the only change proposed is the script itself.
>
> [Test Case]
>
> To ensure no exiting bits being overwritten, the changes are reviewed
> manually.
>
> [Where problems could occur]
>
> These firmware blobs are only referenced in the new hwe/oem kernels, and
> shall not have side effect. On Focal and Jammy, while we have rolled out
> oem and hwe kernels a long time ago, these changes will enable the
> corresponding hardware that were meant to be enabled along with the
> oem/hwe kernel migration at the time.
>
> [Other Info]
>
> While there is no hwe/oem kernel planned for Kinetic, and unlike the
> devel series Lunar, adding this script to it is quite meaningless, so
> it's not nominated for fix here.
>
> ----------------------------------------------------------------
> The following changes since commit e3ff59307071a7867b3436f48bf69a7156abd2f1:
>
> UBUNTU: Ubuntu-20221212.git0707b2f2-0ubuntu1 (2022-12-12 08:51:34 +0100)
>
> are available in the Git repository at:
>
> https://git.launchpad.net/~vicamo/ubuntu/+source/linux-firmware bug-1999406/update-fw-for-hwe-kernels/lunar
>
> for you to fetch changes up to 834d95c984cd90dba9c6e9d497732fa08dd9a920:
>
> UBUNTU: [Packaging] scripts: add back list-lts-update-files (2022-12-13 20:05:46 +0800)
>
> ----------------------------------------------------------------
> You-Sheng Yang (vicamo) (1):
> UBUNTU: [Packaging] scripts: add back list-lts-update-files
>
> debian/scripts/list-lts-update-files | 248 +++++++++++++++++++++++++++++++++++
> 1 file changed, 248 insertions(+)
> create mode 100755 debian/scripts/list-lts-update-files
>
> diff --git a/debian/scripts/list-lts-update-files b/debian/scripts/list-lts-update-files
> new file mode 100755
> index 000000000..46fa3dfff
> --- /dev/null
> +++ b/debian/scripts/list-lts-update-files
> @@ -0,0 +1,248 @@
> +#!/bin/bash
> +#
> +# Used to get a list of firmware files needed to update a LTS
> +# linux-firmware package for a backport kernel. For example to
> +# get a list of files for trusty to support the linux-lts-xenial
> +# kernel run:
> +#
> +# debian/scripts/list-lts-update-files [--dump-filtered] trusty xenial /path/to/xenial/fwinfo
> +#
> +# This will generate a list of files that satisfy all of these
> +# conditions:
> +#
> +# 1) In the xenial branch of linux-firmware
> +# 2) Not in the trusty branch of linux-firmware
> +# 3) In the specified fwinfo file
> +#
> +# This can be used as a starting point for finding upstream
> +# commits needed to add the missing firmware files.
> +#
> +# The fwinfo file can be omitted, in which case the output is a list
> +# of firmware files in @newbranch but not in @oldbranch.
> +
> +#set -ex
> +
> +dump_filtered=
> +if [ $# -ge 1 ]; then
> + case "$1" in
> + --dump-filtered)
> + dump_filtered=1; shift;;
> + *) ;;
> + esac
> +fi
> +
> +if [ $# -lt 2 ]; then
> + echo "Usage $0 [--dump-filtered] <oldbranch> <newbranch> [<fwinfo>]"
> + exit 1
> +fi
> +
> +OLDBRANCH="$1"
> +NEWBRANCH="$2"
> +FWINFO="$3"
> +
> +TMPDIR=$(mktemp -d)
> +
> +function cleanup {
> + rm -r "$TMPDIR"
> +}
> +trap cleanup EXIT
> +
> +function normalize_link {
> + # handle relative paths: a/b/c/../../d/e -> a/d/e
> + local s="$1"
> + s="${s#./}"
> + while true; do
> + case "$s" in
> + */../*)
> + s=$(echo "$s" | sed 's,[^/]\+/../,,')
> + ;;
> + *)
> + break ;;
> + esac
> + done
> + echo "$s"
> +}
> +
> +function grep_links {
> + grep ^WHENCE "$1" | sed "s,^,$2:," | xargs git show | grep ^Link:
> +}
> +
> +if ! git ls-tree -r --name-only "$OLDBRANCH" >"$TMPDIR/files.tmp"
> +then
> + exit 1
> +fi
> +cat "$TMPDIR/files.tmp" | sort >"$TMPDIR/oldbranch-files"
> +
> +if ! git ls-tree -r --name-only "$NEWBRANCH" >"$TMPDIR/files.tmp"
> +then
> + exit 1
> +fi
> +cat "$TMPDIR/files.tmp" | sort >"$TMPDIR/newbranch-files"
> +
> +# Get files in @newbranch not in @oldbranch
> +comm -13 "$TMPDIR/oldbranch-files" "$TMPDIR/newbranch-files" >"$TMPDIR/new-files"
> +
> +if [ "$3" == "" ]; then
> + cat "$TMPDIR/new-files"
> + exit 0
> +fi
> +
> +if ! cat $FWINFO | head -n1 | grep "firmware: " > /dev/null
> +then
> + echo "Invliad fwinfo file $FWINFO"
> + exit 1
> +fi
> +
> +declare -A oldbranch_files
> +while read of; do
> + oldbranch_files["$of"]="old"
> +done < "$TMPDIR/oldbranch-files"
> +
> +while read dc1 ol dc2 of; do
> + if [ -z "${oldbranch_files[$ol]:-}" ]; then
> + oldbranch_files["$ol"]="link"
> + fi
> +done < <(grep_links "$TMPDIR/oldbranch-files" "$OLDBRANCH")
> +
> +declare -A new_files
> +while read nf; do
> + if [ -n "${oldbranch_files["$nf"]:-}" ]; then
> + # new file but matches a previous link
> + new_files["$nf"]="duplicated"
> + elif [ -n "$(git log "$OLDBRANCH" -1 -- "$nf")" ]; then
> + new_files["$nf"]="deleted"
> + else
> + new_files["$nf"]="unused"
> + fi
> +done < "$TMPDIR/new-files"
> +
> +declare -A newlink_files
> +while read dc1 nl dc2 nf; do
> +
> + nf="$(normalize_link "$(dirname "$nl")/$nf")"
> + newlink_files["$nl"]="$nf"
> +done < <(grep_links "$TMPDIR/newbranch-files" "$NEWBRANCH")
> +
> +function check_duplicated {
> + if [ -n "${oldbranch_files["$1"]:-}" ]; then
> + return 0
> + fi
> +
> + if [ -n "${newlink_files["$1"]:-}" ]; then
> + if check_duplicated "${newlink_files["$1"]}"; then
> + return 0
> + fi
> + fi
> +
> + return 1
> +}
> +
> +function mark_if_unused {
> + if [ -n "${new_files["$1"]:-}" ]; then
> + if [ "${new_files["$1"]}" = "unused" ]; then
> + new_files["$1"]="$2"
> + fi
> +
> + return 0
> + fi
> +
> + if [ -n "${newlink_files["$1"]:-}" ]; then
> + if mark_if_unused "${newlink_files["$1"]}" "$2"; then
> + return 0
> + fi
> + fi
> +
> + return 1
> +}
> +
> +while read fw_pattern; do
> + fws=()
> +
> + case "$fw_pattern" in
> + *\**)
> + # pattern with wildcard
> + for nf in "${!new_files[@]}"; do
> + case "$nf" in
> + $fw_pattern)
> + fws+=("$nf");;
> + *)
> + ;;
> + esac
> + done
> + ;;
> + *)
> + fws+=("$fw_pattern")
> + ;;
> + esac
> +
> + for fw in "${fws[@]}"; do
> + case "$fw" in
> + iwlwifi-*.ucode)
> + # check if new_files contains one with lower-equal FW API
> + prefix="${fw%-*}"
> + max_fw_api="${fw##*-}"
> + max_fw_api="${max_fw_api%.ucode}"
> + found=
> + for api in $(seq $max_fw_api -1 1); do
> + nf="$prefix-$api.ucode"
> +
> + if [ -n "$found" ]; then
> + mark_if_unused "$nf" "iwlwifi fwapi skipped" || true
> + continue
> + fi
> +
> + if check_duplicated "$nf"; then
> + mark_if_unused "$nf" "duplicated" || true
> + found=1
> + elif mark_if_unused "$nf" "new"; then
> + found=1
> + fi
> + done
> + ;;
> + *)
> + if check_duplicated "$fw"; then
> + mark_if_unused "$fw" "duplicated" || true
> + else
> + mark_if_unused "$fw" "new" || true
> + fi
> + ;;
> + esac
> + done
> +done < <(cat "$FWINFO" | sed 's/firmware: \+//' | sort)
> +
> +declare -A ath_hidden_fw
> +for nf in "${!new_files[@]}"; do
> + [ "${new_files[$nf]}" = "unused" ] || continue
> +
> + case "$nf" in
> + ath*k/*)
> + model=${nf#*/}
> + model=${model%%/*}
> + if [ -z "${ath_hidden_fw[$model]}" ]; then
> + if ! grep -q "/$model/" "$FWINFO"; then
> + ath_hidden_fw["$model"]=1
> + fi
> + fi
> +
> + # ath*k build firmware path at runtime, so set as new if not
> + # previously marked duplicated.
> + if [ -n "${ath_hidden_fw[$model]:-}" ]; then
> + if check_duplicated "$nf"; then
> + mark_if_unused "$nf" "duplicated" || true
> + else
> + mark_if_unused "$nf" "new" || true
> + fi
> + fi
> + ;;
> + esac
> +done
> +
> +for nf in "${!new_files[@]}"; do
> + [ "${new_files[$nf]}" = "new" ] && echo "$nf"
> +done | sort
> +
> +if [ -n "$dump_filtered" ]; then
> + for nf in "${!new_files[@]}"; do
> + [ "${new_files[$nf]}" = "new" ] || echo "# $nf: ${new_files[$nf]}"
> + done | sort
> +fi
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20230120/c7df1ad5/attachment.sig>
More information about the kernel-team
mailing list