[Bug 2065180] Re: performance regression in dracut-install 060
Benjamin Drung
2065180 at bugs.launchpad.net
Mon Sep 30 12:54:11 UTC 2024
** Description changed:
[ Impact ]
When compared to Ubuntu 23.10, creating intramfs files with update-
initramfs takes 2 to 5 times more time on ARM devices.
IIUC, dracut-install usage was added to initramfs-tools to speed up the
process. But now its way slower. Even running update-initramfs on jammy,
which doesn't use dracut-install, is way faster then the time taken on
Noble.
first bad commit -
https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f780390b874c870
Updating the initrd on a Raspberry Pi Zero 2W on Ubuntu 24.04 (noble)
with initramfs-tools 0.142ubuntu25.1 takes over six minutes:
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 402.751 s ± 5.592 s [User: 166.316 s, System: 228.909 s]
Range (min … max): 394.380 s … 411.445 s 10 runs
```
[ Test Plan ]
1. Measure `update-initramfs -u` before the update.
2. Log the content of the initrd before the update: `lsinitramfs /boot/initrd.img`
3. update dracut-install / initramfs-tools-core
4. Measure `update-initramfs -u`. It should be faster (the performance improvements on amd64 should be very small and might be within the measurement uncertainty).
5. Check with lsinitramfs that the content of the newly generated initrd hasn't changed.
[ Where problems could occur ]
The code that is responsible for including the kernel modules into the
initrd is touched. Negative consequences could be that some needed
kernel modules will not be included any more (should be covered by the
test case) or that building new initrds will fail.
The initramfs-tools fix changes how manual_add_modules behaves.
`manual_add_modules` does not copy kernel modules, but queues them for
being copied when the newly added function `apply_add_modules` is
called.
I checked all instances of calls to `manual_add_modules` for possible
regressions (see comment #15). Only miniramfs needs to be adjusted to
also call `apply_add_modules`. But this change could break consumers of
the `manual_add_modules` function that are outside of the Ubuntu
archive. I googled for `apply_add_modules` but found no public outside
users.
[ Benchmarks ]
Stock noble on a Raspberry Pi Zero 2W:
```
bdrung at zero2w:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 415.664 s ± 6.015 s [User: 166.728 s, System: 232.523 s]
Range (min … max): 409.139 s … 422.632 s 5 runs
```
noble with dracut-install 060+5-1ubuntu3.1 (with linux 6.8.0-1006.6 on
2024-07-01):
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 248.054 s ± 5.569 s [User: 67.410 s, System: 169.412 s]
Range (min … max): 238.909 s … 257.384 s 10 runs
```
noble with dracut-install 060+5-1ubuntu3.2 (with linux 6.8.0-1007 on
2024-07-11):
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 232.001 s ± 5.678 s [User: 55.456 s, System: 166.510 s]
Range (min … max): 222.120 s … 239.610 s 10 runs
```
noble with dracut-install 060+5-1ubuntu3.2 (with linux 6.8.0-1011.12 and
all updates on 2024-09-19):
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
- Time (mean ± σ): 219.439 s ± 4.869 s [User: 45.626 s, System: 163.172 s]
- Range (min … max): 213.340 s … 227.411 s 10 runs
+ Time (mean ± σ): 219.439 s ± 4.869 s [User: 45.626 s, System: 163.172 s]
+ Range (min … max): 213.340 s … 227.411 s 10 runs
+ ```
+
+ noble with initramfs-tools 0.142ubuntu25.2 (with dracut-install
+ 060+5-1ubuntu3.2, linux 6.8.0-1012.13, and all updates on 2024-09-30):
+
+ ```
+ bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
+ Benchmark 1: update-initramfs -u
+ Time (mean ± σ): 217.160 s ± 4.652 s [User: 45.608 s, System: 159.225 s]
+ Range (min … max): 210.736 s … 225.802 s 10 runs
+ ```
+
+ noble with initramfs-tools 0.142ubuntu25.4 (with dracut-install
+ 060+5-1ubuntu3.2, linux 6.8.0-1012.13, and all updates on 2024-09-30):
+
+ ```
+ bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
+ Benchmark 1: update-initramfs -u
+ Time (mean ± σ): 204.143 s ± 4.441 s [User: 37.845 s, System: 153.927 s]
+ Range (min … max): 195.729 s … 208.805 s 10 runs
```
[ Reduce manual_add_modules calls ]
Besides making the dracut-install calls faster, group the dracut-install
calls. Since the fix in oracular can cause regressions in custom hooks
that rely on the current behavior, the SRU takes a safe approach which
includes following packages (stating how many dracut-install calls are
used):
* cryptsetup: 2 -> 1
* lvm2: 8 -> 1
* thin-provisioning-tools: 3 -> 1
* open-iscsi: 9 -> 1
* cloud-initramfs-tools: 5 -> 1
dracut-install calls on a Raspberry Pi Zero 2W:
| area | before | noble SRU | oracular |
|--------------------------------------|--------|-----------|----------|
| auto_add_modules + apply_add_modules | 8 | 5 | 5 |
| calls by hooks + apply_add_modules | 42 | 20 | 2 |
| hidden_dep_add_modules | 1 | 1 | 1 |
| total | 51 | 26 | 8 |
[ Other Info ]
$ lsb_release -rd
No LSB modules are available.
Description: Ubuntu 24.04 LTS
Release: 24.04
$ apt-cache policy dracut-install
dracut-install:
Installed: 060+5-1ubuntu3
Candidate: 060+5-1ubuntu3
Version table:
*** 060+5-1ubuntu3 500
500 http://ports.ubuntu.com/ubuntu-ports noble/main arm64 Packages
100 /var/lib/dpkg/status
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to cryptsetup in Ubuntu.
Matching subscriptions: dracut
https://bugs.launchpad.net/bugs/2065180
Title:
performance regression in dracut-install 060
Status in Dracut:
New
Status in cloud-initramfs-tools package in Ubuntu:
Fix Released
Status in cryptsetup package in Ubuntu:
Fix Released
Status in dracut package in Ubuntu:
Fix Released
Status in initramfs-tools package in Ubuntu:
Fix Released
Status in lvm2 package in Ubuntu:
Fix Released
Status in miniramfs package in Ubuntu:
Fix Released
Status in open-iscsi package in Ubuntu:
Invalid
Status in thin-provisioning-tools package in Ubuntu:
Fix Released
Status in cloud-initramfs-tools source package in Noble:
Fix Released
Status in cryptsetup source package in Noble:
Fix Released
Status in dracut source package in Noble:
Fix Released
Status in initramfs-tools source package in Noble:
Fix Released
Status in lvm2 source package in Noble:
Fix Released
Status in miniramfs source package in Noble:
New
Status in open-iscsi source package in Noble:
Fix Released
Status in thin-provisioning-tools source package in Noble:
Fix Released
Bug description:
[ Impact ]
When compared to Ubuntu 23.10, creating intramfs files with update-
initramfs takes 2 to 5 times more time on ARM devices.
IIUC, dracut-install usage was added to initramfs-tools to speed up
the process. But now its way slower. Even running update-initramfs on
jammy, which doesn't use dracut-install, is way faster then the time
taken on Noble.
first bad commit -
https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f780390b874c870
Updating the initrd on a Raspberry Pi Zero 2W on Ubuntu 24.04 (noble)
with initramfs-tools 0.142ubuntu25.1 takes over six minutes:
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 402.751 s ± 5.592 s [User: 166.316 s, System: 228.909 s]
Range (min … max): 394.380 s … 411.445 s 10 runs
```
[ Test Plan ]
1. Measure `update-initramfs -u` before the update.
2. Log the content of the initrd before the update: `lsinitramfs /boot/initrd.img`
3. update dracut-install / initramfs-tools-core
4. Measure `update-initramfs -u`. It should be faster (the performance improvements on amd64 should be very small and might be within the measurement uncertainty).
5. Check with lsinitramfs that the content of the newly generated initrd hasn't changed.
[ Where problems could occur ]
The code that is responsible for including the kernel modules into the
initrd is touched. Negative consequences could be that some needed
kernel modules will not be included any more (should be covered by the
test case) or that building new initrds will fail.
The initramfs-tools fix changes how manual_add_modules behaves.
`manual_add_modules` does not copy kernel modules, but queues them for
being copied when the newly added function `apply_add_modules` is
called.
I checked all instances of calls to `manual_add_modules` for possible
regressions (see comment #15). Only miniramfs needs to be adjusted to
also call `apply_add_modules`. But this change could break consumers
of the `manual_add_modules` function that are outside of the Ubuntu
archive. I googled for `apply_add_modules` but found no public outside
users.
[ Benchmarks ]
Stock noble on a Raspberry Pi Zero 2W:
```
bdrung at zero2w:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 415.664 s ± 6.015 s [User: 166.728 s, System: 232.523 s]
Range (min … max): 409.139 s … 422.632 s 5 runs
```
noble with dracut-install 060+5-1ubuntu3.1 (with linux 6.8.0-1006.6 on
2024-07-01):
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 248.054 s ± 5.569 s [User: 67.410 s, System: 169.412 s]
Range (min … max): 238.909 s … 257.384 s 10 runs
```
noble with dracut-install 060+5-1ubuntu3.2 (with linux 6.8.0-1007 on
2024-07-11):
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 232.001 s ± 5.678 s [User: 55.456 s, System: 166.510 s]
Range (min … max): 222.120 s … 239.610 s 10 runs
```
noble with dracut-install 060+5-1ubuntu3.2 (with linux 6.8.0-1011.12
and all updates on 2024-09-19):
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Time (mean ± σ): 219.439 s ± 4.869 s [User: 45.626 s, System: 163.172 s]
Range (min … max): 213.340 s … 227.411 s 10 runs
```
noble with initramfs-tools 0.142ubuntu25.2 (with dracut-install
060+5-1ubuntu3.2, linux 6.8.0-1012.13, and all updates on 2024-09-30):
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 217.160 s ± 4.652 s [User: 45.608 s, System: 159.225 s]
Range (min … max): 210.736 s … 225.802 s 10 runs
```
noble with initramfs-tools 0.142ubuntu25.4 (with dracut-install
060+5-1ubuntu3.2, linux 6.8.0-1012.13, and all updates on 2024-09-30):
```
bdrung at zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 204.143 s ± 4.441 s [User: 37.845 s, System: 153.927 s]
Range (min … max): 195.729 s … 208.805 s 10 runs
```
[ Reduce manual_add_modules calls ]
Besides making the dracut-install calls faster, group the dracut-
install calls. Since the fix in oracular can cause regressions in
custom hooks that rely on the current behavior, the SRU takes a safe
approach which includes following packages (stating how many dracut-
install calls are used):
* cryptsetup: 2 -> 1
* lvm2: 8 -> 1
* thin-provisioning-tools: 3 -> 1
* open-iscsi: 9 -> 1
* cloud-initramfs-tools: 5 -> 1
dracut-install calls on a Raspberry Pi Zero 2W:
| area | before | noble SRU | oracular |
|--------------------------------------|--------|-----------|----------|
| auto_add_modules + apply_add_modules | 8 | 5 | 5 |
| calls by hooks + apply_add_modules | 42 | 20 | 2 |
| hidden_dep_add_modules | 1 | 1 | 1 |
| total | 51 | 26 | 8 |
[ Other Info ]
$ lsb_release -rd
No LSB modules are available.
Description: Ubuntu 24.04 LTS
Release: 24.04
$ apt-cache policy dracut-install
dracut-install:
Installed: 060+5-1ubuntu3
Candidate: 060+5-1ubuntu3
Version table:
*** 060+5-1ubuntu3 500
500 http://ports.ubuntu.com/ubuntu-ports noble/main arm64 Packages
100 /var/lib/dpkg/status
To manage notifications about this bug go to:
https://bugs.launchpad.net/dracut/+bug/2065180/+subscriptions
More information about the foundations-bugs
mailing list