[Bug 2085157] Re: mkinitramfs fails with copy_file binary '/libgcc_s.so.[1-9]' not found

Frank Heimes 2085157 at bugs.launchpad.net
Mon Nov 18 13:41:23 UTC 2024 

It was a bit tricky to test this, but did it like this:

I installed a 24.10 LPAR with a broken network configuration to do an offline install (and prevent any updates right during installation).
[For test purposes it would be great to be able to perform offline installs in a more straight-forward and more convenient way ...]
Logged into the system and fixed the network.
Updated the archive index (apt update), installed the initrmfs-tools* packages first (just to be sure) and then upgraded the rest, that incl. the kernel and the system updated the kernel, the update-initramfs worked (even if it still moans about '/usr/lib/udev/dmsetup_env', but that was not part of the patch):

...
Setting up linux-image-6.11.0-9-generic (6.11.0-9.9) ...
I: /boot/vmlinuz.old is now a symlink to vmlinuz-6.11.0-8-generic
I: /boot/initrd.img.old is now a symlink to initrd.img-6.11.0-8-generic
I: /boot/vmlinuz is now a symlink to vmlinuz-6.11.0-9-generic
I: /boot/initrd.img is now a symlink to initrd.img-6.11.0-9-generic
Setting up linux-tools-6.11.0-9 (6.11.0-9.9) ...
Setting up systemd-resolved (256.5-2ubuntu3.1) ...
Setting up linux-tools-6.11.0-9-generic (6.11.0-9.9) ...
Setting up linux-image-generic (6.11.0-9.9) ...
Setting up systemd-sysv (256.5-2ubuntu3.1) ...
Setting up libnss-systemd:s390x (256.5-2ubuntu3.1) ...
Setting up linux-headers-generic (6.11.0-9.9) ...
Setting up libpam-systemd:s390x (256.5-2ubuntu3.1) ...
Setting up linux-generic (6.11.0-9.9) ...
Processing triggers for man-db (2.12.1-3) ...
Processing triggers for dbus (1.14.10-4ubuntu5) ...
Processing triggers for shared-mime-info (2.4-5) ...
Processing triggers for initramfs-tools (0.142ubuntu34.1) ...
update-initramfs: Generating /boot/initrd.img-6.11.0-8-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
Not invoking zipl: initrd doesn't exist yet
Processing triggers for libc-bin (2.40-1ubuntu3) ...
Processing triggers for linux-image-6.11.0-8-generic (6.11.0-8.8+1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.11.0-8-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
Not invoking zipl: initrd doesn't exist yet
/etc/kernel/postinst.d/kdump-tools:
kdump-tools: Generating /var/lib/kdump/initrd.img-6.11.0-8-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
/etc/kernel/postinst.d/zz-zipl:
Not invoking zipl: initrd doesn't exist yet
Processing triggers for linux-image-6.11.0-9-generic (6.11.0-9.9) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.11.0-9-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
Using config file '/etc/zipl.conf'
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Preparing boot device for LD-IPL: dm-0.
Done.
/etc/kernel/postinst.d/kdump-tools:
kdump-tools: Generating /var/lib/kdump/initrd.img-6.11.0-9-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
/etc/kernel/postinst.d/zz-zipl:
Using config file '/etc/zipl.conf'
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Preparing boot device for LD-IPL: dm-0.
Done.
Scanning processes...                                                           
Scanning candidates...                                                          
Scanning processor microcode...                                                 
Scanning linux images...                                                        

Pending kernel upgrade!
Running kernel version:
  6.11.0-8-generic
Diagnostics:
  The currently running kernel version is not the expected kernel version
6.11.0-9-generic.

Restarting the system to load the new kernel will not be handled automatically,
so you should consider rebooting.

Failed to check for processor microcode upgrades.

Restarting services...
 systemctl restart multipathd.service packagekit.service polkit.service rsyslog.service ssh.service udisks2.service

Service restarts being deferred:
 systemctl restart ModemManager.service
 /etc/needrestart/restart.d/dbus.service
 systemctl restart getty at tty1.service
 systemctl restart serial-getty at sclp_line0.service
 systemctl restart serial-getty at ttysclp0.service
 systemctl restart systemd-logind.service
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

User sessions running outdated binaries:
 ubuntu @ session #5: apt[11929], sshd[2676]
 ubuntu @ user manager service: systemd[2205]

No VM guests are running outdated hypervisor (qemu) binaries on this
host.


But I then unfortunately found that the system "hangs" (repeatedly) for about 3min at line:
"Begin: Waiting for udev to settle (multipath) ... done."
It looks like a new udev package:
"Setting up udev (256.5-2ubuntu3.1) ..."
was also installed during my (not only) kernel upgrade (since I upgraded simply everything from -proposed).

Since this is a different issue, it needs to be addressed differently
...

I am calling this bug now as 'verification-done-oracular'.

** Tags removed: verification-needed verification-needed-oracular
** Tags added: verification-done verification-done-oracular

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to initramfs-tools in Ubuntu.
https://bugs.launchpad.net/bugs/2085157

Title:
  mkinitramfs fails with copy_file binary '/libgcc_s.so.[1-9]' not found

Status in Ubuntu on IBM z Systems:
  Fix Committed
Status in initramfs-tools package in Ubuntu:
  Fix Released
Status in initramfs-tools source package in Oracular:
  Fix Committed

Bug description:
  SRU Justification:

  [ Impact ]

   * update-initramfs fails wile running on a multipath system
     (in case multipath-tools-boot package is installed)
     due to issues in /usr/share/initramfs-tools/hook-functions.

   * copy_libgcc fails due to lack of folder handling

   * copy_exec /usr/lib/udev/dmsetup_env fails because
     because there is no /usr/lib/udev/dmsetup_env anymore
     and the line is obsolete

   * The failure of update-initramfs is not obvious and hardly noticeable.
     This leads to the fact that an installation on a multipath environment,
     where the system is online and able to fetch an updated kernel,
     that would trigger an update-initramfs after it's installation,
     would render the system un-bootable - post-install reboot will fail.

  [ Test Plan ]

   * A system with a multipath disk setup is needed,
     that has network access during installation.
     (means no offline install).

   * Perform a 24.10 default installation, using the GA image.

   * Meanwhile at least one newer kernel is available in oracular-updates
     which will be automatically installed by the installer
     (during an online install).

   * Complete the (default) installation and hit "Reboot now" at the
  end.

   * The current initramfs-tools (0.142ubuntu34) will prevent the
     system from doing a successful post-install reboot.

   * An initramfs-tools package with the referenced fix included
     will allow the system to successfully complete the post-install reboot.

  [ Where problems could occur ]

   * Issues could occure in case one relies on the existing 'case' expression:
     copy_libgcc "${x%/*}" || return

   * Or if someone already has an alternate workaround or fix in place
     (see comments for other ways to fix)
     that interferce with the officially suggested fix.

   * The issue affects only system with a multipath disk setup
     (having multipath-tools-boot installed),
     which are the minority of cases,
     but the more enterprise environments.

  [ Other Info ]

   * This got introduced with the recent upgrade of the
     multipath-tools-boot package in oracular.
     Hence only affected plucky and oracular,
     but no Ubuntu release older than oracular.

   * Hence patch need to be applied to the currently development
     release pluck, as well as SRUed to oracular.
  __________

  I recently did an 24.10 install and was surprised that the system
  hangs during the post-install reboot.

  Investigation showed that this happens if the installer applies
  updates, which is done by default, if there is proper network
  connectivity and archive access.

  In case of an install where I disabled the network, hence an offline
  install was done, the installation incl. post-install reboot was
  successful.

  So I took such a successfully installed system (where no updates were applied) and noticed that currently the available updates are:
  $ apt list --upgradable
  iproute2/oracular-updates 6.10.0-2ubuntu1 s390x [upgradable from: 6.10.0-2]
  linux-generic/oracular-updates 6.11.0-9.9 s390x [upgradable from: 6.11.0-8.8]
  linux-headers-generic/oracular-updates 6.11.0-9.9 s390x [upgradable from: 6.11.0-8.8]
  linux-image-generic/oracular-updates 6.11.0-9.9 s390x [upgradable from: 6.11.0-8.8]
  linux-libc-dev/oracular-updates 6.11.0-9.9 s390x [upgradable from: 6.11.0-8.8]
  linux-tools-common/oracular-updates 6.11.0-9.9 all [upgradable from: 6.11.0-8.8]

  So seemed to be kernel related, hence tried to update the kernel manually:
  $ sudo apt install linux-image-generic/oracular-updates
  Selected version '6.11.0-9.9' (Ubuntu:24.10/oracular-updates [s390x]) for 'linux-image-generic'
  Upgrading:
    linux-generic          linux-image-generic  linux-tools-common
    linux-headers-generic  linux-libc-dev

  Installing dependencies:
    linux-headers-6.11.0-9          linux-modules-extra-6.11.0-9-generic
    linux-headers-6.11.0-9-generic  linux-tools-6.11.0-9
    linux-image-6.11.0-9-generic    linux-tools-6.11.0-9-generic
    linux-modules-6.11.0-9-generic

  Suggested packages:
    fdutils  linux-tools

  Summary:
    Upgrading: 5, Installing: 7, Removing: 0, Not Upgrading: 1
    Download size: 65.3 MB
    Space needed: 165 MB / 21.2 GB available
    └─ in /boot:  43.1 MB / 1,876 MB available

  Continue? [Y/n] Y
  ...
  Setting up linux-generic (6.11.0-9.9) ...
  Progress: [ 96%] [███████████████████████████████████████████████████████▋  ]
  Progress: [ 96%] [███████████████████████████████████████████████████████▋  ]
  Progress: [ 96%] [███████████████████████████████████████████████████████▋  ]
  update-initramfs: Generating /boot/initrd.img-6.11.0-9-generic
  mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found█████▋  ]
  mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found
  Using config file '/etc/zipl.conf'
  Run /lib/s390-tools/zipl_helper.device-mapper 252:1
  Building bootmap in '/boot'
  Adding IPL section 'ubuntu' (default)
  Run /lib/s390-tools/zipl_helper.device-mapper 252:1
  Run /lib/s390-tools/zipl_helper.device-mapper 252:1
  Preparing boot device for LD-IPL: dm-0.
  Done.
  /etc/kernel/postinst.d/kdump-tools:
  kdump-tools: Generating /var/lib/kdump/initrd.img-6.11.0-9-generic
  mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
  mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found
  /etc/kernel/postinst.d/zz-zipl:
  Using config file '/etc/zipl.conf'
  Run /lib/s390-tools/zipl_helper.device-mapper 252:1
  Building bootmap in '/boot'
  Adding IPL section 'ubuntu' (default)
  Run /lib/s390-tools/zipl_helper.device-mapper 252:1
  Run /lib/s390-tools/zipl_helper.device-mapper 252:1
  Preparing boot device for LD-IPL: dm-0.
  Done.
  Scanning processes...
  Scanning processor microcode...
  Scanning linux images...

  Pending kernel upgrade!
  Running kernel version:
    6.11.0-8-generic
  Diagnostics:
    The currently running kernel version is not the expected kernel version
  6.11.0-9-generic.

  Restarting the system to load the new kernel will not be handled automatically,
  so you should consider rebooting.

  Failed to check for processor microcode upgrades.

  No services need to be restarted.

  No containers need to be restarted.

  No user sessions are running outdated binaries.

  No VM guests are running outdated hypervisor (qemu) binaries on this
  host.

  I noticed these two lines, that have been spit out by update-initramfs:
  update-initramfs: Generating /boot/initrd.img-6.11.0-9-generic
  mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found█████▋  ]
  mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found

  Hence I tried to ran update-initramfs on the release kernel 6.11.0-8-8
  and it happened there as well:

  $ sudo update-initramfs -k all -u
  update-initramfs: Generating /boot/initrd.img-6.11.0-8-generic
  mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
  mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found
  Using config file '/etc/zipl.conf'
  Run /lib/s390-tools/zipl_helper.device-mapper 252:1
  Building bootmap in '/boot'
  Adding IPL section 'ubuntu' (default)
  Run /lib/s390-tools/zipl_helper.device-mapper 252:1
  Run /lib/s390-tools/zipl_helper.device-mapper 252:1
  Preparing boot device for LD-IPL: dm-0.
  Done.

  So it's a initramfs-tools (mkinitramfs) issue rather than a kernel issue,
  somewhere in copy_libgcc() (or before).

  Seems to be a small issue, but is causing a big impact, since it leads
  to a broken default installation.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-z-systems/+bug/2085157/+subscriptions

More information about the foundations-bugs mailing list