[Bug 1900796] Re: plymouth in proposed cause F-5.8 unable to boot on moonshot ARM64 "Wrong Ramdisk Image Format"

Tue Dec 8 20:33:27 UTC 2020

I think this is a firmware issue. Normally you should see something
like:

288 bytes read in 34 ms (7.8 KiB/s)
## Executing script at 4004000000
10741547 bytes read in 323 ms (31.7 MiB/s)   <- This is loading the kernel
64692535 bytes read in 1672 ms (36.9 MiB/s)    <- This is loading the initramfs
## Booting kernel from Legacy Image at 4002000000 ...
   Image Name:   kernel 5.4.0-56-generic
   Created:      2020-12-08   1:43:18 UTC
   Image Type:   ARM Linux Kernel Image (gzip compressed)
   Data Size:    10741483 Bytes = 10.2 MiB
   Load Address: 00080000
   Entry Point:  00080000
   Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 4005000000 ...
   Image Name:   ramdisk 5.4.0-56-generic
   Created:      2020-12-08   1:43:18 UTC
   Image Type:   ARM Linux RAMDisk Image (uncompressed)
   Data Size:    64692471 Bytes = 61.7 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 4003000000
   Booting using the fdt blob at 0x0000004003000000
   Uncompressing Kernel Image ... OK
   Loading Ramdisk to 4fec24d000, end 4feffff0f7 ... OK
   Loading Device Tree to 0000004000078000, end 000000400007fa67 ... OK

But in your output - where we'd expect to see initramfs load size/rate,
we instead see "invalid extent block". And then later:

Wrong Ramdisk Image Format
Ramdisk image is corrupt or invalid

This could be a u-boot bug, possibly fixed by this patch:
https://lists.denx.de/pipermail/u-boot/2014-January/170802.html

I was able to reproduce this by installing bionic, upgrading to the
5.4-based HWE kernel, rebooting, then upgrading my HWE kernel from
bionic-proposed. Possibly the interesting bit here is that I was running
a 5.4-based ext4 when generating the new uInitrd. I was able to get it
working by defragmenting the file:

$ sudo e4defrag -c /boot/uInitrd
e4defrag 1.44.1 (24-Mar-2018)
<File>                                         now/best       size/ext
/boot/uInitrd                                   15/1           2469 KB

 Total/best extents				15/1
 Average size per extent			2469 KB
 Fragmentation score				1
 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
 This file (/boot/uInitrd) does not need defragmentation.
 Done.
$ sudo e4defrag /boot/uInitrd
e4defrag 1.44.1 (24-Mar-2018)
ext4 defragmentation for /boot/uInitrd
[1/1]/boot/uInitrd:	100%	[ OK ]
 Success:			[1/1]

Since that is ext4-specific, a workaround might be to use an ext2/ext3
/boot. There is newer firmware available for m400s that we could try,
but "An active warranty or support agreement covering Proliant servers
must be linked to your HPE Support Center profile to access this BIOS.",
so I can't access it :(

** Also affects: flash-kernel (Ubuntu)
   Importance: Undecided
       Status: New

** Changed in: plymouth (Ubuntu)
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to plymouth in Ubuntu.
https://bugs.launchpad.net/bugs/1900796

Title:
  plymouth in proposed cause F-5.8 unable to boot on moonshot ARM64
  "Wrong Ramdisk Image Format"

Status in ubuntu-kernel-tests:
  New
Status in flash-kernel package in Ubuntu:
  New
Status in plymouth package in Ubuntu:
  Invalid

Bug description:
  Not sure if we still care, it's the old moonshot ARM64 node.

  Issue found on ARM64 node ms10-35-mcdivittb0-kernel with F-5.8.
  Install the plymouth package from proposed will cause system failed to boot, with error message:
    Wrong Ramdisk Image Format
    Ramdisk image is corrupt or invalid

  It's 100% reproducible.

  Steps to reproduce this:
  1. Deploy Focal on this node (Architecture set to arm64/xgene-uboot, it can't be deployed if set to arm64/generic, PXE boot failed with TFTP error: 'File not found' (1))
  2. Enable proposed pocket.
  3. Install the linux-generic-hwe-20.04-edge and reboot
  4. It's now running with 5.8.0-25-generic on Focal
  5. Install plymouth from proposed and reboot

  In step 5 it will generate the new boot image.

  $ sudo apt install plymouth
  Reading package lists... Done
  Building dependency tree
  Reading state information... Done
  The following additional packages will be installed:
    plymouth-theme-ubuntu-text
  Suggested packages:
    desktop-base plymouth-themes
  The following packages will be upgraded:
    plymouth plymouth-theme-ubuntu-text
  2 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
  Need to get 121 kB of archives.
  After this operation, 0 B of additional disk space will be used.
  Do you want to continue? [Y/n]
  Get:1 http://ports.ubuntu.com/ubuntu-ports focal-proposed/main arm64 plymouth-theme-ubuntu-text arm64 0.9.4git20200323-0ubuntu6.1 [9148 B]
  Get:2 http://ports.ubuntu.com/ubuntu-ports focal-proposed/main arm64 plymouth arm64 0.9.4git20200323-0ubuntu6.1 [112 kB]
  Fetched 121 kB in 0s (326 kB/s)
  (Reading database ... 112559 files and directories currently installed.)
  Preparing to unpack .../plymouth-theme-ubuntu-text_0.9.4git20200323-0ubuntu6.1_arm64.deb ...
  Unpacking plymouth-theme-ubuntu-text (0.9.4git20200323-0ubuntu6.1) over (0.9.4git20200323-0ubuntu6) ...
  Preparing to unpack .../plymouth_0.9.4git20200323-0ubuntu6.1_arm64.deb ...
  Unpacking plymouth (0.9.4git20200323-0ubuntu6.1) over (0.9.4git20200323-0ubuntu6) ...
  Setting up plymouth (0.9.4git20200323-0ubuntu6.1) ...
  update-initramfs: deferring update (trigger activated)
  update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
  update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
  Setting up plymouth-theme-ubuntu-text (0.9.4git20200323-0ubuntu6.1) ...
  update-initramfs: deferring update (trigger activated)
  Processing triggers for man-db (2.9.1-1) ...
  Processing triggers for systemd (245.4-4ubuntu3.2) ...
  Processing triggers for initramfs-tools (0.136ubuntu6.3) ...
  update-initramfs: Generating /boot/initrd.img-5.8.0-25-generic
  flash-kernel: installing version 5.8.0-25-generic
  Generating kernel u-boot image... done.
  Taking backup of uImage.
  Installing new uImage.
  Generating initramfs u-boot image... done.
  Taking backup of uInitrd.
  Installing new uInitrd.
  Generating boot script u-boot image... done.
  Taking backup of boot.scr.
  Installing new boot.scr.

  From the console you will see:
  [  OK  ] Finished Reboot.
  [  OK  ] Reached target Reboot.
  [  290.396974] pci_bus 0000:01: 2-byte config write to 0000:01:00.0 offset 0x4 may corrupt adjacent RW1C bits
  [  290.512964] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x5c may corrupt adjacent RW1C bits
  [  290.629960] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x48 may corrupt adjacent RW1C bits
  [  290.746976] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x4 may corrupt adjacent RW1C bits
  [  290.863165] reboot: Restarting system

  U-Boot 2013.04 (Mar 26 2015 - 11:31:01)

  ProLiant m400 Server Cartridge - U02 (02/26/2015)
  Copyright 2013 - 2015 Hewlett-Packard Development Company, L.P.
  Copyright 2000 - 2012 Wolfgang Denk, DENX Software Engineering, wd at denx.de

  CPU0: APM ARM 64-bit Potenza Rev B0 2400MHz PCP 2400MHz
       32 KB ICACHE, 32 KB DCACHE
       SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
  Boot from SPI-NOR
  Slimpro FW: Ver: 2.3 (build 2015/03/16)
   PMD: 960 mV
   SOC: 950 mV
  I2C:   ready
  DRAM:  ECC 64 GiB @ 1333MHz
  relocation Address is: 0x4ffff27000
  Using default environment

  API sig @ 0x0000004ffdf17170
  In:    serial
  Out:   serial
  Err:   serial
  CPUs:  11111111
  CPLD: 0B
  PCIE3: (RC) X8 GEN-2 link up
    00:00.0     - 19aa:e008 - Bridge device
     01:00.0    - 15b3:1007 - Network controller
  SF: Detected MX25L12805D with page size 64 KiB, total 16 MiB
  SF: 16384 KiB MX25L12805D at 0:0 is now current device

  SF: flash read success (19048 bytes @ 0xe0000)
  .
  SF: flash read success (65568 bytes @ 0xc0000)
  Node Boot Start Time: 2020-10-21T05:29:01
  Node Serial Number: CN7505VJ4B
  Cartridge Chassis Slot ID: 35
  Cartridge Serial Number: CN7505VJ4B
  Chassis Serial Number: USE42207F7
  Chassis Asset Tag:
  Node UUID: 9A026903-E1ED-5E39-9768-ED1FB68A9301
  Product ID: 721717-B21
  Timezone Name: America/New_York
  SCSI:  Target spinup took 0 ms.
  AHCI2 0001.0300 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
  flags: 64bit ncq pm only pmp fbss pio slum part ccc
  scanning bus for devices...
    Device 0: (4:0) Vendor: ATA      Prod.: XR0240GEBLU Rev: HPS4
              Type: Hard Disk
              Capacity: 228936.5 MB = 223.5 GB (468862128 x 512)
  Found 1 device(s).
  Boot: PXE, M.2
  Mellanox ConnectX3 U-Boot driver version 1.1
  Mellanox ConnectX3 Firmware Version 2.32.5330
  Net:   NIC1 [PRIME], NIC2

  Booting PXE
  Requesting DHCP address via NIC1
  BOOTP broadcast 1
  DHCP client bound to address 10.229.65.135
  Retrieving file: pxelinux.cfg/9A026903-E1ED-5E39-9768-ED1FB68A9301
  Using NIC1 device
  TFTP from server 10.229.32.21; our IP address is 10.229.65.135
  Filename 'pxelinux.cfg/9A026903-E1ED-5E39-9768-ED1FB68A9301'.
  Load address: 0x4000800000
  Loading: *
  TFTP error: 'File not found' (1)
  Not retrying...
  Retrieving file: pxelinux.cfg/01-14-58-d0-58-c3-c2
  Using NIC1 device
  TFTP from server 10.229.32.21; our IP address is 10.229.65.135
  Filename 'pxelinux.cfg/01-14-58-d0-58-c3-c2'.
  Load address: 0x4000800000
  Loading: #
    0 Bytes/s
  done
  Bytes transferred = 41 (29 hex)
  Config file found
  1:	local
  PXE: executing localboot
  288 bytes read in 34 ms (7.8 KiB/s)
  ## Executing script at 4004000000
  11349894 bytes read in 312 ms (34.7 MiB/s)
  invalid extent block
  ## Booting kernel from Legacy Image at 4002000000 ...
     Image Name:   kernel 5.8.0-25-generic
     Created:      2020-10-21   5:26:34 UTC
     Image Type:   ARM Linux Kernel Image (gzip compressed)
     Data Size:    11349830 Bytes = 10.8 MiB
     Load Address: 00080000
     Entry Point:  00080000
     Verifying Checksum ... OK
  Wrong Ramdisk Image Format
  Ramdisk image is corrupt or invalid
  Booting M.2

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  Package: plymouth 0.9.4git20200323-0ubuntu6.1
  ProcVersionSignature: Ubuntu 5.8.0-25.26~20.04.1-generic 5.8.14
  Uname: Linux 5.8.0-25-generic aarch64
  ApportVersion: 2.20.11-0ubuntu27.10
  Architecture: arm64
  BootLog: Error: [Errno 2] No such file or directory: '/var/log/boot.log'
  CasperMD5CheckResult: skip
  Date: Wed Oct 21 05:27:35 2020
  Lspci-vt: -[0000:00]---00.0-[01]----00.0  Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  Lsusb-t:

  Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
  ProcCmdLine: console=ttyS0,9600n8r ro
  ProcFB:

  ProcKernelCmdLine: console=ttyS0,9600n8r ro
  SourcePackage: plymouth
  TextPlymouth: /usr/share/plymouth/themes/ubuntu-text/ubuntu-text.plymouth
  UpgradeStatus: No upgrade log present (probably fresh install)
  acpidump:

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1900796/+subscriptions