[Bug 1934192] Re: regression between 2.04-1ubuntu44 and 2.04-1ubuntu44.2 breaks netboot on arm64

dann frazier 1934192 at bugs.launchpad.net
Thu Jul 1 14:40:51 UTC 2021 

@juliank pointed out that 2.04-1ubuntu44 was built w/ the hirsute
toolchain, 2.04-1ubuntu44.2 with focal's toolchain. I'll run some
experiments to see if the issue follows the toolchain.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2-signed in Ubuntu.
https://bugs.launchpad.net/bugs/1934192

Title:
  regression between 2.04-1ubuntu44 and 2.04-1ubuntu44.2 breaks netboot
  on arm64

Status in grub2-signed package in Ubuntu:
  New
Status in grub2-unsigned package in Ubuntu:
  New

Bug description:
  We noticed today that 2 arm64 nodes in our MAAS server are failing to Commission/Deploy. Both happen to be Ampere Altra systems. We have not (yet) noticed a problem with other systems - a Cavium ThunderX system I tested seems unaffected. The symptom is that they successfully PXE boot GRUB, but then GRUB just drops to a prompt[*]. I confirmed that the grubaa64.efi binary is the 2.04-1ubuntu44.2 grubnetaa64.efi.signed from focal-proposed:
   sha256sum: 53916e7cac279a0dc61163dc3edd4580a8e23f63162ced1decff87386fd2cda1

  These systems used to work fine. To try and identify what changed, I reverted MAAS' grubaa64.efi binary to the 2.04-1ubuntu44 grubnetaa64.efi.signed binary:
   sha256sum: e098dd7e372b47de01b27dfc39225d3c62cde2239c984e578421d94c501244ae

  This worked. I then restored the 2.04-1ubuntu44.2 binary, and found it
  once again failed. Since the failing GRUB binary does drop me at a
  shell, I used it to enable debug and retry boot[**]. This seems to
  suggest TFTP requests are failing. Server side, the log shows no such
  requests.

  Note that I've seen something like this before, where GRUB TFTP seems
  to be broken for only Ampere systems. See bug 1900773. It's unlikely
  the same problem though. That bug was only an issue w/ large files,
  whereas this is impacting just fetching the initial grub.cfg.

  [*]
  >>Checking Media Presence......
  >>Media Present......
  >>Start PXE over IPv4 on MAC: 18-C0-4D-82-E0-C8. Press ESC key to abort PXE boot.
    Station IP address is 10.229.193.84

    Server IP address is 10.229.32.21
    NBP filename is grubaa64.efi
    NBP filesize is 1611648 Bytes

  >>Checking Media Presence....GNU GRUB  version 2.04
  >>Media Present......
     Minimal BASH-like line editing is supported. For the first word, TAB
     lists possible command completions. Anywhere else TAB lists possible
     device or file completions.

  grub>

  [**]
  grub> cat (memdisk)/grub.cfg
  if [ -e $prefix/arm64-efi/grub.cfg ]; then
          source $prefix/arm64-efi/grub.cfg
  elif [ -e $prefix/grub.cfg-arm64 ]; then
          source $prefix/grub.cfg-default-arm64
  else
          source $prefix/grub.cfg
  fi

  grub> echo $prefix
  (tftp,10.229.32.21)/grub
  grub> set debug=all
  script/script.c:65: free 0x80bf7ec7120
  script/script.c:65: free 0x80bf7ec7160
  script/script.c:65: free 0x80bf7ec71a0
  script/script.c:65: free 0x80bf7ec6cc0
  script/script.c:65: free 0x80bf7ec6d20
  script/script.c:65: free 0x80bf7ec6d60
  script/script.c:65: free 0x80bf7ec6dc0
  script/script.c:65: free 0x80bf7ec6e20
  script/script.c:65: free 0x80bf7ec6f00
  script/script.c:65: free 0x80bf7ec6fe0
  script/script.c:65: free 0x80bf7ec7040
  script/script.c:65: free 0x80bf7ec7080
  grub> source (memdisk)/grub.cfg
  script/lexer.c:336: token 288 text [source]
  script/script.c:50: malloc 0x80bf7ec6ee0
  script/script.c:50: malloc 0x80bf7ec6ea0
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec6e40
  script/lexer.c:336: token 289 text [(memdisk)/grub.cfg]
  script/script.c:50: malloc 0x80bf7ec6c60
  script/script.c:50: malloc 0x80bf7ec6c00
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec6ba0
  script/lexer.c:336: token 259 text [
  ]
  script/script.c:50: malloc 0x80bf7ec6b40
  script/script.c:50: malloc 0x80bf7ec6b00
  script/script.c:198: cmdline
  script/script.c:50: malloc 0x80bf7ec6aa0
  script/lexer.c:336: token 0 text []
  script/script.c:50: malloc 0x80bf7ec7000
  script/script.c:50: malloc 0x80bf7ec6fc0
  script/script.c:294: append command
  script/script.c:50: malloc 0x80bf7ec6f80
  commands/wildcard.c:535: no expansion needed
  commands/wildcard.c:594: paths[0] = `(memdisk)/grub.cfg'
  kern/disk.c:196: Opening `memdisk'...
  kern/fs.c:56: Detecting zfs...
  kern/disk.c:421: Read out of range: sector 0x100 (attempt to read or write
  outside of disk `memdisk').
  kern/disk.c:421: Read out of range: sector 0x300 (attempt to read or write
  outside of disk `memdisk').
  kern/disk.c:421: Read out of range: sector 0xfffffffffffffd00 (attempt to read
  or write outside of disk `memdisk').
  kern/disk.c:421: Read out of range: sector 0xffffffffffffff00 (attempt to read
  or write outside of disk `memdisk').
  kern/fs.c:78: zfs detection failed.
  kern/fs.c:56: Detecting xfs...
  fs/xfs.c:909: Reading sb
  fs/xfs.c:258: Validating superblock
  kern/fs.c:78: xfs detection failed.
  kern/fs.c:56: Detecting squash4...
  kern/fs.c:78: squash4 detection failed.
  kern/fs.c:56: Detecting ntfs...
  kern/fs.c:78: ntfs detection failed.
  kern/fs.c:56: Detecting iso9660...
  kern/fs.c:78: iso9660 detection failed.
  kern/fs.c:56: Detecting hfsplus...
  kern/fs.c:78: hfsplus detection failed.
  kern/fs.c:56: Detecting fat...
  commands/verifiers.c:88: file: (memdisk)/grub.cfg type: 51
  script/lexer.c:336: token 281 text [if]
  script/script.c:50: malloc 0x80bf7ec1e40
  script/script.c:50: malloc 0x80bf7ec1e00
  script/lexer.c:336: token 289 text [[]
  script/script.c:50: malloc 0x80bf7ec1c20
  script/script.c:50: malloc 0x80bf7ec1be0
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec1b80
  script/lexer.c:336: token 289 text [-e]
  script/script.c:50: malloc 0x80bf7ec1b20
  script/script.c:50: malloc 0x80bf7ec1ae0
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec1a80
  script/lexer.c:336: token 289 text []
  script/script.c:50: malloc 0x80bf7ec1980
  script/script.c:50: malloc 0x80bf7ec1940
  script/lexer.c:336: token 289 text [prefix]
  script/script.c:50: malloc 0x80bf7ec18e0
  script/script.c:50: malloc 0x80bf7ec18a0
  script/lexer.c:336: token 289 text [/arm64-efi/grub.cfg]
  script/script.c:50: malloc 0x80bf7ec1840
  script/script.c:50: malloc 0x80bf7ec17e0
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec1780
  script/lexer.c:336: token 289 text []]
  script/script.c:50: malloc 0x80bf7ec1720
  script/script.c:50: malloc 0x80bf7ec16e0
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec1680
  script/lexer.c:336: token 265 text [;]
  script/script.c:50: malloc 0x80bf7ec1620
  script/script.c:50: malloc 0x80bf7ec15e0
  script/script.c:198: cmdline
  script/script.c:50: malloc 0x80bf7ec1580
  script/script.c:294: append command
  script/script.c:50: malloc 0x80bf7ec1540
  script/lexer.c:336: token 284 text [then]
  script/script.c:50: malloc 0x80bf7ec14e0
  script/script.c:50: malloc 0x80bf7ec14a0
  script/lexer.c:336: token 259 text [
  ]
  script/script.c:50: malloc 0x80bf7ec1440
  script/script.c:50: malloc 0x80bf7ec1400
  script/lexer.c:336: token 288 text [source]
  script/script.c:50: malloc 0x80bf7ec1280
  script/script.c:50: malloc 0x80bf7ec1240
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec11e0
  script/lexer.c:336: token 289 text []
  script/script.c:50: malloc 0x80bf7ec10e0
  script/script.c:50: malloc 0x80bf7ec10a0
  script/lexer.c:336: token 289 text [prefix]
  script/script.c:50: malloc 0x80bf7ec1040
  script/script.c:50: malloc 0x80bf7ec1000
  script/lexer.c:336: token 289 text [/arm64-efi/grub.cfg]
  script/script.c:50: malloc 0x80bf7ec0fa0
  script/script.c:50: malloc 0x80bf7ec0f40
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec0ee0
  script/lexer.c:336: token 259 text [
  ]
  script/script.c:50: malloc 0x80bf7ec0e80
  script/script.c:50: malloc 0x80bf7ec0e40
  script/script.c:198: cmdline
  script/script.c:50: malloc 0x80bf7ec0de0
  script/script.c:294: append command
  script/script.c:50: malloc 0x80bf7ec0da0
  script/lexer.c:336: token 276 text [elif]
  script/script.c:50: malloc 0x80bf7ec0c20
  script/script.c:50: malloc 0x80bf7ec0be0
  script/lexer.c:336: token 289 text [[]
  script/script.c:50: malloc 0x80bf7ec0a40
  script/script.c:50: malloc 0x80bf7ec0a00
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec09a0
  script/lexer.c:336: token 289 text [-e]
  script/script.c:50: malloc 0x80bf7ec0940
  script/script.c:50: malloc 0x80bf7ec0900
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec08a0
  script/lexer.c:336: token 289 text []
  script/script.c:50: malloc 0x80bf7ec07a0
  script/script.c:50: malloc 0x80bf7ec0760
  script/lexer.c:336: token 289 text [prefix]
  script/script.c:50: malloc 0x80bf7ec0700
  script/script.c:50: malloc 0x80bf7ec06c0
  script/lexer.c:336: token 289 text [/grub.cfg-arm64]
  script/script.c:50: malloc 0x80bf7ec0660
  script/script.c:50: malloc 0x80bf7ec0620
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec05c0
  script/lexer.c:336: token 289 text []]
  script/script.c:50: malloc 0x80bf7ec0560
  script/script.c:50: malloc 0x80bf7ec0520
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ec04c0
  script/lexer.c:336: token 265 text [;]
  script/script.c:50: malloc 0x80bf7ec0460
  script/script.c:50: malloc 0x80bf7ec0420
  script/script.c:198: cmdline
  script/script.c:50: malloc 0x80bf7ec03c0
  script/script.c:294: append command
  script/script.c:50: malloc 0x80bf7ec0380
  script/lexer.c:336: token 284 text [then]
  script/script.c:50: malloc 0x80bf7ec0320
  script/script.c:50: malloc 0x80bf7ec02e0
  script/lexer.c:336: token 259 text [
  ]
  script/script.c:50: malloc 0x80bf7ec0280
  script/script.c:50: malloc 0x80bf7ec0240
  script/lexer.c:336: token 288 text [source]
  script/script.c:50: malloc 0x80bf7ec00c0
  script/script.c:50: malloc 0x80bf7ebff80
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ebff20
  script/lexer.c:336: token 289 text []
  script/script.c:50: malloc 0x80bf7ebfe20
  script/script.c:50: malloc 0x80bf7ebfde0
  script/lexer.c:336: token 289 text [prefix]
  script/script.c:50: malloc 0x80bf7ebfd80
  script/script.c:50: malloc 0x80bf7ebfd40
  script/lexer.c:336: token 289 text [/grub.cfg-default-arm64]
  script/script.c:50: malloc 0x80bf7ebfce0
  script/script.c:50: malloc 0x80bf7ebfc80
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ebfc20
  script/lexer.c:336: token 259 text [
  ]
  script/script.c:50: malloc 0x80bf7ebfbc0
  script/script.c:50: malloc 0x80bf7ebfb80
  script/script.c:198: cmdline
  script/script.c:50: malloc 0x80bf7ebfb20
  script/script.c:294: append command
  script/script.c:50: malloc 0x80bf7ebf9e0
  script/lexer.c:336: token 277 text [else]
  script/script.c:50: malloc 0x80bf7ebf880
  script/script.c:50: malloc 0x80bf7ebf840
  script/lexer.c:336: token 259 text [
  ]
  script/script.c:50: malloc 0x80bf7ebf7e0
  script/script.c:50: malloc 0x80bf7ebf7a0
  script/lexer.c:336: token 288 text [source]
  script/script.c:50: malloc 0x80bf7ebf640
  script/script.c:50: malloc 0x80bf7ebf600
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ebf5a0
  script/lexer.c:336: token 289 text []
  script/script.c:50: malloc 0x80bf7ebf4a0
  script/script.c:50: malloc 0x80bf7ebf460
  script/lexer.c:336: token 289 text [prefix]
  script/script.c:50: malloc 0x80bf7ebf400
  script/script.c:50: malloc 0x80bf7ebf3c0
  script/lexer.c:336: token 289 text [/grub.cfg]
  script/script.c:50: malloc 0x80bf7ebf540
  script/script.c:50: malloc 0x80bf7ebf380
  script/script.c:163: arglist
  script/script.c:50: malloc 0x80bf7ebf320
  script/lexer.c:336: token 259 text [
  ]
  script/script.c:50: malloc 0x80bf7ebf2c0
  script/script.c:50: malloc 0x80bf7ebf280
  script/script.c:198: cmdline
  script/script.c:50: malloc 0x80bf7ebf220
  script/script.c:294: append command
  script/script.c:50: malloc 0x80bf7ebf0c0
  script/lexer.c:336: token 279 text [fi]
  script/script.c:50: malloc 0x80bf7ebef60
  script/script.c:50: malloc 0x80bf7ebef20
  script/script.c:223: cmdif
  script/script.c:50: malloc 0x80bf7ebeec0
  script/script.c:223: cmdif
  script/script.c:50: malloc 0x80bf7ebee60
  script/lexer.c:336: token 259 text [
  ]
  script/script.c:50: malloc 0x80bf7ebee00
  script/script.c:50: malloc 0x80bf7ebedc0
  script/lexer.c:336: token 0 text []
  script/script.c:50: malloc 0x80bf7ebf060
  script/script.c:50: malloc 0x80bf7ebf020
  script/script.c:294: append command
  script/script.c:50: malloc 0x80bf7ebed80
  commands/wildcard.c:535: no expansion needed
  commands/wildcard.c:594: paths[0] =
  `(tftp,10.229.32.21)/grub/arm64-efi/grub.cfg'
  kern/disk.c:196: Opening `tftp,10.229.32.21'...
  disk/efi/efidisk.c:482: opening tftp
  kern/disk.c:281: Opening `tftp,10.229.32.21' failed.
  kern/disk.c:295: Closing `tftp'.
  commands/wildcard.c:535: no expansion needed
  commands/wildcard.c:594: paths[0] = `(tftp,10.229.32.21)/grub/grub.cfg-arm64'
  kern/disk.c:196: Opening `tftp,10.229.32.21'...
  disk/efi/efidisk.c:482: opening tftp
  kern/disk.c:281: Opening `tftp,10.229.32.21' failed.
  kern/disk.c:295: Closing `tftp'.
  commands/wildcard.c:535: no expansion needed
  commands/wildcard.c:594: paths[0] = `(tftp,10.229.32.21)/grub/grub.cfg'
  kern/disk.c:196: Opening `tftp,10.229.32.21'...
  disk/efi/efidisk.c:482: opening tftp
  kern/disk.c:281: Opening `tftp,10.229.32.21' failed.
  kern/disk.c:295: Closing `tftp'.
  script/script.c:65: free 0x80bf7ebed80
  script/script.c:65: free 0x80bf7ebf020
  script/script.c:65: free 0x80bf7ebf060
  script/script.c:65: free 0x80bf7ebedc0
  script/script.c:65: free 0x80bf7ebee00
  script/script.c:65: free 0x80bf7ebee60
  script/script.c:65: free 0x80bf7ebeec0
  script/script.c:65: free 0x80bf7ebef20
  script/script.c:65: free 0x80bf7ebef60
  script/script.c:65: free 0x80bf7ebf0c0
  script/script.c:65: free 0x80bf7ebf220
  script/script.c:65: free 0x80bf7ebf280
  script/script.c:65: free 0x80bf7ebf2c0
  script/script.c:65: free 0x80bf7ebf320
  script/script.c:65: free 0x80bf7ebf380
  script/script.c:65: free 0x80bf7ebf540
  script/script.c:65: free 0x80bf7ebf3c0
  script/script.c:65: free 0x80bf7ebf400
  script/script.c:65: free 0x80bf7ebf460
  script/script.c:65: free 0x80bf7ebf4a0
  script/script.c:65: free 0x80bf7ebf5a0
  script/script.c:65: free 0x80bf7ebf600
  script/script.c:65: free 0x80bf7ebf640
  script/script.c:65: free 0x80bf7ebf7a0
  script/script.c:65: free 0x80bf7ebf7e0
  script/script.c:65: free 0x80bf7ebf840
  script/script.c:65: free 0x80bf7ebf880
  script/script.c:65: free 0x80bf7ebf9e0
  script/script.c:65: free 0x80bf7ebfb20
  script/script.c:65: free 0x80bf7ebfb80
  script/script.c:65: free 0x80bf7ebfbc0
  script/script.c:65: free 0x80bf7ebfc20
  script/script.c:65: free 0x80bf7ebfc80
  script/script.c:65: free 0x80bf7ebfce0
  script/script.c:65: free 0x80bf7ebfd40
  script/script.c:65: free 0x80bf7ebfd80
  script/script.c:65: free 0x80bf7ebfde0
  script/script.c:65: free 0x80bf7ebfe20
  script/script.c:65: free 0x80bf7ebff20
  script/script.c:65: free 0x80bf7ebff80
  script/script.c:65: free 0x80bf7ec00c0
  script/script.c:65: free 0x80bf7ec0240
  script/script.c:65: free 0x80bf7ec0280
  script/script.c:65: free 0x80bf7ec02e0
  script/script.c:65: free 0x80bf7ec0320
  script/script.c:65: free 0x80bf7ec0380
  script/script.c:65: free 0x80bf7ec03c0
  script/script.c:65: free 0x80bf7ec0420
  script/script.c:65: free 0x80bf7ec0460
  script/script.c:65: free 0x80bf7ec04c0
  script/script.c:65: free 0x80bf7ec0520
  script/script.c:65: free 0x80bf7ec0560
  script/script.c:65: free 0x80bf7ec05c0
  script/script.c:65: free 0x80bf7ec0620
  script/script.c:65: free 0x80bf7ec0660
  script/script.c:65: free 0x80bf7ec06c0
  script/script.c:65: free 0x80bf7ec0700
  script/script.c:65: free 0x80bf7ec0760
  script/script.c:65: free 0x80bf7ec07a0
  script/script.c:65: free 0x80bf7ec08a0
  script/script.c:65: free 0x80bf7ec0900
  script/script.c:65: free 0x80bf7ec0940
  script/script.c:65: free 0x80bf7ec09a0
  script/script.c:65: free 0x80bf7ec0a00
  script/script.c:65: free 0x80bf7ec0a40
  script/script.c:65: free 0x80bf7ec0be0
  script/script.c:65: free 0x80bf7ec0c20
  script/script.c:65: free 0x80bf7ec0da0
  script/script.c:65: free 0x80bf7ec0de0
  script/script.c:65: free 0x80bf7ec0e40
  script/script.c:65: free 0x80bf7ec0e80
  script/script.c:65: free 0x80bf7ec0ee0
  script/script.c:65: free 0x80bf7ec0f40
  script/script.c:65: free 0x80bf7ec0fa0
  script/script.c:65: free 0x80bf7ec1000
  script/script.c:65: free 0x80bf7ec1040
  script/script.c:65: free 0x80bf7ec10a0
  script/script.c:65: free 0x80bf7ec10e0
  script/script.c:65: free 0x80bf7ec11e0
  script/script.c:65: free 0x80bf7ec1240
  script/script.c:65: free 0x80bf7ec1280
  script/script.c:65: free 0x80bf7ec1400
  script/script.c:65: free 0x80bf7ec1440
  script/script.c:65: free 0x80bf7ec14a0
  script/script.c:65: free 0x80bf7ec14e0
  script/script.c:65: free 0x80bf7ec1540
  script/script.c:65: free 0x80bf7ec1580
  script/script.c:65: free 0x80bf7ec15e0
  script/script.c:65: free 0x80bf7ec1620
  script/script.c:65: free 0x80bf7ec1680
  script/script.c:65: free 0x80bf7ec16e0
  script/script.c:65: free 0x80bf7ec1720
  script/script.c:65: free 0x80bf7ec1780
  script/script.c:65: free 0x80bf7ec17e0
  script/script.c:65: free 0x80bf7ec1840
  script/script.c:65: free 0x80bf7ec18a0
  script/script.c:65: free 0x80bf7ec18e0
  script/script.c:65: free 0x80bf7ec1940
  script/script.c:65: free 0x80bf7ec1980
  script/script.c:65: free 0x80bf7ec1a80
  script/script.c:65: free 0x80bf7ec1ae0
  script/script.c:65: free 0x80bf7ec1b20
  script/script.c:65: free 0x80bf7ec1b80
  script/script.c:65: free 0x80bf7ec1be0
  script/script.c:65: free 0x80bf7ec1c20
  script/script.c:65: free 0x80bf7ec1e00
  script/script.c:65: free 0x80bf7ec1e40
  kern/disk.c:295: Closing `memdisk'.
  script/script.c:65: free 0x80bf7ec6f80
  script/script.c:65: free 0x80bf7ec6fc0
  script/script.c:65: free 0x80bf7ec7000
  script/script.c:65: free 0x80bf7ec6aa0
  script/script.c:65: free 0x80bf7ec6b00
  script/script.c:65: free 0x80bf7ec6b40
  script/script.c:65: free 0x80bf7ec6ba0
  script/script.c:65: free 0x80bf7ec6c00
  script/script.c:65: free 0x80bf7ec6c60
  script/script.c:65: free 0x80bf7ec6e40
  script/script.c:65: free 0x80bf7ec6ea0
  script/script.c:65: free 0x80bf7ec6ee0
  grub>

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2-signed/+bug/1934192/+subscriptions

More information about the foundations-bugs mailing list