[Bug 1922782] Re: MAAS PXE Boot stalls with Grub 2.04 and Broadcom NICs

[Antony Messerli]

Built bootloaders from Hirsute using maas-images, and got the same
result with GRUB 2.04-1ubuntu45.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to grub2 in Ubuntu.
https://bugs.launchpad.net/bugs/1922782

Title:
  MAAS PXE Boot stalls with Grub 2.04 and Broadcom NICs

Status in grub2 package in Ubuntu:
  New

Bug description:
  Environment:

  MAAS version (SNAP):

  2.9/stable:       2.9.2-9164-g.ac176b5c4        2021-02-17 (11851) 150MB
  Grub package_version=2.04-1ubuntu26.9 from ephermeral-v3 maas images

  Servers
  Dell R7525 configured in UEFI mode with both:
  Broadcom Gigabit Ethernet BCM5720
  Broadcom Adv. Dual 10GBASE-T Ethernet BCM57416

  Problem description:

  On commissioning of a new node, the server retrieves bootx64.efi from
  the MAAS server, loads grubx64.efi and then hangs at Booting under
  MAAS direction...  Because it's getting this far, it's loaded Grub and
  a configuration at this point.

  I increased the timeout from 0 to 10 in the MAAS code so that I could
  crack into Grub to debug.  Configuration is getting retrieved from
  MAAS server so I edited the configuration to do a debug=all and loaded
  the configuration.

  Logs show that its attempting to load the kernel and initrd but fails
  when it was previously able to contact the MAAS server via PXE (sample
  of kernel load):

  kern/disk.c:196: Opening 'http,10.127.88.10:5248'...
  disk/efi/efidisk.c:482: opening http
  kern/disk.c:281 Opening 'http,10.127.88.10:5248' failed.
  kern/disk.c:295 Closing 'http'.
  net/http.c:405: opening path
  /images/ubuntu/amd64/hwe-20.04/focal/stable/boot-kernel on host 10.127.88.10 TCP port 5248
  commands/verifiers.c:88: file:
  (http,10.127.88.10:5248)/images/ubuntu/amd64/hwe-20.04/focal/stable/boot-kernel
  type:3
  ....

  last debug ends on:

  loader/efi/linux.c:96: kernel_addr: 0x10000000 handover_offset: 0x190
  params: 0x3d6e1000

  If I switch to Intel NICs in the server, this issue does not occur.
  We are wondering if it may be BCM5720 and PCI-e Gen 4 related as we
  have the BCM5720 NICs in Dell R720s with PCI-e Gen 3 and they can
  commission properly.

  I have seen mention of some newer versions of Grub that may solve some
  HTTP boot issues, but they have not made their way into MAAS yet.  If
  there are good ways to build those bootloaders that would align to how
  MAAS builds them for their images and test them, I can try and test
  them in my environment to see if they resolve the issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1922782/+subscriptions