ACK: [SRU][F, J, N, O][PATCH 0/1] bnx2x: Fix multiple UBSAN array-index-out-of-bounds

Tue Jul 30 00:02:00 UTC 2024

On Mon, Jul 29, 2024 at 6:37 AM Ghadi Elie Rahme
<ghadi.rahme at canonical.com> wrote:
>
> Buglink: https://bugs.launchpad.net/bugs/2074215
>
> [impact]
>
> Currently in the bnx2x kernel driver there are reads/writes that occur out of bounds that have the possibility to cause kernel crashes. No meaningful impact has been observed yet other than UBSAN stack traces.
> I have posted a patch upstream to resolve this issue (134061163ee5 bnx2x: Fix multiple UBSAN array-index-out-of-bounds) and it has been accepted and merged. Although these traces appear only on linux version 6.5 and up, this bug also affects kernels 6.x and 5.x as well but no UBSAN warnings will be printed on these kernels since they were not enforced in these kernels.
>
> [Test Plan]
>
> There are multiple ways to reproduce the issue. But the most hands free way to reproduce it would be to utilize a Qlogic NIC that makes use of the E2 controller on a system with more than 32 cores. Below are both ways this can be reproduced. Please note that both will require a NIC that makes use of the bnx2x driver.
>
> * Normal Reproduction:
>
> 1. start a machine running kernel 6.5 or higher with a number of cores above 32. Please note that these need to be physical cores not threads. The machine also needs to be using a NIC that utilizes an E2 controller.
> 2. In dmesg the following UBSAN warnings can be seen:
>
> UBSAN: array-index-out-of-bounds in
>        drivers/net/ethernet/broadcom/bnx2x/bnx2x_stats.c:1529:11
> index 20 is out of range for type 'stats_query_entry [19]'
> CPU: 12 PID: 858 Comm: systemd-network Not tainted 6.9.0-060900rc7-generic
>       #202405052133
> Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9,
>         BIOS P89 10/21/2019
> Call Trace:
>  <TASK>
>  dump_stack_lvl+0x76/0xa0
>  dump_stack+0x10/0x20
>  __ubsan_handle_out_of_bounds+0xcb/0x110
>  bnx2x_prep_fw_stats_req+0x2e1/0x310 [bnx2x]
>  bnx2x_stats_init+0x156/0x320 [bnx2x]
>  bnx2x_post_irq_nic_init+0x81/0x1a0 [bnx2x]
>  bnx2x_nic_load+0x8e8/0x19e0 [bnx2x]
>  bnx2x_open+0x16b/0x290 [bnx2x]
>  __dev_open+0x10e/0x1d0
> RIP: 0033:0x736223927a0a
> Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca
>       64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00
>       f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89
> RSP: 002b:00007ffc0bb2ada8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
> RAX: ffffffffffffffda RBX: 0000583df50f9c78 RCX: 0000736223927a0a
> RDX: 0000000000000020 RSI: 0000583df50ee510 RDI: 0000000000000003
> RBP: 0000583df50d4940 R08: 00007ffc0bb2adb0 R09: 0000000000000080
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000583df5103ae0
> R13: 000000000000035a R14: 0000583df50f9c30 R15: 0000583ddddddf00
> </TASK>
> ---[ end trace ]---
> ------------[ cut here ]------------
> UBSAN: array-index-out-of-bounds in
>        drivers/net/ethernet/broadcom/bnx2x/bnx2x_stats.c:1546:11
> index 28 is out of range for type 'stats_query_entry [19]'
> CPU: 12 PID: 858 Comm: systemd-network Not tainted 6.9.0-060900rc7-generic
>       #202405052133
> Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9,
>         BIOS P89 10/21/2019
> Call Trace:
> <TASK>
> dump_stack_lvl+0x76/0xa0
> dump_stack+0x10/0x20
> __ubsan_handle_out_of_bounds+0xcb/0x110
> bnx2x_prep_fw_stats_req+0x2fd/0x310 [bnx2x]
> bnx2x_stats_init+0x156/0x320 [bnx2x]
> bnx2x_post_irq_nic_init+0x81/0x1a0 [bnx2x]
> bnx2x_nic_load+0x8e8/0x19e0 [bnx2x]
> bnx2x_open+0x16b/0x290 [bnx2x]
> __dev_open+0x10e/0x1d0
> RIP: 0033:0x736223927a0a
> Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca
>       64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00
>       f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89
> RSP: 002b:00007ffc0bb2ada8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
> RAX: ffffffffffffffda RBX: 0000583df50f9c78 RCX: 0000736223927a0a
> RDX: 0000000000000020 RSI: 0000583df50ee510 RDI: 0000000000000003
> RBP: 0000583df50d4940 R08: 00007ffc0bb2adb0 R09: 0000000000000080
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000583df5103ae0
> R13: 000000000000035a R14: 0000583df50f9c30 R15: 0000583ddddddf00
>  </TASK>
> ---[ end trace ]---
> ------------[ cut here ]------------
> UBSAN: array-index-out-of-bounds in
>        drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.c:1895:8
> index 29 is out of range for type 'stats_query_entry [19]'
> CPU: 13 PID: 163 Comm: kworker/u96:1 Not tainted 6.9.0-060900rc7-generic
>       #202405052133
> Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9,
>         BIOS P89 10/21/2019
> Workqueue: bnx2x bnx2x_sp_task [bnx2x]
> Call Trace:
>  <TASK>
>  dump_stack_lvl+0x76/0xa0
>  dump_stack+0x10/0x20
>  __ubsan_handle_out_of_bounds+0xcb/0x110
>  bnx2x_iov_adjust_stats_req+0x3c4/0x3d0 [bnx2x]
>  bnx2x_storm_stats_post.part.0+0x4a/0x330 [bnx2x]
>  ? bnx2x_hw_stats_post+0x231/0x250 [bnx2x]
>  bnx2x_stats_start+0x44/0x70 [bnx2x]
>  bnx2x_stats_handle+0x149/0x350 [bnx2x]
>  bnx2x_attn_int_asserted+0x998/0x9b0 [bnx2x]
>  bnx2x_sp_task+0x491/0x5c0 [bnx2x]
>  process_one_work+0x18d/0x3f0
>  </TASK>
> ---[ end trace ]---
>
> * Forced reproducer:
>
> 1. Make sure you have a machine running kernel 6.5 and higher with any NIC that makes use of the bnx2x driver (No need for a NIC that utilizes the E2 controller). Also the number of cores the machine has is not important.
>
> 2. once the machine is booted unload the bnx2x module from the kernel:
> $ sudo modprobe -r bnx2x
>
> 3. then load back the driver but while specifying the number of ethernet queues to a value above 16:
> $ sudo modprobe bnx2x num_queues=20
>
> 4. The same stack traces shown above will show up in dmesg.
>
> [Fix]
>
> The fix is already upstream and provided by:
>
> * 134061163ee5 bnx2x: Fix multiple UBSAN array-index-out-of-bounds
>
> [where problems could occur]
>
> * Since the patch increases the firmware stats array size, the driver will utilize slightly more memory, however this is still an insignificant amount.
>
> * Since no logic change has been done to the driver the regression risk is minimal
>
> [workaround]
>
> As stated earlier I have already written a patch to solve the issue, but in the meantime one way to avoid this problem would be to unload the driver and then load it back with a value for num_queues below 16:
> $ sudo modprobe bnx2x num_queues=15
>
> Ghadi Elie Rahme (1):
>   bnx2x: Fix multiple UBSAN array-index-out-of-bounds
>
>  drivers/net/ethernet/broadcom/bnx2x/bnx2x.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> --

Acked-by: Kevin Becker <kevin.becker at canonical.com>