[Bug 1935617] Re: systemd autopkgtest broken on ppc64el with qemu 6.0

Mon Jul 12 15:11:21 UTC 2021

We've had similar cases int he past where new releases glibc or compiler had new instructions breaking the TCG mode.
So I spawned a focal and a bionic image - and then remodeled them to run in emulation mode.

And indeed Bionic works without any crashes, so it must be something in the guest that isn't in Bionic yet. Focal OTOH crashes very similarly - so the userspace feature/instructions was added in 20.04. (BTW 20.04 feels more broken, e.g. it does not at all shut down anymore - but the crashes are just the same).

The focal crashes affect the same programs (rm, find) and look similar:

rm

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000727ed5063b0 in quotearg_buffer_restyled (buffer=buffer at entry=0x727ed530130 <slot0> "'", buffersize=buffersize at entry=256, 
    arg=arg at entry=0x800314f727070000 <error: Cannot access memory at address 0x800314f727070000>, argsize=argsize at entry=18446744073709551615, quoting_style=shell_always_quoting_style, 
    flags=flags at entry=1, quote_these_too=quote_these_too at entry=0x7fffdbb42770, left_quote=0x0, right_quote=0x0) at lib/quotearg.c:400
400	lib/quotearg.c: No such file or directory.
(gdb) bt
#0  0x00000727ed5063b0 in quotearg_buffer_restyled (buffer=buffer at entry=0x727ed530130 <slot0> "'", buffersize=buffersize at entry=256, 
    arg=arg at entry=0x800314f727070000 <error: Cannot access memory at address 0x800314f727070000>, argsize=argsize at entry=18446744073709551615, quoting_style=shell_always_quoting_style, 
    flags=flags at entry=1, quote_these_too=quote_these_too at entry=0x7fffdbb42770, left_quote=0x0, right_quote=0x0) at lib/quotearg.c:400
#1  0x00000727ed509144 in quotearg_n_options (options=0x7fffdbb42768, argsize=18446744073709551615, arg=0x800314f727070000 <error: Cannot access memory at address 0x800314f727070000>, n=0)
    at lib/quotearg.c:907
#2  quotearg_n_style (arg=0x800314f727070000 <error: Cannot access memory at address 0x800314f727070000>, s=shell_escape_always_quoting_style, n=0) at lib/quotearg.c:958
#3  quotearg_style (s=shell_escape_always_quoting_style, arg=0x800314f727070000 <error: Cannot access memory at address 0x800314f727070000>) at lib/quotearg.c:972
#4  0x00000727ed50377c in excise (ent=ent at entry=0x727f71415b0, x=x at entry=0x7fffdbb42bb8, is_dir=is_dir at entry=false, fts=<optimized out>) at src/remove.c:406
#5  0x00000727ed504644 in rm_fts (x=<optimized out>, ent=0x727f71415b0, fts=<optimized out>) at src/remove.c:548
#6  rm (file=<optimized out>, x=0x7fffdbb42bb8) at src/remove.c:607
#7  0x00000727ed502d98 in main (argc=<optimized out>, argv=<optimized out>) at src/rm.c:370

find
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_power8 () at ../sysdeps/powerpc/powerpc64/power8/strlen.S:36
36	../sysdeps/powerpc/powerpc64/power8/strlen.S: No such file or directory.
(gdb) bt
#0  __strlen_power8 () at ../sysdeps/powerpc/powerpc64/power8/strlen.S:36
#1  0x00007e0f566971f8 in __vfprintf_internal (s=0x7e0f568417e8 <_IO_2_1_stdout_>, format=0x6b007417f10 "%s\n", ap=0x7ffff96f4690 "", mode_flags=<optimized out>) at vfprintf-internal.c:1688
#2  0x00007e0f56772750 in ___fprintf_chk (fp=<optimized out>, flag=<optimized out>, format=<optimized out>) at fprintf_chk.c:33
#3  0x000006b0073d7960 in fprintf (__fmt=0x6b007417f10 "%s\n", __stream=<optimized out>) at /usr/include/powerpc64le-linux-gnu/bits/stdio2.h:100
#4  print_quoted (fp=<optimized out>, qopts=<optimized out>, dest_is_tty=<optimized out>, format=0x6b007417f10 "%s\n", s=<optimized out>) at printquoted.c:77
#5  0x000006b0073bad44 in pred_print () at pred.c:554
#6  0x000006b0073c8814 in apply_predicate (pathname=0x601e4c45b0060000 <error: Cannot access memory at address 0x601e4c45b0060000>, stat_buf=0x7ffff96f6888, p=0x6b0454c10d0) at util.c:1093
#7  0x000006b0073ba368 in pred_and (pred_ptr=0x6b0454c0f90, stat_buf=0x7ffff96f6888, pathname=0x601e4c45b0060000 <error: Cannot access memory at address 0x601e4c45b0060000>) at pred.c:224
#8  pred_and (pathname=0x601e4c45b0060000 <error: Cannot access memory at address 0x601e4c45b0060000>, stat_buf=0x7ffff96f6888, pred_ptr=0x6b0454c0f90) at pred.c:219
#9  0x000006b0073c8814 in apply_predicate (pathname=0x601e4c45b0060000 <error: Cannot access memory at address 0x601e4c45b0060000>, stat_buf=0x7ffff96f6888, p=0x6b0454c0f90) at util.c:1093
#10 0x000006b0073b89b8 in visit (pstat=<optimized out>, ent=0x6b0454c1540, p=0x6b0454c1390) at ftsfind.c:181
#11 consider_visiting (p=0x6b0454c1390, ent=0x6b0454c1540) at ftsfind.c:507
#12 0x000006b0073b8f80 in find (arg=0x7ffff96fff63 "/var/lib/update-notifier/updates-available") at ftsfind.c:584
#13 0x000006b0073b7f90 in process_all_startpoints (argv=<optimized out>, argc=<optimized out>) at ftsfind.c:625
#14 main (argc=<optimized out>, argv=<optimized out>) at ftsfind.c:734

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1935617

Title:
  systemd autopkgtest broken on ppc64el with qemu 6.0

Status in qemu package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  New

Bug description:
  I'm not sure yet if this is flaky or a real issue, but I'm filing it
  to avoid multiple people analyzing the same.

  The Qemu 6.0 upload https://launchpad.net/ubuntu/+source/qemu/1:6.0+dfsg-1~ubuntu2 triggers a test failure like
  https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/ppc64el/s/systemd/20210708_223311_e3bbb@/log.gz

  I have tested the new qemu on ppc64 and it worked fine for device emulation and migration cases.
  But this is suspicious.
  Of the last tests exactly and only those with the new qemu failed.

  impish
    ppc64el
      tests-in-lxd                   (F  5% f  0% S  0% B  0% => P 95%/) F.........F.............................
      systemd-fsckd                  (F  0% f  0% S 100% B  0% => P  0%/) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
      upstream-1                     (F 15% f  0% S  0% B  0% => P 85%/) F..FFFF....F............................
      upstream-2                     (F 12% f  0% S  0% B  0% => P 87%/) F..FFFF.................................

  For an insight in flakyness/reproducibility I've retriggered the missing qemu and the non-qemu cases a few times. If those reproduce all-bad vs all-good again this would further indicate a real issue.

  Unfortunately the ppc maas seems down right now and canonistack also
  isn't too nice this week - overall that inhibits the testing a bit :-/

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1935617/+subscriptions