[Bug 1842020] Re: ceph patch as of 8/29 segfaults all bluestore osds

Harry Coin hgcoin at gmail.com
Sat Aug 31 14:10:22 UTC 2019


And with debug symbols:

(gdb) run
Starting program: /usr/bin/ceph-bluestore-tool 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program received signal SIGILL, Illegal instruction.
0x0000555555743984 in eth_dev_init_cb_lists ()
(gdb) backtrace full
#0  0x0000555555743984 in eth_dev_init_cb_lists ()
No symbol table info available.
#1  0x0000555555dc045d in __libc_csu_init ()
No symbol table info available.
#2  0x00007fffee524e2e in __libc_start_main (main=0x5555557346b0 <main(int, char**)>, argc=1, argv=0x7fffffffe328, init=0x555555dc0410 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe318)
at ../csu/libc-start.c:264
result = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737351328816, 5390835150573769728, 140737353589600, 140737353586152, 1, 140737488347944, 140737488347960, 140737354007706}, mask_was_saved = 8}}, priv = {pad = {0x1, 
0x7fffffffe328, 0x7fffffffe338, 0x7ffff7ffe190}, data = {prev = 0x1, cleanup = 0x7fffffffe328, canceltype = -7368}}}
not_first_call = <optimized out>
#3  0x000055555581e47e in _start () at /usr/include/c++/9/ostream:108
No symbol table info available.
(gdb) backtrace full
#0  0x0000555555743984 in eth_dev_init_cb_lists () at /usr/include/c++/9/ostream:108
No symbol table info available.
#1  0x0000555555dc045d in __libc_csu_init ()
No symbol table info available.
#2  0x00007fffee524e2e in __libc_start_main (main=0x5555557346b0 <main(int, char**)>, argc=1, argv=0x7fffffffe328, init=0x555555dc0410 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe318)
at ../csu/libc-start.c:264
result = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737351328816, 5390835150573769728, 140737353589600, 140737353586152, 1, 140737488347944, 140737488347960, 140737354007706}, mask_was_saved = 8}}, priv = {pad = {0x1, 
0x7fffffffe328, 0x7fffffffe338, 0x7ffff7ffe190}, data = {prev = 0x1, cleanup = 0x7fffffffe328, canceltype = -7368}}}
not_first_call = <optimized out>
#3  0x000055555581e47e in _start () at /usr/include/c++/9/ostream:108
No symbol table info available.
(gdb) info registers
rax            0x555555fe0340      93825003291456
rbx            0x36                54
rcx            0xb                 11
rdx            0x5555568921a0      93825012408736
rsi            0x7fffffffe328      140737488347944
rdi            0x1                 1
rbp            0xc5                0xc5
rsp            0x7fffffffe208      0x7fffffffe208
r8             0x0                 0
r9             0x0                 0
r10            0x642e6264626f6c62  7218815436009204834
r11            0x20                32
r12            0x555555f2b510      93825002550544
r13            0x1                 1
r14            0x7fffffffe328      140737488347944
r15            0x5555568921a0      93825012408736
rip            0x555555743984      0x555555743984 <eth_dev_init_cb_lists+68>
eflags         0x10212             [ AF IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
(gdb) x/16i $pc
=> 0x555555743984 <eth_dev_init_cb_lists+68>:   pextrq $0x1,%xmm2,0x40c0(%rax)
0x55555574398f <eth_dev_init_cb_lists+79>:   pextrq $0x1,%xmm1,0xc1c0(%rax)
0x55555574399a <eth_dev_init_cb_lists+90>:   movdqa 0x6c2eae(%rip),%xmm2        # 0x555555e06850
0x5555557439a2 <eth_dev_init_cb_lists+98>:   movq   %xmm1,0x8140(%rax)
0x5555557439aa <eth_dev_init_cb_lists+106>:  movdqa 0x6c2eae(%rip),%xmm1        # 0x555555e06860
0x5555557439b2 <eth_dev_init_cb_lists+114>:  movq   $0x0,0x8138(%rax)
0x5555557439bd <eth_dev_init_cb_lists+125>:  paddq  %xmm0,%xmm2
0x5555557439c1 <eth_dev_init_cb_lists+129>:  movq   %xmm2,0x10240(%rax)
0x5555557439c9 <eth_dev_init_cb_lists+137>:  paddq  %xmm0,%xmm1
0x5555557439cd <eth_dev_init_cb_lists+141>:  pextrq $0x1,%xmm2,0x142c0(%rax)
0x5555557439d8 <eth_dev_init_cb_lists+152>:  movdqa 0x6c2e90(%rip),%xmm2        # 0x555555e06870
0x5555557439e0 <eth_dev_init_cb_lists+160>:  movq   %xmm1,0x18340(%rax)
0x5555557439e8 <eth_dev_init_cb_lists+168>:  pextrq $0x1,%xmm1,0x1c3c0(%rax)
0x5555557439f3 <eth_dev_init_cb_lists+179>:  movdqa 0x6c2e85(%rip),%xmm1        # 0x555555e06880
0x5555557439fb <eth_dev_init_cb_lists+187>:  movq   $0x0,0xc1b8(%rax)
0x555555743a06 <eth_dev_init_cb_lists+198>:  paddq  %xmm0,%xmm2
(gdb) thread apply all backtrace
Thread 1 (Thread 0x7fffee0e20c0 (LWP 825)):
#0  0x0000555555743984 in eth_dev_init_cb_lists () at /usr/include/c++/9/ostream:108
#1  0x0000555555dc045d in __libc_csu_init ()
#2  0x00007fffee524e2e in __libc_start_main (main=0x5555557346b0 <main(int, char**)>, argc=1, argv=0x7fffffffe328, init=0x555555dc0410 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe318) at ../csu/libc-start.c:264
#3  0x000055555581e47e in _start () at /usr/include/c++/9/ostream:108
(gdb)

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1842020

Title:
  ceph patch as of 8/29 segfaults all bluestore osds

Status in ceph package in Ubuntu:
  New

Bug description:
  The ceph patch on eoan distributed 8/29 crashes all bluestore OSD's
  and is unusable at least on some systems.  Failed on an old dual Xeon
  E5345 box.

  Easy test.  Run:

  /usr/bin/ceph-bluestore-tool

  On working systems it reports a help message.  On the latest eoan release distributed by canonical it
  turns in to an illegal instruction process kill, just after reading /proc/<...>/auxv

  looks like something to do with vsock issues.

  strace -k -y /usr/bin/ceph-bluestore-tool 
  is instructive.

  Notice valgrind against that program reports thousands of memory allocation issues.
  Reverting the system to the snapshot before the apt upgrade restores full operations.
  --- 
  ProblemType: Bug
  ApportVersion: 2.20.11-0ubuntu7
  Architecture: amd64
  DistroRelease: Ubuntu 19.10
  InstallationDate: Installed on 2019-07-30 (30 days ago)
  InstallationMedia: Ubuntu-MATE 19.10 "Eoan Ermine" - Alpha amd64 (20190726)
  Package: ceph 14.2.2-0ubuntu2
  PackageArchitecture: amd64
  ProcEnviron:
   LANGUAGE=en_US
   TERM=xterm-256color
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcVersionSignature: Ubuntu 5.2.0-15.16-generic 5.2.9
  Tags:  eoan
  Uname: Linux 5.2.0-15-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups:
   
  _MarkForUpload: True
  mtime.conffile..etc.default.apport: 2019-08-30T11:35:29.463071

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1842020/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list