[Bug 1832867] Re: Blocked devices unlinked after upgrade of packages via landscape

James Page james.page at ubuntu.com
Thu Jul 18 07:16:33 UTC 2019


tl;dr upgrading the ceph packages should not tinker with ceph osd
devices.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1832867

Title:
  Blocked devices unlinked after upgrade of packages via landscape

Status in OpenStack ceph-osd charm:
  Invalid
Status in ceph package in Ubuntu:
  Incomplete

Bug description:
  Overview Description:
  =====================

  During landscape upgrades on customer cloud, numerous ceph related packages were upgraded.
  During package configurations process the upgrade resulted in the removal of:

  /var/lib/ceph/osd/ceph-XXXX/block symlink from a number of machines
  and the charm did not restore the link

  engineers attempted to use udevadm to trigger block device detection and symlink creation:
  eventually discovered "ceph-volume lvm activate --all" which recreated the symlinks

  
  Cloud Current versions:
  =======================

  Bionic
  Queens
  ceph version 12.2.11

  Packages upgraded:

  Additional required changes
  Upgrade package libdns1100 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package libgcc-7-dev 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package libgcc1 1:8.2.0-1ubuntu2~18.04 to 1:8.3.0-6ubuntu1~18.04
  Upgrade package libgfortran4 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package libgomp1 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package libirs160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package libisc169 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package libisccc160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package libisccfg160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package libitm1 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package liblsan0 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package liblwres160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package libmagic-mgc 1:5.32-2ubuntu0.1 to 1:5.32-2ubuntu0.2
  Upgrade package libmagic1 1:5.32-2ubuntu0.1 to 1:5.32-2ubuntu0.2
  Upgrade package libmpx2 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package libquadmath0 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package librados2 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package libradosstriper1 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package librbd1 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package librgw2 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package libstdc++-7-dev 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package libstdc++6 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package libtsan0 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package libubsan0 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package python-cephfs 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package python-rados 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package python-rbd 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package python3-apport 2.20.9-0ubuntu7.5 to 2.20.9-0ubuntu7.6
  Upgrade package python3-debconf 1.5.66 to 1.5.66ubuntu1
  Upgrade package radosgw 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  User requested changes
  Upgrade package apport 2.20.9-0ubuntu7.5 to 2.20.9-0ubuntu7.6
  Upgrade package apt 1.6.8 to 1.6.11
  Upgrade package apt-transport-https 1.6.8 to 1.6.11
  Upgrade package apt-utils 1.6.8 to 1.6.11
  Upgrade package bash 4.4.18-2ubuntu1 to 4.4.18-2ubuntu1.1
  Upgrade package bind9-host 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package binutils 2.30-21ubuntu1~18.04 to 2.30-21ubuntu1~18.04.1
  Upgrade package binutils-common 2.30-21ubuntu1~18.04 to 2.30-21ubuntu1~18.04.1
  Upgrade package binutils-x86-64-linux-gnu 2.30-21ubuntu1~18.04 to 2.30-21ubuntu1~18.04.1
  Upgrade package busybox-initramfs 1:1.27.2-2ubuntu3 to 1:1.27.2-2ubuntu3.2
  Upgrade package busybox-static 1:1.27.2-2ubuntu3 to 1:1.27.2-2ubuntu3.2
  Upgrade package ceph 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package ceph-base 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package ceph-common 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package ceph-mgr 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package ceph-mon 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package ceph-osd 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package cloud-init 18.4-0ubuntu1~18.04.1 to 19.1-1-gbaa47854-0ubuntu1~18.04.1
  Upgrade package console-setup 1.178ubuntu2.7 to 1.178ubuntu2.9
  Upgrade package console-setup-linux 1.178ubuntu2.7 to 1.178ubuntu2.9
  Upgrade package cpp 4:7.3.0-3ubuntu2.1 to 4:7.4.0-1ubuntu2.2
  Upgrade package cpp-7 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package curl 7.58.0-2ubuntu3.6 to 7.58.0-2ubuntu3.7
  Upgrade package dbus 1.12.2-1ubuntu1 to 1.12.2-1ubuntu1.1
  Upgrade package debconf 1.5.66 to 1.5.66ubuntu1
  Upgrade package debconf-i18n 1.5.66 to 1.5.66ubuntu1
  Upgrade package distro-info 0.18 to 0.18ubuntu0.18.04.1
  Upgrade package distro-info-data 0.37ubuntu0.2 to 0.37ubuntu0.5
  Upgrade package dnsutils 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package file 1:5.32-2ubuntu0.1 to 1:5.32-2ubuntu0.2
  Upgrade package filebeat 5.6.15 to 5.6.16
  Upgrade package g++ 4:7.3.0-3ubuntu2.1 to 4:7.4.0-1ubuntu2.2
  Upgrade package g++-7 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package gcc 4:7.3.0-3ubuntu2.1 to 4:7.4.0-1ubuntu2.2
  Upgrade package gcc-7 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package gcc-7-base 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package gcc-8-base 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package gettext-base 0.19.8.1-6ubuntu0.1 to 0.19.8.1-6ubuntu0.3
  Upgrade package grub-common 2.02-2ubuntu8.12 to 2.02-2ubuntu8.13
  Upgrade package grub-efi-amd64 2.02-2ubuntu8.12 to 2.02-2ubuntu8.13
  Upgrade package grub-efi-amd64-bin 2.02-2ubuntu8.12 to 2.02-2ubuntu8.13
  Upgrade package grub-efi-amd64-signed 1.93.13+2.02-2ubuntu8.12 to 1.93.14+2.02-2ubuntu8.13
  Upgrade package grub2-common 2.02-2ubuntu8.12 to 2.02-2ubuntu8.13
  Upgrade package initramfs-tools 0.130ubuntu3.7 to 0.130ubuntu3.8
  Upgrade package initramfs-tools-bin 0.130ubuntu3.7 to 0.130ubuntu3.8
  Upgrade package initramfs-tools-core 0.130ubuntu3.7 to 0.130ubuntu3.8
  Upgrade package intel-microcode 3.20180807a.0ubuntu0.18.04.1 to 3.20190514.0ubuntu0.18.04.3
  Upgrade package ipvsadm 1:1.28-3build1 to 1:1.28-3ubuntu0.18.04.1
  Upgrade package isc-dhcp-client 4.3.5-3ubuntu7 to 4.3.5-3ubuntu7.1
  Upgrade package isc-dhcp-common 4.3.5-3ubuntu7 to 4.3.5-3ubuntu7.1
  Upgrade package keepalived 1:1.3.9-1ubuntu0.18.04.1 to 1:1.3.9-1ubuntu0.18.04.2
  Upgrade package keyboard-configuration 1.178ubuntu2.7 to 1.178ubuntu2.9
  Upgrade package language-selector-common 0.188.1 to 0.188.2
  Upgrade package libaio1 0.3.110-5 to 0.3.110-5ubuntu0.1
  Upgrade package libapt-inst2.0 1.6.8 to 1.6.11
  Upgrade package libapt-pkg5.0 1.6.8 to 1.6.11
  Upgrade package libasan4 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package libatomic1 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package libbind9-160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package libbinutils 2.30-21ubuntu1~18.04 to 2.30-21ubuntu1~18.04.1
  Upgrade package libcc1-0 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
  Upgrade package libcephfs2 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
  Upgrade package libcilkrts5 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
  Upgrade package libcups2 2.2.7-1ubuntu2.3 to 2.2.7-1ubuntu2.6
  Upgrade package libcurl3-gnutls 7.58.0-2ubuntu3.6 to 7.58.0-2ubuntu3.7
  Upgrade package libcurl4 7.58.0-2ubuntu3.6 to 7.58.0-2ubuntu3.7
  Upgrade package libdb5.3 5.3.28-13.1ubuntu1 to 5.3.28-13.1ubuntu1.1
  Upgrade package libdbus-1-3 1.12.2-1ubuntu1 to 1.12.2-1ubuntu1.1

  
  Yesterday at 11:55 UTC

  Additional required changes
  Upgrade package libpam-systemd 237-3ubuntu10.13 to 237-3ubuntu10.22
  Upgrade package libsystemd0 237-3ubuntu10.13 to 237-3ubuntu10.22
  Upgrade package ntfs-3g 1:2017.3.23-2 to 1:2017.3.23-2ubuntu0.18.04.2
  Upgrade package systemd 237-3ubuntu10.13 to 237-3ubuntu10.22
  User requested changes
  Upgrade package libdns-export1100 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package libdw1 0.170-0.4 to 0.170-0.4ubuntu0.1
  Upgrade package libelf1 0.170-0.4 to 0.170-0.4ubuntu0.1
  Upgrade package libglib2.0-0 2.56.3-0ubuntu0.18.04.1 to 2.56.4-0ubuntu0.18.04.3
  Upgrade package libglib2.0-data 2.56.3-0ubuntu0.18.04.1 to 2.56.4-0ubuntu0.18.04.3
  Upgrade package libgnutls30 3.5.18-1ubuntu1 to 3.5.18-1ubuntu1.1
  Upgrade package libidn11 1.33-2.1ubuntu1 to 1.33-2.1ubuntu1.2
  Upgrade package libisc-export169 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
  Upgrade package libldap-2.4-2 2.4.45+dfsg-1ubuntu1.1 to 2.4.45+dfsg-1ubuntu1.2
  Upgrade package libldap-common 2.4.45+dfsg-1ubuntu1.1 to 2.4.45+dfsg-1ubuntu1.2
  Upgrade package liblua5.3-0 5.3.3-1 to 5.3.3-1ubuntu0.18.04.1
  Upgrade package libmysqlclient20 5.7.25-0ubuntu0.18.04.2 to 5.7.26-0ubuntu0.18.04.1
  Upgrade package libnss-systemd 237-3ubuntu10.13 to 237-3ubuntu10.22
  Upgrade package libntfs-3g88 1:2017.3.23-2 to 1:2017.3.23-2ubuntu0.18.04.2
  Upgrade package libnuma1 2.0.11-2.1 to 2.0.11-2.1ubuntu0.1
  Upgrade package libpam-modules 1.1.8-3.6ubuntu2 to 1.1.8-3.6ubuntu2.18.04.1
  Upgrade package libpam-modules-bin 1.1.8-3.6ubuntu2 to 1.1.8-3.6ubuntu2.18.04.1
  Upgrade package libpam-runtime 1.1.8-3.6ubuntu2 to 1.1.8-3.6ubuntu2.18.04.1

  Actual Results:
  ===============

  symlink was missing

  Expected Results:
  =================

  Something like:

  root at ob-sv-ostk05:/home/ubuntu# ls -ld /var/lib/ceph/osd/ceph-10/block
  lrwxrwxrwx 1 ceph ceph 93 Jun 13 16:22 /var/lib/ceph/osd/ceph-10/block -> /dev/ceph-35724d7c-a9c3-41b0-8c54-022b7e723686/osd-block-35724d7c-a9c3-41b0-8c54-022b7e723686

  
  Reproducibility:
  ================

  Difficult: but avoidable: suggest checking for the symlink in the
  charm, if its not present and other characteristics?  run "ceph-volume
  lvm activate --all"

  Resolved by:
  ============

  for i in <list of osd numbers>; 
      systemctl stop ceph-osd@$i; 
  done; 

  ceph-volume lvm activate --all

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-osd/+bug/1832867/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list