[Bug 1832867] Re: Blocked devices unlinked after upgrade of packages via landscape
James Page
james.page at ubuntu.com
Thu Jul 18 07:16:33 UTC 2019
tl;dr upgrading the ceph packages should not tinker with ceph osd
devices.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1832867
Title:
Blocked devices unlinked after upgrade of packages via landscape
Status in OpenStack ceph-osd charm:
Invalid
Status in ceph package in Ubuntu:
Incomplete
Bug description:
Overview Description:
=====================
During landscape upgrades on customer cloud, numerous ceph related packages were upgraded.
During package configurations process the upgrade resulted in the removal of:
/var/lib/ceph/osd/ceph-XXXX/block symlink from a number of machines
and the charm did not restore the link
engineers attempted to use udevadm to trigger block device detection and symlink creation:
eventually discovered "ceph-volume lvm activate --all" which recreated the symlinks
Cloud Current versions:
=======================
Bionic
Queens
ceph version 12.2.11
Packages upgraded:
Additional required changes
Upgrade package libdns1100 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package libgcc-7-dev 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package libgcc1 1:8.2.0-1ubuntu2~18.04 to 1:8.3.0-6ubuntu1~18.04
Upgrade package libgfortran4 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package libgomp1 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package libirs160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package libisc169 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package libisccc160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package libisccfg160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package libitm1 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package liblsan0 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package liblwres160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package libmagic-mgc 1:5.32-2ubuntu0.1 to 1:5.32-2ubuntu0.2
Upgrade package libmagic1 1:5.32-2ubuntu0.1 to 1:5.32-2ubuntu0.2
Upgrade package libmpx2 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package libquadmath0 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package librados2 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package libradosstriper1 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package librbd1 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package librgw2 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package libstdc++-7-dev 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package libstdc++6 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package libtsan0 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package libubsan0 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package python-cephfs 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package python-rados 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package python-rbd 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package python3-apport 2.20.9-0ubuntu7.5 to 2.20.9-0ubuntu7.6
Upgrade package python3-debconf 1.5.66 to 1.5.66ubuntu1
Upgrade package radosgw 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
User requested changes
Upgrade package apport 2.20.9-0ubuntu7.5 to 2.20.9-0ubuntu7.6
Upgrade package apt 1.6.8 to 1.6.11
Upgrade package apt-transport-https 1.6.8 to 1.6.11
Upgrade package apt-utils 1.6.8 to 1.6.11
Upgrade package bash 4.4.18-2ubuntu1 to 4.4.18-2ubuntu1.1
Upgrade package bind9-host 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package binutils 2.30-21ubuntu1~18.04 to 2.30-21ubuntu1~18.04.1
Upgrade package binutils-common 2.30-21ubuntu1~18.04 to 2.30-21ubuntu1~18.04.1
Upgrade package binutils-x86-64-linux-gnu 2.30-21ubuntu1~18.04 to 2.30-21ubuntu1~18.04.1
Upgrade package busybox-initramfs 1:1.27.2-2ubuntu3 to 1:1.27.2-2ubuntu3.2
Upgrade package busybox-static 1:1.27.2-2ubuntu3 to 1:1.27.2-2ubuntu3.2
Upgrade package ceph 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package ceph-base 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package ceph-common 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package ceph-mgr 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package ceph-mon 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package ceph-osd 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package cloud-init 18.4-0ubuntu1~18.04.1 to 19.1-1-gbaa47854-0ubuntu1~18.04.1
Upgrade package console-setup 1.178ubuntu2.7 to 1.178ubuntu2.9
Upgrade package console-setup-linux 1.178ubuntu2.7 to 1.178ubuntu2.9
Upgrade package cpp 4:7.3.0-3ubuntu2.1 to 4:7.4.0-1ubuntu2.2
Upgrade package cpp-7 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package curl 7.58.0-2ubuntu3.6 to 7.58.0-2ubuntu3.7
Upgrade package dbus 1.12.2-1ubuntu1 to 1.12.2-1ubuntu1.1
Upgrade package debconf 1.5.66 to 1.5.66ubuntu1
Upgrade package debconf-i18n 1.5.66 to 1.5.66ubuntu1
Upgrade package distro-info 0.18 to 0.18ubuntu0.18.04.1
Upgrade package distro-info-data 0.37ubuntu0.2 to 0.37ubuntu0.5
Upgrade package dnsutils 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package file 1:5.32-2ubuntu0.1 to 1:5.32-2ubuntu0.2
Upgrade package filebeat 5.6.15 to 5.6.16
Upgrade package g++ 4:7.3.0-3ubuntu2.1 to 4:7.4.0-1ubuntu2.2
Upgrade package g++-7 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package gcc 4:7.3.0-3ubuntu2.1 to 4:7.4.0-1ubuntu2.2
Upgrade package gcc-7 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package gcc-7-base 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package gcc-8-base 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package gettext-base 0.19.8.1-6ubuntu0.1 to 0.19.8.1-6ubuntu0.3
Upgrade package grub-common 2.02-2ubuntu8.12 to 2.02-2ubuntu8.13
Upgrade package grub-efi-amd64 2.02-2ubuntu8.12 to 2.02-2ubuntu8.13
Upgrade package grub-efi-amd64-bin 2.02-2ubuntu8.12 to 2.02-2ubuntu8.13
Upgrade package grub-efi-amd64-signed 1.93.13+2.02-2ubuntu8.12 to 1.93.14+2.02-2ubuntu8.13
Upgrade package grub2-common 2.02-2ubuntu8.12 to 2.02-2ubuntu8.13
Upgrade package initramfs-tools 0.130ubuntu3.7 to 0.130ubuntu3.8
Upgrade package initramfs-tools-bin 0.130ubuntu3.7 to 0.130ubuntu3.8
Upgrade package initramfs-tools-core 0.130ubuntu3.7 to 0.130ubuntu3.8
Upgrade package intel-microcode 3.20180807a.0ubuntu0.18.04.1 to 3.20190514.0ubuntu0.18.04.3
Upgrade package ipvsadm 1:1.28-3build1 to 1:1.28-3ubuntu0.18.04.1
Upgrade package isc-dhcp-client 4.3.5-3ubuntu7 to 4.3.5-3ubuntu7.1
Upgrade package isc-dhcp-common 4.3.5-3ubuntu7 to 4.3.5-3ubuntu7.1
Upgrade package keepalived 1:1.3.9-1ubuntu0.18.04.1 to 1:1.3.9-1ubuntu0.18.04.2
Upgrade package keyboard-configuration 1.178ubuntu2.7 to 1.178ubuntu2.9
Upgrade package language-selector-common 0.188.1 to 0.188.2
Upgrade package libaio1 0.3.110-5 to 0.3.110-5ubuntu0.1
Upgrade package libapt-inst2.0 1.6.8 to 1.6.11
Upgrade package libapt-pkg5.0 1.6.8 to 1.6.11
Upgrade package libasan4 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package libatomic1 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package libbind9-160 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package libbinutils 2.30-21ubuntu1~18.04 to 2.30-21ubuntu1~18.04.1
Upgrade package libcc1-0 8.2.0-1ubuntu2~18.04 to 8.3.0-6ubuntu1~18.04
Upgrade package libcephfs2 12.2.8-0ubuntu0.18.04.2 to 12.2.11-0ubuntu0.18.04.2
Upgrade package libcilkrts5 7.3.0-27ubuntu1~18.04 to 7.4.0-1ubuntu1~18.04
Upgrade package libcups2 2.2.7-1ubuntu2.3 to 2.2.7-1ubuntu2.6
Upgrade package libcurl3-gnutls 7.58.0-2ubuntu3.6 to 7.58.0-2ubuntu3.7
Upgrade package libcurl4 7.58.0-2ubuntu3.6 to 7.58.0-2ubuntu3.7
Upgrade package libdb5.3 5.3.28-13.1ubuntu1 to 5.3.28-13.1ubuntu1.1
Upgrade package libdbus-1-3 1.12.2-1ubuntu1 to 1.12.2-1ubuntu1.1
Yesterday at 11:55 UTC
Additional required changes
Upgrade package libpam-systemd 237-3ubuntu10.13 to 237-3ubuntu10.22
Upgrade package libsystemd0 237-3ubuntu10.13 to 237-3ubuntu10.22
Upgrade package ntfs-3g 1:2017.3.23-2 to 1:2017.3.23-2ubuntu0.18.04.2
Upgrade package systemd 237-3ubuntu10.13 to 237-3ubuntu10.22
User requested changes
Upgrade package libdns-export1100 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package libdw1 0.170-0.4 to 0.170-0.4ubuntu0.1
Upgrade package libelf1 0.170-0.4 to 0.170-0.4ubuntu0.1
Upgrade package libglib2.0-0 2.56.3-0ubuntu0.18.04.1 to 2.56.4-0ubuntu0.18.04.3
Upgrade package libglib2.0-data 2.56.3-0ubuntu0.18.04.1 to 2.56.4-0ubuntu0.18.04.3
Upgrade package libgnutls30 3.5.18-1ubuntu1 to 3.5.18-1ubuntu1.1
Upgrade package libidn11 1.33-2.1ubuntu1 to 1.33-2.1ubuntu1.2
Upgrade package libisc-export169 1:9.11.3+dfsg-1ubuntu1.5 to 1:9.11.3+dfsg-1ubuntu1.7
Upgrade package libldap-2.4-2 2.4.45+dfsg-1ubuntu1.1 to 2.4.45+dfsg-1ubuntu1.2
Upgrade package libldap-common 2.4.45+dfsg-1ubuntu1.1 to 2.4.45+dfsg-1ubuntu1.2
Upgrade package liblua5.3-0 5.3.3-1 to 5.3.3-1ubuntu0.18.04.1
Upgrade package libmysqlclient20 5.7.25-0ubuntu0.18.04.2 to 5.7.26-0ubuntu0.18.04.1
Upgrade package libnss-systemd 237-3ubuntu10.13 to 237-3ubuntu10.22
Upgrade package libntfs-3g88 1:2017.3.23-2 to 1:2017.3.23-2ubuntu0.18.04.2
Upgrade package libnuma1 2.0.11-2.1 to 2.0.11-2.1ubuntu0.1
Upgrade package libpam-modules 1.1.8-3.6ubuntu2 to 1.1.8-3.6ubuntu2.18.04.1
Upgrade package libpam-modules-bin 1.1.8-3.6ubuntu2 to 1.1.8-3.6ubuntu2.18.04.1
Upgrade package libpam-runtime 1.1.8-3.6ubuntu2 to 1.1.8-3.6ubuntu2.18.04.1
Actual Results:
===============
symlink was missing
Expected Results:
=================
Something like:
root at ob-sv-ostk05:/home/ubuntu# ls -ld /var/lib/ceph/osd/ceph-10/block
lrwxrwxrwx 1 ceph ceph 93 Jun 13 16:22 /var/lib/ceph/osd/ceph-10/block -> /dev/ceph-35724d7c-a9c3-41b0-8c54-022b7e723686/osd-block-35724d7c-a9c3-41b0-8c54-022b7e723686
Reproducibility:
================
Difficult: but avoidable: suggest checking for the symlink in the
charm, if its not present and other characteristics? run "ceph-volume
lvm activate --all"
Resolved by:
============
for i in <list of osd numbers>;
systemctl stop ceph-osd@$i;
done;
ceph-volume lvm activate --all
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ceph-osd/+bug/1832867/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list