[Bug 1741364] Re: ceph ephemeral info not updated during live migrate
Erlon R. Cruz
1741364 at bugs.launchpad.net
Tue Oct 6 19:46:49 UTC 2026
** Also affects: cloud-archive
Importance: Undecided
Status: New
** Also affects: nova (Ubuntu)
Importance: Undecided
Status: New
** Also affects: nova (Ubuntu Stonking)
Importance: Undecided
Status: New
** Also affects: nova (Ubuntu Noble)
Importance: Undecided
Status: New
** Also affects: nova (Ubuntu Resolute)
Importance: Undecided
Status: New
** Changed in: nova (Ubuntu Stonking)
Status: New => Fix Released
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1741364
Title:
ceph ephemeral info not updated during live migrate
Status in Ubuntu Cloud Archive:
New
Status in Ubuntu Cloud Archive dalmatian series:
New
Status in Ubuntu Cloud Archive epoxy series:
New
Status in Ubuntu Cloud Archive flamingo series:
New
Status in OpenStack Compute (nova):
Fix Released
Status in nova package in Ubuntu:
Fix Released
Status in nova source package in Noble:
New
Status in nova source package in Resolute:
New
Status in nova source package in Stonking:
Fix Released
Bug description:
[Impact]
With Ceph RBD instance disks ([libvirt] images_type = rbd), the domain XML
embeds the Ceph monitor addresses valid at spawn time. On live migration
nova refreshes Cinder volume sources but copies the nova-managed RBD disks
(root/ephemeral/config drive) unchanged. After monitors are replaced or
re-addressed, the destination gets a stale monitor list: if no old monitor
is reachable the live migration fails, otherwise the stale list is carried
forward. Ceph monitor maintenance therefore blocks draining hypervisors
without downtime (cold migration, shelve or hard reboot).
[Test Plan]
On a deployment with 2+ computes using images_type=rbd and 1+ ceph-
mons:
1. Boot an image-backed instance (optionally attach an RBD Cinder volume)
and note the <host> list from `virsh dumpxml` for vms/<uuid>_disk.
2. Replace all ceph-mons (e.g. juju add-unit ceph-mon -n 3, then remove
the original units); confirm with `ceph mon dump` and ceph.conf.
3. openstack server migrate --live-migration --wait <vm>
Current packages: migration fails (destination cannot open the image).
With -proposed: migration succeeds and `virsh dumpxml` on the
destination lists exactly the monitors from `ceph mon dump`; the Cinder
volume disk is unchanged and I/O works on both disks.
4. Regression: live migrate a non-RBD (qcow2) and a boot-from-volume
instance; cold migrate and evacuate an RBD instance.
[Where problems could occur]
* Only live migration of images_type=rbd instances is affected; other
backends return early.
* The source compute now runs `ceph mon dump` on each RBD live migration;
if it fails (unreachable mons, keyring issues) the migration fails where
a still-valid stale list may previously have worked.
* Cinder volumes are skipped by heuristic (alias "ua-*" or a <serial>).
A volume with neither would get nova's monitor list, which breaks it if
Cinder uses a different Ceph cluster.
[Other Info]
* Per agreement with the Nova developers, this fix will not be backported
to upstream branches older than "Gazpacho", and will leave only on
Ubuntu packages.
* For 2025.2 and older the regression-test fixture was adjusted (tests
only: no <alias> in the fake network-disk XML).
* Unit and functional tests pass on all patched branches, except two
pre-existing Python 3.12 failures on 2024.1 that also fail unpatched.
---
rbdimages information isn't updated during a live migration.
I have a user that has updated their ceph cluster and added a new monitor IP.
He adds a new monitor to his ceph cluster and updates ceph.conf on all of the openstack nodes to reflect the new monitor IP.
They have running VMs that are boot from ceph volume and also has attached ceph volumes.
He does a live migration to try and get nova to update the libvirt.xml and it seems that only the volumes section is updated, not the vms section.
This is a copy of xml after the live migrate.
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='nova'>
<secret type='ceph' uuid='820ccd0b-b180-4528-93ed-76ae82edf832'/>
</auth>
<source protocol='rbd' name='vms/3b97914e-3f9b-410a-b3d9-6c1a83244136_disk'> <-- this one is NOT changed, old ips
<host name='192.168.200.12' port='6789'/>
<host name='192.168.200.14' port='6789'/>
<host name='192.168.200.24' port='6789'/>
<host name='192.168.240.17' port='6789'/>
<host name='192.168.240.23' port='6789'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='nova'>
<secret type='ceph' uuid='820ccd0b-b180-4528-93ed-76ae82edf832'/>
</auth>
<source protocol='rbd' name='volumes/volume-6d04520d-0029-499c-af81-516a7ba37a54'> <-- this one is changed, new ips
<host name='192.168.200.12' port='6789'/>
<host name='192.168.200.14' port='6789'/>
<host name='192.168.200.24' port='6789'/>
<host name='192.168.210.15' port='6789'/>
<host name='192.168.240.17' port='6789'/>
<host name='192.168.240.23' port='6789'/>
</source>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<serial>6d04520d-0029-499c-af81-516a7ba37a54</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1741364/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list