[Bug 2061701] Re: [SRU] check_can_live_migrate_source fails with mkfs.vfat error: Label can be no longer than 11 characters
Taichi Maeda
2061701 at bugs.launchpad.net
Tue Oct 6 09:09:19 UTC 2026
Thanks Timo. I completed SRU verification for Noble/Caracal:
Test environment:
- Charmed OpenStack Noble/Caracal
- Two nova-compute hosts using QEMU and QCOW2
- VFAT as the default ephemeral disk format
My original report reproduced the bug with Nova 3:29.2.0-0ubuntu1.8. I
then installed Nova 3:29.2.0-0ubuntu1.9 from noble-proposed and repeated
the same live migration test:
$ openstack hypervisor list
+--------------------------------------+--------------------------------------+-----------------+--------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+--------------------------------------+-----------------+--------------+-------+
| 7777747f-b883-4082-85bb-7bd0aadbfa3e | juju-422ac8-noble-caracal- | QEMU | 10.159.67.29 | up |
| | proposed2-9 | | | |
| e59e4df7-e78e-4dd6-b22b-8206ec939d65 | juju-422ac8-noble-caracal- | QEMU | 10.159.67.19 | up |
| | proposed2-8 | | | |
+--------------------------------------+--------------------------------------+-----------------+--------------+-------+
$ openstack server show testsvr -f value -c OS-EXT-SRV-ATTR:host
juju-422ac8-noble-caracal-proposed2-9
$ export OS_COMPUTE_API_VERSION=2.30
$ openstack server migrate \
--live-migration \
--block-migration \
--host juju-422ac8-noble-caracal-proposed2-8 \
testsvr
$ openstack server show testsvr -f value -c OS-EXT-SRV-ATTR:host
juju-422ac8-noble-caracal-proposed2-8
The live migration completed successfully. On the destination host, Nova
created the ephemeral backing file using the short filesystem label:
$ sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log
2026-10-06 08:45:25.867 70730 DEBUG oslo_concurrency.processutils [None req-ad61f3b1-b7e8-4e1c-88bb-c994bc11cb55 a27b307fc2c34f65a6a7089b4a100387 7d2979840fa54dcb8975fc50898d73b7 - - 0f0d335453c04c108aec607880fc6908 0f0d335453c04c108aec607880fc6908] Running cmd (subprocess): mkfs -t vfat -n ephemeral0 /var/lib/nova/instances/_base/ephemeral_5_0706d66 execute /usr/lib/python3/dist-packages/oslo_concurrency/processutils.py:390
2026-10-06 08:45:25.965 70730 DEBUG oslo_concurrency.processutils [None req-ad61f3b1-b7e8-4e1c-88bb-c994bc11cb55 a27b307fc2c34f65a6a7089b4a100387 7d2979840fa54dcb8975fc50898d73b7 - - 0f0d335453c04c108aec607880fc6908 0f0d335453c04c108aec607880fc6908] CMD "mkfs -t vfat -n ephemeral0 /var/lib/nova/instances/_base/ephemeral_5_0706d66" returned: 0 in 0.098s execute /usr/lib/python3/dist-packages/oslo_concurrency/processutils.py:428
The proposed package fixes the issue on Noble.
** Changed in: cloud-archive/caracal
Status: In Progress => Fix Committed
** Changed in: cloud-archive/yoga
Status: In Progress => Fix Committed
** Changed in: cloud-archive/epoxy
Status: In Progress => Fix Committed
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2061701
Title:
[SRU] check_can_live_migrate_source fails with mkfs.vfat error: Label
can be no longer than 11 characters
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive antelope series:
Won't Fix
Status in Ubuntu Cloud Archive bobcat series:
Won't Fix
Status in Ubuntu Cloud Archive caracal series:
Fix Committed
Status in Ubuntu Cloud Archive dalmatian series:
Won't Fix
Status in Ubuntu Cloud Archive epoxy series:
Fix Committed
Status in Ubuntu Cloud Archive flamingo series:
Fix Released
Status in Ubuntu Cloud Archive gazpacho series:
Fix Released
Status in Ubuntu Cloud Archive hibiscus series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
Fix Committed
Status in Ubuntu Cloud Archive zed series:
Won't Fix
Status in OpenStack Compute (nova):
Fix Released
Status in nova package in Ubuntu:
Fix Released
Status in nova source package in Jammy:
Fix Committed
Status in nova source package in Noble:
Fix Committed
Status in nova source package in Resolute:
Fix Released
Status in nova source package in Stonking:
Fix Released
Bug description:
[Impact]
Live migration of an instance with an ephemeral disk fails when the
destination host needs to create a missing ephemeral backing file
while using QCOW2.
Nova uses a long name such as 'ephemeral_10_0706d66' as the filesystem
label. VFAT only supports labels up to 11 characters, so mkfs fails
and the live migration does not complete:
mkfs -t vfat -n ephemeral_10_0706d66 /var/lib/nova/instances/_base/ephemeral_10_0706d66
mkfs.vfat: Label can be no longer than 11 characters
The upstream fix is:
https://review.opendev.org/c/openstack/nova/+/947541
The fix uses a short name such as 'ephemeral0', while keeping the
backing file name unchanged:
mkfs -t vfat -n ephemeral0
/var/lib/nova/instances/_base/ephemeral_10_0706d66
I tested the bug in the following environments:
- Jammy/Yoga: Nova 3:25.2.1-0ubuntu2.11
- Noble/Caracal: Nova 3:29.2.0-0ubuntu1.8
- Noble/Epoxy: Nova 3:31.0.0-0ubuntu1.2~cloud3
- Resolute/Gazpacho: Nova 3:33.0.0-0ubuntu3.1
- Stonking/Hibiscus: Nova 3:34.0.0~rc1-0ubuntu1
The bug reproduced with the above error message on Jammy/Yoga,
Noble/Caracal and Noble/Epoxy.
For Resolute/Gazpacho, I used a single-node setup due to compatibility
issues. I deleted the backing file on the same host and hard-rebooted
the instance, which triggered the same execution path described in the
upstream change. I then confirmed that the bug did not reproduce.
For Stonking/Hibiscus, I also confirmed that the bug did not reproduce
with the same single-node setup, although I had to manually override
the broken unit file to run glance-api using WSGI.
The bug has been present in Nova for more than 12 years. The original
call to _create_ephemeral() in _create_images_and_backing() was added
by this commit in 2014:
https://review.opendev.org/c/openstack/nova/+/68658
In upstream, the fix was backported to all maintained and unmaintained
branches. The fix is available in Dalmatian 30.3.0 and Epoxy 31.3.1,
and in the initial releases of Flamingo and Gazpacho.
In Ubuntu, the fix is present in Resolute and Stonking (devel) for UA,
and in Flamingo and Gazpacho for UCA. The affected releases are Jammy
and Noble for UA, and Yoga, Caracal and Epoxy for UCA.
I applied the upstream patch and verified that the same live migration
succeeded. The missing ephemeral backing file was created with the
short label 'ephemeral0', and mkfs completed successfully.
[Test Case]
The test requires an environment with two Nova compute hosts.
Configure nova-compute on both hosts to use QEMU, qcow2 and VFAT:
sudo apt update
sudo apt install -y crudini
sudo crudini --set /etc/nova/nova.conf libvirt virt_type qemu
sudo crudini --set /etc/nova/nova.conf libvirt images_type qcow2
sudo crudini --set /etc/nova/nova.conf libvirt cpu_mode custom
sudo crudini --set /etc/nova/nova.conf libvirt cpu_model qemu64
sudo crudini --set /etc/nova/nova.conf DEFAULT default_ephemeral_format vfat
sudo crudini --set /etc/nova/nova.conf DEFAULT debug true
sudo systemctl restart nova-compute
Create a test image:
curl -fL -o /tmp/cirros.img \
https://download.cirros-cloud.net/0.6.3/cirros-0.6.3-x86_64-disk.img
openstack image create testimg \
--file /tmp/cirros.img \
--disk-format qcow2 \
--container-format bare \
--private
Create a test flavour:
openstack flavor create testflv \
--ram 1024 \
--disk 5 \
--ephemeral 5 \
--vcpus 1
Launch a Nova instance:
openstack server create \
--image testimg \
--flavor testflv \
--network private \
--wait \
testsvr
Identify the host names of the source and destination hosts:
openstack hypervisor list
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
| 957ab3ee-e579-454b-a9ae-6c69ea0852c6 | juju-ecb72d-noble-caracal-9 | QEMU | 10.159.67.47 | up |
| f326dd68-279d-4897-88f7-074a656a18c2 | juju-ecb72d-noble-caracal-8 | QEMU | 10.159.67.105 | up |
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
openstack server show testsvr -f value -c OS-EXT-SRV-ATTR:host
juju-ecb72d-noble-caracal-8
SOURCE_HOST=juju-ecb72d-noble-caracal-8
DEST_HOST=juju-ecb72d-noble-caracal-9 # Choose the other host
Request a live migration to the destination host:
export OS_COMPUTE_API_VERSION=2.30
openstack server migrate \
--live-migration \
--block-migration \
--host "$DEST_HOST" \
testsvr
Wait for the live migration to finish:
openstack server show testsvr \
-f value \
-c OS-EXT-STS:task_state \
-c OS-EXT-SRV-ATTR:host
SSH into the destination host and search for the mkfs error message:
sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
...
2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Command: mkfs -t vfat -n ephemeral_5_0706d66 /var/lib/nova/instances/_base/ephemeral_5_0706d66
2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
Now apply the upstream patch and follow the same steps. Make sure to
create a test flavour with a different value for --ephemeral so that
the existing backing file is not reused:
# Change --ephemeral 5 to 6
openstack flavor create testflv2 \
--ram 1024 \
--disk 5 \
--ephemeral 6 \
--vcpus 1
openstack server create \
--image testimg \
--flavor testflv2 \
--network private \
--wait \
testsvr2
After following the rest of the steps, I verified that mkfs was called
with the short label and that the live migration succeeded with the
patch:
sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
...
2026-09-18 09:22:49.407 3788355 DEBUG oslo_concurrency.processutils [None req-d2e72a6c-00c1-4df2-b70b-cd3b167330b0 73d496ce27ea461a8e5f4a1397d90d91 409a8037cb4f414f9d62a5ac852fe243 - - 6adae1c4f64747f595b2980622166731 6adae1c4f64747f595b2980622166731] Running cmd (subprocess): mkfs -t vfat -n ephemeral0 /var/lib/nova/instances/_base/ephemeral_6_0706d66 execute /usr/lib/python3/dist-packages/oslo_concurrency/processutils.py:390
...
[Regression Potential]
The change only affects the filesystem label when Nova creates an
ephemeral disk or recreates a missing backing file. The backing file
name is unchanged.
A regression could prevent a new ephemeral disk from being formatted,
or could affect guest software that depends on the filesystem label.
[Discussion]
Although UCA Epoxy was initially marked Fix Released on the bug
report, the current UCA Epoxy package does not contain the fix. I have
therefore updated the status to In Progress.
Also note that a second live migration attempt to the same destination
host may appear to succeed because the first attempt leaves the
backing file in place. However, this is not a workaround because the
backing file is left unformatted:
sudo find /var/lib/nova/instances/_base \
-maxdepth 1 -type f \
-name 'ephemeral_*'
...
/var/lib/nova/instances/_base/ephemeral_11_0706d66
sudo blkid /var/lib/nova/instances/_base/ephemeral_11_0706d66 && echo formatted || echo unformatted
unformatted
Original Bug Description Below
===========
When testing NBD TLS live-migration the initial attempt failed. Looks
like a generated filesystem label is too long.
Subsequent attempts succeeded
$ nova-compute --version
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
27.2.1
oslo_messaging.rpc.client.RemoteError: Remote error: ProcessExecutionError Unexpected error while running command.
Command: mkfs -t vfat -n ephemeral_1_0706d66 /var/lib/nova/instances/_base/ephemeral_1_0706d66
Exit code: 1
Stdout: 'mkfs.fat 4.2 (2021-01-31)\n'
Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2061701/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list