[Bug 2061701] Re: check_can_live_migrate_source fails with mkfs.vfat error: Label can be no longer than 11 characters
Taichi Maeda
2061701 at bugs.launchpad.net
Fri Sep 18 09:38:55 UTC 2026
** Description changed:
+ [ Impact ]
+
+ Live migration of an instance with an ephemeral disk fails when the
+ destination host needs to create a missing ephemeral backing file while
+ using QCOW2.
+
+ Nova uses a long name such as 'ephemeral_10_0706d66' as the filesystem
+ label. VFAT only supports labels up to 11 characters, so mkfs fails and
+ the live migration does not complete:
+
+ mkfs -t vfat -n ephemeral_10_0706d66 /var/lib/nova/instances/_base/ephemeral_10_0706d66
+ mkfs.vfat: Label can be no longer than 11 characters
+
+ The upstream fix is:
+
+ https://review.opendev.org/c/openstack/nova/+/947541
+
+ The fix uses a short name such as 'ephemeral0', while keeping the
+ backing file name unchanged:
+
+ mkfs -t vfat -n ephemeral0
+ /var/lib/nova/instances/_base/ephemeral_10_0706d66
+
+ I tested the bug in the following environments:
+
+ - Jammy/Yoga: Nova 3:25.2.1-0ubuntu2.11
+ - Noble/Caracal: Nova 3:29.2.0-0ubuntu1.8
+ - Noble/Epoxy: Nova 3:31.0.0-0ubuntu1.2~cloud3
+ - Resolute/Gazpacho: Nova 3:33.0.0-0ubuntu3.1
+ - Stonking/Hibiscus: Nova 3:34.0.0~rc1-0ubuntu1
+
+ The bug reproduced with the above error message on Jammy/Yoga,
+ Noble/Caracal and Noble/Epoxy.
+
+ For Resolute/Gazpacho, I used a single-node setup due to compatibility
+ issues. I deleted the backing file on the same host and hard-rebooted
+ the instance, which triggered the same execution path described in the
+ upstream change. I then confirmed that the bug did not reproduce.
+
+ For Stonking/Hibiscus, I also confirmed that the bug did not reproduce
+ with the same single-node setup, although I had to manually override the
+ broken unit file to run glance-api using WSGI.
+
+ The bug has been present in Nova for more than 12 years. The original
+ call to _create_ephemeral() in _create_images_and_backing() was added by
+ this commit in 2014:
+
+ https://review.opendev.org/c/openstack/nova/+/68658
+
+ In upstream, the fix was backported to all maintained and unmaintained
+ branches. The fix is available in Dalmatian 30.3.0 and Epoxy 31.3.1, and
+ in the initial releases of Flamingo and Gazpacho.
+
+ In Ubuntu, the fix is present in Resolute and Stonking (devel) for UA,
+ and in Flamingo and Gazpacho for UCA. The affected releases are Jammy
+ and Noble for UA, and Yoga, Caracal and Epoxy for UCA.
+
+ I applied the upstream patch and verified that the same live migration
+ succeeded. The missing ephemeral backing file was created with the short
+ label 'ephemeral0', and mkfs completed successfully.
+
+ [ Test Plan ]
+
+ The test requires an environment with two Nova compute hosts.
+
+ Configure nova-compute on both hosts to use QEMU, qcow2 and VFAT:
+
+ sudo apt update
+ sudo apt install -y crudini
+ sudo crudini --set /etc/nova/nova.conf libvirt virt_type qemu
+ sudo crudini --set /etc/nova/nova.conf libvirt images_type qcow2
+ sudo crudini --set /etc/nova/nova.conf libvirt cpu_mode custom
+ sudo crudini --set /etc/nova/nova.conf libvirt cpu_model qemu64
+ sudo crudini --set /etc/nova/nova.conf DEFAULT default_ephemeral_format vfat
+ sudo crudini --set /etc/nova/nova.conf DEFAULT debug true
+ sudo systemctl restart nova-compute
+
+ Create a test image:
+
+ curl -fL -o /tmp/cirros.img \
+ https://download.cirros-cloud.net/0.6.3/cirros-0.6.3-x86_64-disk.img
+
+ openstack image create testimg \
+ --file /tmp/cirros.img \
+ --disk-format qcow2 \
+ --container-format bare \
+ --private
+
+ Create a test flavour:
+
+ openstack flavor create testflv \
+ --ram 1024 \
+ --disk 5 \
+ --ephemeral 5 \
+ --vcpus 1
+
+ Launch a Nova instance:
+
+ openstack server create \
+ --image testimg \
+ --flavor testflv \
+ --network private \
+ --wait \
+ testsvr
+
+ Identify the host names of the source and destination hosts:
+
+ openstack hypervisor list
+ +--------------------------------------+-----------------------------+-----------------+---------------+-------+
+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+ +--------------------------------------+-----------------------------+-----------------+---------------+-------+
+ | 957ab3ee-e579-454b-a9ae-6c69ea0852c6 | juju-ecb72d-noble-caracal-9 | QEMU | 10.159.67.47 | up |
+ | f326dd68-279d-4897-88f7-074a656a18c2 | juju-ecb72d-noble-caracal-8 | QEMU | 10.159.67.105 | up |
+ +--------------------------------------+-----------------------------+-----------------+---------------+-------+
+
+ openstack server show testsvr -f value -c OS-EXT-SRV-ATTR:host
+ juju-ecb72d-noble-caracal-8
+
+ SOURCE_HOST=juju-ecb72d-noble-caracal-8
+ DEST_HOST=juju-ecb72d-noble-caracal-9 # Choose the other host
+
+ Request a live migration to the destination host:
+
+ export OS_COMPUTE_API_VERSION=2.30
+ openstack server migrate \
+ --live-migration \
+ --block-migration \
+ --host "$DEST_HOST" \
+ testsvr
+
+ Wait for the live migration to finish:
+
+ openstack server show testsvr \
+ -f value \
+ -c OS-EXT-STS:task_state \
+ -c OS-EXT-SRV-ATTR:host
+
+ SSH into the destination host and search for the mkfs error message:
+
+ sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
+ ...
+ 2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Command: mkfs -t vfat -n ephemeral_5_0706d66 /var/lib/nova/instances/_base/ephemeral_5_0706d66
+ 2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
+
+ Now apply the upstream patch and follow the same steps. Make sure to
+ create a test flavour with a different value for --ephemeral so that the
+ existing backing file is not reused:
+
+ # Change --ephemeral 5 to 6
+ openstack flavor create testflv2 \
+ --ram 1024 \
+ --disk 5 \
+ --ephemeral 6 \
+ --vcpus 1
+ openstack server create \
+ --image testimg \
+ --flavor testflv2 \
+ --network private \
+ --wait \
+ testsvr2
+
+ After following the rest of the steps, I verified that mkfs was called
+ with the short label and that the live migration succeeded with the
+ patch:
+
+ sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
+ ...
+ 2026-09-18 09:22:49.407 3788355 DEBUG oslo_concurrency.processutils [None req-d2e72a6c-00c1-4df2-b70b-cd3b167330b0 73d496ce27ea461a8e5f4a1397d90d91 409a8037cb4f414f9d62a5ac852fe243 - - 6adae1c4f64747f595b2980622166731 6adae1c4f64747f595b2980622166731] Running cmd (subprocess): mkfs -t vfat -n ephemeral0 /var/lib/nova/instances/_base/ephemeral_6_0706d66 execute /usr/lib/python3/dist-packages/oslo_concurrency/processutils.py:390
+ ...
+
+ [ Where problems could occur ]
+
+ The change only affects the filesystem label when Nova creates an
+ ephemeral disk or recreates a missing backing file. The backing file
+ name is unchanged.
+
+ The new label matches the short names already used during normal
+ instance creation. The upstream change also includes a unit test that
+ verifies that a recreated ephemeral backing file uses the short names.
+
+ A regression could cause failures on systems that depend on filesystem
+ labels to identify volumes, but this should be rare because the labels
+ are not unique.
+
+ [ Other Info ]
+
+ Although UCA Epoxy was initially marked Fix Released on the bug report,
+ the current UCA Epoxy package does not contain the fix. I have therefore
+ updated the status to In Progress.
+
+ Also note that a second live migration attempt to the same destination
+ host may appear to succeed because the first attempt leaves the backing
+ file in place. However, this is not a workaround because the backing
+ file is left unformatted:
+
+ sudo find /var/lib/nova/instances/_base \
+ -maxdepth 1 -type f \
+ -name 'ephemeral_*'
+ ...
+ /var/lib/nova/instances/_base/ephemeral_17_0706d66
+
+ sudo blkid /var/lib/nova/instances/_base/ephemeral_17_0706d66 && echo formatted || echo unformatted
+ unformatted
+
+ Original Bug Description Below
+ ===========
+
When testing NBD TLS live-migration the initial attempt failed. Looks
like a generated filesystem label is too long.
Subsequent attempts succeeded
$ nova-compute --version
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
27.2.1
oslo_messaging.rpc.client.RemoteError: Remote error: ProcessExecutionError Unexpected error while running command.
Command: mkfs -t vfat -n ephemeral_1_0706d66 /var/lib/nova/instances/_base/ephemeral_1_0706d66
Exit code: 1
Stdout: 'mkfs.fat 4.2 (2021-01-31)\n'
Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
** Summary changed:
- check_can_live_migrate_source fails with mkfs.vfat error: Label can be no longer than 11 characters
+ [SRU] check_can_live_migrate_source fails with mkfs.vfat error: Label can be no longer than 11 characters
** Description changed:
[ Impact ]
Live migration of an instance with an ephemeral disk fails when the
destination host needs to create a missing ephemeral backing file while
using QCOW2.
Nova uses a long name such as 'ephemeral_10_0706d66' as the filesystem
label. VFAT only supports labels up to 11 characters, so mkfs fails and
the live migration does not complete:
mkfs -t vfat -n ephemeral_10_0706d66 /var/lib/nova/instances/_base/ephemeral_10_0706d66
mkfs.vfat: Label can be no longer than 11 characters
The upstream fix is:
https://review.opendev.org/c/openstack/nova/+/947541
The fix uses a short name such as 'ephemeral0', while keeping the
backing file name unchanged:
mkfs -t vfat -n ephemeral0
/var/lib/nova/instances/_base/ephemeral_10_0706d66
I tested the bug in the following environments:
- Jammy/Yoga: Nova 3:25.2.1-0ubuntu2.11
- Noble/Caracal: Nova 3:29.2.0-0ubuntu1.8
- Noble/Epoxy: Nova 3:31.0.0-0ubuntu1.2~cloud3
- Resolute/Gazpacho: Nova 3:33.0.0-0ubuntu3.1
- Stonking/Hibiscus: Nova 3:34.0.0~rc1-0ubuntu1
The bug reproduced with the above error message on Jammy/Yoga,
Noble/Caracal and Noble/Epoxy.
For Resolute/Gazpacho, I used a single-node setup due to compatibility
issues. I deleted the backing file on the same host and hard-rebooted
the instance, which triggered the same execution path described in the
upstream change. I then confirmed that the bug did not reproduce.
For Stonking/Hibiscus, I also confirmed that the bug did not reproduce
with the same single-node setup, although I had to manually override the
broken unit file to run glance-api using WSGI.
The bug has been present in Nova for more than 12 years. The original
call to _create_ephemeral() in _create_images_and_backing() was added by
this commit in 2014:
https://review.opendev.org/c/openstack/nova/+/68658
In upstream, the fix was backported to all maintained and unmaintained
branches. The fix is available in Dalmatian 30.3.0 and Epoxy 31.3.1, and
in the initial releases of Flamingo and Gazpacho.
In Ubuntu, the fix is present in Resolute and Stonking (devel) for UA,
and in Flamingo and Gazpacho for UCA. The affected releases are Jammy
and Noble for UA, and Yoga, Caracal and Epoxy for UCA.
I applied the upstream patch and verified that the same live migration
succeeded. The missing ephemeral backing file was created with the short
label 'ephemeral0', and mkfs completed successfully.
[ Test Plan ]
The test requires an environment with two Nova compute hosts.
Configure nova-compute on both hosts to use QEMU, qcow2 and VFAT:
sudo apt update
sudo apt install -y crudini
sudo crudini --set /etc/nova/nova.conf libvirt virt_type qemu
sudo crudini --set /etc/nova/nova.conf libvirt images_type qcow2
sudo crudini --set /etc/nova/nova.conf libvirt cpu_mode custom
sudo crudini --set /etc/nova/nova.conf libvirt cpu_model qemu64
sudo crudini --set /etc/nova/nova.conf DEFAULT default_ephemeral_format vfat
sudo crudini --set /etc/nova/nova.conf DEFAULT debug true
sudo systemctl restart nova-compute
Create a test image:
curl -fL -o /tmp/cirros.img \
- https://download.cirros-cloud.net/0.6.3/cirros-0.6.3-x86_64-disk.img
+ https://download.cirros-cloud.net/0.6.3/cirros-0.6.3-x86_64-disk.img
openstack image create testimg \
- --file /tmp/cirros.img \
- --disk-format qcow2 \
- --container-format bare \
- --private
+ --file /tmp/cirros.img \
+ --disk-format qcow2 \
+ --container-format bare \
+ --private
Create a test flavour:
openstack flavor create testflv \
- --ram 1024 \
- --disk 5 \
- --ephemeral 5 \
- --vcpus 1
+ --ram 1024 \
+ --disk 5 \
+ --ephemeral 5 \
+ --vcpus 1
Launch a Nova instance:
openstack server create \
- --image testimg \
- --flavor testflv \
- --network private \
- --wait \
- testsvr
+ --image testimg \
+ --flavor testflv \
+ --network private \
+ --wait \
+ testsvr
Identify the host names of the source and destination hosts:
openstack hypervisor list
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
| 957ab3ee-e579-454b-a9ae-6c69ea0852c6 | juju-ecb72d-noble-caracal-9 | QEMU | 10.159.67.47 | up |
| f326dd68-279d-4897-88f7-074a656a18c2 | juju-ecb72d-noble-caracal-8 | QEMU | 10.159.67.105 | up |
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
openstack server show testsvr -f value -c OS-EXT-SRV-ATTR:host
juju-ecb72d-noble-caracal-8
SOURCE_HOST=juju-ecb72d-noble-caracal-8
DEST_HOST=juju-ecb72d-noble-caracal-9 # Choose the other host
Request a live migration to the destination host:
export OS_COMPUTE_API_VERSION=2.30
openstack server migrate \
- --live-migration \
- --block-migration \
- --host "$DEST_HOST" \
- testsvr
+ --live-migration \
+ --block-migration \
+ --host "$DEST_HOST" \
+ testsvr
Wait for the live migration to finish:
openstack server show testsvr \
- -f value \
- -c OS-EXT-STS:task_state \
- -c OS-EXT-SRV-ATTR:host
+ -f value \
+ -c OS-EXT-STS:task_state \
+ -c OS-EXT-SRV-ATTR:host
SSH into the destination host and search for the mkfs error message:
sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
...
2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Command: mkfs -t vfat -n ephemeral_5_0706d66 /var/lib/nova/instances/_base/ephemeral_5_0706d66
2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
Now apply the upstream patch and follow the same steps. Make sure to
create a test flavour with a different value for --ephemeral so that the
existing backing file is not reused:
# Change --ephemeral 5 to 6
openstack flavor create testflv2 \
- --ram 1024 \
- --disk 5 \
- --ephemeral 6 \
- --vcpus 1
+ --ram 1024 \
+ --disk 5 \
+ --ephemeral 6 \
+ --vcpus 1
openstack server create \
- --image testimg \
- --flavor testflv2 \
- --network private \
- --wait \
- testsvr2
+ --image testimg \
+ --flavor testflv2 \
+ --network private \
+ --wait \
+ testsvr2
After following the rest of the steps, I verified that mkfs was called
with the short label and that the live migration succeeded with the
patch:
sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
...
2026-09-18 09:22:49.407 3788355 DEBUG oslo_concurrency.processutils [None req-d2e72a6c-00c1-4df2-b70b-cd3b167330b0 73d496ce27ea461a8e5f4a1397d90d91 409a8037cb4f414f9d62a5ac852fe243 - - 6adae1c4f64747f595b2980622166731 6adae1c4f64747f595b2980622166731] Running cmd (subprocess): mkfs -t vfat -n ephemeral0 /var/lib/nova/instances/_base/ephemeral_6_0706d66 execute /usr/lib/python3/dist-packages/oslo_concurrency/processutils.py:390
...
[ Where problems could occur ]
The change only affects the filesystem label when Nova creates an
ephemeral disk or recreates a missing backing file. The backing file
name is unchanged.
The new label matches the short names already used during normal
instance creation. The upstream change also includes a unit test that
verifies that a recreated ephemeral backing file uses the short names.
A regression could cause failures on systems that depend on filesystem
labels to identify volumes, but this should be rare because the labels
are not unique.
[ Other Info ]
Although UCA Epoxy was initially marked Fix Released on the bug report,
the current UCA Epoxy package does not contain the fix. I have therefore
updated the status to In Progress.
Also note that a second live migration attempt to the same destination
host may appear to succeed because the first attempt leaves the backing
file in place. However, this is not a workaround because the backing
file is left unformatted:
sudo find /var/lib/nova/instances/_base \
- -maxdepth 1 -type f \
- -name 'ephemeral_*'
+ -maxdepth 1 -type f \
+ -name 'ephemeral_*'
...
/var/lib/nova/instances/_base/ephemeral_17_0706d66
sudo blkid /var/lib/nova/instances/_base/ephemeral_17_0706d66 && echo formatted || echo unformatted
unformatted
+
Original Bug Description Below
===========
When testing NBD TLS live-migration the initial attempt failed. Looks
like a generated filesystem label is too long.
Subsequent attempts succeeded
$ nova-compute --version
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
27.2.1
oslo_messaging.rpc.client.RemoteError: Remote error: ProcessExecutionError Unexpected error while running command.
Command: mkfs -t vfat -n ephemeral_1_0706d66 /var/lib/nova/instances/_base/ephemeral_1_0706d66
Exit code: 1
Stdout: 'mkfs.fat 4.2 (2021-01-31)\n'
Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
** Description changed:
[ Impact ]
Live migration of an instance with an ephemeral disk fails when the
destination host needs to create a missing ephemeral backing file while
using QCOW2.
Nova uses a long name such as 'ephemeral_10_0706d66' as the filesystem
label. VFAT only supports labels up to 11 characters, so mkfs fails and
the live migration does not complete:
mkfs -t vfat -n ephemeral_10_0706d66 /var/lib/nova/instances/_base/ephemeral_10_0706d66
mkfs.vfat: Label can be no longer than 11 characters
The upstream fix is:
https://review.opendev.org/c/openstack/nova/+/947541
The fix uses a short name such as 'ephemeral0', while keeping the
backing file name unchanged:
mkfs -t vfat -n ephemeral0
/var/lib/nova/instances/_base/ephemeral_10_0706d66
I tested the bug in the following environments:
- Jammy/Yoga: Nova 3:25.2.1-0ubuntu2.11
- Noble/Caracal: Nova 3:29.2.0-0ubuntu1.8
- Noble/Epoxy: Nova 3:31.0.0-0ubuntu1.2~cloud3
- Resolute/Gazpacho: Nova 3:33.0.0-0ubuntu3.1
- Stonking/Hibiscus: Nova 3:34.0.0~rc1-0ubuntu1
The bug reproduced with the above error message on Jammy/Yoga,
Noble/Caracal and Noble/Epoxy.
For Resolute/Gazpacho, I used a single-node setup due to compatibility
issues. I deleted the backing file on the same host and hard-rebooted
the instance, which triggered the same execution path described in the
upstream change. I then confirmed that the bug did not reproduce.
For Stonking/Hibiscus, I also confirmed that the bug did not reproduce
with the same single-node setup, although I had to manually override the
broken unit file to run glance-api using WSGI.
The bug has been present in Nova for more than 12 years. The original
call to _create_ephemeral() in _create_images_and_backing() was added by
this commit in 2014:
https://review.opendev.org/c/openstack/nova/+/68658
In upstream, the fix was backported to all maintained and unmaintained
branches. The fix is available in Dalmatian 30.3.0 and Epoxy 31.3.1, and
in the initial releases of Flamingo and Gazpacho.
In Ubuntu, the fix is present in Resolute and Stonking (devel) for UA,
and in Flamingo and Gazpacho for UCA. The affected releases are Jammy
and Noble for UA, and Yoga, Caracal and Epoxy for UCA.
I applied the upstream patch and verified that the same live migration
succeeded. The missing ephemeral backing file was created with the short
label 'ephemeral0', and mkfs completed successfully.
[ Test Plan ]
The test requires an environment with two Nova compute hosts.
Configure nova-compute on both hosts to use QEMU, qcow2 and VFAT:
sudo apt update
sudo apt install -y crudini
sudo crudini --set /etc/nova/nova.conf libvirt virt_type qemu
sudo crudini --set /etc/nova/nova.conf libvirt images_type qcow2
sudo crudini --set /etc/nova/nova.conf libvirt cpu_mode custom
sudo crudini --set /etc/nova/nova.conf libvirt cpu_model qemu64
sudo crudini --set /etc/nova/nova.conf DEFAULT default_ephemeral_format vfat
sudo crudini --set /etc/nova/nova.conf DEFAULT debug true
sudo systemctl restart nova-compute
Create a test image:
curl -fL -o /tmp/cirros.img \
https://download.cirros-cloud.net/0.6.3/cirros-0.6.3-x86_64-disk.img
openstack image create testimg \
--file /tmp/cirros.img \
--disk-format qcow2 \
--container-format bare \
--private
Create a test flavour:
openstack flavor create testflv \
--ram 1024 \
--disk 5 \
--ephemeral 5 \
--vcpus 1
Launch a Nova instance:
openstack server create \
--image testimg \
--flavor testflv \
--network private \
--wait \
testsvr
Identify the host names of the source and destination hosts:
openstack hypervisor list
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
| 957ab3ee-e579-454b-a9ae-6c69ea0852c6 | juju-ecb72d-noble-caracal-9 | QEMU | 10.159.67.47 | up |
| f326dd68-279d-4897-88f7-074a656a18c2 | juju-ecb72d-noble-caracal-8 | QEMU | 10.159.67.105 | up |
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
openstack server show testsvr -f value -c OS-EXT-SRV-ATTR:host
juju-ecb72d-noble-caracal-8
SOURCE_HOST=juju-ecb72d-noble-caracal-8
DEST_HOST=juju-ecb72d-noble-caracal-9 # Choose the other host
Request a live migration to the destination host:
export OS_COMPUTE_API_VERSION=2.30
openstack server migrate \
--live-migration \
--block-migration \
--host "$DEST_HOST" \
testsvr
Wait for the live migration to finish:
openstack server show testsvr \
-f value \
-c OS-EXT-STS:task_state \
-c OS-EXT-SRV-ATTR:host
SSH into the destination host and search for the mkfs error message:
sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
...
2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Command: mkfs -t vfat -n ephemeral_5_0706d66 /var/lib/nova/instances/_base/ephemeral_5_0706d66
2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
Now apply the upstream patch and follow the same steps. Make sure to
create a test flavour with a different value for --ephemeral so that the
existing backing file is not reused:
# Change --ephemeral 5 to 6
openstack flavor create testflv2 \
--ram 1024 \
--disk 5 \
--ephemeral 6 \
--vcpus 1
openstack server create \
--image testimg \
--flavor testflv2 \
--network private \
--wait \
testsvr2
After following the rest of the steps, I verified that mkfs was called
with the short label and that the live migration succeeded with the
patch:
sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
...
2026-09-18 09:22:49.407 3788355 DEBUG oslo_concurrency.processutils [None req-d2e72a6c-00c1-4df2-b70b-cd3b167330b0 73d496ce27ea461a8e5f4a1397d90d91 409a8037cb4f414f9d62a5ac852fe243 - - 6adae1c4f64747f595b2980622166731 6adae1c4f64747f595b2980622166731] Running cmd (subprocess): mkfs -t vfat -n ephemeral0 /var/lib/nova/instances/_base/ephemeral_6_0706d66 execute /usr/lib/python3/dist-packages/oslo_concurrency/processutils.py:390
...
[ Where problems could occur ]
The change only affects the filesystem label when Nova creates an
ephemeral disk or recreates a missing backing file. The backing file
name is unchanged.
The new label matches the short names already used during normal
instance creation. The upstream change also includes a unit test that
verifies that a recreated ephemeral backing file uses the short names.
A regression could cause failures on systems that depend on filesystem
labels to identify volumes, but this should be rare because the labels
are not unique.
[ Other Info ]
Although UCA Epoxy was initially marked Fix Released on the bug report,
the current UCA Epoxy package does not contain the fix. I have therefore
updated the status to In Progress.
Also note that a second live migration attempt to the same destination
host may appear to succeed because the first attempt leaves the backing
file in place. However, this is not a workaround because the backing
file is left unformatted:
sudo find /var/lib/nova/instances/_base \
-maxdepth 1 -type f \
-name 'ephemeral_*'
...
- /var/lib/nova/instances/_base/ephemeral_17_0706d66
-
- sudo blkid /var/lib/nova/instances/_base/ephemeral_17_0706d66 && echo formatted || echo unformatted
+ /var/lib/nova/instances/_base/ephemeral_11_0706d66
+
+ sudo blkid /var/lib/nova/instances/_base/ephemeral_11_0706d66 && echo formatted || echo unformatted
unformatted
-
Original Bug Description Below
===========
When testing NBD TLS live-migration the initial attempt failed. Looks
like a generated filesystem label is too long.
Subsequent attempts succeeded
$ nova-compute --version
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
27.2.1
oslo_messaging.rpc.client.RemoteError: Remote error: ProcessExecutionError Unexpected error while running command.
Command: mkfs -t vfat -n ephemeral_1_0706d66 /var/lib/nova/instances/_base/ephemeral_1_0706d66
Exit code: 1
Stdout: 'mkfs.fat 4.2 (2021-01-31)\n'
Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2061701
Title:
[SRU] check_can_live_migrate_source fails with mkfs.vfat error: Label
can be no longer than 11 characters
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive antelope series:
Won't Fix
Status in Ubuntu Cloud Archive bobcat series:
Won't Fix
Status in Ubuntu Cloud Archive caracal series:
In Progress
Status in Ubuntu Cloud Archive dalmatian series:
Won't Fix
Status in Ubuntu Cloud Archive epoxy series:
In Progress
Status in Ubuntu Cloud Archive flamingo series:
Fix Released
Status in Ubuntu Cloud Archive gazpacho series:
Fix Released
Status in Ubuntu Cloud Archive hibiscus series:
Fix Released
Status in Ubuntu Cloud Archive yoga series:
In Progress
Status in Ubuntu Cloud Archive zed series:
Won't Fix
Status in OpenStack Compute (nova):
Fix Released
Status in nova package in Ubuntu:
Fix Released
Status in nova source package in Jammy:
In Progress
Status in nova source package in Noble:
In Progress
Status in nova source package in Resolute:
Fix Released
Status in nova source package in Stonking:
Fix Released
Bug description:
[ Impact ]
Live migration of an instance with an ephemeral disk fails when the
destination host needs to create a missing ephemeral backing file
while using QCOW2.
Nova uses a long name such as 'ephemeral_10_0706d66' as the filesystem
label. VFAT only supports labels up to 11 characters, so mkfs fails
and the live migration does not complete:
mkfs -t vfat -n ephemeral_10_0706d66 /var/lib/nova/instances/_base/ephemeral_10_0706d66
mkfs.vfat: Label can be no longer than 11 characters
The upstream fix is:
https://review.opendev.org/c/openstack/nova/+/947541
The fix uses a short name such as 'ephemeral0', while keeping the
backing file name unchanged:
mkfs -t vfat -n ephemeral0
/var/lib/nova/instances/_base/ephemeral_10_0706d66
I tested the bug in the following environments:
- Jammy/Yoga: Nova 3:25.2.1-0ubuntu2.11
- Noble/Caracal: Nova 3:29.2.0-0ubuntu1.8
- Noble/Epoxy: Nova 3:31.0.0-0ubuntu1.2~cloud3
- Resolute/Gazpacho: Nova 3:33.0.0-0ubuntu3.1
- Stonking/Hibiscus: Nova 3:34.0.0~rc1-0ubuntu1
The bug reproduced with the above error message on Jammy/Yoga,
Noble/Caracal and Noble/Epoxy.
For Resolute/Gazpacho, I used a single-node setup due to compatibility
issues. I deleted the backing file on the same host and hard-rebooted
the instance, which triggered the same execution path described in the
upstream change. I then confirmed that the bug did not reproduce.
For Stonking/Hibiscus, I also confirmed that the bug did not reproduce
with the same single-node setup, although I had to manually override
the broken unit file to run glance-api using WSGI.
The bug has been present in Nova for more than 12 years. The original
call to _create_ephemeral() in _create_images_and_backing() was added
by this commit in 2014:
https://review.opendev.org/c/openstack/nova/+/68658
In upstream, the fix was backported to all maintained and unmaintained
branches. The fix is available in Dalmatian 30.3.0 and Epoxy 31.3.1,
and in the initial releases of Flamingo and Gazpacho.
In Ubuntu, the fix is present in Resolute and Stonking (devel) for UA,
and in Flamingo and Gazpacho for UCA. The affected releases are Jammy
and Noble for UA, and Yoga, Caracal and Epoxy for UCA.
I applied the upstream patch and verified that the same live migration
succeeded. The missing ephemeral backing file was created with the
short label 'ephemeral0', and mkfs completed successfully.
[ Test Plan ]
The test requires an environment with two Nova compute hosts.
Configure nova-compute on both hosts to use QEMU, qcow2 and VFAT:
sudo apt update
sudo apt install -y crudini
sudo crudini --set /etc/nova/nova.conf libvirt virt_type qemu
sudo crudini --set /etc/nova/nova.conf libvirt images_type qcow2
sudo crudini --set /etc/nova/nova.conf libvirt cpu_mode custom
sudo crudini --set /etc/nova/nova.conf libvirt cpu_model qemu64
sudo crudini --set /etc/nova/nova.conf DEFAULT default_ephemeral_format vfat
sudo crudini --set /etc/nova/nova.conf DEFAULT debug true
sudo systemctl restart nova-compute
Create a test image:
curl -fL -o /tmp/cirros.img \
https://download.cirros-cloud.net/0.6.3/cirros-0.6.3-x86_64-disk.img
openstack image create testimg \
--file /tmp/cirros.img \
--disk-format qcow2 \
--container-format bare \
--private
Create a test flavour:
openstack flavor create testflv \
--ram 1024 \
--disk 5 \
--ephemeral 5 \
--vcpus 1
Launch a Nova instance:
openstack server create \
--image testimg \
--flavor testflv \
--network private \
--wait \
testsvr
Identify the host names of the source and destination hosts:
openstack hypervisor list
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
| 957ab3ee-e579-454b-a9ae-6c69ea0852c6 | juju-ecb72d-noble-caracal-9 | QEMU | 10.159.67.47 | up |
| f326dd68-279d-4897-88f7-074a656a18c2 | juju-ecb72d-noble-caracal-8 | QEMU | 10.159.67.105 | up |
+--------------------------------------+-----------------------------+-----------------+---------------+-------+
openstack server show testsvr -f value -c OS-EXT-SRV-ATTR:host
juju-ecb72d-noble-caracal-8
SOURCE_HOST=juju-ecb72d-noble-caracal-8
DEST_HOST=juju-ecb72d-noble-caracal-9 # Choose the other host
Request a live migration to the destination host:
export OS_COMPUTE_API_VERSION=2.30
openstack server migrate \
--live-migration \
--block-migration \
--host "$DEST_HOST" \
testsvr
Wait for the live migration to finish:
openstack server show testsvr \
-f value \
-c OS-EXT-STS:task_state \
-c OS-EXT-SRV-ATTR:host
SSH into the destination host and search for the mkfs error message:
sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
...
2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Command: mkfs -t vfat -n ephemeral_5_0706d66 /var/lib/nova/instances/_base/ephemeral_5_0706d66
2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
Now apply the upstream patch and follow the same steps. Make sure to
create a test flavour with a different value for --ephemeral so that
the existing backing file is not reused:
# Change --ephemeral 5 to 6
openstack flavor create testflv2 \
--ram 1024 \
--disk 5 \
--ephemeral 6 \
--vcpus 1
openstack server create \
--image testimg \
--flavor testflv2 \
--network private \
--wait \
testsvr2
After following the rest of the steps, I verified that mkfs was called
with the short label and that the live migration succeeded with the
patch:
sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail
...
2026-09-18 09:22:49.407 3788355 DEBUG oslo_concurrency.processutils [None req-d2e72a6c-00c1-4df2-b70b-cd3b167330b0 73d496ce27ea461a8e5f4a1397d90d91 409a8037cb4f414f9d62a5ac852fe243 - - 6adae1c4f64747f595b2980622166731 6adae1c4f64747f595b2980622166731] Running cmd (subprocess): mkfs -t vfat -n ephemeral0 /var/lib/nova/instances/_base/ephemeral_6_0706d66 execute /usr/lib/python3/dist-packages/oslo_concurrency/processutils.py:390
...
[ Where problems could occur ]
The change only affects the filesystem label when Nova creates an
ephemeral disk or recreates a missing backing file. The backing file
name is unchanged.
The new label matches the short names already used during normal
instance creation. The upstream change also includes a unit test that
verifies that a recreated ephemeral backing file uses the short names.
A regression could cause failures on systems that depend on filesystem
labels to identify volumes, but this should be rare because the labels
are not unique.
[ Other Info ]
Although UCA Epoxy was initially marked Fix Released on the bug
report, the current UCA Epoxy package does not contain the fix. I have
therefore updated the status to In Progress.
Also note that a second live migration attempt to the same destination
host may appear to succeed because the first attempt leaves the
backing file in place. However, this is not a workaround because the
backing file is left unformatted:
sudo find /var/lib/nova/instances/_base \
-maxdepth 1 -type f \
-name 'ephemeral_*'
...
/var/lib/nova/instances/_base/ephemeral_11_0706d66
sudo blkid /var/lib/nova/instances/_base/ephemeral_11_0706d66 && echo formatted || echo unformatted
unformatted
Original Bug Description Below
===========
When testing NBD TLS live-migration the initial attempt failed. Looks
like a generated filesystem label is too long.
Subsequent attempts succeeded
$ nova-compute --version
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
27.2.1
oslo_messaging.rpc.client.RemoteError: Remote error: ProcessExecutionError Unexpected error while running command.
Command: mkfs -t vfat -n ephemeral_1_0706d66 /var/lib/nova/instances/_base/ephemeral_1_0706d66
Exit code: 1
Stdout: 'mkfs.fat 4.2 (2021-01-31)\n'
Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2061701/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list