[Bug 1718966] Re: Cannot install snaps on Ubuntu 14.04 with /var on its own partition
Rafael David Tinoco
rafael.tinoco at canonical.com
Wed Sep 27 20:50:57 UTC 2017
I was also able to reproduce this, here are my notes for now:
## /etc/fstab
LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 1
LABEL=TESTE /teste ext4 defaults 0 1
Right after boot:
inaddy at trustylivepatch:~$ systemctl list-units --all | grep fsck
systemd-...el-LVVAR.service error inactive dead systemd-fsck at dev-disk-by\x2dlabel-LVVAR.service
systemd-...el-TESTE.service error inactive dead systemd-fsck at dev-disk-by\x2dlabel-TESTE.service
This indicates that UPSTART is the one mounting the block devices, NOT
SYSTEMD using its mount units. SNAPS are mounting the SQUASH filesystems
using SYSTEMD UNITS, despite UPSTART scripts.
It is likely that this wasn't noticed, on systems mounting "/" only,
because the "-.mount" SYSTEMD UNIT doesn't depend on "systemd-
fsck at .service" unit, it depends only on "systemd-fsck-root.service", non
existent in TRUSTY's SYSTEMD version. Probably this made SYSTEMD to act
like no error existed.
For SYSTEMD mount units to work, it is needed that no fsck unit error
exists - like when having /var or any other mounting besides root
filesystem - allowing all SYSTEMD units created by snappy to work.
Comparing default setups for TRUSTY and ZESTY:
--------
## TRUSTY
$ dpkg -L systemd | grep fsck
/lib/systemd/systemd-fsck
$ systemctl list-units --all | grep fsck
systemd-...el-LVVAR.service error inactive dead systemd-fsck at dev-disk-by\x2dlabel-LVVAR.service
systemd-...el-TESTE.service error inactive dead systemd-fsck at dev-disk-by\x2dlabel-TESTE.service
$ systemctl list-units --all | grep mount
-.mount loaded active mounted /
teste.mount loaded active mounted /teste
var.mount loaded active mounted /var
umount.target loaded inactive dead Unmount All Filesystems
## ZESTY
$ dpkg -L systemd | grep fsck
/lib/systemd/system/systemd-fsck-root.service
/lib/systemd/system/systemd-fsck at .service
/lib/systemd/system/systemd-fsckd.service
/lib/systemd/system/systemd-fsckd.socket
/lib/systemd/systemd-fsck
/lib/systemd/systemd-fsckd
$ systemctl list-unit-files | grep fsck
systemd-fsck-root.service static
systemd-fsck at .service static
systemd-fsck at dev-disk-by\x2dlabel-TESTE.service static
systemd-fsckd.service static
systemd-fsckd.socket static
$ systemctl list-unit-files | grep mount
-.mount generated
home-inaddy-work.mount generated
mnt.mount static
mountall.service masked
umountfs.service masked
umountroot.service masked
umount.target static
--------
SYSTEMD in TRUSTY was treated differently for FSCK. TRUSTY's version
contains systemd-fsck but not systemd-fsckd, the daemon responsible for
consolidating all fsck information for SYSTEMD journal. It is also clear
that TRUSTY did not include any unit file for systemd-fsck at .service,
that might still be considered for the automatically generated mount
unit files.
You can reproduce this by trying to use TRUSTY SYSTEMD mount units:
--------
## TRUSTY
$ mount /teste
$ mount | grep teste
/dev/sdb2 on /teste type ext4 (rw)
$ systemctl status teste.mount
teste.mount - /teste
Loaded: loaded (/etc/fstab)
Active: active (mounted) since Wed 2017-09-27 17:27:21 BRT; 10s ago
Where: /teste
What: /dev/sdb2
Process: 1754 ExecUnmount=/bin/umount /teste (code=exited, status=0/SUCCESS)
$ systemctl stop teste.mount
$ systemctl status teste.mount
teste.mount - /teste
Loaded: loaded (/etc/fstab)
Active: inactive (dead) since Wed 2017-09-27 17:27:33 BRT; 2s ago
Where: /teste
What: /dev/disk/by-label/TESTE
Process: 1778 ExecUnmount=/bin/umount /teste (code=exited, status=0/SUCCESS)
$ mount | grep teste
$ systemctl start teste.mount
Failed to issue method call: Unit systemd-fsck at dev-disk-by\x2dlabel-TESTE.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck at dev-disk-by\x2dlabel-TESTE.service' for details.
--------
Also, in TRUSTY, SYSTEMD creates the mount units based on /etc/fstab
entries, just like the recent SYSTEMD does, but on them THERE IS NO
setting for fsck dependencies (Requires/After), and, still, it appears
to be considering those fsck dependencies when you try to mount a
".mount" unit, based on errors I showed above for "systemctl start
XXX.mount".
--------
## TRUSTY
$ cat var.mount
# Automatically generated by systemd-fstab-generator
[Unit]
SourcePath=/etc/fstab
DefaultDependencies=no
After=local-fs-pre.target
Conflicts=umount.target
Before=umount.target
Before=local-fs.target
[Mount]
What=/dev/disk/by-label/LVVAR
Where=/var
Type=ext4
FsckPassNo=1
## ZESTY
$ systemctl edit --full mnt.mount
[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target
Requires=systemd-fsck at dev-disk-by\x2dlabel-TESTE.service
After=systemd-fsck at dev-disk-by\x2dlabel-TESTE.service
[Mount]
What=/dev/disk/by-label/TESTE
Where=/mnt
Type=ext4
--------
Unfortunately creating the fsck service unit didn't seem to help for
TRUSTY, and, according to this:
$ for file in `dpkg -L systemd`; do [ -f $file ] && grep systemd-fsck $file; done
Binary file /lib/systemd/systemd matches
Binary file /bin/systemd matches
It is extremely likely that systemd-fsck at .service was hardcoded as a
dependency for systemd mount unit files, and the fsck units systemd-
fsck@<DISK>.service are NOT BEING CREATED automatically. This is the
real problem to be fixed IMO.
This case should be targeted to both: snapd (trusty) AND systemd
(trusty)
** Changed in: snapd
Assignee: (unassigned) => Rafael David Tinoco (inaddy)
** Changed in: snapd
Status: New => Confirmed
** Also affects: systemd (Ubuntu)
Importance: Undecided
Status: New
** Changed in: systemd (Ubuntu)
Assignee: (unassigned) => Rafael David Tinoco (inaddy)
** Changed in: systemd (Ubuntu)
Importance: Undecided => Medium
** Changed in: systemd (Ubuntu)
Status: New => In Progress
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1718966
Title:
Cannot install snaps on Ubuntu 14.04 with /var on its own partition
Status in snapd:
Confirmed
Status in systemd package in Ubuntu:
In Progress
Status in systemd source package in Trusty:
New
Bug description:
Installing snaps is not possible on Ubuntu 14.04 when having /var on
its own partition, whether its LVM or not.
The issue is with the core snap being unable to mount:
The error with /var isolated and using LVM:
root at ubuntu:~# snap install canonical-livepatch
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck at dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck at dev-mapper-vg00\x2dvarvol.service' for details.
)
The error with /var isolated but without using LVM:
root at ubuntu:~# snap install canonical-livepatch
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck at dev-disk-by\x2duuid-7383abd2\x2d019c\x2d46c2\x2d8b36\x2d34633cc8f3ca.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck at dev-disk-by\x2duuid-7383abd2\x2d019c\x2d46c2\x2d8b36\x2d34633cc8f3ca.service' for details.
)
The same error happens if I try to install the hello-world snap (with
LVM in this example):
root at ubuntu:~# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck at dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck at dev-mapper-vg00\x2dvarvol.service' for details.
)
I cannot reproduce the issue in Ubuntu 16.04.
I couldn't reproduce this issue by using the Ubuntu 14.04 cloud image
which doesn't isolate /var on its own partition. I tried adding a
secondary disk to that cloud image VM and create a dummy VG and LV,
but couldn't reproduce the issue.
Also could not reproduce using Ubuntu 14.04 (with LVM or not) but with
only a / partition and swap.
Steps to reproduce:
===================
# Install Ubuntu 14.04 in KVM (I used the 14.04.4 server iso) and
configure /, /var and swap on their own partitions (with LVM or not,
the issue happens in both situations).
root at ubuntu:~# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
rootvol vg00 -wi-ao--- 3.72g
swap vg00 -wi-ao--- 3.72g
varvol vg00 -wi-ao--- 3.72g
root at ubuntu:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 484M 4.0K 484M 1% /dev
tmpfs 100M 988K 99M 1% /run
/dev/dm-0 3.7G 1.7G 1.8G 49% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 497M 0 497M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/mapper/vg00-varvol 3.7G 716M 2.8G 21% /var
# Upgrade system, install snapd and reboot
root at ubuntu:~# apt update
root at ubuntu:~# apt upgrade -y
root at ubuntu:~# apt install -y snapd
root at ubuntu:~# reboot
# After reboot, check kernel version and try to install the canonical-livepatch snap:
root at ubuntu:~# uname -a
Linux ubuntu 4.4.0-96-generic #119~14.04.1-Ubuntu SMP Wed Sep 13 08:40:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root at ubuntu:~# snap list
No snaps are installed yet. Try "snap install hello-world".
root at ubuntu:~# snap install canonical-livepatch
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck at dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck at dev-mapper-vg00\x2dvarvol.service' for details.
)
To manage notifications about this bug go to:
https://bugs.launchpad.net/snapd/+bug/1718966/+subscriptions
More information about the foundations-bugs
mailing list