[Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

Dimitri John Ledkov launchpad at surgut.co.uk
Thu Sep 13 13:30:51 UTC 2018 

Note to self.

       umount.target
           A special target unit that unmounts all mount and automount points on system shutdown.

           Mounts that shall be unmounted on system shutdown shall add Conflicts dependencies to this unit for their mount unit, which is implicitly done
           when DefaultDependencies=yes is set (the default).

Given that /media/root-ro is a pre-requisite for /, imho it should be
declared with DefaultDependencies=no.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /        ext4   defaults        0 0
    #LABEL=UEFI     /boot/efi       vfat    defaults        0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
      nomodeset iscsi_initiator=maas-enlist
      iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
      iscsi_target_port=3260 iscsi_initiator=maas-enlist
      ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
      root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
      overlayroot=tmpfs console=ttyS0
      ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
      # cat /etc/fstab
      #
      #  This fstab is in an overlay. The real one can be found at
      #  /media/root-ro/etc/fstab
      #  The original entry for '/' and other mounts have been updated to be placed
      #  under /media/root-ro.
      #  To permanently modify this (or any other file), you should change-root into
      #  a writable view of the underlying filesystem using:
      #      sudo overlayroot-chroot
      #
      #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
      /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
      #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
      LABEL=cloudimg-rootfs   /        ext4   defaults        0 0
      #LABEL=UEFI     /boot/efi       vfat    defaults        0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the initramfs
   mount -t tmpfs tmpfs-root /media/root-rw
   mkdir /media/root-rw/overlay-workdir/_
   mount --move $ROOTMNT /media/root-ro
   mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root
   mkdir $ROOTMNT/media/root-ro
   mkdir $ROOTMNT/media/root-rw
   mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
   mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

   # then, if 'ro' on the command line, it mounts /root read-only.
   mount -o remount,ro $ROOTMNT

  The script then exits, as ROOTMNT is now set up with a read-only mount
  of the overlayroot.  All the mounts it has done have been moved
  under ROOTMNT.

  On failure systemd reports:
     [  104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start
     [  104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start
     [  104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start
     [  104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided

  On successful boot, we can login and see:

  $  find /run/systemd/ -name "*.mount" | xargs ls -l
  -rw-r--r-- 1 root root 322 Aug 21 13:55 /run/systemd/generator/-.mount
  lrwxrwxrwx 1 root root  10 Aug 21 13:55 /run/systemd/generator/local-fs.target.requires/-.mount -> ../-.mount
  lrwxrwxrwx 1 root root  32 Aug 21 13:55 /run/systemd/units/invocation:dev-hugepages.mount -> b847fc2d06b54b1ab86a6f526d149522
  lrwxrwxrwx 1 root root  32 Aug 21 13:55 /run/systemd/units/invocation:dev-mqueue.mount -> f991d3bfe35d4e2bb9aa86677ff31d70
  lrwxrwxrwx 1 root root  32 Aug 21 13:55 /run/systemd/units/invocation:sys-fs-fuse-connections.mount -> a561df591a28437f990f94269fa08be9
  lrwxrwxrwx 1 root root  32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-config.mount -> 141e9bbaca9041e48458749e29bc58fb
  lrwxrwxrwx 1 root root  32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-debug.mount -> a406163f265c4b02b37e6f92ddbfc029

  $ for f in $(find /run/systemd/ -name '*.mount' -type f); do echo == $f ==; cat $f; done
  == /run/systemd/generator/-.mount ==
  # Automatically generated by systemd-fstab-generator

  [Unit]
  SourcePath=/etc/fstab
  Documentation=man:fstab(5) man:systemd-fstab-generator(8)
  Before=local-fs.target

  [Mount]
  Where=/
  What=/media/root-ro/
  Type=overlay
  Options=lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_

  --
  [1] https://git.launchpad.net/cloud-initramfs-tools/tree/overlayroot/scripts/init-bottom/overlayroot

  Related bugs:
   * bug 1723183: [overlayroot] transient systemd ordering issue when using overlayroot

  ProblemType: Bug
  DistroRelease: Ubuntu 18.10
  Package: systemd 237-3ubuntu10
  ProcVersionSignature: Ubuntu 4.17.0-6.7-generic 4.17.9
  Uname: Linux 4.17.0-6-generic x86_64
  ApportVersion: 2.20.10-0ubuntu7
  Architecture: amd64
  Date: Tue Aug 21 14:06:24 2018
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
  ProcEnviron:
   TERM=vt220
   PATH=(custom, no user)
   LANG=C.UTF-8
   SHELL=/bin/bash
  ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash
  SourcePackage: systemd
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 04/01/2014
  dmi.bios.vendor: SeaBIOS
  dmi.bios.version: 1.11.1-1
  dmi.chassis.type: 1
  dmi.chassis.vendor: QEMU
  dmi.chassis.version: pc-i440fx-cosmic
  dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.1-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-cosmic:cvnQEMU:ct1:cvrpc-i440fx-cosmic:
  dmi.product.name: Standard PC (i440FX + PIIX, 1996)
  dmi.product.version: pc-i440fx-cosmic
  dmi.sys.vendor: QEMU

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1788188/+subscriptions

More information about the foundations-bugs mailing list