[ubuntu-cloud] Cloud-config and mount points

[Bo Shi]

Results in case anyone else is interested.  Given a #cloud-config consisting of:

    mounts:
     - [ ephemeral0, /var/log, auto, "defaults,noexec" ]
     - [ ephemeral1, /mnt, auto, "defaults,noexec" ]

    # Re-order cloud_init_modules and cloud_config_modules
    # so that "rsyslog" is run after "mounts"

    cloud_init_modules:
     - bootcmd
     - resizefs
     - mounts
     - set_hostname
     - update_hostname
     - update_etc_hosts
     - ca-certs
     - rsyslog
     - ssh

    cloud_config_modules:
     - ssh-import-id
     - locale
     - set-passwords
     - grub-dpkg
     - apt-pipelining
     - apt-update-upgrade
     - landscape
     - timezone
     - puppet
     - chef
     - salt-minion
     - mcollective
     - disable-ec2-metadata
     - runcmd
     - byobu


The resulting instance appears to be working great.  Upon inspection of the
files that got overwritten, it isn't _too_ bad - just the initial
cloud-init.log.  And there are ways of getting at it.


    ubuntu at ip-10-35-59-91:~$ sudo mount --bind / /tmp/root-bind-view/


    ubuntu at ip-10-35-59-91:~$ ls -Rl /tmp/root-bind-view/var/log
    /tmp/root-bind-view/var/log:
    total 8
    -rw-r--r-- 1 syslog adm  2933 Jul 12 00:56 cloud-init.log
    drwxr-xr-x 2 root   root 4096 Jul 12 00:56 landscape

    /tmp/root-bind-view/var/log/landscape:
    total 0
    -rw-r--r-- 1 root root 0 Jul 12 00:56 sysinfo.log
    ubuntu at ip-10-35-59-91:~$ lsof /tmp/root-bind-view/var/log/


It also appears that we don't have any weird open file descriptors accessing
the old /var/log/ directory... another positive!

    ubuntu at ip-10-35-59-91:~$ lsof /tmp/root-bind-view/var/log/
    ubuntu at ip-10-35-59-91:~$


Some context; we're using the Canonical AMI's as the base distribution to build
our own images in our continuous integration pipeline.  One concern is that
overriding cloud_init_modules and cloud_config_modules will make upgrading more
error prone as we will need to manually check that the default listing has not
changed.


On Wed, Jul 11, 2012 at 5:28 PM, Bo Shi <bo at simpler.co> wrote:
> Thanks Scott, I'll give it a try.  As for ideas/code, I'm not
> qualified to make any statement as I don't even know where the code
> lives for the init code.  I'll try to report back when I get some time
> to dive in.
>
> On Wed, Jul 11, 2012 at 10:38 AM, Scott Moser <smoser at ubuntu.com> wrote:
>> On Tue, 10 Jul 2012, Bo Shi wrote:
>>
>>> Hi,
>>>
>>> On an AWS EC2 instance, I'd like to switch /var/log from the root disk
>>> to an ephemeral disk.  I can successfully change the mount point using
>>> cloud-config:
>>>
>>>    mounts:
>>>      - [ ephemeral0, /var/log, auto, "defaults,noexec" ]
>>>
>>>
>>> One problem with this approach is that the actual mounts appear to be
>>> occurring *after* rsyslog is started which blows away any existing
>>> logs.  Does anyone know of a workaround?
>>
>>
>> At times I've considered trying to make the mounts try to copy data
>> across, but it seemed complex.  Ie, in the case given above, the code
>> would have to know that rsyslog has to be restarted because it has handles
>> to /var/log.
>>
>> That said, I think you might be able to accomplish what you want by
>> running the 'mounts' cloud-config module as a 'cloud_init_modules' rather
>> than 'cloud_config_modules'.
>>
>> basically, through user-data or image modification of
>> /etc/cloud/cloud.cfg you should be able to do this:
>>  | mounts:
>>  |  - [ ephemeral0, /var/log, auto, "defaults,noexec" ]
>>  | cloud_init_modules:
>>  |  - bootcmd
>>  |  - resizefs
>>  |  - set_hostname
>>  |  - update_hostname
>>  |  - update_etc_hosts
>>  |  - ca-certs
>>  |  - rsyslog
>>  |  - ssh
>>  | cloud_config_modules:
>>  |  - mounts
>>  |  - ssh-import-id
>>  |  - locale
>>  |  - set-passwords
>>  |  - grub-dpkg
>>  |  - apt-pipelining
>>  |  - apt-update-upgrade
>>  |  - landscape
>>  |  - timezone
>>  |  - puppet
>>  |  - chef
>>  |  - salt-minion
>>  |  - mcollective
>>  |  - disable-ec2-metadata
>>  |  - runcmd
>>  |  - byobu
>>
>> Above, I basically copied content from /etc/cloud/cloud.cfg the data
>> provided by user-data will override those stored in the image.
>>
>> I think that when run as a cloud-init module, cloud-init will actually
>> block the running of rsyslog until it has finished.  So, at that point
>> rsyslog should not have populated it.
>>
>> However, cloud-inti *will* have written some stuff to
>> /var/log/cloud-init.log. so... thats one example of why this doesn't
>> really work as well as you'd like.
>>
>> I'm open to ideas and code on how to make it magically work