LD_PRELOAD work in progress

[Stéphane Graber]

On Tue, Feb 24, 2015 at 04:40:21PM -0600, Jamie Strandboge wrote:
> On 02/24/2015 03:29 PM, Michael Terry wrote:
> > Hello!  So I worked on a proof-of-concept for faking an overlayfs scenario by
> > doing path-redirection in an LD_PRELOAD library.
> > 
> > I've got working code here: lp:~mterry/+junk/snappy-preload
> > 
> This is a very intriguing idea.
> 
> Making debs from the archive reusable in snaps easily is a really great goal.
> The suggested overlayfs approach in my mind has two major drawbacks though: 1)
> backporting overlayfs to earlier kernels is hard and not porting-friendly and 2)
> there is a difficult disconnected path issue with apparmor with the current
> implementation[1]. In addition to those, I've been quite concerned about the
> complexity that adding overlayfs mounts brings to the snappy user and developer
> experience. This LD_PRELOAD idea removes the two major drawbacks and IMO makes
> for the possibility of a better experience.
> 
> The security team is tasked with looking at the viability of overlayfs wrt the
> apparmor issue[1] (there is some promising upstream work happening that may
> help) and we will add examining this LD_PRELOAD approach.
> 
> I'm very curious what others have to say.

So the main problem with the LD_PRELOAD approach and why I discarded it
back in CPT is that it will not work for file system access coming from
a static binary.

So if I write a Go piece of software which opens some file from
/usr/share provided by the overlay either in the app itself or one of
its frameworks, it won't be able to access the file because it won't be
using the local libc.


My alternative to overlayfs at the time was a bind-mount farm in a
separate mount namespace, that is, giving the exact same fs view as
you'd get using read-only overlayfs but using read-only bind-mounts and
tmpfs mounts instead (possibly hundreds/thousands of them).

So if you have to stack 3 trees like so:

 == tree 1 (rootfs) ==
    - bin
    - boot
    - data
    - dev
    - etc
    - home
    - initrd.img
    - lib
    - lib64
    - lost+found
    - media
    - mnt
    - opt
    - proc
    - root
    - run
    - sbin
    - srv
    - sys
    - tmp
    - usr
    - var
    - vmlinuz

 == tree 2 ==
    - usr/share/somestuff/blah

 == tree 3 ==
    - usr/bin/abc


Now if you want to set that up without overlays and have the resulting
tree in /mnt, you'd do (pseudo C/shell code):
 - clone(NEWNS)
# tree 1
 - mount -o rbind / /mnt

# tree 2
 - mount -t tmpfs -o size=4096,mode=0755 /mnt/usr/share
 - for entry in /usr/share/*; do
    [ -d $entry ] && mkdir /mnt/$entry && mount -o bind,ro $entry /mnt/$entry
    [ ! -d $entry ] && touch /mnt/$entry && mount -o bind,ro $entry /mnt/$entry
   done
 - mkdir /mnt/usr/share/somestuff
 - mount -o bind,ro /app/.../overlay/usr/share/somestuff /mnt/usr/share/somestuff

# tree 3
 - mount -t tmpfs -o size=4096,mode=0755 /mnt/ussr/bin
 - for entry in /usr/bin/*; do
    [ -d $entry ] && mkdir /mnt/$entry && mount -o bind,ro $entry /mnt/$entry
    [ ! -d $entry ] && touch /mnt/$entry && mount -o bind,ro $entry /mnt/$entry
   done
 - touch /mnt/usr/bin/abc
 - mount -o bind,ro /app/.../overlay/usr/bin/abc /mnt/usr/bin/abc


The result from the app point of view would be identical to overlayfs as
in it'd work with just about any userspace application. However there is
the small concern of having multiple thousand-entries long mount tables
on the system. Working with containers where we do (ab)use mount
namespace a fair bit, the kernel should behave just fine, though that'd
be a fair amount of kernel memory being used just for all of those
mounts and tmpfs.


All that to say that we have alternatives but I think it's important
that we keep in mind that we are basically recommending people do static
builds of their stuff for snappy, so LD_PRELOAD will be a problem.
overlayfs may be painful to backport (though at the time I was told we
could assume a current kernel which apparently got changed later down
the line?) but the alternatives to me seem even worse.

> 
> Thanks Michael!
> 
> [1]https://launchpad.net/bugs/1408106
> 
> > I've tested a few simple programs like bsdgame's boggle and xterm and they run
> > fine.  I'm going to try harder things and see what issues I run into.
> > 
> > Here's how I tested:
> > - Took a deb's contents, copy it all to a temporary directory, uninstall that
> > package
> > - Run the binary like:
> > LD_PRELOAD=/home/mike/Work/code/snappy-preload/trunk/builddir/src/libsnappypreload.so
> > SNAPPY_PRELOAD=/tmp/xterm /tmp/xterm/usr/bin/xterm
> > 
> > (LD_PRELOAD pointing to the built redirection library, and SNAPPY_PRELOAD to
> > point at the rootfs for the overlay.)
> > 
> > What's the state of the art for sucking a deb package and its dependencies into
> > an arbitrary filesystem tree?
> > 
> > There are dpkg hooks and such that would be good to run in such cases, I
> > imagine.  But I don't know much about getting dpkg to install to arbitrary
> > locations and forget about the package afterwards.  (You can do it a bit with
> > dpkg --root, but that situation seems to want all packages to be in the root,
> > rather than accepting that some deps come from the system.)
> > 
> > I also bet the logic to grab dependencies down to a certain level is probably
> > out there already, I just don't know where.
> > 
> > -- 
> > -mt
> > 
> > 
> 
> 
> -- 
> Jamie Strandboge                 http://www.ubuntu.com/
> 



> -- 
> snappy-devel mailing list
> snappy-devel at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snappy-devel


-- 
Stéphane
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/snappy-devel/attachments/20150224/4066bf74/attachment.pgp>