RFC on Cloud Images: Make /tmp a tmpfs

[Clint Byrum]

Excerpts from Dustin Kirkland's message of 2016-01-14 02:27:58 -0800:
> On Thu, Jan 14, 2016 at 12:05 AM, Seth Arnold <seth.arnold at canonical.com> wrote:
> > On Wed, Jan 13, 2016 at 11:00:16PM +0100, Martin Pitt wrote:
> >> In a perfect world we'd have some clever tmpfs file system which would
> >> use RAM as available and start overflowing onto a disk partition
> >> (which could be LUKS with a random key) when necessary.. But even
> >
> > In fact this is what happens, 'unused' data from tmpfs heads to swap. So
> > just configure swap space on your systems.
> 
> Moreover, just 'sudo apt-get install swapspace' and watch as swapfiles
> are created/deleted as needed.  If your root disk is lvm-encrypted,
> then obviously such swap files are encrypted, too.
> 

This is a neat idea. However, at that point, I'm not sure what this
actually gets you over just letting /tmp be on the filesystem, which will
have a much more limited effect. With a regular /tmp on a filesystem,
you mostly have non-unlinked named temporary files for the purpose of
sharing small amounts of data between processes. That use is definitely
a win for any tmpfs use case, as metadata for these is unlikely to get
paged out.  For the other typical use, you have one program spooling
to an unlinked file. This file will only go to disk when the number of
dirty buffers gets too high, and this flushing can be re-ordered for the
elevator, and will be heavily optimized by the VFS layer. IMO this is a
loss for tmpfs, because effectively you don't get through that optimized
code path, since swap files are never treated as a closed, unlinked file,
but instead a file which must be synced to immediately as soon as a block
has been written, and thus blows holes in the elevator queuing since it
has an immediate need.

Also, since swap cannot be on sparse files, I assume a number of files
must be created at install time, and if more are needed, large amounts
of 0-filled files must be created on the fly. Also, swapoff _must_ move
any used pages out of a swap space before removing it, causing that data
to be re-read, and likely soon thereafter re-written to disk _twice_
if the system is still low on available RAM at the time of swapoff.

Can anyone point to an actual measurement that actually proves the effects
over time when tmpfs is used for /tmp? It feels like we've reasoned
through a few things, but perhaps somebody has done experiments and
published a paper or two that we can refer to for details.