ZFS feature flags

[Richard Laager]

TL;DR: I think the installer should default to all features, so I think
the current behavior is correct. I would support the installer gaining
an option to default to a subset of features (e.g. using the future
features=portable mechanism) to make the pool more portable. If the
desktop GUI partitioning tool gains support for creating ZFS pools, I
would like to see a similar option there, probably with the opposite
default (i.e. defaulting to portable).


I was asked to comment on this thread. I maintain the upstream
root-on-ZFS HOWTO which was used as a starting point for the Ubuntu
installer design. However, unless otherwise noted, my comments are my
own and do not necessarily reflect a ZoL/OpenZFS consensus.


An incrementing version number was simple. It worked great when ZFS
development was centralized at Sun only. When they opened things up with
OpenSolaris, it worked okay enough. But as ZFS development started
happening in parallel at various companies and projects, it was no
longer feasible to have a simple integer version number. If company A
develops feature X as zpool version 29 and company B develops feature Y
as zpool version 29, you have a problem. The feature flags allow for
parallel development and for features to land on different platforms in
any order. The "read-only compatible" flag on features is a nice detail,
too. It allows using a lot more features on the boot pool than would
otherwise be the case.

A lot of new features originate in ZFS-on-Linux, but features also
originate from Illumos and FreeBSD. (Oracle is not involved in OpenZFS
development in any way. They continue to have their own, closed-source
ZFS implementation on Solaris which is not interoperable with pool
features at all.)

A downside of parallel development is that not all platforms are
guaranteed to have the same features, or land the features in the same
order. So cross-platform compatibility is definitely more complicated
now. If you care about cross-platform compatibility for a particular
pool, you have to limit your enabled features to the common denominator
among the platforms you wish to support with that particular pool.
Alternatively, you can use a minimal set of old features, or even
version=28, but then you miss out on useful enhancements.

There has been recent (March 2019) talk upstream of adding a
"features=portable" property. This would act as a feature mask, limiting
the enabled features (on zpool create and zpool upgrade) to a set that
is portable across "tier 1" OpenZFS platforms. The exact semantics of
what counts as a "tier 1" platform was part of that discussion; the
notes (remember this was March) say "FreeBSD (11.X), ZoL (0.7.X),
illumos-gate (from 1 year ago)". Further discussion was that the latest
Ubuntu LTS would be considered a "tier 1" platform. The idea is that the
definition of features=portable would be rolling, with unchanging
features=portable-YYYY (e.g. portable-2018, portable-2019) complementing
it as another option. This would significantly simplify configuration of
pool features, but is not yet implemented.

The same implementation could (and if I have my way, will) be used to
provide a features=grub mask. This would be used for the boot pool
(bpool) to limit it to the features supported by GRUB. This would avoid
the dangerous message in "zpool status" which tells you to run "zpool
upgrade" on your bpool which would then break booting from it.

If/when features=portable lands, it's probably reasonable to expose that
in the installer for the root pool. This could be a checkbox, for
example. It might read something like, "Disable some pool features to
maximize compatibility with other OSes" or "Maximize compatibility with
other OSes by disabling some pool features". The default could go either
way, but based on the existing precedent and my assumption that the
majority use case is a single OS install, I would have the installer
_default_ to the "more features, less portable" trade-off. Those who
want the "less features, more portable" trade-off could change the checkbox.

I think there's a stronger argument for features=portable (or otherwise
limiting features) as the default for data pools than for the root pool.
Data pools are more likely to move between systems than a root pool.
This would be especially true for something like an external hard drive.
They're also likely to be larger, making pool rebuilds more burdensome.
For a _new install_ on a root pool, the need for portable is far less;
dual booting seems like the most obvious and important case, which is a
good use case for the checkbox. If the desktop GUI partitioning tool
gains support for creating ZFS pools, I think features=portable would be
a reasonable default there (again with a checkbox), under the assumption
that it is creating data pools.

Whatever the default, there are obvious exceptions. For example, if/when
the installer supports ZFS encryption, if the user selects encryption,
the installer must enable the encryption feature even if it would
otherwise default to features=portable. This same idea still applies
(ideally with a warning in the installer) even if the user has set the
checkbox for features=portable--i.e. keep the portable set and add
encryption only, but not enable every feature.

Another exception would be that if you're creating a data pool using a
special vdev, you need the allocation_classes feature that adds support
for special vdevs. Otherwise it simply won't work.

Whether features=portable is set or not, the in-place OS update and
upgrade processes definitely should NOT call `zpool upgrade` right away,
as that would limit the ability to roll back. It would, however, be a
good idea to prompt the user "a while" after a system upgrade to upgrade
to new pool features.

As an aside, FreeBSD is in the process of merging and rebasing their ZFS
implementation to use the current ZFS-on-Linux project as its upstream.
This effort will result in Linux and FreeBSD being equal peers in the
codebase, with the repository being renamed from "zfsonlinux" to
"openzfs" on GitHub, replacing the current Illumos-based openzfs
repository. Apropos to this discussion, this will reduce the feature
delta between Linux and FreeBSD. Of course, due to different release
cycles, there will likely always be some delta between ZFS features in
released versions of, e.g., Ubuntu and FreeBSD.

-- 
Richard