[MERGE] Rule-based preferences (EOL part 1 of 3)

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Pool wrote:
| The way we tackle this ought to be consistent with what we do in other
| places in Bazaar, and informed by experiences on other features or in
| other systems.
|
| I think we should add something to the developer documentation about
| the general way we approach these issues, and have made a start on it.
|  If we merge this in we can edit it after settling this issue.
|
| (I can think of quite a few more things that we have learned that we
| could usefully  add, but for now I'll stick to this particular issue.)
|
|>> I think one reason to be careful with a new file in .bzr/branch is that
|>> we have so far *largely* kept from having user editable files in control
|>> directories; if the expectation is that there is a bzr UI for managing
|>> these files that would be nice.
|
| .. and the converse is that everything outside of .bzr is a user file,
| propagated and committed in the usual way, and the user can make
| arbitrary edits to it.  This doesn't mean bzr itself will never
| consult or modify those files, but it's in a fairly restricted way.
| It is basically data not metadata.
|
|> I agree that principle has served us well. branch.conf is the only
|> exception I know of. Shell hooks might live under there one day but
|> don't yet.
|
| To me the question is really whether we want to have a third category
| of file that is neither a control file nor a user file, and how such a
| category would behave.  If we take branch.conf as an example then I
| believe it is never rewritten by bzr, never propagated, and not
| changed by upgrades.  So it is very close to what is done by
| locations.conf, but actually stored in the tree.  I can see how people
| might also want some rules to be configured that way but it would be
| unusual.
|
| If we are going to do this more widely maybe we should make a
| directory clearly just for this use, say .bzr/etc/.
|
| I would suggest that generally hook scripts should be stored in the
| tree because they're part of the project and should be consistent
| between contributors.  But for security perhaps the configuration to
| turn them on should be in .bzr/etc/ or ~/.bazaar/.
|
|>>> Personally, I'd be happy putting a .bzrrules file in the tree because
|>>> we have a file management architecture (status + diff + merge + commit
|>>> + update/pull/push + ...) for files in that location. But, as you know,
|>>> Robert (and others) will reject that because it locks us into a "format"
|>>> which is more difficult to upgrade. I don't agree it's as big an issue
|>>> as others, but I also don't have their experience and we all need to
|>>> support this.
|>> I won't reject it; I think it would be a mistake so I certainly wouldn't
|>> approve it, but perhaps we simply have to learn this lesson again, if
|>> Aaron and I are failing to express the issues satisfactorily.
|
|> I hope I understand your concerns, namely:
|>
|> 1. Putting data about the tree in the tree is evil. This causes
|>   ugliness like special hacks in export to skip over it. It also
|>   makes it next to impossible to abstract how we deliver functionality
|>   in the future or in custom applications.
|
| To separate them out.
|
| 1a - The working tree is the user's namespace.  Some people would say
| even the  .bzr directory is an intrusion, and they like the way that
| Perforce stores no metadata in the tree at all, or that svn has hidden
| properties (and dislike the way there is a .svn in every directory).
| But, we've already decided it's acceptable to have at least .bzr and
| .bzrignore there, and users generally seem to think that's reasonable.
|  This does have the advantage of a bright line between files they
| should modify and files they should not, and making plain diffs to the
| tree include changes to this file.
|
| 1b - The question of abstraction: are we storing the particular bytes
| the user committed, or the general meaning of them which could be
| given different representations?  Again at the moment it's pretty
| straightforward: things in the tree we store the bytes; things in .bzr
| we can change the representation in upgrades.
|
|> 2. Exposing metadata like this locks in the format reducing
|>   upgrade options and forcing us to always carry supporting code,
|>   however obsolete that format is in the future.
|
| That issue exists in either case, though it is slightly more serious
| if we store it as bytes: we might eventually remove support for the
| weave format, but we still need to be able to read the .bzrignore
| files committed then, because it will be carried over by upgrades.
|
|> 3. The future consequences are unclear but are likely to include
|>   lower performance.
|
| I think the specific performance concern was that having one list of
| globs which is matched against every file (or every file in scope for
| a particular operation) may be slow.  Secondarily we are using an
| ini-style format, which is apparently pretty quick to parse may not be
| the ultimate.  I think Robert's real concern though is that if we
| treat it as a user file, we'll be promising to always support that
| representation in future.
|
| However, we're not totally locked in: if we do discover we really want
| to use some other format, it's possible to add that as an alternative
| or to suggest people switch.
|
| The svn properties approach is far more problematic because it both
| promises to store arbitrary user data and multiplies it out by the
| number of files.
|
| I believe Robert had one further significant concern which is that if
| this is introduced without a format bump, people may use old clients
| on such a tree and it will ignore the rules, therefore potentially
| committing changes with the wrong line endings.  If we did have a
| format bump, people would get an error that they could not access the
| tree at all.  In fact, we would also need a repository format bump to
| make sure people didn't pull such a tree into an existing checkout.
|
| I think most of our users would not see that as a good cost benefit tradeoff.
|
| Even if we did this, because new attributes can be handled by plugins
| it would not really give total protection.  That's why I suggested to
| Ian that we might in future let branches or repositories have
| user-configured requirements for bzr versions and plugins, in case
| projects want to set that policy.
|
|> On the other hand, we gain rules that are stored per revision, we
|> get a UI for diff/merge/etc., we don't force a format upgrade on
|> everyone and users of most popular systems are already comfortable
|> with managing .xxxignore files this same way, however imperfect it is.
|
| And not only do we get a UI for merge/diff/conflict, etc for free, but
| that UI is understandable: it's in the tree, so it clearly just works
| the same as everything else in there.
|
|> It's a question of trade-offs at some point. I fully agree with
|> the benefits of typed data. I also think our policy of new semantics
|> means new format is good. But I can live with limited magic files
|> at the top of a tree, all marked as ".bzryyyyyy". To give us breathing
|> room for format bumps, we could introduce an optional format marker
|> either as a file extension (.bzrrules.v2) or top line of file, e.g.
|>
|>  # format 1.6
|
| I think those options give a reasonable escape route if we want to
| change in the future.  I do think the format marker should be optional
| because needing to get it just right will feel bureaucratic.
|
|> My point is that there are solutions (of varying quality) either way.
|>
|> IMO, "mistake" is a harsh word - I see it as being less black and
|> white than that. At some point, our guidelines conflict or imply
|> lots of work. Take these:
|>
|> 1. metadata in the tree is evil.
|> 2. Users shouldn't edit things under .bzr.
|>
|> For complex metadata which is best expressed as a config file
|> complete with embedded user comments, editing this directly rather
|> than providing a command-line UI for a hidden file is a good choice.
|> I'd prefer it wasn't in the tree but we don't have an architecture
|> for many out-of-tree operations on files like this. So, putting some
|> (potentially format-marked) files only at the top of the tree now
|> is a trade-off I'm ok with. We're forever carry code for that format
|> but I think many of the other problems are oversold at times.
|
| I agree.
|
|>>> 1. Is it really true that everything under .bzr is determined by
|>>>    the repo/branch/tree "formats"? Is it illegal for plugins to
|>>>    add files there and if so, where should they put them? It really
|>>>    frustrates me that we have a beautiful open code architecture
|>>>    (plugins) yet our data architecture is so closed, requiring
|>>>    multiple upgrades per year for many users. I struggle to see,
|>>>    and to explain to Joe Average users, why adding some optional
|>>>    preferences to a branch is worthy of a branch upgrade, at least
|>>>    until such time as those things are versioned.
|>> Plugins that need user settings can use the get_user_option api to get
|>> settings (side-note: why doesn't that suffice?). Yes it is illegal to
|>> put files there - consider 'bzr-svn' which provides branch and tree and
|>> repository objects - there is no .bzr directory for them, and its having
|>> a clear abstraction barrier there that allows it to work as seamlessly
|>> as it does. Typed data for the win. There isn't even a guarantee that a
|>> branch at a url actually *has* a directory attached to it - it can just
|>> as reasonably be a postgresql database or some other system. Making it
|>> expose a read-write file system is not very clean.
|
| (Having just been cleaning up some of the code that touches or
| implements control_files and seeing some code depends on it being
| present, I am a little surprised this has not caused failures with
| bzr-svn.  I suppose the particular cases have been addressed in either
| bzr-svn or bzr.  Even now, RemoteBranch needs to provide a fake
| LockableFiles to satisfy this.... but I'll get it.)
|
|>> I've argued strongly *for* landing the core components of this with no
|>> other dependencies, and I certainly don't think you need to
|>> touch .bzrignore.
|> I have no complaints and don't feel you've been blocking me in any way.
|> I did feel there was no point putting up something you felt
|> passionately against though. Please continue to argue for what you
|> feel is right, while understanding that my views on trade-offs
|> will differ from time to time.
|
| I think landing the per-user-configured part would be worthwhile as it
| is, and would reduce the size of the outstanding diff.  But having
| been through this thread I do think having it in the tree is
| reasonable.
|
|

I like the document you wrote up. Care to submit it in the form of a patch?

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkg0PGAACgkQJdeBCYSNAAM7MQCfalKC7rf0t6UxjENb07fHNpG2
oI0AoMH/epkIlFrRb9lQylp515zekcBm
=puY8
-----END PGP SIGNATURE-----