Experiences with Bazaar in a Commercial Environment

[A. S. Budden]

Dear all,

Apologies for the long email...

We have been using Bazaar as a version control system at my workplace
for the last two years.  As of today, we have decided to gradually
migrate from Bazaar to Mercurial, so I thought I would write up some of
our experiences and the reasons for this change in case it's of any
interest to others.  It is my sincere hope that none of this is seen as
insulting or disparaging to the developers or community in any way; I
merely wanted to document the reasons for our choice.  I would like to
thank the community for the incredibly helpful support I've received
over the past few years; we have commercial support for a lot of
applications that we use and if they could live up to the Bazaar mailing
list it would be a massive improvement!

Before we used Bazaar, we had used PVCS for a long time and the many
frustrations associated with that --CENSORED-- VCS led to us looking
elsewhere.  After comparing a number of different options (the three
open source DVCSs, Perforce and a couple of others), we opted for Bazaar
for a number of reasons including:

* It had the best GUI by a very long way (most of the other developers
  in my team like GUIs: I'm the only hardened CLI user, although I still
  use qlog rather a lot).
* It supported the 'heavyweight checkout' model which gave the advantage
  of optional local commits (when the network was being temperamental)
  but with all changes going to the server by default (unlike git etc).
  This means that there are no un-pushed changes that aren't being
  safely backed-up without needing to run SyncToy or the equivalent on a
  regular basis.
* Bazaar Externals allowed common library code to be shared across
  projects (as long as you don't use lightweight checkouts).
* It allowed empty directories to be put under version control, which is
  useful with some of the less-polished toolchains in use here.
* It's well supported on Windows, which is a bit fundamental for us!
* There was a migration path away if ever we needed it (which we're now
  using).

I have never heard a single developer in the company say anything along
the lines of "I wish we'd stuck with PVCS", so in many ways it was a big
success: moving away from PVCS could only be a good thing, whichever way
you look at it.  However, there were some issues that we had.  Some we
were able to overcome, some we could probably have overcome if we put
the effort in, some were just frustrating (but relatively manageable)
and some weren't likely to go away any time soon.  As a result, we're
converting all of our (~90) repositories to Mercurial (which now has a
very nice GUI, unlike the one it had two years ago).  I'll try to group
the reasons into the categories above for the sake of some sense of
order in this (probably over-long) email.  As you get further down the
following list, the issues do become significantly more minor!

Problems that weren't going away:

1.  Lack of tool support.  Okay, this is not really Bazaar or
    Canonical's fault, but it doesn't have the user base for companies
    like Atlassian (we use JIRA) to go to the effort of adding support.
2.  Bazaar Explorer is very very flaky.  This is probably top of the list
    of problems.  At the moment I'm getting about two people a day coming
    to my desk and saying "Bazaar's crashed again" (by which they mean
    Bazaar Explorer).  A couple of the developers have simply stopped
    committing due to the bugs in Bazaar Explorer (and this is a VERY
    bad thing).  This is mostly due to changes in one of the most recent
    releases and it can be improved a bit by turning of the file system
    monitor, but we still get crashes regularly.  Before 2.4.0, we
    didn't really have these problems (although we had plenty of other
    ones that were fixed in 2.4.0), but this has turned into a bit of a
    show-stopper for us.  It's probably not that helpful to my team that
    I never see these problems as I almost exclusively use the command
    line...
3.  Bazaar Externals doesn't seem very good at locking revisions down.  We
    put a specific revision into the .bzrmeta/externals file and then the
    build server tells us a build has failed.  On investigation it turns
    out it's pulled the latest version of the library module and not the
    one specified in .bzrmeta/externals.  This doesn't happen often, but
    it's annoying when it does.
4.  Bazaar Externals isn't supported by the GUI.

Could have overcome with some effort:

5.  Merges get intertwined with commits.  This was very frustrating and
    probably our fault for using checkouts rather than a proper
    distributed model.  When you go to do a commit, if your working
    directory is out of date with the server, you update and Bazaar merges
    your working directory with the server state.  This means you have a
    commit that is a combination of a merge and a change and it's very
    difficult to track what just happened.  We could have overcome this by
    switching to a distributed model or making more use of "ci --local",
    but we didn't.
6.  Merges are completely automatic.  I guess this is related to the
    above, but our experiences with merges trashing the working copy has
    made us very nervous of a merge that isn't very heavily reviewed.  I
    quite like the way TortoiseHG allows you to choose whether the
    auto-merge happens or not.  I guess I could have written a merge
    plugin to disable all merges in some way, but I didn't really want to!
7.  GUI diffs are a bit awkward.  In Bazaar's qlog, I right-click on a
    revision and ask for a diff in Beyond Compare.  It opens the first
    file in Beyond Compare.  When Beyond Compare quits, it opens the
    next one, etc etc.  If Beyond Compare is running already, it tells
    Beyond Compare to open them all in tabs, but deletes the files
    before Beyond Compare can look at them (because the Beyond Compare
    process returns immediately).  In Mercurial, it opens up a Beyond
    Compare 'Folder Compare' view with all of the changed files and I
    can compare them in whatever order I choose.

Frustrating but sort-of manageable:

8.  Binary conflicts are handled in a user-unfriendly way.  In TortoiseHG
    you get two big buttons: "Take This" and "Take Other".  This would be
    a REALLY good addition to Bazaar.  In Bazaar you get a load of files
    in the file system, some registered as renames, some as untracked
    files and you get to sort it out yourself.  I never really mastered
    this one...
9.  Directories are versioned.  I know I put this under the reasons to go
    for Bazaar rather than the others above, but having used it a lot,
    I've changed my mind on this one.  This feature results in a lot of
    '.moved' directories all over the place.  For example, use "Switch
    Checkout" to create a new branch, add a new directory with a test
    harness in it.  Compile and run the test harness etc etc, committing
    as you go.  Decide it's ready to merge to trunk, so switch to trunk.
    The trunk doesn't have the 'Test' directory, so Bazaar wants to remove
    it; however, there are lots of '.o' files in the directory, so Bazaar
    can't remove it safely: hence .moved.  Mercurial just ignores the .o
    files and leaves them where they were with no problems found.  In
    Mercurial we can work round the flaky toolchains needing directories
    to exist by using .placeholder files or whatever.  We rarely rename
    directories in practice, so Bazaar's benefits here are not
    necessarily that significant to us.

Easily able to overcome:

10. Differences between the GUI and the CLI: "bzr co" gets a heavyweight
    checkout; "bzr qco" gets a heavyweight checkout but Bazaar Explorer's
    checkout gets a lightweight one (overcome with a Bazaar Explorer
    plugin);
11. Setting up a project for checkouts and branches is complicated:
        bzr init-repo --no-trees F:/path/to/repo
        bzr init f:/path/to/repo/trunk
        bzr co f:/path/to/repo/trunk c:/workspace/project
    I got round this with a 'remote feature branches' plugin that does
    the whole lot of the above from Bazaar Explorer's 'Initialise'
    screen.
12. Terminology: In Bazaar Explorer (without the above plugin), you use
    the 'Branch' command with the bind option set to checkout your code
    and the 'Switch Checkout' command to create a branch.  The number of
    times I had to explain "Branch to Checkout, Switch Checkout to Branch"
    was REALLY starting to get to me; feature branches aren't really
    created here often enough for it to be completely ingrained.  Partly
    this was overcome with the plugin, but also partly by people getting
    used to working in Bazaar.  I think a lot of this comes down to what
    I see as the key problem with Bazaar's (very nice) support for lots
    of working models: the GUI is built around one and only seems to
    support the others as an afterthought (see http://bit.ly/wzbKE2).
13. The name: If ONE more person comes to my desk and says "I've got a
    Bizarre problem with Bazaar!", I think I'm going to scream!  Despite
    being an Englishman, I've never been a fan of puns!

----

How Mercurial stacks up on these specific issues:

1. Better tool support.
2. I guess I'll have to wait and see whether we have issues with
   TortoiseHG.  I suspect I'll be rather unpopular if we do!
3. Mercurial's subrepos seems to work differently - there's no revision
   locking by editing a file, it is effectively the same as only having
   .bzrmeta/externals-snapshot.
4. Mercurial's subrepos is standard, so supported by the GUI.
5. We're stuck with the distributed model, so this gets cured
   automatically.
6. I've already commented on this one.
7. Ditto.
8. Ditto.
9. Ditto.
10. I haven't found any significant inconsistencies yet (but I may!).
11. Only one working model, so this is a bit simpler.
12. Again: only one working model, so the terminology is relatively
    consistent.
13. I guess I'll have to wait and see what puns people come up with for
    Mercurial!

----

Now that all probably sounds rather negative, but having said all of
that, I'll be the first to admit that there are probably things we could
have done differently in a lot of cases.  There are a mixture of
developers in the department, some who want to know how things work and
others who want a big 'COMMIT' button and nothing else.  Maybe if
everyone wanted to understand the underlying concepts it would have been
better in some cases.  Maybe if we'd kicked them hard enough that they
HAD to learn the concepts (and maybe even the CLI!) it would have been
better in even more cases.

I for one still like Bazaar.  I have a number of projects on my home PC
that I work on, many in Bazaar, some in Mercurial and a few in Git
(although I'm getting increasingly tempted to migrate away from Git).  I
have no intention of moving away from Bazaar for the projects that use
it as I see no reason to do so.  I also expect to use Bazaar on new
projects where it seems appropriate.  There are many many things that I
love about Bazaar.  TortoiseHG Workbench is really nice, but qlog is
GORGEOUS and it's the only log viewer I've ever seen for any VCS that
allows you to fold up branches (the little +/- icon) to make the screen
more readable.  I love Bazaar's '--remember' option: this is so much
easier than having to edit .hg/hgrc.  I still maintain that Bazaar has
one of the most helpful and responsive communities of any of the VCSs
that I've used.  Heavyweight checkouts still seem like a very good thing
to me.  Plugins are easy to write (once you've worked out that the best
place to start is the existing command implementations rather than
trying to read the API docs).  I also really like "bzr uncommit" for the
simple reason that I like to be able to easily do this:

    bzr ci -m 'No typos in ths comit message.'
    bzr uncommit
    bzr ci -m 'No typos in this commit message.'

but maybe that's just me...

Also I really like the way that Bazaar makes it very very hard to
actually change history (bzr uncommit keeps the old revision).  I'm
hoping that none of my colleagues figure out how to do this in Mercurial
as then I'll inevitably have to fix it and probably won't be able to.
Oh, and the one big advantage of Bazaar Externals being a plugin is that
I can turn it off.  If the Mercurial subrepository gets corrupted, I
have to get into some really weird 'hg debug...' commands to try to
recover my work from the parent repository!

----

I guess there is a chance that Mercurial will be deemed a failure and
we'll be looking at Bazaar again in a couple of years (although I might
be looking for a new job at that point!).  If so, I'll be sure to report
on the experiences again and I am confident I'll be welcomed back by the
ever friendly Bazaar community.  Bazaar has certainly been an invaluable
stepping stone for us and I guess it's quite possible that had we gone
for Mercurial two years ago we'd now be moving to Bazaar due to problems
that we have yet to discover.

As I said at the start, please don't take this email as a massive
criticism.  It's difficult to provide completely objective feedback
after having had any significant issues in a commercial environment and
inevitably that will have 'coloured' the contents of the email.
Overall, Bazaar has been an extremely good thing from the company's
point of view and there was a long and drawn-out debate over whether we
should move away from it (by contrast the decision to move away from
PVCS was quick and easy!).  I've written this email in the hope that it
will provide some useful and hopefully constructive feedback and I'd
welcome any comments people have on all the mistakes we've inevitably
made in our use of Bazaar (although I doubt it'll change our decision).

If you've made it this far through my email, thanks for your patience
and, more importantly, thanks once again for Bazaar, for QBzr and for
all of the prompt and professional support over the last couple of
years.  I'll still be lurking on this list as a personal Bazaar user,
but I guess I'll be reading fewer of the emails now.  I will certainly
be looking forward to seeing the way that Bazaar progresses in the next
few years and I'm sure I'll continue to be impressed by the rate of
progress.

Al