Thoughts and experiences on Bazaar & Subversion in an enterprise environment

[Philip Peitsch]

Hi everybody,

After yesterdays fun I've experienced the rather fun morning of answering
some hard questions from management (pro tip: advocacy of any new tech is
dangerous).  Unfortunately, management has decreed that for now Bazaar not
an acceptable official tool (I'm still negotiating any level of personal
use, but this has some ghost issues I'll discuss later).  Though I'm sad at
this decision, I do feel I have some experiences here that might be valuable
to the community. I've been pondering a few things about bazaar and the
Enterprise experience, and thought it particularly relevant now based on
some recent road-map discussions.

First, I want to say that bazaar has definitely made coding a more efficient
& pleasurable experience for me.  I don't want this email to be read as a
"Dear John" letter of any sort (I definitely still consider myself a heavy
bzr user and participant)... I intend for this email to contain some of the
difficulties that I've hit deploying this within the organization that I
work at, in hope that some of these can help future experiences and
potentially make some salient points about bazaar in general :)

So, to give a bit of background, I work at a reasonable-sized company (300
employees at the campus I'm at), on one of the pivotal company projects with
roughly 15-20 other devs & architects.  The company had a long history with
perforce being the central CVS for a while, migrating approximately 3 years
ago to subversion.  Approximately 5-ish months ago I started using bzr
personally against the svn trunk.  Other developers gained some interest in
the workflows and methods I was using with bazaar, and so usage gradually
grew up to 5 others, and eventually the majority (14/15 main devs) on
bazaar.  As it was, due to some of the tools we use not supporting bazaar in
any official capacity (TeamCity and JIRA primarily), we were not able to
switch to using bzr core, but continued to use the main repository in
subversion, with bazaar feature branches being stored in a central network
location.

Some of the difficulties / learning experiences I've had during the process
(some of these are bzr-svn specific, others are general bzr):

   - Ran into the 2Gb memory limit for python for initial checkouts of
   trunk.  We're a Windows XP 32-bit dev shop, so from what I can see this is a
   python limit, nothing Bazaar has any control over.  In our case, I used a
   linux VM to make the initial conversion to bazaar, and this "seed" was then
   distributed to all other devs. Having said that, it's been exciting watching
   the memory optimisations being made with the knowledge that your work is
   reducing this pain for others.
   - Ghost-revisions.  This one is a world of pain and required some very
   special management.  We hit annotate crashes, log crashes etc. caused by the
   incomplete revision data ghost revs tended to introduce into the system.
   These all have existing bug reports by the way, and in some cases some
   progress is being made on them.  Even now, I still don't know how to
   properly manage our setup (svn core, bzr branches that merge into svn)
   without these ghosts causing us to essentially throw away all extra data on
   a merge, and worse, crash other bzr clients that do not have access to this
   revision history.  The workaround I introduced here was a central shared
   repository, with a local bound repository for each dev.  This meant the devs
   could play with bzr on their local file systems for qlog, qdiff, qswitch
   etc. (required as not all had full samba access to the shared location),
   while the shared location always had a copy of all the unique revisions
   bazaar new about, and a ghost-free copy of svn trunk with full revision
   history for all merges.
   - Bazaar didn't have any standard practices for working with a large
   checkout.  In our case, trunk is roughly 2Gb, making rapid branch creation
   with a full-checkout impossible.  I had to read pretty far and wide to come
   up with a good process using switch and some custom bat files to make
   branching easy using a single working checkout.  I have noticed in recent
   weeks the colo-branch stuff taking shape, which is again very exciting to
   know that soon there will be a well accepted method for working in this
   fashion with decent GUI integration.
   - Shared repositories.  These are a very dangerous single point of
   failure.  Since bazaar entered heavy use in the last 3 months, I've been
   bitten by two separate bugs (https://bugs.launchpad.net/bzr/+bug/485601and
   https://bugs.launchpad.net/bzr-svn/+bug/587819).  In both cases, these
   bugs caused the shared network repository to no longer allow commits,
   effectively forcing me to either disconnect the devs from the central repo
   and accept new ghost revisions creeping into svn, or hold up the devs as I
   madly attempted to repair the shared repository.  With a bzr reconcile
   taking ~5hrs for our trunk (or 20hours with the most recent bug), the
   decision is hard to make, and costly either way.  If I disconnect the devs,
   we are running with the knowledge the shared repository can never get a
   complete history for the appropriate svn trunk, as well as the devs having
   no backup of their code.... If I hold up the devs while the problem is
   solved, well... thats 15man-days lost, on a project costing $300,000 per
   month... so definitely expensive :)