No subject
Tue Sep 21 04:50:38 BST 2010
Ghost-revisions - this concept needs a lot of work (as I have no doubt
everyone involved already knew :)...). The key difficulty was that I could
not have devs working in effective isolation (which would have reduced the
impact of the previous two shared repo bugs to one or two devs most likely)
as their checkins would break annotate, log etc. for other bzr users. Even
now, if multiple devs decide to use bzr for personal use, we will be
interferring with each others ability to view change history and will likely
need to revert back to svn to see things like blame and log. Ghosts also
seems to cause a lot of inconsistent rev history warnings when going to and
from the network shared repo.
Shared repo bugs - these have two major challenges. Firstly, they are very
high risk, in that a broken shared repo halts dev work on that project
effectively. Secondly, they are very time consuming to fix, in that a full
reconcile needs to be done to recover, meaning a lot of down time. Both
these are difficult to accept in an enterprise environment. In my ignorance
I don't know how these can be effectively addressed, but some random (and
foolish) ideas would be things like:
- ability to iteratively recover a repository (so main dev activities are
not halted). Potentially isolating the blocks of revisions to be fixed to
specific branches, revision ranges so I can fix the currently broken
branches in active use, and recover the inactive/merged revisions after
hours. In my experience here, if it takes more than 12hours, it's a weekend
fix effectively so needs a quicker workaround to prevent losing an entire
day of work
- ability to easily resync shared repositories. In my case, it takes
roughly 30-60mins to create a brand new bzr repo and checkout set even using
the seed. I knew there were repo issues, but had no easy method for the
devs to quickly resync broken or inconsistent revisions between the network
repo and their local ones such that they could resume committing. I tried a
few ways to quickly get around the problematic revisions (e.g., commit on
the server and pull the result to the client... this threw other errors
though), but just had no way of forcing the local shared repository to
clobber its own view with the network's more correct view without clearing
out a bunch of caches and redownloading the whole 5Gb repo
A slightly unrelated suggestion is that it would be handy to have some way
of creating simple path aliases for checkouts. So that rather than
requiring a custom plugin to write a company-specific version of
lp:<user>/<branch>, I can simply build up a path prefix and handle the
management on the file system. e.g.,
[DirectoryAliases]
# Prefix=Path
myproj=bzr+ssh://$SSH_USER@projhost/path/to/remote/location
which then allows use as:
bzr co myproj:~user/branch
I'll see how much time I end up with at various points, I may even try and
create this plugin myself though if nothing already exists :)
As this story ends for now, I'm currently finishing the repair work on the
main repository, and will likely continue to use bazaar myself (coping with
the somewhat unsolvable ghost-revision issue). For main project use, bazaar
has been dropped though. The upside of the experience has been that over
half of the existing devs were so in love with bazaar and the wider DCVS
workflows they will now make sure management is aware of the pain subversion
is causing them... hopefully this cry will help us get future projects on a
DCVS of some description (I still would prefer bazaar of course :)).
I want to repeat again, that this email isn't intended to be insulting to
the devs or community in any way. I really appreciate the help I have
received through this period, and all suggestions I've had have been prompt,
professional and thorough. The primary intent of this email is to document
my journey, and hopefully provide some gain for us all :)
Thanks for your patience and eyes... I welcome any feedback and suggestions
people might have for various pain-points I've listed. I am still excited
by watching where bazaar goes, and hope for more stability, speed and magic
unicorns in future releases!
--
Philip Peitsch
Mob: 0439 810 260
--0016e6de048ac349130491846494
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Hi everybody,<br><br>After yesterdays fun I've experienced the rather f=
un morning of answering some hard questions from management (pro tip: advoc=
acy of any new tech is dangerous).=A0 Unfortunately, management has decreed=
that for now Bazaar not an acceptable official tool (I'm still negotia=
ting any level of personal use, but this has some ghost issues I'll dis=
cuss later).=A0 Though I'm sad at this decision, I do feel I have some =
experiences here that might be valuable to the community. I've been pon=
dering a few things about bazaar and the Enterprise experience, and thought=
it particularly relevant now based on some recent road-map discussions. <b=
r>
<br>First, I want to say that bazaar has definitely made coding a more effi=
cient & pleasurable experience for me.=A0 I don't want this email t=
o be read as a "Dear John" letter of any sort (I definitely still=
consider myself a heavy bzr user and participant)... I intend for this ema=
il to contain some of the difficulties that I've hit deploying this wit=
hin the organization that I work at, in hope that some of these can help fu=
ture experiences and potentially make some salient points about bazaar in g=
eneral :)<br>
<br>So, to give a bit of background, I work at a reasonable-sized company (=
300 employees at the campus I'm at), on one of the pivotal company proj=
ects with roughly 15-20 other devs & architects.=A0 The company had a l=
ong history with perforce being the central CVS for a while, migrating appr=
oximately 3 years ago to subversion.=A0 Approximately 5-ish months ago I st=
arted using bzr personally against the svn trunk.=A0 Other developers gaine=
d some interest in the workflows and methods I was using with bazaar, and s=
o usage gradually grew up to 5 others, and eventually the majority (14/15 m=
ain devs) on bazaar.=A0 As it was, due to some of the tools we use not supp=
orting bazaar in any official capacity (TeamCity and JIRA primarily), we we=
re not able to switch to using bzr core, but continued to use the main repo=
sitory in subversion, with bazaar feature branches being stored in a centra=
l network location.<br>
<br>Some of the difficulties / learning experiences I've had during the=
process (some of these are bzr-svn specific, others are general bzr):<br><=
ul><li>Ran into the 2Gb memory limit for python for initial checkouts of tr=
unk.=A0 We're a Windows XP 32-bit dev shop, so from what I can see this=
is a python limit, nothing Bazaar has any control over.=A0 In our case, I =
used a linux VM to make the initial conversion to bazaar, and this "se=
ed" was then distributed to all other devs. Having said that, it's=
been exciting watching the memory optimisations=20
being made with the knowledge that your work is reducing this pain for=20
others.<br></li><li>Ghost-revisions.=A0 This one is a world of pain and req=
uired some very special management.=A0 We hit annotate crashes, log crashes=
etc. caused by the incomplete revision data ghost revs tended to introduce=
into the system.=A0 These all have existing bug reports by the way, and in=
some cases some progress is being made on them.=A0 Even now, I still don&#=
39;t know how to properly manage our setup (svn core, bzr branches that mer=
ge into svn) without these ghosts causing us to essentially throw away all =
extra data on a merge, and worse, crash other bzr clients that do not have =
access to this revision history.=A0 The workaround I introduced here was a =
central shared repository, with a local bound repository for each dev.=A0 T=
his meant the devs could play with bzr on their local file systems for qlog=
, qdiff, qswitch etc. (required as not all had full samba access to the sha=
red location), while the shared location always had a copy of all the uniqu=
e revisions bazaar new about, and a ghost-free copy of svn trunk with full =
revision history for all merges.</li>
<li>Bazaar didn't have any standard practices for working with a large =
checkout.=A0 In our case, trunk is roughly 2Gb, making rapid branch creatio=
n with a full-checkout impossible.=A0 I had to read pretty far and wide to =
come up with a good process using switch and some custom bat files to make =
branching easy using a single working checkout.=A0 I have noticed in recent=
weeks the colo-branch stuff taking shape, which is again very exciting to =
know that soon there will be a well accepted method for working in this fas=
hion with decent GUI integration.</li>
<li>Shared repositories.=A0 These are a very dangerous single point of fail=
ure.=A0 Since bazaar entered heavy use in the last 3 months, I've been =
bitten by two separate bugs (<a href=3D"https://bugs.launchpad.net/bzr/+bug=
/485601" target=3D"_blank">https://bugs.launchpad.net/bzr/+bug/485601</a> a=
nd <a href=3D"https://bugs.launchpad.net/bzr-svn/+bug/587819" target=3D"_bl=
ank">https://bugs.launchpad.net/bzr-svn/+bug/587819</a>).=A0 In both cases,=
these bugs caused the shared network repository to no longer allow commits=
, effectively forcing me to either disconnect the devs from the central rep=
o and accept new ghost revisions creeping into svn, or hold up the devs as =
I madly attempted to repair the shared repository.=A0 With a bzr reconcile =
taking ~5hrs for our trunk (or 20hours with the most recent bug), the decis=
ion is hard to make, and costly either way.=A0 If I disconnect the devs, we=
are running with the knowledge the shared repository can never get a compl=
ete history for the appropriate svn trunk, as well as the devs having no ba=
ckup of their code.... If I hold up the devs while the problem is solved, w=
ell... thats 15man-days lost, on a project costing $300,000 per month... so=
definitely expensive :)</li>
</ul>From all the previously listed bits, I'd like highlight what I thi=
nk the main difficulties are/were:<br><br>Ghost-revisions - this concept ne=
eds a lot of work (as I have no doubt everyone involved already knew :)...)=
.=A0 The key difficulty was that I could not have devs working in effective=
isolation (which would have reduced the impact of the previous two shared =
repo bugs to one or two devs most likely) as their checkins would break ann=
otate, log etc. for other bzr users.=A0 Even now, if multiple devs decide t=
o use bzr for personal use, we will be interferring with each others abilit=
y to view change history and will likely need to revert back to svn to see =
things like blame and log.=A0 Ghosts also seems to cause a lot of inconsist=
ent rev history warnings when going to and from the network shared repo.<br=
>
<br>Shared repo bugs - these have two major challenges.=A0 Firstly, they ar=
e very high risk, in that a broken shared repo halts dev work on that proje=
ct effectively.=A0 Secondly, they are very time consuming to fix, in that a=
full reconcile needs to be done to recover, meaning a lot of down time.=A0=
Both these are difficult to accept in an enterprise environment.=A0 In my =
ignorance I don't know how these can be effectively addressed, but some=
random (and foolish) ideas would be things like:<br>
<ul><li>ability to iteratively recover a repository (so main dev activities=
are not halted).=A0 Potentially isolating the blocks of revisions to be fi=
xed to specific branches, revision ranges so I can fix the currently broken=
branches in active use, and recover the inactive/merged revisions after ho=
urs.=A0 In my experience here, if it takes more than 12hours, it's a we=
ekend fix effectively so needs a quicker workaround to prevent losing an en=
tire day of work<br>
</li><li>ability to easily resync shared repositories. In my case, it takes=
roughly 30-60mins to create a brand new bzr repo and checkout set even usi=
ng the seed.=A0 I knew there were repo issues, but had no easy method for t=
he devs to quickly resync broken or inconsistent revisions between the netw=
ork repo and their local ones such that they could resume committing.=A0 I =
tried a few ways to quickly get around the problematic revisions (e.g., com=
mit on the server and pull the result to the client... this threw other err=
ors though), but just had no way of forcing the local shared repository to =
clobber its own view with the network's more correct view without clear=
ing out a bunch of caches and redownloading the whole 5Gb repo</li>
</ul>A slightly unrelated suggestion is that it would be handy to have some=
way of creating simple path aliases for checkouts.=A0 So that rather than =
requiring a custom plugin to write a company-specific version of lp:<use=
r>/<branch>, I can simply build up a path prefix and handle the ma=
nagement on the file system.=A0 e.g., <br>
<br>[DirectoryAliases]<br># Prefix=3DPath<br>myproj=3Dbzr+ssh://$SSH_USER@p=
rojhost/path/to/remote/location<br><br>which then allows use as:<br><br>bzr=
co myproj:~user/branch<br><br>I'll see how much time I end up with at =
various points, I may even try and create this plugin myself though if noth=
ing already exists :)<br>
<br>As this story ends for now, I'm currently finishing the repair work=
on the main repository, and will likely continue to use bazaar myself (cop=
ing with the somewhat unsolvable ghost-revision issue). For main project us=
e, bazaar has been dropped though. The upside of the experience has been th=
at over half of the existing devs were so in love with bazaar and the wider=
DCVS workflows they will now make sure management is aware of the pain sub=
version is causing them... hopefully this cry will help us get future proje=
cts on a DCVS of some description (I still would prefer bazaar of course :)=
).<br>
<br>I want to repeat again, that this email isn't intended to be insult=
ing to the devs or community in any way.=A0 I really appreciate the help I =
have received through this period, and all suggestions I've had have be=
en prompt, professional and thorough.=A0 The primary intent of this email i=
s to document my journey, and hopefully provide some gain for us all :)<br>
<br>Thanks for your patience and eyes... I welcome any feedback and suggest=
ions people might have for various pain-points I've listed.=A0 I am sti=
ll excited by watching where bazaar goes, and hope for more stability, spee=
d and magic unicorns in future releases!<br>
<br>-- <br>Philip Peitsch<br>Mob: 0439 810 260<br>
--0016e6de048ac349130491846494--
More information about the bazaar
mailing list