[OT] Re: Corrupt bazaar repositories

[Martin Steigerwald]

Am Freitag 22 Juni 2007 schrieb David Cournapeau:
> Martin Steigerwald wrote:
> > Hello!
> >
> > I have quite some corrupt Bazaar repositories after I tried out
> > xfs_fsr - the online defragmentation tool for XFS. I had something
> > similar with KMail index files and a Mercurial repository[1]. I dunno
> > whether it is related to xfs_fsr or not. At least I could not
> > reproduce this up to now - neither with KMail indexes nor with a bzr
> > repository, but aside from the broken ones I had no Bazaar repository
> > with fragmented files.
>
> I know this won't solve your problem, but using XFS on PC-quality
> hardware really is a bad idea, this has been mentioned many times by
> knowledgable people, see for example [1].
>
> [1] http://linuxmafia.com/faq/Filesystems/reiserfs.html

Hello!

Well I know XFS quite deeply from first hand experience and since a quite 
long time. And since 2.6.17.7 it has been really stable. Before that 
there could have been crashes when write caches were enabled cause write 
barriers were not supported / enabled by default. Thats a problem that 
other journaling filesystem face too, may be that XFS is a bit more 
vulnerable to it, but technically spoken any journalling filesystem needs 
some write ordering in order to prevent corruption on abnormal write 
termination. See the XFS FAQ or my article for details[1][2]. Other 
filesystems like ext3, ReiserFS3, Reiser4 also have write barrier support 
for exactly that reason.

I use XFS on two ThinkPads and we use it on 10 Dell workstations in the 
company and we also use it on some servers. And its just stable since 
2.6.17.7 or before, too, *when* write caches were disabled - even 2.6.16 
were I had the most problems with enabled write caches was stable with 
disabled write cache. 2.6.17.1 and 2.6.17.7 had a slight directory 
corruption bug, that is fixable by a current xfs_repair, also mentioned 
in the FAQ.

From all I seen on the XFS mailing list, the XFS people respond quickly to 
any corruption issues reported. Other filesystems are not free from 
corruption issues as well, just search "ext3 corruption" or "ext3 
corruption 2007" on Google to get an idea. Likewise for ReiserFS3, Reiser 
4 and JFS.

Just to spread some up-to-date information on this topic.

I can not yet prove / reproduce it, but it seemed that xfs_fsr - the 
online defragmentation tool for XFS - corrupted some files. I found those 
corrupted Bazaar and Mercurial repos as well as KMail index files shortly 
after I can xfs_fsr. I had no meta data corruption on XFS itself at all. 

Actually I didn't really need to use xfs_fsr since the filesystems were 
not badly fragmented at all. I just liked to try it out. It does not seem 
that many people use it. But granted if it really is xfs_fsr that 
corrupted the files this is bad. I informed the XFS people already about 
the problem and if I can reproduce it I will file a bug report about it - 
maybe even if not.

It seems to be typical file content corruption - file content changed, but 
date as well as size remained the same. I was in luck, cause rsync didn't 
copy those corrupted repositories to my backup drive exactly for that 
reason, except for the Bazaar article repo, which was one of the last 
articles I worked on.

Thus fixing my bazaar article repository would mean to correct those 
corrupted bytes... well maybe I can see a pattern, when "comparing" a 
good and a broken repository with rsync -ac ;-). This should give me the 
broken files and then I can cmp them.

[1] http://oss.sgi.com/projects/xfs/faq.html#wcache

[2] Published in issue http://www.linux-magazine.com/issue/78 (SysAdmin: 
Write Barrier). Not yet available online.

Regards,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7