very slow deletions on ext3 USB drive

[Rashkae]

Karl Auer wrote:
> Hullo all.
> 
> I have a hard disk, 250Gb or so, in an external USB 2.0 HDD enclosure.
> The disk is now full. Specifically, it is full of old backups. Each
> backup is in its own directory, and there are about 45 directories at
> the top level. Each directory contains about 300,000 files and
> directories, mostly very small. The files are pretty evenly distributed
> through the directories and subdirectories. The disk is an ext3
> filesystem.
> 
> I now want to recycle the drive and start filling it up with new
> backups. So I have started deleting the old backups. I don't want to
> delete ALL the backups - just the oldest ones - so I can't just reformat
> the disk.
> 
> Problem is, it is taking literally hours to remove each backup
> directory. Why is this so? Is there any faster way than "rm -fr" to
> remove a directory? Should I use a different filesystem type next time?
> 
> The only wrinkle here is that the backup directories were created using
> rsynch, hard linking to identical files rather than copying them anew.
> But that is a simple link count - I can't see it adding *this* much pain
> to a simple deletion!
> 
> Clues?
> 
> Regards, K.
> 
> 

I see you've already improved on the situation.. but I wanted to suggest
a couple things you could look into.

1. make sure this really is a USB2 enclosure, and nothing is degrading
the Usb level to 1.  The quickest way I can do that is to simply start
reading from the drive and check your transfer rate.

dd if=/dev/usbdrive of=/dev/null  and use another terminal to check
iostat (though personally, I prefer using gkrellm for these kinds of
quick and dirty visual checks.)  You should be getting transfer rate of
approx 30 MB/s.

2.  If you are trying to stuff over than 10,000 small files into a
single directory, it is *vital* that the ext3 file system on your hard
drive has the dir_index option.  This should be the default on new file
systems now, but might not have been if your drive was formatted years
ago.  You will recognize this problem if file system operations (such as
deleting the backups) causes 100% system cpu utilization (one one cpu,
for smp systems.)

Check your drive with tune2fs -l /dev/usbdevice.  Make certain that
Filesystem Options line includes dir_index.  If it doesn't. use tune2fs
-O dir_index /dev/... followed with a e2fsck -D /dev/...