Best filesystem to use for a specific type of application

[Ioannis Vranos]

On Mon, Feb 13, 2012 at 6:40 PM, Steve Flynn <anothermindbomb at gmail.com> wrote:
> On 13 February 2012 16:20, Rashkae <ubuntu at tigershaunt.com> wrote:
>> On 02/13/2012 11:08 AM, Steve Flynn wrote:
>
>> ReiserFS is deceptive.  It was advertised as being very high performance
>> with small files.  But that was only taking into account benchmarks that
>> favor it.  In my experience with this kind of workload, one very important
>> 'benchmark' is the speed in which you can read all files in the order they
>> are returned by Readdir.  (The function that lists all files in a
>> directory.)  For the reading of small files to be efficient, they have to be
>> read in the order they are laid out on the hard drive.  Otherwise, the head
>> has to thrash with lots of random seeks, and that will drop your read
>> performance to only a few MB/s.
>
> Yup - we've seen that kind of pain already with much smaller datasets
> (just over a million files). So much so that we ended up  splitting
> the directory structure up into "no more than 1000 files in a
> directory" and effectively building a tree structure of directories
> within directories to make sure we actually spend time reading the
> contents of the files rather than reading the contents of the
> directories. Worked OK for the smaller stuff but now I'm looking at 40
> times more files so we need a better method.
>
>> I haven't tested it much for this, but I think EXT4 has overcome this
>> limitation.  XFS was by far the fastest filesystem I tested for this
>> workload (before EXT4 was released.)  Reiser, and EXT3 were practically
>> unusable.
>
> Perfect. I'll move those two up my list a bit!

Keep in mind that ext4 (and btrfs) are aggressively developed, and
kernel 3.2 comes with even more optimisations for ext4.

Perhaps what you are looking for is the Phoronix web site:

http://www.phoronix.com

Search for comparison benchmarks there.


-- 
Ioannis Vranos

http://cppsoftware.binhoster.com