The default file descriptor limit (ulimit -n 1024) is too low

[Scott Ritchie]

On 09/27/2010 01:50 PM, Aigars Mahinovs wrote:
> On 20 September 2010 13:21, Scott Ritchie <scott at open-vote.org> wrote:
>> Would there be any harm in raising this?
>>
>> I ask because I've seen a real world application hit this limit.  The
>> application in question is multithreaded and opens separate threads and
>> files to work on for each core; on a 4 core machine it stays under the
>> limit, while on an 8 core machine it hits it and runs into strange errors.
>>
>> I feel that, as we get more and more cores on machines applications like
>> this are going to increasingly be a problem with a ulimit of only 1024.
> 
> I've seen this limit being hit on a far more pedestrian application -
> Vuze. Imagine a typical backwater teen launching a BitTorrent client
> and downloading all his favorite ... Linux distributions. Let's say
> there is are 20 downloads going at once, each tries to download from
> 60 people and upload to 40 people. Oops, that's over 2000 open file
> handles.
> 
> P.S. I am assuming that this was the reason why the person in question
> saw the Cannot open file 'too many open files' error that went away
> after changing the limit in /etc/security/limits.conf . It is also
> possible that there was a lot of files in these download and that the
> seeds and peers gave random pieces of random files in such a way that
> most of the files were open at the time.
> 

I hadn't considered that use case, but it definitely sounds like a
desktop application that might have a problem too.

The specific app in question was Visual Studio running via Wine and
compiling a large project.  On 4 cores it stayed under the limit, on 8
it would compile more in parallel at once and hit the limit.  According
to AppDB, it seems that Quicken runs into the same problem.

For what it's worth, I believe processes on modern Windows have a
default open files limit of 512, but can be coded to set themselves to
2048 using a C function:
http://msdn.microsoft.com/en-us/library/6e3b887c%28v=VS.80%29.aspx

For Wine to properly support this we'd need to go up to more than 2048,
as Wine needs some of its own files too (and currently opens 2 handles
for every file open a program asks for).

-Scott Ritchie