The default file descriptor limit (ulimit -n 1024) is too low

[Kees Cook]

On Mon, Sep 20, 2010 at 03:21:29AM -0700, Scott Ritchie wrote:
> Would there be any harm in raising this?
> 
> I ask because I've seen a real world application hit this limit.  The
> application in question is multithreaded and opens separate threads and
> files to work on for each core; on a 4 core machine it stays under the
> limit, while on an 8 core machine it hits it and runs into strange errors.
> 
> I feel that, as we get more and more cores on machines applications like
> this are going to increasingly be a problem with a ulimit of only 1024.

There are some unexpected things that can happen when open fd limit
(RLIMIT_NOFILE) is raised above 1024, and the main one is that the libc
select() helper macros don't expect the vectors to ever be more than
1024[1] bits long. To quote "man select":

       An  fd_set is a fixed size buffer.  Executing FD_CLR() or FD_SET() with
       a value of fd that is negative or is equal to or larger than FD_SETSIZE
       will result in undefined behavior.

"Undefined behavior" here mean, "will write all over your stack/heap,
etc".  While there are defensive workarounds (see Apache, SSH, etc),
some applications using select can be made to crash if the system
RLIMIT_NOFILE is larger than 1024.

So, while it can certainly make sense to raise it for individual processes
that have been validated to do the right thing with the select macros,
I would strongly recommend against raising it system-wide until glibc
fixes this bug[2] and everything that uses the macros has been recompiled
(or changed to not use select() any more).

-Kees

[1] FD_SETSIZE on Linux is 1024:
    /usr/include/sys/select.h:#define   FD_SETSIZE      __FD_SETSIZE
    /usr/include/bits/typesizes.h:#define   __FD_SETSIZE        1024

[2] http://sourceware.org/bugzilla/show_bug.cgi?id=10352
    But since this is "won't fix", I guess I really just mean "never".

-- 
Kees Cook
Ubuntu Security Team