read() wont allow me to read files larger than 2 gig (on a 64bit)

[glide creme]

Thanks for all your replies,
but it seems that its not possible to use posix read to read file
chunks larger than 2.1 gig.
This is according to the linus
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=e28cc715

At somepoint I guess it was possible but the kernel has regressed to
only allow smaller chunks being read.

kernelsourcetree/fs/read_write.c
---------------------------------------------------------
/*
 * rw_verify_area doesn't like huge counts. We limit
 * them to something that fits in "int" so that others
 * won't have to do range checks all the time.
 */
#define MAX_RW_COUNT (INT_MAX & PAGE_CACHE_MASK)
int rw_verify_area(int read_write, struct file *file, loff_t *ppos,
size_t count);
---------------------------------------------------------
This was rather annoying, it would have been nice if this had been
documented outside of the kernelsource and kernel git.

I'm aware that the returnvalue from the 'read' function can be smaller
than the number of bytes expected.

But I think it should be clarified in the documentation, that the
function can't return a value larger than 2147479552.

Instead of the strange "  If  count is greater than SSIZE_MAX, the
result is unspecified.", which implies that a value larger than SIGNED
and smaller than
SSIZE_MAX is actually supported.

It would be sensible to write that the posix read is still limited to
a 32bit value.

This is the sort of not incorrect documentation, that is misleading
and makes the transition to 64bit worse than it has to be.

I don't think many are aware of this 32bit limitation on a 64bit platform.

Thanks again for your replies
On Fri, Dec 25, 2009 at 9:29 AM, glide creme <glidecreme at gmail.com> wrote:
> Hi
> Sorry if I'm asking on the wrong mailinglist,
> but I've asked on various forums and I havn't found a solution.
>
> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
> the problem is not related to the allocation itself, which works.
> Attached is a sample program that illustrates this.
>
> I've tried with various, compiler flags like
> -D_FILE_OFFSET_BITS=64
>
> According to man 2 read, the maximum is limited by SSIZE_MAX
> which is defined in
>
> /usr/include/bits/posix1_lim.h
> # define SSIZE_MAX LONG_MAX
>
> And LONG_MAX is defined in /usr/include/limits.h as
> # if __WORDSIZE == 64
> # define LONG_MAX 9223372036854775807L
> # else
> # define LONG_MAX 2147483647L
> # endif
> # define LONG_MIN (-LONG_MAX - 1L)
>
> Is this the expected behaviour for the read function on a 64bit platform,
> or is something broken
>
> Thanks in advance
>
> edit:
>
> by the way
>
> readelf -h ./a.out
> ELF Header:
>   Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>   Class:                             ELF64
>   Data:                              2's complement, little endian
>   Version:                           1 (current)
>   OS/ABI:                            UNIX - System V
>   ABI Version:                       0
>   Type:                              EXEC (Executable file)
>   Machine:                           Advanced Micro Devices X86-64
>   Version:                           0x1
>   Entry point address:               0x400750
>   Start of program headers:          64 (bytes into file)
>   Start of section headers:          5312 (bytes into file)
>   Flags:                             0x0
>   Size of this header:               64 (bytes)
>   Size of program headers:           56 (bytes)
>   Number of program headers:         9
>   Size of section headers:           64 (bytes)
>   Number of section headers:         37
>   Section header string table index: 34
>
> ldd ./a.out
>         linux-vdso.so.1 =>  (0x00007fff689ff000)
>         libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
>         libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
>         libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
>         libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)
>
> uname -a
> Linux csi 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
> 2009 x86_64 GNU/Linux
>