read() wont allow me to read files larger than 2 gig (on a 64bit)

[Stefan Bader]

glide creme wrote:
> Thanks for your reply
> 
> Yes stat, calculates the filesize correctly,
> 
> This is the output of the program
> 
> $ls -l bigfile.dat
> -rw-r--r-- 1 tmp tmp 2163946253 2009-12-23 07:57 bigfile.dat
> $ ./a.out
> LONG_MAX:9223372036854775807
> SSIZE_MAX:9223372036854775807
> a.out: Read only 2147479552 of 2163946253 bytes: Succes
> 
> This was I find strange,
> because the LONG_MAX is 64bit, and SSIZE_MAX should be defined to be LONG_MAX,
> but it looks like read is only limited to 32bit.
> -------------------------------------------------------------
> #include <stdio.h>
> #include <stdlib.h>
> #include <err.h>
> #include <fcntl.h>
> #include <sysexits.h>
> #include <unistd.h>
> #include <sys/stat.h>
> #include <limits.h>
> #include <bits/posix1_lim.h>
> #include <sys/types.h>
> // get bytesize of file
> size_t fsize(const char* fname){
>   struct stat st ;
>   stat(fname,&st);
>   return st.st_size;
> }
> 
> 
> 
> 
> int main() {
>   const char *infile = "bigfile.dat";
>   int fd;
>   size_t bytes_read, bytes_expected = fsize(infile);
>   char *data;
>   printf("LONG_MAX:%lu\n",LONG_MAX);
>   printf("SSIZE_MAX:%lu\n",SSIZE_MAX);
> 
>   if ((fd = open(infile,O_RDONLY)) < 0)
>     err(EX_NOINPUT, "%s", infile);
> 
>   if ((data =(char *) malloc(bytes_expected)) == NULL)
>     err(EX_OSERR, "data malloc");
> 
>   bytes_read = read(fd, data, bytes_expected);
> 
>   if (bytes_read != bytes_expected)
>     err(EX_DATAERR, "Read only %lu of %lu bytes",bytes_read, bytes_expected);
> 
>   /* ... operate on data ... */
> 
>   free(data);
> 
>   exit(EX_OK);
> }
> -------------------------------------------------------------
> 
> On Fri, Dec 25, 2009 at 10:04 AM, AceLan Kao <acelan.kao at canonical.com> wrote:
>> Dear Glide,
>>
>> Does the stat() return the correct file size?
>> I remember that there is a 64bits version of stat(), stat64(), maybe
>> that's the problem?
>>
>> Best regards,
>> AceLan Kao.
>>
>> 2009/12/25 glide creme <glidecreme at gmail.com>:
>>> Hi
>>> Sorry if I'm asking on the wrong mailinglist,
>>> but I've asked on various forums and I havn't found a solution.
>>>
>>> I can't read files larger than 2 gig on my ubuntu64bit using 'read()',
>>> the problem is not related to the allocation itself, which works.
>>> Attached is a sample program that illustrates this.
>>>
>>> I've tried with various, compiler flags like
>>> -D_FILE_OFFSET_BITS=64
>>>
>>> According to man 2 read, the maximum is limited by SSIZE_MAX
>>> which is defined in
>>>
>>> /usr/include/bits/posix1_lim.h
>>> # define SSIZE_MAX LONG_MAX
>>>
>>> And LONG_MAX is defined in /usr/include/limits.h as
>>> # if __WORDSIZE == 64
>>> # define LONG_MAX 9223372036854775807L
>>> # else
>>> # define LONG_MAX 2147483647L
>>> # endif
>>> # define LONG_MIN (-LONG_MAX - 1L)
>>>
>>> Is this the expected behaviour for the read function on a 64bit platform,
>>> or is something broken
>>>
>>> Thanks in advance
>>>
>>> edit:
>>>
>>> by the way
>>>
>>> readelf -h ./a.out
>>> ELF Header:
>>>   Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
>>>   Class:                             ELF64
>>>   Data:                              2's complement, little endian
>>>   Version:                           1 (current)
>>>   OS/ABI:                            UNIX - System V
>>>   ABI Version:                       0
>>>   Type:                              EXEC (Executable file)
>>>   Machine:                           Advanced Micro Devices X86-64
>>>   Version:                           0x1
>>>   Entry point address:               0x400750
>>>   Start of program headers:          64 (bytes into file)
>>>   Start of section headers:          5312 (bytes into file)
>>>   Flags:                             0x0
>>>   Size of this header:               64 (bytes)
>>>   Size of program headers:           56 (bytes)
>>>   Number of program headers:         9
>>>   Size of section headers:           64 (bytes)
>>>   Number of section headers:         37
>>>   Section header string table index: 34
>>>
>>> ldd ./a.out
>>>         linux-vdso.so.1 =>  (0x00007fff689ff000)
>>>         libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
>>>         libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
>>>         libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
>>>         libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
>>>         /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)
>>>
>>> uname -a
>>> Linux csi 2.6.31-15-generic #50-Ubuntu SMP Tue Nov 10 14:53:52 UTC
>>> 2009 x86_64 GNU/Linux
>>>
>>> --
>>> kernel-team mailing list
>>> kernel-team at lists.ubuntu.com
>>> https://lists.ubuntu.com/mailman/listinfo/kernel-team
>>>
>>>
>>
>>
>> --
>> Chia-Lin Kao(AceLan)
>> http://blog.acelan.idv.tw/
>> E-Mail: acelan.kaoATcanonical.com (s/AT/@/)
>>
> 

Have you tried the following?

-Stefan

FEATURE_TEST_MACROS(7)     Linux Programmer's Manual    FEATURE_TEST_MACROS(7)

NAME
       feature_test_macros - feature test macros

SYNOPSIS
       #include <features.h>

       _FILE_OFFSET_BITS
              Defining this macro with the  value  64  automatically  converts
              references  to  32-bit  functions and data types related to file
              I/O and file system operations into references to  their  64-bit
              counterparts.   This is useful for performing I/O on large files
              (> 2 Gigabytes) on 32-bit systems.  (Defining this macro permits
              correctly written programs to use large files with only a recom‐
              pilation being required.)  64-bit systems naturally permit  file
              sizes  greater than 2 Gigabytes, and on those systems this macro
              has no effect.