Large files on 32 and 64 bits ystem

Large files on 32 and 64 bits ystem

[Olivier Scalbert]

Hello everybody,

Here is my weekly question ....


I need to create a file that has 2540160000 bytes, that is a little more 
than 2**31 bytes.

On a 64 bits linux box, it is ok, but on a 32 bits linux box, I have:
raised ADA.IO_EXCEPTIONS.DEVICE_ERROR : s-fileio.adb:1135

The file length is: 2147483647, which is 2**31 - 1

Is it possible to write more than 2**31 bytes with an Ada program on a 
32 bits linux ?

Thanks for your help,

Olivier.

Re: Large files on 32 and 64 bits ystem

[Ludovic Brenta]

On May 25, 9:13 pm, Olivier Scalbert <olivier.scalb...@algosyn.com>
wrote:
> Hello everybody,
>
> Here is my weekly question ....
>
> I need to create a file that has 2540160000 bytes, that is a little more
> than 2**31 bytes.
>
> On a 64 bits linux box, it is ok, but on a 32 bits linux box, I have:
> raised ADA.IO_EXCEPTIONS.DEVICE_ERROR : s-fileio.adb:1135

That line indicates that the GNAT run-time library delegates the write
to fwrite(3), so your question really boils down to whether the C run-
time library has support for large files or not.  What filesystem type
do you use on your machines? (I use XFS which supports files up to 8
exabytes :) )

> The file length is: 2147483647, which is 2**31 - 1

This limitation exists on FAT16 ("msdos" in Linux parlance). Newer
filesystems have higher limits.

> Is it possible to write more than 2**31 bytes with an Ada program on a
> 32 bits linux ?

32-bit Linux has "large file support" using either a dedicated 64-bit
API or the O_LARGEFILE flag supported in open(2) since glibc 2.2. It
might be a good idea to check how GNAT's run-time library deals with
this API but I don't have the time right now. Anyone?

--
Ludovic Brenta.

Re: Large files on 32 and 64 bits ystem

[anon]

With GNAT:
 
Ada.Direct_IO uses a file indexes that has the type of Count or Positive_Count. 
And these types are based on the positive range which is define as the size of a 
long_Integer which is normally in a 32-bit machine set to positive value of 
( 2**63 - 1 ).
 
So it is possible from Ada, but the interface links between Ada and the OS 
may limit the size. Or in some cases it could be the OS or device-drivers 
that is limiting the file size.

But in looking at the routine System.FileIO.Write_Buf where the exception 
occurred and the Interfaces.C.Streams they both limit the file size to 
Standard'Address_Size or in the case of GNAT 32-bit version, to a positive 
range of 32-bit word aka (2GB -1). 

So, the answer is: On a 64-bit machine is limited to 2**64-1 file size 
                     and 
                     on a 32-bit machine is limited to 2**32-1 file size.
 

In <4a1aedda$0$2855$ba620e4c@news.skynet.be>, Olivier Scalbert <olivier.scalbert@algosyn.com> writes:
>Hello everybody,
>
>Here is my weekly question ....
>
>
>I need to create a file that has 2540160000 bytes, that is a little more 
>than 2**31 bytes.
>
>On a 64 bits linux box, it is ok, but on a 32 bits linux box, I have:
>raised ADA.IO_EXCEPTIONS.DEVICE_ERROR : s-fileio.adb:1135
>
>The file length is: 2147483647, which is 2**31 - 1
>
>Is it possible to write more than 2**31 bytes with an Ada program on a 
>32 bits linux ?
>
>Thanks for your help,
>
>Olivier.

Re: Large files on 32 and 64 bits ystem

[Olivier Scalbert]

Ludovic Brenta wrote:

> That line indicates that the GNAT run-time library delegates the write
> to fwrite(3), so your question really boils down to whether the C run-
> time library has support for large files or not.  What filesystem type
> do you use on your machines? (I use XFS which supports files up to 8
> exabytes :) )
I use ext3.

Olivier

Re: Large files on 32 and 64 bits ystem

[Ludovic Brenta]

Olivier Scalbert wrote on comp.lang.ada:
> Ludovic Brenta wrote:
> > That line indicates that the GNAT run-time library delegates the write
> > to fwrite(3), so your question really boils down to whether the C run-
> > time library has support for large files or not.  What filesystem type
> > do you use on your machines? (I use XFS which supports files up to 8
> > exabytes :) )
>
> I use ext3.

In this filesystem, files can grow to 16 GiB to 2 TiB depending on
block size.

[1] explains how large file support works in GNU/Linux. Basically, a C
program supports LFS if it is compiled with the -D_FILE_OFFSET_BITS=64
preprocessor option on the command line. This changes the definition
of fopen(3), fwrite(3) et al to use 64-bit file offsets instead of the
default 32-bit file offsets.

Unfortunately, the GNAT run-time library directly imports these
functions from glibc without any preprocessor in between, so is
restricted to 32-bit file offsets, and so does not support large
files.

It would be an interesting project for a beginning GCC hacker to
implement LFS in libgnat. This would involve:
- wrapper functions in adaint.c that call fopen(3), fwrite(3), etc.
- compiling adaint.c with -D_FILE_OFFSET_BITS=64
- calling the wrappers instead of the glibc functions in
System.File_IO et al. (it may be a little bit tricky to import them
properly on all platforms, with and without LFS).

[1] http://www.suse.de/~aj/linux_lfs.html

--
Ludovic Brenta.

Re: Large files on 32 and 64 bits ystem

[Robert A Duff]

Ludovic Brenta <ludovic@ludovic-brenta.org> writes:

> Unfortunately, the GNAT run-time library directly imports these
> functions from glibc without any preprocessor in between, so is
> restricted to 32-bit file offsets, and so does not support large
> files.

Recent versions of GNAT have better support for large files.
I don't know if this has made it into any public versions yet
-- probably not.

- Bob