Re: A little trouble with very large arrays.

[Niklas Holsti <niklas.holsti@tidorum.invalid>]

On 18-10-06 09:40 , Jacob Sparre Andersen wrote:
> Niklas Holsti wrote:
>
>> If your FITS files are not much larger than your RAM, the fastest
>> approach is probably to "mmap" the file into your virtual address
>> space and then compute the address of any given image pixel with the
>> flattening method. If your FITS files are larger than your RAM, your
>> program should process the file as a stream, which may or may not be
>> practical, depending on what the program should output.
>
> Why not leave the transport between disk and RAM to the operating
> system, and use memory mapping even if the file is larger than the RAM
> of the system?

One could use mmap even for very large files, I guess, but on a 32-bit 
system the virtual address space could run out. On a 64-bit system, 
probably not.

This was advice based on my feeling of what would work best. If the file 
is processed as a stream, the OS is likely to use read-ahead to speed 
things up. If the file is mmap'ed to use the virtual-memory paging 
system, I'm not sure if the OS will do read-ahead, but perhaps modern 
OS's have some such adaptive optimisations even for mmap'ed files.

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .