From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable autolearn_force=no version=3.4.4 Path: eternal-september.org!reader01.eternal-september.org!reader02.eternal-september.org!feeder.eternal-september.org!news.swapon.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail From: Niklas Holsti Newsgroups: comp.lang.ada Subject: Re: A little trouble with very large arrays. Date: Sat, 6 Oct 2018 12:35:21 +0300 Organization: Tidorum Ltd Message-ID: References: <3f2828df-d54a-4427-bc3c-dc5ef0dc8069@googlegroups.com> <871s93tw6h.fsf@jacob-sparre.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Trace: individual.net q3y2tt1zA9hjd2PzcYglLA1669GeNGInAHA7nrfmjEIHW80LEj Cancel-Lock: sha1:MZZsFKqzxkEikvjJPW9zTqCLma0= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 In-Reply-To: <871s93tw6h.fsf@jacob-sparre.dk> Xref: reader02.eternal-september.org comp.lang.ada:54502 Date: 2018-10-06T12:35:21+03:00 List-Id: On 18-10-06 09:40 , Jacob Sparre Andersen wrote: > Niklas Holsti wrote: > >> If your FITS files are not much larger than your RAM, the fastest >> approach is probably to "mmap" the file into your virtual address >> space and then compute the address of any given image pixel with the >> flattening method. If your FITS files are larger than your RAM, your >> program should process the file as a stream, which may or may not be >> practical, depending on what the program should output. > > Why not leave the transport between disk and RAM to the operating > system, and use memory mapping even if the file is larger than the RAM > of the system? One could use mmap even for very large files, I guess, but on a 32-bit system the virtual address space could run out. On a 64-bit system, probably not. This was advice based on my feeling of what would work best. If the file is processed as a stream, the OS is likely to use read-ahead to speed things up. If the file is mmap'ed to use the virtual-memory paging system, I'm not sure if the OS will do read-ahead, but perhaps modern OS's have some such adaptive optimisations even for mmap'ed files. -- Niklas Holsti Tidorum Ltd niklas holsti tidorum fi . @ .