Re: A little trouble with very large arrays.

[Niklas Holsti <niklas.holsti@tidorum.invalid>]

On 18-10-05 19:47 , Shark8 wrote:
> On Friday, October 5, 2018 at 12:20:34 AM UTC-6, Niklas Holsti wrote:
>> On 18-10-05 00:38 , Shark8 wrote:
>>> I'm trying to implement a FITS library for work -- see
>>> https://fits.gsfc.nasa.gov/standard40/fits_standard40aa.pdf --
>>> and have come across some rather interesting problems implementing it.
>>>
>>> The main-problem right now is the "Primary Data Array" which can
>>> have  a dimensionality in 1..999, each itself with some non-zero range.
>>> (In the files these are specified by keywords in the file like
>>> NAXIS = n, NAXIS1 = n_1, NAXIS2 = n_2, and so on until the NAXISn = n_n
>>> keyword/value pair is encountered.)
>>>
>>> Relatively straightforward, no?
>>
>> No. Handling arrays with a variable number of dimensions is not simple.
>>
>>> Well, I'd thought I could handle  everything with a dimensionality-array
>>> and generic like:
>>>
>>>     Type Axis_Count is range 0..999 with Size => 10;
>>>     Type Axis_Dimensions is Array (Axis_Count range <>) of Positive
>>>       with Default_Component_Value => 1;
>>> ...
>>> Generic
>>> 	Type Element is (<>);
>>> 	Dim : Axis_Dimensions:= (1..999 => 1);
>>> Package FITS.Data with Pure is
>>>
>>> 	Type Data_Array is not null access Array(
>>> 			1..Dim( 1),1..Dim( 2),1..Dim( 3),1..Dim( 4),
>>>   			1..Dim( 5),1..Dim( 6),1..Dim( 7),1..Dim( 8),
>>>   			--...
>>> 			1..Dim( 997),1..Dim( 998),1..Dim( 999)
>>> 		) of Element
>>
>> Give it some thought. Even if each dimension would have the smallest
>> sensible length, which is two index values, the total number of elements
>> in that array would be 2**999, somewhat larger than the memories of
>> current computers.
>
> No, the smallest sensible number of indices is 1, for everything except
> maybe the first two or three dimensions: eg Image data from a camera, or
> perhaps topological data from a map (longitude, latitude, elevation).

FITS images can have more dimensions than that. Further dimensions might 
be the frequency of the light (spectral imaging); polarisation; time 
when image was taken; and perhaps a couple more that don't come to mind 
immediately.

I understand what you tried to do, including having length-one 
dimensions, but I don't think that it is a sensible approach to handling 
up to 999 dimensions. I agree with the flattening approach that Dmitry 
suggested.

If your FITS files are not much larger than your RAM, the fastest 
approach is probably to "mmap" the file into your virtual address space 
and then compute the address of any given image pixel with the 
flattening method. If your FITS files are larger than your RAM, your 
program should process the file as a stream, which may or may not be 
practical, depending on what the program should output.

> FITS was developed for handling "image" transport by the astronomy
> world, back when there were 9-bit bytes and such.

I know, I used to work in astronomy. What's your point about 9-bit 
bytes? FITS standard version 4.0 defines "byte" as 8 bits, and allows 
only 8, 16, 32 and 64-bit pixels. No 9-bit pixels.

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .