Re: A little trouble with very large arrays.

[Niklas Holsti <niklas.holsti@tidorum.invalid>]

On 18-10-05 00:38 , Shark8 wrote:
> I'm trying to implement a FITS library for work -- see
> https://fits.gsfc.nasa.gov/standard40/fits_standard40aa.pdf --
> and have come across some rather interesting problems implementing it.
>
> The main-problem right now is the "Primary Data Array" which can
> have  a dimensionality in 1..999, each itself with some non-zero range.
> (In the files these are specified by keywords in the file like
> NAXIS = n, NAXIS1 = n_1, NAXIS2 = n_2, and so on until the NAXISn = n_n
> keyword/value pair is encountered.)
>
> Relatively straightforward, no?

No. Handling arrays with a variable number of dimensions is not simple.

> Well, I'd thought I could handle  everything with a dimensionality-array
> and generic like:
>
>     Type Axis_Count is range 0..999 with Size => 10;
>     Type Axis_Dimensions is Array (Axis_Count range <>) of Positive
>       with Default_Component_Value => 1;
> ...
> Generic
> 	Type Element is (<>);
> 	Dim : Axis_Dimensions:= (1..999 => 1);
> Package FITS.Data with Pure is
>
> 	Type Data_Array is not null access Array(
> 			1..Dim( 1),1..Dim( 2),1..Dim( 3),1..Dim( 4),
>   			1..Dim( 5),1..Dim( 6),1..Dim( 7),1..Dim( 8),
>   			--...
> 			1..Dim( 997),1..Dim( 998),1..Dim( 999)
> 		) of Element

Give it some thought. Even if each dimension would have the smallest 
sensible length, which is two index values, the total number of elements 
in that array would be 2**999, somewhat larger than the memories of 
current computers.

> What's the proper way to go about doing this?

If you really want to support up to 999 dimensions (though I doubt that 
any real FITS file will be close to that number), your program has to 
manage the data in blocks of some practical size.

> (As another interesting constraint, the file-format mandates a sort
> of block-structure of 2880 bytes [23040 bits], and while I don't
> anticipate this being an issue, something that might be relevant.)

Perhaps that is the solution, not a new problem.

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .