Re: Array and memory usage

["Randy Brukardt" <randy@rrsoftware.com>]

Robert A Duff wrote in message ...
>mheaney@on2.com (Matthew Heaney) writes:
>
>> Robert A Duff <bobduff@shell01.TheWorld.com> wrote in message
news:<wccr89ge7hw.fsf@shell01.TheWorld.com>...
>> >
>> > If you use this trick on Windows, it will allocate backing store
for all
>> > that virtual memory.  I believe most Unix systems are smarter than
that.
>> >
>> > This trick is sometimes more efficient than linked lists for
growable
>> > data structures.  The links waste space, plus if you're not
careful, the
>> > list gets scattered all over the place, damaging cache behavior.
The
>> > nice thing about arrays is that they are contiguous.
>>
>> One thing you can on Windows is simply reserve the virtual memory for
>> the array, without committing it to physical memory, and then use SEH
>> ("structured exception handling") to handle the error.
>>
>> The SEH mechanism supports resumption semantics, so you can handle
the
>> exception by committing the page you need, and then resuming from the
>> spot from where the exception was raised.
>>
>> This is documented in Jeff Richter's Programming Applications book.
>
>I thought it happened automatically.  That is, if I say:
>
>    X: array (1..100_000_000) of Integer;
>
>it would allocate 400,000,000 bytes of virtual memory, but only
allocate
>physical pages on demand.  I believe that much works on Windows
(without
>calling windows-specific stuff like SEH).  (Windows 2000, Windows XP.)
>The problem I found was that it allocated 400,000,000 bytes of backing
>store, which was annoying.
>
>Am I misinformed?


Sort of. Windows has three stages of memory allocation, address space
allocation, virtual page allocation, and physical page allocation. The
first two are (partially) under control of the programmer.

Swap space is allocated when virtual pages are allocated (called
"commit"). If the pages aren't commited, an attempt to access them
causes a page fault. (That's the idea Matt was talking about).

Memory that is allocated in a load image (even as address space) is
always commited, so that it can be accessed without a fault -- but that
requires swap space to be allocated (even if no physical memory will
ever be used).

The proper way to deal with this is to avoid allocating the address
space at compile-time; rather use dynamic allocation to do it. That is
easy to do in Ada by writing a storage pool that allocates the address
space immediately, then commits additional memory only as needed.

In my application, I did it by using accesses to unconstrained arrays,
and having an "Expand" routine. When one became full, I called the
Expand routine, which determined if uncommitted address space followed
the acces, and if it did, commited the needed memory and adjusted the
array descriptor for the access type. Thus, all of the expansion was
done in place. Of course, adjusting the array descriptor is not a safe
or portable operation, and it might not even be possible on some
compilers. If the memory couldn't be expanded, then I would allocate the
needed larger array from the pool, copy it, and free the old array --
that is, the standard way of expanding dynamic arrays.

One warning: address space has to be allocated in 64K chunks. If you try
to use smaller chunks than that, it get rounded up to 64K. (I found that
out the hard way).

              Randy.