From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.4 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: 103376,d02eb5c33ac65d9 X-Google-Attributes: gid103376,public X-Google-ArrivalTime: 2003-03-10 16:47:57 PST Path: archiver1.google.com!news1.google.com!newsfeed.stanford.edu!news-spur1.maxwell.syr.edu!news.maxwell.syr.edu!sn-xit-03!sn-xit-06!sn-post-01!supernews.com!corp.supernews.com!not-for-mail From: "Randy Brukardt" Newsgroups: comp.lang.ada Subject: Re: Array and memory usage Date: Mon, 10 Mar 2003 18:40:27 -0600 Organization: Posted via Supernews, http://www.supernews.com Message-ID: References: <1ec946d1.0303100713.7f7bcbb7@posting.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Newsreader: Microsoft Outlook Express 4.72.3612.1700 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3719.2500 X-Complaints-To: abuse@supernews.com Xref: archiver1.google.com comp.lang.ada:35158 Date: 2003-03-10T18:40:27-06:00 List-Id: Robert A Duff wrote in message ... >mheaney@on2.com (Matthew Heaney) writes: > >> Robert A Duff wrote in message news:... >> > >> > If you use this trick on Windows, it will allocate backing store for all >> > that virtual memory. I believe most Unix systems are smarter than that. >> > >> > This trick is sometimes more efficient than linked lists for growable >> > data structures. The links waste space, plus if you're not careful, the >> > list gets scattered all over the place, damaging cache behavior. The >> > nice thing about arrays is that they are contiguous. >> >> One thing you can on Windows is simply reserve the virtual memory for >> the array, without committing it to physical memory, and then use SEH >> ("structured exception handling") to handle the error. >> >> The SEH mechanism supports resumption semantics, so you can handle the >> exception by committing the page you need, and then resuming from the >> spot from where the exception was raised. >> >> This is documented in Jeff Richter's Programming Applications book. > >I thought it happened automatically. That is, if I say: > > X: array (1..100_000_000) of Integer; > >it would allocate 400,000,000 bytes of virtual memory, but only allocate >physical pages on demand. I believe that much works on Windows (without >calling windows-specific stuff like SEH). (Windows 2000, Windows XP.) >The problem I found was that it allocated 400,000,000 bytes of backing >store, which was annoying. > >Am I misinformed? Sort of. Windows has three stages of memory allocation, address space allocation, virtual page allocation, and physical page allocation. The first two are (partially) under control of the programmer. Swap space is allocated when virtual pages are allocated (called "commit"). If the pages aren't commited, an attempt to access them causes a page fault. (That's the idea Matt was talking about). Memory that is allocated in a load image (even as address space) is always commited, so that it can be accessed without a fault -- but that requires swap space to be allocated (even if no physical memory will ever be used). The proper way to deal with this is to avoid allocating the address space at compile-time; rather use dynamic allocation to do it. That is easy to do in Ada by writing a storage pool that allocates the address space immediately, then commits additional memory only as needed. In my application, I did it by using accesses to unconstrained arrays, and having an "Expand" routine. When one became full, I called the Expand routine, which determined if uncommitted address space followed the acces, and if it did, commited the needed memory and adjusted the array descriptor for the access type. Thus, all of the expansion was done in place. Of course, adjusting the array descriptor is not a safe or portable operation, and it might not even be possible on some compilers. If the memory couldn't be expanded, then I would allocate the needed larger array from the pool, copy it, and free the old array -- that is, the standard way of expanding dynamic arrays. One warning: address space has to be allocated in 64K chunks. If you try to use smaller chunks than that, it get rounded up to 64K. (I found that out the hard way). Randy.