Re: The future of Spark . Spark 2014 : a wreckage

["Randy Brukardt" <randy@rrsoftware.com>]

"Simon Wright" <simon@pushface.org> wrote in message 
news:lyppupvcs3.fsf@pushface.org...
> "Randy Brukardt" <randy@rrsoftware.com> writes:
>
>> problems with fragmentation. (Although those are way overblown,
>> they're very unlikely to occur in practice.)
>
> That depends on the capabilities of your OS's memory manager! In our
> VxWorks 5.5/Ada 95 system, it didn't take long for the heap to become so
> fragmented that the system couldn't meet real-time constraints (such as
> failing to kick the watchdog timer).

I suppose I should know better than to presume most programmers are 
competent. ;-)

My personal experience is that fragmentation is rare at best. We had a bit 
of trouble with it in our MS-DOS compilers, that mainly came from allocating 
lots of objects, each one slightly larger than the last (pretty much the 
worst case for heap use). Changing to a quadradic size increase (rather than 
a linear one) got rid of that problem (as did moving some tiny objects to 
their own manager [allocated once and never freed back to the main heap] - 
today I'd use a custom pool for that).

I was very worried that the long running web server and spam filter would 
suffer from fragmentation, but I've never seen any evidence of that, even 
after running two months and having processed a half a million requests. 
Apparently, the Windows heap manager is smart enough to prevent it, as the 
fragmentation goes up a bit in the first few hours after the servers get 
started and then stays pretty constant after that.

> It was a fairly straightforward job to provide our own version of GNAT's
> System.Memory to use preallocated binary-sized bins (we instrumented it
> to determine the maximum number of blocks for each bin and allocated
> that number at startup, with the ability to allocate further blocks at
> run time if the measured maximum requirement grew).
>
> I don't remember why use of Ada storage pools wasn't the right solution;
> maybe the some of the allocations were by the runtime without the option
> to use a user-defined storage pool? (things like task control blocks).

Obviously, the best solution is to replace a bad heap by a good one. Won't 
work in every circumstance, but it will in 98% of them.

                          Randy.