Re: Looking for keyed file package

[kilgallen@eisner.decus.org (Larry Kilgallen)]

In article <7sijv6$tc8$1@nnrp1.deja.com>, Robert Dewar <robert_dewar@my-deja.com> writes:

>> but I think that all started back when disk and memory space
>> were much more dear.  I/O bandwidth one area of computer
>> performance that does not seem to be advancing so much as the
>> others, so these days typical advice might be to engage in
>> extensive caching of the index data if performance is
>> important.
> 
> The whole *point* of key compression is to allow this caching
> to be more efficient. You want as much of the key index in main
> memory as possible (and if you are really pushing things, in
> cache as possible). Key compression is vitally important for
> this, and in fact it is MORE important with modern architectures
> than with older ones, precisely because of the increasing gap
> between processor/memory speed and I/O speed.

Since you say "in main memory as possible (and ...in cache",
I gather you are talking about processor memory cache, whereas
what I meant was having the index in the caches created by the
indexed file system in main memory.

Presuming the actual data will likely require a disk read,
I think the difference between index data being in processor
cache and being in the file system cache in main memory is
not of consequence.

The data I have been dealing with this month have a maximum
of 16 bytes of key data for a record length of 600 bytes,
so the standard recommendation to set the cache size large
enough to hold the entire index tree in memory seems reasonable.
Memory is cheap enough, however, that even not having the
keys compressed would be acceptable in this setting. That
would be quite different with 584 bytes of key data for
a record length of 600 bytes.  Of course particular key
data can be more or less susceptible to compression.

Larry Kilgallen