Re: Reading a line of arbitrary length

[Rex Reges <reges@mds.lmco.com>]

Robert Dewar wrote:

> If your point is that it is non-trivial to ensure efficient execution of
> programs that use variable length strings that are millions of characters
> long, then yes, that is true, it is non-trivial, and also non-important!
> 

What you say about very large variable length strings
is true. I did not clearly express what I was talking about:
one very long string and that string is of fixed length.

The case I was thinking of is a distributed real-time simulator.  It was
distributed in that it consisted of several Ada mains which may or may 
not run on the same CPUs within the same computer.  It was real-time
in that some of the processes had to repeatedly complete a specific 
amount of work within set frames of 120 Hz, 60Hz and 30Hz.  

The problem was how to pass string information quickly.  This 
string information consisted of error messages, symbol names, and 
commands.  The solution used was to create a string heap that 
was similar to the regular heap except that persistent strings
were compressed and that each process could add to its string heap 
from entries in another processes string heap.  

Integer IDs for strings were provided at string creation time and
were gauranteed to be unique among all the processes.  The code
which used the string heap was unaware of the compression and 
communication mechanisms.  This allowed strings to be passed 
around as if using access types, but provided global distributed
strings.

The large string issue arose in the string heap itself.
A single large fixed-length string was used as the heap.
Start index and size were used to keep track of each string
heap entry.  

It was possible to save the string heap to a file and then 
restore it later.  This was useful for the symbol table data
which doesn't change from one execution to another.  This
caused a problem on the Vax: not being able to read in a 
5 million character string in one chunk.  

I've left out a lot of details which are hopefully 
irrelevant to this discussion, but I would like to 
mention the compression needs.  The symbol information
consisted of 70,000 symbols which were fully qualified
Ada names.  It also included the enumeration values 
for those symbols which were enumeration types.  This
information alone was over 5 megabytes.  The compression
mechanism was simple: a vocabulary of frequently
occuring substrings was created, then each symbol was 
saved as an array of vocabulary entries.  

The constraints of the real-time response
precluded the usual fixes like storing strings in a file.
It was necessary to keep the memory usage to a minimum
so that the programs could be locked in memory to prevent
page swapping.
 
It's good news to hear about the pattern matching!  Up to
now I've been satisfied writing string tools as needed, but 
the general feeling I got from other Ada programmers is that 
Ada doesn't do strings.  


-- 
Rex Reges                      or you can call me The Fixer 
Systems Analyst                or you can call me The Lawyer
Lockheed Martin, M&DS          or you can call me The Doctor
(610)354-5047                  or you can call me Rexasaurus