Re: Reading the while standard input into a String

[Natasha Kerensikova <lithiumcat@gmail.com>]

Hello,

On 2011-06-06, Ludovic Brenta <ludovic@ludovic-brenta.org> wrote:
> Natasha Kerensikova wrote on comp.lang.ada:
>> I have encountered what was a very easily solved problem in C: dump the
>> whole contents of the standard input into some place in memory, in order
>> to process it afterwards.
>
> Personally I think this is a fundamentally bad idea.  Consider:
>
> $ my_program < /dev/random
>
> Do you consider it acceptable to use all memory and all swap space
> before finally bailing out with a Storage_Error and not doing any
> actual processing?

Actually, in principle, yes. I mean, I want to be able to write code
that behave that way as far as the "reading the whole standard input"
goes, because that's this part I'm trying to learn about.

Of course, I don't want a real program to behave that way, so I would
probably add a "reasonable" limit (like 16 MB) that interrupt the loop,
but that's beyond the scope of my question. And anyway, I don't trust
any program to not use all memory and all swap space, so I have posix
ulimits set accordingly.

> I'd rather process the input one fixed-length chunk at a time and
> discard each chunk after processing (possibly by recycling the in-
> memory buffer for the next chunk).

I would do so if online processing was an option. However in the program
containing code similar to what I posted, I cannot do any processing
before reaching the end of input, because the processed format allows
forward references, so references have to be gathered in a first pass on
the whole input, before doing a second pass where dereference can take
place.

> Note that reading one character at a time from a stream is quite fast
> since streams normally use the operating system's buffered IO. Adding
> a second layer of buffering inside your program is probably counter-
> productive.

That second layer of buffer was actually meant to reduce the number of
Ada.Strings.Unbounded.Append calls (and the expensive reallocations that
go with them).

However I thought that something similar to C's fread(), which can use
efficient bulk memory copy, would be better than a loop of single
character read (even when buffered in memory).


Thanks for your comments,
Natasha