I/O streaming with custom data transport

I/O streaming with custom data transport

[Maciej Sobczak]

Hi,

In C++ the IOStreams library allows to vary the behaviour of the stream 
by decoupling formatting from data buffering and transport to the 
physical device, which are in turn strategies for the stream object. 
This means that having millions of functions like this:

void foo(ostream &s)
{
     s << "Hello";
     s << someObject;
     s << someOtherObject;
     // ...
}

(and *there are* millions of functions like this)

it is possible to reuse these functions with different stream objects, 
from standard console output:

foo(cout);

...to file output:

foo(myFile);

...to network socket output:

foo(someSocketStreamConnectedSomewhere);

...to whatever else comes up to the programmer's mind. It's just the 
matter of providing the implementation for the stream buffer.
The key point here is that the mass of code out there need not be aware 
of it and can be reused with new stream objects without any 
modifications (that's OO in action).

I'm looking for something like this in Ada.

The basic I/O facilities in the standard library don't seem to provide 
anything like this.
I hoped that Ada.Streams allows this by subclassing Root_Stream_Type and 
providing some overriding operations, but unfortunately I cannot even 
find the specification of Root_Stream_Type (looks like there isn't any 
and this type is just a name placeholder in ARM).

If the standard library does not provide the functionality that I'm 
looking for (and I hope that I'm still misunderstanding something here), 
did anybody tackle the problem by implementing some replacement IO 
library with the functionality similar to that of IOStreams in C++?

-- 
Maciej Sobczak : http://www.msobczak.com/
Programming    : http://www.msobczak.com/prog/

Re: I/O streaming with custom data transport

[gautier_niouzes]

Perhaps you are looking for something like Unzip.Streams (here, a
custom input stream) ? You can browse from...

http://homepage.sunrise.ch/mysunrise/gdm/uza_html/teunzstr__adb.htm

HTH, Gautier

Re: I/O streaming with custom data transport

[Alex R. Mosteo]

Maciej Sobczak wrote:

> Hi,
> 
> In C++ the IOStreams library allows to vary the behaviour of the stream
> by decoupling formatting from data buffering and transport to the
> physical device, which are in turn strategies for the stream object.
> This means that having millions of functions like this:
> 
> void foo(ostream &s)
> {
>      s << "Hello";
>      s << someObject;
>      s << someOtherObject;
>      // ...
> }
> 
> (and *there are* millions of functions like this)
> 
> it is possible to reuse these functions with different stream objects,
> from standard console output:
> 
> foo(cout);
> 
> ...to file output:
> 
> foo(myFile);
> 
> ...to network socket output:
> 
> foo(someSocketStreamConnectedSomewhere);
> 
> ...to whatever else comes up to the programmer's mind. It's just the
> matter of providing the implementation for the stream buffer.
> The key point here is that the mass of code out there need not be aware
> of it and can be reused with new stream objects without any
> modifications (that's OO in action).
> 
> I'm looking for something like this in Ada.
> 
> The basic I/O facilities in the standard library don't seem to provide
> anything like this.
> I hoped that Ada.Streams allows this by subclassing Root_Stream_Type and
> providing some overriding operations, but unfortunately I cannot even
> find the specification of Root_Stream_Type (looks like there isn't any
> and this type is just a name placeholder in ARM).

I think you haven't looked right. That's precisely how it's done. I have
chainable streams for compression, buffering, file I/O, memory streaming,
etc. And you do this just as you say: you extend
Ada.Streams.Root_Stream_Type and provide the read/write subprograms. Then,
you can dispatch on these or use the 'Input/'Output attributes of any type.
Conversely, you can override these to provide your own stream
representation of any type.

(I mention calling the subprograms directly instead of the 'attributes
because for some array types it's much more efficient -- at least it was
with GNAT 3.15p. But, usually, you just want to use the attributes).

Maybe is a compiler matter. In GNAT you can inspect the specification, but
if this is done by compiler magic in your case, still you should be able to
override the Read/Write subprograms in your derived stream type. The
package spec is in the RM05 13.13.1.

> If the standard library does not provide the functionality that I'm
> looking for (and I hope that I'm still misunderstanding something here),
> did anybody tackle the problem by implementing some replacement IO
> library with the functionality similar to that of IOStreams in C++?
 
If you're using GPS, go to Help -> GNAT runtime -> Ada -> Streams. You have
there the spec of Root_Stream_Type.

Ada 2005 adds new features for some kind of magic needed when inputting
types from an stream. I have had no need for this, but you may be as well
interested. It's all in the Barnes' rationale articles.

Re: I/O streaming with custom data transport

[Dmitry A. Kazakov]

On Tue, 21 Nov 2006 16:11:02 +0100, Maciej Sobczak wrote:

> In C++ the IOStreams library allows to vary the behaviour of the stream 
> by decoupling formatting from data buffering and transport to the 
> physical device, which are in turn strategies for the stream object. 
> This means that having millions of functions like this:
> 
> void foo(ostream &s)
> {
>      s << "Hello";
>      s << someObject;
>      s << someOtherObject;
>      // ...
> }
> 
> (and *there are* millions of functions like this)

Alas, because it is a poor design based on ad-hoc polymorphism. All these
millions are overloaded. They should be overridden, but that were
impossible due to lack of multiple dispatch.

> I'm looking for something like this in Ada.

In Ada it is exactly same. Consider:

   Put (S : in out My_Root_Stream; X : String);
   Put (S : in out My_Root_Stream; X : SomeObject);
   ...

An alternative design (still non-MD) is:

   Put (S : in out Root_Stream'Class; X : String);
   Put (S : in out Root_Stream'Class; X : SomeObject);
   ...

This achieves what you want. You can implement I/O on an object type like
String using some common class-wide functionality of streams and then
re-use it over all possible streams.

> The basic I/O facilities in the standard library don't seem to provide 
> anything like this.
> I hoped that Ada.Streams allows this by subclassing Root_Stream_Type and 
> providing some overriding operations, but unfortunately I cannot even 
> find the specification of Root_Stream_Type (looks like there isn't any 
> and this type is just a name placeholder in ARM).

As I said it is basically same. But Ada also offers a sort of hard-wired
double dispatch through S'Read/Write, S'Input/Output attributes. See ARM
13.13.2.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: I/O streaming with custom data transport

[Maciej Sobczak]

Alex R. Mosteo wrote:

>> I'm looking for something like this in Ada.
>>
>> The basic I/O facilities in the standard library don't seem to provide
>> anything like this.
>> I hoped that Ada.Streams allows this by subclassing Root_Stream_Type and
>> providing some overriding operations, but unfortunately I cannot even
>> find the specification of Root_Stream_Type (looks like there isn't any
>> and this type is just a name placeholder in ARM).
> 
> I think you haven't looked right.

Indeed - it's in 13.13.1.

> That's precisely how it's done.

> And you do this just as you say: you extend
> Ada.Streams.Root_Stream_Type and provide the read/write subprograms.

And I actually managed to do this.
Consider the following example of custom stream that for the sake of 
presentation is bound to standard IO and just converts characters to 
uppercase when writing (the point is that if I can do *this*, I can do 
anything else):

-- file my_streams.ads:
with Ada.Streams;
package My_Streams is
    use Ada.Streams;

    type My_Stream is new Root_Stream_Type with null record;

    procedure Read(Stream : in out My_Stream;
                   Item : out Stream_Element_Array;
                   Last : out Stream_Element_Offset);

    procedure Write(Stream : in out My_Stream;
                    Item : in Stream_Element_Array);
end My_Streams;

-- file my_streams.adb:
with Ada.Text_IO.Text_Streams;
with Ada.Characters.Handling;
package body My_Streams is
    use Ada.Text_IO;
    use Ada.Text_IO.Text_Streams;

    Std_Stream : Stream_Access := Stream(Current_Output);

    procedure Read(Stream : in out My_Stream;
                   Item : out Stream_Element_Array;
                   Last : out Stream_Element_Offset) is
    begin
       -- forward to standard streams:
       Read(Std_Stream.all, Item, Last);
    end Read;

    procedure Write(Stream : in out My_Stream;
                    Item : in Stream_Element_Array) is
       Item_Uppercase : Stream_Element_Array := Item;
       C : Character;
    begin
       for I in Item_Uppercase'Range loop
          C := Character'Val(Item_Uppercase(I));
          C := Ada.Characters.Handling.To_Upper(C);
          Item_Uppercase(I) := Stream_Element(Character'Pos(C));
       end loop;

       -- forward to standard streams:
       Write(Std_Stream.all, Item_Uppercase);
    end Write;
end My_Streams;

-- file hello.adb:
with Ada.Text_IO.Text_Streams;
with My_Streams;

procedure Hello is
    use Ada.Text_IO;
    use Ada.Text_IO.Text_Streams;
    use My_Streams;

    procedure Write_Hello(Stream : in out Stream_Access) is
    begin
       String'Write(Stream, "Hello");
    end Write_Hello;

    S1 : Stream_Access := Stream(Current_Output);
    S2 : Stream_Access := new My_Stream;
begin
    Write_Hello(S1);
    New_Line;
    Write_Hello(S2);
    New_Line;
end Hello;

The result is:

$ ./hello
Hello
HELLO
$

The point here is that the Write_Hello procedure can be reused with 
various streams, just like my foo functions in C++ (from initial post).

Is the code above correct? Any traps or problems that I don't see at the 
moment?


-- 
Maciej Sobczak : http://www.msobczak.com/
Programming    : http://www.msobczak.com/prog/

Re: I/O streaming with custom data transport

[Maciej Sobczak]

Dmitry A. Kazakov wrote:

>> In C++ the IOStreams library allows to vary the behaviour of the stream 
>> by decoupling formatting from data buffering and transport to the 
>> physical device, which are in turn strategies for the stream object. 
>> This means that having millions of functions like this:
>>
>> void foo(ostream &s)
>> {
>>      s << "Hello";
>>      s << someObject;
>>      s << someOtherObject;
>>      // ...
>> }
>>
>> (and *there are* millions of functions like this)
> 
> Alas, because it is a poor design based on ad-hoc polymorphism. All these
> millions are overloaded.

I don't understand. The focus here is on foo - which represents the set 
of functions that can use the stream without really caring what the 
stream is.

> In Ada it is exactly same. Consider:
> 
>    Put (S : in out My_Root_Stream; X : String);
>    Put (S : in out My_Root_Stream; X : SomeObject);
>    ...
> 
> An alternative design (still non-MD) is:
> 
>    Put (S : in out Root_Stream'Class; X : String);
>    Put (S : in out Root_Stream'Class; X : SomeObject);
>    ...
> 
> This achieves what you want. You can implement I/O on an object type like
> String using some common class-wide functionality of streams and then
> re-use it over all possible streams.

OK, now I see - there is a bit of misunderstanding. I'm not asking about 
how to provide new formatting, but rather how to provide new data 
transport. The practical intention is that I want to write some code 
that will be reused both with file streams and with network streams (for 
example).

Please see the example code in my reply to Alex, I think I've found the 
functionality that I'm looking for.

-- 
Maciej Sobczak : http://www.msobczak.com/
Programming    : http://www.msobczak.com/prog/

Re: I/O streaming with custom data transport

[Alex R. Mosteo]

Maciej Sobczak wrote:

> The point here is that the Write_Hello procedure can be reused with
> various streams, just like my foo functions in C++ (from initial post).
> 
> Is the code above correct? Any traps or problems that I don't see at the
> moment?

That's it. 

About traps, more complex streams may require special care: for example,
imagine your characters are multi-byte and you receive an incomplete
character code in the write function. This could happen if, for example, a
previous buffering stream doesn't respect boundaries of received data. In
those cases, you need to start worrying about what to do, but I'd say this
is pretty much implementation detail not related to the general stream
pattern.

Just a nitpick: in your example, instead of allocating a new stream I would
simply get an access to a stack-allocated one:

declare
   S2 : aliased My_Stream;
begin
� �Write_Hello (S2'Access);
end;

Re: I/O streaming with custom data transport

[Dmitry A. Kazakov]

On Wed, 22 Nov 2006 10:21:02 +0100, Maciej Sobczak wrote:

> I don't understand. The focus here is on foo - which represents the set 
> of functions that can use the stream without really caring what the 
> stream is.

OK, then it is just what Ada calls a class-wide subprogram:

   type My_Abstract_Stream is abstract new Root_Stream_Type ...;

   -- Somewhere else
   procedure Foo (S : in out My_Abstract_Stream'Class);

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: I/O streaming with custom data transport

[Maciej Sobczak]

Alex R. Mosteo wrote:

> About traps, more complex streams may require special care: for example,
> imagine your characters are multi-byte and you receive an incomplete
> character code in the write function.

Sure.

But what about buffering the input/output. Is this already done (where?) 
or is this something I have to take care of?

Another issue - what about the text formatting? The standard Put(...) 
procedures are quite powerful, is there any way to reuse them (or at 
least their logic) with custom streams?

> Just a nitpick: in your example, instead of allocating a new stream I would
> simply get an access to a stack-allocated one:
> 
> declare
>    S2 : aliased My_Stream;
> begin
>    Write_Hello (S2'Access);
> end;

Yes, thanks.

-- 
Maciej Sobczak : http://www.msobczak.com/
Programming    : http://www.msobczak.com/prog/

Re: I/O streaming with custom data transport

[Robert A Duff]

Maciej Sobczak <no.spam@no.spam.com> writes:

>    procedure Write_Hello(Stream : in out Stream_Access) is
>    begin
>       String'Write(Stream, "Hello");
>    end Write_Hello;

I don't understand this.  Ada streams are streams of Stream_Elements
(usually bytes), and the 'Read/'Write attributes encode things in
some way that is not necessarily human-readable.  For example,
I think the above outputs the bounds of the string in some
binary form.

But the original question was about streams of characters,
intended to be human-readable.  Right?

So how do Ada's streams help, here?

It seems to me what you want is an abstract Character_Output_Stream
type, with a primitive (or class-wide?) operation that puts a Character
to the stream.  Then you can make derived types for sending to a file,
to an in-memory buffer, etc.

- Bob

Re: I/O streaming with custom data transport

[Maciej Sobczak]

Robert A Duff wrote:

>>    procedure Write_Hello(Stream : in out Stream_Access) is
>>    begin
>>       String'Write(Stream, "Hello");
>>    end Write_Hello;
> 
> I don't understand this.  Ada streams are streams of Stream_Elements
> (usually bytes), and the 'Read/'Write attributes encode things in
> some way that is not necessarily human-readable.

That's right, but the above was written for experimental/presentation 
purposes and was based on the assumption that the implementation will 
not unnecessarily complicate things so that the output will anyway be 
easy to verify to show that the principle works.

>  For example,
> I think the above outputs the bounds of the string in some
> binary form.

T'Input/'Output do, but T'Write/'Read don't.

> But the original question was about streams of characters,
> intended to be human-readable.  Right?

Not necessarily. The point was only to make sure that there is a 
possiblity to use streams in a polimorphic way. Or to put it more 
precisely, where the existing code can be reused with different data 
transport strategies and where the reuse method will be based on 
run-time polymorphism and not, for example, on generics.

> So how do Ada's streams help, here?

Taking into account these objective, they do. :-)

Of course, the question about text streams is still valid - and was left 
open in one of my other posts.

For the moment I assume that in order to do this with text, I have to 
first format it in some string buffer and then do:

String'Write(S, Buf);

(just like with the "Hello" above)

Taking into account that T'Write does not output constraints, it might 
be a valid method to insert text into the stream. Is this correct?

> It seems to me what you want is an abstract Character_Output_Stream
> type, with a primitive (or class-wide?) operation that puts a Character
> to the stream.  Then you can make derived types for sending to a file,
> to an in-memory buffer, etc.

That makes sense as well, but seems to be further off from what the 
standard library offers out of the box.

-- 
Maciej Sobczak : http://www.msobczak.com/
Programming    : http://www.msobczak.com/prog/

Re: I/O streaming with custom data transport

[Dmitry A. Kazakov]

On Wed, 22 Nov 2006 11:39:48 +0100, Maciej Sobczak wrote:

> Another issue - what about the text formatting? The standard Put(...) 
> procedures are quite powerful, is there any way to reuse them (or at 
> least their logic) with custom streams?

?

Stream-oriented I/O is unformatted per concept of. It is a stream of
elements, not even characters. Formatted I/O is when there are some things
beyond characters, like lines, columns, pages etc.

For formatted buffered I/O use strings. As a block/record/line use String.
You can find an example of string formatting library here:

   http://www.dmitry-kazakov.de/ada/strings_edit.htm

For physical I/O, which is a separate issue, you can still use streams,
files, sockets, whatsoever. Just define a primitive operation Write_Line on
the abstract type of the "recipient," which takes String argument. For all
further types of objects use that primitive operation.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: I/O streaming with custom data transport

[Georg Bauhaus]

On Wed, 2006-11-22 at 15:37 +0100, Maciej Sobczak wrote:

> For the moment I assume that in order to do this with text, I have to 
> first format it in some string buffer and then do:
> 
> String'Write(S, Buf);

Have you considered 
 for T'Write use Your_Function...

Also, Ada.Text_IO.Text_Streams is an example of how to use
the attribute functions with text file streams. So you'd have
the formatting in 'Write and the transport mechanics in the
new byte stream type.

Re: I/O streaming with custom data transport

[Alex R. Mosteo]

Maciej Sobczak wrote:

> Alex R. Mosteo wrote:
> 
>> About traps, more complex streams may require special care: for example,
>> imagine your characters are multi-byte and you receive an incomplete
>> character code in the write function.
> 
> Sure.
> 
> But what about buffering the input/output. Is this already done (where?)
> or is this something I have to take care of?

Not in the standard library AFAIK.

Re: I/O streaming with custom data transport

[Simon Wright]

Maciej Sobczak <no.spam@no.spam.com> writes:

> But what about buffering the input/output. Is this already done
> (where?) or is this something I have to take care of?

I would expect the OS to do this in most cases. However .. I have in
mind, for a work problem, using an alternate transport layer for a
message stream where we use GNAT.Sockets at present, and for that I
will have to do the buffering that the new transport requires
myself. The users will be unaware, though there will be an issue when
the stream is closed (the last buffer will have to be padded and
flushed). This sort of thing could be handled in a Close operation on
the new stream type. And of course there will have to be some means of
associating the new stream type with the underlying stream.