Ada x <whatever> Datagram Sockets

Ada x <whatever> Datagram Sockets

[Rego, P.]

I am trying to communicate a Windows Ada application with another application in the same machine (an specific C++ porting called MQL) using datagram sockets. The core Ada app was extracted from GNAT.Sockets documentation, from the ping-pong caller. 

with GNAT.Sockets;
with Text_IO;
with Ada.Exceptions; use Ada.Exceptions;

procedure DG_SERVER is
   package SOCKETS renames GNAT.Sockets;
   Socket  : SOCKETS.Socket_Type;
   Address : SOCKETS.Sock_Addr_Type;
   Channel : SOCKETS.Stream_Access;
   Group : constant String := "239.255.128.128";

begin

   SOCKETS.Initialize;
   SOCKETS.Create_Socket (Socket, SOCKETS.Family_Inet, SOCKETS.Socket_Datagram);
   SOCKETS.Set_Socket_Option
     (Socket,
      SOCKETS.Socket_Level,
      (SOCKETS.Reuse_Address, True));
   SOCKETS.Set_Socket_Option
     (Socket,
      SOCKETS.IP_Protocol_For_IP_Level,
      (SOCKETS.Multicast_TTL, 1));
   SOCKETS.Set_Socket_Option
     (Socket,
      SOCKETS.IP_Protocol_For_IP_Level,
      (SOCKETS.Multicast_Loop, True));
   Address.Addr := SOCKETS.Any_Inet_Addr;
   Address.Port := 55505;
   SOCKETS.Bind_Socket (Socket, Address);
   SOCKETS.Set_Socket_Option
     (Socket,
      SOCKETS.IP_Protocol_For_IP_Level,
      (SOCKETS.Add_Membership, SOCKETS.Inet_Addr (Group), SOCKETS.Any_Inet_Addr));
   Address.Addr := SOCKETS.Inet_Addr (Group);
   loop
      Channel := SOCKETS.Stream (Socket, Address);
      declare
         Message : String := String'Input (Channel);
      begin
         Address := SOCKETS.Get_Address (Channel);
         Text_IO.Put_Line (Message & " from " & SOCKETS.Image (Address));
         String'Output (Channel, Message);
      end;
   end loop;
exception
   when The_Error : others =>
      Text_IO.Put_Line("!!! "&Ada.Exceptions.Exception_Information (The_Error));
end DG_SERVER;

The problem is that when I send the message (from MQL side), my Ada server returns

raised GNAT.SOCKETS.SOCKET_ERROR : [10040] Message too long
Call stack traceback locations:
0x424e6c 0x426ddb 0x426e17 0x4200a8 0x42174e 0x4019fc 0x40246d 0x4013db 0x74198482 0x778f3ab6 0x778f3a86

and investigating the libraries, Channel : SOCKETS.Stream_Access will really receive any data size.

So... how can avoid the exception?

Thanks!

Re: Ada x <whatever> Datagram Sockets

[Jere]

On Wednesday, February 6, 2019 at 6:10:37 PM UTC-5, Rego, P. wrote:
> I am trying to communicate a Windows Ada application with another application in the same machine (an specific C++ porting called MQL) using datagram sockets. The core Ada app was extracted from GNAT.Sockets documentation, from the ping-pong caller. 
> *SNIPPED*
> 
> The problem is that when I send the message (from MQL side), my Ada server returns
> 
> raised GNAT.SOCKETS.SOCKET_ERROR : [10040] Message too long
> Call stack traceback locations:
> 0x424e6c 0x426ddb 0x426e17 0x4200a8 0x42174e 0x4019fc 0x40246d 0x4013db 0x74198482 0x778f3ab6 0x778f3a86
> 
> and investigating the libraries, Channel : SOCKETS.Stream_Access will really receive any data size.
> 
> So... how can avoid the exception?
> 
> Thanks!

How big of a message are you trying to send?  Datagram sockets have
an upper limit on the payload size (65507 bytes for ipv4) regardless
of the language.

Re: Ada x <whatever> Datagram Sockets

[Rego, P.]

On Wednesday, February 6, 2019 at 10:42:56 PM UTC-2, Jere wrote:
> On Wednesday, February 6, 2019 at 6:10:37 PM UTC-5, Rego, P. wrote:
> > I am trying to communicate a Windows Ada application with another application in the same machine (an specific C++ porting called MQL) using datagram sockets. The core Ada app was extracted from GNAT.Sockets documentation, from the ping-pong caller. 
> > *SNIPPED*
> > 
> > The problem is that when I send the message (from MQL side), my Ada server returns
> > 
> > raised GNAT.SOCKETS.SOCKET_ERROR : [10040] Message too long
> > Call stack traceback locations:
> > 0x424e6c 0x426ddb 0x426e17 0x4200a8 0x42174e 0x4019fc 0x40246d 0x4013db 0x74198482 0x778f3ab6 0x778f3a86
> > 
> > and investigating the libraries, Channel : SOCKETS.Stream_Access will really receive any data size.
> > 
> > So... how can avoid the exception?
> > 
> > Thanks!
> 
> How big of a message are you trying to send?  Datagram sockets have
> an upper limit on the payload size (65507 bytes for ipv4) regardless
> of the language.

Just the string "test". I also confirmed that the MQL container (uchar data[]) has really the size 5.

I also forgot to mention that the communication from both client-server applications show no problem at all, I mean, MQL to MQL and Ada to Ada have no errors. The problem comes only when trying MQL to Ada.

Re: Ada x <whatever> Datagram Sockets

[Egil H H]

> > On Wednesday, February 6, 2019 at 6:10:37 PM UTC-5, Rego, P. wrote:
> 
> Just the string "test". I also confirmed that the MQL container (uchar data[]) has really the size 5.
> 

Right, Ada strings are not zero-terminated, and the size of the string needs 
to come from somewhere, which is why String'Input will interpret the first x bits (probably 32, but may depend on your platform) as the length. So in this
case, "test" becomes a big number and 'Input will most likely try to allocate
just under 2GB  on your stack (and probably at least twice, in order to copy
the result into your string).

Don't use Ada stream attributes to communicate with other languages.

And, as some will tell you, don't use them at all, especially for arrays, as
each array element may end up being transferred as a single data packet, destroying performance.


-- 
~egilhh

Re: Ada x <whatever> Datagram Sockets

[Rego, P.]

On Thursday, February 7, 2019 at 4:00:11 AM UTC-2, Egil H H wrote:
> > > On Wednesday, February 6, 2019 at 6:10:37 PM UTC-5, Rego, P. wrote:
> > 
> > Just the string "test". I also confirmed that the MQL container (uchar data[]) has really the size 5.
> > 
> 
> Right, Ada strings are not zero-terminated, and the size of the string needs 
> to come from somewhere, which is why String'Input will interpret the first x bits (probably 32, but may depend on your platform) as the length. So in this
> case, "test" becomes a big number and 'Input will most likely try to allocate
> just under 2GB  on your stack (and probably at least twice, in order to copy
> the result into your string).
A C++ uchar in Windows machine is usually 1 byte, so assuming that both use the same signature, both messages (from C++ and from Ada) should be of same size, right? 

(I checked both send commands used by Ada sockets and MQL sockets, and they both are using standard win32 signatures, I mean for C++ case

int sendto(
  SOCKET         s,
  const char     *buf,
  int            len,
  int            flags,
  const sockaddr *to,
  int            tolen
);

and in Ada case a imported function
   function C_Sendto
     (S     : C.int;
      Msg   : System.Address;
      Len   : C.int;
      Flags : C.int;
      To    : System.Address;
      Tolen : C.int) return C.int;
)

> 
> Don't use Ada stream attributes to communicate with other languages.
> 
> And, as some will tell you, don't use them at all, especially for arrays, as
> each array element may end up being transferred as a single data packet, destroying performance.

This is quite controversial, in this case. I'd usually agree with you, however the use of Ada stream is not by choice. The whole GNAT Ada package uses Ada streams, and actually the above code was entirely extracted from the comments from g-sockets.ads with minimum modification. I don't think it would be wise to re-implement Ada Sockets pkg to be more efficient.

Re: Ada x <whatever> Datagram Sockets

[Egil H H]

On Thursday, February 7, 2019 at 7:41:28 AM UTC+1, Rego, P. wrote:
> On Thursday, February 7, 2019 at 4:00:11 AM UTC-2, Egil H H wrote:
> > > > On Wednesday, February 6, 2019 at 6:10:37 PM UTC-5, Rego, P. wrote:
> > > 
> > > Just the string "test". I also confirmed that the MQL container (uchar data[]) has really the size 5.
> > > 
> > 
> > Right, Ada strings are not zero-terminated, and the size of the string needs 
> > to come from somewhere, which is why String'Input will interpret the first x bits (probably 32, but may depend on your platform) as the length. So in this
> > case, "test" becomes a big number and 'Input will most likely try to allocate
> > just under 2GB  on your stack (and probably at least twice, in order to copy
> > the result into your string).
> A C++ uchar in Windows machine is usually 1 byte, so assuming that both use the same signature, both messages (from C++ and from Ada) should be of same size, right? 
> 

The *message* may be of the same size, but that doesn't matter, as long as
it's interpreted differently in C++ vs Ada.
C++ apparently puts an extra character in your message (you said "test"
was length 5...)
Ada interprets your string (presumably the first 4 bytes, which is usually
the size of Positive/Integer) as a very big number.

> > 
> > Don't use Ada stream attributes to communicate with other languages.
> > 
> > And, as some will tell you, don't use them at all, especially for arrays, as
> > each array element may end up being transferred as a single data packet, destroying performance.
> 
> This is quite controversial, in this case. I'd usually agree with you, however the use of Ada stream is not by choice. The whole GNAT Ada package uses Ada streams, and actually the above code was entirely extracted from the comments from g-sockets.ads with minimum modification. I don't think it would be wise to re-implement Ada Sockets pkg to be more efficient.


How is it controversial? The example works fine when the server and client
are both written in Ada, as they would then agree on the protocol.
But when dealing with cross-language communication, you're better off using
Send_Socket/Receive_Socket directly. No need to re-implement

-- 
~egilhh

Re: Ada x <whatever> Datagram Sockets

[Jere]

On Thursday, February 7, 2019 at 2:23:32 AM UTC-5, Egil H H wrote:
> C++ apparently puts an extra character in your message (you said "test"
> was length 5...)

It terminates all character strings with '\0' which is just the
nul character (0x00 in hex).

Re: Ada x <whatever> Datagram Sockets

[Rego, P.]

> How is it controversial? The example works fine when the server and client
> are both written in Ada, as they would then agree on the protocol.
> But when dealing with cross-language communication, you're better off using
> Send_Socket/Receive_Socket directly. No need to re-implement

I am trying this approach, but it seems that both g-socket.ads Send_Socket signatures also use Ada Streams

   procedure Receive_Socket
     (Socket : Socket_Type;
      Item   : out Ada.Streams.Stream_Element_Array;
      Last   : out Ada.Streams.Stream_Element_Offset;
      Flags  : Request_Flag_Type := No_Request_Flag);

   procedure Receive_Socket
     (Socket : Socket_Type;
      Item   : out Ada.Streams.Stream_Element_Array;
      Last   : out Ada.Streams.Stream_Element_Offset;
      From   : out Sock_Addr_Type;
      Flags  : Request_Flag_Type := No_Request_Flag);

Re: Ada x <whatever> Datagram Sockets

[Dmitry A. Kazakov]

On 2019-02-08 20:41, Rego, P. wrote:
>> How is it controversial? The example works fine when the server and client
>> are both written in Ada, as they would then agree on the protocol.
>> But when dealing with cross-language communication, you're better off using
>> Send_Socket/Receive_Socket directly. No need to re-implement
> 
> I am trying this approach, but it seems that both g-socket.ads Send_Socket signatures also use Ada Streams
> 
>     procedure Receive_Socket
>       (Socket : Socket_Type;
>        Item   : out Ada.Streams.Stream_Element_Array;
>        Last   : out Ada.Streams.Stream_Element_Offset;
>        Flags  : Request_Flag_Type := No_Request_Flag);
> 
>     procedure Receive_Socket
>       (Socket : Socket_Type;
>        Item   : out Ada.Streams.Stream_Element_Array;
>        Last   : out Ada.Streams.Stream_Element_Offset;
>        From   : out Sock_Addr_Type;
>        Flags  : Request_Flag_Type := No_Request_Flag);

No. Stream_Element_Array is a plain array. Receive_Socket fill it and 
return the index of last updated element.

    declare
       Buffer : Stream_Element_Array (1..Max_Packet_Length);
       Last   : Stream_Element_Offset;
    begin
       Receive_Socket (Socket, Buffer, Last);
       -- Buffer (1..Last) is the receipt

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Ada x <whatever> Datagram Sockets

[Rego, P.]

> No. Stream_Element_Array is a plain array. Receive_Socket fill it and 
> return the index of last updated element.
> 
>     declare
>        Buffer : Stream_Element_Array (1..Max_Packet_Length);
>        Last   : Stream_Element_Offset;
>     begin
>        Receive_Socket (Socket, Buffer, Last);
>        -- Buffer (1..Last) is the receipt

Interesting... I get it now.

Re: Ada x <whatever> Datagram Sockets

[Dmitry A. Kazakov]

On 2019-02-07 07:41, Rego, P. wrote:

> This is quite controversial, in this case. I'd usually agree with you, however the use of Ada stream is not by choice. The whole GNAT Ada package uses Ada streams, and actually the above code was entirely extracted from the comments from g-sockets.ads with minimum modification. I don't think it would be wise to re-implement Ada Sockets pkg to be more efficient.

Well, using streams directly with UDP is calling for trouble (and highly 
non-portable too). You should probably never use streams with TCP 
either. Certainly not if socket options like NO_DELAY are planned. 
Streams will never ever work with non-blocking sockets, obviously.

(There are other concerns with stream attributes beyond performance that 
however do not apply to strings)

Anyway, the way to do it is to have an outgoing packet buffer of 
Stream_Element_Array. You put data there and then send all buffer to the 
socket (write to the file etc), in one piece.

If you want to use streams. Put a memory-mapped stream on top of the 
Stream_Element_Array. Write packet whole. Flush the buffer contents to 
the socket/file.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Ada x <whatever> Datagram Sockets

[Simon Wright]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> Well, using streams directly with UDP is calling for trouble (and
> highly non-portable too). You should probably never use streams with
> TCP either. Certainly not if socket options like NO_DELAY are
> planned.

The reason for this is to be found in ARM 13.13.2(9)[1] for 'Write:

   For composite types, the Write or Read attribute for each component
   is called in canonical order

which will result in multiple calls to the underlying OS services; bad
for TCP_NODELAY, very very bad for UDP.

See GCC PR9535[2]. Note, the change applied really doesn't address the
problem at all, in spite of being essentially my patch! I lost interest,
having already decided to do as Dmitry says ...

> Anyway, the way to do it is to have an outgoing packet buffer of
> Stream_Element_Array. You put data there and then send all buffer to
> the socket (write to the file etc), in one piece.
>
> If you want to use streams. Put a memory-mapped stream on top of the
> Stream_Element_Array. Write packet whole. Flush the buffer contents to
> the socket/file.

... see, in particular, comments 2, 6 in [2].

[1] http://www.ada-auth.org/standards/rm12_w_tc1/html/RM-13-13-2.html#p9
[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=9535

Re: Ada x <whatever> Datagram Sockets

[Randy Brukardt]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> wrote in message 
news:q3gq7i$o27$1@gioia.aioe.org...
...
> Anyway, the way to do it is to have an outgoing packet buffer of 
> Stream_Element_Array. You put data there and then send all buffer to the 
> socket (write to the file etc), in one piece.

Note that Ada 2020 will have a package FIFO_Streams for this purpose (that 
is, marshalling/unmarshalling streams). See AI12-0293-1 or 13.13.1 in the 
draft 16 or later RM: 
http://www.ada-auth.org/standards/2xrm/html/RM-13-13-1.html (toward the 
bottom).

We had to write such a package for Claw, in order to allow using stream 
attributes to put/get items from Windows data structures like the registry 
or clipboard. It's something that you can hardly live without, which is why 
it will be standardized.

                  Randy.

Re: Ada x <whatever> Datagram Sockets

[Simon Wright]

"Randy Brukardt" <randy@rrsoftware.com> writes:

> Note that Ada 2020 will have a package FIFO_Streams

Names are in short supply - but - is there an implication there that
streams in general need not be FIFO? (I can just about see that you
might have some form of FIFO-within-priority ordering, though that does
seem to be a bit of a perversion of the underlying idea)

Re: Ada x <whatever> Datagram Sockets

[Dmitry A. Kazakov]

On 2019-02-08 09:25, Simon Wright wrote:
> "Randy Brukardt" <randy@rrsoftware.com> writes:
> 
>> Note that Ada 2020 will have a package FIFO_Streams
> 
> Names are in short supply - but - is there an implication there that
> streams in general need not be FIFO? (I can just about see that you
> might have some form of FIFO-within-priority ordering, though that does
> seem to be a bit of a perversion of the underlying idea)

There might be more than one type of stream actually.

The FIFO_Streams, I guess, serve the purpose of providing an inter-task 
pipe line. Which BTW should also provide blocking on being empty/full 
rather than raising End_Error.

Other memory-mapped streams are streams on top of a contiguous regular 
array. You write into it via stream interface and you get a slice of the 
written data. And conversely you store a slice, and then you read it out 
via the stream. This is a bit different from FIFO / ring buffer stream.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Ada x <whatever> Datagram Sockets

[Randy Brukardt]

"Simon Wright" <simon@pushface.org> wrote in message 
news:lyef8iy9dj.fsf@pushface.org...
> "Randy Brukardt" <randy@rrsoftware.com> writes:
>
>> Note that Ada 2020 will have a package FIFO_Streams
>
> Names are in short supply - but - is there an implication there that
> streams in general need not be FIFO? (I can just about see that you
> might have some form of FIFO-within-priority ordering, though that does
> seem to be a bit of a perversion of the underlying idea)

We tried a lot of names and FIFO was the winner. I'm not sure it is very 
descriptive of what the package actually does.

                 Randy.

Re: Ada x <whatever> Datagram Sockets

[Simon Wright]

"Randy Brukardt" <randy@rrsoftware.com> writes:

> "Simon Wright" <simon@pushface.org> wrote in message 
> news:lyef8iy9dj.fsf@pushface.org...
>> "Randy Brukardt" <randy@rrsoftware.com> writes:
>>
>>> Note that Ada 2020 will have a package FIFO_Streams
>>
>> Names are in short supply - but - is there an implication there that
>> streams in general need not be FIFO? (I can just about see that you
>> might have some form of FIFO-within-priority ordering, though that
>> does seem to be a bit of a perversion of the underlying idea)
>
> We tried a lot of names and FIFO was the winner. I'm not sure it is
> very descriptive of what the package actually does.

I called it Memory_Streams, e.g.
https://github.com/simonjwright/coldframe/blob/master/lib/coldframe-memory_streams.ads

The main use was to allow messages for UDP streams to be prepared
simply, as discussed above, so the typical usage was

   declare
      a Memory_Stream instance of the appropriate size
   begin
      fill the Memory_Stream instance
      write its contents to the UDP stream[1]
   end

A further benefit of using streams was that we were communicating
between machines of different endianness, and GNAT contains an optional
stream implementation that will transparently convert to/from network
byte order.

[1] not sure that FIFO_Streams offers this without copying the contents
out?

Re: Ada x <whatever> Datagram Sockets

[Dmitry A. Kazakov]

On 2019-02-10 18:54, Simon Wright wrote:

> A further benefit of using streams was that we were communicating
> between machines of different endianness, and GNAT contains an optional
> stream implementation that will transparently convert to/from network
> byte order.

That does not make much sense. Most protocols of the layers above simply 
ignore that and use the order (encoding, actually) they prefer for 
whatever reason. Furthermore encoding integers is the least problem.

Anyway, there are two major and quite different cases: piped I/O vs. 
encoding/decoding. You refer to the latter and FIFO is likely the former.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Ada x <whatever> Datagram Sockets

[Simon Wright]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> On 2019-02-10 18:54, Simon Wright wrote:
>
>> A further benefit of using streams was that we were communicating
>> between machines of different endianness, and GNAT contains an
>> optional stream implementation that will transparently convert
>> to/from network byte order.
>
> That does not make much sense. Most protocols of the layers above
> simply ignore that and use the order (encoding, actually) they prefer
> for whatever reason. Furthermore encoding integers is the least
> problem.

Machine A (a PowerPC) will automatically send the contents of a record
in network byte order. Machine B (x86_84) will handle the protocol
without problem but requires the content of the the record to be
converted to host byte order. Best, especially if the record content is
subject to change, if that conversion can be automatic.

A consideration on that project was that the target hardware was all
PowerPC, and conversion load was to be avoided if possible, while the
x86_64 part was all about logging and analysis, and had cycles to spare.

> Anyway, there are two major and quite different cases: piped I/O
> vs. encoding/decoding. You refer to the latter and FIFO is likely the
> former.

If you're communicating within one program, use an indefinite queue. If
between two programs, I don't see there's a lot of difference really
(OK, if the programs are both on x86_64, doesn't make much sense to
convert the contents to network byte order! which is a disadvantage of
using a modified runtime, of course).

Re: Ada x <whatever> Datagram Sockets

[Dmitry A. Kazakov]

On 2019-02-11 14:35, Simon Wright wrote:
> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
> 
>> On 2019-02-10 18:54, Simon Wright wrote:
>>
>>> A further benefit of using streams was that we were communicating
>>> between machines of different endianness, and GNAT contains an
>>> optional stream implementation that will transparently convert
>>> to/from network byte order.
>>
>> That does not make much sense. Most protocols of the layers above
>> simply ignore that and use the order (encoding, actually) they prefer
>> for whatever reason. Furthermore encoding integers is the least
>> problem.
> 
> Machine A (a PowerPC) will automatically send the contents of a record
> in network byte order.

It does not make sense in most cases. The Machine A sends a packet 
encoded as defined by the protocol. Usually there is no any Ada record 
type corresponding to the packet, because the semantics of packet 
sending is not sending a record, but doing some protocol-defined stuff, 
e.g. subscribe to the MQTT topic XXX using the policy YYY. It is safer 
and simpler to form the packet in a buffer straight away without any 
intermediate record types, especially because in most cases you would 
not be able declare such a record type at all.

> A consideration on that project was that the target hardware was all
> PowerPC, and conversion load was to be avoided if possible, while the
> x86_64 part was all about logging and analysis, and had cycles to spare.

There is no conversion at all. You simply encode a packet and decode it. 
That is it. Ada's machine representation of the types in the application 
plays no role.

>> Anyway, there are two major and quite different cases: piped I/O
>> vs. encoding/decoding. You refer to the latter and FIFO is likely the
>> former.
> 
> If you're communicating within one program, use an indefinite queue. If
> between two programs, I don't see there's a lot of difference really
> (OK, if the programs are both on x86_64, doesn't make much sense to
> convert the contents to network byte order! which is a disadvantage of
> using a modified runtime, of course).

Stream is not queue unless you mean a queue of stream elements. I 
suppose that using a queue of stream elements will inflict too much 
overhead if you put a stream interface on top of it to be usable.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Ada x <whatever> Datagram Sockets

[Simon Wright]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> On 2019-02-11 14:35, Simon Wright wrote:
>> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>>
>>> On 2019-02-10 18:54, Simon Wright wrote:

>> A consideration on that project was that the target hardware was all
>> PowerPC, and conversion load was to be avoided if possible, while the
>> x86_64 part was all about logging and analysis, and had cycles to
>> spare.
>
> There is no conversion at all. You simply encode a packet and decode
> it. That is it. Ada's machine representation of the types in the
> application plays no role.

You say encode/decode, in the case I'm talking about it seemed to me
that conversion was an equally valid way of stating it. And, just in
case it's not clear, the Ada types in the application were of primary
importance.

>>> Anyway, there are two major and quite different cases: piped I/O
>>> vs. encoding/decoding. You refer to the latter and FIFO is likely the
>>> former.
>>
>> If you're communicating within one program, use an indefinite
>> queue. If between two programs, I don't see there's a lot of
>> difference really (OK, if the programs are both on x86_64, doesn't
>> make much sense to convert the contents to network byte order! which
>> is a disadvantage of using a modified runtime, of course).
>
> Stream is not queue unless you mean a queue of stream elements. I
> suppose that using a queue of stream elements will inflict too much
> overhead if you put a stream interface on top of it to be usable.

Within one program, why use streams at all? Just have an indefinite
queue of objects.

Re: Ada x <whatever> Datagram Sockets

[Dmitry A. Kazakov]

On 2019-02-11 16:19, Simon Wright wrote:
> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
> 
>> On 2019-02-11 14:35, Simon Wright wrote:
>>> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>>>
>>>> On 2019-02-10 18:54, Simon Wright wrote:
> 
>>> A consideration on that project was that the target hardware was all
>>> PowerPC, and conversion load was to be avoided if possible, while the
>>> x86_64 part was all about logging and analysis, and had cycles to
>>> spare.
>>
>> There is no conversion at all. You simply encode a packet and decode
>> it. That is it. Ada's machine representation of the types in the
>> application plays no role.
> 
> You say encode/decode, in the case I'm talking about it seemed to me
> that conversion was an equally valid way of stating it. And, just in
> case it's not clear, the Ada types in the application were of primary
> importance.

I only say that in practice it is a very rare case. Even if your 
objective is to simply communicate between two piers, you would end up 
rather with some standardised protocol instead of sending custom records 
around. The problem with the latter is maintaining states on the both 
sides and error recovery. Beyond some very simple cases records would 
not scale/work.

>>>> Anyway, there are two major and quite different cases: piped I/O
>>>> vs. encoding/decoding. You refer to the latter and FIFO is likely the
>>>> former.
>>>
>>> If you're communicating within one program, use an indefinite
>>> queue. If between two programs, I don't see there's a lot of
>>> difference really (OK, if the programs are both on x86_64, doesn't
>>> make much sense to convert the contents to network byte order! which
>>> is a disadvantage of using a modified runtime, of course).
>>
>> Stream is not queue unless you mean a queue of stream elements. I
>> suppose that using a queue of stream elements will inflict too much
>> overhead if you put a stream interface on top of it to be usable.
> 
> Within one program, why use streams at all? Just have an indefinite
> queue of objects.

The use case is when a component is designed to work with an external 
file. Quite frequently one wishes to replace file I/O with a general 
provider/consumer interface. Ada stream is a convenient common denominator.

Queue of [indefinite] typed objects is a higher-level and better 
abstraction. It would work, and would be safer too, if we had an 
object-based OS and persistency layer instead of files, all written in 
Ada. Unfortunately we do not.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Ada x <whatever> Datagram Sockets

[Randy Brukardt]

"Simon Wright" <simon@pushface.org> wrote in message 
news:ly5ztry1f8.fsf@pushface.org...
> "Randy Brukardt" <randy@rrsoftware.com> writes:
...
> [1] not sure that FIFO_Streams offers this without copying the contents
> out?

It doesn't, but I'm not sure how it could. It's not possible for a single 
data type to be two different streams at the same time. Moreover, at least 
some of the time, the result isn't going to a stream at all. And I haven't 
personally seen a case where the extra cost of copying the data is 
significant (the target stream/API being much slower). I could see where it 
might matter, particularly in a transfer to some other kind of memory.

                    Randy.

Re: Ada x <whatever> Datagram Sockets

[Simon Wright]

"Randy Brukardt" <randy@rrsoftware.com> writes:

> "Simon Wright" <simon@pushface.org> wrote in message 
> news:ly5ztry1f8.fsf@pushface.org...
>> "Randy Brukardt" <randy@rrsoftware.com> writes:
> ...
>> [1] not sure that FIFO_Streams offers this without copying the contents
>> out?
>
> It doesn't, but I'm not sure how it could. It's not possible for a single 
> data type to be two different streams at the same time. Moreover, at least 
> some of the time, the result isn't going to a stream at all. And I haven't 
> personally seen a case where the extra cost of copying the data is 
> significant (the target stream/API being much slower). I could see where it 
> might matter, particularly in a transfer to some other kind of memory.

You may well be right about the extra cost. As I saw it, I needed to
eliminate the need for the users to declare a buffer, read into it, and
send it to the socket (represented in GNAT.Sockets as a stream), when
there was already a perfectly good buffer in the memory stream. There
were quite a few users.

I had fun getting FIFO_Streams to build in Ada 2012. Obviously had to
remove the 2020 aspects, and recast the (declare ...). Only the bounded
version, because the unbounded version would be much more
work. Unfortunately this means GNATprove couldn't be used, because
SPARK2014 doesn't allow discriminants on derived types (SPARK RM
3.7(2)). All the same, the assertions were a huge help.

Test code:

   with Ada.Streams.FIFO_Streams.Bounded;
   with Ada.Text_IO; use Ada.Text_IO;
   procedure Test is
      S : aliased Ada.Streams.FIFO_Streams.Bounded.Stream_Type (31);
   begin
      for J in 1 .. 9 loop
         String'Output (S'Access, "hello!" & J'Image);
         declare
            Buff : Ada.Streams.Stream_Element_Array (1 .. 32);
            Last : Ada.Streams.Stream_Element_Offset;
         begin
            S.Read (Item   => Buff,
                    Last   => Last);
            for K in 1 .. Last loop
               Put (Buff (K)'Image);
            end loop;
            New_Line;
         end;
      end loop;
   end Test;

Output:

   $ ./test
    1 0 0 0 8 0 0 0 104 101 108 108 111 33 32 49
    1 0 0 0 8 0 0 0 104 101 108 108 111 33 32 50
    1 0 0 0 8 0 0 0 104 101 108 108 111 33 32 51
    1 0 0 0 8 0 0 0 104 101 108 108 111 33 32 52
    1 0 0 0 8 0 0 0 104 101 108 108 111 33 32 53
    1 0 0 0 8 0 0 0 104 101 108 108 111 33 32 54
    1 0 0 0 8 0 0 0 104 101 108 108 111 33 32 55
    1 0 0 0 8 0 0 0 104 101 108 108 111 33 32 56
    1 0 0 0 8 0 0 0 104 101 108 108 111 33 32 57

Re: Ada x <whatever> Datagram Sockets

[Rego, P.]

> If you want to use streams. Put a memory-mapped stream on top of the 
> Stream_Element_Array. Write packet whole. Flush the buffer contents to 
> the socket/file.

(I don't want to use streams) But in this case I am receiving a data with at principle unknown size, and I don't have control on the write side.

Re: Ada x <whatever> Datagram Sockets

[Jere]

On Thursday, February 7, 2019 at 1:41:28 AM UTC-5, Rego, P. wrote:
> On Thursday, February 7, 2019 at 4:00:11 AM UTC-2, Egil H H wrote:
> > > > On Wednesday, February 6, 2019 at 6:10:37 PM UTC-5, Rego, P. wrote:
> > > 
> > > Just the string "test". I also confirmed that the MQL container (uchar data[]) has really the size 5.
> > > 
> > 
> > Right, Ada strings are not zero-terminated, and the size of the string needs 
> > to come from somewhere, which is why String'Input will interpret the first x bits (probably 32, but may depend on your platform) as the length. So in this
> > case, "test" becomes a big number and 'Input will most likely try to allocate
> > just under 2GB  on your stack (and probably at least twice, in order to copy
> > the result into your string).
> A C++ uchar in Windows machine is usually 1 byte, so assuming that both use the same signature, both messages (from C++ and from Ada) should be of same size, right? 
> 
The problem is you are telling it to send it one way and receive it
in another way.  In C++ you tell the code to send:
't' 'e' 's' 't' '\0'

In Ada you are telling it to receive:

<length byte 1..4> <data byte 1..length>

if it reads your c++ packet in that format, assuming a little endian 
system:

<0x74736574> <\0 + random junk>

That translates to specifying a length of 1953719668 bytes, which is 
above the 65507 limit.

For giggles try modifying your receive code a bit:

      declare
         subtype Test_String is String(1..5);
         Message : Test_String := Test_String'Input (Channel);
      begin
         Address := SOCKETS.Get_Address (Channel);
         Text_IO.Put_Line (Message & " from " & SOCKETS.Image (Address));
         Test_String'Output (Channel, Message);
      end; 

I don't recall if C actually sends the '\0' at the end or not
for sockets, so if it doesn't, then change the Test_String declaration
to

      subtype Test_String is String(1..4);

However, I think it does send the '\0', so you will also have to
handle that in your Ada code (since strings don't terminate with a
'\0' in Ada).

Changing String to a constrained subtype might change the pattern
to be closer to the C++ pattern sent.  This doesn't solve your problem
because you never know how long the data is for real data, but would
at least (hopefully) get you talking.

> 
> > 
> > Don't use Ada stream attributes to communicate with other languages.
> > 
> > And, as some will tell you, don't use them at all, especially for arrays, as
> > each array element may end up being transferred as a single data packet, destroying performance.
> 
> This is quite controversial, in this case. I'd usually agree with you, however the use of Ada stream is not by choice. The whole GNAT Ada package uses Ada streams, and actually the above code was entirely extracted from the comments from g-sockets.ads with minimum modification. I don't think it would be wise to re-implement Ada Sockets pkg to be more efficient.

I've never used streams with the GNAT socket package for production code,
so I cannot comment on them, but the package definitely offers other
options besides streams.  I generally have a task that periodically
reads up to a specific number of bytes into a buffer and then my other
tasks can handle the data as they see fit.  You can even have the OS
tell you how many bytes are available to read before you do.

Re: Ada x <whatever> Datagram Sockets

[Jeffrey R. Carter]

On 2/7/19 12:47 PM, Jere wrote:
> 
> In Ada you are telling it to receive:
> 
> <length byte 1..4> <data byte 1..length>

Wrong. According to ARM 13.13.2(26/3) says, "If T is an array type, S'Output 
first writes the bounds, and S'Input first reads the bounds." Ada arrays do not 
necessarily have a lower bound of 1.

So String'Input will first try to read 2 integers, the lower and upper bounds of 
the array. Presumably it runs out of data while reading the 2nd, leading to the 
exception.

What the OP needs to do is

1. Get Length, the length of the data.
2. Create C : Interfaces.C.char_array (1 .. Length)
3. Transfer the data into C
4. Use Interfaces.C.To_Ada (C) to transform C into an Ada String

-- 
Jeff Carter
"Now go away or I shall taunt you a second time."
Monty Python & the Holy Grail
07

Re: Ada x <whatever> Datagram Sockets

[Rego, P.]

> What the OP needs to do is
> 
> 1. Get Length, the length of the data.
> 2. Create C : Interfaces.C.char_array (1 .. Length)
> 3. Transfer the data into C
> 4. Use Interfaces.C.To_Ada (C) to transform C into an Ada String

Simple enough, would you know how transfer the data into this C? I am trying in this path

   loop
      --  Receive and print message from client Ping
      Channel := SOCKETS.Stream (Socket, Address);

      Text_IO.Put_Line (Integer'Image (Channel'Size));
      
      declare
         Channel_Size : Integer := Integer (Channel'Size);
         type Buffer_Type is new Interfaces.C.char_array (1 .. Interfaces.C.size_t (Channel_Size));
         type Stream_Buffer_Type is new String (1 .. Integer (Channel_Size)); 

         function Copy_Arr is new Ada.Unchecked_Conversion (Buffer_Type, Stream_Buffer_Type); 
         
         --Buffer : Buffer_Type;
         Stream_Buffer : Stream_Buffer_Type;
         
      begin
         --Buffer := String'Input (Channel);
         
         --!!!! Stream_Buffer := Stream_Buffer_Type (String'Input (Channel));
      end;
end loop;

Re: Ada x <whatever> Datagram Sockets

[Jeffrey R. Carter]

On 2/8/19 9:35 PM, Rego, P. wrote:
>> What the OP needs to do is
>>
>> 1. Get Length, the length of the data.
>> 2. Create C : Interfaces.C.char_array (1 .. Length)
>> 3. Transfer the data into C
>> 4. Use Interfaces.C.To_Ada (C) to transform C into an Ada String

Now that I look at it in more detail, it's probably better to do something like

function To_String (Buffer : Ada.Streams.Stream_Element_Array) return String is
    Result : String (1 .. Buffer'Length);
    Last   : Natural := 0;
begin -- To_String
    Convert : for I in Buffer'range loop
       exit Convert when Buffer (I) = 0; -- If NUL terminated

       Last := Last + 1;
       Result (Last) := Character'Val (Buffer (I) );
    end loop Convert;

    return Result (1 .. Last);
end To_String;

Buffer : Ada.Streams.Stream_Element_Array (1 .. Big_Enough);
Last   : Ada.Streams.Stream_Element_Offset;
...
Receive_Socket (Socket => Socket, Item => Buffer, Last => Last);

Do_Something (To => To_String (Buffer (1 .. Last) ) );

-- 
Jeff Carter
"I've seen projects fail miserably for blindly
applying the Agile catechism: we're Agile, we
don't need to stop and think, we just go ahead
and code!"
Bertrand Meyer
150

Re: Ada x <whatever> Datagram Sockets

[Rego, P.]

On Friday, February 8, 2019 at 7:26:20 PM UTC-2, Jeffrey R. Carter wrote:
> Now that I look at it in more detail, it's probably better to do something like
> 
> function To_String (Buffer : Ada.Streams.Stream_Element_Array) return String is
>     Result : String (1 .. Buffer'Length);
>     Last   : Natural := 0;
> begin -- To_String
>     Convert : for I in Buffer'range loop
>        exit Convert when Buffer (I) = 0; -- If NUL terminated
> 
>        Last := Last + 1;
>        Result (Last) := Character'Val (Buffer (I) );
>     end loop Convert;
> 
>     return Result (1 .. Last);
> end To_String;
> 
> Buffer : Ada.Streams.Stream_Element_Array (1 .. Big_Enough);
> Last   : Ada.Streams.Stream_Element_Offset;
> ...
> Receive_Socket (Socket => Socket, Item => Buffer, Last => Last);
> 
> Do_Something (To => To_String (Buffer (1 .. Last) ) );

It worked perfectly. Thank you Jeff and everybody.
Now I am going to make a proper design of the system :-)

Re: Ada x <whatever> Datagram Sockets

[Dmitry A. Kazakov]

On 2019-02-08 21:35, Rego, P. wrote:
>> What the OP needs to do is
>>
>> 1. Get Length, the length of the data.
>> 2. Create C : Interfaces.C.char_array (1 .. Length)
>> 3. Transfer the data into C
>> 4. Use Interfaces.C.To_Ada (C) to transform C into an Ada String
> 
> Simple enough, would you know how transfer the data into this C? I am trying in this path
> 
>     loop
>        --  Receive and print message from client Ping
>        Channel := SOCKETS.Stream (Socket, Address);
> 
>        Text_IO.Put_Line (Integer'Image (Channel'Size));
>        
>        declare
>           Channel_Size : Integer := Integer (Channel'Size);
>           type Buffer_Type is new Interfaces.C.char_array (1 .. Interfaces.C.size_t (Channel_Size));
>           type Stream_Buffer_Type is new String (1 .. Integer (Channel_Size));
> 
>           function Copy_Arr is new Ada.Unchecked_Conversion (Buffer_Type, Stream_Buffer_Type);
>           
>           --Buffer : Buffer_Type;
>           Stream_Buffer : Stream_Buffer_Type;
>           
>        begin
>           --Buffer := String'Input (Channel);
>           
>           --!!!! Stream_Buffer := Stream_Buffer_Type (String'Input (Channel));
>        end;
> end loop;

    declare
       Packet : Stream_Element_Array (1..Max_Size);
       Last   : Stream_Element_Offset;
       From   : Sock_Addr_Type;
    loop
       Receive_Socket (Socket, Packet, Last, From); -- UDP from anyone
       declare
          Text : String (1..Natural (Last));
       begin
          for Index in 1..Last loop
             Text (Integer (Index)) := Character'Val (Packet (Index));
          end loop;
          Put_Line (Image (From) & ">|" & Text & "|");
       end;
    end loop;

For UDP there is no need to have packet length because packet=frame.

For TCP, you read the header first which usually determine the length. 
Then you read the length stream elements. After that you start to decode.

P.S. It is good practice to keep framing (packet I/O) separate from 
payload encoding/decoding.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Ada x <whatever> Datagram Sockets

[Rego, P.]

> For giggles try modifying your receive code a bit:
> 
>       declare
>          subtype Test_String is String(1..5);
>          Message : Test_String := Test_String'Input (Channel);
>       begin
>          Address := SOCKETS.Get_Address (Channel);
>          Text_IO.Put_Line (Message & " from " & SOCKETS.Image (Address));
>          Test_String'Output (Channel, Message);
>       end; 

The exception continues to come, but now the catcher comes from 
>          Message : Test_String := Test_String'Input (Channel);
line.

> I don't recall if C actually sends the '\0' at the end or not
> for sockets, so if it doesn't, then change the Test_String declaration
> to
> 
>       subtype Test_String is String(1..4);
> 
> However, I think it does send the '\0', so you will also have to
> handle that in your Ada code (since strings don't terminate with a
> '\0' in Ada).
It does not seem to be an '\0' problem, since the reception container is not limited (so, I don't mind receiving a 5 lenght message instead of 4), and the message (should be) is too small. The real messages will not be so long as well, in the real case it should have the same issue.

> I've never used streams with the GNAT socket package for production code,
> so I cannot comment on them, but the package definitely offers other
> options besides streams.  

Well... specifically GNAT.Sockets, from g-socket.ads I couldn't find any Receive_Sockets -like signature using any data container different than streams. 


> I generally have a task that periodically
> reads up to a specific number of bytes into a buffer and then my other
> tasks can handle the data as they see fit.  

This will be the next step :-) 

Re: Ada x <whatever> Datagram Sockets

[Simon Wright]

"Rego, P." <pvrego@gmail.com> writes:

> Just the string "test". I also confirmed that the MQL container (uchar
> data[]) has really the size 5.
>
> I also forgot to mention that the communication from both
> client-server applications show no problem at all, I mean, MQL to MQL
> and Ada to Ada have no errors. The problem comes only when trying MQL
> to Ada.

Since you (sadly) need to change on the Ada side, the best start might
be to use Wireshark, or read the MQL message into a Stream_Element_Array
buffer, to see what actually comes over the wire, and then work from
that.

Re: Ada x <whatever> Datagram Sockets

[Rego, P.]

> Since you (sadly) need to change on the Ada side, the best start might
> be to use Wireshark, or read the MQL message into a Stream_Element_Array
> buffer, to see what actually comes over the wire, and then work from
> that.

I will try wireshark, just installing now.