Using representation clauses in networking software

Using representation clauses in networking software

[Florian Weimer]

Have there been any efforts to fix representation clauses so that they
are suitable for expressing elements commonly found in networking
protocols?

My understanding is that representation clauses are totally unsuitable
for this purpose because they do not allow the programmer to express
endianness.  I dimly recall a technical report which argued that
expressing endianness portably was impossible.  However, Erlang's bit
syntax seems to be a counterexample, so I wonder if that issue has
been picked up in recent years.

Re: Using representation clauses in networking software

[Yannick Duchêne (Hibou57)]

Hello Florian
Le Sun, 15 Aug 2010 13:33:32 +0200, Florian Weimer <fw@deneb.enyo.de> a  
écrit:
> Have there been any efforts to fix representation clauses so that they
> are suitable for expressing elements commonly found in networking
> protocols?

General case first and yours then

For representation issue, the common scheme is to use conversion when  
receiving something from the outside of the application and use conversion  
when sending something outside. This is done this way to be more efficient  
as this preserve the best representation for all internal uses.

This conversion may be implicitly defined via a type conversion between a  
type and a derived type if one has a representation clause applying. Ex. a  
type T1 is a type with its universal definition, a second type T2 derived  
 from T1 is to be stored in file or retrieved from a device using the size  
of X bits. Both are the same in some sense, except T2 has a representation  
clause. When you do “T1 (Object_Of_Type_T1)” there is an implicit  
conversion. The other way conversion is the same.

Your case now (without representation attribute, as this one does not  
exist so far)

I do not know neither any representation clause for Byte Ordering.  
However, you probably have a set of basic type like 16 bits words, 64 bits  
words or anything else. The best is to have a similar thing as the above,  
except this would be implemented explicitly. You would define a function  
From_Local_To_Network and and From_Network_To_Local (with two different  
types for Local_Type and Network_Type to avoid to erroneously mixing it).

Just to try to meet your request (something else later) : I do not know a  
way to have a method which would automatically know the bit order of its  
platform, except perhaps receiving a standard data, like 16#1234#  
(0x1234), and then check if it looks like 16#3412# or like 16#1234#. Or  
this may be a link to a tiny-tiny library which would only contains a  
simple data word, which could be checked the same way. Ex. a library which  
would contains the same 6#1234# which would be linked as an external C  
stuff and could be tested for the same way.

The other way (which does not meet you requirement) : use the ability of  
Ada to have multiple package body for the same package specification and  
define two package body for the same network<->platform conversion package  
specification. Then, use a simple configuration option to build the  
application using one or the other (you will not face thousand of cases,  
as there are not some much commonly used platform in real life).

You may write it or reuse one (pretty sure something like this already  
exist in whatever license).

Or else you may suggest it for a future Ada revision (via this newsgroup  
or else ada-auth.org ).

Re: Using representation clauses in networking software

[Dmitry A. Kazakov]

On Sun, 15 Aug 2010 15:44:34 +0200, Yannick Duch�ne (Hibou57) wrote:

(I agree with what you wrote. I am programming a lot of communication
stuff, but never used representation clauses to handle endianness.)

> I do not know neither any representation clause for Byte Ordering. 

Byte ordering is what S'Bit_Ordering is, when bytes are addressable.

IMO, Bit_Ordering was an unfortunate choice. The attribute name suggests
ordering of bits in some machine storage unit, which it is not.

> Just to try to meet your request (something else later) : I do not know a  
> way to have a method which would automatically know the bit order of its  
> platform,

Bit order is useless if bits are not directly addressable.

> except perhaps receiving a standard data, like 16#1234#  
> (0x1234), and then check if it looks like 16#3412# or like 16#1234#.

This would determine byte ordering. Bit ordering is when you serialize a
sequence of bits {0, 0, 0, 0, 0, 0, 0, 1} and get 2#1000_0000# (little
endian) 2#0000_0001# (big endian).

Most types of hardware have opaque bytes compatible with the machine, e.g.
UARTs. If the hardware is not, then representation clause would not be my
choice anyway. I would recode bytes or whatever units immediately as
obtained from the device:

   type Strange_Octet is mod 2**8;
   Decode_Strange_Octet : array (Strange_Octet) of Unsigned_8 :=
      (  2#0000_0001# => 2#0000_1000#,
         2#0000_0010# => 2#0000_0100#,
         2#0000_0011# => 2#0000_1100#,
         ...
      );

> Or else you may suggest it for a future Ada revision (via this newsgroup  
> or else ada-auth.org ).

One could introduce enumeration representation clauses for modular types,
e.g.

   type Strange_Octet is mod 2**8;
   for Strange_Octet use (2#0000_0001# => 2#0000_1000#, ...);

Better would be to allow user-defined integer literals, so that the
programmer could define its own type.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Using representation clauses in networking software

[Florian Weimer]

* Dmitry A. Kazakov:

> On Sun, 15 Aug 2010 15:44:34 +0200, Yannick Duch�ne (Hibou57) wrote:
>
> (I agree with what you wrote. I am programming a lot of communication
> stuff, but never used representation clauses to handle endianness.)

This is not surprising because the current representation clauses
cannot handle it.

I wonder if there has been any attempt to enhance representation
clauses so that they are useful in this context.

Re: Using representation clauses in networking software

[Dmitry A. Kazakov]

On Sun, 15 Aug 2010 16:44:33 +0200, Florian Weimer wrote:

> * Dmitry A. Kazakov:
> 
>> On Sun, 15 Aug 2010 15:44:34 +0200, Yannick Duch�ne (Hibou57) wrote:
>>
>> (I agree with what you wrote. I am programming a lot of communication
>> stuff, but never used representation clauses to handle endianness.)
> 
> This is not surprising because the current representation clauses
> cannot handle it.

What for, if there are better ways to handle it? Representation layout
clause was invented for handling something in place. It is no more actual,
because reading out/writing in with an appropriate recoding is cleaner and
possibly cheaper on modern hardware.

> I wonder if there has been any attempt to enhance representation
> clauses so that they are useful in this context.

To handle all possible cases of mixed-endian? To handle non-2's complement
integers? IEEE, IBM, DEC float layouts? Layouts of Character like EBCDIC,
KOI-7/8 etc? It would be a mammoth task to accomplish. What is the gain?

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Using representation clauses in networking software

[Yannick Duchêne (Hibou57)]

Le Sun, 15 Aug 2010 16:32:47 +0200, Dmitry A. Kazakov  
<mailbox@dmitry-kazakov.de> a écrit:
> Byte ordering is what S'Bit_Ordering is, when bytes are addressable.
Do you mean when byte and bit are the same ? I am sure to understand

Or else may be I should go back and read again the 'Bit_Order attribute  
specification.


>> Just to try to meet your request (something else later) : I do not know  
>> a
>> way to have a method which would automatically know the bit order of its
>> platform,
>
> Bit order is useless if bits are not directly addressable.

I wrote bit ordered where I meant byte ordering (clumsy mistake of mine)


-- 
There is even better than a pragma Assert: a SPARK --# check.
--# check C and WhoKnowWhat and YouKnowWho;
--# assert Ada;
--  i.e. forget about previous premises which leads to conclusion
--  and start with new conclusion as premise.

Re: Using representation clauses in networking software

[Yannick Duchêne (Hibou57)]

Le Sun, 15 Aug 2010 16:44:33 +0200, Florian Weimer <fw@deneb.enyo.de> a  
écrit:
> I wonder if there has been any attempt to enhance representation
> clauses so that they are useful in this context.
What do you mean with “enhance” here ? What enhancement over a user  
defined function/procedure would you expect ?


-- 
There is even better than a pragma Assert: a SPARK --# check.
--# check C and WhoKnowWhat and YouKnowWho;
--# assert Ada;
--  i.e. forget about previous premises which leads to conclusion
--  and start with new conclusion as premise.

Re: Using representation clauses in networking software

[Florian Weimer]

* Dmitry A. Kazakov:

> On Sun, 15 Aug 2010 16:44:33 +0200, Florian Weimer wrote:
>
>> * Dmitry A. Kazakov:
>> 
>>> On Sun, 15 Aug 2010 15:44:34 +0200, Yannick Duch�ne (Hibou57) wrote:
>>>
>>> (I agree with what you wrote. I am programming a lot of communication
>>> stuff, but never used representation clauses to handle endianness.)
>> 
>> This is not surprising because the current representation clauses
>> cannot handle it.
>
> What for, if there are better ways to handle it? Representation layout
> clause was invented for handling something in place. It is no more actual,
> because reading out/writing in with an appropriate recoding is cleaner and
> possibly cheaper on modern hardware.

It's quite difficult to come up with the best approach to
deserialization for a particular architecture.  There are some pretty
common CPUs that handle unaligned loads quite well, so it's unclear
that the load-a-byte-at-a-time-and-shift approach is a win.  Same for
serialization.

>> I wonder if there has been any attempt to enhance representation
>> clauses so that they are useful in this context.
>
> To handle all possible cases of mixed-endian? To handle non-2's complement
> integers? IEEE, IBM, DEC float layouts? Layouts of Character like EBCDIC,
> KOI-7/8 etc? It would be a mammoth task to accomplish. What is the gain?

So why were representation clauses added to Ada in the first place?
Would it make sense to deprecate them?

(Endian issues arise even when controlling devices because most PCI
devices you can plug into a big-endian box expect little endian
memory-mapped I/O.)

Re: Using representation clauses in networking software

[Yannick Duchêne (Hibou57)]

Le Sun, 15 Aug 2010 17:04:15 +0200, Dmitry A. Kazakov  
<mailbox@dmitry-kazakov.de> a écrit:
> Representation layout clause was invented for handling something in
> place. It is no more  actual, because reading out/writing in with an
> appropriate recoding is cleaner and possibly cheaper on modern hardware.

Conversion at the hardware level ?

-- 
There is even better than a pragma Assert: a SPARK --# check.
--# check C and WhoKnowWhat and YouKnowWho;
--# assert Ada;
--  i.e. forget about previous premises which leads to conclusion
--  and start with new conclusion as premise.

Re: Using representation clauses in networking software

[Simon Wright]

Florian Weimer <fw@deneb.enyo.de> writes:

> Have there been any efforts to fix representation clauses so that they
> are suitable for expressing elements commonly found in networking
> protocols?
>
> My understanding is that representation clauses are totally unsuitable
> for this purpose because they do not allow the programmer to express
> endianness.  I dimly recall a technical report which argued that
> expressing endianness portably was impossible.  However, Erlang's bit
> syntax seems to be a counterexample, so I wonder if that issue has
> been picked up in recent years.

I haven't tried it but
.. http://www.adaic.com/standards/05rm/html/RM-13-5-3.html

Re: Using representation clauses in networking software

[Florian Weimer]

* Simon Wright:

> I haven't tried it but
> .. http://www.adaic.com/standards/05rm/html/RM-13-5-3.html

This only affects the interpretation of bit positions.  It does not
change the memory layout of components.

Re: Using representation clauses in networking software

[Dmitry A. Kazakov]

On Sun, 15 Aug 2010 17:32:02 +0200, Florian Weimer wrote:

> It's quite difficult to come up with the best approach to
> deserialization for a particular architecture.  There are some pretty
> common CPUs that handle unaligned loads quite well, so it's unclear
> that the load-a-byte-at-a-time-and-shift approach is a win.  Same for
> serialization.

Usually there are next 2-4 protocol levels, so anything you might gain at
this level will be lost anyway if you tried to keep things encoded. Worse
than that, some of the layers may explicitly state endiannes at run time.
(I know at least two examples of such protocols) Dynamic representation
clauses, shudder?

> So why were representation clauses added to Ada in the first place?

I think they were due to dominance of dual-ported memory and
non-standardized hardware that time.

> Would it make sense to deprecate them?

Maybe, especially because they do not allow writing portable programs
anyway. Under portability I understand that the hardware is fixed and the
target machine varies.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Using representation clauses in networking software

[Dmitry A. Kazakov]

On Sun, 15 Aug 2010 17:30:29 +0200, Yannick Duch�ne (Hibou57) wrote:

> Le Sun, 15 Aug 2010 16:32:47 +0200, Dmitry A. Kazakov  
> <mailbox@dmitry-kazakov.de> a �crit:
>> Byte ordering is what S'Bit_Ordering is, when bytes are addressable.
> Do you mean when byte and bit are the same ? I am sure to understand

The attribute is semantically defined for Word_Size > Storage_Unit.
Therefore it is actually "unit ordering", or "byte ordering" when
Storage_Unit is byte.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Using representation clauses in networking software

[Yannick Duchêne (Hibou57)]

Le Sun, 15 Aug 2010 18:10:08 +0200, Dmitry A. Kazakov  
<mailbox@dmitry-kazakov.de> a écrit:
>> Would it make sense to deprecate them?
>
> Maybe, especially because they do not allow writing portable programs
> anyway.
How do you handle this then ? How would interface API as an example ?  
Formated memory blocks ?

Can you give an example case where this shows to be contradictory with  
portability ?


-- 
There is even better than a pragma Assert: a SPARK --# check.
--# check C and WhoKnowWhat and YouKnowWho;
--# assert Ada;
--  i.e. forget about previous premises which leads to conclusion
--  and start with new conclusion as premise.

Re: Using representation clauses in networking software

[Dmitry A. Kazakov]

On Sun, 15 Aug 2010 18:40:15 +0200, Yannick Duch�ne (Hibou57) wrote:

> Le Sun, 15 Aug 2010 18:10:08 +0200, Dmitry A. Kazakov  
> <mailbox@dmitry-kazakov.de> a �crit:
>>> Would it make sense to deprecate them?
>>
>> Maybe, especially because they do not allow writing portable programs
>> anyway.
> How do you handle this then ?

Dynamically:

   Value :=  Unsigned_16 (Buffer (Index)) * 256 +
      Unsigned_16 (Buffer (Index + 1));
   Index := Index + 2;
   ...
exception
   when Constraint_Error =>
      Raise_Exception (Protocol_Error'Identity, "Malformed telegram at ...

BTW, an ability to handle protocol errors is another good argument against
representation clauses.

> How would interface API as an example ?  
> Formated memory blocks ?

It is usually layered, at least: raw I/O, telegrams, application layer. In
reality it can be more, because modern time protocols themselves are
layered. Many of them are dynamic. You just cannot statically describe all
queries and responses. You ask the device capacities, build query lists,
configure events, start measures, stop measures. It is far too complex for
representation clauses.

> Can you give an example case where this shows to be contradictory with  
> portability ?

When using representation clauses? In order to use them you have two
parameters in the equation: the internal endianness and external
endianness. How can it be statically independent on the former (=portable)?
Well, with conditional expressions of Ada 201X, maybe, it could be possible
to describe. But I don't want to be one to maintain that mess. I'd better
stick to good old means.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Using representation clauses in networking software

[Shark8]

Here's what I've been using to swap endian-ness... as you can see
though it is the "programmer's responsibility" to use it and use it
correctly; a representation clause for endian-ness might not be a bad
thing, it would make implementing the protocols which do specify the
endian-ness easier though.

   -- 8-bit 'word' [Byte]
   Type Word8 is mod 2**8;
    For Word8'Size use 8;

   -- Array of some number of bits... I am assuming that a single unit-
size will not exceed 255-bits.
   Type Bit_Array is Array (Word8 range<>) of Boolean;
   Pragma Pack(Bit_Array);


---- Function for swapping big-endian to little-endian and vice versa.

   Procedure Swap( B : in out Bit_Array ) is

      Low, High : Bit_Array(1..B'Length/2);
      For Low'Address use B'Address;
      For High'Address use B(1+B'Length/2)'Address;
   begin
      Case B'Length is
         when 64 | 32 | 16 =>
		Swap( Low );
		Swap( High );
		B(1..B'Length):=  High & Low;
         when 8 | 4 | 2 | 1 | 0 =>
            null;
         when others => raise Program_Error;  -- Non powers of 2 are
not supported; and sizes in excess of 64-bits.
      end case;
   end Swap;

Re: Using representation clauses in networking software

[Simon Wright]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> When using representation clauses? In order to use them you have two
> parameters in the equation: the internal endianness and external
> endianness. How can it be statically independent on the former (=portable)?
> Well, with conditional expressions of Ada 201X, maybe, it could be possible
> to describe. But I don't want to be one to maintain that mess. I'd better
> stick to good old means.

Yes, indeed.

Stick to network-byte-order on the wire, define messages explicitly,
preferably with a pictorial representation that is extremely clear as to
which byte is sent in which order and how this corresponds to machine
storage.

I don't think you can take too much trouble with specifying any sort of
interface; endianness differences just make matters more critical.

Re: Using representation clauses in networking software

[Maciej Sobczak]

On 15 Sie, 21:15, Simon Wright <si...@pushface.org> wrote:

> Stick to network-byte-order on the wire,

Just when ~99% of hardware in use is little-endian?
Sounds like a lost optimization opportunity with absolutely no overall
gain.

Hint: it most likely does not make much difference for individual
words, but things get more interesting with arrays - the longer the
array, the more interesting.

--
Maciej Sobczak * http://www.inspirel.com

YAMI4 - Messaging Solution for Distributed Systems
http://www.inspirel.com/yami4

Re: Using representation clauses in networking software

[Simon Wright]

Maciej Sobczak <see.my.homepage@gmail.com> writes:

> On 15 Sie, 21:15, Simon Wright <si...@pushface.org> wrote:
>
>> Stick to network-byte-order on the wire,
>
> Just when ~99% of hardware in use is little-endian?  Sounds like a
> lost optimization opportunity with absolutely no overall gain.

Perhaps I'm too used to working in an environment where it's fairly
evenly split! And in which the place where performance is crucial is on
a PowerPC.

If you know that your project will never talk to BE systems then even I
would have to agree there is a case.

Re: Using representation clauses in networking software

[Dmitry A. Kazakov]

On Sun, 15 Aug 2010 13:25:33 -0700 (PDT), Maciej Sobczak wrote:

> On 15 Sie, 21:15, Simon Wright <si...@pushface.org> wrote:
> 
>> Stick to network-byte-order on the wire,
> 
> Just when ~99% of hardware in use is little-endian?

This does not change anything.

The fact is that the *relevant* endianness is one on the wire. Believe or
not, but it is normal practice that two little-endian machines would
exchange big-endian encoded numbers if the protocol mandates it.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Using representation clauses in networking software

[anon]

In <8739ugqfeb.fsf@mid.deneb.enyo.de>, Florian Weimer <fw@deneb.enyo.de> writes:
>Have there been any efforts to fix representation clauses so that they
>are suitable for expressing elements commonly found in networking
>protocols?
>
>My understanding is that representation clauses are totally unsuitable
>for this purpose because they do not allow the programmer to express
>endianness.  I dimly recall a technical report which argued that
>expressing endianness portably was impossible.  However, Erlang's bit
>syntax seems to be a counterexample, so I wonder if that issue has
>been picked up in recent years.

  The representation clauses be used in endianness but it is a little 
  tricky the first time. After that it is easy to do. And it does not 
  matter if you using little or big endian Ada using the representation 
  clauses can handle them as is. So, no modications is needed.

  And one of the first uses of the enumeration representation clause was 
  to aid in converting programs files written in IBM EBCDIC character set 
  into ASCII set back in the mid 1980s using Ada 83.

--------
> So why were representation clauses added to Ada in the first place?

  First, to be able to assign internal values to enumeration value such 
  as the character set (see Standard package) or

    "type Boolean is ( False, True ) ;"  

  In most C compiler today False has an internal value 0, but there are 
  C compilers like Miscrosoft that were used back in the 70 to the mid 
  90s and the compiled code is still use today that uses -1, while others 
  C compilers assigned True to 0.  The representation clause allowed an 
  Ada programmer to correct this problem without having to deal with the 
  close source or binary code. Just interface and use the representation 
  clause to match the Ada code to the external world and the program is 
  good to go.

  Second, is to set size and location of elements in a pack record. A 
  simple example is the status registers, like a UART.  Does not matter 
  what the IO port it uses the status word values and bit pattern is 
  standardize and using the representation clause with a pack record 
  can allow a programmer to use simple Boolean logic instead of dealing 
  with integer masking algorithms. In other words, let the compiler do 
  the hard work.

  The Address representation clause has 100s of uses. Like setting up 
  DOS and no-OS type of interrupts. Also can be use to calculate the 
  physical memory size for an Ada OS. 

  All of which allows Ada to translate the outside environment into 
  the Ada world.  Making the Ada program more portable and universal 
  which extents the life of the program!

--------
> Would it make sense to deprecate them?

  The answer is NO!!! And they should not be expanded! They have a 
  special place to aid the programmer in special areas. Such as insuring 
  the code is portable when dealing with others areas that are not 
  standard or in a pre-standard design! 


  An example is in writhing a keyboard driver.  Is it easier to replace 
  the keyboard layout every time a different keyboard is used and then 
  re-booting or is it better to add an additional translation package 
  using representation clause. Then allow the program to ask which layout 
  to use from the hardware or in some cases the user.

Re: Using representation clauses in networking software

[Stephen Leake]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> On Sun, 15 Aug 2010 15:44:34 +0200, Yannick Duchêne (Hibou57) wrote:
>
> (I agree with what you wrote. I am programming a lot of communication
> stuff, but never used representation clauses to handle endianness.)
>
>> I do not know neither any representation clause for Byte Ordering. 
>
> Byte ordering is what S'Bit_Ordering is, when bytes are addressable.

Not exactly.

> IMO, Bit_Ordering was an unfortunate choice. The attribute name suggests
> ordering of bits in some machine storage unit, which it is not.

Bit_ordering specifies how to interpret the bit numbers in Ada record
representation clauses.

The restriction of 'Bit_Order only being valid for the target endianness
was removed in Ada 2005.

It is useful for representing record layouts that may be used in
systems with different byte orders.

See http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ais/ai-00133.txt?rev=1.17
for a good discussion. The concept of 'machine scalars' is important.
Only part of this discussion made it into the ARM or AARM, so they are
hard to understand.

Still, S'Bit_ordering does _not_ solve the inter-machine byte endianness
problem. It could be part of a solution.

-- 
-- Stephe

Re: Using representation clauses in networking software

[Randy Brukardt]

"Florian Weimer" <fw@deneb.enyo.de> wrote in message 
news:8762zbhnhy.fsf@mid.deneb.enyo.de...
>* Simon Wright:
>
>> I haven't tried it but
>> .. http://www.adaic.com/standards/05rm/html/RM-13-5-3.html
>
> This only affects the interpretation of bit positions.  It does not
> change the memory layout of components.

When we discussed this the last time, it was concluded that supporting out 
of order byte reads (especially those for unusual numbers of bits) is just 
too expensive for implementations for the relatively small amount of use 
that they would get. Moreover, such things could not be volatile or atomic 
on most hardware (because they would require multiple reads/writes of the 
various pieces - the pieces wouldn't necesssarily be contiguous in memory 
for the non-native format), so they couldn't be used for hardware 
interfaces. That would eliminate half of the potential uses.

Remember that read/writes of memory are probably the most fundamental thing 
that a compiler does; supporting split reads would require changes to 
virtually every part of a compiler. (For Janus/Ada, we would have to add 
intermediate code instructions to support such reads, with costs in 
everything that handles intermediate code and target code generation.)

                         Randy.

Re: Using representation clauses in networking software

[Florian Weimer]

* Maciej Sobczak:

> On 15 Sie, 21:15, Simon Wright <si...@pushface.org> wrote:
>
>> Stick to network-byte-order on the wire,
>
> Just when ~99% of hardware in use is little-endian?
> Sounds like a lost optimization opportunity with absolutely no overall
> gain.

On most CPUs, dealing with misaligned loads when converting from a
byte stream is more expensive than the endianness conversion.