AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

It seems that 2 major use-cases of AI12-0218 are getting short shrift.

http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ai12s/ai12-0218-1.txt?rev=1.3&raw=N

USE-CASE:  network byte order:
1) Nearly all telecom protocols operate on big-endian octet ordering.  Big-endian octet ordering in all the Internet protocols' packet formats is so common that IETF calls it “network byte ordering”.  Conversely, the 3 most widely deployed ISAs on the planet (i.e., IA-32, AMD64, ARM*) are all little-endian.  This matters not for line-rate packets and frames processed by ASICs or FPGAs or DSPs, but for the packets whose processing is at software speed in general-purpose processors, byte-swapping is what makes the Internet work on the computer that you are using right this instant (unless you are using a PowerPC-, POWER, 68K-, MIPS, or mainframe-based computer or an Alpha- or Itanium-based computer configured in the rare big-endian configuration).  So this is not a rare use-case at all!

* Like many RISC processors, endianness is a choice at PCB engineering-time.  The vast vast majority of ARM-based designs choose little-endian.

Instead of the use-case being rare, what is apparently rare is for Ada to be utilized to write such network-protocol software on Intel and ARM processors instead of C or C++.  Most of the military uses PowerPC as their general-purpose processors, and rarely ARM and even rarer Intel.

USE-CASE:  memory-mapped registers on peripheral ICs:
2) When writing device drivers for many ICs, the IC's register set often is memory-mapped at a fixed address.  This IC has either big-endian or little-endian octet ordering.  The general-purpose processor is either big-endian or little-endian.  Often ICs from big-endian manufacturers are utilized on processors that are little-endian.  Often ICs from little-endian manufacturers are utilized on processors that are big-endian.  Sometimes endianness is configurable in those ICs; often it is not.

So in AI12-0218 when I see resistance of ARG members to standardizing a widespread use-case that was (in the 1970s-/1980s-/1990s-era Ada-is-the-future mantra) supposed to be Ada's sweet-spot (i.e., writing device drivers portably), I can see one crucial reason that Ada was frowned upon in writing packet-processing and device-driver software an Intel and ARM platforms, which are the most popular by far of all ISAs.

Little-endian platforms (who need byte-swapping for network byte order):
all Apple iOS devices, all ARM-based Android devices, all Windows 10 computers, all Windows 8.1 computers, all Windows 7 computers, all Windows Server 2008, all Windows Server 2012, all Windows Server 2016, all Linux on IA-32, all Linux on AMD64, all Linux on ARM

Big-endian platforms (who don't need byte-swapping for network byte order):
the other less-than-small-fraction-of-1% of all other computers on the planet.  No hyperbole:  far far less than 1% of extant computers throughout planet Earth.

Good to know that (for ≥34 years!) Ada has been standardized for the less-than-small-fraction-of-1% of computers on the planet nowadays.  This isn't regarding the old arcane convert-COBOL-to-Ada featureset of Ada either.  This is regarding device drivers interacting with hardware ICs, which is supposedly one of Ada's A#1 key must-do marketspaces.

And the endian feature of AI12-0218 is not being embraced by the ARG.  Wow!  It is not viewed by the ARG as a ≥34-year-old embarrassment that might have actually caused unpopularity of Ada over the decades in its potential key advocates:  realtime software engineers writing device drivers in Ada's supposed key marketspace of reading & writing IC registers, meeting realtime deadlines before packet-arrival over-run (or whatever mission the IC is accomplishing in realtime).

The Challenge:
Without the proposed solution in AI12-0218, I challenge anyone to write an Ada representation clause that is •portable• to both little-endian and big-endian processors for either of the following IETF packet formats that must always be in big-endian network byte order:
https://en.wikipedia.org/wiki/IPv4#Packet_structure
https://en.wikipedia.org/wiki/IPv6_packet

Hint #1:  in C or C++, different definitions of preprocessor macros (controlled by #if conditional compilation directives)) would typically be utilized for portably cloaking lack-of-byte-swap-no-op on big-endian processors and byte-swapping on little-endian processors, but Ada lacks macros, so the ••representation clause•• in AI12-0218 (on only GNAT for now) is the only game in Adatown, it would seem.  And other than on GNAT in the past few years, the representation clause as standardized cannot accomplish the goal portably.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Thursday, May 10, 2018 at 12:45:43 PM UTC-5, Dan'l Miller wrote:
> … byte-swapping is what makes the Internet work on the computer that you are using right this instant
> (unless you are using a PowerPC-, POWER-, 68K-, MIPS-, or mainframe-based computer or an Alpha- or
> Itanium-based computer configured in the rare big-endian configuration).

Oops.  In that list, I forgot SPARC there for another 0.01%* of the computers on planet Earth.

* I am completely uncertain whether that is hyperbole or fact.  I pulled that number out of thin air, but it sounds about right.

And just in case anyone is thinking, “Oh, every single one of the new designs of ISAs have been big-endian for decades; all those little-endian ISAs are all for antiquated legacy reasons:  ARM derived from little-endian 6502; Alpha derived from little-endian VAX; Intel derived from Fairchild”, the newest (open-source!) hot-shot kid on the block in the ISA neighborhood is •little-endian•:  RISC-V.

https://en.wikipedia.org/wiki/RISC-V

RISC-V looks promising too, likely to give PowerPC, MIPS, SPARC, and ARM a run for their money.  Here is Esperanto's debut RISC-V product, which as a leap forward has practically no direct competitor in any other ISA:
https://fuse.wikichip.org/news/686/esperanto-exits-stealth-mode-aims-at-ai-with-a-4096-core-7nm-risc-v-monster

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

"Dan'l Miller" <optikos@verizon.net> writes:

> so the ••representation clause•• in AI12-0218 (on only GNAT for now)
> is the only game in Adatown

I don't have a reference, but there was a cunning plan once involving
the fact that Boolean'Pos (False) is 0, and Boolean'Pos (True) is 1. So
you could write expressions involving

   BE : constant :=
      Boolean'Pos (System.Default_Bit_Order = System.High_Order_First);

& likewise for LE, and you could then write representation components as

   (le-value * LE + be-value * BE)

(something like that, anyway). Ugh.

But you still have to byte-swap.

The worst case I came across was where the remote system was LE, the
network was LE (there's British for you), we were BE, and the interface
device - in a spirit of helpfulness - did byte-swapping for you assuming
that the whole message consisted of 16-bit values (because the message
header did, and that was the only bit they needed to deal with; they
were on a BE device). Bad luck if one of the message fields was 32-bits.

It is a pain in the neck, and something that newer programmers found
hard to get their heads round, but it is only needed at the edges of the
system (letting byte-swapping inside the system proper is going to be
very expensive).

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dmitry A. Kazakov]

On 2018-05-10 19:45, Dan'l Miller wrote:
> It seems that 2 major use-cases of AI12-0218 are getting short shrift.
> 
> http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ai12s/ai12-0218-1.txt?rev=1.3&raw=N
[...]
> The Challenge:
> Without the proposed solution in AI12-0218, I challenge anyone to write an Ada representation clause that is •portable• to both little-endian and big-endian processors for either of the following IETF packet formats that must always be in big-endian network byte order:
> https://en.wikipedia.org/wiki/IPv4#Packet_structure
> https://en.wikipedia.org/wiki/IPv6_packet

Representation clauses are useless for network protocols and 
serialization, for multiple reasons. I never used them in any of the 
protocol implementations I wrote.

The only place representation clauses could have is mapping special 
machine registers, which naturally excludes any questions of portability.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Thursday, May 10, 2018 at 2:34:39 PM UTC-5, Dmitry A. Kazakov wrote:
> Representation clauses are useless for network protocols and 
> serialization

Serialization?  LoL!!  Streamed field-by-field (even textual?) serialization like OO designs do?  Hahahaha.  That is not even on-topic at all regarding fixed-header binary packet layouts in this thread.  When the Dmitry Engineering Task Force (DETF) designs the Dmitrynet protocol stack to replace the Internet protocol stack at & near layer 3, you can rework all of IETF's fixed packet-header binary layouts to be your ‘serialization’.  But until then, we're not tilting at windmills; we live in IETF's reality of their packet-header formats.  Now show how to process them portably in Ada without AI12-0218.

>, for multiple reasons. I never used them in any of the 
> protocol implementations I wrote.

LoL!!  Oh, do tell why a representation clause on an IETF packet's header layout is “useless”.  (For decades and decades, IETF intentionally designs its packet headers to be typecasted as meticulously-laid-out C structs.)

> The only place representation clauses could have is mapping special 
> machine registers, which naturally excludes any questions of portability.

LoL!!  Oh, so an IC manufacturer doesn't want to sell their chips to both big-endian and little-endian accounts?  Or is it that the IC manufacturer wants to write entirely different source code for different processors?  Or is it that the IC manufacturer never provides the source code to a reference-implementation device driver to their customers at all?  Job security by elimination of all portability of software; write a fresh variant for every target, even if they differ only slightly.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Paul Rubin]

"Dan'l Miller" <optikos@verizon.net> writes:
> And just in case anyone is thinking, “Oh, every single one of the new
> designs of ISAs have been big-endian for decades

No it's for real, you do have to consider enddianness in any network
program, though like Dmitry I'm unconvinced that representation clauses
are the way to do it in Ada.  In C programs you usually just call some
macros to convert machine integers to and from network byte order.  In
the case where the machine int's bytes are already in network order, the
macro does nothing.

Example 4 of this page

  http://erlang.org/doc/programming_examples/bit_syntax.html

shows how you describe an IP datagram in Erlang:

    -define(IP_VERSION, 4).
    -define(IP_MIN_HDR_LEN, 5).

    DgramSize = byte_size(Dgram),
    case Dgram of 
        <<?IP_VERSION:4, HLen:4, SrvcType:8, TotLen:16, 
          ID:16, Flgs:3, FragOff:13,
          TTL:8, Proto:8, HdrChkSum:16,
          SrcIP:32,
          DestIP:32, RestDgram/binary>> when HLen>=5, 4*HLen=<DgramSize ->
            OptsLen = 4*(HLen - ?IP_MIN_HDR_LEN),
            <<Opts:OptsLen/binary,Data/binary>> = RestDgram,
        ...
    end.

It isn't bad, if you are used to Erlang syntax.  Those fields like
FragOff:13 are bit fields of the specified lengths.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Thursday, May 10, 2018 at 3:32:11 PM UTC-5, Paul Rubin wrote:
> "Dan'l Miller" writes:
> …  In C programs you usually just call some
> macros to convert machine integers to and from network byte order.  In
> the case where the machine int's bytes are already in network order, the
> macro does nothing.
> 
> Example 4 of this page
> 
>   http://erlang.org/doc/programming_examples/bit_syntax.html
> 
> shows how you describe an IP datagram in Erlang:
> 
>     -define(IP_VERSION, 4).
>     -define(IP_MIN_HDR_LEN, 5).
> 
>     DgramSize = byte_size(Dgram),
>     case Dgram of 
>         <<?IP_VERSION:4, HLen:4, SrvcType:8, TotLen:16, 
>           ID:16, Flgs:3, FragOff:13,
>           TTL:8, Proto:8, HdrChkSum:16,
>           SrcIP:32,
>           DestIP:32, RestDgram/binary>> when HLen>=5, 4*HLen=<DgramSize ->
>             OptsLen = 4*(HLen - ?IP_MIN_HDR_LEN),
>             <<Opts:OptsLen/binary,Data/binary>> = RestDgram,
>         ...
>     end.
> 
> It isn't bad, if you are used to Erlang syntax.  Those fields like
> FragOff:13 are bit fields of the specified lengths.

Ummmmmm, Paul, neither your example of C-esque bitfields in Erlang nor the Erlang tutorial page that you provided nor the Erland reference manual link in that tutorial page provided a solution to the byte-swapping.  How to actually accomplish the byte-swapping portably is the core topic of this entire thread and the core topic of AI12-0218.  Why post clearly off-topic material?  (Not only Erlang is prima facie off-topic, but bit fields in local-machine's endianness without any byte swapping is evermore off-topic, which could have been made somewhat on-topic if anything in Erlang demonstrated byte-swapping in opposing-endianness inter-machine communications, with an air of:  if Erlang can elegantly declare the byte-swapping portably on heterogenous endianness, then is there an analogous design in pre-AI12-0218 Ada?)

Everyone:
For example, in standard Ada without AI12-0218's proposal, how can one code up byte-swapping the fields of network byte order (big-endian) IETF packet headers on a little-endian machine?  Is there some clever switcheroo with processor-specific choice of child packages that contain little- versus big-endian subroutines analogous to the conditionally-compiled different definitions of C macros that both Paul and I have mentioned?

Paul:
Btw, bitfields in C and in Erlang are •the• roughly analogous language feature corresponding to representation clauses in Ada; all the undefined behavior regarding bitfields in C notwithstanding.  Indeed, AIUI, C's bitfields & their 1970s-known undefined behavior when porting among different processors were the public enemy #1 on Ichbiah et al's minds when they put bitwise representation clauses in Ada.  So you undermined your own line of reasoning there.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Simon Wright" <simon@pushface.org> wrote in message 
news:ly4ljfgvim.fsf@pushface.org...
> "Dan'l Miller" <optikos@verizon.net> writes:
>
>> so the ..representation clause.. in AI12-0218 (on only GNAT for now)
>> is the only game in Adatown
>
> I don't have a reference, but there was a cunning plan once involving
> the fact that Boolean'Pos (False) is 0, and Boolean'Pos (True) is 1. So
> you could write expressions involving
>
>   BE : constant :=
>      Boolean'Pos (System.Default_Bit_Order = System.High_Order_First);
>
> & likewise for LE, and you could then write representation components as
>
>   (le-value * LE + be-value * BE)
>
> (something like that, anyway). Ugh.

That's an Ada 83 approch. With Ada 2012, you could be more direct:

     (if BE then be-value else le-value)

> But you still have to byte-swap.

Right. The problem with AI12-0218-1 (besides the fact that no one other than 
the author understands it, and the author has not volunteered to help), is 
that it makes it way too easy to put very expensive byte-swapping code into 
data structures that occur everywhere.

It's also very expensive to implement, since every compiler back-end has to 
be aware of it. (If you just implement it in the front-end using code 
equivalent to unchecked_conversions, you're forcing all of the operations 
into memory and don't allow any hardware support to get used.) Ada-specific 
back-ends definitely don't support byte-swapping (it never appears anywhere 
in existing Ada, the rules are designed to prevent it). A change to the 
back-end can be much more expensive than changes to front-ends, simply 
because there are many more of them and every individual target has to be 
tested carefully. (Front-ends only need to be tested carefully once; other 
testing only can surface latent back-end bugs.)

                            Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Paul Rubin]

"Dan'l Miller" <optikos@verizon.net> writes:
> Serialization?  LoL!!  Streamed field-by-field (even textual?)
> serialization like OO designs do?

Serialization just means representing a data structure as a series of
bytes.  Some languages can do it automatically by reflection on the
datatype description, giving a convenient representation for exchanging
those objects with another program written in the same language.  In the
case of an IP packet you'd write the serialization code yourself so that
the representation would be packet format sent over the wire.

> LoL!!  Oh, so an IC manufacturer doesn't want to sell their chips to
> both big-endian and little-endian accounts?  Or is it that the IC
> manufacturer wants to write entirely different source code for
> different processors?

Do you write any actual programs, bro?  To switch between big and little
endian processors in C you change a #ifdef to redefine some byte
swapping macros, not write entirely different source code.  Here:

   https://linux.die.net/man/3/byteorder

It's inconceivable to me that Ada networking programs don't do something
similar.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Niklas Holsti]

On 18-05-11 01:24 , Dan'l Miller wrote:
> On Thursday, May 10, 2018 at 3:32:11 PM UTC-5, Paul Rubin wrote:
>> "Dan'l Miller" writes:
>> …  In C programs you usually just call some
>> macros to convert machine integers to and from network byte order.  In
>> the case where the machine int's bytes are already in network order, the
>> macro does nothing.
>>
>> Example 4 of this page
>>
>>   http://erlang.org/doc/programming_examples/bit_syntax.html
>>
>> shows how you describe an IP datagram in Erlang:
>>
>>     -define(IP_VERSION, 4).
>>     -define(IP_MIN_HDR_LEN, 5).
>>
>>     DgramSize = byte_size(Dgram),
>>     case Dgram of
>>         <<?IP_VERSION:4, HLen:4, SrvcType:8, TotLen:16,
>>           ID:16, Flgs:3, FragOff:13,
>>           TTL:8, Proto:8, HdrChkSum:16,
>>           SrcIP:32,
>>           DestIP:32, RestDgram/binary>> when HLen>=5, 4*HLen=<DgramSize ->
>>             OptsLen = 4*(HLen - ?IP_MIN_HDR_LEN),
>>             <<Opts:OptsLen/binary,Data/binary>> = RestDgram,
>>         ...
>>     end.
>>
>> It isn't bad, if you are used to Erlang syntax.  Those fields like
>> FragOff:13 are bit fields of the specified lengths.
>
> Ummmmmm, Paul, neither your example of C-esque bitfields in Erlang
> nor the Erlang tutorial page that you provided nor the Erland
> reference manual link in that tutorial page provided a solution
> to the byte-swapping.

Both the example, and the Erlang reference manual section linked from 
the example, describe an "endianness" attribute that can be given one of 
the values "big", "little", "native", with default "big". However, it 
seems this attribute applies only to a restricted set of types: integer, 
utf16, utf32 and float.

It is not clearly explained how endianness is interpreted for fields 
that do not align with octet boundaries.

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Dan'l Miller" <optikos@verizon.net> wrote in message 
news:2821b8e0-2c64-4621-b65e-337027fc121e@googlegroups.com...

>But until then, we're not tilting at windmills; we live in IETF's reality 
>of their
>packet-header formats.  Now show how to process them portably in Ada
>without AI12-0218.

Why would you want to process them portably? Low-level hardware specific 
code (and socket drivers are always that) is never portable (too much 
hardware dependence); the usual Ada plan is to carefully encapsulate such 
stuff in a package and provide several bodies for different targets. Code 
using such a package is portable of course (any differences are filed-off by 
the package).

Portable representation clauses are a waste of time, especially as machines 
are inconsistent in the ways that they number bits. Just don't go there. 
(218 solves one such problem, but there are many others.)

                     Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Thursday, May 10, 2018 at 5:40:49 PM UTC-5, Randy Brukardt wrote:
> … If you just implement it in the front-end using code 
> equivalent to unchecked_conversions, you're forcing all of the operations 
> into memory

Wait, what is the actual extant-in-reality use-case for byte-swapping between registers (within the same processor!) of (supposedly-)heterogenous endianness?  (Not even the PDP-11's mixed endianness did that!)  Does any processor exhibit such odd endianness within its own processor('s registers)?  Methinks this is pure bicycle-shedding without any basis in reality.

> don't allow any hardware support to get used.

Hmmm, that one is somewhat more compelling if the processor has byte-swapping instructions.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Paul Rubin]

"Dan'l Miller" <optikos@verizon.net> writes:
> Btw, bitfields in C and in Erlang are •the• roughly analogous language
> feature corresponding to representation clauses in Ada

Can you suggest a more complete description of representation clauses
than in Ada 95 Distilled?  If they mean what I expected them to mean,
they actually specify the memory layout of the data object.

That is, if you use a hypothetical endianness representation clause to
declare something like "type bigint32 is int32 big_endian", then a
record containing "x, y : bigint32" will actually require x and y to be
stored in memory in big-endian byte order, and will show up that way if
you dump the memory region to disk.

That particularly means if the hardware happens to be little-endian and
you want to add x and y together with a 32-bit "add" machine
instruction, you are requiring the compiler to generate code that reads
x and y from memory and performs byte swaps before doing the addition.

There is an analogous situation where you declare a decimal integer in
PL/I or COBOL on a binary machine (i.e. almost any machine of the past
several decades).  In that case the integer is stored in BCD and
converted before and after any arithmetic.  It is ok to do that because
those sorts of programs simply don't do much arithmetic, so the
performance hit is not so bad.

I've never seen a network program do it like that though.  Everyone just
makes sure that network integers are converted to the host format before
being stored in memory, and converted back when sent over the network.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Paul Rubin]

Niklas Holsti <niklas.holsti@tidorum.invalid> writes:
> It is not clearly explained how endianness is interpreted for fields
> that do not align with octet boundaries.

I don't see an unclarity as long as the machine is either big or little
endian.  That specifies whether the more significant bits are at the
higher addresses or the lower ones.  There could possibly be some
confusion with the weird old VAX representation that was neither big nor
little, but the Erlang page doesn't try to deal with that case.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Thursday, May 10, 2018 at 6:07:20 PM UTC-5, Paul Rubin wrote:
> "Dan'l Miller" writes:
> > Btw, bitfields in C and in Erlang are •the• roughly analogous language
> > feature corresponding to representation clauses in Ada
> 
> Can you suggest a more complete description of representation clauses
> than in Ada 95 Distilled?

http://www.ada-auth.org/standards/aarm12_w_tc1/AA-Final.pdf
§13.5.1 Record Representation Clauses, page 571

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Paul Rubin]

"Dan'l Miller" <optikos@verizon.net> writes:
> http://www.ada-auth.org/standards/aarm12_w_tc1/AA-Final.pdf
> §13.5.1 Record Representation Clauses, page 571

Thanks, I've downloaded that manual though my likelihood of actually
reading it (1332 pages!) isn't great.  The section on representation
clauses is very verbose and I'd have to check it microscopically to make
sure, but I do get the impression that it's intended to specify actual
memory layout.  That is a poor way to handle network byte order for the
reasons I gave.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Thursday, May 10, 2018 at 7:30:43 PM UTC-5, Paul Rubin wrote:
> … it's intended to specify actual memory layout.

vis a vis

> That is a poor way to handle network byte order …

How does specifying “actual memory layout“ not include the topic of byte order?  Endianness is quite honestly the ultimate very first topic in an ISA's definition of “actual memory layout”, either at processor engineering-time or (in bi-endian processors) at PCB engineering-time of pulling a pin high or low.  Endianness even predates word size:  even before software choses to write or read a 16- or 32- or 64-bit integer, its endianness in predetermined beforehand (months or years or decades prior to software's johnny-come-lately decision of word size on which to operate).

Btw, what is nonactual memory layout or actual nonmemory layout or actual memory nonlayout?  I have no idea what the word actual is meaning there or what actual’s presence/absence means.  It would seem that “memory layout” and “actual memory layout” mean exactly the same thing.  And the layout of words and bytes in memory includes their endian order.  (And to Randy's concerns about front-end implementation forcing layout of memory instead of intra-processor registers, there is no such thing as layout of nonmemory (i.e., registers within a processor) because registers do not have addresses, and hence lack endianness of which byte is at the lower-valued address).

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Paul Rubin]

"Dan'l Miller" <optikos@verizon.net> writes:
>> … it's intended to specify actual memory layout.
>> That is a poor way to handle network byte order …
> How does specifying “actual memory layout“ not include the topic of
> byte order?

I don't think representation clauses in Ada try to specify byte order of
integers.  A 32-bit integer is treated as an opaque object.  It is
represented in memory as 4 bytes but in to the doc I looked at, the byte
order wasn't specified.

In C, the byte order is observable because you can take the address of a
32-bit integer, cast it into a char pointer, and then look at the 4
bytes one at a time.  But I think in Ada, you aren't supposed to cast
addresses like that.  An int is an int and a byte is a byte.

> Endianness is quite honestly the ultimate very first topic in an ISA's
> definition of “actual memory layout”

Yes, and if Ada programs are supposed to be portable they shouldn't care
what the byte order is.  You can convert network to native byte order
portably, something like:

  uint32 n = byte0 << 24 | byte1 << 16 | byte2 << 8 | byte3;

where byte0, byte1, byte2, and byte3 are the 4 bytes from the network
packet, giving the integer in big-endian order.  That should work
regardless of the cpu endianness.  But, if the cpu is already big-endian
then you have done a bunch of arithmetic that could have been replaced
with a no-op.  So in practice you use a conditionally-compiled macro.

> Btw, what is nonactual memory layout or actual nonmemory layout or
> actual memory nonlayout?

I just mean if you say "uint32 n = 123456789;" that normally means you
want 4 bytes in memory holding the machine's representation of that
integer in the machine's native byte order, whatever that order happens
to be.  In big-endian order the 4 bytes (starting from the lowest
address) would be 07 5B CD 15, while in little-endian order they would
be 15 CD 5B 07.  You leave it up to the compiler to generate this layout
machine-dependently, instead of specifying it yourself.

By "specify the actual memory layout" I mean you hypothetically write
something like "big_endian_32 n = 123456789;" and that means you want
the 4 bytes in memory to be 07 5B CD 15 in that order, even if the
machine is little-endian.  That means if you want to increment n on a
little-endian machine, you have to read it from memory, swap the bytes
around, increment, swap around the bytes of the result, and write them
back out to memory.

Normally a network programmer wouldn't do that.  They'd read the
big-endian sequence 07 5B CD 15 from the network, convert it (using a
macro) to the endianness of whatever machine they are running on, and
store it in memory in the machine's native byte order.  Then if they
want to increment it, they just use an increment instruction with no
byte swapping needed.  Finally if they want to write it back out to the
network, they convert it to network byte order before sending.

I hope that helps.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Thursday, May 10, 2018 at 8:34:46 PM UTC-5, Paul Rubin wrote:
> … if you want to increment n on a
> little-endian machine, you have to read it from memory, swap the bytes
> around, increment, swap around the bytes of the result, and write them
> back out to memory.

Reading and assignment are the only operations that are essential to support on fields whose representation is governed by endian representation clauses.  Arithmetic on temporaries extracted from the oddball representation would be an expendable luxury.

> Normally a network programmer wouldn't do that.

My point is normally a network programmer does exactly what you describe ••in C••, not Ada, because the engineer years or decades ago established that C's macros and pointer arithmetic can do what Ada's record representation clauses cannot, and C's macros and pointer arithmetic can do what is disdained as uncouth to be performed by Ada's various accesses: byte manipulations.

> They'd read the
> big-endian sequence 07 5B CD 15 from the network, convert it (using a
> macro)

Ada doesn't have such a macro processor.

> to the endianness of whatever machine they are running on, and
> store it in memory in the machine's native byte order.  Then if they
> want to increment it, they just use an increment instruction with no
> byte swapping needed.  Finally if they want to write it back out to the
> network, they convert it to network byte order before sending.

Of course this whole topic is focused on the perimeter of the system at the incoming edge of input from network (or reading from registers in a memory-mapped IC) and at the outgoing edge of output to network (or writing to registers in a memory-mapped IC).  Of course this is not about heterogenous endianness utilized deep throughout the guts of software designs.  (Nor is this about [preposterous] heterogenous endianness among the registers within a single processor—memory, memory, memory of a transient artifact incoming to the system or outgoing from the system.)

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Thursday, May 10, 2018 at 5:44:21 PM UTC-5, Niklas Holsti wrote:
> Both the example, and the Erlang reference manual section linked from 
> the example, describe an "endianness" attribute that can be given one of 
> the values "big", "little", "native", with default "big".

Ah, yes, now I see it.  Thank you for pointing that out.  I apologize, Paul.

> However, it 
> seems this attribute applies only to a restricted set of types: integer, 
> utf16, utf32 and float.
> 
> It is not clearly explained how endianness is interpreted for fields 
> that do not align with octet boundaries.

Ah, Paul & Randy & Dmitry, right there might be the best card for the opposition to play:  endianness (independent of Ada's record representation clauses) has absolutely nothing at all to do with arbitrary bitfields.  Endianness exists when reading 16-, 32-, 64-, 80-, 128-, 256-, and 512-bit words from memory or writing them to memory.  Hence, incorrect endianness of processing, say, an IETF packet header depends on the ••word size (arbitrarily chosen to be) utilized by the software•• to (capriciously) access the fields within (or crossing the boundaries of) that 16-, 32-, 64-, etc-bit word, not the •record itself•.  It only becomes part of the record per se when trying to utilize Ada's record representation clauses or C's struct-of-bitfields type-casting.  Endianness is on a plane above the fields of the record—the parent nodes of a tree (of word boundaries) hovering above the record, casting a shadow of these boundary-lines, so to speak, on the bitwise fields that might not be byte or word aligned at all, within the words or crossing the boundary of words.

Still, standard Ada has no good Ada-esque solution to heterogenous endianness at the perimeter of a system, other than writing C-esque pointer-arithmetic code with the various unchecked_ constructs (which even C programmers don't generally do; they utilize conditionally-compiled macros that correctly type-cast meticulously*-laid-out structs-of-bitfields onto that packet's header or IC register's word.

* on a per-processor, per-toolchain basis, selected via the conditional compilation via the C preprocessor.  This selection (out of nonportable categorization) is what accomplishes the portability.  Ada has no equivalent other than AI32-0218's/GNAT's solution.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

"Dan'l Miller" <optikos@verizon.net> writes:

> Still, standard Ada has no good Ada-esque solution to heterogenous
> endianness at the perimeter of a system, other than writing C-esque
> pointer-arithmetic code with the various unchecked_ constructs (which
> even C programmers don't generally do; they utilize
> conditionally-compiled macros that correctly type-cast
> meticulously*-laid-out structs-of-bitfields onto that packet's header
> or IC register's word.

I've used (effectively) conditionally-compiled sections of code for
this.

From an SNTP implementation: in BE, the first 32 bits of a packet are

   --   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   --  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   --  |LI | VN  |Mode |    Stratum    |     Poll      |   Precision   |
   --  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

and looking just at the top (first) byte, the package spec contains

   type Leap_Indicator is (No_Warning,
                           Last_Minute_Has_61_Seconds,
                           Last_Minute_Has_59_Seconds,
                           Alarm_Condition);

   type Version is range 3 .. 4;

   type Mode is (Reserved,
                 Symmetric_Active,
                 Symmetric_Passive,
                 Client,
                 Server,
                 Broadcast,
                 Reserved_For_NTP_Control_Message,
                 Reserved_For_Private_Use);

   type Status is record
      LI : Leap_Indicator;
      VN : Version;
      M : Mode;
   end record;

in native layout (nowadays I'd be a lot more specific about
sizes). Conversions:

   function To_Stream_Element (S : Status) return Ada.Streams.Stream_Element;

   function To_Status (S : Ada.Streams.Stream_Element) return Status;

In the body,

   Big_Endian : constant Boolean
     := System."=" (System.Default_Bit_Order, System.High_Order_First);

and

   function To_Status (S : Ada.Streams.Stream_Element) return Status is
   begin
      --  these two sections are conditionally compiled (by GNAT,
      --  anyway) because Big_Endian is constant.
      if Big_Endian then
         declare
            --  create a BE type with BE representation
            type Host_Status is record
               LI : Leap_Indicator;
               VN : Version;
               M : Mode;
            end record;
            for Host_Status use record
               LI at 0 range 0 .. 1;
               VN at 0 range 2 .. 4;
               M  at 0 range 5 .. 7;
            end record;
            for Host_Status'Size use 8;
            function Convert
            is new Ada.Unchecked_Conversion (Ada.Streams.Stream_Element,
                                             Host_Status);
            V : constant Host_Status := Convert (S);
         begin
            --  let the compiler convert from BE representation to host
            return (LI => V.LI, VN => V.VN, M => V.M);
         end;
      else
         declare
            --  create an LE type with LE representation
            type Host_Status is record
               LI : Leap_Indicator;
               VN : Version;
               M : Mode;
            end record;
            for Host_Status use record
               LI at 0 range 6 .. 7;
               VN at 0 range 3 .. 5;
               M  at 0 range 0 .. 2;
            end record;
            for Host_Status'Size use 8;
            function Convert
            is new Ada.Unchecked_Conversion (Ada.Streams.Stream_Element,
                                             Host_Status);
            V : constant Host_Status := Convert (S);
         begin
            --  let the compiler convert from LE representation to host
            return (LI => V.LI, VN => V.VN, M => V.M);
         end;
      end if;
   end To_Status;

No denying it's a lot of work. Much easier for 2-, 4-, 8-byte objects:

   type SNTP_Timestamp is delta 2.0 ** (-32) range -2.0 ** 31 .. 2.0 ** 31;
   for SNTP_Timestamp'Size use 64;

   subtype Timestamp_Slice is Ada.Streams.Stream_Element_Array (1 .. 8);

   function To_SNTP_Timestamp (T : Timestamp_Slice) return SNTP_Timestamp is
      function Convert is new Ada.Unchecked_Conversion (Timestamp_Slice,
                                                        SNTP_Timestamp);
   begin
      if Big_Endian then
         return Convert (T);
      else
         return Convert ((1 => T (8),
                          2 => T (7),
                          3 => T (6),
                          4 => T (5),
                          5 => T (4),
                          6 => T (3),
                          7 => T (2),
                          8 => T (1)));
      end if;
   end To_SNTP_Timestamp;

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

Paul Rubin <no.email@nospam.invalid> writes:

> "Dan'l Miller" <optikos@verizon.net> writes:
>> Btw, bitfields in C and in Erlang are •the• roughly analogous language
>> feature corresponding to representation clauses in Ada
[...]
> That is, if you use a hypothetical endianness representation clause to
> declare something like "type bigint32 is int32 big_endian", then a
> record containing "x, y : bigint32" will actually require x and y to
> be stored in memory in big-endian byte order, and will show up that
> way if you dump the memory region to disk.
>
> That particularly means if the hardware happens to be little-endian
> and you want to add x and y together with a 32-bit "add" machine
> instruction, you are requiring the compiler to generate code that
> reads x and y from memory and performs byte swaps before doing the
> addition.

One would hope that the design review would catch this mistake.

> I've never seen a network program do it like that though.  Everyone
> just makes sure that network integers are converted to the host format
> before being stored in memory, and converted back when sent over the
> network.

Which is exactly what a competent Ada team would do, too. No different
from ensuring that the units are sensible: just because the network data
is in feet and pounds is no reason for the system to work internally in
other than metric units.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

"Randy Brukardt" <randy@rrsoftware.com> writes:

> Right. The problem with AI12-0218-1 (besides the fact that no one
> other than the author understands it, and the author has not
> volunteered to help), is that it makes it way too easy to put very
> expensive byte-swapping code into data structures that occur
> everywhere.

Indeed. A project might almost make a rule that types with
representation clauses should only appear in external-interface packages
(and perhaps only in package bodies!)

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Niklas Holsti]

On 18-05-11 01:50 , Randy Brukardt wrote:

> Portable representation clauses are a waste of time, especially as machines
> are inconsistent in the ways that they number bits. Just don't go there.
> (218 solves one such problem, but there are many others.)

The ability to specify Bit_Order for record representations is very 
useful to me, as it allows my programs to be executed both on the target 
HW (big-endian SPARC) and on the development workstations (little-endian 
x86) with exactly the same HW-level input and output.

So thank you, ARG, for this!

There are some gotchas and limitations (most severe is the inability to 
specify the ordering of the components of packed arrays within records) 
but on the whole it works.

My current project has 115 cases of "for ...'Bit_Order use ...".

(And about three cases of having to manually expand packed arrays into 
individual record components because of the inability mentioned above.)

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dmitry A. Kazakov]

On 10/05/2018 22:06, Dan'l Miller wrote:

> LoL!!  Oh, do tell why a representation clause on an IETF packet's header layout is “useless”.

Because the language of representation clauses is too weak to handle 
layouts. One could design and integrate a full blown data-definition 
language (with conditional and looping constructs) into Ada but that 
would be a gigantic waste of time. It just does not make sense. The 
result of this are data types one cannot use effectively. To name a few 
examples from real life:

1. Integer types with reserved bit-patterns for exceptional states. E.g. 
16#FFFF#, 16#FFFE#, 16#FFFD# are not numbers but states of the hardware.

2. Non-contiguous integers. E.g. low-order 16 bits in one word, 
high-order 8 bits in a word two words ahead.

3. Middle-endian numbers.

4. Non-2's complement signed integer numbers

Therefore the programmer must copy them into machine-friendly Ada native 
types anyway. It is simpler to do this right while encoding/decoding the 
packet.

Then there is error handling and sanity checks issues which 
representation clauses do not do. So the programmer will get some 
rubbish in the memory *already* interpreted as a valid value of Ada 
type. Read - the type system breach. He must then re-check everything 
inside by re-scanning the I/O buffer one more time, because X'Valid is 
no help here.

Furthermore, representation clauses require complete packet or its parts 
read into the buffer or written away. Communication, e.g. asynchronous 
I/O just does not work this way. The application must deal with data 
chunks of almost arbitrary length. Representation clauses would require 
additional assembling/disassembling of data into chunks and thus extra 
copying.

Ergo. It works for a few cases on paper. In real life it is simpler, 
faster, safer and cleaner not to use representation clauses at all.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Friday, May 11, 2018 at 4:40:19 AM UTC-5, Niklas Holsti wrote:
> On 18-05-11 01:50 , Randy Brukardt wrote:
> 
> > Portable representation clauses are a waste of time, especially as machines
> > are inconsistent in the ways that they number bits. Just don't go there.
> > (218 solves one such problem, but there are many others.)
> 
> The ability to specify Bit_Order for record representations is very 
> useful to me, as it allows my programs to be executed both on the target 
> HW (big-endian SPARC) and on the development workstations (little-endian 
> x86) with exactly the same HW-level input and output.
> 
> So thank you, ARG, for this!
> 
> There are some gotchas and limitations (most severe is the inability to 
> specify the ordering of the components of packed arrays within records) 
> but on the whole it works.
> 
> My current project has 115 cases of "for ...'Bit_Order use ...".
> 
> (And about three cases of having to manually expand packed arrays into 
> individual record components because of the inability mentioned above.)
> 
> -- 
> Niklas Holsti
> Tidorum Ltd
> niklas holsti tidorum fi
>        .      @       .

Please provide an example of a Bit_Order-based-representation-clause record that is portable between SPARC and IA-32/AMD64 (that is not utilizing the GNAT feature in A12-0218, and completely ignoring the packed-array troublesome case).  There is a perennial debate whether the •proper• usage of Bit_Order on •bug-free• representation clauses in an Ada compiler already solves the entire topic of A12-0218, making moot A12-0218 and GNAT's already-extant feature therein.  Your long 115-cases track-record of success with Bit_Order representation for endianness portability between SPARC and IA-32/AMD64 seems to not only concur, but strongly contains the wisdom necessary to decisively blow A12-0218 out of the water.  Please clearly lead us; please clearly teach us how to utilize Bit_Order properly.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Lucretia]

On Friday, 11 May 2018 08:55:28 UTC+1, Simon Wright  wrote:
> In the body,
> 
>    Big_Endian : constant Boolean
>      := System."=" (System.Default_Bit_Order, System.High_Order_First);
> 

Is it not possible to use derived types with different rep clauses to implement byte swapping on assignment, like the pack/unpack trick can?

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

Simon Wright <simon@pushface.org> writes:

>    type Leap_Indicator is (No_Warning,
>                            Last_Minute_Has_61_Seconds,
>                            Last_Minute_Has_59_Seconds,
>                            Alarm_Condition);
>
>    type Version is range 3 .. 4;
>
>    type Mode is (Reserved,
>                  Symmetric_Active,
>                  Symmetric_Passive,
>                  Client,
>                  Server,
>                  Broadcast,
>                  Reserved_For_NTP_Control_Message,
>                  Reserved_For_Private_Use);
>
>    type Status is record
>       LI : Leap_Indicator;
>       VN : Version;
>       M : Mode;
>    end record;
>
> in native layout (nowadays I'd be a lot more specific about
> sizes).

Actually, I don't think I would: I realise that in that code I was
leaving it up to the compiler to work out the best representation for
use in the vast majority of the program. Just specify enumerations,
ranges and don't overthink it.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

Lucretia <laguest9000@googlemail.com> writes:

> On Friday, 11 May 2018 08:55:28 UTC+1, Simon Wright  wrote:
>> In the body,
>> 
>>    Big_Endian : constant Boolean
>>      := System."=" (System.Default_Bit_Order, System.High_Order_First);
>> 
>
> Is it not possible to use derived types with different rep clauses to
> implement byte swapping on assignment, like the pack/unpack trick can?

Maybe nowadays; not when that code was written.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[AdaMagica]

Am Donnerstag, 10. Mai 2018 19:45:43 UTC+2 schrieb Dan'l Miller:
> The Challenge:
> Without the proposed solution in AI12-0218, I challenge anyone to write an Ada representation clause that is •portable• to both little-endian and big-endian processors for either of the following IETF packet formats that must always be in big-endian network byte order:
> https://en.wikipedia.org/wiki/IPv4#Packet_structure
> https://en.wikipedia.org/wiki/IPv6_packet

I'm not sure this is what you are looking for:

  for IPv4 use record
    Version                at  0 range  0 ..  3;
    IHL                    at  0 range  4 ..  7;
    DSCP                   at  0 range  8 .. 13;
    ECN                    at  0 range 14 .. 15;
    Total_Length           at  0 range 16 .. 31;
    Identification         at  4 range  0 .. 15;
    Flags                  at  4 range 16 .. 18;
    Fragment_Offset        at  4 range 19 .. 31;
    Time_to_Live           at  8 range  0 ..  7;
    Protocol               at  8 range  8 .. 15;
    Header_Checksum        at  8 range 16 .. 31;
    Source_IP_Address      at 12 range  0 .. 31;
    Destination_IP_Address at 16 range  0 .. 31;
  end record;
  for IPv4'Bit_Order use System.High_Order_First;

Of course you have to byte-swap after a transfer.

GNAT reports this upne compilation:
reverse bit order in machine scalar of length 32
little-endian range for component "Version" is 28 .. 31
reverse bit order in machine scalar of length 32
little-endian range for component "IHL" is 24 .. 27
reverse bit order in machine scalar of length 32
little-endian range for component "DSCP" is 18 .. 23
reverse bit order in machine scalar of length 32
little-endian range for component "ECN" is 16 .. 17
reverse bit order in machine scalar of length 32
little-endian range for component "Total_Length" is 0 .. 15
reverse bit order in machine scalar of length 32
little-endian range for component "Identification" is 16 .. 31
reverse bit order in machine scalar of length 32
little-endian range for component "Flags" is 13 .. 15
reverse bit order in machine scalar of length 32
little-endian range for component "Fragment_Offset" is 0 .. 12
reverse bit order in machine scalar of length 32
little-endian range for component "Time_to_Live" is 24 .. 31
reverse bit order in machine scalar of length 32
little-endian range for component "Protocol" is 16 .. 23
reverse bit order in machine scalar of length 32
little-endian range for component "Header_Checksum" is 0 .. 15
reverse bit order in machine scalar of length 32
little-endian range for component "Source_IP_Address" is 0 .. 31
reverse bit order in machine scalar of length 32
little-endian range for component "Destination_IP_Address" is 0 .. 31

You might want to look at:
https://en.wikibooks.org/wiki/Ada_Programming/Attributes/%27Bit_Order

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Jeffrey R. Carter]

On 05/11/2018 03:49 PM, Simon Wright wrote:
> Lucretia <laguest9000@googlemail.com> writes:
> 
>> On Friday, 11 May 2018 08:55:28 UTC+1, Simon Wright  wrote:
>>> In the body,
>>>
>>>     Big_Endian : constant Boolean
>>>       := System."=" (System.Default_Bit_Order, System.High_Order_First);
>>>
>>
>> Is it not possible to use derived types with different rep clauses to
>> implement byte swapping on assignment, like the pack/unpack trick can?
> 
> Maybe nowadays; not when that code was written.

You can, and have been able to since Ada 83.

-- 
Jeff Carter
"C++: The power, elegance and simplicity of a hand grenade."
Ole-Hjalmar Kristensen
90

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

"Jeffrey R. Carter" <spam.jrcarter.not@spam.not.acm.org> writes:

> On 05/11/2018 03:49 PM, Simon Wright wrote:
>> Lucretia <laguest9000@googlemail.com> writes:
>>
>>> On Friday, 11 May 2018 08:55:28 UTC+1, Simon Wright  wrote:
>>>> In the body,
>>>>
>>>>     Big_Endian : constant Boolean
>>>>       := System."=" (System.Default_Bit_Order, System.High_Order_First);
>>>>
>>>
>>> Is it not possible to use derived types with different rep clauses to
>>> implement byte swapping on assignment, like the pack/unpack trick can?
>>
>> Maybe nowadays; not when that code was written.
>
> You can, and have been able to since Ada 83.

I must be missing something.

   type T is record
      J : Integer;
   end T;

   type BE_T is new T;
   for BE_T use record
      ?????
   end record;

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Jeffrey R. Carter]

On 05/11/2018 06:48 PM, Simon Wright wrote:
> "Jeffrey R. Carter" <spam.jrcarter.not@spam.not.acm.org> writes:
> 
>> On 05/11/2018 03:49 PM, Simon Wright wrote:
>>> Lucretia <laguest9000@googlemail.com> writes:
>>>
>>>> On Friday, 11 May 2018 08:55:28 UTC+1, Simon Wright  wrote:
>>>>> In the body,
>>>>>
>>>>>      Big_Endian : constant Boolean
>>>>>        := System."=" (System.Default_Bit_Order, System.High_Order_First);
>>>>>
>>>>
>>>> Is it not possible to use derived types with different rep clauses to
>>>> implement byte swapping on assignment, like the pack/unpack trick can?
>>>
>>> Maybe nowadays; not when that code was written.
>>
>> You can, and have been able to since Ada 83.
> 
> I must be missing something.
> 
>     type T is record
>        J : Integer;
>     end T;
> 
>     type BE_T is new T;
>     for BE_T use record
>        ?????
>     end record;

In Ada 83, it would have been

with Text_IO;
with Unchecked_Conversion;
procedure Byte_Swap is
    type I16 is range -(2 ** 15) .. 2 ** 15 - 1;
    for I16'Size use 16;

    type Byte is range -128 .. 127;
    for Byte'Size use 8;

    type LE16 is record -- We'll assume this is the native order
       LSB : Byte;
       MSB : Byte;
    end record;
    for LE16'Size use 16;
    for LE16 use record
       LSB at 0 range 0 .. 7;
       MSB at 1 range 0 .. 7;
    end record;

    type BE16 is new LE16;
    for BE16'Size use 16;
    for BE16 use record
       LSB at 1 range 0 .. 7;
       MSB at 0 range 0 .. 7;
    end record;

    function To_LE  is new Unchecked_Conversion (Source => I16,  Target => LE16);
    function To_I16 is new Unchecked_Conversion (Source => BE16, Target => I16);

    A : I16 := 299;
    B : I16 := To_I16 (BE16 (To_LE (A) ) ); -- Bytes swapped
begin -- Byte_Swap
    Text_IO.Put_Line (Item => I16'Image (B) );
end Byte_Swap;

Because BE16 is derived from LE16, there are conversions between them that do a 
change of representation; in this case, byte swapping.

$ gnatmake -gnat83 -gnatan -gnato2 -O2 -fstack-check byte_swap.adb
$ ./byte_swap
  11009

-- 
Jeff Carter
"Since I strongly believe that overpopulation is by
far the greatest problem in the world, this [Soylent
Green] would be my only message movie."
Charleton Heston
123

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Niklas Holsti]

On 18-05-11 14:40 , Dan'l Miller wrote:
> On Friday, May 11, 2018 at 4:40:19 AM UTC-5, Niklas Holsti wrote:
 >>
>> The ability to specify Bit_Order for record representations is very
>> useful to me, as it allows my programs to be executed both on the target
>> HW (big-endian SPARC) and on the development workstations (little-endian
>> x86) with exactly the same HW-level input and output.
>
> Please provide an example of a Bit_Order-based-representation-clause
> record that is portable between SPARC and IA-32/AMD64 (that is not
> utilizing the GNAT feature in A12-0218, and completely ignoring the
> packed-array troublesome case).  There is a perennial debate whether
> the •proper• usage of Bit_Order on •bug-free• representation clauses
> in an Ada compiler already solves the entire topic of A12-0218,
> making moot A12-0218 and GNAT's already-extant feature therein.

Bit_Order is not Byte_Order; it does not address the byte-endianness 
problem. Bit_Order merely specifies the order in which bits are numbered 
in a representation clause.

I use record types with specified Bit_Order to access structured 32-bit 
HW/peripheral control registers (real registers on the SPARC, just RAM 
variables in the PC). The record types are also 32 bits in size, and the 
registers are written and read as 32-bit words, not as sequences of 
8-bit storage units.

My record representation clauses always say "at 0 range ..." and use bit 
numbers between 0 and 31, so there is never a reference to a ´particular 
storage unit (octet) within the record: never "at 1 range ..." or some 
other non-zero storage-unit offset.

By using the same Bit_Order on the SPARC and the PC, it is ensured that 
the 32-bit record values are identical on the SPARC and on the PC, for 
example when unchecked-converted to 32-bit unsigned integers and 
displayed in hex.

The octet order in memory is still different on the SPARC and the PC, 
but that is irrelevant to me, because the registers are always accessed 
as 32-bit words. (On the SPARC, the octet order is not really defined, 
because the HW control registers must be accessed as whole 32-bit words 
and cannot be accessed in smaller units.)

So my successful use of Bit_Order is not relevant to ai12-0218, which is 
about the order of storage units in memory, not about the order of 
numbering bits in a word in record representation clauses.

ai12-0218 is relevant to records that are accessed _both_ as a sequence 
of storage units (for example, for stream I/O) and as a 
multi-storage-unit record. This is not my case; for stream I/O I have 
the same approach as Dmitry (I believe): decode/encode octet by octet.

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

"Jeffrey R. Carter" <spam.jrcarter.not@spam.not.acm.org> writes:

> On 05/11/2018 06:48 PM, Simon Wright wrote:
>> "Jeffrey R. Carter" <spam.jrcarter.not@spam.not.acm.org> writes:
>>
>>> On 05/11/2018 03:49 PM, Simon Wright wrote:
>>>> Lucretia <laguest9000@googlemail.com> writes:
>>>>
>>>>> Is it not possible to use derived types with different rep clauses
>>>>> to implement byte swapping on assignment, like the pack/unpack
>>>>> trick can?
>>>>
>>>> Maybe nowadays; not when that code was written.
>>>
>>> You can, and have been able to since Ada 83.
>>
>> I must be missing something.
>>
>>     type T is record
>>        J : Integer;
>>     end T;
>>
>>     type BE_T is new T;
>>     for BE_T use record
>>        ?????
>>     end record;
>
> In Ada 83, it would have been
>
> with Text_IO;
> with Unchecked_Conversion;
> procedure Byte_Swap is
>    type I16 is range -(2 ** 15) .. 2 ** 15 - 1;
>    for I16'Size use 16;
>
>    type Byte is range -128 .. 127;
>    for Byte'Size use 8;
>
>    type LE16 is record -- We'll assume this is the native order
>       LSB : Byte;
>       MSB : Byte;
>    end record;
>    for LE16'Size use 16;
>    for LE16 use record
>       LSB at 0 range 0 .. 7;
>       MSB at 1 range 0 .. 7;
>    end record;
>
>    type BE16 is new LE16;
>    for BE16'Size use 16;
>    for BE16 use record
>       LSB at 1 range 0 .. 7;
>       MSB at 0 range 0 .. 7;
>    end record;

I had somewhere got the idea that the parent type couldn't have
representation clauses.

But I see that Ada83 LRM 13.6 says

    "Hence, if an alternative representation is needed, it is necessary
    to declare a second type, derived from the first, and to specify a
    different representation for the second type."

whereas Ada2012 says

    "To convert a record from one representation to another, two record
    types with a common ancestor type need to be declared, with no
    inherited subprograms."

which seems to require _three_ types.

But in any case your code doesn't answer my question: what
representation trick could convert between

   type T is record
      J : Integer;
   end T;

and

   type BE_T is new T;
   for BE_T use record
       ?????
   end record;

One approach (the C one) would be to use htonl(), of course.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Jeffrey R. Carter]

On 05/11/2018 11:39 PM, Simon Wright wrote:
> 
> whereas Ada2012 says
> 
>      "To convert a record from one representation to another, two record
>      types with a common ancestor type need to be declared, with no
>      inherited subprograms."
> 
> which seems to require _three_ types.

The example in 13.6 shows this with only 2 types. This is because a type is 
defined to be its own ancestor (ARM 3.4.1).

> But in any case your code doesn't answer my question: what
> representation trick could convert between
> 
>     type T is record
>        J : Integer;
>     end T;
> 
> and
> 
>     type BE_T is new T;
>     for BE_T use record
>         ?????
>     end record;

I was showing how to use records with representations to get the compiler to do 
byte swapping for you. There's no way to use them to change the byte order of an 
integer type directly.

-- 
Jeff Carter
"Since I strongly believe that overpopulation is by
far the greatest problem in the world, this [Soylent
Green] would be my only message movie."
Charleton Heston
123

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Dan'l Miller" <optikos@verizon.net> wrote in message 
news:c3a4f872-7e24-4d66-a425-776a1aa063f5@googlegroups.com...
On Thursday, May 10, 2018 at 5:40:49 PM UTC-5, Randy Brukardt wrote:
>> . If you just implement it in the front-end using code
>> equivalent to unchecked_conversions, you're forcing all of the operations
>> into memory
>
>Wait, what is the actual extant-in-reality use-case for byte-swapping 
>between
>registers (within the same processor!) of (supposedly-)heterogenous 
>endianness?
>(Not even the PDP-11's mixed endianness did that!)  Does any processor 
>exhibit
>such odd endianness within its own processor('s registers)?  Methinks this 
>is pure
>bicycle-shedding without any basis in reality.

You'd need that to implement AI12-0218-1, or a complete replacement of the 
existing code for handling representation clauses. For Janus/Ada, at least, 
we read the memory into a register, then do various masking an shifting 
operations. One would want to do byte-swapping the same way.

I don't know of any way on the x86 processor of reading a byte-swapped 
32-bit integer into a register so it can be used. If you don't do the byte 
swapping in a register after reading, you would have to do it by doing a-la 
Unchecked_Conversion, copying each byte from one memory location to another 
temporary location in the reverse order, *then* reading the now-swapped 
value into a register. That would be extremely expensive in Janus/Ada, since 
we don't have any way to allocate temporary memory in many places where such 
reads would occur (the fall back would be heap allocation/deallocation!).

Besides, the x86 has instructions like Xchg AL, AH which do byte swapping in 
registers. One would rather use those rather than do extra memory 
operations.

Thus, I conclude that back-end changes would be needed.

I don't know the situation on other processors vis-a-vis byte swapping, but 
one can always generate the best possible code if the back-end is aware of 
the need. Emulating it in the front-end can often end up sub-optimal.

>> don't allow any hardware support to get used.
>
>Hmmm, that one is somewhat more compelling if the processor has 
>byte-swapping instructions.

If such instructions aren't available, you have either the choice of 
Unchecked_Conversion (memory to memory swapping, followed by usual reading) 
or doing it in a pair of registers with shifting/masking. I'd rather do the 
latter, since the easiest way to speed up a program is to reduce the amount 
of memory use. (The correlary is the reverse is also true. :-)

                            Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Dan'l Miller" <optikos@verizon.net> wrote in message 
news:5c9b9f90-884f-4de7-8663-d39a67949f4f@googlegroups.com...
>* on a per-processor, per-toolchain basis, selected via the conditional 
>compilation
>via the C preprocessor.  This selection (out of nonportable categorization) 
>is
>what accomplishes the portability.  Ada has no equivalent other than
>AI32-0218's/GNAT's solution.

The best way to do this in Ada is with different package bodies (and 
sometimes specs) for each target. That's how Janus/Ada is designed, and it 
works great. I know the GNAT project system even has facilities to make this 
happen automatically (by selecting the unit to compile based on a version 
id).

The problem is that existing version control systems cannot handle such 
designs properly. That was definitely true in the late 1980's (so I designed 
a wrapper around our version control to deal with this), and I haven't seen 
any that deal with it properly to date. (The main issue being that when a 
bug is fixed in one such package, you want a notification to check if the 
same is needed for the other versions. No version control that I'm aware of 
can handle this - they all seem focused on merging development for a single 
end-product.)

Shortcomings in version control are way outside of anything that the 
language can control. And using a sub-optimal design because ancillary tools 
are broken seems to be letting the tail wag the dog.

                 Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Paul Rubin" <no.email@nospam.invalid> wrote in message 
news:87po23yusb.fsf@nightsong.com...
>...
> That particularly means if the hardware happens to be little-endian and
> you want to add x and y together with a 32-bit "add" machine
> instruction, you are requiring the compiler to generate code that reads
> x and y from memory and performs byte swaps before doing the addition.

Yes, that's precisely what AI12-0218-1 is intended to require, and  also 
explains why there is so much resistance to it. Ada to-date has adopted 
rules that do not require byte-swapping although it does require supporting 
the non-native big-endianness.

                           Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Dan'l Miller" <optikos@verizon.net> wrote in message 
news:3c2c787e-227e-4066-b397-47b23bad3c43@googlegroups.com...
...
>> Normally a network programmer wouldn't do that.
>
>My point is normally a network programmer does exactly what you
>describe ..in C.., not Ada, because the engineer years or decades ago
>established that C's macros and pointer arithmetic can do what Ada's
>record representation clauses cannot, and C's macros and pointer
>arithmetic can do what is disdained as uncouth to be performed by
> Ada's various accesses: byte manipulations.

A pretty silly programmer, if you ask me. An experienced Ada programmer 
wouldn't use macros and pointer arithmetic because the job can be trivially 
done with Unchecked_Conversion. (Jeff Carter showed how in another message). 
I've done so several times in my career, and it works fine and even isn't 
expensive if implemented correctly. (Unfortunately, GNAT doesn't take 
advantage of the permissions to share, so it does a lot more operations than 
necessary - or at least it did a few years back.)

Using a representation clause to swap bytes might slightly simplify some 
code, but it is hugely expensive to (non-GNAT) implementations, prone to 
misuse, and it is very rarely used, so the cost-benefit ratio is out of 
whack.

                                      Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Paul Rubin]

"Randy Brukardt" <randy@rrsoftware.com> writes:
> I don't know of any way on the x86 processor of reading a byte-swapped 
> 32-bit integer into a register so it can be used.

Load it into a register, then BSWAP.  This could be made into a compiler
intrinsic if necessary.  GCC has __builtin_bswap16, __builtin_bswap32,
and __builtin_bswap64 for this purpose.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

"Jeffrey R. Carter" <spam.jrcarter.not@spam.not.acm.org> writes:

> On 05/11/2018 11:39 PM, Simon Wright wrote:
>>
>> whereas Ada2012 says
>>
>>      "To convert a record from one representation to another, two
>>      record types with a common ancestor type need to be declared,
>>      with no inherited subprograms."
>>
>> which seems to require _three_ types.
>
> The example in 13.6 shows this with only 2 types. This is because a
> type is defined to be its own ancestor (ARM 3.4.1).

I saw the example. I stand by my statement that it's confusing.

Perhaps there needs to be a way to mark a language usage as
domain-specific rather than natural; in English, I am not my own
ancestor (or descendant).

>> But in any case your code doesn't answer my question: what
>> representation trick could convert between
>>
>>     type T is record
>>        J : Integer;
>>     end T;
>>
>> and
>>
>>     type BE_T is new T;
>>     for BE_T use record
>>         ?????
>>     end record;
>
> I was showing how to use records with representations to get the
> compiler to do byte swapping for you. There's no way to use them to
> change the byte order of an integer type directly.

Not quite the same, but I think I showed upthread a conversion that does
this:

   function To_Fixed_32_16 (S : Four_Byte_Slice) return Fixed_32_16 is
      function Convert is new Ada.Unchecked_Conversion (Four_Byte_Slice,
                                                        Fixed_32_16);
   begin
      if Big_Endian then
         return Convert (S);
      else
         return Convert ((1 => S (4),
                          2 => S (3),
                          3 => S (2),
                          4 => S (1)));
      end if;
   end To_Fixed_32_16;

Of course, in C you'd use ntohl(), but we'd have to wrap something like
the above in a generic.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Jeffrey R. Carter]

On 05/12/2018 09:08 AM, Simon Wright wrote:
> 
> I saw the example. I stand by my statement that it's confusing.

I agree that the terminology is not intuitive.

> 
> Not quite the same, but I think I showed upthread a conversion that does
> this:
> 
>     function To_Fixed_32_16 (S : Four_Byte_Slice) return Fixed_32_16 is
>        function Convert is new Ada.Unchecked_Conversion (Four_Byte_Slice,
>                                                          Fixed_32_16);
>     begin
>        if Big_Endian then
>           return Convert (S);
>        else
>           return Convert ((1 => S (4),
>                            2 => S (3),
>                            3 => S (2),
>                            4 => S (1)));
>        end if;
>     end To_Fixed_32_16;

For the case where it does swapping, I don't see that this is very different 
from my version, at least if the intention is to go from Integer to Integer, 
which is what I thought you were asking. I used change-of-representation to get 
the swapping and this does the swapping itself, and this hides one of the 
unchecked conversions. My version was intended to allow conversion in both 
directions, while this seems to only go one way. I used Ada 83 to show that such 
change-of-representation was in the language then, and you're using a more 
recent version.

-- 
Jeff Carter
"Death awaits you all, with nasty, big, pointy teeth!"
Monty Python & the Holy Grail
20

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Björn Lundin]

On 2018-05-12 00:12, Randy Brukardt wrote:
> 
> The best way to do this in Ada is with different package bodies (and 
> sometimes specs) for each target. 

We do this too.


> The problem is that existing version control systems cannot handle such 
> designs properly. 

To get around that we have subdirectories matching the target

-w32_x86
-lnx_x64
-aix_ppc

each of these directories would contain the native body and spec for the
platform. Each directory and file is version_controlled.

The build system (first homebrew, now gpr) only includes the directory
matching the correct target. So when building on AIX, the compiler never
sees the directories lnx_x64 not w32_x86.

Each platform tests on an environment variable we set that has one of
the above 3 values.

So in essence we limit the files the compiler sees to the ones
belonging to the current platform.


-- 
--
Björn

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

Björn Lundin <b.f.lundin@gmail.com> writes:

> On 2018-05-12 00:12, Randy Brukardt wrote:
>> 
>> The best way to do this in Ada is with different package bodies (and 
>> sometimes specs) for each target. 
>
> We do this too.
>
>> The problem is that existing version control systems cannot handle such 
>> designs properly. 
>
> To get around that we have subdirectories matching the target
>
> -w32_x86
> -lnx_x64
> -aix_ppc
>
> each of these directories would contain the native body and spec for the
> platform. Each directory and file is version_controlled.
>
> The build system (first homebrew, now gpr) only includes the directory
> matching the correct target. So when building on AIX, the compiler never
> sees the directories lnx_x64 not w32_x86.
>
> Each platform tests on an environment variable we set that has one of
> the above 3 values.
>
> So in essence we limit the files the compiler sees to the ones
> belonging to the current platform.

I have to say that my experience at Cortex GNAT RTS[1] is similar to
Randy's.

At the moment I have 6 usable branches; the gcc7* and gcc8* ones are
live:

   gcc6 (FSF GCC 6, GNAT GPL 2016)
   gcc7 (FSF GCC 7)
   gcc-finalization (FSF GCC 7 + finalization)
   gcc8 (FSF GCC 8)
   gcc-finalization (FSF GCC 8 + finalization)
   gnat-gpl-2017

The reason for the distinction is that the interface between the
compiler and the RTS changes; between gcc7 and gcc8 10 files changed,
and between gcc8 and gcc8-finalization 30 files changed (not including
tests).

The way I've "managed" this is to choose a branch to develop a change
on, implement the changes, commit, then checkout the other branch that
the development applies to and cherry-pick the commit(s) (i.e the
changes) from the first branch. This is error-prone, to say the
least. It'd be quite hard to do at all on my previously preferred DVCS,
Mercurial.

Your remarks about using directories for this are very interesting, I'll
be looking into them. I suspect the problem of "change a spec; apply
matching changes to all the variant package bodies" will remain, though.

[1] https://github.com/simonjwright/cortex-gnat-rts

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Björn Lundin]

On 2018-05-12 15:08, Simon Wright wrote:
> Your remarks about using directories for this are very interesting, I'll
> be looking into them.

It is a way of getting all files into version handling.

> I suspect the problem of "change a spec; apply
> matching changes to all the variant package bodies" will remain, though.

unfortuantly yes.

-- 
--
Björn

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Simon Wright" <simon@pushface.org> wrote in message 
news:lypo21ic5j.fsf@pushface.org...
> "Jeffrey R. Carter" <spam.jrcarter.not@spam.not.acm.org> writes:
>
>> On 05/11/2018 11:39 PM, Simon Wright wrote:
>>>
>>> whereas Ada2012 says
>>>
>>>      "To convert a record from one representation to another, two
>>>      record types with a common ancestor type need to be declared,
>>>      with no inherited subprograms."
>>>
>>> which seems to require _three_ types.
>>
>> The example in 13.6 shows this with only 2 types. This is because a
>> type is defined to be its own ancestor (ARM 3.4.1).
>
> I saw the example. I stand by my statement that it's confusing.
>
> Perhaps there needs to be a way to mark a language usage as
> domain-specific rather than natural; in English, I am not my own
> ancestor (or descendant).

That's easy compared to "part" (also a technical term, also includes the 
entire item, also gets confused all the time, even by people who know 
better).

If a word in the RM is more than 6 characters or so, assume it is a 
technical term and look it up; only assume the English meaning if it isn't 
found when looking it up. (And you still get caught by "part", "named", and 
others.) We try hard to make the technical terms at least someone intuitive, 
but it's not possible to do that completely (else a technical term would not 
be needed).

Modern ISO standards are supposed to have a glossary of all defined 
technical terms; adding one of those for Ada would take a years work 
(because a lot of the definitions are folded into other text), and there are 
a LOT of them.

                      Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Paul Rubin" <no.email@nospam.invalid> wrote in message 
news:87lgcphdxq.fsf@nightsong.com...
> "Randy Brukardt" <randy@rrsoftware.com> writes:
>> I don't know of any way on the x86 processor of reading a byte-swapped
>> 32-bit integer into a register so it can be used.
>
> Load it into a register, then BSWAP.  This could be made into a compiler
> intrinsic if necessary.  GCC has __builtin_bswap16, __builtin_bswap32,
> and __builtin_bswap64 for this purpose.

What's "BSWAP"? It's not an instruction I'm familar with on x86. 
(Admittedly, I don't usually use newer instructions because of the need to 
have some sort of alternative to run on older processors.)

In any case, one has to build such support into the back-end. (Compiler 
front-ends are almost always target-independent.) "intrinsics" are just a 
fancy way of identifying things that are directly supported in the back-end.

                          Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Paul Rubin]

> What's "BSWAP"? It's not an instruction I'm familar with on x86. 

http://www.felixcloutier.com/x86/BSWAP.html

> (Admittedly, I don't usually use newer instructions because of the
> need to have some sort of alternative to run on older processors.)

It was introduced with the 486 so I think it runs on almost any x86 that
anyone is currently using.

> In any case, one has to build such support into the back-end.

In GCC you can write intrinsics by using pragmas that include the
desired assembly code.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dennis Lee Bieber]

On Mon, 14 May 2018 17:54:52 -0500, "Randy Brukardt" <randy@rrsoftware.com>
declaimed the following:


>What's "BSWAP"? It's not an instruction I'm familar with on x86. 
>(Admittedly, I don't usually use newer instructions because of the need to 
>have some sort of alternative to run on older processors.)
>
	Really old <G>

https://en.wikipedia.org/wiki/X86_instruction_listings#Added_with_80486



-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
	wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/ 

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Paul Rubin" <no.email@nospam.invalid> wrote in message 
news:87lgcld9yu.fsf@nightsong.com...
>> What's "BSWAP"? It's not an instruction I'm familar with on x86.
>
> http://www.felixcloutier.com/x86/BSWAP.html
>
>> (Admittedly, I don't usually use newer instructions because of the
>> need to have some sort of alternative to run on older processors.)
>
> It was introduced with the 486 so I think it runs on almost any x86 that
> anyone is currently using.

My bad; it definitely is in the printed Pentium manual on my shelf. Never 
had a reason to use it, I guess.

>> In any case, one has to build such support into the back-end.
>
> In GCC you can write intrinsics by using pragmas that include the
> desired assembly code.

You can write assembly code in Janus/Ada, but that would be terrible quality 
as that doesn't tie into the register manager and thus has to use memory to 
get operands in and out.

                 Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Thursday, May 10, 2018 at 12:45:43 PM UTC-5, Dan'l Miller wrote:
> It seems that 2 major use-cases of AI12-0218 are getting short shrift.
> 
> http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ai12s/ai12-0218-1.txt?rev=1.3&raw=N

Why is Norman Cohen's solution in the PDF at the URL below not the way for the ARG to merely dismiss AI12-0218 as not-a-problem that needs to be solved, due to already having a solution ever since Record Representation Clause was introduced.

http://www.ada-auth.org/ai-files/grab_bag/bitorder.pdf

On the final page 9, Norman Cohen seems to leave the solution hanging on one final point though:

How to pick the proper Record Representation Clause •portably• in standard Ada?  Hint:  there isn't a standardized way to do that selection of alternative Record Representation Clauses among, say, alternative child packages.

Also, Norman Cohen seems to misuse the word proposal in that document.  He isn't proposing any change to Ada.  He is merely educating, isn't he?  I guess that he is proposing that we learn.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[AdaMagica]

Am Samstag, 26. Mai 2018 18:15:30 UTC+2 schrieb Dan'l Miller:
> On Thursday, May 10, 2018 at 12:45:43 PM UTC-5, Dan'l Miller wrote:
> > It seems that 2 major use-cases of AI12-0218 are getting short shrift.
> > 
> > http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ai12s/ai12-0218-1.txt?rev=1.3&raw=N
> 
> Why is Norman Cohen's solution in the PDF at the URL below not the way for the ARG to merely dismiss AI12-0218 as not-a-problem that needs to be solved, due to already having a solution ever since Record Representation Clause was introduced.
> 
> http://www.ada-auth.org/ai-files/grab_bag/bitorder.pdf

This is what has been implemented in Ada 2005. See my post of May 11.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Saturday, May 26, 2018 at 2:02:12 PM UTC-5, AdaMagica wrote:
> Am Samstag, 26. Mai 2018 18:15:30 UTC+2 schrieb Dan'l Miller:
> > On Thursday, May 10, 2018 at 12:45:43 PM UTC-5, Dan'l Miller wrote:
> > > It seems that 2 major use-cases of AI12-0218 are getting short shrift.
> > > 
> > > http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ai12s/ai12-0218-1.txt?rev=1.3&raw=N
> > 
> > Why is Norman Cohen's solution in the PDF at the URL below not the way for the ARG to merely dismiss AI12-0218 as not-a-problem that needs to be solved, due to already having a solution ever since Record Representation Clause was introduced.
> > 
> > http://www.ada-auth.org/ai-files/grab_bag/bitorder.pdf
> 
> This is what has been implemented in Ada 2005. See my post of May 11.

Yes, I know.  It seems that AI12-0218 and Scalar_Storage_Order are focused on the wrong topic (i.e., the topic of a 2nd solution to the endian byte-swapping).  A replacement to AI12-0218 should be focused on •portably• (among all Ada compilers) choosing between the 2 alternative byte-swapping Record Representation Clauses on different targets.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[AdaMagica]

Am Samstag, 26. Mai 2018 23:01:28 UTC+2 schrieb Dan'l Miller:
> On Saturday, May 26, 2018 at 2:02:12 PM UTC-5, AdaMagica wrote:
> > Am Samstag, 26. Mai 2018 18:15:30 UTC+2 schrieb Dan'l Miller:
> > > Why is Norman Cohen's solution in the PDF at the URL below not the way for the ARG to merely dismiss AI12-0218 as not-a-problem that needs to be solved, due to already having a solution ever since Record Representation Clause was introduced.
> > >

If you look at the last post in this AI, nobody is willing to work on it. And if I remember correctly, Randy once said that nobody except the author of the AI understands it.

> > > http://www.ada-auth.org/ai-files/grab_bag/bitorder.pdf
> > 
> > This is what has been implemented in Ada 2005. See my post of May 11.
> 
> Yes, I know.  It seems that AI12-0218 and Scalar_Storage_Order are focused on the wrong topic (i.e., the topic of a 2nd solution to the endian byte-swapping).  A replacement to AI12-0218 should be focused on •portably• (among all Ada compilers) choosing between the 2 alternative byte-swapping Record Representation Clauses on different targets.

I do not understand. As far as I can tell, the Ada 2005 solution with its machine scalars is portable.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

AdaMagica <christ-usch.grein@t-online.de> writes:

> Am Samstag, 26. Mai 2018 23:01:28 UTC+2 schrieb Dan'l Miller:
>> On Saturday, May 26, 2018 at 2:02:12 PM UTC-5, AdaMagica wrote:
>> > Am Samstag, 26. Mai 2018 18:15:30 UTC+2 schrieb Dan'l Miller:
>> > > Why is Norman Cohen's solution in the PDF at the URL below not
>> > > the way for the ARG to merely dismiss AI12-0218 as not-a-problem
>> > > that needs to be solved, due to already having a solution ever
>> > > since Record Representation Clause was introduced.
>> > >
>
> If you look at the last post in this AI, nobody is willing to work on
> it. And if I remember correctly, Randy once said that nobody except
> the author of the AI understands it.
>
>> > > http://www.ada-auth.org/ai-files/grab_bag/bitorder.pdf
>> >
>> > This is what has been implemented in Ada 2005. See my post of May 11.

I just don't see this in AALRM 2012, but 13.5.{2,3} are fairly
opaque. The paper is opaque, too.

AI95-00133 contains extensive references to the paper, but important
things like

   "The interpretation of component_clauses in the nondefault bit order
   is based on machine scalars, which are chunks of storage that can be
   natively loaded and stored by the machine. All of the
   component_clauses at a given offset are considered to be part of the
   same machine scalar, and the first_bit and last_bit are interpreted
   as bit offsets within that machine scalar. This makes it possible to
   write endian-independent record_representation_clauses."

only appear in the AI, not the ALRM.

And I think I'm right in saying that the size of the machine scalar
starting at a given offset is determined by the largest last_bit of any
of the components starting at that offset (so you'd better have a filler
component extending into the top byte if none of the actual components
do).

>> Yes, I know.  It seems that AI12-0218 and Scalar_Storage_Order are
>> focused on the wrong topic (i.e., the topic of a 2nd solution to the
>> endian byte-swapping).  A replacement to AI12-0218 should be focused
>> on •portably• (among all Ada compilers) choosing between the 2
>> alternative byte-swapping Record Representation Clauses on different
>> targets.
>
> I do not understand. As far as I can tell, the Ada 2005 solution with
> its machine scalars is portable.

But it doesn't do byte swapping.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Sunday, May 27, 2018 at 9:58:57 AM UTC-5, AdaMagica wrote:
> Am Samstag, 26. Mai 2018 23:01:28 UTC+2 schrieb Dan'l Miller:
> > On Saturday, May 26, 2018 at 2:02:12 PM UTC-5, AdaMagica wrote:
> > > Am Samstag, 26. Mai 2018 18:15:30 UTC+2 schrieb Dan'l Miller:
> > > > Why is Norman Cohen's solution in the PDF at the URL below not the way for the ARG to merely dismiss AI12-0218 as not-a-problem that needs to be solved, due to already having a solution ever since Record Representation Clause was introduced.
> > > >
> 
> If you look at the last post in this AI, nobody is willing to work on it. And if I remember correctly, Randy once said that nobody except the author of the AI understands it.

There is a chance that the ARG is slow-walking AI12-0218 because of the political difficulty to tell AdaCore that there already existed a solution to the problem as of Ada2005, and that Scalar_Storage_Order was the 2nd solution to the problem, not the first.

Given that the 1st solution already exists in standard Ada2005 (except for the finishing-touch topic below) and that the Scalar_Storage_Order obviously* is nontrivial to implement, I am baffled why the ARG doesn't definitively reject or kill off AI12-0218 with stronger language.

* by both the AI's expansive touch-lots-of-stuff content and Randy's analogous that-would-wreak-lots-of-havoc-in-Janus-Ada's-source-code general response

> > > > http://www.ada-auth.org/ai-files/grab_bag/bitorder.pdf
> > > 
> > > This is what has been implemented in Ada 2005. See my post of May 11.
> > 
> > Yes, I know.  It seems that AI12-0218 and Scalar_Storage_Order are focused on the wrong topic (i.e., the topic of a 2nd solution to the endian byte-swapping).  A replacement to AI12-0218 should be focused on •portably• (among all Ada compilers) choosing between the 2 alternative byte-swapping Record Representation Clauses on different targets.
> 
> I do not understand. As far as I can tell, the Ada 2005 solution with its machine scalars is portable.

Yes, the 2 alternative Record Representation Clauses themselves are perfectly portable (because they are in fact syntax in ISO standard Ada).

What is not portable (due to the •lack• of being syntax in ISO standard Ada) is the build-time cleverness to choose one child package (containing one of the two) versus another child package (containing the other of the two) on a per-target/per-ISA basis.  That language-undefined** selectivity is (needlessly) peculiar** to each Ada compiler's build tool.  I am pretty sure that,—if Jean Ichbiah were alive today to see this state of affairs on AI12-0218 vis a vis Norman Cohen's use-3-existing-Ada-features-together-as-they-are-already-designed pedagogical paper,—he'd say that the choice of child package to elaborate/use for this target should be overt Ada-language syntax.  I'm pretty sure that he say that it should not be some nonportable gotchya afterthought in gprbuild or other outside-of-Ada syntax/toolset that is peculiar (differently!) to each Ada toolchain.

** in roughly the same “undefined behavior” ballpark conceptually as ISO standard C++'s historical lack of control on elaboration order of libraries/objects for what C++ calls static instances.  Usually Ada cares passionately about eliminating target-specific undefined-in-the-language behavior, but, in this case of portably choosing between alternative child packages based on target/ISA via standard Ada syntax (pragma or aspect or otherwise), not so much.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Simon Wright" <simon@pushface.org> wrote in message 
news:ly4litj7qx.fsf@pushface.org...
> AdaMagica <christ-usch.grein@t-online.de> writes:
>
>> Am Samstag, 26. Mai 2018 23:01:28 UTC+2 schrieb Dan'l Miller:
>>> On Saturday, May 26, 2018 at 2:02:12 PM UTC-5, AdaMagica wrote:
>>> > Am Samstag, 26. Mai 2018 18:15:30 UTC+2 schrieb Dan'l Miller:
>>> > > Why is Norman Cohen's solution in the PDF at the URL below not
>>> > > the way for the ARG to merely dismiss AI12-0218 as not-a-problem
>>> > > that needs to be solved, due to already having a solution ever
>>> > > since Record Representation Clause was introduced.
>>> > >
>>
>> If you look at the last post in this AI, nobody is willing to work on
>> it. And if I remember correctly, Randy once said that nobody except
>> the author of the AI understands it.
>>
>>> > > http://www.ada-auth.org/ai-files/grab_bag/bitorder.pdf
>>> >
>>> > This is what has been implemented in Ada 2005. See my post of May 11.
>
> I just don't see this in AALRM 2012, but 13.5.{2,3} are fairly
> opaque. The paper is opaque, too.
>
> AI95-00133 contains extensive references to the paper, but important
> things like
>
>   "The interpretation of component_clauses in the nondefault bit order
>   is based on machine scalars, which are chunks of storage that can be
>   natively loaded and stored by the machine. All of the
>   component_clauses at a given offset are considered to be part of the
>   same machine scalar, and the first_bit and last_bit are interpreted
>   as bit offsets within that machine scalar. This makes it possible to
>   write endian-independent record_representation_clauses."
>
> only appear in the AI, not the ALRM.
>
> And I think I'm right in saying that the size of the machine scalar
> starting at a given offset is determined by the largest last_bit of any
> of the components starting at that offset (so you'd better have a filler
> component extending into the top byte if none of the actual components
> do).
>
>>> Yes, I know.  It seems that AI12-0218 and Scalar_Storage_Order are
>>> focused on the wrong topic (i.e., the topic of a 2nd solution to the
>>> endian byte-swapping).  A replacement to AI12-0218 should be focused
>>> on .portably. (among all Ada compilers) choosing between the 2
>>> alternative byte-swapping Record Representation Clauses on different
>>> targets.
>>
>> I do not understand. As far as I can tell, the Ada 2005 solution with
>> its machine scalars is portable.
>
> But it doesn't do byte swapping.

Thank goodness. People use rep. clauses all the time in code that permeates 
an entire program; having very expensive byte swapping code easily triggered 
would give Ada a reputation of being a slug (rather than understanding that 
the code misuses the features). The whole idea of portable representation 
clauses is close to misuse of the feature (the entire value is to ensure 
that compilers for the same target intepret the clauses the same way).

Byte-swapping by its nature has to be restricted to an interface layer, and 
that layer is almost always target-dependent. Making it easy to "leak" 
interface info out of that layer is not helping anything.

                                                             Randy.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

"Randy Brukardt" <randy@rrsoftware.com> writes:

> "Simon Wright" <simon@pushface.org> wrote in message 
> news:ly4litj7qx.fsf@pushface.org...

>> But it doesn't do byte swapping.
>
> Thank goodness. People use rep. clauses all the time in code that
> permeates an entire program; having very expensive byte swapping code
> easily triggered would give Ada a reputation of being a slug (rather
> than understanding that the code misuses the features). The whole idea
> of portable representation clauses is close to misuse of the feature
> (the entire value is to ensure that compilers for the same target
> intepret the clauses the same way).

(a) In that case, the LRM should point out this potential problem in red
block caps; or eliminate rep clauses altogether and make us use C-style
shifts and masks (I think that's what Dmitry does anyway).

(b) My experience of rep clauses is communication with other machines,
be they NTP time servers over a network or hardware interface cards. I
don't understand "ensure that compilers for the same target intepret the
clauses the same way".

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dmitry A. Kazakov]

On 2018-05-30 08:39 AM, Simon Wright wrote:
> "Randy Brukardt" <randy@rrsoftware.com> writes:
> 
>> "Simon Wright" <simon@pushface.org> wrote in message
>> news:ly4litj7qx.fsf@pushface.org...
> 
>>> But it doesn't do byte swapping.
>>
>> Thank goodness. People use rep. clauses all the time in code that
>> permeates an entire program; having very expensive byte swapping code
>> easily triggered would give Ada a reputation of being a slug (rather
>> than understanding that the code misuses the features). The whole idea
>> of portable representation clauses is close to misuse of the feature
>> (the entire value is to ensure that compilers for the same target
>> intepret the clauses the same way).
> 
> (a) In that case, the LRM should point out this potential problem in red
> block caps; or eliminate rep clauses altogether and make us use C-style
> shifts and masks (I think that's what Dmitry does anyway).

Exactly. But why is it C-style? Differently to C, Ada has a well defined 
semantics of all operations, shifts included. It is pretty much 
Ada-style to me, which cannot be said about representation clauses 
*misused* for this purpose. In my view it is impossible to make 
representation clauses a machine-independent data definition language. 
The *only* place for them is machine-dependent low-level code. Anything 
else is asking for trouble and against Ada principles of keeping code 
clean. I doubt many people could explain what exactly a complex 
representation clause does, what it does not do, where and when that 
happens or not.

P.S. It is an old debate about imperative vs. declarative, in which 
declarative (representation clauses in this case) is always to lose.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Simon Wright]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> But why is it C-style? Differently to C, Ada has a well defined
> semantics of all operations, shifts included

I guess I meant, the style found in Unix network programming
texts. Which typically show examples in C.

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Dan'l Miller]

On Wednesday, May 30, 2018 at 10:01:03 AM UTC-5, Simon Wright wrote:
> "Dmitry A. Kazakov" writes:
> 
> > But why is it C-style? Differently to C, Ada has a well defined
> > semantics of all operations, shifts included
> 
> I guess I meant, the style found in Unix network programming
> texts. Which typically show examples in C.

C++ instead of C, but conforming to Simon's point vis a vis Dmitry's position:
http://www.open-std.org/jtc1/sc22/wg21/docs/TR18015.pdf

Well, to take the per-IC-register piecemeal bit-twiddling approach to the extreme, there is ISO TR18015 from C++-world which (in Appendix B from page 129 through page 153) ad nauseam explains the iohw approach that is more in Dmitry's* school of thought than in mine**.  To be fair, iohw handles the other major case of I/O-port ICs in addition to memory-mapped ICs (by uncouthly making nice civilized memory-mapped ICs have a clunky barbaric I/O-port-esque personality*, but hey that was their mantra, and they stuck to it).

cryptosyntax:
* per-IC-register piecemeal bit-twiddling in imperative code that uses magic numbers (or named constants thereof) to identify the registers numerically (where those magic numbers might be von-Neumann addresses mapped into memory or might be I/O port numbers)

lucid readable code true to Jean Ichbiah's overall goals for readable lucid Ada:
** typecasting a record-with-tightly-controlled-representation onto memory-mapped address with the offsets calculated implicitly by the compiler as the octet or word offset from the IC's single memory-mapped base address

Re: AI12-0218: What is the portable representation clause for processing IETF packets on little-endian machines?

[Randy Brukardt]

"Simon Wright" <simon@pushface.org> wrote in message 
news:lysh69hcl0.fsf@pushface.org...
> "Randy Brukardt" <randy@rrsoftware.com> writes:
>
>> "Simon Wright" <simon@pushface.org> wrote in message
>> news:ly4litj7qx.fsf@pushface.org...
>
>>> But it doesn't do byte swapping.
>>
>> Thank goodness. People use rep. clauses all the time in code that
>> permeates an entire program; having very expensive byte swapping code
>> easily triggered would give Ada a reputation of being a slug (rather
>> than understanding that the code misuses the features). The whole idea
>> of portable representation clauses is close to misuse of the feature
>> (the entire value is to ensure that compilers for the same target
>> intepret the clauses the same way).
>
> (a) In that case, the LRM should point out this potential problem in red
> block caps; or eliminate rep clauses altogether and make us use C-style
> shifts and masks (I think that's what Dmitry does anyway).

Rep. clauses are very useful where byte swaps are not needed (even there, 
though, it is best to avoid them in code that needs to have highest 
performance).  And it's impossible to legislate good program design in a 
programming language -- but that doesn't mean that we have to encourage bad 
design.

> (b) My experience of rep clauses is communication with other machines,
> be they NTP time servers over a network or hardware interface cards. I
> don't understand "ensure that compilers for the same target intepret the
> clauses the same way".

I assume that a single compiler will always interpret rep. clauses the same 
way (otherwise, that would be very unfriendly to their customers). But there 
are lots of things in Ada (or any other programming language), if left to 
their own interpretation, will end up different for different compilers even 
for the same target. (For example, the exact interpretation of file names).

The point of Claw, of instance, was to work on *all* Windows Ada compilers, 
and for that, rep. clauses, pragma Import, and a variety of other things 
have to work the same way. And that was a struggle - even a premimum vendor 
that truly cared like Rational in the late 1990s got lots of things 
wrong/not portable.

Most of the value of Ada's portability features is to prevent vendor 
lock-in, so you can move your code to a new vendor without having to make a 
lot of code changes.

When the target changes (especially the processor), that advantage is much 
less, since all of those things that prevent real portability are going to 
be different anyway. I'm sure there are occassional exceptions here and 
there (there *always* are exceptions to any rule!), but those are much less 
likely. (Ada is quite portable even without that target-specific stuff 
that's almost never portable, but that requires careful program design to 
keep the things likely to be different very isolated.)

                                          Randy.