Binary files vs Portablity vs Ada

Binary files vs Portablity vs Ada

[John Halleck]

Background:

  At one time I wrote a version of the Standard Hash Algorithm (SHA, 
  it went with the old government Digital Signature Standard)
  which I wrote in C...

  Lately, because GNAT is now around, I've hauled it out again,
  and redid it in Ada.  I then used it as an example for a coworker
  as to how you can be portable in Ada.

  The actual algorithm is defined for a bit stream, although almost
  everyone implements it for bits in byte size chunks. (As I did)
  Because of this, any file should be Hashable, even on machines
  with perverse word and byte sizes. 

Question:

  Is there even a portable way to get the "raw" bits for a file that
  works on all machines supporting Ada?  (So that it can be handed
  to a bit oriented version of SHA.)

  It is not hard to do something that just grabs bytes from a file
  until done, but that assumes the file is some multiple of eight
  bits long, and that all data for the file is in the bytes returned.
  On machines like the Sperry 1100 series, the native word size is
  36 bits (Usually chopped into four nine-bit bytes for Text use)
  And the file is "control word -> image"  instead of "embeded
  control."

  Can this be done portabily?  (I suspect not, but I'm not an Ada
  expert.)  I have something that seems to work for PC's, Mac's, 
  and Unix boxes, but it bugs me that I don't know how to do it in
  any general way.

Re: Binary files vs Portablity vs Ada

[Robert Dewar]

In article <7vt67r$qv0$1@coward.ks.cc.utah.edu>,
  nahaj@u.cc.utah.edu (John Halleck) wrote:
>   Is there even a portable way to get the "raw" bits for a
>   file that works on all machines supporting Ada?  (So that it
>   can be handed to a bit oriented version of SHA.)


Sure, have a look at Ada.Streams.Stream_IO


Sent via Deja.com http://www.deja.com/
Before you buy.

Re: Binary files vs Portablity vs Ada

[Ted Dennison]

In article <7vurt3$ojd$1@nnrp1.deja.com>,
  Robert Dewar <robert_dewar@my-deja.com> wrote:
> In article <7vt67r$qv0$1@coward.ks.cc.utah.edu>,
>   nahaj@u.cc.utah.edu (John Halleck) wrote:
> >   Is there even a portable way to get the "raw" bits for a
> >   file that works on all machines supporting Ada?  (So that it
> >   can be handed to a bit oriented version of SHA.)
>
> Sure, have a look at Ada.Streams.Stream_IO


You do have to be somewhat careful there. There's a whole lot about
streams that is implmentation dependent. For one thing, there's no
guarantee that you are really reading the "raw" (whatever that means)
bits from the file when you do a stream IO operation, unless you wrote
the stream's Read and Write routines yourself. There's no reason to
expect that your vendor would do something goofy with the data (like
compress it or something), but there's nothing stopping them either.

There may be portability issues with the IO attributes as well. For
instance, what exactly does Integer'Write send to the stream's Write
procedure? The language doesn't specify it. In one instance here we were
trying to 'Read data on one machine that was 'Written on another using
the same code compiled with a different vendor's compiler (big
mistake!). Among other portability problems, one vendor had implemented
'Write on arrays to send the elements in the array to the stream in
*reverse* order. I believe that's probably against the LRM (It says
elements are written in "cannonical" order), but the point is that
unless you write everything yourself, the only thing you can really
count on when you are using default stream implementations is that 'Read
will be able to retrieve what 'Write wrote, if both were compiled with
the same compiler on the same platform.

--
T.E.D.


Sent via Deja.com http://www.deja.com/
Before you buy.

Re: Binary files vs Portablity vs Ada

[Robert Dewar]

In article <7vuto0$pv0$1@nnrp1.deja.com>,
  Ted Dennison <dennison@telepath.com> wrote:
> There's no reason to
> expect that your vendor would do something goofy with the data
> (like compress it or something), but there's nothing stopping
> them either.

This is unnecessary FUD :-)

There is specific implementation advice on this subject:

                            Implementation Advice

17   If a stream element is the same size as a storage element,
then the normal in-memory representation should be used by Read
and Write for scalar objects.  Otherwise, Read and Write should
use the smallest number of stream elements needed to represent
all values in the base range of the scalar type.

Now an implementation could conceivably refuse to follow this
implementation advice, but then it must document this fact:

1   The Ada language allows for certain machine dependences in a
controlled manner.  Each Ada implementation must document all
implementation-defined characteristics:

    2  Whether or not each recommendation given in
       Implementation Advice is followed.  See 1.1.2(37).

So if you are really worried by Ted's concerns, then just check
your compiler documentation, for GNAT for example you will find:

@cindex Stream oriented attributes
@item 13.13.2(17): Stream Oriented Attributes

... (repeat of quote above)

Followed.

Reassuring you that GNAT does not do crazy things. Note that
if you start worrying about craziness at this level, almost
no compiler is useful for anything :-)


> I believe that's probably against the LRM (It says
> elements are written in "cannonical" order),

yes it's a plain bug!!!

>  but the point is that
> unless you write everything yourself, the only thing you can
really
> count on when you are using default stream implementations is
that 'Read
> will be able to retrieve what 'Write wrote, if both were
compiled with
> the same compiler on the same platform.

No, that's too pessimistic.


Sent via Deja.com http://www.deja.com/
Before you buy.

Re: Binary files vs Portablity vs Ada

[Ted Dennison]

In article <7vvrin$gp9$1@nnrp1.deja.com>,
  Robert Dewar <robert_dewar@my-deja.com> wrote:
> In article <7vuto0$pv0$1@nnrp1.deja.com>,
>   Ted Dennison <dennison@telepath.com> wrote:
> > There's no reason to
> > expect that your vendor would do something goofy with the data
> > (like compress it or something), but there's nothing stopping
> > them either.
> There is specific implementation advice on this subject:

Perhaps. But advice is just that; advice. No guarantees. If you are
trying to do something universally portable, you can't count on
implementation advice.


> Now an implementation could conceivably refuse to follow this
> implementation advice, but then it must document this fact:

That's only an issue if you have two particular implementations you want
it to work between. But no one asked about portability between two
particular compilers. The issue is portability in general.

Even in the two compiler case, you're nessecarily not off the hook. The
"required" documentation is often inadaquate, out of date, or just plain
wrong. I have found two different vendors for which that applies (not
ACT, but then I haven't read through ACT's annex M). When I mentioned
this here, I was told there's no "validation test suite" for
documentation. So apparently a crappy annex M implementation (or perhaps
even *no* annex M at all) will not in any way hamper a vendor's ability
to get or keep an Ada validation.

--
T.E.D.


Sent via Deja.com http://www.deja.com/
Before you buy.

Re: Binary files vs Portablity vs Ada

[Tucker Taft]

Ted Dennison wrote:
> 
> In article <7vvrin$gp9$1@nnrp1.deja.com>,
>   Robert Dewar <robert_dewar@my-deja.com> wrote:
> > In article <7vuto0$pv0$1@nnrp1.deja.com>,
> >   Ted Dennison <dennison@telepath.com> wrote:
> > > There's no reason to
> > > expect that your vendor would do something goofy with the data
> > > (like compress it or something), but there's nothing stopping
> > > them either.
> > There is specific implementation advice on this subject:
> 
> Perhaps. But advice is just that; advice. No guarantees. If you are
> trying to do something universally portable, you can't count on
> implementation advice.
> 
> > Now an implementation could conceivably refuse to follow this
> > implementation advice, but then it must document this fact:
> 
> That's only an issue if you have two particular implementations you want
> it to work between. But no one asked about portability between two
> particular compilers. The issue is portability in general.
> ...

It is always a bit frustrating to follow discussions like this
where people keep arguing about how this or that is not "guaranteed"
by the Ada RM, when other languages have far fewer guarantees to begin
with, and no strictly-obeyed standard and validation suite available.

Portability is not an absolute.  It is defined relative to the platforms
and compilers of interest.  The Ada RM tries to avoid overspecification,
partly because it is trying to handle any sort of target environment, with
storage elements varying from 8 bits each to 60 bits each. 

If you limit yourself to things which are "guaranteed" portable by
the RM, you will probably be disappointed.  If instead you concentrate
on localizing your target-specific assumptions, rather than spreading them
around, you use some common sense, and try your code on whatever compilers 
you have ready access to, you will probably be happier...

> T.E.D.

-- 
-Tucker Taft   stt@averstar.com   http://www.averstar.com/~stt/
Technical Director, Distributed IT Solutions  (www.averstar.com/tools)
AverStar (formerly Intermetrics, Inc.)   Burlington, MA  USA

Re: Binary files vs Portablity vs Ada

[Robert Dewar]

In article <3827555C.D8E87932@averstar.com>,
  Tucker Taft <stt@averstar.com> wrote:
> It is always a bit frustrating to follow discussions like this
> where people keep arguing about how this or that is not
> "guaranteed" by the Ada RM, when other languages have far
> fewer guarantees to begin with, and no strictly-obeyed
> standard and validation suite available.

I very much agree. The real issue here is that if you want to
write portable code, you need to combine a lot of relevant
pragmatic knowledge with the material in the RM. The IA helps
provide that knowledge. Yes, I know people would like to reduce
portability to simple compliance with what is guaranteed in
the standard, but as Tuck says, you will be disappointed if
you try this approach. There is no substitute for intelligent
application of pragmatic knowledge when it comes to writing
portable code.

In the case of C, portable code typically contains many things
that are not guaranteed by the standard, e.g. that int has a
sensible range. What Ada does is to make this kind of thing
very explicit in the RM (in the form of implementation advice),
and then vendors are required to note if they do not follow this
advice, so you are on much more well defined ground than with
any other standardized language when it comes to writing
portable code.


Sent via Deja.com http://www.deja.com/
Before you buy.

Re: Binary files vs Portablity vs Ada

[Robert A Duff]

Ted Dennison <dennison@telepath.com> writes:

> Perhaps. But advice is just that; advice. No guarantees. If you are
> trying to do something universally portable, you can't count on
> implementation advice.

The reason certain rules are written as Implementation Advice is that we
didn't know how to formalize the rule.  It's not that Impl Advice is any
less important than the "real" rules.  So, I think you can count on
responsible implementers to not disobey the advice without good reason.
If you find otherwise, send in a bug report.  If you find poor
documentation (whether or not the RM "required" that documentation),
send in a bug report.

- Bob

Advice, or *Advice*? (was: Binary files vs Portablity vs Ada)

[Ted Dennison]

In article <wccaeonfthh.fsf@world.std.com>,
  Robert A Duff <bobduff@world.std.com> wrote:

> The reason certain rules are written as Implementation Advice is that
> we didn't know how to formalize the rule.  It's not that Impl Advice
> is any less important than the "real" rules.  So, I think you can

But are all the advice at that level, or are some really just
suggestions? From context and the attitudes I've seen from vendors, a
lot of it appears to be the latter.

> count on responsible implementers to not disobey the advice without
> good reason. If you find otherwise, send in a bug report.  If you find

Here's an example: Until recently 2 of the 3 Ada compilers we have here
at work did not follow the implementation advice in 11.4.1(19), putting
*nothing* in Exception_Message. (Disclaimer: One of these was Gnat,
which I *believe* had this problem. I don't have the "luxury" of
having obsolete compiler documentation for Gnat laying around, like
I do for my other compilers :-) ).

When I complained to other compiler's support group about this over a
year ago, it was treated as a feature enhancement request, not as a
*bug*. We finally got a version that followed the advice about 4 weeks
ago. As a point of comparison, this same company has taken no longer
than 2 months to fix any of the other bugs that I submitted that were
non-advice LRM compliance issues.

As for the other compiler (well...ok, Gnat), I have heard reports that
the situation has improved there recently, but I haven't yet checked it
out. Others will be more qualified than I to say what it is doing, and
what their policy towards this "bug" is/has been over the past year or
so.

--
T.E.D.


Sent via Deja.com http://www.deja.com/
Before you buy.

Re: Advice, or *Advice*? (was: Binary files vs Portablity vs Ada)

[Robert A Duff]

Ted Dennison <dennison@telepath.com> writes:

> Here's an example: Until recently 2 of the 3 Ada compilers we have here
> at work did not follow the implementation advice in 11.4.1(19), putting
> *nothing* in Exception_Message.

I would probably consider that a bug.

(But I fear I might well work for one of the companies that you're
griping about.  ;-) )

What's a "bug" and what's a "feature" and what's an "enhancement" is
somewhat open to interpretation.  It's certainly true that a compiler
can strictly obey the RM, and still have (what I would consider to be)
bugs.

- Bob

Re: Binary files vs Portablity vs Ada

[Robert Dewar]

In article <8074m8$bk8$1@nnrp1.deja.com>,
  Ted Dennison <dennison@telepath.com> wrote:

> Even in the two compiler case, you're nessecarily not off the
hook. The
> "required" documentation is often inadaquate, out of date, or
just plain
> wrong. I have found two different vendors for which that
applies (not
> ACT, but then I haven't read through ACT's annex M). When I
mentioned
> this here, I was told there's no "validation test suite" for
> documentation. So apparently a crappy annex M implementation
(or perhaps
> even *no* annex M at all) will not in any way hamper a
vendor's ability
> to get or keep an Ada validation.

No, but it should hinder you choosing a vendor who does not
provide this required essential documentation.

The point here is that you have to be reasonable about
portability. The implementation advice is really a set of
things which you can reasonably assume about code in writing
portable code.

Writing portable code is not about fanatically sticking to the
set of things that is absolutely guaranteed by the RM. Indeed
no programs can be written under that requirement, since any
program is allowed to raise Storage_Error at any point.

Writing portable code is about writing code that in practice
ports with minimum effort. You need to have a good sense of
what is and what is not reasonable to assume in this task, and
the implementation advice is precisely intended to help you.

You should indeed insist contractually on your vendor telling
you whether implementation advice is followed. If you are taking
the position that a compiler being validated is sufficient
guarantee that it is acceptable for your use, then that's
passing the buck. Ada users have to be prepared to do some
reasonable homework in choosing an Ada compiler. Checking out
the documentation is part of that homework.

I definitely would insist on a complete annex M, it is an
important requirement. If your vendor says they conform to the
RM, then they must provide this documentation (validation does
not prove conformance, and in particular there is no practical
way to formally validate documentation -- one could have a
requirement that annex M be present, but that's too weak to
be useful).

My advice: when choosing an Ada compiler, put the provision of
a complete and useful Annex M on your check list!

And, going back to the original issue, this particular IA is
something you can reasonably count on. Sure it is technically
possible for a compiler to refuse to follow this IA, but
unlikely, probably much less likely than having a bug in the
implementation, and no amount of reading the RM protects you
against that.


Sent via Deja.com http://www.deja.com/
Before you buy.

Re: Binary files vs Portablity vs Ada

[Larry Kilgallen]

In article <7vt67r$qv0$1@coward.ks.cc.utah.edu>, nahaj@u.cc.utah.edu (John Halleck) writes:

>   It is not hard to do something that just grabs bytes from a file
>   until done, but that assumes the file is some multiple of eight
>   bits long, and that all data for the file is in the bytes returned.
>   On machines like the Sperry 1100 series, the native word size is
>   36 bits (Usually chopped into four nine-bit bytes for Text use)
>   And the file is "control word -> image"  instead of "embeded
>   control."
> 
>   Can this be done portabily?  (I suspect not, but I'm not an Ada
>   expert.)  I have something that seems to work for PC's, Mac's, 
>   and Unix boxes, but it bugs me that I don't know how to do it in
>   any general way.

Does your Mac version do the resource fork or just the data fork ?

Windows NT (NTFS) actually supports more than two forks, although
most programs don't make use of them.

On VMS, you would have to decide what treatment you would give to
various file organizations and record formats provided by the
operating system.

On MVS, I get the feeling that dealing with a "partitioned data
set" would not be at all portable to anything.

So hardware data size is not your only issue.

Larry Kilgallen

Re: Binary files vs Portablity vs Ada

[John Halleck]

Larry Kilgallen (kilgallen@eisner.decus.org) wrote:
: [...] nahaj@u.cc.utah.edu (John Halleck) writes:

: > [...]

: >   It is not hard to do something that just grabs bytes from a file
: >   until done, but that assumes the file is some multiple of eight
: >   bits long, and that all data for the file is in the bytes returned.
: >   On machines like the Sperry 1100 series, the native word size is
: >   36 bits (Usually chopped into four nine-bit bytes for Text use)
: >   And the file is "control word -> image"  instead of "embeded
: >   control."
: > 
: >   Can this be done portabily?  (I suspect not, but I'm not an Ada
: >   expert.)  I have something that seems to work for PC's, Mac's, 
                                      *****
: >   and Unix boxes, but it bugs me that I don't know how to do it in
: >   any general way.

: Does your Mac version do the resource fork or just the data fork ?

  (:  Only Data  :)  The project was only to show a co-worker about
  some of the ease of doing things portably in Ada.  It was not
  for a production product...  

: > [... good examples of other portablitity issues ommitted ...]

: So hardware data size is not your only issue.

  True enough.

  For the purpose of getting the bits for a hash there are other
  issues which I would expect the Language Designers not to have
  addressed.

  For example, on many OS's, files can have unallocated regions
  within the body of the file.  How does one handle that for a
  Hash?  (Or even doing direct IO?)  There are ugly issues everywhere.


  This discussion seems to be drifting from the "Do it in Ada" question
  to "Boy the world is ugly portablity wise."

  I'm more than willing to continue this thread in private email with all
  interested parties.

: Larry Kilgallen

Re: Binary files vs Portablity vs Ada

[Matthew Heaney]

In article <7vt67r$qv0$1@coward.ks.cc.utah.edu> , nahaj@u.cc.utah.edu 
(John Halleck) wrote:

>   Is there even a portable way to get the "raw" bits for a file that
>   works on all machines supporting Ada?  (So that it can be handed
>   to a bit oriented version of SHA.)

Are you using Stream_IO to read in the raw data?  Does Stream_Element
type already handle machine differences?

Matt
--
Evolution is "only a theory"?  "Creationism is not *even* a theory,
because its proponents have decided in advance that no amount of
evidence will change their beliefs."

Steve Walsh and Thomas Demere, "Facts, Faith, and Fairness"

Re: Binary files vs Portablity vs Ada

[Nick Roberts]

John Halleck wrote:
> [how to manipulate file bits portably]

I'd have thought you could go a long way towards a portable
implementation if you could code an algorithm that can work on words of
a parametric size (number of bits), and then use
Ada.Streams.Stream_Element'Size as the parameter.

Then you can use Ada.Streams.Stream_IO to do input and output directly
on (arrays of) stream elements. I think this would bypass any
implementation-specific characteristic of reading or writing the data,
in practice.

To get the bits out of a mod type (portably), you have to resort to
arithmetic (X and 2**N), and hope the compiler optimises correctly.

I think it's a subject that has been mentioned before in comp.lang.ada,
but possibly another suggestion for the next Ada revision might be two
new predefined types:

   type Bit_Integer is mod 2;

   type Bit_Array is array (Natural range <>) of Bit_Integer;

and then a new conversion introduced between Bit_Array (or types derived
from it) and any mod type whose modulus was a power of two (and
static?). This conversion would stipulate that the bit with the lowest
index (in the array) was the bit with the least significance (in the
integer). 

-- 
Nick Roberts
Computer Consultant (UK)
http://www.callnetuk.com/home/nickroberts
http://www.adapower.com/lab/adaos

Re: Binary files vs Portablity vs Ada

[Ted Dennison]

In article <3826FD9C.37446C29@callnetuk.com>,
  Nick Roberts <nickroberts@callnetuk.com> wrote:

> but possibly another suggestion for the next Ada revision might be two
> new predefined types:
>
>    type Bit_Integer is mod 2;
>
>    type Bit_Array is array (Natural range <>) of Bit_Integer;
>
> and then a new conversion introduced between Bit_Array (or types
derived
> from it) and any mod type whose modulus was a power of two (and
> static?). This conversion would stipulate that the bit with the lowest
> index (in the array) was the bit with the least significance (in the
> integer).

Why a one bit integer, as opposed to a boolean array?

--
T.E.D.


Sent via Deja.com http://www.deja.com/
Before you buy.

Re: Binary files vs Portablity vs Ada

[Robert Dewar]

In article <3826FD9C.37446C29@callnetuk.com>,
  Nick Roberts <nickroberts@callnetuk.com> wrote:
>    type Bit_Integer is mod 2;
>    type Bit_Array is array (Natural range <>) of Bit_Integer;

I don't see the introduction of these types in the RM as being
a helpful step in improving portability. Can you give an example
of something that *in practice* is a portability problem now,
that would be no longer a problem. I am not really interested
in problems that might happen in theory, but do not happen in
practice.


Sent via Deja.com http://www.deja.com/
Before you buy.