Advice on low level file handling.

Advice on low level file handling.

[Peter C. Chapin]

I'm working on a program that needs to read an input file on a byte by byte 
basis and examine bit fields and do bitwise shifting in some of these 
bytes. Other parts of the file are to be treated as uninterpreted data 
(this is an OpenPGP message file). Right now I'm using Interfaces.Unsigned_
8 as the type to hold a single byte from the file and I'm instantiating 
Ada.Sequential_IO with this type to get the necessary file reading 
subprograms. So far this seems okay, but I'm wondering if this is the most 
appropriate way to do this. Am I abusing package Interfaces? Should I be 
defining my own modular type or should I be using representation clauses 
with custom record types?

Also I need to count the number of bytes I process and the quantity may 
exceed 2**32. I can define a 64 bit modular type and that seems to work 
fine on gnat. Is that portable? I'm not extremely worried about 
portability, but I'd rather not sacrifice it for no reason.

Thanks for any advice you can offer.

Peter

Re: Advice on low level file handling.

[Georg Bauhaus]

Peter C. Chapin wrote:

> Also I need to count the number of bytes I process and the quantity may 
> exceed 2**32. I can define a 64 bit modular type and that seems to work 
> fine on gnat. Is that portable? I'm not extremely worried about 
> portability, but I'd rather not sacrifice it for no reason.

That's nice to hear when someone at least thinks about
portability :-)

package More is

   type Count is range 0 .. 2**50;

end More;

with More;
procedure More.try_it is
   x: Count;
begin
   x := 2**40;
end;

Using similar front ends (first is ObjectAda for Windows native,
second is AppletMagic), but different backends,

...\> adamake more.try_it
more.ada: Error: line 3 col 26 LRM:3.5.4(6),
  subtype bounds must be between System.Min_Int and System.Max_Int,
  returning plastic subtype
(etc, failing)

... \> adajava more.try_it

Front end of ..\more.ads succeeded with no errors.

Re: Advice on low level file handling.

[Randy Brukardt]

"Peter C. Chapin" <pchapin@sover.net> wrote in message
news:Xns9798DDB851DB4pchapinsovernet@198.186.192.137...
>
> I'm working on a program that needs to read an input file on a byte by
byte
> basis and examine bit fields and do bitwise shifting in some of these
> bytes. Other parts of the file are to be treated as uninterpreted data
> (this is an OpenPGP message file). Right now I'm using
Interfaces.Unsigned_
> 8 as the type to hold a single byte from the file and I'm instantiating
> Ada.Sequential_IO with this type to get the necessary file reading
> subprograms. So far this seems okay, but I'm wondering if this is the most
> appropriate way to do this.

Why aren't you using Stream_IO for this? You don't need to declare your own
types for this sort of I/O. Perhaps you thought that you could only use
Stream_IO with the stream attributes? Actually, I think it is more useful to
use by itself, especially in the sort of case you have here. Byte at a time
I/O can be painfully slow; Stream_IO lets you input a batch of stuff and
then process it.

>...Is that portable? I'm not extremely worried about
> portability, but I'd rather not sacrifice it for no reason.

A machine might not have Unsigned_8 (like the U2200, it has Unsigned_9
instead); but there is going to be a definition of Stream_Element. On the
vast majority of machines, that will be an 8-bit entity.

OTOH, if you *must* have an 8-bit byte, then I'd suggest declaring it
yourself and using Sequential_IO as you did.

> I can define a 64 bit modular type and that seems to work
> fine on gnat. Is that portable?

Not really, but an Ada compiler will reject the program if it doesn't work.
(As someone else pointed out, many Ada compilers don't support 64-bit
numbers.) But it's probably portable enough for your purposes; there isn't
much point in working around the lack of 64-bit math until you actually have
to work on a compiler that doesn't support it.

                                     Randy.

Re: Advice on low level file handling.

[Peter C. Chapin]

"Randy Brukardt" <randy@rrsoftware.com> wrote in
news:R_adnVLVrp7jUazZRVn-rw@megapath.net: 

> Why aren't you using Stream_IO for this? You don't need to declare
> your own types for this sort of I/O. Perhaps you thought that you
> could only use Stream_IO with the stream attributes? Actually, I think
> it is more useful to use by itself, especially in the sort of case you
> have here. Byte at a time I/O can be painfully slow; Stream_IO lets
> you input a batch of stuff and then process it.

I see. Thanks for the suggestion; I'll take a look at it.

> OTOH, if you *must* have an 8-bit byte, then I'd suggest declaring it
> yourself and using Sequential_IO as you did.

Well, the RFC is described in terms of 8-bit bytes so using anything 
else will result in an unclear correspondence between the code and the 
specification. At least, that would be my expectation.

On the other hand, it might make sense to define a suitable record type 
that has the necessary fields already specified in the right places and 
then just read a chunk of the file into such a record. However, part of 
the file needs to be uninterpreted data---it's not a file of records---
so this approach may not work out well either.

> Not really, but an Ada compiler will reject the program if it doesn't
> work. (As someone else pointed out, many Ada compilers don't support
> 64-bit numbers.) But it's probably portable enough for your purposes;
> there isn't much point in working around the lack of 64-bit math until
> you actually have to work on a compiler that doesn't support it.

I can see why the C99 standard added a 64 bit type to the standard. In 
today's world, objects larger than 2/4 GBytes are not unusual. It's 
awkward talking about such large sizes without a standard type to 
represent them. Does Ada 2005 have a built-in type that is 64 bits (or 
more)? 

Peter

Re: Advice on low level file handling.

[Michael Paus]

Peter C. Chapin wrote:

> "Randy Brukardt" <randy@rrsoftware.com> wrote in
> news:R_adnVLVrp7jUazZRVn-rw@megapath.net: 

> On the other hand, it might make sense to define a suitable record type 
> that has the necessary fields already specified in the right places and 
> then just read a chunk of the file into such a record. However, part of 
> the file needs to be uninterpreted data---it's not a file of records---
> so this approach may not work out well either.

That's exactly what Stream_IO is made for. The big advantage of Stream_IO
is that you can read mixed types from a single stream. So you can read
pre-defined record types and uninterpreted data (bytes) just as you like.

Michael

Re: Advice on low level file handling.

[Peter C. Chapin]

Michael Paus <pausnospam@nospamib-paus.com> wrote in news:e12grg$5nr$1
@online.de:

> That's exactly what Stream_IO is made for. The big advantage of Stream_IO
> is that you can read mixed types from a single stream. So you can read
> pre-defined record types and uninterpreted data (bytes) just as you like.

Okay, I'll take a look at Stream_IO. However, as I think about this more 
I'm not sure there is much to gain by reading the header data directly into 
a record. I have to do endianness conversions on the multibyte quantities 
(big endian file, little endian CPU). Given that, it might be just as easy 
to read the data one byte at a time and build up the record fields as I go.

In other words: I probably need to treat the file as a blob of 
unstructured, typeless data and impose structure on it at run time rather 
than with compile time declarations. This seems somewhat contrary to the 
ideals of strong typing, but I'm not sure I see a good alternative in a 
case like this. I suppose this is why practical strongly typed languages 
have escapes from their typing mechanisms. This problem must come up 
frequently when writing programs that conform to externally defined binary 
standards.

Peter

Re: Advice on low level file handling.

[Dmitry A. Kazakov]

On 06 Apr 2006 12:21:08 GMT, Peter C. Chapin wrote:

> In other words: I probably need to treat the file as a blob of 
> unstructured, typeless data and impose structure on it at run time rather 
> than with compile time declarations. This seems somewhat contrary to the 
> ideals of strong typing, but I'm not sure I see a good alternative in a 
> case like this.

One way is to have a better control over the representation.

Another is to provide abstract record interface. The problem is that the
record view you want to use when dealing with read data does not correspond
to the memory layout. The question is, should it really correspond? It
might be better and easier for the language to provide user-defined
getter/setter interface supported by a predefined library for dealing with
endianess than to blow up representation clauses.

> I suppose this is why practical strongly typed languages 
> have escapes from their typing mechanisms. This problem must come up 
> frequently when writing programs that conform to externally defined binary 
> standards.

Not only this. The whole purpose of constructors and destructors is to
circumvent typing mechanism.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Advice on low level file handling.

[Bob Spooner]

"Peter C. Chapin" <pchapin@sover.net> wrote in message 
news:Xns979D55134B90Cpchapinsovernet@198.186.192.137...
> Michael Paus <pausnospam@nospamib-paus.com> wrote in news:e12grg$5nr$1
> @online.de:
>
>> That's exactly what Stream_IO is made for. The big advantage of Stream_IO
>> is that you can read mixed types from a single stream. So you can read
>> pre-defined record types and uninterpreted data (bytes) just as you like.
>
> Okay, I'll take a look at Stream_IO. However, as I think about this more
> I'm not sure there is much to gain by reading the header data directly 
> into
> a record. I have to do endianness conversions on the multibyte quantities
> (big endian file, little endian CPU). Given that, it might be just as easy
> to read the data one byte at a time and build up the record fields as I 
> go.
>
I think you should be able to do endian conversions by overriding the 'Read 
attributes for the types in your record where this is needed and doing the 
conversions in the read routines as you read in the record. This may not 
work for floating point values if the wrong endianness of the data 
occasionally produces NAN.

Bob 

Re: Advice on low level file handling.

[Stephen Leake]

"Peter C. Chapin" <pchapin@sover.net> writes:

> In other words: I probably need to treat the file as a blob of 
> unstructured, typeless data and impose structure on it at run time rather 
> than with compile time declarations. 

Yes, that is typically the case when reading files of data.

> This seems somewhat contrary to the ideals of strong typing, but I'm
> not sure I see a good alternative in a case like this. I suppose
> this is why practical strongly typed languages have escapes from
> their typing mechanisms. This problem must come up frequently when
> writing programs that conform to externally defined binary
> standards.

Yes. For one fairly well structured solution, see
http://www.toadmail.com/~ada_wizard/ada/sal.html , particularly the
"network_order" packages (sal-network_order.ads, sal-*-network_order.*).


-- 
-- Stephe