'Write attribute vs Write procedure.

'Write attribute vs Write procedure.

[Jano]

Hi everybody.

I'm getting abysmal differences in speed when writing large chunks (64k) 
of data to a stream. The stream is from a socket of Gnat.Sockets, I 
mention it because I don't know if it has something to do.

See the two following snippets of code:

----common-----
Buffer : Stream_element_array (1 .. 
            Stream_element_offset (Upload_buffer_size));
----common-----

----case A-----
Write (This.Link.all, 
   Buffer (1 .. Stream_element_offset (Chunk)));
----case A-----

----case B-----
Stream_element_array'Write (This.Link, 
   Buffer (1 .. Stream_element_offset (Chunk)));
----case B-----

In the above example (without optimizations and -O2 tried, both the 
same), the Write procedure is taking no noticeable amount of time/cpu. 
In the contrary, the 'Write attribute takes like 3-4 seconds of 100% CPU 
usage, shared more or less equally by my program and the SYSTEM process, 
whatever it is, on WinXP. Compiler is Gnat 3.15p, computer Pentium III 
450Mhz.

I'm asking only out of curiosity, and to prevent any other surprise in 
the future. Could it be related to alignment checks/adjustments?

P.s: The 'Read attribute is also fast, no noticeable delay.

P.s2: I've tried also with strings, only the 'Write version of course, 
and results are the same.

-- 
-------------------------
Jano
402450.at.cepsz.unizar.es
-------------------------

Re: 'Write attribute vs Write procedure.

[tmoran]

> Write (This.Link.all,
>    Buffer (1 .. Stream_element_offset (Chunk)));
> Stream_element_array'Write (This.Link,
>    Buffer (1 .. Stream_element_offset (Chunk)));
>...
> the Write procedure is taking no noticeable amount of time/cpu.
>In the contrary, the 'Write attribute takes like 3-4 seconds of 100% CPU
  Perhaps Gnat's 'Write is doing one 'Write(Stream_Element) call for
each element of the Buffer, vs a single call for the whole thing in
the procedure version?  Since you've already got a Stream_Element_Array,
perhaps you could call
  Ada.Streams.Write(Ada.Streams.Root_Stream_Type'class(This.Link),
                    Buffer (1 .. Stream_element_offset (Chunk)));
directly?
  Personally, I think using the default 'write is only reasonable
in the very simplest cases.  If something is big, or complicated,
I'd write my own 'write for it.

Re: 'Write attribute vs Write procedure.

[Ole-Hjalmar Kristensen]

tmoran@acm.org writes:

> > Write (This.Link.all,
> >    Buffer (1 .. Stream_element_offset (Chunk)));
> > Stream_element_array'Write (This.Link,
> >    Buffer (1 .. Stream_element_offset (Chunk)));
> >...
> > the Write procedure is taking no noticeable amount of time/cpu.
> >In the contrary, the 'Write attribute takes like 3-4 seconds of 100% CPU
>   Perhaps Gnat's 'Write is doing one 'Write(Stream_Element) call for
> each element of the Buffer, vs a single call for the whole thing in
> the procedure version?  Since you've already got a Stream_Element_Array,
> perhaps you could call

You hit the nail on the head.

>   Ada.Streams.Write(Ada.Streams.Root_Stream_Type'class(This.Link),
>                     Buffer (1 .. Stream_element_offset (Chunk)));
> directly?
>   Personally, I think using the default 'write is only reasonable
> in the very simplest cases.  If something is big, or complicated,
> I'd write my own 'write for it.

-- 
Ole-Hj. Kristensen

******************************************************************************
* You cannot consistently believe this sentence.
******************************************************************************

Re: 'Write attribute vs Write procedure.

[Ole-Hjalmar Kristensen]

tmoran@acm.org writes:

> > Write (This.Link.all,
> >    Buffer (1 .. Stream_element_offset (Chunk)));
> > Stream_element_array'Write (This.Link,
> >    Buffer (1 .. Stream_element_offset (Chunk)));
> >...
> > the Write procedure is taking no noticeable amount of time/cpu.
> >In the contrary, the 'Write attribute takes like 3-4 seconds of 100% CPU
>   Perhaps Gnat's 'Write is doing one 'Write(Stream_Element) call for
> each element of the Buffer, vs a single call for the whole thing in
> the procedure version?  Since you've already got a Stream_Element_Array,
> perhaps you could call
>   Ada.Streams.Write(Ada.Streams.Root_Stream_Type'class(This.Link),
>                     Buffer (1 .. Stream_element_offset (Chunk)));
> directly?
>   Personally, I think using the default 'write is only reasonable
> in the very simplest cases.  If something is big, or complicated,
> I'd write my own 'write for it.

Yes, but that removes much of the attraction of using a stream in the
first place. It should only be necessary to do your own write if you
wanted another representation of it in the external format.  A prime
example of rendering a very nice feature of the language pretty much
unusable because of a poor implementation.

-- 
Ole-Hj. Kristensen

******************************************************************************
* Forth: You yourself foot in shoot
******************************************************************************

Re: 'Write attribute vs Write procedure.

[tmoran]

>>   Personally, I think using the default 'write is only reasonable
>> in the very simplest cases.  If something is big, or complicated,
>> I'd write my own 'write for it.
>
>Yes, but that removes much of the attraction of using a stream in the
>first place. It should only be necessary to do your own write if you
>wanted another representation of it in the external format.  A prime
>example of rendering a very nice feature of the language pretty much
>unusable because of a poor implementation.

  I disagree.  It is convenient when you use it that you needn't
distinguish between a 'write that will write in a canonical, vs a special
external format.  In the OP's case, it was probably simpler to get the
program up and running with the standard 'write, rather than having to
make his own, and then worry about doing something special as a matter of
performance tuning.

You also still have use at a higher level, eg
  type Sound_Bytes is record
    Time  : Ada.Calendar.Time;
    Audio : Wave;
    Title : String(1 .. 10);
  end record;
If you have a special Wave'write, you can say
  Sound_Bytes'write
and it will do the standard thing for Time and Name but call your
special routine for Audio, so you probably don't need to make your
own Sound_Bytes'write.  At it's own abstraction level, Sound_Bytes is
neither big nor complicated, so you probably don't need a special 'write.

Re: 'Write attribute vs Write procedure.

[tmoran]

>>   Personally, I think using the default 'write is only reasonable
>> in the very simplest cases.  If something is big, or complicated,
>> I'd write my own 'write for it.
>
>Yes, but that removes much of the attraction of using a stream in the
>first place. It should only be necessary to do your own write if you
>wanted another representation of it in the external format.  A prime
>example of rendering a very nice feature of the language pretty much
>unusable because of a poor implementation.

  I disagree.  It is convenient when you use it that you needn't
distinguish between a 'write that will write in a canonical, vs a special
external format.  In the OP's case, it was probably simpler to get the
program up and running with the standard 'write, rather than having to
make his own, and then worry about doing something special as a matter of
performance tuning.

You also still have use at a higher level, eg
  type Sound_Bytes is record
    Time  : Ada.Calendar.Time;
    Audio : Wave;
    Title : String(1 .. 10);
  end record;
If you have a special Wave'write, you can say
  Sound_Bytes'write
and it will do the standard thing for Time and Name but call your
special routine for Audio, so you probably don't need to make your
own Sound_Bytes'write.  At it's own abstraction level, Sound_Bytes is
neither big nor complicated, so you probably don't need a special 'write.

Re: 'Write attribute vs Write procedure.

[Simon Wright]

Ole-Hjalmar Kristensen <oleh@vlinux.voxelvision.no> writes:

> tmoran@acm.org writes:
> 
> > > Write (This.Link.all,
> > >    Buffer (1 .. Stream_element_offset (Chunk)));
> > > Stream_element_array'Write (This.Link,
> > >    Buffer (1 .. Stream_element_offset (Chunk)));
> > >...
> > > the Write procedure is taking no noticeable amount of time/cpu.
> > >In the contrary, the 'Write attribute takes like 3-4 seconds of 100% CPU
> >   Perhaps Gnat's 'Write is doing one 'Write(Stream_Element) call for
> > each element of the Buffer, vs a single call for the whole thing in
> > the procedure version?  Since you've already got a Stream_Element_Array,
> > perhaps you could call
> >   Ada.Streams.Write(Ada.Streams.Root_Stream_Type'class(This.Link),
> >                     Buffer (1 .. Stream_element_offset (Chunk)));
> > directly?
> >   Personally, I think using the default 'write is only reasonable
> > in the very simplest cases.  If something is big, or complicated,
> > I'd write my own 'write for it.
> 
> Yes, but that removes much of the attraction of using a stream in the
> first place. It should only be necessary to do your own write if you
> wanted another representation of it in the external format.  A prime
> example of rendering a very nice feature of the language pretty much
> unusable because of a poor implementation.

ARM 13.13.2(9) says

   [...] For composite types, the Write or Read attribute for each
   component is called in canonical order, which is last dimension
   varying fastest for an array, and positional aggregate order for a
   record. Bounds are not included in the stream if T is an array
   type. [...]

I'm not sure whether an implementation has licence to produce code
which works "as if" this happened. What about the exceptional cases?
(where there isn't enough room in the stream for all the elements, for
example).

Re: 'Write attribute vs Write procedure.

[Ole-Hjalmar Kristensen]

tmoran@acm.org writes:

> >>   Personally, I think using the default 'write is only reasonable
> >> in the very simplest cases.  If something is big, or complicated,
> >> I'd write my own 'write for it.
> >
> >Yes, but that removes much of the attraction of using a stream in the
> >first place. It should only be necessary to do your own write if you
> >wanted another representation of it in the external format.  A prime
> >example of rendering a very nice feature of the language pretty much
> >unusable because of a poor implementation.
> 
>   I disagree.  It is convenient when you use it that you needn't
> distinguish between a 'write that will write in a canonical, vs a special
> external format.  In the OP's case, it was probably simpler to get the
> program up and running with the standard 'write, rather than having to
> make his own, and then worry about doing something special as a matter of
> performance tuning.

Yes, but why should performance tuning be necessary for the simple
cases, like a record without tags or discriminants? The compiler has
all the information necessary to make an *efficient* write. I figure
the answer is that if you do it the simple-minded way by calling the
write procedure of every component, you don't have to do any special
compile time analysis, just note that this type has its own write
already defined, and don't generate one for it. But the opposite
strategy should also be simple : if on of the components has a special
write, do it by calling write on every component, else just transfer
it as a blob.

> 
> You also still have use at a higher level, eg
>   type Sound_Bytes is record
>     Time  : Ada.Calendar.Time;
>     Audio : Wave;
>     Title : String(1 .. 10);
>   end record;
> If you have a special Wave'write, you can say
>   Sound_Bytes'write
> and it will do the standard thing for Time and Name but call your
> special routine for Audio, so you probably don't need to make your
> own Sound_Bytes'write.  At it's own abstraction level, Sound_Bytes is
> neither big nor complicated, so you probably don't need a special 'write.

-- 
Ole-Hj. Kristensen

******************************************************************************
* You cannot consistently believe this sentence.
******************************************************************************

Re: 'Write attribute vs Write procedure.

[Ole-Hjalmar Kristensen]

Simon Wright <simon@pushface.org> writes:

> Ole-Hjalmar Kristensen <oleh@vlinux.voxelvision.no> writes:
> 
> > tmoran@acm.org writes:
> > 
> > > > Write (This.Link.all,
> > > >    Buffer (1 .. Stream_element_offset (Chunk)));
> > > > Stream_element_array'Write (This.Link,
> > > >    Buffer (1 .. Stream_element_offset (Chunk)));
> > > >...
> > > > the Write procedure is taking no noticeable amount of time/cpu.
> > > >In the contrary, the 'Write attribute takes like 3-4 seconds of 100% CPU
> > >   Perhaps Gnat's 'Write is doing one 'Write(Stream_Element) call for
> > > each element of the Buffer, vs a single call for the whole thing in
> > > the procedure version?  Since you've already got a Stream_Element_Array,
> > > perhaps you could call
> > >   Ada.Streams.Write(Ada.Streams.Root_Stream_Type'class(This.Link),
> > >                     Buffer (1 .. Stream_element_offset (Chunk)));
> > > directly?
> > >   Personally, I think using the default 'write is only reasonable
> > > in the very simplest cases.  If something is big, or complicated,
> > > I'd write my own 'write for it.
> > 
> > Yes, but that removes much of the attraction of using a stream in the
> > first place. It should only be necessary to do your own write if you
> > wanted another representation of it in the external format.  A prime
> > example of rendering a very nice feature of the language pretty much
> > unusable because of a poor implementation.
> 
> ARM 13.13.2(9) says
> 
>    [...] For composite types, the Write or Read attribute for each
>    component is called in canonical order, which is last dimension
>    varying fastest for an array, and positional aggregate order for a
>    record. Bounds are not included in the stream if T is an array
>    type. [...]
> 
> I'm not sure whether an implementation has licence to produce code
> which works "as if" this happened. What about the exceptional cases?
> (where there isn't enough room in the stream for all the elements, for
> example).

Too bad. But I cannot see what stops the compiler to produce code
which works "as if" this happened in the case where every read/write
attribute is compiler generated. After all, the standard does not
specify how the compiler should implement read and write for
elementary types.

-- 
Ole-Hj. Kristensen

******************************************************************************
* C: You shoot yourself in the foot
* Ada: If you are actually dumb enough to program in this language, the DOD will
* kidnap you, stand you up in front of a firing squad, and tell the soldiers:
* "Shoot at his feet"
******************************************************************************

Re: 'Write attribute vs Write procedure.

[Marc A. Criley]

Simon Wright <simon@pushface.org> wrote in message news:<x7v65o4j8y4.fsf@smaug.pushface.org>...

> I'm not sure whether an implementation has licence to produce code
> which works "as if" this happened. What about the exceptional cases?
> (where there isn't enough room in the stream for all the elements, for
> example).

According to Robert Dewar, compilers do have such "as if" license:

"Remember that a compiler is always allowed to treat a language
definition 'as-if', which means that any generated code that makes a
compiler run the code the same way as some other way is equivalent. " 
Posted 9/11/97 in comp.lang.ada.

(If you want to see the whole context of the post, do a Google groups
search in comp.lang.ada with keywords:   Dewar "as if" "no semantic
effect"
The thread is titled "The Red Language".  I didn't include the URL
because it's quite long and unwieldy.

Marc A. Criley

Re: 'Write attribute vs Write procedure.

[Simon Wright]

mcq95@earthlink.net (Marc A. Criley) writes:

> Simon Wright <simon@pushface.org> wrote in message news:<x7v65o4j8y4.fsf@smaug.pushface.org>...
> 
> > I'm not sure whether an implementation has licence to produce code
> > which works "as if" this happened. What about the exceptional cases?
> > (where there isn't enough room in the stream for all the elements, for
> > example).
> 
> According to Robert Dewar, compilers do have such "as if" license:
> 
> "Remember that a compiler is always allowed to treat a language
> definition 'as-if', which means that any generated code that makes a
> compiler run the code the same way as some other way is equivalent. " 
> Posted 9/11/97 in comp.lang.ada.
> 
> (If you want to see the whole context of the post, do a Google groups
> search in comp.lang.ada with keywords:   Dewar "as if" "no semantic
> effect"
> The thread is titled "The Red Language".  I didn't include the URL
> because it's quite long and unwieldy.

I meant to say, it isn't clear that a failure due to overrun when
writing as a blob will have the same external effect as writing
individual elements; and I'm not sure whether the "as if" permission
includes "unless an exceptional condition occurs, in which case you
are allowed to behave differently".

Re: 'Write attribute vs Write procedure.

[Craig Carey]

On Wed, 21 May 2003 00:42:33 GMT, tmoran@acm.org wrote:

>> Write (This.Link.all,
>>    Buffer (1 .. Stream_element_offset (Chunk)));
>> Stream_element_array'Write (This.Link,
>>    Buffer (1 .. Stream_element_offset (Chunk)));
>>...
>> the Write procedure is taking no noticeable amount of time/cpu.
>>In the contrary, the 'Write attribute takes like 3-4 seconds of 100% CPU
>  Perhaps Gnat's 'Write is doing one 'Write(Stream_Element) call for
>each element of the Buffer, vs a single call for the whole thing in
>the procedure version?  Since you've already got a Stream_Element_Array,
>perhaps you could call
>  Ada.Streams.Write(Ada.Streams.Root_Stream_Type'class(This.Link),
>                    Buffer (1 .. Stream_element_offset (Chunk)));
>directly?
>  Personally, I think using the default 'write is only reasonable
>in the very simplest cases.  [...]


X'Write() would have to come out best in a comparison involving at least
two choices. Some other principle, presumably something 'unreasonable',
shows up when there is not much Ada 95 code.


--

PS. GNAT is allocating tracking numbers for requests on GNAT.Sockets,
including for requests for big rewrites. ACT might not be planning on
rewriting the package (I don't know). I got this subject header back:

"[C516-004 public] Programming style of exception raising in GNAT.Sockets,
   etc."

So similarly, GNAT.Sockets could be altered so that the comments in it
suggesting use of Ada 95's X'Read feature, are deleted or discouraged.

X'Read() has no parameter for diagnostics on what the malfunction was.
The language can be revised so that X'Read is deleted or else it returns
error information out directly so that it is available to very next
line in the source code. If Ada becomes a language where malfunctioning
programs don't know why, and are really cryptic in describing the
precise problem (e.g. since code in exception handlers tries to handle too
many errors), then it could be all made reasonable (insufficiently
reasonable) by the new style of new unneeded code that seems to do
ntohing to a single byte but just lose the error info so that the next
line can't get it (it has to be the line after if the code is in an
exception handler).

However ACT is saying that customers do not seek big changes in
GNAT.Sockets.


Craig Carey
http://www.ijs.co.nz/ada_95.htm <- mailing lists

Re: 'Write attribute vs Write procedure.

[Simon Wright]

Craig Carey <research@ijs.co.nz> writes:

> So similarly, GNAT.Sockets could be altered so that the comments in it
> suggesting use of Ada 95's X'Read feature, are deleted or discouraged.
> 
> X'Read() has no parameter for diagnostics on what the malfunction was.
> The language can be revised so that X'Read is deleted or else it returns
> error information out directly so that it is available to very next
> line in the source code. If Ada becomes a language where malfunctioning
> programs don't know why, and are really cryptic in describing the
> precise problem (e.g. since code in exception handlers tries to handle too
> many errors), then it could be all made reasonable (insufficiently
> reasonable) by the new style of new unneeded code that seems to do
> ntohing to a single byte but just lose the error info so that the next
> line can't get it (it has to be the line after if the code is in an
> exception handler).

I don't understand this ..

A "simple" solution to your problem (not to mine!) might be to have a
package -- _not_ a revised GNAT.Sockets, please! -- which doesn't deal
with Streams, but with something else.

You still have the option to (for example) encapsulate the data you're
actually interested in in a "datagram" preceded by length information,
so you can read the "datagram" into a Stream_Element_Array and then
internally stream from that (this is the approach we are adopting for
UDP sockets).

> However ACT is saying that customers do not seek big changes in
> GNAT.Sockets.

There is certainly a problem with reading classwide objects from UDP
sockets.

AI-248 and counting the elements in an external file

[Larry Kilgallen]

I suppose my lack of experience with Ada95 streams ('read, etc.)
may be the reason for my confusion, but in reading

	http://www.ada-auth.org/cgi-bin/cvsweb.cgi/AIs/AI-00248.TXT?rev=1.16

I see in several places the text:

	The size of an external file is the number of
	stream elements contained in the file.

That is straightforward for a file of some record type, or a file of
integers, but what if the file is one created with Ada95 stream semantics ?
I thought that 'write and write() relied upon the reading program to know
"how" to read the file, and that implementations would not have kept a
tally of how many elements were in the file.

I don't know enough about this to know if it is possible to do a linear
search of a file to count the elements, but even if it were, that would
be a very slow operation.

Is it envisioned by the authors of AI-248 that implementations would
preserve metadata containing the element count ?  If so, what about
stream files that were written under Ada95 ?  Or can someone suggest
what it is that I don't understand about Ada streams ?

Re: AI-248 and counting the elements in an external file

[Stephen Leake]

Kilgallen@SpamCop.net (Larry Kilgallen) writes:

> I suppose my lack of experience with Ada95 streams ('read, etc.)
> may be the reason for my confusion, but in reading
> 
> 	http://www.ada-auth.org/cgi-bin/cvsweb.cgi/AIs/AI-00248.TXT?rev=1.16
> 
> I see in several places the text:
> 
> 	The size of an external file is the number of
> 	stream elements contained in the file.

a "stream element" is just a "byte"; it's Ada-ese for "smallest
writeable chunk". On all "normal" OS's, it's 8 bits.

I think perhaps you are thinking of "stream element" as some
higher-level object?

> That is straightforward for a file of some record type, or a file of
> integers, but what if the file is one created with Ada95 stream
> semantics ? 

It makes no difference. All of these types must be converted to raw
stream elements (bytes) before they can be written to the file. That's
the nature of files!

> I thought that 'write and write() relied upon the reading program to
> know "how" to read the file, and that implementations would not have
> kept a tally of how many elements were in the file.

The OS certainly has to know how big the file is; that's all that
matters here.

> I don't know enough about this to know if it is possible to do a
> linear search of a file to count the elements, but even if it were,
> that would be a very slow operation.

Yes, but the OS keeps track.

> Is it envisioned by the authors of AI-248 that implementations would
> preserve metadata containing the element count ? 

Only if the underlying OS doesn't; in which case this whole package
makes no sense anyway.

> If so, what about stream files that were written under Ada95 ? 

I don't see why you think stream files are different from other files,
as far as this package is concerned.

> Or can someone suggest what it is that I don't understand about Ada
> streams ?

-- 
-- Stephe

Re: AI-248 and counting the elements in an external file

[Randy Brukardt]

Stephen Leake wrote in message ...
>Kilgallen@SpamCop.net (Larry Kilgallen) writes:
>
>> I suppose my lack of experience with Ada95 streams ('read, etc.)
>> may be the reason for my confusion, but in reading
>>
>> http://www.ada-auth.org/cgi-bin/cvsweb.cgi/AIs/AI-00248.TXT?rev=1.16
>>
>> I see in several places the text:
>>
>> The size of an external file is the number of
>> stream elements contained in the file.
>
>a "stream element" is just a "byte"; it's Ada-ese for "smallest
>writeable chunk". On all "normal" OS's, it's 8 bits.
>
>I think perhaps you are thinking of "stream element" as some
>higher-level object?
>
>> That is straightforward for a file of some record type, or a file of
>> integers, but what if the file is one created with Ada95 stream
>> semantics ?
>
>It makes no difference. All of these types must be converted to raw
>stream elements (bytes) before they can be written to the file. That's
>the nature of files!


Right, the intent is that this value is whatever the OS returns for the
file size. That's true for all of the results of this package (with a
few exceptions). There is no attempt to tie the file size to any other
Ada semantics (that would be a mess).

            Randy.

Re: AI-248 and counting the elements in an external file

[Larry Kilgallen]

In article <u4r3l5pvw.fsf@nasa.gov>, Stephen Leake <Stephe.Leake@nasa.gov> writes:
> Kilgallen@SpamCop.net (Larry Kilgallen) writes:
> 
>> I suppose my lack of experience with Ada95 streams ('read, etc.)
>> may be the reason for my confusion, but in reading
>> 
>> 	http://www.ada-auth.org/cgi-bin/cvsweb.cgi/AIs/AI-00248.TXT?rev=1.16
>> 
>> I see in several places the text:
>> 
>> 	The size of an external file is the number of
>> 	stream elements contained in the file.
> 
> a "stream element" is just a "byte"; it's Ada-ese for "smallest
> writeable chunk". On all "normal" OS's, it's 8 bits.
> 
> I think perhaps you are thinking of "stream element" as some
> higher-level object?

Yes, that was my problem.

>> I don't know enough about this to know if it is possible to do a
>> linear search of a file to count the elements, but even if it were,
>> that would be a very slow operation.
> 
> Yes, but the OS keeps track.

On VMS that would be true if one writes a fixed-length record file,
or a stream-lf, stream-cr or stream-crlf file, but not if one is
writing a (VMS default) variable length record file.
> 
>> Is it envisioned by the authors of AI-248 that implementations would
>> preserve metadata containing the element count ? 
> 
> Only if the underlying OS doesn't; in which case this whole package
> makes no sense anyway.

In DEC Ada83, there is no function to return "bytes in a file",
and certainly an Ada program reading a normal VMS file would not
know down to the byte level how large a file is if it had been
written by a program from another language.

File length statistics on VMS are kept in disk blocks, and the number
of actual data bytes in standard (variable length record) files will
vary according to the number of records, whether they have odd or even
lengths, and whether they evenly fill an integral number of blocks.

Does anyone reading this use GNAT Ada95 on VMS, and know whether they
have dealt with any of this (I thought they had some GNAT-specific
directory functions).

Re: AI-248 and counting the elements in an external file

[Larry Kilgallen]

In article <vct5imh4199m7f@corp.supernews.com>, "Randy Brukardt" <randy@rrsoftware.com> writes:
> Stephen Leake wrote in message ...
>>Kilgallen@SpamCop.net (Larry Kilgallen) writes:

>>> That is straightforward for a file of some record type, or a file of
>>> integers, but what if the file is one created with Ada95 stream
>>> semantics ?
>>
>>It makes no difference. All of these types must be converted to raw
>>stream elements (bytes) before they can be written to the file. That's
>>the nature of files!
> 
> 
> Right, the intent is that this value is whatever the OS returns for the
> file size. That's true for all of the results of this package (with a
> few exceptions). There is no attempt to tie the file size to any other
> Ada semantics (that would be a mess).

Meaning it is permissible to return the number of 512-byte disk blocks
(or file-specific-size tape blocks) for this function on VMS ?  Or is the
answer supposed to be normalized to bytes even though the number of
overhead bytes required (and thus the file size) will vary, for instance,
when copying a file from disk to tape ?

Re: 'Write attribute vs Write procedure.

[Craig Carey]

On Wed, 21 May 2003 02:07:05 +0200, Jano <nono@celes.unizar.es> wrote:

>Hi everybody.
>
>I'm getting abysmal differences in speed when writing large chunks (64k) 
>of data to a stream. The stream is from a socket of Gnat.Sockets, I 
>mention it because I don't know if it has something to do.
>
...

>In the contrary, the 'Write attribute takes like 3-4 seconds of 100% CPU 
>usage, shared more or less equally by my program and the SYSTEM process, 
...
>P.s2: I've tried also with strings, only the 'Write version of course, 
>and results are the same.

I guess that the ACT vendor designed that part of their Ada compiler to
be intolerably inefficient. It would not affect authors of programs
written in a fairly good style.

That speculation could be tested by:
(1) finding out where ACT could have speeded up their X'Write() for TCP
 data, by adding a "pragma Suppress()" pragma statement.
(2) Then that information would be reported in the form of a bug report.
(3) Then the speculation is suggested to be incorrect if a tracking number
  comes back in the tailored response if any.

If the text in the GNAT.Sockets package caused a loss of time in doing
timing tests, then a bug report about the comments in the package could
be sent in to ACT. 

If the topic is about TCP Ada streams then it probably about extremely
simple programs that might be much modified later.


Craig Carey
Bindings to GNAT.Sockets: http://www.ijs.co.nz/code/ada95_http_hdrs.zip

Re: AI-248 and counting the elements in an external file

[Robert I. Eachus]

Larry Kilgallen wrote:

> Meaning it is permissible to return the number of 512-byte disk blocks
> (or file-specific-size tape blocks) for this function on VMS ?  Or is the
> answer supposed to be normalized to bytes even though the number of
> overhead bytes required (and thus the file size) will vary, for instance,
> when copying a file from disk to tape ?

Let me answer differently.  VMS supports many file formats.  Ada stream 
files would map nicely to VMS record oriented ANSI tape files.  There 
are other VMS formats that I would not tend to support stream files for. 
  You could do the blocking and unblocking in the file system, but why 
not use the OS to do what it does best, and only try to read and write 
file types that support stream semantics as stream files.

What if the program needs one type of file, and you have the data in 
another?  Write a program to read the file in and write it in as the 
necessary file type.  Won't be the first time I had to do that.  I 
remember one case where I had to distribute a data file to several 
projects.  One file format covered all the Unix systems, but I needed 
three different formats for the three groups with VAXen.  One of them 
actually overrode the default block size with 4K. (To save tape.) Argh!

Re: AI-248 and counting the elements in an external file

[Larry Kilgallen]

In article <3ECFEB59.4010401@attbi.com>, "Robert I. Eachus" <rieachus@attbi.com> writes:
> Larry Kilgallen wrote:
> 
>> Meaning it is permissible to return the number of 512-byte disk blocks
>> (or file-specific-size tape blocks) for this function on VMS ?  Or is the
>> answer supposed to be normalized to bytes even though the number of
>> overhead bytes required (and thus the file size) will vary, for instance,
>> when copying a file from disk to tape ?
> 
> Let me answer differently.  VMS supports many file formats.  Ada stream 
> files would map nicely to VMS record oriented ANSI tape files.  There 
> are other VMS formats that I would not tend to support stream files for. 

But my understanding is that the Ada.Directory functions are to support
all files that might be accessed from Ada, and that is not limited to
stream files, nor to files written by Ada.

Re: AI-248 and counting the elements in an external file

[Randy Brukardt]

Larry Kilgallen wrote in message ...
>In article <3ECFEB59.4010401@attbi.com>, "Robert I. Eachus"
<rieachus@attbi.com> writes:
>> Larry Kilgallen wrote:
>>
>>> Meaning it is permissible to return the number of 512-byte disk
blocks
>>> (or file-specific-size tape blocks) for this function on VMS ?  Or
is the
>>> answer supposed to be normalized to bytes even though the number of
>>> overhead bytes required (and thus the file size) will vary, for
instance,
>>> when copying a file from disk to tape ?
>>
>> Let me answer differently.  VMS supports many file formats.  Ada
stream
>> files would map nicely to VMS record oriented ANSI tape files.  There
>> are other VMS formats that I would not tend to support stream files
for.
>
>But my understanding is that the Ada.Directory functions are to support
>all files that might be accessed from Ada, and that is not limited to
>stream files, nor to files written by Ada.

The intent is that it be well-defined (which might include raising
Use_Error if if cannot be figured out, as for a keyboard) for files that
can be read or written by Ada. If Ada doesn't understand the files at
all, then we're not trying to say anything.

I would expect that this functionality would take more work on some OSes
than it does on Windows or Unix. On the original CP/M, for instance, you
had to figure out the file size by finding out the block size and
multiplying it by the number of blocks. That's fine, and keep in mind
that this result really is intended to report the physical size of the
file, which may or may not be directly relatable to the 'logical' size
returned by Direct_IO.Size and Stream_IO.Size.

             Randy.

Re: AI-248 and counting the elements in an external file

[Larry Kilgallen]

In article <vd7hvtd5dkqr17@corp.supernews.com>, "Randy Brukardt" <randy@rrsoftware.com> writes:
> Larry Kilgallen wrote in message ...
>>In article <3ECFEB59.4010401@attbi.com>, "Robert I. Eachus"
> <rieachus@attbi.com> writes:
>>> Larry Kilgallen wrote:
>>>
>>>> Meaning it is permissible to return the number of 512-byte disk
> blocks
>>>> (or file-specific-size tape blocks) for this function on VMS ?  Or
> is the
>>>> answer supposed to be normalized to bytes even though the number of
>>>> overhead bytes required (and thus the file size) will vary, for
> instance,
>>>> when copying a file from disk to tape ?

> The intent is that it be well-defined (which might include raising
> Use_Error if if cannot be figured out, as for a keyboard) for files that
> can be read or written by Ada. If Ada doesn't understand the files at
> all, then we're not trying to say anything.
> 
> I would expect that this functionality would take more work on some OSes
> than it does on Windows or Unix. On the original CP/M, for instance, you
> had to figure out the file size by finding out the block size and
> multiplying it by the number of blocks. That's fine, and keep in mind
> that this result really is intended to report the physical size of the
> file,

The idea of "physical size -- does not have to accurately represent
size of data contained" was not at all clear to me reading AI-248.
Is that stated in there, or are there plans for an AI- about AI-248 ?

Re: AI-248 and counting the elements in an external file

[Randy Brukardt]

Larry Kilgallen wrote in message
<$ghCR$mci5sd@eisner.encompasserve.org>...
>In article <vd7hvtd5dkqr17@corp.supernews.com>, "Randy Brukardt"
<randy@rrsoftware.com> writes:
>
>> The intent is that it be well-defined (which might include raising
>> Use_Error if if cannot be figured out, as for a keyboard) for files
that
>> can be read or written by Ada. If Ada doesn't understand the files at
>> all, then we're not trying to say anything.
>>
>> I would expect that this functionality would take more work on some
OSes
>> than it does on Windows or Unix. On the original CP/M, for instance,
you
>> had to figure out the file size by finding out the block size and
>> multiplying it by the number of blocks. That's fine, and keep in mind
>> that this result really is intended to report the physical size of
the
>> file,
>
>The idea of "physical size -- does not have to accurately represent
>size of data contained" was not at all clear to me reading AI-248.
>Is that stated in there, or are there plans for an AI- about AI-248 ?

Why would you expect otherwise? The 'size in Storage_Elements' for a
Direct_IO file certainly doesn't have to be the number of records
(Count) * the size of a record, even though it is often implemented that
way. There may be control information, or (like on the CP/M systems I
mentioned previously), there may be some rounding up. Certainly the
standard has nothing to say about this, and Ada.Directories is not
intended to change anything in this area. It was carefully written with
the intent of not adding any unintended requirements about file
representation, because that would be incompatible, and for no good
reason.

Anyway, AI-248 appears to require that Ada.Directories.Size =
Ada.Streams.Stream_IO.Size for a stream file, but that ought to not have
any implementation burden. But for any file type that is not a stream
file (that is, could not be Created by Stream_IO.Open) (or is a
non-positionable stream file), the meaning of Ada.Directories.Size is
essentially implementation defined, because the standard says nothing
about the representation of Sequential_IO, Text_IO, and Direct_IO files,
and thus the maping to stream elements is not defined by the standard.

You have to remember that the wording part of an AI is written to be
part of the standard. That means if it doesn't say something, that means
that the interpretation is being left to the implementor. It's not
practical to put every possible nuance into an AI; indeed, when I've
done that in the past, I've generally been asked to remove such
discussions as clutter. Too much clutter makes an AI hard to read and
digest.

If you have a specific suggestion for an improvement, I'd be happy to
hear it. (Even better, submit it to Ada-Comment to put it into the
official record.)

                   Randy Brukardt
                   ARG Editor

Re: AI-248 and counting the elements in an external file

[Brian Gaffney]

Kilgallen@SpamCop.net (Larry Kilgallen) wrote in message news:<X5uMu8X9J4P6@eisner.encompasserve.org>...
> In article <u4r3l5pvw.fsf@nasa.gov>, Stephen Leake <Stephe.Leake@nasa.gov> writes:
> > Kilgallen@SpamCop.net (Larry Kilgallen) writes:
> > 
> >> I suppose my lack of experience with Ada95 streams ('read, etc.)
> >> may be the reason for my confusion, but in reading
> >> 
> >> 	http://www.ada-auth.org/cgi-bin/cvsweb.cgi/AIs/AI-00248.TXT?rev=1.16
> >> 
> >> I see in several places the text:
> >> 
> >> 	The size of an external file is the number of
> >> 	stream elements contained in the file.
> > 
> > a "stream element" is just a "byte"; it's Ada-ese for "smallest
> > writeable chunk". On all "normal" OS's, it's 8 bits.
> > 
> > I think perhaps you are thinking of "stream element" as some
> > higher-level object?
> 
> Yes, that was my problem.
> 
> >> I don't know enough about this to know if it is possible to do a
> >> linear search of a file to count the elements, but even if it were,
> >> that would be a very slow operation.
> > 
> > Yes, but the OS keeps track.
> 
> On VMS that would be true if one writes a fixed-length record file,
> or a stream-lf, stream-cr or stream-crlf file, but not if one is
> writing a (VMS default) variable length record file.
> > 
> >> Is it envisioned by the authors of AI-248 that implementations would
> >> preserve metadata containing the element count ? 
> > 
> > Only if the underlying OS doesn't; in which case this whole package
> > makes no sense anyway.
> 
> In DEC Ada83, there is no function to return "bytes in a file",
> and certainly an Ada program reading a normal VMS file would not
> know down to the byte level how large a file is if it had been
> written by a program from another language.
> 
> File length statistics on VMS are kept in disk blocks, and the number
> of actual data bytes in standard (variable length record) files will
> vary according to the number of records, whether they have odd or even
> lengths, and whether they evenly fill an integral number of blocks.
> 

My suggestion for VMS would be for File_Size to represent the size in
blocks, since that is what the VMS user would be used to working with.
 This would imply File_Size would have an implementation-defined unit
as well as range.  (BTW why is stream elements used as the size
definition, shouldn't it be whatever is natural to the
implementation/OS enviroment?)  (Or should a stream element on VMS be
a block?:-)

File_Size isn't the only issue VMS would have with this package.  What
would an implementation do about Logicals in returning a Full_Name? 
Depending on how the Logical is defined, sometimes they are translated
to give the full path, and sometimes they are not (i.e. used like the
drive letter in DOS).  The user may never see the true "full paths",
and the system administrator may have it set up that way
intentionally.

Also, Set_Directory raises an exception if the target doesn't exist. 
This is not usual in VMS (from a user's perspective). I can SET DEF to
a nonexistant directory, and only get an error if I try to access it. 
(I have found this to be a feature, since I can still log in if the
drive my home directory is on has crashed.)

VMS is not the only enviroment where this package, as written, would
cause problems.  For example, the Discussion for the function Compose
states that a null Name is not valid, however a file with no Name and
an Extension (as defined in the AI), is allowed in DOS/Windows and is
actually a 'feature' in Linux (and Unix?).

                     --Brian

Re: AI-248 and counting the elements in an external file

[Larry Kilgallen]

In article <5e9b8c34.0305281317.3a6eb6a4@posting.google.com>, Brian.Gaffney@myrealbox.com (Brian Gaffney) writes:

> My suggestion for VMS would be for File_Size to represent the size in
> blocks, since that is what the VMS user would be used to working with.

But the last I read from Randy in this topic indicated to me the
expectation was to normalize to bytes.

> File_Size isn't the only issue VMS would have with this package.  What
> would an implementation do about Logicals in returning a Full_Name? 
> Depending on how the Logical is defined, sometimes they are translated
> to give the full path, and sometimes they are not (i.e. used like the
> drive letter in DOS).  The user may never see the true "full paths",
> and the system administrator may have it set up that way
> intentionally.

That part seemed clear to me in the VMS context, depending on the setting
of the "concealed" flag on the logical names.

> Also, Set_Directory raises an exception if the target doesn't exist. 
> This is not usual in VMS (from a user's perspective). I can SET DEF to
> a nonexistant directory, and only get an error if I try to access it. 

But it certainly is easy to program in the Ada package to make it obey
the specification.

> (I have found this to be a feature, since I can still log in if the
> drive my home directory is on has crashed.)

If one were to rewrite LOGINOUT in Ada :-), it could likewise be written
to ignore that exception.