implementation question about writing and reading from files

implementation question about writing and reading from files

[Francis Crick]

*** post for FREE via your newsreader at post.newsfeeds.com ***

Hi Ada people...I'm teaching myself Ada95, and trying to write programs I
actually want to use at the same time...

I've been messing around for a bit trying to figure out how things work, but
I'm starting to get a little frustrated so I thought i might seek help.

The component I'm writing is for managing files by manipulating small chunks
of them.  The basic idea is that if the component is eventually given every
chunk of an original file, it will be able to reconstruct the original file.
The chunks will be roughly 1mb and the files are probably all going to be
100+mb....

so in addition to other things i'm trying to write procedures that:

read a chunk of any size from any part of a file and write it into memory
and pass back the pointer.

write a chunk of any size into any part of any file.

I haven't really done a lot of this type of work in any other languages, so
I'm not totally sure how I want to do this...i usually code on a much higher
level...

should I use streams?  If so, which stream components?  Also, whats the best
format to keep a big bunch of bytes in memory?  Should that be a stream of
some sort, or a big array or what?

Any suggestion, links, pointers, specs, sample code, or encouraging comments
would be much appreciated.

Thanks,

Francis Crick










-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----==  Over 90,000 Newsgroups - 16 Different Servers! =-----

Re: implementation question about writing and reading from files

[tmoran]

  I'd suggest Ada.Streams.Stream_IO

> read a chunk of any size from any part of a file and write it into memory
> and pass back the pointer.
    procedure Read (File : in     File_Type;
                    Item :    out Stream_Element_Array;
                    Last :    out Stream_Element_Offset;
                    From : in     Positive_Count);

> write a chunk of any size into any part of any file.
  assuming "any file" does not include, for instance, printer output,
    procedure Write(File : in     File_Type;
                    Item : in     Stream_Element_Array;
                    From : in     Positive_Count);

Re: implementation question about writing and reading from files

[Ted Dennison]

In article <3b6dc4ed@post.newsfeeds.com>, Francis Crick says...
>read a chunk of any size from any part of a file and write it into memory
>and pass back the pointer.
>
>write a chunk of any size into any part of any file.
>
>should I use streams?  If so, which stream components?  Also, whats the best
>format to keep a big bunch of bytes in memory?  Should that be a stream of
>some sort, or a big array or what?

I'd probably use streams for this, yes. See Tom Moran's message for the specific
calls. You could just use the stream attributes instead of Read and Write, but
that would create one I/O per byte, which can be rather slow.

---
T.E.D.    homepage   - http://www.telepath.com/dennison/Ted/TED.html
          home email - mailto:dennison@telepath.com

Re: implementation question about writing and reading from files

[Francis Crick]

I'm still really confused about these calls...anyone willing to offer me
some extra help?

I've been playing around with these calls but i really don't understand what
the parameters mean...obviously File_Type is the file i'm getting, but what
about the Item and Last parameters?  Is Stream_Element a generic type I have
to declare, and if so, how and where do I do that?  If not, how can I ensure
I'm getting the ammount of data out of the file that I want on any platform?

The From param in Read - what does this represent exactly?  from byte #?
from Stream_Element # in the stream?

And also, if I want to use a stream for keeping data in memory, how do i
that?  How do i instantiate a stream that I can just write to and read from
at my leisure?  Also, the data I'm moving is large, and chances are I won't
be using it for much else other than writing back to other file streams, so
how can i make the reading and writing as fast as possible?  Could I make it
read a whole mb in one go as a single unit?

Sorry most of these are no-brainers...I'm still a total Ada newbie and I
haven't really got a grasp of the way things work yet...

TIA!

Francis Crick

<tmoran@acm.org> wrote in message
news:j_jb7.33402$Kd7.20784576@news1.rdc1.sfba.home.com...
>   I'd suggest Ada.Streams.Stream_IO
>
> > read a chunk of any size from any part of a file and write it into
memory
> > and pass back the pointer.
>     procedure Read (File : in     File_Type;
>                     Item :    out Stream_Element_Array;
>                     Last :    out Stream_Element_Offset;
>                     From : in     Positive_Count);
>
> > write a chunk of any size into any part of any file.
>   assuming "any file" does not include, for instance, printer output,
>     procedure Write(File : in     File_Type;
>                     Item : in     Stream_Element_Array;
>                     From : in     Positive_Count);

Re: implementation question about writing and reading from files

[tmoran]

>the parameters mean...obviously File_Type is the file i'm getting, but what
>about the Item and Last parameters?  Is Stream_Element a generic type I have
>to declare, and if so, how and where do I do that?  If not, how can I ensure
>I'm getting the amount of data out of the file that I want on any platform?
>
>The From param in Read - what does this represent exactly?  from byte #?
>from Stream_Element # in the stream?
  You need a book, or look at www.adapower.com for the Ada 95 Reference
Manual (ARM).  A.12.1(31) says "The position of the first element in the
file is 1."  The Ada.Streams.Stream_IO Read and Write operate with
Stream_Element_Array's, which are defined in Ada.Streams, where ARM
13.13.1(4) says
  type Stream_Element is mod (implementation defined);
Probably a Stream_Element is a single byte, but there are weird systems
where it might be a 20 bit word or something.  Check your compiler's docs.

>How do i instantiate a stream that I can just write to and read from
>at my leisure?
  You can certainly do Ada.Streams.Stream_IO Read and Write at your
leisure.  There's nothing to instantiate, just use it.  It's even
simpler than Text/Sequential/Direct_IO.

>Also, the data I'm moving is large, and chances are I won't
>be using it for much else other than writing back to other file streams, so
>how can i make the reading and writing as fast as possible?  Could I make it
>read a whole mb in one go as a single unit?
  Ada.Streams.Stream_IO.Read/Write ought to be fast.  They don't do
any formatting or buffering or anything, just straightforward byte
(actually Stream_Element_Array) IO.  If you Declare
  Glob : Ada.Streams.Stream_Element_Array(1 .. 1_000_000);
you can read or write the whole thing in one call.

>And also, if I want to use a stream for keeping data in memory, how do i
>that?
  Not with Ada.Streams.Stream_IO.  There is something else entirely,
Ada.Streams, that handles automatically reading and writing everything
from a boolean to an array of tagged records.  It doesn't actually do
any IO - it just handles converting things to and from byte streams
(well, actually Stream_Element_Arrays) and calls your routine to do
the actual IO.  If you want to move things in memory, have your
routine do that, and not do any IO.

What is the level of abstraction here?  If you aren't looking at the
contents of the data, but just shoving it around, use
Ada.Streams.Stream_IO.  If you are looking at is as structured
information that you want to pay attention to, then use Ada.Streams
as well, or even use Ada.Direct_IO if it's just a series of fixed
size records.

Re: implementation question about writing and reading from files

[Ted Dennison]

In article <9kv5rs$6hosn$2@ID-102190.news.dfncis.de>, Francis Crick says...
>And also, if I want to use a stream for keeping data in memory, how do i
>that?  How do i instantiate a stream that I can just write to and read from
>at my leisure? 

The only language-defined streams are in Ada.Streams.Stream_IO,
Ada.Text_IO.Text_Streams, and Ada.Wide_Text_IO.Text_Streams. I believe they all
deal with files. If you want to use one to make yourself a buffer, you will have
to code it yourself by deriving it from Ada.Streams.Root_Stream_Type and
overriding "Read" and "Write". 

>               Also, the data I'm moving is large, and chances are I won't
>be using it for much else other than writing back to other file streams, so
>how can i make the reading and writing as fast as possible?  Could I make it
>read a whole mb in one go as a single unit?

The easiest way to do that is to unchecked_conversion your data (or better yet,
a pointer to it) into a Ada.Streams.Stream_Element_Array, then call
Ada.Streams.Write directly. If you just use the stream attributes, it will
perform one write for each element, which I've found to be significantly slower
(even when the target is a memory-based stream).

A good Ada book should go over the basics of this. Stream use is a complicated
enough topic that you really ought to have such a resource to learn the basics
from.

---
T.E.D.    homepage   - http://www.telepath.com/dennison/Ted/TED.html
          home email - mailto:dennison@telepath.com

Re: implementation question about writing and reading from files

[Warren W. Gay VE3WWG]

Ted Dennison wrote:
> In article <9kv5rs$6hosn$2@ID-102190.news.dfncis.de>, Francis Crick says...
> >And also, if I want to use a stream for keeping data in memory, how do i
> >that?  How do i instantiate a stream that I can just write to and read from
> >at my leisure?
> 
> The only language-defined streams are in Ada.Streams.Stream_IO,
> Ada.Text_IO.Text_Streams, and Ada.Wide_Text_IO.Text_Streams. I believe they all
> deal with files. If you want to use one to make yourself a buffer, you will have
> to code it yourself by deriving it from Ada.Streams.Root_Stream_Type and
> overriding "Read" and "Write".

If you want a number of examples of streams code, download the AdaVox software
from: http://members.home.net/ve3wwg

I am currently re-vamping AdaVox, because there are a number of design problems
in it, but in the originally released code (see above), you will find
some code that implements a stream buffer. Look for file wc-streams-buffer.ads.

I am sure you can improve upon it, but it will get you started at least.

> 
> >               Also, the data I'm moving is large, and chances are I won't
> >be using it for much else other than writing back to other file streams, so
> >how can i make the reading and writing as fast as possible?  Could I make it
> >read a whole mb in one go as a single unit?
> 
> The easiest way to do that is to unchecked_conversion your data (or better yet,
> a pointer to it) into a Ada.Streams.Stream_Element_Array, then call
> Ada.Streams.Write directly. If you just use the stream attributes, it will
> perform one write for each element, which I've found to be significantly slower
> (even when the target is a memory-based stream).

I think the GNAT implementation is built upon the stdio (C) package, ultimately.
One way to improve performance some, is to increase the buffer size of the 
underlying "file". You'll need to look at the GNAT User Guide to find out how.
You will need to open a C file, and then turn it into an Ada stream.
Maybe someone else can volunteer the details here (I'm too pressed for
time to look it up).

This won't compensate for the procedure(attribute) calls (that Ted referred to), 
but it will make each I/O a larger one.

> A good Ada book should go over the basics of this. Stream use is a complicated
> enough topic that you really ought to have such a resource to learn the basics
> from.

Otherwise, be prepared to look at the streams specs a lot (as I did).
-- 
Warren W. Gay VE3WWG
http://members.home.net/ve3wwg