Advanced file manipulation (multiple question)

Advanced file manipulation (multiple question)

[Steph-ADA]

Hello, all!
I'm new on this board so please excuse my lack of habits!
I have some question about how manipulating files in ADA. Here's a
kind of list, if you have any informations about how solving these
problems, it would be very kind!

- If I want to copy a file, if it's a 10kb gif file or a big 100Mb avi
one, should I consider it as binary files, and use the same
Ada.sequential_io and Ada.direct_io packages? Or is there special
methods to copy/edit/write in/delete large files?

- Do I need to verify the integrity of the copy? CRC-error control is
possible with ADA? or another I don't know for the moment... ?

Next, few another questions, maybe :)
Thanks a lot for your help !

Steph

Re: Advanced file manipulation (multiple question)

[Peter Hermann]

Steph-ADA <france.suisse@gmail.com> wrote:
> Hello, all!
> I'm new on this board so please excuse my lack of habits!
> I have some question about how manipulating files in ADA. Here's a

Steph-ADA writes "ADA". ok. lack of habit. Has nothing to do with Ada
;-)

> kind of list, if you have any informations about how solving these
> problems, it would be very kind!
> 
> - If I want to copy a file, if it's a 10kb gif file or a big 100Mb avi
> one, should I consider it as binary files, and use the same
> Ada.sequential_io and Ada.direct_io packages? Or is there special

better use streams: Ada95LRM A.12

Re: Advanced file manipulation (multiple question)

[Larry Kilgallen]

In article <74a78c42.0503010130.785f178f@posting.google.com>, france.suisse@gmail.com (Steph-ADA) writes:

> - If I want to copy a file, if it's a 10kb gif file or a big 101Mb avi
> one, should I consider it as binary files, and use the same
> Ada.sequential_io and Ada.direct_io packages? Or is there special
> methods to copy/edit/write in/delete large files?

Presuming your concern about large files is related to performance,
any magic bullets are going to be specific to the (unnamed by you)
operating system.  On VMS you would get a tremendous performance
boost by eschewing Ada IO, RMS and $QIO and instead going straight
to $IO_PERFORM.  Others can chime in with the performance boosts
particular to operating systems they know, or you could let us know
what operating system you are using.  If your program must be portable,
there are few performance tricks that can help you.

> - Do I need to verify the integrity of the copy? CRC-error control is
> possible with ADA? or another I don't know for the moment... ?

CRC would do something if the file involved is stored with a CRC
value.  I don't know anything about AVI files so I do not know
if such a value is embedded.

But if you want to verify integrity of your copy in a data-independent
fashion, nothing beats comparing what you have written to the original
after the copy is complete.

Consider the possible places where corruption of file data might occur:

  1.	Before the original got to your disk

  2.	While the original was on your disk

  3.	Reading from your disk into memory

  4.	Manipulating in Ada

  5.	Writing from memory to your disk

  6.	After the copy is on your disk

No one technique covers all of those, and if AVI does not have an
embedded checksum, you have no hope on numbers 1, 2 or 6.

Comparison after the fact will cover 3-5.

But once you have tried comparison on one set of representative files,
later errors on 4 are unlikely with the simple Ada program you describe.

Re: Advanced file manipulation (multiple question)

[Jeffrey Carter]

Steph-ADA wrote:

> I'm new on this board so please excuse my lack of habits!
> I have some question about how manipulating files in ADA. Here's a
> kind of list, if you have any informations about how solving these
> problems, it would be very kind!

It's Ada, named for Ada King, Countess Lovelace, who wrote programs for 
  Babbage's Analytical Engine.

> - If I want to copy a file, if it's a 10kb gif file or a big 100Mb avi
> one, should I consider it as binary files, and use the same
> Ada.sequential_io and Ada.direct_io packages? Or is there special
> methods to copy/edit/write in/delete large files?

Probably the best way is to make an OS call for this. That won't be 
portable at the moment, but should be when Ada 0X becomes available, you 
should be able to use Ada.Directories.Copy_File portably.

If you must do this portably in Ada, one way is to use 
Ada.Streams.Stream_IO to open the source file and obtain the size of the 
file, then read and write chunks of the file that will fit in memory. 
When there's not enough data left in the file to fill a chunk, decrease 
the chunk size by a large factor. Repeat until you're copying the last 
few (< 10) bytes byte-by-byte (technically, Stream_Elements, one-by-one).

-- 
Jeff Carter
"Ada has made you lazy and careless. You can write programs in C that
are just as safe by the simple application of super-human diligence."
E. Robert Tisdale
72

Re: Advanced file manipulation (multiple question)

[Steve]

Two solutions come to mind:
  1) Make a call to the underlying OS to copy the file.

  2) Use Ada.Direct_IO.

If you use Ada.Direct_Io, there is a nice way to read the file in one read, 
but it takes two steps.
  First create an instance of Ada.Direct_Io of bytes.
  Open the file as bytes.
  Use the size function to get the size of the file.
  Close the file
  Create an instance of Ada.Direct_IO with an element that is the size of 
the file (an array of bytes maybe?).
  Read the file (a single read)
  Close the file.

This approach tends to be quite fast since there is no iteration involved 
over the bytes of the file.

Steve
(The Duck)


"Steph-ADA" <france.suisse@gmail.com> wrote in message 
news:74a78c42.0503010130.785f178f@posting.google.com...
> Hello, all!
> I'm new on this board so please excuse my lack of habits!
> I have some question about how manipulating files in ADA. Here's a
> kind of list, if you have any informations about how solving these
> problems, it would be very kind!
>
> - If I want to copy a file, if it's a 10kb gif file or a big 100Mb avi
> one, should I consider it as binary files, and use the same
> Ada.sequential_io and Ada.direct_io packages? Or is there special
> methods to copy/edit/write in/delete large files?
>
> - Do I need to verify the integrity of the copy? CRC-error control is
> possible with ADA? or another I don't know for the moment... ?
>
> Next, few another questions, maybe :)
> Thanks a lot for your help !
>
> Steph 

Re: Advanced file manipulation (multiple question)

[Steph-ADA]

Hello!

Thanks for your helpful advices, and sorry for 1/my english,
2/abscence of details and 3/my error about Lady Lovelace name, Ada
(and not ADA or anything else). I Hope this next post would be better
:)

So, first, my OS: Windows 2000 Pro. I use Gnat 3.15 (public).
I started Ada as a student, with few advices and then... concurrent
programming the next week (semaphore etc...), that was rather... hard,
for beginners!
So, my excuse if I don't know how call OS functions... maybe you have
links where I can find tutorials about this ?

> If you use Ada.Direct_Io, there is a nice way to read the file in one read, 
> but it takes two steps.
>   First create an instance of Ada.Direct_Io of bytes.
>   Open the file as bytes.
>   Use the size function to get the size of the file.
>   Close the file
>   Create an instance of Ada.Direct_IO with an element that is the size of 
> the file (an array of bytes maybe?).
>   Read the file (a single read)
>   Close the file.

This answer seems easier for me, and I'll add a "compare-file"
procedure... I'm a bit paranoid :)

Thanks a lot for your help

Steph

Re: Advanced file manipulation (multiple question)

[Steve]

It sounds like you have downloaded Gnat 3.15p.
To make OS calls, you'll also want to get gnatwin-3.15p.exe
You can find it here:
  http://libre.act-europe.fr/GNAT/

gnatwin includes bindings to most of the Win32 API.

For example in the file win32-winbase.ads

You will find the binding for CopyFile.

   function CopyFile (lpExistingFileName : Win32.LPCSTR;
                      lpNewFileName : Win32.LPCSTR;
                      bFailIfExists : Win32.BOOL)
                     return Win32.BOOL
     renames CopyFileA; --  winbase.h :4919

It takes a little practice to get all of the right arguments set up with the 
right types to make these OS calls.  Fortunately you don't have to do it 
very often.

Steve
(The Duck)

"Steph-ADA" <france.suisse@gmail.com> wrote in message 
news:74a78c42.0503020231.1dc804b6@posting.google.com...
> Hello!
>
> Thanks for your helpful advices, and sorry for 1/my english,
> 2/abscence of details and 3/my error about Lady Lovelace name, Ada
> (and not ADA or anything else). I Hope this next post would be better
> :)
>
> So, first, my OS: Windows 2000 Pro. I use Gnat 3.15 (public).
> I started Ada as a student, with few advices and then... concurrent
> programming the next week (semaphore etc...), that was rather... hard,
> for beginners!
> So, my excuse if I don't know how call OS functions... maybe you have
> links where I can find tutorials about this ?
>
>> If you use Ada.Direct_Io, there is a nice way to read the file in one 
>> read,
>> but it takes two steps.
>>   First create an instance of Ada.Direct_Io of bytes.
>>   Open the file as bytes.
>>   Use the size function to get the size of the file.
>>   Close the file
>>   Create an instance of Ada.Direct_IO with an element that is the size of
>> the file (an array of bytes maybe?).
>>   Read the file (a single read)
>>   Close the file.
>
> This answer seems easier for me, and I'll add a "compare-file"
> procedure... I'm a bit paranoid :)
>
> Thanks a lot for your help
>
> Steph 

Re: Advanced file manipulation (multiple question)

[Jeffrey Carter]

Steve wrote:

> If you use Ada.Direct_Io, there is a nice way to read the file in one read, 
> but it takes two steps.
>   First create an instance of Ada.Direct_Io of bytes.
>   Open the file as bytes.

System.Storage_Elements.Storage_Element.

>   Use the size function to get the size of the file.
>   Close the file
>   Create an instance of Ada.Direct_IO with an element that is the size of 
> the file (an array of bytes maybe?).

System.Storage_Elements.Storage_Array.

>   Read the file (a single read)
>   Close the file.
> 
> This approach tends to be quite fast since there is no iteration involved 
> over the bytes of the file.

Right, but this is limited to files that will fit in memory. The streams 
approach I described using varying-size chunks will work for all files, 
and is equivalent to this approach for smaller files. In addition, it 
does not require opening the file twice.

-- 
Jeff Carter
"What I wouldn't give for a large sock with horse manure in it."
Annie Hall
42

Re: Advanced file manipulation (multiple question)

[Larry Kilgallen]

Since we seem to have lost the attribution that the original poster
was interested in Windows...

In article <QemVd.3382$L17.2090@newsread3.news.pas.earthlink.net>, Jeffrey Carter <spam@spam.com> writes:

> Right, but this is limited to files that will fit in memory. The streams 
> approach I described using varying-size chunks will work for all files, 
> and is equivalent to this approach for smaller files. In addition, it 
> does not require opening the file twice.

All Windows files ?

Certainly not all files.

Does that method preserve out-of-band record and page marks on other
operating systems ?

Certainly it does not preserve all the information, as would be the
case from calling an OS-specific procedure.

Re: Advanced file manipulation (multiple question)

[Jeffrey Carter]

Larry Kilgallen wrote:

> Since we seem to have lost the attribution that the original poster
> was interested in Windows...

We may not still be quoting it, but it's still part of the thread.

> Certainly not all files.
> 
> Does that method preserve out-of-band record and page marks on other
> operating systems ?

Not a factor for the OS of interest.

> Certainly it does not preserve all the information, as would be the
> case from calling an OS-specific procedure.

Certainly all files under the OS of interest.

-- 
Jeff Carter
"This school was here before you came,
and it'll be here before you go."
Horse Feathers
48

Re: Advanced file manipulation (multiple question)

[Martin Krischik]

Jeffrey Carter wrote:

> Larry Kilgallen wrote:

>> Certainly not all files.
>> 
>> Does that method preserve out-of-band record and page marks on other
>> operating systems ?
> 
> Not a factor for the OS of interest.

>> Certainly it does not preserve all the information, as would be the
>> case from calling an OS-specific procedure.

> Certainly all files under the OS of interest.

Sure? May I mention "Extended Attributes", "Access Controll Lists" and
"Multiple Data Streams" here - all which are available for NTFS.
 
Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: Advanced file manipulation (multiple question)

[Steph-ADA]

Wow! I didn't thought copying files would make so interesting debates
:)

If I read carefully, Copyfile is ok from Windows 95 to Windows 2000...
I mean FAT-32, but also FAT? And Windows NT 3.1, according to Mr
Randy... do you mean NTFS ?
Thanks a lot Mr Steve, for you piece of code! It must be interesting
for me (few stuff I don't really understand... but I will :) )

Mr Carter, I'm sure all the problems you talk about are really
important, but as I've said, I'm just a beginner (engineer, but
beginner ;) ). You probably think a program is good when all
situations have been taken in consideration, but... I just can't! Not
enought experience!! That's why this would be first an "alpha"
version... ;)

Well... maybe, the best I can tell you is what I want to do:
A proggy that would be able to make:
- recurrent directories: I say "n directories", and it build it, with
incremental name, from a list, or numbers.
- file copying: from a directory to another, single or multi-files.
- "deltree like": cut all the files from all the directories in a
single "root" directory, and delete all the others directories.

... with all the problem that could happen (from fat->ntfs, ntfs->fat,
file(s) size too big for copying, priority of task, crash system
prevention... and so on!)

You see, it's a challenge, and just for myself! Not any commercial
goal... maybe freeware, if it works :)

Thanks for your actual (and future, I'll need it!) help!

Re: Advanced file manipulation (multiple question)

[Jean-Pierre Rosen]

Steph-ADA a �crit :
> Well... maybe, the best I can tell you is what I want to do:
> A proggy that would be able to make:
> - recurrent directories: I say "n directories", and it build it, with
> incremental name, from a list, or numbers.
> - file copying: from a directory to another, single or multi-files.
> - "deltree like": cut all the files from all the directories in a
> single "root" directory, and delete all the others directories.

Shameless plug:
You should maybe consider using the package OS_Services, freely 
available from http://www.adalog.fr/compo2.htm

-- 
---------------------------------------------------------
            J-P. Rosen (rosen@adalog.fr)
Visit Adalog's web site at http://www.adalog.fr

Re: Advanced file manipulation (multiple question)

[Steph-ADA]

Back!

Here's a few notes:

- "It sounds like you have downloaded Gnat 3.15p.
To make OS calls, you'll also want to get gnatwin-3.15p.exe
You can find it here:
  http://libre.act-europe.fr/GNAT/"
---> Thanks a lot, Mr Steve ! 

- "Right, but this is limited to files that will fit in memory. The
streams
approach I described using varying-size chunks will work for all
files,
and is equivalent to this approach for smaller files. In addition, it 
does not require opening the file twice."
---> Mr Carter, in the case that my file or my packet of files is
bigger than the amount of RAM memory, do you mean It'll crash? In this
case, the "streams approach", as you call it, seems better. And does
this use OS-copying properties so I need to use win32-winbase.ads (or
another, I didn't read so much about Win32 API's at the moment!). And
what about the use of this tiny prog with Windows 98, for example?
APIs will probably be proper to the OS... but this is not the main
problem, for me! I want, first, to make it run with Windows 2000,
then, we'll see !

SO!
I'll use gnatwin, to make OS calls. I'll take a long time to read
about it, and then I'll give you what I've learned :)

All of those who helped me... thank you so much!!

PS: if you have other tricks, no problem I'm ok to read them !!!

Re: Advanced file manipulation (multiple question)

[Jeffrey Carter]

Steph-ADA wrote:

> ---> Mr Carter, in the case that my file or my packet of files is
> bigger than the amount of RAM memory, do you mean It'll crash? In this
> case, the "streams approach", as you call it, seems better. And does
> this use OS-copying properties so I need to use win32-winbase.ads (or
> another, I didn't read so much about Win32 API's at the moment!). And
> what about the use of this tiny prog with Windows 98, for example?
> APIs will probably be proper to the OS... but this is not the main
> problem, for me! I want, first, to make it run with Windows 2000,
> then, we'll see !

If you're willing to be tied to the OS, then using an OS call is 
probably the best approach. The other methods are for when you need to 
be portable among different platforms.

-- 
Jeff Carter
"When danger reared its ugly head, he bravely
turned his tail and fled."
Monty Python and the Holy Grail
60

Re: Advanced file manipulation (multiple question)

[Randy Brukardt]

"Steph-ADA" <france.suisse@gmail.com> wrote in message
news:74a78c42.0503030056.5995d08c@posting.google.com...
...
> And does
> this use OS-copying properties so I need to use win32-winbase.ads (or
> another, I didn't read so much about Win32 API's at the moment!). And
> what about the use of this tiny prog with Windows 98, for example?
> APIs will probably be proper to the OS... but this is not the main
> problem, for me! I want, first, to make it run with Windows 2000,
> then, we'll see !

According to the documentation in MSDN, CopyFile is available back to
Windows NT 3.1 and Windows 95. You're not going to be able to run a 32-bit
PE file on something older anyway. (Well, there is a way to do it on Windows
3.1 running on MS-DOS, but come on, who wants to do that these days?)

                                 Randy.

Re: Advanced file manipulation (multiple question)

[Steve]

Here is a (partial) example of binding to the Win32 copy file:

WITH Win32;
WITH Win32.WinBase;
WITH Win32.WinNT;
WITH Interfaces.C;

PACKAGE BODY DFUdiskFileUtils IS

  PACKAGE WinBase RENAMES Win32.WinBase;
  PACKAGE WinNT RENAMES Win32.WinNT;
  PACKAGE C RENAMES Interfaces.C;

  USE TYPE Win32.BOOL;
  USE TYPE Win32.DWORD;
  USE TYPE WinNT.HANDLE;

  PROCEDURE CopyFileDFU( sourceFileNameDFU,
                         destFileNameDFU : STRING;
                         copiedDFU : out BOOLEAN ) IS
    result     : Win32.BOOL;
    sourceName : ALIASED C.Char_Array := C.To_C( sourceFileNameDFU );
    destName   : ALIASED C.Char_Array := C.To_C( destFileNameDFU );
  BEGIN
    result := WinBase.CopyFile( sourceName(0)'UNCHECKED_ACCESS,
                                destName(0)'UNCHECKED_ACCESS,
                                Win32.FALSE );
    copiedDFU := result /= Win32.FALSE;
  END CopyFileDFU;

It's part of a package I made to wrap a number of OS dependent system calls.

Also... when Ada 2005 becomes available you will be able to use the 
Copy_File function in the package Ada.Directories.  There is a preliminary 
implementation at:

http://www.martin.dowie.btinternet.co.uk/

I haven't tried it, but I appears that you can add the implementation to 
your GNAT installation and have the Ada.Directories package with GNAT 3.15p.

In any case it is a good idea to at least be aware of what it takes to make 
OS calls, for those cases where you need to do something that isn't built 
into the standard libraries.

Steve
(The Duck)