Re: Reading and writing a big file in Ada (GNAT) on Windows XP

[Quarc <quarc2000@hotmail.com>]

Fionn Mac Cumhaill wrote:
> On Sat, 28 Apr 2007 19:12:33 GMT, "Jeffrey R. Carter"
> <jrcarter@acm.org> wrote:
> 
>> Fionn Mac Cumhaill wrote:
>>> All it does is read lines in a loop from a text file with
>>> Ada.Text_IO.Get_Line, does minor modifications on about 80% of the
>>> lines that it reads, and writes the lines to an output file with
>>> Put_Line. 
>>>
>>> The modifications consist of replacing a slice of text at the end of a
>>> line with another bit of text. The biggest slice is 10 characters, and
>>> the replacement slice is always smaller than the original slice. An
>>> occasional line of text is about 6000 characters long, but most are
>>> about 700 haracters.  Get_Line reads them into a String variable that
>>> is 10,000 characters long.
>>>
>>> The problem is that the input file has more than 10 million lines of
>>> text in it. The program works perfectly, but takes about 5 hours to
>>> run. The Cygwin version of wc can count the lines in the input file in
>>> less than one minute.
>>>
>>> Why is this so slow? 
>>> Do I have an Ada problem, a GNAT problem, or a MinGW problem?

Sorry, but I don't have your full question here so trying to figure out 
your problem for the quote above. I woul assume the big difference 
between what you are doing and what WC does is that wc only reads the 
file, and can therefore do it in one go.

I am speculation now, but I think one explanation for this would be that 
you are reading, then writing each line by itself. Depending on the OS 
you are running on, and also the binding to the OS this could 
potentially create a loot of seek times on the disk. This could explain 
the long times I think. One of two things happen I think (or potentially 
both). 1) You read 700 bytes from disk, you write 700 bytes to disk. 
Assume average seek times of 5 ms and you get to 2*50 000 s = 100 000 s 
= 28 hours !!!! 2) Instead you creating the seektimes on the disk 
between reads, ther are other processes waiting for disk access in the 
OS as well. SO qite often some other process get to the top of the 
waiting queue, and can access disk, causing the seektimes.

If one of the above is causing your problem, the only solution is to 
read in larger chunks at once, and then write larger chunks of data as 
well. I don't remember how to set that up, and it is of course depending 
on compiler and OS, but hopefully you should be able to make sure you 
don't read line by line from disk.

regards

Peter Atterfj�ll
(haven't been working professionally with Ada for 14 years now :-)