Benchmarking

Benchmarking

[ravege]

I have a program that exports binary records to text format.  A sample
record might be 196MB and the resulting export text file 607MB.  I've
been using Text_Io to write out the text file, but inspired by the K-
Nucleotide thread decided to give stream_io a try.  So now I want to
compare.  I run the old Text_Io version and it takes 133 seconds to
complete.  I run the Stream_Io version, it completes in 88 seconds!
Nice improvement, I want to get some more test points so I run the old
version again and get a 99 second time, okay... I run the new version
and get a 145 second time to completion.  Is there some optimization I
should be turning on/off, some OS caching I can configure so I can get
less variation in my testing?  I'm compiling with GNAT 20080521 build
on Windows XP.

Re: Benchmarking

[Yannick Duchêne Hibou57]

On 10 août, 17:39, ravege <mhamel...@yahoo.com> wrote:
> Is there some optimization I
> should be turning on/off, some OS caching I can configure so I can get
> less variation in my testing?  I'm compiling with GNAT 20080521 build
> on Windows XP.

Ensure you do not have any other program running, stop any uneeded
processes, and run the program to test multiple times before you
retain any measures.

But this will still not be a more real measurement of anything,
beceause all the variations you may get in other context... are still
valid measures too.

You may think about question like : is the program to be executed
alone on a system ? Is it to be launched from time to time on a system
which is most of time busy to something else ?

Think about the execution context as a part of the stuff

Then, you've just tested on Windows XP, so be sure you will get
different result on an other system. Do not see it as a trouble. Each
system is focused on something specific, whih its own runtime
strategy, so if the efficiency of your application vary depending of
each system specific properties (and on the fly state), this is just a
normal thing.

Re: Benchmarking

[Georg Bauhaus]

ravege wrote:
> I have a program that exports binary records to text format.  A sample
> record might be 196MB and the resulting export text file 607MB.  I've
> been using Text_Io to write out the text file, but inspired by the K-
> Nucleotide thread decided to give stream_io a try.  So now I want to
> compare.  I run the old Text_Io version and it takes 133 seconds to
> complete.  I run the Stream_Io version, it completes in 88 seconds!
> Nice improvement, I want to get some more test points so I run the old
> version again and get a 99 second time, okay... I run the new version
> and get a 145 second time to completion.  Is there some optimization I
> should be turning on/off, some OS caching I can configure so I can get
> less variation in my testing?

A dumb measure might be to write a fresh file that the
OS hasn't seen yet.  Otherwise, could it be possible to
formalize Yannick's observations into a testing strategy
that would measure either program in a loop, for a a few
hours or so?

Re: Benchmarking

[Gautier write-only]

Hello!
Here is the generic Timing procedure I'm using for "shooting-out" a
bit. The instanciation for one test holds then in one line:
  with Timing, Fasta_O, Fasta; procedure Fasta_Timing is new Timing
( 100, "Fasta 2005", "Fasta 2009", Fasta_O, Fasta);
HTH
Gautier

-- Timing of two algorithms P1, P2 (wall clock time).
-- P1 and P2 are run "runs" times (+1 as "tour de chauffe").

generic
  runs: Positive;
  n1, n2: String;
  with procedure P1;
  with procedure P2;
procedure Timing;

with Ada.Text_IO;                       use Ada.Text_IO;
with Ada.Calendar;                      use Ada.Calendar;
with Ada.Numerics.Float_Random;         use Ada.Numerics.Float_Random;

procedure Timing is
  a1,z1,a2,z2: Time;
  f, t1s,t2s: Long_Float;
  package LFIO is new Ada.Text_IO.Float_IO(Long_Float); use LFIO;
  gen: Generator;
begin
  Put_Line("Heating..."); -- Skip (most of) cache effects
  P1;
  P2;
  -- Now we start for real:
  t1s:= 0.0;
  t2s:= 0.0;
  Reset(gen);
  for run in 1..runs loop
    Put_Line(Standard_Error, "========= Run" & run'img & " of" &
runs'img);
    if Random(gen) > 0.5 then
      a1:= Clock;  P1;  z1:= Clock;
      a2:= Clock;  P2;  z2:= Clock;
    else
      a2:= Clock;  P2;  z2:= Clock;
      a1:= Clock;  P1;  z1:= Clock;
    end if;
    Put_Line(Standard_Error, "Time in seconds for...");
    Put(Standard_Error, "P1 ("&n1&"): " & Duration'Image(z1-a1));
    Put(Standard_Error, " -- average so far: ");
    Put(Standard_Error, t1s / Long_Float(run),0,5,0);
    New_Line(Standard_Error);
    Put(Standard_Error, "P2 ("&n2&"): " & Duration'Image(z2-a2));
    Put(Standard_Error, " -- average so far: ");
    Put(Standard_Error, t2s / Long_Float(run),0,5,0);
    New_Line(Standard_Error);
    f:= Long_Float(z2-a2) / Long_Float(z1-a1);
    Put(Standard_Error, "\--> time factor (T2/T1): "); Put
(Standard_Error, f,0,5,0);
    t1s:= t1s + Long_Float(z1-a1);
    t2s:= t2s + Long_Float(z2-a2);
    Put(Standard_Error, " -- average so far: ");
    Put(Standard_Error, t2s / t1s,0,5,0);
    New_Line(Standard_Error);
  end loop;
  -- Put("[press return]"); Skip_Line;
end;