Performance Benchmarking

[lm@logm1.logm.se ("Lennart M}nsson")]

Mark Roseman, University of Manitoba, <ROSEMAN@ccu.UManitoba.CA>
 or <ROSEMAN@UOFMCC.BITNET>, asks for methodological advice when 
doing performance benchmarking. 

I did some heavy testing a couple of years ago. It was on a VAX and 
not on UNIX but the problem with disturbing processes seems to be 
analog. 

My conclusion was that the only comparable measure is achieved when 
trying to eliminate the measurment noise as far as possible. Thus I 
always runned the tests on night time when no other people was supposed
to use the system. I ALSO CONTROLLED THAT IT WAS IN FACT SO. Secondly
I turned of all possibly interfering process, which seems to correspond 
you turning of all deamons running around. (If they are needed for your 
Ada program to run, well then they should be part of the mesaure)

Thirdly I ran each test three times each night checking that I got 
stable measurment. ( I made up a script calculating means and issuing 
warnings for to big deviations, it also made printouts of the reports, 
be sure not to let them interfere with the next test, printing is often 
done in parallell with other program execution) Above this I repetead 
the tests another night to check long time stability.  

Now, if you only see to that resorces are as memory usage etc is returned to 
system for each test, you can fairly well rely that you mesure the capacity
of your Ada code, under no disturbancies. What you really is interested in
a how it perform on a normally loaded system. But then you can't compare
figures between different implementations if you can't set up a standard
load situation or do so many iterations that you can count the mean 
disturbancy as a standard situation.

Testing is not easily done !

Lennart Mansson
Telelogic AB, Box 4148, S-203 12 MALM\, SWEDEN 
Phone:  +46-40-25 46 36
Fax:    +46-40-25 46 25
E-mail: lm@logm.se

------