performance benchmarking

[roseman@ccu.UManitoba.CA (roseman)]

I'm involved in doing a bit of performance benchmarking work on a couple
of Ada compilers.  I'm wondering if there is anyone else out there doing
similar kind of work.  I've also got a few questions.

Right now we're using the PIWG (Performance Issues Working Group) test
suite to do the tests.  This seems to be "the" standard Ada test suite
out there.  I'm wondering first off if people are using other tests,
and if so, what?  (Furthermore, where did they come from, why are you
using them, etc?)

Second, the machine we're running PIWG on is a Unix based system.  With
the PIWG, we're running into some problems.  The tests themselves are
very very short - total times including iteration is well under a second
for most of them!

The problem with that is its almost impossible to get any accurate
measurements that way - you've got all the little Unix daemons popping in
and out and using up some time.  We have tests which vary from 0 usecs
to almost 4 (per iteration that is), which is most unacceptable!

What can you do to correct things?  Run tests 25 (e.g.) times and take
the best?  The average?  Increase the iteration count to some ridiculous
amount to try to compensate?

I guess this is getting into general benchmarking procedures (any digests
or lists devoted to this out there?).. but how are tests like this supposed
to be used?  Surely, this must be an old problem.  You have various companies
out there who are publishing PIWG numbers for their compilers, but what
are they measuring?  Is it reasonable to measure on a souped-up system
(e.g. high priority, kill the daeomons), or do people want to see results
on a real Unix system?

If anyone has any answers, comments, pointers to any papers covering these
issues, etc. I would very much like to hear from you.  I ask only that if
you post to the list that you also send a copy to my userid directly, as
my time is so tight these days I can't keep up with the digest.  Thanks.

Mark Roseman, University of Manitoba
<ROSEMAN@ccu.UManitoba.CA> or <ROSEMAN@UOFMCC.BITNET>