Re: 10 rules for benchmarking (was Re: Teaching new tricks to an old dog (C++ -->Ada))

[Georg Bauhaus <sb463ba@user1.uni-duisburg.de>]

CTips wrote:

> Like I said, if one knows what one is doing (which he does) then one can 
> disregard the rules. If one does not know what one is doing (which you 
> don't) then one should follow the rules.

If one defines what is the right thing to do and does it
then there is nothing to argue about. OTOH, who says
who is doing right because they are referring to some machine
that allows them to explain that they know what they are
doing? This may sound like far fetched, but there are other
references, e.g. (semi-)formal economic, or organisational.
If
  "To produce a fast program, and a fast enough one,
   but not the fastest will be easier, and sooner,"
is true, then,
  "so do that"
is an argument outperforming every benchmark argument out
there (yes, the arguement is not always true).

If someone is utterly ignorant of anything inside the
computer on the desktop and they ask, "can this VB
program format 30 documents per hour?" then they know
very well what they are asking, and they are right when
they are suspicious of someone explaining to them that
this program is wasting a dose of valuable instruction
cycles and that with proper knowledge, language, and effort,
the same box could format 110 documents per hour.
   In the commercial frame of reference, _this_ is
ignorant in their view, because in this case the benchmarkers
don't follow the rules by wasting brain cycles on
silly speed things ;-).

Some people do know (very well) how to write a benchmark,
I'm not one of them. This does not however imply that
a benchmark incorporates all the references to the important
things to measure or to discuss. E.g. a compiler has a
"human user interface", and its output is not just
performance. A compiler's  input is, de facto, not just
instructions to a specific computing machine.
So combining "human user interface" and "not just
instructions" yields guidance as to what should be
measured when comparing relative merits of a programming
language in a given situation.


Georg