Re: 10 rules for benchmarking (was Re: Teaching new tricks to an old dog (C++ -->Ada))

[CTips <ctips@bestweb.net>]

Pascal Obry wrote:
> CTips <ctips@bestweb.net> writes:
> 
> 
>>Since it appears that several people out here are making some very basic
>>mistakes when they report benchmark numbers, I thought I'd write a small note
>>on how to do benchmarking.
> 
> 
> Thanks a lot :)
> 
> Did you see my note that a good benchmark had to be done on the real
> application ? I pretty well know that a stupid benchmark like this means
> nothing. But well, you did (I think it was you, sorry if not) use this
> argument by bindly say that Ada was slow. This just because an unknown web
> site with very bad code (at least for the Ada part) was showing that Ada was
> lot slower!
> 
> At least I have showed that a decent Ada implementation (very close to the C++
> one) is running at the same speed. But I have not hours to spend to provide
> all the data as described in your post.
> 
> Pascal.
> 

Nope; your run-time was way too small for the number to be at all 
reliable. Try running the null program:
int main(void) { return 0; }
under cygwin/windows. On my machine, the best of 5 returns:
real    0m0.017s
user    0m0.030s
sys     0m0.000s

Contrast this with the matrix code [I removed the printf()]
real    0m0.018s
user    0m0.030s
sys     0m0.015s

In other words, the startup code dominates all the numbers, and you 
can't judge anything from it.

All you've proved is that you don't understand how to benchmark 
programs, which means that you've never had to *really* worry about 
performance.

Based on this thread, I think most of the Ada advocates here (and maybe 
most Ada programmers)
- don't know how a processor works
- don't know how to measure performance
- don't know how to program for performance
- don't know where Ada will hurt performance
- don't really care about performance

This is pretty much in line with my experience. The Ada advocates (I've 
known several) don't really know how to program for performance. All the 
people I know who program for performance pick C (if they have the choice).