From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: ** X-Spam-Status: No, score=2.6 required=5.0 tests=BAYES_40,INVALID_DATE, REPLYTO_WITHOUT_TO_CC autolearn=no autolearn_force=no version=3.4.4 Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!ukma!tut.cis.ohio-state.edu!ucbvax!logm1.logm.se!lm From: lm@logm1.logm.se ("Lennart M}nsson") Newsgroups: comp.lang.ada Subject: Performance Benchmarking Message-ID: <8908290951.AA01910@logm.se> Date: 30 Aug 89 07:01:39 GMT Sender: usenet@ucbvax.BERKELEY.EDU Reply-To: "Lennart M}nsson" Organization: The Internet X-Unparsable-Date: 29 Aug 89 11:31:00 GMT-1:00 List-Id: Mark Roseman, University of Manitoba, or , asks for methodological advice when doing performance benchmarking. I did some heavy testing a couple of years ago. It was on a VAX and not on UNIX but the problem with disturbing processes seems to be analog. My conclusion was that the only comparable measure is achieved when trying to eliminate the measurment noise as far as possible. Thus I always runned the tests on night time when no other people was supposed to use the system. I ALSO CONTROLLED THAT IT WAS IN FACT SO. Secondly I turned of all possibly interfering process, which seems to correspond you turning of all deamons running around. (If they are needed for your Ada program to run, well then they should be part of the mesaure) Thirdly I ran each test three times each night checking that I got stable measurment. ( I made up a script calculating means and issuing warnings for to big deviations, it also made printouts of the reports, be sure not to let them interfere with the next test, printing is often done in parallell with other program execution) Above this I repetead the tests another night to check long time stability. Now, if you only see to that resorces are as memory usage etc is returned to system for each test, you can fairly well rely that you mesure the capacity of your Ada code, under no disturbancies. What you really is interested in a how it perform on a normally loaded system. But then you can't compare figures between different implementations if you can't set up a standard load situation or do so many iterations that you can count the mean disturbancy as a standard situation. Testing is not easily done ! Lennart Mansson Telelogic AB, Box 4148, S-203 12 MALM\, SWEDEN Phone: +46-40-25 46 36 Fax: +46-40-25 46 25 E-mail: lm@logm.se ------