From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.3 required=5.0 tests=BAYES_00,INVALID_MSGID autolearn=no autolearn_force=no version=3.4.4 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: 103376,b736111afc6e20ee X-Google-Attributes: gid103376,public From: phil.brashear@acm.org Subject: Re: Embedded Processor/Compiler Selection Date: 1998/02/25 Message-ID: <6d14ju$ue$1@nnrp1.dejanews.com>#1/1 X-Deja-AN: 328477744 References: <98022314312394@psavax.pwfl.com> Organization: Deja News - The Leader in Internet Discussion X-Article-Creation-Date: Wed Feb 25 12:59:42 1998 GMT Newsgroups: comp.lang.ada X-Http-User-Agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) X-Originating-IP-Addr: 199.228.142.7 Date: 1998-02-25T00:00:00+00:00 List-Id: In article , dewar@merv.cs.nyu.edu (Robert Dewar) wrote: > > This advice from Phil, as one of the creators of ACES, is not a surprise :-) > Right! As I said, this is my standard reply. > But I have a question, has anyone successfully used the latest version of the > full ACES suite. We had a couple of customers look at using it and decide > it was far too much work. One customer used the Quick-Look facility, but > I would not base much on the results from this, since it is a rather > miscellaneous collection of tests. Yes, I have exchanged information with various people who appear to be using ACES successfully. (Sometimes I wish they were just a little bit less successful, so they could pay me to help -- but that's rather mercenary of me, isn't it!) To cite one example of ACES use, the primary software developer on an ongoing major weapon used ACES to do formal evaluations of several Ada implementations as part of selecting an appropriate combination of platform and compiler. Yes, it's a lot of work, though less than previous versions, due to the much- improved user interface. The hardest part, of course, is deciding what you're trying to accomplish, and selecting the appropriate tests and analysis techniques. > > Any set of canned benchmarks is always a bit dubious. I think you usually > do better to create some test cases of your own, trying to mirror your > intended application as closely as possible. > One of the nice features of the ACES is that it provides a facility for inserting your own benchmarks into the ACES testing and analysis processes. It also provides a mechanism for selecting, processing, and analyzing subsets tailored to your needs. All this said, the fact remains that ACES is big, and the amount of information produced can be overwhelming if you aren't selective. Performance evaluation is a very tricky thing, not because ACES is hard to use, but because the concept of measuring performance is loaded with difficulties. The worst thing is that after all the work of selection, processing, analyzing, and understanding the results, you run into a manager who wants you to give him a single number representing the performance of each implementation. One might as well give such a manager the Douglas Adams answer: 42. Phil -----== Posted via Deja News, The Leader in Internet Discussion ==----- http://www.dejanews.com/ Now offering spam-free web-based newsreading