From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=BAYES_00,INVALID_DATE, LOTS_OF_MONEY,MSGID_SHORT autolearn=no autolearn_force=no version=3.4.4 Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!sample.eng.ohio-state.edu!cis.ohio-state.edu!sei.cmu.edu!ajpo!griest From: griest@ajpo.sei.cmu.edu (Tom Griest) Newsgroups: comp.lang.ada Subject: Re: Critique of SEI's Ada DARK project Summary: Comments on your critique. Message-ID: <781@ajpo.sei.cmu.edu> Date: 29 May 91 21:45:11 GMT References: Distribution: comp.lang.ada List-Id: In Article 5010 Gregory Aharonian writes: > I am giving a talk during the first week of June on cost/benefit > analysis and economics as applied to software reuse and engineering. One > example I plan to use is the Software Engineering Institute (SEI) Distributed > Ada Real-Time Kernel (DARK). My contention is that the costs in terms of tax > dollars far outweighed the benefits the country received in terms of any new > original information from the project' results. I am interested in what others > think about the cost and benefits of this project. First, I would not dream of giving a talk on something that you appear to have so many fuzzy facts about. If anyone in the audience knows about DARK you are likely to get ripped apart. Even if you get lucky and the audience accepts your guesses, because they are just guesses you are just as likely to give the wrong impression as the correct one. > The DARK project was started a few years ago at SEI('88 or '89), funded > by the Air Force's Electronic Systems Division. I do not know the exact > amount of money allocated, but assuming that the effort required at least four > man-years, had a small room of equipment to test the kernel, and average > overhead rates, I figure that it cost at least 325,000 dollars for this > project (four man years @ 50000, equipment at 50000 and overhead at 75000). If you want to do a true cost analysis, I suggest you find out exactly how many person-hours were spent on it and how much it cost (and who paid the bill). You must also study all of the spinoffs that resulted from this work if you want to be accurate. > [STUFF ABOUT DARK's PURPOSE DELETED] > After developing and experimenting with the code, the SEI tried > licensing the kernel to industry in 1990. I assume that this effort failed, > because in 1991, the Department of Commerce's National Technical Information > Service (NTIS) announced it was selling the DARK system (the cost is a few > hundred dollars for a magnetic tape and documentation). To the best of my > knowledge, this is the current official status of DARK. I have not examined > the code, but I assume that like other Ada code developed at SEI, the quality > of the Ada source code is very high and reliable. Licensing software source is an extremely tricky business. Since the mission of the SEI is to make information available for wide dissemination, I am not surprised that they decided to make data available via NTIS. I hope your not trying to imply that the project as a whole was a failure because they are providing the results through NTIS! Virtually all government funded projects provide their results through NTIS (or DTIC for classified data). > > As a background note, before, during and after this project, there > were (and still are) at least a half dozen private ventures concerning real > time Ada kernels, some from compiler companies like Alsys, some from real time > kernel vendors such as Ready Systems, a few university efforts and at least > one European Community funded project. Don't forget that DARK is specifically for DISTRIBUTED systems, and NOT simply a message passing mechanism on top of a network. However, many other researchers are also working on distributed operating systems. > > My question is, how did (or how could) the country benefit by this > investment? As with all research, the results can be very instructive to those WHO GET THE RESULTS AND STUDY THEM. Commercial vendors doing research, as you mention above, are very reluctant to publish the whole truth of their work (there is a financial disincentive unless it has been patented). Universities do provide useful information and sometimes at a lower cost because they occasionally "make do" with limited resources and often depend on graduate students to provide slave labor. However sometimes the results of the work reflects the inexperience of these graduate students. The SEI pays enough to get experienced people who are more likely to see subtle details that effect the results of studies than less experienced students. [ University folks please flame to /dev/null ... I think the vast majority of Univ. research is fine, but some of it could benefit from more review by industry. There is a need for both university and industrial researchers. Note also that SEI is run by CMU.] Since Ada is going through a revision process, the results of the SEI work could influence the Ada9X process. Alternatively is could provide guidance for others wanting to do similar projects with Ada. Even when research "fails" to meet it's objective, careful review of the work often provides ideas to other researches which benefit from the results of the prior projects. It's rather silly to suggest that because the results of a research project weren't watched like CNN during the first days of the Air War that the work was a waste. It is hard to predict the results of research and often the full results are not known for many years after the project has completed. > If the goal (as they stated in their documents) was to prove that it > is possible to build decent real-time, distributed Ada systems, then the DoD > and the country could have learned this by sitting back and watching the > activities of industry developments with Ada kernels, and actual embedded > projects, for free. A low cost effort to summarize the results of existing > activities (like the CECOM project) would have been sufficient. For this goal, > the net cost/benefit of DARK is negative. I would really like to see the equation you used to compute a negative number here. Other research which provides results "at a lower cost" does not indicate that the net benefit of DARK will not be positive. One point that must be made when observing work being done on "real" projects (that is, not research work) is that usually these projects do not have a time budget that allows finding out WHY everything operates they way it does. If a problem arises during a development, usually the quickest way to eliminate the problem is chosen. Often code is replaced with something "we know will work" rather than taking the time to find out why some "elegant solution" did not work. The emphasis is on coming up with any suitable solution, not an optimal one. If research was limited to the study of what developers did, we would probably use only the bubble sort for all sorting. Summaries of ongoing work and research are both needed to provide a balanced picture of the issues. > > An alternate goal might have been to have some SEI staff gain > experience in this area. However the DoD could have sent them to one of these > companies for training, at a much lower cost than what the DoD funded the SEI. > Also, there was a rather nice ARMY CECOM effort (finished in 1989) that > studied the problems of developing real time Ada code, and also offered Ada > source code examples, as well as over 1000 pages of documentation, at no cost > (contact Mary Bender at 201-544-2105). Again, for this goal, the net > cost/benefit of DARK is negative. Nothing is free. I can only guess that your "no cost" refers to the cost of getting the documentation. I know CECOM pays the people who do their research. The ARMY will sometimes pay for the duplication of reports, especially during periods when reports are being transitioned to NTIS. I suspect that these reports are now available from NTIS for a charge. I have also received free documentation from SEI. > > If a goal was to provide the country with an Ada real time kernel to > insure that people use Ada (which must have been an unstated goal since they > did try to license DARK to industry), then the venture failed because vendors > developed their own real time kernels, and recently most agreed to offer or > integrate in Ready Systems' real time kernel. Again, the cost/benefit of DARK > is negative. Besides, the government shouldn't be in the position of > developing products. In light of the fact (that you point out below) that Ready Systems is out of the Ada business, I fail to see the clear cost benefit in having a proprietary runtime which is no longer supported by the developer. > > If a goal was to provide a freely available, reusable piece of Ada > real-time source code, then someone screwed up by having DARK available thru > the NTIS, instead of being placed on the public archives on Internet. (It may > be that DARK is available from SIMTEL, but to the best of my knowledge, DARK > is not available by anonymous ftp from any Internet site). Again, the net > cost/benefit of DARK is negative. Not everyone has access to the net, but... Try anonymous ftp from: fg.sei.cmu.edu looks like you're the one who is in error. > > If a goal was to provide an example of a self-contained project whose > costs and benefits could be analyzed as an example of cost-benefit analysis, > a project of immeasurable benefit to the Ada community, well, I haven't seen > a single report from SEI (or an independent third party) performing such an > analysis. In fact, in March of 1989 the General Accounting Office, in a > report (GAO/IMTEC-89-9) on the general progress of Ada inside the DoD, > complained (and the DoD concurred) that no one is collecting the data to > justify the claims made on the benefits of reuse and Ada. The DARK project > might have been a good opportunity to do so. > There have been dozens of reports and public presentations on DARK. You may wish to get "Experiences Porting the Distributed Ada Real-Time Kernel", Brian Smith, Boeing Military Airplanes, June 1990 (Available through DTIC and NTIS: AD-A226 693, I think I paid $5 from DTIC.) I'm pretty sure that cost-benefit analysis of software reuse or government sponsored research were NOT the stated objectives of DARK. This is YOUR area of interest (and GAO's) and it seems you are not adequately prepared for it. You had better start with getting a way to measure software productivity. (When you figure that one out, let me know. I have a long list of people interested in THE answer!) > My contention is that while the country benefited from this project, > the costs to the country were much greater, and that this analysis could have > been performed before the project had ever started, by considering the social > and economic aspects of software development and reuse on a national scale. > All of the benefits could have been obtained at a much lower cost. There may > be other benefits to this project, or that the costs were lower, but they > escape me at the moment. Now 300,000 dollars is not much, but if similar waste > is going on elsewhere, it soon adds up to a lot of tax dollars. I can believe > great things about Ada, but I would like to see some numbers, especially when > I am paying for it. ".... by considering the social and economic aspects of software development and reuse on a national scale" ??? That certainly seems easy to do! -:) The computer industry is so volatile I don't see how a realistic cost analysis could be done. If it could be done, the cost analysis would certainly be as big a job as the research and probably cost more and have less reliable results. Do you know how many tons of money have been spent on actual development projects only to have the industry change direction before the product could get to market? Look at OS/2. I saw estimates by market forecasters in 1988 that it would be the OS on 70%-90% of the PC platforms by 1991. Now analysts say it is all but dead, being replace by MS Windows and Unix. Who knows? Will it be another PC Jr. or will it be what VMS used to be? > > (Of interest is that Ready Systems got out of the real-time Ada kernel > business because there was no profit in it as a product. Finding out why would > be a beneficial SEI study, and also why there are so few Ada library companies > and fewer that survive.) > How much would you pay for this study? Where is your cost-benefit analysis that this study would benefit American Tax Payers? I can probably guess the results of your study and save the tax payers $325,000: 1). The Ada compiler/support tool business is not exactly the "goose that laid the golden egg". 2). There is no standard interface between Ada compilers and the underlying runtime. This makes it costly to support a wide variety of compilers. (By the way, this issue may be addressed by Ada9X.) 3). Runtime vendors are somewhat dependent on the compiler vendors to obtain a customer base. As soon as the runtime vendor's business begins to show a large profit, the compiler vendor may invest the money it takes to provide nearly the same capability from their "new high-performance" runtime. It is extremely hard to compete against the compiler vendor which has complete control over the interface and front-line access to the customer base. 4). In the case of Ready Systems, there may have been "electro-political" forces in effect that caused a falling out between them and one of their compiler team-mates. This type of condition can cause runtime suppliers to get a "grass is greener" viewpoint and switch to a concentration on technologies that are less dependent on other software houses. 5). The future of runtime vendors will be brighter if a standard runtime interface is proposed in Ada9X, however I suspect that only companies that have a long commitment to Ada and good technology that is difficult to duplicate (or patented) will be profitable at supplying independent runtimes. There! You can make the check payable to: -))) Tom Griest LabTek Corporation IMHO, I think you should either pick a different project that you know more about, or spend the time to find out the details about DARK. If you can foretell all of the benefits of research before it is conducted as you suggest, you may want to make yourself available to SDIO, the Human Genome Project, the Cold Fusion Institute, High Temperature Superconductor Project, and others. Your time is much too valuable to waste on giving talks. Final note: I am much more interested in discussing the philosophy behind the model of distributed execution for DARK than trying to guess if this one project will "pay for itself". There are serious questions about what Ada9X is doing in the area of distribution and how DARK may or may not relate to this. ---------------- Disclaimer: I am not employed by either the AJPO, SEI, or CMU and have never received any compensation from the SEI. LabTek Corp. is an affiliate of the SEI and has found benefit from several of their projects. The opinions expressed above are my own and do not necessarily reflect those of LabTek Corporation or any of its customers.