From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,LOTS_OF_MONEY autolearn=unavailable autolearn_force=no version=3.4.4 Newsgroups: comp.lang.ada Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!wuarchive!uunet!world!srctran From: srctran@world.std.com (Gregory Aharonian) Subject: Critique of SEI's Ada DARK project Message-ID: Sender: srctran@world.std.com (Gregory Aharonian) Organization: The World Distribution: comp.lang.ada Date: Tue, 21 May 1991 19:02:21 GMT List-Id: I am giving a talk during the first week of June on cost/benefit analysis and economics as applied to software reuse and engineering. One example I plan to use is the Software Engineering Institute (SEI) Distributed Ada Real-Time Kernel (DARK). My contention is that the costs in terms of tax dollars far outweighed the benefits the country received in terms of any new original information from the project' results. I am interested in what others think about the cost and benefits of this project. The DARK project was started a few years ago at SEI('88 or '89), funded by the Air Force's Electronic Systems Division. I do not know the exact amount of money allocated, but assuming that the effort required at least four man-years, had a small room of equipment to test the kernel, and average overhead rates, I figure that it cost at least 325,000 dollars for this project (four man years @ 50000, equipment at 50000 and overhead at 75000). "The purpose of DARK is to be a mechanism for supporting the execution of distributed real-time Ada applications in embedded computer systems. It provides a solution to scheduling and distributing tasks without modifying the Ada language or vendor-supplied runtime systems. The goal of the DARK project is not to develop any radically new technological breakthroughs, but to demonstrate that well-understood, familiar and reliable methods of building real-time, distributed systems could be done in Ada as well as any programming language". (This paragraph comes from some SEI report abstracts). After developing and experimenting with the code, the SEI tried licensing the kernel to industry in 1990. I assume that this effort failed, because in 1991, the Department of Commerce's National Technical Information Service (NTIS) announced it was selling the DARK system (the cost is a few hundred dollars for a magnetic tape and documentation). To the best of my knowledge, this is the current official status of DARK. I have not examined the code, but I assume that like other Ada code developed at SEI, the quality of the Ada source code is very high and reliable. As a background note, before, during and after this project, there were (and still are) at least a half dozen private ventures concerning real time Ada kernels, some from compiler companies like Alsys, some from real time kernel vendors such as Ready Systems, a few university efforts and at least one European Community funded project. My question is, how did (or how could) the country benefit by this investment? If the goal (as they stated in their documents) was to prove that it is possible to build decent real-time, distributed Ada systemss, then the DoD and the country could have learned this by sitting back and watching the activities of industry developments with Ada kernels, and actual embedded projects, for free. A low cost effort to summarize the results of existing activities (like the CECOM project) would have been sufficient. For this goal, the net cost/benefit of DARK is negative. An alternate goal might have been to have some SEI staff gain experience in this area. However the DoD could have sent them to one of these companies for training, at a much lower cost than what the DoD funded the SEI. Also, there was a rather nice ARMY CECOM effort (finished in 1989) that studied the problems of developing real time Ada code, and also offered Ada source code examples, as well as over 1000 pages of documentation, at no cost (contact Mary Bender at 201-544-2105). Again, for this goal, the net cost/benefit of DARK is negative. If a goal was to provide the country with an Ada real time kernel to insure that people use Ada (which must have been an unstated goal since they did try to license DARK to industry), then the venture failed because vendors developed their own real time kernels, and recently most agreed to offer or integrate in Ready Systems' real time kernel. Again, the cost/benefit of DARK is negative. Besides, the government shouldn't be in the position of developing products. If a goal was to provide a freely available, reusable piece of Ada real-time source code, then someone screwed up by having DARK available thru the NTIS, instead of being placed on the public archives on Internet. (It may be that DARK is available from SIMTEL, but to the best of my knowledge, DARK is not available by anonymous ftp from any Internet site). Again, the net cost/benefit of DARK is negative. If a goal was to provide an example of a self-contained project whose costs and benefits could be analyzed as an example of cost-benefit analysis, a project of immeasurable benefit to the Ada community, well, I haven't seen a single report from SEI (or an independent third party) performing such an analysis. In fact, in March of 1989 the General Accounting Office, in a report (GAO/IMTEC-89-9) on the general progress of Ada inside the DoD, complained (and the DoD concurred) that no one is collecting the data to justify the claims made on the benefits of reuse and Ada. The DARK project might have been a good opportunity to do so. My contention is that while the country benefitted from this project, the costs to the country were much greater, and that this analysis could have been performed before the project had ever started, by considering the social and economic aspects of software development and reuse on a national scale. All of the benefits could have been obtained at a much lower cost. There may be other benefits to this project, or that the costs were lower, but they escape me at the moment. Now 300,000 dollars is not much, but if similar waste is going on elsewhere, it soon adds up to a lot of tax dollars. I can believe great things about Ada, but I would like to see some numbers, especially when I am paying for it. (Of interest is that Ready Systems got out of the real-time Ada kernel business because there was no profit in it as a product. Finding out why would be a beneficial SEI study, and also why there are so few Ada library companies and fewer that survive.) I'd appreciate any comments of my analysis, either posted, or emailed to me at srctran@world.std.com (if you want privacy). I'll post the results. Attached are comments from one reviewer. Thanks. Gregory Aharonian Source Translation & Optimization "The best tools to use to defend are the ones being defended" ================================= ****************************************************************************** Comment: This is interesting to me because of my position at the *************, which is a state supported academic/industry/government consortium with the avowed goal of helping improve the state of the art and the state of the practice in software engineering, especially as it might benefit software-related industry. A smidgin more background about the center: The state provides `matching' funds against industrial contribution, with the goal (among other aspects of our mission) of making us a `cheap date' for industrial members to support or collaborate on ventures they might never do all by themselves. So while we aren't set up at all like SEI, there are rough similarities. I bring this up because DARK is along the same model of projects we might consider taking on: DARK probably LOOKED like * something worth doing (if nothing else, as a proof of concept that such things could be done in Ada.) * a somewhat risky project (people probably weren't very confident that it could be done.) * something that industrial teams might not do on their own (because of the risk). All of these apply just about uniformly to most things we do take on. But the main thing we do differently is that we NEVER take on a project without industrial involvement. Sometimes we feel that this is a straightjacket, since it's hard to convince places to support some things that we think need to be done, but don't look relevant enough over the usual 1-3yr horizons that they operate under. I think this is a difficult issue: DoD/DARPA, NSF, et al probably should support software technology research for which researchers can make a good case benefits computer science and engineering long term, but not if industry will develop it anyway. How can you know this in advance? Conclusions: 1. DARK was probably a waste; compounded by poor midcourse decisions like not generally releasing the software. 2. Had people at SEI been a little smarter they might have realized this before it happened. But that this was probably close enough to the borderline of the kinds of things that SEI should take on that I have a hard time assigning much blame. I might have made the same mistake.