Re: Critique of SEI's Ada DARK project

[griest@ajpo.sei.cmu.edu (Tom Griest)]

In Article 5010  Gregory Aharonian writes:

>         I am giving a talk during the first week of June on cost/benefit
> analysis and economics as applied to software reuse and engineering.  One
> example I plan to use is the Software Engineering Institute (SEI) Distributed
> Ada Real-Time Kernel (DARK). My contention is that the costs in terms of tax
> dollars far outweighed the benefits the country received in terms of any new
> original information from the project' results. I am interested in what others
> think about the cost and benefits of this project.
 
   First, I would not dream of giving a talk on something that you appear to
have so many fuzzy facts about.  If anyone in the audience knows about DARK
you are likely to get ripped apart.  Even if you get lucky and the audience
accepts your guesses, because they are just guesses you are just as likely
to give the wrong impression as the correct one.

>      The DARK project was started a few years ago at SEI('88 or '89), funded
> by the Air Force's Electronic Systems Division.  I do not know the exact
> amount of money allocated, but assuming that the effort required at least four
> man-years, had a small room of equipment to test the kernel, and average
> overhead rates, I figure that it cost at least 325,000 dollars for this
> project (four man years @ 50000, equipment at 50000 and overhead at 75000).

   If you want to do a true cost analysis, I suggest you find out exactly 
how many person-hours were spent on it and how much it cost (and who 
paid the bill).  You must also study all of the spinoffs that resulted
from this work if you want to be accurate.

>  [STUFF ABOUT DARK's PURPOSE DELETED]

>       After developing and experimenting with the code, the SEI tried
> licensing the kernel to industry in 1990. I assume that this effort failed,
> because in 1991, the Department of Commerce's National Technical Information
> Service (NTIS) announced it was selling the DARK system (the cost is a few
> hundred dollars for a magnetic tape and documentation).  To the best of my
> knowledge, this is the current official status of DARK. I have not examined
> the code, but I assume that like other Ada code developed at SEI, the quality
> of the Ada source code is very high and reliable.

   Licensing software source is an extremely tricky business.  Since the
mission of the SEI is to make information available for wide dissemination,
I am not surprised that they decided to make data available via NTIS.  I
hope your not trying to imply that the project as a whole was a failure 
because they are providing the results through NTIS!  Virtually all
government funded projects provide their results through NTIS (or DTIC
for classified data).

>  
>         As a background note, before, during and after this project, there
> were (and still are) at least a half dozen private ventures concerning real
> time Ada kernels, some from compiler companies like Alsys, some from real time
> kernel vendors such as Ready Systems, a few university efforts and at least
> one European Community funded project.

   Don't forget that DARK is specifically for DISTRIBUTED systems, and NOT
simply a message passing mechanism on top of a network.  However, 
many other researchers are also working on distributed operating systems.

>  
>        My question is, how did (or how could) the country benefit by this
> investment?

   As with all research, the results can be very instructive to those WHO
GET THE RESULTS AND STUDY THEM.  Commercial vendors doing research, as
you mention above, are very reluctant to publish the whole truth of their
work (there is a financial disincentive unless it has been patented).
Universities do provide useful information and sometimes at a lower cost
because they occasionally "make do" with limited resources and often depend
on graduate students to provide slave labor.  However sometimes the results
of the work reflects the inexperience of these graduate students.  The
SEI pays enough to get experienced people who are more likely to see
subtle details that effect the results of studies than less experienced
students.   [ University folks please flame to /dev/null ... I think 
the vast majority of Univ. research is fine, but some of it could benefit
from more review by industry.  There is a need for both university and
industrial researchers.  Note also that SEI is run by CMU.]

   Since Ada is going through a revision process, the results of the SEI
work could influence the Ada9X process.  Alternatively is could provide
guidance for others wanting to do similar projects with Ada.  Even when
research "fails" to meet it's objective, careful review of the work
often provides ideas to other researches which benefit from the results
of the prior projects.  It's rather silly to suggest that because
the results of a research project weren't watched like CNN during
the first days of the Air War that the work was a waste.  It is
hard to predict the results of research and often the full results are
not known for many years after the project has completed.

>         If the goal (as they stated in their documents) was to prove that it
> is possible to build decent real-time, distributed Ada systems, then the DoD
> and the country could have learned this by sitting back and watching the
> activities of industry developments with Ada kernels, and actual embedded
> projects, for free. A low cost effort to summarize the results of existing
> activities (like the CECOM project) would have been sufficient. For this goal,
> the net cost/benefit of DARK is negative.

   I would really like to see the equation you used to compute a negative 
number here.  Other research which provides results "at a lower cost"
does not indicate that the net benefit of DARK will not be positive.

   One point that must be made when observing work being done on "real"
projects (that is, not research work) is that usually these projects
do not have a time budget that allows finding out WHY everything operates
they way it does.  If a problem arises during a development, usually
the quickest way to eliminate the problem is chosen.  Often code is
replaced with something "we know will work" rather than taking the time
to find out why some "elegant solution" did not work.  The emphasis is
on coming up with any suitable solution, not an optimal one.  If research
was limited to the study of what developers did, we would probably use
only the bubble sort for all sorting.  Summaries of ongoing work
and research are both needed to provide a balanced picture of the issues.

>  
>          An alternate goal might have been to have some SEI staff gain
> experience in this area. However the DoD could have sent them to one of these
> companies for training, at a much lower cost than what the DoD funded the SEI.
> Also, there was a rather nice ARMY CECOM effort (finished in 1989) that
> studied the problems of developing real time Ada code, and also offered Ada
> source code examples, as well as over 1000 pages of documentation, at no cost
> (contact Mary Bender at 201-544-2105). Again, for this goal, the net
> cost/benefit of DARK is negative.

    Nothing is free.  I can only guess that your "no cost" refers to
the cost of getting the documentation.  I know CECOM pays the people who
do their research.  The ARMY will sometimes pay for the duplication of 
reports, especially during periods when reports are being transitioned
to NTIS.  I suspect that these reports are now available from NTIS for a charge.
I have also received free documentation from SEI.

>  
>          If a goal was to provide the country with an Ada real time kernel to
> insure that people use Ada (which must have been an unstated goal since they
> did try to license DARK to industry), then the venture failed because vendors
> developed their own real time kernels, and recently most agreed to offer or
> integrate in Ready Systems' real time kernel. Again, the cost/benefit of DARK
> is negative. Besides, the government shouldn't be in the position of
> developing products.

In light of the fact (that you point out below) that Ready Systems is out of
the Ada business, I fail to see the clear cost benefit in having a proprietary
runtime which is no longer supported by the developer.

> 
>          If a goal was to provide a freely available, reusable piece of Ada
> real-time source code, then someone screwed up by having DARK available thru
> the NTIS, instead of being placed on the public archives on Internet. (It may
> be that DARK is available from SIMTEL, but to the best of my knowledge, DARK
> is not available by anonymous ftp from any Internet site). Again, the net
> cost/benefit of DARK is negative.

Not everyone has access to the net, but...

Try anonymous ftp from:  fg.sei.cmu.edu

looks like you're the one who is in error.

>  
>        If a goal was to provide an example of a self-contained project whose
> costs and benefits could be analyzed as an example of cost-benefit analysis,
> a project of immeasurable benefit to the Ada community, well, I haven't seen
> a single report from SEI (or an independent third party) performing such an
> analysis. In fact, in March of 1989 the General Accounting Office, in a
> report (GAO/IMTEC-89-9) on the general progress of Ada inside the DoD,
> complained (and the DoD concurred) that no one is collecting the data to
> justify the claims made on the benefits of reuse and Ada. The DARK project
> might have been a good opportunity to do so.
>  
   There have been dozens of reports and public presentations on DARK.  You
may wish to get "Experiences Porting the Distributed Ada Real-Time Kernel",
Brian Smith, Boeing Military Airplanes, June 1990  (Available through DTIC
and NTIS: AD-A226 693,  I think I paid $5 from DTIC.)

I'm pretty sure that cost-benefit analysis of software reuse or government
sponsored research were NOT the stated objectives of DARK.  This is YOUR
area of interest (and GAO's) and it seems you are not adequately prepared
for it.  You had better start with getting a way to measure software 
productivity.  (When you figure that one out, let me know.  I have a long 
list of people interested in THE answer!)

>         My contention is that while the country benefited from this project,
> the costs to the country were much greater, and that this analysis could have
> been performed before the project had ever started, by considering the social
> and economic aspects of software development and reuse on a national scale.
> All of the benefits could have been obtained at a much lower cost. There may
> be other benefits to this project, or that the costs were lower, but they
> escape me at the moment. Now 300,000 dollars is not much, but if similar waste
> is going on elsewhere, it soon adds up to a lot of tax dollars. I can believe
> great things about Ada, but I would like to see some numbers, especially when
> I am paying for it.

".... by considering the social and economic aspects of software development
and reuse on a national scale" ???  That certainly seems easy to do! -:)
The computer industry is so volatile I don't see how a realistic cost
analysis could be done.  If it could be done, the cost analysis would
certainly be as big a job as the research and probably cost more and have
less reliable results.  Do you know how many tons of money have been spent
on actual development projects only to have the industry change direction
before the product could get to market?  Look at OS/2.  I saw estimates
by market forecasters in 1988 that it would be the OS on 70%-90% of the 
PC platforms by 1991.  Now analysts say it is all but dead, being replace
by MS Windows and Unix.  Who knows?  Will it be another PC Jr. or will
it be what VMS used to be?


>  
>         (Of interest is that Ready Systems got out of the real-time Ada kernel
> business because there was no profit in it as a product. Finding out why would
> be a beneficial SEI study, and also why there are so few Ada library companies
> and fewer that survive.)
>  


How much would you pay for this study?  Where is your cost-benefit analysis
that this study would benefit American Tax Payers?  

I can probably guess the results of your study and save the tax payers
$325,000:

      1). The Ada compiler/support tool business is not exactly the
          "goose that laid the golden egg".

      2). There is no standard interface between Ada compilers and the
          underlying runtime.  This makes it costly to support a wide
          variety of compilers.  (By the way, this issue may be addressed
          by Ada9X.)

      3). Runtime vendors are somewhat dependent on the compiler vendors
          to obtain a customer base.  As soon as the runtime vendor's
          business begins to show a large profit, the compiler vendor may
          invest the money it takes to provide nearly the same capability
          from their "new high-performance" runtime.  It is extremely hard
          to compete against the compiler vendor which has complete
          control over the interface and front-line access to the customer
          base.

      4). In the case of Ready Systems, there may have been "electro-political"
          forces in effect that caused a falling out between them and one of
          their compiler team-mates.  This type of condition can cause
          runtime suppliers to get a "grass is greener" viewpoint and switch
          to a concentration on technologies that are less dependent on
          other software houses.

      5). The future of runtime vendors will be brighter if a standard
          runtime interface is proposed in Ada9X, however I suspect that
          only companies that have a long commitment to Ada and good technology
          that is difficult to duplicate (or patented) will be profitable
          at supplying independent runtimes.



There!  You can make the check payable to:     -)))

            Tom Griest
            LabTek Corporation


IMHO, I think you should either pick a different project that you know
more about, or spend the time to find out the details about DARK.  If you
can foretell all of the benefits of research before it is conducted as
you suggest, you may want to make yourself available to SDIO, the Human 
Genome Project, the Cold Fusion Institute, High Temperature Superconductor 
Project, and others.  Your time is much too valuable to waste on giving talks.

Final note: I am much more interested in discussing the philosophy behind
the model of distributed execution for DARK than trying to guess if this
one project will "pay for itself".  There are serious questions about what
Ada9X is doing in the area of distribution and how DARK may or may not
relate to this.

----------------

Disclaimer: I am not employed by either the AJPO, SEI, or CMU and have
never received any compensation from the SEI.  LabTek Corp. is an affiliate
of the SEI and has found benefit from several of their projects.
The opinions expressed above are my own and do not necessarily reflect
those of LabTek Corporation or any of its customers.