Re: Required Metrics

["Ken Garlington" <Ken.Garlington@computer.org>]

"Robert Dewar" <dewar@gnat.com> wrote in message
news:8es5sk$5cr$1@nnrp1.deja.com...
> That's a strawman. Obviously no one can sign the DOC if they
> have deliberately failed to implement some aspect of the
> standard, whether or not it is tested in the ACAATS tests

Why not, if the vendor believes it's an unreasonable requirement? For
example, if a vendor believes that "the documentation requirements of the
standard have almost no effect on compiler writers," then he might be
justified in signing off without ever examining his documentation for
compliance.

I also wonder about inadvertant failures. Let's take a specific example:
Before ACT signs off, what items -- other than ensuring all ACAATS tests
pass -- are on your checklist?

> > In addition, if vendors alone are permitted to decide what
> really _isn't_ in
> > the standard, why can't vendors alone decide what's really
> _in_ the
> > standard? Why spend all this time fooling with ISO
> standardization
> > procedures if vendors have, in essence, a veto? Is it just for
> the publicity
> > value?
>
> That's an absurd over-reaction to the thread at hand, not based
> in any reality.

How can questions be "over-reactions?" Sounds like _someone_ doesn't have
any answers handy... :)

More to the point, I have to at least assume this is a possibility, given
that there (a) is evidence that vendors can choose what they want to
implement and (b) no obvious counter-evidence that there are any limits
beyond the ACAATS tests as to what vendors might ignore.

> Remember this thread is all about a requirement that is clearly
> semantically meaningless in formal terms (the RM does not even
> describe what the word "documentation" means, and that's a
> serious omission. Note that we could if we wanted perfectly
> well declare that our sources are part of the documentation
> and constitute the full compliance with the requirements
> of annex M. That would meet the RM standards from a legal
> point of view, but it would be useless to our customers.

But (and this is the key point, which I will keep repeating until someone
bothers to address it), isn't there an intrinsic value in meeting the
standards "from a legal point of view"? Or is it all just based on what a
particular vendor's customers want?

Secondly, if there is something in the standards that is unreasonable, isn't
there an intrinsic value in going through the formal process to fix it?

> I really think the documentation requirements of the standard
> have almost no effect on compiler writers. Documentation is
> written for the benefit of users. It is presumptious in any
> case for the RM to think it can know what kinds of documentation
> the user will require. Furthermore, if it requires documentation
> without any indication of what the requirement means, then the
> requirement is plain useless.

> But Ken, extending this to clearly defined technical features,
> where the requirements are meaningful and semantically sound
> makes no sense at all. Bob Duff was quite clear in saying that
> in THIS area, vendors follow the RM closely even if they don't
> like what it says, and even if the tests do not test something.
> His examples were well chosen ones. I do indeed think that ATC
> was a huge mistake in the design of Ada, but Ada Core
> Technologies has invested significant resources in making this
> feature work completely and well, not just to pass the tests,
> but to meet the requirements of our users.

I was really feeling good until the last line ("meet the requirements of our
users"). Again, I think you're missing the point. I assume you're investing
in ATC because users are asking you for this feature. I assume Sun invested
in a good implementation of java.net without benefit of an ISO standard, as
well.

Without the benefit of knowing what your customers are asking from you, let
me take a stab in the dark at another example, that maybe will get the point
across. Let's say that a vendor claims to fully implement the Safety and
Secuity annex. Pragma Reviewable is part of that annex. It has a series of
requirements identified as "implementation requirements", not documentation
requirements. Are these "clearly defined technical features?" Are they
"semantically sound"? I know there's compilers (including Ada83 compilers)
that meet these requirements, so I assume that at least some vendors believe
they are implementable. I also know that it's really expensive to implement
these (because I kept having to pay big bucks to get them implemented,
because they weren't part of the standard).

Before this conversation, I would have said "If a vendor claims conformance
to Annex H, these requirements have to be implemented, or the vendor has to
explicitly say that they aren't". However, let's say this vendor doesn't
even identify pragma Reviewable in their documentation, much less implement
the required behavior. If I complained about this, would I have a leg to
stand on? Unless I could get some other users to agree, sounds like I'm out
of luck.

It sounds like we may have gone to all the trouble to identify requirements
that IHMO are really important for safety-critical systems, and managed to
get it into the standard so that we could share the cost among a larger
group of users, just to end up in the same fix we were in back in 1985. I
can hear the conversation now:

"You claim conformance to the standard, but you don't implement these
features."
"Well, they don't affect program execution, and they're not tested as part
of validation, and no other users have asked for them, so we never did
them."
"Maybe that's because we're the first safety-critical application to use
your product?"
"Well, if you don't like it, pick another vendor!"
"Unfortunately, you're the only vendor for our platform!"
"OK - pay us and we'll do it."

That conversation is easy to reproduce, because I heard it a lot over the
years. The main difference? Before, we said "yes". Now, we say "C". :)

(The next time anyone complains about why companies are abandoning Ada, keep
this conversation in mind...)

> Your concerns about vendors running amok would make more sense
> if you would give us some nice examples of what it is that you
> do not like :-)

Are you saying that something really bad has to happen before anyone
considers it a risk? Remember what I was saying earlier about good software
engineering practices?

Here's an example from the Ada83 days. We had built an application on a
particular vendor's product, and it worked fine. We delivered the product.
Then, we ported it to a different platform and a different vendor. It
wouldn't compile. It turned out we had a subprogram, declared in a package
body, which used another subprogram in the same package body before it was
declared. The old (validated) compiler, we discovered, had a few problems in
enforcing the visibility rules. This probably isn't a common programming
error, so no other users had complained. (Actually, I think we wrote about
75% of all the problem reports this vendor _ever_ received.) Obviously, it
wasn't caught by the then-ACVC. It wasn't that hard to change the package
body (although I was now stuck with two variants to maintain, at least until
the "new" package body could be ported back to the old system.) So, it
sounds like I really didn't have any reason to complain about the compiler's
behavior, based on the criteria presented above?