Test driver for internal subpgms?

Test driver for internal subpgms?

[Owen Chang]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 6318 bytes --]


I have been coding in Ada on and off for ten years and it bothered me
that I could not find a satisfactory way to unit test internal
subprograms (not visible outside of it�s enclosing package).  The
problem comes with finding a way to write a driver which will ping the
subprogram with the full range of possible inputs.  I came up with an
idea (which I found out later has been already used by others) which
involves declaring a new procedure for the package for the purpose of
unit test only and declaring it as a subunit (separate). 

There is disagreement at work about whether this is good programming
practice or not.  So I wrote this message hoping I can get some
feedback - good or bad.  This kind of topic is not covered in Ada
books I have seen.  And I have not seen the topic discussed anywhere.
I don�t  believe this is an issue with C because it is so easy to
provide visibility to a driver although I suppose it would be one with
C++.  (I haven�t really pursued this avenue).  Any leads on where I
can find more info would be much appreciated.

INDIRECT APPROACH
The most straight forward way to ping  a �unit under test� which is an
internal subprogram with the full range of possible input values is to
determine a set of inputs for an external subprogram which in turn
pings the unit under test.  I have found that often this set of inputs
is difficult to determine because of the extra variables and logic
that must be taken into account.  Also it is not obvious to a
programmer other than the one who wrote the unit test what is being
done.

CUT  AND PASTE APPROACH
In these cases one could cut and paste the unit under test into the
code containing the test driver.  The drawback to this approach is
that any supporting declarations of global variables, supporting
subprograms, instantiations, etc. must also be found, cut and pasted.
Also regression testing after a platform change or code fix is
painfully tedious.

DEBUGGER APPROACH
Another approach would be to go into a debugger and manually set the
inputs.   This could be scripted if the debugger has a scripting
feature.  The disadvantage here is that  it takes time to master the
scripting features of a debugger.  And every environments debugger is
different, so if you go to a new environment the scripts need to be
rewritten and a new debugger must be learned.  And it can be tedious
to adjust the script when doing a regression test for code that has
been modified.

TROJAN HORSE APPROACH
The approach inserting a new subunit is shown by example below:

file 1
-------
package PKG_A is
   procedure PROC_A (X : in INTEGER);
   procedure TEST;                                      -- inserted
for test purposes only

end PKG_A;

file 2
------
package body PKG_A is
   procedure PROC_B is             -- internal subprogram unit under
test
   begin
        <whatever>
   end;
   procedure PROC_C is              -- internal subprogram unit under
test
   begin
        <whatever>
   end;
   procedure PROC_A (X : in INTEGER) is  -- external  subprogram
   begin
        <whatever>
   end;

   procedure TEST is separate;                      -- inserted for
test purposes only

end PKG_A;

file 3
------
separate (PKG_A)
procedure TEST is
   procedure TEST_PROC_B is                  -- code to test PROC_B
   begin
         PROC_B(5);
         <whatever>
   end;
   procedure TEST_PROC_C is                   -- code to test PROC_C
   begin
         PROC_C(5);
         <whatever>
   end;
begin -- TEST
   TEST_PROC_B;
   TEST_PROC_C;
end TEST;

with PKG_A;
procedure DRIVER_MAIN is                       -- driver main
procedure
begin
     PKG_A.TEST;
end DRIVER_MAIN;

Files 1 and 2 contain the spec and body of the package containing the
unit under test.  File 3 contains two library units.  It contains the
body for the �Trojan horse� subunit   And it contains the main
procedure of the driver executable which will be used to invoke the
unit(s) under test with the desired inputs for unit test.  A single
line of code is inserted into file 1 and file 2 in order to declare
the �Trojan horse� subunit TEST.   DRIVER_MAIN calls TEST and TEST
pings the unit(s) under test with whatever inputs desired.

This approach has nice advantages.  Modification to the package
containing the unit under test is limited to one line in the spec and
one line in the body.  The driver has access to all internal
subprograms and data structures, generic instantiations, etc. that are
visible inside the package.  Regression test requires almost no code
changes.

The disadvantage of this approach is that one must remove one line of
code from the spec and body before delivery to the customer unless
quality assurance standards make special allowances for code inserted
for test purposes only.  Also packages needed to support unit test
must be with�ed into the package containing the unit under test.  In
this case the number of lines added to the package body can be more
than one line.


Hint : If you want to link PKG_A with your application�s main (vs.
DRIVER_MAIN), you can keep the package spec as is by replacing
�separate� with �begin null; end;� in the package body.  This will
satisfy the linker�s need for the body of TEST and you don�t have to
recompile the PKG_A spec.

I thought the �Trojan horse� approach was a great solution for unit
testing internal subprograms, but there are major concerns about it.  

It was argued that it violates the principle that the code under test
cannot be altered in any way (even if it is only one line).  To alter
it means that any results from the test are not valid.  And there are
quality assurance standard that makes the code unsuitable for
delivery.

It was argued that one should be able to come up with the needed set
of inputs to an external subprogram .And if an input cannot be found
such that the unit under test is called with a particular value, then
there is no need to worry about it since the unit under test will
never get it.  A subprogram so low level as to require special
scrutiny should be moved into a support package and made external.  So
there should never be a case where the �indirect approach� cannot
work.  It is inefficient use of time to write driver code dedicated to
an internal subprogram.

What do you think ?

Owen

Re: Test driver for internal subpgms?

[George Haddad]

Owen Chang wrote:
> for test purposes only.  Also packages needed to support unit test
> must be with�ed into the package containing the unit under test.  In
> this case the number of lines added to the package body can be more
> than one line.

   I am not sure what you mean here, but (if I correctly understand what
you want) could you not merely 'with' the test-specific support units
into the body of the test procedure? I.e.:

with Test_Support_0;
with Test_Support_1;
separate(PKG_A)
procedure Test is
  ...

> It was argued that one should be able to come up with the needed set
> of inputs to an external subprogram .And if an input cannot be found
> such that the unit under test is called with a particular value, then
> there is no need to worry about it since the unit under test will
> never get it.  A subprogram so low level as to require special
> scrutiny should be moved into a support package and made external.  So
> there should never be a case where the �indirect approach� cannot
> work.  It is inefficient use of time to write driver code dedicated to
> an internal subprogram.
> 
> What do you think ?

   In a development effort where there are path coverage requirements,
it would seem that the internal code needs to be tested.  If external
inputs can't produce a given internal execution path, then one would
think the internal logic should be changed to eliminate the unreachable
path; however, sometimes the software test engineer isn't the author of
the package in question.  (At least on our projects  :-))  In fact, if
the package is being reused, one may not be permitted to rewrite it, but
one still has to test it.  So, it would seem prudent to have a method
for so doing.

   However, in Ada 95 (Disclaimer: I have only the barest acquaintance
with Ada 95), rather than having a TEST procedure, could you not declare
a child package (which can presumably access the internals of PKG_A),
and then invoke PKG_A.Test.Proc_B (for example) from your driver?
-- 
I found these opinions on my doorstep, would you please give them a good
home?