Test driver for internal subprograms

Test driver for internal subprograms

[Owen Chang]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 6334 bytes --]


I have been coding in Ada on and off for ten years and it bothered me
that I could not find a satisfactory way to unit test internal
subprograms (not visible outside of it�s enclosing package).  The
problem comes with finding a way to write a driver which will ping the
subprogram with the full range of possible inputs.  I came up with an
idea (which I found out later has been already used by others) which
involves declaring a new procedure for the package for the purpose of
unit test only and declaring it as a subunit (separate). 

There is disagreement among my peers at work about whether this is
good programming practice or not.  So I wrote this message hoping I
can get some feedback - good or bad.  This kind of topic is not
covered in Ada books I have seen.  And I have not seen the topic
discussed anywhere.  I don�t  believe this is an issue with C because
it is so easy to provide visibility to a driver although I suppose it
would be one with C++.  (I haven�t really pursued this avenue).  Any
leads on where I can find more info would be much appreciated.

INDIRECT APPROACH
The most straight forward way to ping  a �unit under test� which is an
internal subprogram with the full range of possible input values is to
determine a set of inputs for an external subprogram which in turn
pings the unit under test.  I have found that often this set of inputs
is difficult to determine because of the extra variables and logic
that must be taken into account.  Also it is not obvious to a
programmer other than the one who wrote the unit test what is being
done.

CUT  AND PASTE APPROACH
In these cases one could cut and paste the unit under test into the
code containing the test driver.  The drawback to this approach is
that any supporting declarations of global variables, supporting
subprograms, instantiations, etc. must also be found, cut and pasted.
Also regression testing after a platform change or code fix is
painfully tedious.

DEBUGGER APPROACH
Another approach would be to go into a debugger and manually set the
inputs.   This could be scripted if the debugger has a scripting
feature.  The disadvantage here is that  it takes time to master the
scripting features of a debugger.  And every environments debugger is
different, so if you go to a new environment the scripts need to be
rewritten and a new debugger must be learned.  And it can be tedious
to adjust the script when doing a regression test for code that has
been modified.

TROJAN HORSE APPROACH
The approach inserting a new subunit is shown by example below:

file 1
-------
package PKG_A is
   procedure PROC_A (X : in INTEGER);
   procedure TEST;                                      -- inserted
for test purposes only

end PKG_A;

file 2
------
package body PKG_A is
   procedure PROC_B is             -- internal subprogram unit under
test
   begin
        <whatever>
   end;
   procedure PROC_C is              -- internal subprogram unit under
test
   begin
        <whatever>
   end;
   procedure PROC_A (X : in INTEGER) is  -- external  subprogram
   begin
        <whatever>
   end;

   procedure TEST is separate;                      -- inserted for
test purposes only

end PKG_A;

file 3
------
separate (PKG_A)
procedure TEST is
   procedure TEST_PROC_B is                  -- code to test PROC_B
   begin
         PROC_B(5);
         <whatever>
   end;
   procedure TEST_PROC_C is                   -- code to test PROC_C
   begin
         PROC_C(5);
         <whatever>
   end;
begin -- TEST
   TEST_PROC_B;
   TEST_PROC_C;
end TEST;

with PKG_A;
procedure DRIVER_MAIN is                       -- driver main
procedure
begin
     PKG_A.TEST;
end DRIVER_MAIN;

Files 1 and 2 contain the spec and body of the package containing the
unit under test.  File 3 contains two library units.  It contains the
body for the �Trojan horse� subunit   And it contains the main
procedure of the driver executable which will be used to invoke the
unit(s) under test with the desired inputs for unit test.  A single
line of code is inserted into file 1 and file 2 in order to declare
the �Trojan horse� subunit TEST.   DRIVER_MAIN calls TEST and TEST
pings the unit(s) under test with whatever inputs desired.

This approach has nice advantages.  Modification to the package
containing the unit under test is limited to one line in the spec and
one line in the body.  The driver has access to all internal
subprograms and data structures, generic instantiations, etc. that are
visible inside the package.  Regression test requires almost no code
changes.

The disadvantage of this approach is that one must remove one line of
code from the spec and body before delivery to the customer unless
quality assurance standards make special allowances for code inserted
for test purposes only.  Also packages needed to support unit test
must be with�ed into the package containing the unit under test.  In
this case the number of lines added to the package body can be more
than one line.


Hint : If you want to link PKG_A with your application�s main (vs.
DRIVER_MAIN), you can keep the package spec as is by replacing
�separate� with �begin null; end;� in the package body.  This will
satisfy the linker�s need for the body of TEST and you don�t have to
recompile the PKG_A spec.

I thought the �Trojan horse� approach was a great solution for unit
testing internal subprograms, but there are major concerns about it.  

It was argued that it violates the principle that the code under test
cannot be altered in any way (even if it is only one line).  To alter
it means that any results from the test are not valid.  And there are
quality assurance standard that makes the code unsuitable for
delivery.

It was argued that one should be able to come up with the needed set
of inputs to an external subprogram .And if an input cannot be found
such that the unit under test is called with a particular value, then
there is no need to worry about it since the unit under test will
never get it.  A subprogram so low level as to require special
scrutiny should be moved into a support package and made external.  So
there should never be a case where the �indirect approach� cannot
work.  It is inefficient use of time to write driver code dedicated to
an internal subprogram.

What do you think ?

Owen

Re: Test driver for internal subprograms

[Robert A Duff]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2970 bytes --]


In article <32aa09cb.517338@netnews.voicenet.com>,
Owen Chang <oachang@omni.voicenet.com> wrote:
>..  I came up with an
>idea (which I found out later has been already used by others) which
>involves declaring a new procedure for the package for the purpose of
>unit test only and declaring it as a subunit (separate). 

You could do something similar, but using child units instead of
subunits.  This will sometimes require you to put things in a private
child, when you might otherwise have put them in the body.  The
advantage is that you don't have to modify the source code in order to
get rid of the test driver -- just don't link it into your program when
you don't want it.

>The disadvantage of this approach is that one must remove one line of
>code from the spec and body before delivery to the customer unless
>quality assurance standards make special allowances for code inserted
>for test purposes only.

Well, you could leave those lines in, but recompile/relink with a
different (empty) version of the subunit.

>...  Also packages needed to support unit test
>must be with�ed into the package containing the unit under test.  In
>this case the number of lines added to the package body can be more
>than one line.

Subunits can have their own with_clauses.  So can child units.

>Hint : If you want to link PKG_A with your application�s main (vs.
>DRIVER_MAIN), you can keep the package spec as is by replacing
>�separate� with �begin null; end;� in the package body.  This will
>satisfy the linker�s need for the body of TEST and you don�t have to
>recompile the PKG_A spec.

Better to keep it separate, but replace it with "begin null; end;".

>I thought the �Trojan horse� approach was a great solution for unit
>testing internal subprograms, but there are major concerns about it.  
>
>It was argued that it violates the principle that the code under test
>cannot be altered in any way (even if it is only one line).  To alter
>it means that any results from the test are not valid.  And there are
>quality assurance standard that makes the code unsuitable for
>delivery.

Important concerns.  

>It was argued that one should be able to come up with the needed set
>of inputs to an external subprogram .And if an input cannot be found
>such that the unit under test is called with a particular value, then
>there is no need to worry about it since the unit under test will
>never get it.  A subprogram so low level as to require special
>scrutiny should be moved into a support package and made external.  So
>there should never be a case where the �indirect approach� cannot
>work.  It is inefficient use of time to write driver code dedicated to
>an internal subprogram.

I tend to agree with that -- most test drivers do just fine without
meddling in internal details of the unit under test.  And if they want
to meddle, maybe those details should be separated out into a child
package, and tested separately.

- Bob