Extended comm loss and TCP/IP

Extended comm loss and TCP/IP

[Mark McKercher]

I'm having the following problem.  We're using Ada95 on both Solaris and
HP-UX and our orb is ORBexpress.

I'm working on a project on which a client workstation might be a
satellite link away from the server.  Loss of comm may be frequent and
sometimes last tens of minutes at a time.

Normally, if we issue a CORBA request while comm is down, we'd see a
COMM_FAILURE exception.  I can deal with that.

If comm is lost while in the middle of a CORBA request, if comm is
re-established within a reasonable time, then the  request will complete
normally.  I can deal with that too.

If comm is lost while in the middle of a CORBA request and it takes 10
or more minutes to re-establish comm, then there is trouble.  It appears
as though the server side completes the request but abandons the return
to the client after 10 minutes.  The client, on the otherhand, is
waiting infinitely for the return.  Hence, the client side is locked up
and never frees up.

Our orb vendor has looked at the problem and has determined that it is a
TCP/IP problem... it never sends any indication to orb of a problem.

Has anyone else experienced such problems with TCP/IP and are there any
workarounds for such a problem?

Thanks for you  help!

Mark M.

Re: Extended comm loss and TCP/IP

[tim gilbert]

For some info do "man setsockopt" and "man 2 socket" and read the parts
about SO_KEEPALIVE, I've been told that without SO_KEEPALIVE being set
that the client would remain blocked at the recv call forever.
Hope this helps.

Tim

Mark McKercher wrote:
> 
> I'm having the following problem.  We're using Ada95 on both Solaris and
> HP-UX and our orb is ORBexpress.
> 
> I'm working on a project on which a client workstation might be a
> satellite link away from the server.  Loss of comm may be frequent and
> sometimes last tens of minutes at a time.
> 
> Normally, if we issue a CORBA request while comm is down, we'd see a
> COMM_FAILURE exception.  I can deal with that.
> 
> If comm is lost while in the middle of a CORBA request, if comm is
> re-established within a reasonable time, then the  request will complete
> normally.  I can deal with that too.
> 
> If comm is lost while in the middle of a CORBA request and it takes 10
> or more minutes to re-establish comm, then there is trouble.  It appears
> as though the server side completes the request but abandons the return
> to the client after 10 minutes.  The client, on the otherhand, is
> waiting infinitely for the return.  Hence, the client side is locked up
> and never frees up.
> 
> Our orb vendor has looked at the problem and has determined that it is a
> TCP/IP problem... it never sends any indication to orb of a problem.
> 
> Has anyone else experienced such problems with TCP/IP and are there any
> workarounds for such a problem?
> 
> Thanks for you  help!
> 
> Mark M.

Re: Extended comm loss and TCP/IP

[Corey Ashford]

Mark McKercher wrote:
[snip]
> If comm is lost while in the middle of a CORBA request and it takes 10
> or more minutes to re-establish comm, then there is trouble.  It appears
> as though the server side completes the request but abandons the return
> to the client after 10 minutes.  The client, on the otherhand, is
> waiting infinitely for the return.  Hence, the client side is locked up
> and never frees up.
> 
> Our orb vendor has looked at the problem and has determined that it is a
> TCP/IP problem... it never sends any indication to orb of a problem.
> 
> Has anyone else experienced such problems with TCP/IP and are there any
> workarounds for such a problem?

It seems like the server side ought to close the socket if it believes there's
an unrecoverable error and has given up waiting for a response.
That way when the client finally establishes contact
again, it will be made aware by TCP/IP that the socket has closed.  At this
point the client ought to be able to reestablish a new connection.

If the socket is just left open, and the server has given up on the protocol,
it's a problem waiting to happen.

Another possible solution would be to change the server so that it just never times
out.

- Corey

Re: Extended comm loss and TCP/IP

[SpamSpamSpam]

Hi Mark,

Your problem is CORBA specific, though if ORBExpress raised a COMMS_FAILURE
exception within the client this would help! Theres a book "CORBA Design
Patterns" that
you might consider.  Its on my list..........

For an Ada view :)...

You could make ALL the calls "oneway" and then enter a guarded "select"
which accepts
a callback with the specified delay period or assumes a comms failure, but
this requires alot
of extra work for an exceptional condition the ORB should detect and report.

It would be better if ORBExpress on the client side "polled" the server
while awaiting responses,
raising the COMMS_FAILURE  exception if the ORB server side failed to
respond. After all this
is all meant to be invisible to you (including the TCP/IP networking).
Alternative - Wrap your calls
in tasks, assume comms failure and surprise yourself with a completion
exception !

-------------------------------------------------------------------------------------------

 with CorbaCallXWrap ;
procedure Client is
begin

    -- blar, blar , blar .....
    declare
       CorbaCall : CorbaCallXWrap.Object ; -- kick off call
       RoughTimeout : constant Duration := 600 ;
    begin
       Delay ( RoughTimeout ) ;
       -- Comms fail code here
       -- raise COMMS_FAILURE for example
    exception
       when CorbaCallXWrap.Finished => null ;
    end ;
    -- blar, blar , blar
end Client ;
---------------------------------------------------------------------------------

package CorbaCallXWrap is

    task type Object ;
    Finished : exception ;

end CorbaCallXWrap ;

-----------------------------------------------------------------------------------

package body CorbaCallXWrap is

    -- Dummy real would with in
    procedure CorbaCall is begin null ; end ;

    task body Object is
       Finished : exception ;
    begin
          CorbaCall ;
          raise Finished ;
    end Object ;

end CorbaCallXWrap ;

Cheers Glen

Re: Extended comm loss and TCP/IP

[SpamSpamSpam]

>     task body Object is
>        Finished : exception ; <----------delete

Sorry, that extra "Finished" declaration should come out - tut! two min solutions!

Glen

Re: Extended comm loss and TCP/IP

[SpamSpamSpam]

.. to escape the declare block on comms failure, you also have to call "abort CorbaCall"after
the timeout delay.  If the actual Corba Call is a rendezvous with the ORBExpress
implementation, then it will still be held, the task can't terminate and you can't leave the
declare block............

>        Delay ( RoughTimeout ) ;

>        abort CorbaCall ;

Re: Extended comm loss and TCP/IP

[SpamSpamSpam]

... or even simply ( with Ada95 - a great language gets even better :- ) .....

with CorbaCall ; -- no wrapper !
procedure Client is
begin

   select
       delay ( 6.0 ) ;
       -- comms fail code
    then abort
       CorbaCall.xxxx ;
    end select ;

end Client ;

... I'm going to let one of the language gurus on CLA pick up now.... but its been fun.
Glen