Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[awdorrin]

I am getting some random seg faults from my program:

system.secondary_stack.ss_mark () at s-secsta.adb:467

and I'm at a loss for what is going on. I'm guessing that GNAT thinks the stack is bad when the call is made.

The backtrace of my code is:

#0  system.secondary_stack.ss_mark () at s-secsta.adb:467
#1  0x081ec693 in r_de.get_de_int (de_num=79500, to_unit=none)
    at /home/aldorr/lasis/code/rtsim/rde_b.ada:2288
#2  0x081244b8 in g_sv.svs_get_value (svs_num=79500) at ../code/common/gsv_b.ada:172
#3  0x082e867b in sc_mfr.execute () at ../code/sensor/comm/scmfr_b.ada:4627
#4  0x080ff25f in execute_senctl () at ../code/rtctl/esnctl_b.ada:589
#5  0x08050fe2 in e_sequencer (e_cycle_count=56707) at e_seq.c:511
#6  0x080efb92 in e_mainline.exec_subfunctions () at ../code/rtctl/e_mainline_b.ada:453
#7  0x080f044e in e_mainline.exec_main_line_normal () at ../code/rtctl/e_mainline_b.ada:697
#8  0x080ee726 in e_mainline () at ../code/rtctl/e_mainline_b.ada:771
#9  0x080536ba in rtctl_rts () at gosim.c:380
#10 0xb7a198a6 in task_wrapper (task=0x8aa5310) at ltaskLib.c:669
#11 0xb7fba955 in start_thread (arg=0xb663fb70) at pthread_create.c:300
#12 0xb797c5ee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Looking at the assembly at Frame 1:

0x81ec677 <r_de__get_de_int>    push   %ebp
0x81ec678 <r_de__get_de_int+1>  mov    %esp,%ebp
0x81ec67a <r_de__get_de_int+3>  push   %esi
0x81ec67b <r_de__get_de_int+4>  push   %ebx
0x81ec67c <r_de__get_de_int+5>  sub    $0xb0,%esp
0x81ec682 <r_de__get_de_int+11> mov    0xc(%ebp),%eax
0x81ec685 <r_de__get_de_int+14> mov    %al,-0x5c(%ebp)
0x81ec688 <r_de__get_de_int+17> lea    -0x68(%ebp),%eax
0x81ec68b <r_de__get_de_int+20> mov    %eax,(%esp)
0x81ec68e <r_de__get_de_int+23> call   0x804e908 <system__secondary_stack__ss_mark@plt> 
0x81ec693 <r_de__get_de_int+28> sub    $0x4,%esp  

Then the assembly at Frame 0:

0xb7e180e0 <system__secondary_stack__ss_mark>           push   %ebp
0xb7e180e1 <system__secondary_stack__ss_mark+1>         mov    %esp,%ebp
0xb7e180e3 <system__secondary_stack__ss_mark+3>         sub    $0x8,%esp
0xb7e180e6 <system__secondary_stack__ss_mark+6>         mov    %ebx,(%esp)
0xb7e180e9 <system__secondary_stack__ss_mark+9>         call   0xb7cb0ca7 <__i686.get_pc_thunk.bx>
0xb7e180ee <system__secondary_stack__ss_mark+14>        add    $0x9748a,%ebx
0xb7e180f4 <system__secondary_stack__ss_mark+20>        mov    %esi,0x4(%esp)
0xb7e180f8 <system__secondary_stack__ss_mark+24>        mov    0x8(%ebp),%esi
0xb7e180fb <system__secondary_stack__ss_mark+27>        mov    -0x298(%ebx),%eax
0xb7e18101 <system__secondary_stack__ss_mark+33>        call   *(%eax)
0xb7e18103 <system__secondary_stack__ss_mark+35>        mov    (%eax),%edx

I can tell that the 'GET_DE_INT' function call hasn't even gotten to the first line of code within the function, but I'm really not sure what is going on. The data structures themselves look fine. So I'm not sure if GNAT thinks I'm blowing the stack, or what.

My assembly knowledge is more than a little rusty... :)

The corresponding Ada code is:
Frame 2:

DE_NUM := R_DE.DE_NUM_TYPE'val(SVS_NUM);
TEMP   := R_DE.GET_DE_INT(DE_NUM);

Then in Frame 1:

function GET_DE_INT
  (DE_NUM: DE_NUM_TYPE;
   TO_UNIT : R_CONVERT.UNIT_TYPE := R_CONVERT.NONE )
  return INT32 is
  DE_VAL_FLT : FLOAT64;
  RC : INT32;
begin
  GET_DE( DE_NUM, DE_DATA(DE_NUM).ADDR, TO_UNIT, DE_VAL_FLT, RC );


At the time of the crash, I don't think the GET_DE_INT function has even gotten as far as validating the parameters (since I see some __gnat_rcheck calls further down in the assembly code)

Anyone have any suggestions on what I can look at?

Thanks!

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[Ludovic Brenta]

awdorrin <awdorrin@gmail.com> writes:
> My assembly knowledge is more than a little rusty... :)
>
> Anyone have any suggestions on what I can look at?

Try compiling your program with -gnatDG and look at the file
rde_b.ada.dg, this is an intermediate representation of your program in
a language looking like Ada.  This might at least tell you why this
function uses the secondary stack; I see nothing in its spec that
requires the secondary stack (i.e. returning objects of unconstrained
type or declaring controlled objects on the stack).

see
http://gcc.gnu.org/onlinedocs/gcc-4.6.3/gnat_ugn_unw/Debugging-Control.html

Sorry I cannot help more, HTH anyway.

-- 
Ludovic Brenta.

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[awdorrin]

The -gnatDG results, for the function in question:

   function r_de__get_de_int (de_num : in r_de__de_num_type; to_unit :
     in r_convert.r_convert__unit_type := r_convert.r_convert__none)
     return global_types__int32 is
      M875b : system__secondary_stack__mark_id :=
        $system__secondary_stack__ss_mark;

      procedure r_de__get_de_int___clean is
      begin
         $system__secondary_stack__ss_release (M875b);
         return;
      end r_de__get_de_int___clean;
   begin
      de_val_flt : global_types__float64;
      rc : global_types__int32;
      [constraint_error when
        r_de__de_data = null
        "access check failed"]
      [constraint_error when
        not (r_de__Tde_num_typeB(de_num) in 1 .. r_de__de_data.all'last)
        "index check failed"]
      r_de__get_de (de_num, r_de__de_data.all (de_num).addr, to_unit,
        de_val_flt, rc);
      if rc /= 0 then
         subtype r_de__get_de_int__TS863bS is string (1 .. 11);
         S863b : string (1 .. 11);
         P864b : natural;
         $system__img_int__image_integer (integer(rc), S863b, P864b);
         R865b : constant natural := P864b;
         B869b : declare
         begin
            g_error.g_error__add_to_q ($system__string_ops__str_concat
              ("R_DE.GET_DE_INT: FAILED.  RC = ", S863b (1 .. R865b)),
              error_type => g_common.g_common__non_severe);
         end B869b;
         if rc = -1 then
            $__gnat_raise_exception (ada__exceptions__exception_id!(
              r_de__illegal_conversion'reference), "rde_b.ada:2305");
         elsif rc = -2 then
            $__gnat_raise_exception (ada__exceptions__exception_id!(
              r_de__get_de_exception'reference), "rde_b.ada:2308");
         end if;
      end if;
      [constraint_error when
        not (de_val_flt > (-(4503599628419072.0*2**(-21))) and then
          de_val_flt < (9007199252643840.0*2**(-22)))
        "overflow check failed"]
      return global_types__int32(de_val_flt);
   at end
      r_de__get_de_int___clean;
   end r_de__get_de_int;


I'm still not sure what to make of that...

Could GNAT be misinterpreting the 'int32' return type? (Global_Types.Int32 is defined as: subtype INT32 is Integer;)

Or could the 'DE_DATA' structure be causing the issues?

It is defined as in the 'rde_b.ada' file as:

type DE_ARRAY_TYPE is array (DE_NUM_TYPE range <>) of DE_REC_TYPE;
NUM_DES: constant INT32 := G_COMMON.NUM_LINES(DE_CFG_FN);
subtype CONSTRAINED_DE_ARRAY is DE_ARRAY_TYPE( DE_NUM_TYPE range 1 .. DE_NUM_TYPE(NUM_DES));
type DE_ARRAY_PTR_TYPE is access all CONSTRAINED_DE_ARRAY;
DE_DATA : DE_ARRAY_PTR_TYPE;

DE_DATA is a pointer to memory allocated by another process (The original developers allocated memory in a HeapMgr package they developed...)

I just don't know if this legacy goofiness with the heapmgr allocating memory is incompatible with the newer Ada95/2005 approaches... or if I'm getting side tracked from the real problem.

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[Ludovic Brenta]

awdorrin <awdorrin@gmail.com> writes:
> The -gnatDG results, for the function in question:
>
>    function r_de__get_de_int (de_num : in r_de__de_num_type; to_unit :
>      in r_convert.r_convert__unit_type := r_convert.r_convert__none)
>      return global_types__int32 is
>       M875b : system__secondary_stack__mark_id :=
>         $system__secondary_stack__ss_mark;
>
>       procedure r_de__get_de_int___clean is
>       begin
>          $system__secondary_stack__ss_release (M875b);
>          return;
>       end r_de__get_de_int___clean;
>    begin
>       de_val_flt : global_types__float64;
>       rc : global_types__int32;
>       [constraint_error when
>         r_de__de_data = null
>         "access check failed"]
>       [constraint_error when
>         not (r_de__Tde_num_typeB(de_num) in 1 .. r_de__de_data.all'last)
>         "index check failed"]
>       r_de__get_de (de_num, r_de__de_data.all (de_num).addr, to_unit,
>         de_val_flt, rc);
>       if rc /= 0 then
>          subtype r_de__get_de_int__TS863bS is string (1 .. 11);
>          S863b : string (1 .. 11);
>          P864b : natural;
>          $system__img_int__image_integer (integer(rc), S863b, P864b);
>          R865b : constant natural := P864b;
>          B869b : declare
>          begin
>             g_error.g_error__add_to_q ($system__string_ops__str_concat
>               ("R_DE.GET_DE_INT: FAILED.  RC = ", S863b (1 .. R865b)),
>               error_type => g_common.g_common__non_severe);
>          end B869b;
>          if rc = -1 then
>             $__gnat_raise_exception (ada__exceptions__exception_id!(
>               r_de__illegal_conversion'reference), "rde_b.ada:2305");
>          elsif rc = -2 then
>             $__gnat_raise_exception (ada__exceptions__exception_id!(
>               r_de__get_de_exception'reference), "rde_b.ada:2308");
>          end if;
>       end if;
>       [constraint_error when
>         not (de_val_flt > (-(4503599628419072.0*2**(-21))) and then
>           de_val_flt < (9007199252643840.0*2**(-22)))
>         "overflow check failed"]
>       return global_types__int32(de_val_flt);
>    at end
>       r_de__get_de_int___clean;
>    end r_de__get_de_int;
>
>
> I'm still not sure what to make of that...

It appears that the secondary stack is made necessary by this part:

>          begin
>             g_error.g_error__add_to_q ($system__string_ops__str_concat
>               ("R_DE.GET_DE_INT: FAILED.  RC = ", S863b (1 .. R865b)),
>               error_type => g_common.g_common__non_severe);
>          end B869b;

The call to the string concatenation operator "&" returns a string of
unconstrained type on the secondary stack (even though both operands
have a fixed size known at compile time).

I suggest you comment out this error reporting and see whether this
eliminates the secondary stack and, more importantly, the run-time error
it causes.  If that works, maybe you can reintroduce the error reporting
without using "&", e.g. by using Unbounded_String.Append or simply
fixed-length string slices instead.

Also I suggest you investigate why you're getting errors when executing
$system__secondary_stack__ss_mark; are you running on a bare-board
system that does not support the secondary stack?  Or running out of
memory?

HTH

-- 
Ludovic Brenta.

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[awdorrin]

Ah that is interesting. How did you know that was using the secondary stack?

My system is a Intel based Dell pc running Debian Linux. I think it has 8gb RAM (at home now will have to double check tomorrow.) So i shouldn't be running out of memory, although I would not be surprised if this code contains memory leaks that could be effecting the stack. 

I spent most of today trying to track this secondary stack issue down. 

Thanks!

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[awdorrin]

I have done some more investigation this morning.

As suggested, I removed the string concat in the ADD_TO_Q() call.
Checking the rde_b.ada.dg file showed that there was no references to the secondary stack, within that function. (Unfortunately, looking at all the other .dg files generated, I saw over 5000 references to usage of the secondary stack.)

I verified that each pthread is being provided with 8MB of space for their stack. (This program starts off from C and spawns threads, which are mixed Ada and C.)

I'm not yet sure how to verify the default stack/secondary stack sizes in GNAT (although I've seen the -d/-D gnat flags for explicitly setting the sizes) - still researching.

Yesterday, this program could run for random amounts of time before seg faulting, sometimes over 30 minutes, so I have it running again to see what I can get to happen. I figure it very well could crash in another routine which uses the secondary stack - until I figure out what the root cause is...

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[Ludovic Brenta]

awdorrin wrote on comp.lang.ada:
> I verified that each pthread is being provided with 8MB of space for
> their stack. (This program starts off from C and spawns threads,
> which are mixed Ada and C.)
> 
> I'm not yet sure how to verify the default stack/secondary stack
> sizes in GNAT (although I've seen the -d/-D gnat flags for
> explicitly setting the sizes) - still researching.

The secondary stack is actually a region of the heap.  See the package
GNAT.Secondary_Stack_Info if you want to monitor usage and see
http://gcc.gnu.org/onlinedocs/gnat_ugn_unw/Switches-for-gnatbind.html
for a quick introduction (look for "-Dnn[k|m]").  The gritty details
of the secondary stack are in the sources of GCC: s-secsta.ad[bs]; on
Debian and most OSes with virtual memory, the secondary stack is
theoretically unlimited in size because
System.Parameters.Sec_Stack_Percentage is set to Dynamic.

HTH

-- 
Ludovic Brenta.

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[Simon Wright]

awdorrin <awdorrin@gmail.com> writes:

> I verified that each pthread is being provided with 8MB of space for
> their stack. (This program starts off from C and spawns threads, which
> are mixed Ada and C.)

Have you registered these threads with the Ada RTS? See
g-thread.ads. You need to do this before doing anything (such as
accessing the secondary stack) which requires Ada-specific setup in the
TCB or equivalent.

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[awdorrin]

On Friday, September 7, 2012 11:35:42 AM UTC-4, Simon Wright wrote:
> 
> Have you registered these threads with the Ada RTS? See
> g-thread.ads. You need to do this before doing anything (such as
> accessing the secondary stack) which requires Ada-specific setup in the
> TCB or equivalent.

I thought I was, but perhaps I'm not doing it correctly.

I have all of the Ada code compiled into a library (libsimlib.a) and I initialize the Ada through a generated function: 'simlibinit()'

The C code then makes multiple pthread_create() calls to spawn off each thread.
The function pointer passed to pthread_create() is a C function that wraps the Ada call. For example:

extern e_mainline();
int rtctl_rts() {
  e_mainline();
  return 0;
}

pthread_create(&tid, attrp, &rtctl_rts, NULL);

where 'e_mainline' is exported from the RTCTL package with:
pragma EXPORT(Ada, E_MAINLINE, "e_mainline");

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[awdorrin]

After looking at the g-thread.ads file on my system, I'm thinking that perhaps I need to change the thread function to something like:

extern void *__gnat_register_thread();

int rtctl_rts() {
  void *task_id = __gnat_register_thread();
  e_mainline();
  __gnat_unregister_thread();
  return 0;
}

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[Simon Wright]

awdorrin <awdorrin@gmail.com> writes:

> After looking at the g-thread.ads file on my system, I'm thinking that
> perhaps I need to change the thread function to something like:
>
> extern void *__gnat_register_thread();
>
> int rtctl_rts() {
>   void *task_id = __gnat_register_thread();
>   e_mainline();
>   __gnat_unregister_thread();
>   return 0;
> }

That's what I was thinking (when I came across this problem, the Ada
code was a callback from a VxWorks task, so not the same scenario
exactly).

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[awdorrin]

On Friday, September 7, 2012 1:07:36 PM UTC-4, Simon Wright wrote:
> 
> That's what I was thinking (when I came across this problem, the Ada
> code was a callback from a VxWorks task, so not the same scenario
> exactly).

The code that I am working on migrating to Linux came from a VxWorks system, so it may be a more similar scenario than you thought. ;-)

The code compiled and ran with those changes. I haven't seen any Seg Faults yet. However now I appear to have another issue likely related to trying to run at RT priorities.

Thanks once again, to both of you, for the help! 
Hopefully I'll get this next priority issue worked out and will be able to verify that this did resolve the secondary stack issues.

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[francois_fabien]

This question is a little off topic, but may be of interest in general.
When moving from VxWorks to Linux, how are the performances in terms of speed ? i.e. do you have a rating between the execution on Linux vs  Vxworks ?
I ask the question because the performance of Ada RT code vs C/C++ is often a matter of debate.
  

Re: Stumped: Seg Fault with System.Secondary_Stack.SS_Mark()

[awdorrin]

On Saturday, September 8, 2012 9:23:26 AM UTC-4, (unknown) wrote:
> This question is a little off topic, but may be of interest in general.
> 
> When moving from VxWorks to Linux, how are the performances in terms of speed ? i.e. do you have a rating between the execution on Linux vs  Vxworks ?
> 
> I ask the question because the performance of Ada RT code vs C/C++ is often a matter of debate.

At this point, I have no performance data gathered between the applications.

The largest concern for our application is how quickly and consistently we can respond to the interrupts generated by an interface card in the system. I'm using a Linux kernel with the PREEMPT_RT patches installed, and I believe the system will perform fine. 

But, until I get things stable and can evaluate our interface card using hardware interrupts, I won't know for sure.