Re: Ravenscar and context switching for Cortex-M4

[Patrick Noffke <patrick.noffke@gmail.com>]

On Thursday, February 19, 2015 at 4:45:00 PM UTC-6, Patrick Noffke wrote:
> On Thursday, February 19, 2015 at 4:13:51 PM UTC-6, Patrick Noffke wrote:
> > On Thursday, February 19, 2015 at 2:14:45 PM UTC-6, Patrick Noffke wrote:
> > > On Monday, February 16, 2015 at 3:28:08 PM UTC-6, Simon Wright wrote:
> > > > Patrick Noffke writes: 
> > > >  
> > > > > Here's what happens now (the order of the interrupts may change 
> > > > > between runs, but this is for one capture): 
> > > > > 
> > > > > 1. UART interrupt triggers.  2. PO1's entry executes. 
> > > >  
> > > > because the entry body is executed in interrupt context. See 
> > > > below. 
> > > >  
> > > > > 3. SPI interrupt triggers twice (see below).  4. PO2's entry 
> > > > > executes.  5. T1 (UART task) executes.  This is the first thing 
> > > > > wrong.  T2 is higher priority than T1 so T2 should run first. 
> > > > > 6. T2 (SPI task) executes twice.  Upon the second execution, I 
> > > > > get a program error because Object.Entry_Queue is null.  The 
> > > > > exception is 
> > > >  
> > > > Entry_Queue is *not* null, as you said in the next post. 
> > > >  
> > > > > raised in s-tposen-raven.adb (line 167 in my copy) in 
> > > > > Protected_Single_Entry_Call. 
> > > > > 
> > > > > This may be relevant -- the SPI interrupt triggers twice.  This 
> > > > > is because the interrupt is for a DMA completion, and it fires 
> > > > > both when TX and RX complete (since it's SPI, they complete at 
> > > > > the same time).  I take care in my interrupt handler to release 
> > > > > the entry from only one of the two interrupts.  Perhaps with the 
> > > > > interrupt firing twice, the runtime may get confused and 
> > > > > activate the task twice (even though the entry only executes 
> > > > > once).  But for the above run, the entry was released during the 
> > > > > second SPI interrupt. 
> > > >  
> > > > The RTS does this (I hope I have it right): 
> > > >  
> > > >    The entry call (Protected_Single_Entry_Call):
> > > > 
> > > >      locks the entry
> > > >      if the barrier is open then
> > > >        asserts that Call_In_Progress isn't set
> > > >        sets Call_In_Progress
> > > >        calls the entry body wrapper
> > > >        clears Call_In_Progress
> > > >        unlocks the entry
> > > >      else
> > > >        if the Entry_Queue isn't null then
> > > >          unlocks the entry
> > > >          raises PE
> > > >        end if
> > > >        sets the Entry_Queue
> > > >        unlocks the entry
> > > >        sleeps
> > > >      end if
> > > > 
> > > >    The handler wrapper:
> > > > 
> > > >      locks the entry
> > > >      calls another wrapper for the handler itself
> > > >      calls Service_Entry
> > > >      exits
> > > > 
> > > >    Service_Entry:
> > > >      if the Entry_Queue is set and the barrier is open then
> > > >        clears the Entry_Queue
> > > >        asserts that Call_In_Progress isn't set
> > > >        sets Call_In_Progress
> > > >        calls the entry body wrapper
> > > >        clears Call_In_Progress
> > > >        saves the caller task_id
> > > >        unlocks the entry
> > > >        wakes the caller
> > > >      else
> > > >        unlocks the entry
> > > >      end if
> > > > 
> > > > I really don't see how the sequnece you describe happens!
> > > > 
> > > > One thing that puzzles me is the locking/unlocking of the entry: this is
> > > > done (in that RTS) by raising the caller task's priority to the ceiling
> > > > priority of the task, if necessary. So what about interrupts? And when
> > > > the handler wrapper (you can see this by compiling the package with the
> > > > PO in with -gnatdg) locks the entry, it seems to raise the current
> > > > task's priority, where the current task has nothing to do with the PO at
> > > > all!
> > > > 
> > > 
> > > Thank you for all this!  It helps a lot.  I didn't know about -gnatdg -- very useful.
> > > 
> > > I suspect the problem may stem from the fact that Leave_Kernel in s-bbprot.adb can insert a suspended task into the thread queue.  I am guessing somehow this is tripping up the runtime when multiple tasks become runnable at the same time.
> > > 
> > > I stepped through the debugger at startup, and I can see the suspended task going into the queue.  Then another task is woken up and put at the front of the queue, so the first task is Runnable and the Next task is Suspended.  Then when Leave_Kernel resumes running (it enables interrupts after inserting the suspended task) it may not call Extract (since the running thread state is Runnable) -- it never considers that the Suspended task it inserted might be later in the queue.
> > > 
> > > I haven't yet been able to directly correlate the Leave_Kernel behavior with the task incorrectly waking up twice.  I'll keep trying things, but I wanted to share what I found so far.
> > > 
> > > What I have confirmed is that I can get the system into a state where there are two Runnable tasks in the thread queue, and the "Next" field of the last one points to the first one.  That is:
> > > 
> > > First_Thread_Table (CPU_Id) = First_Thread_Table (CPU_Id).Next.Next
> > > 
> > > Right before this happens is when the two tasks are woken up at the same time.
> > > 
> > > It appears the task that runs twice (when I get the Program_Error I reported earlier) is the one that gets put in the queue in the suspended state at startup (an empirical data point after changing priorities of my tasks).
> > > 
> > 
> > I think I know what's going on.  There is a problem with the Insert procedure in s-bbthqu.adb.  If (1) a thread is already in the queue, (2) it is not at the head of the queue, and (3) the priority is greater than the thread at the head of the queue, it can get inserted twice.  The error is in the "elsif" condition of this code:
> > 
> >      if First_Thread_Table (CPU_Id) = Thread then
> >          null;
> > 
> >       --  Insert at the head of queue if there is no other thread with a higher
> >       --  priority.
> > 
> >       elsif First_Thread_Table (CPU_Id) = Null_Thread_Id
> >         or else
> >           Thread.Active_Priority > First_Thread_Table (CPU_Id).Active_Priority
> >       then
> >          Thread.Next := First_Thread_Table (CPU_Id);
> >          First_Thread_Table (CPU_Id) := Thread;
> > 
> >       --  Middle or tail insertion
> > 
> >       else
> >          --  Look for the Aux_Pointer to insert the thread just after it
> >          ...
> > 
> > Here is what's happening in my case:
> > 
> > (1) UART is at the head of the queue in the suspended state with priority 10.  Leave_Kernel code is pending after calling Enable_Interrupts.
> > (2) SPI interrupt fires.
> > (3) SPI task priority is 252 (interrupt priority).
> > (4) SPI task gets inserted at head of queue in the Runnable state.
> > (5) SPI task priority gets adjusted to 15 when Unlock_Entry is called.  It is left at the head of the queue.  SPI_task.Next = UART_task.
> > (6) UART interrupt fires.
> > (7) UART task priority is 250, and task is set to Runnable.
> > (8) UART task gets inserted at head of queue since its priority is greater than SPI_task.  Now UART_task.Next = SPI_task, and SPI_task.Next = UART_task.
> > (9) UART task priority is adjusted to 10 when Unlock_Entry is called.  Now SPI_task is head of queue, and SPI_task.Next = UART_task.  UART_task.Next = UART_task.
> > (10) SPI task runs.
> > (11) UART task runs twice.  Boom.
> > 
> > I think the Insert procedure needs to be modified to look through the entire queue for the thread to be inserted.  Might be simplest to just remove it if it exists before entering the if statement.
> > 
> 
> This version of the entire Insert procedure works for me:
> 
>    ------------
>    -- Insert --
>    ------------
> 
>    procedure Insert (Thread : Thread_Id) is
>       Aux_Pointer : Thread_Id;
>       CPU_Id      : constant CPU := Get_CPU (Thread);
> 
>    begin
> 
>       --  ??? This pragma is disabled because the Tasks_Activated only
>       --  represents the end of activation for one package not all the
>       --  packages. We have to find a better milestone for the end of
>       --  tasks activation.
> 
>       --  --  A CPU can only insert alarm in its own queue, except during
>       --  --  initialization.
> 
>       --  pragma Assert (CPU_Id = Current_CPU or else not Tasks_Activated);
> 
>       --  It may be the case that we try to insert a task that is already in
>       --  the queue. This can only happen if the task was not runnable and its
>       --  context was being used for handling an interrupt. Hence, if the task
>       --  is already in the queue and we try to insert it, we need to check
>       --  whether it is in the correct place.
> 
>       --  No insertion if the task is already at the head of the queue
> 
>       if First_Thread_Table (CPU_Id) = Thread then
>          null;
> 
>          --  Insert at the head of queue if there is no other thread
>          --  with a higher priority.
> 
>       elsif First_Thread_Table (CPU_Id) = Null_Thread_Id then
>          Thread.Next := First_Thread_Table (CPU_Id);
>          First_Thread_Table (CPU_Id) := Thread;
> 
>       else
>          --  Middle or tail insertion
> 
>          --  Remove the thread if it is already in the queue.  We know
>          --  the first thread is not null.
>          Aux_Pointer := First_Thread_Table (CPU_Id);
>          while Aux_Pointer.Next /= Null_Thread_Id
>            and then Aux_Pointer.Next /= Thread
>          loop
>             Aux_Pointer := Aux_Pointer.Next;
>          end loop;
> 
>          if Aux_Pointer.Next = Thread then
>             Aux_Pointer.Next := Thread.Next;
>          end if;
> 
>          if
>            Thread.Active_Priority > First_Thread_Table (CPU_Id).Active_Priority
>          then
>             Thread.Next := First_Thread_Table (CPU_Id);
>             First_Thread_Table (CPU_Id) := Thread;
>          else
>             --  Look for the Aux_Pointer to insert the thread just after it
> 
>             Aux_Pointer := First_Thread_Table (CPU_Id);
>             while Aux_Pointer.Next /= Null_Thread_Id
>               and then Aux_Pointer.Next.Active_Priority >=
>               Thread.Active_Priority
>             loop
>                Aux_Pointer := Aux_Pointer.Next;
>             end loop;
> 
>             --  Insert the thread after the Aux_Pointer
> 
>             Thread.Next := Aux_Pointer.Next;
>             Aux_Pointer.Next := Thread;
>          end if;
> 
>       end if;
>    end Insert;

FYI - This problem is not fixed in GNAT GPL 2015.  I reported it to AdaCore back in February.