GNAT and large number of threads

GNAT and large number of threads

[Wiktor Moskwa]

Hi,

I'm using GNAT GPL 2006 on Linux x86.
Recently I had a problem with creating large number of tasks(*) in my 
Ada program. I create tasks in a loop and when I reach the limit of
stack address space the main thread freezes - no exceptions, nothing.

Using ltrace tool reveals:
...
pthread_create(0x9656e98, 0xbffca3d0, 0x80c3280, 0x9656e58, 10256) = 0
pthread_create(0x965a670, 0xbffca3d0, 0x80c3280, 0x965a630, 10256) = 0
pthread_create(0xbfd86048, 0xbffca3d0, 0x80c3280, 0xbfd86008, 10256)= 12
--- SIGSEGV (Segmentation fault) --- 

Error 12 is ENOMEM (the system lacked the necessary resources to
create another thread). GNAT seems to completely ignore this error.
Quick search in GNAT GPL sources shows that only EAGAIN error is taken
into account, others are silently ignored.
=======================================================================
[file: s-taprop-linux.adb; procedure: Create_Task]
Result := pthread_create
  (T.Common.LL.Thread'Access,
   Attributes'Access,
   Thread_Body_Access (Wrapper),
   To_Address (T));
pragma Assert (Result = 0 or else Result = EAGAIN);

Succeeded := Result = 0;

Result := pthread_attr_destroy (Attributes'Access);
pragma Assert (Result = 0);

Set_Priority (T, Priority);
=======================================================================
Succeeded (out paramter) is set to False in my case but I guess that
pthread_attr_destroy or Set_Priority cannot be invoked after failure
of pthread_create. I'm not sure because I don't know neither pthreads
nor GNAT compiler. 
Nevertheless it looks like a GNAT bug to me, what do you think?

My question is how can my program predict/calculate (approximetly) the 
maximum number of tasks that can be created with known maximum stack 
size per task (specified with pragma Storage_Size)?

[*] I'm writing a simulator of a distributed protocol wiht one task per
node so I really need large number of threads to simulate many nodes.

Thanks,
Wiktor

-- 
Wiktor Moskwa

Re: GNAT and large number of threads

[Gene]

On Apr 26, 6:50 pm, Wiktor Moskwa <wiktorDOTmos...@gmail.com> wrote:
> Hi,
>
> I'm using GNAT GPL 2006 on Linux x86.
> Recently I had a problem with creating large number of tasks(*) in my
> Ada program. I create tasks in a loop and when I reach the limit of
> stack address space the main thread freezes - no exceptions, nothing.
>
> Using ltrace tool reveals:
> ...
> pthread_create(0x9656e98, 0xbffca3d0, 0x80c3280, 0x9656e58, 10256) = 0
> pthread_create(0x965a670, 0xbffca3d0, 0x80c3280, 0x965a630, 10256) = 0
> pthread_create(0xbfd86048, 0xbffca3d0, 0x80c3280, 0xbfd86008, 10256)= 12
> --- SIGSEGV (Segmentation fault) ---
>
> Error 12 is ENOMEM (the system lacked the necessary resources to
> create another thread). GNAT seems to completely ignore this error.
> Quick search in GNAT GPL sources shows that only EAGAIN error is taken
> into account, others are silently ignored.
> =======================================================================
> [file: s-taprop-linux.adb; procedure: Create_Task]
> Result := pthread_create
>   (T.Common.LL.Thread'Access,
>    Attributes'Access,
>    Thread_Body_Access (Wrapper),
>    To_Address (T));
> pragma Assert (Result = 0 or else Result = EAGAIN);
>
> Succeeded := Result = 0;
>
> Result := pthread_attr_destroy (Attributes'Access);
> pragma Assert (Result = 0);
>
> Set_Priority (T, Priority);
> =======================================================================
> Succeeded (out paramter) is set to False in my case but I guess that
> pthread_attr_destroy or Set_Priority cannot be invoked after failure
> of pthread_create. I'm not sure because I don't know neither pthreads
> nor GNAT compiler.
> Nevertheless it looks like a GNAT bug to me, what do you think?
>
> My question is how can my program predict/calculate (approximetly) the
> maximum number of tasks that can be created with known maximum stack
> size per task (specified with pragma Storage_Size)?
>
> [*] I'm writing a simulator of a distributed protocol wiht one task per
> node so I really need large number of threads to simulate many nodes.
>
> Thanks,
> Wiktor
>
> --
> Wiktor Moskwa

I can't answer your question directly, but the approach of using tasks
as abstract data structures is asking for trouble.

Tasks are appropriate when you _need_ independent computation threads:
I/O with multiple devices, editing a file while a copy is being
written to disk, you get the idea.  Protocol simulation doesn't fit
this bill.  Nodes are essentially data. If you have only a single
processor, you should run in the environment task.  If you need to
exploit an MP environment, then use a task pool of N tasks for N
processors.

In other words, represent data with data structures and computation
threads with tasks.  For your problem, you might want to represent
each node as a set of input and output queues.  Apply one (or more)
computation thread(s) to identify an appropriate node, remove inputs,
create and add outputs to respective queues, then move on to another
node. A heap keyed on number of inputs left to define will always have
a node with all inputs defined at the top. Multiple threads from a
pool work exactly the same way except the queues are protected for
synchronization.

Re: GNAT and large number of threads

[Jeffrey R. Carter]

Wiktor Moskwa wrote:
> 
> I'm using GNAT GPL 2006 on Linux x86.
> Recently I had a problem with creating large number of tasks(*) in my 
> Ada program. I create tasks in a loop and when I reach the limit of
> stack address space the main thread freezes - no exceptions, nothing.

Are you compiling with -fstack-check?

Note that without -gnato -fstack-check, GNAT is not Ada.

-- 
Jeff Carter
"I fart in your general direction."
Monty Python & the Holy Grail
05

Re: GNAT and large number of threads

[Stefan Bellon]

Gene wrote:
> On Apr 26, 6:50 pm, Wiktor Moskwa <wiktorDOTmos...@gmail.com> wrote:

> > I'm using GNAT GPL 2006 on Linux x86.
> > Recently I had a problem with creating large number of tasks(*) in
> > my Ada program. I create tasks in a loop and when I reach the limit
> > of stack address space the main thread freezes - no exceptions,
> > nothing.

I made the same observation with GNAT Pro 6.0.1 on x86 GNU/Linux. See
posting <20070418201307.18a85fd9@cube.tz.axivion.com> where I still
thought it's just an effect of context switching eating up all the
time. Later on I noticed that from a certain number of tasks on,
nothing happens anymore at all.

> If you need to exploit an MP environment, then use a task pool of N
> tasks for N processors.

Is there a way of finding out the number of available processors in Ada
(or GNAT) that works at least on GNU/Linux, Windows, and Solaris?

-- 
Stefan Bellon

Re: GNAT and large number of threads

[Wiktor Moskwa]

On 27.04.2007, Jeffrey R. Carter <jrcarter@acm.org> wrote:
> Wiktor Moskwa wrote:
>> I'm using GNAT GPL 2006 on Linux x86.
>> Recently I had a problem with creating large number of tasks(*) in my 
>> Ada program. I create tasks in a loop and when I reach the limit of
>> stack address space the main thread freezes - no exceptions, nothing.
>
> Are you compiling with -fstack-check?
>
> Note that without -gnato -fstack-check, GNAT is not Ada.
>

Yes and additionaly with -gnatf -gnatwa.

-- 
Wiktor Moskwa

Re: GNAT and large number of threads

[Wiktor Moskwa]

On 26.04.2007, Gene <gene.ressler@gmail.com> wrote:
>
> I can't answer your question directly, but the approach of using tasks
> as abstract data structures is asking for trouble.
>
> Tasks are appropriate when you _need_ independent computation threads:
> I/O with multiple devices, editing a file while a copy is being
> written to disk, you get the idea.  Protocol simulation doesn't fit
> this bill.  Nodes are essentially data. If you have only a single
> processor, you should run in the environment task.  If you need to
> exploit an MP environment, then use a task pool of N tasks for N
> processors.
>
> In other words, represent data with data structures and computation
> threads with tasks.  For your problem, you might want to represent
> each node as a set of input and output queues.  Apply one (or more)
> computation thread(s) to identify an appropriate node, remove inputs,
> create and add outputs to respective queues, then move on to another
> node. 

I decided to use one thread per node to simulate nodes executing
"in the same time" (in reality sequentially but at least with 
interleaving operations). An operating system schedules my threads and 
switches between them so I don't have to choose which node should be 
processed.

Thank you for your advice, it showed me new way to look at such
applications. Unfortunately I lack some experience in multi-threaded 
programming.

> A heap keyed on number of inputs left to define will always have
> a node with all inputs defined at the top. Multiple threads from a
> pool work exactly the same way except the queues are protected for
> synchronization.

It would be a good solution to scheduling if nodes produced output
messages based only on input messages. In case of this protocol(*)
every node periodically produces some messages to discover new nodes
in the network and to update internal references to other nodes.

It could be simplified that every N seconds worker threads execute
periodic procedures for all nodes and then go back to normal processing
based on a priority queue that you suggested. But I have no idea (yet) 
how to do it in case when periodic procedures can be invoked for each
node at different time.
Another thing to consider is how to modify this algorithm to reduce
chance of starving some nodes.

Thanks for answer and ideas,
Wiktor

[*] Chord distributed hash lookup protocol

-- 
Wiktor Moskwa

Re: GNAT and large number of threads

[Alex R. Mosteo]

Wiktor Moskwa wrote:

> Hi,
> 
> I'm using GNAT GPL 2006 on Linux x86.
> Recently I had a problem with creating large number of tasks(*) in my
> Ada program. I create tasks in a loop and when I reach the limit of
> stack address space the main thread freezes - no exceptions, nothing.

This happened to me in 3.15p for windows long ago, but I suppose it's the
same issue.

You can have thousands of tasks with a reasonable stack size. I don't know
how to know the number in advance, sorry.

Re: GNAT and large number of threads

[Pascal Obry]

Alex R. Mosteo a �crit :
> You can have thousands of tasks with a reasonable stack size. I don't know
> how to know the number in advance, sorry.

You need to know precisely the tasks stack size then divide 2GB by this
number. This should give you a good approximation.

Pascal.

-- 

--|------------------------------------------------------
--| Pascal Obry                           Team-Ada Member
--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE
--|------------------------------------------------------
--|              http://www.obry.net
--| "The best way to travel is by means of imagination"
--|
--| gpg --keyserver wwwkeys.pgp.net --recv-key C1082595