Re: Fun with Tasking

["Jeffrey R. Carter" <jrcarter@acm.org>]

mhamel_98@yahoo.com wrote:
> 
> Anyhoo Jeff, I'll try to explain in more detail what my program is
> doing and hopefully you can see where I'm going wrong.  The program
> has a number of packages, each handling what I call a subsystem, to be
> verbose: ephemeris, residual, vector, covariance and a few others
> (this sort of equates to the "exec" procedure in my sample).  Each of
> these systems has a number of procedures of interest, like a
> differencing of quantities, a statistical analysis, etc.  Now each of
> these procedures of each of these subsystems can have up to 60 (and
> more as we go on) objects - each one I handle as it's own task.  I've
> found that throwing 60 threads at the OS (W2K in this case) is, while
> better then serially and single-threaded on multi-cpu's, not terribly
> efficient.  For this application, I've found most procedures operate
> best if N+1 threads are running, N being the processors available.
> The output is all .csv files.  For awhile, to show an ensemble of
> data, I would do a lot of cut and pasting between .csv's, one day, I
> thought, why not have the program do this?  So I implemented these
> "collating" tasks that all 60 or so threads would send some
> information to for some sort of unified processing.  Each procedure of
> each subsystem/package is slightly different either in the data types
> being handled or the manner in which the data is processed so a
> "generic" collating task isn't really feasible.
> 
> That's what's happening, now how do I handle it?  As the program
> starts, I have each subsystem loads its procedures (via procedure
> pointers) to a task manager.  A command is recieved via CLI and the
> task manager then inserts, let's say, 60 copies of a specified
> procedure into a list which are doled out to worker threads so that
> only N+1 are active at any time.  The issue I'm having is, how do you
> have a unique collating task created for those 60 processes to send
> updates to?  As it is now, I have all of these (nearly a dozen) unique
> collating tasks defined in the task manager.  I would rather have them
> defined in the subsystem package to which they really belong.  As the
> subsystem packages "push" their procedures to the task manager, the
> task manager has no visibility into the subsystem.  My aim was to
> define an area (a System.Address for example) in a control block
> record that the task manager would have space for "a task".  What this
> task was, what it did, where it came from the task manager would not
> know nor care, but it would be the owner, or master, in some sense so
> the "preprocess" or initialization component could exit.  I hope this
> makes a little sense.

There's still a lot of implementation in here. Talking about procedure 
pointers and addresses is not only implementation, it's quite low level. 
One of Ada's great strengths is its ability to express, implement, and 
use high-level abstractions. The low-level stuff is still available, but 
generally best hidden well down in the implementation of such an 
abstraction.

What I get from your description is: You have classes of computations 
("subsystems"); each class can have numerous computations carried on at 
the same time; each computation produces output; you want some of the 
output from all the computations of the same class to be subjected to 
collating.

Everything else seems like implementation detail that may be blocking 
your view of a solution.

Steve has given you some ideas about how to abstract the idea of 
computations and controlling their scheduling. He didn't have get into 
the resulting output or how to subject some of it, within a class of 
computations, to collating.

I would probably take a different approach than Steve, with different 
subsystems more independent, in order to ease the collating within the 
subsystems, but that may be more a matter of style than substance.

If you can produce high-level package specifications that only address 
the high-level concepts of your problem, without worrying about 
implementation (private parts, bodies), with a road map of how they work 
together, you'll probably find that you can then work out a way to 
implement them.

-- 
Jeff Carter
"Don't knock masturbation. It's sex with someone I love."
Annie Hall
45