Re: RFC: Prototype for a user threading library in Ada

[rieachus@comcast.net]

On Friday, June 17, 2016 at 5:44:18 AM UTC-4, Hadrien Grasland wrote:
> So, a while ago, after playing with the nice user-mode threading libraries that are available these days in C++, like Intel TBB and HPX, I thought it would be nice if Ada had something similar.

I've been following this thread for a while, and I keep thinking that there is a lot of barking up the wrong tree.  But then again, I am used to thinking about the size of problem that takes supercomputers.  Ada is a decent fit but the real issues are language independent--and as you may have grown tired of seeing me say, the optimal program--in any language will finish within a few seconds of the estimate from only looking at moving data.  Even if you have thousands of CPU hours of number crunching to do, your program looks like this:

Distribute the program code to the correct number of nodes.
Start the program.  The nodes will collect the data from the data storage system, interact (with hopefully only adjacent nodes) to complete the computation.
Collect and assemble the finished data for presentation.

The hard part of supercomputer programming today is not writing the code that does the work, it is distributing the program and data, and then assembling the results.

How to do this?  Let's forget about Ada tasking for a moment.  It may be great for tying together the (relatively small) number of CPUs at a node which share memory, and in some cases even cache memory.  What you need to start with assuming you wrote your program as a thousand or more distributed annex programs, is either to use tools that are part of the supercomputer to distribute thousands of copies of the identical code to different nodes, or to create your program with a hierarchical tree structure.  Otherwise you will spend all your time just getting the code distributed from a single node.

The other end of the program presents the same issues.  Imagine you want to simulate the evolution of the universe over a space that grows to a cube a million parsecs on a side and a period of a billion years.  After each time step you want to collect the data for that epoch for later viewing.  Again, if all the nodes are hammering on a single file, your program won't run very fast.

One solution is to store result data on a per node basis, then when the program is finished, run a program that converts from slices through time to 3d images at a given point in time.  This program may run several times as long as your big number crunching program.  But you can do it on your workstation over the weekend. ;-)

Another, tricky but it works, is to skew time in the main program.  This can result in duplicate number crunching at the boundaries, but as I keep saying, that is not the problem.  Now you feed the data into files, but you have several hundred files corresponding to different points in time, all open and collecting data.

Why post all this here?  The Ada distributed annex is actually a good fit for the tools available on most supercomputers.  Combining the distributed annex with Ada tasking on the nodes provide by modern CPUs, is a good fit.  Unfortunately, or we just have to learn to live with it, supercomputers are being taken over by GPUs.  The CPUs are still there, but they end up delegated to moving data between nodes, and doing whatever synchronization is needed.

Technically the tools are there to write code in high-level languages and run it on GPUs.  But right now, you end up with lots of structural artifacts in your program that make it non-portable.  (The biggest of these is the number of shader processors per GPU.  AMD has done a nice thing in breaking the shaders into groups of 64 on all their cards.  Helps a lot when working on a single machine, but...  Besides, right now most supercomputers use nVidia cards.  This may change if AMD gets their GFLOPS/watt down to where nVidia is.)