Re: How to optimize use of RAM/disk access ?

[reinert <reinkor@gmail.com>]

Hello, I have been away for a while.

I find no memory leak in my program (it is easy to check memory use under linux - for example via "top").

In general I would like to have an Ada function which tells how much memory (in general hardware resources) is available. Maybe there is a "system function" I have overlooked?

reinert 


On Wednesday, January 24, 2018 at 4:18:25 PM UTC+1, Robert Eachus wrote:
> On Saturday, January 20, 2018 at 1:16:00 AM UTC-5, reinert wrote:
> > 
> > Any hint for how I can optimize a bit smarter?
> > 
> 
> First, realize that you are not the only one with this problem.  There are programs that run on supercomputers for megayears of CPU time.  It may take weeks (often on smaller systems) to figure out the "right" parameters for a given run.  Why does it take so long?  The usual approach is to create a linear regression model usually with linear and squared values for each model parameter, and sometimes cross-products.  Now take your regression model and choose enough test points to get a decent result.  Usually this is on the order of three or four data points for each model parameter.  For example, your model might be t/p = 1/m + 1/m^2 + 1/N+ 1/N^2 + s + s^2 + s^3 + d/s + d/(s^2) where
> t is time in seconds per iteration, p is the number of processors, m is memory size per CPU core in Gigabytes, N is an internal model sizing parameter, s is problem size in data points, and d is total (free) disk space in Gigabytes.
> 
> Now you pick say 30 or so points, including some where you expect the model to crash or run impossibly slow.  Do the runs, with probably a 1000 second limit per iteration per run.  Now eliminate any time outs or crashes (you are not going to do big runs in that parameter space) and find the parameter values.  From experience you are going to repeat the experiment on the big machine, with test parameters close to what you expect on a full run, but again with one to a few time steps.
> 
> Now you know enough to ask for time (and number of CPU cores) on the big machine.  Today, you will probably want to try running on both the CPU cores and on the GPUs.
> 
> Is this a lot of work?  Sure, but if it saves a few CPU centuries, it is worth the effort.
> 
> In your case, you might want to "fool around" with various model parameters that are combinations of your N and memory per CPU.  Oh, and I often have algorithm parameters which correspond to L1, L2 and L3 data cache sizes.  A typical result for a "simple" matrix multiplication (A*B=C) might have A fitted to L1, and B to L2.  If you are doing something expressed in linear algebra, check out the ATLAS version of the BLAS library: http://math-atlas.sourceforge.net/  The big advantage of using ATLAS is that it will give good results for ALL > n^2 functions in terms of matrix multiplication.  So even if you use some other BLAS, you can use the ATLAS libraries for some LAPACK calls.  (There are several Ada bindings to BLAS floating around.  I'm not choosing one, since your choices of OS and compiler will affect your choice.)
> 
> Too much information?  Probably.  But if you do have a program that requires CPU years to run, or one that can be simplified by using LAPACK or BLAS?  Have at it.