From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM autolearn=unavailable autolearn_force=no version=3.4.4 X-Received: by 10.107.184.139 with SMTP id i133mr3788591iof.85.1518240451764; Fri, 09 Feb 2018 21:27:31 -0800 (PST) X-Received: by 10.157.40.82 with SMTP id h18mr214368otd.13.1518240451674; Fri, 09 Feb 2018 21:27:31 -0800 (PST) Path: eternal-september.org!reader01.eternal-september.org!reader02.eternal-september.org!feeder.eternal-september.org!news.uzoreto.com!weretis.net!feeder6.news.weretis.net!feeder.usenetexpress.com!feeder-in1.iad1.usenetexpress.com!border1.nntp.dca1.giganews.com!nntp.giganews.com!w142no151547ita.0!news-out.google.com!s63ni197itb.0!nntp.google.com!o66no151987ita.0!postnews.google.com!glegroupsg2000goo.googlegroups.com!not-for-mail Newsgroups: comp.lang.ada Date: Fri, 9 Feb 2018 21:27:31 -0800 (PST) In-Reply-To: <1892f04b-0223-4060-90a7-91983f775f18@googlegroups.com> Complaints-To: groups-abuse@google.com Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=85.167.212.79; posting-account=bPTmZAoAAAC_6HP9XLKB9aAAxBa6BuOR NNTP-Posting-Host: 85.167.212.79 References: <1892f04b-0223-4060-90a7-91983f775f18@googlegroups.com> User-Agent: G2/1.0 MIME-Version: 1.0 Message-ID: <752bb990-b70a-4fce-8cd7-91d4f8ee7ea1@googlegroups.com> Subject: Re: How to optimize use of RAM/disk access ? From: reinert Injection-Date: Sat, 10 Feb 2018 05:27:31 +0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Xref: reader02.eternal-september.org comp.lang.ada:50357 Date: 2018-02-09T21:27:31-08:00 List-Id: Hello, I have been away for a while. I find no memory leak in my program (it is easy to check memory use under l= inux - for example via "top"). In general I would like to have an Ada function which tells how much memory= (in general hardware resources) is available. Maybe there is a "system fun= ction" I have overlooked? reinert=20 On Wednesday, January 24, 2018 at 4:18:25 PM UTC+1, Robert Eachus wrote: > On Saturday, January 20, 2018 at 1:16:00 AM UTC-5, reinert wrote: > >=20 > > Any hint for how I can optimize a bit smarter? > >=20 >=20 > First, realize that you are not the only one with this problem. There ar= e programs that run on supercomputers for megayears of CPU time. It may ta= ke weeks (often on smaller systems) to figure out the "right" parameters fo= r a given run. Why does it take so long? The usual approach is to create = a linear regression model usually with linear and squared values for each m= odel parameter, and sometimes cross-products. Now take your regression mod= el and choose enough test points to get a decent result. Usually this is o= n the order of three or four data points for each model parameter. For exa= mple, your model might be t/p =3D 1/m + 1/m^2 + 1/N+ 1/N^2 + s + s^2 + s^3 = + d/s + d/(s^2) where > t is time in seconds per iteration, p is the number of processors, m is m= emory size per CPU core in Gigabytes, N is an internal model sizing paramet= er, s is problem size in data points, and d is total (free) disk space in G= igabytes. >=20 > Now you pick say 30 or so points, including some where you expect the mod= el to crash or run impossibly slow. Do the runs, with probably a 1000 seco= nd limit per iteration per run. Now eliminate any time outs or crashes (yo= u are not going to do big runs in that parameter space) and find the parame= ter values. From experience you are going to repeat the experiment on the = big machine, with test parameters close to what you expect on a full run, b= ut again with one to a few time steps. >=20 > Now you know enough to ask for time (and number of CPU cores) on the big = machine. Today, you will probably want to try running on both the CPU core= s and on the GPUs. >=20 > Is this a lot of work? Sure, but if it saves a few CPU centuries, it is = worth the effort. >=20 > In your case, you might want to "fool around" with various model paramete= rs that are combinations of your N and memory per CPU. Oh, and I often hav= e algorithm parameters which correspond to L1, L2 and L3 data cache sizes. = A typical result for a "simple" matrix multiplication (A*B=3DC) might have= A fitted to L1, and B to L2. If you are doing something expressed in line= ar algebra, check out the ATLAS version of the BLAS library: http://math-at= las.sourceforge.net/ The big advantage of using ATLAS is that it will give= good results for ALL > n^2 functions in terms of matrix multiplication. S= o even if you use some other BLAS, you can use the ATLAS libraries for some= LAPACK calls. (There are several Ada bindings to BLAS floating around. I= 'm not choosing one, since your choices of OS and compiler will affect your= choice.) >=20 > Too much information? Probably. But if you do have a program that requi= res CPU years to run, or one that can be simplified by using LAPACK or BLAS= ? Have at it.