* How to optimize use of RAM/disk access ? @ 2018-01-20 6:15 reinert 2018-01-20 8:45 ` Jacob Sparre Andersen ` (3 more replies) 0 siblings, 4 replies; 18+ messages in thread From: reinert @ 2018-01-20 6:15 UTC (permalink / raw) Hello, I am developing an Ada program which I want automatically, more or less, to optimize its use of RAM and disk access. Assume a number, N, controls/defines the size of a data set (for example N represents number of images which the program stores internally). Just now I am guessing a number, N, which I experience make my computer run the program the best way. However, when I change computer, I have to guess again :-) Any hint for how I can optimize a bit smarter? reinert ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-01-20 6:15 How to optimize use of RAM/disk access ? reinert @ 2018-01-20 8:45 ` Jacob Sparre Andersen 2018-01-21 7:26 ` reinert 2018-01-22 8:53 ` gautier_niouzes ` (2 subsequent siblings) 3 siblings, 1 reply; 18+ messages in thread From: Jacob Sparre Andersen @ 2018-01-20 8:45 UTC (permalink / raw) reinert <reinkor@gmail.com> writes: > I am developing an Ada program which I want automatically, more or > less, to optimize its use of RAM and disk access. Assume a number, N, > controls/defines the size of a data set (for example N represents > number of images which the program stores internally). Just now I am > guessing a number, N, which I experience make my computer run the > program the best way. However, when I change computer, I have to guess > again :-) > > Any hint for how I can optimize a bit smarter? Let your operating system worry about those things. Load what you need into the address space of your program, and leave it to the operating system and CPU to decide what is stored on which cache level between disk and CPU registers. Greetings, Jacob -- »Saving keystrokes is the job of the text editor, not the programming language.« -- Preben Randhol ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-01-20 8:45 ` Jacob Sparre Andersen @ 2018-01-21 7:26 ` reinert 2018-01-21 23:49 ` Shark8 2018-01-22 5:53 ` darkestkhan 0 siblings, 2 replies; 18+ messages in thread From: reinert @ 2018-01-21 7:26 UTC (permalink / raw) The reason I asked is that I develop the program on a desktop computer where it runs fast. When I run it on a laptop, it goes slow and "free-ing" pointers seems to make it run easier (on the laptop which seems to have slower disk (?)). reinert ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-01-21 7:26 ` reinert @ 2018-01-21 23:49 ` Shark8 2018-01-22 5:53 ` darkestkhan 1 sibling, 0 replies; 18+ messages in thread From: Shark8 @ 2018-01-21 23:49 UTC (permalink / raw) On Sunday, January 21, 2018 at 12:26:17 AM UTC-7, reinert wrote: > The reason I asked is that I develop the program on a desktop computer where it runs fast. When I run it on a laptop, it goes slow and "free-ing" pointers seems to make it run easier (on the laptop which seems to have slower disk (?)). The variance of HW isn't something you should be optimizing for on a standard "user-level" program; that's the job of the OS which provides the "general interface" for the program you're writing. (There are exceptions, such as lower-level utilities.) The best advice I can give, at such a high level (and with as little information as you provided), is to take a look at your algorithms and data-structures and ensure those are appropriate -- in this vein, you can use pools/subpools to free "everything" at that pool's level at once via clever use of scoping. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-01-21 7:26 ` reinert 2018-01-21 23:49 ` Shark8 @ 2018-01-22 5:53 ` darkestkhan 1 sibling, 0 replies; 18+ messages in thread From: darkestkhan @ 2018-01-22 5:53 UTC (permalink / raw) On Sunday, January 21, 2018 at 7:26:17 AM UTC, reinert wrote: > The reason I asked is that I develop the program on a desktop computer where it runs fast. When I run it on a laptop, it goes slow and "free-ing" pointers seems to make it run easier (on the laptop which seems to have slower disk (?)). > > reinert Do you have memory leaks? Cause it sounds like this. Also make sure that you don't allocate gigabytes of data... laptops tend to have less RAM. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-01-20 6:15 How to optimize use of RAM/disk access ? reinert 2018-01-20 8:45 ` Jacob Sparre Andersen @ 2018-01-22 8:53 ` gautier_niouzes 2018-01-22 15:01 ` Mr. Man-wai Chang 2018-01-24 15:18 ` Robert Eachus 3 siblings, 0 replies; 18+ messages in thread From: gautier_niouzes @ 2018-01-22 8:53 UTC (permalink / raw) It would be better to store the guessworked parameter in a config file (*) that would be set up once per machine. These RAM - Disk things are tricky anyway (for instance, disk is cached in its turn by RAM!) and the RAM usage by other processes can fluctuate massively during the run-time of your program, so even if your program thinks having the right amount at time t0 it might be way off at time t1. ___ (*) shameless plug: https://sf.net/projects/ini-files/ ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-01-20 6:15 How to optimize use of RAM/disk access ? reinert 2018-01-20 8:45 ` Jacob Sparre Andersen 2018-01-22 8:53 ` gautier_niouzes @ 2018-01-22 15:01 ` Mr. Man-wai Chang 2018-01-24 15:18 ` Robert Eachus 3 siblings, 0 replies; 18+ messages in thread From: Mr. Man-wai Chang @ 2018-01-22 15:01 UTC (permalink / raw) On 20/1/2018 14:15, reinert wrote: > Hello, > > I am developing an Ada program which I want automatically, more or less, to optimize its use of RAM and disk access. Assume a number, N, controls/defines the size of a data set (for example N represents number of images which the program stores internally). Just now I am guessing a number, N, which I experience make my computer run the program the best way. However, when I change computer, I have to guess again :-) You sure your algorithm could do a better job than the operating system in which your compiled programs run? -- @~@ Remain silent! Drink, Blink, Stretch! Live long and prosper!! / v \ Simplicity is Beauty! /( _ )\ May the Force and farces be with you! ^ ^ (x86_64 Ubuntu 9.10) Linux 2.6.39.3 不借貸! 不詐騙! 不援交! 不打交! 不打劫! 不自殺! 請考慮綜援 (CSSA): http://www.swd.gov.hk/tc/index/site_pubsvc/page_socsecu/sub_addressesa ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-01-20 6:15 How to optimize use of RAM/disk access ? reinert ` (2 preceding siblings ...) 2018-01-22 15:01 ` Mr. Man-wai Chang @ 2018-01-24 15:18 ` Robert Eachus 2018-02-10 5:27 ` reinert 3 siblings, 1 reply; 18+ messages in thread From: Robert Eachus @ 2018-01-24 15:18 UTC (permalink / raw) On Saturday, January 20, 2018 at 1:16:00 AM UTC-5, reinert wrote: > > Any hint for how I can optimize a bit smarter? > First, realize that you are not the only one with this problem. There are programs that run on supercomputers for megayears of CPU time. It may take weeks (often on smaller systems) to figure out the "right" parameters for a given run. Why does it take so long? The usual approach is to create a linear regression model usually with linear and squared values for each model parameter, and sometimes cross-products. Now take your regression model and choose enough test points to get a decent result. Usually this is on the order of three or four data points for each model parameter. For example, your model might be t/p = 1/m + 1/m^2 + 1/N+ 1/N^2 + s + s^2 + s^3 + d/s + d/(s^2) where t is time in seconds per iteration, p is the number of processors, m is memory size per CPU core in Gigabytes, N is an internal model sizing parameter, s is problem size in data points, and d is total (free) disk space in Gigabytes. Now you pick say 30 or so points, including some where you expect the model to crash or run impossibly slow. Do the runs, with probably a 1000 second limit per iteration per run. Now eliminate any time outs or crashes (you are not going to do big runs in that parameter space) and find the parameter values. From experience you are going to repeat the experiment on the big machine, with test parameters close to what you expect on a full run, but again with one to a few time steps. Now you know enough to ask for time (and number of CPU cores) on the big machine. Today, you will probably want to try running on both the CPU cores and on the GPUs. Is this a lot of work? Sure, but if it saves a few CPU centuries, it is worth the effort. In your case, you might want to "fool around" with various model parameters that are combinations of your N and memory per CPU. Oh, and I often have algorithm parameters which correspond to L1, L2 and L3 data cache sizes. A typical result for a "simple" matrix multiplication (A*B=C) might have A fitted to L1, and B to L2. If you are doing something expressed in linear algebra, check out the ATLAS version of the BLAS library: http://math-atlas.sourceforge.net/ The big advantage of using ATLAS is that it will give good results for ALL > n^2 functions in terms of matrix multiplication. So even if you use some other BLAS, you can use the ATLAS libraries for some LAPACK calls. (There are several Ada bindings to BLAS floating around. I'm not choosing one, since your choices of OS and compiler will affect your choice.) Too much information? Probably. But if you do have a program that requires CPU years to run, or one that can be simplified by using LAPACK or BLAS? Have at it. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-01-24 15:18 ` Robert Eachus @ 2018-02-10 5:27 ` reinert 2018-02-10 23:31 ` Robert Eachus 2018-02-11 10:58 ` darkestkhan 0 siblings, 2 replies; 18+ messages in thread From: reinert @ 2018-02-10 5:27 UTC (permalink / raw) Hello, I have been away for a while. I find no memory leak in my program (it is easy to check memory use under linux - for example via "top"). In general I would like to have an Ada function which tells how much memory (in general hardware resources) is available. Maybe there is a "system function" I have overlooked? reinert On Wednesday, January 24, 2018 at 4:18:25 PM UTC+1, Robert Eachus wrote: > On Saturday, January 20, 2018 at 1:16:00 AM UTC-5, reinert wrote: > > > > Any hint for how I can optimize a bit smarter? > > > > First, realize that you are not the only one with this problem. There are programs that run on supercomputers for megayears of CPU time. It may take weeks (often on smaller systems) to figure out the "right" parameters for a given run. Why does it take so long? The usual approach is to create a linear regression model usually with linear and squared values for each model parameter, and sometimes cross-products. Now take your regression model and choose enough test points to get a decent result. Usually this is on the order of three or four data points for each model parameter. For example, your model might be t/p = 1/m + 1/m^2 + 1/N+ 1/N^2 + s + s^2 + s^3 + d/s + d/(s^2) where > t is time in seconds per iteration, p is the number of processors, m is memory size per CPU core in Gigabytes, N is an internal model sizing parameter, s is problem size in data points, and d is total (free) disk space in Gigabytes. > > Now you pick say 30 or so points, including some where you expect the model to crash or run impossibly slow. Do the runs, with probably a 1000 second limit per iteration per run. Now eliminate any time outs or crashes (you are not going to do big runs in that parameter space) and find the parameter values. From experience you are going to repeat the experiment on the big machine, with test parameters close to what you expect on a full run, but again with one to a few time steps. > > Now you know enough to ask for time (and number of CPU cores) on the big machine. Today, you will probably want to try running on both the CPU cores and on the GPUs. > > Is this a lot of work? Sure, but if it saves a few CPU centuries, it is worth the effort. > > In your case, you might want to "fool around" with various model parameters that are combinations of your N and memory per CPU. Oh, and I often have algorithm parameters which correspond to L1, L2 and L3 data cache sizes. A typical result for a "simple" matrix multiplication (A*B=C) might have A fitted to L1, and B to L2. If you are doing something expressed in linear algebra, check out the ATLAS version of the BLAS library: http://math-atlas.sourceforge.net/ The big advantage of using ATLAS is that it will give good results for ALL > n^2 functions in terms of matrix multiplication. So even if you use some other BLAS, you can use the ATLAS libraries for some LAPACK calls. (There are several Ada bindings to BLAS floating around. I'm not choosing one, since your choices of OS and compiler will affect your choice.) > > Too much information? Probably. But if you do have a program that requires CPU years to run, or one that can be simplified by using LAPACK or BLAS? Have at it. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-02-10 5:27 ` reinert @ 2018-02-10 23:31 ` Robert Eachus 2018-02-11 10:58 ` darkestkhan 1 sibling, 0 replies; 18+ messages in thread From: Robert Eachus @ 2018-02-10 23:31 UTC (permalink / raw) On Saturday, February 10, 2018 at 12:27:33 AM UTC-5, reinert wrote: > In general I would like to have an Ada function which tells how much memory (in general hardware resources) is available. Maybe there is a "system function" I have overlooked? If System.Memory_Size will help you, great. Look in your compiler's reference manual to see what sort of a value you will get. (The Ada RM says implementation defined and doesn't give hints to the implementer.) I haven't found it of much use, since it often defines the limit of addressable (virtual) memory. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-02-10 5:27 ` reinert 2018-02-10 23:31 ` Robert Eachus @ 2018-02-11 10:58 ` darkestkhan 2018-02-11 11:38 ` Dmitry A. Kazakov 1 sibling, 1 reply; 18+ messages in thread From: darkestkhan @ 2018-02-11 10:58 UTC (permalink / raw) On Saturday, February 10, 2018 at 5:27:33 AM UTC, reinert wrote: > Hello, I have been away for a while. > > I find no memory leak in my program (it is easy to check memory use under linux - for example via "top"). > > In general I would like to have an Ada function which tells how much memory (in general hardware resources) is available. Maybe there is a "system function" I have overlooked? > > reinert > Use valgrind for checking memory leaks - top would show you only big ones (I know that few bytes may not sound like much, but IT IS A LOT - for example windows 10 audio driver seems to leak memory [otherwise you can't explain why it uses over 500MB of memory after just 2-3 weeks of running] - how? small leak over prolonged period of time becomes big one). ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-02-11 10:58 ` darkestkhan @ 2018-02-11 11:38 ` Dmitry A. Kazakov 2018-02-16 4:59 ` reinert 2018-02-16 5:04 ` reinert 0 siblings, 2 replies; 18+ messages in thread From: Dmitry A. Kazakov @ 2018-02-11 11:38 UTC (permalink / raw) On 2018-02-11 11:58, darkestkhan wrote: > Use valgrind for checking memory leaks - top would show you only big ones (I know that few bytes may not sound like much, but IT IS A LOT - for example windows 10 audio driver seems to leak memory [otherwise you can't explain why it uses over 500MB of memory after just 2-3 weeks of running] - how? small leak over prolonged period of time becomes big one). For memory leaks I would recommend gnatmem. It is extremely easy to use. No code change required. It groups similar memory allocations together. It also finds leaks related to C code, which is the main source of leaks when using bindings to low-level C libraries. -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-02-11 11:38 ` Dmitry A. Kazakov @ 2018-02-16 4:59 ` reinert 2018-02-16 5:04 ` reinert 1 sibling, 0 replies; 18+ messages in thread From: reinert @ 2018-02-16 4:59 UTC (permalink / raw) Thanks for the help. I have startet to test out. reinert ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-02-11 11:38 ` Dmitry A. Kazakov 2018-02-16 4:59 ` reinert @ 2018-02-16 5:04 ` reinert 2018-02-16 8:18 ` Björn Lundin 1 sibling, 1 reply; 18+ messages in thread From: reinert @ 2018-02-16 5:04 UTC (permalink / raw) Thanks all for the help. I have started to test out. A point here (for me) seems to be that some computers are much slower on disk operations than others. My program is mush slower on my laptop than on my desktop. So optimization for a laptop seems to be somehow different as compared to a desktop. reinert ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-02-16 5:04 ` reinert @ 2018-02-16 8:18 ` Björn Lundin 2018-02-17 6:15 ` reinert 0 siblings, 1 reply; 18+ messages in thread From: Björn Lundin @ 2018-02-16 8:18 UTC (permalink / raw) On 2018-02-16 06:04, reinert wrote: >My program is mush slower on my laptop than on my desktop. >So optimization for a laptop seems to be somehow different as compared to a desktop. Or that is just because laptops usually have slower disks. 10-15 years ago or so an ordinary laptop would have an 4200 rpm IDE disk, while a desktop would spin at 5400 rpm or even 7200 rpm. And servers would have scsi disks at 10000 or 15000 rpms. But now - a large amount of laptops have SSDs instead. And some are faster some are slower. So dividing into laptop/desktop for general disk access optimisation seems strange to me. If you run your tests on an external drive and compare the output from both laptops and desktops - is there a difference? -- -- Björn ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-02-16 8:18 ` Björn Lundin @ 2018-02-17 6:15 ` reinert 2018-02-17 18:36 ` Dennis Lee Bieber 0 siblings, 1 reply; 18+ messages in thread From: reinert @ 2018-02-17 6:15 UTC (permalink / raw) Obs, you mean to try to use an external disk for swap space? (I use linux, Debian). See if I can spend some days on hacking :-) reinert On Friday, February 16, 2018 at 9:18:29 AM UTC+1, björn lundin wrote: > On 2018-02-16 06:04, reinert wrote: > > >My program is mush slower on my laptop than on my desktop. > >So optimization for a laptop seems to be somehow different as compared to a desktop. > > Or that is just because laptops usually have slower disks. > 10-15 years ago or so an ordinary laptop would have an 4200 rpm IDE > disk, while a desktop would spin at 5400 rpm or even 7200 rpm. > And servers would have scsi disks at 10000 or 15000 rpms. > > But now - a large amount of laptops have SSDs instead. And some are > faster some are slower. > > So dividing into laptop/desktop for general disk access optimisation > seems strange to me. > > If you run your tests on an external drive and compare the output > from both laptops and desktops - is there a difference? > > > > -- > -- > Björn ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-02-17 6:15 ` reinert @ 2018-02-17 18:36 ` Dennis Lee Bieber 2018-02-18 15:21 ` Björn Lundin 0 siblings, 1 reply; 18+ messages in thread From: Dennis Lee Bieber @ 2018-02-17 18:36 UTC (permalink / raw) On Fri, 16 Feb 2018 22:15:36 -0800 (PST), reinert <reinkor@gmail.com> declaimed the following: >Obs, you mean to try to use an external disk for swap space? (I use linux, Debian). See if I can spend some days on hacking :-) > I suspect the suggestion is to put the entire test application (program, data, swap) on the external drive. And to use the same drive on both computers (allowing for possibly needing to rebuild the application if the architecture is different). That would isolate disk I/O to the single drive, and any performance differences should then be a result of difference in processor and USB (unless you've got an old FireWire drive <G>) support. -- Wulfraed Dennis Lee Bieber AF6VN wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/ ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: How to optimize use of RAM/disk access ? 2018-02-17 18:36 ` Dennis Lee Bieber @ 2018-02-18 15:21 ` Björn Lundin 0 siblings, 0 replies; 18+ messages in thread From: Björn Lundin @ 2018-02-18 15:21 UTC (permalink / raw) On 2018-02-17 19:36, Dennis Lee Bieber wrote: > I suspect the suggestion is to put the entire test application > (program, data, swap) on the external drive. And to use the same drive on > both computers (allowing for possibly needing to rebuild the application if > the architecture is different). > > That would isolate disk I/O to the single drive, and any performance > differences should then be a result of difference in processor and USB > (unless you've got an old FireWire drive <G>) support. > Yes, this is what I meant -- -- Björn ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2018-02-18 15:21 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-01-20 6:15 How to optimize use of RAM/disk access ? reinert 2018-01-20 8:45 ` Jacob Sparre Andersen 2018-01-21 7:26 ` reinert 2018-01-21 23:49 ` Shark8 2018-01-22 5:53 ` darkestkhan 2018-01-22 8:53 ` gautier_niouzes 2018-01-22 15:01 ` Mr. Man-wai Chang 2018-01-24 15:18 ` Robert Eachus 2018-02-10 5:27 ` reinert 2018-02-10 23:31 ` Robert Eachus 2018-02-11 10:58 ` darkestkhan 2018-02-11 11:38 ` Dmitry A. Kazakov 2018-02-16 4:59 ` reinert 2018-02-16 5:04 ` reinert 2018-02-16 8:18 ` Björn Lundin 2018-02-17 6:15 ` reinert 2018-02-17 18:36 ` Dennis Lee Bieber 2018-02-18 15:21 ` Björn Lundin
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox