GNAT corrupted builds with below second changes

GNAT corrupted builds with below second changes

[Alejandro R. Mosteo]

Hi,

I'm doing some automated testing that involves very fast file generation 
and recompilation. Platform is linux, GPL 2017 or FSF 7.2. I've observed 
that gprbuild seems to not detect that a file is changed if less than a 
second has elapsed. I will guess here that the timestamp being used has 
second precision. This leads to supposedly up-to-date builds that 
aren't, among other funny effects.

I faintly remember having read about this somewhere, but I'm unable to 
find anything related in web searches. Anyone has experienced this?

Thanks,
Álex.

Re: GNAT corrupted builds with below second changes

[Randy Brukardt]

"Alejandro R. Mosteo" <alejandro@mosteo.com> wrote in message 
news:p8atvh$74r$1@dont-email.me...
> Hi,
>
> I'm doing some automated testing that involves very fast file generation 
> and recompilation. Platform is linux, GPL 2017 or FSF 7.2. I've observed 
> that gprbuild seems to not detect that a file is changed if less than a 
> second has elapsed. I will guess here that the timestamp being used has 
> second precision. This leads to supposedly up-to-date builds that aren't, 
> among other funny effects.
>
> I faintly remember having read about this somewhere, but I'm unable to 
> find anything related in web searches. Anyone has experienced this?

Not with GNAT, but we used to have such problems with Janus/Ada programs. I 
believe it caused certain ACATS tests to intermittently fail. I think I 
"fixed" that by introducing a delay into the compiler in certain 
circumstances (the delay being barely noticable to humans, but long enough 
to ensure that the time stamps are unique). [I don't remember for sure 
whether I fixed the problem, but since I don't remember it happening in the 
last 5 years, it seems likely that I did something to prevent it.] A better 
fix would have been to stop using time for such checks (compiles having 
gotten fast enough to make identical timestamps probable, and more so with 
every new generation of computers), but that would have required a lot of 
compiler and library change. (Definitely not the quick fix I was looking for 
at the time.)

It wouldn't surprise me if GNAT ran into similar issues -- the file 
timestamps on Windows (which is what we were/are using) have a 2 second 
granularity.

                                       Randy.

Re: GNAT corrupted builds with below second changes

[Alejandro R. Mosteo]

On 14/03/18 23:56, Randy Brukardt wrote:
> "Alejandro R. Mosteo" <alejandro@mosteo.com> wrote in message
> news:p8atvh$74r$1@dont-email.me...
>> Hi,
>>
>> I'm doing some automated testing that involves very fast file generation
>> and recompilation. Platform is linux, GPL 2017 or FSF 7.2. I've observed
>> that gprbuild seems to not detect that a file is changed if less than a
>> second has elapsed. I will guess here that the timestamp being used has
>> second precision. This leads to supposedly up-to-date builds that aren't,
>> among other funny effects.
>>
>> I faintly remember having read about this somewhere, but I'm unable to
>> find anything related in web searches. Anyone has experienced this?
> 
> Not with GNAT, but we used to have such problems with Janus/Ada programs. I
> believe it caused certain ACATS tests to intermittently fail. I think I
> "fixed" that by introducing a delay into the compiler in certain
> circumstances (the delay being barely noticable to humans, but long enough
> to ensure that the time stamps are unique).

I did this initially but it mounted up, so I'm now manually deleting 
.o/.ali files for a few critical files only.

> [I don't remember for sure
> whether I fixed the problem, but since I don't remember it happening in the
> last 5 years, it seems likely that I did something to prevent it.] A better
> fix would have been to stop using time for such checks (compiles having
> gotten fast enough to make identical timestamps probable, and more so with
> every new generation of computers), but that would have required a lot of
> compiler and library change. (Definitely not the quick fix I was looking for
> at the time.)
> 
> It wouldn't surprise me if GNAT ran into similar issues -- the file
> timestamps on Windows (which is what we were/are using) have a 2 second
> granularity.
> 
>                                         Randy.

The funny thing is I'm almost sure of having read somewhere that this 
had been addressed in gnat specifically, but I can't locate it.

Thanks,
Álex.

Re: GNAT corrupted builds with below second changes

[Shark8]

On Wednesday, March 14, 2018 at 4:36:34 AM UTC-6, Alejandro R. Mosteo wrote:
> 
> I will guess here that the timestamp being used has 
> second precision. This leads to supposedly up-to-date builds that 
> aren't, among other funny effects.

This is one reason I advocate a DB-based solution rather than an FS-based solution; after all, the FS solution *IS* simply using the FS as an ad-hoc and anemic database, and using the timestamp-field this way is essentially analogous to querying a 'time' column when what we *REALLY* want is a 'version' column (not "1.2.3" but more like "build # X").

A lot of problems with build-systems come from (a) using the FS as an ad-hoc DB, and (b) the differences in the FS like "/ vs \ vs :" directory separators, or case-insensitive vs case-sensitive.

Re: GNAT corrupted builds with below second changes

[Bojan Bozovic]

On Thursday, March 15, 2018 at 7:52:20 PM UTC+1, Shark8 wrote:
> On Wednesday, March 14, 2018 at 4:36:34 AM UTC-6, Alejandro R. Mosteo wrote:
> > 
> > I will guess here that the timestamp being used has 
> > second precision. This leads to supposedly up-to-date builds that 
> > aren't, among other funny effects.
> 
> This is one reason I advocate a DB-based solution rather than an FS-based solution; after all, the FS solution *IS* simply using the FS as an ad-hoc and anemic database, and using the timestamp-field this way is essentially analogous to querying a 'time' column when what we *REALLY* want is a 'version' column (not "1.2.3" but more like "build # X").
> 
> A lot of problems with build-systems come from (a) using the FS as an ad-hoc DB, and (b) the differences in the FS like "/ vs \ vs :" directory separators, or case-insensitive vs case-sensitive.

Versioning file system is then what's needed, and that has been in VMS/VAX-VMS/OpenVMS since seventies (and since seventies, VAX had 128 bit integers as well). There's no good reason mainstream OS don't have it (like Windows, Linux and OS X), it's absent only due to inertia. In linux there's NILFS but it's not used in mainstream distributions. DB is partial solution, as files belong to the file system, not the database.

Re: GNAT corrupted builds with below second changes

[Jeffrey R. Carter]

On 03/15/2018 07:52 PM, Shark8 wrote:
> 
> This is one reason I advocate a DB-based solution rather than an FS-based solution; after all, the FS solution *IS* simply using the FS as an ad-hoc and anemic database, and using the timestamp-field this way is essentially analogous to querying a 'time' column when what we *REALLY* want is a 'version' column (not "1.2.3" but more like "build # X").

This is somewhat similar to the "program library" required by Ada 83 (ARM-83 10.4).

http://archive.adaic.com/standards/83lrm/html/lrm-10-04.html#10.4

-- 
Jeff Carter
"Beyond 100,000 lines of code you
should probably be coding in Ada."
P. J. Plauger
26

Re: GNAT corrupted builds with below second changes

[Shark8]

On Thursday, March 15, 2018 at 3:03:24 PM UTC-6, Jeffrey R. Carter wrote:
> 
> This is somewhat similar to the "program library" required by Ada 83 (ARM-83 10.4).
> 
> http://archive.adaic.com/standards/83lrm/html/lrm-10-04.html#10.4

Yep; there were/are a lot of things in the 83 Standard that were really done quite well.

Re: GNAT corrupted builds with below second changes

[Shark8]

On Thursday, March 15, 2018 at 1:59:21 PM UTC-6, Bojan Bozovic wrote:
> 
> Versioning file system is then what's needed, and that has been in VMS/VAX-VMS/OpenVMS since seventies (and since seventies, VAX had 128 bit integers as well).

I agree that a versioning FS is needed; but this only addresses the specific issue raised here (the timestamp-resolution) and even a versioned FS is inadequate to the general problem.

Take, for example, moving a file or directory and the impact that has on a makefile/build-system: it invalidates the entire object in question and destroys the ability to build. -- As you can see, using the FS as an anemic & ad-hoc DB is simply not addressing the entire problem.

> There's no good reason mainstream OS don't have it (like Windows, Linux and OS X), it's absent only due to inertia. In linux there's NILFS but it's not used in mainstream distributions. DB is partial solution, as files belong to the file system, not the database.

*sigh* -- Programs needn't BE files.

We could [and should] store them as parsed, processed semantically-valid structures. (Yes, there *should* be an interface for import/export to files/text; this operation could be done by the Types for the structure themselves as serialize/deserialize operations.) -- While it's possible these structures could be stored in a mere file, that's not handling things very intelligently: if the above structure is suitable to being stored in a database then [with a little cleverness] we inherit an entire entire system for dealing with ensuring that data is valid.

Yes, there's the ultra-trivial example of renaming -- pseudo-SQL: UPDATE 'm' = 'MINIMUM' WHERE SCOPE = 'My_Function' -- which now won't have the text-search and replace probleMINIMUM where there's extraneous replaceMINIMUMents in coMINIMUMMINIMUMents and the like.

But that's a ridiculously simple/trivial example; what about something like having the ability to make a subprogram generic with a simple command -- perhaps pseudo-SQL: INSERT Generic_Foo INTO 'Functions' WHERE SELECT x FROM 'Functions' WHERE x.Name = 'Some_Fn' REPLACING x.param['example'].type WITH GENERIC_TYPE

Or something like that. (Of course these sorts of manipulations on the source/DB should NOT be visible to the user, but are merely here to disabuse you of the notion that all editing/manipulation must be done as text. [A bit ironic considering the text-based medium being used here.])

Aside from all this, the usage of DBs can be a huge improvement to FS-based systems precisely because of properties of the DB. Take for example this alternative to "Continuous Integration" from the mid 80s -- http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.26.2533&rep=rep1&type=pdf -- by partitioning the project into workspaces and using a hierarchical database, only merging "upward" when everything on the "current level" is consistant [compilable], you end up with a DB where the root node *ALWAYS* has "the last compilable version". (Combine that with versioning and you have a record of EVERY COMPILING VERSION; *BAM!* no messed up/non-compiling crap in your repository.)

Re: GNAT corrupted builds with below second changes

[Bojan Bozovic]

On Friday, March 16, 2018 at 4:27:46 AM UTC+1, Shark8 wrote:
> On Thursday, March 15, 2018 at 1:59:21 PM UTC-6, Bojan Bozovic wrote:
> > 
> > Versioning file system is then what's needed, and that has been in VMS/VAX-VMS/OpenVMS since seventies (and since seventies, VAX had 128 bit integers as well).
> 
> I agree that a versioning FS is needed; but this only addresses the specific issue raised here (the timestamp-resolution) and even a versioned FS is inadequate to the general problem.
> 
> Take, for example, moving a file or directory and the impact that has on a makefile/build-system: it invalidates the entire object in question and destroys the ability to build. -- As you can see, using the FS as an anemic & ad-hoc DB is simply not addressing the entire problem.
> 
> > There's no good reason mainstream OS don't have it (like Windows, Linux and OS X), it's absent only due to inertia. In linux there's NILFS but it's not used in mainstream distributions. DB is partial solution, as files belong to the file system, not the database.
> 
> *sigh* -- Programs needn't BE files.
> 
> We could [and should] store them as parsed, processed semantically-valid structures. (Yes, there *should* be an interface for import/export to files/text; this operation could be done by the Types for the structure themselves as serialize/deserialize operations.) -- While it's possible these structures could be stored in a mere file, that's not handling things very intelligently: if the above structure is suitable to being stored in a database then [with a little cleverness] we inherit an entire entire system for dealing with ensuring that data is valid.
> 
> Yes, there's the ultra-trivial example of renaming -- pseudo-SQL: UPDATE 'm' = 'MINIMUM' WHERE SCOPE = 'My_Function' -- which now won't have the text-search and replace probleMINIMUM where there's extraneous replaceMINIMUMents in coMINIMUMMINIMUMents and the like.
> 
> But that's a ridiculously simple/trivial example; what about something like having the ability to make a subprogram generic with a simple command -- perhaps pseudo-SQL: INSERT Generic_Foo INTO 'Functions' WHERE SELECT x FROM 'Functions' WHERE x.Name = 'Some_Fn' REPLACING x.param['example'].type WITH GENERIC_TYPE
> 
> Or something like that. (Of course these sorts of manipulations on the source/DB should NOT be visible to the user, but are merely here to disabuse you of the notion that all editing/manipulation must be done as text. [A bit ironic considering the text-based medium being used here.])
> 
> Aside from all this, the usage of DBs can be a huge improvement to FS-based systems precisely because of properties of the DB. Take for example this alternative to "Continuous Integration" from the mid 80s -- http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.26.2533&rep=rep1&type=pdf -- by partitioning the project into workspaces and using a hierarchical database, only merging "upward" when everything on the "current level" is consistant [compilable], you end up with a DB where the root node *ALWAYS* has "the last compilable version". (Combine that with versioning and you have a record of EVERY COMPILING VERSION; *BAM!* no messed up/non-compiling crap in your repository.)

Of course you could implement any DB functionality in the file system, I don't  deny, but it in fact should be system wide. Problem is the concept isn't  marketable to non-technical users. I myself would use compiler which stores source in a DB, provided that copies on the FS exist, and I can pick the source that is to be used (DB or FS), and that it can be turned off for source on removable media. In this era we have terabytes of disk space available anyway, duplication of space used isn't an issue.

Re: GNAT corrupted builds with below second changes

[J-P. Rosen]

Le 16/03/2018 à 04:27, Shark8 a écrit :
> We could [and should] store them as parsed, processed semantically-valid structures.

This would require a machine/OS entirely devoted to software development
- the Rational R-1000. Unfortunately, the concept was not commercially
succesful, although users were really happy with it.

-- 
J-P. Rosen
Adalog
2 rue du Docteur Lombard, 92441 Issy-les-Moulineaux CEDEX
Tel: +33 1 45 29 21 52, Fax: +33 1 45 29 25 00
http://www.adalog.fr

Re: GNAT corrupted builds with below second changes

[Dmitry A. Kazakov]

On 16/03/2018 04:27, Shark8 wrote:
> On Thursday, March 15, 2018 at 1:59:21 PM UTC-6, Bojan Bozovic wrote:
>>
>> Versioning file system is then what's needed, and that has been in VMS/VAX-VMS/OpenVMS since seventies (and since seventies, VAX had 128 bit integers as well).
> 
> I agree that a versioning FS is needed; but this only addresses the specific issue raised here (the timestamp-resolution) and even a versioned FS is inadequate to the general problem.

Yes, versioning files /= versioning projects

> Take, for example, moving a file or directory and the impact that has on a makefile/build-system: it invalidates the entire object in question and destroys the ability to build. -- As you can see, using the FS as an anemic & ad-hoc DB is simply not addressing the entire problem.

Moving directory on the project level is checking out everything for 
delete, then committing delete, then deleting items from the project, or 
all project.

> We could [and should] store them as parsed, processed semantically-valid structures. (Yes, there *should* be an interface for import/export to files/text; this operation could be done by the Types for the structure themselves as serialize/deserialize operations.) -- While it's possible these structures could be stored in a mere file, that's not handling things very intelligently: if the above structure is suitable to being stored in a database then [with a little cleverness] we inherit an entire entire system for dealing with ensuring that data is valid.

Yes, an OO persistent storage.

BTW, RDBMS is notoriously poor on that. Relational algebra is no help 
here, at all.

> Or something like that. (Of course these sorts of manipulations on the source/DB should NOT be visible to the user, but are merely here to disabuse you of the notion that all editing/manipulation must be done as text.

No, that would be a text buffer with tags for colorization and hyperlink 
references.

> Aside from all this, the usage of DBs can be a huge improvement to FS-based systems precisely because of properties of the DB. Take for example this alternative to "Continuous Integration" from the mid 80s -- http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.26.2533&rep=rep1&type=pdf -- by partitioning the project into workspaces and using a hierarchical database, only merging "upward" when everything on the "current level" is consistant [compilable], you end up with a DB where the root node *ALWAYS* has "the last compilable version". (Combine that with versioning and you have a record of EVERY COMPILING VERSION; *BAM!* no messed up/non-compiling crap in your repository.)

Instead of partitioning projects decent software engineering uses the 
project (library) as a partition. The larger projects consist of/depend 
on projects versioned in a way that incompatible versions cannot be 
mixed. Any IDE, any OS distribution follows this model, only file 
versioning systems do not (Rational ClearCase does).

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: GNAT corrupted builds with below second changes

[Robert I. Eachus]

On 3/15/2018 3:59 PM, Bojan Bozovic wrote:

> Versioning file system is then what's needed, and that has been in VMS/VAX-VMS/OpenVMS since seventies (and since seventies, VAX had 128 bit integers as well). There's no good reason mainstream OS don't have it (like Windows, Linux and OS X), it's absent only due to inertia. In linux there's NILFS but it's not used in mainstream distributions. DB is partial solution, as files belong to the file system, not the database.
>
Multics had it before that, it was just so invisible (unless you needed 
it) that many users never knew it was there.  Actually, since this was 
Multics, I should talk about segments.  The dumper daemon, ran around 
behind the scene and copied modified segments, usually to tape, but 
could be to hard disk.  How did it know if a file was modified? It 
looked to see if the last changed date was before this pass started. 
What if you made multiple changes in a second?  First, the timestamps 
were guaranteed unique, the timestamp might normally be printed to the 
second, I think the timestamps went to the microsecond.  Even if the 
system clock was slower, and it usually was, any call to the clock got 
the clock plus the number of clock calls since the last clock tic. 
(Shriek names, that began with an exclamation point were guaranteed to 
be system unique and were created by hashing the value from a clock call.)

The policy on dumping segments allowed for segments growing, so every 
keystroke didn't result in a dumped file.  Also tools like compilers 
used !names to insure they wouldn't be dumped, except for those 
resulting segments that were renamed when finished. But for example 
deleting the contents of a segment gave you a new blank segment, and 
Multics held onto the old version, if necessary, until it was dumped.

I only used the capability once. I was testing changes to a terminal 
driver written in Multics Lisp, and wrote all over the source segment 
while testing it.  No problem.  The file system offered me 25 versions, 
and offered to look for more. ;-)

The only real problem was that all system times were in UTS with no 
daylight savings, but printing files would use the local time offset. 
So around the start and end of daylight savings time it could confuse 
users. (The timestamps didn't change, just how they were printed.)

Anyway, I don't care how they do it anything that depends on timestamps 
should make them unique.  Like Multics, the milliseconds or microseconds 
don't need to be accurate, just correctly ordered and unique.

Re: GNAT corrupted builds with below second changes

[Robert I. Eachus]

On 3/16/2018 1:37 AM, J-P. Rosen wrote:
> Le 16/03/2018 à 04:27, Shark8 a écrit :
>> We could [and should] store them as parsed, processed semantically-valid structures.
> 
> This would require a machine/OS entirely devoted to software development
> - the Rational R-1000. Unfortunately, the concept was not commercially
> succesful, although users were really happy with it.
> 
Don't forget Multics. It did have other uses, but if you had a Multics 
system, all software development seemed to migrate there.  Honeywell in 
Billerica had a Multics system which was used to develop much of the 
software for Honeywell's line of small systems: Level 6, DPS 6, etc. 
Same at large systems in Phoenix.

The reason this is important is that most of the people involved in 
developing Green, later Ada, including Jean Ichbiah, were Multics users. 
  So a lot of Multics concepts became part of Ada.

Re: GNAT corrupted builds with below second changes

[David Thompson]

On Wed, 14 Mar 2018 17:56:13 -0500, "Randy Brukardt"
<randy@rrsoftware.com> wrote:

> "Alejandro R. Mosteo" <alejandro@mosteo.com> wrote in message 
> news:p8atvh$74r$1@dont-email.me...
<snip: gprbuild confused if modtimes within 1 second>

> ... we used to have such problems with Janus/Ada programs. I 
> believe it caused certain ACATS tests to intermittently fail. I think I 
> "fixed" that by introducing a delay into the compiler in certain 
> circumstances (the delay being barely noticable to humans, but long enough 
> to ensure that the time stamps are unique). ...
> It wouldn't surprise me if GNAT ran into similar issues -- the file 
> timestamps on Windows (which is what we were/are using) have a 2 second 
> granularity.

_FAT_, which was the file system used on MSDOS and (thus) early
Windows, has 2 second granularity. Windows NT introduced NTFS, with
granularity 100ns (and epoch in 1600!), but you can still have FAT for
data disks -- and for removable or detachable devices may prefer to,
as NTFS permissions get screwed up if you move a disk between machines
which are not both in the same 'domain' (in the MS sense, not DNS).

It is possible to get FAT (and NTFS) filesystem drivers on at least
some Unices, if you want to, and cause the problem there.

Re: GNAT corrupted builds with below second changes

[Simon Wright]

"Alejandro R. Mosteo" <alejandro@mosteo.com> writes:

> I've observed that gprbuild seems to not detect that a file is changed
> if less than a second has elapsed. I will guess here that the
> timestamp being used has second precision.

Looking at a .ali file, the dependencies are recorded in lines like

D ada.ads               20170106103348 76789da1 ada%s

where 20170106103348 is the date/time to 1 second precision, and
76789da1 is a checksum of the file's contents. I'd imagine that gprbuild
looks at the timestamp first; not sure about the checksum, but certainly
needed for -m processing (only recompile if semantically modified).

Re: GNAT corrupted builds with below second changes

[J-P. Rosen]

Le 22/04/2018 à 09:31, Simon Wright a écrit :
> "Alejandro R. Mosteo" <alejandro@mosteo.com> writes:
> 
>> I've observed that gprbuild seems to not detect that a file is changed
>> if less than a second has elapsed. I will guess here that the
>> timestamp being used has second precision.
> 
> Looking at a .ali file, the dependencies are recorded in lines like
> 
> D ada.ads               20170106103348 76789da1 ada%s
> 
> where 20170106103348 is the date/time to 1 second precision, and
> 76789da1 is a checksum of the file's contents. I'd imagine that gprbuild
> looks at the timestamp first; not sure about the checksum, but certainly
> needed for -m processing (only recompile if semantically modified).
> 
Yes, it's a checksum that ignores casing, comments, blank lines,
indentation and the like.

-- 
J-P. Rosen
Adalog
2 rue du Docteur Lombard, 92441 Issy-les-Moulineaux CEDEX
Tel: +33 1 45 29 21 52, Fax: +33 1 45 29 25 00
http://www.adalog.fr