Config_Files proposal

Config_Files proposal

[Stephen Leake]

I've posted another example spec and implementation, at 

http://users.erols.com/leakstan/Stephe/Ada/Config_Files/config_files.html

This second implementation uses the Java property style file format.

I also changed the spec some. In particular, I got rid of the
"Section" parameters; keys now just use dotted notation.

I'd like to have a semi-formal vote on which file format to go with.
Then I'll move this into Grace, and we can finish up all the details.

To vote, either post here with your preference, or send me email.
Voting closes in a week (June 25), but I'm not going to be picky about
time of day :).

I've listed some advantages and disadvantages of each proposed file
format on the web page. Feel free to tell me about advantages or
disadvantages I've missed.

Based on that list, I vote for the Java property style format. I also
think most people here agree it's a good format.

Note that we are only voting on the general file format. The remaining
details of the file format (like how to represent comments) and spec
will be worked out later.

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> Note that we are only voting on the general file format. The remaining
> details of the file format (like how to represent comments) and spec
> will be worked out later.

I think this is kind of the wrong way around. Asking which of two completely
different file formats is better, and then trying to build an API based on
that, seems backwards.

That said, there's some things missing from the spec:

1) When you read three config files off the path, change a value that was in
the second one, and write it out again, where does it get written?

2) Can you have a "local" and a "global" config file, such that changes to
the local override the global, but it's possible to have only some of the
information in the local file? E.g., the system administrator says "the mail
server is a smtp-server.example.com, but each user fills in his own "from"
email address"?

3) If you enable #2, and then the user changes her email address, does it
also store back the mail server address, thereby preventing the
administrator from changing it?

4) If you have a list of items, such as a recent-file list, how are they
managed? Does it make sense to use [] notation or something?

5) Do comments go in the files? If so, how does that interact with the
previous questions?

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> Stephen Leake wrote:
> > Note that we are only voting on the general file format. The remaining
> > details of the file format (like how to represent comments) and spec
> > will be worked out later.
> 
> I think this is kind of the wrong way around. Asking which of two completely
> different file formats is better, and then trying to build an API based on
> that, seems backwards.

That isn't what I meant to say :). The API _is_ orthogonal to the file
format.

What I meant to say was that I'd like to decide now whether we will go
forward with Java properties style or XML (or possibly some other
format). Given that choice, there are still (minor) issues to decide
about the file format.

In addition to that decision, there are decisions about the package
spec to be made. Note that the file "grace-config_files_java.ads"
is a package spec that could easily have a body that uses an XML file
format. I should have used a different name :(.

> That said, there's some things missing from the spec:
> 
> 1) When you read three config files off the path, change a value that was in
> the second one, and write it out again, where does it get written?

I tried to answer that in requirement 10, but I can see it isn't very
clear.

The intent is that only one file be writable, and that is where all
values are written.

For example, if you open ~/.myconfig (writable), /usr/local/app/config
(readonly) and /etc/app/config (readonly), then any values you change
are written to ~/.myconfig.

> 2) Can you have a "local" and a "global" config file, such that
> changes to the local override the global, but it's possible to have
> only some of the information in the local file? E.g., the system
> administrator says "the mail server is a smtp-server.example.com,
> but each user fills in his own "from" email address"?

Yes, but any values the user changes will be written to the local
file. Same example as above.

The final doc for the Config_Files library will have examples of these
use cases.

> 3) If you enable #2, and then the user changes her email address,
> does it also store back the mail server address, thereby preventing
> the administrator from changing it?

Hmm. I was going to say "no, the user did not change that value". That
is my intent. But my draft implementation does not follow that intent;
it writes all the values to the writeable (local) file.

I'll declare that implementation broken, in this respect. I guess I
need a "value changed since read" marker, and only changed values get
written. That's not bad. Then I have to enhance the unit test to test
this. That will be harder :).

The requirements need to be clearer on this as well.

Thanks for pointing this out!

Hmm. Now I think we need the queries "what file was this value read from"
and "what file will this value be written to".

> 4) If you have a list of items, such as a recent-file list, how are
> they managed? 

Given the current requirements and spec, you need a unique key for
each scalar item; foo.bar.item.1, foo.bar.item.2, ... foo.bar.item.n.

> Does it make sense to use [] notation or something?

I'm leaving that for a future enhancement. The biggest problem I have
with it at the moment is that it requires some notion of a "list" of
values an arbitrary scalar type in the Config_Files spec. This could
probably be handled with a generic or a tagged type or an
unconstrained array. At the moment, we have no explicit lists in the
API, which keeps things simple (the search path is a list in the body,
but not in the visible part of the spec). Since the same effect can be
accomplished via numbered keys, I don't think it's worth pursuing at
this time.

Hmm, I'm wrong. There is a requirement to return "a list of keys". I
guess that will be a list of strings. When I implement that, I'll
think about this issue again :).

> 5) Do comments go in the files? If so, how does that interact with
> the previous questions?

The requirements say we must support comments. Neither of my
implementations do so. That's one of the "minor" details that remain
to be worked out.

Once again, thanks for your insightful comments.

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> That isn't what I meant to say :). The API _is_ orthogonal to the file
> format.

I don't think the API covers everything you'd want yet.
 
> > That said, there's some things missing from the spec:
> >
> > 1) When you read three config files off the path, change a value that was in
> > the second one, and write it out again, where does it get written?
> 
> I tried to answer that in requirement 10, but I can see it isn't very
> clear.
> 
> The intent is that only one file be writable, and that is where all
> values are written.
> 
> For example, if you open ~/.myconfig (writable), /usr/local/app/config
> (readonly) and /etc/app/config (readonly), then any values you change
> are written to ~/.myconfig.

What about values you don't change? What about values that you change back
to match what's in /usr/local/app/config? What about values you don't change
but which came from ~/.myconfig?

There's no obvious place I saw in the .ADS that shows multiple file names
being offered for reading, such that they'd be linked in any way.

> Yes, but any values the user changes will be written to the local
> file. Same example as above.

OK. That isn't clear from the docs. At least not to me.
 
> > 3) If you enable #2, and then the user changes her email address,
> > does it also store back the mail server address, thereby preventing
> > the administrator from changing it?
> 
> Hmm. I was going to say "no, the user did not change that value". That
> is my intent. But my draft implementation does not follow that intent;
> it writes all the values to the writeable (local) file.
> 
> I'll declare that implementation broken, in this respect. I guess I
> need a "value changed since read" marker, and only changed values get
> written.

Still not good enough. If I change the user email address this session,
write it out, then change the window position next time, I don't want to
lose the email address when I write it.

I think you need to keep track, for each value, what file it was read from
and whether it changed. Then when you write it out, write out any values
that have changed as well as any values that originally came from the file
you're rewriting.

> Hmm. Now I think we need the queries "what file was this value read from"
> and "what file will this value be written to".

Something close to that, yes. :)

Oh. One other question. How often are the config files read? If someone
changes a config file while I'm running, do I see the change reflected? What
if I run two copies of the app at once, one of which I use to configure a
bunch of stuff, then write it back out, and the other which I move a window
but don't change anything else. Does it reread the "changed" file, make the
updates, and write it back out? 

I think if you're not careful, there's going to be very unintuitive
behavior.

> but not in the visible part of the spec). Since the same effect can be
> accomplished via numbered keys, I don't think it's worth pursuing at
> this time.

Fair enough.
 
> Hmm, I'm wrong. There is a requirement to return "a list of keys". I
> guess that will be a list of strings. When I implement that, I'll
> think about this issue again :).

I'd think an unconstrained array of strings would be best there. Of course,
if it's part of Grace, you have th elists.
 
> Once again, thanks for your insightful comments.

Hey, it's fun. :-)

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> Stephen Leake wrote:
> > That isn't what I meant to say :). The API _is_ orthogonal to the file
> > format.
> 
> I don't think the API covers everything you'd want yet.

Ok. Do you think that prevents us from agreeing on which file format
to go forward with?

It's a pain maintaining two example implementations. The stuff we are
talking about here is totally independent of file format.

> What about values you don't change? 

As I said below, only values changed since being read from the config
file will be written; unchanged values are not written.

> What about values that you change back to match what's in
> /usr/local/app/config? 

Well, I guess we could provide an "undo" command. I guess "undo" can
simply delete the value from the writeable tree, which will expose the
value in the read-only tree (assuming there is one).

Perhaps also a total "erase", which would discard all changes.

> What about values you don't change but which came from ~/.myconfig?

Hmm. Good point. I guess all values from a writeable file must be
written, not just changed ones.

> There's no obvious place I saw in the .ADS that shows multiple file
> names being offered for reading, such that they'd be linked in any
> way.

Yes, that is not implemented.

> > Yes, but any values the user changes will be written to the local
> > file. Same example as above.
> 
> OK. That isn't clear from the docs. At least not to me.

Right, it needs to be added.

> > > 3) If you enable #2, and then the user changes her email
> > > address, does it also store back the mail server address,
> > > thereby preventing the administrator from changing it?
> > 
> > Hmm. I was going to say "no, the user did not change that value". That
> > is my intent. But my draft implementation does not follow that intent;
> > it writes all the values to the writeable (local) file.
> > 
> > I'll declare that implementation broken, in this respect. I guess I
> > need a "value changed since read" marker, and only changed values get
> > written.
> 
> Still not good enough. If I change the user email address this session,
> write it out, then change the window position next time, I don't want to
> lose the email address when I write it.
> 
> I think you need to keep track, for each value, what file it was read from
> and whether it changed. Then when you write it out, write out any values
> that have changed as well as any values that originally came from the file
> you're rewriting.

Yes.

> Oh. One other question. How often are the config files read? 

When you call "open".

> If someone changes a config file while I'm running, do I see the
> change reflected? 

We explicitly rule out multiple parallel processes accessing the same
files. So that's erroneous usage, and you may format your hard disk :).

> What if I run two copies of the app at once, one of which I use to
> configure a bunch of stuff, then write it back out, and the other
> which I move a window but don't change anything else. Does it reread
> the "changed" file, make the updates, and write it back out?

Don't do that.

Seriously, you've set up a race condition, which is an indication of
bad design. If you want multiple instances of your app running, each
with its own persistent window positions, you need multiple config
files. That is precisely one of my use cases, and why I argued for
specifying the config file on the app command line.

Or, if you really want one value shared among processes, use the
registry, since it is designed for that. 

> I think if you're not careful, there's going to be very unintuitive
> behavior.

Which is why we ruled out multiple processes from the beginning.

> > Hmm, I'm wrong. There is a requirement to return "a list of keys". I
> > guess that will be a list of strings. When I implement that, I'll
> > think about this issue again :).
> 
> I'd think an unconstrained array of strings would be best there. Of course,
> if it's part of Grace, you have th elists.

Ah. I guess I can rely on other Grace components. Good point.

> > Once again, thanks for your insightful comments.
> 
> Hey, it's fun. :-)

Less fun from this end :). But still Ok.

Let me ask again; can we choose the general file format now, or do you
think we need to work on the API more first?

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> Ok. Do you think that prevents us from agreeing on which file format
> to go forward with?

I'm not sure why you're implementing two bodies while the spec is still in
flux. :-)  Either file format you want to support is OK by me.
 
> > What about values you don't change?
> 
> As I said below, only values changed since being read from the config
> file will be written; unchanged values are not written.

Then you're screwed. If I load 20 values out of a config file, change one,
and save it, then I'll lose 19 config values?
 
> > What about values that you change back to match what's in
> > /usr/local/app/config?
> 
> Well, I guess we could provide an "undo" command. I guess "undo" can
> simply delete the value from the writeable tree, which will expose the
> value in the read-only tree (assuming there is one).

I don't think that's quite what I meant. I meant that default values
probably shouldn't be stored, even if you got them by assigning the default
to the key.

I think until you (we) work out the semantics of what's going on,
implementing multiple bodies is relatively unhelpful to progressing the
spec. Of course, if you're enjoying it and all, that's something else. :-)

> Seriously, you've set up a race condition, which is an indication of
> bad design. If you want multiple instances of your app running, each
> with its own persistent window positions, you need multiple config
> files. That is precisely one of my use cases, and why I argued for
> specifying the config file on the app command line.

That's not what I'm saying.

I'm saying I open my text editor for file XYZ.ads, and another for XYZ.adb,
and then I move XYZ.adb off to the side. Then, in XYZ.ads, I configure all
kinds of things to do the proper indenting for Ada, and then I close
XYZ.ads. Since I want XYZ.adb to see these changes, I close that, and reopen
it. Unfortunately, XYZ.adb didn't reload any changes I made in the indenting
configuration section, so all that work's gone.

Bad news.

Relatively easy to fix - when a config file is saved, it is first read off
the disk, then any values in the in-memory configuration that differ from
the just-loaded configuration and which have been changed since they were
loaded (or since the last save?) are copied to the just-loaded
configuration, then the just-loaded configuration is written back out.

> > I think if you're not careful, there's going to be very unintuitive
> > behavior.
> 
> Which is why we ruled out multiple processes from the beginning.

OK. I'd misinterpreted, thinking you meant only one process could read or
write at a time, rather than only one process could be using a configuration
at a time.
 
> Let me ask again; can we choose the general file format now, or do you
> think we need to work on the API more first?

I think you can pick the file format, sure. If you have "heirarchy of keys
mapped to string values", I don't see much more you need to know for that.

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> Stephen Leake wrote:
> > Ok. Do you think that prevents us from agreeing on which file format
> > to go forward with?
> 
> I'm not sure why you're implementing two bodies while the spec is still in
> flux. :-)  Either file format you want to support is OK by me.

Fair enough.

> > > What about values you don't change?
> > 
> > As I said below, only values changed since being read from the config
> > file will be written; unchanged values are not written.
> 
> Then you're screwed. If I load 20 values out of a config file, change one,
> and save it, then I'll lose 19 config values?

Yes, this is bad.

> > > What about values that you change back to match what's in
> > > /usr/local/app/config?
> > 
> > Well, I guess we could provide an "undo" command. I guess "undo" can
> > simply delete the value from the writeable tree, which will expose the
> > value in the read-only tree (assuming there is one).
> 
> I don't think that's quite what I meant. I meant that default values
> probably shouldn't be stored, even if you got them by assigning the default
> to the key.

Yes. The default is provided again with every call to "read", so it
does not need to be stored.

> I think until you (we) work out the semantics of what's going on,
> implementing multiple bodies is relatively unhelpful to progressing
> the spec. 

Well, I usually find problems with a spec when I get around to
implementing the body. I did these two implementations to see if there
were any problems with the spec that were related to the file format.
I don't think there are.

> Of course, if you're enjoying it and all, that's something else. :-)

I was trying to eliminate one of the variables in the problem. If we
agree on what the file format is, that's one less thing to think about.

I think we are ready to do that, and it will allow people to focus
solely on the spec, rather than on the XML vs ini etc.

> > Seriously, you've set up a race condition, which is an indication
> > of bad design. If you want multiple instances of your app running,
> > each with its own persistent window positions, you need multiple
> > config files. That is precisely one of my use cases, and why I
> > argued for specifying the config file on the app command line.
> 
> That's not what I'm saying.
> 
> I'm saying I open my text editor for file XYZ.ads, and another for XYZ.adb,
> and then I move XYZ.adb off to the side. Then, in XYZ.ads, I configure all
> kinds of things to do the proper indenting for Ada, and then I close
> XYZ.ads. Since I want XYZ.adb to see these changes, I close that, and reopen
> it. Unfortunately, XYZ.adb didn't reload any changes I made in the indenting
> configuration section, so all that work's gone.

Let's see. You have two instances of "text_edit", sharing one config
file. That is explicitly forbidden by requirement 1.

So if you want this behavior, you have to use the registry or gconf,
not Config_Files.

> Bad news.
> 
> Relatively easy to fix - when a config file is saved, it is first read off
> the disk, then any values in the in-memory configuration that differ from
> the just-loaded configuration and which have been changed since they were
> loaded (or since the last save?) are copied to the just-loaded
> configuration, then the just-loaded configuration is written back out.

This is a race condition. Suppose two processes happen to start a
"flush" operation "at the same time"? Whichever goes last wins.

That's why we have requirement 1. Config_Files don't do this.

If you feel that makes Config_Files totally useless, I guess we can
talk about removing requirement 1 and see if we can come up with
locking semantics to fix it. But I'd rather not, especially since the
registry and gconf have already done that.

In general, I suggest that there are other ways to fix an apparent
need for concurrent access to config files.

For the specific case of a text editor, you can use a text editor that
supports multiple documents; that provides automatic sharing of config
file settings between XYZ.adb and XYZ.ads.

> > > I think if you're not careful, there's going to be very
> > > unintuitive behavior.
> > 
> > Which is why we ruled out multiple processes from the beginning.
> 
> OK. I'd misinterpreted, thinking you meant only one process could read or
> write at a time, rather than only one process could be using a configuration
> at a time.

Ok, I guess I need to make that clearer. I thought one implied the
other :). Your example is more complex than I had in mind; I'll try to
include something like it in the final rationale document.

> > Let me ask again; can we choose the general file format now, or do
> > you think we need to work on the API more first?
> 
> I think you can pick the file format, sure. If you have "heirarchy of keys
> mapped to string values", I don't see much more you need to know for that.

But I want _you_ to pick a format, so we can get consensus. Hmm. You said
above either is ok; I guess that counts as consensus for the Java
property format.

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> Let's see. You have two instances of "text_edit", sharing one config
> file. That is explicitly forbidden by requirement 1.

Then requirement one should speak of concurrent use, rather than
simultaneous use. :-)

> This is a race condition. Suppose two processes happen to start a
> "flush" operation "at the same time"? Whichever goes last wins.

That would be simultaneous use. ;-)
 
> If you feel that makes Config_Files totally useless, I guess we can
> talk about removing requirement 1 and see if we can come up with
> locking semantics to fix it. But I'd rather not, especially since the
> registry and gconf have already done that.

I don't know that we need locking semantics. I just thought it would be easy
to apply the changes and merge them appropriately. For that matter, you're
going to have a similar problem if you have two threads in the same program
doing this.

Certainly it at least has to be made clear in the specs.

> > OK. I'd misinterpreted, thinking you meant only one process could read or
> > write at a time, rather than only one process could be using a configuration
> > at a time.
> 
> Ok, I guess I need to make that clearer. I thought one implied the
> other :). Your example is more complex than I had in mind; I'll try to
> include something like it in the final rationale document.

OK. I'm just trying to have ideas about problems before anyone puts a lot of
work into something, only to find out that the end-users find it too
unintuitive to use.

Something like "list API" is one thing, since only programmers who read the
instructions are going to see that. Something like a config file where the
user can set configurations is another thing. Perhaps it's worth having an
option to create a lock file of some sort.

> But I want _you_ to pick a format, so we can get consensus. Hmm. You said
> above either is ok; I guess that counts as consensus for the Java
> property format.

Yes. I've been pushing towards something where there are minimal syntactic
interdependencies between keys, rather than something like XML. I'm more
concerned about precise semantics than about the niggly details of the file
format, tho.

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Jacob Sparre Andersen]

Darren New wrote:
> 
> Stephen Leake wrote:
> > Let's see. You have two instances of "text_edit", sharing one config
> > file. That is explicitly forbidden by requirement 1.
> 
> Then requirement one should speak of concurrent use, rather than
> simultaneous use. :-)

This discussion made me think of how my e-mail client works
(typically, I have five running instances of it). It somehow
keeps an eye on changes to the configuration file, so if one
instance has saved a changed configuration, the other
instances will (when they happen to feel like looking at the
configuration file) complain, and ask if I want to use the
in-memory or the disk configuration.

I don't know how trivial the implementation of this feature
is, but just keeping a copy of the time stamp of the file
with the in-memory configuration would help a bit, although
one also could use a file check-sum.

Jacob
-- 
People in cars cause accidents. Accidents in cars cause
people.

Re: Config_Files proposal

[Stephen Leake]

Jacob Sparre Andersen <sparre@nbi.dk> writes:

> Darren New wrote:
> > 
> > Stephen Leake wrote:
> > > Let's see. You have two instances of "text_edit", sharing one config
> > > file. That is explicitly forbidden by requirement 1.
> > 
> > Then requirement one should speak of concurrent use, rather than
> > simultaneous use. :-)
> 
> This discussion made me think of how my e-mail client works
> (typically, I have five running instances of it). It somehow
> keeps an eye on changes to the configuration file, so if one
> instance has saved a changed configuration, the other
> instances will (when they happen to feel like looking at the
> configuration file) complain, and ask if I want to use the
> in-memory or the disk configuration.
> 
> I don't know how trivial the implementation of this feature
> is, but just keeping a copy of the time stamp of the file
> with the in-memory configuration would help a bit, although
> one also could use a file check-sum.

Yes, there are ways to do it. 

I think we have three choices here:

1) Simply forbid simultaneous and/or concurrent access to a single
   config file (as the requirements currently state).

2) Allow simultaneous and/or concurrent access, and define the
   behavior to be similar to Jacob's email program.

3) Allow full concurrent access to keys, the way the registry does.

I chose 1) originally to keep things simple, and because it meets my
needs.

I think we should avoid 3), because it's way too hard (you get into
"change notification messages"), and because the registry and gconf
already do this nicely.

I could go for 2), if enough people think it is important.

Or we could agree to do 2) later, after everything else.

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> 1) Simply forbid simultaneous and/or concurrent access to a single
>    config file (as the requirements currently state).

I think "forbid simultaneous access" is fine. But that's not what you're 
doing. You're doing "screw the user if there's simultaneous access." Kind of 
the way C "forbids" indexing off the end of an array. ;-)

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> Stephen Leake wrote:
> > 1) Simply forbid simultaneous and/or concurrent access to a single
> >    config file (as the requirements currently state).
> 
> I think "forbid simultaneous access" is fine. But that's not what
> you're doing. You're doing "screw the user if there's simultaneous
> access." Kind of the way C "forbids" indexing off the end of an array.
> ;-)

Well, that's true; I'm only "forbidding" in the requirements and the
user guide, not in the code. But what else can I do, without assuming
more about what the operating system can do?

I'd like to use either Ada.Text_IO, or some Gnat package to do all the
file management stuff. I don't think there is any support for locking
in there, but I'll admit I haven't really looked.

It would be nice if we only use Ada.Text_IO, because then it will be
more portable to other compilers.

-- 
-- Stephe

Re: Config_Files proposal

[Ted Dennison]

Darren New wrote:
> I think "forbid simultaneous access" is fine. But that's not what you're 
> doing. You're doing "screw the user if there's simultaneous access." 
> Kind of the way C "forbids" indexing off the end of an array. ;-)

I'm probably missing something here. How is this worse than the current 
situation with Ada.Text_IO?

Re: Config_Files proposal

[Marin David Condic]

I'd like to understand how to do some form of file lock-out for simultaneous
access by different processes without immediately getting into some kind of
OS dependant code. I can imagine a variety of semaphore schemes that will
depend on the OS or at least semi-portable facilities, but if the objective
is to to it strictly within the bounds of Ada, I can't think of a way to do
that.

If the goal is pure, portable, run-anywhere Ada code, I'm betting that it
will come down to "Don't do simultaneous access - it would be Bad..."

MDC
--
Marin David Condic
Senior Software Engineer
Pace Micro Technology Americas    www.pacemicro.com
Enabling the digital revolution
e-Mail:    marin.condic@pacemicro.com


"Ted Dennison" <dennison@telepath.com> wrote in message
news:3D1289D2.9090107@telepath.com...
> Darren New wrote:
> > I think "forbid simultaneous access" is fine. But that's not what you're
> > doing. You're doing "screw the user if there's simultaneous access."
> > Kind of the way C "forbids" indexing off the end of an array. ;-)
>
> I'm probably missing something here. How is this worse than the current
> situation with Ada.Text_IO?
>

Re: Config_Files proposal

[Stephen Leake]

"Marin David Condic" <dont.bother.mcondic.auntie.spam@[acm.org> writes:

> I'd like to understand how to do some form of file lock-out for simultaneous
> access by different processes without immediately getting into some kind of
> OS dependant code. 

You can't; file locking is OS dependant.

However, that in itself is ok for Config_Files. There is no
_requirement_ that it be possible to implement Config_Files in an OS
independent way. It would be nice, of course.

> I can imagine a variety of semaphore schemes that will depend on the
> OS or at least semi-portable facilities, but if the objective is to
> to it strictly within the bounds of Ada, I can't think of a way to
> do that.

That is _not_ one of the objectives. The main objective is to produce
something useful; sometimes that requires OS dependence.

> If the goal is pure, portable, run-anywhere Ada code, I'm betting
> that it will come down to "Don't do simultaneous access - it would
> be Bad..."

Not the goal.

My objection to getting into file locking is twofold:

1) It's complex, and hard to get right.

2) The registry already does it, so use that if you need it.

I believe there are many applications for config files that don't need
file locking. But I'm willing to be convinced otherwise. 

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Ted Dennison wrote:
> Darren New wrote:
> 
>> I think "forbid simultaneous access" is fine. But that's not what 
>> you're doing. You're doing "screw the user if there's simultaneous 
>> access." Kind of the way C "forbids" indexing off the end of an array. 
>> ;-)
> 
> 
> I'm probably missing something here. How is this worse than the current 
> situation with Ada.Text_IO?

If I have a text editor program and a compiler,  I can save the source from 
my editor window (using Text_IO to write it) and then compile it with the 
compiler (using Text_IO to read it). This works like the user expects.

What the user doesn't expect is to run two copies of the text editor, change 
the configuration in one, and have that configuration disappear simply 
because they closed the other window later.

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

R> Ted Dennison wrote:
> > Darren New wrote:
> >
> >> I think "forbid simultaneous access" is fine. But that's not what
> >> you're doing. You're doing "screw the user if there's simultaneous
> >> access." Kind of the way C "forbids" indexing off the end of an
> >> array. ;-)
> > I'm probably missing something here. How is this worse than the
> > current situation with Ada.Text_IO?
> 
> If I have a text editor program and a compiler,  I can save the source
> from my editor window (using Text_IO to write it) and then compile it
> with the compiler (using Text_IO to read it). This works like the user
> expects.
> 
> What the user doesn't expect is to run two copies of the text editor,
> change the configuration in one, and have that configuration disappear
> simply because they closed the other window later.

That's what happens when you run two copies of the text editor, change
the same source file in both, and save from both. One set of changes
is lost.

Why should config files be different?

Hmm, one answer is "because the config file is hidden". That makes
sense; we can expect the user to be aware of loading the same source
file into to copies of the text editor, but not immediately aware of
sharing one config file between the two copies of the text editor.

But I still think the right answer in this case is one of:

"use a better (multi-document) text editor".

"use the registry for resources shared by multiple processes".

"use separate config files for separate instances of applications"

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> That's what happens when you run two copies of the text editor, change
> the same source file in both, and save from both. One set of changes
> is lost.

Well, maybe in *your* editors, based on file systems as horrificly bad as
UNIX. ;-)

> Why should config files be different?

Because reading a config file and writing a config file are both invisible
operations, as far as the user is concerned.
 
> Hmm, one answer is "because the config file is hidden". That makes
> sense; we can expect the user to be aware of loading the same source
> file into to copies of the text editor, but not immediately aware of
> sharing one config file between the two copies of the text editor.

Exactly. And there are commands to save the files and to load the files.
 
> But I still think the right answer in this case is one of:
> "use a better (multi-document) text editor".
> "use the registry for resources shared by multiple processes".
> "use separate config files for separate instances of applications"

What's wrong with #4 - merge changes into the most recent config file,
rather than just overwriting it?

How about when you go to save the new config file, you check the
size/date/checksum/contents/whatever against what it was when you loaded it,
and you throw an exception if it's different, *unless* the overwrite flag is
set? Then those who want to merge the changes can do so "manually" in their
code?

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> Stephen Leake wrote:
> > That's what happens when you run two copies of the text editor, change
> > the same source file in both, and save from both. One set of changes
> > is lost.
> 
> Well, maybe in *your* editors, based on file systems as horrificly bad as
> UNIX. ;-)
> 
> > Why should config files be different?
> 
> Because reading a config file and writing a config file are both invisible
> operations, as far as the user is concerned.
>  
> > Hmm, one answer is "because the config file is hidden". That makes
> > sense; we can expect the user to be aware of loading the same source
> > file into to copies of the text editor, but not immediately aware of
> > sharing one config file between the two copies of the text editor.
> 
> Exactly. And there are commands to save the files and to load the files.
>  
> > But I still think the right answer in this case is one of:
> > "use a better (multi-document) text editor".
> > "use the registry for resources shared by multiple processes".
> > "use separate config files for separate instances of applications"
> 
> What's wrong with #4 - merge changes into the most recent config file,
> rather than just overwriting it?

Because you need "real locks", and you need to guard against all the
possible race conditions. The registry already does that.

When _any_ other process may write to the config file at _any_ time,
it is not possible to define "the most recent config file". You have
to lock it to get exclusive access, then merge your changes.

What's wrong with the answers I gave (besides "I don't like them" :)?

> How about when you go to save the new config file, you check the
> size/date/checksum/contents/whatever against what it was when you
> loaded it, and you throw an exception if it's different, *unless*
> the overwrite flag is set? Then those who want to merge the changes
> can do so "manually" in their code?

If we want to support this scenario (multiple processes reading and
writing to shared config file), we need "real locks". We also need
"real notification" so apps know when the config files have changed.
"real locks" is doable on most reasonable OSes, it's just complicated.
"real notification" is much harder. The registry already does both.

In light of Robert Eachus's post, I need to play with implementing at
least an exclusive read/write lock on a config file. When I do that,
I'll see how hard it would be to do the "merge changes when write"
approach. Maybe that will be easy to add.

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> Because you need "real locks", and you need to guard against all the
> possible race conditions. The registry already does that.

But you can eliminate a large number of race conditions by implementing a
merge. You can change it from "if you start the same program twice, you're
screwed" to "if you click the button on both instances in the length of time
it takes to write the file, you're screwed." I think that's reasonable. You
might not.
 
> What's wrong with the answers I gave (besides "I don't like them" :)?

Well, it's an opinion. 
 
> If we want to support this scenario (multiple processes reading and
> writing to shared config file), we need "real locks".

Sure, there's a race condition if you write the file between the time you
check if it has changed and the time you write it out. I think that's a much
smaller window than the window between the time you start the program and
the time you exit it.

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> Stephen Leake wrote:
> > Because you need "real locks", and you need to guard against all the
> > possible race conditions. The registry already does that.
> 
> But you can eliminate a large number of race conditions by implementing a
> merge. You can change it from "if you start the same program twice, you're
> screwed" to "if you click the button on both instances in the length of time
> it takes to write the file, you're screwed." I think that's reasonable. 

I see.

> You might not.

Right, I don't. I tend to be very picky; a file locking protocol is
either "right", or it's "wrong"; no room for "mostly right" :).

> > What's wrong with the answers I gave (besides "I don't like them"
> > :)?
> 
> Well, it's an opinion. 

I was hoping for something technical, that would allow judging the
benefit against the cost of doing file locking right.

> > If we want to support this scenario (multiple processes reading
> > and writing to shared config file), we need "real locks".
> 
> Sure, there's a race condition if you write the file between the time you
> check if it has changed and the time you write it out. I think that's a much
> smaller window than the window between the time you start the program and
> the time you exit it.

It is smaller, but I'm willing to bet someone will hit it. For
example, when you shutdown the computer, all the active editor
instances will write to the config file at once. Maybe us "real"
programmers don't leave editors active when we shutdown, but a lot of
people do!

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> Right, I don't. I tend to be very picky; a file locking protocol is
> either "right", or it's "wrong"; no room for "mostly right" :).

Avoid Linux, then. Its file locking has starvation problems built into the
kernel. ;-)

> I was hoping for something technical, that would allow judging the
> benefit against the cost of doing file locking right.

Well, there ya have what I can give. :-) By "it's my opinion" I meant "it's
my opinion that it's worth doing the merge." Technically, it's not
especially difficult. Also technically, I guess one could really do this
manually if one wanted to, loading up a second "config" object and doing the
comparison oneself, so I guess I can withdraw my objections. I.e., one can
"code around" the problem, and with the proper iterators built into the
config API (allowing you to iterate over all keys, for example), one need
only code this once.

> It is smaller, but I'm willing to bet someone will hit it. For
> example, when you shutdown the computer, all the active editor
> instances will write to the config file at once. Maybe us "real"
> programmers don't leave editors active when we shutdown, but a lot of
> people do!

Good point. I hadn't thought of that. Hmmm...  Of course, there's no way to
catch the system shutting down out from under you in a portable way either,
so in reality it's more likely to lose *both* sets of changes. :-)

Ok. Nevermind. Rock on. You have convinced me with your superior logic. :-)

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Ted Dennison]

Stephen Leake wrote:
> Darren New <dnew@san.rr.com> writes:
> 
>>But you can eliminate a large number of race conditions by implementing a
>>merge. You can change it from "if you start the same program twice, you're
>>screwed" to "if you click the button on both instances in the length of time
>>it takes to write the file, you're screwed." I think that's reasonable. 

I think when they fixed the Emacs ada-mode case exceptions file writing 
algorithm, they did something like this. I was satisfied at the time, 
for the exact reason Darren mentions.

> 
> Right, I don't. I tend to be very picky; a file locking protocol is
> either "right", or it's "wrong"; no room for "mostly right" :).

I'd have to agree with this. For a single application like Emacs, it may 
be good enough to come close. For something that is going to be an 
official part of Ada, I don't think we should put in a feature that 
proports to solve a problem that it doesn't actually solve.

> It is smaller, but I'm willing to bet someone will hit it. For
> example, when you shutdown the computer, all the active editor
> instances will write to the config file at once. Maybe us "real"

Exactly. And when someone gets this problem, hoses their config file 
that they had setup *juuuust* right, and tracks the problem down, they 
are going to be quite ticked that the language advertised something as 
safe when it in fact wasn't.

> programmers don't leave editors active when we shutdown, but a lot of
> people do!

Actually, I would probably not have that happen to me, just because I 
never purposely shut my systems down. They only way they go down is the 
hard way. :-)

Re: Config_Files proposal

[Randy Brukardt]

To stick my head into a debate that I probably should stay out of:

Stephan said:

>But I still think the right answer in this case is one of:
>
>"use a better (multi-document) text editor".
>
>"use the registry for resources shared by multiple processes".
>
>"use separate config files for separate instances of applications"


I think these are silly. I've always been dubious of this package, and
if this is the result, then there is no reason to bother, because it
would be totally useless.

Let's look at each of these "solutions"

>"use a better (multi-document) text editor".

This doesn't have any effect on the problem. On modern (multi-tasking)
systems, it is possible to run multiple copies of a program. If that
happens, you still have the problem. For instance, you can run multiple
copies of Word, even though there is no NEED to do so. And, as someone
pointed out, users won't read the manual about that anyway. If you
believe in solid applications (and if you don't, why are you using
Ada?), "user beware" is not an option.

>"use separate config files for separate instances of applications"

This doesn't seem to be much of a solution, either. You're requiring the
user to specify the "separate" file in some way. (You could use a PID or
the like for this, but then it hardly is a config file, as it would be
different for every invokation). Since this is likely to be a pain,
you'll have a default. And then you'll be back to the first situation,
because users won't read the manual, and they'll run multiple copies

>"use the registry for resources shared by multiple processes".

This of course works, but now you've lost the portability. Moreover,
since I don't believe that any carefully written apps could ever assume
single access, the Config_Files package could never be used. So why
bother to define it?

Summary: Requirement 1 has to go. For this package to be useful, it has
to make it easy to create bullet-proof configuration files, and more
portably than relying on the registry. (If you just want to use the
registry, use Claw.Registry, which is plenty portable on Windows.) Yes,
this is a lot of work. But if this was easy, there wouldn't be any need
for the package. (Just writing a value to a file hardly requires a fancy
package.)

                Randy Brukardt.

Re: Config_Files proposal

[Ted Dennison]

OK. I think I'm finally getting the problem here. This is my old issue 
with the emacs ada-mode case exceptions file from two years ago, come 
back to haunt me. :-(

Randy Brukardt wrote:
> I think these are silly. I've always been dubious of this package, and
> if this is the result, then there is no reason to bother, because it
> would be totally useless.

I think that's a bit extreme. At most, I think one could say that it 
would not make for a good solution in some cases. In particular, it 
would be a problem any time the program itself is both reading and 
writing config files, *and* there is the possibility of multiple copies 
of the program running simultaniously. Remember that we must include 
networked installations of a program under this umbrella.

This situation includes just about any desktop user app (eg: an Ada text 
editor). However, it happens to exclude every situation where I've found 
myself using a configuration file in the past (generally systems 
software). So I think calling it "useless" is overstating the case, 
possibly to the detriment of the point.

> Summary: Requirement 1 has to go. For this package to be useful, it has
> to make it easy to create bullet-proof configuration files, and more

I should point out here that the solution for this problem with Emacs 
ada-mode case exceptions is not "bullet-proof" either. However, it was 
close enough for me (the original complainer), and I have yet to hear 
anyone call ada-mode's case exceptions "useless" since then.

Say we do remove requirement 1. What do we replace it with? One 
possiblity is "nothing". But in that case we will make a portable 
reference implementation that doesn't bother with this case, nearly 
every vendor will just use the reference, and you will still not have 
this situation handled.

So what's the alternative you'd like? In other words, how does one solve 
this problem so that its impossible to loose changes due to simultanious 
edits?

Probably the way to think of this of this problem is in terms of 
revision-control systems.

We could only allow files to be updated with "delta" operations. But 
even then, you have the old revision-control problem of merging 
conflicting changes. I've yet to see a merging algorithm that didn't 
have situations that require human intervention. However, with our file 
format we probably couldn't detect these conflicts, so we'd just be back 
to the problem of wiped-out changes (but with a much smaller window for 
the wipeout to occur). This is sort of like an unlocked checkout in 
revision control systems. I think this is roughly the solution they used 
for the emacs ada-mode problem.

Another possibility would be some kind of explicit "edit" call that 
locks the file and reloads everything, and an "end edit" call that saves 
and unlocks it. But the client would still need to be very careful to 
display the read-in configurations for editing, not the ones that the 
program read in when it started. This is the "locked check-out" 
approach. The problem with this approach is that the locker may go AWOL, 
holding up everyone's changes for an indefinite period.

Re: Config_Files proposal

[Randy Brukardt]

Ted Dennison wrote in message <3D1D22BB.7000305@telepath.com>...
>OK. I think I'm finally getting the problem here. This is my old issue
>with the emacs ada-mode case exceptions file from two years ago, come
>back to haunt me. :-(
>
>Randy Brukardt wrote:
>> I think these are silly. I've always been dubious of this package,
and
>> if this is the result, then there is no reason to bother, because it
>> would be totally useless.
>
>I think that's a bit extreme. At most, I think one could say that it
>would not make for a good solution in some cases. In particular, it
>would be a problem any time the program itself is both reading and
>writing config files, *and* there is the possibility of multiple copies
>of the program running simultaniously. Remember that we must include
>networked installations of a program under this umbrella.
>
>This situation includes just about any desktop user app (eg: an Ada
text
>editor). However, it happens to exclude every situation where I've
found
>myself using a configuration file in the past (generally systems
>software). So I think calling it "useless" is overstating the case,
>possibly to the detriment of the point.


Well, "useless" is a bit extreme, but not much. You say that pretty much
all desktop apps couldn't use the package if Requirement 1 remains.
Moreover, most embedded applications can't use the package, because they
don't have a file system. So that leaves how many applications? Let's be
generous and say 10%. Why should we bother with a "standard" package
that can safely be used in only 10% of the applications? That hardly
needs to be standardized!

>> Summary: Requirement 1 has to go. For this package to be useful, it
has
>> to make it easy to create bullet-proof configuration files, and more
>
>I should point out here that the solution for this problem with Emacs
>ada-mode case exceptions is not "bullet-proof" either. However, it was
>close enough for me (the original complainer), and I have yet to hear
>anyone call ada-mode's case exceptions "useless" since then.
>
>Say we do remove requirement 1. What do we replace it with? One
>possiblity is "nothing". But in that case we will make a portable
>reference implementation that doesn't bother with this case, nearly
>every vendor will just use the reference, and you will still not have
>this situation handled.
>
>So what's the alternative you'd like? In other words, how does one
solve
>this problem so that its impossible to loose changes due to
simultanious
>edits?


You can't exactly (and probably shouldn't in every possible case), but
you can adopt rules that make it unlikely that trouble will occur. I
suggest:

1) If two applications write different, unrelated keys in the same file,
both changes will be reflected, and nothing else will be changed.

2) If an application writes a group of keys, it needs to be possible
that the application can insure that the entire group is updated as a
set.

These the first rule means that the only time a simulateous edit will be
lost is when two apps write the same key. There isn't much that we can
do about that. The second rule means that an app can insure that some
consistent set of keys is written, even if another app is trying to
write the same keys. (Imagine a set of keys representing a screen
position, getting written at shutdown of a bunch of copies of an editor.
We don't care which position is saved, but we certainly want to insure
that a single complete position is saved, rather than parts of several.

>Another possibility would be some kind of explicit "edit" call that
>locks the file and reloads everything, and an "end edit" call that
saves
>and unlocks it. But the client would still need to be very careful to
>display the read-in configurations for editing, not the ones that the
>program read in when it started. This is the "locked check-out"
>approach. The problem with this approach is that the locker may go
AWOL,
>holding up everyone's changes for an indefinite period.


This can be mitigated by using a controlled object "lock" for editing.
Then, if the locking app is aborted or has an unhandled exception, the
lock will get freed. That prevents program bugs from keeping the file
locked. (You'll still need some way to deal with OS termination of the
application's process; probably the lock will have to have a time limit
on it.)

                  Randy.

Re: Config_Files proposal

[Ted Dennison]

Randy Brukardt wrote:
> Well, "useless" is a bit extreme, but not much. You say that pretty much
> all desktop apps couldn't use the package if Requirement 1 remains.
> Moreover, most embedded applications can't use the package, because they
> don't have a file system. So that leaves how many applications? Let's be

That leaves non-embedded real-time apps, which accout for all but one of 
the projects I've ever worked on. Every single one of those projects 
(with one trivial exception) had a configuration file.

> generous and say 10%. Why should we bother with a "standard" package

I don't think we have enough information to make that estimate. If I 
were to do it based on my past experience, it would be roughly come out 
to %75 of all Ada projects. Further, it would be %100 of all Ada 
projects that use configuration files. If the mood struck me, I could 
even try and argue something close to these numbers, based on Ada's 
percieved heavier usage in real-time apps than in desktop apps.

Of course this is unscientific, but that's my point; we don't have 
scientific numbers.

>>So what's the alternative you'd like? In other words, how does one
> solve
>>this problem so that its impossible to loose changes due to
> simultanious
>>edits?
> 
> You can't exactly (and probably shouldn't in every possible case), but

Are saying that you think we should put something in the Ada spec that 
doesn't solve the problem it was introduced for; something that meerly 
lessens the probablility of a problem occuring, but leaves the 
probablility there? There are probably instances of such things in the 
existing Ada spec, but I don't like the idea much.

> you can adopt rules that make it unlikely that trouble will occur. I
> suggest:
> 
> 1) If two applications write different, unrelated keys in the same file,
> both changes will be reflected, and nothing else will be changed.
 >
> 2) If an application writes a group of keys, it needs to be possible
> that the application can insure that the entire group is updated as a
> set.
> 
> These the first rule means that the only time a simulateous edit will be
> lost is when two apps write the same key. There isn't much that we can
> do about that. 

I don't understand this thought. If we were to have users explicitly 
lock files for editing (and reread the values immediately after doing 
so), we could prevent that. If we don't do that, then there's no way to 
perform rule #2.

>>Another possibility would be some kind of explicit "edit" call that
>>locks the file and reloads everything, and an "end edit" call that
...
>>approach. The problem with this approach is that the locker may go
> AWOL,
>>holding up everyone's changes for an indefinite period.
> 
> This can be mitigated by using a controlled object "lock" for editing.

No it can't. Controlled-objects aren't system-wide. This kind of locking 
can only be done through OS calls.

Even if they were system-wide (which they aren't), there's nothing that 
says that if the OS kills your Ada process then your controlled objects 
will all get finalized nicely for you. On every Unix OS I've used, a 
"kill -9" on an Ada program will *not* do that.

Re: Config_Files proposal

[Randy Brukardt]

Ted Dennison wrote in message <3D20F3E3.8090104@telepath.com>...

Sigh. I *knew* I shouldn't get involved in this one. It would help a lot
of people would read what I wrote...


>Randy Brukardt wrote:
>> Well, "useless" is a bit extreme, but not much. You say that pretty
much
>> all desktop apps couldn't use the package if Requirement 1 remains.
>> Moreover, most embedded applications can't use the package, because
they
>> don't have a file system. So that leaves how many applications? Let's
be
>
>That leaves non-embedded real-time apps, which accout for all but one
of
>the projects I've ever worked on. Every single one of those projects
>(with one trivial exception) had a configuration file.

Huh? Obviously we have very different definitions of these things,
because there can be no such thing. As a first approximation, apps break
down into two categories - the kind that users explicitly launch on OSes
(which you somewhat inaccurately called desktop apps), and embedded apps
that aren't explicitly run. Whatever your projects have been either are
(non-traditional) embedded apps [which would be my guess], or "regular"
OS-hosted apps.

*All* user-lanuched apps need protection against multiple launches. You
better not fry the patient because someone started the program twice.
Admittedly, such protection could be done at a higher layer. But it
seems unusual to say the least to create a package which is designed to
fail in a likely case; after all, Ada is a safe language.

>> generous and say 10%. Why should we bother with a "standard" package
>
>I don't think we have enough information to make that estimate. If I
>were to do it based on my past experience, it would be roughly come out
>to %75 of all Ada projects. Further, it would be %100 of all Ada
>projects that use configuration files. If the mood struck me, I could
>even try and argue something close to these numbers, based on Ada's
>percieved heavier usage in real-time apps than in desktop apps.
>
>Of course this is unscientific, but that's my point; we don't have
>scientific numbers.


Since you're discussing a category that doesn't exist, its hard to
respond. Probably, your thinking of embedded application where there is
a file system. This seems to be a reasonably rare case in general,
although clearly there are specific instances of it. That would be the
only case where a package without write protection of some sort could be
useful. Obviously, your experience varies from mine a lot.

Anyway, if the package is defined without locking, it can only be useful
to Ted's type of applications (whatever they are). If the package is
defined with some sort of locking, it can be useful to most
applications. Moreover, an implementation specifically for Ted's
applications could leave out the locking, and still meet the
specification. So I find it hard to imagine that we would want to define
this stuff to be unsafe for the majority of applications, when it is not
hard to make it safe enough for most apps.

>>>So what's the alternative you'd like? In other words, how does one
solve
>>>this problem so that its impossible to loose changes due to
simultanious
>>>edits?
>>
>> You can't exactly (and probably shouldn't in every possible case),
but
>
>Are saying that you think we should put something in the Ada spec that
>doesn't solve the problem it was introduced for; something that meerly
>lessens the probablility of a problem occuring, but leaves the
>probablility there? There are probably instances of such things in the
>existing Ada spec, but I don't like the idea much.

No, I'm saying that if there are two changes to the same key at the same
time, which one gets stored is implementation-defined. I don't see much
benefit in trying to define what happens in that case; usually one or
the other is good enough anyway. If its not, you'll probably need to use
a first-class database, which is clearly out of scope.

But if two apps write different keys to the same file at the same time,
both changes should be reflected. A race condition is a bad thing.

You seem to worry about about the idea that writing the same key at the
same time doesn't have a defined result. But that's no different than
writing the same variable twice at the same time, and that certainly
doesn't have a defined result.

One of the biggest advantages of using a "canned" package like this (or
like Claw for that matter), rather than rolling your own, is simply that
the "canned" package is much more likely to handle unlikely but
dangerous cases correctly. Especially if it is widely used. It is rare
that the normal case is enough work that it is worth figuring out
someone else's package -- it is the handling of unusual conditions (like
locking) that can be the big benefit.

>> you can adopt rules that make it unlikely that trouble will occur. I
>> suggest:
>>
>> 1) If two applications write different, unrelated keys in the same
file,
>> both changes will be reflected, and nothing else will be changed.
> >
>> 2) If an application writes a group of keys, it needs to be possible
>> that the application can insure that the entire group is updated as a
>> set.
>>
>> These the first rule means that the only time a simulateous edit will
be
>> lost is when two apps write the same key. There isn't much that we
can
>> do about that.
>
>I don't understand this thought. If we were to have users explicitly
>lock files for editing (and reread the values immediately after doing
>so), we could prevent that. If we don't do that, then there's no way to
>perform rule #2.


A *user* could prevent that. But the default case should allow rule #1,
even if there is no explicit locking. Rule #1 should *always* hold - for
*any* write. Indeed, rule #1 here is much more important than rule #2.
With rule #1, you can arrange your own locking to implement rule #2.
Without rule #1, you can't assume much of anything.

This package should be reasonably bullet-proof -- a simple use should be
reasonably safe by default. That means it must do some locking
internally.

Note that if the package supports "groups" of keys (as had been
suggested elsewhere), we don't need rule #2 at all. We can just provide
operations for updating an entire group of keys at once, and leave the
locking implicit. That may be preferable.

>>>Another possibility would be some kind of explicit "edit" call that
>>>locks the file and reloads everything, and an "end edit" call that
>...
>>>approach. The problem with this approach is that the locker may go
AWOL,
>>>holding up everyone's changes for an indefinite period.
>>
>> This can be mitigated by using a controlled object "lock" for
editing.
>
>No it can't. Controlled-objects aren't system-wide. This kind of
locking
>can only be done through OS calls.

>
>Even if they were system-wide (which they aren't), there's nothing that
>says that if the OS kills your Ada process then your controlled objects
>will all get finalized nicely for you. On every Unix OS I've used, a
>"kill -9" on an Ada program will *not* do that.


You obviously didn't read any of the rest of what I wrote here (and that
you didn't quote), which covered all of that.

Let me quote myself:

"Then, if the locking app is aborted or has an unhandled exception, the
lock will get freed. That prevents program bugs from keeping the file
locked. (You'll still need some way to deal with OS termination of the
application's process; probably the lock will have to have a time limit
on it.)"

Of course the lock has to be at the OS level. But you don't want to hold
it any longer than you have to. A controlled object provides protection
against program bugs holding the lock too long, by automatically
releasing the OS lock. It doesn't provide the entire solution (OS
termination - "kill" to Unix-heads - still has to be handled with a time
limit), but it lets the time limit case be quite rare, so we can leave
the lock held for quite a while before forcibly releasing it. That makes
the lock much safer, because if you let the lock be freed too soon (when
an App really is using it), you have all of the problems again -- only
worse, because the programmer has an expectation of safety.

               Randy.

Re: Config_Files proposal

[Robert I. Eachus]

Randy Brukardt wrote:


> "Then, if the locking app is aborted or has an unhandled exception, the
> lock will get freed. That prevents program bugs from keeping the file
> locked. (You'll still need some way to deal with OS termination of the
> application's process; probably the lock will have to have a time limit
> on it.)"
> 
> Of course the lock has to be at the OS level. But you don't want to hold
> it any longer than you have to. A controlled object provides protection
> against program bugs holding the lock too long, by automatically
> releasing the OS lock. It doesn't provide the entire solution (OS
> termination - "kill" to Unix-heads - still has to be handled with a time
> limit), but it lets the time limit case be quite rare, so we can leave
> the lock held for quite a while before forcibly releasing it. That makes
> the lock much safer, because if you let the lock be freed too soon (when
> an App really is using it), you have all of the problems again -- only
> worse, because the programmer has an expectation of safety.


Exactly.  The Ada standard has to be easy to use but prevent the 
"stupid" cases.  There will always be a way outside the Ada interface to 
muck things up:  Text editors, registry editors, system crashes, hard 
drive failures, etc.

What we have to worry about is the case of two (often different) Ada 
programs accessing the same configuration file and making a mess of it. 
  In other words, we want to guarantee that as long as all access is 
through the standard Ada package interface, strange things don't happen. 
  If strange things happen in the external environment that foul up the 
configuration file that is not our problem.

As far as I am concerned an interface that provided two types of handles 
should be sufficient.  You either have (shared) read access or unshared 
read-write access to file elements.  Whether the granularity is per 
file, per object, or some definition of group is something worth 
discussing.  But we have to have consistancy.  If one implementation 
locks files (Unix) and another locks objects (Windows Registry) then 
applications are not portable.

Since I feel that simplicity is best, I like the idea of configuration 
file locking.  Normal access is to open the file read it, then close it. 
  If your program wants to update the file it re-opens it read write. 
Yes this will not provide notification if the file changes after a 
program starts, but I think that is reasonable. I just find it hard to 
imagine situations where two programs update different but intersecting 
attributes.  For example let us say that one program changes the font 
size used in an output window, and another sets the size of the window. 
  I certainly want these actions to be done in some consistant order, 
but IMHO it would be bad programming if only one of the programs 
computed the correct line-length information...

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> Stephen Leake wrote:
> > Let's see. You have two instances of "text_edit", sharing one config
> > file. That is explicitly forbidden by requirement 1.
> 
> Then requirement one should speak of concurrent use, rather than
> simultaneous use. :-)

Yes, that is better.

> > This is a race condition. Suppose two processes happen to start a
> > "flush" operation "at the same time"? Whichever goes last wins.
> 
> That would be simultaneous use. ;-)

Ok, neither simultaneous nor concurrent use is supported.

> > If you feel that makes Config_Files totally useless, I guess we
> > can talk about removing requirement 1 and see if we can come up
> > with locking semantics to fix it. But I'd rather not, especially
> > since the registry and gconf have already done that.
> 
> I don't know that we need locking semantics. I just thought it would be easy
> to apply the changes and merge them appropriately. For that matter, you're
> going to have a similar problem if you have two threads in the same program
> doing this.

Yes, but Ada programmers are supposed to be aware of things like that.
I guess it wouldn't hurt to mention it.

> Certainly it at least has to be made clear in the specs.

yes.

> > > OK. I'd misinterpreted, thinking you meant only one process
> > > could read or write at a time, rather than only one process
> > > could be using a configuration at a time.
> > 
> > Ok, I guess I need to make that clearer. I thought one implied the
> > other :). Your example is more complex than I had in mind; I'll try to
> > include something like it in the final rationale document.
> 
> OK. I'm just trying to have ideas about problems before anyone puts a lot of
> work into something, only to find out that the end-users find it too
> unintuitive to use.

Yes, and I do appreciate that.

> Something like "list API" is one thing, since only programmers who
> read the instructions are going to see that. Something like a config
> file where the user can set configurations is another thing. Perhaps
> it's worth having an option to create a lock file of some sort.

Only if we can define it well.

> > But I want _you_ to pick a format, so we can get consensus. Hmm.
> > You said above either is ok; I guess that counts as consensus for
> > the Java property format.
> 
> Yes. I've been pushing towards something where there are minimal syntactic
> interdependencies between keys, rather than something like XML. I'm more
> concerned about precise semantics than about the niggly details of the file
> format, tho.

Ok, I'll put you on the Java side.

-- 
-- Stephe

Re: Config_Files proposal

[Dr. Michael Paus]

Stephen Leake wrote:

> I'd like to have a semi-formal vote on which file format to go with.
> Then I'll move this into Grace, and we can finish up all the details.
> 
> To vote, either post here with your preference, or send me email.
> Voting closes in a week (June 25), but I'm not going to be picky about
> time of day :).

I vote for the Java version. For the implementation you should try to
stay as close as possible to the Java version as well.

Michael

Re: Config_Files proposal

[Ted Dennison]

"Dr. Michael Paus" <paus@ib-paus.com> wrote in message news:<3D102A4E.2010900@ib-paus.com>...
> I vote for the Java version. For the implementation you should try to
> stay as close as possible to the Java version as well.

Another good example of my problem with votes. *Why* do you feel this way?

Re: Config_Files proposal

[Ted Dennison]

Stephen Leake <stephen.a.leake.1@gsfc.nasa.gov> wrote in message news:<uwuswy0qr.fsf@gsfc.nasa.gov>...
> I've posted another example spec and implementation, at 
> 
> http://users.erols.com/leakstan/Stephe/Ada/Config_Files/config_files.html
> 
> This second implementation uses the Java property style file format.
> 
> I also changed the spec some. In particular, I got rid of the
> "Section" parameters; keys now just use dotted notation.

I think either the Java or the X style would be acceptable. With a bit
of work, someone could probably sell me on the ini style too though.

> I'd like to have a semi-formal vote on which file format to go with.
> Then I'll move this into Grace, and we can finish up all the details.

I hate to do this, but I have to go on record as being strongly
against voting for things. Everything in Grace needs to be decided by
general consensus. The only reason there isn't an official policy on
this at the Grace website is because I haven't had time to put it up.

I'm not saying everyone has to agree that one of these styles is the
best of all possible styles. But there should be some kind of general
agreement that one of them is meritous and acceptable, if not
everyone's perfect option. If we get down to two that seem to be
equally good with no huge contraversy between them, then perhaps a
vote or coin flip or executive decision or something similar would be
in order.

BTW: I have recently appointed Stephen to lead the config file effort
for Grace (see http://savannah.gnu.org/projects/grace/ ). So if there
were an "executive decision" to be made in the context of the Grace
project, it would be up to him to make it.

This is a good example of why we need to go with this approach. We may
now have consensus against using XML, but there really hasn't been
much discussion here on the relative merits of the other approaches
(other than that some folks think INI's should be reseved for
Windows). If I were to just vote now between the other 3 options, it
would be based on looks. Another thread here has recently illustrated
quite well that this would be quite succeptable to parrochial biases.
:-)

So really a better question right now would be to ask if anyone has
any big objections to any of the approaches presented (other than XML,
to which the objections are already well recorded). Also, does anyone
have any strong reasons why they think one is much better than the
others?

Re: Config_Files proposal

[Robert I. Eachus]

Ted Dennison wrote:

 
> So really a better question right now would be to ask if anyone has
> any big objections to any of the approaches presented (other than XML,
> to which the objections are already well recorded). Also, does anyone
> have any strong reasons why they think one is much better than the
> others?


Yes.  But the problem is not with the various file formats, it is with 
the approach.  At this point we should be debating a set of requirements 
and a strawman interface package specification.  If the result favors 
one or the other file format, so be it.  My expectation is that a good 
set of requirements and interface will allow more than one underlying 
implementation.  I'd love to see a result that maps to registry, .ini 
files, and .cfg files with no differences in usage outside the create 
and/or open parameters.

Re: Config_Files proposal

[Darren New]

"Robert I. Eachus" wrote:
> Yes.  But the problem is not with the various file formats, it is with
> the approach.  At this point we should be debating a set of requirements
> and a strawman interface package specification.  If the result favors
> one or the other file format, so be it.

Seconded! That's what I've been saying. Work out an API first, and the file
format will pretty much either fall out of it or be arbitrary where it
doesn't.

>  My expectation is that a good
> set of requirements and interface will allow more than one underlying
> implementation.  I'd love to see a result that maps to registry, .ini
> files, and .cfg files with no differences in usage outside the create
> and/or open parameters.

Seconded again.

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> "Robert I. Eachus" wrote:
> > Yes.  But the problem is not with the various file formats, it is with
> > the approach.  At this point we should be debating a set of requirements
> > and a strawman interface package specification.  If the result favors
> > one or the other file format, so be it.
> 
> Seconded! That's what I've been saying. Work out an API first, and the file
> format will pretty much either fall out of it or be arbitrary where it
> doesn't.

Since we keep saying the API _must_ be independent of the file format,
I don't see how finishing the API can help decide what file format to
use.

Note that Robert said "requirements and package spec", not just
"package spec".

> >  My expectation is that a good set of requirements and interface
> > will allow more than one underlying implementation. I'd love to
> > see a result that maps to registry, .ini files, and .cfg files
> > with no differences in usage outside the create and/or open
> > parameters.
> 
> Seconded again.

Well, I thought we did that. Do I need to wait longer than "a couple
weeks"? 

-- 
-- Stephe

Re: Config_Files proposal

[Marin David Condic]

Well, if, for example, the decisions for the API lead to some kind of
complex tree-like structure, you might have a really hard time storing this
in some simpler file format like .CSV or as simple attribute/value pairs.

I suggested elsewhere that establishing a package spec now and leaving the
file format for later would be a wise thing. It starts some progress and
helps clarify the problems. Assumimg that the API is going to have something
pretty basic like "Load/Store" operations with a file name and a data
structure, you really don't have to settle the file format issues until
later & it creates the possibility of trying out many different ones.

MDC
--
Marin David Condic
Senior Software Engineer
Pace Micro Technology Americas    www.pacemicro.com
Enabling the digital revolution
e-Mail:    marin.condic@pacemicro.com


"Stephen Leake" <stephen.a.leake.1@gsfc.nasa.gov> wrote in message
news:u4rfzw2kx.fsf@gsfc.nasa.gov...
>
> Since we keep saying the API _must_ be independent of the file format,
> I don't see how finishing the API can help decide what file format to
> use.
>

Re: Config_Files proposal

[Stephen Leake]

"Marin David Condic" <dont.bother.mcondic.auntie.spam@[acm.org> writes:

> Well, if, for example, the decisions for the API lead to some kind of
> complex tree-like structure, you might have a really hard time storing this
> in some simpler file format like .CSV or as simple attribute/value pairs.

But that would violate the requirement that the API _must_ be
independent of the file format, and would therefore be a bad API
decision.

> I suggested elsewhere that establishing a package spec now and
> leaving the file format for later would be a wise thing. It starts
> some progress and helps clarify the problems. 

If I generalize this to "make some decisions now", choosing the file
format now is also wise :).

> Assumimg that the API is going to have something pretty basic like
> "Load/Store" operations with a file name and a data structure, you
> really don't have to settle the file format issues until later & it
> creates the possibility of trying out many different ones.

Since we need to settle on one file format, and we've had about a
month of discussion on it, what more needs to be done to pick a file
format? 

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> Since we need to settle on one file format, and we've had about a
> month of discussion on it, what more needs to be done to pick a file
> format?

This is like asking "Well, since we've been arguing for a month about
whether we'll drive or fly on our vacation, what's wrong with picking a
vehicle and then deciding where to go?"

The file format cannot be independent of the API. You just have some very
loose and informal ideas of what the API ought to support, but without
making them concrete, solidifying the file format is premature. 

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> Stephen Leake wrote:
> > Since we need to settle on one file format, and we've had about a
> > month of discussion on it, what more needs to be done to pick a file
> > format?
> 
> This is like asking "Well, since we've been arguing for a month about
> whether we'll drive or fly on our vacation, what's wrong with picking a
> vehicle and then deciding where to go?"
> 
> The file format cannot be independent of the API. 

Well, ok, that's a definite change. Nobody has said that before, but I
have said the opposite many times.

> You just have some very loose and informal ideas of what the API
> ought to support, but without making them concrete, solidifying the
> file format is premature.

I guess I did not make an explicit requirement on the web page that
the API be independent of the file format, but I thought that was an
accepted principle.

Can we agree that the choice of file format is between Java properties
and XML? Given that, I believe the two example implementations I've
done show that the API is independent of the file format.

Granted the API is not finished, but the things we are still talking
about have no impact on the file format. Or am I wrong about that?

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> I guess I did not make an explicit requirement on the web page that
> the API be independent of the file format, but I thought that was an
> accepted principle.

Great. Now, my API wants to store binary, and I don't want to have to give
keys to every value, and I want to be able to store the file as part of a
larger file.

My point is that the file format makes certain assumptions. You're already
assuming heirarchical keys, each with a single textual value, that the order
of keys is independent, etc. You just haven't made that especially concrete.
I think it's a good idea to make it concrete before deciding on the file
format. But since you're doing the work and I'm just complaining, this is
merely a suggestion. :-)

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Stephen Leake]

Darren New <dnew@san.rr.com> writes:

> Stephen Leake wrote:
> > I guess I did not make an explicit requirement on the web page that
> > the API be independent of the file format, but I thought that was an
> > accepted principle.
> 
> Great. Now, my API wants to store binary, 

As base64; no problem.

> and I don't want to have to give keys to every value, 

hmm. I guess you mean you want a record or array type accessed by a
single key. I guess I'll agree; that has an impact on the file format.

However, that's not in the list of "things we are talking about". It
contradicts API requirement 21.

> and I want to be able to store the file as part of a larger file.

Say what? If the "larger file" has a sufficient protocol, you can do
this now.

> My point is that the file format makes certain assumptions. You're
> already assuming heirarchical keys, each with a single textual
> value, that the order of keys is independent, etc. You just haven't
> made that especially concrete. 

Well, that's been in the API requirements from the start, and not
seriously challenged.

> I think it's a good idea to make it concrete before deciding on the
> file format. But since you're doing the work and I'm just
> complaining, this is merely a suggestion. :-)

Actually, it is concrete; API requirements 18 thru 21. If you are
suggesting that these need to change, we can talk about that.


I guess I need to update the API requirements list, and show those
that we have consensus on, and those that are still under discussion.

-- 
-- Stephe

Re: Config_Files proposal

[Darren New]

Stephen Leake wrote:
> I guess I need to update the API requirements list, and show those
> that we have consensus on, and those that are still under discussion.

I think that's reasonable. I guess all I'm saying is I'd like to see the API
requirements list complete (even if not "agreed on") before putting anything
concrete in the file. Otherwise, I think it's premature. If you say (for
example) that the order of keys don't matter, and you want to be able to
associate a comment with multiple keys, your file format might need to be
radically different, just because you added requirements that weren't there
before.

Maybe I'm just misinterpreting the API requirements or something, trying to
either read more or less into them than you intended. :-)

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

     My brain needs a "back" button so I can
         remember where I left my coffee mug.

Re: Config_Files proposal

[Georg Bauhaus]

Stephen Leake <stephen.a.leake.1@gsfc.nasa.gov> wrote:
: hmm. I guess you mean you want a record or array type accessed by a
: single key. I guess I'll agree; that has an impact on the file format.

Isn't this drifting a bit towards a convenient general file
based storage mechanism for structured values?  Too much in my
view for Config_Files maybe. (I would build wrapper subprograms to
update record or array values.)

-- Georg

Re: Config_Files proposal

[Georg Bauhaus]

Stephen Leake <stephen.a.leake.1@gsfc.nasa.gov> wrote:
: Granted the API is not finished, but the things we are still talking
: about have no impact on the file format. Or am I wrong about that?

Looks like an XML/Ada based child package implementation of the current
API is unaffected, with or without file locking.

-- Georg

Re: Config_Files proposal

[Georg Bauhaus]

Darren New <dnew@san.rr.com> wrote:
: "Robert I. Eachus" wrote:
:> Yes.  But the problem is not with the various file formats, it is with
:> the approach.  At this point we should be debating a set of requirements
:> and a strawman interface package specification.  If the result favors
:> one or the other file format, so be it.
: 
: Seconded! That's what I've been saying. Work out an API first, and the file
: format will pretty much either fall out of it or be arbitrary where it
: doesn't.

O.K., and I say, for the last time, that this consensus rules out
any configuration process requirements that are not as
close to computers as interfaces and file formats.
I'm hoping and confident that Ada applications are configured by people
competent enough to do well with properties files.
As Stephen has said, don't use Config_Files otherwise.

Cheers,
 Georg

Re: Config_Files proposal

[Stephen Leake]

"Robert I. Eachus" <rieachus@attbi.com> writes:

> Ted Dennison wrote:
> 
>  
> > So really a better question right now would be to ask if anyone has
> > any big objections to any of the approaches presented (other than XML,
> > to which the objections are already well recorded). Also, does anyone
> > have any strong reasons why they think one is much better than the
> > others?
> 
> 
> Yes.  But the problem is not with the various file formats, it is with
> the approach.  At this point we should be debating a set of
> requirements and a strawman interface package specification. 

Well, I posted those a couple weeks ago, so I thought we did that. It
did not stimulate much discussion of the specific requirements or
package spec; all of the discussion was on the file format.

> If the result favors one or the other file format, so be it. 

It certainly went against XML. The choice among the other three is
less clear.

Do you find the rationale I've posted at
http://users.erols.com/leakstan/Stephe/Ada/Config_Files/config_files.html
to be insufficient? What's missing?

> My expectation is that a good set of requirements and interface will
> allow more than one underlying implementation. 

Yes. But we agreed at the very start that a standard file format was
required.

> I'd love to see a result that maps to registry, .ini files, and .cfg
> files with no differences in usage outside the create and/or open
> parameters.

Ok, but that's explicitly outside the scope of the Config_Files
project.

-- 
-- Stephe

Re: Config_Files proposal

[Robert I. Eachus]

Stephen Leake wrote:


> Well, I posted those a couple weeks ago, so I thought we did that. It
> did not stimulate much discussion of the specific requirements or
> package spec; all of the discussion was on the file format.


Ah, what was missing was not your draft requirements and interface, what 
was missing was the debate about the requirements.  That is finally 
starting to show up, so let me weigh in on the concurrent access issue. ;-)

There are really two separate sets of concerns here.  First, an 
application which expects users to edit the config file implicitly 
during program installation.  Later edits will be with a text editor, or 
a separate program that changes the config data.  For such uses, the 
semantics are pretty simple.  The program checks a potentially multiple 
set of config at initialization time, changes nothing, and edits while 
the program is running can either be left implementation dependent, or 
defined to be not seen.

A program which is going to change its config attributes during 
operation is a horse of a different color.  As I see it, the three 
choices are:

1) Opening the config for writing means that other instances of the 
program, or other programs that use the same file must either die with 
some reasonable behavior, or connect somehow to the first running 
instance.  Defining how to do the latter is well beyond the scope here.

2)  A program which can change the config file is required to follow a 
standard protocol. An example would be:

     a) If the config data is changed, the program uses a locking 
protocol of some sort to insure that there are no race conditions.  (For 
example, under Windows opening the file before the change works.)  The 
important thing from the specification view here is that the attempt to 
change a value or values can fail.  Whether the user view is an explicit 
  transaction protocol, or a multiple change commit protocol that can 
raise an exception is a detailed design decision.  (Although I favor the 
latter, with an exception occurance string that should be meaningful to 
the actual end user.)

     b) If the config file is changed by some other program or some 
other instance, the program will check for changes at defined points. 
Of course, these points will be defined by the program, but a good 
starting point would be to always check for changes before creating a 
new window or a new file, and of course before changing any config 
parameters.

     c)  If the program "rolls up" a set of config data from several 
locations, all values inherited from files other than the last will not 
be checked for changes while the program is running.  If you have 27 
files in the config path, only the last will be modified, or checked for 
changes while the program is running.

3) Chaos.

Re: Config_Files proposal

[Stephen Leake]

"Robert I. Eachus" <rieachus@attbi.com> writes:

> Stephen Leake wrote:
> 
> 
> > Well, I posted those a couple weeks ago, so I thought we did that. It
> > did not stimulate much discussion of the specific requirements or
> > package spec; all of the discussion was on the file format.
> 
> 
> Ah, what was missing was not your draft requirements and interface,
> what was missing was the debate about the requirements.  

Ok, I guess I'm being impatient :).

> That is finally starting to show up, so let me weigh in on the
> concurrent access issue. ;-)
> 
> There are really two separate sets of concerns here.  First, an
> application which expects users to edit the config file implicitly
> during program installation.  Later edits will be with a text editor,
> or a separate program that changes the config data.  For such uses,
> the semantics are pretty simple.  The program checks a potentially
> multiple set of config at initialization time, changes nothing, and
> edits while the program is running can either be left implementation
> dependent, or defined to be not seen.

Yes, that is an important use case that I should capture. 

> A program which is going to change its config attributes during
> operation is a horse of a different color. As I see it, the three
> choices are:
> 
> 1) Opening the config for writing means that other instances of the
> program, or other programs that use the same file must either die with
> some reasonable behavior, or connect somehow to the first running
> instance.  Defining how to do the latter is well beyond the scope here.

Yes. I think that conforms to the current "forbid simultaneous access"
requirement, but adds the requirement for an enforcement mechanism.

I don't think the GNAT library supports file locking of this type, but
it shouldn't be hard to do. It would make the package much safer.

> 2) A program which can change the config file is required to follow
> a standard protocol. An example would be:
> 
>      a) If the config data is changed, the program uses a locking
> protocol of some sort to insure that there are no race conditions.
> (For example, under Windows opening the file before the change works.)

Windows (and I assume other reasonable OSes) supports "open locked for
write", so that no other program can "open for read" while it is
locked. Then we need to define what the second program does when it
attempts the read; fail with an error message? or wait and timeout
with an error message?

> The important thing from the specification view here is that the
> attempt to change a value or values can fail. 

Read can also fail, if the file is locked for write.

> Whether the user view is an explicit transaction protocol, or a
> multiple change commit protocol that can raise an exception is a
> detailed design decision. (Although I favor the latter, with an
> exception occurance string that should be meaningful to the actual
> end user.)

I can see that this would work. I don't want to implement it :).

We could agree to do choice 1) above first, with this protocol as a
later enhancement.

>      b) If the config file is changed by some other program or some
> other instance, the program will check for changes at defined
> points. Of course, these points will be defined by the program, but
> a good starting point would be to always check for changes before
> creating a new window or a new file, and of course before changing
> any config parameters.

Ok, that makes sense. 

One alternative is to have the writer broadcast some sort of message,
which is way more complex. That's what the Windows registry does; I
assume Gnome gconf does something similar.

Without instant notification of changes it will be pretty easy to set
up situations that will confuse the user. But maybe we can live with
it. 

>      c) If the program "rolls up" a set of config data from several
> locations, all values inherited from files other than the last will
> not be checked for changes while the program is running. If you have
> 27 files in the config path, only the last will be modified, or
> checked for changes while the program is running.

Hmm. The current app can only change the "last" config file. But some
other app may have one of the other config files as its "writable"
file, so any of the 27 may change. I don't think we want to ignore
those changes.

> 3) Chaos.

Hmm. Are you saying that simply forbidding access by multiple
processes, without an enforcement mechanism, is a recipe chaos? I
guess that could be true. Users will ignore warnings in the
application documentation, since they won't read it :).

-- 
-- Stephe

Re: Config_Files proposal

[Marin David Condic]

XML is significantly better than some of the other formats suggested here
for one factor: Other tools will be able to make use of it. If it comes to
displaying an initialization file or using the data in any other way, you
gain the ability to make them work in entirely new contexts without having
to do some form of data conversion.

All other things being equal, gaining an XML parser in the process would be
A Good Thing. Imagine if one day when everything is sunshine and rainbows
and pastel unicorns and Ada has some kind of standard means of displaying
web page stuff, your initialization file XML data slides right in there.
Imagine that once the tools are there to handle XML on input and output how
many other things might be built on top of Grace that could leverage this
capability.

IOW, use this as an excuse to build an XML DOM & SAX. (I'd like to see an OO
implementation of this using tagged records and closely following the W3C
DOM model, despite the very nice non-OO implementation done by XML/Ada.)

There's my $0.02 on it. :-)

MDC
--
Marin David Condic
Senior Software Engineer
Pace Micro Technology Americas    www.pacemicro.com
Enabling the digital revolution
e-Mail:    marin.condic@pacemicro.com


"Ted Dennison" <dennison@telepath.com> wrote in message
news:4519e058.0206190708.2ef205e4@posting.google.com...
>
> So really a better question right now would be to ask if anyone has
> any big objections to any of the approaches presented (other than XML,
> to which the objections are already well recorded). Also, does anyone
> have any strong reasons why they think one is much better than the
> others?

Re: Config_Files proposal

[Georg Bauhaus]

Marin David Condic <dont.bother.mcondic.auntie.spam@[acm.org> wrote:

[...] Yes, I'll second that! (A configuration process in a multi-
company setting with people from a very heterogenous set of professions
might profit from a standard that was inspired by data representation
issues for more than 30 years.)

: I'd like to see an OO
: implementation of this using tagged records and closely following the W3C
: DOM model, despite the very nice non-OO implementation done by XML/Ada.

From what I can see, there isn't too much to be done, since the OO
approach is already reflected in discriminated subtypes and corresponding
packages. DOM2 support seems quite complete, the IDL interfaces have
been mostly implemented in XML/Ada.

: There's my $0.02 on it. :-)
Add EUR 0.02. :-)
 
-- Georg

Re: Config_Files proposal

[Marin David Condic]

Its not that the XML/Ada thing is "bad" - just that I would prefer one that
built the whole thing up with tagged records, dispatching, etc., rather than
using discriminated records. Its almost like it is saying "Oh, the DOM
specified by the W3C is very OO and could readily be recognized in *real* OO
languages like C++ or Java - complete with inheritance and dispatching and
all the cool OO stuff, but since Ada isn't *really* an OO language it has to
kind of kludge the DOM together with discriminated records and all that."

That, and I've always felt that discriminated records were a pain in the
posterior. Maybe its just me, but they always seem to cause trouble as you
run into language rules with them that don't let them do what you want to
do. Tagged records just seem like a much more elegant solution.

I've looked over the XML/Ada stuff just enough to know that trying to
umbundle the discriminated records and turn them into tagged records with
inheritance and dispatching and all that, would be something beyond
"trivial". I wouldn't see the point in even attempting to do so unless that
was the direction chosen for the future of the project by the guys who
currently own it. (Would you really want to see two divergent
implementations, or would it be better to have just one?)

MDC
--
Marin David Condic
Senior Software Engineer
Pace Micro Technology Americas    www.pacemicro.com
Enabling the digital revolution
e-Mail:    marin.condic@pacemicro.com


"Georg Bauhaus" <sb463ba@l1-hrz.uni-duisburg.de> wrote in message
news:aesjsv$7kv$1@a1-hrz.uni-duisburg.de...
>
> From what I can see, there isn't too much to be done, since the OO
> approach is already reflected in discriminated subtypes and corresponding
> packages. DOM2 support seems quite complete, the IDL interfaces have
> been mostly implemented in XML/Ada.
>

Re: Config_Files proposal

[Stephen Leake]

"Marin David Condic" <dont.bother.mcondic.auntie.spam@[acm.org> writes:

> XML is significantly better than some of the other formats suggested here
> for one factor: Other tools will be able to make use of it. If it comes to
> displaying an initialization file or using the data in any other way, you
> gain the ability to make them work in entirely new contexts without having
> to do some form of data conversion.

True.

> All other things being equal, gaining an XML parser in the process
> would be A Good Thing. 

Yes, but I think you'll agree that all other things are not equal
today. And since I am _not_ proposing to write an XML parser, but only
to reuse an existing one, we are not "gaining" anything; those parsers
are already there.

> Imagine if one day when everything is sunshine and rainbows and
> pastel unicorns and Ada has some kind of standard means of
> displaying web page stuff, your initialization file XML data slides
> right in there. 

XML is not "web page stuff"; Netscape doesn't know how to display it.
It has to be processed into HTML first (unless I am really missing
something). It's just as easy to process a Config_File into HTML. If
the Config_File standard catches on, it would be reasonable to define
and implement a standard translation to XML and HTML and whatever else
comes along.

> Imagine that once the tools are there to handle XML on input and
> output how many other things might be built on top of Grace that
> could leverage this capability.

Yes, that would be nice. But it is way outside the scope of the
Config_File mandate.

Even if we had better XML tools in Grace today, I think I would not
use XML for Config_Files. It just doesn't meet the requirements;
hand-editing is important, and too hard in XML.

> IOW, use this as an excuse to build an XML DOM & SAX. (I'd like to
> see an OO implementation of this using tagged records and closely
> following the W3C DOM model, despite the very nice non-OO
> implementation done by XML/Ada.)

I don't think we need an excuse for that :).

-- 
-- Stephe

Re: Config_Files proposal

[Ted Dennison]

"Marin David Condic" <dont.bother.mcondic.auntie.spam@[acm.org> wrote in message news:<aeqcl6$sgg$1@nh.pace.co.uk>...
> All other things being equal, gaining an XML parser in the process would be
> A Good Thing. Imagine if one day when everything is sunshine and rainbows

Won't happen. At best you'd get something that can only parse the
limited syntax that the config file happens to use.

If folks really want an XML parser in the language, that should be a
separate effort. Perhaps the one Gnat developed could be used, now
that the license has changed?

If not, I don't see any hope of putting XML in the language either.
The effort would just be too much. XML/Ada hasn't gotten to 1.0 yet
after more than a year of effort, and I believe that effort has been
supported by ACT (or ACT-Europe?)

Re: Config_Files proposal

[Stephen Leake]

dennison@telepath.com (Ted Dennison) writes:

> I hate to do this, but I have to go on record as being strongly
> against voting for things. Everything in Grace needs to be decided by
> general consensus. The only reason there isn't an official policy on
> this at the Grace website is because I haven't had time to put it up.
> 
> I'm not saying everyone has to agree that one of these styles is the
> best of all possible styles. But there should be some kind of general
> agreement that one of them is meritous and acceptable, if not
> everyone's perfect option. If we get down to two that seem to be
> equally good with no huge contraversy between them, then perhaps a
> vote or coin flip or executive decision or something similar would be
> in order.

I think that's where we are. I guess I could have said "I'd like to
reach consensus on this". But I don't know how to actually do that in
a newsgroup forum. In person, I can go around the room and ask "do you
have anything else to add".

Asking for a vote is pretty simple, and most people know what it means.

> BTW: I have recently appointed Stephen to lead the config file
> effort for Grace (see http://savannah.gnu.org/projects/grace/ ). So
> if there were an "executive decision" to be made in the context of
> the Grace project, it would be up to him to make it.

Yes, I guess I never said that here. Thanks.

> This is a good example of why we need to go with this approach. We
> may now have consensus against using XML, but there really hasn't
> been much discussion here on the relative merits of the other
> approaches (other than that some folks think INI's should be reseved
> for Windows). If I were to just vote now between the other 3
> options, it would be based on looks. Another thread here has
> recently illustrated quite well that this would be quite succeptable
> to parrochial biases. :-)

Ok, I'll accept that.

> So really a better question right now would be to ask if anyone has
> any big objections to any of the approaches presented (other than
> XML, to which the objections are already well recorded). Also, does
> anyone have any strong reasons why they think one is much better
> than the others?

Ok. Good question. I guess I'm learning how to do consensus on a
newsgroup.

I went with the choice between Java and XML because there was
definitely consensus on multi-level keys (rules out ini), and I felt
there was sufficient complaint about "complex syntax" to rule out X
style.

Note that I did ask people to vote for any of the file formats.

If we end up with an overwhelming majority for one format, I'll take
that as consensus. If it's close, or there are only a few votes, I'll
be back :).

-- 
-- Stephe

Re: Config_Files proposal

[Ted Dennison]

Stephen Leake wrote:
> Ok. Good question. I guess I'm learning how to do consensus on a
> newsgroup.

With the Lists stuff, often I would just pick a choice, start arguing 
its merits, and wait to see if anyone screamed. If no one complains: 
bingo! :-)

You should probably at least post the merits and drawback of each 
approach, as you see it. Presumably others will do the same if they 
think you missed something.

> If we end up with an overwhelming majority for one format, I'll take
> that as consensus. If it's close, or there are only a few votes, I'll
> be back :).

The problem with a vote is there is no real weight on it, and no good 
idea of the rationale behind the numbers. The nightmare scenario is that 
the majority was just going with the flow to get things moving, while 
the minority voting against it had a damn good reason.

The other issue with getting consensus is that we want to make sure we 
don't leave %49 or %32 of everyone behind every time we make a decision. 
We need as near to a universal buy-in as we can get.

Re: Config_Files proposal

[Stephen Leake]

Ted Dennison <dennison@telepath.com> writes:

> Stephen Leake wrote:
> > Ok. Good question. I guess I'm learning how to do consensus on a
> > newsgroup.
> 
> With the Lists stuff, often I would just pick a choice, start arguing
> its merits, and wait to see if anyone screamed. If no one complains:
> bingo! :-)

That approach does have a certain appeal.

> You should probably at least post the merits and drawback of each
> approach, as you see it. Presumably others will do the same if they
> think you missed something.

Hmm. I did that on the web page. By "post", do you mean here on
comp.lang.ada? 

> > If we end up with an overwhelming majority for one format, I'll
> > take that as consensus. If it's close, or there are only a few
> > votes, I'll be back :).
> 
> The problem with a vote is there is no real weight on it, and no good
> idea of the rationale behind the numbers. The nightmare scenario is
> that the majority was just going with the flow to get things moving,
> while the minority voting against it had a damn good reason.

Yes. However, that's why I posted the rationale (or at least the start
of one). I hoped that the ensuing discussion would bring out all the
"good reasons". I wanted a vote to put a specific end to the
discussion; sometimes that motivates people to respond. But I could
just as easily have said "I will pick a format on Tuesday; post your
comments by then".

> The other issue with getting consensus is that we want to make sure
> we don't leave %49 or %32 of everyone behind every time we make a
> decision. We need as near to a universal buy-in as we can get.

Well, we could define a vote to be successful only if 80% agree.
Either way, we are just counting people's opinions.

My impression so far is that there is strong sentiment against XML as
being too complex to hand-edit. And there is strong sentiment against
ini because it doesn't have multi-level keys. Other than that, nobody
cares much. So it probably is easier for me to say "I'm picking Java
property format, unless anyone has serious objections".

That also avoids the issue of whether we get a quorum in the voting
process. 

Thanks for putting up with my learning curve :).

-- 
-- Stephe

Re: Config_Files proposal

[Jacob Sparre Andersen]

Stephen Leake wrote:

> Ted Dennison <dennison@telepath.com> writes:

> > You should probably at least post the merits and drawback of each
> > approach, as you see it. Presumably others will do the same if they
> > think you missed something.
> 
> Hmm. I did that on the web page. By "post", do you mean here on
> comp.lang.ada?

I don't know what _Ted_ meant, but it is my experience that
you get much more discussion going, if you post the text on
the newsgroup, so it is trivial for people to quote and
comment it.

Jacob
-- 
"Navier-Stokes-ligningen er som Biblen,
 det er randbetingelserne der bestemmer det hele."

Re: Config_Files proposal

[Stephen Leake]

Jacob Sparre Andersen <sparre@nbi.dk> writes:

> Stephen Leake wrote:
> 
> > Ted Dennison <dennison@telepath.com> writes:
> 
> > > You should probably at least post the merits and drawback of each
> > > approach, as you see it. Presumably others will do the same if they
> > > think you missed something.
> > 
> > Hmm. I did that on the web page. By "post", do you mean here on
> > comp.lang.ada?
> 
> I don't know what _Ted_ meant, but it is my experience that
> you get much more discussion going, if you post the text on
> the newsgroup, so it is trivial for people to quote and
> comment it.

Ok. But when the text gets long enough, I find it more awkward. Maybe
my threshold for "long enough" is too low :).

-- 
-- Stephe

Re: Config_Files proposal

[Ted Dennison]

Jacob Sparre Andersen wrote:
> Stephen Leake wrote:
> 
>>Ted Dennison <dennison@telepath.com> writes:
>>>You should probably at least post the merits and drawback of each
>>>approach, as you see it. Presumably others will do the same if they
>>>think you missed something.
>>
>>Hmm. I did that on the web page. By "post", do you mean here on
>>comp.lang.ada?
> 
> I don't know what _Ted_ meant, but it is my experience that
> you get much more discussion going, if you post the text on
> the newsgroup, so it is trivial for people to quote and
> comment it.


Quite true. However, I'll admit I missed that on the website, which was 
my bad. After looking it over more carefully, I also have to admit I 
have nothing to add to it (which I guess leads me towards the same 
conclusion Stephen has been getting to).

Re: Config_Files proposal

[Stephen Leake]

Ted Dennison <dennison@telepath.com> writes:

> Jacob Sparre Andersen wrote:
> > Stephen Leake wrote:
> >
> >>Ted Dennison <dennison@telepath.com> writes:
> >>>You should probably at least post the merits and drawback of each
> >>>approach, as you see it. Presumably others will do the same if they
> >>>think you missed something.
> >>
> >>Hmm. I did that on the web page. By "post", do you mean here on
> >>comp.lang.ada?
> > I don't know what _Ted_ meant, but it is my experience that
> > you get much more discussion going, if you post the text on
> > the newsgroup, so it is trivial for people to quote and
> > comment it.
> 
> 
> Quite true. However, I'll admit I missed that on the website, which
> was my bad. 

Or mine, for making a less-than-wizzy website :). I also realized that
I have not been advertising the web url very often, so here it is
again:

http://users.erols.com/leakstan/Stephe/Ada/Config_Files/config_files.html

> After looking it over more carefully, I also have to admit I have
> nothing to add to it (which I guess leads me towards the same
> conclusion Stephen has been getting to).

Ok, making progress :).

-- 
-- Stephe

Re: Config_Files proposal

[Ted Dennison]

Stephen Leake wrote:

> I went with the choice between Java and XML because there was
> definitely consensus on multi-level keys (rules out ini), and I felt
> there was sufficient complaint about "complex syntax" to rule out X
> style.

That's a good point. The X style does look like a real pain to parse. 
It'd probably be nearly as tough as the XML style to explain over the 
phone too.

Re: Config_Files proposal {long}

[Bobby D. Bryant]

On Tue, 18 Jun 2002 11:07:08 -0600, Stephen Leake wrote:

> I've posted another example spec and implementation, at
> 
> http://users.erols.com/leakstan/Stephe/Ada/Config_Files/config_files.html

I'm sorry that I haven't had time to follow these discussions
carefully, but I'd still like to offer some thoughts on it.  (Please
pardon anything that has already been discussed.)


"11. Additional files may be opened for read-only simultaneously in
one Config_File object, using an append-read-only operation. Keys
are searched for first in the writeable file, then in the additional
read-only files. Keys that are created or modified are written to
the writeable file when flushed."

I'm not sure I understand that.  If it's saying "allow a config file
to chain in other config files", great.  If it's saying something
else, then please add what I just said to your list of requirements.


"22. Provide a way to read and write opaque binary values (ie
bitmaps for icons)."

Do you really want binary data in-line with text data?  I think a
better solution would be for the config file to just give the
filename for binary data, so that after fetching the filename from
the config file the program would use special-purpose library
functions to read/write binary data.

People are going to need to specify filenames in config files from
time to time anyway, so what I'm suggesting won't require any new
mechanisms, and it would let you completely off-load responsibility
for arbitrary binary formats.


"23. Provide a way to write comments. Comments are associated with a
key (possibly a hierarchy level), and are preserved thru open and
flush. Comments are intended to guide manually editing the file."

I agree with this, but I want to point out that it is _extremely_
problematic.  If you support comments then when people hand-edit a
config file they will occasionaly (and rightly) add a comment
explaining why that value is there.  But if a program later changes
that value, how is it to know when the comment has been vitiated?


Add:

I would like to see direct, high-level support for loading/saving 2-D
tables, with the tables laid out in 2-D fashion (with row and column
headers) in the config file.  For instance, suppose the program in
question is a wargame.  Most such games require a table of Unit_Type
x Terrain_Type --> Movement_Cost (and many other 2-D tables as well),
and it would make life very easy for the scenario designer if s/he
could simply type the data into the config file _as_a_2-D_table_.

Also, it would be nice to support a certain type of dynamism in such
tables.  To continue the wargame example, it is becoming
increasingly popular to design games that are just minimalist
engines driven by externally specified data.  So the scenario
designer might wish to enumerate the unit and terrain types in the
config file and then follow them with the table of movement costs.
It would be a very useful feature, IMO, if the table-loading were
able to do some sanity checking as it loaded the table.  (Minimally,
ensure that the table has the correct dimensions; optimally, have
it actually verify that the row and column headers have the correct
values; ideally, by direct reference to the values of the defining
fields elsewhere in the config file.)

Higher-dimensional tables are sometimes needed as well, though
seemingly less often.  They are also somewhat difficult to
represent in a text file.  So I would suggest that
higher-dimensional tables be built up by "stacking"
lower-dimensional tables appearing iteratively in the config file.
(The same logic could of course be applied to the construction of
2-D tables, but they are a frequent and natural data type for
humans, and they _can_ be specified in tabular fashion in a text
file, so I think it would make good sense to support them directly
with a high-level API.)


Other food for thought:

I think I mentioned a couple of months ago that I am using GUILE
(Scheme) for my config files.  I won't go so far as recommend
adopting that as a standard, but I would like mention a couple of
things about it to provoke thought.

First off, compare the Scheme-like syntax to XML.  Here's the
example from your page:

<?xml version="1.0"?> <Config>
  <Numeric>
    <Interfaces>
      <C>
        <An_Unsigned> 124076833</An_Unsigned>
        <An_Int> 2</An_Int>
      </C>
    </Interfaces>
    <Float>
      <A_Float> 3.14159E+00</A_Float>
    </Float>
  </Numeric>
  <Strings>
    <Quoted>  he said &quot;hi there &amp; goodbye&quot; </Quoted>
    <Violins>Stradivarious</Violins>
  </Strings>
</Config>

And here's how the same thing would look in one of my config files
right now:

(configuration
  (numeric
    (interfaces
      (C
        (An_Unsigned 124076833)
        (An_Int              2)
      )
    )
    (Float
      (A_Float 3.14159E+00)
    )
  )
  (strings
    (Quoted  he said "hi there & goodbye")
    (Violins Stradivarious)
  )
)

The latter is, IMO, *much* easier to read and comprehend.  It's
also about (guessingly) 40% smaller.  While I don't advocate
smallness for smallness' sake (and I think that "bloat" is usually
merely a slur that people invoke against software systems that they
don't like but can't come up with any cogent criticism of), the
lean syntax can be very important in config files because it
promotes readability -- partly through reduced clutter, and partly
because there will be fewer times when non-semantic line wraps are
required.

[Per my example, an actual Scheme representation for the "Quoted"
field would need to be expressed somewhat differently; I gave
instead what I think it would look like if you adopted the
parentheses syntax for what you're trying to do.  The only thing
that would need special treatment would be parentheses -- in fact,
only close-parentheses in data that did not also include
matching open parentheses.]

The only disadvantage I can think of is that the lack of labeled
end markers makes it hard to see where very long lists end.  When
this becomes a problem for me I merely address it with a cosmetic
comment, thus:

...
   (Floats
      (F1 1.2)
      (F2 2.3)
      (F3 4.2)
      (F4 1.9)
       ...
      (F468 7.2)
      (Comment: End of "Floats" section.)
   )
...


As a side note, notice also that using Scheme for config files
means that there is no formal distinction between config files and
script files, since code and data are represented the same way in
Scheme.  However, this requres a run-time system such as GUILE to
make it work, and is thus far out of the scope of what you are
trying to do.  OTOH, if you *are* thinking about support for
scripting sometime in the future, give the mechanism some thought
now.  I find it extremely convenient to include certain kinds of
code in my config files, thus:

...
   (Penalty_Function (log n))
...


Finally, if I may say so without stepping on any toes, I would like
to call attention to the curiously "flat" examples that you have
for the Java and XML sections on your Web page, and the emphasis on
data types rather than on semantics.  For comparison, here is what
one of my real config files looks like (substantially reduced and
re-commented for the purposes of this post) -

(configuration

 (Comment:      The next three values are symbolic names that will be
                used as keys for lookups in 1-D tables further down.)
 (use-problem             legion-i)
 (use-workers             first-five)
 (use-solution-strategy   fixed-size-250)

 (worker-plot-colors
  "RGB:FF/00/00"
  "RGB:00/BF/00"
  "RGB:00/00/FF"
  )

 (problems
  (Comment:     "problems" is a list of problem-specific data, to
                be selected by the "use-probem" key defined above.
                I.e., the program grabs "use-problem" and extracts
                its value, then grabs "problems" and uses the
                previously obtained value of "use-problem" to
                fetch the proper record out of "problems".
                Due to the nature of the application, the data
                varies from problem to problem, and the program
                must decide what fields to ask for based on the
                value of the key.  I.e., if "use-problem" is
                "phalanx-1" it will not ask for "barbarian-rate".
                That is hard-coded behavior embedded in a case
                statement in the application program, and I don't
                see any other obvious way to do it.)
  (legion-i
   (Comment:    "use-map" is a key that will be looked up in a
                table below.)
   (use-map            21x21+3cities)
   (number-of-legions        5)
   (barbarian-rate         1.0)
   (game-length            200)
   (movement-granularity  10.0)
   (games-per-generation     3)
   (population-size        500)
   )
  (phalanx-1
   (use-map           41x31+5cities)
   (units-per-side         24)
   (turns-per-game         30)
   (games-per-generation    5)
   (population-size        10)
   )
  )

 (maps
  (Comment:       "maps" is another lookup table.  The key is
                  obtained from the problem definition, above.)
  (21x21+3cities
   (map-width              21)
   (map-height             21)
   (number-of-cities        3)
   )
  (41x31+5cities
   (map-width              41)
   (map-height             31)
   (number-of-cities        5)
   )
  )

 (worker-configurations
  (Comment:       "worker-configurations" is another lookup table.)
  (first-five
   (Comment:      Notice that "worker-random-seeds" is a list that
                  my program reads into an array.  Currently I do
                  this by iterating within the program, but it
                  would be nice if the config-file parser did that
                  for me.)
   (worker-random-seeds 1 2 3 4 5)
   (minimum-workers-required-to-start 1)
   )
  (three-require-three
   (worker-random-seeds 1 2 3)
   (minimum-workers-required-to-start 3)
   )
  )

 (solution-strategies
  (quick-test
    (hof-size        20)
    (children-each   20)
    (min-initial-hidden-units  1)
    (epochs
     (epoch
      (generations             2)
      (weight-mutation-rate  .01)
      (size-penalty         none)
      )
     )
    )
  (fixed-size-250
    (hof-size         50)
    (children-each    10)
    (min-initial-hidden-units  2)
    (epochs
     (Comment:     Very importantly, notice that this is a
                   *list* of records of the same type.  The
                   config-file parser must not simply grab
                   the first and ignore the rest; rather,
                   it must give them to me sequentially on
                   demand.)
     (epoch
      (generations            50)
      (weight-mutation-rate  .01)
      (size-penalty         none)
      )
     (epoch
      (generations           100)
      (weight-mutation-rate  .05)
      (size-penalty         none)
      )
     (epoch
      (generations           100)
      (weight-mutation-rate  .10)
      (size-penalty         none)
      )
     )
    )
  )

)

In addition to the raw GUILE bindings I have a package of
higher-level support routines that makes life easier for
the programmer.  Included are a "Lookup" function and an
overloaded-by-return-type "Second" function for extracting the
values.  For example, after appropriate variable declarations
and with the config file already loaded but not processed, my
code has things like this:

	Problem := Lookup("use-problem", Config);
        Map     := Lookup("use-map", Problem);
	X       := Second(Lookup("map-width", Map));


Please look at some of the things I'm doing in the example, and
consider how you would be able to do it (and what it would look
like) with the various proposed syntaxes.

FWIW, if I were doing it myself I would probably start with an
X-like syntax and remove the need for "end" markers by making the
indentation semantically significant.  I used GUILE instead because
it saved me writing my own parser (and because I may need the
scripting capabilities later), but the GUILE/Scheme syntax, when
pretty-printed, is very similar to X-like with semantic
indentations.  To repeat your XML example yet again, you would get
something like:

configuration
  numeric
    interfaces
      C
        An_Unsigned 124076833
        An_Int              2
    Float
      A_Float 3.14159E+00
  strings
    Quoted  he said "hi there & goodbye"
    Violins Stradivarious

Most human-friendly of all, IMO, but possibly prone to error, and
surely difficult for syntax-highlighting editors unless you chose
to limit the field names to a predefined set.


Sorry for the length; hopefully someone will find an interesting
thing or two somewhere in there.

Bobby Bryant
Austin, Texas

Re: Config_Files proposal {long}

[Bobby D. Bryant]

On Sat, 29 Jun 2002 05:03:17 -0600, Bobby D. Bryant wrote:

> Finally, if I may say so without stepping on any toes, I would
> like to call attention to the curiously "flat" examples that you
> have for the Java and XML sections on your Web page, and the
> emphasis on data types rather than on semantics.

After taking my morning walk I think I can elaborate on this
usefully.

First, regarding the "flatness" of some of the examples.  Notice
that the apparent depth of the Java and XML examples arises from the
fact that you have given the path names through the Ada package
hierarchy to the Ada type definitions as "structure" in the config
file.  But if you replaced those path names with aliases and then
used those aliases in your config file, it would immediately become
apparent that in those two examples the data is just a flat list of
{type, name, value} tuples (with a huge pile of useless syntax
thrown in for the XML implementation, to boot).

Compare that to the X example, where the nesting in the config file
actually represents some structure in the data.

Sure, you *could* represent the X data as a flat list, but
presumably it is intended to be isomorphic to some nested data
structure that has nested meaning in the actual program.  In fact
Ada will let you nest record structures to an (ideally) unbounded
depth, and in practice I do find myself going several layers deep
from time to time.  And for ease of comprehension I think it
important to represent nested structures as nested structures in the
config file.

Accordingly, please consider changing -

"15. At least one layer of key hierarchy is supported."

with -

"15. At unbounded depth of key hierarchy is supported."



Also, if you want people to look at the examples on your Web page to
help them decide which syntactic system is best for the job, then I
would think that the following would be absolutely essential -

(a) Level the playing field:  Each example needs to specify
identical data.

(b) Raise the bar:  For the examples, use the most demanding example
rather than a simpler one.   (Solicit to see what people feel that
config files must support, and use that feedback to construct a
'wicked' example.)

I suspect you will find that if you level the playing field and
raise the bar, some suggested syntaxes will rule themselves out
immediately.  Also, by working with a 'wicked' example you can
ensure that we're not here revisiting the syntax question again in
2-3 years when people discover that the standard only supports the
most rudimentary data representation problems and does not actually
meet the requirements of complex programs.


Notice that I'm not commenting on the specification of data types in
the config file.  If that's necessary or desirable then it should be
done, but it should be an "also" added to the structured
representation of structured data, not an "instead of".  The key,
IMO, is to get the representation of the structured data right,
after which type-tagging should be a relatively minor detail.

When I say "structured data" I mean, minimally, support for records,
iterative structures (arrays or lists, possibly with identical
representation in the config file), 2-D tables, etc., including
unbounded nesting of structures.

later,

Bobby Bryant
Austin, Texas

p.s. - Wandering yet farther from my main point...  For arrays it
would be nice if the API had a function call that would let your
proram peek to see how long an array was in the config file,
so that you could peek, allocate the array, and then load it, with
two calls to the API and a "new" statement in between.  For lists,
there needs to be some mechanism for easily saying "OK, now give me
the next one."

Re: Config_Files proposal {long}

[Mark Biggar]

"Bobby D. Bryant" wrote:

> I would like to see direct, high-level support for loading/saving 2-D
> tables, with the tables laid out in 2-D fashion (with row and column
> headers) in the config file.  For instance, suppose the program in
> question is a wargame.  Most such games require a table of Unit_Type
> x Terrain_Type --> Movement_Cost (and many other 2-D tables as well),
> and it would make life very easy for the scenario designer if s/he
> could simply type the data into the config file _as_a_2-D_table_.

This seems to be a specialized need that should be handled like
binary data: have a file-name in the configfile.

--
Mark Biggar
mark.a.biggar@attbi.com

Re: Config_Files proposal {long}

[Georg Bauhaus]

Bobby D. Bryant <bdbryant@mail.utexas.edu> wrote:
[Scheme vs XML]
: 
: The latter is, IMO, *much* easier to read and comprehend.  It's
: also about (guessingly) 40% smaller. ...
:  the
: lean syntax can be very important in config files because it
: promotes readability -- partly through reduced clutter, and partly
: because there will be fewer times when non-semantic line wraps are
: required.

All this is usually addressed in good DTD design (and you will
be pleased to hear, if you haven't yet, that the ISO standard
accompanying SGML, DSSSL, is actually describing a specialised
Scheme).

It is a sad mistake that people don't see tags as carrying essential
meaning (like your identifiers in function name position do),
and not syntax clutter.
Tags and Attributes as important means of adding explicit additional
infomration about the structure of the document, mutual relations,
data types, restrictions, constraints....

As long as we will see uninformed non-sensical abuse like I have
to from time to time:

<ingredient>
<number>1</number><colon/>
 <type>flour<variant>wheat</variant></type>
 <amount>125<unit>g</unit></amount>
</ingredient>

there will be a lot to criticize about the "features" of XML.
It could be written, not loosing anything:

<ingredient amount="125" unit="g">
 <flour type="wheat"/>
</ingredient>

Moreover, something makes me start thinking there is a general dislike
for anything but parentheses in some parts of the Ada community :->
Anything but dots and parens and semicola is called clutter?

: The only disadvantage I can think of is that the lack of labeled
: end markers makes it hard to see where very long lists end.

Add to this the varying skill of customers at using Lisp-syntax aware
editors...

:  cosmetic
: comment, thus:
: ...
:   (Floats
:      (F1 1.2)
:      (F2 2.3)
:      (F3 4.2)
:      (F4 1.9)
:       ...
:      (F468 7.2)
:      (Comment: End of "Floats" section.)
:   )

Are these cosmetic comments easily checked to belong to where
they should belong? Same for ()?


Really, these SGML/XML guys have spent more than 30 years on how to
add marking syntax to data carrying document text--I think
they have done something that should at least be considered a
valuable source of information. At least in this corner of config
file discussions. Of course you can't compute with XML like you
can with Scheme.

If you look closely, you will note that, to Lisp programmers,
there is a familiar nesting in SGML/XML :-)

<Floats desc="whatever these numbers mean...">
 <F id="north" v="1.2"/>
 <F id="west"  v="2.3"/>
 <F id="south" v="4.2"/>
 <F id="east"  v="1.9"/>
 <!-- ... -->
</Float>


-- Georg