OpenToken 6.0 released

OpenToken 6.0 released

[Stephen Leake]

OpenToken 6.0 is now available on the website:
http://stephe-leake.org/ada/opentoken.html

See http://stephe-leake.org/ada/opentoken.html#History for a description
of the changes.

The main change is support of generalized LALR (spawn parallel parsers)
to handle conflicts in the Ada parser runtime (previous releases only
supported this in the parse table generator; the runtime was provided
only in Emacs elisp).

The API of the OpenToken packages changes significantly, due to
reorganizing/cleaning up to support generalized LALR. So existing
projects will have to be edited. The changes are only in the
instantiations, not in user code logic.

--
-- Stephe

Re: OpenToken 6.0 released

[Shark8]

I am quite happy to hear this.
Thank you for your work on OpenToken.

Re: OpenToken 6.0 released

[Shark8]

> html_lexer-tags.ads:39:121: (style) this line is too long

Fixed by inserting CR on the commas:
pragma Elaborate_All (
			OpenToken.Recognizer.Character_Set,
			OpenToken.Recognizer.End_Of_File,
			OpenToken.Recognizer.Keyword,
			OpenToken.Recognizer.Nothing,
			OpenToken.Recognizer.Separator,
			OpenToken.Recognizer.String
		);

> opentoken-token-enumerated-list-print.ads:27:41: "Print" conflicts with declaration at opentoken-token-enumerated-list.ads:119

Solved by renaming Print line #119 (adb #229) it Print_Item, altering:
opentoken-production-print.adb line #40.
opentoken-production-parser-lalr-parser_lists.adb line #724.

After that the gprbuild and gprinstall seem to have gone on w/o incident.

Re: OpenToken 6.0 released

[AdaMagica]

Am Freitag, 17. April 2015 00:55:47 UTC+2 schrieb Shark8:
> > html_lexer-tags.ads:39:121: (style) this line is too long
> 
> Fixed by inserting CR on the commas:
> pragma Elaborate_All (
> 			OpenToken.Recognizer.Character_Set,
> 			OpenToken.Recognizer.End_Of_File,
> 			OpenToken.Recognizer.Keyword,
> 			OpenToken.Recognizer.Nothing,
> 			OpenToken.Recognizer.Separator,
> 			OpenToken.Recognizer.String
> 		);

Just a proposal: Instead of having Elaborate_All at any and all units withing another unit X, put an Elaborate_Body on specification of X (if that is possible; for the recognizers, this should generally be feasible). Thus you are sure you'll never get elaboration problems.

This is IMHO preferable.

Re: OpenToken 6.0 released

[Stephen Leake]

Shark8 <onewingedshark@gmail.com> writes:

>> html_lexer-tags.ads:39:121: (style) this line is too long

I don't get this in my testing.

What compile command did you use?

>> opentoken-token-enumerated-list-print.ads:27:41: "Print" conflicts
>> with declaration at opentoken-token-enumerated-list.ads:119

This package should have been deleted; it was merged into
enumerated-list. Sorry about that.

Again, I don't get this error in my testing.

-- 
-- Stephe

Re: OpenToken 6.0 released

[AdaMagica]

Am Mittwoch, 15. April 2015 10:09:39 UTC+2 schrieb Stephen Leake:
> OpenToken 6.0 is now available on the website:
> http://stephe-leake.org/ada/opentoken.html

I see a problem with the Ada Lexer:

  Comment_T => Tokenizer.Get (OpenToken.Recognizer.Line_Comment.Get (" --")),

A comment starts with "--", not with a blank prefixed.

Re: OpenToken 6.0 released

[AdaMagica]

Am Dienstag, 21. April 2015 19:26:06 UTC+2 schrieb AdaMagica:
> Am Mittwoch, 15. April 2015 10:09:39 UTC+2 schrieb Stephen Leake:
> > OpenToken 6.0 is now available on the website:
> > http://stephe-leake.org/ada/opentoken.html
> 
> I see a problem with the Ada Lexer:
> 
>   Comment_T => Tokenizer.Get (OpenToken.Recognizer.Line_Comment.Get (" --")),
> 
> A comment starts with "--", not with a blank prefixed.

This bug was not present in OpenToken 4.0b.

Re: OpenToken 6.0 released

[Jacob Sparre Andersen]

AdaMagica <christ-usch.grein@t-online.de> writes:

> This bug was not present in OpenToken 4.0b.

Isn't there a publicly visible version control repository for OpenToken
somewhere?

Greetings,

Jacob
-- 
"... while the C compiler will happily generate code for
 almost anything produced by leaning on the keyboard."

Re: OpenToken 6.0 released

[jan.de.kruyf]

Stephe, hallo.

Thanks for all the good work, and here is some small inconsistency in the makefile:

org.opentoken-6.0a/build/release/Install.make 

line 30:
	cp lib/libopentoken* $(I_BIN)

I am puzzled why the lib files should be copied to the bin directory
all the more since you dont check that a bin directory is created, and it also does not get cleaned up in the 
install-clean :
rule.

Cheers

j.

Re: OpenToken 6.0 released

[Shark8]

On Monday, April 20, 2015 at 9:50:53 AM UTC-6, Stephen Leake wrote:
> Shark8 writes:
> 
> >> html_lexer-tags.ads:39:121: (style) this line is too long
> 
> I don't get this in my testing.
> 
> What compile command did you use?

Just gprbuild, so it should be using the options in the GPR file.
 
> >> opentoken-token-enumerated-list-print.ads:27:41: "Print" conflicts
> >> with declaration at opentoken-token-enumerated-list.ads:119
> 
> This package should have been deleted; it was merged into
> enumerated-list. Sorry about that.
> 
> Again, I don't get this error in my testing.

That's odd.
Perhaps AdaCore altered the name-clash detection algorithm.

Re: OpenToken 6.0 released

[Stephen Leake]

Jacob Sparre Andersen <jacob@jacob-sparre.dk> writes:

> AdaMagica <christ-usch.grein@t-online.de> writes:
>
>> This bug was not present in OpenToken 4.0b.
>
> Isn't there a publicly visible version control repository for OpenToken
> somewhere?

as it says on http://stephe-leake.org/ada/opentoken.html, monotone
server www.ada-france.org, branch org.opentoken.  

-- 
-- Stephe

Re: OpenToken 6.0 released

[Stephen Leake]

jan.de.kruyf@gmail.com writes:

> Stephe, hallo.
>
> Thanks for all the good work, and here is some small inconsistency in the makefile:
>
> org.opentoken-6.0a/build/release/Install.make 
>
> line 30:
> 	cp lib/libopentoken* $(I_BIN)
>
> I am puzzled why the lib files should be copied to the bin directory

The only file that matches that pattern is libopentoken.dll, which needs
to be in the bin directory on Windows (as the comment says).

Patches for linux welcome, but my general approach is to leave
installation to the platform installer packages.

> all the more since you dont check that a bin directory is created, 

It's computed starting on line 6:

INSTALL_DIR ?= $(dir $(shell which gnatls))..
prefix=$(INSTALL_DIR)
I_BIN = $(prefix)/bin

so it has to exist if you can compile OpenToken

> and it also does not get cleaned up in the install-clean : rule.

That is an oversight.

-- 
-- Stephe

Re: OpenToken 6.0 released

[jan.de.kruyf]

Yes ok.
I appreciate your reasoning, and it probably in general will do no damage, but although you compute the bin directory, on a private installation Opentoken does not compile since the bin directory does not exist. If it is installed in the gnat directory it would compile ofcourse since the bin directory pre-exists.

cheers,

j.

Re: OpenToken 6.0 released

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> OpenToken 6.0 is now available on the website:
> http://stephe-leake.org/ada/opentoken.html

Several people have reported minor bugs; I'll do a bug fix release soon.
So if you have found any problems that you have not reported yet, please
report them.

I didn't realize there was so much interest in OpenToken!

-- 
-- Stephe

Using Monotone to access the OpenToken sources (Was: OpenToken 6.0 released)

[Jacob Sparre Andersen]

Stephen Leake wrote:

> as it says on http://stephe-leake.org/ada/opentoken.html, monotone
> server www.ada-france.org, branch org.opentoken.

It is probably just because I haven't used Monotone before, but when I
use the commands there I just get a single SQLite3 database file out of
it.

I've tried:

   mtn --db ~/.monotone/ada-france.db checkout --branch org.opentoken

but the result is just

   mtn: misuse: branch 'org.opentoken' is empty

Any idea what I've done wrong?  I can't see what I'm doing differently
from the various Monotone examples I've found.

Greetings,

Jacob
-- 
Who guards the guardians?

Re: OpenToken 6.0 released

[AdaMagica]

Am Freitag, 24. April 2015 15:30:10 UTC+2 schrieb Stephen Leake:
> Stephen Leake <stephen_leake@stephe-leake.org> writes:
> 
> > OpenToken 6.0 is now available on the website:
> > http://stephe-leake.org/ada/opentoken.html
> 
> Several people have reported minor bugs; I'll do a bug fix release soon.
> So if you have found any problems that you have not reported yet, please
> report them.

May I propose to add pragma Elaborate_Body to all Recognizer subpackages? This will prevent any elaboration problems with calling the Get procedures - otherwise you have to use Elaborate{_All} in every caller of Get.

(I see that there is no such pragma e.g. in the Ada Lexer, but I think it is only a coincidence that elaboration does not fail.)

Re: Using Monotone to access the OpenToken sources (Was: OpenToken 6.0 released)

[Qun-Ying]

Jacob Sparre Andersen wrote:
> Stephen Leake wrote:
>
> I've tried:
>
>     mtn --db ~/.monotone/ada-france.db checkout --branch org.opentoken
>
> but the result is just
>
>     mtn: misuse: branch 'org.opentoken' is empty
>
> Any idea what I've done wrong?  I can't see what I'm doing differently
> from the various Monotone examples I've found.
>
> Greetings,
>
> Jacob
>

You will need to use "org.opentoken.stephe" as the branch name.

Re: Using Monotone to access the OpenToken sources

[Stephen Leake]

Qun-Ying <zhu.qunying@gmail.com> writes:

> Jacob Sparre Andersen wrote:
>> Stephen Leake wrote:
>>
>> I've tried:
>>
>>     mtn --db ~/.monotone/ada-france.db checkout --branch org.opentoken
>>
>> but the result is just
>>
>>     mtn: misuse: branch 'org.opentoken' is empty
>>
>> Any idea what I've done wrong?  I can't see what I'm doing differently
>> from the various Monotone examples I've found.
>>
>> Greetings,
>>
>> Jacob
>>
>
> You will need to use "org.opentoken.stephe" as the branch name.

No, that's the development branch; org.opentoken is the release branch.

-- 
-- Stephe

Re: Using Monotone to access the OpenToken sources

[Stephen Leake]

Jacob Sparre Andersen <jacob@jacob-sparre.dk> writes:

> Stephen Leake wrote:
>
>> as it says on http://stephe-leake.org/ada/opentoken.html, monotone
>> server www.ada-france.org, branch org.opentoken.
>
> It is probably just because I haven't used Monotone before, but when I
> use the commands there I just get a single SQLite3 database file out of
> it.
>
> I've tried:
>
>    mtn --db ~/.monotone/ada-france.db checkout --branch org.opentoken
>
> but the result is just
>
>    mtn: misuse: branch 'org.opentoken' is empty
>
> Any idea what I've done wrong?  I can't see what I'm doing differently
> from the various Monotone examples I've found.

You must have a sync problem; possibly a default sync branch filter?
Make sure the filter covers "org.opentoken".

The branch filter is displayed when you execute "mtn sync"; it's set by
a "--set-default" option to the sync command.

-- 
-- Stephe

Re: OpenToken 6.0 released

[Stephen Leake]

jan.de.kruyf@gmail.com writes:

> but although you compute the bin directory, on a private
> installation Opentoken does not compile since the bin directory does
> not exist. 

By _definition_, "the bin directory" is "the directory containing
gnatls". So if you are compiling with GNAT, that directory exists.

If you are compiling with a different compiler, you have a problem. No
makefile can cover _all_ options; that's what target-specific installer
packages are for.

> If it is installed in the gnat directory it would compile ofcourse
> since the bin directory pre-exists.

I guess you want the option to install somewhere else? You can override
INSTALL_DIR on the make command line:

make INSTALL_DIR=<whatever> install


-- 
-- Stephe

Re: OpenToken 6.0 released

[Stephen Leake]

AdaMagica <christ-usch.grein@t-online.de> writes:

> Am Freitag, 24. April 2015 15:30:10 UTC+2 schrieb Stephen Leake:
>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>> 
>> > OpenToken 6.0 is now available on the website:
>> > http://stephe-leake.org/ada/opentoken.html
>> 
>> Several people have reported minor bugs; I'll do a bug fix release soon.
>> So if you have found any problems that you have not reported yet, please
>> report them.
>
> May I propose to add pragma Elaborate_Body to all Recognizer
> subpackages? This will prevent any elaboration problems with calling
> the Get procedures - otherwise you have to use Elaborate{_All} in
> every caller of Get.
>
> (I see that there is no such pragma e.g. in the Ada Lexer, but I think
> it is only a coincidence that elaboration does not fail.)

I've never had an elaboration problem when using the default GNAT
elaboration algorithm.

Are you using another compiler?

-- 
-- Stephe

Re: Using Monotone to access the OpenToken sources

[Simon Wright]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> You must have a sync problem; possibly a default sync branch filter?
> Make sure the filter covers "org.opentoken".

I think I must have had it set to "org.opentoken.*". Anyway, now using

  mtn sync \
    --db ada-france.db \
    www.ada-france.org \
    "org.emacs*" \
    "org.opentoken*"

and all is good.

> The branch filter is displayed when you execute "mtn sync"; it's set by
> a "--set-default" option to the sync command.

Using mtn 0.48 (the only one working on Mac OS X, last time I looked)
"mtn sync" doesn't display the default pattern, and I can't see any way
to find out what it's set to.

Re: OpenToken 6.0 released

[AdaMagica]

Am Samstag, 25. April 2015 15:31:42 UTC+2 schrieb Stephen Leake:
> AdaMagica <christ-usch.grein@t-online.de> writes:
> > May I propose to add pragma Elaborate_Body to all Recognizer
> > subpackages? This will prevent any elaboration problems with calling
> > the Get procedures - otherwise you have to use Elaborate{_All} in
> > every caller of Get.
> >
> > (I see that there is no such pragma e.g. in the Ada Lexer, but I think
> > it is only a coincidence that elaboration does not fail.)
> 
> I've never had an elaboration problem when using the default GNAT
> elaboration algorithm.
> Are you using another compiler?

Gnat in the standard mode implicitly uses an Elaborate_All, you have to use -gnatE (IIRC) to get Ada RM behaviour.

Thus other compilers *might* generate elaboration problems. Elaborate_Body (if possible; under certain circumstances, it cannot be used) prevents any problems.

The Recognizers, those that I checked, should allow El_Bd. Your test procedures should show where the pragma cannot be used.

Re: OpenToken 6.0 released

[Stephen Leake]

AdaMagica <christ-usch.grein@t-online.de> writes:

> Am Samstag, 25. April 2015 15:31:42 UTC+2 schrieb Stephen Leake:
>> AdaMagica <christ-usch.grein@t-online.de> writes:
>> > May I propose to add pragma Elaborate_Body to all Recognizer
>> > subpackages? This will prevent any elaboration problems with calling
>> > the Get procedures - otherwise you have to use Elaborate{_All} in
>> > every caller of Get.
>> >
>> > (I see that there is no such pragma e.g. in the Ada Lexer, but I think
>> > it is only a coincidence that elaboration does not fail.)
>> 
>> I've never had an elaboration problem when using the default GNAT
>> elaboration algorithm.
>> Are you using another compiler?
>
> Gnat in the standard mode implicitly uses an Elaborate_All, you have
> to use -gnatE (IIRC) to get Ada RM behaviour.
>
> Thus other compilers *might* generate elaboration problems.
> Elaborate_Body (if possible; under certain circumstances, it cannot be
> used) prevents any problems.

I gather the answer to my question is "no, I'm not using another
compiler, but someone else might".

Ok; I'll wait until someone reports an actual problem. I'm not getting
paid for this!

-- 
-- Stephe

Re: Using Monotone to access the OpenToken sources

[Qun-Ying]

Stephen Leake wrote:
>>
>> You will need to use "org.opentoken.stephe" as the branch name.
>
> No, that's the development branch; org.opentoken is the release branch.
>
Oh, I got the wrong impression from your page 
http://www.stephe-leake.org/ada-france-access.html.

"org.opentoken.stephe
     OpenToken extended to produce Elisp and Ada source for generalized 
LALR(1) parser, used by wisi indentation engine."

It would be better to make a change on the web site to point to the 
right branch name.

Re: Using Monotone to access the OpenToken sources

[Stephen Leake]

Qun-Ying <zhu.qunying@gmail.com> writes:

> Stephen Leake wrote:
>>>
>>> You will need to use "org.opentoken.stephe" as the branch name.
>>
>> No, that's the development branch; org.opentoken is the release branch.
>>
> Oh, I got the wrong impression from your page
> http://www.stephe-leake.org/ada-france-access.html.
>
> "org.opentoken.stephe
>     OpenToken extended to produce Elisp and Ada source for generalized
> LALR(1) parser, used by wisi indentation engine."
>
> It would be better to make a change on the web site to point to the
> right branch name.

Thanks, I forgot to update that page. Fixed now.

-- 
-- Stephe

Re: OpenToken 6.0 released

[AdaMagica]

Am Montag, 27. April 2015 18:43:28 UTC+2 schrieb Stephen Leake:
> Ok; I'll wait until someone reports an actual problem. I'm not getting
> paid for this!

I know - I wasn't paid either when I added a lot of recognizers and the sample lexers.

Thanks for the work.