about OpenToken

about OpenToken

[Lionel Draghi]

Is 3.0b the latest available OpenToken version?
I'am not so sure, because some of the examples don't compile
using gnat 4.0.3 (Debian 4.0.2-4).

Here is my problem : I am implementing a recursive descent parser, and 
my grammar contains optionnals keywords, that will be ignored.
Here is an exemple:
- do this and this
- do this and also this
"also" is an optionnal keyword.

I am wondering what is the best way to deal with this.
What comes to my mind is to create a selection (In OpenToken parlance) 
between the keyword "also" and a null token, child of the abstract 
"Recognizer.Nothing" token.

It seems far to complex to be the right way. Any hints?

-- 
Lionel Draghi

PS : and once more question : is there still a mail list for OpenToken 
on AdaPower site? (I would like to have a look at the archive)

Re: about OpenToken

[Dmitry A. Kazakov]

On Sun, 11 Dec 2005 00:54:49 +0100, Lionel Draghi wrote:

> Here is my problem : I am implementing a recursive descent parser, and 
> my grammar contains optionnals keywords, that will be ignored.
> Here is an exemple:
> - do this and this
> - do this and also this
> "also" is an optionnal keyword.
> 
> I am wondering what is the best way to deal with this.
> What comes to my mind is to create a selection (In OpenToken parlance) 
> between the keyword "also" and a null token, child of the abstract 
> "Recognizer.Nothing" token.

If optional keywords cannot appear in any context in any number then this
will not work.

> It seems far to complex to be the right way. Any hints?

Far too simple you mean...

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: about OpenToken

[Stephen Leake]

Lionel Draghi <Lionel.nospam.Draghi@Ada-France.org> writes:

> Is 3.0b the latest available OpenToken version?

I'm using a version of OpenToken that I have patched and enhanced. I'm
no longer sure what version I got originally, but since the OpenToken
website is dated 9/6/00, it's quite probable I started with 3.0b.

Perhaps if there is enough interest, we should start a SourceForge project.

> I'am not so sure, because some of the examples don't compile using
> gnat 4.0.3 (Debian 4.0.2-4).

I have not tried that yet, but I am using GNAT 5.03a.

What sorts of errors are you getting? Perhaps it's one of the Ada 2006
features that is incompatible with Ada 95 (like the 'interface' keyword).

> Here is my problem : I am implementing a recursive descent parser, and
> my grammar contains optionnals keywords, that will be ignored.
> Here is an exemple:
> - do this and this
> - do this and also this
> "also" is an optionnal keyword.
>
> I am wondering what is the best way to deal with this.
> What comes to my mind is to create a selection (In OpenToken parlance)
> between the keyword "also" and a null token, child of the abstract
> "Recognizer.Nothing" token.

I'm not using recursive descent, but I did play with selections and
couldn't get them to work. So I just code all variants of the syntax:

   Grammar : constant Production_List.Instance :=
     Tokens.Statement <= Go_Statement and
     Go_Statement <= --  go;
        Nonterminal.Get (Go_ID) + Execute_Action and
     Go_Statement <= --  go sync;
        Nonterminal.Get (Go_ID) & Nonterminal.Get (Sync_ID) + Execute_Action and
     Go_Statement <= --  go count;
        Nonterminal.Get (Go_ID) & Tokens.Integer + Execute_Action and
     Go_Statement <= --  go hardware;
        Nonterminal.Get (Go_ID) & Nonterminal.Get (Hardware_ID) + Execute_Action and
     Go_Statement <= --  go hardware sync;
        Nonterminal.Get (Go_ID) & Nonterminal.Get (Hardware_ID) & Nonterminal.Get (Sync_ID) + Execute_Action and
     Go_Statement <= --  go hardware count;
        Nonterminal.Get (Go_ID) & Nonterminal.Get (Hardware_ID) & Tokens.Integer + Execute_Action;

> PS : and once more question : is there still a mail list for OpenToken
> on AdaPower site? (I would like to have a look at the archive)

The only things about OpenToken I found on AdaPower point to the
OpenToken web site.

-- 
-- Stephe

Re: about OpenToken example compilation error

[Lionel Draghi]

Stephen Leake a ï¿œcrit :
...
> I have not tried that yet, but I am using GNAT 5.03a.
> 
> What sorts of errors are you getting? Perhaps it's one of the Ada 2006
> features that is incompatible with Ada 95 (like the 'interface' keyword).
In the ASU_Example_5_10 :
make
...
gcc-4.0 -c -I../../ asu_example_5_10-run.adb
asu_example_5_10.ads:125:55: "Token_ID" is not a visible entity of 
"Master_Token"
asu_example_5_10.ads:141:40: invalid operand types for operator "+"
asu_example_5_10.ads:141:40: left operand has private type "Instance" 
defined at opentoken-token-enumerated-list.ads:47, instance at line 64
asu_example_5_10.ads:141:40: right operand has type access to 
"Print_Value" defined at line 141

It's probably easy to fix, I just wonder if I am using the up to date 
version.

>>PS : and once more question : is there still a mail list for OpenToken
>>on AdaPower site? (I would like to have a look at the archive)
> 
> The only things about OpenToken I found on AdaPower point to the
> OpenToken web site.

I saw the reference to the list on this site :
"There's a email list avilable for OpenToken. To subscribe, send a 
message to OpenToken-list-request@AdaPower.com with the word "subscribe" 
in the body."

-- 
Lionel Draghi

Re: about OpenToken

[Lionel Draghi]

Dmitry A. Kazakov a ï¿œcrit :
...
> If optional keywords cannot appear in any context in any number then this
> will not work.
OK, I won't try.

>>It seems far to complex to be the right way. Any hints?
> Far too simple you mean...

I expect the code representing the grammar to be almost as simple as the 
BNF expression.
If an optionnal word requires a bunch of code, I would say that there's 
something to improve.

-- 
Lionel Draghi

Re: about OpenToken

[Lionel Draghi]

Stephen Leake a ï¿œcrit :
> I'm using a version of OpenToken that I have patched and enhanced. I'm
> no longer sure what version I got originally, but since the OpenToken
> website is dated 9/6/00, it's quite probable I started with 3.0b.
> 
> Perhaps if there is enough interest, we should start a SourceForge project.

That's a good idea, at least to start from your up-to-date sources.

For the future, it may be interesting to compare OpenToken with similar 
library like ANTLR (http://www.antlr.org/).
Is OpenToken a good basis, and is it worth the time, or is it better to 
interface with another existing library?

-- 
Lionel Draghi

Re: about OpenToken

[Dmitry A. Kazakov]

On Sun, 11 Dec 2005 19:23:38 +0100, Lionel Draghi wrote:

> Dmitry A. Kazakov a �crit :
> ...
>>>It seems far to complex to be the right way. Any hints?
>> Far too simple you mean...
> 
> I expect the code representing the grammar to be almost as simple as the 
> BNF expression.

Well, that depends. Consider comments and line feeds allowed between the
keywords. To represent this in BNF, I don't think it will be such simple
expression.

> If an optionnal word requires a bunch of code, I would say that there's 
> something to improve.

There are many approaches to parsing. I am using table driven parsers. For
a descent recursive parser I just change the table to match the source
against depending on the context. I presume that something similar can be
done with OpenToken as well.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: about OpenToken

[Stephen Leake]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> On Sun, 11 Dec 2005 19:23:38 +0100, Lionel Draghi wrote:
>
>> I expect the code representing the grammar to be almost as simple as the 
>> BNF expression.
>
> Well, that depends. Consider comments and line feeds allowed between the
> keywords. To represent this in BNF, I don't think it will be such simple
> expression.

OpenToken does comments and white space in the lexer level, not the
grammar level.

I don't think you can define a keyword as whitespace, but it's an
interesting idea.

>> If an optionnal word requires a bunch of code, I would say that there's 
>> something to improve.
>
> There are many approaches to parsing. I am using table driven parsers. For
> a descent recursive parser I just change the table to match the source
> against depending on the context. I presume that something similar can be
> done with OpenToken as well.

Yes, you can.

-- 
-- Stephe

Re: about OpenToken

[Stephen Leake]

Lionel Draghi <Lionel.nospam.Draghi@Ada-France.org> writes:

> Stephen Leake a �crit :
>> Perhaps if there is enough interest, we should start a SourceForge
>> project.
>
> That's a good idea, at least to start from your up-to-date sources.

We need interest from more than two people to make it worth setting up
on SourceForge. I can email you my sources if you'd like.

> For the future, it may be interesting to compare OpenToken with
> similar library like ANTLR (http://www.antlr.org/).

I just took a peek at ANTLR.

It is similar to YACC, in that it is a pre-processor; it generates
Java, C#, C++, or Python code that then does the actual parsing (note
Ada is missing; it could presumably be added). It appears to be
written in C++ (I haven't downloaded the full source). It has a BSD
license. 

> Is OpenToken a good basis, and is it worth the time, or is it better
> to interface with another existing library?

OpenToken takes a different approach. The parse table is built at
run-time; there is no pre-processor generated code. This makes it
easier to use, if less efficient at run time. For example, one pass
with gnatmake is sufficient to find all compilation errors; you don't
have to build makefiles to keep track of the pre-processor sources.
This also makes it possible to modify the language parsed based on
startup parameters; I don't do that, but it seems an interesting idea.

It would probably be possible to dump the parse table and reload it,
to save the time it takes to build it. In my projects, that's not
worth the trouble.

OpenToken is also fully object-oriented; the run-time parser
dispatches to the various actions. That also makes it easier to use.  

OpenToken is written in Ada, and has the GMGPL license.

I'm not sure if OpenToken has the "latest LL(k)" grammar generator
algorithms, as ANTLR claims. So far, it's good enough for my projects.
And since it's in Ada, I can fix it :).

I'll stick with OpenToken.

-- 
-- Stephe

Re: about OpenToken example compilation error

[Stephen Leake]

Lionel Draghi <Lionel.nospam.Draghi@Ada-France.org> writes:

> Stephen Leake a �crit :
> ...
>> I have not tried that yet, but I am using GNAT 5.03a.
>> What sorts of errors are you getting? Perhaps it's one of the Ada
>> 2006
>> features that is incompatible with Ada 95 (like the 'interface' keyword).
> In the ASU_Example_5_10 :
> make
> ...
> gcc-4.0 -c -I../../ asu_example_5_10-run.adb
> asu_example_5_10.ads:125:55: "Token_ID" is not a visible entity of
> "Master_Token"
> asu_example_5_10.ads:141:40: invalid operand types for operator "+"
> asu_example_5_10.ads:141:40: left operand has private type "Instance"
> defined at opentoken-token-enumerated-list.ads:47, instance at line 64
> asu_example_5_10.ads:141:40: right operand has type access to
> "Print_Value" defined at line 141
>
> It's probably easy to fix, I just wonder if I am using the up to date
> version.

Hmm. I don't think I compiled that example. But it's probably due to
GNAT getting better at enforcing the Ada 95 language; I fixed similar
errors in my sources.

>>>PS : and once more question : is there still a mail list for OpenToken
>>>on AdaPower site? (I would like to have a look at the archive)
>> The only things about OpenToken I found on AdaPower point to the
>> OpenToken web site.
>
> I saw the reference to the list on this site :

By 'this site' do you mean 'comp.lang.ada'?

> "There's a email list avilable for OpenToken. To subscribe, send a
> message to OpenToken-list-request@AdaPower.com with the word
> "subscribe" in the body."

I just sent a subscription request to that address; it bounced. So 
that email list does not exist.

-- 
-- Stephe

Re: about OpenToken

[Lionel Draghi]

Dmitry A. Kazakov a ï¿œcrit :
..
> There are many approaches to parsing. I am using table driven parsers. For
> a descent recursive parser I just change the table to match the source
> against depending on the context. I presume that something similar can be
> done with OpenToken as well.

I have no special preferences. I have a simple grammar to implement, and 
  my main criteria is simplicity.
When reading OpenToken's doc, I had the impression that a descent 
recursive parser was easier to write (even if beta).
That's probably wrong. For example, my grammar is not LL1, and this 
cause an extra work when using descent parser in OpenToken.

The fact is that I have spent to much time on this topic.
I am about to give-up doing the full analysis with OpenToken, and be 
back to "hand made" analysis. I know that this will becomes a nigthmare 
when the grammar will grow in complexity, but for now it's still OK.

-- 
Lionel Draghi

Re: about OpenToken example compilation error

[Lionel Draghi]

Stephen Leake a ï¿œcrit :
..
> By 'this site' do you mean 'comp.lang.ada'?
Sorry, I meant this one 
http://www.telepath.com/~dennison/Ted/OpenToken/OpenToken.html

-- 
Lionel Draghi

Re: about OpenToken

[Dmitry A. Kazakov]

On Mon, 12 Dec 2005 00:01:25 +0100, Lionel Draghi wrote:

> Dmitry A. Kazakov a �crit :
> ..
>> There are many approaches to parsing. I am using table driven parsers. For
>> a descent recursive parser I just change the table to match the source
>> against depending on the context. I presume that something similar can be
>> done with OpenToken as well.
> 
> I have no special preferences. I have a simple grammar to implement, and 
> my main criteria is simplicity.
> When reading OpenToken's doc, I had the impression that a descent 
> recursive parser was easier to write (even if beta).
> That's probably wrong. For example, my grammar is not LL1, and this 
> cause an extra work when using descent parser in OpenToken.

You know, what humans count as simple is not necessarily grammatically
simple. The parser in our heads works differently. Consider Ada's "and",
"and then", "then". Simple?

For things like that I'm using "modifiers", which change the operation
stack before firing the actions stored there. So "then" could look around
and decide whether it should modify "and" to "and then" or else shift. To
me it is simple, at least simpler that writing BNF standing on my head...

Another variant is to use patterns as tokens. I used this approach too, but
I feel uncomfortable with patterns. They aren't simple! (:-))

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: about OpenToken

[Marc A. Criley]

Lionel Draghi wrote:
> I have no special preferences. I have a simple grammar to implement, and 
>  my main criteria is simplicity.
> When reading OpenToken's doc, I had the impression that a descent 
> recursive parser was easier to write (even if beta).
> That's probably wrong. For example, my grammar is not LL1, and this 
> cause an extra work when using descent parser in OpenToken.
> 
> The fact is that I have spent to much time on this topic.
> I am about to give-up doing the full analysis with OpenToken, and be 
> back to "hand made" analysis. I know that this will becomes a nigthmare 
> when the grammar will grow in complexity, but for now it's still OK.

I tried out OpenToken, and while I think it's probably pretty good for 
nice, rigid grammars, if your needs go beyond that you might need to 
look elsewhere.  (In no way am I criticising OpenToken, there are lots 
of areas where simple, rigid grammars are all that's needed and 
OpenToken does an admirable job with them.)

For XPath In Ada (XIA) (http://www.mckae.com/xia.html) I used AdaGOOP: 
http://unicoi.kennesaw.edu/ase/ase02_02/tools/usafa/adagoop.

It was a little gnarly to set up and use, since it interacts with lex 
and yacc variants called scaflex and scayacc, which I had to dork around 
with to build and use.  Once that was done, though, it worked great 
building me a parser for the XPath grammar.  Though of course I had to 
construct the grammar in the format that AdaGOOP expected, which also 
caused me a few headaches (the grammar, not the format).

Once done, it was done though.  Using AdaGOOP is more work than 
something like OpenToken, but it works well on those more complex 
grammar's, like XPath's, and since I needed XPath querying for the 
pending version of DTraq I was motivated to grind through it and get it 
working.  (And then make XIA available to the Ada community :-)

-- Marc A. Criley
-- McKae Technologies
-- www.mckae.com
-- DTraq - XPath In Ada - XML EZ Out

Re: about OpenToken

[Martin Krischik]

Stephen Leake wrote:

> We need interest from more than two people to make it worth setting up
> on SourceForge. I can email you my sources if you'd like.

I disagree here. Sourceforge make a good archiving tool. Imagine you had
been ill this week then you two might not have found each other and would
not have been able comunicate with each other.

With Sourceforge the newest sources are allways available.

Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: about OpenToken

[qun-ying]

Stephen Leake wrote:
> Lionel Draghi <Lionel.nospam.Draghi@Ada-France.org> writes:
>
> > Stephen Leake a écrit :
> >> Perhaps if there is enough interest, we should start a SourceForge
> >> project.
> >
> > That's a good idea, at least to start from your up-to-date sources.
>
> We need interest from more than two people to make it worth setting up
> on SourceForge. I can email you my sources if you'd like.
I have some interest on it, I think once you setup the SF project,
there will be more people will show their interest.

Re: about OpenToken

[Ludovic Brenta]

"qun-ying" <zhu.qunying@gmail.com> writes:

> Stephen Leake wrote:
>> Lionel Draghi <Lionel.nospam.Draghi@Ada-France.org> writes:
>>
>> > Stephen Leake a écrit :
>> >> Perhaps if there is enough interest, we should start a SourceForge
>> >> project.
>> >
>> > That's a good idea, at least to start from your up-to-date sources.
>>
>> We need interest from more than two people to make it worth setting up
>> on SourceForge. I can email you my sources if you'd like.
> I have some interest on it, I think once you setup the SF project,
> there will be more people will show their interest.

As the maintainer of OpenToken in Debian, I am also interested and
will keep the package up to date if and when improvements appear.
FYI, the current version in Debian is 3.0b-1.

-- 
Ludovic Brenta.

Re: about OpenToken

[Lionel Draghi]

Marc A. Criley a ï¿œcrit :
...
> For XPath In Ada (XIA) (http://www.mckae.com/xia.html) I used AdaGOOP: 
> http://unicoi.kennesaw.edu/ase/ase02_02/tools/usafa/adagoop.
> 
> It was a little gnarly to set up and use, since it interacts with lex 
> and yacc variants called scaflex and scayacc,
This is why I did not investigate further on AdaGOOP: I felt it was to 
complex to settle down for my simple needs (and the link to SCATC was 
broken).

BTW, I just noticed that there is an Ada version of the LL(k) Coco/R 
(http://www.ssw.uni-linz.ac.at/Research/Projects/Coco/)
here :
http://www.ada-ru.org/files/cocor_ada-1.53.1.tar.gz

Yet another alternative to check... :-)

-- 
Lionel Draghi

Re: about OpenToken

[Lionel Draghi]

Lionel Draghi a ï¿œcrit :
...
> For the future, it may be interesting to compare OpenToken with similar 
> library like ANTLR (http://www.antlr.org/).

Closer to OpenToken than ANTLR is Spirit (http://spirit.sourceforge.net/).

Spirit is addressing moderately complex grammars, inlined in the C++ 
code, and seems to be easy to use (as far as I can undestand C++ code).

-- 
Lionel Draghi

Re: about OpenToken

[Georg Bauhaus]

On Tue, 2005-12-13 at 01:01 +0100, Lionel Draghi wrote:
> Lionel Draghi a écrit :
> ...
> > For the future, it may be interesting to compare OpenToken with similar 
> > library like ANTLR (http://www.antlr.org/).
> 
> Closer to OpenToken than ANTLR is Spirit (http://spirit.sourceforge.net/).
> 
> Spirit is addressing moderately complex grammars, inlined in the C++ 
> code, and seems to be easy to use (as far as I can undestand C++ code).
> 

If you consider Spirit, have a look at GNAT.Spitbol.Patterns.
Plain Ada (GNAT actually), and easy to use in my view.

"The pattern language is modeled on context free grammars,
with context sensitive extensions that provide full (type 0)
computational capabilities." (g-spipat.ads)

That's actually more than Spirit can offer unless things have
changed. The notation is fairly concise, permits assignment
of matched substrings and subprogram calls while the
scanner is moving (back and forth).


-- Georg 

Re: about OpenToken

[christoph.grein]

qun-ying wrote:
> I have some interest on it, I think once you setup the SF project,
> there will be more people will show their interest.

I wrote a few new recognizers and modified a lot of the others to
create the Ada and Java lexers delivered with OpenToken.

So I would also be interested in a more lively OpenToken package. (Has
Ted drowned in work as an employee because we have not heard a mumbling
word of him for ages?)

But before setting up such a project, you should try to contact him via
e-mail. As far as I know, he received some updates from other users
(including me) which he did not yet make public.

There is also an unresolved problem in the Ada lexer: it cannot handle
correctly the following code fragment:
    Character'(''')