Re: OpenToken: Handling the empty word token

["Randy Brukardt" <randy@rrsoftware.com>]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> wrote in message 
news:1jvlv7i0tn14u.b5d2cwsqhl2h$.dlg@40tude.net...
> On Fri, 27 Jan 2012 08:22:12 -0800 (PST), mtrenkmann wrote:
>
>> Is there a way to instrument the parser to silently accept the epsilon
>> token whenever it expects it without consuming a token from the lexer,
>> or is it a common convention to translate each grammar into a epsilon-
>> free representation?
>
> I use neither explicit grammars nor OpenToken, so it is possible that I
> didn't really understand the problem you have.

Like Dmitry, I don't use OpenToken, but I do use a LALR(1) parser generator 
(ours originates in a University of Wisconsin research project from the late 
1970s).

In all of the grammars I've seen, you don't write anything for an epsilon 
production; that's because you are matching nothing. But there is no problem 
in matching nothing, so long as your grammar generator is powerful enough 
(uses at least LALR(1) parsing, or perhaps LR(1) parsing). In that case, 
matching nothing works so long as the follow sets are disjoint (something 
that fails to be true periodically in our Ada grammar).

For instance, here's the grammar for parameter modes from the Janus/Ada 
compiler grammar:

mode ::= IN ## 93
    | OUT ## 94
    | IN OUT ## 95
    |   ## 198

Note that the last production is an epsilon production. The ## part gives an 
action number associated with the matching of that particular alternative of 
this production. The ## part also marks the end of the production (it's 
optional, and | also ends a production -- but it's required on the last 
alternative as the grammar of our grammar uses insignificant line endings 
like Ada does).

I'd be surprised if OpenToken didn't have something similar; and if it 
doesn't, you probably need to upgrade to a better grammar generator.

                                                           Randy.