OpenToken: Handling the empty word token

[mtrenkmann <martin.trenkmann@googlemail.com>]

Hello all.

Very often grammars have so called epsilon-productions where one
alternative for a non-terminal symbol points to the empty word
(epsilon).

For example: Optional -> Something | epsilon

In OpenToken I modeled the epsilon token as an
OpenToken.Recognizer.Nothing.Instance and defined the production like
this:

Optional <= Something and
Optional <= epsilon

Now I realized that the lexer would actually never emit the epsilon
token, because of it's pure formal meaning, and thus the second
production would never be detected.

Is there a way to instrument the parser to silently accept the epsilon
token whenever it expects it without consuming a token from the lexer,
or is it a common convention to translate each grammar into a epsilon-
free representation?

Thanks in advance.

-- Martin