Re: OpenToken: Parsing Ada (subset)?

[Georg Bauhaus <bauhaus@futureapps.invalid>]

On 17.06.15 19:58, Stephen Leake wrote:
> Ok. I guess I'd like to see some actual examples of hand-written lexers.

My silly Ada source highlighter effectively used look-back, too, as a
state in the tokenizer, IIRC.  Bigger language tokens would be built
from smaller token pieces; there are definitions of Delimiter_1 and
Delimiter_2, a constraint expressing a relation between Operator_1 and
Delimiter_1 (of inclusion); a number of Character_Set-s and Is_*
functions for classifying; but still not all tokens are classified
correctly (bugs or leniency, depending on POV of this program).

Incidentally, the ubiquitous highlite program has similar shortcomings
when parsing Ada or Perl (they share the '''), last time I looked.

Some more input for testing, omitting spaces and combinations:

Character'(''');
Character(''');
Character'('(');
Character('(');

Character'(-''');
Character'('-'-'-');
Character'(-'''-'-');

Character'(''')'Alignment;
Name(''')'Has_Same_Storage;
Character'Pos(''');
Character'Base'(''');
Character'Base'Pos(''');