Re: OpenToken version 3.1 preview

[AdaMagica <christoph.grein@eurocopter.com>]

On Jul 23, 3:41 am, Stephen Leake <stephen_le...@stephe-leake.org>
wrote:
> AdaMagica <christoph.gr...@eurocopter.com> writes:
> > There is a problem with Bracketed_Comment. If it extends over more
> > than one line, the token is correctly recognized, but the lexeme
> > fails.
>
> The line feed characters are dropped from the lexeme, on Windows.

Also on Linux.

> I don't suppose you have an idea of how to fix it?

You guessed right - I haven't. I shortly browsed the code, but found
no simple solution.

> It will be interesting to figure out how to make that test portable
> between Windows and Gnu/Linux. The easiest way to identify which line
> ending to use that I know of is to look at
> GNAT.Directory_Operations.Dir_Separator; it's '\' for CR LF, '/' for
> LF. Don't know how to deal with Mac!

There are other OSs where an end of line is not a character in the
stream. Can OpenToken handle these?
We could do a Get_Line and insert a LF irrespective of what the OS
uses. If then a lexeme was output that comprises several lines
(currently only Bracketed_Comment I think), the output routine would
have to translate this back to the OS's New_Line (this has of course
to be documented in the recognizer).

There is a declaration EOL_Character in package OpenToken.