Re: OpenToken version 3.1 preview

[Stephen Leake <stephen_leake@stephe-leake.org>]

AdaMagica <christoph.grein@eurocopter.com> writes:

> On Jul 23, 3:41�am, Stephen Leake <stephen_le...@stephe-leake.org>
> wrote:
>> AdaMagica <christoph.gr...@eurocopter.com> writes:
>> > There is a problem with Bracketed_Comment. If it extends over more
>> > than one line, the token is correctly recognized, but the lexeme
>> > fails.
>>
>> The line feed characters are dropped from the lexeme, on Windows.
>
> Also on Linux.
>
>> I don't suppose you have an idea of how to fix it?
>
> You guessed right - I haven't. I shortly browsed the code, but found
> no simple solution.
>
>> It will be interesting to figure out how to make that test portable
>> between Windows and Gnu/Linux. The easiest way to identify which line
>> ending to use that I know of is to look at
>> GNAT.Directory_Operations.Dir_Separator; it's '\' for CR LF, '/' for
>> LF. Don't know how to deal with Mac!
>
> There are other OSs where an end of line is not a character in the
> stream. Can OpenToken handle these?

The current file Text_Feeder uses Ada.Text_IO, so it should do "the
right thing" for any OS.

> We could do a Get_Line and insert a LF irrespective of what the OS
> uses. 

That's what the text feeder does now. Actually, it inserts
EOL_Character (see below).

So the LF must be dropped after that; I'll have to look harder.

> If then a lexeme was output that comprises several lines (currently
> only Bracketed_Comment I think), the output routine would have to
> translate this back to the OS's New_Line (this has of course to be
> documented in the recognizer).

Right.

> There is a declaration EOL_Character in package OpenToken.

Which has a comment to change it for your OS; not very friendly, as
it's a constant!

It's used in OpenToken.Recognizer.Character_Set.Standard_Whitespace,
OpenToken.Recognizer.Line_Comment.Analyze,
OpenToken.Recognizer.String.Analyze.

I'll change the comment to "we use this regardless of OS, since we
need a standard way of representing an end of line in a string
buffer".

-- 
-- Stephe