Re: OpenToken version 3.1 preview

["Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>]

On Wed, 22 Jul 2009 22:09:27 -0700 (PDT), AdaMagica wrote:

> On Jul 23, 3:41�am, Stephen Leake <stephen_le...@stephe-leake.org>
> wrote:
>> AdaMagica <christoph.gr...@eurocopter.com> writes:
>>> There is a problem with Bracketed_Comment. If it extends over more
>>> than one line, the token is correctly recognized, but the lexeme
>>> fails.
>>
>> The line feed characters are dropped from the lexeme, on Windows.
> 
> Also on Linux.
> 
>> I don't suppose you have an idea of how to fix it?
> 
> You guessed right - I haven't. I shortly browsed the code, but found
> no simple solution.
> 
>> It will be interesting to figure out how to make that test portable
>> between Windows and Gnu/Linux. The easiest way to identify which line
>> ending to use that I know of is to look at
>> GNAT.Directory_Operations.Dir_Separator; it's '\' for CR LF, '/' for
>> LF. Don't know how to deal with Mac!
> 
> There are other OSs where an end of line is not a character in the
> stream. Can OpenToken handle these?
> We could do a Get_Line and insert a LF irrespective of what the OS
> uses. If then a lexeme was output that comprises several lines
> (currently only Bracketed_Comment I think), the output routine would
> have to translate this back to the OS's New_Line (this has of course
> to be documented in the recognizer).

You could do what I did in the Simple Components for Ada parser. I
decoupled sources from the parser itself. The source is an abstract object
that provides basic operations like "get next line" and "forward to the
next line". The obvious advantage is that you need not to care about LF, CR
in the parser, and can use files, streams, strings, GUI text buffers, etc,
as a source to the same parser.

My 2 cents.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de