Re: OpenToken version 3.1 preview

[Stephen Leake <stephen_leake@stephe-leake.org>]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> AdaMagica <christoph.grein@eurocopter.com> writes:
>
>> On Jul 23, 3:41�am, Stephen Leake <stephen_le...@stephe-leake.org>
>> wrote:
>>> AdaMagica <christoph.gr...@eurocopter.com> writes:
>>> > There is a problem with Bracketed_Comment. If it extends over more
>>> > than one line, the token is correctly recognized, but the lexeme
>>> > fails.
>>>
>>> The line feed characters are dropped from the lexeme, on Windows.
>>

Here is the explanation of this symptom.

Text_Feeder uses Ada.Text_IO.Get_Line, so it never sees the "CR LF" on
DOS, nor the "LF" on Linux. It does insert a EOL_Character = CR for
each line break. That's why it appears to be dropping the LF.

So for a file created like this:

      Text1 : constant String := "/* A comment that starts here";
      Text2 : constant String := "   and keeps going";
      Text3 : constant String := "   and finally ends here *.*..";

      Create (File, Out_File, File_Name);
      Put_Line (File, Text1);
      Put_Line (File, Text2);
      Put_Line (File, Text3);
      Close (File);

the expected lexeme is:

      Expected_Lexeme : constant String := 
        Text1 & OpenToken.EOL_Character & 
        Text2 & OpenToken.EOL_Character & 
        Text3;

I've added a test that demonstrates this, and a comment to
opentoken-recognizer-bracketed_comment.ads to document it.

If the purpose of the lexer is to just recognize comments and skip
them, this is fine.

If the purpose of the lexer is to be able to later reconstruct the
code, the reconstruction routine will need a way to turn EOL_Character
back into OS-specific newlines; using Ada.Text_IO.Put_Line will do
that.

-- 
-- Stephe