From mboxrd@z Thu Jan  1 00:00:00 1970
X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham
	autolearn_force=no version=3.4.4
X-Google-Language: ENGLISH,ASCII
Path: 
 g2news2.google.com!postnews.google.com!c1g2000yqi.googlegroups.com!not-for-mail
From: AdaMagica <christoph.grein@eurocopter.com>
Newsgroups: comp.lang.ada
Subject: Re: OpenToken version 3.1 preview
Date: Wed, 22 Jul 2009 22:09:27 -0700 (PDT)
Organization: http://groups.google.com
Message-ID: <17b17a5b-6b54-4486-8494-650827a58dad@c1g2000yqi.googlegroups.com>
References: <ueiscnz8i.fsf@stephe-leake.org>
 <aa18196d-e374-436d-b6c3-28426d9ea58e@a26g2000yqn.googlegroups.com>
	<uab2wb2h6.fsf@stephe-leake.org>
NNTP-Posting-Host: 80.156.44.178
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Trace: posting.google.com 1248325768 20949 127.0.0.1 (23 Jul 2009 05:09:28
 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Thu, 23 Jul 2009 05:09:28 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: c1g2000yqi.googlegroups.com; posting-host=80.156.44.178;
	posting-account=rmHyLAoAAADSQmMWJF0a_815Fdd96RDf
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;
 .NET
	CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.04506.30;
 .NET CLR
	3.0.4506.2152; .NET CLR 3.5.30729),gzip(gfe),gzip(gfe)
X-HTTP-Via: 1.1 webwasher (Webwasher 6.8.3.4555)
Xref: g2news2.google.com comp.lang.ada:7287
Date: 2009-07-22T22:09:27-07:00
List-Id: <comp.lang.ada>

On Jul 23, 3:41=A0am, Stephen Leake <stephen_le...@stephe-leake.org>
wrote:
> AdaMagica <christoph.gr...@eurocopter.com> writes:
> > There is a problem with Bracketed_Comment. If it extends over more
> > than one line, the token is correctly recognized, but the lexeme
> > fails.
>
> The line feed characters are dropped from the lexeme, on Windows.

Also on Linux.

> I don't suppose you have an idea of how to fix it?

You guessed right - I haven't. I shortly browsed the code, but found
no simple solution.

> It will be interesting to figure out how to make that test portable
> between Windows and Gnu/Linux. The easiest way to identify which line
> ending to use that I know of is to look at
> GNAT.Directory_Operations.Dir_Separator; it's '\' for CR LF, '/' for
> LF. Don't know how to deal with Mac!

There are other OSs where an end of line is not a character in the
stream. Can OpenToken handle these?
We could do a Get_Line and insert a LF irrespective of what the OS
uses. If then a lexeme was output that comprises several lines
(currently only Bracketed_Comment I think), the output routine would
have to translate this back to the OS's New_Line (this has of course
to be documented in the recognizer).

There is a declaration EOL_Character in package OpenToken.