help / color / mirror / Atom feed
From: Niklas Holsti <niklas.holsti@tidorum.invalid>
Subject: Re: Status of ayacc and aflex?
Date: Fri, 22 Aug 2008 20:17:46 +0300
Date: 2008-08-22T20:17:46+03:00	[thread overview]
Message-ID: <48aef41e$0$23587$> (raw)
In-Reply-To: <>

Ludovic Brenta wrote:
> Niklas Holsti wrote:
>>I have not used the parsing functions in OpenToken...
> I too like the absence of a generator, and I'm used to writing my own
> parsers by hand. Did you find that OpenToken helped a lot in doing
> that?

It does its job: lexical analysis. The parser gets to look at the 
tokens one by one; OpenToken provides a function to return the 
identity of the current token (an enumeration), another function to 
return the text string of the current token, and a procedure to 
advance to the next token. That is what I expect of a lexical 
analyser, and OpenToken gives me that (at least; I haven't really 
studied it thoroughly to see what else there may be).

As far as I recall, the only wart I have found has to do with the 
error reporting when the input text has a sequence of characters 
that does not match any token -- I had to add a special "invalid 
token" definition (Opentoken.Recognizer.Nothing.Get) to find the 
line-number and column-number of the erroneous text. A minor detail.

If I try to think of what might be missing, perhaps the main thing 
is context-dependent lexical analysis: the ability to say, for 
example, that I expect the next token to be an identifier, so 
please ignore the definitions of reserved keywords and just 
consider them identifiers, too. Of course I have designed my own 
languages not to need this (keywords are really reserved).

As far as I know OpenToken has no general look-ahead facility, but 
it should not be too hard to build one yourself, on top of 
OpenToken, if you need it.

Another good point about OpenToken and the absence of a generator 
phase: one can have several instances of OpenToken in the same 
application, for different languages, without any clashes of names 
or data. My application needs that.

A caveat: The amounts of text that my applications scan with 
OpenToken are small. I have no idea of the scanning speed; it has 
been quite enough for my needs.

Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .

  reply	other threads:[~2008-08-22 17:17 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-08-21  2:35 Status of ayacc and aflex? Peter C. Chapin
2008-08-21  4:39 ` J. David Bryan
2008-08-21 10:35   ` Peter C. Chapin
2008-08-21  6:27 ` ficorax
2008-08-21 10:36   ` Peter C. Chapin
2008-08-21  8:36 ` Dmitry A. Kazakov
2008-08-21  9:56 ` Stephen Leake
2008-08-21 10:34   ` Peter C. Chapin
2008-08-21 14:55 ` gautier_niouzes
2008-08-21 23:00   ` Peter C. Chapin
2008-08-22 11:02 ` Colin Paul Gloster
2008-08-22 12:02   ` Ludovic Brenta
2008-08-22 12:56     ` Dmitry A. Kazakov
2008-08-22 13:29     ` Niklas Holsti
2008-08-22 14:17       ` Ludovic Brenta
2008-08-22 17:17         ` Niklas Holsti [this message]
2008-08-24 15:02     ` Stephen Leake
2008-08-28 13:38       ` OpenToken (was: Status of ayacc and aflex?) Ludovic Brenta
2008-08-29 22:51         ` OpenToken Stephen Leake
2008-08-30 10:43           ` OpenToken Ludovic Brenta
2008-08-31 13:27             ` OpenToken Stephen Leake
2008-08-31 14:03               ` OpenToken Ludovic Brenta
2008-09-01 12:40                 ` OpenToken Stephen Leake
2008-09-12 18:40                   ` OpenToken Ludovic Brenta
2023-05-31 20:42 ` Status of ayacc and aflex? Gautier write-only address
replies disabled

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox