Announcement: OpenToken 1.1 released

Announcement: OpenToken 1.1 released

[Ted Dennison]

Version 1.1 of the OpenToken package has been released. It is available
at http://www.telepath.com/dennison/Ted/OpenToken/OpenToken.html . It is
also available from links on my homepage and AdaPower.

The new features are:

   * A default text feeder function that can be redirected to whatever
     file you want via Text_IO.
   * A user's guide with a tutorial.
   * An example program implementing the syntax in the Aho/Sethi/Ullman
     Compilers book's (aka: "The Dragon Book") example 3.6.

Relevant portions of the Readme file are attached below:
-----------------------------------------
OpenToken Package Readme

                                   Version 1.1


The OpenToken package is a facility for performing token analysis within
the Ada language. It is designed to provide
all the functionality of a traditional lexical analyzer generator, such
as lex. But due to the magic of inheritance and
runtime polymorphism it is implemented entirely in Ada as withed-in
code. No precompilation step is required, and
no messy tool-generated source code is created.

Additionally, the technique of using classes of recognizers promises to
make most token specifications as simple as
making an easy to read procedure call. The most error prone part of
generating analyzers, the token pattern
matching, has been taken from the typical user's hands and placed into
reusable classes. Over time I hope to see the
addition of enough reusable recognizer classes that very few users will
ever need to write a custom one.

Ada's type safety features should also make misbehaving analyzers easier
to debug. All this will hopefully add up to
token analyzers that are much simpler and faster to create, easier to
get working properly, and easier to understand.

History

Version 1.1

The main code change to this version is a default text feeder function
that has been added to the analyzer. It reads its
input from Ada.Text_IO.Current_Input, so you can change the file to
whatever you want fairly easily. The capability
to create and use your own feeder function still exists, but it should
not be nessecary in most cases. If you already
have code that does this, it should still compile and work properly.

The other addition is the first version of the OpenToken user's guide.
All it contains right now is a user manual
walking through the steps needed to make a simple token analyzer.
Feedback and/or ideas on this are welcome.

Version 1.0

This is the very first publicly released version. This package is based
on work I did while working on the JPATS
trainer for FlightSafety International. The germ of this idea came while
I was trying to port a fairly ambitious, but
fatally buggy Ada 83 token recognition package written for a previous
simulator. But once I was done, I was rather
suprised at the flexibility of the final product. Seeing the possible
benefit to the community, and to the company
through user-submitted enhancement and debugging, I suggested that this
code be released as Open Source. They
were open-minded enough to agree. Bravo!


Future

I do plan on actively maintaining this package. How actively depends on
user submissions, my own schedule, and
what pans out for my studies. I am proposing to do my master's thesis on
this, but they seem to think it "isn't enough",
even with the proposed parsing facility added. I'll admit developing a
parser generator in inlined code using OO
technology to do what used to have to be done with a precompiler doesn't
seem to be an amazingly clever idea in
hindsight, but as near as I can tell this facility is unique. It
certainly should be enough work. We will see what
happens.

Things I would like to add soon:

     A string literal recognizer

Things on my plate for later:

     Look into changing the feeder function into a stream reference. I
was unfamiliar with streams when I wrote this
     package. It looks like they would make several things much easier
to deal with, but the devil's always in the
     details...
     More recognizers
     The Biggie: A parsing facility in the same vein as this token
analysis facility!

Things you can help with:

     More recognizers - The more of these there are, the more useful
this facility is. If you make 'em, please send
     'em in!
     Well isolated bug reports (or even fixes). Version 1.0 has been
fairly thoroughly wrung out already, but that
     doesn't mean you won't find any problems.

Again, I hope you find this package useful for your needs.
--------------------
--
T.E.D.

Home - mailto:dennison@telepath.net  Work - mailto:dennison@ssd.fsi.com
WWW  - http://www.telepath.com/dennison/Ted/TED.html  ICQ  - 10545591

Re: Announcement: OpenToken 1.1 released

[Ted Dennison]

David Botton wrote:

> I added a link to your project on the Ada Lab at:
>
> http://www.adapower.com/lab
>

Cool. Thanks.

> You may consider creating a collaboration page in the lab if you get
> other people involved in expanding your project.

Right now I really need the flexibility of being able to upload changes to the
website at the push of a button. For example, I've already received reports of
extraneous stuff in the tar.gz and typos in the user manual. Both have already
been fixed and uploaded to the website. When things settle down a little, and
I get a little better at this, I probably will want the whole project moved to
an independent site like AdaPower.

Personally I think its a great thing you are doing; giving small projects a
place to call home that has some presence. One of my concerns as a user of a
small Open Source project like OpenToken would be the possible sudden
dissapearance of the whole project when its current maintainer has his account
removed from his ISP for whatever reason. A site hosted on a user web page
would be much more likely to have that happen. So a spot on AdaPower I think
would lend a bit more legitimacy to a project.

--
T.E.D.

Home - mailto:dennison@telepath.net  Work - mailto:dennison@ssd.fsi.com
WWW  - http://www.telepath.com/dennison/Ted/TED.html  ICQ  - 10545591

Re: Announcement: OpenToken 1.1 released

[Ted Dennison]

Ted Dennison wrote:

> Version 1.1 of the OpenToken package has been released. It is available

I have had one person suggest I post html-ized versions of the sources
themselves. Its an interesting idea, but would drasticly increase the amount
of effort I have to go through whenever I release a new version. I'm sure
there are ways to do things less manually, but figuring that out would also
be effort taken away from other things. Does anyone else think that would be
a good idea?

--
T.E.D.

Home - mailto:dennison@telepath.net  Work - mailto:dennison@ssd.fsi.com
WWW  - http://www.telepath.com/dennison/Ted/TED.html  ICQ  - 10545591

Re: Announcement: OpenToken 1.1 released

[David Botton]

I added a link to your project on the Ada Lab at:

http://www.adapower.com/lab

You may consider creating a collaboration page in the lab if you get
other people involved in expanding your project.

David Botton


Ted Dennison wrote:
> 
> Version 1.1 of the OpenToken package has been released. It is available
> at http://www.telepath.com/dennison/Ted/OpenToken/OpenToken.html . It is
> also available from links on my homepage and AdaPower.
> 
> The new features are:
> 
>    * A default text feeder function that can be redirected to whatever
>      file you want via Text_IO.
>    * A user's guide with a tutorial.
>    * An example program implementing the syntax in the Aho/Sethi/Ullman
>      Compilers book's (aka: "The Dragon Book") example 3.6.
> 
> Relevant portions of the Readme file are attached below:
> -----------------------------------------
> OpenToken Package Readme
> 
>                                    Version 1.1
> 
> The OpenToken package is a facility for performing token analysis within
> the Ada language. It is designed to provide
> all the functionality of a traditional lexical analyzer generator, such
> as lex. But due to the magic of inheritance and
> runtime polymorphism it is implemented entirely in Ada as withed-in
> code. No precompilation step is required, and
> no messy tool-generated source code is created.
> 
> Additionally, the technique of using classes of recognizers promises to
> make most token specifications as simple as
> making an easy to read procedure call. The most error prone part of
> generating analyzers, the token pattern
> matching, has been taken from the typical user's hands and placed into
> reusable classes. Over time I hope to see the
> addition of enough reusable recognizer classes that very few users will
> ever need to write a custom one.
> 
> Ada's type safety features should also make misbehaving analyzers easier
> to debug. All this will hopefully add up to
> token analyzers that are much simpler and faster to create, easier to
> get working properly, and easier to understand.
> 
> History
> 
> Version 1.1
> 
> The main code change to this version is a default text feeder function
> that has been added to the analyzer. It reads its
> input from Ada.Text_IO.Current_Input, so you can change the file to
> whatever you want fairly easily. The capability
> to create and use your own feeder function still exists, but it should
> not be nessecary in most cases. If you already
> have code that does this, it should still compile and work properly.
> 
> The other addition is the first version of the OpenToken user's guide.
> All it contains right now is a user manual
> walking through the steps needed to make a simple token analyzer.
> Feedback and/or ideas on this are welcome.
> 
> Version 1.0
> 
> This is the very first publicly released version. This package is based
> on work I did while working on the JPATS
> trainer for FlightSafety International. The germ of this idea came while
> I was trying to port a fairly ambitious, but
> fatally buggy Ada 83 token recognition package written for a previous
> simulator. But once I was done, I was rather
> suprised at the flexibility of the final product. Seeing the possible
> benefit to the community, and to the company
> through user-submitted enhancement and debugging, I suggested that this
> code be released as Open Source. They
> were open-minded enough to agree. Bravo!
> 
> Future
> 
> I do plan on actively maintaining this package. How actively depends on
> user submissions, my own schedule, and
> what pans out for my studies. I am proposing to do my master's thesis on
> this, but they seem to think it "isn't enough",
> even with the proposed parsing facility added. I'll admit developing a
> parser generator in inlined code using OO
> technology to do what used to have to be done with a precompiler doesn't
> seem to be an amazingly clever idea in
> hindsight, but as near as I can tell this facility is unique. It
> certainly should be enough work. We will see what
> happens.
> 
> Things I would like to add soon:
> 
>      A string literal recognizer
> 
> Things on my plate for later:
> 
>      Look into changing the feeder function into a stream reference. I
> was unfamiliar with streams when I wrote this
>      package. It looks like they would make several things much easier
> to deal with, but the devil's always in the
>      details...
>      More recognizers
>      The Biggie: A parsing facility in the same vein as this token
> analysis facility!
> 
> Things you can help with:
> 
>      More recognizers - The more of these there are, the more useful
> this facility is. If you make 'em, please send
>      'em in!
>      Well isolated bug reports (or even fixes). Version 1.0 has been
> fairly thoroughly wrung out already, but that
>      doesn't mean you won't find any problems.
> 
> Again, I hope you find this package useful for your needs.
> --------------------
> --
> T.E.D.
> 
> Home - mailto:dennison@telepath.net  Work - mailto:dennison@ssd.fsi.com
> WWW  - http://www.telepath.com/dennison/Ted/TED.html  ICQ  - 10545591

Re: Announcement: OpenToken 1.1 released

[David Botton]

This will convert the specs to HTML with links and all:

ftp://ftp.seas.gwu.edu/pub/ada/adalib_html/

I've used it often for my own projects.

Also,

http://www.adahome.com/Resources/Tools/ada2html/ada2html.htm

Will do the src too, but requires Unix, Perl, and gnatf.

David Botton


Ted Dennison wrote:
> 
> Ted Dennison wrote:
> 
> > Version 1.1 of the OpenToken package has been released. It is available
> 
> I have had one person suggest I post html-ized versions of the sources
> themselves. Its an interesting idea, but would drasticly increase the amount
> of effort I have to go through whenever I release a new version. I'm sure
> there are ways to do things less manually, but figuring that out would also
> be effort taken away from other things. Does anyone else think that would be
> a good idea?
> 
> --
> T.E.D.
> 
> Home - mailto:dennison@telepath.net  Work - mailto:dennison@ssd.fsi.com
> WWW  - http://www.telepath.com/dennison/Ted/TED.html  ICQ  - 10545591

Re: Announcement: OpenToken 1.1 released

[Simon Wright]

David Botton <David@Botton.com> writes:

> This will convert the specs to HTML with links and all:
> 
> ftp://ftp.seas.gwu.edu/pub/ada/adalib_html/
> 
> I've used it often for my own projects.
> 
> Also,
> 
> http://www.adahome.com/Resources/Tools/ada2html/ada2html.htm
> 
> Will do the src too, but requires Unix, Perl, and gnatf.

I haven't looked to see exactly what's required (at least Perl,
obviously), but GNAT 3.11p's src/ada directory contains gnathtml.pl
which seems to do a brilliant job (I've only just tried it out the
once).

Re: Announcement: OpenToken 1.1 released

[James S. Rogers]

Simon Wright wrote in message ...
>David Botton <David@Botton.com> writes:
>
>> This will convert the specs to HTML with links and all:
>>
>> ftp://ftp.seas.gwu.edu/pub/ada/adalib_html/
>>
>> I've used it often for my own projects.
>>
>> Also,
>>
>> http://www.adahome.com/Resources/Tools/ada2html/ada2html.htm
>>
>> Will do the src too, but requires Unix, Perl, and gnatf.
>
>I haven't looked to see exactly what's required (at least Perl,
>obviously), but GNAT 3.11p's src/ada directory contains gnathtml.pl
>which seems to do a brilliant job (I've only just tried it out the
>once).

Yes, gnathtml.pl does require Perl.  I have used it several times.
The only drawback it has is its requirement that the source code compiles
cleanly under GNAT. It does not have an option to ignore a non-compiling
package and allow you to build html pages for all the other packages.

I intend to use this as part of a Software Design Document (SDD) and
Interface Design Document (IDD) for my current project in robotics.

Jim Rogers
Colorado Springs, Colorado USA

Re: Announcement: OpenToken 1.1 released

[dennison]

In article <7i9qbb$t28$1@bgtnsc03.worldnet.att.net>,
  "James S. Rogers" <jimmaureenrogers@worldnet.att.net> wrote:
>
> Simon Wright wrote in message ...
> >I haven't looked to see exactly what's required (at least Perl,
> >obviously), but GNAT 3.11p's src/ada directory contains gnathtml.pl
> >which seems to do a brilliant job (I've only just tried it out the
> >once).
>
> Yes, gnathtml.pl does require Perl.  I have used it several times.
> The only drawback it has is its requirement that the source code
compiles

I don't know Perl, but from looking over the source it appears to
require gnat ".ali" files for everything you want converted. And of
course there's the little matter of the Perl interpriter. Its a shame
they couldn't have provided this tool in a format that would work
straight out of the box, rather than requiring you to go download YASC
(yet another scripting language).

Hmmm. I wonder how much work would be involved in porting it to Ada.
OpenToken might be of some help... :-)

--
T.E.D.


--== Sent via Deja.com http://www.deja.com/ ==--
---Share what you know. Learn what you don't.---

Re: Announcement: OpenToken 1.1 released

[Robert Dewar]

In article <7ic0m5$848$1@nnrp1.deja.com>,
  dennison@telepath.com wrote:

> Its a shame
> they couldn't have provided this tool in a format that would
> work straight out of the box, rather than requiring you to go
> download YASC (yet another scripting language).

There has been no demand generated for converting this tool
to Ada. Our customers have had no trouble running the Perl
version.

But Ted, it would certainly be constructive and useful (at
least to you, and perhaps to others :-) if you
want to port this tool to Ada. The sources are available, so
by all means go ahead!

Robert Dewar
Ada Core Technologies



--== Sent via Deja.com http://www.deja.com/ ==--
---Share what you know. Learn what you don't.---

Re: Announcement: OpenToken 1.1 released

[Stephen Leake]

dennison@telepath.com writes:

> I don't know Perl, but from looking over the source it appears to
> require gnat ".ali" files for everything you want converted. And of
> course there's the little matter of the Perl interpriter. Its a shame
> they couldn't have provided this tool in a format that would work
> straight out of the box, rather than requiring you to go download YASC
> (yet another scripting language).
> 
> Hmmm. I wonder how much work would be involved in porting it to Ada.
> OpenToken might be of some help... :-)

Before you do that, be sure to see the adalib_html tool at:

http://www.seas.gwu.edu/seas/eecs/Research/ada/adalib_html/

(this site was mentioned eariler in this thread). This tool is written
in Ada, and compiles nicely on Win95 with GNAT (by which I mean, all I
had to was type "gnatmake -O3"). I use it to publish the specs to
Windex - one of these days I'll compare it's output to that of the
Perl script that comes with GNAT.

-- Stephe

Re: Announcement: OpenToken 1.1 released

[Thomas Quinot]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 310 bytes --]

Dans comp.lang.ada,  �crit :

>Hmmm. I wonder how much work would be involved in porting it to Ada.
>OpenToken might be of some help... :-)

Alternatively you may want to consider implementing it as an ASIS
application.

Thomas.

-- 
    Thomas.Quinot@Cuivre.FR.EU.ORG   <URL:http://web.fdn.fr/~tquinot/>