Re: Announce: OpenToken 2.0 released

[Hyman Rosen <hymie@prolifics.com>]

Ted Dennison <dennison@telepath.com> writes:
> Proper binary support is not in OT because it has just never come up
> before. But as you can see, it could be modified fairly easily to
> support parsing binaries. But using a sentinel character for the end of
> file has always seemed like a nice simplification. So what are the uses
> of parsing binaries? I kinda thought that binaries are, by their very
> nature, already parsed.

Well, at one point I was writing code to parse Adobe PDF files.
They have a binary format, where arbitrary 8-bit bytes can appear,
and a structure which I think lends itself well to syntax-oriented
parsing.

In general, I like to avoid arbitrary restrictions in tools. Before
GNU, most classic UNIX utilities had arbitrary limits, especially
on line size. This led to unexpected and sometimes silent breakage
when the tools were fed files with lines which were too large. And
the tool reporting the problem isn't of much help, when I still have
that file I need to process and the tool won't work.

By the way, the normal C/C++ style for handling EOF is to have the
return type of the character reader be such that it can hold any
value of the character set, plus an out-of-band value representing
EOF. The usual is '#define EOF -1' and 'int getchar()'.