Re: Tentative design of markdown parser

[Natasha Kerensikova <lithiumcat@gmail.com>]

Hello,

On 2011-05-30, Georg Bauhaus <rm.dash-bauhaus@futureapps.de> wrote:
> On 28.05.11 15:22, Natasha Kerensikova wrote:
>
>> This is actually the only point I don't like about the design: there
>> should be a way to check at compile time that in a Span_Element the
>> Renderer_Callback can actually handle what is produced by the
>> Lexer_Callback. However I can't find any.
>
> When I know all the types of objects that Lexer_Callback
> can produce, I like to list them in an enumeration type,
> type Token_Kind is (A, B, C, ...).

Yes, I do like that too, and then using a variant record to store the
token-specific parameters rather than a tagged type.

However, that means knowing statically all kinds of tokens, which
prevents any extension that adds new tokens. That is how my C library
was designed, but then I found out that not all Markdown extensions fit
in the Markdown-strict set of tokens. One common example is tables. So I
ended up adding tokens for tables, which means some of the features are
inside the parser while some others are completely outside, and I really
don't like that.

So with that experience and the emphasis on extensibility from various
people in my previous thread, I was trying to design a parser that would
be extensible both by adding new tokens and new ways of rendering a set
of tokens. It does seem like a much harder problem to solve. If it turns
out impossible, or unreasonably complex, to solve, then I will fall back
on the fixed set of tokens.

Here I used a tagged type for Token only to have what is effectively a
variant record whose discriminant and variations would be extensible
(using the tag instead of a discriminant). It feels a bit like an abuse
of the tagged type system, but I haven't been able to come up with
anything better.


Thanks for your comment,
Natasha