RFC: markdown to HTML library

RFC: markdown to HTML library

[Natasha Kerensikova]

Hello,

I have been working on an Ada library that parses lightweight markup
languages and render them in various output format (somewhat like
pandoc, except I'm not sure my architecture scales easily to a feature
set as big as pandoc's).

I wanted to integrate it in the server for my website and let it run in
production for a while before formally realising it, however for various
reasons it may take a while before I reach that point.

Currently, the library is fully functional with only Markdown front-end
and (X)HTML back-end, it passes the official markdown test suite (that I
don't distribute because of licence uncertainty) and a decently-covering
homegrown test suite (according to gcov, it covers 1112 lines out of
1217 in official markdown front-end, 657/732 lines in markdown
extensions, and 348/398 lines in (X)HTML back-end).

Since recently there has been discussions here about Ada for the web,
and there's even a fosdem talk about it, so maybe Markdown-to-HTML is of
interest too.

I would be greatly interested in hearing any comment or criticism or
event bug reports about it.

Features request are welcome too, though I can't tell for now when I
will manage to look into them. Currently reStructuedText front-end and
and fully-configurable ODT back-end are on my radar.

I can get into the details of how it works internally, but I won't bore
you with it if it's not necessary.

The code is released under ISC licence and currently available on
github at http://github.com/faelys/markup-ada and eventually the
"official" fossil repository will be on my aforementioned website.


Thanks in advance for your interest,
Natasha

Re: RFC: markdown to HTML library

[Dmitry A. Kazakov]

On Sun, 19 Jan 2014 22:26:04 +0000 (UTC), Natasha Kerensikova wrote:

> Features request are welcome too, though I can't tell for now when I
> will manage to look into them. Currently reStructuedText front-end and
> and fully-configurable ODT back-end are on my radar.
> 
> I can get into the details of how it works internally, but I won't bore
> you with it if it's not necessary.

There are many different requirements depending on the usage, I guess.

Now to the way my HTTP server works. The content is generated and not
stored in a file, though the latter is possible of course. That is because
the server is intended for embedded platforms, which may have no disk space
or it could be undesirable to write anything on the disk.

For the same reason the content should be fed to a state machine in pieces
that fit into the outgoing buffer. This is because the server does not
allocate memory dynamically by itself and because it is driven by the
buffer-ready event.

So the library for generating content should be able to work in the
corresponding manner. E.g. upon a GET request the content must be prepared
and its sending only initiated. Then another time on another context chunks
of the content are sequentially requested as they are sent away. I.e. the
thing must maintain internal state and not spilling all guts in one shot.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: RFC: markdown to HTML library

[Natasha Kerensikova]

On 2014-01-20, Dmitry A. Kazakov <mailbox@dmitry-kazakov.de> wrote:
> On Sun, 19 Jan 2014 22:26:04 +0000 (UTC), Natasha Kerensikova wrote:
>
>> Features request are welcome too, though I can't tell for now when I
>> will manage to look into them. Currently reStructuedText front-end and
>> and fully-configurable ODT back-end are on my radar.
>> 
>> I can get into the details of how it works internally, but I won't bore
>> you with it if it's not necessary.
>
> There are many different requirements depending on the usage, I guess.

Indeed, but it feels like I'm a lunatic doing crazy stuff that nobody
cares about or is interested in, so the only requirements that really
matter when I write code are mine.

Then when someone does not want to roll their own library, they compare
their requirements against what is provided in existing libraries they
know about. And the actual code and what it can handle is of no matter
when you can't make it to that list.

> Now to the way my HTTP server works. The content is generated and not
> stored in a file, though the latter is possible of course. That is because
> the server is intended for embedded platforms, which may have no disk space
> or it could be undesirable to write anything on the disk.
>
> For the same reason the content should be fed to a state machine in pieces
> that fit into the outgoing buffer. This is because the server does not
> allocate memory dynamically by itself and because it is driven by the
> buffer-ready event.
>
> So the library for generating content should be able to work in the
> corresponding manner. E.g. upon a GET request the content must be prepared
> and its sending only initiated. Then another time on another context chunks
> of the content are sequentially requested as they are sent away. I.e. the
> thing must maintain internal state and not spilling all guts in one shot.

In such a situation, I would strongly advise against Markdown in the
first place, because the language doesn't really lend itself to bounded
environments, with its references that could be explicitly absolutely
anywhere within the source, some language elements that require
unbounded look-ahead to be distinguished, etc.

So if were in charge of such a webserver, I would first try to negotiate
my way out of supporting Markdown.

If supporting Markdown were an absolute requirement, I would try to
process it outside of the webserver (on client side, on dedicated
unbounded side machines, etc).

And if supporting Markdown inside the bounded webserver were an abolute
requirement, I would try negotiate the removal of the most problematic
language features (URL references, lists of blocks elements, etc) and
with carefully chosen upper bounds it might be possible. However I'm
afraid that even then, it would be beyond what is achievable with my
current programming skills.


Now as far as my library goes outside of Markdown, there is nothing
structurally that prevents it from being totally bounded. So it is
certainly possible to write bounded front-ends and bounded back-ends
that can fit within your constraints.

The HTML back-end is actually not that far away, the only current
blocker is the mechanism to share the internal state between all
language element instances, which involves dynamic allocations. But with
well-written storage pools, I believe even this mechanism could be used
in your environments. So you would only lack a front-end to make use of
my library.


Thanks for the discussion,
Natasha