Re: S-expression I/O in Ada

[Natacha Kerensikova <lithiumcat@gmail.com>]

On Aug 11, 12:37 pm, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
wrote:
> On Wed, 11 Aug 2010 02:43:58 -0700 (PDT), Natacha Kerensikova wrote:
> > On Aug 10, 5:46 pm, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
> > wrote:
> >> Why don't you send it at once?
>
> > As I said, I can't just insert the raw object in the stream,
>
> You can encode it.

Indeed, but to encode anything I need to access it, i.e. having it
live in the RAM. That's exactly what my in-memory serialized are for.

> > I need at least to know its size.
>
> No, you don't and you cannot, because the object's size has nothing to do
> to the object representation in the encoding used for the stream. Consider
> two systems using common vocabulary. Instead of sending objects you do the
> ids asking the partner to construct an object for this id.

I meant the size of the serialized representation. I can't store or
transmit something when I don't know the size of what is to be stored
or to be transmitted. If you want to transfer id instead of objects,
fine, then the id is the transmitted object and I have to know its
size. Using id is just moving the problem without solving it.

>  What is the intended S-expression implementation according to the
> OSI classification?

The primary intended use of this implementation is disk file I/O,
which doesn't fit well into the OSI classification.

When I used its C counterpart over a network, it has always been over
stream sockets (UNIX stream sockets or TCP sockets), so it's clearly
above the transport layer. I've never been able to figure out what is
what on the level 5 and above, but that's where my implementation
would be.

Now real-life S-expressions are used as a part of SPKI and IMAP. I
think IMAP is on level 7, but I'm not that tells us where IMAP S-
expressions are; and I haven't found enough information about SPKI to
know where it is, but from its purpose I guess it's probably also on
level 5 or above.

I don't know how this information helps you though.

> >>>> What was the problem then?
>
> >>> The problem is to organize different objects inside a single file. S-
> >>> expression standardize the organization and relations between objects,
> >>> while something else has to be done beyond S-expression to turn
> >>> objects into representations suitable to live in a file.
>
> >>> [...]
>
> >> Yes, I don't see how S-expression might help there. They do not add value,
> >> because the work of serialization or equivalent to serialization is already
> >> done while construction of the expression object.
>
> > There are two things that are to be serialized: objects into atoms,
> > and relations between objects into S-expression-specific stuff. The S-
> > expression object is an unserialized in-memory representation of
> > relations between serialized representations of objects. The writing
> > of an S-expression into a stream is the final serialization-of-
> > relations stage.
>
> It is #2? (see below)

When we are talking about a S-expression object, considering I
understand the word "object" as something that lives in RAM, then yes
we are in #2.

But this isn't required: my proposed Sexp_Stream never actually
construct a S-expression object, it serializes S-expression stuff
directly into the underlying stream, around the provided
serializations that are atoms.

> >> There are two questions to discuss:
>
> >> 1. The external storage format: S-expressions vs. other
> >> 2. Construction of an intermediate S-expression object in the memory
>
> >> You justified #1 by an argument to legacy. You cannot [re-]use that
> >> argument for #2. (OK, since Ludovic had already done it, you could (:-))
>
> > I don't re-use that argument. And actually if you followed the
> > description of my Sexp_Stream, I don't need a S-expression object in
> > memory, I only need serialized representation of atoms. The rest can
> > be directly send into a stream.
>
> It is not #2, only #1?

#1 using S-expression storage format is a choice driven by existing
data being in that format and personal taste.

#2 constructing a S-expression object in memory, is something that may
or may not be needed, depending on the application. Parsing a
configuration like my tcp-connect example does not, hence my
proposition of Sexp_Stream which functions without S-expression
objects in memory. My S-expression templates still seem (to me) to
require a S-expression object in memory (to be handed around to the
correct handler after dispatch).

When I proposed my Sexp_Stream, I postponed the design of a S-
expression object library, because it would be built upon the
Sexp_Stream which isn't fixed yet. And depending on Ludovic Brenta's
progress, I might never design my S-expression object and re-use his.

> >>>> Why do you need S-sequence in the memory, while dumping
> >>>> objects directly into files as S-sequences (if you insist on having them)
> >>>> is simpler, cleaner, thinner, faster.
>
> >>> Because I need to examine the S-sequence before writing it to disk, in
> >>> order to have enough information to write S-expression metadata. At
> >>> the very lest, I need to know the total size of the atom before
> >>> allowing its first byte to be send into the file.
>
> >> That does not look like a stream! But this is again about abstraction
> >> layers. Why do you care?
>
> > The "verbatim encoding" of an atom, which is the only one allowed in
> > canonical representation of a S-expression, is defined as follow: a
> > size, represented as the ASCII encoding of the decimal representation
> > of the number of octets in the atom, without leading zero (therefore
> > of variable length); followed by the ASCII character ':'; followed by
> > the octet sequence of the atom.
>
> And TCP requires checksums, port numbers, destination address etc. Do you
> care of them writing a socket?

I would if I were implementing a TCP library based on something of
lower OSI-level.

Now I'm writing a S-expression library based on TCP sockets (or disk
streams), so I don't care about TCP details. However I do care about S-
expression details, among which is the "verbatim encoding".

> >>> That sounds like a very nice way of doing it. So in the most common
> >>> case, there will still be a stream, provided by the platform-specific
> >>> socket facilities, which will accept an array-of-octets, and said
> >>> array would have to be created from objects by custom code, right?
>
> >> Yes, if TCP sockets is what you use. There is a hell of other protocols
> >> even on the Ethernet, some of which are not stream-oriented.
>
> > But you were talking about Octet'Read and Octet'Write. Aren't these
> > Ada Stream based stuff?
>
> It is a stream interface. You can hang this interface on something that is
> not a stream, e.g. a text file. (We are hobbling in circles because of
> mixed abstractions.)

And what do you deduce from this?

In the fourth level of quote above, I'm basically asking whether I
correctly understood your idea by reformulating it and asking for
confirmation ("…, right?").

Third level of quote, you seem to agree with a restriction.

Second level of quote, I try to understand the restriction.

And now I still don't know whether the idea in the fourth quote level
is right or wrong, and if it's wrong what it needs to be set right.
That's indeed a sad circle, and I don't even have a clue of what mixed
abstractions have to do with the circle. That's almost the point where
I stop trying to understand and just give up everything, Ada looks way
above my brain's capabilities.

> >> This is what caused the questions. Because if the problem is serialization,
> >> then S-expression does not look good.
>
> > Why? Because it's a partial serialization?
>
> Yes
>
> > Because it serializes stuff you deem as useless?
>
> Yes, it does a structure of nested brackets. It does not serialize objects,
> you need a lot of meat on the carcass. So if the problem is serialization
> (of objects), then S-expression yet is not a solution.

Indeed, because S-expressions assume the serialization of objects
(into atoms) is already solved, and address the problem of serializing
a higher level of information about these objects.

> Operation is executed, no matter by what. PPN is executable. Whatever you
> want to be executed has to be expressed in PPN. The PPN itself can be
> pushed onto the stack if you wanted (=a von Neumann machine). I don't know
> why would you need that stuff (~trampoline), however.

I still haven't found what PPN is, but from your description it looks
similar to my S-expression with semantics provided by an interpreter.

What I need can be basically examplified as the following S-expression
being sent to a list-of-object objects:
("<h1>Contents of " (title) " list</h1>"
    (for-each-item "<h2>" (title) "</h2><p>" (text) "</p>"))

In the first list line, atoms (double-quoted) are taken as-is into the
output HTML stream. (title) is a call to the list's "title" function,
which is substituted by that particular list's title. Then the "for-
each-item" function is called, which sends its arguments to each of
item of the list. Here again, following atoms are copied as-is, and
this time (title) is interpreted by item objects and substituted by
the item's title, and then the item's text substitutes (text).

That doesn't sound like an alien use of operations being passed as an
argument to another operation.



Regards,
Natacha