Re: S-expression I/O in Ada

["Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>]

On Wed, 11 Aug 2010 02:43:58 -0700 (PDT), Natacha Kerensikova wrote:

> On Aug 10, 5:46�pm, "Dmitry A. Kazakov" <mail...@dmitry-kazakov.de>
> wrote:

>> Why don't you send it at once?
> 
> As I said, I can't just insert the raw object in the stream,

You can encode it.

> I need at least to know its size.

No, you don't and you cannot, because the object's size has nothing to do
to the object representation in the encoding used for the stream. Consider
two systems using common vocabulary. Instead of sending objects you do the
ids asking the partner to construct an object for this id.

> For example, S-expressions small enough to fit in one packet can be
> easily transferred over UDP. S-expression parsers (or at least mine)
> handle well fragmented data (even when unevenly fragmented) but fail
> when data is missing or mis-ordered, which prevent large S-expressions
> to be simply spread over as many packets as needed. However one might
> solve this issue by adding a sequence number inside the UDP payload,
> along with a mechanism to re-send lost packet; however that would be
> (at least partially) re-inventing TCP.

You can recode/pack S-expressions into something else. That does not answer
the question why. And from you responses it becomes less and less clear
where. What is the intended S-expression implementation according to the
OSI classification? We tried almost all levels, it looks pretty much
everything and nothing, a Ding-an-sich.
 
>>>> What was the problem then?
>>
>>> The problem is to organize different objects inside a single file. S-
>>> expression standardize the organization and relations between objects,
>>> while something else has to be done beyond S-expression to turn
>>> objects into representations suitable to live in a file.
>>
>>> [...]
>>
>> Yes, I don't see how S-expression might help there. They do not add value,
>> because the work of serialization or equivalent to serialization is already
>> done while construction of the expression object.
> 
> There are two things that are to be serialized: objects into atoms,
> and relations between objects into S-expression-specific stuff. The S-
> expression object is an unserialized in-memory representation of
> relations between serialized representations of objects. The writing
> of an S-expression into a stream is the final serialization-of-
> relations stage.

It is #2? (see below)

>> There are two questions to discuss:
>>
>> 1. The external storage format: S-expressions vs. other
>> 2. Construction of an intermediate S-expression object in the memory
>>
>> You justified #1 by an argument to legacy. You cannot [re-]use that
>> argument for #2. (OK, since Ludovic had already done it, you could (:-))
> 
> I don't re-use that argument. And actually if you followed the
> description of my Sexp_Stream, I don't need a S-expression object in
> memory, I only need serialized representation of atoms. The rest can
> be directly send into a stream.

It is not #2, only #1?

>>>> Why do you need S-sequence in the memory, while dumping
>>>> objects directly into files as S-sequences (if you insist on having them)
>>>> is simpler, cleaner, thinner, faster.
>>
>>> Because I need to examine the S-sequence before writing it to disk, in
>>> order to have enough information to write S-expression metadata. At
>>> the very lest, I need to know the total size of the atom before
>>> allowing its first byte to be send into the file.
>>
>> That does not look like a stream! But this is again about abstraction
>> layers. Why do you care?
> 
> The "verbatim encoding" of an atom, which is the only one allowed in
> canonical representation of a S-expression, is defined as follow: a
> size, represented as the ASCII encoding of the decimal representation
> of the number of octets in the atom, without leading zero (therefore
> of variable length); followed by the ASCII character ':'; followed by
> the octet sequence of the atom.

And TCP requires checksums, port numbers, destination address etc. Do you
care of them writing a socket?

>>> That sounds like a very nice way of doing it. So in the most common
>>> case, there will still be a stream, provided by the platform-specific
>>> socket facilities, which will accept an array-of-octets, and said
>>> array would have to be created from objects by custom code, right?
>>
>> Yes, if TCP sockets is what you use. There is a hell of other protocols
>> even on the Ethernet, some of which are not stream-oriented.
> 
> But you were talking about Octet'Read and Octet'Write. Aren't these
> Ada Stream based stuff?

It is a stream interface. You can hang this interface on something that is
not a stream, e.g. a text file. (We are hobbling in circles because of
mixed abstractions.)

>> This is what caused the questions. Because if the problem is serialization,
>> then S-expression does not look good.
> 
> Why? Because it's a partial serialization?

Yes

> Because it serializes stuff you deem as useless?

Yes, it does a structure of nested brackets. It does not serialize objects,
you need a lot of meat on the carcass. So if the problem is serialization
(of objects), then S-expression yet is not a solution.

>> The PPN, a simple stack machine. Push arguments onto the stack, pop to
>> execute an operation. Push the results back. Repeat.
> 
> Does that allows to push an operation and its arguments, to have it
> executed by another operation?

Operation is executed, no matter by what. PPN is executable. Whatever you
want to be executed has to be expressed in PPN. The PPN itself can be
pushed onto the stack if you wanted (=a von Neumann machine). I don't know
why would you need that stuff (~trampoline), however.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de