protobuff for Ada

protobuff for Ada

[Per Sandberg]

I managed to resurrect an old master thesis work that was done by Niklas 
Ekendahl in 2013 and put it on https://github.com/persan/protobuf-ada
the plan is to get it in working shape.

/P

Re: protobuff for Ada

[Shark8]

Cool!
More libs, bindings, and implementations in Ada is a good thing.

Though, it should be noted that ASN.1 is *probably* the better technology in cases where ProtoBufs are being considered:
http://ttsiodras.github.io/asn1.html

Re: protobuff for Ada

[Dmitry A. Kazakov]

On 2018-12-29 06:53, Shark8 wrote:

> Though, it should be noted that ASN.1 is *probably* the better technology in cases where ProtoBufs are being considered:
> http://ttsiodras.github.io/asn1.html

Sorry to disappoint you in this festive time, but this approach has the 
same fundamental flaw as prepared SQL statements do. You have to bind 
native Ada objects to protocol/serialized/persistent objects forth and 
back. This does not work well in practice. In fact, it barely work at 
all considering the overhead and hazards of type conversions.

A different approach is Ada's representation clauses which describe both 
objects same. Beyond simple textbook cases that does not work either.

The best practical method so far is using manually written stream 
attributes. Unfortunately it has shortcomings too:

1. Reuse is limited and composition is unsafe because stream attributes 
are non-primitive operations.

2. Introspection is almost non-existed. Only tagged types could have it.

3. No support of error handling and versioning. Though it is possible to 
do manually that is extremely error-prone and totally lacks static 
verification when the number of test cases is huge to potentially 
infinite. Even worse, the offending cases do not show up in a normally 
functioning system. So, when detected, it is always too late.

P.S. Needless to say, the problems 1-3 fully apply to other two methods 
as well.

P.P.S. And nothing was said about referential and recursive types...

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: protobuff for Ada

[joakimds]

Den fredag 28 december 2018 kl. 19:57:43 UTC+1 skrev Per Sandberg:
> I managed to resurrect an old master thesis work that was done by Niklas 
> Ekendahl in 2013 and put it on https://github.com/persan/protobuf-ada
> the plan is to get it in working shape.
> 
> /P

Hi Per! Great news and thanks for uploading!

When running ./autogen.sh I get the error message:
Could not find source code.  Make sure you are running this script from the
root of the distribution tree.

Is there source code missing in the repository? A missing /src directory?

Best regards,
Joakim

Re: protobuff for Ada

[Per Sandberg]

Joakim its still in non working alpha state and should not work.
/P

On 12/29/18 2:01 PM, joakimds@kth.se wrote:
> Den fredag 28 december 2018 kl. 19:57:43 UTC+1 skrev Per Sandberg:
>> I managed to resurrect an old master thesis work that was done by Niklas
>> Ekendahl in 2013 and put it on https://github.com/persan/protobuf-ada
>> the plan is to get it in working shape.
>>
>> /P
> 
> Hi Per! Great news and thanks for uploading!
> 
> When running ./autogen.sh I get the error message:
> Could not find source code.  Make sure you are running this script from the
> root of the distribution tree.
> 
> Is there source code missing in the repository? A missing /src directory?
> 
> Best regards,
> Joakim
> 

Re: protobuff for Ada

[joakimds]

> Joakim its still in non working alpha state and should not work.

Hi Per!

I have taken a look at this and the result can be found at:
https://github.com/joakim-strandberg/temp_protobuf

As you remember it wasn't possible to execute "./autogen.sh" without error message. And the error message indicated that the "src/" directory was missing. To improve on this situation I cloned the repository

https://github.com/protocolbuffers/protobuf

which contains the latest version of the official C++ protocol buffer compiler and took the "src/" directory. In addition I've taken the latest version of "autogen.sh", "configure.ac", "Makefile.am" and "Makefile.in". Also I took the directories "benchmarks" and "conformance" since these directories were non-existant in the 2013 version of the protobuf implementation for Ada. In the 2013 protobuf repository the contents of directory "m4" was deleted and then replenished with the the latest versions from the official C++ repository.
With these changes it was then possible to run "./autogen.sh" without error and then successfully run "make".

Next step after this was to compile the C++ code in the "compiler" directory by executing "gprbuild -p -P protoc_ada.gpr". This was not possible due to compilation errors. To fix the compilation errors the following changes were done to the C++ code:

scoped_ptr --> std::unique_ptr
scoped_array --> std::vector
map --> std::map
numeric_limits --> std::numeric_limits

It seems the protocol buffer C++ code from 2013 had a dependency upon Boost  which has been removed. With the changes in place it is possible to compile the C++ code. The protoc_ada.gpr file also needed adjustments to be able to link against libprotoc and libprotobuf that are created when running make above.
I hope this feedback will help accelerate the Ada binding from "non working alpha state" into a "working alpha state".

Best regards,
Joakim

Re: protobuff for Ada

[joakimds]

> Joakim its still in non working alpha state and should not work.

Hi Per!

I have taken a look at this and the result can be found at:
https://github.com/joakim-strandberg/temp_protobuf

As you remember it wasn't possible to execute "./autogen.sh" without error message. And the error message indicated that the "src/" directory was missing. To improve on this situation I cloned the repository

https://github.com/protocolbuffers/protobuf

which contains the latest version of the official C++ protocol buffer compiler and took the "src/" directory. In addition I've taken the latest version of "autogen.sh", "configure.ac", "Makefile.am" and "Makefile.in". Also I took the directories "benchmarks" and "conformance" since these directories were non-existant in the 2013 version of the protobuf implementation for Ada. In the 2013 protobuf repository the contents of directory "m4" was deleted and then replenished with the the latest versions from the official C++ repository.
With these changes it was then possible to run "./autogen.sh" without error and then successfully run "make".

Next step after this was to compile the C++ code in the "compiler" directory by executing "gprbuild -p -P protoc_ada.gpr". This was not possible due to compilation errors. To fix the compilation errors the following changes were done to the C++ code:

scoped_ptr --> std::unique_ptr
scoped_array --> std::vector
map --> std::map
numeric_limits --> std::numeric_limits

It seems the protocol buffer C++ code from 2013 had a dependency upon Boost  which has been removed. With the changes in place it is possible to compile the C++ code. The protoc_ada.gpr file also needed adjustments to be able to link against libprotoc and libprotobuf that are created in the "src/.libs" directory when running make above. I hope this feedback will help accelerate the Ada binding from "non working alpha state" into a "working alpha state".

Best regards,
Joakim

Re: protobuff for Ada

[Per Sandberg]

The repo is in heavy flux and shall not build yet.
/P

On 12/30/18 1:28 AM, joakimds@kth.se wrote:
>> Joakim its still in non working alpha state and should not work.
> 
> Hi Per!
> 
> I have taken a look at this and the result can be found at:
> https://github.com/joakim-strandberg/temp_protobuf
> 
> As you remember it wasn't possible to execute "./autogen.sh" without error message. And the error message indicated that the "src/" directory was missing. To improve on this situation I cloned the repository
> 
> https://github.com/protocolbuffers/protobuf
> 
> which contains the latest version of the official C++ protocol buffer compiler and took the "src/" directory. In addition I've taken the latest version of "autogen.sh", "configure.ac", "Makefile.am" and "Makefile.in". Also I took the directories "benchmarks" and "conformance" since these directories were non-existant in the 2013 version of the protobuf implementation for Ada. In the 2013 protobuf repository the contents of directory "m4" was deleted and then replenished with the the latest versions from the official C++ repository.
> With these changes it was then possible to run "./autogen.sh" without error and then successfully run "make".
> 
> Next step after this was to compile the C++ code in the "compiler" directory by executing "gprbuild -p -P protoc_ada.gpr". This was not possible due to compilation errors. To fix the compilation errors the following changes were done to the C++ code:
> 
> scoped_ptr --> std::unique_ptr
> scoped_array --> std::vector
> map --> std::map
> numeric_limits --> std::numeric_limits
> 
> It seems the protocol buffer C++ code from 2013 had a dependency upon Boost  which has been removed. With the changes in place it is possible to compile the C++ code. The protoc_ada.gpr file also needed adjustments to be able to link against libprotoc and libprotobuf that are created in the "src/.libs" directory when running make above. I hope this feedback will help accelerate the Ada binding from "non working alpha state" into a "working alpha state".
> 
> Best regards,
> Joakim
> 

Re: protobuff for Ada

[Olivier Henley]

Interesting. I do not grasp the problem in full though...

When you say "Sorry to disappoint you in this festive time", do you mean trying a solution from ASN.1 or only trying at Protobuff?

I think I get why a Protobuff could not cover "complete" transfer of Ada types around, but how does other languages do? (Almost everyone has it) Some of these languages have relatively "complex" type system..? 

How do they achieve it? They express any complex types with a limited subset of primitive types(string, int32, etc)?

Can you give a more pragmatic example that exemplifies the limitations in Ada?

Thx,

olivier 

Re: protobuff for Ada

[Dmitry A. Kazakov]

On 2018-12-31 17:55, Olivier Henley wrote:
> Interesting. I do not grasp the problem in full though...
> 
> When you say "Sorry to disappoint you in this festive time", do you mean trying a solution from ASN.1 or only trying at Protobuff?

Both. They are useless, up to harmful.

> I think I get why a Protobuff could not cover "complete" transfer of Ada types around, but how does other languages do? (Almost everyone has it) Some of these languages have relatively "complex" type system..?

The very concept of a data definition/description language (DDL) is 
wrong as I tried to explain. It has a very long and sad history in 
process automation, control, communication (e.g. CORBA), databases (e.g. 
SQL). Almost everybody and everyone tried it and failed. There are 
countless protocol describing "languages" around in process automation. 
I fought with them for decades, wrote several compilers for this mess. 
One could save huge amount of money and time if there were a law to 
punish people introducing this stuff... (:-))

> How do they achieve it? They express any complex types with a limited subset of primitive types(string, int32, etc)?

You cannot express a type in a DDL. Data /= Type. Type = data + 
operations. If you want to express complex typed objects you lose before 
you start with a DDL. You throw all type semantics overboard.

*If* you are OK without semantics then there is no need to introduce 
this mess. Use Ada stream attributes and simply read and write what you 
want and how you want. It is clean, easy, fast and 100% Ada.

> Can you give a more pragmatic example that exemplifies the limitations in Ada?

Any limitations Ada might have are unrelated to the issue of language 
impedance: DDL vs Ada unless you make DDL embedded like embedded SQL, 
which does not work either.

I believe AdaCore has a product of the sort. Though I don't think that 
would be much better, but I would rather trust them than anybody else...

Happy New Year!

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: protobuff for Ada

[Per Sandberg]

Well

 From my perspective absolutely biggest flaw with technologies like 
protobuf is:
* Its backed by a large corporation.
* The technology is well known.
* 99.9% of the programming population think that they are the salvation 
to serialization.
* The licensing is open.

And on top.
* The are significantly more then one project where the lack of protobuf 
support has ruled out Ada as implementation technology.

/And my intent was to eliminate at least the last points even if the 
technology is inferior.

/P


On 12/29/18 12:05 PM, Dmitry A. Kazakov wrote:
> On 2018-12-29 06:53, Shark8 wrote:
> 
>> Though, it should be noted that ASN.1 is *probably* the better 
>> technology in cases where ProtoBufs are being considered:
>> http://ttsiodras.github.io/asn1.html
> 
> Sorry to disappoint you in this festive time, but this approach has the 
> same fundamental flaw as prepared SQL statements do. You have to bind 
> native Ada objects to protocol/serialized/persistent objects forth and 
> back. This does not work well in practice. In fact, it barely work at 
> all considering the overhead and hazards of type conversions.
> 
> A different approach is Ada's representation clauses which describe both 
> objects same. Beyond simple textbook cases that does not work either.
> 
> The best practical method so far is using manually written stream 
> attributes. Unfortunately it has shortcomings too:
> 
> 1. Reuse is limited and composition is unsafe because stream attributes 
> are non-primitive operations.
> 
> 2. Introspection is almost non-existed. Only tagged types could have it.
> 
> 3. No support of error handling and versioning. Though it is possible to 
> do manually that is extremely error-prone and totally lacks static 
> verification when the number of test cases is huge to potentially 
> infinite. Even worse, the offending cases do not show up in a normally 
> functioning system. So, when detected, it is always too late.
> 
> P.S. Needless to say, the problems 1-3 fully apply to other two methods 
> as well.
> 
> P.P.S. And nothing was said about referential and recursive types...
> 

Re: protobuff for Ada

[Olivier Henley]

Thx for the clear answer.

> One could save huge amount of money and time if there were a law to 
> punish people introducing this stuff... (:-))

LOL

> Happy New Year!


Happy New Year sir! :)

Re: protobuff for Ada

[Olivier Henley]

> * Its backed by a large corporation.

true

> * The technology is well known.

true

> * 99.9% of the programming population think that they are the salvation 
> to serialization.

true, until they switch to Cap'N Proto ... or any other one. We may be good for 10 years though.

> And on top.
> * The are significantly more then one project where the lack of protobuf 
> support has ruled out Ada as implementation technology.

very true (grpc which would give a low hanging DGraph(https://dgraph.io/) Ada client)

> /And my intent was to eliminate at least the last points even if the 
> technology is inferior.

+1

Re: protobuff for Ada

[G. B.]

Dmitry A. Kazakov <mailbox@dmitry-kazakov.de> wrote:

> *If* you are OK without semantics then there is no need to introduce 
> this mess. Use Ada stream attributes and simply read and write what you 
> want and how you want. It is clean, easy, fast and 100% Ada.

What kind of stream do you write for your partners in business? Three of
them have different needs than you WRT data and none of them is using Ada.

Re: protobuff for Ada

[Dmitry A. Kazakov]

On 2019-01-02 07:57, G. B. wrote:
> Dmitry A. Kazakov <mailbox@dmitry-kazakov.de> wrote:
> 
>> *If* you are OK without semantics then there is no need to introduce
>> this mess. Use Ada stream attributes and simply read and write what you
>> want and how you want. It is clean, easy, fast and 100% Ada.
> 
> What kind of stream do you write for your partners in business?

Stream of octets.

> Three of
> them have different needs than you WRT data and none of them is using Ada.

They still can read and write the stream. You are confusing description 
of a protocol with the implementation of.

The OP suggested having descriptions in protobuff and partial 
implementation generated from that. It is a bad idea.

-------------------------------
BTW, it is very easy to write things like protobuff straight in Ada with 
Simple Components

    http://www.dmitry-kazakov.de/ada/components.htm#17.2.1

This feature is rarely used because, as I said, the concept is too 
limited and constraining if not wrong altogether.

Here is a small example. Consider an example in protobuff:

    message Person {
       required string name = 1;
       required int32 id = 2;
       optional string email = 3;
    }

This direct Ada code:

    type Person is new State_Machine with
       Name  : String_Data_Item (Max_String_Length);
       ID    : Unsigned_32_Data_Item;
       Email : String_Data_Item (Max_String_Length);
    end record;

Thanks to Ada's "introspection" that is all. It will be read or written 
by the connections server automatically. On the packet receipt callback, 
you get values like Person_Session.ID.Value. Before sending a new packet 
you assign Person_Session.ID.Value. Note, this is Ada 95, no fancy stuff.

I didn't show here alternation for using optional fields because the 
transport level representation would be different anyway. Which is the 
point actually. Such key details are all left unspecified in the 
protobuff "description" above along with endianness and other encoding 
issues. Yet exactly these details are essential in practice where the 
protocol is already defined. Present or not bits might kept combined in 
the message header, special values of integers are reserved to indicate 
exceptional states and so on and so forth. And, again, no semantics 
whatsoever, just buckets of bits.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de