HTTP with Simple Components: Status.Kind always File

HTTP with Simple Components: Status.Kind always File

[Felix Krause]

I am trying to implement an HTTP server with Simple Components. My 
problem is that whenever I send a request to the server, in Do_Get, 
Status.Kind is File instead of URI. Relevant parts of my code:

    package Yaml.Servers is
       package HTTP renames GNAT.Sockets.Connection_State_Machine.HTTP_Server;
       package Server renames GNAT.Sockets.Server;

       type Yaml_Factory (Request_Length  : Positive;
                          Input_Size      : Server.Buffer_Length;
                          Output_Size     : Server.Buffer_Length;
                          Max_Connections : Positive) is
         new Server.Connections_Factory with null record;

       type Yaml_Client is new HTTP.HTTP_Client with null record;

       overriding function Create (Factory  : access Yaml_Factory;
                                   Listener : access 
Server.Connections_Server'Class;
                                   From     : GNAT.Sockets.Sock_Addr_Type)
                                   return Server.Connection_Ptr is
          (new Yaml_Client (Listener       => Listener.all'Unchecked_Access,
                            Request_Length => Factory.Request_Length,
                            Input_Size     => Factory.Input_Size,
                            Output_Size    => Factory.Output_Size));

       overriding procedure Do_Get (Client : in out Yaml_Client);
    end Yaml.Servers;

    package body Yaml.Servers is
       procedure Do_Get (Client : in out Yaml_Client) is
          Status : HTTP.Status_Line renames Get_Status_Line (Client);
       begin
          case Status.Kind is
             when HTTP.None =>
                ...
             when HTTP.File =>
                ...
             when HTTP.URI =>
                ...
          end case;
       end Do_Get;
    end Yaml.Servers;

    procedure Yaml.Server is
       Factory : aliased Servers.Yaml_Factory (200, 80, 8192, 100);
       Server  : Servers.Server.Connections_Server (Factory'Access, 8088);
       pragma Unreferenced (Server);
    begin
       Ada.Text_IO.Put_Line ("HTTP server started.");
       loop
          delay 60.0;
       end loop;
    end Yaml.Server;

Now I run this server behind an nginx, and it responds as expected. 
However, when I access it with an URI `http://ada.yaml.io/widget/`, 
inside Do_Get, Status.Kind is File. Why is this? With the File kind, I 
cannot possibly parse the query at this point, because I cannot 
differentiate between `%26` and `&` (entities have already been 
resolved).

I also tested this locally with `http://localhost:8888/widget/`, and 
that also yields a file.

How can I make the HTTP server to yield an URI instead of a File?

Cheers,
Felix

Re: HTTP with Simple Components: Status.Kind always File

[Dmitry A. Kazakov]

On 2017-07-17 20:30, Felix Krause wrote:
> I am trying to implement an HTTP server with Simple Components. My 
> problem is that whenever I send a request to the server, in Do_Get, 
> Status.Kind is File instead of URI. Relevant parts of my code:
> 
>     package Yaml.Servers is
>        package HTTP renames 
> GNAT.Sockets.Connection_State_Machine.HTTP_Server;
>        package Server renames GNAT.Sockets.Server;
> 
>        type Yaml_Factory (Request_Length  : Positive;
>                           Input_Size      : Server.Buffer_Length;
>                           Output_Size     : Server.Buffer_Length;
>                           Max_Connections : Positive) is
>          new Server.Connections_Factory with null record;
> 
>        type Yaml_Client is new HTTP.HTTP_Client with null record;
> 
>        overriding function Create (Factory  : access Yaml_Factory;
>                                    Listener : access 
> Server.Connections_Server'Class;
>                                    From     : GNAT.Sockets.Sock_Addr_Type)
>                                    return Server.Connection_Ptr is
>           (new Yaml_Client (Listener       => 
> Listener.all'Unchecked_Access,
>                             Request_Length => Factory.Request_Length,
>                             Input_Size     => Factory.Input_Size,
>                             Output_Size    => Factory.Output_Size));
> 
>        overriding procedure Do_Get (Client : in out Yaml_Client);
>     end Yaml.Servers;
> 
>     package body Yaml.Servers is
>        procedure Do_Get (Client : in out Yaml_Client) is
>           Status : HTTP.Status_Line renames Get_Status_Line (Client);
>        begin
>           case Status.Kind is
>              when HTTP.None =>
>                 ...
>              when HTTP.File =>
>                 ...
>              when HTTP.URI =>
>                 ...
>           end case;
>        end Do_Get;
>     end Yaml.Servers;
> 
>     procedure Yaml.Server is
>        Factory : aliased Servers.Yaml_Factory (200, 80, 8192, 100);
>        Server  : Servers.Server.Connections_Server (Factory'Access, 8088);
>        pragma Unreferenced (Server);
>     begin
>        Ada.Text_IO.Put_Line ("HTTP server started.");
>        loop
>           delay 60.0;
>        end loop;
>     end Yaml.Server;
> 
> Now I run this server behind an nginx, and it responds as expected. 
> However, when I access it with an URI `http://ada.yaml.io/widget/`, 
> inside Do_Get, Status.Kind is File. Why is this? With the File kind, I 
> cannot possibly parse the query at this point, because I cannot 
> differentiate between `%26` and `&` (entities have already been resolved).
> 
> I also tested this locally with `http://localhost:8888/widget/`, and 
> that also yields a file.
> 
> How can I make the HTTP server to yield an URI instead of a File?

If I correctly understand the issue, the HTTP request line must contain 
a schema.

If you take a look into the source, parsing gets the method (e.g. GET) 
then it checks (gnat-sockets-connection_state_machine-http_server.adb, 
1540):

    if Request (Pointer) = '*' then
       ...
    elsif Request (Pointer) = '/' then
       ...
    else
       ...
    end if;

The first does nothing, the second creates Kind => File, the third 
creates Kind => URI. So Kind = File should be when the request like 
looked like:

    GET /something/somewhere...

URI when:

    GET http://www.something/somewhere...

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: HTTP with Simple Components: Status.Kind always File

[Felix Krause]

On 2017-07-17 20:59:32 +0000, Dmitry A. Kazakov said:
> 
> If I correctly understand the issue, the HTTP request line must contain 
> a schema.
> 
> If you take a look into the source, parsing gets the method (e.g. GET) 
> then it checks (gnat-sockets-connection_state_machine-http_server.adb, 
> 1540):
> 
>     if Request (Pointer) = '*' then
>        ...
>     elsif Request (Pointer) = '/' then
>        ...
>     else
>        ...
>     end if;
> 
> The first does nothing, the second creates Kind => File, the third 
> creates Kind => URI. So Kind = File should be when the request like 
> looked like:
> 
>     GET /something/somewhere...
> 
> URI when:
> 
>     GET http://www.something/somewhere...

Well, this is the behavior I expect, but I *am* querying the server 
with an URI containing a schema. For example:

    curl http://localhost:8088/

This still yields a File as Status.Kind. This is what bothers me.

Re: HTTP with Simple Components: Status.Kind always File

[Dmitry A. Kazakov]

On 2017-07-19 18:09, Felix Krause wrote:
> On 2017-07-17 20:59:32 +0000, Dmitry A. Kazakov said:
>>
>> If I correctly understand the issue, the HTTP request line must 
>> contain a schema.
>>
>> If you take a look into the source, parsing gets the method (e.g. GET) 
>> then it checks (gnat-sockets-connection_state_machine-http_server.adb, 
>> 1540):
>>
>>     if Request (Pointer) = '*' then
>>        ...
>>     elsif Request (Pointer) = '/' then
>>        ...
>>     else
>>        ...
>>     end if;
>>
>> The first does nothing, the second creates Kind => File, the third 
>> creates Kind => URI. So Kind = File should be when the request like 
>> looked like:
>>
>>     GET /something/somewhere...
>>
>> URI when:
>>
>>     GET http://www.something/somewhere...
> 
> Well, this is the behavior I expect, but I *am* querying the server with 
> an URI containing a schema.

How do you know? How exactly looks the request? If you don't trust the 
integrated trace you can use Wireshark to be sure.

> For example:
> 
>     curl http://localhost:8088/
> 
> This still yields a File as Status.Kind.

I must see what is sent from the client in order to tell if it is a bug 
or correct behavior. A command line tells nothing.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: HTTP with Simple Components: Status.Kind always File

[Randy Brukardt]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> wrote in message 
news:oko1kk$1pf7$1@gioia.aioe.org...
> On 2017-07-19 18:09, Felix Krause wrote:
...
>> For example:
>>
>>     curl http://localhost:8088/
>>
>> This still yields a File as Status.Kind.
>
> I must see what is sent from the client in order to tell if it is a bug or 
> correct behavior. A command line tells nothing.

Right, because many programs read the schema to determine how to contact the 
server, then remove it (because it was used). Indeed, it's a bit unusual to 
pass the schema to be passed to a GET command (although it happens enough 
that my code in Ada Server is prepared to deal with it); for HTTP 1.1, the 
domain ("host") is passed separately so one typically only passes the URI 
path (and not the domain or schema).

                                            Randy.

Re: HTTP with Simple Components: Status.Kind always File

[Felix Krause]

On 2017-07-19 16:33:15 +0000, Dmitry A. Kazakov said:

> On 2017-07-19 18:09, Felix Krause wrote:
>> 
>> Well, this is the behavior I expect, but I *am* querying the server 
>> with an URI containing a schema.
> 
> How do you know? How exactly looks the request? If you don't trust the 
> integrated trace you can use Wireshark to be sure.

I enabled the internal trace and in fact, there is no schema on the 
incoming GET.

> I must see what is sent from the client in order to tell if it is a bug 
> or correct behavior. A command line tells nothing.

In my view, it is not correct behavior that Status.Kind is File when 
the request has been sent via HTTP, regardless of whether the schema is 
part of GET URI or not. As I explained earlier, for a File, I only get 
the Path string. I cannot parse this properly since HTTP server already 
unescaped the escaping sequences. Let me give an example:

    curl "http://localhost:8088/foo?key=value"

    curl "http://localhost:8088/foo%3Fkey=value"

The first command should fetch a resource with the path `/foo` and the 
query `?key=value`. The second command should fetch a resource with the 
path `/foo?key=value`. However, I do not see the difference in the Path 
string I get from the HTTP server, which makes me unable to 
differentiate between those two cases. Thus, the HTTP server should 
give me an URI instead, even if the path on the GET line does not 
contain a scheme.

Re: HTTP with Simple Components: Status.Kind always File

[Dmitry A. Kazakov]

On 19/07/2017 23:13, Felix Krause wrote:
> On 2017-07-19 16:33:15 +0000, Dmitry A. Kazakov said:
> 
>> On 2017-07-19 18:09, Felix Krause wrote:
>>>
>>> Well, this is the behavior I expect, but I *am* querying the server 
>>> with an URI containing a schema.
>>
>> How do you know? How exactly looks the request? If you don't trust the 
>> integrated trace you can use Wireshark to be sure.
> 
> I enabled the internal trace and in fact, there is no schema on the 
> incoming GET.

See, there is no bug so far.

>> I must see what is sent from the client in order to tell if it is a 
>> bug or correct behavior. A command line tells nothing.
> 
> In my view, it is not correct behavior that Status.Kind is File when the 
> request has been sent via HTTP, regardless of whether the schema is part 
> of GET URI or not.

Well, it processes what the client sends.

> As I explained earlier, for a File, I only get the 
> Path string. I cannot parse this properly since HTTP server already 
> unescaped the escaping sequences. Let me give an example:
> 
>     curl "http://localhost:8088/foo?key=value"
> 
>     curl "http://localhost:8088/foo%3Fkey=value"

I see, you want to the recognize the query part (and possibly the 
fragment part) even when no scheme present.

I think what you want is illegal. I might be wrong, people claim ARM is 
complicated, they should read RFCs!

Anyway, my interpretation of RFC 3986 is that the scheme part must be 
present:

    https://tools.ietf.org/html/rfc3986#section-1.1.1

No scheme, no query part. Therefore in both

    GET /foo?key=value
    GET foo%3Fkey=value

"foo?key=value" must be the path.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: HTTP with Simple Components: Status.Kind always File

[Felix Krause]

On 2017-07-20 08:57:57 +0000, Dmitry A. Kazakov said:
> 
>> As I explained earlier, for a File, I only get the Path string. I 
>> cannot parse this properly since HTTP server already unescaped the 
>> escaping sequences. Let me give an example:
>> 
>> curl "http://localhost:8088/foo?key=value"
>> 
>> curl "http://localhost:8088/foo%3Fkey=value"
> 
> I see, you want to the recognize the query part (and possibly the 
> fragment part) even when no scheme present.
> 
> I think what you want is illegal. I might be wrong, people claim ARM is 
> complicated, they should read RFCs!
> 
> Anyway, my interpretation of RFC 3986 is that the scheme part must be present:
> 
>     https://tools.ietf.org/html/rfc3986#section-1.1.1
> 
> No scheme, no query part. Therefore in both
> 
>     GET /foo?key=value
>     GET foo%3Fkey=value
> 
> "foo?key=value" must be the path.

I read a bit into the HTTP 1.1 and the URI spec. I think your 
interpretation is correct with one minor error:

According to HTTP/1.1:

    https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html

The Request-URI of an HTTP request is defined as follows:

    Request-URI    = "*" | absoluteURI | abs_path | authority

"*" and authority are only for certain request types, so let's ignore 
them. So the URI can either be
an absoluteURI or and abs_path, both of which are defined in the URI 
spec you linked:

    absolute-URI  = scheme ":" hier-part [ "?" query ]

abs_path is declared obsolete in the URI spec; it defines that the 
translation of the obsolete rule is
path-absolute:

    path-absolute = "/" [ segment-nz *( "/" segment ) ]
    segment       = *pchar
    pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"

Since '?' is not part of unreserved nor sub-delims, it is, according to 
the spec, completely illegal in
a path. So,

    GET /foo?key=value

should be rejected as it is illegal syntax. But

    GET /foo%3Fkey=value

is legal since pchar may contain pct-encoded.


Now, my problem is that this argument is quite far away from reality. I 
tested various HTTP client
implementations (curl, Chrome, Firefox), and every one, when I tell it to get

    http://example.com/?key=value

sends the following Request-Line:

    GET /?key=value HTTP/1.1

So, HTTP clients do not seem to respect this part of the HTTP 
specification. That means for me that if
I want to build a server supporting the wide-spread HTTP clients, I 
need a server implementation that
supports this kind of GET request even though it violates the HTTP 
specification. I understand your
point that you want to conform to the spec, but this seems to mean that 
it will not work well with
existing HTTP clients.

My suggestion is thus to parse the query even if you get a path. As I 
see it, this will not directly
violate the spec. HTTP defines that:

    Servers SHOULD respond to invalid Request-URIs with an appropriate 
status code.

If a path contains a '?', it is an invalid Request-URI. The SHOULD 
allows you to handle this
differently without violating the spec.

I look forward to your opinion.

Cheers,
Felix

Re: HTTP with Simple Components: Status.Kind always File

[Dmitry A. Kazakov]

On 20/07/2017 15:35, Felix Krause wrote:

> My suggestion is thus to parse the query even if you get a path. As I 
> see it, this will not directly
> violate the spec. HTTP defines that:
> 
>     Servers SHOULD respond to invalid Request-URIs with an appropriate 
> status code.
> 
> If a path contains a '?', it is an invalid Request-URI. The SHOULD 
> allows you to handle this differently without violating the spec.
> 
> I look forward to your opinion.

How about this. I could add a Boolean parameter to control the behavior:

1. If False everything is as it is now;

2. If True and the path contains either '?' or '#' (something else?), it 
is parsed as if HTTP_Scheme was there.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: HTTP with Simple Components: Status.Kind always File

[Felix Krause]

On 2017-07-20 13:57:54 +0000, Dmitry A. Kazakov said:

> On 20/07/2017 15:35, Felix Krause wrote:
> 
>> My suggestion is thus to parse the query even if you get a path. As I 
>> see it, this will not directly
>> violate the spec. HTTP defines that:
>> 
>> Servers SHOULD respond to invalid Request-URIs with an appropriate status code.
>> 
>> If a path contains a '?', it is an invalid Request-URI. The SHOULD 
>> allows you to handle this differently without violating the spec.
>> 
>> I look forward to your opinion.
> 
> How about this. I could add a Boolean parameter to control the behavior:
> 
> 1. If False everything is as it is now;
> 
> 2. If True and the path contains either '?' or '#' (something else?), 
> it is parsed as if HTTP_Scheme was there.

This would be fully sufficient solution. I would be grateful if you added this.

Cheers,
Felix

Re: HTTP with Simple Components: Status.Kind always File

[Simon Wright]

Felix Krause <contact@flyx.org> writes:

> Now, my problem is that this argument is quite far away from
> reality. I tested various HTTP client
> implementations (curl, Chrome, Firefox), and every one, when I tell it to get
>
>    http://example.com/?key=value
>
> sends the following Request-Line:
>
>    GET /?key=value HTTP/1.1
>
> So, HTTP clients do not seem to respect this part of the HTTP
> specification.

The RFC that Dmitri quotes is for URIs. You want the one that defines
the HTTP protocol, currently RFC7230 - see section 3, Message Format.
https://tools.ietf.org/html/rfc7230#section-3

Re: HTTP with Simple Components: Status.Kind always File

[Dmitry A. Kazakov]

On 2017-07-20 22:29, Simon Wright wrote:
> Felix Krause <contact@flyx.org> writes:
> 
>> Now, my problem is that this argument is quite far away from
>> reality. I tested various HTTP client
>> implementations (curl, Chrome, Firefox), and every one, when I tell it to get
>>
>>     http://example.com/?key=value
>>
>> sends the following Request-Line:
>>
>>     GET /?key=value HTTP/1.1
>>
>> So, HTTP clients do not seem to respect this part of the HTTP
>> specification.
> 
> The RFC that Dmitri quotes is for URIs. You want the one that defines
> the HTTP protocol, currently RFC7230 - see section 3, Message Format.
> https://tools.ietf.org/html/rfc7230#section-3

So, is it OK to always recognize query part in the path. Correct?

What is the scheme by default? http? None?

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: HTTP with Simple Components: Status.Kind always File

[Felix Krause]

On 2017-07-20 20:57:35 +0000, Dmitry A. Kazakov said:

> On 2017-07-20 22:29, Simon Wright wrote:
>> 
>> The RFC that Dmitri quotes is for URIs. You want the one that defines
>> the HTTP protocol, currently RFC7230 - see section 3, Message Format.
>> https://tools.ietf.org/html/rfc7230#section-3

Ah, I looked at an older RFC for HTTP 1.1; I wasn't aware there was a newer
one. This newer one indeed defines query as part of the request target:

    https://tools.ietf.org/html/rfc7230#section-5.3

    request-target = origin-form
                    / absolute-form
                    / authority-form
                    / asterisk-form
    origin-form    = absolute-path [ "?" query ]
> 
> So, is it OK to always recognize query part in the path. Correct?

Yes, although to avoid confusion, it may be a good idea to not name it „Path“.
Reading the spec, „Origin“ might be more appropriate. But I guess that change
would break backwards compatibility.

> 
> What is the scheme by default? http? None?

There is no scheme in the origin-form. I would keep this separated from the
full URI and just supply Path and Query in the record.

Re: HTTP with Simple Components: Status.Kind always File

[Simon Wright]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

>> The RFC that Dmitri quotes is for URIs. You want the one that defines
>> the HTTP protocol, currently RFC7230 - see section 3, Message Format.
>> https://tools.ietf.org/html/rfc7230#section-3
>
> So, is it OK to always recognize query part in the path. Correct?
>
> What is the scheme by default? http? None?

See https://tools.ietf.org/html/rfc7230#section-5.3.1 :

   For example, a client wishing to retrieve a representation of the
   resource identified as

     http://www.example.org/where?q=now

   directly from the origin server would open (or reuse) a TCP
   connection to port 80 of the host "www.example.org" and send the
   lines:

     GET /where?q=now HTTP/1.1
     Host: www.example.org

   followed by the remainder of the request message.

Re: HTTP with Simple Components: Status.Kind always File

[Dmitry A. Kazakov]

On 20/07/2017 23:12, Felix Krause wrote:
> On 2017-07-20 20:57:35 +0000, Dmitry A. Kazakov said:
> 
>> On 2017-07-20 22:29, Simon Wright wrote:
>>>
>>> The RFC that Dmitri quotes is for URIs. You want the one that defines
>>> the HTTP protocol, currently RFC7230 - see section 3, Message Format.
>>> https://tools.ietf.org/html/rfc7230#section-3
> 
> Ah, I looked at an older RFC for HTTP 1.1; I wasn't aware there was a newer
> one. This newer one indeed defines query as part of the request target:
> 
>     https://tools.ietf.org/html/rfc7230#section-5.3
> 
>     request-target = origin-form
>                     / absolute-form
>                     / authority-form
>                     / asterisk-form
>     origin-form    = absolute-path [ "?" query ]
>>
>> So, is it OK to always recognize query part in the path. Correct?
> 
> Yes, although to avoid confusion, it may be a good idea to not name
> it „Path“. Reading the spec, „Origin“ might be more appropriate. But
> I guess that change would break backwards compatibility.

I think I will just move the Query member out of the variant choice like 
this:

    type Status_Line
         (  Kind         : Status_Line_Type;
            Path_Length  : Natural;
            Host_Length  : Natural;
            Query_Length : Natural
         )  is
    record
       Query : String (1..Query_Length);
       case Kind is
          when None =>
             null;
          when File =>
             File : String (1..Path_Length);
          when URI =>
             Scheme : Scheme_Type;
             Host   : String (1..Host_Length);
             Port   : Port_Type;
             Path   : String (1..Path_Length);
       end case;
    end record;

That should keep it backward compatible.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: HTTP with Simple Components: Status.Kind always File

[Dmitry A. Kazakov]

On 21/07/2017 09:06, Simon Wright wrote:
> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
> 
>>> The RFC that Dmitri quotes is for URIs. You want the one that defines
>>> the HTTP protocol, currently RFC7230 - see section 3, Message Format.
>>> https://tools.ietf.org/html/rfc7230#section-3
>>
>> So, is it OK to always recognize query part in the path. Correct?
>>
>> What is the scheme by default? http? None?
> 
> See https://tools.ietf.org/html/rfc7230#section-5.3.1 :
> 
>     For example, a client wishing to retrieve a representation of the
>     resource identified as
> 
>       http://www.example.org/where?q=now
> 
>     directly from the origin server would open (or reuse) a TCP
>     connection to port 80 of the host "www.example.org" and send the
>     lines:
> 
>       GET /where?q=now HTTP/1.1
>       Host: www.example.org
> 
>     followed by the remainder of the request message.

OK, thanks for clarification.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de