Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

["Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>]

On 2016-03-31 19:02, Olivier Henley wrote:
> On Thursday, March 31, 2016 at 3:39:10 AM UTC-4, Dmitry A. Kazakov wrote:
>
>> Still it could be indication of a problem or a bug. Usual suspects are:
>>
>> 1. The measurement procedure.
>>
>> 1.1. Artifacts of process switching.
>> 1.2. Wrong clock used. System clocks may be very coarse. In many OSes
>> they are driven by the timer interrupts or from an inferior clock
>> source. Real-time clock at the processor's or the front bus' tact must
>> be used in round-trip measurements.
>
> Debian 8 x64 using Adacore GNAT 2015 and within Ada.Real_Time

AFAIK GNAT's Ada.Real_Time and Ada.Calendar use the same time source and 
it must be TSC under Linux.

To test clock quality call Clock twice and compare the results. Good 
clock must have different reading each time you call it.

>> 2. HTTP. It is a very poor protocol. Much depends on the
>> connection/session settings. E.g. if a new session is created for each
>> measurement it is to expect a quite poor performance. Creating and
>> connecting sockets is expensive.
>
> I am using Gnoga and expect everything to go through Websocket and
> not  HTTP (handshake done). I don't suspect my using of the framework, nor
> the framework, to create new connections every time I communicate with a
> particular client.

WebSocket uses a HTTP session. You first establish a HTTP session and 
then switch to WebSocket. An awful idea, but everything related to HTTP 
is awful.

>> 3. Socket settings
>>
>> 3.1. TCP_NO_DELAY and sending packets in full is essential to short
>> latencies. The frame coalescing algorithm timeout depends on the system
>> settings. Usually it is close to 100ms. Which means that you could have
>> sporadic latency spikes depending on the state of the TCP buffer when
>> the traffic is low.
>
> I need to investigate this one.

Yes. This is the major source of latencies. TCP/IP is optimized for 
throughout, not for short latencies.

>> 4. OS, TCP stack system settings
>>
>> 4.1. Poor Ethernet controller, especially bad are integrated ones. The
>> worst are ones connected over USB/COM. Many boards have ones sitting on
>> top of an UART.
>
> ping 127.0.0.1 gives me like 0.02 ms. So to have a ~80ms
> discrepancy,
> I would not suspect my hardware.

OK, loop-back is not related to Ethernet.

>> Then, it could be a software bug. The Simple Components server gets a
>> notification when there is free space in the output buffer, ready to
>> send more data out. When the traffic is low, the output buffer is never
>> full and this means that, in effect, the buffer-ready signal is always
>> set. Thus dealing with the socket output becomes in effect busy waiting.
>> That in turn leads 100% load of the CPU's core running the server. In
>> order to prevent that the server stops waiting for the buffer-ready
>> signal when there was nothing to send. When a new portion of output data
>> appears the socket waiting is resumed. There is a certain timeout to
>> look after the socket regardless. If there is a bug somewhere that
>> prevents the resuming, it may expose itself as sporadically increased
>> latencies during sending data out. The latency must then be close to the
>> timeout value. [This is similar to the effect of TCP_NO_DELAY not set]
>
> Good. In a sense it reassures me because I am effectively sending
> almost nothing, couple of bytes, every half seconds. I should stress
> test and see if it stays the same or even maybe reduce latency.

I would start with TCP_NO_DELAY. Ask David if Gnoga supports it. Note 
that it must be set on both sides if the communication is 
bi-directional. I don't know if it were possible on the client's script 
side.

> Ok but would it hold if I tell you that using Firefox I get around 120ms instead?

That looks consistent with TCP_NO_DELAY being not set.

> What is my best plan to investigate further? Profiling my exec,
> analysing network traffic etc?

In Wireshark (on LAN) you can see the structure of frames. When 
TCP_NO_DELAY is active packets are small because the TCP stack bypasses 
coalescing and sends data as soon as possible regardless the frame size. 
TCP_NO_DELAY is a dangerous thing that leads to flooding the LAN.

If TCP_NO_DELAY does not help and there is a suspicion that there could 
be something wrong with the server and/or client, I would suggest to 
write a small echo server and client program to test QoS of the server 
and the client separately.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de