Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[Olivier Henley]

Hey,

I am trying a small multiplayer 2d game using Gnoga and Phaser.io and and I
am experiencing some 80 ms for any roundtrip test when trying my game at
127.0.0.1:8080. I profiled javascript and it looks like it take 0.4 ms on
the client side to process round_trip().

Anyone has an idea about what is wrong?

Thx,

Olivier

p.s: Here is my roundtrip code:

---- Ada code ----

App_Data.Roundtrip_Input.On_Change_Handler (Handler => On_Roundtrip'Access);

...

loop
    App_Data.Last_Roundtrip_Start := Clock;
    Gnoga.Server.Connection.Execute_Script (ID => App_Data.Window.Connection_ID,Script => "round_trip();");
    delay 0.5;
end loop;

...

procedure On_Roundtrip (Object : in out Gnoga.Gui.Base.Base_Type'Class) is
    App_Data : App_Data_Access := App_Data_Access (Object.Connection_Data);
    Roundtrip_Duration : Duration;
begin
    App_Data.Roundtrip_Span := Clock - App_Data.Last_Roundtrip_Start;
    Roundtrip_Duration := To_Duration (App_Data.Roundtrip_Span);
    Put_Line ("---- Roundtrip ---- :  " & Duration'Image(Roundtrip_Duration));
 
-- output about 0.080 using Chrome
-- output about 0.120 using Firefox

end On_Roundtrip;


---- Javascript code ----

function round_trip()
{
    var round_trip_date = new Date();
    $('#rt').attr('value', round_trip_date.getTime()).change();
}

Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[rieachus]

On Wednesday, March 30, 2016 at 11:35:59 AM UTC-4, Olivier Henley wrote:
> Hey,
> 
> I am trying a small multiplayer 2d game using Gnoga and Phaser.io and and I
> am experiencing some 80 ms for any roundtrip test when trying my game at
> 127.0.0.1:8080. I profiled javascript and it looks like it take 0.4 ms on
> the client side to process round_trip().
> 
> Anyone has an idea about what is wrong?
 
What is the (real) tick for the clock you are using?  Some OS clocks have about a 20 ms tick, and if you call them twice will return previous tick plus 1 millisecond or even  one microsecond.  Other system clocks will delay until the next tick.  Some software depends on unique time-stamps, so the system clock gets diddled one way or the other.

Oh, the other thing that can happen is that there is a clock buried in the tasking/threading/process switching software, and interprocess messages only get delivered when that clock ticks, or waiting software only gets started every so many ticks. (The idea is to insure that threads have enough time to actually do something before someone with higher priority grabs the system back.  All this cruft is still buried in OS software, even though most systems have more than one CPU...)

I suspect that one or the other of these is going on, due to the 2 to 3 ratio between the two results.  I remember fighting these gremlins years ago when trying to time various message passing protocols. 

Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[Jeffrey R. Carter]

On Wednesday, March 30, 2016 at 11:35:59 AM UTC-4, Olivier Henley wrote:
>
> I am trying a small multiplayer 2d game using Gnoga and Phaser.io and and I
> am experiencing some 80 ms for any roundtrip test when trying my game at
> 127.0.0.1:8080. I profiled javascript and it looks like it take 0.4 ms on
> the client side to process round_trip().
>
> Anyone has an idea about what is wrong?

AFAICT, nothing is wrong, unless you have timing requirements that aren't being
met. Since you're talking about a game, and 80 ms is too short for a human to
notice, I don't see that you have a problem.

-- 
Jeff Carter
"Many times we're given rhymes that are quite unsingable."
Monty Python and the Holy Grail
57

Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[Dmitry A. Kazakov]

On 31/03/2016 07:23, Jeffrey R. Carter wrote:
> On Wednesday, March 30, 2016 at 11:35:59 AM UTC-4, Olivier Henley wrote:
>>
>> I am trying a small multiplayer 2d game using Gnoga and Phaser.io and and I
>> am experiencing some 80 ms for any roundtrip test when trying my game at
>> 127.0.0.1:8080. I profiled javascript and it looks like it take 0.4 ms on
>> the client side to process round_trip().
>>
>> Anyone has an idea about what is wrong?
>
> AFAICT, nothing is wrong, unless you have timing requirements that aren't being
> met. Since you're talking about a game, and 80 ms is too short for a human to
> notice, I don't see that you have a problem.

Still it could be indication of a problem or a bug. Usual suspects are:

1. The measurement procedure.

1.1. Artifacts of process switching.
1.2. Wrong clock used. System clocks may be very coarse. In many OSes 
they are driven by the timer interrupts or from an inferior clock 
source. Real-time clock at the processor's or the front bus' tact must 
be used in round-trip measurements.

2. HTTP. It is a very poor protocol. Much depends on the 
connection/session settings. E.g. if a new session is created for each 
measurement it is to expect a quite poor performance. Creating and 
connecting sockets is expensive.

3. Socket settings

3.1. TCP_NO_DELAY and sending packets in full is essential to short 
latencies. The frame coalescing algorithm timeout depends on the system 
settings. Usually it is close to 100ms. Which means that you could have 
sporadic latency spikes depending on the state of the TCP buffer when 
the traffic is low.

4. OS, TCP stack system settings

4.1. Poor Ethernet controller, especially bad are integrated ones. The 
worst are ones connected over USB/COM. Many boards have ones sitting on 
top of an UART.

Then, it could be a software bug. The Simple Components server gets a 
notification when there is free space in the output buffer, ready to 
send more data out. When the traffic is low, the output buffer is never 
full and this means that, in effect, the buffer-ready signal is always 
set. Thus dealing with the socket output becomes in effect busy waiting. 
That in turn leads 100% load of the CPU's core running the server. In 
order to prevent that the server stops waiting for the buffer-ready 
signal when there was nothing to send. When a new portion of output data 
appears the socket waiting is resumed. There is a certain timeout to 
look after the socket regardless. If there is a bug somewhere that 
prevents the resuming, it may expose itself as sporadically increased 
latencies during sending data out. The latency must then be close to the 
timeout value. [This is similar to the effect of TCP_NO_DELAY not set]

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[Olivier Henley]

On Thursday, March 31, 2016 at 3:39:10 AM UTC-4, Dmitry A. Kazakov wrote:

> Still it could be indication of a problem or a bug. Usual suspects are:
> 
> 1. The measurement procedure.
> 
> 1.1. Artifacts of process switching.
> 1.2. Wrong clock used. System clocks may be very coarse. In many OSes 
> they are driven by the timer interrupts or from an inferior clock 
> source. Real-time clock at the processor's or the front bus' tact must 
> be used in round-trip measurements.

Debian 8 x64 using Adacore GNAT 2015 and within Ada.Real_Time
 
> 2. HTTP. It is a very poor protocol. Much depends on the 
> connection/session settings. E.g. if a new session is created for each 
> measurement it is to expect a quite poor performance. Creating and 
> connecting sockets is expensive.

I am using Gnoga and expect everything to go through Websocket and not HTTP (handshake done). I don't suspect my using of the framework, nor the framework, to create new connections every time I communicate with a particular client.

> 3. Socket settings
> 
> 3.1. TCP_NO_DELAY and sending packets in full is essential to short 
> latencies. The frame coalescing algorithm timeout depends on the system 
> settings. Usually it is close to 100ms. Which means that you could have 
> sporadic latency spikes depending on the state of the TCP buffer when 
> the traffic is low.

I need to investigate this one.


> 4. OS, TCP stack system settings
> 
> 4.1. Poor Ethernet controller, especially bad are integrated ones. The 
> worst are ones connected over USB/COM. Many boards have ones sitting on 
> top of an UART.

ping 127.0.0.1 gives me like 0.02 ms. So to have a ~80ms discrepancy, I would not suspect my hardware.

> Then, it could be a software bug. The Simple Components server gets a 
> notification when there is free space in the output buffer, ready to 
> send more data out. When the traffic is low, the output buffer is never 
> full and this means that, in effect, the buffer-ready signal is always 
> set. Thus dealing with the socket output becomes in effect busy waiting. 
> That in turn leads 100% load of the CPU's core running the server. In 
> order to prevent that the server stops waiting for the buffer-ready 
> signal when there was nothing to send. When a new portion of output data 
> appears the socket waiting is resumed. There is a certain timeout to 
> look after the socket regardless. If there is a bug somewhere that 
> prevents the resuming, it may expose itself as sporadically increased 
> latencies during sending data out. The latency must then be close to the 
> timeout value. [This is similar to the effect of TCP_NO_DELAY not set]

Good. In a sense it reassures me because I am effectively sending almost nothing, couple of bytes, every half seconds. I should stress test and see if it stays the same or even maybe reduce latency.

Ok but would it hold if I tell you that using Firefox I get around 120ms instead?

What is my best plan to investigate further? Profiling my exec, analysing network traffic etc?

Thx 

Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[Dmitry A. Kazakov]

On 2016-03-31 19:02, Olivier Henley wrote:
> On Thursday, March 31, 2016 at 3:39:10 AM UTC-4, Dmitry A. Kazakov wrote:
>
>> Still it could be indication of a problem or a bug. Usual suspects are:
>>
>> 1. The measurement procedure.
>>
>> 1.1. Artifacts of process switching.
>> 1.2. Wrong clock used. System clocks may be very coarse. In many OSes
>> they are driven by the timer interrupts or from an inferior clock
>> source. Real-time clock at the processor's or the front bus' tact must
>> be used in round-trip measurements.
>
> Debian 8 x64 using Adacore GNAT 2015 and within Ada.Real_Time

AFAIK GNAT's Ada.Real_Time and Ada.Calendar use the same time source and 
it must be TSC under Linux.

To test clock quality call Clock twice and compare the results. Good 
clock must have different reading each time you call it.

>> 2. HTTP. It is a very poor protocol. Much depends on the
>> connection/session settings. E.g. if a new session is created for each
>> measurement it is to expect a quite poor performance. Creating and
>> connecting sockets is expensive.
>
> I am using Gnoga and expect everything to go through Websocket and
> not  HTTP (handshake done). I don't suspect my using of the framework, nor
> the framework, to create new connections every time I communicate with a
> particular client.

WebSocket uses a HTTP session. You first establish a HTTP session and 
then switch to WebSocket. An awful idea, but everything related to HTTP 
is awful.

>> 3. Socket settings
>>
>> 3.1. TCP_NO_DELAY and sending packets in full is essential to short
>> latencies. The frame coalescing algorithm timeout depends on the system
>> settings. Usually it is close to 100ms. Which means that you could have
>> sporadic latency spikes depending on the state of the TCP buffer when
>> the traffic is low.
>
> I need to investigate this one.

Yes. This is the major source of latencies. TCP/IP is optimized for 
throughout, not for short latencies.

>> 4. OS, TCP stack system settings
>>
>> 4.1. Poor Ethernet controller, especially bad are integrated ones. The
>> worst are ones connected over USB/COM. Many boards have ones sitting on
>> top of an UART.
>
> ping 127.0.0.1 gives me like 0.02 ms. So to have a ~80ms
> discrepancy,
> I would not suspect my hardware.

OK, loop-back is not related to Ethernet.

>> Then, it could be a software bug. The Simple Components server gets a
>> notification when there is free space in the output buffer, ready to
>> send more data out. When the traffic is low, the output buffer is never
>> full and this means that, in effect, the buffer-ready signal is always
>> set. Thus dealing with the socket output becomes in effect busy waiting.
>> That in turn leads 100% load of the CPU's core running the server. In
>> order to prevent that the server stops waiting for the buffer-ready
>> signal when there was nothing to send. When a new portion of output data
>> appears the socket waiting is resumed. There is a certain timeout to
>> look after the socket regardless. If there is a bug somewhere that
>> prevents the resuming, it may expose itself as sporadically increased
>> latencies during sending data out. The latency must then be close to the
>> timeout value. [This is similar to the effect of TCP_NO_DELAY not set]
>
> Good. In a sense it reassures me because I am effectively sending
> almost nothing, couple of bytes, every half seconds. I should stress
> test and see if it stays the same or even maybe reduce latency.

I would start with TCP_NO_DELAY. Ask David if Gnoga supports it. Note 
that it must be set on both sides if the communication is 
bi-directional. I don't know if it were possible on the client's script 
side.

> Ok but would it hold if I tell you that using Firefox I get around 120ms instead?

That looks consistent with TCP_NO_DELAY being not set.

> What is my best plan to investigate further? Profiling my exec,
> analysing network traffic etc?

In Wireshark (on LAN) you can see the structure of frames. When 
TCP_NO_DELAY is active packets are small because the TCP stack bypasses 
coalescing and sends data as soon as possible regardless the frame size. 
TCP_NO_DELAY is a dangerous thing that leads to flooding the LAN.

If TCP_NO_DELAY does not help and there is a suspicion that there could 
be something wrong with the server and/or client, I would suggest to 
write a small echo server and client program to test QoS of the server 
and the client separately.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[Olivier Henley]

On Thursday, March 31, 2016 at 1:24:03 AM UTC-4, Jeffrey R. Carter wrote:

> met. Since you're talking about a game, and 80 ms is too short for a human to
> notice, I don't see that you have a problem.

33.3 ms gives you a 30 fps game which is standard and noticeable.
16.6 ms gives you a 60 fps game which is the quality standard. The extent to which it is noticeable is still a debate.

That said, a multiplayer game can tolerate more or less 150 ms maximum roundtrip and for this to be acceptable it has to implement Client-Side Prediction, Server Reconciliation, Entity Interpolation and Lag Compensation.

I figured that having 80ms of lag on the loopback does not look good to start with.

Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[Jeffrey R. Carter]

On 03/31/2016 09:39 AM, Olivier Henley wrote:
> 
> 33.3 ms gives you a 30 fps game which is standard and noticeable.
> 16.6 ms gives you a 60 fps game which is the quality standard. The extent to which it is noticeable is still a debate.
>
> That said, a multiplayer game can tolerate more or less 150 ms maximum roundtrip and for this to be acceptable it has to implement Client-Side Prediction, Server Reconciliation, Entity Interpolation and Lag Compensation.

PAL TV was 25 fps, or 40 ms/frame. Film movies are shot at 24 fps, or 41.7
ms/frame. Silent movies were shot at 18 fps, or 55.6 ms/frame. Traditional,
hand-drawn animation is shot at 12 fps, or 83.3 ms/frame, and looks fine (see
"Fantasia" for an example). Anything under 100 ms/frame will look smooth.

Still, I can see how 80 ms on the local machine might be a concern if the
intention is for this to work over the internet.

-- 
Jeff Carter
"My little plum, I am like Robin Hood. I take from
the rich, and I give to the poor. ... Us poor."
Poppy
96

Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[Dmitry A. Kazakov]

On 2016-03-31 22:46, Jeffrey R. Carter wrote:
> On 03/31/2016 09:39 AM, Olivier Henley wrote:
>>
>> 33.3 ms gives you a 30 fps game which is standard and noticeable.
>> 16.6 ms gives you a 60 fps game which is the quality standard. The extent to which it is noticeable is still a debate.
>>
>> That said, a multiplayer game can tolerate more or less 150 ms
>> maximum roundtrip and for this to be acceptable it has to implement
>> Client-Side Prediction, Server Reconciliation, Entity Interpolation and
>> Lag Compensation.
>
> PAL TV was 25 fps, or 40 ms/frame. Film movies are shot at 24 fps, or 41.7
> ms/frame. Silent movies were shot at 18 fps, or 55.6 ms/frame. Traditional,
> hand-drawn animation is shot at 12 fps, or 83.3 ms/frame, and looks fine (see
> "Fantasia" for an example). Anything under 100 ms/frame will look smooth.

It is more complicated. An old TV set is not sharp. Its dithering mask 
work as a sort of anti-aliasing filter. The same applies to the live 
images, which perception is different from artificial images. E.g. 
running curves on a high-resolution screen without anti-aliasing appear 
jerking even at 20ms rate. Furthermore, jitter around 10ms is visible 
too. So, I understand how 80ms might become a problem.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Roundtrip latency problem using Gnoga, on Linux, when testing at localhost address

[Dennis Lee Bieber]

On Thu, 31 Mar 2016 13:46:23 -0700, "Jeffrey R. Carter"
<spam.jrcarter.not@spam.not.acm.org> declaimed the following:


>
>PAL TV was 25 fps, or 40 ms/frame. Film movies are shot at 24 fps, or 41.7
>ms/frame. Silent movies were shot at 18 fps, or 55.6 ms/frame. Traditional,
>hand-drawn animation is shot at 12 fps, or 83.3 ms/frame, and looks fine (see
>"Fantasia" for an example). Anything under 100 ms/frame will look smooth.
>
	Traditional, quality, animation was shot "on-twos" -- that is, each
still drawing was photographed twice, so while the animation moves on
12fps, the film is still fully 24fps.
 
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/