* Read-write mutex sometimes fails on deadlock
@ 2017-10-28 20:02 pascal.malaise
2017-11-06 18:24 ` Robert Eachus
0 siblings, 1 reply; 8+ messages in thread
From: pascal.malaise @ 2017-10-28 20:02 UTC (permalink / raw)
Hi,
[long post because it includes the full sources and output, sorry]
I have a package that implements several kinds of mutexes, including a
read-write mutex. Here is the full source (spec and body) with only the
read-write part.
mutexes.ads
-----------
private with Ada.Task_Identification;
-- Mutex (single and Read_Write) management
package Mutexes is
-- Kind of mutex
-- Read_Write allows several readers but one writer at a time.
-- This implementation is fair but somewhat CPU-consuming because
-- the arrival of a reader while writer(s) are waiting triggers a
-- re-evaluation to let it pass if it has a higher (task) priority.
-- Kind of requested access for a Read_Write mutex
type Access_Kind is (Read, Write);
-- Mutex object, free at creation
type Mutex is tagged limited private;
-- Get access to a mutex.
-- With RW mutex (Read_Write), simultaneous read are possible,
-- but there is only one writer at a time and no reader at that time.
-- If delay is negative, wait until mutex is got,
-- If delay is null, try and give up if not free,
-- If delay is positive, try during the specified delay.
-- Raises Already_Got if the current task if it has already got the RW mutex
-- for write.
-- Note that there is no check of "Read then Write" deadlock.
Already_Got : exception;
function Get (A_Mutex : in out Mutex;
Waiting_Time : in Duration;
Kind : in Access_Kind := Read) return Boolean;
-- Get access to a mutex : infinite wait.
procedure Get (A_Mutex : in out Mutex;
Kind : in Access_Kind := Read);
function Get (A_Mutex : in out Mutex) return Boolean;
-- Release access to a mutex.
-- Raises Not_Owner if current task doesn't own the the RW mutex for write
-- (no check when releasing RW mutex aquired for read).
Not_Owner : exception;
procedure Release (A_Mutex : in out Mutex);
-- Is current task the "owner" (access got) of a simple mutex
-- Is it the writer in case of a RW mutex
function Is_Owner (A_Mutex : in Mutex) return Boolean;
private
-- Read/write mutex
-- We want to queue all requests (read and write) within the same queue
-- to ensure fairness (avoid starvation).
-- But the condition for read is "no writer" (not Writer)
-- and for write it is "no writer and no reader"
-- (not Writer and then Readers = 0)
-- So the guard of the common queue is "no writer", but a writer may pass it
-- while there are reader. In this case, we would like to requeue it in the
-- same queue and in the beginning of it; note that "requeue" requeues at
-- the end :-(. The artifact is to requeue the task, and all other
-- tasks queueing after it as well, in an alternate queue (Swapping).
-- Benefit: a new reader may enter while writer(s) are queueing, if its
-- priority is better. Fairness ++.
-- Drawback: each new reader while writer(s) are queueing triggers an open
-- then a swap of the queues. Perfo --.
-- So there are two queues
type Queue_Range is mod 2;
-- The read/write access lock and queues. No time.
protected type Rw_Mutex_Protect is
-- Gets the lock. Blocking.
entry Mutex_Get (Kind : in Access_Kind);
-- Releases the lock
procedure Mutex_Release;
-- Is current task write owner of the lock
function Mutex_Owns return Boolean;
private
-- Number of readers
Readers : Natural := 0;
-- Writer identification
Owner : Ada.Task_Identification.Task_Id;
-- Numer of times it has got the write lock
Writer : Natural := 0;
-- Two queues, one is active at a time
entry Queues (Queue_Range) (Kind : in Access_Kind);
Current_Queue : Queue_Range := Queue_Range'First;
-- The status of the queue:
-- Swapping or not
Swapping : Boolean := False;
-- If not swapping, is the mutex open or not
-- if swapping, will it be open or not
Open : Boolean := True;
end Rw_Mutex_Protect;
-- The general purpose mutex
type Mutex is tagged limited record
Rw_Mutex : Rw_Mutex_Protect;
end record;
end Mutexes;
mutexes.adb
-----------
with Ada.Text_Io;
package body Mutexes is
use type Ada.Task_Identification.Task_Id;
function Image (Id : in Ada.Task_Identification.Task_Id) return String
renames Ada.Task_Identification.Image;
Debug_Set : Boolean := False;
Debug_On : Boolean := False;
procedure Trace (Id : in Ada.Task_Identification.Task_Id;
Msg : in String) is
begin
if not Debug_Set then
-- Initial setup
Debug_Set := True;
Debug_On := True;
end if;
if Debug_On then
Ada.Text_Io.Put_Line (Msg & " " & Image (Id));
end if;
end Trace;
-- The protected object which implements the read/write mutex
protected body Rw_Mutex_Protect is
-- Gets the lock. Blocking.
-- Do not let a new request be inserted in the middle of a queue while
-- we are swapping queues.
entry Mutex_Get (Kind : in Access_Kind) when not Swapping is
begin
-- Are we already the writer
if Writer /= 0 and then Mutex_Get'Caller = Owner then
raise Already_Got;
end if;
-- Queue the request in the active queue
Trace (Mutex_Get'Caller, "Read_Write get queueing");
requeue Queues(Current_Queue) with abort;
end Mutex_Get;
-- Releases the lock. No Check of kind but the lock must have been
-- got.
procedure Mutex_Release is
begin
if Readers > 0 then
-- There are readers, one of them is releasing the lock
Readers := Readers - 1;
if Readers = 0 then
-- The last reader leaves, so the lock becomes available
-- for writers
Trace (Ada.Task_Identification.Current_Task, "Last reader releases");
Open := True;
else
Trace (Ada.Task_Identification.Current_Task, "Reader releases");
end if;
elsif Writer /= 0
and then Ada.Task_Identification.Current_Task = Owner then
-- The writer is releasing the lock
if Writer /= 1 then
Writer := Writer - 1;
return;
end if;
-- Really releasing
Trace (Ada.Task_Identification.Current_Task, "Writer releases");
Writer := 0;
Open := True;
else
-- Called while no lock was got or not called by the writer
raise Not_Owner;
end if;
end Mutex_Release;
function Mutex_Owns return Boolean is
(Writer /= 0 and then Ada.Task_Identification.Current_Task = Owner);
-- Two queues, one active at a time
-- Passes when swapping queues or else when open
entry Queues (for Queue in Queue_Range) (Kind : in Access_Kind)
when Queue = Current_Queue
and then (Swapping or else Open) is
begin
if Swapping then
-- Swapping queueing tasks from one queue to the other
Trace (Queues'Caller, "Swapping");
if Queues(Queue)'Count = 0 then
-- This is the last task: end of swapping
-- Open remains unchanged (maybe open by a release)
Swapping := False;
Current_Queue := Current_Queue + 1;
Trace (Queues'Caller, "End swapping");
end if;
-- Requeue the task on the other queue
requeue Queues(Queue + 1) with abort;
else
-- The queue is open: The lock is either free or allocated to reader(s)
if Kind = Read then
Trace (Queues'Caller, "Another reader");
-- Read lock
Readers := Readers + 1;
else
-- Write lock:
-- If we are here, it means that the gate is open so no writer
-- has already got the lock
-- Either we get the lock (queue is closed until we release)
-- or we queue (no currently queueing read can pass)
-- so in both case, the queue is closed
-- Note that a new request may re-open the queue and enter
-- before us if it as a better prio
Open := False;
if Readers = 0 then
-- No reader => we get the write lock
Writer := 1;
Owner := Queues'Caller;
Trace (Queues'Caller, "Writer has got lock");
else
-- We have to wait until last reader releases the lock
-- If we are alone, requeue ourself. Otherwise
-- requeue in the alternate queue this task, then all the other
-- queueing tasks, so we remain first (if same prios)
Swapping := Queues(Queue)'Count > 0;
if Swapping then
Trace (Queues'Caller, "Start swapping");
requeue Queues(Queue + 1) with abort;
else
Trace (Queues'Caller, "Writer waits for current reader");
requeue Queues(Queue) with abort;
end if;
end if;
end if;
end if;
end Queues;
end Rw_Mutex_Protect;
function Get (A_Mutex : in out Mutex;
Waiting_Time : in Duration;
Kind : in Access_Kind := Read) return Boolean is
Result : Boolean;
begin
if Waiting_Time < 0.0 then
-- Negative delay : unconditional waiting
A_Mutex.Rw_Mutex.Mutex_Get (Kind);
Result := True;
else
select
A_Mutex.Rw_Mutex.Mutex_Get (Kind);
Result := True;
or
delay Waiting_Time;
Result := False;
end select;
end if;
return Result;
end Get;
function Get (A_Mutex : in out Mutex) return Boolean is
(Get (A_Mutex, -1.0, Read));
-- Get a mutex : infinite wait
procedure Get (A_Mutex : in out Mutex;
Kind : in Access_Kind := Read) is
Dummy : Boolean;
begin
Dummy := Get (A_Mutex, -1.0, Kind);
end Get;
-- Release a mutex
procedure Release (A_Mutex : in out Mutex) is
begin
-- Request releasing
A_Mutex.Rw_Mutex.Mutex_Release;
end Release;
-- Does current task own the mutex (for write)
function Is_Owner (A_Mutex : Mutex) return Boolean is
begin
return A_Mutex.Rw_Mutex.Mutex_Owns;
end Is_Owner;
end Mutexes;
Finally, I have a test program of the read-write mutex, which creates 10 tasks,
each of them taking the mutex, for read or write, during some time. The tasks
randomly terminate, and when all of them are done then the test is OK.
t_read_write.adb
----------------
with Ada.Text_Io, Ada.Command_Line, Ada.Calendar, Ada.Task_Identification,
Ada.Numerics.Float_Random;
with Gnat.Calendar;
with Mutexes;
-- Test Read-Write mutex
procedure T_Read_Write is
pragma Priority(10);
-- True if gnat, otherwise remove usage of Gnat.Calendar
Has_Gnat : constant Boolean := True;
-- The number of tasks
subtype Range_Actor is Natural range 0 .. 10;
Main_Index : constant Range_Actor := 0;
subtype Range_Task is Positive range 1 .. Range_Actor'Last;
-- Date of last output of a task
Last_Time : Ada.Calendar.Time;
use type Ada.Calendar.Time;
-- Delay of inactivity
Inactivity : constant Duration := 5.0;
-- Random generator
Gen : Ada.Numerics.Float_Random.Generator;
function Random (Mini : in Integer := 0;
Maxi : in Integer := 1) return Integer is
F : Float;
Res : Integer;
begin
F := Float(Mini) + Ada.Numerics.Float_Random.Random(Gen)
* Float(Maxi - Mini);
Res := Integer (F);
while Res > Maxi and then Res > Mini loop
Res := Res - 1;
end loop;
return Res;
end Random;
-- Image of a date
function Image (Date : Ada.Calendar.Time) return String is
Year : Ada.Calendar.Year_Number;
Month : Ada.Calendar.Month_Number;
Day : Ada.Calendar.Day_Number;
Hour : Gnat.Calendar.Hour_Number;
Minute : Gnat.Calendar.Minute_Number;
Second : Gnat.Calendar.Second_Number;
Sub_Second : Gnat.Calendar.Second_Duration;
function Img (D : Gnat.Calendar.Second_Duration) return String is
S : constant String := D'Img;
begin
return S(3 .. S'Last);
end Img;
Dur : Ada.Calendar.Day_Duration;
function Img (N : Integer) return String is
S : constant String := N'Img;
begin
if S'Last = 2 then
-- " d", 1 digit => "0" & d
return "0" & S(2);
else
-- " dd", d
return S(2 .. S'Last);
end if;
end Img;
begin
if Has_Gnat then
Gnat.Calendar.Split (Date, Year, Month, Day, Hour, Minute, Second,
Sub_Second);
return Img (Year) & "-" & Img (Month) & "-" & Img (Day) & "T"
& Img (Hour) & ":" & Img (Minute) & ":" & Img (Second)
& Img (Sub_Second);
else
Ada.Calendar.Split (Date, Year, Month, Day, Dur);
return Img (Year) & "/" & Img (Month) & "/" & Img (Day) & Dur'Img;
end if;
end Image;
-- Put a task activity
procedure Put_Line (Index : in Range_Actor; Msg : in String) is
begin
Last_Time := Ada.Calendar.Clock;
declare
Date : constant String := Image (Last_Time);
Str : constant String
:= (if Index in Range_Task then Index'Img else "Main")
& " " & Msg;
begin
Ada.Text_Io.Put_Line (Date & " " & Str);
end;
end Put_Line;
Lock : access Mutexes.Mutex;
-- The client tasks
task type T is
pragma Priority(10);
entry Num (I : in Range_Task);
entry Done;
end T;
function Image (D : Duration) return String is
Str : constant String := D'Img;
begin
return Str (1 .. 4);
end Image;
task body T is
Index : Range_Task;
Dur : Duration;
Kind : Mutexes.Access_Kind;
subtype Str5 is String (1 .. 5);
Kind_Strs : constant array (Mutexes.Access_Kind) of Str5 := (
Mutexes.Read => " Read",
Mutexes.Write => "Write");
Res : Boolean;
begin
-- Get name
accept Num (I : in Range_Task) do
Index := I;
end Num;
Put_Line (Index ,
"Task started, id "
& Ada.Task_Identification.Image (Ada.Task_Identification.Current_Task));
-- Work until termination requested in Critical
loop
delay 0.01;
-- Wait from -0.1 to 0.5
Dur := Duration (Random (-1, 5)) / 10.0;
-- 10% chances to write
if Random (0, 9) = 0 then
Kind := Mutexes.Write;
else
Kind := Mutexes.Read;
end if;
-- Get lock
Put_Line (Index, "get " & Kind_Strs(Kind) & " " & Image(Dur));
Res := Lock.Get (Dur, Kind);
-- Trace result
if Res then
Put_Line (Index, "OK");
else
Put_Line (Index, "NOK");
end if;
if Res then
-- Got it: Work (wait) from 0.1 to 0.5
Put_Line (Index, "waiting");
delay Duration (Random (1, 5)) / 10.0;
Put_Line (Index, "waited");
-- Release lock
Put_Line (Index, "release");
Lock.Release;
end if;
-- 10% chances to terminate
exit when Random (0, 9) = 0;
end loop;
Put_Line (Index, "terminate");
-- Ready to end
accept Done;
end T;
Tasks : array (Range_Task) of T;
Runs : array (Range_Task) of Boolean;
Nb_Run : Natural;
-- The main: activate and monitor the tasks
begin
Lock := new Mutexes.Mutex;
-- Randomize
Ada.Numerics.Float_Random.Reset (Gen,
Integer (Ada.Calendar.Seconds (Ada.Calendar.Clock)));
-- Give to each actor it's name
Put_Line (Main_Index, "Starting");
Nb_Run := 0;
for I in Range_Task loop
Tasks(I).Num (I);
Runs(I) := True;
Nb_Run := Nb_Run + 1;
end loop;
-- Wait until termination of all tasks
Main: while Nb_Run /= 0 loop
-- Try to terminate tasks: wait 1s altogether
for I in Range_Task loop
if Runs(I) then
select
Tasks(I).Done;
Runs(I) := False;
Nb_Run := Nb_Run - 1;
or
delay 1.0 / Duration(Nb_Run);
end select;
end if;
end loop;
-- Monitor activity
if Ada.Calendar.Clock - Last_Time > Inactivity then
-- Deadlock detected => abort tasks
Put_Line (Main_Index, "Deadlock detected, aborting");
Ada.Command_Line.Set_Exit_Status (1);
for I in Range_Task loop
if Runs(I) then
abort Tasks(I);
-- This will exit Main
Nb_Run := Nb_Run - 1;
end if;
end loop;
end if;
end loop Main;
Put_Line (Main_Index, "Done");
end T_Read_Write;
Most of the time it works, but from time to time (every 100 to 1000 iterations)
the test fails: the main detects a deadlock because of no output activity during
more than 5 secs, and exits on error.
You can test on Unix with the following scripts (launch "./Loop"):
trw
---
#!/bin/bash
./t_read_write >out 2>err
Loop
----
#!/bin/bash
# Loop until failure
res=0
let n=0+1
while [ $res -eq 0 ] ; do
echo `date +"%Y-%m-%dT%H:%M:%S"`" Iteration "$n":"
./trw
if [ $? -ne 0 ] ; then
echo "Failed!"
exit 1
fi
let n=$n+1
done
When everything is OK all the tasks terminate one after the other, but in case
of failure, here is the output:
2017-10-28T21:15:21.139556000 Main Starting
2017-10-28T21:15:21.139602000 1 Task started, id tasks(1)_0000000000E63F10
2017-10-28T21:15:21.139618000 2 Task started, id tasks(2)_0000000000E68040
2017-10-28T21:15:21.139644000 3 Task started, id tasks(3)_0000000000E6B650
2017-10-28T21:15:21.139662000 4 Task started, id tasks(4)_0000000000E6EC60
2017-10-28T21:15:21.139679000 5 Task started, id tasks(5)_0000000000E72270
2017-10-28T21:15:21.139695000 6 Task started, id tasks(6)_0000000000E75880
2017-10-28T21:15:21.139709000 7 Task started, id tasks(7)_0000000000E78E90
2017-10-28T21:15:21.139732000 8 Task started, id tasks(8)_0000000000E7C4A0
2017-10-28T21:15:21.139752000 9 Task started, id tasks(9)_0000000000E7FAB0
2017-10-28T21:15:21.139759000 10 Task started, id tasks(10)_0000000000E830C0
2017-10-28T21:15:21.149687000 1 get Read 0.0
2017-10-28T21:15:21.149690000 3 get Write 0.1
2017-10-28T21:15:21.149700000 2 get Read 0.0
Read_Write get queueing tasks(1)_0000000000E63F10
2017-10-28T21:15:21.149744000 5 get Read 0.0
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:21.149782000 4 get Write 0.5
Read_Write get queueing tasks(3)_0000000000E6B650
Writer waits for current reader tasks(3)_0000000000E6B650
2017-10-28T21:15:21.149804000 8 get Read 0.1
2017-10-28T21:15:21.149786000 1 OK
2017-10-28T21:15:21.149769000 6 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
2017-10-28T21:15:21.149825000 9 get Read 0.5
2017-10-28T21:15:21.149829000 2 NOK
Read_Write get queueing tasks(9)_0000000000E7FAB0
2017-10-28T21:15:21.149855000 10 get Read 0.0
2017-10-28T21:15:21.149875000 7 get Read 0.2
2017-10-28T21:15:21.149816000 1 waiting
Read_Write get queueing tasks(4)_0000000000E6EC60
Read_Write get queueing tasks(8)_0000000000E7C4A0
Read_Write get queueing tasks(6)_0000000000E75880
Read_Write get queueing tasks(5)_0000000000E72270
2017-10-28T21:15:21.149917000 6 NOK
2017-10-28T21:15:21.149926000 5 NOK
Read_Write get queueing tasks(10)_0000000000E830C0
Read_Write get queueing tasks(7)_0000000000E78E90
2017-10-28T21:15:21.149940000 10 NOK
2017-10-28T21:15:21.159909000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
2017-10-28T21:15:21.159990000 5 get Read 0.3
Read_Write get queueing tasks(5)_0000000000E72270
2017-10-28T21:15:21.160006000 10 get Read 0.5
2017-10-28T21:15:21.159993000 6 get Read 0.0
Read_Write get queueing tasks(6)_0000000000E75880
2017-10-28T21:15:21.160032000 6 NOK
Read_Write get queueing tasks(10)_0000000000E830C0
2017-10-28T21:15:21.170103000 6 get Read 0.4
Read_Write get queueing tasks(6)_0000000000E75880
2017-10-28T21:15:21.249871000 3 NOK
2017-10-28T21:15:21.249967000 8 NOK
2017-10-28T21:15:21.259960000 2 NOK
2017-10-28T21:15:21.259964000 3 get Read 0.1
Read_Write get queueing tasks(3)_0000000000E6B650
2017-10-28T21:15:21.260036000 8 get Read -0.1
Read_Write get queueing tasks(8)_0000000000E7C4A0
2017-10-28T21:15:21.270048000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
2017-10-28T21:15:21.270066000 2 NOK
2017-10-28T21:15:21.280139000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
2017-10-28T21:15:21.349953000 1 waited
2017-10-28T21:15:21.349976000 1 release
Last reader releases tasks(1)_0000000000E63F10
Another reader tasks(9)_0000000000E7FAB0
Start swapping tasks(4)_0000000000E6EC60
Swapping tasks(7)_0000000000E78E90
2017-10-28T21:15:21.350007000 9 OK
2017-10-28T21:15:21.350051000 9 waiting
Swapping tasks(5)_0000000000E72270
2017-10-28T21:15:21.360060000 1 get Read 0.0
Swapping tasks(10)_0000000000E830C0
Swapping tasks(6)_0000000000E75880
Swapping tasks(8)_0000000000E7C4A0
Swapping tasks(2)_0000000000E68040
End swapping tasks(2)_0000000000E68040
Read_Write get queueing tasks(1)_0000000000E63F10
2017-10-28T21:15:21.360109000 3 NOK
2017-10-28T21:15:21.360121000 1 NOK
2017-10-28T21:15:21.370195000 3 get Read 0.3
Read_Write get queueing tasks(3)_0000000000E6B650
2017-10-28T21:15:21.370204000 1 get Write 0.0
Read_Write get queueing tasks(1)_0000000000E63F10
2017-10-28T21:15:21.370242000 1 NOK
2017-10-28T21:15:21.380223000 2 NOK
2017-10-28T21:15:21.380324000 1 get Read -0.1
Read_Write get queueing tasks(1)_0000000000E63F10
2017-10-28T21:15:21.390298000 2 get Read 0.2
Read_Write get queueing tasks(2)_0000000000E68040
2017-10-28T21:15:21.460071000 5 NOK
2017-10-28T21:15:21.470153000 5 get Read 0.0
Read_Write get queueing tasks(5)_0000000000E72270
2017-10-28T21:15:21.470170000 5 NOK
2017-10-28T21:15:21.480242000 5 get Read 0.1
Read_Write get queueing tasks(5)_0000000000E72270
2017-10-28T21:15:21.570194000 6 NOK
2017-10-28T21:15:21.580275000 6 get Read 0.0
Read_Write get queueing tasks(6)_0000000000E75880
2017-10-28T21:15:21.580293000 6 NOK
2017-10-28T21:15:21.580321000 5 NOK
2017-10-28T21:15:21.590365000 2 NOK
2017-10-28T21:15:21.590374000 6 get Read 0.4
Read_Write get queueing tasks(6)_0000000000E75880
2017-10-28T21:15:21.590401000 5 get Read 0.2
Read_Write get queueing tasks(5)_0000000000E72270
2017-10-28T21:15:21.600438000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
2017-10-28T21:15:21.649962000 4 NOK
2017-10-28T21:15:21.660056000 4 get Read 0.2
Read_Write get queueing tasks(4)_0000000000E6EC60
2017-10-28T21:15:21.660107000 10 NOK
2017-10-28T21:15:21.670184000 10 get Read 0.0
Read_Write get queueing tasks(10)_0000000000E830C0
2017-10-28T21:15:21.670203000 10 NOK
2017-10-28T21:15:21.670286000 3 NOK
2017-10-28T21:15:21.680277000 10 get Read -0.1
Read_Write get queueing tasks(10)_0000000000E830C0
2017-10-28T21:15:21.680365000 3 get Read 0.3
Read_Write get queueing tasks(3)_0000000000E6B650
2017-10-28T21:15:21.750125000 9 waited
2017-10-28T21:15:21.750140000 9 release
Last reader releases tasks(9)_0000000000E7FAB0
Another reader tasks(8)_0000000000E7C4A0
Another reader tasks(1)_0000000000E63F10
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:21.750191000 1 OK
2017-10-28T21:15:21.750202000 6 OK
Another reader tasks(5)_0000000000E72270
2017-10-28T21:15:21.750215000 6 waiting
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:21.750166000 8 OK
2017-10-28T21:15:21.750251000 5 OK
Another reader tasks(4)_0000000000E6EC60
2017-10-28T21:15:21.750264000 5 waiting
2017-10-28T21:15:21.750275000 4 OK
2017-10-28T21:15:21.750286000 2 OK
2017-10-28T21:15:21.750298000 2 waiting
Another reader tasks(10)_0000000000E830C0
2017-10-28T21:15:21.750287000 4 waiting
2017-10-28T21:15:21.750254000 8 waiting
Another reader tasks(3)_0000000000E6B650
2017-10-28T21:15:21.750353000 9 terminate
2017-10-28T21:15:21.750356000 3 OK
2017-10-28T21:15:21.750205000 1 waiting
2017-10-28T21:15:21.750381000 3 waiting
2017-10-28T21:15:21.750338000 10 OK
2017-10-28T21:15:21.750420000 10 waiting
2017-10-28T21:15:21.950478000 3 waited
2017-10-28T21:15:21.950503000 3 release
Reader releases tasks(3)_0000000000E6B650
2017-10-28T21:15:21.960583000 3 get Read 0.0
Read_Write get queueing tasks(3)_0000000000E6B650
Another reader tasks(3)_0000000000E6B650
2017-10-28T21:15:21.960607000 3 OK
2017-10-28T21:15:21.960615000 3 waiting
2017-10-28T21:15:22.150289000 6 waited
2017-10-28T21:15:22.150308000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:22.150351000 5 waited
2017-10-28T21:15:22.150361000 5 release
Reader releases tasks(5)_0000000000E72270
2017-10-28T21:15:22.150377000 2 waited
2017-10-28T21:15:22.150386000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:22.150411000 4 waited
2017-10-28T21:15:22.150425000 4 release
Reader releases tasks(4)_0000000000E6EC60
2017-10-28T21:15:22.150444000 8 waited
2017-10-28T21:15:22.150453000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:22.150488000 10 waited
2017-10-28T21:15:22.150498000 10 release
Reader releases tasks(10)_0000000000E830C0
2017-10-28T21:15:22.160386000 5 get Read 0.3
2017-10-28T21:15:22.160388000 6 get Read 0.3
Read_Write get queueing tasks(5)_0000000000E72270
Another reader tasks(5)_0000000000E72270
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:22.160455000 2 get Read 0.1
2017-10-28T21:15:22.160498000 4 get Read 0.2
2017-10-28T21:15:22.160457000 6 OK
2017-10-28T21:15:22.160517000 6 waiting
Read_Write get queueing tasks(2)_0000000000E68040
2017-10-28T21:15:22.160433000 5 OK
2017-10-28T21:15:22.160542000 8 get Read 0.0
2017-10-28T21:15:22.160547000 5 waiting
Another reader tasks(2)_0000000000E68040
Read_Write get queueing tasks(4)_0000000000E6EC60
Another reader tasks(4)_0000000000E6EC60
2017-10-28T21:15:22.160580000 2 OK
2017-10-28T21:15:22.160611000 4 OK
Read_Write get queueing tasks(8)_0000000000E7C4A0
2017-10-28T21:15:22.160578000 10 get Read 0.4
2017-10-28T21:15:22.160624000 4 waiting
Another reader tasks(8)_0000000000E7C4A0
Read_Write get queueing tasks(10)_0000000000E830C0
Another reader tasks(10)_0000000000E830C0
2017-10-28T21:15:22.160615000 2 waiting
2017-10-28T21:15:22.160655000 8 OK
2017-10-28T21:15:22.160674000 10 OK
2017-10-28T21:15:22.160692000 8 waiting
2017-10-28T21:15:22.160702000 10 waiting
2017-10-28T21:15:22.250462000 1 waited
2017-10-28T21:15:22.250475000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:22.260551000 1 get Read 0.0
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:22.260571000 1 OK
2017-10-28T21:15:22.260579000 1 waiting
2017-10-28T21:15:22.260775000 10 waited
2017-10-28T21:15:22.260785000 10 release
Reader releases tasks(10)_0000000000E830C0
2017-10-28T21:15:22.270860000 10 get Read -0.1
Read_Write get queueing tasks(10)_0000000000E830C0
Another reader tasks(10)_0000000000E830C0
2017-10-28T21:15:22.270879000 10 OK
2017-10-28T21:15:22.270887000 10 waiting
2017-10-28T21:15:22.360594000 6 waited
2017-10-28T21:15:22.360619000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:22.360646000 1 waited
2017-10-28T21:15:22.360658000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:22.360684000 4 waited
2017-10-28T21:15:22.360686000 3 waited
2017-10-28T21:15:22.360696000 4 release
Reader releases tasks(4)_0000000000E6EC60
2017-10-28T21:15:22.360706000 3 release
2017-10-28T21:15:22.360723000 4 terminate
Reader releases tasks(3)_0000000000E6B650
2017-10-28T21:15:22.360743000 2 waited
2017-10-28T21:15:22.360767000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:22.370699000 6 get Read -0.1
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:22.370720000 6 OK
2017-10-28T21:15:22.370728000 6 waiting
2017-10-28T21:15:22.370735000 1 get Read 0.0
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:22.370754000 1 OK
2017-10-28T21:15:22.370762000 1 waiting
2017-10-28T21:15:22.370817000 3 get Read 0.0
Read_Write get queueing tasks(3)_0000000000E6B650
Another reader tasks(3)_0000000000E6B650
2017-10-28T21:15:22.370835000 3 OK
2017-10-28T21:15:22.370838000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:22.370843000 3 waiting
2017-10-28T21:15:22.370866000 2 OK
2017-10-28T21:15:22.370882000 2 waiting
2017-10-28T21:15:22.460633000 5 waited
2017-10-28T21:15:22.460653000 5 release
Reader releases tasks(5)_0000000000E72270
2017-10-28T21:15:22.460667000 5 terminate
2017-10-28T21:15:22.560780000 8 waited
2017-10-28T21:15:22.560798000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:22.570880000 8 get Read 0.3
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:22.570902000 8 OK
2017-10-28T21:15:22.570911000 8 waiting
2017-10-28T21:15:22.570949000 2 waited
2017-10-28T21:15:22.570959000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:22.581035000 2 get Read 0.2
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:22.581054000 2 OK
2017-10-28T21:15:22.581062000 2 waiting
2017-10-28T21:15:22.670800000 6 waited
2017-10-28T21:15:22.670856000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:22.680902000 6 get Read 0.2
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:22.680930000 6 OK
2017-10-28T21:15:22.680938000 6 waiting
2017-10-28T21:15:22.770837000 1 waited
2017-10-28T21:15:22.770857000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:22.770938000 3 waited
2017-10-28T21:15:22.770939000 10 waited
2017-10-28T21:15:22.770951000 3 release
Reader releases tasks(3)_0000000000E6B650
2017-10-28T21:15:22.770965000 10 release
Reader releases tasks(10)_0000000000E830C0
2017-10-28T21:15:22.780936000 1 get Read 0.1
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:22.780975000 1 OK
2017-10-28T21:15:22.780983000 1 waiting
2017-10-28T21:15:22.781041000 3 get Read 0.2
Read_Write get queueing tasks(3)_0000000000E6B650
Another reader tasks(3)_0000000000E6B650
2017-10-28T21:15:22.781059000 10 get Read 0.0
Read_Write get queueing tasks(10)_0000000000E830C0
Another reader tasks(10)_0000000000E830C0
2017-10-28T21:15:22.781059000 3 OK
2017-10-28T21:15:22.781092000 10 OK
2017-10-28T21:15:22.781102000 3 waiting
2017-10-28T21:15:22.781112000 10 waiting
2017-10-28T21:15:22.881010000 6 waited
2017-10-28T21:15:22.881023000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:22.891099000 6 get Read 0.4
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:22.891118000 6 OK
2017-10-28T21:15:22.891126000 6 waiting
2017-10-28T21:15:22.981185000 3 waited
2017-10-28T21:15:22.981200000 3 release
Reader releases tasks(3)_0000000000E6B650
2017-10-28T21:15:22.991277000 3 get Read 0.3
Read_Write get queueing tasks(3)_0000000000E6B650
Another reader tasks(3)_0000000000E6B650
2017-10-28T21:15:22.991297000 3 OK
2017-10-28T21:15:22.991305000 3 waiting
2017-10-28T21:15:23.070985000 8 waited
2017-10-28T21:15:23.071001000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:23.081054000 1 waited
2017-10-28T21:15:23.081066000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:23.081078000 8 get Read 0.4
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:23.081098000 8 OK
2017-10-28T21:15:23.081106000 8 waiting
2017-10-28T21:15:23.081130000 2 waited
2017-10-28T21:15:23.081139000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:23.091142000 1 get Read 0.0
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:23.091162000 1 OK
2017-10-28T21:15:23.091170000 1 waiting
2017-10-28T21:15:23.091220000 2 get Read 0.2
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:23.091242000 2 OK
2017-10-28T21:15:23.091250000 2 waiting
2017-10-28T21:15:23.091369000 3 waited
2017-10-28T21:15:23.091379000 3 release
Reader releases tasks(3)_0000000000E6B650
2017-10-28T21:15:23.091391000 3 terminate
2017-10-28T21:15:23.181190000 10 waited
2017-10-28T21:15:23.181206000 10 release
Reader releases tasks(10)_0000000000E830C0
2017-10-28T21:15:23.191284000 10 get Read 0.4
Read_Write get queueing tasks(10)_0000000000E830C0
Another reader tasks(10)_0000000000E830C0
2017-10-28T21:15:23.191304000 10 OK
2017-10-28T21:15:23.191312000 10 waiting
2017-10-28T21:15:23.291193000 6 waited
2017-10-28T21:15:23.291207000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:23.291316000 2 waited
2017-10-28T21:15:23.291329000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:23.301282000 6 get Read 0.1
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:23.301302000 6 OK
2017-10-28T21:15:23.301310000 6 waiting
2017-10-28T21:15:23.301406000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:23.301442000 2 OK
2017-10-28T21:15:23.301450000 2 waiting
2017-10-28T21:15:23.381183000 8 waited
2017-10-28T21:15:23.381228000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:23.391314000 8 get Read 0.2
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:23.391339000 8 OK
2017-10-28T21:15:23.391347000 8 waiting
2017-10-28T21:15:23.501526000 2 waited
2017-10-28T21:15:23.501556000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:23.511639000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:23.511663000 2 OK
2017-10-28T21:15:23.511671000 2 waiting
2017-10-28T21:15:23.591239000 1 waited
2017-10-28T21:15:23.591261000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:23.591417000 8 waited
2017-10-28T21:15:23.591432000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:23.601334000 1 get Read 0.2
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:23.601356000 1 OK
2017-10-28T21:15:23.601365000 1 waiting
2017-10-28T21:15:23.601513000 8 get Read 0.5
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:23.601535000 8 OK
2017-10-28T21:15:23.601544000 8 waiting
2017-10-28T21:15:23.691386000 10 waited
2017-10-28T21:15:23.691426000 10 release
Reader releases tasks(10)_0000000000E830C0
2017-10-28T21:15:23.701508000 10 get Read -0.1
Read_Write get queueing tasks(10)_0000000000E830C0
Another reader tasks(10)_0000000000E830C0
2017-10-28T21:15:23.701530000 10 OK
2017-10-28T21:15:23.701538000 10 waiting
2017-10-28T21:15:23.801385000 6 waited
2017-10-28T21:15:23.801402000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:23.811480000 6 get Read 0.3
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:23.811501000 6 OK
2017-10-28T21:15:23.811509000 6 waiting
2017-10-28T21:15:23.901439000 1 waited
2017-10-28T21:15:23.901452000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:23.901611000 10 waited
2017-10-28T21:15:23.901623000 10 release
Reader releases tasks(10)_0000000000E830C0
2017-10-28T21:15:23.901637000 10 terminate
2017-10-28T21:15:23.911529000 1 get Read 0.0
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:23.911553000 1 OK
2017-10-28T21:15:23.911561000 1 waiting
2017-10-28T21:15:23.911744000 2 waited
2017-10-28T21:15:23.911758000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:23.921835000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:23.921856000 2 OK
2017-10-28T21:15:23.921864000 2 waiting
2017-10-28T21:15:24.101621000 8 waited
2017-10-28T21:15:24.101640000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:24.111575000 6 waited
2017-10-28T21:15:24.111589000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:24.111717000 8 get Read 0.3
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:24.111737000 8 OK
2017-10-28T21:15:24.111745000 8 waiting
2017-10-28T21:15:24.121665000 6 get Read 0.2
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:24.121684000 6 OK
2017-10-28T21:15:24.121692000 6 waiting
2017-10-28T21:15:24.211634000 1 waited
2017-10-28T21:15:24.211658000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:24.221736000 6 waited
2017-10-28T21:15:24.221737000 1 get Read 0.0
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:24.221767000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:24.221793000 1 OK
2017-10-28T21:15:24.221834000 1 waiting
2017-10-28T21:15:24.221931000 2 waited
2017-10-28T21:15:24.221941000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:24.231875000 6 get Read 0.1
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:24.231895000 6 OK
2017-10-28T21:15:24.231903000 6 waiting
2017-10-28T21:15:24.232011000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:24.232030000 2 OK
2017-10-28T21:15:24.232037000 2 waiting
2017-10-28T21:15:24.411815000 8 waited
2017-10-28T21:15:24.411839000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:24.421919000 8 get Read -0.1
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:24.421940000 8 OK
2017-10-28T21:15:24.421948000 8 waiting
2017-10-28T21:15:24.431974000 6 waited
2017-10-28T21:15:24.431986000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:24.442064000 6 get Read 0.3
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:24.442087000 6 OK
2017-10-28T21:15:24.442094000 6 waiting
2017-10-28T21:15:24.521907000 1 waited
2017-10-28T21:15:24.521922000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:24.531998000 1 get Read 0.2
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:24.532018000 1 OK
2017-10-28T21:15:24.532026000 1 waiting
2017-10-28T21:15:24.532108000 2 waited
2017-10-28T21:15:24.532119000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:24.542201000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:24.542224000 2 OK
2017-10-28T21:15:24.542232000 2 waiting
2017-10-28T21:15:24.622024000 8 waited
2017-10-28T21:15:24.622040000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:24.632119000 8 get Read 0.2
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:24.632139000 8 OK
2017-10-28T21:15:24.632147000 8 waiting
2017-10-28T21:15:24.832096000 1 waited
2017-10-28T21:15:24.832133000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:24.842170000 6 waited
2017-10-28T21:15:24.842188000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:24.842214000 1 get Read -0.1
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:24.842235000 1 OK
2017-10-28T21:15:24.842243000 1 waiting
2017-10-28T21:15:24.852266000 6 get Read 0.0
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:24.852287000 6 OK
2017-10-28T21:15:24.852295000 6 waiting
2017-10-28T21:15:24.932222000 8 waited
2017-10-28T21:15:24.932238000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:24.942304000 8 get Read 0.5
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:24.942309000 2 waited
2017-10-28T21:15:24.942346000 8 OK
2017-10-28T21:15:24.942357000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:24.942367000 8 waiting
2017-10-28T21:15:24.952453000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:24.952480000 2 OK
2017-10-28T21:15:24.952488000 2 waiting
2017-10-28T21:15:25.052369000 6 waited
2017-10-28T21:15:25.052384000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:25.062461000 6 get Read 0.4
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:25.062481000 6 OK
2017-10-28T21:15:25.062489000 6 waiting
2017-10-28T21:15:25.162556000 6 waited
2017-10-28T21:15:25.162575000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:25.172668000 6 get Read -0.1
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:25.172689000 6 OK
2017-10-28T21:15:25.172697000 6 waiting
2017-10-28T21:15:25.242313000 1 waited
2017-10-28T21:15:25.242329000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:25.252407000 1 get Read 0.0
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:25.252428000 1 OK
2017-10-28T21:15:25.252436000 1 waiting
2017-10-28T21:15:25.342454000 8 waited
2017-10-28T21:15:25.342491000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:25.352573000 8 get Read 0.4
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:25.352594000 8 OK
2017-10-28T21:15:25.352602000 8 waiting
2017-10-28T21:15:25.452564000 2 waited
2017-10-28T21:15:25.452584000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:25.452672000 8 waited
2017-10-28T21:15:25.452686000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:25.462663000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:25.462683000 2 OK
2017-10-28T21:15:25.462691000 2 waiting
2017-10-28T21:15:25.462762000 8 get Read 0.2
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:25.462781000 8 OK
2017-10-28T21:15:25.462789000 8 waiting
2017-10-28T21:15:25.572774000 6 waited
2017-10-28T21:15:25.572789000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:25.582868000 6 get Read 0.1
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:25.582889000 6 OK
2017-10-28T21:15:25.582897000 6 waiting
2017-10-28T21:15:25.662860000 8 waited
2017-10-28T21:15:25.662894000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:25.672970000 8 get Read 0.5
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:25.672993000 8 OK
2017-10-28T21:15:25.673001000 8 waiting
2017-10-28T21:15:25.752510000 1 waited
2017-10-28T21:15:25.752531000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:25.762609000 1 get Read 0.3
Read_Write get queueing tasks(1)_0000000000E63F10
Another reader tasks(1)_0000000000E63F10
2017-10-28T21:15:25.762630000 1 OK
2017-10-28T21:15:25.762638000 1 waiting
2017-10-28T21:15:25.862708000 1 waited
2017-10-28T21:15:25.862720000 1 release
Reader releases tasks(1)_0000000000E63F10
2017-10-28T21:15:25.862732000 1 terminate
2017-10-28T21:15:25.862767000 2 waited
2017-10-28T21:15:25.862783000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:25.872861000 2 get Read 0.2
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:25.872882000 2 OK
2017-10-28T21:15:25.872890000 2 waiting
2017-10-28T21:15:26.073070000 8 waited
2017-10-28T21:15:26.073087000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:26.082972000 6 waited
2017-10-28T21:15:26.082988000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:26.083163000 8 get Read 0.2
Read_Write get queueing tasks(8)_0000000000E7C4A0
Another reader tasks(8)_0000000000E7C4A0
2017-10-28T21:15:26.083181000 8 OK
2017-10-28T21:15:26.083189000 8 waiting
2017-10-28T21:15:26.093071000 6 get Read 0.3
Read_Write get queueing tasks(6)_0000000000E75880
Another reader tasks(6)_0000000000E75880
2017-10-28T21:15:26.093095000 6 OK
2017-10-28T21:15:26.093103000 6 waiting
2017-10-28T21:15:26.172971000 2 waited
2017-10-28T21:15:26.172987000 2 release
Reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:26.183059000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:26.183080000 2 OK
2017-10-28T21:15:26.183106000 2 waiting
2017-10-28T21:15:26.383260000 8 waited
2017-10-28T21:15:26.383286000 8 release
Reader releases tasks(8)_0000000000E7C4A0
2017-10-28T21:15:26.383301000 8 terminate
2017-10-28T21:15:26.393182000 6 waited
2017-10-28T21:15:26.393198000 6 release
Reader releases tasks(6)_0000000000E75880
2017-10-28T21:15:26.393212000 6 terminate
2017-10-28T21:15:26.483178000 2 waited
2017-10-28T21:15:26.483208000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:26.493292000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:26.493316000 2 OK
2017-10-28T21:15:26.493325000 2 waiting
2017-10-28T21:15:26.993401000 2 waited
2017-10-28T21:15:26.993437000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:27.003520000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:27.003542000 2 OK
2017-10-28T21:15:27.003550000 2 waiting
2017-10-28T21:15:27.203619000 2 waited
2017-10-28T21:15:27.203647000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:27.213722000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:27.213746000 2 OK
2017-10-28T21:15:27.213754000 2 waiting
2017-10-28T21:15:27.513830000 2 waited
2017-10-28T21:15:27.513855000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:27.523934000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:27.523956000 2 OK
2017-10-28T21:15:27.523964000 2 waiting
2017-10-28T21:15:27.624041000 2 waited
2017-10-28T21:15:27.624061000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:27.634138000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:27.634166000 2 OK
2017-10-28T21:15:27.634175000 2 waiting
2017-10-28T21:15:28.034250000 2 waited
2017-10-28T21:15:28.034286000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:28.044367000 2 get Read 0.2
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:28.044390000 2 OK
2017-10-28T21:15:28.044398000 2 waiting
2017-10-28T21:15:28.144463000 2 waited
2017-10-28T21:15:28.144477000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:28.154554000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:28.154574000 2 OK
2017-10-28T21:15:28.154582000 2 waiting
2017-10-28T21:15:28.554654000 2 waited
2017-10-28T21:15:28.554694000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:28.564788000 2 get Read 0.2
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:28.564809000 2 OK
2017-10-28T21:15:28.564817000 2 waiting
2017-10-28T21:15:28.964892000 2 waited
2017-10-28T21:15:28.964930000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:28.975011000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:28.975033000 2 OK
2017-10-28T21:15:28.975041000 2 waiting
2017-10-28T21:15:29.375116000 2 waited
2017-10-28T21:15:29.375152000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:29.385233000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:29.385256000 2 OK
2017-10-28T21:15:29.385264000 2 waiting
2017-10-28T21:15:29.585340000 2 waited
2017-10-28T21:15:29.585369000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:29.595450000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:29.595472000 2 OK
2017-10-28T21:15:29.595499000 2 waiting
2017-10-28T21:15:29.995580000 2 waited
2017-10-28T21:15:29.995616000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:30.005699000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:30.005720000 2 OK
2017-10-28T21:15:30.005728000 2 waiting
2017-10-28T21:15:30.405807000 2 waited
2017-10-28T21:15:30.405841000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:30.415924000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:30.415949000 2 OK
2017-10-28T21:15:30.415957000 2 waiting
2017-10-28T21:15:30.916029000 2 waited
2017-10-28T21:15:30.916066000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:30.926148000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:30.926170000 2 OK
2017-10-28T21:15:30.926178000 2 waiting
2017-10-28T21:15:31.026251000 2 waited
2017-10-28T21:15:31.026271000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:31.036349000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:31.036370000 2 OK
2017-10-28T21:15:31.036378000 2 waiting
2017-10-28T21:15:31.436457000 2 waited
2017-10-28T21:15:31.436496000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:31.446579000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:31.446603000 2 OK
2017-10-28T21:15:31.446611000 2 waiting
2017-10-28T21:15:31.746681000 2 waited
2017-10-28T21:15:31.746717000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:31.756798000 2 get Read 0.5
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:31.756820000 2 OK
2017-10-28T21:15:31.756828000 2 waiting
2017-10-28T21:15:32.056900000 2 waited
2017-10-28T21:15:32.056916000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:32.066994000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:32.067014000 2 OK
2017-10-28T21:15:32.067022000 2 waiting
2017-10-28T21:15:32.167094000 2 waited
2017-10-28T21:15:32.167110000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:32.177187000 2 get Write 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Writer has got lock tasks(2)_0000000000E68040
2017-10-28T21:15:32.177207000 2 OK
2017-10-28T21:15:32.177214000 2 waiting
2017-10-28T21:15:32.477286000 2 waited
2017-10-28T21:15:32.477325000 2 release
Writer releases tasks(2)_0000000000E68040
2017-10-28T21:15:32.487402000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:32.487424000 2 OK
2017-10-28T21:15:32.487432000 2 waiting
2017-10-28T21:15:32.687509000 2 waited
2017-10-28T21:15:32.687547000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:32.697628000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:32.697651000 2 OK
2017-10-28T21:15:32.697659000 2 waiting
2017-10-28T21:15:32.897731000 2 waited
2017-10-28T21:15:32.897745000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:32.907822000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:32.907842000 2 OK
2017-10-28T21:15:32.907850000 2 waiting
2017-10-28T21:15:33.307919000 2 waited
2017-10-28T21:15:33.307952000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:33.318032000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:33.318071000 2 OK
2017-10-28T21:15:33.318080000 2 waiting
2017-10-28T21:15:33.718160000 2 waited
2017-10-28T21:15:33.718215000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:33.728307000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:33.728329000 2 OK
2017-10-28T21:15:33.728337000 2 waiting
2017-10-28T21:15:34.228412000 2 waited
2017-10-28T21:15:34.228445000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:34.238526000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:34.238548000 2 OK
2017-10-28T21:15:34.238556000 2 waiting
2017-10-28T21:15:34.538629000 2 waited
2017-10-28T21:15:34.538647000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:34.548728000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:34.548752000 2 OK
2017-10-28T21:15:34.548761000 2 waiting
2017-10-28T21:15:34.948840000 2 waited
2017-10-28T21:15:34.948878000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:34.958963000 2 get Write 0.2
Read_Write get queueing tasks(2)_0000000000E68040
Writer has got lock tasks(2)_0000000000E68040
2017-10-28T21:15:34.958987000 2 OK
2017-10-28T21:15:34.958995000 2 waiting
2017-10-28T21:15:35.259070000 2 waited
2017-10-28T21:15:35.259088000 2 release
Writer releases tasks(2)_0000000000E68040
2017-10-28T21:15:35.269165000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:35.269186000 2 OK
2017-10-28T21:15:35.269194000 2 waiting
2017-10-28T21:15:35.569274000 2 waited
2017-10-28T21:15:35.569311000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:35.579396000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:35.579420000 2 OK
2017-10-28T21:15:35.579428000 2 waiting
2017-10-28T21:15:35.879506000 2 waited
2017-10-28T21:15:35.879541000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:35.889623000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:35.889644000 2 OK
2017-10-28T21:15:35.889652000 2 waiting
2017-10-28T21:15:36.189721000 2 waited
2017-10-28T21:15:36.189744000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:36.199823000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:36.199844000 2 OK
2017-10-28T21:15:36.199852000 2 waiting
2017-10-28T21:15:36.599930000 2 waited
2017-10-28T21:15:36.599964000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:36.610050000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:36.610076000 2 OK
2017-10-28T21:15:36.610084000 2 waiting
2017-10-28T21:15:37.010162000 2 waited
2017-10-28T21:15:37.010200000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:37.020282000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:37.020305000 2 OK
2017-10-28T21:15:37.020313000 2 waiting
2017-10-28T21:15:37.420390000 2 waited
2017-10-28T21:15:37.420417000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:37.430498000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:37.430519000 2 OK
2017-10-28T21:15:37.430527000 2 waiting
2017-10-28T21:15:37.730596000 2 waited
2017-10-28T21:15:37.730633000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:37.740713000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:37.740754000 2 OK
2017-10-28T21:15:37.740762000 2 waiting
2017-10-28T21:15:38.140840000 2 waited
2017-10-28T21:15:38.140862000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:38.150942000 2 get Read 0.5
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:38.150963000 2 OK
2017-10-28T21:15:38.150972000 2 waiting
2017-10-28T21:15:38.551046000 2 waited
2017-10-28T21:15:38.551084000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:38.561169000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:38.561193000 2 OK
2017-10-28T21:15:38.561201000 2 waiting
2017-10-28T21:15:38.761271000 2 waited
2017-10-28T21:15:38.761315000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:38.771407000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:38.771429000 2 OK
2017-10-28T21:15:38.771437000 2 waiting
2017-10-28T21:15:39.171511000 2 waited
2017-10-28T21:15:39.171539000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:39.181620000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:39.181645000 2 OK
2017-10-28T21:15:39.181653000 2 waiting
2017-10-28T21:15:39.381735000 2 waited
2017-10-28T21:15:39.381777000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:39.391859000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:39.391881000 2 OK
2017-10-28T21:15:39.391889000 2 waiting
2017-10-28T21:15:39.691968000 2 waited
2017-10-28T21:15:39.692008000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:39.702089000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:39.702111000 2 OK
2017-10-28T21:15:39.702119000 2 waiting
2017-10-28T21:15:40.002188000 2 waited
2017-10-28T21:15:40.002214000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:40.012293000 2 get Write 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Writer has got lock tasks(2)_0000000000E68040
2017-10-28T21:15:40.012314000 2 OK
2017-10-28T21:15:40.012322000 2 waiting
2017-10-28T21:15:40.412390000 2 waited
2017-10-28T21:15:40.412415000 2 release
Writer releases tasks(2)_0000000000E68040
2017-10-28T21:15:40.422496000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:40.422517000 2 OK
2017-10-28T21:15:40.422525000 2 waiting
2017-10-28T21:15:40.822601000 2 waited
2017-10-28T21:15:40.822637000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:40.832719000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:40.832741000 2 OK
2017-10-28T21:15:40.832749000 2 waiting
2017-10-28T21:15:41.132824000 2 waited
2017-10-28T21:15:41.132845000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:41.142924000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:41.142945000 2 OK
2017-10-28T21:15:41.142953000 2 waiting
2017-10-28T21:15:41.443034000 2 waited
2017-10-28T21:15:41.443077000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:41.453160000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:41.453184000 2 OK
2017-10-28T21:15:41.453192000 2 waiting
2017-10-28T21:15:41.753267000 2 waited
2017-10-28T21:15:41.753306000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:41.763387000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:41.763426000 2 OK
2017-10-28T21:15:41.763434000 2 waiting
2017-10-28T21:15:41.863510000 2 waited
2017-10-28T21:15:41.863525000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:41.873605000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:41.873629000 2 OK
2017-10-28T21:15:41.873637000 2 waiting
2017-10-28T21:15:42.273708000 2 waited
2017-10-28T21:15:42.273741000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:42.283819000 2 get Read 0.3
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:42.283844000 2 OK
2017-10-28T21:15:42.283853000 2 waiting
2017-10-28T21:15:42.583929000 2 waited
2017-10-28T21:15:42.583954000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:42.594034000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:42.594056000 2 OK
2017-10-28T21:15:42.594065000 2 waiting
2017-10-28T21:15:42.794142000 2 waited
2017-10-28T21:15:42.794182000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:42.804264000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:42.804287000 2 OK
2017-10-28T21:15:42.804296000 2 waiting
2017-10-28T21:15:43.104374000 2 waited
2017-10-28T21:15:43.104400000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:43.114481000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:43.114504000 2 OK
2017-10-28T21:15:43.114513000 2 waiting
2017-10-28T21:15:43.314585000 2 waited
2017-10-28T21:15:43.314605000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:43.324684000 2 get Read 0.0
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:43.324707000 2 OK
2017-10-28T21:15:43.324715000 2 waiting
2017-10-28T21:15:43.724984000 2 waited
2017-10-28T21:15:43.725023000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:43.735103000 2 get Read -0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:43.735134000 2 OK
2017-10-28T21:15:43.735143000 2 waiting
2017-10-28T21:15:44.035217000 2 waited
2017-10-28T21:15:44.035255000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:44.045346000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:44.045369000 2 OK
2017-10-28T21:15:44.045377000 2 waiting
2017-10-28T21:15:44.245450000 2 waited
2017-10-28T21:15:44.245471000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:44.255551000 2 get Read 0.5
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:44.255574000 2 OK
2017-10-28T21:15:44.255582000 2 waiting
2017-10-28T21:15:44.355662000 2 waited
2017-10-28T21:15:44.355703000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:44.365786000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:44.365812000 2 OK
2017-10-28T21:15:44.365821000 2 waiting
2017-10-28T21:15:44.565899000 2 waited
2017-10-28T21:15:44.565930000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:44.576016000 2 get Read 0.1
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:44.576041000 2 OK
2017-10-28T21:15:44.576050000 2 waiting
2017-10-28T21:15:44.876120000 2 waited
2017-10-28T21:15:44.876158000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:44.886240000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:44.886281000 2 OK
2017-10-28T21:15:44.886290000 2 waiting
2017-10-28T21:15:45.186362000 2 waited
2017-10-28T21:15:45.186384000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:45.196462000 2 get Read 0.4
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:45.196483000 2 OK
2017-10-28T21:15:45.196492000 2 waiting
2017-10-28T21:15:45.296562000 2 waited
2017-10-28T21:15:45.296585000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:45.306664000 2 get Read 0.5
Read_Write get queueing tasks(2)_0000000000E68040
Another reader tasks(2)_0000000000E68040
2017-10-28T21:15:45.306685000 2 OK
2017-10-28T21:15:45.306693000 2 waiting
2017-10-28T21:15:45.706766000 2 waited
2017-10-28T21:15:45.706804000 2 release
Last reader releases tasks(2)_0000000000E68040
2017-10-28T21:15:45.706821000 2 terminate
2017-10-28T21:15:50.729534000 Main Deadlock detected, aborting
2017-10-28T21:15:50.729586000 Main Done
Task 7 did not terminate.
Last report of it was "21:15:21.149875000 7 get Read 0.2"
At this time, 1 has got the lock for read, 3 is requesting it for write and
the other tasks are queueing for read or write (7 is queueing for read).
When 1 releases the lock: 017-10-28T21:15:21.349976000 1 release
the log of swapping shows that task 4, requesting for write, triggers the
swapping: Start swapping tasks(4)_0000000000E6EC60
At that time, all the queueing tasks are logged as being swapped except 7,
and task 7 seems to be blocked somewhere and is never activated.
Can you reproduce it? On Windows?
Is the bug in the test program, or in the mutex implementation, in Linux, or in Gnat?
Thanks
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Read-write mutex sometimes fails on deadlock
2017-10-28 20:02 Read-write mutex sometimes fails on deadlock pascal.malaise
@ 2017-11-06 18:24 ` Robert Eachus
2017-11-06 18:31 ` Simon Wright
2017-11-06 21:01 ` Dmitry A. Kazakov
0 siblings, 2 replies; 8+ messages in thread
From: Robert Eachus @ 2017-11-06 18:24 UTC (permalink / raw)
On Saturday, October 28, 2017 at 4:02:33 PM UTC-4, pascal....@gmail.com wrote:
> Hi,
>
> [long post because it includes the full sources and output, sorry]
>
> I have a package that implements several kinds of mutexes, including a
> read-write mutex. Here is the full source (spec and body) with only the
> read-write part.
You got started down the wrong road and never turned around. The right abstraction for this in Ada is a protected object. A typical mutex looks like:
protected type Mutex is
entry Write(D: in Data):
function Read return Data;
private
The_Data: Data := No_Data;
end Mutex;
protected body Mutex is
entry Write (D: in Data) is
begin
The_Data := D;
Some_Data := True;
Print(D);
delay 0.1;
end Write;
function Read return Data is
return The_Data;
end Read;
end Mutex;
The choice of type Data, the value No_Data and the procedure Print is left to the reader. If you want to insure that calls to Print(D) are sequentialized, you can wrap it in its own protected object:
protected Writer is
procedure Write (D: in Data);
end Writer;
protected body Writer is
procedure Write(D: in Data) is
begin
-- do some output
end Write;
end Writer;
Now you can use Write to do some formatting of your output and be sure even if it is called by your tasks outside the Mutex, for example your tasks could print the data read.
There are thousands of useful patterns in Ada using protected objects or protected values. For example, you could allow exactly one reader after a Write, but allow two successive Writes. Or you could stack up reads before the first write, etc.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Read-write mutex sometimes fails on deadlock
2017-11-06 18:24 ` Robert Eachus
@ 2017-11-06 18:31 ` Simon Wright
2017-11-12 4:33 ` Robert Eachus
2017-11-06 21:01 ` Dmitry A. Kazakov
1 sibling, 1 reply; 8+ messages in thread
From: Simon Wright @ 2017-11-06 18:31 UTC (permalink / raw)
Robert Eachus <rieachus@comcast.net> writes:
> entry Write (D: in Data) is
> begin
> The_Data := D;
> Some_Data := True;
> Print(D);
> delay 0.1;
> end Write;
Is this not illegal? or at the very least wrong?
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Read-write mutex sometimes fails on deadlock
2017-11-06 18:31 ` Simon Wright
@ 2017-11-12 4:33 ` Robert Eachus
2017-11-12 5:21 ` J-P. Rosen
0 siblings, 1 reply; 8+ messages in thread
From: Robert Eachus @ 2017-11-12 4:33 UTC (permalink / raw)
On Monday, November 6, 2017 at 1:31:07 PM UTC-5, Simon Wright wrote:
> Robert Eachus <rieachus@comcast.net> writes:
>
> > entry Write (D: in Data) is
> > begin
> > The_Data := D;
> > Some_Data := True;
> > Print(D);
> > delay 0.1;
> > end Write;
>
> Is this not illegal? or at the very least wrong?
Don't get wrapped around the axle by random names in partial examples. Print might be a (non-blocking) routine that eventually calls an I/O routine like Ada.Text_IO.Put.
Having said that, the delay 0.1 is very naughty, but on most hardware it will be implemented as a counting loop. (Make it delay 0.0001 if that makes you happier. ;-) It is not illegal, and is only erroneous if it results in a scheduling action.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Read-write mutex sometimes fails on deadlock
2017-11-12 4:33 ` Robert Eachus
@ 2017-11-12 5:21 ` J-P. Rosen
2017-11-15 22:57 ` Randy Brukardt
0 siblings, 1 reply; 8+ messages in thread
From: J-P. Rosen @ 2017-11-12 5:21 UTC (permalink / raw)
Le 12/11/2017 à 05:33, Robert Eachus a écrit :
> On Monday, November 6, 2017 at 1:31:07 PM UTC-5, Simon Wright wrote:
>> Robert Eachus <rieachus@comcast.net> writes:
>>
>>> entry Write (D: in Data) is
>>> begin
>>> The_Data := D;
>>> Some_Data := True;
>>> Print(D);
>>> delay 0.1;
>>> end Write;
>>
> Having said that, the delay 0.1 is very naughty, but on most hardware it will be implemented as a counting loop. (Make it delay 0.0001 if that makes you happier. ;-) It is not illegal, and is only erroneous if it results in a scheduling action.
>
Delay is potentially blocking operation, so this is definitely a bounded
error. But it is allowed for a bounded error situation to give the
proper result...
--
J-P. Rosen
Adalog
2 rue du Docteur Lombard, 92441 Issy-les-Moulineaux CEDEX
Tel: +33 1 45 29 21 52, Fax: +33 1 45 29 25 00
http://www.adalog.fr
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Read-write mutex sometimes fails on deadlock
2017-11-12 5:21 ` J-P. Rosen
@ 2017-11-15 22:57 ` Randy Brukardt
2017-11-15 23:05 ` Randy Brukardt
0 siblings, 1 reply; 8+ messages in thread
From: Randy Brukardt @ 2017-11-15 22:57 UTC (permalink / raw)
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1308 bytes --]
"J-P. Rosen" <rosen@adalog.fr> wrote in message
news:ou8lod$3iq$3@dont-email.me...
> Le 12/11/2017 à 05:33, Robert Eachus a écrit :
>> On Monday, November 6, 2017 at 1:31:07 PM UTC-5, Simon Wright wrote:
>>> Robert Eachus <rieachus@comcast.net> writes:
>>>
>>>> entry Write (D: in Data) is
>>>> begin
>>>> The_Data := D;
>>>> Some_Data := True;
>>>> Print(D);
>>>> delay 0.1;
>>>> end Write;
>>>
>> Having said that, the delay 0.1 is very naughty, but on most hardware it
>> will be implemented as a counting loop. (Make it delay 0.0001 if that
>> makes you happier. ;-) It is not illegal, and is only erroneous if it
>> results in a scheduling action.
>>
> Delay is potentially blocking operation, so this is definitely a bounded
> error. But it is allowed for a bounded error situation to give the
> proper result...
Unless pragma Detect_Blocking is given (which should have been the default
in Ada, IMHO -- predictable is always better than "might work -- or not").
In that case, raising Program_Error is required.
If the new aspect Nonblocking is used, the delay is statically illegal.
(That *really* should have been the default, but it took us 20 years to
figure out the right set of rules -- too late to mandate.)
Randy.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Read-write mutex sometimes fails on deadlock
2017-11-15 22:57 ` Randy Brukardt
@ 2017-11-15 23:05 ` Randy Brukardt
0 siblings, 0 replies; 8+ messages in thread
From: Randy Brukardt @ 2017-11-15 23:05 UTC (permalink / raw)
I wrote:
...
> If the new aspect Nonblocking is used, the delay is statically illegal.
> (That *really* should have been the default, but it took us 20 years to
> figure out the right set of rules -- too late to mandate.)
"New" here means Ada 2020; it was just approved at our most recent ARG
meeting and isn't yet in the draft RM. So it might not be in your favorite
compiler for a while.
The aspect is a big win for most things, as it makes turns the "potentially
nonblocking" rules into static checks, so the portability issues [can you
use Text_IO in a protected object or not?] and "tripping hazards"
(Program_Error being raised unexpectedly) are eliminated.
To show how this works, let's look at the Eachus example using the aspect
Nonblocking:
protected type Mutex
with Nonblocking is
entry Write(D: in Data):
function Read return Data;
private
The_Data: Data := No_Data;
end Mutex;
protected body Mutex is
entry Write (D: in Data) is
begin
The_Data := D;
Some_Data := True;
Print(D); -- Illegal unless Print is declared with Nonblocking = True.
delay 0.1; -- Illegal.
end Write;
function Read return Data is
return The_Data;
end Read;
end Mutex;
So this code only will compile if it is guarenteed to work on all
implementations. (The original code would have raised Program_Error on
Janus/Ada for the delay, Print may or may not have worked.)
Randy.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Read-write mutex sometimes fails on deadlock
2017-11-06 18:24 ` Robert Eachus
2017-11-06 18:31 ` Simon Wright
@ 2017-11-06 21:01 ` Dmitry A. Kazakov
1 sibling, 0 replies; 8+ messages in thread
From: Dmitry A. Kazakov @ 2017-11-06 21:01 UTC (permalink / raw)
On 2017-11-06 19:24, Robert Eachus wrote:
> On Saturday, October 28, 2017 at 4:02:33 PM UTC-4, pascal....@gmail.com wrote:
>>
>> I have a package that implements several kinds of mutexes, including a
>> read-write mutex. Here is the full source (spec and body) with only the
>> read-write part.
>
> You got started down the wrong road and never turned around. The
> right abstraction for this in Ada is a protected object. A typical
> mutex looks like:
>
> protected type Mutex is
> entry Write(D: in Data):
> function Read return Data;
> private
> The_Data: Data := No_Data;
> end Mutex;
This is not mutex. Mutex is a semaphore with k=1.
> protected body Mutex is
> entry Write (D: in Data) is
> begin
> The_Data := D;
> Some_Data := True;
> Print(D);
This is illegal, because Print is potentially blocking.
With a proper mutex printing is done outside any protected action. Here
is a proper (not re-entrant, not read-write) mutex is:
protected type Mutex is
entry Seize;
procedure Release;
private
Owned : Boolean := False;
end Mutex;
protected body Mutex is
entry Seize when not Owned is
begin
Owned := True;
end Seize;
procedure Release is
begin
if Owned then
Owned := False;
else
raise Use_Error with "Releasing not owned mutex";
end if;
end Release;
end Mutex;
Printing goes as follows (an elaborated example):
Lock : Mutex;
...
select
Lock.Seize;
or delay Default_Timeout;
raise Time_Error with "Printer mutex deadlocked";
end select;
begin
Print (Something);
Lock.Release;
exception
when others =>
Lock.Release;
raise;
end;
P.S. It is advisable to have a controlled holder object with
initialization seizing and finalization releasing mutex. Then printing
goes Like:
declare
Exclusion : Holder (Lock'Access); -- Seize Lock
begin
Print (Something);
end; -- Release Lock
P.P.S. Re-entrant mutex is one the same task may seize consequently.
P.P.P.S. Read-write mutex is one with two levels of access. Read access
is allowed to several tasks. Write access is to only one. Read-write
mutexes have issues with pro-/demotion operations, that is when a task
owning the mutext changes the access level without releasing it.
--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2017-11-15 23:05 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-10-28 20:02 Read-write mutex sometimes fails on deadlock pascal.malaise
2017-11-06 18:24 ` Robert Eachus
2017-11-06 18:31 ` Simon Wright
2017-11-12 4:33 ` Robert Eachus
2017-11-12 5:21 ` J-P. Rosen
2017-11-15 22:57 ` Randy Brukardt
2017-11-15 23:05 ` Randy Brukardt
2017-11-06 21:01 ` Dmitry A. Kazakov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox