Stream venting (long)

Stream venting (long)

[dennison]

I have this situation where I need a general-purpose method for a task to save
off whatever data it chooses in real-time and possibly read it in at real-time
later. This seems like a natural for streams, right?

Well, I've run into a few complications that make streams not so helpful.

  1. Stream 'Writes and 'Reads are inherently task unsafe.

Possible solution 1a: You can of course try to implement your stream in a
task safe manner. But as a stream implementor all you will see is a series of
Write calls. A single 'Write from the client will cause an indeterminate
number of these. There's no way to tell if a Write or Read call is the first
or last in a series. If you need to make the enitre composite type's 'Write
or 'Read atomic, you're SOL.

1b: You can pick task priorites for the reader and writer tasks and a task
scheduling policy that make such operations atomic. EG: If you are using
strict priority scheduling, giving every involved task a differenty priority
will do the job. However, if your scheduling policy ever has to change, (eg:
code port, multiple processor system, etc.) the code will break in a very
hard to detect and hard to fix way.

1c: Have the client explicitly lock and unlock the stream before using it.
Doesn't have the above drawbacks, but makes what should be a simple one liner
for the client into a threee step process.

  2. Streams don't support 'Read when all the data may not have arrived yet.

Possible Solution 2a: Have the clients wait until the rest of the data comes
in. This is unacceptable for real-time systems, as it causes really nasty
prioity inversion. You just can't have a 60Hz task waiting for data to get
read in from the hard disk.

2b: Have the clients handle the exeption when the end of the stream is
reached, and try again later. This won't work because in a composite 'Read,
some of the reads may have completed already. Thus the client's target data
object is now corrupt (and they don't even know where!)

2c: Provide a hook for clients to check the amount of data in the stream
before doing their 'Read. This looks promising, but unfortunately there's no
way for the client to know how much data they need ahead of time. There's
nothing stopping a client from making their own 'Write/Reads that transmit a
*variable* amount of data, based on the value of internal fields. In fact, I
suspect the default 'Write for variant records works in just that way.

2d: Have the clients attempt their 'Reads into temp variables, and copy the
data over into the working location only if no exception occurrs. The problem
with this is that some fields in that record of theirs may be "limited
private" types. 'Read/Write have ways to handle those fields safely, but an
assignment will raise Program_Error.

2e: Provide the client with a hook to write their own data checking routine
which will verify if the stream is safe for them to read. The problem is that
if they just want to use the default 'Read and 'Write for anything, they
would have no way to know how to write the checking routine (other than try
2d above).

--
T.E.D.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[robert_dewar]

In article <768sng$6r9$1@nnrp1.dejanews.com>,
  dennison@telepath.com wrote:
>   1. Stream 'Writes and 'Reads are inherently task
>      unsafe. Possible solution 1a: You can of course try
>      to implement your stream in a
>      task safe manner. But as a stream implementor all
>      you will see is a series of
>      Write calls. A single 'Write from the client will
>      cause an indeterminate number of these. There's no
>      way to tell if a Write or Read call is the first
>      or last in a series. If you need to make the enitre
>      composite type's 'Write or 'Read atomic, you're SOL.

I find this complaint bizarre, I must not understand it
properly.

First, the idea of different tasks writing asynchronously
to a common stream seems an entirely bizarre program
organization.

Second, if you really *do* want such an organization, it
seems quite easy to program stream functions so that they
are properly atomic. I see no difficulty in doing this.
You just need a single lock, with a test that avoids
messing with it if it is already set.


-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[dennison]

In article <769g3r$moq$1@nnrp1.dejanews.com>,
  robert_dewar@my-dejanews.com wrote:
> In article <768sng$6r9$1@nnrp1.dejanews.com>,
>   dennison@telepath.com wrote:
> >   1. Stream 'Writes and 'Reads are inherently task
> >      unsafe. Possible solution 1a: You can of course try
> >      to implement your stream in a
> >      task safe manner. But as a stream implementor all
> >      you will see is a series of
> >      Write calls. A single 'Write from the client will
> >      cause an indeterminate number of these. There's no
> >      way to tell if a Write or Read call is the first
> >      or last in a series. If you need to make the enitre
> >      composite type's 'Write or 'Read atomic, you're SOL.
>
> I find this complaint bizarre, I must not understand it
> properly.
>
> First, the idea of different tasks writing asynchronously
> to a common stream seems an entirely bizarre program
> organization.

The bizarre program orginization you refer to is commonly called "logging".
It is also a good way to implement checkpointing or persistent objects,
unless you really *want* one file per task. Hmmm...Now I guess in my
particular situation I could use one file per task...perhaps that's the
solution.

> Second, if you really *do* want such an organization, it
> seems quite easy to program stream functions so that they
> are properly atomic. I see no difficulty in doing this.
> You just need a single lock, with a test that avoids
> messing with it if it is already set.

The problem is when to release the lock. You can't acquire and release it
after every call to the stream's Read or Write procedures, or you get
interleaved data. You'd have to do it *manually* before the client's first
top level 'Write call and after their last. That means the client has to do
it. Yuk.

--
T.E.D.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[robert_dewar]

In article <76aor7$l4q$1@nnrp1.dejanews.com>,
  dennison@telepath.com wrote:
> The problem is when to release the lock. You can't
> acquire and release it after every call to the stream's
> Read or Write procedures, or you get interleaved data.
> You'd have to do it *manually* before the client's first
> top level 'Write call and after their last. That means
> the client has to do
> it. Yuk.

I already gave the (perfectly trivial) solution for this
problem. Associate a lock with the stream. When a stream
routine is called, claim the lock IF IT IS NOT ALREADY
CLAIMED by you! On completion, release the lock IF YOU
CLAIMED it on entry. This should be perfectly simple to
program, and completely solves your problem.

Less Yuk, more thinking :-)

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[dennison]

In article <76baib$4ne$1@nnrp1.dejanews.com>,
  robert_dewar@my-dejanews.com wrote:
> In article <76aor7$l4q$1@nnrp1.dejanews.com>,
>   dennison@telepath.com wrote:
> > The problem is when to release the lock. You can't
> > acquire and release it after every call to the stream's
> > Read or Write procedures, or you get interleaved data.
> > You'd have to do it *manually* before the client's first
> > top level 'Write call and after their last. That means
> > the client has to do
> > it. Yuk.
>
> I already gave the (perfectly trivial) solution for this
> problem. Associate a lock with the stream. When a stream
> routine is called, claim the lock IF IT IS NOT ALREADY
> CLAIMED by you! On completion, release the lock IF YOU
> CLAIMED it on entry. This should be perfectly simple to
> program, and completely solves your problem.

Hmm. It appears you think my problem is with algorithmicly handling the
clients locking when locking is happening interally already. That's not the
case; my problem is making the clients do their own pre- and postprocessing.
That's a crappy interface. My code should be handling messy details like that
for them (not so incidently, removing one possible source for errors). They
should just be able to do a 'Write on their data structure and be done with
it.

--
T.E.D.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[robert_dewar]

In article <76de7d$opb$1@nnrp1.dejanews.com>,
  dennison@telepath.com wrote:

> Hmm. It appears you think my problem is with
> algorithmicly handling the clients locking when locking
> is happening interally already. That's not the case; my
> problem is making the clients do their own pre- and
> postprocessing. That's a crappy interface. My code should
> be handling messy details like that for them (not so
> incidently, removing one possible source for errors).
> They should just be able to do a 'Write on their data
> structure and be done with it.

I agree, a crappy interface, why on earth would you do it
that way. The details of the locking should be buried in
the stream attribute routines that you write. If you want
the locking mechanism to be general, you could even make
an extended stream type that had the necessary lock field,
but most likely it would be perfectly convenient to just
use a single global lock.

I really don't see a significant problem here, it seems a
straightforward programming excercise, but most certainly
you don't want the client calling the stream routine to
have to do the locking, and of course (reread my messages)
I never suggested such a thing -- that was your suggestion!

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[dennison]

In article <76dh6m$r63$1@nnrp1.dejanews.com>,
  robert_dewar@my-dejanews.com wrote:
> In article <76de7d$opb$1@nnrp1.dejanews.com>,
> that way. The details of the locking should be buried in
> the stream attribute routines that you write. If you want
> the locking mechanism to be general, you could even make
> an extended stream type that had the necessary lock field,
> but most likely it would be perfectly convenient to just
> use a single global lock.
>

Ahh. I see we have different ideas about what "client" means. To me the
"client" of my stream package *is* the writer of the stream attribute
routines. I have no clue how to write those routines for them, as I have no
prior knowledge of what they want to write (or access into their packages).
In fact, they may want to use the default routines in many cases. So we're
stuck with the crappy interface.

--
T.E.D.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[robert_dewar]

In article <76doir$1ek$1@nnrp1.dejanews.com>,
  dennison@telepath.com wrote:
> Ahh. I see we have different ideas about what "client"
> means. To me the "client" of my stream package *is* the
> writer of the stream attribute routines. I have no clue
> how to write those routines for them, as I have no
> prior knowledge of what they want to write (or access
> into their packages). In fact, they may want to use the
> default routines in many cases. So we're stuck with the
> crappy interface.


This is getting more and more peculiar. Of *course* the
implementation of the stream routine itself cannot provide
atomicity. This is fundamental, since as you say, it cannot
know what is being written.

The only point in the system where you know what level of
atomicity is required is at the point where you write stuff
to the streams. This requires locks. There is no magic to
get rid of the need to lock here. If the built in stream
attributes were task safe (that would incidentally be a
very poor choice), then of course the locking would have
to be in the default attribute routines.

Of *course* the locking belongs in the attribute routines,
it could not be anywhere else. As for crappy interface, it
is perfectly trivial to make an interface that simply
allows:

   Stream_Lock;
   ...
   Stream_Unlock;

to be placed in each attribute routine for which atomicity
is required (these primitives would properly take care of
nesting).

(a) what is so crappy about this interface?
(b) how could it possibly be avoided

If you want to use tasking, you have to take responsibility
for allocating resources properly, there is no magic that
can do this for you.

Yes, the language could have been designed so that the
default attribute routines had this kind of locking, but
that would have been an extremely bad idea for two reasons:

a) user written stream attribute routines would have to
lock manually, and might easily not do so. But caller's
would expect automatic locking.

b) this locking would introduce nasty overhead in the
normal case at a point where efficiency is very important.

After all, we do not even bother to lock Text_IO
automatically, so two tasks cannot do Put_Line's without
worrying about locking. It would be bizarre to decide that
streams should behave differently.

Yes, the logging case is a legitimate one, but frankly I
would do things differently, I would create an agent,
either a task or a protected type, that was charged with
the responsibility of doing all necessary logging, that is
I would make a higher level interface to the logging
routines. If, as seems the case from your somewhat
confusing description, you insist on having individual
tasks execute stream attributes directly, then the locks
belong in the stream attributes, no other place for them!


-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[dewar]

In article <368B6D96.D728F0B7@erols.com>,
  LeakyStain <leakstan@erols.com> wrote:
> You could implement a lock using a Controlled type, where
> Initialize locks and Finalize unlocks, giving this usage:
>
>     declare
> 	Stream_Locker;
>     begin
> 	stream use
>     end;
>
> That way, you don't have to worry about unlocking on
> exceptions. I suspect Ted Dennison still won't be happy.


I am not sure what would make Ted happy here, he seems to
expect some kind of impossible magic :-)

Note incidentally that internally, GNAT provides a cleanup
handler possibility, the pseudo-syntax (e.g. that provided
by -gnatG or -gnatdg in older versions) is

   begin
      ...
   at end
      ...
   end;

and the stuff in the "at end" handler is guaranteed to be
done on any exit from the block including an exceptional
exit. We use this all the time for internal expansions,
and it is tempting to make it available as a language
extension (under control of the -gnatX switch of course :-)

Robert Dewar
Ada Core Technologies

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[dennison]

In article <76g9ql$uvs$1@nnrp1.dejanews.com>,
  dewar@gnat.com wrote:
> In article <368B6D96.D728F0B7@erols.com>,
>   LeakyStain <leakstan@erols.com> wrote:
> > That way, you don't have to worry about unlocking on
> > exceptions. I suspect Ted Dennison still won't be happy.
>
> I am not sure what would make Ted happy here, he seems to
> expect some kind of impossible magic :-)
>

Magic would be nice. I tried poking voodoo pins in a collaboration diagram,
but it didn't help. :-) But as Fredrick Brooks pointed out, *we* are the
magicians.

As the title says, I was mostly venting. I had some hope there was some
"magical" solution I was missing. Failing that, at least some acceptable
solution. It seems I have to settle for acceptable. For the new year, I'll
resolve to try to be happy about it. :-)

--
T.E.D.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    

Re: Stream venting (long)

[LeakyStain]

robert_dewar@my-dejanews.com wrote:
> 
> Of *course* the locking belongs in the attribute routines,
> it could not be anywhere else. As for crappy interface, it
> is perfectly trivial to make an interface that simply
> allows:
> 
>    Stream_Lock;
>    ...
>    Stream_Unlock;
> 
> to be placed in each attribute routine for which atomicity
> is required (these primitives would properly take care of
> nesting).
> 
> (a) what is so crappy about this interface?

You could implement a lock using a Controlled type, where Initialize
locks and Finalize unlocks, giving this usage:

    declare
	Stream_Locker;
    begin
	stream use
    end;

That way, you don't have to worry about unlocking on exceptions. I
suspect Ted Dennison still won't be happy.

> (b) how could it possibly be avoided

No way, as you say, this is where it needs to be.

-- Stephe