Re: For the AdaOS folks

["Warren W. Gay VE3WWG" <ve3wwg@NoSpam.cogeco.ca>]

Nick Roberts wrote:

> "Warren W. Gay VE3WWG" <ve3wwg@NoSPAM.cogeco.ca> wrote:
>>By "process migration", are you referring to "thread-migration"?
> 
> No, I'm talking about moving an entire process (its memory, threads, and all
> other resources held) from one workstation to another (within the network)
> at /any/ arbitrary point during its execution (the execution of its
> threads).

Ok, it wasn't clear from the context.

> I'm assuming that the threads within a process will be closely-coupled --
> they will share the same memory address space -- so as to support the
> typical monoprocessor and SMP configurations of contemporary PCs.
> 
> As such, Mach's concept of thread migration seems inappropriate. Mach was
> designed to accomodate experimentation with NUMA, but NUMA machines have not
> come into the mainstream marketplace yet, so this is one way in which the
> experimental nature of Mach's design is less than ideal, to me.

Mach's migrating threads do in fact cross into other
"tasks". Instead of a caller thread stopping when a message is
queued, and the receiving thread starting (from a blocked state),
a migrating thread continues from the caller, works
through the server code, and then returns back to the caller (in
the caller's "task"). This is a simplification of what happens,
because there are "activations" and stack games that must happen
to make this work - but it is a migrating thread.

>>>but since there is only a binary interface and a C interface (no
>>>published Ada interface), this doesn't turn out to be such an advantage.
>>>Similar comments apply to other kernels, including Mach.
>>
>>I looked at their API and it doesn't look too bad actually. They map a
>>number of things into MR (Message Registers), which would be real easy to
>>do in Ada, using the for Object'Address use ...;  A tiny bit of _Asm would
>>fix the rest. So I don't see that as much of a problem. Define a package
>>(or few) and the rest is a slam dunk.
> 
> They take the attitude that the C interface must remain the same, regardless
> of the machine. I think that's impractical. 

In what way?

> I've designed the Bachar
> interface so that it will use real machine registers, rather than pretend
> registers. 

But don't forget the MRs are immediately loaded into registers
after all of your "deposits" are made. I don't think you
can level valid efficiency arguments against L4. AFAICS, it
is the best performing microkernel available.

> It means the interface changes from machine to machine, but it
> also means that the interface is as efficient as possible for each machine.
> Note that the differences in Bachar's interface, from machine to machine,
> will vary only in certain details (such as call convention and parameter
> passing).

I think L4 has done a nice job of saving the programmer from the
hassle of potential processor differences. They've apparently
done a good job of it, because it performs quite well in the
benchmark tests compared to the others.

>>What it _does_ of course require is the need for formatted messages. By
>>this I mean that you cannot just send a stream of bytes from one task to
>>another, and include a Port somewhere in the middle. The kernel must know
>>when a port is being copied/moved, so that it can play its kernel magic
>>before the receiving task gets the message (and thus inherit the port
>>right). So from this point of view, I would agree with you. But otherwise,
>>I don't see it as complex at all.
> 
> Assume we are considering one distributed process making an RPC to another
> one. Every such call must be dynamically routed, since the target process
> could be on any workstation in the network. Not only must the router store a
> table of the location (which workstation) of every process, but there must
> also be a protocol that allows a router on another workstation to forward
> the call (because the process has just moved) and notify the originating
> router (so it updates its locations table). It is also necessary to deal
> with network partitions (a partial breakdown of the network), and it is
> necessary in AdaOS for such RPCs to be made in the context of transactions,
> so that if the call unrecoverably fails, the transaction can be aborted. I
> don't think all that network routing complexity should be inside a
> microkernel.

Agreed. What you talk about is at least one abstraction layer above
the one I was speaking of.

>>The thing is, depending upon design of course, you could easily use
>>thousands of send-rights, with little or no overhead. If you must
>>associate the message endpoint to a thread, barring lazy allocation
>>techniques, you impose all of the overheads that a thread must impose
>>(stacks, state etc.)  I just feel that the port paradigm is more flexible
>>for the OS designer (it costs less). It is also conceptually cleaner to
>>say you have two end-points (ports) of a message queue, than saying you
>>have two-threads when threads aren't part of the abstraction.
> 
> I have taken the approach of local IPC being supported by the microkernel
> (and lower levels of TCB software), and network IPC being supported by a
> higher layer of TCB software (Avarus).

Yes, I agree with that layered approach. I was just stating that the
Mach message port interface is extremely convenient at the microkernel
level, for building your higher level layers.

>>But if you look at the Mach paper above, you can do the very same thing
>>with ports. Even without thread-migration, you can interpose proxies. With
>>thread-migration enabled RPC, you achieve both and provide the OS designer
>>a nicer interface.
> 
> But if we did that, what would be the advantage?

Its cheaper. To have 1000 send rights, costs near nothing. 1000 send
rights in one task costs nothing (just increments ref count). 1000 in
a 1000 different tasks, costs one pointer in each task. Nothing further.

1000 threads OTOH, costs you state information for each of those
1000 threads (same task or different). The amount of state
information needed will vary with platform.

>>>most microkernels do not seem to support this functionality. Bachar
>>>provides functionality that allows any process to be frozen, dissected,
>>>reconstructed (on the target machine), and then reanimated. These
>>>functions can be done by a higher layer, but it makes more sense for
>>>them to be supported directly by the kernel.
>>
>>I was planning to do this for my OS outside of the mk. The primary reason
>>for this is perhaps two-fold:
>>
>>  1. I want to choose my own form of networking for the purpose
>>  2. It may require some other OS "coordination"
>>
>>So I still favour a minimalist microkernel, but allowing for those nicer
>>abstractions above it.
> 
> I don't see the point in making the microkernel /that/ minimalist, to be
> honest. Whatever the upper layer did, it would have to be dependent upon
> intimate details of the kernel. So why not just lump them together?

But in your answer:

 > I have taken the approach of local IPC being supported by the microkernel
 > (and lower levels of TCB software), and network IPC being supported by a
 > higher layer of TCB software (Avarus).

If I understand correctly, you are saying that your network "layer"
is already built on top of lower levels (like microkernel).

>>To make this work, as I envision it, requires that all participating nodes
>>in a cluster see the file system the same way. IOW, there can only be one
>>root file system, which obviously is only local to one node. To all other
>>nodes, root will be access NFS-like (ie. over the network cable).
> 
> 
> This is a fundamental precept of fully distributed networking.

It would seem so, but I think there is room for creative
thinking here. Perhaps a "context sensitive view" is
also possible.

>>This consistent file view is necessary to make tasks mobile to any node in
>>the cluster.
> 
> Not only must the filesystem view by fully distributed (the same from every
> workstation in the network), but also the temporary memory view must be.

Good point.

> Since AdaOS allows a process to access multiple memory spaces (called
> 'regions'), distributed processes must have a distributed way of accessing
> all of its regions. This is provided a program calssed as a 'distributed
> storage manager' (DSM); there will be a DSM program called Cortex, which
> automatically shuffles pages around the network on demand.

This is nothing new of course. An example program was given the
Programming under Mach in 1992, and I am sure others did this
well before that book came out.

>>>Exactly. The main problem with the Registry is that it is not a proper
>>>database system.
>>
>>Well, define "proper" and define what the "database system" solves that
>>the present system doesn't?
> 
> The Windows Registry doesn't provide indexing, joining, field selection, or

   - Indexes are a performance issue (not an API issue, per se)
   - Joining (hard pressed to think of any registry entry I wanted
     to do a join for - I can't think of any)
   - Field selection (already part of the registry API)

> filtering (record selection).

   - I've never needed this. My programs usually just ask questions
     like "where is object such-n-such".

> These would all be useful capabilities. It
> also doesn't provide the kind of administrative functionality (e.g. backup)
> that a full database system usually does.

File systems are easily backed up, and so are individual files. This
isn't a difficult problem to solve, no matter what the form of the
registry.

>>SQL databases are designed to deal with large volumes of similar data, in
>>SELECTs, joins etc. However, when you look at what goes into the registry,
>>a lot of it is custom, hierarchically structured information.
> 
> No, the kind of data that goes into the Registry would be naturally
> organised in many complex ways. For example, many different applications
> store their own font information, but a font disinstaller may well wish to
> be able to 'cut across' this font data, to make appropriate adjustments (for
> all the applications). As another example, some applications may wish to
> store data for several different users, but an adminstrative program may
> wish to change all the data (across many applications) for a particular
> user.

The same argument can be made for changing file systems into databases.
So far, the filesystems approach seems to be the clear winner for
that kind of data.
--
Warren W. Gay VE3WWG
http://home.cogeco.ca/~ve3wwg