load and use a ".o" file?

load and use a ".o" file?

[lifetime n00b]

I've been trying to educate myself on how to do this, but I can't seem 
to find the answers I need. In concept, what I need to be able to do 
seems like it should be simple, and so I'll give a simple example.

There is a main program loop already running, and there is a ".o" 
compiled object file (compiled with gnat) which *hasn't* been linked 
into the main program, but it has a procedure or function in it I want 
to call. Let's say it is very simple function which was compiled from this:

function HiThere return String is
begin
	return "Hi there!";
end HiThere;

Instead of linking this into an executable, it is just compiled with 
say, "gnatmake -c hithere", to get the "hithere.o" file we're interested in.

Now, how can I load this "hithere.o" file into an already running 
program (without linking it in beforehand) and then call the "HiThere" 
function from that program (which was also written in Ada)? Would the 
information in the corresponding .ali file be of use? Maybe I'm thinking 
about it wrong, and the source which compiles to the object file needs 
to have some access types or some other mechanisms in place? Note that 
I'm not talking about using a .dll or .so file, just the .o file.

Any guidance would be appreciated.

Re: load and use a ".o" file?

[Stephen Leake]

lifetime n00b <anonymous@yahoo.com> writes:

> There is a main program loop already running, and there is a ".o"
> compiled object file (compiled with gnat) which *hasn't* been linked
> into the main program, but it has a procedure or function in it I want
> to call. 

You need to compile and link it into a "dynamically linked library"
(dll on windows, so (shared object) on Unix). Then you need to import
your OS's dynamic linking API (which may be done somewhere already),
and you're all set.

To get more specific, you need to say what OS you are on. And someone
else will have to reply; I've not actually done this as yet.

-- 
-- Stephe

Re: load and use a ".o" file?

[tmoran]

>Let's say it is very simple function which was compiled from this:
>
>function HiThere return String is
>begin
>        return "Hi there!";
>end HiThere;
>
>Instead of linking this into an executable, it is just compiled with
>say, "gnatmake -c hithere", to get the "hithere.o" file we're interested in.
>
>Now, how can I load this "hithere.o" file into an already running
>program (without linking it in beforehand) and then call the "HiThere"
>...
>I'm not talking about using a .dll or .so file, just the .o file.

  If you are not talking about using an existing dynamic linker/loader,
then you are clearly talking about developing your own.  In your example,
HiThere may need to make calls (for memory allocation for the string, for
instance) on the Ada Run Time System of the main program.  So you will
need to load the code from the .o file and then modify call addresses, ie,
link, the loaded HiThere function.  You should be able to find the
information you need by looking at the Gnat source code.
  What problem are you trying to solve?  Perhaps there's an easier way.

Re: load and use a ".o" file?

[lifetime n00b]

tmoran@acm.org wrote:
>   If you are not talking about using an existing dynamic linker/loader,
> then you are clearly talking about developing your own.  In your example,
> HiThere may need to make calls (for memory allocation for the string, for
> instance) on the Ada Run Time System of the main program.  So you will
> need to load the code from the .o file and then modify call addresses, ie,
> link, the loaded HiThere function.  You should be able to find the
> information you need by looking at the Gnat source code.
>   What problem are you trying to solve?  Perhaps there's an easier way.


Well, the "Big Picture" is that I'm wanting to write a Common Lisp 
system in Ada (yes I know, it probably sounds like heresy to both the 
Lisp and Ada camps), and I want to use gnat as the back-end compiler. 
Contrary to popular opinion, Lisp is not simply an interpreted language. 
One feature of a complete Common Lisp system is that source files can be 
compiled and loaded while the top-level system is running. Some 
implementations (such as CLISP) use byte-code compilation, but most 
compile to native code. Some use their own built-in compilers, while 
others first translate the Lisp code to C and use gcc as the back-end 
compiler and then load and use that compiled file. From what I can tell, 
it seems that GCL and ECL use gcc to compile the files as dynamic load 
(.so) files and then call dlopen() on that file.

I was thinking that it would be more efficient (and more portable) to 
not use .dll or .so files, but to load and use the .o object files 
directly in some way. And if I could figure out how to do it with the 
example I described in the OP, then I could just expand on that to 
accomplish what I'm ultimately aiming towards.

Re: load and use a ".o" file?

[Stephen Leake]

lifetime n00b <anonymous@yahoo.com> writes:

> I was thinking that it would be more efficient (and more portable) to
> not use .dll or .so files, but to load and use the .o object files
> directly in some way. 

.dll and .so files are designed to do _precisely_ what you are trying
to do; they are the most efficient solution!

Hmm. Well, they were designed in the context of a lot of heritage. If
you want to design an object file format and dynamic linker from
scratch, you might be able to do better. But there's no point; .dll
and .so are the only reasonable choices at this point.

-- 
-- Stephe

Re: load and use a ".o" file?

[Simon Wright]

lifetime n00b <anonymous@yahoo.com> writes:

> I was thinking that it would be more efficient (and more portable)
> to not use .dll or .so files, but to load and use the .o object
> files directly in some way. And if I could figure out how to do it
> with the example I described in the OP, then I could just expand on
> that to accomplish what I'm ultimately aiming towards.

I realise it is most unlikely that your target OS is VxWorks, but
there may be some help to be gleaned ..

The way you build Ada for VxWorks (well, with GNAT) is

* you do the bind as normal, this generates (among other things) the
  elaboration code in adainit and a call to it followed by a call to
  your program entry point

* the link is done (like any other VxWorks link) as a partial link (I
  think that's the right word, the basic command is ld -r), so that no
  VxWorks runtime is included but the GNAT runtime is

* the result is downloaded to the target and relocations resolved
  there, against the symbol table (I am a little hazy about exact
  details here, but it is certainly possible to do this download from
  eg flash memory, so the symbols must be resident).

For C under VxWorks there is no elaboration so you can just download a
.o.

For Ada it would be more tricky. You really do need that elaboration
code to get called. And you have to be sure that the runtime is
already present. And it may need elaboration too. And it may be wrong
to re-elaborate parts of the runtime that have been elaborated by the
already-running program.

All this sounds like a very interesting project!

One thought -- I'm not sure what the difference is between a .o and a
.so, they are both ELF files (on Linux anyway), I wonder if dlopen etc
would work with .o's?

-S

Re: load and use a ".o" file?

[lifetime n00b]

Stephen Leake wrote:
> .dll and .so files are designed to do _precisely_ what you are trying
> to do; they are the most efficient solution!

Well, not _precisely_, since a typical program using the .dll or .so 
files needs to know about those files when it is compiled itself, right? 
Maybe I wasn't clear on exactly what I need to do. The program that 
needs to load and use the object file is *already running* before the 
object file is compiled. This "main program" has no prior information 
about the data or routines which may be in the object file it is being 
told to load.

The "main program" operates much like a command prompt, and when 
(compile-file "hithere") is typed at the prompt, gnat will be called to 
compile an object file. Then when (load "hithere") is typed at the 
prompt, the object file is loaded and the data and/or functions in 
"hithere" are now available to the main program. Some additional 
information may have to be included in a supporting file produced at the 
same time the object file is produced, and this supporting file would be 
loaded when the object file is loaded.

That's the idea anyway. But I'm having a difficult time trying to figure 
out how to accomplish this seemingly straightforward task.

Re: load and use a ".o" file?

[tmoran]

>The "main program" operates much like a command prompt, and when
>(compile-file "hithere") is typed at the prompt, gnat will be called to
>compile an object file. Then when (load "hithere") is typed at the
>prompt, the object file is loaded and the data and/or functions in
>"hithere" are now available to the main program. Some additional
  How about making the freshly generated "hithere" file a server program
with remotely accessible procedures etc and use Annex E.

Re: load and use a ".o" file?

[lifetime n00b]

tmoran@acm.org wrote:
>   How about making the freshly generated "hithere" file a server program
> with remotely accessible procedures etc and use Annex E.

That would be horribly inefficient. Conceptually, the contents of these 
object files become part of the "main program" when they are loaded and 
share the same address space with the main program. For example, a call 
to a function normally involves putting the arguments on the stack, 
executing the function, and reading the return value from a register. In 
the distributed case, the partition containing the function must be 
located and placed on the stream, the arguments to the function are 
placed on the stream, the stream is sent to the receiving partition 
which executes the function locally, and the result of the function call 
is placed back on the stream which is then read as the return value.

Re: load and use a ".o" file?

[tmoran]

> > with remotely accessible procedures etc and use Annex E.
>
> That would be horribly inefficient.
  The question is, would it take too long.  If you are going to make
billions of these calls and subroutine call efficiency is a real problem,
perhaps you should write out an entire new program, including the
freshly generated code, and compile that with full global optimization,
subroutine inlining, etc.

Re: load and use a ".o" file?

[Simon Wright]

lifetime n00b <anonymous@yahoo.com> writes:

> Well, not _precisely_, since a typical program using the .dll or .so
> files needs to know about those files when it is compiled itself,
> right? Maybe I wasn't clear on exactly what I need to do. The
> program that needs to load and use the object file is *already
> running* before the object file is compiled. This "main program" has
> no prior information about the data or routines which may be in the
> object file it is being told to load.

man dlopen (assuming you're on a Unix box of course). What the
equivalent for win32 is I don't know, but I'm sure there is one.

-- 
Simon Wright                               100% Ada, no bugs.

Re: load and use a ".o" file?

[Stephen Leake]

lifetime n00b <anonymous@yahoo.com> writes:

> Stephen Leake wrote:
> > .dll and .so files are designed to do _precisely_ what you are trying
> > to do; they are the most efficient solution!
> 
> Well, not _precisely_, since a typical program using the .dll or .so
> files needs to know about those files when it is compiled itself,
> right? 

That's one way to use dynamically linked libraries. But if you look
under the hood, you will see that is just a variant of what you are
doing.

When a program that uses dlls starts up, it dynamically links the dll.
What you want is a program that dynamically links a dll at some later
time. In either case, the dynamic linking is the key.

> Maybe I wasn't clear on exactly what I need to do. The program that
> needs to load and use the object file is *already running* before
> the object file is compiled. This "main program" has no prior
> information about the data or routines which may be in the object
> file it is being told to load.

Well, it can't have absolutely _no_ information. It has to know the
list of subroutines (and their parameter profiles) that are in the
dll. Or, in COM style, it has to have a way of finding out what
subroutines are in the dll; which means it has to know the names of
the introspection subroutines in the dll.

When you dynamically link the compiled file into the main program, you
look up the addresses of those known routines.

> The "main program" operates much like a command prompt, and when
> (compile-file "hithere") is typed at the prompt, gnat will be called
> to compile an object file. Then when (load "hithere") is typed at
> the prompt, the object file is loaded and the data and/or functions
> in "hithere" are now available to the main program. 

Yes, that is what I thought you were doing. But how does the main
program know the names of the functions in "hithere"?

> Some additional information may have to be included in a supporting
> file produced at the same time the object file is produced, and this
> supporting file would be loaded when the object file is loaded.

Ah; perhaps that's the names of the functions? In Windows, that's a
.def file, which lists the names of the functions in a dll.

> That's the idea anyway. But I'm having a difficult time trying to
> figure out how to accomplish this seemingly straightforward task.

Go read the manuals on dlls, and dynamic linking in general, and on
COM. You are re-inventing the wheel! It may seem straightforward, but
it gets quite complicated. Fortunately, it has been solved.

Most good ideas have been thought of before :).

-- 
-- Stephe

Re: load and use a ".o" file?

[Mark H Johnson]

lifetime n00b wrote:
> Stephen Leake wrote:
> 
>> .dll and .so files are designed to do _precisely_ what you are trying
>> to do; they are the most efficient solution!
> 
> 
> Well, not _precisely_, since a typical program using the .dll or .so 
> files needs to know about those files when it is compiled itself, right? 
> Maybe I wasn't clear on exactly what I need to do. The program that 
> needs to load and use the object file is *already running* before the 
> object file is compiled. This "main program" has no prior information 
> about the data or routines which may be in the object file it is being 
> told to load.
> 
What you are describing sounds a lot like the mechanism used on Multics 
for dynamic linking of object files. There is a better explanation in 
the references at
   http://www.multicians.org/
or by reading
   "The Multics System: An Examination of its Structure" by E.I. Organick
but a quick summary follows.

When a file is compiled on Multics, an object file is created. Part of 
the data in that object file is a list of public symbols, including 
entry points. For a subroutine named xyz, compiled from xyz.fortran, the 
  entry point would simply be xyz. Let's say you have a main program 
main, calling xyz. This is an external reference to xyz (since it is not 
found in the current segment), and the first call works something like this:
  - call xyz (dispatched to an activation function)
  - the activation function checks first the active segments for a 
symbol within them called xyz. If found, it fixes up the dispatch to go 
directly to that xyz and resumes execution.
  - if not in an active segment, it uses the search path (similar to 
PATH on Unix) to find a file named xyz. If found, it loads that file 
into a segment, makes it active, and does the work of the second step.
  - if no such file is found, it calls the command line interpreter with 
an error status. At this point, you can fix your search path, create the 
object file xyz, start the debugger, or abandon execution. If you either 
of the first two, you can resume and the attempt to load xyz repeats.

The subsequent calls to xyz is done just like any other cross segment 
function call, basically running at (or near) full speed. It is this 
part that eliminates most of the concerns about performance since the 
expensive look up is only done once per symbol.

This capability, and I/O redirection are two of the things I really miss 
from Multics. The development flexibility provided were great to enhance 
the performance of developers and the overhead was quite modest with the 
hardware provided. The lack of effective paged / segmentation hardware 
is one thing that makes this hard to implement in software.

You can probably get close if you modify the build of objects to 
generate .so's instead of .o's and then put a wrapper around dlopen to 
do the semantics you are looking for. If you use GNAT, you could 
probably even bury this in the runtime by adding a pragma (or command 
line option) to denote external references as dynamic calls.

If you get this to work, I'd like to hear about it.
   --Mark

Re: load and use a ".o" file?

[lifetime n00b]

Stephen Leake wrote:
> You are re-inventing the wheel! 

No, I'm just trying to find the wheel I need! Honest. If I thought I'd 
have to reinvent the thing, I'd be the first to point out that I'm the 
last one qualified to do so, and then I'd just try to find a square I 
could use instead (in this case, maybe a byte-code VM?). :-)

> It may seem straightforward, but it gets quite complicated. 

So I have found.

 > Fortunately, it has been solved.

I think I may be able to use dlopen and dlsym on Linux with an Ada 
wrapper, but I was hoping for an Ada-only solution that would be 
cross-platform (letting gnat take all the nasty details from me). But 
that's just me hoping for a wheel that can fly...

> Most good ideas have been thought of before :).

And almost all of them were thought of by Kepler (just don't tell that 
to Newton)! :)

Re: load and use a ".o" file?

[lifetime n00b]

Mark H Johnson wrote:
>  - call xyz (dispatched to an activation function)
>  - the activation function checks first the active segments for a symbol 
> within them called xyz. If found, it fixes up the dispatch to go 
> directly to that xyz and resumes execution.
>  - if not in an active segment, it uses the search path (similar to PATH 
> on Unix) to find a file named xyz. If found, it loads that file into a 
> segment, makes it active, and does the work of the second step.
>  - if no such file is found, it calls the command line interpreter with 
> an error status. 

This actually sounds a lot like using a distributed system (Annex E) 
with both active and passive partitions on the same local machine, 
though probably more efficient.


> The subsequent calls to xyz is done just like any other cross segment 
> function call, basically running at (or near) full speed. It is this 
> part that eliminates most of the concerns about performance since the 
> expensive look up is only done once per symbol.

Unfortunately, I won't be programming this for Multics. I may need to 
suck it up and just do some benchmarks using the Annex E method. Passing 
data, arguments, and return values along streams between paritions just 
to call a function sure seems like a huge performance drain, but maybe 
in practice it's not as bad as I imagine.

> If you get this to work, I'd like to hear about it.

Me too! :)

Re: load and use a ".o" file?

[Jon S. Anthony]

Stephen Leake <stephen_leake@acm.org> writes:

> lifetime n00b <anonymous@yahoo.com> writes:
> 
...
> > Maybe I wasn't clear on exactly what I need to do. The program that
> > needs to load and use the object file is *already running* before
> > the object file is compiled. This "main program" has no prior
> > information about the data or routines which may be in the object
> > file it is being told to load.

This sort of thing is perfectly reasonable and something that is
extremely useful in a wide range of applications.

What I don't understand is why people don't just use technology that
has solved this problem since "for ever".  No, dll's and so's are not
the answer - they are intended to solve a different (related) problem
and are at best a poor mans hack for what you are talking about.

What you really "need"[1] here is a true dynamic language.


> Most good ideas have been thought of before :).

Indeed.  I see that Greenspun's 10th is alive and well on c.l.a :-|...


/Jon

Re: load and use a ".o" file?

[Mark H Johnson]

lifetime n00b wrote:

> Mark H Johnson wrote:
> 
>>  - call xyz (dispatched to an activation function)
>>  - the activation function checks first the active segments for a 
>> symbol within them called xyz. If found, it fixes up the dispatch to 
>> go directly to that xyz and resumes execution.
>>  - if not in an active segment, it uses the search path (similar to 
>> PATH on Unix) to find a file named xyz. If found, it loads that file 
>> into a segment, makes it active, and does the work of the second step.
>>  - if no such file is found, it calls the command line interpreter 
>> with an error status. 
> 
> 
> This actually sounds a lot like using a distributed system (Annex E) 
> with both active and passive partitions on the same local machine, 
> though probably more efficient.
> 
No, they were actually subroutine calls. The compiler generated slightly 
different code for an "in segment" call and a "cross segment" call due 
to the indirection. The glue code (to do the lookup) was basically a 
system service done by the OS to facilitate activation of new segments.

By the way, when I say segment - I mean a segment of the same address 
space, not a partition. Think to the 386 processor where you have 32 k 
segments in certain processing modes. On Multics, the segments were 256 
Kwords (1 Mbyte) in size which worked quite well in practice.
   --Mark

Re: load and use a ".o" file?

[Stephen Leake]

j-anthony@rcn.com (Jon S. Anthony) writes:

> This sort of thing is perfectly reasonable and something that is
> extremely useful in a wide range of applications.
> 
> What I don't understand is why people don't just use technology that
> has solved this problem since "for ever".  No, dll's and so's are not
> the answer - they are intended to solve a different (related) problem
> and are at best a poor mans hack for what you are talking about.
> 
> What you really "need"[1] here is a true dynamic language.

Ok, I'll bite. What's a "true dynamic language"? Do you mean
"interpreted from source", like some versions of Basic, and most shell
languages? Or "interpreted from some intermediate byte code", like
most Visual Basic, Lisp implementations, Java, C#, etc? Or something
else?

All interpreters make the job of adding random code at run-time much
easier. 

-- 
-- Stephe

Re: load and use a ".o" file?

[tmoran]

>All interpreters make the job of adding random code at run-time much
>easier.
  If I understand correctly, the problem is not in adding code, but in
finding code.  The compiler has put the code, in some form, in a .o file
and the OP wants to pluck it out so he can call it - perhaps with an
indirect jump, ie, a software simulation of Multics et al.

Re: load and use a ".o" file?

[Robert I. Eachus]

Mark H Johnson wrote:

> No, they were actually subroutine calls. The compiler generated slightly 
> different code for an "in segment" call and a "cross segment" call due 
> to the indirection. The glue code (to do the lookup) was basically a 
> system service done by the OS to facilitate activation of new segments.

Technically not correct.  The compiler could "in line" some local calls, 
but the binder couldn't.   What happened was that when a segment called 
an entry point for which the local link was null, it trapped.  The trap 
routine tried to snap the link using the current environment, and if it 
failed, bounced out to the user.  You literally could, when an error of 
this type occurred, change the search paths, or write the missing 
routine, compile it, then continue from the condition.  Done it myself....

> By the way, when I say segment - I mean a segment of the same address 
> space, not a partition. Think to the 386 processor where you have 32 k 
> segments in certain processing modes. On Multics, the segments were 256 
> Kwords (1 Mbyte) in size which worked quite well in practice.

Correct.  As I said in another post, there were multisegment files which 
allowed you to have objects larger than 1 Meg.  But in practice they 
were very infrequently used.

-- 
                                           Robert I. Eachus

"The war on terror is a different kind of war, waged capture by capture, 
cell by cell, and victory by victory. Our security is assured by our 
perseverance and by our sure belief in the success of liberty." -- 
George W. Bush

Re: load and use a ".o" file?

[Mark H Johnson]

Robert I. Eachus wrote:

> Mark H Johnson wrote:
> 
>> No, they were actually subroutine calls. The compiler generated 
>> slightly different code for an "in segment" call and a "cross segment" 
>> call due to the indirection. The glue code (to do the lookup) was 
>> basically a system service done by the OS to facilitate activation of 
>> new segments.
> 
> 
> Technically not correct.  The compiler could "in line" some local calls, 
> but the binder couldn't. 
I did not mention the binder (which in many ways I equate to having a 
linker on Unix), just the compiler.

> What happened was that when a segment called 
> an entry point for which the local link was null, it trapped.  The trap 
> routine tried to snap the link using the current environment, and if it 
> failed, bounced out to the user.  You literally could, when an error of 
> this type occurred, change the search paths, or write the missing 
> routine, compile it, then continue from the condition.  Done it myself....
> 
Me to. That is why I described that mechanism in terms the OP would 
likely understand in a previous message.

   --Mark

Re: load and use a ".o" file?

[Robert I. Eachus]

Mark H Johnson wrote:

> Robert I. Eachus wrote:
> 
>> Technically not correct.  The compiler could "in line" some local 
>> calls, but the binder couldn't. 
> 
> I did not mention the binder (which in many ways I equate to having a 
> linker on Unix), just the compiler.

I was not trying to be confrontational or whatever.  My point was, that 
in segments created by the binder, all that was done was to provide a 
(static, code-relative)target for the call.  It didn't change the 
calling code at all.  So if you had a program compiled as a collection 
of separate segments, and ran it that way, the first time each entry 
point was used, the link was snapped, and from then on you had no 
additional run-time overhead.  If you used the binder, it eliminated 
some of the run-time link snapping, but the code was exactly as 
efficient as if it had been compiled from one huge source file.

> Me to. That is why I described that mechanism in terms the OP would 
> likely understand in a previous message.

I still miss Multics.  I had accounts on several Multics machines when I 
worked at Honeywell.  When I moved to Stratus, even though Stratus was a 
'spritual' descendent of Multics, I kept an account on MIT_Multics.  (I 
had some Multics PL/I tools I wanted to port to VOS.)  But it has been 
fifteen years now, and I still don't have a development environment as 
good as Multics.

However, Ada is now a much better language than PL/I. ;-)

-- 
                                           Robert I. Eachus

"The war on terror is a different kind of war, waged capture by capture, 
cell by cell, and victory by victory. Our security is assured by our 
perseverance and by our sure belief in the success of liberty." -- 
George W. Bush

Re: load and use a ".o" file?

[Jon S. Anthony]

Stephen Leake <stephen_leake@acm.org> writes:

> j-anthony@rcn.com (Jon S. Anthony) writes:
> 
> > This sort of thing is perfectly reasonable and something that is
> > extremely useful in a wide range of applications.
> > 
> > What I don't understand is why people don't just use technology that
> > has solved this problem since "for ever".  No, dll's and so's are not
> > the answer - they are intended to solve a different (related) problem
> > and are at best a poor mans hack for what you are talking about.
> > 
> > What you really "need"[1] here is a true dynamic language.
> 
> Ok, I'll bite. What's a "true dynamic language"? 

It is true that many folks will/do have different ideas about what
"true dynamic language" means.  In this respect I fear it is a bit
like "true object oriented language".

One characteristic that is neither necessary nor sufficient for any
notion of this is that it be an "interpreted language".  While some
languages (for example, most scripting languages) do imply (mostly
because the language specification _is_ the implementation)
interpretation as part of their definition, "interpreted" interpreted
properly is really an _implementation_ issue, _not_ a language issue.

I think most people in the "dynamic community" would say you need at
least the following characteristics to be "truly dynamic":

1. Dynamic typing.  This is, strong typing on _objects_ as
   differentiated from "variables/locations".

2. Complete introspection of the language's computation and type
   model.  This is directly and explicitly available at the level of
   the programmer (not just the implementer).  NOTE: This does _not_
   mean or imply access to the underlying _implementation_.

   Some things this _does_ imply:

   * New function objects may be added to a running system at any time
     and existing functions may be so modified at any time.

   * The ability to create new types and/or classes at runtime; modify
     existing classes (with complete instance migration).


A couple examples which fully have this would be Common Lisp and
Smalltalk.  Neither are "interpreter" based[1].  Most Common Lisp
implementations have optimizing native compilers that typically
produce code within about 1.5-2.5 of well crafted C code[2].  So,
"interpretation" of either source or byte code is not a relevant
factor/issue.

For more information, here are a couple of links:

"Open Source/Free Software" Implementations:

http://www.cliki.net/Common%20Lisp%20implementation

Commercial Implementations:

http://alu.cliki.net/Implementation

General community information:

http://www.cliki.net/index and, of course, c.l.l


Here are a couple very simple examples concerning the issue at hand of
the OP.  These are done in Allegro Common Lisp at the REPL (read eval
print loop), typically this sort of interaction would be through the
IDE.


CL-USER(1):
 (defun foo (l r)
   (declare (fixnum l r) (optimize (speed 3 safety 1 debug 1)))
   (+ l r))
=> FOO

CL-USER(2): (compile 'foo)
=> FOO

CL-USER(3): (disassemble 'foo) ; Just to show we are native (on Intel...)
;; disassembly of #<Function FOO>
;; formals: L R

;; code start: #x7140a65c:
   0: 83 f9 02    cmpl	ecx,$2
   3: 74 02       jz	7
   5: cd 61       int	$97             ; EXCL::TRAP-ARGERR
   7: 80 7f 97 00 cmpb	[edi-105],$0    ; SYS::C_INTERRUPT
  11: 74 02       jz	15
  13: cd 64       int	$100            ; EXCL::TRAP-SIGNAL-HIT
  15: 03 c2       addl	eax,edx
  17: f8          clc
  18: 8b 75 fc    movl	esi,[ebp-4]
  21: c3          ret




This example shows compiling and loading a set of resources from a
file under program control.

;;;--------------------junk.cl------------

(in-package :cl-user)

(defun foo (l r)
  (declare (fixnum l r) (optimize (speed 3 safety 1 debug 1)))
  (+ l r))


(defun bar (msg)
  (format nil "Here's a message: ~A" msg))

;;;-----------------end junk.cl------------



;;;---------------------load-from.cl-----------

(in-package :cl-user)

;;; Load the function definitions (and other forms) in the file
;;; identified by FILESPEC (a pathname or a string which is a legal
;;; filespec for the underlying OS filesystem).  Compile the file
;;; before loading it.
;;;
;;; WHEN-UNDEFINED indicates a level of conditionality for compiling
;;; and loading the file.  If not given, or explicitly set to NIL,
;;; unconditionally compile and load the file.
;;;
;;; If given, it must be a list of function names, which are tested to
;;; see if they are currently defined.  If _none_ of them are, then
;;; compile and load the file.
;;;
;;; NOTE: no current provision for checking to see if the definitions
;;; in the file are of a later time stamp than the current existing
;;; function definitions (which could be used to conditionally compile
;;; and load when definitions are superceded).
;;;
;;; Errors and serious conditions are handled by sending a message on
;;; the condition to the standard error stream.
;;;
;;; Returns t if file is successfully compiled and loaded, :noaction
;;; if no action is taken (see above), :error if an error is signaled
;;; or :serious if a serious condition is signaled.
;;;
(defun load-from (filespec &key (when-undefined nil))
  (assert (listp when-undefined))
  (handler-case
   (if (or (eql when-undefined nil)
	   (reduce #'(lambda (x y)
		       (and x (not (fboundp y))))
		   when-undefined :initial-value t))
       (load (compile-file filespec))
     :noaction)
   
   ;; Exceptions
   ;;
   (error (condition)
     (format *error-output* "LOAD-FROM crashed with ~S" condition)
     :error)
   (serious-condition (condition)
     (format *error-output* "LOAD-FROM questionable: ~S" condition)
     :serious)))
;;;-----------------end load-from.cl-----------


;;; Load the previously compiled load-from object

CL-USER(1): (load "load-from")
; Fast loading load-from.fasl
T

CL-USER(2): (foo 1 2)
Error: attempt to call `FOO' which is an undefined function.
  [condition type: UNDEFINED-FUNCTION]

CL-USER(4): (fboundp 'foo)
NIL

foo is undefined...


CL-USER(5): (load-from "./junk.cl" :when-undefined '(load-from))
:NOACTION

Since load-from is already defined, this does nothing per the spec.


CL-USER(6): (load-from "./junk.cl" :when-undefined '(foo bar))
;;; Compiling file ./junk.cl
;;; Writing fasl file ./junk.fasl
;;; Fasl write complete
; Fast loading /opt/GoldenThread/ExampleApps/ECDemo/junk.fasl
T


CL-USER(7): (foo 1 2)
3
CL-USER(8): (bar "Hello world!")
"Here's a message: Hello world!"



> Do you mean "interpreted from source"

No, as this is an implementation issue.


> Or "interpreted from some intermediate byte code", like most Visual
> Basic, Lisp implementations, Java, C#, etc?

No, as this is an implementation issue.

BTW, _most_ Lisp implementations have _native compilers_.  Some (e.g.,
CMUCL) _only_ have native compilers - no interpreter at all.

> Or something else?

Yes, as described above.


> All interpreters make the job of adding random code at run-time much
> easier.

Actually, this only seems to be true; with fully compiled versions
_this aspect_ is just as easy.  The hard part, of course, is still the
bits that make up the difference between a compiler and interpreter.


/Jon


[1] The ANSI Common Lisp specification
    http://www.lispworks.com/reference/HyperSpec/Front/index.htm
    requires a compiler, though not necessarily a native code
    generator.

    I'm much more familiar with Common Lisp than Smalltalk, but
    VisualWorks, Squeak, and VisualAge all have JIT compilers.

    Python offers some of this as well, but the "definition/reference
    implementations" are interpreted.

[2] There are several types of basic block structures where CL can
    perform better than such C code.

Re: load and use a ".o" file?

[Stephen Leake]

j-anthony@rcn.com (Jon S. Anthony) writes:

> Stephen Leake <stephen_leake@acm.org> writes:
> 
> > j-anthony@rcn.com (Jon S. Anthony) writes:
> > 
> > > This sort of thing is perfectly reasonable and something that is
> > > extremely useful in a wide range of applications.
> > > 
> > > What I don't understand is why people don't just use technology that
> > > has solved this problem since "for ever".  No, dll's and so's are not
> > > the answer - they are intended to solve a different (related) problem
> > > and are at best a poor mans hack for what you are talking about.
> > > 
> > > What you really "need"[1] here is a true dynamic language.
> > 
> > Ok, I'll bite. What's a "true dynamic language"? 
> 
> It is true that many folks will/do have different ideas about what
> "true dynamic language" means.  In this respect I fear it is a bit
> like "true object oriented language".

That's why I asked :).

> I think most people in the "dynamic community" would say you need at
> least the following characteristics to be "truly dynamic":
> 
> 1. Dynamic typing.  This is, strong typing on _objects_ as
>    differentiated from "variables/locations".

Ok. I don't think that impacts on the issue of adding new code at runtime.

> 2. Complete introspection of the language's computation and type
> model. This is directly and explicitly available at the level of the
> programmer (not just the implementer). NOTE: This does _not_ mean or
> imply access to the underlying _implementation_.
> 
>    Some things this _does_ imply:
> 
>    * New function objects may be added to a running system at any time
>      and existing functions may be so modified at any time.

Hmm. The English meaning of "introspection" is "to examine oneself";
that's not the same as creating new code. But you get to use the term
the way you want to ...

>    * The ability to create new types and/or classes at runtime;
>    modify existing classes (with complete instance migration).
> 
> 
> A couple examples which fully have this would be Common Lisp and
> Smalltalk.  Neither are "interpreter" based[1].  Most Common Lisp
> implementations have optimizing native compilers that typically
> produce code within about 1.5-2.5 of well crafted C code[2].  

Hmm. Do those compiled versions include the ability to create new code
at runtime, where the new code is also compiled, not interpreted? I'd
be very surprised if that were true (I have not used a system where
that is true, but I have used only a couple Lisps, and no Smalltalks).
If it is true, what is the linking mechanism used?

> <snip> Here are a couple very simple examples concerning the issue
> at hand of the OP. These are done in Allegro Common Lisp at the REPL
> (read eval print loop), typically this sort of interaction would be
> through the IDE.
> 
> CL-USER(1):
>  (defun foo (l r)
>    (declare (fixnum l r) (optimize (speed 3 safety 1 debug 1)))
>    (+ l r))
> => FOO
> 
> CL-USER(2): (compile 'foo)
> => FOO
> 
> CL-USER(3): (disassemble 'foo) ; Just to show we are native (on Intel...)
> ;; disassembly of #<Function FOO>
> ;; formals: L R
> 
> ;; code start: #x7140a65c:
>    0: 83 f9 02    cmpl	ecx,$2
>    3: 74 02       jz	7
>    5: cd 61       int	$97             ; EXCL::TRAP-ARGERR
>    7: 80 7f 97 00 cmpb	[edi-105],$0    ; SYS::C_INTERRUPT
>   11: 74 02       jz	15
>   13: cd 64       int	$100            ; EXCL::TRAP-SIGNAL-HIT
>   15: 03 c2       addl	eax,edx
>   17: f8          clc
>   18: 8b 75 fc    movl	esi,[ebp-4]
>   21: c3          ret

Hmm. You don't actually demonstrate calling foo, but I'll grant you
that :).

Ok, I stand corrected. Interpretation is not necessary for dynamically
generated and linked code.

But I'd still like to understand what the linking process implied by
(compile 'foo). As far as I can see, it must do the equivalent of
Windows DLL or Unix so. Just because that mechanism is hidden from the
user (as it should be) doesn't mean it isn't there!

That was the OP's question; how to get dynamically generated code
linked in to a running program. Just saying "use CL" doesn't explain
the mechanism.

-- 
-- Stephe

Re: load and use a ".o" file?

[Georg Bauhaus]

Stephen Leake <stephen_leake@acm.org> wrote:
: Hmm. Do those compiled versions include the ability to create new code
: at runtime, where the new code is also compiled, not interpreted?

If I have understood Jon correctly, there are Lisp compilers without
an interpreter, and with the ability to load compiled code, so
I'd guess, yes. (I think I've seen the thing, too.)

For SNOBOL4, a minimal RE(P)L which runs the compiler at
run time is

read                        :<code(input ' :(read)')>
end

SPITBOL is a "true" SNOBOL4 compiler afaik, so it might offer
the same capability[*]. (If you are interested,
the :<...>  triggers a jump to the beginning of ..., which is compiled
code() from ``input ' :(read)'''; input reads what you
type, ' :(read)' is an unconditional jump to label 'read',
appended to every such input.)

 [*]From the SPITBOL description at www.snobol4.com:
"New program fragments can be compiled during
program execution...
For very complex problems, your program can "evovle" and
modify itself during execution in response to the data it is
processing. New patterns can be built during execution."

Re: load and use a ".o" file?

[Jon S. Anthony]

Stephen Leake <stephen_leake@acm.org> writes:

> (Jon S. Anthony) writes:
> 
> > Stephen Leake <stephen_leake@acm.org> writes:
> > 
> > >  (Jon S. Anthony) writes:
...

> > I think most people in the "dynamic community" would say you need at
> > least the following characteristics to be "truly dynamic":
> > 
> > 1. Dynamic typing.  This is, strong typing on _objects_ as
> >    differentiated from "variables/locations".
> 
> Ok. I don't think that impacts on the issue of adding new code at runtime.

Actually it does, but in non obvious ways.  For this discussion, it is
safe to ignore it.

> > 2. Complete introspection of the language's computation and type
> > model. This is directly and explicitly available at the level of the
> > programmer (not just the implementer). NOTE: This does _not_ mean or
> > imply access to the underlying _implementation_.
> > 
> >    Some things this _does_ imply:
> > 
> >    * New function objects may be added to a running system at any time
> >      and existing functions may be so modified at any time.
> 
> Hmm. The English meaning of "introspection" is "to examine oneself";
> that's not the same as creating new code. But you get to use the term
> the way you want to ...

Fair enough; while the explicit addition of "self modifying" would be
clearer for c.l.a, I believe that is taken for granted in the "dynamic
community".  It is not always easy to clearly change gears in usenet
discussions...

> >    * The ability to create new types and/or classes at runtime;
> >    modify existing classes (with complete instance migration).
> > 
> > 
> > A couple examples which fully have this would be Common Lisp and
> > Smalltalk.  Neither are "interpreter" based[1].  Most Common Lisp
> > implementations have optimizing native compilers that typically
> > produce code within about 1.5-2.5 of well crafted C code[2].  
> 
> Hmm. Do those compiled versions include the ability to create new code
> at runtime, where the new code is also compiled, not interpreted?

Yes.

> I'd be very surprised if that were true (I have not used a system
> where that is true, but I have used only a couple Lisps, and no
> Smalltalks).  If it is true, what is the linking mechanism used?

It is a pleasent surprise, I hope.  The "linking" mechanism is simply
resolution on a symbol's function slot.  This is part of the
introspective aspect of a Lisp system.  The native code is "just"
placed in a code vector.


> > CL-USER(1):
> >  (defun foo (l r)
> >    (declare (fixnum l r) (optimize (speed 3 safety 1 debug 1)))
> >    (+ l r))
> > => FOO
> > 
> > CL-USER(2): (compile 'foo)
> > => FOO
> > 
> > CL-USER(3): (disassemble 'foo) ; Just to show we are native (on Intel...)
> > ;; disassembly of #<Function FOO>
> > ;; formals: L R
> > 
> > ;; code start: #x7140a65c:
> >    0: 83 f9 02    cmpl	ecx,$2
> >    3: 74 02       jz	7
> >    5: cd 61       int	$97             ; EXCL::TRAP-ARGERR
> >    7: 80 7f 97 00 cmpb	[edi-105],$0    ; SYS::C_INTERRUPT
> >   11: 74 02       jz	15
> >   13: cd 64       int	$100            ; EXCL::TRAP-SIGNAL-HIT
> >   15: 03 c2       addl	eax,edx
> >   17: f8          clc
> >   18: 8b 75 fc    movl	esi,[ebp-4]
> >   21: c3          ret
> 
> Hmm. You don't actually demonstrate calling foo, but I'll grant you
> that :).

That was a little further below in the example which compiles and
loads foo from a file.


> But I'd still like to understand what the linking process implied by
> (compile 'foo). As far as I can see, it must do the equivalent of
> Windows DLL or Unix so.

No, this is part of the underlying introspective environment.  It is
actually much simpler than the contortions involved in DLL/so type
stuff.  It is also fine grained - a function at a time or even a
functions symbol at a time.

> Just because that mechanism is hidden from the user (as it should
> be) doesn't mean it isn't there!

Yes, it is there, and you can even get at the important user
(programmer) level bits yourself.  Here is a few interactions after
the compile and load of the junk.cl file:

CL-USER(18): (inspect 'bar)
The symbol BAR @ #x7140990f
  which is an INTERNAL symbol in the COMMON-LISP-USER package
   0 type ---------> Bit field: #x07
   1 flags --------> Bit field: #x00
   2 hash ---------> Bit field: #x61d7
   3 value --------> ..unbound..
   4 package ------> The COMMON-LISP-USER package
   5 function -----> #<Function BAR>
   6 name ---------> A simple-string (3) "BAR"
   7 plist --------> <...>, a proper list with 2 elements
[1i] CL-USER(19): (symbol-function 'bar)
#<Function BAR>
[1i] CL-USER(20): (symbol-function 'my-new-foo)
Error: Symbol MY-NEW-FOO does not have a function definition.
  [condition type: SIMPLE-ERROR]

Restart actions (select using :continue):
 0: Return to Debug Level 1 (an "abort" restart).
 1: Return to Top Level (an "abort" restart).
 2: Abort entirely from this process.
[2] CL-USER(21): :cont 0
[1i] CL-USER(22): :i q ; get out of the inspection of 'bar
BAR
CL-USER(23): (defun my-new-foo (x) x) ; == cl:identity function

MY-NEW-FOO
CL-USER(24): (symbol-function 'my-new-foo)
#<Interpreted Function MY-NEW-FOO>
CL-USER(25): (compile 'my-new-foo)
MY-NEW-FOO
NIL
NIL
CL-USER(26): (symbol-function 'my-new-foo)
#<Function MY-NEW-FOO>
CL-USER(27): (disassemble 'my-new-foo)
;; disassembly of #<Function MY-NEW-FOO>
;; formals: X

;; code start: #x714242c4:
   0: 83 f9 01    cmpl	ecx,$1
   3: 74 02       jz	7
   5: cd 61       int	$97             ; EXCL::TRAP-ARGERR
   7: 80 7f 97 00 cmpb	[edi-105],$0    ; SYS::C_INTERRUPT
  11: 74 02       jz	15
  13: cd 64       int	$100            ; EXCL::TRAP-SIGNAL-HIT
  15: f8          clc
  16: 8b 75 fc    movl	esi,[ebp-4]
  19: c3          ret
CL-USER(28): (my-new-foo (list 1 2 3))
(1 2 3)
CL-USER(29): 
CL-USER(33): (inspect 'my-new-foo)
The symbol MY-NEW-FOO @ #x714221d7
  which is an INTERNAL symbol in the COMMON-LISP-USER package
   0 type ---------> Bit field: #x07
   1 flags --------> Bit field: #x00
   2 hash ---------> Bit field: #xb1bc
   3 value --------> ..unbound..
   4 package ------> The COMMON-LISP-USER package
   5 function -----> #<Function MY-NEW-FOO>
   6 name ---------> A simple-string (10) "MY-NEW-FOO"
   7 plist --------> (EXCL::.ARGS. NIL), a proper list with 2 elements
[1i] CL-USER(34): :i 5 ; let's look into the function slot
#<Function MY-NEW-FOO>
  lambda-list: (X)
   0 excl-type ----> Bit field: #x08
   1 flags --------> Bit field: #xa8
   2 start --------> Bit field: #x714242c4
   3 hash ---------> Bit field: #x00005b4c
   4 symdef -------> The symbol MY-NEW-FOO
   5 code ---------> simple CODE vector (14) = #(63875 29697 52482...)
   6 formals ------> (X), a proper list with 1 element
   7 cframe-size --> fixnum 0 [#x00000000]
   8 immed-args ---> fixnum 0 [#x00000000]
   9 locals -------> fixnum 0 [#x00000000]
[1i] CL-USER(35): :i 5 ; let's look at the code vector
A simple CODE vector (14) @ #x714242d2
   0-> The field #xf983
   1-> The field #x7401
   2-> The field #xcd02
   3-> The field #x8061
   4-> The field #x977f
   5-> The field #x7400
   6-> The field #xcd02
   7-> The field #xf864
   8-> The field #x758b
   9-> The field #xc3fc
  10-> The field #x0000
  11-> The field #x0000
  12-> The field #x0000
  13-> The field #x0000
[1i] CL-USER(36): :i q ; get out of 'my-new-foo inspection
MY-NEW-FOO
CL-USER(38): 


> That was the OP's question; how to get dynamically generated code
> linked in to a running program. Just saying "use CL" doesn't explain
> the mechanism.

In CL you don't need an explaination of the mechanism unless you are
_implementing_ a Common Lisp.  For the user, you don't need to do
anything other than (compile-file ...) (load ...) to get all the
resources in native compiled form "linked" in and ready to use.  Or
for interactive work, just (compile 'your-foo-here).  That's it.  A
pleasent thing indeed.


/Jon

Re: load and use a ".o" file?

[Jon S. Anthony]

Georg Bauhaus <sb463ba@l1-hrz.uni-duisburg.de> writes:

> Stephen Leake <stephen_leake@acm.org> wrote:
> : Hmm. Do those compiled versions include the ability to create new code
> : at runtime, where the new code is also compiled, not interpreted?
> 
> If I have understood Jon correctly, there are Lisp compilers without
> an interpreter, and with the ability to load compiled code, so
> I'd guess, yes.

Yes.  Though also having an interpreter as part of the implementation
does not change this (it's orthogonal to the discussion).

> (I think I've seen the thing, too.)

Most do this, so you probably did :-)

/Jon

Re: load and use a ".o" file?

[lifetime n00b]

Jon S. Anthony wrote:

> In CL you don't need an explaination of the mechanism unless you are
> _implementing_ a Common Lisp.  For the user, you don't need to do
> anything other than (compile-file ...) (load ...) to get all the
> resources in native compiled form "linked" in and ready to use.  Or
> for interactive work, just (compile 'your-foo-here).  That's it.  A
> pleasent thing indeed.

See, I just *knew* this was going to turn out more difficult than it 
sounded! :-) First let me say I am not trying to build a complete 
"Common Lisp" implementation, at least not yet. Common Lisp is similar 
to Ada in that both language definitions are very rich and include many 
"libraries" as part of the definition itself that other languages have 
to add on as third party packages. However, one of the nicest features 
of Lisp is that the fundamental syntax of the language is extremely 
elegant and simple.

Right now, it's just this elegant and simple core that I am trying to 
implement, using gnat as the machine-code compiler. It's like a Siren, 
calling me into the deep waters. "Look how beautiful I am! Of course you 
can obtain me, just come a little closer..."

Once I have this core actually working, maybe then I can begin to think 
in terms of adding the myriad functions and operators that make up the 
rest of the Common Lisp definition. And I realize that (compile ...) and 
(load ...) are not part of the core syntax but rather part of the 
"extras" that make up a complete Common Lisp. However, without these, 
there is not much point in continuing on, since the whole point of using 
gnat as the compiler is in the context of these functions.

Re: load and use a ".o" file?

[Stephen Leake]

j-anthony@rcn.com (Jon S. Anthony) writes:

> Stephen Leake <stephen_leake@acm.org> writes:
> 
> > But I'd still like to understand what the linking process implied by
> > (compile 'foo). As far as I can see, it must do the equivalent of
> > Windows DLL or Unix so.
> 
> No, this is part of the underlying introspective environment.  It is
> actually much simpler than the contortions involved in DLL/so type
> stuff.  It is also fine grained - a function at a time or even a
> functions symbol at a time.

Well, ok. I don't have a DLL manual handy. Fine-grained would be nice.

Maybe sometime I'll find the time to read up on Lisp implementations.

I suspect lambda binding, which is very dynamic, actually simplifies
the issue of dynamic code loading.

> In CL you don't need an explaination of the mechanism unless you are
> _implementing_ a Common Lisp.  

Ok. I'm interested in the implementation details.

> For the user, you don't need to do anything other than (compile-file
> ...) (load ...) to get all the resources in native compiled form
> "linked" in and ready to use. Or for interactive work, just (compile
> 'your-foo-here). That's it. A pleasent thing indeed.

Right. In VxWorks, you get almost the same convenience, except you
have to do the compile on the host, and execute on the target.

-- 
-- Stephe