Multi-arch

Multi-arch

[Ludovic Brenta]

I am following up on the questions raised here about running both
32-bit and 64-bit binary code on the same machine. This is called
biarch. Currently, the gcc-4.1 source package in Debian supports biarch
for a few languages, but this support is maintenance-intensive and
partial. Most notably, Ada does not currently support biarch (and
neither does AdaCore). Furthermore, administration of a biarch system
is more complex than that of a single-arch system, as not all binary
packages support biarch, and some support packages are required, e.g.
ia32-libs.

Canonical, the company behind Ubuntu (a derivative of Debian), has
written two very interesting papers about multi-arch, including an
introduction, problems, and a proposed solution. Multi-arch is a
generalisation of biarch, and allows mixing packages for several
architectures that are compatible with a processor. For example, an
Athlon or Opteron system can run binaries for all of amd64, i386, i486,
i586 and i686. The proposed design would solve the current problems
with biarch, at the expense of rewriting dpkg almost from scratch.

http://multiarch.alioth.debian.org/

Thoughts?

-- 
Ludovic Brenta.

Re: Multi-arch

[Georg Bauhaus]

Ludovic Brenta wrote:

> Canonical, the company behind Ubuntu (a derivative of Debian), has
> written two very interesting papers about multi-arch,

IIRC, HP has always played with mounts per architecture.
They also have an interesting file system standard.
So I trust there is good reason to rewrite dpkg, given
that only uni-arch is probably reasonably simple?

Re: Multi-arch

[gshapovalov]

Ludovic Brenta wrote:
> I am following up on the questions raised here about running both
> 32-bit and 64-bit binary code on the same machine. This is called
> biarch. Currently, the gcc-4.1 source package in Debian supports biarch
[skipped]
> i586 and i686. The proposed design would solve the current problems
> with biarch, at the expense of rewriting dpkg almost from scratch.
>
> http://multiarch.alioth.debian.org/
>
> Thoughts?

Biarch, multiarch, khm, whatever :).
On the topic:
Yes, it can be done and is very well worth it. On Gentoo it is called
multilib and is supported distribution-wide, not only for x86 and
amd64, but ppc and sparc have theirs multilibs too IIRC (I think both
of them, but may be only sparc actually).  gcc is supported, as well
as Ada and most of the libs and apps (to the point where you can
have wine running 32-bit Windows code on otherwise
64bit-clean system - kernel and userspace).

This is regulated primarily via multilib and toolchain eclasses, which
is
basically the way to contain common controlling code. I recently put
Ada
on the same rails, so that now, on a multilib system, (that is for
users who
selected the multilib profile) two sets of rts libs are generated and
compiler can generate 64bit or 32bit code. Although, as there were no
requests so far, I haven't yet automatized switching between the
multilib
sub-profiles, only the usual niceties - like having FSF's 3.4.x, 4.1.x
based and AdaCore's gnat-gpl compilers installed side-by-side and
activated as necessary..

You can see the code here:
http://www.gentoo.org/cgi-bin/viewcvs.cgi/eclass/
You would be looking for the toolchain.eclass, multilib.eclass and
possibly
some other eclasses. And here:
http://www.gentoo.org/cgi-bin/viewcvs.cgi/dev-lang/gnat-gcc/
http://www.gentoo.org/cgi-bin/viewcvs.cgi/dev-lang/gnat-gpl/

The ebuilds and eclasses are mostly just a bash code, so should be
familiar, aside from a few vars that have special meaning. Some
Ada-specific implementation details and discussion can be found
in this bug:
https://bugs.gentoo.org/show_bug.cgi?id=111340
and you can catch me on irc,freenode.net in channels #gentoo and
#gentoo-dev if you would like to discuss this further (my nick there
is georges, email of course works too: george at gentoo.org), although
I suppose the way it will have to be done on Debian would be quite
different..

George

Re: Multi-arch

[Ludovic Brenta]

> Yes, it can be done and is very well worth it. On Gentoo it is called
> multilib and is supported distribution-wide, not only for x86 and
> amd64, but ppc and sparc have theirs multilibs too IIRC (I think both
> of them, but may be only sparc actually).  gcc is supported, as well
> as Ada and most of the libs and apps (to the point where you can
> have wine running 32-bit Windows code on otherwise
> 64bit-clean system - kernel and userspace).

Interesting. I've looked at multilib.eclass and gnat-gcc-4.1.0.ebuild.
I have no detailed knowledge of the Gentoo portage system, so I can
only guess, but it seems to me that the current infrastructure means
that each source package builds several binary packages, one for each
arch, on the machine where you build.  So, for example, if you have an
amd64 machine, you'd get two binaries, i386 and amd64. Furthermore, the
i386 package would install libraries in /usr/lib32, whereas the same
package built on a i386 machine would install libraries in /usr/lib, so
the two packages would be slightly different, and incompatible, even
though built for the same architecture. Debian is currently in a
similar situation, except that it has not deployed this scheme
distribution-wide but only in a few important packages, binutils and
gcc being the most prominent ones.

In the proposal, a source package would only produce one binary package
for the machine doing the build; thus your amd64 box would only produce
the amd64 binary. Then, if you want to install the i386 binary as well,
you'd take the package built by the i386 autobuilder, modify it on the
fly (this is one of the proposed changes to dpkg), and install it
alongside your amd64 package.

Of course, this scheme only makes sense in the context of a binary
distribution like Debian, but there are several benefits:

- it reduces the workload of the autobuilders
- it reduces the size of the binary distribution, and load on the
mirrors
- it simplifies system administration
- it simplifies the job of package maintainers and reduces the
opportinuties for bugs

>From what I understand, Gentoo people might not be very interested in
these benefits, because:

- Gentoo has no autobuilders, as each user recompiles the world on
their machine
- Gentoo has no binary distribution apart from the minimal
bootstrapping system
- Gentoo users seem to like system administration :)
- Gentoo package maintainers seem to like difficult problems :)

The proposal also hints at the LSB. I think it would be necessary to
standardise the library paths across all distros. The current /usr/lib,
/usr/lib32, and /usr/lib64 directories are not general enough. Consider
that some HP processors can run i386, amd64, ia64, hppa *and* hppa64
binaries on the same machine :) And what about Cell processors and
other future asymmetric multiprocessors? What about binaries intended
to run on GPUs or other coprocessors?

> you can catch me on irc,freenode.net in channels #gentoo and
> #gentoo-dev if you would like to discuss this further (my nick there
> is georges, email of course works too: george at gentoo.org)

I'd rather keep the discussion here on c.l.a as it it not specific to
Gentoo, or Debian, or any particular distribution.

-- 
Ludovic Brenta.

Re: Multi-arch

[gshapovalov]

Ludovic Brenta wrote:
> only guess, but it seems to me that the current infrastructure means
> that each source package builds several binary packages, one for each
> arch, on the machine where you build.  So, for example, if you have an
> amd64 machine, you'd get two binaries, i386 and amd64. Furthermore, the
That sums up the principle pretty well. Of course there is also a lot
of glue code to hold it all together and make it "just work"..

> i386 package would install libraries in /usr/lib32, whereas the same
> package built on a i386 machine would install libraries in /usr/lib, so
> the two packages would be slightly different, and incompatible, even
> though built for the same architecture. Debian is currently in a
Yes, that would have made them (incompatible) if we were distributing
binary packages. As it is, everything is built locally against
installed libs. So this is not an issue in our case.

> In the proposal, a source package would only produce one binary package
> for the machine doing the build; thus your amd64 box would only produce
> the amd64 binary. Then, if you want to install the i386 binary as well,
> you'd take the package built by the i386 autobuilder, modify it on the
> fly (this is one of the proposed changes to dpkg), and install it
> alongside your amd64 package.
This is actually quite similar to what is happening in Gentoo, as it
does not make sense to make two versions of *every* package. Only the
principal libs (parts of glibc and gcc rts) plus compatibility libs (of
these actually only the ones that are dependencies of requested
packages) are produced "by default". Then user is free to mix and match
in a usual fashion, although most people just stick with defaults of
course :).

[skipping this part, as you rightly point out, most of these issues do
not apply in our case]
> - Gentoo users seem to like system administration :)
> - Gentoo package maintainers seem to like difficult problems :)
:).
Although many people claim that their system administration efforts
were reduced after switching to Gentoo :). Well, as usual, this is an
issue of how you think and what tools you like I guess..

> The proposal also hints at the LSB. I think it would be necessary to
> standardise the library paths across all distros. The current /usr/lib,
> /usr/lib32, and /usr/lib64 directories are not general enough. Consider
> that some HP processors can run i386, amd64, ia64, hppa *and* hppa64
> binaries on the same machine :) And what about Cell processors and
> other future asymmetric multiprocessors? What about binaries intended
> to run on GPUs or other coprocessors?
Definitely, this will have to be done. Unfortunately I do not see it
happening *just yet* - having seen how much it takes to organize
anything on a large scale :). However in 2-3 years and when we can
persuade LSB people that Linux/BSD/FOSS is not limited to Red Hat..
(admittedly they are getting better at that lately). But we can start
by having a discussion among Gentoo and Debian toolchain people (or
what gcc/glibc "subproject" is called in Debian?)

George

PS
Does anybody know of a public newsserver that would allow to post? I
would really like to do all my email/messaging from one place (a trusty
kontact :)), rather than having to browse to google groups just to post
the reply..

Re: Multi-arch

[Björn Persson]

I understand how multi-arch is useful if you want to use proprietary 
binaries that are available only for certain architectures. Otherwise I 
don't see a reason to do it. I'd think free code would just be compiled 
for the "native" architecture. Is multi-arch useful in an altogether 
free system in some way that eludes me?

-- 
Bj�rn Persson                              PGP key A88682FD
                    omb jor ers @sv ge.
                    r o.b n.p son eri nu

Re: Multi-arch

[Ludovic Brenta]

Björn Persson a écrit :
> Is multi-arch useful in an altogether free system in some way that eludes me?

Yes.

Today, for example, OpenOffice works only on 32-bit architectures; if
you want to run it on an amd64 machine, you need biarch or multiarch.
There are probably other cases where compiling for one of the
architectures in a multi-arch system is problematic.

Also, 32-bit binaries use less memory than 64-bit binaries do. It makes
sense to run 32-bit binaries on 64-bit machines, if those binaries do
not benefit from the larger address space or 64-bit instructions.

Another benefit of multiarch is that you can e.g. compile i386 binaries
on an amd64 machine and run them on i386 machines.

-- 
Ludovic Brenta.

Re: Multi-arch

[Björn Persson]

Ludovic Brenta skrev:
> Bj�rn Persson a �crit :
>> Is multi-arch useful in an altogether free system in some way that eludes me?
> 
> Yes.
> 
> Today, for example, OpenOffice works only on 32-bit architectures; if
> you want to run it on an amd64 machine, you need biarch or multiarch.
> There are probably other cases where compiling for one of the
> architectures in a multi-arch system is problematic.

That sounds like a defect in OpenOffice, so in that case multi-arch 
support functions as a workaround for a buggy program.

> Also, 32-bit binaries use less memory than 64-bit binaries do. It makes
> sense to run 32-bit binaries on 64-bit machines, if those binaries do
> not benefit from the larger address space or 64-bit instructions.

Ah. Yes, that's a good reason if the difference in memory usage is 
significant. (A factor two perhaps?)

> Another benefit of multiarch is that you can e.g. compile i386 binaries
> on an amd64 machine and run them on i386 machines.

And then the compiler would also be an i386 program I presume. What are 
the benefits of that over an AMD64 to i386 cross compiler?

-- 
Bj�rn Persson                              PGP key A88682FD
                    omb jor ers @sv ge.
                    r o.b n.p son eri nu

Re: Multi-arch

[Ludovic Brenta]

Björn Persson <spam-away@nowhere.nil> writes:
> Ludovic Brenta skrev:
>> Björn Persson a écrit :
>>> Is multi-arch useful in an altogether free system in some way that
>>> eludes me?
>> Yes.
>> Today, for example, OpenOffice works only on 32-bit architectures; if
>> you want to run it on an amd64 machine, you need biarch or multiarch.
>> There are probably other cases where compiling for one of the
>> architectures in a multi-arch system is problematic.
>
> That sounds like a defect in OpenOffice, so in that case multi-arch
> support functions as a workaround for a buggy program.

I think it's a bit more complex than that; it may well be a defect in
OpenOffice, but also in the compiler or in any of the numerous
libraries that OpenOffice uses.  But I agree, in this case multi-arch
is a workaround; this does not make multi-arch a bad idea, though.

>> Also, 32-bit binaries use less memory than 64-bit binaries do. It makes
>> sense to run 32-bit binaries on 64-bit machines, if those binaries do
>> not benefit from the larger address space or 64-bit instructions.
>
> Ah. Yes, that's a good reason if the difference in memory usage is
> significant. (A factor two perhaps?)

A factor two for pointers and integers, yes, but not for Strings or
other data structures.  If you have many pointers, the increase in
memory usage is quite significant.

>> Another benefit of multiarch is that you can e.g. compile i386 binaries
>> on an amd64 machine and run them on i386 machines.
>
> And then the compiler would also be an i386 program I presume. What
> are the benefits of that over an AMD64 to i386 cross compiler?

On the amd64 machine, you need either an amd64-to-i386 cross-compiler
(gcc -m32 does just that), or a native i386 compiler.  Either will do,
but then you also need the i386 libraries to link against.  A proper
multi-arch design allows you to have these libraries alongside the
amd64 libraries, in a clean way as opposed to local hacks.

-- 
Ludovic Brenta.

Re: Multi-arch

[Björn Persson]

Thanks for your answers, Ludovic!

Canonical's proposal seems like "The Right Thing", although I can't say 
whether it's worth the effort. If there are people willing to do the 
work I certainly wouldn't try to stop them. (Not that I should have any 
influence over Debian.)

-- 
Björn Persson                              PGP key A88682FD
                    omb jor ers @sv ge.
                    r o.b n.p son eri nu

Re: Multi-arch

[Dr. Adrian Wrigley]

On Wed, 17 May 2006 02:22:57 -0700, Ludovic Brenta wrote:

> I am following up on the questions raised here about running both
> 32-bit and 64-bit binary code on the same machine. This is called
> biarch.

...
> Thoughts?

Since I have been raising this as an issue in the past few months,
I thought I'd give an update on my progress.

I am now running a prototype stock trading system on an amd64
system running Fedora Core 5 (x86_64).

The problem I had been trying to solve was accessing very large
memory-mapped market data arrays (several GB).  I had run out
of addressing range on ia32 systems.

Moving to amd64, however, I found I couldn't link in commercial 32-bit
library code, available in binary form only.

The system is based around Annex E distributed computing, with
client partitions accessing server code using the 32-bit library
via a Remote_Call_Interface partition.

I now build the system using amd64 architecture for all the
partitions except this Remote_Call_Interface partition, which
is built for i386.  There is no problem mixing architectures
in a program as long as each partition only links in code from
a permitted architecture. (note that it will build invalid
binaries with mixed architecture!)

The build process relies on a full i386 install of FC5 in a
separate set of partitions from the x86_64.  I installed a
complete suite of i386 GNAT tools while running in 32-bit mode.
The machine is dual-bootable, for convenience, but the build
scripts use linux32 and chroot to compile 32-bit when booted
as x86_64.

I have the same source code checked out from cvs in two
separate directories.  The build script checks the architecture,
and if it is x86_64, it builds all the partitions except those
which need the 32-bit library.  Otherwise, it builds all the
partitions.

I build one source tree in a 64 bit environment, the other
in a 32-bit chrooted environment.  I copy the 32-bit-only
partition once built into the 64-bit build directory.
This way, I get two complete working builds.

The GtkAda component 'gate' does not seem to work on x86_64.
This is invoked by the build script in a 32-bit chroot.

I had experimented with -m32 in GNAT, and this produced
working 32-bit .o files, but needed messing with at the linking
stage, since it seemed to be looking in the wrong directories
for the libraries.  This might be made to work fairly easily, but
I cannot get the -m32 switch passed from gnatdist as a compiler
argument.  This seems to be a simple bug(?).

I think having two separate build directories is necessary for
building a mixture of architectures, even if each partition is
needed once, since some .o files are needed in both architectures
(for pure partitions, for example).  I haven't yet arranged it
to build the binaries in side directories from the same source
files.  This would help ensure a consistent code base.

I would like to thank Ludovic Brenta and Martin Krischik (and
others) for working on these issues, providing helpful posts
here at c.l.a, and for providing the necessary packages for
running GNAT on these systems.

The 64-bit system is still at a development stage on a test
machine.  I may try Debian for the production system, but I think
the same build process will be needed until the issues mention
above are fixed.
--
Adrian

Re: Multi-arch

[Ludovic Brenta]

Dr. Adrian Wrigley writes:
> The build process relies on a full i386 install of FC5 in a separate
> set of partitions from the x86_64.  I installed a complete suite of
> i386 GNAT tools while running in 32-bit mode.  The machine is
> dual-bootable, for convenience, but the build scripts use linux32
> and chroot to compile 32-bit when booted as x86_64.
>
> I have the same source code checked out from cvs in two separate
> directories.  The build script checks the architecture, and if it is
> x86_64, it builds all the partitions except those which need the
> 32-bit library.  Otherwise, it builds all the partitions.

I guess you could have just one checkout, but two different object and
executable directories.  This is quite easy to set up in GNAT project
files or, failing that, Makefiles.

I don't know if the proposal from Canonical will be accepted or
implemented, much less in what time frame.  What do you think I should
do in Debian?  Continue supporting single-arch only (which still
allows you to use a chrooted system for the other arch, like you are
doing now) and wait for dpkg 2.0, or try provide multi-arch GNAT and
libraries now?

-- 
Ludovic Brenta.

Re: Multi-arch

[Dr. Adrian Wrigley]

On Thu, 18 May 2006 21:24:44 +0000, Bj�rn Persson wrote:

> I understand how multi-arch is useful if you want to use proprietary 
> binaries that are available only for certain architectures. Otherwise I 
> don't see a reason to do it. I'd think free code would just be compiled 
> for the "native" architecture. Is multi-arch useful in an altogether 
> free system in some way that eludes me?

Several reasons.

I suggest that people who run a mixture of machines may want
to share binaries across the network, but build them on their
newest machines.

People also want to avoid recompiling working code with another
architecture, even if they have source code.  You can't be sure
it'll work exactly right without testing, which may be expensive.
I found a couple of latent errors when builing for 64-bit.
One was calling a C varargs function incorrectly from Ada.
Another was caused by undocumented members of a C struct
overwriting the Ada stack.  Changing architectures is a risk.

If you have written some code in assembly language, this will
constrain the architecture until alternative code is available.

Finally, some people store records directly in data files
for various reasons.  Changing architecture would need
recreatoin of those files.

It only takes one old architecture library to be a show-stopper,
whether that is a third-party or handwritten codec, a buggy
library or whatever. 

multi-arch eases the transition to a new architecture.
--
Adrian

Re: Multi-arch

[Dr. Adrian Wrigley]

On Sun, 21 May 2006 15:41:49 +0200, Ludovic Brenta wrote:
...
> I don't know if the proposal from Canonical will be accepted or
> implemented, much less in what time frame.  What do you think I should
> do in Debian?  Continue supporting single-arch only (which still
> allows you to use a chrooted system for the other arch, like you are
> doing now) and wait for dpkg 2.0, or try provide multi-arch GNAT and
> libraries now?

I found that I could build code with -m32 in FC5 with Martin's rpms,
provided I messed around with linker commands and copied the 32-bit
libraries from elsewhere.  This would be OK, provided there was a
"how to" document somewhere explaining it.  It's what you call
"local hacks" in another post.

But the lack of a functioning gnatdist -m32 option, and lack of 'gate'
meant I had to use the chroot method.  Using chroot took a lot of time
to set up, with a whole new install and 'bind' for directories in
fstab, and implementing a dchroot script, needing sudo etc.
If gate and gnatdist were fixed, and the -m32 linking were documented,
it wouldn't be worth the effort providing multi-arch GNAT libraries
at this stage.  It's only people with special requirements like
mine who would really benefit.

Perhaps a year from now, once there are more amd64 users, and once
higher priorities are addressed, looking at the problem again would
be a good idea.

I don't know enough about the amount and value of work needed
on multi-arch GNAT to be able to be sure whether it makes sense.
But I'm sure people value a streamlined robust, current and complete
set of packages for each separate architecture.
--
Adrian