Miloping



On systems that can address up to 32 bits of memory, pointers have to be large enough to store any address – up to 32 bits – and this makes them each cost 4 bytes. Generally, this means that 32-bit systems have 4-byte pointers whereas 64-bit systems have 8-byte pointers. And that a program on a 64-bit system would consume twice as much memory as the same program on a 32-bit system.

(Okay, so it's actually more complicated.)

The other day I read about running 32-bit programs on 64-bit systems by setting up a chroot specifically for running 32-bit programs. Why would I have a program running on a 64-bit system when you could set up a 32-bit chroot in which to run the same program for half the memory? Yeah, I couldn't think of why either. So I set about seeing what was needed.

My specific cause was the pair of Mongrels which normally eat up at least 80MB each (and which now eat less; still the top two memory hungry processes on the slice, but hey):

$ ps -C mongrel_rails -o %mem,size,cmd
%MEM    SZ CMD
16.4 48244 /usr/bin/ruby1.8 /usr/bin/mongrel_rails start -d -e production -a 127.0.0.1 -c
16.3 47992 /usr/bin/ruby1.8 /usr/bin/mongrel_rails start -d -e production -a 127.0.0.1 -c

That's right. They are now taking up about 40MB – half the hungriness. Ok, let's get on with it.

Creating the chroot

First, install debootstrap and dchroot, which are important for setting up the chroot.

• debootstrap, installs a Debian base system into a directory. Use this to install a 32-bit Debian system.
• dchroot, allows us to run a program inside a chroot.

I installed everything through aptitude:

--\ Packages to be installed
debootstrap
dchroot
--\ Packages being automatically installed to satisfy dependencies
libboost-program-options1.33.1
libboost-regex1.33.1
libicu36
liblockdev1
schroot

Next, create a directory to hold the 32-bit Debian system. Then, bootstrap it.

# ~/ mkdir /var/etch-386-chroot
# debootstrap --arch i386 etch /var/etch-386-chroot http://ftp.debian.org/debian/

Notice the “–arch i386” bit in the command – this is the specific parameter to get a 32-bit system. When deboostrap finishes, ls the directory and you should see familiar root folders.

Before we can chroot into the new Debian system, add these lines to the /etc/ld.so.conf:

/var/etch-386-chroot/lib
/var/etch-386-chroot/usr/lib
/var/etch-386-chroot/usr/local/lib

Also, add the 32-bit linker to the libs by creating a symlink.

# cd /lib
# ln -s /var/etch-386-chroot/lib/ld-linux.so.2 ld-linux.so.2
# ldconfig

This next step is important: mount the /home, /tmp, /dev, and /proc directories. The /proc mount is important for /etc/init.d/mongrel_cluster to know of currently running Mongrel processes. Open up /etc/fstab and add the following lines:

/home /var/etch-386-chroot/home none rw,bind 0 0
/proc /var/etch-386-chroot/proc none rw,bind 0 0
/tmp /var/etch-386-chroot/tmp none rw,bind 0 0
/dev /var/etch-386-chroot/dev none rw,bind 0 0

Then mount them.

# mount /var/etch-386-chroot/home
# mount /var/etch-386-chroot/proc
# mount /var/etch-386-chroot/tmp
# mount /var/etch-386-chroot/dev

N.B. Please note that before deleting this chroot (i.e. rm -rf /var/etch-386-chroot) you must unmount these partitions or they will be REMOVED along with the chroot.

Now, we can test out our new 32-bit chroot.

# chroot /var/etch-386-chroot

Setting up the Rails stack

On the 32-bit chroot I only needed to install ruby, rubygems, mongrel, rails, and the mysql client libraries. It doesn't really matter how these programs get installed, but I used aptitude for ruby1.8, ruby1.8-dev, and libmysqlclient15-dev; built rubygems from source; then used rubygems to install gems for mongrel_cluster, rails and mysql.

One thing to note: if you don't choose to install ri and rdoc, then when making rubygems from source add --no-rdoc --no-ri:

~/rubygems-1.3.1# ruby setup.rb --no-rdoc --no-ri

I recall an error I had building “native extensions” in the mongrel_cluster gem. Mongrel contains some non-ruby code which needs to be built natively, so make sure to have g++ or the build-essentials package installed.

Lastly, mount the directory containing the mysqld sock.

# mkdir /var/etch-386-chroot/var/run/mysqld
# mount -o bind /var/run/mysqld /var/etch-386-chroot/var/run/mysqld

By this point, you should be able to run the Ruby part of the Rails stack entirely within the 32-bit chroot. Now the trick is calling up the stack from outside of the chroot. Remember the dchroot program installed earlier? Let's set that up.

Run 32-bit programs from outside the chroot

Add this line to /etc/dchroot.conf:

etch386    /var/etch-386-chroot

Now you can start up programs inside the chroot without having to enter it. I took this wrapper script from a helpful Debian article. Create the file /usr/bin/local/runas32bit with the following lines.

#!/bin/sh
ARGS=""
for i in "$@" ; do
        ARGS="$ARGS '$i'"
done

exec dchroot -c etch386 -d -q "`basename $0`" "$ARGS"

Then create a symlink for each part of the Rails stack we want to run outside the chroot. The symlinks allow to run the programs transparently so that users don't notice the programs are actually running from the chroot.

/usr/local/bin# ln -s runas32bit ruby1.8
/usr/local/bin# ln -s runas32bit mongrel_rails
/usr/local/bin# ln -s runas32bit mongrel_cluster

You can test that the ruby1.8 running is the 32-bit version.

# ruby1.8 --version
ruby 1.8.5 (2006-08-25) [i486-linux]

At this point, you can set up /etc/mongrel_cluster and also start up /etc/init.d/mongrel_cluster. Just remember, though, the mongrel_cluster is running inside the chroot so the configuration file needs to be symlinked or something so that it is accessible to those inside the chroot.

# cd /var/etch-386-chroot/etc/mongrel_cluster
# ln -s /etc/mongrel_cluster/mongrel_cluster.yml mongrel_cluster.yml

And, that's it. Hopefully, for those who are stuck with a 64-bit system the 32-bit chroot frees up some memory by running things inside the chroot.

Got a lot of help from this Debian article.