Of VMWare, OpenSSL, and 64 Bit Linux

This post (rant) is probably only of potential interest to Linux nerds. If you are not such a nerd, perhaps I can recommend something more universally enjoyable.

OK. You are still with me.

So, at work, we recently received three new Dell desktops. Specifically, three new Inspiron 530 desktops. They were cheap (like $725 each, including a 20" flat panel display) and perhaps best of all, we could buy them with FreeDOS to avoid the "Windows Tax". (They are/were also available with Ubuntu pre-installed; for some reason it made more sense to order them w/ FreeDOS and install Ubuntu ourselves though.)

So, this week I've been working to get them set up. The process has been frustrating. Hence, this rant.

To begin, the Ubuntu install went pretty smoothly. Except, "out of the box," the network card isn't recognized. (This despite the fact that Dell sells this machine, with Ubuntu (!?)).

With the aid of the internet, that wasn't too hard to get around. Just download and install the Intel provided driver for the network card. Yay Ubuntu Forums and the Blogosphere for getting me past "issue #1".

Although we wanted to use Ubuntu as the host OS for these new machines, we're a practical bunch, and recognized we'd need to put on Windows as well. We have a ton of old unused Windows 2000 Professional licenses, so the plan was to throw up a Windows 2000 virtual machine on the desktop using VMWare Server.

In a typical scenario this all works out pretty nicely. We have one caveat that makes it a nightmare. We recently set up an LDAP server for centralized user account management and want to use that LDAP server to manage the users and groups for this machine.

Here is where the problems start.

Getting the machine generally to authenticate against LDAP went smoothly. Change the appropriate files in /etc/pam.d/ and you are all set. And I was tempted to think that because everything else that uses PAM for authentication was so easily set up this way, we could take care of VMWare's authentication settings (getting it to hit LDAP as well) by just changing /etc/pam.d/vmware-authd. How wrong I was.

Here's the problem. Our desktops are 64-bit systems. So, /lib/security/pam_ldap.so is 64-bit. But the VMWare binaries are 32-bit. So, when VMWare tries to load the 64-bit pam_ldap.so, that bombs out.

That's all fairly reasonable and understandable. Apple worked out universal binaries that load under both PowerPC and Intel architectures, so it would be nice if you could have shared libraries that worked with both 64- and 32-bit programs (it would be very nice) but I don't find this "can't load 64-bit libraries in 32-programs" restriction unreasonable.

What I do find somewhat unreasonable is that VMWare isn't shipping a 64-bit version of their Server product yet. OK, they have 64-bit drivers and extensions and things ... and they can even run virtual machines with 64-bit guest operating systems ... but look, this is 2007 - VMWare's core binaries should be available as 64-bit. I have to guess that a good majority, if not a super-majority, of machines that run VMWare Server, are 64-bit machines running 64-bit operating systems. Why VMWare is still providing 32-bit binaries to run on the 64-bit machines really doesn't make sense to me. (And it would help me get past part of my problem with loading pam_ldap.so so VMWare Server will play nice with our LDAP authentication scheme.)

So, anyway, the answer looks to be with compiling a 32-bit version of pam_ldap.so that VMWare's binaries will be happy to interact with. I've tried this, but it didn't work.

It's gets even more complicated though. And this is where the third item in my post's title, OpenSSL, enters the picture.

VMWare's binaries link to OpenSSL. And, for whatever reason, the good people that have given their time to develop OpenSSL have developed it such that it's shared libary's SONAME includes the version number. So, this causes a mess because VMWare links to (and provides in it's /usr/lib/vmware/lib directory) OpenSSL 0.9.7. But, recent Linux distributions ship with 0.9.8.

So, to get around that whole mess, you basically have to compile your own OpenSSL version 0.9.7. Then, compile OpenLDAP, linking to the OpenSSL you just compiled. And then compile pam_ldap linking to that OpenLDAP you just compile. And, you don't want to make any of these things are compiling available system wide, because they are probably different versions from what your OS's package manager has installed, so when it comes time to do make install for each, be careful. Ugh. What a hassle.

And if you have a 64-bit machine running a 64-bit OS, you need to do all of those compilations targeting 32-bits. Ugh (again).

And the above series of compilations is exactly what I tried to do, but for whatever reason things still aren't working for me. VMWare is now starting to load my specially compiled pam_ldap.so, but it bombs out doing so because it can't find libldap-2.3.so.0, despite that in my /etc/xintetd.d/vmware-authd file I have:

env = LD_LIBRARY_PATH=/usr/lib/vmware/lib/libldap-2.3.so.0:/usr/lib/vmware/lib/liblber-2.3.so.0

specified (and I have my specially compiled libldap-2.3.so.0 living at that file path).

Having tried this way too many times, I'm now giving up on this whole approach and trowing in the towel and making these new desktops authenticate locally instead of against the LDAP server. What a shame.

I'll also note a couple related things though, should someone want to take up trying to get this work for themselves.

First, for users that authenticate via LDAP, here's what happens when they try to launch vmware-server-console without all the library and pam_ldap stuff set up correctly:


$ vmware-server-console
Cannot get temporary directory for log file.

VMware Server Console Error:
VMware Server Console unrecoverable error: (vmui)
Cannot get temporary directory for log file.
Please request support.
To collect files to submit to VMware support, run vm-support.
We will respond on the basis of your support entitlement.


Press "Enter" to continue...


Helpful, huh!?.

(Based on what I've found on the VMWare Forums, it looks like vmware-server-console tries to look up the current user's name via PAM and since the user comes from LDAP, that fails, and vmware-server-console can't do the look up so it's fails.)

There are some good forum threads and blog entries concerning the whole old-version-of-OpenSSL-dependency thing that are worth referencing:

• darkness - Using pam_ldap with VMware Server on Fedora Core 5


• VMWare Forums - Auth problems (LDAP backend with TLS)


• VMWare Forums - RHEL5 ldap auth

Best of luck to anyone who is trying to get a similar set up to what I tried to work. If anyone does have success, I'd love to hear about it!

Update: Per the suggestion of a friend (which I wish had been as obvious to me as it was to him), I made a second attempt at this, but starting with the 32-bit version of Ubuntu 7.04. There's really no particularly compelling reason to use the 64-bit version on these workstations. When I made my initial choice my thought process just sort of when "they have 64-bit processors, I'll go with the 64-bit version of the OS."

After starting with the 32-bit version of the OS, everything went smoothly. I did have to specially compile OpenLDAP so that VMWare would use pam_ldap correctly. Following the steps referenced above as provided by "darkness" ("Using pam_ldap with VMware Server on Fedora Core 5") worked great. In terms of a bash command history, here's what that winded up looking like:

mkdir -p ~/tmp/root/usr
wget http://www.openssl.org/source/openssl-0.9.7m.tar.gz
wget ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-2.3.38.tgz
tar xzvf openssl-0.9.7m.tar.gz 
tar xzvf openldap-2.3.38.tgz 
cd openssl-0.9.7m/
./config --shared --prefix=~/tmp/root/usr
MAKEFLAGS=-j8 make && make install
cd ..
ls -al root/usr/lib/libssl.so # just to check it exists
ls -al ~/tmp/root/usr/lib/libssl.so # just to check it exists
cd openldap-2.3.38/
CPPFLAGS=-I$HOME/tmp/root/usr/include LDFLAGS=-L$HOME/tmp/root/usr/lib \
   LD_LIBRARY_PATH=$HOME/tmp/root/usr/lib ./configure --prefix=/usr \ 
   --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --disable-slapd \
   --disable-slurpd --disable-backends --disable-overlays --with-threads=posix \
   --disable-static -enable-shared --enable-dynamic --enable-local --with-tls
make depend && MAKEFLAGS=-j8 make
make install DESTDIR=$HOME/tmp/root
ls -al ~/tmp/root/usr/lib/libldap.so # just to check it exists
cd ~/tmp/root/
sudo install -o 0 -g 0 -m 755 -d /usr/lib/vmware/lib/libldap-2.3.so.0
sudo install -o 0 -g 0 -m 444 usr/lib/libldap-2.3.so.0.2.26 \
  /usr/lib/vmware/lib/libldap-2.3.so.0/
sudo ln -s libldap-2.3.so.0.2.26 \
  /usr/lib/vmware/lib/libldap-2.3.so.0/libldap-2.3.so.0
ls -la /usr/lib/vmware/lib/libldap-2.3.so.0/libldap-2.3.so.0 # just to check
sudo install -o 0 -g 0 -m 755 -d /usr/lib/vmware/lib/liblber-2.3.so.0
sudo install -o 0 -g 0 -m 444 usr/lib/liblber-2.3.so.0.2.26 \
  /usr/lib/vmware/lib/liblber-2.3.so.0/
sudo ln -s liblber-2.3.so.0.2.26 \
  /usr/lib/vmware/lib/liblber-2.3.so.0/liblber-2.3.so.0
ls -la /usr/lib/vmware/lib/liblber-2.3.so.0/liblber-2.3.so.0 # just to check
ls -la /usr/lib/vmware/lib/libldap-2.3.so.0 # just to check
ls -la /usr/lib/vmware/lib/liblber-2.3.so.0 # just to check

Then, edit /etc/xinetd.d/vmware-authd to be:

# default: on
# description: The VMware remote access authentification daemon
service vmware-authd
{
    disable         = no
    port            = 902
    socket_type     = stream
    protocol        = tcp
    wait            = no
    user            = root
    server          = /usr/sbin/vmware-authd
    type            = unlisted
    env             = LD_LIBRARY_PATH=/usr/lib/vmware/lib/libldap-2.3.so.0:/usr/lib/vmware/lib/liblber-2.3.so.0
}

And finally, edit /etc/pam.d/vmware-authd to the following:

#%PAM-1.0

#ORIG CONFIG
#auth       sufficient       /lib/security/pam_unix2.so shadow nullok
#auth       required         /lib/security/pam_unix_auth.so shadow nullok
#account    sufficient       /lib/security/pam_unix2.so
#account    required         /lib/security/pam_unix_acct.so

# Changes to enable LDAP

auth       sufficient       pam_ldap.so
auth       sufficient       /lib/security/pam_unix2.so shadow nullok
auth       required         /lib/security/pam_unix_auth.so shadow nullok

account    sufficient       pam_ldap.so
account    sufficient       /lib/security/pam_unix2.so
account    required         /lib/security/pam_unix_acct.so