Friday, August 17, 2007

Of VMWare, OpenSSL, and 64 Bit Linux

This post (rant) is probably only of potential interest to Linux nerds. If you are not such a nerd, perhaps I can recommend something more universally enjoyable.

OK. You are still with me.

So, at work, we recently received three new Dell desktops. Specifically, three new Inspiron 530 desktops. They were cheap (like $725 each, including a 20" flat panel display) and perhaps best of all, we could buy them with FreeDOS to avoid the "Windows Tax". (They are/were also available with Ubuntu pre-installed; for some reason it made more sense to order them w/ FreeDOS and install Ubuntu ourselves though.)

So, this week I've been working to get them set up. The process has been frustrating. Hence, this rant.

To begin, the Ubuntu install went pretty smoothly. Except, "out of the box," the network card isn't recognized. (This despite the fact that Dell sells this machine, with Ubuntu (!?)).

With the aid of the internet, that wasn't too hard to get around. Just download and install the Intel provided driver for the network card. Yay Ubuntu Forums and the Blogosphere for getting me past "issue #1".

Although we wanted to use Ubuntu as the host OS for these new machines, we're a practical bunch, and recognized we'd need to put on Windows as well. We have a ton of old unused Windows 2000 Professional licenses, so the plan was to throw up a Windows 2000 virtual machine on the desktop using VMWare Server.

In a typical scenario this all works out pretty nicely. We have one caveat that makes it a nightmare. We recently set up an LDAP server for centralized user account management and want to use that LDAP server to manage the users and groups for this machine.

Here is where the problems start.

Getting the machine generally to authenticate against LDAP went smoothly. Change the appropriate files in /etc/pam.d/ and you are all set. And I was tempted to think that because everything else that uses PAM for authentication was so easily set up this way, we could take care of VMWare's authentication settings (getting it to hit LDAP as well) by just changing /etc/pam.d/vmware-authd. How wrong I was.

Here's the problem. Our desktops are 64-bit systems. So, /lib/security/ is 64-bit. But the VMWare binaries are 32-bit. So, when VMWare tries to load the 64-bit, that bombs out.

That's all fairly reasonable and understandable. Apple worked out universal binaries that load under both PowerPC and Intel architectures, so it would be nice if you could have shared libraries that worked with both 64- and 32-bit programs (it would be very nice) but I don't find this "can't load 64-bit libraries in 32-programs" restriction unreasonable.

What I do find somewhat unreasonable is that VMWare isn't shipping a 64-bit version of their Server product yet. OK, they have 64-bit drivers and extensions and things ... and they can even run virtual machines with 64-bit guest operating systems ... but look, this is 2007 - VMWare's core binaries should be available as 64-bit. I have to guess that a good majority, if not a super-majority, of machines that run VMWare Server, are 64-bit machines running 64-bit operating systems. Why VMWare is still providing 32-bit binaries to run on the 64-bit machines really doesn't make sense to me. (And it would help me get past part of my problem with loading so VMWare Server will play nice with our LDAP authentication scheme.)

So, anyway, the answer looks to be with compiling a 32-bit version of that VMWare's binaries will be happy to interact with. I've tried this, but it didn't work.

It's gets even more complicated though. And this is where the third item in my post's title, OpenSSL, enters the picture.

VMWare's binaries link to OpenSSL. And, for whatever reason, the good people that have given their time to develop OpenSSL have developed it such that it's shared libary's SONAME includes the version number. So, this causes a mess because VMWare links to (and provides in it's /usr/lib/vmware/lib directory) OpenSSL 0.9.7. But, recent Linux distributions ship with 0.9.8.

So, to get around that whole mess, you basically have to compile your own OpenSSL version 0.9.7. Then, compile OpenLDAP, linking to the OpenSSL you just compiled. And then compile pam_ldap linking to that OpenLDAP you just compile. And, you don't want to make any of these things are compiling available system wide, because they are probably different versions from what your OS's package manager has installed, so when it comes time to do make install for each, be careful. Ugh. What a hassle.

And if you have a 64-bit machine running a 64-bit OS, you need to do all of those compilations targeting 32-bits. Ugh (again).

And the above series of compilations is exactly what I tried to do, but for whatever reason things still aren't working for me. VMWare is now starting to load my specially compiled, but it bombs out doing so because it can't find, despite that in my /etc/xintetd.d/vmware-authd file I have:
env = LD_LIBRARY_PATH=/usr/lib/vmware/lib/

specified (and I have my specially compiled living at that file path).

Having tried this way too many times, I'm now giving up on this whole approach and trowing in the towel and making these new desktops authenticate locally instead of against the LDAP server. What a shame.

I'll also note a couple related things though, should someone want to take up trying to get this work for themselves.

First, for users that authenticate via LDAP, here's what happens when they try to launch vmware-server-console without all the library and pam_ldap stuff set up correctly:

$ vmware-server-console
Cannot get temporary directory for log file.

VMware Server Console Error:
VMware Server Console unrecoverable error: (vmui)
Cannot get temporary directory for log file.
Please request support.
To collect files to submit to VMware support, run vm-support.
We will respond on the basis of your support entitlement.

Press "Enter" to continue...

Helpful, huh!?.

(Based on what I've found on the VMWare Forums, it looks like vmware-server-console tries to look up the current user's name via PAM and since the user comes from LDAP, that fails, and vmware-server-console can't do the look up so it's fails.)

There are some good forum threads and blog entries concerning the whole old-version-of-OpenSSL-dependency thing that are worth referencing:

Best of luck to anyone who is trying to get a similar set up to what I tried to work. If anyone does have success, I'd love to hear about it!

Update: Per the suggestion of a friend (which I wish had been as obvious to me as it was to him), I made a second attempt at this, but starting with the 32-bit version of Ubuntu 7.04. There's really no particularly compelling reason to use the 64-bit version on these workstations. When I made my initial choice my thought process just sort of when "they have 64-bit processors, I'll go with the 64-bit version of the OS."

After starting with the 32-bit version of the OS, everything went smoothly. I did have to specially compile OpenLDAP so that VMWare would use pam_ldap correctly. Following the steps referenced above as provided by "darkness" ("Using pam_ldap with VMware Server on Fedora Core 5") worked great. In terms of a bash command history, here's what that winded up looking like:

mkdir -p ~/tmp/root/usr
tar xzvf openssl-0.9.7m.tar.gz
tar xzvf openldap-2.3.38.tgz
cd openssl-0.9.7m/
./config --shared --prefix=~/tmp/root/usr
MAKEFLAGS=-j8 make && make install
cd ..
ls -al root/usr/lib/ # just to check it exists
ls -al ~/tmp/root/usr/lib/ # just to check it exists
cd openldap-2.3.38/
CPPFLAGS=-I$HOME/tmp/root/usr/include LDFLAGS=-L$HOME/tmp/root/usr/lib \
LD_LIBRARY_PATH=$HOME/tmp/root/usr/lib ./configure --prefix=/usr \
--sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --disable-slapd \
--disable-slurpd --disable-backends --disable-overlays --with-threads=posix \
--disable-static -enable-shared --enable-dynamic --enable-local --with-tls
make depend && MAKEFLAGS=-j8 make
make install DESTDIR=$HOME/tmp/root
ls -al ~/tmp/root/usr/lib/ # just to check it exists
cd ~/tmp/root/
sudo install -o 0 -g 0 -m 755 -d /usr/lib/vmware/lib/
sudo install -o 0 -g 0 -m 444 usr/lib/ \
sudo ln -s \
ls -la /usr/lib/vmware/lib/ # just to check
sudo install -o 0 -g 0 -m 755 -d /usr/lib/vmware/lib/
sudo install -o 0 -g 0 -m 444 usr/lib/ \
sudo ln -s \
ls -la /usr/lib/vmware/lib/ # just to check
ls -la /usr/lib/vmware/lib/ # just to check
ls -la /usr/lib/vmware/lib/ # just to check

Then, edit /etc/xinetd.d/vmware-authd to be:
# default: on
# description: The VMware remote access authentification daemon
service vmware-authd
disable = no
port = 902
socket_type = stream
protocol = tcp
wait = no
user = root
server = /usr/sbin/vmware-authd
type = unlisted
env = LD_LIBRARY_PATH=/usr/lib/vmware/lib/

And finally, edit /etc/pam.d/vmware-authd to the following:


#auth sufficient /lib/security/ shadow nullok
#auth required /lib/security/ shadow nullok
#account sufficient /lib/security/
#account required /lib/security/

# Changes to enable LDAP

auth sufficient
auth sufficient /lib/security/ shadow nullok
auth required /lib/security/ shadow nullok

account sufficient
account sufficient /lib/security/
account required /lib/security/


Anonymous said...

It may seem strange that the "Linux Nerd" post is the only one to attract a comment lately, but here here you go...I am such a nerd, although I haven't done anything nearly intense as your post lately. Something about linux nerds is they love to find problems like this: something so complicated, intense, and strangely mysterous that it must be conquored. There were lots of them when the kernel switched from 2.4 to 2.6, I remember having to install RedHat 7.4 or something even though Fedora was available...I couldn't get a driver to compile and work with the new kernel. I currently have a problem with evolution, an e-mail client. I only started using it to dump messages from my e-mail server to keep an archive. Some time ago evolution updated automaticaly, and now it locks up my computer when I try to read a message. I can still get in there and download the messages, and I read them on the web, but it is a lingering issue that I haven't tackled yet. You can only do so much in a day, so you have issues like these that linger and make other people think: "why Linux? you have so much trouble with it" But I think Linux nerds like to have projects like that in the back of their mind for three reasons:
1. To have something do work on when they get tired of reading Slashdot
2. Because they like the "community" of Linux users and love seeing a problem like this come to light, be picked up by someone and fixed, summarized into a "howto", and be hit #2 on Google. "The comminity is working, just like they said it would!"
3. To stick it to those companys that say they support Linux but do stupid things that cause Linux Nerds to go on a rant.

kazoolist said...

Matt - Thanks for the comment.

To echo a lot of what you said, I've found the Linux experience these days is a very reasonable one. Many (most) things "just work." I can remember the days where just trying to get Linux installed on a box was a challenge. The standard Linux disto installs these days are as easy as any other OS.

When you try to do something less standard, like authentication via LDAP, that's where things always start to get a bit challenging. And, frankly, I don't mind the ocassional challenge. When you take one a challenge and get things to work, it's pretty rewarding.

And, many of those challenges are getting easier to conquer due to the efforts of the FOSS community and more-and-better documentation on teh Interwebs. Googling an error message gets you to an answer maybe 8 out of 10 times. HowtoForge has great step-by-step instructions for things. And then blogs and forums nd mailing lists... etc etc etc.

My hope with my rant, asside from just venting some frustration, is that someone having the same problem might find it and be able to get a bit of a head start. And then, if they can get things working, they'll share how.