This isn’t supported by VMware obviously or you’d be looking at their documentation.  However, this is an extremely useful way of provisioning ESXi on to servers.  For starters, suppose you have a server running some accounting software during the day and you want to test that server by running virtual machines on it (VMs will be mounted from a NAS or something somewhere else).

One of the options I’ve seen from IBM is that you can buy their cool thumb drive which will set you back perhaps a few $100 ?  I don’t know the price.  The other option is to say “Thumb drives are so 2001.  Get with the times man!”  And then pay nothing by just network booting the image.  Network booting is the only way to have real control of your data center and make it agile.   Otherwise you’re stuck with the same scenario of the fool with the crash cart which is what you’re trying to avoid, right?  If you’re using a crash cart to configure your servers, then you probably don’t have a cloud, nor an agile data center.

Ok grasshopper, this is how it is done:
1. Download the image from VMware. In my example, I’m getting the ESXi 4.0 image specialized for IBM hardware so that I can get the remote console to work. (That’s what happens if you don’t get the right one).

2. Mount the image and copy the contents of the file to some place. I copied all the files to /install/esxi4/x86_64/

3.  cd /install/esxi4/x86_64/;  tar zxvf image.tgz

4.  This should create the directory usr/lib/vmware/installer.  Now extract this image:

  • cd usr/lib/vmware/installer
  • bunzip2 VMware-VMvisor-big-4.0.0-208167-x86_64.dd.bz2

5.  Take a look at the sectors to see which one we need to get:

  • fdisk -lu VMware-VMvisor-big*dd
# fdisk -lu VMware-VMvisor-big-4.0.0-208167-x86_64.dd
last_lba(): I don't know how to handle files with mode 81a4
You must set cylinders.
You can do this from the extra functions menu.

Disk VMware-VMvisor-big-4.0.0-208167-x86_64.dd: 0 MB, 0 bytes
64 heads, 32 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes

 Device Boot      Start         End      Blocks   Id  System
VMware-VMvisor-big-4.0.0-208167-x86_64.dd1            8192     1843199      917504    5  Extended
VMware-VMvisor-big-4.0.0-208167-x86_64.dd4   *          32        8191        4080    4  FAT16 <32M
VMware-VMvisor-big-4.0.0-208167-x86_64.dd5            8224      520191      255984    6  FAT16
VMware-VMvisor-big-4.0.0-208167-x86_64.dd6          520224     1032191      255984    6  FAT16
VMware-VMvisor-big-4.0.0-208167-x86_64.dd7         1032224     1257471      112624   fc  VMware VMKCORE
VMware-VMvisor-big-4.0.0-208167-x86_64.dd8         1257504     1843199      292848    6  FAT16

Partition table entries are not in disk order

The goods are inside *dd5.  We see that the sector starts at 8224.  Lets mount this partition.

6.  mkdir /install/stage

7.  mount VMware-VMvisor*dd /install/stage -o loop,offset=4210668

  • offset is 4210668 because 512 * 8224 !

8.  ls /install/stage

boot.cfg    cim.vgz      mod.tgz  pkgdb.tgz  vmkboot.gz
cimstg.tgz  license.tgz  oem.tgz  sys.vgz    vmk.gz

9.  Lets modify these images so that we can SSH into it.  SSH is just so useful!  This is done by making our own mod.tgz.  If you extract the one you see there then you’ll find that its empty!  So lets put something in it.

  1. mkdir -p tmpdir/etc/vmware/init/init.d/
  2. Put a file called ’47.enable-ssh’ that has the following:
#!/bin/sh
sed -ie 's/#ssh/ssh/' /etc/inetd.conf #ssh is too nice not to have
return ${SUCCESS}

3.  copy an /etc/shadow into tmpdir/etc/shadow that has root’s password so you can ssh in with your password.

4.  Go back to your tmpdir root and tar it up:  tar czvf mod.tgz etc/

5.  Copy this mod.tgz over the empty one.

Nice work!  Now you’re ready to rock.  You’ll need to set up dhcp and tftp to deliver the goods.  Here’s what we do:

  1. Put all those files (boot.cfg, cimstg,tgz, you new mod.tgz file, mboot.c32, oem.tgz, pkgdb.tgz….. all those) into /tftpboot/esxi4/
  2. Make a PXE file for the node.  Mine looks like this:
#netboot esxi4
DEFAULT xCAT
LABEL xCAT
 KERNEL /esxi4//mboot.c32
 APPEND /esxi4/vmkboot.gz ---\
/esxi4/vmk.gz --- \
/esxi4/sys.vgz --- \
/esxi4/cim.vgz --- \
/esxi4/oem.tgz --- \
/esxi4/license.tgz --- \
/esxi4/mod.tgz --- \
/esxi4/cimstg.tgz

Provided you have tftpboot and dhcp setup correctly, you can now just pxeboot these files and watch as your node comes up in a stateless ESXi hypervisor.  Now I know what you’re saying:  What about keeping track of virtual machines on the server?  If it reboots, it comes up fresh and I don’t know what virtual machines should be there?  This is true.  But I’m not going into that right now.  You’ll have to pay extra for that :-)

Anyway this may all seem like a lot of work to get up and running, but the convenience is awesome.  All this information was gleaned from the xCAT source code that we wrote.  xCAT does this for you automatically.  Or for a more painless setup, you could use the Sumavisor to do it.  Anyway, hope this helps and gets you thinking!

Ah, one last tidbit:

If you are on the console and want to log in directly to ESXi4, just do Alt+F1 and type: unsupported, then the prompt will come up (telling you its unsupported) and then you can just enter the root password you had from your /etc/shadow that you copied in and have access.