Deep Intro to VMware ESX, Part 2

In the first article I explained what a VM is and how you can identify all the files. In this article I'm showing you some best practices for storage.

This article describes VMware ESX 3.5

The Service Console on VMware ESX Server is based on RedHat Linux. All the data in this section relates to that fact.

Maximum number of partitions (which can contain data) per SCSI disk (=logical drive on RAID systems) in Linux kernel 2.4.x based systems is 15. The Service Console is based on kernel version 2.4.9. Of these max 15 partitions 3 are primary (1-3), 1 is extended (4), and 11 are logical (5-15). Extended partitions never contain any actual data. It merely acts as a container for the logical partitions.

The 14 data partition (partitions 1-3, and 5-15) limit could force you to create smaller logical drives than you wanted.

It is a nice practice to use primary partitions for non-vmfs filesystems, and logical partitions for vmfs filesystems. Nothing forces you to do that, it is just a choice.

SCSI disk naming and a word about ISO images

SCSI disks are named as /dev/sda (first), /dev/sdb (second), /dev/sdc (third). The first disk is always the boot disk and (practically always) internal to the server. The other disks are usually on an external (usually fibre-connected) disk enclosure. Please note, that many IDE-RAID controllers show up as a SCSI controller, but are currently not supported by VMware ESX Server. These disk will be supported in Vmware ESX 4.

It is strongly recommended to use CD/DVD ISO images, and/or floppy images. Images are very handy for OS and application installations. They are much more reliable and convenient than the actual physical media. Usually some portion of the server internal disk space is reserved for these ISO images. Example commands on how to create these images are given in one of the next chapters.

VMFS (vmfs, virtual machine file system) and accessibility modes

Partitions (dubbed as volumes in VMware ESX Server) formatted as vmfs do not support subdirectories. It is one thing given up for the fast speed of vmfs.

VMware ESX Server v2.x and v3.x uses version 2 of vmfs dubbed VMFS-2. These volumes (partitions) have 2 possible accessibility settings:

• public
  This is the default. On a fibre channel or similar SAN, make this volume (partition) visible to more than one ESX Server concurrently. ESX Server locks individual .vmdk files so that only one ESX Server can access it at a time. That way file system integrity is ensured.
• shared
  Shared is the same as public with one difference: The .vmdk files can be accessed by multiple ESX Servers at the same time. This accessibility setting is only recommended for clustered VMs which need to share virtual SCSI disks (.vmdk files). The most common example are Netware and Windows fault-tolerant clusters.

VMFS-2 also allows spanning up to 32 extents. Use spanning only if you really have to. It is not as bad as RAID-0, but you still have a major risk of losing it all if one volume has severe problems/fails. VMFS-1 volumes can be converted to VMFS-2 with vmkfstools command.

When you are creating a .vmdk file to a shared mode volume, the volume might go to read-only mode. You can enable writing to the volume by entering command

"vmkfstools --config writable <vmhba-name>". 

A note on .vmdk files (which are seen as physical SCSI disks in the VM)

The virtual SCSI disks (.vmdk files) have four different modes:

• persistent,
• non-persistent,
• undoable,
• append.

Last two modes are invaluable when upgrading or testing the software in a VM, as you can easily revert to the point you enabled that mode. Effectively you can undo any changes to the file system.

Strategy for placing all the files - VMware ESX Server Service Console files, and the VM files

It is wise to install VMware ESX Server to the local/internal disks (single RAID-1 SCSI logical drive preferably with a hot-spare). Dual 36 GB drives make plenty of space for mid-size systems. Dual 72 GB is actually today more common and scales up to high-end systems. You might consider buying and using 15000RPM disks because of the much higher swap.

Place all the VM-related files (guest OS's, configs and disks) to external disks (logical drives) on fibre channel (or similar).

Standard: Partitioning the VMware ESX Server local/internal disk (/dev/sda)

This is where the VMware ESX Server and its Service Console will reside. Partitioning examples are for a system with mirrored 36 GB local/internal drives.

Please note, that the hard drive gigabytes are most often not the real gigabytes. You get less than you thought.

Server internal hard drive (/dev/sda) partitioning

DeviceName PartType Size MntPoint FStype Remarks

• /dev/sda1 primary 2500 / ext3 Service Console root.
• /dev/sda2 extended 32500+ Logical partition container.
• /dev/sda5 logical 1000 swap swap Service Console swap.
• /dev/sda6 logical ~14500 /local ext3 See notes below.
• /dev/sda7 logical 100 vmcore vmcore VMware Core Dump.
• /dev/sda8 logical ~16000 /vmfs/esxswap vmfs See notes below.

Note: Your partition numbers are probably different from the example above. You will only have these partition numbers if you do partitioning with fdisk.

/vmfs/esxswap is of accessibility type "public", and formatted with volume label "esxswap". You do not define the name during install, but later by setting a volume label for the volume. It is used only for VMware swap, and nothing else. The size of this volume (partition) equals to the size of VMware swap that is created.

/local is used for CD/DVD ISO images, floppy images, etc. It is also used for backup purposes. You can also export .vmdk files faster to this volume, than to an VMFS volume.

Balance between /local and /vmfs/esxswap. Make /vmfs/esxswap just big enough for the VMware swap. Nothing else goes there. Allocate as big /local as possible.

Use /local as a backup target of the Service Console with a tar command. VMware swap file should be 1 - 2 times the amount of physical memory on the system. The default is 1 time the amount of memory. This size together with the physical RAM amount determines the possible amount of memory over-subscribing. If you think you will be adding more memory to the system, you should prepare for that when determining the size of this partition.

Service Console swap space should be 1 - 2 times the amount of memory reserved for the Service Console. That reserved memory amount depends on how many VMs are powered on and running at the same time. VMware ESX Server manual and the MUI tells how to calculate this memory amount. In the example above, the maximum required amount of Service Console memory, 512 MB is presumed. 2 times 512 MB approximates the 1000 MB swap above. This swap is defined early in the installation and cannot easily be changed. That is why it is a good idea to plan for the largest possible configuration. You don't save a lot of disk space by reducing this swap partition to 400 MB, the smallest recommended.

The maximum amount of memory to reserve for Service Console is 800 MB. It is required rather rarely.

Tuned: Partitioning the VMware ESX Server local/internal disk (/dev/sda)

This is really for experienced GNU/Linux administrators only. Use this partitioning scheme instead of the above one on VMware ESX Servers with higher security requirements. For simplicity, the possible System/Service partition is not shown.

This fine-tuning makes it possible to make the Service Console even more secure. You do not want to apply this plan unless you are very familiar with Linux and the Service Console.

DeviceName PartType Size MntPoint FStype Remarks

• /dev/sda1 primary 100 /boot ext3 Service Console kernel is here.
• /dev/sda2 primary 500 / ext3 Service Console root.
• /dev/sda3 extended 34000+ Logical partition container.
• /dev/sda5 logical 500 /tmp ext3 This is really not much used.
• /dev/sda6 logical 1000 /var ext3 All the local log files will be here.
• /dev/sda7 logical 1500 /usr ext3 All standard programs are here.
• /dev/sda8 logical 50 /usr/local ext3 All server local scripts/programs.
• /dev/sda9 logical 350 /opt ext3 Optional 3rd party programs.
• /dev/sda10 logical 1000 swap swap Service Console swap. See note below.
• /dev/sda11 logical 11000 /local ext3 See notes below.
• /dev/sda12 logical 100 vmcore vmcore VMware Core Dump.
• /dev/sda13 logical 16000 /vmfs/esxswap vmfs See notes below.

Important! Please also see the notes above in the standard partitioning example.

Actual Service Console partition order and numbers might differ a bit from the above example unless you do partitioning manually with fdisk.

Some further site and/or installation dependent fine-tuning on partition sizes might be necessary.

For increased security, you can mount /boot, /usr, /usr/local, and maybe /opt (depending on installed software) as read-only when in production. Also, the mounting of /local could be made non-automatic.

You might be able to unmount /boot after system startup.

/home is not used in VMware ESX Server, and thus not required, nor defined.

You might need a bit larger /opt in some cases. Check and know your optional 3rd party software.

Use /local for backing up the Service Console with a tar command. More information on that later in this document.

About 2000 MB of internal disk space will be left unpartitioned in the extended partition for possible future requirement.

Partitioning and otherwise handling the external disk space for VMs

There are two grand schemes with partitioning the external disk space where VMs reside. You can combine the schemes, if required. Both schemes are presented below:

• "Scheme A": Place all small files (.vmx, .log, .txt, and nvram) belonging to one VM into one ext3 formatted directory (/vmconf/vm001/), and all big .vmdk, and .REDO files, and the .vmss files into a single large vmfs formatted partition (/vmfs/public01/).
• "Scheme B": Place all small files (.vmx, .log, .txt, and nvram) belonging to one VM into one ext3 formatted directory (/vmconf/vm001/). Then create a vmfs partition for the VM, which is big enough for all its .vmdk, and .REDO files, and the .vmss file. The vmfs partition should be named (during formatting) after the VM (vm001), so it can be accessed as /vmfs/vm001/.

Please note, that both schemes place VM configuration, and other related small files to a subdirectory under the /vmconf directory.

Scheme A is absolutely the most common, and used about 99 % of the time. It is however possible to mix both schemes at the same time. Some organisations use scheme B for the most important VMs.

As you see it is a lot of information. In the last article of this series I will add some scripts to do this automatically. That will prevent you from having a lot of work.

You can read Part 1 here: Deep Intro to VMware ESX, Part 1