So I’ve decided I need to get with the times and build myself a small VMware ESXi 4 lab at home. I will be using the lab to consolidate both my pfsense router (currently on an old P3 desktop), my Debian mail/web/dhcp/dns server (currently on an old Compaq EVO P4 1.6 desktop) and to use as a test environment to play with and train myself on the latest technologies. I estimate I will be running around 6 VMs at one time. My file server will remain physical.

I am prepared aiming to spend around $2000AU on this project.

I spent a bit of time toying with whether I would be better purchasing a second hand ~4 year old Dell/HP/IBM server off eBay or whether I was prepared to have a go building a “whitebox” ESX server. The benefit of a brand name server was guaranteed hardware compatibility and also good disk performance using SAS drives and hardware RAID controllers – disk IO and performance is critical in VMware. In the end I came to the conclusion that if I chose the right compatible parts, I would have a much faster system for a similar price if I built it myself. I also figured that using comodotity off the shelf parts it would be far easier to get replacement parts from the local computer shop if something failed rather than hunting around on eBay and waiting for delivery. The spare 4RU Antec rackmount case I had lying around may have persuaded me as well!

Once I’d made that decision it was time to sort out a CPU platform to use. Looking at the parts for sale at my local computer shop, the choice I had was between using an older Core 2 Quad Q9400 platform or a new Core i7 930 platform. Obviously the Core i7 was going to be faster but it was also more expensive! So I did some more research. The decision was soon made when I discovered that the maximum RAM I was going to be able to use with the Q9400 was 8GB whereas the Core i7 chipsets supported up to 24GB. Core i7 it is!

Now it was time to sort out a motherboard. This is where compatibility with ESX can be a problem! I considered a standard desktop board but didn’t want to have to pay for a deluxe model with useless features just to get dual LAN and wanted something I knew would be 24/7/365 server quality. I had a look around on various whitebox ESX forums and and soon came across the Supermicro X8SAX, an Intel X58 chipset server motherboard with 2x ESX compatible Gigabit Intel LAN ports – handy! The problem was… Distribution of Supermicro gear is pretty limited in Australia and this motherboard was nowhere to be found. I started hunting around on eBay and soon found the motherboard for roughly $350AU delivered from the US. Done!

Researching online, it soon became clear that no onboard SATA/RAID controller was going to be up to the task of deliverying good disk performance for VMware ESX. Being familiar with HP servers everyday in my job, I soon honed in on the HP P400i 512MB BBWC SATA/SAS PCIe controller. This controller can be quite expensive however I managed to pick one up for a steal on eBay and there are many more still to be found. If using the P400i, make sure you purchase the additional battery with 512MB cache and not just the standard 256MB model as the controller is not able to do write caching without it and write performance will be poor.

Update 31/03/2010:

I’ve had to make some changes to my original server hardware configuration after realising that Intel Core i7 processors do not support ECC memory. I have now decided to go with an Intel Xeon W3520 processor for around $20 more than the Core i7 930 I was originally planning on using.

Update 13/04/2010:

After doing some research, I’ve decided not to go with a redundant power supply and have purchased an Antec TruePower 550W. The ATX form factor redundant power supplies I found were generally quite expensive and not very high wattage. I would also find it difficult to find a replacement power unit if one were to fail. The TruePower is a good quality power supply and given the server will be protected by an APC Smart-UPS I envisage it lasting a long time. If it does happen to fail then I can pick another one up from the local computer shop!

I was really worried about the performance I would achieve with SATA disks, even WD VelociRaptors, in a ESX environment so bit the bullet and decided to go with SAS disks. Unfortunately SSD was out of the question given the cost per gigabyte. I managed to find brand new HP 146GB 15000RPM SAS drives on eBay for an absolute bargain so have purchased 4 disks. I plan to run the disks in a RAID10 for maximum performance.

I have purchased a SNT-3051 5 disk SATA/SAS hotswap enclosure from eBay to install the SAS drives in. These are available in both single and dual port configurations. I am yet to test how well these work but they are excellent value for money and make it easier to monitor the status of individual drives and replace them in case of failure. If they work well then I will purchase another two enclosures for my file server.

Project cost so far: ~$3000AU. $1000 over budget

Update 20/04/2010:

It’s alive! I still haven’t received the SAS disks I ordered off eBay, although TNT reports they are in Perth, but in my excitement last weekend I decided I would setup the server with a single SATA disk for now and migrate the VMs to the SAS RAID when the disks arrive. I purchased a 4GB Kingston USB memory stick to install VMware ESXi on so I didn’t have to worry about moving the install later (instructions here). The X8SAX has internal onboard USB ports which makes this a very sexy setup!

Little did I know the issues I was in for with my new ESXi server!

I put everything together and the server POST first time perfectly… however, the spare SATA disk I had was faulty (unbeknownst to me) and the HP P400i controller would not pick it up. I did not realise that with no disks detected the controller would not present the F8 option to boot into the controller settings so spent quite a long time trying to figure out if there was something wrong with the RAID controller. In desperation I tried another SATA disk, it was picked up straight away and the controller now let me select F8 at POST and enter into the controller settings.

Now it was time to install ESXi… where I ran into more problems, which I will document soon as I need some sleep! Stay tuned.

Update 25/04/2010:

Unfortunately I have been quite busy so have not had time to update this post. My new ESXi server is now complete and is running brilliantly. I have now installed the SAS disks and have decided to use a RAID5 configuration to try and get more usable space and also to allow for easier expandability later on if required. I realise I may be giving up some performance over RAID10 but this a sacrifice I’m prepared to make. Given the low load of this server as a home lab setup, and the high cost of SAS disks, I think next time I would definitely consider higher capacity SATA disks in a RAID10 configuration rather than SAS for my purposes, I’m running 6 VMs (4x Windows Server 2008) the VMs flew even when I was running with a single SATA disk! Despite not being quite the quality of a HP/Dell/IBM enclosure, The SNT-3051 5 disk SATA/SAS hotswap enclosure I purchased is working brilliantly with both the SAS and SATA disks and comes highly recommended!

So anyway, about the issues I ran into with the X8SAX… basically, despite this motherboard being listed as compatible in the whitebox HCL, I was unable to get VMware ESXi to install or boot with ACPI enabled. The server would simply display a PSOD (Purple Screen of Death – it’s definitely more pink than purple!) after loading the initial boot images.

The error I received was near identical to the one described in this VMware Communities forum post.

Panic from another CPU cpu 0 world 0

To install ESX I was forced to disable the ACPI APIC Support option in the BIOS under ACPI Configuration. I assume I could also have passed the noapic option to the installer but did not try this. Trying to find the source of the issue I tried everything from reseating RAM, changing video cards, removing the RAID controller, various BIOS settings (I am running the latest 1.1 BIOS revision) and everything inbetween but no luck.

With ACPI APIC Support disabled ESX would install and run without issue however would PSOD on shutdown with an ACPI or APIC related error. I have now disabled the VMkernel.Boot.ACPI option on the ESX server (Configuration > Advanced Settings on the host in the vSphere Client) and the server shutdowns cleanly.

Unfortunately without ACPI support I cannot use hyperthreading or any power management features. I have received many hits on this blog specific to the X8SAX and VMware ESX so plan to contact Super Micro to see if there are any obvious solutions or reasons behind the ACPI issue.

[IMPORTANT] Update 15/04/2010

Thanks to Adrian’s comment, I was motivated to spend a bit more time investigating this issue. I checked the post I had made on the vm-help.com forums regarding the X8SAX, and was pleasantly surprised to find 2 users had replied saying that they had managed to get ACPI functioning by using the 1.0c BIOS revision.

I went to SuperMicro’s support website to try and download 1.0c but was unable to find it. What I did find however, was BIOS revision 1.1a which was released only 2 weeks ago on 29/04/2010.

I have now flashed my BIOS with version 1.1a and have enabled ACPI. I am no longer suffering from any ACPI issues and now have hyperthreading and power management!

The hardware:

• CPU – Intel Core i7 930 2.8Ghz Xeon W3520 2.66Ghz (Quad Core)
  
• Motherboard – Supermicro X8SAX
• Storage Controller – HP P400i 512MB BBWC SATA/SAS
• RAM – 12GB un-buffered ECC DDR3-1333
• Power Supply – Antec TruePower 550W Modular Power Supply
• Disks – 4x HP 146GB 15000RPM 3.5″ SAS
• Case – Antec 4U22ATX
• 4GB Kingston USB Memory Stick
  

I will keep this post updated as I move forward with this project.

Some links I have found useful:

• http://www.vm-help.com/esx40i/esx40_whitebox_HCL.php
• http://virtualfuture.info/2009/02/esx-whitebox-local-storage-perfomance-tip/
• http://episteme.arstechnica.com/eve/forums/a/tpc/f/77909774/m/167002658931
• http://www.techhead.co.uk/installing-vmware-esxi-4-0-on-a-usb-memory-stick-the-official-way