I was just wondering whether to use use VMFS or rather RDM for my home NAS running on ESXi 4.1. Now I see RDM is the right way. I’m running my ESXi host on Atom D525 so I’m curious about the performance. I’ll keep you updated.

RJ, did you try running your test also on Windows Server 2008?

Bye, Lukas, Czech republic

I currently run ESXi 4.1 (dual core 3.16Ghz 4GB DDR2 RAM) and on it an Openfiler VM with 4×1.5TB running in software raid5. This same ESX server houses my 2k8 server, debian and centos machines also. I run it purely for consolidation reasons and power consumption (the server runs at ~140watts) for my training/home server kit.

Coming from using software raid on a dedicated machine I ran through quite a few options to test out the performance hit. I tried VMFS both with and without software raid using Debian(mdadm), FreeNAS (graid) and openfiler and then I also tried the ‘unofficial’ method of RDM for the sata drives. These are all on hosts with 1×3.16Ghz core and 512MB RAM (I ran a physical P4 3Ghz and 512MB RAM and had throughput of ~100MB/s). Here’s my results:

Sustained Read/Write per second – VMFS

Debian: 42MB/38MB
FreeNAS: 55MB/43MB
Openfiler: 25MB/25MB

Sustained Read/Write per second RDM

Debian: 78MB/71MB
FreeNAS: 65MB/65MB
Openfiler: 72MB/62MB

I was prepared to take the performance hit as this meant i could turn off another computer reducing my power consumption, which was my primary goal. However after a few days it got to me a bit because by all right’s the performance shouldn’t be THAT different so i started monitoring the filer. I noticed on large reads the filer would max out in memory so i bumped it to 1GB of RAM. That has enabled me to get 100MB/s speeds across the network again! I have tried iSCSI from this also but performance is not ideal both from the server -> another physical and presenting the iSCSI to the esx instance for datastores.

I have my VM system volumes on a 10k rpm cheetah drive and use the raid volume for backups and storage only. Now would i recommend this ? for a play/home setup absolutely, it’s very flexible and can cope in those burst periods for file transfers.

I wouldn’t recommend it for business. I would imagine running the VM and having a controller to handle the RAID it would be fine but definitely dont run it like i do for production stuff for anything except storage.

At the same time however maybe a beefcake core i7 or xeon system with 16GB+ of RAM would free up these bottlenecks?? It would be much faster thats for sure, whether or not it hits limitations within ESX i dont know – i dont have the luxury of trying out better hardware at this time

RJ

This sounds like an interesting and creative solution however I am not sure about the sort of performance you would achieve with a “virtual SAN” on the same box. Also, with some of the issues I’ve run into myself, I am now firmly of the belief of keeping things simple!

You can pickup HP P400 and Dell PERC 5/6 RAID cards fairly cheaply now so I would definitely be looking at taking the RAID card path if you can.

Sam

Great info mate. Im thinking of building up the same sort of homebrew server for a home training lab.

I know that ESXi wont support ICH10R RAID which is why you went for the dedicated P400i card, but I had a thought and would be interested to know what you think before I go and buy a bunch of kit.

I would be looking at installing ESXi onto a USB key and installing 6 HDD’s directly connected to the onboard SATA of the X8SAX board. I would then create a VM (on the USB key) using something like freenas/openfiler and using RDM to present those physical disks to that VM and create a ZFS pool out of them. I can then use iSCSI to present disk to ESXi for the storage of VMDK’s.

I know that ESXi doesnt support RDM (well not officially), but it does work as I was able to get physical disks presented to a VM with some testing I did a while ago.

Im interested in what you think? is it too much mucking around, or should I spend the extra $$ on a supported RAID card?

Hard to say what I’d do if I was doing things again…

I think you need to set a clear plan and budget from the start as I quickly went from a $2k investment to what ended up being over a $3.5k AU investment when I decided I wanted “just that little bit more”. Something that stung me was the price of unbuffered ECC memory. It costs a fortune compared to regular registered/buffered ECC which you normally see in multiprocessor servers.

Although my CPU easily deals with as many VMs as I can throw at it, and disk performance is excellent, the most limiting factor seems to be the amount of RAM I have, especially with 64-bit Server 2008 VMs loving RAM (remember, less available RAM just means more paging which means more DISK I/O!). This makes me think I could probably have made do with an i7 and SATA disks and choosing to invest in RAM instead. In saying that, I am very pleased with the machine and see it serving me well for a few years to come! (Once I purchase some more RAM! $$)

For a budget “homebrew” setup, I think that a Core i7, ASUS motherboard (check the whitebox HCL: http://www.vm-help.com/esx40i/esx40_whitebox_HCL.php), non-ECC DDR3, 4 – 6 disk SATA RAID5 or even RAID10 would be more than sufficient!

If you want a Xeon CPU i’d be more inclined to save money and go with a W3530 or similar workstation level Xeon. I’m not sure i see the benefit of purchasing a multiprocessor capable E5640 for nearly 3 times the price to achieve only slightly higher performance and possibly slightly lower heat and power consumption with the smaller die size.

I haven’t upgraded to vSphere 4.1 just yet but plan to soon and don’t foresee any issues. Will update my blog when I upgrade!

Hope that helps!

I wanted to ask, if you were to complete a build again would you run a similar setup? I was looking at the option to buy a ASUS board with i7 but find that if I spend a little more I can get the better quality board etc.

Also wanted to ask have you tired Vsphere 4.1 yet? I assume that this shouldn’t have any issues working.

Cheers!

Just remember that you will need un-buffered ECC DDR3 on this motherboard – unfortunately it costs about twice as much as buffered ECC!

With SATA disks I would encourage RAID10 if possible for maximum performance. If you are able to move to Exchange 2010 you will see even more performance improvements as it has 50% less disk I/O than Exchange 2007 and is optimised for SATA disks. See here: http://www.microsoft.com/exchange/2010/en/my/storage.aspx

Many thanks for the info. My specs for the one I’m planning to build is Xeon E5520, 24gig ecc, adapted 2405 and four 1 tera wd black caviar. do you think this will be able to handle two vm manchines one for primary domain controller and the second one will run msexchange 2007 with 150users

Many thanks sam and more power

I purchased the motherboard from the following eBay seller: http://cgi.ebay.com.au/SUPERMICRO-X8SAX-MBD-X8SAX-B-BULK-MB-FREE-BIOS-UPDATE-/270391328078?cmd=ViewItem&pt=Motherboards&hash=item3ef4943d4e

I am running 8 concurrent virtual machines but am finding 12GB of RAM limiting when running 64bit Windows Server 2008 VMs. In my environment, the CPU and disk performance is ample and I am sure would cope with more concurrent virtual machines with ease. I plan to upgrade to 24GB of RAM soon to test this out.

The board does allow you to mount the stock CPU cooler that comes with the Xeon w3520 but does not come with any additional brackets. I assume the BXSTS100C requires a metal bracket to mount the cooler? If this is the case, you may need to purchase the bracket separately.

Sam