You don't NEED to install a base Windows OS on a machine; ESX can be 'installed' as the base 'host ' OS and managed from another machine to configure virtual 'guest' Windows machines.
The thing with running the VMWare as an application on a base installed Windows Domain Controller, is that VMWare will create virtual NIC cards on top of the existing NICs on the domain controller, unless you're real careful with your NIC/IP setup and DNS, you can screw up ActiveDirectory/DNS registration for domain workstations and member servers, virtual or not.
As for performance, I have two HP DC5850 Small Form Factor PC's running VMWare ESXi, with an entire test domain running virtually.(I picked these up off of ebay for 200-300 bucks depending on the deal)
Granted, I have upgraded these to AMD Quad Core Opteron 1352's (with BIOS update) and 8 GB RAM, but that's just me.
One of my host machines is running two server 2008 servers(one 32 bit DC and the other 64 bit with Exchange 2010), another 2003 member server for messing with Sharepoint and an XP client This is about as nasty as I can get on that host without really pushing it.
By no means am I telling you to get what I got, I wanted the physically smaller Small Form Factor cases to play around with VMWare clustering/HA and didn't want two(more) big honking server chassis sitting in my office. I had to search for a SATA card that was supported under ESX.
I got a lot of helpful info for finding whitebox VMWare servers and HCL info here;
http://www.vm-help.com/