I'm reading the question a bit differently, so I figured I might as well add to the answer in case it's helpful:
Sounds to me the question isn't VMware ESX vs Server, rather running Linux appliance vs Windows VM. If that's the case, please note that the VE appliances are Linux just like the hardware appliances, not Windows. So it sounds to me the real question is whether you can mix and match virtual appliances and hardware. As John wrote, the answer is 'yes' with the lastest updates.
(In fact, if I didn't know better, the question truly reads that you're looking to run a VM on VMware Server in a Windows VM on ESX. If so, that's a highly curious config, so I assume not.)