Realistically, that is not unexpected as a virtual machine has to share resources with other VMs and also the host machine. I assume that the VM has the same RAM and CPU setup as the physical machine you used as your benchmark, but even in those circumstances you won't get equivalent performance.
So the answer to your original question is yes - you can host Ghost on a virtual server. It is now up to your client/boss to decide whether they need the absolute performance available from a physical machine or are satisfied with the slower VM performance.
Just one final thought - does Xen server allow you to dedicate one NIC in the host machine to a specific VM, and does it allow a VM to have a dedicated disk volume? This might offer some performance improvement as the bottlenecks in Ghost are disk and NIC transfer rates.