Hi,
I hope this KB helps clarifying the issue a bit better:
http://www.symantec.com/docs/HOWTO58882
There is a solution map provided in the article please read through it.
It might help to restart the MTA in these cases to free up RAM.
The memory allocated to the appliacne should also be reserved since Memory ballooning and other advanced memory allocation techniques which VmWare would do are not supported by the software.
Shold you be running on version 9.5.3 and still experienceing this issue please open up a support ticket.