Hi,
Have you tested out to make sure what was slowing down the server? The Symantec Management Agent itself does not slow down the OS, but the execution of tasks and policies can be the reason why this is happening.
For instance the Inventory Policy, if not correctly configured can really slow down a machine.
First, try to narrow down the problem.
Regards.