Hello,
This also happens if install SEPM on Windows Server 2008 64bit. If so, please run following to run IIS as 64 bit mode.
Set the Enable32bitAppOnWin64 option to False at the Parent level under the master Web site level. To do this, follow these steps:
1. In IIS Manager, click Application Pools.
2. In the Actions pane, click Set Application Pool Defaults.
3. In the General section, set the Enable32bitAppOnWin64 option to False.
Check this Article:
Java -1 Errors and HTTP 500 errors when connecting to the Symantec Endpoint Protection Manager on a Windows Server 2008 64 bit
http://www.symantec.com/docs/TECH96006
How do you run the Symantec Endpoint Protection Manager (SEPM) on a 64-bit machine - IIS (7 or 6), the Database (Embedded or SQL), ASP(1.0 and 2.0), Java as 32-bit or 64-bit
http://www.symantec.com/docs/TECH96970
Again, The minimum SEP 11 version you should be using is in the RU7 family; 11.0.7000, 11.0.7100 or latest of the SEP 11's, 11.0.7200.
Is this a fresh installation or was the SEPM working previously?
If this is a Fresh Installation, the first suggestion would be to install SEP 11.0.7200 rather than older version of SEP 11.0 RU5. Check this -
Latest Symantec Endpoint Protection Releases - SEP 12.1 RU1 MP1 and SEP 11.0. RU7 MP2
Hope that helps!!