check if you have any file related to symantec endpoint in c:\windows\temp folder, a tempporary file, if so delete that file
try increasing the java heap size as per this document and restart SEPM service.
- Click Start> Run.
- In the Run dialog box, type regedit
- Press Enter.
- Locate the following registry key:
HKLM\SYSTEM\CurrentControlSet\Services\semsrv\Parameters\
- Locate the following keys:
JVM Option Number 0
JVM Option Number 1
- Adjust the key values upward, and for optimal performance, match the key values.
For example, to create a 1 GB static heap, set: "JVM Option Number 0" to "-Xms512m", and set "JVM Option Number 1" to "-Xmx1024m".
- Close Regedit.
- Click Start> Settings> Control Panel> Administrative Tools
- In the Services dialog box, right-click Symantec Endpoint Protection Manager, and then click Restart.