Video Screencast Help

Be Rid of Symantec

Created: 22 Oct 2013 • Updated: 22 Oct 2013 | 30 comments

As I suspected and the reason that after 20 years I never seek so called technical support- there is none. . So I am going and would advise everyone is to rip out Symantec completely from their networks and go somewhere else. None of us have time for this nonsense. In my case that will involve about 8 companies and more than 3000 workstations and further i will be on my radio show in Dallas/Fort Worth and will be blasting Symantec for the next 6 weeks and advising people not to waste their time with nonsense.

 

Thanks guys for the experience It has been much like living in the jungle,.

 

Tom in Dallas

Operating Systems:

Comments 30 CommentsJump to latest comment

Mithun Sanghavi's picture

Hello,

Is there something which I could assist you with?

Could you please assist us and explain us the issue you are facing with?

In case you have created a case with Symantec, please PM me your case # and I would surely assist you.

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

Jazzwineman's picture

For the last 5 weeks (now 6 weeks) in a row on every Monday Morning at around 3:00-3:50 am, I will get the following message emailed to me:

 

Message from:
    Server name: xxxxx1
    Server IP: 192.168.xx.xx
    
The Symantec Endpoint Protection Manager database has gone down and needs immediate attention.

 

The corresponding event id reads as follow:

 

Event Type:    Error
Event Source:    SQLANY 12.0
Event Category:    None
Event ID:    1
Date:        10/14/2013
Time:        3:19:36 AM
User:        N/A
Computer:    xxxxx
Description:
SQLANYs_sem5

 

----------

 

I go to the server (2003 SBS and no it was not my choice) go to the Symantec embedded Database which is set to automatic and simply start from services and everything runs fine the entire week until the next Sunday am.

 

This only happens then. This is a closed company of 19 users with Exchange 2003 and MDaemon Server 13.03 running on another machine (win XP) and SEPM 12.1.3001.165

 

There is zero happening at 3am on Monday morning with the exception of an email or two coming into exchange and be ready for delivery in the morning when they log in with the workstations.

 

We do backups every night/morning of the week with the exception of Sunday/Monday morning am. No one is there. The company is closed, The only activity that shows before this happens is an on-line defrag of the Exchange databases. It starts and finishes, but this is some 20-60 minutes before (I stopped this from the previous week- so now on the last error- zip/zero is happening on the server). and this occurs at the same time 7 days a week, but only on Monday early morning  does the database go down. It does not have to be rebooted, there is no database corruption. Just the service has to be restarted. What gives, any explanation of why this occurs when nothing is going on?

 

Live update is occurring once a day at 12:15 pm. There is no database replication errors showing in SEPM.

The system could not have less activity at this time than almost any other time and beyond the one event error message- there is nothing else.

 

I would appreciate any help or ideas. Attached are all the relevant logs (catalina.out and catalina.err and scm-server-0.log) and while it seems to indicate it can't find something from Java- it only occurs at this one point at this one time and zero problems any other time.

 

I find it depressing to see that Symantec is everyday becoming more like Microsoft and that is NOT a compliment. Here is a funny example. The spell check for these postings find Symantec to be an incorrect spelling- come on and get your act together.

 

Thanks

 

Tom in Dallas

AttachmentSize
logs_13.rar 523.77 KB
.Brian's picture

He's asking if you've opened a case and have a case number that you can PM him so he can look into it further, if you did open a case.

Now, I've looked at the logs and wasn't able to discern much in the short time I had to look at it (I'm by no means a guru with this particular problem) and since this problem is ongoing, it is advised to reach out to support. Have you done that?

Just for future reference, this site is primarily for users, such as yourself, who are looking for solutions to problems. While I consider it one of the best sites in terms of community support, it should not be considered *official* support. Symantec employees do frequent the board quite often and are very helpful but in cases like yours, which I consider to be advanced, you should reach out to support for advanced troubleshooting. People who contribute here do so on their free time. Please don't think support is bad just because you've not received help here. The fact is this is a more difficult (or might be) problem and support is likely the only one that has the chops to handle.

If you have a case number, send it over to Mithun. He is always willing to help and is very good at what he does.

Brian

Please click the "Mark as solution" link at bottom left on the post that best answers your question. This will benefit admins looking for a solution to the same problem.

Mithun Sanghavi's picture

Hello,

Isn't this the same issue which you have stated in a similar Thread for -

https://www-secure.symantec.com/connect/forums/sepm-still-not-able-load-database

Did you create a case for this issue?

If "YES", PM me the case #, and in case "NO", I would request you to please create a case for the same.

Check these Steps below on :

How to create a new case in MySymantec

http://www.symantec.com/business/support/index?page=content&id=TECH58873

You can also contact Symantec Technical Support via Phone numbers given below - 

Regional Support Telephone Numbers:

  • United States: 800-342-0652 (407-357-7600 from outside the United States)
  • Australia: 1300 365510 (+61 2 8220 7111 from outside Australia)
  • United Kingdom: +44 (0) 870 606 6000

Additional contact numbers: http://www.symantec.com/business/support/contact_t...

 

As per the Logs, we see lot of errors - 

catalina.out shows 

 
### The error occurred while executing an update
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: [Sybase][JDBC Driver][SQL Anywhere]Database server not found
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:365)
at com.sun.proxy.$Proxy103.update(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:251)
at com.symantec.sepm.server.module.dashboard.DashboardDAOImpl.updateLatestDefsOnSymcServer(DashboardDAOImpl.java:549)
at com.symantec.sepm.server.module.dashboard.StatusTask.updateStatus(StatusTask.java:80)
at com.symantec.sepm.server.module.dashboard.StatusJob.executeInternal(StatusJob.java:49)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113)
at org.quartz.core.JobRunShell.run(JobRunShell.java:206)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: [Sybase][JDBC Driver][SQL Anywhere]Database server not found
### The error may exist in file [C:\Program Files\Symantec\Symantec Endpoint Protection Manager\tomcat\webapps_ws\sepm\WEB-INF\classes\com\symantec\sepm\server\module\dashboard\dashboard_mapper_ibatis.xml]
### The error may involve com.symantec.sepm.server.module.dashboard.updateGUIPARMSsql
### The error occurred while executing an update
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: [Sybase][JDBC Driver][SQL Anywhere]Database server not found
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:23)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:147)
at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:355)
... 8 more
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: [Sybase][JDBC Driver][SQL Anywhere]Database server not found
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:80)
at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:66)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:279)

 

catalina.out shows

 

ServerId: 0B92F5A4C0A810020133919A6C9BB040
SystemEventId: 1281
EventDesc: Unexpected server error.
MessageId: 1
ErrorCode: 268500992
java.lang.NullPointerException
 
com.sygate.scm.server.util.ServerException: Unexpected server error.
at com.sygate.scm.server.util.ServerLogger.log(ServerLogger.java:399)
 
2013-10-21 08:35:37.444 THREAD 42 SEVERE: SEM: Connect to database failed
java.sql.SQLException: SEM: Connect to database failed
at com.sygate.scm.server.db.util.DatabaseUtilities.getDefaultDatabaseConnection(DatabaseUtilities.java:338)
at com.sygate.scm.server.util.logging.DbLogHandler.publish(DbLogHandler.java:58)
 
2013-10-21 08:35:39.835 THREAD 31 SEVERE: 
java.sql.SQLException: [Sybase][JDBC Driver][SQL Anywhere]Database server not found
at sybase.jdbc4.sqlanywhere.IDriver.makeODBCConnection(Native Method)

 

 

catalina.err shows 

 

Caused by: java.sql.SQLException: [Sybase][JDBC Driver][SQL Anywhere]Database server not found
at sybase.jdbc4.sqlanywhere.IDriver.makeODBCConnection(Native Method)
at sybase.jdbc4.sqlanywhere.IDriver.connect(IDriver.java:772)
 
com.sygate.scm.server.util.ServerException: Unexpected server error.
at com.sygate.scm.server.configmanager.ConfigManagerCache.getDatabaseChecksum(ConfigManagerCache.java:343)
at com.sygate.scm.server.configmanager.ConfigManagerCache.getTopLevelObject(ConfigManagerCache.java:244)

 

When was the last time you restarted the server machine?

Could you please check if there is enough disk space on the machine?

Is there any application running on port 2638?

Could you please provide us the size of "sem5.log" and do you have a backup of the Symantec database?

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

Knottyropes's picture

is exchange doing anything at that time?

mine would start running some tasks at about that time which is heavy disk intensive on creating white space and my IOPS would skyrocket. If your database is on the same drive then that could be an issue

Jazzwineman's picture

Not anything. Exchange on a completely different drive and I have no backups, no online defrag of exchange going on, no updates to Symantec at the time- there is nothing going on. Company closed, no users, most workstations off. Perhaps one or two emails coming into exchange being queued for delivery. That is what is so strange.

 

Thanks

 

Tom

Jazzwineman's picture

I do not have a case number and really don't have the time to go through a number of try this and try that. What you are seeing hapeens one time each week. period. There are no updates occurring at that time . There is nothing going on in the system The machine has been restarted at least 6 times in the last 2 weeks- last time about 6 days ago.. We have 3 drives, each one has more than 250 gigs free. Symantec is on drive C. There is no space problem. The system does an update once a day and that is at 12:15 pm.

netstat -ano show nothing running or listening on port 2638

 

sem5.log is 676 mbs.

 

yes I have a backup of the database.

 

Tom

Mithun Sanghavi's picture

Hello,

Do not worry. By creating the case, I would keep a track of the case personally.

Let me create a Case for you quickly, if you could simply PM me your Full Name, Customer Id # and email address.

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

Swathi Turlapaty's picture

Tom, please send me an IM with your contact information, and I shall have someone from the product team reach out to you soon.

SMLatCST's picture

Given the consistent timing, I'd suggest try disbling the native database re-indexing task in the SEPM console (found under ADMIN -> Servers -> highlight the "localhost" entry representing the Embedded DB -> Hit "Edit Database Properties" -> Disable or reschedule the "Rebuild Indexes" task), and monitor to see if the DB falling-over either stops or follows the new schedule.

Assuming it is the reindexing that is somehow causing the DB to face-plant, I'd suggest looking in the err.log file in the DB folder (typically "\Program Files\Symantec\Symantec Endpoint Protection Manager\db") to look for specific errors.

Jazzwineman's picture

Thank you. While what you are referring to does not appear to be the problem- it did lead me to the problem. At 3am every Monday morning the SEPM is set to to do backups. Apparently that is what is failing. If anyone has had experience with why it does so that would be very helpful in solving or otherwise, I will just not do backups. Now we do backups of the entire system Monday- Saturday- so I could probably restore from there is a problem came up- however I would like to do normal SEPM backups and wonder what I need to d to fix.

 

Thanks

 

Tom in Dallas

SMLatCST's picture

Does a manual backup (using the native SEPM Backup and Restore wizard) cause the DB to crash as well?

As it still sounds like a potential DB issue, can look to see if the err.log file exists and what it says?

Jazzwineman's picture

Here is the error log at the time of the problem:

09/23 03:19:11. *** ERROR *** Assertion failed: 101412 (12.0.1.3554)[sem5]
Page number on page does not match page requested
09/30 03:19:18. *** ERROR *** Assertion failed: 101412 (12.0.1.3554)[sem5]
Page number on page does not match page requested
10/07 03:00:30. *** ERROR *** Assertion failed: 101412 (12.0.1.3554)[sem5]
Page number on page does not match page requested
10/07 09:07:57. *** ERROR *** Assertion failed: 101412 (12.0.1.3554)[sem5]
Page number on page does not match page requested
10/14 03:19:36. *** ERROR *** Assertion failed: 101412 (12.0.1.3554)[sem5]
Page number on page does not match page requested
10/21 03:00:27. *** ERROR *** Assertion failed: 101412 (12.0.1.3554)[sem5]
Page number on page does not match page requested

 

TBB

 

SMLatCST's picture

In that case, does a manual backup using the native tool replicate the DB crash?

There are a number of articles about the "Assertion Failed" error, with solutions varying from a full DR recovery, to just recreating the transaction log:

http://www.symantec.com/docs/TECH145875
http://www.symantec.com/docs/TECH169664
http://www.symantec.com/docs/TECH134782 (this is for the RU3 transaction log recreation steps)

Jazzwineman's picture

fails with same error. I will look at articles and see what they say when I get time later in the day. This just started some 6 weeks ago and there was nothing abnormal going on or different that would have seemed to have been a cause /effect.

 

TBB

SMLatCST's picture

Yeah, it's gonna be difficult to investigate what caused the DB corruption at this point.

I'd personally go with the transaction log recreation first, followed by a DB restore as a second path, with a full DR as a last ditch recovery attempt (possibly even with an empty DB).

Mithun Sanghavi's picture

Hello,

This is what I was afraid of and that was the reason I asked if you have a Database Backup.

I would suggest you to perform a Disaster Recovery.

Symantec Endpoint Protection 12.1: Best Practices for Disaster Recovery with the Symantec Endpoint Protection Manager

http://www.symantec.com/business/support/index?page=content&id=TECH160736

Hope that helps!!

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

Jazzwineman's picture

and what is the purpose of doing a database recovery?  In the event of a total failure on the system we have mutiple backups of everything on the system and I would probably just reinstall from scratch if necessary the SEPM. I am going to look at the the solutions before I would take this much time to s=d this. My real concern was getting rid of the database going offline and I know now the cause.. What other issues might this cause?

 

TBB

Jazzwineman's picture

is there a possibilty that the database needs to be trimmed in size or if I take it offline would it backup?

 

T

Jazzwineman's picture

here is the run by the tool you requested. Please see file attached

 

Tom

AttachmentSize
IBI001-1__2013_10_23__16_35_09.rar 13.95 MB
Cameron_W's picture

I would recommend opening a case with support. You have already done the basic troubleshooting steps and this would need additional research.

To perform this research and most likely collect additional data from your enivornment a open support case would be the most appropriate method for communication back and forth with you during this process.

If I was able to help resolve your issue please mark my post as solution.

Jazzwineman's picture

This has already been done, but thanks anyway.

 

TB

Cameron_W's picture

This has already been done as in there is currently a open support case or have you had a support case opened in the past? Either way you can PM me the case number and I can take a quick look. If it is a currently opened case I would recommend continuing to work with the assigned TSE.

If I was able to help resolve your issue please mark my post as solution.

Jazzwineman's picture

I just give up. No more Symantec for me ever again or for any of my clients.. Been a MCSE for 15 years worked on thousands of computers and no way to even get the file they want to them - prohibited by email size on their end, posted above as an attachment and they cannot get to it as all of us can, pm to them and they can't see it,FTP will not see the password as correct. No one has this much time to waste on this.

 

Tom in Dallas

Mithun Sanghavi's picture

Hello,

I have uploaded the log files to the case. 

Sincerely apologize for the trouble caused.

Regards,

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

Jazzwineman's picture

Look, I have a manual backup to the DB. I can restore from our daily backups anything/. If there is a a way to rebuild a clean database, then tell me what it is. I do NOT NOT have the time going through  endless phone calls, emails  and tests ,when the problem is a database that cannot be backed up. Computer are protested. Computers are being updated. I have used eircar testing  and get what would expect back.Daily reports to me are correct.The only thing I cannot do is backup the database from Symantec native. Stop trying to make this more complicated than it is. I can restart the embedded database anytime I want..

 

If the database has a problem, what are the step to rebuild. I have done a repair install to no avail. There comes apoint where time spent has to be worth it. I have spent some 15 hours on this already. I have to manage some 3500 computerd for more that 16 companies. At this point- it would simply be easier to take it all out and start from scratch- instead of dealing with people that think we have unlimited time to engage in an easter egg hunt.

 

So does anyone have a way to rebuild the database in a simple way???????????????????????????????

 

Tom

SMLatCST's picture

Assuming you've already tried the transaction log recreation I mentioned earlier, then the only way I know if to recreate the DB in its entirety is to run through the DR steps but skipping the DB restore:

http://www.symantec.com/docs/TECH160736

Just make sure you have a copy of the "Server Private Key Backup" folder and that you've noted any/all custom SEP domains, exported policies, and grabbed anything else you think you'll need, before starting.

Jazzwineman's picture

I know you are just doing your job and meaning well. But at this point I have spent more than 10 hours back and forth with you to end up where I already knew that I could do a disaster recovery. What you completely fail to understand - that at my billing rate the client is going to be charged $1650 for what amounts to nothing new. I can rip out all of Symantec, put in a new product for less than what they are going to pay me or rip out Symantec and start over from scratch.

I was trying to find out and still do not have the answer as to what corrupted the database and in what way. Considering this is a fairly frequent occurrence, why have you not developed a repair tool. Even the idiots at Microsoft have repair toos for most of their database formatted programs- even Outlook which has constant problems.

 

Please try and live in the real world. Justify to my client , if you will, how they should be wasting time getting support that gets us no where other than where we were before

SMLatCST's picture
Erm, only those with the orange "Symantec Employee" or dark "Admin" sign beside their names actually work for Symantec.  The rest of us (myself, Brian and knottyropes included) are doing this purely out of the goodness of our hearts (and possibly because we like a puzzle).
 
For my part, I was answering your question on how to clear out the entire DB.  Plus, I did mention on 23rd Oct that it's going to be difficult to find the root cause of the corruption 6-7 weeks after the symptom first manifested itself.  Also, as external parties, we haven't the foggiest what you've already gone through with the Symantec lads.
 
As far as justification goes, I'm afraid it's something only you can decide.  While the community can help out with the odd technical bit of info, it's your call as to if it's worth the time & effort.  All we can really say is "here's the info, good luck".
 
On that note, did you know there was a dbvalidator tool?
http://www.symantec.com/docs/HOWTO39461
Jazzwineman's picture

I appreciate your help, know and have used the tool. My comment was to them, nit you and that is why I dislike the so-called customer support and can count on one hand the number of times I have used it to spend much time accomplishing nothing.  Your help is appreciated and I try t do the same for others.

 

Thanks again

 

T