Video Screencast Help
Symantec Appoints Michael A. Brown CEO. Learn more.

SQL job fails with error code 2

Created: 02 Aug 2013 • Updated: 12 Aug 2013 | 14 comments
This issue has been solved. See solution.

I'm very new to Netbackup, well backups in general as I've taken over this system when employees left. I'm getting an error code 2 on SQL jobs and it seems specific to the TIDB portion of the backup. Any insight would be greatly appreciated. See below for more detail.

 

8/2/2013 11:18:16 AM - Info nbjm(pid=6340) starting backup job (jobid=20507) for client SFSSQLPRD, policy SQL_SFSSQLPRD_TLOGS, schedule TLOGS 
8/2/2013 11:18:16 AM - Info nbjm(pid=6340) requesting MEDIA_SERVER_WITH_ATTRIBUTES resources from RB for backup job (jobid=20507, request id:{BDED6CBC-A625-4E20-941E-C0C60E7FCE43}) 
8/2/2013 11:18:16 AM - requesting resource sbckprd02-hcart-robot-tld-1
8/2/2013 11:18:16 AM - requesting resource sbckprd02.ahcccs.state.az.us.NBU_CLIENT.MAXJOBS.SFSSQLPRD
8/2/2013 11:18:16 AM - requesting resource sbckprd02.ahcccs.state.az.us.NBU_POLICY.MAXJOBS.SQL_SFSSQLPRD_TLOGS
8/2/2013 11:18:16 AM - granted resource sbckprd02.ahcccs.state.az.us.NBU_CLIENT.MAXJOBS.SFSSQLPRD
8/2/2013 11:18:16 AM - granted resource sbckprd02.ahcccs.state.az.us.NBU_POLICY.MAXJOBS.SQL_SFSSQLPRD_TLOGS
8/2/2013 11:18:16 AM - granted resource sbckprd02-hcart-robot-tld-1
8/2/2013 11:18:16 AM - estimated 0 Kbytes needed
8/2/2013 11:18:16 AM - Info nbjm(pid=6340) started backup (backupid=SFSSQLPRD_1375467496) job for client SFSSQLPRD, policy SQL_SFSSQLPRD_TLOGS, schedule TLOGS on storage unit sbckprd02-hcart-robot-tld-1
8/2/2013 11:18:16 AM - started process bpbrm (9068)
8/2/2013 11:18:18 AM - Info bpbrm(pid=9068) SFSSQLPRD is the host to backup data from    
8/2/2013 11:18:18 AM - Info bpbrm(pid=9068) reading file list from client       
8/2/2013 11:18:18 AM - connecting
8/2/2013 11:18:20 AM - Info bpbrm(pid=9068) starting bphdb on client        
8/2/2013 11:18:20 AM - connected; connect time: 00:00:02
8/2/2013 11:18:21 AM - Info bphdb(pid=38084) Backup started          
8/2/2013 11:18:23 AM - Info dbclient(pid=39276) INF - BACKUP STARTED USING       
8/2/2013 11:18:23 AM - Info dbclient(pid=39276) Microsoft SQL Server 2012 - 11.0.2100.60 (X64)     
8/2/2013 11:18:23 AM - Info dbclient(pid=39276) Feb 10 2012 19:39:15        
8/2/2013 11:18:23 AM - Info dbclient(pid=39276) Copyright (c) Microsoft Corporation        
8/2/2013 11:18:23 AM - Info dbclient(pid=39276) Standard Edition (64-bit) on Windows NT 6.1 <X64> (Build 7601: Service Pack 1)
8/2/2013 11:18:23 AM - Info dbclient(pid=39276) Batch = sfssqlprd_tidb_tlog.bch, Op# = 1      
8/2/2013 11:18:23 AM - Info dbclient(pid=39276) INF - Using backup image SFSSQLPRD01.MSSQL7.SFSSQLPRD.trx.TIDB.~.7.001of001.20130802111822..C      
8/2/2013 11:18:23 AM - Info dbclient(pid=39276) INF - backup log "TIDB" to VIRTUAL_DEVICE='VNBU0-39276-38400-1375467503' with  stats = 10, blocksize = 65536, maxtransfersize = 4194304, buffercount = 2
8/2/2013 11:18:24 AM - Info dbclient(pid=39276) INF - Number of stripes: 1, Number of buffers per stripe 2.
8/2/2013 11:18:24 AM - Info dbclient(pid=39276) INF - Created VDI object for SQL Server instance <SFSSQLPRD>. Connection timeout is <300> seconds.
8/2/2013 11:23:26 AM - Info dbclient(pid=39276) ERR - Error in GetConfiguration: 0x80770003.      
8/2/2013 11:23:26 AM - Info dbclient(pid=39276)     CONTINUATION: - The api was waiting and the timeout interval had elapsed.
8/2/2013 11:23:31 AM - Info dbclient(pid=39276) ERR - Error in VDS->Close: 0x80770004.      
8/2/2013 11:23:31 AM - Info dbclient(pid=39276)     CONTINUATION: - An abort request is preventing anything except termination actions.
8/2/2013 11:23:32 AM - Info dbclient(pid=39276) INF - OPERATION #1 of batch sfssqlprd_tidb_tlog.bch FAILED with STATUS 1 (0 is normal). Elapsed time = 309(309) seconds.
8/2/2013 11:23:34 AM - Info dbclient(pid=39276) INF - Results of executing <sfssqlprd_tidb_tlog.bch>:      
8/2/2013 11:23:34 AM - Info dbclient(pid=39276) <0> operations succeeded. <1> operations failed.      
8/2/2013 11:23:34 AM - Info dbclient(pid=39276) INF - The following object(s) were not backed up successfully.  
8/2/2013 11:23:34 AM - Info dbclient(pid=39276) INF - TIDB         
8/2/2013 11:23:34 AM - Error bpbrm(pid=9068) from client SFSSQLPRD: ERR - command failed: none of the requested files were backed up (2)
8/2/2013 11:23:34 AM - Error bpbrm(pid=9068) from client SFSSQLPRD: ERR - bphdb exit status = 2: none of the requested files were backed up
8/2/2013 11:23:35 AM - Error bpbrm(pid=9068) cannot send mail to        
8/2/2013 11:23:36 AM - Info bphdb(pid=38084) done. status: 2: none of the requested files were backed up 
8/2/2013 11:23:36 AM - end writing
none of the requested files were backed up(2)

Operating Systems:

Comments 14 CommentsJump to latest comment

RonCaplinger's picture

To me, it looks like you are just hitting a client timeout value:

8/2/2013 11:18:24 AM - Info dbclient(pid=39276) INF - Created VDI object for SQL Server instance <SFSSQLPRD>. Connection timeout is <300> seconds.
8/2/2013 11:23:26 AM - Info dbclient(pid=39276) ERR - Error in GetConfiguration: 0x80770003.      
8/2/2013 11:23:26 AM - Info dbclient(pid=39276)     CONTINUATION: - The api was waiting and the timeout interval had elapsed.
8/2/2013 11:23:31 AM - Info dbclient(pid=39276) ERR - Error in VDS->Close: 0x80770004.      
8/2/2013 11:23:31 AM - Info dbclient(pid=39276)     CONTINUATION: - An abort request is preventing anything except termination actions.
8/2/2013 11:23:32 AM - Info dbclient(pid=39276) INF - OPERATION #1 of batch sfssqlprd_tidb_tlog.bch FAILED with STATUS 1 (0 is normal). Elapsed time = 309(309) seconds.

The SQL server is taking 309 seconds to respond, but the timeout is set for 300 seconds, which is a standard default time for many timeouts.  Check your NetBackup Management -> Host Properties -> Master Servers properties for the "Timeouts" section.  Make sure the Client Connect timeout and Client Read Timeout are set higher than 300 seconds (600?), then also check the client's properties for the same Timeouts section and make sure the Client Read Timeout is also set to more than 300 seconds.

SOLUTION
Vince L.'s picture

Thanks so much for the info! I did as you suggested and tried to rerun the job, however, I get the same error and I noticed the log still shows timeout is 300s. But if I go back into the master server properties, as well as the client properties, it shows the new 600s. I stopped and restarted all services and the error continues. Is there something else I'm missing for the new setting to take effect? Thank you!

Marianne's picture

Do you have a single master/media server or separate media servers?

Timeouts go on the Media server that is backing up this client. Host Properties -> Media Servers -> Timeouts. 

If increasing timeouts does not help, we need SQL VDI and Errorlog to see what is happening during this phase:

Info dbclient(pid=39276) INF - Created VDI object for SQL Server instance <SFSSQLPRD>. Connection timeout is <300> seconds.

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Omar Villa's picture

Rise CLIENT_CONNECT_TIMEOUT and CLIENT_READ_TIMEOUT to a big value like 3600 or 7200 I have see many times that SQL takes long time to connect or respond to read requests, this goes on the client side, not on the master or the media, you can do it on the master server but will be a global value for every client and you dont want to do that.

 

Hope this helps.

Regards.

Omar Villa

Netbackup Expert

Twiter: @omarvillaNBU

 

Vince L.'s picture

Hi All, thanks so much for the replies! I changed the CLIENT_READ_TIMEOUT on the client to 600s and am still not seeing a change. We have a seperate Media server as well and I changed it there also.I initially changed both CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT on the master server but didn't see a change when the job tries to run. I try to rerun the job and the logs show a connection timeout of 300s. Here's an updated log. Thank you all so much for your help.

8/3/2013 2:23:56 PM - Info dbclient(pid=42276) INF - Created VDI object for SQL Server instance <SFSSQLPRD>. Connection timeout is <300> seconds.
8/3/2013 2:28:59 PM - Info dbclient(pid=42276) ERR - Error in GetConfiguration: 0x80770003.      
8/3/2013 2:28:59 PM - Info dbclient(pid=42276)     CONTINUATION: - The api was waiting and the timeout interval had elapsed.
8/3/2013 2:29:04 PM - Info dbclient(pid=42276) ERR - Error in VDS->Close: 0x80770004.      
8/3/2013 2:29:05 PM - Info dbclient(pid=42276)     CONTINUATION: - An abort request is preventing anything except termination actions.
8/3/2013 2:29:05 PM - Info dbclient(pid=42276) INF - OPERATION #1 of batch sfssqlprd_tidb_tlog.bch FAILED with STATUS 1 (0 is normal). Elapsed time = 311(311) seconds.
8/3/2013 2:29:07 PM - Info dbclient(pid=42276) INF - Results of executing <sfssqlprd_tidb_tlog.bch>:      
8/3/2013 2:29:07 PM - Info dbclient(pid=42276) <0> operations succeeded. <1> operations failed.      
8/3/2013 2:29:07 PM - Info dbclient(pid=42276) INF - The following object(s) were not backed up successfully.  
8/3/2013 2:29:07 PM - Info dbclient(pid=42276) INF - TIDB         
8/3/2013 2:29:07 PM - Error bpbrm(pid=13816) from client SFSSQLPRD: ERR - command failed: none of the requested files were backed up (2)
8/3/2013 2:29:07 PM - Error bpbrm(pid=13816) from client SFSSQLPRD: ERR - bphdb exit status = 2: none of the requested files were backed up
8/3/2013 2:29:08 PM - Error bpbrm(pid=13816) cannot send mail to    
8/3/2013 2:29:10 PM - Info bphdb(pid=42712) done. status: 2: none of the requested files were backed up 
8/3/2013 2:29:10 PM - end writing
none of the requested files were backed up(2)

RonCaplinger's picture

Following Marianne's advice: check the Media Server "Timeouts" values.  Are any of them still set to 300?  If so, increase all of them to 600, then retry and see if that fixes it (or at least changes "Connection timeout is <300> seconds" message!).

Michael G Andersen's picture

Could also be caused by permissions on the SQL server.

I would look in the dbclient log on the SQL server for indication of this

We have discovered that the netbackup service user (usually system) needs sysadmin on the SQL servers, which is not defaukt anymore

 

Marianne's picture

Please add the following to the SQL backup script to match Client Connect and Client Read timeouts on the MEDIA SERVER:

VDITIMEOUTSECONDS 600

If you change timeouts on Media Server to a higher value (e.g. 1800), adjust above timeout in the script accordingly.

Hope this helps.
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION
Vince L.'s picture

Thank you all so much for your help!

Marianne, where would I find that script? Do you think there's a bigger problem since it is not working by increasing timeouts on the master/media server? Many or our SQL jobs fail in the same way, with the timeout.

Will Restore's picture

the script is the SQL batch file (.bch) normally located in client folder

C:\Program Files\VERITAS\NetBackup\DbExt\Mssql\

Will Restore -- where there is a Will there is a way

Vince L.'s picture

Thanks so much wr! I appreciate the info! I'll add the text as Marianne suggested and see if that helps.

Vince L.'s picture

This is getting stranger. I was able to make the change to the script as suggested and now it shows correctly a timeout of 600s. However, when I rerun the job, it still fails with a wait time of 605s, just over the limit as before.

8/8/2013 3:19:47 PM - Info dbclient(pid=3664) INF - Created VDI object for SQL Server instance <SSQLPRD05>. Connection timeout is <600> seconds.
8/8/2013 3:29:48 PM - Info dbclient(pid=3664) ERR - Error in GetConfiguration: 0x80770003.      
8/8/2013 3:29:48 PM - Info dbclient(pid=3664)     CONTINUATION: - The api was waiting and the timeout interval had elapsed.
8/8/2013 3:29:51 PM - Info dbclient(pid=3664) ERR - Error in VDS->Close: 0x80770004.      
8/8/2013 3:29:51 PM - Info dbclient(pid=3664)     CONTINUATION: - An abort request is preventing anything except termination actions.
8/8/2013 3:29:51 PM - Info dbclient(pid=3664) INF - OPERATION #1 of batch ssqlprd05_diff.bch FAILED with STATUS 1 (0 is normal). Elapsed time = 605(605) seconds.

Will Restore's picture

from an older thread:

 check your task manager whether there is any leftover dbbackex.exe, kill them if there are, and re-run backup.

https://www-secure.symantec.com/connect/forums/having-problem-mssql-agent-backup#comment-7406511

 

otherwise, you can try higher timeout value such as 1800  Marianne recommended above.

Will Restore -- where there is a Will there is a way

SOLUTION
Vince L.'s picture

Finally! Thank you all so much! I found a hung dbbackex.exe process (had to select show all users). that along with the increase in TIMEOUT seems to have fixed it. I just ran my first successful backup of that server in over a week. Thank you all so much! This backup newbie is very appreciative!