Video Screencast Help

BE 2012 Storage offline

Created: 30 May 2012 | 47 comments
Ralph Meyer's picture

 Hello All

Perhaps someone can help me. We have a Backup Exec Installation with the following problem.

Environment:

Backup Server:  HP DL380 G7 (Windows 2008 R2)  / local Storage (140 GB OS, 780GB LokalBackup Storage)

Backup Software: Backup Exec 2012

Storage : HP Lefthand P4000 Multisite Cluster (iSCSI Connection)  (StorageLUN 1.95TB)

We configured a LocalStorage of 780GB for special VM’s, and configured a 1.95TB StorageLUN on the HP Lefthand for the classical backup. Now we have the problem, that at the beginning always at 4AM the LUN (StorageLUN) goes offline. At 4AM also the Backup Exec DB Maintenance is running. We have also tired to change the DB Maintenance Time to 8AM, and after that change the LUN goes offline at 8AM. Also if we disable the DB Maintenance the LUN will be offline a couple seconds after 8AM. If we restart the BE Services, the LUN will goes online.

For BE 2012 is the LUN offline, but on the Microsoft level we are able to write files into the LUN.

Many thanks in advance for your help.

 

Kind regards Chris

Comments 47 CommentsJump to latest comment

CraigV's picture

Hi,

 

I think this has been posted before on Connect so maybe do a search for that too.

In any case, just make sure that BE 2012 is fully patched...since its release, there have been a couple of new hotfixes that Symantec released. Get those and update the media server and any other remote servers.

Check the P4000 as well and make sure that there are no errors being registered there that could point to a hardware issue.

You can also consider removing the iSCSI LUNs, and then adding them back, and then doing the same with the B2Ds (DON'T delete them totally or you will lose your data!).

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Kiran Bandi's picture

Recent Discussion on the same issue: https://www-secure.symantec.com/connect/forums/database-management-job-brings-disk-folders-offline

Sush...'s picture

The issue that you are facing seems to be a Known issue with BE 2012. This issue has been escalated further. If you have a support contract then I would suggest you to open a case with Symantec.

 I will provide the detail links to the technotes later.

Thanks,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

Colin Weaver's picture

We are aware of an issue where B2D Locations on SAN volumes (particularly where multipath might be involved) go offline during DB maintenance or service restart. I suspect your condition might be related. However as it is early in the investigations and your problem is not exactly the same I would suggest you log a formal support case for your condition.

With regards the B2D issue, for some reason the TECH article for this issue is not yet public

 

Ralph Meyer's picture

Hello All

Thanks for the prompt answers. Yes, I already saw the other posts. I have also tried to build a new LUN, to install Backup Exec Server from scratch, to use different methods for the ISCSI Connection (HP MPIO, MS MPIO) and so on.  I saw also, that some are writing jobs, which restarts the BE services after the DB maintenance. But that’s not a solution for me.

I will open a case and post the solution if there is one.

 

Kind regards Chris

TTT's picture

I'm also having B2D volumes go offline, using Compellent SAN w/FC and native Windows MPIO.  I can bring the B2D back online inside BE without a problem- but it'll go offline the next day.  Is my problem the same as the one you describe?

Sush...'s picture

Yes as per the description that you provided it seems that you are facing the same issue. If you have a support contract with Symantec then I would suggest that you open a case with Symantec as we do have some positive development on this issue.

 

Thanks,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

MDCnet's picture

I am experiencing the same issue with iSCSI volumes. Even disabling the database maintenance makes no difference. Symantec when will a hotfix be released to resolve this problem? Note I have applied BE 2012 SP1 already

Sush...'s picture

The current workaround is to run database maintenance when no jobs are running.

 

Thanks,

-Sush...

 

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

HNeumann's picture

This is a joke, isn´t it? What a workaround. :-(

I hope this is fixed as soon as possible...

teiva-boy's picture

If the DB Maintenance is run and completes, does the volume come back online?  

If not why not shedule a task mid-day or in the morning to bring it back online?  Seems like a pretty easy workaround until a hotfix can be made.  Saves from havivng to log back in each day to fix it.

Change maintenance to working hours, create batch file to bring volume back online as needed shortly after the maintenance task runs.  (Assuming you know how long it typically takes, so it can be timed accordingly)

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

Chris Morf's picture

Hello All

It's me again. Now with my own account. I started this case with my college’s account. As recommended I have opened a case to the Symantec support.  We had a Webex session and the solution was to delete all the ISCSI Volumes and to connect a new one. Reboot the Backup Exec Server and defined a short DB Maintenance Tests 5 minutes later. And it worked fine. The next test was to set the DB Maintenance to run at 4 AM. One the next 2 days the Volume was still online.
After those test we installed SP1 and created five B2D jobs, witch wrote the backup to the new LUN.
On the next day the Storage was offline after the DB Maintenance at 4 AM.
OK, as we have read, that there are some problems with SP1 we uninstalled the SP1 and rebooted the Backup Exec Server. In about 11 hours I will see if the Storage will be online or offline after the DB Maintenance.
If it will be again offline, then the last test will be to remove again all the LUN’s and reconnect a new LUN. Test if the Storage will stay online after DB Maintenance and then to create only one B2D Backup job and test again if the Storage stays online after DB Maintenance.

If I will have still offline problems, then we will reinstall Backup Exec 2010 R3 and change the product after the maintenance contracts will be finished.

Kind regards Chris
 

Chris Morf's picture

Hello Sush

As already mention, we opened a support case and also had a WebEx session with the Symantec support team.  We deleted all the LUN’s, also on the storage. After that we created a new LUN an connected the new LUN to the Backup Server and added the LUN to Backup Exec 2012. 

Finally we run once the DB maintenance process and the LUN was still online.  All that work we did together with the Symantec support team. On the next day, the LUN was still online.

Then we started to configure some B2D jobs, and one the next day we were running into the same old problems, that after the DB maintenance the LUN goes and stays offline.

Sorry, but in my opinion there is still no fix available.

Regards Chris

Sush...'s picture

You may ask to re-open this case as the issue has immediately re-occured. We do have progress on the solution for this issue as currently is in testing phase.

SP1a does not have solution for this issue. This issue is different than what the following poster has said about the service crash.

 

Thanks,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

MDCnet's picture

SP1a has been released which appears to address the problem

http://www.symantec.com/business/support/index?page=content&id=TECH189907

- Backup Exec server service crashes 4 AM after installing Backup Exec 2012 Service Pack 1 (TECH190472)

I am applying the update now

fwolf's picture

Hi MDCnet,

did the SP1a solve your problem?

MDCnet's picture

Hi fwolf,

No, as Sush mentioned SP1a doesnt include the fix for the storage offline issue during database maintenance.

Regards

Mark

HNeumann's picture

Hi,

MDCnet is right, my B2D Folder was offline at 11.00 AM :-(

Regards

 

HNeumann

Sush...'s picture

Thats right. SP1a does not have the code to fix this issue. But we so have development on this issue and request you open a case so that the (Private) Fix can be provided.

 

Thanks,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

David Willoughby's picture

Is this private fix already available? I already have an open ticket for this exact same issue.

fwolf's picture

Hi David,

I was told they're still working on a fix. My supportcase is still open and I was told that I will be informed if they got a solution for this problem.

Regards

Florian

Javed Nawaz's picture

Hi Ralph do you have any update on this case ?

Chris Morf's picture

Hello Javed

Sorry. Still the same problem. We are restarting every day after the DB Maintenance all backup exec services

and we are in the process to evaluate an other backup solution.

Kind regards Chris

fwolf's picture

Hello,

my problem was solved by symantec support with a not public hotfix. I was told they are working on a public hotfix. A date for the release doesn't exist right now.

My workaround was to schedule the database maintenance at a time slot when I was at work. I always put my B2D folders online again when database maintenance was finished.

Florian

HNeumann's picture

The workaround you explained is ok, but what is on weekends :-( I had to dial in to put the folder back online, too...

fwolf's picture

I scheduled all weekend backups before the database maintenance on saturday.

teiva-boy's picture

Why not a scheduled Windows task to restart BE services or bring the folders online via BECMD (or whatever the CLI is)?  

 

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

fwolf's picture

In my case restarting the BE services could also cause the B2D folders to go offline and until now I didn't knew that it's possible to bring them online by CLI.

geldridge's picture

Was the "private fix" ever made public?  I've got this on 2 of my media servers and it is driving me nuts.

Sush...'s picture

Hello there,

    The fix is still Private and not made public. There is no ETA about the fix release for public at this moment. I would request you to open a Support case with Symantec to get the private fix.

 

Thanks,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

geldridge's picture

Thanks Sush, I have opened a support case.

Chris Morf's picture

Hello Geldridge

I am really interested if you will get a solution from symantec (Public or Private).
I have still the same problem since almost 4 month. My solution that I got, is to restart
every day all the BE services.

Kind regards Chris

 

fwolf's picture

Hi Chris,

please open a new request at symantec support. In my case I had to deliver some Logs and after a wile I got the not public hotfix.

The problem returns if I make any updates of Backup Exec. After updates I always have to install the Fix again.

 

Regards Florian

HNeumann's picture

Hi Chris,

fwolf is right, symantec provides a private hotfix. After patching the system you always have to apply the hotfix again. In my case symantec delivered a new pvlsvr.exe, which i have to copy to the backupe exec folder...
I wish symantec would integrate the new .exe in SP2.

 

Regards HNeumann

geldridge's picture

I was not successfull with getting support to give me the private hotfix.  It really is about time this fix was released.  Backups should be a very reliable and low maintenance activity for admins.  Not something that needs daily babysitting!  If I can't trust my software to backup and adequately cover us in a disaster I will look elsewhere, budgeting time is coming up. 

Please, don't ask me to open another case as I'm tired of all the going back and forth with support, release the hotfix already.

GeoNetGuy's picture

Any updates on this issue? I have similar with BE2012 attaching HP P4500 LUNS for B2D operations. B2D LUNS go off-line. Reschedule DB maintenace and then disable DB maint with no positive results.

If any one has resolved this problem I would be appreciative of an update that may help me solve this ongoing maintenance nighmare.

fwolf's picture

Hi GeoNetGuy,

my system is up to date but after the last hotfix installation the problem still occured. I had to use the private hotfix provided by symantec support again. So this problem isn't solved until now.

Regards
Florian

 

TTT's picture

Thanks fwolf, I'm about to set up B2D2T (currently only doing B2T with BE2012, our old BE11d is still handling B2D) on our FC SAN (Compellent Series 40).

I'm surprised this issue still exists- I ran across it doing testing back in April 2012 (same hardware, same SAN).

Good to know there's a super secret private hotfix that can solve it.  I'll let you guys know what happens to my setup.

Gurvinder Rait's picture

Please refer this TECHNOTE http://www.symantec.com/business/support/index?pag.... Hotfix 199190 was released which had the fix for this Issue but because of a beserver crash which was observed on a few setups we pulled it back. A new version of the DDI fix will be released would have the fixes which are part of HF199190 plus the code changes which caused the beserver  crash with HF199190.

Workaround can be followed in the meantime as suggested in the above article

GeoNetGuy's picture

Thanks everyone.

The workaround in the noted article does not work in my instance. Neither disabling maintenance or moving it to another time slot worked to resolve the off line B2D folders. BE2012 is up date, all patches installed, live update says current.

Symantec support replied to my ticket with the following but the hot fix is not available. Is there any way for me to get the private hot fix?

.......

Are you experiencing the same issue as shown in http://www.symantec.com/business/support/index?pag...?

Thanks & Regards,
Ajith S
Technical Support Analyst
Backup Exec Technical Support, Symantec Corporation
 

 

GeoNetGuy's picture

Hello,

The workaround doesn't work for me.

Symantec points me to the same article:

Are you experiencing the same issue as shown in http://www.symantec.com/business/support/index?page=content&id=TECH189831?

Anyone know how I can get the super secret private hot fix?

Colin Weaver's picture

The public fix (replacement for withdrawn Hotfix 199190) should be available in next few days (barring any late identified problems anyway)

TTT's picture

Great!  Will this come through via LiveUpdate to BE 2012 SP1A or do we have to install it manually?

Gurvinder Rait's picture

This would be available on Live Update after 25th feb (may change, could be earlier or later)

Sush...'s picture

Just an update that this is now available on Live Updates too .

 

Thanks,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

GeoNetGuy's picture

Seems to be working in regard to fixing LUNs falling off line. Fingers crossed.