Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

MSDP pool down - can't get it back up - win 2008 r2 enterprise master / media servers.

Created: 17 Jan 2013 • Updated: 17 Jan 2013 | 2 comments
This issue has been solved. See solution.

Master / Media servers all on Win Server 2008 R2 enterprise. Client is Red Hat 6.2.

Media server has MSDP pool and attached tape drive, and has had no issues lately. Ran into issue with SLP duplication job taking forever (almost 2 days - didnt error out, but won't finish)..so, I cleared out this job using nbstutil command. No pending SLP jobs for this media server now. Also, this media server only has this one client.

 

Anywho, now I can't get my disk pool to come up - tried through the GUI, and using "nbdevconfig -changestate -stype -PureDisk -dp mydiskpoolname -state UP"...this returns successfully, however, re-running nbdevquery right after still shows this disk pool as being in the 'Down' status.

 

Anything else I can try?

Also, if I cancel out a pending duplication job for this SLP, the backup is still there, it's just not duplicating to tape, correct? Or by blowing away the duplication job, the backup is lost as well. Thanks in advnace for any assistance.

 

-Scott

 

 

Comments 2 CommentsJump to latest comment

lovethatcheese's picture

If this helps, this is the log from spoold:

Manager: Initialized in 0.13 seconds.
January 17 20:32:11 INFO [0000000000DF8EA0]: Storage Cache Manager: initialization complete
January 17 20:32:11 INFO [0000000000DF8EA0]: Spooler Cache Manager: initializing
January 17 20:32:11 INFO [0000000000DF8EA0]: Spooler Cache Manager: initialization complete
January 17 20:32:11 INFO [0000000000DF8EA0]: DataStore Manager: initializing
January 17 20:32:11 INFO [0000000000DF8EA0]: Task thread stack size: 0 bytes
January 17 20:32:11 ERR [0000000000DF8EA0]: 25017: FileCopyA: could not open destination file N:\dedupe\data/journal/0.bin
January 17 20:32:11 ERR [0000000000DF8EA0]: 25017: FileMoveA: copy failed
January 17 20:32:11 ERR [0000000000DF8EA0]: 25001: dcOutPlaceUpdateReplay: could not move N:\dedupe\data/6248.bin to N:\dedupe\data/journal/0.bin (object already exists)
January 17 20:32:11 ERR [0000000000DF8EA0]: 25001: _storeSingleDCCompactRecover: dcOutPlaceUpdateReplay (object already exists)
January 17 20:32:11 ERR [0000000000DF8EA0]: 25001: _storeInit: failed to recover from single container compaction
January 17 20:32:11 ERR [0000000000DF8EA0]: 25001: Could not initialize DataStore Manager
January 17 20:32:12 INFO [0000000000DF8EA0]: WSRequestExt: submitting &request=4&login=agent_3_9549&passwd=***************************January 17 20:32:19 INFO: set entire process max log size to 0
January 17 20:32:19 INFO: set entire process max log size to 0
January 17 20:32:19 INFO: _getmaxstdio() before = 512
January 17 20:32:19 INFO: _getmaxstdio() after  = 2048
January 17 20:32:19 INFO: Startup: occurred at Thu Jan 17 20:32:19 2013
January 17 20:32:19 INFO: Database Manager: initializing
January 17 20:32:19 INFO: Database Manager: initialization complete
January 17 20:32:19 INFO: Database Manager: closing storage database connection
January 17 20:32:19 INFO: Database Manager: shutdown
January 17 20:32:19 INFO: Startup: loading configuration from N:\dedupe\etc\puredisk\contentrouter.cfg
January 17 20:32:19 INFO: 25022: NetCheckIPRange: invalid NULL or empty parameter
January 17 20:32:19 INFO: N:\dedupe\data: extended attribute support enabled
January 17 20:32:19 INFO: Manager thread stack size: 0 bytes
January 17 20:32:19 INFO: Task thread stack size: 0 bytes
January 17 20:32:19 INFO: Max DO size: 335544320 bytes
January 17 20:32:19 INFO: Auto-sized value for N:\dedupe\etc\puredisk\contentrouter.cfg: section Cache, entry Bits to 22 bits
January 17 20:32:19 INFO [0000000000CE8EA0]: set entire process max log size to 10000000
January 17 20:32:19 INFO [0000000000CE8EA0]: set entire process max log size to 10000000
January 17 20:32:19 INFO [0000000000CE8EA0]: Startup: occurred at Thu Jan 17 20:32:19 2013
January 17 20:32:19 INFO [0000000000CE8EA0]: Successfully loaded configuration from N:\dedupe\etc\puredisk\contentrouter.cfg
January 17 20:32:19 INFO [0000000000CE8EA0]: Startup: Symantec PureDisk Content Router Version 7.0003.0012.0915.
January 17 20:32:19 INFO [0000000000CE8EA0]: Startup: using Symantec: libdct 6.0.0.0, July 7, 2004
January 17 20:32:19 INFO [0000000000CE8EA0]: Startup: using Symantec PureDisk: libcr 6.1.0.0, December 13, 2006
January 17 20:32:19 INFO [0000000000CE8EA0]: Memory Manager: initializing
January 17 20:32:19 INFO [0000000000CE8EA0]: Memory Manager: initialization complete
January 17 20:32:19 INFO [0000000000CE8EA0]: Authorization Manager: initializing
January 17 20:32:19 INFO [0000000000CE8EA0]: Authorization Manager: initialization complete
January 17 20:32:19 INFO [0000000000CE8EA0]: Database Manager: initializing
January 17 20:32:19 INFO [0000000000CE8EA0]: Database Manager: initialization complete
January 17 20:32:19 INFO [0000000000CE8EA0]: Storage Cache Manager: initializing
January 17 20:32:19 INFO [0000000000CE8EA0]: Database Manager: initializing
January 17 20:32:19 INFO [0000000000CE8EA0]: Database Manager: initialization complete
January 17 20:32:19 INFO [0000000000CE8EA0]: Database Manager: initializing
January 17 20:32:19 INFO [0000000000CE8EA0]: Database Manager: initialization complete
January 17 20:32:19 INFO [0000000000CE8EA0]: Storage Cache Manager: Initialized in 0.13 seconds.
January 17 20:32:19 INFO [0000000000CE8EA0]: Storage Cache Manager: initialization complete
January 17 20:32:19 INFO [0000000000CE8EA0]: Spooler Cache Manager: initializing
January 17 20:32:19 INFO [0000000000CE8EA0]: Spooler Cache Manager: initialization complete
January 17 20:32:19 INFO [0000000000CE8EA0]: DataStore Manager: initializing
January 17 20:32:19 INFO [0000000000CE8EA0]: Task thread stack size: 0 bytes
January 17 20:32:20 ERR [0000000000CE8EA0]: 25017: FileCopyA: could not open destination file N:\dedupe\data/journal/0.bin
January 17 20:32:20 ERR [0000000000CE8EA0]: 25017: FileMoveA: copy failed
January 17 20:32:20 ERR [0000000000CE8EA0]: 25001: dcOutPlaceUpdateReplay: could not move N:\dedupe\data/6248.bin to N:\dedupe\data/journal/0.bin (object already exists)
January 17 20:32:20 ERR [0000000000CE8EA0]: 25001: _storeSingleDCCompactRecover: dcOutPlaceUpdateReplay (object already exists)
January 17 20:32:20 ERR [0000000000CE8EA0]: 25001: _storeInit: failed to recover from single container compaction
January 17 20:32:20 ERR [0000000000CE8EA0]: 25001: Could not initialize DataStore Manager
January 17 20:32:20 INFO [0000000000CE8EA0]: WSRequestExt: submitting &request=4&login=agent_3_9549&passwd=********************************&action=newevent&data=EVENT%7Bversion%3A1%3Btype%3A1%3Bid%3A0%3Bdate%3A1358472732%3B%7BLEGACYEVENT%7Bpayload%3Asev%3D1%3Btype%3D1017%3Bmsg%3DFailed%20to%20Activate%20the%20Store%20Manager.%0A%0APlease%20check%20the%20server%20log%20for%20additional%20information%20and%20probable%0Acause%20of%20this%20error.%0A%0AThe%20application%20has%20been%20terminated.%3B%7D%7D
January 17 20:32:12 INFO [0000000000DF8EA0]: sessionStartAgent: Server is Version 7.0003.0012.091, Protocol Version 6.6.1
January 17 20:32:12 INFO [0000000000DF8EA0]: WSRequestExt: submitting &request=4&login=agent_3_9549&passwd=********************************&action=newevent&data=EVENT%7Bversion%3A1%3Btype%3A4%3Bid%3A0%3Bdate%3A1358472732%3B%7BSERVERSTATUSEVENT%7Bstate%3A1%3Bserver%3ASymantec%20PureDisk%20Content%20Router%3B%7D%7D
January 17 20:32:12 ERR [0000000000DF8EA0]: 26016: Store Manager: Activate failure.
*****&action=newevent&data=EVENT%7Bversion%3A1%3Btype%3A1%3Bid%3A0%3Bdate%3A1358472740%3B%7BLEGACYEVENT%7Bpayload%3Asev%3D1%3Btype%3D1017%3Bmsg%3DFailed%20to%20Activate%20the%20Store%20Manager.%0A%0APlease%20check%20the%20server%20log%20for%20additional%20information%20and%20probable%0Acause%20of%20this%20error.%0A%0AThe%20application%20has%20been%20terminated.%3B%7D%7D
January 17 20:32:20 INFO [0000000000CE8EA0]: sessionStartAgent: Server is Version 7.0003.0012.091, Protocol Version 6.6.1
January 17 20:32:20 INFO [0000000000CE8EA0]: WSRequestExt: submitting &request=4&login=agent_3_9549&passwd=********************************&action=newevent&data=EVENT%7Bversion%3A1%3Btype%3A4%3Bid%3A0%3Bdate%3A1358472740%3B%7BSERVERSTATUSEVENT%7Bstate%3A1%3Bserver%3ASymantec%20PureDisk%20Content%20Router%3B%7D%7D
January 17 20:32:20 ERR [0000000000CE8EA0]: 26016: Store Manager: Activate failure.
 

_____

Current Environment - NB 7.5.0.4 on Master / Media Servers. 7+ and above on clients (Red Hat).

OS - Windows Server 2008 R2 Enterprise on all

All media servers have MSDP / attached tape drive / enabled SLP's

___

lovethatcheese's picture

uly 28 09:46:10 ERR [00000000009299E0]: 25001: dcOutPlaceUpdateReplay: could not move F:\Storage\data/4866.bin to F:\Storage\data/journal/0.bin (object already exists)

July 28 09:46:10 ERR [00000000009299E0]: 25001: _storeSingleDCCompactRecover: dcOutPlaceUpdateReplay (object already exists)
July 28 09:46:10 ERR [00000000009299E0]: 25001: _storeInit: failed to recover from single container compaction
July 28 09:46:10 ERR [00000000009299E0]: 25001: Could not initialize DataStore Manager

 

Environment

 Windows 2008 R2 

Netbackup 7.5

Solution

Move F:\Storage\data/journal/0.bin file to another space or rename it.

1,Stope Netbackup service.

2,Rename F:\Storage\data/journal/0.bin to F:\Storage\data/journal/0.bin.old

3,Start Netbackup servcie.

 

_____

Current Environment - NB 7.5.0.4 on Master / Media Servers. 7+ and above on clients (Red Hat).

OS - Windows Server 2008 R2 Enterprise on all

All media servers have MSDP / attached tape drive / enabled SLP's

___

SOLUTION