Hotfix Bundle NB_PDE_6.6.1.2_EEB20-rollup2.tar provides fixes to Symantec NetBackup (tm) PureDisk

Article:TECH162680  |  Created: 2011-06-19  |  Updated: 2012-01-12  |  Article URL
Article Type
Technical Solution


 Name: NB_PDE_6.6.1.2_EEB20-ET2399563_rollup2
Date: July 13, 2011

This Engineering Binary provides features and fixes to the Symantec NetBackup
PureDisk Remote Office Edition 6.6 software.

WARNING: When you apply this Engineering Binary, all PureDisk services stop
         and start.  This can cause running jobs to abort.



Install PureDisk before you install this Engineering Binary.


Health checker:
This Engineering Binary contains the PureDisk Health Checker v1.90 under /opt/pdconfigure/scripts/support/,
which now support multi-node environments with '-m' flag. For other options, execute:
/opt/pdag/bin/php /opt/pdconfigure/scripts/support/health_checker.php --help

The health checker is now automatically called by PDgetlogs and saved as health_check.log in the PDgetlogs tarball.

General fixes:
ET2240413 - MBE-CLI_MBGarbageCollection fail/error 'Query failed: ERROR: relation "tmp_expired_history_ds_* ...does not exist'
Metabase Garbage Collection jobs, with 'Log deleted records' enabled, fail with:
[2]pg_query(): Query failed: ERROR: relation "tmp_expired_history_ds_20" does not exist

This change in behavior no longer uses intermediate table in Metabase database,
but logs deleted records directly to output files under /Storage/log/gc/.

ET2252992 - Datamining workflow hangs/fails, "JobStep has failed because it was not running on the agent"
Metabase data mining job fail with error message:
End JobStep: Failed to parse logfile. Reported error was: Failed to open input file: /Storage/tmp/workflow.9746
This JobStep has failed because it was not running on the agent while it should have been running according to the workflow engine.

ET2187410 - Stability improvements to 'pdbackup' for AIX client agents

ET2262000 - Sync of topology failed: [2]fopen(/Component.lck): failed to open stream
After installation of PureDisk, the following alerts show every hour:
Sync of topology failed: ***ERROR***nr: 999999severity: 6server:source:
Application_Component Application not yet initialized; latest captured error:
[2]fopen(/Component.lck): failed to open stream: Permission denied

ET2262831 - Always allow to start rerouting of storage pool in the PureDisk GUI

ET2262865 - Rerouting fails because job fails to connect to CR to retrieve progress info
If the rerouting process is in the "WaitCRDelete" or the "ReRoutingMonitorCR"
step, then it contacts the CR every 3 or 5 seconds to check for the
progress of this step. Every time these steps contact the CR there is also an
authentication check. In an environment with many CRs (up to 7 CRs) and if the
Storage Pool is under load during rerouting, the progress check in
"WaitCRDelete" or "ReRoutingMonitorCR" can fail with an authorisation error.

This causes the entire rerouting job to fail. The entire rerouting job needs
to be restarted. Sometimes multiple times.

Change the check interval from 3 and 5 seconds, to 5 minutes.

ET2273819 - PureDisk replication job gives incorrect message about PDDO replication
A PureDisk replication job on a non-PDDO data selection shows this incorrect
warning message:
This is a PDDO scheduled replication, not forwarding
NBU tag files (NBU_PD_MARKER) and (NBU_PD_SERVER) in the top level directory.

ET2270169 - MonitorStatistics is run immediately on startup of PureDisk services by pdagent.
This can result in erroneous Critical Alert if other services are slow to startup.
Change is to not execute MonitorStatistics on the agent in the first 10 minutes after

ET2271180 - Unable to create UNC data selection in PureDisk UI.
When expanding the domainname during UNC data selection creation, the following error is
shown: "Unable to execute dirlist on agent <agentname>(id) for path: /<domainname>"

ET2314380 - Replication failing, Error: 2 : Could not load route table /Storage/tmp/<n>.recommended
Replication fails if Maintenance workflow removes temporary files from /Storage/tmp, that
are required by the replication job.

- Replication routing tables are now stored in /Storage/var/rt/, not /Storage/tmp/
- Maintenance workflow checks if files have not been accessed for more than 24 hours before removing them

ET2350549 - Files on AIX larger than 2 GB are not backed up by PureDisk, but the backup job is successful

ET2343216 - Installation of PureDisk agent on Windows prevents removal of mapped drives.
Upon successfully install of PD agent sw, mapped drives to agent cannot be
removed with the "net use" command. Attempt to remove the mapped drives via
explorer also reports failure.

ET2369869 - PureDisk client agent leaks memory, causing various errors and failures

ET2269894 - Unable to browse files with commas in the name.

ET2253079 - Database connection leak in PD 6.6.1 Authorization daemon.

ET2343634 - OpsCenter 7.1 is unable to collect job data from PD
OpsCenter 7.1 fails to collect job data for PureDisk replication jobs. A NumberFormatException
is thrown while parsing the SOAP response.

ET2349540 - i18N: GUI got exception: org.xml.sax.SAXParseException when using Manual Backup Wizard - unexpected byte sequence (0x01)
When creating a backup job using 'Manual Backup Wizard', an exception error message is shown at the Data selection definition screen.
; nested exception is: org.xml.sax.SAXParseException: An invalid XML character
(Unicode: 0x1) was found in the element content of the document.
When click on root directory, the error message disappears, but the directory and files cannot be listed.

ET2369885 - Files smaller than 14 bytes are ignored during Content Router cache load, resulting in data loss
During Content Router startup, the cache load ignores files smaller than 14 bytes. As these fingerprint are not in the
cache, they are removed from the backend data store by the compaction process, resulting in missing data and segment

ET2363637 - Metabase real time web services use dslast/dslast_pddo depending on query filter
ET2395644 - Change the query for the metabase so does not operate off of the view
Filterless queries from NBU 7.1 to PD cause "hanging" postgres processes*,
as query to dslast_pddo_2 on MBE takes forever to complete.

Querying the active queries, shows more than 60 queries on dslast_pddo_2, all running for more than 1 hour. This causes
all opt-dupe jobs to fail and disk pool to DOWN.

ET2363637 - Raw segment data is not converted and causes duplication/restore failures
For certain upgrade paths, the segment data might be stored in the data container without a header (raw segment). During
retrieval (restore or duplication), this generates an error.

ET2249814 - Patch installation with rollback generates error event "Cannot find correct installer for Agent <name> (<id>)
with OS version 21". Issue is caused by an attempt to rollback to server agents, instead of only the client agents.

ET2347424 - Granular part of Exchange backup failing with vss_cmd_server_DB error. Phase 2 enumeration could not complete
Customer is attempting to backup his Exchange database with the Granular technology, the backup using PureDisk ends
successfully with errors which results in the granular part failing.

ET2409674 - /tmp/mbestart.tmp and /tmp/mbestop.tmp are not automatically cleaned up - grow exponentially whenever pdmbe is stopped/started

ET1918923 - Workflow Engine failing with "Timeout reached for connection from" causing all jobs to hang
The Workflow Engine log shows "Timeout reached for connection from", due to a Controller issues which uses up all
available connections and thread on the workflow daemon.

ET2432014 Unable to run SP Consistency Check work flow against replicated dataselections.

PureDisk GUI fixes:
ET2109691 - Error when viewing the alerts page in GUI
Fixes the following error occuring when attempting to view alerts page:
nested exception is: org.xml.sax.SAXParseException: An invalid XML character
(Unicode: 0xe) was found in the element content of the document.:Unable to view
all critical alerts

ET2227180 - Using "storage pool configuration wizard" to add a service to PD needs a warning if it is going to re-install PD.
Improvement to the PureDisk Installer interface, to prevent accidently storage pool re-installation and data loss:
1.Add "Confirm" message box on the last page of wizard when it is a re-installation.
2.A summary on last page of wizard about which action is going to be taken (But only shows summary when it is re-installation)
3.Fix incorrect warning message at "Broker configuration" page.

ET2043474 - Restored folder gets named with wrong symbols if it contains special characters in the folder name
A PureDisk restore to a folder with "special characters" will create a new restore destination folder with incorrect symbols.

ET2221041 - Not able to choose dataselection in data removal policy
PureDisk GUI improvement to display replicated clients and data selections in the data removal policy

ET2380137 The button 'Reroute Storage Pool' is missing in the GUI > Settings > Topology

ET2391154 Rollback Upgrade Patch 'Browse'/'Ok' fails to upload patches on VCS clustered PureDisk 6.6.1.x storagepools

PDDO Fixes:
ET2178573 - Optimized duplication failing between PDDO to PDDO storage units using AIX PDDO plug-in
Log example for failed duplication jobs using AIX PDDO plug-in:
Critical bpdm (pid=409846) sts_copy_extent failed: error 2060013 no more entries
Critical bpdm (pid=409846) image copy failed: error 2060013: no more entries
Error bpdm (pid=409846) cannot copy image from disk,bytesCopied = 18446744073709551615
Critical bpdm (pid=409846) sts_close_handle failed:2060022 software error

ET2209928 - PDDO backup to 6.6.1 fails w/setup_pdvfs_image_cache error Bad file descriptor
Backups to PDDO fail with status 83 and this error in the bptm log:
setup_pdvfs_image_cache error (-1 9 Bad file descriptor) PDVFS_IOCTL_ADD_DO_FP_CACHE

ET2215029 - Backups to dedupe STU failing with status 84 when attempting to write first fragment header.

ET2280249 - Rerouting hangs with the data backed up from NetBackup 7.0.1
When backing up from NBU 7.0.1 to a PureDisk storage pool, rerouting cannot proceed to the end, the data
transfer is always in progress. The newly added CR node is filled with data.

ET2272289 - PDDO Restores during Re-Routing fails with "Could not send get operations for DOs: zero-sized object"
Example of log:
PDVFS: [1] pdvfs_lib_log: Could not send get operations for DOs: zero-sized object
PDVFS: [1] pdvfs_lib_log: Could not send all DO get operations via recommended route table: zero-sized object
PDVFS: [1] pdvfs_pread: CRDOReaderRead for failed: 47 (zero-sized object)
verify_mapfile_version error reading mapfile version (5 Input/output error)

ET2251112 - VmWare VCB Flashbackup (Type 3 Virtual machine) backups to PDDO storage unit fail with status 84.

ET2244429 - Content Router hangs due to deadlock caused by parallel Disaster Recovery backup and restore or duplication job on
NetBackup side. Batch prefetching is disabled by fix this (PREFETCH_SIZE=0) to prevent hang.

ET2278855 - PDDO Replication policy replicates expired images to destination storage pool
PDDO Replication will replicate all images from PDDO data selection, including images that have been
expired from NetBackup, but not yet fully cleaned by the PureDisk maintenance workflows.

- Replicate only valid PDDO images, by using dslast database view instead of ds database view

ET2341505 - PDDO optimized duplications between two PureDisk storage pools have no compression or bandwidth set
The compression and bandwidth parameters for all PDDO optimized duplication between PureDisk storage
pools is empty, causing uncompressed data on the target storage pool and the bandwidth not being honoured.

ET2327808 - A PureDisk storage pool registered to NetBackup 7.1 received critical events "getSDKversion"
Event occurs ever hour and has description: "Validation error: action has an invalid value: getSDKVersion"

IMPORTANT: A 'client agent update' job will NOT be created for PDDO agents. Please
follow the procedure 'Upgrading PDDO client agents on Media Servers'.

Content Router Queue Processing:
ET2246881 - Detailed progress logging on a more regular time interval during CR queue processing
Logging for Content Router queue processing steps (sorting, tlog processing, delayed file processing,
creating database index, ...) is more verbose and more frequently logged, so a distinction can be made
between a hanging and a slow queue processing job.

ET2253050 - ET2215448 - ET2222209 - ET2222453 - ET2248352: CR queue processing improvements for stability and supportability.

These improvements are made to provide more stability and easier troubleshooting for CR queue processing:
1. Read the tlog file again right after it was closed.
2. Create copy of the tlog file in processed directory (/Storage/processed) at the start of queue processing.
3. Delete the tlog copy after queue processing completes successfully.

The Content Router process will also shut down after five consecutive failures during queue processing.
This behavior prevents the queue from grower any further and forces the customer to take action before continuing,

ET2312853 - Queue processing fails to create new .tlog in queue directory.
Queue processing error: Could not move spool log /Storage/spool/8748.tlog to /Storage/spoold/8748.tlog (The file exists.)

ET2353696 - Keep unprocessed tlogs in /Storage/processed/ directory if CR queue processing fails.

1. Use rcp or scp to copy the tar file, NB_PDE_6.6.1.2_EEB20-ET2399563_rollup2.tar,
to the /root directory of the PureDisk node that hosts the storage pool authority.

2. Log on as root to the node that hosts the storage pool authority.

3. Type cd /root to go to the /root directory.

4. Type the following command to extract the README file:

tar -C / -xf /root/NB_PDE_6.6.1.2_EEB20-ET2399563_rollup2.tar ./NB_PDE_6.6.1.2_EEB20-ET2399563_rollup2.README


1.Log into the PureDisk administrative WebUI and make sure that no PureDisk jobs are running or are scheduled to run. Stop the jobs that are running currently. Content router queue processing job(s) can not be manually stopped and will need to complete before proceeding.
Note: The patch installation process stops all jobs that are running. If you want to stop any jobs gracefully, make sure to stop them before you proceed.

2.Log out from the administrative Web UI.

3.Confirm only PureDisk required partitions are mounted. If additonal partitions have been mounted for disaster recovery or other reasons, these need to be unmounted before proceeding.

4.(Conditional) Freeze PD service groups for clustered PD server.

Perform this step if the storage pool is installed with VCS cluster
software. Use the Cluster Manager Java Console, and freeze all the service

For information about how to freeze and unfreeze clustered storage pools,
see the Veritas NetBackup PureDisk Storage Pool Installation Guide.

The The following command will confirm the service groups are frozen

/opt/VRTSvcs/bin/hastatus -sum

Look for the following section and confirm all nodes are frozen (there should be a group for each active node in the cluster).

-- Group

G pd_group1
G pd_group2

5.(Conditional) Freeze any additional service groups in VCS which were configured outside of the PureDisk installer. This may include a cluster services monitor.

6.Run the health checker script to confirm the environment is in a good state. The health checker will also provide changes in the configuration files which also need to be updated in the configuration portion of the GUI.

/opt/pdag/bin/php /opt/pdconfigure/scripts/support/health_checker.php


IMPORTANT: A patch installation using the rollback functionality requires additional free space.
If the partition free space on any PureDisk node is low, it is strongly recommended to *NOT* use the rollback installation option.

Uploading patch
1 Log into the storage pool configuration wizard as root.
2 On the storage pool configuration wizard's Welcome page, click Manage Patches.
3 On the Upgrade and Roll Back page, click the Available Software tab.
4 Specify a patch to upload. Use one of the following methods:
A. Click Browse, and select the patch you want to install from your browser's upload menu. For example, navigate to the C: drive and select a patch.
B. Type the path to the patch you want into the Choose file to upload: field.
5 Verify that the file in the Choose file to upload field is correct and click OK.

Installing a patch
1.Complete the "Upgrade and Roll Back - Patch List" page.Perform the following steps on this page:
A. Click the "Available Software" tab, and verify that the patch you want to install appears in the list of patches.
If the patch you want to install does not appear, perform the following procedures see uploading a patch section above.
B. In the Patch Version column, click the patch number of the patch you want.
C. Read the "Software Details" pop-up page for this patch, and verify the prerequisites. The Software Details page explains
the patch purpose, explains the rollback possibilities for this patch, and contains other information. Verify that your
storage pool meets the patch prerequisites, and click Close.
2.Click Install.
3.Complete the "Upgrade and Roll Back - Installation Options" page.
Depending on the nature of the patch you want to install, the 'Upgrade and Roll Back - Installation Options' page displays installation options.
(Optional) Select one or more of the following possible options:
A. Create a complete checkpoint for this patch. Most patches let you create a system checkpoint before the installation starts.
A checkpoint is a record of your storage pool at a specific time. Later, if the patch has a negative effect on your storage pool,
you can remove the patch and reinstate the checkpoint.
B. Upgrade agents automatically after patch installation completes Some patches let you specify whether or not to upgrade the agents automatically.
Other patches upgrade the agents without asking you to confirm this action.
4.Click Start Installation.
5.Complete the Upgrade and Roll Back - Installation Progress page. Perform the following steps on this page:
A. Wait for the installation to complete.
B. Click Go To Installed Software or Close.
If you click Go To Installed Software, the wizard returns to the Upgrade and Rollback - Patch List page.
If you click Close, the upgrade and roll back pages close. The configuration wizard remains open in another window.
6.If you created a checkpoint, you have the following options:
A. Enter a test period and commit the patch after the test period ends.
B. Roll back the patch.

If you did not create a checkpoint, you can not commit or remove the patch.

(Conditional) Unfreeze PD service groups for clustered PD server.

Perform this step if the storage pool is installed with VCS cluster

The Veritas Cluster Server (VCS) software might detect some faults
during the upgrade process. If any upgrade actions generate a VCS fault,
use the Cluster Manager Java Console to clear the fault and probe that
resource group before you unfreeze the cluster.

To commit a patch
1.Log into the storage pool configuration wizard as root.
2.On the storage pool configuration wizard's Welcome page, click Manage Patches.
3.On the Upgrade and Roll Back page, click the Installed Software tab.
4.Click Commit.

To rollback a patch.
Note: This can only be performed when checkpoints were enabled during the patch installation and the patch has not been committed.
1.Log into the storage pool configuration wizard as root.
2.On the storage pool configuration wizard's Welcome page, click Manage Patches.
3.On the UpgradeandRollBack-PatchList page, click the Installed Software tab.
4.Click Roll back.
5.On the pop-up that appears, click OK to confirm that you want to perform a rollback.
6.Wait for the rollback to complete.
7.On the UpgradeandRollBack-RollbackProgress page, click GoToInstalled Software or Close.

About checkpoints
PureDisk considers a storage pool to be in checkpoint mode if you specified that you wanted to create a checkpoint during
the installation of a patch. You can have more than one checkpoint on your system at a time. After you commit a patch, PureDisk
removes the checkpoint for the patch.
If one or more checkpoints exist, the PureDisk administrative Web UI displays the message Storage pool is in checkpoint mode.
The storage pool configuration wizard also displays a checkpoint mode message. Depending on how full your system is, you might
be able to leave a storage pool in checkpoint mode for a few days. The length of time also depends on the change rate of the
clients being backed up. Garbage collection does not occur when a storage pool is in checkpoint mode. Monitor your system closely
and remove the checkpoint as soon as you can.

Do not attempt to remove a checkpoint manually. Always use the storage pool configuration wizard to remove checkpoints.

Managing multiple checkpoints
You can have up to three checkpoints on a storage pool at a time. For each patch that lets you create a checkpoint, you can create
one checkpoint. Some patches do not permit you to create a checkpoint. When you commit a patch, PureDisk removes the patch's checkpoint.
Storage pool functionality while in checkpoint mode PureDisk functionality is reduced when the storage pool is in checkpoint mode.
Specifically, checkpointing has the following effects on the storage pool:
- You can start the storage pool configuration wizard, but its available functions are severely reduced. You can manage patches,
but you cannot perform other functions. For example, you cannot add a new node while the storage pool is in checkpoint mode because
the addition would invalidate the checkpoint.
- To prevent performance degradation and data loss, PureDisk prevents certain storage pool operations from occurring when the storage pool
is in checkpoint mode. PureDisk deduplication option (PDDO) operations, rerouting, replication, disaster recovery restores, and garbage
collection cannot run when the storage pool is in checkpoint mode.


1. Type the following command to unpack the Engineering Binary software:

tar -C / -xf /root/NB_PDE_6.6.1.2_EEB20-ET2399563_rollup2.tar ./

4. Type the following command to run and install the Engineering Binary:


If the topology.ini file is encrypted, the software prompts you for the
password to decrypt this file.

The Engineering Binary automatically pushes the software to all nodes in the
storage pool and to all clients.

At the end of a successful installation, the software prompts you to encrypt
the topology.ini file.

5. (Conditional) Unfreeze PD service groups for clustered PD server.

Perform this step if the storage pool is installed with VCS cluster

The Veritas Cluster Server (VCS) software might detect some faults
during the upgrade process. If any upgrade actions generate a VCS fault,
use the Cluster Manager Java Console to clear the fault and probe that
resource group before you unfreeze the cluster.

6. Monitor the client agent update jobs.

Take the following actions to monitor the client agent upgrade jobs:

A. Invoke the Web UI.
B. Click the Workflows tab.
C. Select the Agent update workflow.

If one of the upgrade jobs fails, PureDisk deactivates those client agents.
You will need to activate these agents again and to select the client to
upgrade to. Follow the steps below:

A. Invoke the Web UI.
B. Click the Data management tab.
C. In the middle pane, select the 'Deactivated client agents by storage pool' view,
which is the third view from the left.
D. Select the 'storage pool' level in the middle pane
and choose 'Activate Agents' from the left hand pane.
E. Refresh the middle pane.
It will switch back to the default view with all clients.
F. Select the 'storage pool' level in the middle pane
and choose 'Upgrade Agents' from the left hand pane.
Choose the version you want to upgrade to.

Monitor these upgrade jobs again, start over if they fail or time-out.

7. Upgrade the PDDO agent software on PDDO media server

Perform the procedure called "UPGRADING PDDO AGENTS ON MEDIA SERVER
CLIENTS", which follows. The release update includes upgraded
PDDO agents for Linux, Solaris, and Windows clients.

1. Make sure that the storage pool software for the PDDO clients has been

2. Refresh the PureDisk landing page.

The landing page can be found on:


For <your-puredisk-spa>, type the hostname or the IP address of your
PureDisk Storaga Pool Authority.

Be aware that this release update includes both the PureDisk 6.6
base release agents and revised agents for Linux, Solaris, and Windows
clients. The revised agents include updates for PDDO agent installation
only. A later step in this procedure instructs you to download the
revised agents if you want to upgrade the PDDO agent on a client.

3. Log in to each PDDO client.

4. From the PDDO media server client, download the new PDDO agent software.

The landing page includes more than one software version of the PDDO
agent, so take care to download the most recent agent.

5. Stop all NetBackup services on the NetBackup media server.

6. Install the agent software included with this release update.

* On Windows systems, double-click the PureDisk agent icon. This starts
the Windows installation Wizard.

* On Linux or Solaris systems, you can use either the attended or the
unattended installation method. The installer prompts you to confirm
the upgrade. For example, type the following command to upgrade the
Solaris 10 agent:

# sh

For NNNNN, type the name of the PureDisk upgrade release number.

NOTE: The upgrade software removes the old PDDO agent software

For more information about how to install the PDDO agent, see the Veritas
NetBackup PureDisk Deduplication Option Guide.

7. Start all the NetBackup services on the NetBackup media server.



Hotfix Bundle NB_PDE_6.6.1.2_EEB20-rollup2.tar provides fixes to Symantec NetBackup (tm) PureDisk
NB_PDE_6.6.1.2_EEB20-ET2399563_rollup2.tar (243.4 MBytes)

Article URL

Terms of use for this information are found in Legal Notices