Video Screencast Help

Client install Checking to see if the PDDE configuration needs upgrading hang

Created: 21 Jan 2014 • Updated: 14 Apr 2014 | 40 comments
Michael G Andersen's picture
This issue has been solved. See solution.

Hello

The of install Netbackup 7.5 client on AIX 7.1 hangs on Checking to see if the PDDE configuration needs upgrading

Have found a technote regarding siilar issue on HP-UX which caused by automount being on, checked and automount (cdromd) was not running on the AIX machine

There is a firewall between the master and the AIX machine

Has somebody seen something like this ?

Regards

Michael

 

Operating Systems:

Comments 40 CommentsJump to latest comment

Marianne's picture

Have not seen it and can only find the same TN that you did.

Is this a new NBU7.5 Client installation? Or and upgrade?

Did you remember to use gtar to extract the Client binaries?
See http://www.symantec.com/docs/TECH154080

Do you have the installation log file? (/tmp/install_trace.###)

Hopefully the log will give a clue...

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Michael G Andersen's picture

It is a new NBU7.5 Client installation. Can't remember which tar I used to extract the files as that was done quite a while ago on our NFS software share.

Have the install log file(s) (latest attached with .txt extension), seems to me that they just stops too.

Can see that pdcfg runs 100% some of the CPU's, think there is a process for each failed install attempt

AttachmentSize
install_trace.6225938.txt 3.16 KB
Michael G Andersen's picture

When running sort offline, it seems that perl is using 100% of on CPU

Michael G Andersen's picture

Seems sort also have ani issue with this AIX machine, get no report to upload

Marianne's picture

SORT as in https://sort.symantec.com/ ?

or OS sort command?

Do you suspect something wrong at OS level? 

All I can suggest is to kill hung processes, remove already installed software packages and try again, but if we don't know why the installation got hung in the first place, there is no guarantee that it won't happen again. 
Looking at the log, it seems that installation should be more or less completed? 
Wonder what would happen if you kill hung pdcfg, check that bpcd and vnetd processes are running and then try a small test backup?

I have had a look at Operating system patches and updates in NBU 7.5 Release Notes - there are OS patch requirements listed for AIX 5.3 and 6.1. Nothing for AIX 7.1.

Do you have other AIX clients? Was same software used to install?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Michael G Andersen's picture

yes sort from tps://sort.symantec.com/ as I hoped to get an error or warning that indicated what problem was

dbzq's picture

I am running into the same issue. Install hangs at PDDE install but log file does not indicate a problem this includes the PDDE log flle from /var/log/puredisk which shows PDDE agent being successfully installed. There is a firewall between the client and the master server. Just FYI my hardware platform is Power5.

AttachmentSize
install_trace.txt 3.27 KB
2014-01-21_10_38-pdde-install.log_.txt 3.12 KB
Michael G Andersen's picture

Thanks. Just checked my PDDE log also indicates that PDDE has been installed successfully

Marianne's picture

What you have in common is the firewall between Client and master.

Although Dedupe is not used by the client, it might be worth your while to ask Firewall Admins to monitor network between Master and Client to see if any comms is initiated from the client side during the installation.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Matt-D's picture

I am wondering if this issue is related to AIX 7.1 TL3. I have exactly the same issue when installing a NB master (so no firewalls involved here). Same symptons - hang on pdcfg and it uses lots of CPU.

A truss of the process process reveals nothing either, so it seems to be 100% busy doing absolutley nothing.

Killing the pdcfg processes (had to kill several) then completes the install, seemingly OK, and as far as I can tell the install is good.

AIX 7.1 TL2 installs of NB work fine.

Is everyone else who has this issue using AIX 7.1 TL3?

 

 

Michael G Andersen's picture

Our AIX in question is a 7.1 TL3 (oslevel -sq  first entry 7100-03-01-1341)

Michael G Andersen's picture

Nothing showed up in the firewall on the dedup ports 10082, 10085, 10102

The client install completed after I had killed pdfcg twice with ./pdinstall killed messsages 

Michael G Andersen's picture

Have created Case # 05899237 - Client install hangs on "Checking to see if the PDDE configuration needs upgrading"

Matt-D's picture

Good idea,

I think that is the only reasonable course to take now. Could you update this thread with the outcome. Good luck!

 

Michael G Andersen's picture

According to the supporter on the case it is a known issue on AIX 7.1 TL3 for Netbackup 7.5. Have asked if there is tech note and if yes a link to it

 

Michael G Andersen's picture

From our AIX machine:

# ldd pdcfg
pdcfg needs:
Cannot find libz.so
Cannot find libssl.so
Cannot find libcrypto.so
         /usr/lib/libpthread.a(shr_xpg5_64.o)
Cannot find libcurl.so
Cannot find libxml2.so
Cannot find libACE.so
Cannot find libACE_SSL.so
Cannot find libssh2.so
Cannot find libdct.so
Cannot find libfc.so
Cannot find libcr.so
Cannot find libmetab.so
Cannot find libpd.so
Cannot find libpuredisk++.so
Cannot find libfsysora.so
Cannot find libstdc++.a(libstdc++.so.6)
Cannot find libgcc_s.a(shr.o)
         /usr/lib/libc.a(shr_64.o)
         /unix
         /usr/lib/libcrypt.a(shr_64.o)
 

Thinking that we either are missing some libraries or they have changed name from TL2 to TL3

Matt-D's picture

Not sure if this might be a red herring here. On a TL2 system I get exactley the same.

oslevel -s:
7100-02-03-1334
 

>ldd /usr/openv/pdde/pdag/bin/.bin/pdcfg
/usr/openv/pdde/pdag/bin/.bin/pdcfg needs:
Cannot find libz.so
Cannot find libssl.so
Cannot find libcrypto.so
         /usr/lib/libpthread.a(shr_xpg5_64.o)
Cannot find libcurl.so
Cannot find libxml2.so
Cannot find libACE.so
Cannot find libACE_SSL.so
Cannot find libssh2.so
Cannot find libdct.so
Cannot find libfc.so
Cannot find libcr.so
Cannot find libmetab.so
Cannot find libpd.so
Cannot find libpuredisk++.so
Cannot find libfsysora.so
Cannot find libstdc++.a(libstdc++.so.6)
Cannot find libgcc_s.a(shr.o)
         /usr/lib/libc.a(shr_64.o)
         /unix
         /usr/lib/libcrypt.a(shr_64.o)
 

Marianne's picture

Let us know what the outcome is for TN request.

I would think that NBU 7.6 LBN should be updated with this info as well.

I have search NBU 7.6 documentation for OS requirement for AIX 7.1 and could not find anything...

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Michael G Andersen's picture

Seems some of the issue is the relative paths used in setting LIBPATH

 

Example with absolute paths

 

# echo $LIBPATH

/usr/openv/pdde/pdopensource/lib:/usr/lib:/usr/openv/pdde/pdshared/lib

#

 

# ldd pdcfg

pdcfg needs:

         /usr/openv/pdde/pdopensource/lib/libz.so

         /usr/openv/pdde/pdopensource/lib/libssl.so

         /usr/openv/pdde/pdopensource/lib/libcrypto.so

         /usr/lib/libpthread.a(shr_xpg5_64.o)

         /usr/openv/pdde/pdopensource/lib/libcurl.so

         /usr/openv/pdde/pdopensource/lib/libxml2.so

         /usr/openv/pdde/pdopensource/lib/libACE.so

         /usr/openv/pdde/pdopensource/lib/libACE_SSL.so

         /usr/openv/pdde/pdopensource/lib/libssh2.so

         /usr/openv/pdde/pdshared/lib/libdct.so

         /usr/openv/pdde/pdshared/lib/libfc.so

         /usr/openv/pdde/pdshared/lib/libcr.so

         /usr/openv/pdde/pdshared/lib/libmetab.so

         /usr/openv/pdde/pdshared/lib/libpd.so

         /usr/openv/pdde/pdshared/lib/libpuredisk++.so

         /usr/openv/pdde/pdshared/lib/libfsysora.so

         /usr/openv/pdde/pdopensource/lib/libstdc++.a(libstdc++.so.6)

         /usr/openv/pdde/pdopensource/lib/libgcc_s.a(shr.o)

         /usr/lib/libc.a(shr_64.o)

         /usr/lib/libpthreads.a(shr_xpg5_64.o)

         /unix

         /usr/lib/libxti.a(shr_64.o)

         /usr/lib/libiconv.a(shr4_64.o)

         /usr/lib/libcrypt.a(shr_64.o)

         /usr/lib/libc.a(pse_64.o)

Matt can you confirm on your TL2 system ?

John Hink's picture

This is a known issue and engineering is currently working on it with IBM. The newest AIX TLs (6.1 TL9 and 7.1 TL3) have a problem with the libACE shipped with the MSDP component and when libACE is initilized it results in the proccess hang.  Note this does not apply to the previous AIX TL's. Short term, the recommended workaround is to use the previous TL. Hopefully IBM will be able to help identify what changed in the new TLs caused this issue.  I'll be sure to update this thread when I get more information.  

Marianne's picture

This is a known issue .....

My favourite response from Support........... After wasting LOTS of time testing, searching Symantec knowledge base, forums, looking at logs, documentation.... 
And then to hear: This is a known issue  ... 

Thanks for letting us know!

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

John Hink's picture

I certainly can sympathize with you there. But let me ensure you this was very recently discovered. As I said, we are working with IBM to  resolve this problem as quickly as possible. 

Michael G Andersen's picture

Have suggested that a tech note is created so customer can subscribe and see when the issue is solved. Might already have happened, must admit I havn't checked

smond's picture

Encountered the same issue.

Logged a case and was advised that as the AIX machine is a client with no media server capability, to use NBU7.6 with a workaround.

# touch /tmp/PDDE_SKIP_PACKAGE_SCRIPTS

will allow the installation to complete, but will not allow the server to be used as an MSDP server, nor use client-direct or PDDE storage units..

As for a tech note to check when it is resolved,

There is an internal tech note available for this issue but until the joint team of IBM and Symantec engineering finds root cause, this article will still remain as internal. I would recommend that you check below link from time to time to see if article becomes publicly available which would mean root cause is found.

http://www.symantec.com/business/support/index?page=content&id=TECH214505

Currently this link has no content.

Michael G Andersen's picture

Does that mean 7.6 does not work either ?

ask as the first suggestion from support was to install the 7.6 Client on our AIX7.1TL3 machine

smond's picture

If you are referring to full functionality of NBU7.6 client, it does not work.

However, it is good enough for me as I would only need to backup the AIX client flat files directly to tapes at this time.
Backup and restore tests directly to tapes looks to be working.

Michael G Andersen's picture

Done a little more checking and can see some of the libraries for pdde is back from AIX 5.3A/5.3L

Seems to me some of the problem is that Symantec does not recompile the client software at major AIX version e.g. 6.1 & 7.1

irnltd's picture

I'm unable to install NBU7.6 master server on AIX7.1 TL3 SP1 for the same reason (pdcfg hangs).

There is no warning in support matrix and customer is not willing to reinstall AIX with lower TL...

Any chance for timely solution?

Michael G Andersen's picture

Unfortunately don't really get much in the way of update on my Support case.

My best offer is to create a support case too, if you have not already. Think the more customers create cases on this problem the more attention Symantec gives it

And in the mean time just kill the pdcfg processes so your customer can run backup/restore on their AIX 7.1TL3 clients, albeit without client deduplication

Marianne's picture

Michael, are you insisting on feedback from Backline and/or Engineering?

Maybe all of you should exchange case #'s via PM and submit to your respective case engineers.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Michael G Andersen's picture

I try to insist on getting some feedback and I am getting this:

==========================================SNIP ============================

At this point of time we don’t have much more details except below.

 

1.      Issue is somewhat with the AIX OS platform architecture.

2.      NO fault from NBU perspective.

3.      IBM to share their inputs & work together with Symantec to find a mutual solution on this issue. Engineering Team is on it.

4.      For more you could verify TSAnet call ref PMR 70884,003,756. Next steps are determined by the outcome of this ticket.

==================================END SNIP ================================

Have stated that I think point 2 is not entirely correct as Symantec has not recompiled the client software at the major AIX releases 6.1 & 7.1

My case is 05899237

John Hink's picture

Hi everybody, here's an update:

This will be fixed in 7.6.0.2. You will have to touch /tmp/PDDE_SKIP_PACKAGE_SCRIPTS  to complete the 7.6.0.1 install and then apply the 7.6.0.2 patch. 

There will also be an EEB solution for the meantime. Look for TECH214505 to be published by friday (3/14/) with additional details. 

RichardA's picture

Hi,

I have run into the same issue with upgrading a NetBackup Master Server from 7.5.0.6 to 7.5.0.7 while running on AIX 6.1 TL9. (I currently have cases open for this too)

Will this fix be rolled out for server upgrades as well as client upgrades? Also will it be rolled out for the 7.5.0.x version as well as the 7.6.0.x version?

The reason I ask is that I am dealing with a bpdbm issue that requires an EEB that Backline/Engineering have developed for 7.5.0.7 to be rolled out.

Thanks

CRZ's picture

Will this fix be rolled out for server upgrades as well as client upgrades? Also will it be rolled out for the 7.5.0.x version as well as the 7.6.0.x version?

Yes and yes, but you'll still need to work it via a support case, I think.  Since you already have one open, ask your TSE to check out the TechNote John has mentioned above (TECH214505) and they should be able to help you get to 7.5.0.7 right now...or escalate it if not.


bit.ly/76LBN | APPLBN | 75LBN

Michael G Andersen's picture

For AIX 7.1 there is APAR IV56004 which should solve this when the fix is released

John Hink's picture

Please check out TECH214505 here:

http://www.symantec.com/business/support/index?pag...

It has all the information and the download links for the IBM iFixes. 

I still stongly suggest you upgrade to 7.6.0.2 (when available) before any further IBM TL updates are applied as there is no guarantee the IBM fix will be in the next TL.  

SOLUTION
jgordo02's picture

As a workaround using 7.5.0.4 is to run the install and when it hangs on the pdde process and kill that but don't kill the whole install process.  We have used this on 100's of new Aix 14r1 clients and it works like a charm.