Video Screencast Help
Scheduled Maintenance: Symantec Connect is scheduled to be down Saturday, April 19 from 10am to 2pm Pacific Standard Time (GMT: 5pm to 9pm) for server migration and upgrades.
Please accept our apologies in advance for any inconvenience this might cause.

Bpbkar Process hung on client

Created: 16 Dec 2012 • Updated: 17 Dec 2012 | 16 comments
This issue has been solved. See solution.

Master Server -7.1 (unix)

Client 6.5.6 (unix)

Bpbkar processes are hung on client and not killing automatically even after server reboot.Only /var backup is failing with 13 error and for that i have enabled the bpbkar logs.

Now need to know what Files are actually causing issue,so that i can check with SA.

Also need to know that the below Mount point is causing an issue:

 

16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv2 on /appl/logarchive
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/logarchive are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>

16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/logarchive is /appl/logarchive
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv3 on /appl/logs
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/logs are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/logs is /appl/logs
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv4 on /appl/lsinfra
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/lsinfra are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/lsinfra is /appl/lsinfra
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv5 on /appl/monitor
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/monitor are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/monitor is /appl/monitor
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv6 on /appl/problem
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/problem are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/problem is /appl/problem
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv7 on /usr/local/opt/tools
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /usr/local/opt/tools are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
...skipping...
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd9var on /var
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /var is /var
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd3 on /tmp
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /tmp is /tmp
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd1 on /home
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /home is /home
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd10opt on /opt
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /opt is /opt
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hsadm_lv on /home/sadm
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /home/sadm is /home/sadm
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/sysintro_lv on /siopt
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /siopt is /siopt
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (procfs) /proc on /proc
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /proc is /proc
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/local_lv on /usr/local
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/local is /usr/local
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/localcw_lv on /usr/localcw
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/localcw is /usr/localcw
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/patrol_lv on /usr/localcw/opt/patrol
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/localcw/opt/patrol is /usr/localcw/opt/patrol
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/cischk_lv on /var/adm/cischk
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /var/adm/cischk is /var/adm/cischk
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/husers_lv on /home/users
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /home/users is /home/users
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/security_lv on /usr/localcw/opt/security
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/localcw/opt/security is /usr/localcw/opt/security
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/cfeng_lv on /usr/localcw/opt/cfengine
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /usr/localcw/opt/cfengine is /usr/localcw/opt/cfengine
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/vsf_lv on /tmp/vsf
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /tmp/vsf is /tmp/vsf
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/livedump_lv on /var/adm/ras/livedump
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /var/adm/ras/livedump is /var/adm/ras/livedump
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/hd11admin on /admin
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /admin is /admin
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/varsa_lv on /var/adm/sa
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /var/adm/sa is /var/adm/sa
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (local) /dev/perfstats_lv on /perfstats
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /perfstats is /perfstats
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv2 on /appl/logarchive
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/logarchive are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.535 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/logarchive is /appl/logarchive
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv3 on /appl/logs
16:22:59.535 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/logs are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/logs is /appl/logs
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv4 on /appl/lsinfra
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/lsinfra are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/lsinfra is /appl/lsinfra
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv5 on /appl/monitor
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/monitor are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/monitor is /appl/monitor
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv6 on /appl/problem
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /appl/problem are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
16:22:59.536 [7405736] <2> mount resolve_mount_path: INF - Actual mount point of /appl/problem is /appl/problem
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Processing (vxfs) /dev/vx/dsk/crm_vg/crm_lv7 on /usr/local/opt/tools
16:22:59.536 [7405736] <2> mount build_mount_list: INF - Mount options for /usr/local/opt/tools are <rw,delaylog,suid,ioerror=mwdisable,qio,largefiles>
log.121512 (0%)
 

Is there ant specific way to find out under /var which files are causing issue bcoz when we check with SA ,as per them VAR is ok.......

Any help will be appreciated.....

Discussion Filed Under:

Comments 16 CommentsJump to latest comment

2013's picture

One more thing i have noticed is that the SelectFile..please check the below  bpbkar logs:

16:22:59.603 [7405736] <2> mount build_mount_list: INF - Processing (NFS) /vol/v00_fs01_common/q00_fs01_common on /nas/common
16:22:59.603 [7405736] <2> mount build_mount_list: INF - Processing (NFS) /vol/v00_fs01_admin/q00_fs01_admin/ibm on /nas/usr/sbc
16:22:59.603 [7405736] <2> bpbkar SelectFile: INF - cwd = NULL
16:22:59.603 [7405736] <2> bpbkar SelectFile: INF - path = /appl/lsinfra
16:22:59.603 [7405736] <2> bpbkar resolve_path: INF - Actual mount point of /appl is /appl
16:22:59.603 [7405736] <2> bpbkar SelectFile: INF - Resolved_path = /appl/lsinfra
16:22:59.603 [7405736] <4> bpbkar SelectFile: INF - VxFS filesystem is /appl/lsinfra for /appl/lsinfra
16:22:59.606 [7405736] <2> bpbkar resolve_path: INF - Actual mount point of /appl/lsinfra is /appl/lsinfra
16:22:59.606 [7405736] <4> bpbkar PrintFile: /
16:22:59.606 [7405736] <4> bpbkar PrintFile: /appl/
16:22:59.612 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/
16:22:59.627 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.627 [7405736] <2> bpbkar SelectFile: INF - path = .lastlogin
16:22:59.627 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/.lastlogin
16:22:59.627 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.627 [7405736] <2> bpbkar SelectFile: INF - path = .sh_history
16:22:59.628 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/.sh_history
16:22:59.628 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.628 [7405736] <2> bpbkar SelectFile: INF - path = lost+found
16:22:59.629 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/lost+found/
16:22:59.629 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.629 [7405736] <2> bpbkar SelectFile: INF - path = scripts
16:22:59.630 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/
16:22:59.630 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.630 [7405736] <2> bpbkar SelectFile: INF - path = load-FP39-6-1-release-17.ksh
16:22:59.636 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-FP39-6-1-release-17.ksh
16:22:59.636 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.637 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-17-FP39-HTTPServer1.ksh
16:22:59.637 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-17-FP39-HTTPServer1.ksh
16:22:59.637 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.637 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-17-FP39-HTTPServer2.ksh
16:22:59.638 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-17-FP39-HTTPServer2.ksh
16:22:59.638 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.638 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-17-FP39-HTTPServerDiag.ksh
16:22:59.645 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-17-FP39-HTTPServerDiag.ksh
16:22:59.645 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.645 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-18-HTTPServer.ksh
16:22:59.653 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-18-HTTPServer.ksh
16:22:59.653 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.653 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-IBMHTTPD-6-1-release-18.ksh
16:22:59.654 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-IBMHTTPD-6-1-release-18.ksh
16:22:59.654 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.654 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-17-FP39-HTTPServerDiag.ksh.diag
16:22:59.654 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-17-FP39-HTTPServerDiag.ksh.diag
16:22:59.654 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.654 [7405736] <2> bpbkar SelectFile: INF - path = config
16:22:59.655 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/
16:22:59.655 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config
16:22:59.655 [7405736] <2> bpbkar SelectFile: INF - path = HTTPServer1
16:22:59.656 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer1/
16:22:59.656 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer1
16:22:59.656 [7405736] <2> bpbkar SelectFile: INF - path = install1-ihs-39.txt
16:22:59.658 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer1/install1-ihs-39.txt
16:22:59.658 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer1
16:22:59.658 [7405736] <2> bpbkar SelectFile: INF - path = install1-plg-39.txt
...skipping...
16:22:59.603 [7405736] <2> mount build_mount_list: INF - Processing (NFS) /vol/v00_fs01_common/q00_fs01_common on /nas/common
16:22:59.603 [7405736] <2> mount build_mount_list: INF - Processing (NFS) /vol/v00_fs01_admin/q00_fs01_admin/ibm on /nas/usr/sbc
16:22:59.603 [7405736] <2> bpbkar SelectFile: INF - cwd = NULL
16:22:59.603 [7405736] <2> bpbkar SelectFile: INF - path = /appl/lsinfra
16:22:59.603 [7405736] <2> bpbkar resolve_path: INF - Actual mount point of /appl is /appl
16:22:59.603 [7405736] <2> bpbkar SelectFile: INF - Resolved_path = /appl/lsinfra
16:22:59.603 [7405736] <4> bpbkar SelectFile: INF - VxFS filesystem is /appl/lsinfra for /appl/lsinfra
16:22:59.606 [7405736] <2> bpbkar resolve_path: INF - Actual mount point of /appl/lsinfra is /appl/lsinfra
16:22:59.606 [7405736] <4> bpbkar PrintFile: /
16:22:59.606 [7405736] <4> bpbkar PrintFile: /appl/
16:22:59.612 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/
16:22:59.627 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.627 [7405736] <2> bpbkar SelectFile: INF - path = .lastlogin
16:22:59.627 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/.lastlogin
16:22:59.627 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.627 [7405736] <2> bpbkar SelectFile: INF - path = .sh_history
16:22:59.628 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/.sh_history
16:22:59.628 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.628 [7405736] <2> bpbkar SelectFile: INF - path = lost+found
16:22:59.629 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/lost+found/
16:22:59.629 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.629 [7405736] <2> bpbkar SelectFile: INF - path = scripts
16:22:59.630 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/
16:22:59.630 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.630 [7405736] <2> bpbkar SelectFile: INF - path = load-FP39-6-1-release-17.ksh
16:22:59.636 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-FP39-6-1-release-17.ksh
16:22:59.636 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.637 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-17-FP39-HTTPServer1.ksh
16:22:59.637 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-17-FP39-HTTPServer1.ksh
16:22:59.637 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.637 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-17-FP39-HTTPServer2.ksh
16:22:59.638 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-17-FP39-HTTPServer2.ksh
16:22:59.638 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.638 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-17-FP39-HTTPServerDiag.ksh
16:22:59.645 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-17-FP39-HTTPServerDiag.ksh
16:22:59.645 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.645 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-18-HTTPServer.ksh
16:22:59.653 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-18-HTTPServer.ksh
16:22:59.653 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.653 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-IBMHTTPD-6-1-release-18.ksh
16:22:59.654 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-IBMHTTPD-6-1-release-18.ksh
16:22:59.654 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/scripts
16:22:59.654 [7405736] <2> bpbkar SelectFile: INF - path = load-ihs-6-1-release-17-FP39-HTTPServerDiag.ksh.diag
16:22:59.654 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/scripts/load-ihs-6-1-release-17-FP39-HTTPServerDiag.ksh.diag
16:22:59.654 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.654 [7405736] <2> bpbkar SelectFile: INF - path = config
16:22:59.655 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/
16:22:59.655 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config
16:22:59.655 [7405736] <2> bpbkar SelectFile: INF - path = HTTPServer1
16:22:59.656 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer1/
16:22:59.656 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer1
16:22:59.656 [7405736] <2> bpbkar SelectFile: INF - path = install1-ihs-39.txt
16:22:59.658 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer1/install1-ihs-39.txt
16:22:59.658 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer1
16:22:59.658 [7405736] <2> bpbkar SelectFile: INF - path = install1-plg-39.txt
16:22:59.659 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer1/install1-plg-39.txt
16:22:59.659 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer1
16:22:59.659 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-ihs-install.txt
16:22:59.659 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer1/responsefile1-ihs-install.txt
16:22:59.659 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer1
16:22:59.659 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-ihs-PLG-install.txt
16:22:59.663 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer1/responsefile1-ihs-PLG-install.txt
16:22:59.663 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer1
16:22:59.663 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-updiinstaller-ihs.txt
16:22:59.664 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer1/responsefile1-updiinstaller-ihs.txt
16:22:59.664 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config
16:22:59.664 [7405736] <2> bpbkar SelectFile: INF - path = HTTPServer2
16:22:59.665 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer2/
16:22:59.665 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer2
16:22:59.665 [7405736] <2> bpbkar SelectFile: INF - path = install1-ihs-39.txt
16:22:59.665 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer2/install1-ihs-39.txt
16:22:59.665 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer2
16:22:59.665 [7405736] <2> bpbkar SelectFile: INF - path = install1-plg-39.txt
16:22:59.666 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer2/install1-plg-39.txt
16:22:59.666 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer2
16:22:59.666 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-ihs-install.txt
16:22:59.667 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer2/responsefile1-ihs-install.txt
16:22:59.667 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer2
16:22:59.667 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-ihs-PLG-install.txt
16:22:59.668 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer2/responsefile1-ihs-PLG-install.txt
16:22:59.668 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServer2
16:22:59.668 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-updiinstaller-ihs.txt
16:22:59.668 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServer2/responsefile1-updiinstaller-ihs.txt
16:22:59.668 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config
16:22:59.668 [7405736] <2> bpbkar SelectFile: INF - path = HTTPServerDiag
16:22:59.669 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServerDiag/
16:22:59.669 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServerDiag
16:22:59.669 [7405736] <2> bpbkar SelectFile: INF - path = install1-ihs-39.txt
16:22:59.670 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServerDiag/install1-ihs-39.txt
16:22:59.670 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServerDiag
16:22:59.670 [7405736] <2> bpbkar SelectFile: INF - path = install1-plg-39.txt
16:22:59.671 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServerDiag/install1-plg-39.txt
16:22:59.671 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServerDiag
16:22:59.671 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-ihs-install.txt
16:22:59.672 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServerDiag/responsefile1-ihs-install.txt
16:22:59.672 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServerDiag
16:22:59.672 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-ihs-PLG-install.txt
16:22:59.673 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServerDiag/responsefile1-ihs-PLG-install.txt
16:22:59.673 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/HTTPServerDiag
16:22:59.673 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-updiinstaller-ihs.txt
16:22:59.674 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/HTTPServerDiag/responsefile1-updiinstaller-ihs.txt
16:22:59.674 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config
16:22:59.674 [7405736] <2> bpbkar SelectFile: INF - path = IBMHTTPD
16:22:59.674 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/
16:22:59.675 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.675 [7405736] <2> bpbkar SelectFile: INF - path = install1-ihs-39.txt
16:22:59.675 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/install1-ihs-39.txt
16:22:59.675 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.676 [7405736] <2> bpbkar SelectFile: INF - path = install1-plg-39.txt
16:22:59.677 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/install1-plg-39.txt
16:22:59.677 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.677 [7405736] <2> bpbkar SelectFile: INF - path = install-ihs-39-IBMHTTPD.txt
16:22:59.677 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/install-ihs-39-IBMHTTPD.txt
16:22:59.677 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.677 [7405736] <2> bpbkar SelectFile: INF - path = install-plg-39-IBMHTTPD.txt
16:22:59.678 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/install-plg-39-IBMHTTPD.txt
16:22:59.678 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.678 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-ihs-install.txt
16:22:59.679 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/responsefile1-ihs-install.txt
16:22:59.679 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.679 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-ihs-PLG-install.txt
16:22:59.680 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/responsefile1-ihs-PLG-install.txt
16:22:59.680 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.680 [7405736] <2> bpbkar SelectFile: INF - path = responsefile1-updiinstaller-ihs.txt
16:22:59.680 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/responsefile1-updiinstaller-ihs.txt
16:22:59.680 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.680 [7405736] <2> bpbkar SelectFile: INF - path = responsefile-ihs-IBMHTTPD.txt
16:22:59.682 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/responsefile-ihs-IBMHTTPD.txt
16:22:59.682 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.682 [7405736] <2> bpbkar SelectFile: INF - path = responsefile-ihs-PLG-IBMHTTPD.txt
16:22:59.692 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/responsefile-ihs-PLG-IBMHTTPD.txt
16:22:59.692 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/config/IBMHTTPD
16:22:59.692 [7405736] <2> bpbkar SelectFile: INF - path = responsefile-updiinstaller-ihs-IBMHTTPD.txt
16:22:59.693 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/config/IBMHTTPD/responsefile-updiinstaller-ihs-IBMHTTPD.txt
16:22:59.693 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.693 [7405736] <2> bpbkar SelectFile: INF - path = .ssh
16:22:59.694 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/.ssh/
16:22:59.694 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra/.ssh
16:22:59.694 [7405736] <2> bpbkar SelectFile: INF - path = known_hosts
16:22:59.694 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/.ssh/known_hosts
16:22:59.694 [7405736] <2> bpbkar SelectFile: INF - cwd = /appl/lsinfra
16:22:59.694 [7405736] <2> bpbkar SelectFile: INF - path = software
16:22:59.695 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/software/

 

SelectFIle :Is it the culprit?But only var backup is failing with error:13......
 

revaroo's picture

So is it hanging at this point?

16:22:59.694 [7405736] <2> bpbkar SelectFile: INF - path = software
16:22:59.695 [7405736] <4> bpbkar PrintFile: /appl/lsinfra/software/

 

If so, what is /appl/lsinfra/software? Does it need to be backed up? Is it live data that needs to be stopped before backing up?

2013's picture

Its taking good backup of /appl.We are facing issue only when /var backup started and it failed with 13 though we have enabled Multple Streams havinvg Backup Selection "All Local Drives"...All streams completed successfully except /var......Snpapshot attached......Need to know which /var files is causing issue..Any specific way to check those files which is causing issue.....

BAR_client.JPG appl.JPG
Marianne's picture

Create a separate policy for /var, rename current bpbkar log file, increase logging level on client to level 5, then kick off this backup.
Post log as file attachment.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

2013's picture

Basically 2 issues on same client:

a.All streams completed successfully except /var...Need to know which /var files is causing issue.

b.When /var backup failed bpbkar process hung on the client and as a result CPU utilization will get higher on client system....and if we cancel the/var backup even then the bpbbkar process showing hung on the client.We tried to kill-9 in many ways but no luck.....

 

2013's picture

Please find the attached bpbkar logs..though the backup is still running but it doesnot write any more....Sending that snapshot too...

client_snapshot.JPG
AttachmentSize
bpbkar_log_client.txt 152.48 KB
Nagalla's picture

hey,,,

what is your currnet exclude list?  ( their might be lot that we can exclude from /var)

and read time out ?( you can try increasing the timeout)

and when you run ls -R /var  see how much time its taking to compleate the command... ( should not be more than the read timeout value)

Marianne's picture

Last entries in bpbkar log:

 

16:15:02.022 [9961618] <2> bpbkar SelectFile: INF - cwd = /var/adm/cron

16:15:02.022 [9961618] <2> bpbkar SelectFile: INF - path = log
 
Logging stopped in less than a minute.....
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Nicolai's picture

GENERAL ERROR: The bpbkar process hangs when attempting to back up a NFS file system.

http://www.symantec.com/docs/TECH29475

You can use the procedure outlined in this technote to see what file/directory that is causing the bpbkar hang.

 

 

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

2013's picture

Hi All,

1.Please find the exclude file.

/tmp
/cdrom
/proc
/var/spool/sockets
core
*.arc
*.dbf
/var/tmp
/var/crash
/oradata/*
/oraarch/*
/oraredo/*
/opt/app/*/oradata*
/opt/app/*/oraarch*
/opt/app/*/oraredo*
/oraflsh/*
/opt/app/*/oraflsh*
 

2.Client read Timeout :1800

 

3.Also attaching the output of ls -R /var.It seems many var files we need to exclude....Please let me know what files we can exclude from /var......

 

AttachmentSize
output_of_var.txt 39.06 KB
2013's picture

Can i exclude the files in ls -R /var which is starting from :The file access permissions do not allow the specified action.

Nagalla's picture

hi,

exclude list is defined based on your organization /appteam/SA requirement..

 

below is the list that i generally use for /var , you can check them also for test, but before making exclude as perminent you need to check with your Appteam or SA team.to avoide future issues.. 
 
/var/crash
/var/adm/crash
/var/run
/var/tmp

###now for testing 

1) exclude entire /var from backup and test the backup to make sure /var is causing the failurs.

2) once you get the confirmation from  saying that the failure is because of /var, then start updating the above exclude list and see how it works..

and also.. 

I am seeing below erroes in the output file, did you run the command as root , even with the root its showing access isssue.?

if you did not run with root, run the command as root and see how much time its taking to compleate the command ls -R.. it should not be more then 1800.

 

ls: /var/websm/lwi/lib: The file access permissions do not allow the specified action.

/var/websm/lwi/logs:
ls: /var/websm/lwi/logs: The file access permissions do not allow the specified action.

/var/websm/lwi/native:
ls: /var/websm/lwi/native: The file access permissions do not allow the specified action.

/var/websm/lwi/rollback_config:
ls: /var/websm/lwi/rollback_config: The file access permissions do not allow the specified action.

/var/websm/lwi/runtime:
ls: /var/websm/lwi/runtime: The file access permissions do not allow the specified action.

/var/websm/lwi/security:
ls: /var/websm/lwi/security: The file access permissions do not allow the specified action.

/var/websm/lwi/update.images:
ls: /var/websm/lwi/update.images: The file access permissions do not allow the specified action.

hope this helps.. 

2013's picture

Hi Nagalla,

Thanks for the update.

1.Excluded /var and Backup completed successfully.

2.Regarding the exclude file i have checked on others servers its the same file.

3.I have not ran the command from root.Sending the output of ls -R/var which run from root and yes it completed very fast...

4.Also i want someone please check bpbkar logs and can tell which files we can exclude.

AttachmentSize
bpbkar_log_client.txt 152.48 KB
var output.txt 269.17 KB
2013's picture

Hi All,

As per my findings..i have seen bpbkar logs and i found the NBU is unable to read the below file:

16:15:02.019 [9961618] <2> bpbkar SelectFile: INF - path = corrals
16:15:02.019 [9961618] <4> bpbkar PrintFile: /var/adm/corrals/
16:15:02.019 [9961618] <2> bpbkar SelectFile: INF - cwd = /var/adm
16:15:02.019 [9961618] <2> bpbkar SelectFile: INF - path = cron
16:15:02.020 [9961618] <4> bpbkar PrintFile: /var/adm/cron/
16:15:02.020 [9961618] <2> bpbkar SelectFile: INF - cwd = /var/adm/cron
16:15:02.021 [9961618] <2> bpbkar SelectFile: INF - path = FIFO
16:15:02.021 [9961618] <4> bpbkar PrintFile: /var/adm/cron/FIFO
---------------------------------------------------------------------------------------------------------------------------------------------------

Below are the files which are creating issue:

/var/adm
/var/eksh.install.out
/var/ncs
/var/LicenseUseManagement
/var/VRTSat
/var/VRTSweb
/var/aacct

 I am thinking to put all these files in exclde list....If i am wrong then correct me....

SOLUTION
2013's picture

Issue is now fixed.Cron file was the culprit...

Marianne's picture

I have pointed out over here: https://www-secure.symantec.com/connect/forums/bpbkar-process-hung-client#comment-8094771 that logging stopped when /var/adm/cron was hit.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links