Bug:  1950048  

Symptom(s):
Backups are failing with status 233. bpdbm core dumps during add_files, stack shows that its in NBE_NameWrapper::addNewName

Log Files:
bpbrm shows a premature end of file encountered failure coming from db_FLISTsend -
15:03:42.482 [22656] <2> db_end: Need to collect reply
15:03:46.716 [22656] <2> get_long: (2) premature end of file (byte 1)
15:03:46.716 [22656] <2> db_getdata: get_string() failed: I/O error (5) premature end of file encountered (-9)
15:03:46.716 [22656] <2> db_end: no DONE from db_getreply(): premature eof encountered
15:03:46.716 [22656] <2> db_FLISTsend: db_end_sts() failed: premature eof encountered (233)
15:03:46.716 [22656] <16> bpbrm main: db_FLISTsend failed: premature eof encountered (233)


bpdbm log shows that the last function called was add_files -
16:46:28.193 [2320] <2> image_path_tmp_r: ?
16:46:28.193 [2320] <2> getIDIRSTRUCT: ?
16:46:28.193 [2320] <2> image_dir_name: ?
16:46:28.216 [2320] <2> add_files: creating /usr/openv/netbackup/db/images/<client>/<ctime>/tmp/<image>.f


bpdbm core dumped during a call to add_files, stack trace shows that its in NBE_NameWrapper::addNewName.
Run pstack command to get the stack trace -

# pstack <core>
core '/core' of 2320:   /usr/openv/netbackup/bin/bpdbm
ffffffff7ec2a5f0 int NBE_NameWrapper::addNewName(const char*,unsigned,int)
(100945f88, ffffffff7ffee21e, 47, ffffffff7ea03738, 4, f6) + 268
ffffffff7ec2a7b0 void NBE_NameWrapper::addName(const char*,unsigned)
(100945f88, ffffffff7ffee21e, 4, 0, ffffffff726f3eac, 1) + 10
ffffffff7ec48674 void NBE_CatImage::updateRecordData(int,Cat_Entry_Rec*)
(100924540, 2e75, ffffffff7ffedfb0, 20578, 20400, 20578) + 16c
ffffffff7ec487d8 void NBE_CatImage::addRecordData(int,Cat_Entry_Rec*,int,int)
(100924540, 2e75, ffffffff7ffedfb0, ffffffff80002997, 4dd, 100944ab8) + 60
ffffffff7ec40870 int NBE_CatImage::addNewNode(Cat_Entry_Rec*,int*) (100924540,
ffffffff7ffedfb0, 20570, 10087dec0, 20400, 80002997) + c58
ffffffff7ec2d624 CatImg_put_file_rec (100537af0, 100840ee0, 1006ce6b0,
1006ce6e3, 0, 32) + 414
00000001000dd694 add_files (ffffffff7fff9098, ffffffffffffffed,
ffffffff7fff90f0, 173a8, 2e75, 0) + 1238
00000001000d9428 image_db (4e, a, 1002cc000, 2, 1004501e4, 1002cc3e8) + be8
00000001000b53f8 process_request (a, 4e, 25, 0, 0, 1) + 778
00000001000b3e00 listen_loop (100400, 15180, 8, 4b5e1fbb, 0, 0) + 11bc
00000001000b1f0c main (100455, 100460990, 10010d000, 100400, 100400, 0) + 1980
00000001000afdbc _start (0, 0, 0, 0, 0, 0) + 17c


Workaround:
There is no known workaround for this issue at this time.

Fix:
The problem occurs when trying to access the UserGroupHashTable catstore file.
bpdbm tries to read a block of memory that had previously been backed by the
memory-mapped file, but which has since been released.

bpdbm's memory-mapped files keep several "windows" in memory at once, but when it
has too many, it releases some. The first window of the UserGroupHashTable file
is assumed to always be in memory. At some point, it is removed from memory,
but bpdbm still continues to point to that freed memory.

Re-loading the desired window into memory each time is needed to resolve the problem.\
NDMP hash table may have the same problem.


ETA of Fix:
This issue is resolved in the NB_7.0_ET1950017_1_355095 Hotfix for NetBackup 7.0 Master Servers, available in the Related Documents section, below.