Agent crashing on Domino after upgrade to 12.
Updated: 25 Sep 2010 | 55 comments
We've been on 10d for quite a while, then a few weeks ago upgraded to 11d with no issues. When the notification of 12 came out I jumped on it, and I haven't had a good Domino backup since because the agent keeps crashing. The agents on our other non-Domino servers seem to be fine. I recall this being a common issue with 11d when it was first released. Here is the error that shows in the event log on the Domino server:
Faulting application beremote.exe, version 12.0.1364.0, faulting module bedsnote.dll, version 12.0.1364.0, fault address 0x00022040.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Any ideas? I'll go back to 11d if I have to, but I'd much rather get this one issue worked out and stick with 12.
discussion Filed Under:
Comments
Message Edited by Randolfini on 04-28-2008 02:19 AM
If anyone else has any solution or has a possibility to investigate together with Symantec please update this thread.
I have the same problem, with Domino 8.0.1 and BE-12. I used the agent for some weeks, without problems. Now I cannot backup the domino server because the agent crash. I edited the mentioned registry keys and I deleted ucj file, but I didn't solved the problem. Now I cannot browse domino database, and on Domino console, every minute, I see in the log this error: "c:\program files\symantec\belnapi.exe has terminated abnormally".
Event Source: Application Error
Event Category: (100)
Event ID: 1000
Date: 6/23/2008
Time: 8:46:08 PM
User: N/A
Computer: DOMINOMAIL01
Description:
Faulting application beremote.exe, version 12.0.1364.0, faulting module bedsnote.dll, version 12.0.1364.0, fault address 0x00022040.
Data:
0000: 41 70 70 6c 69 63 61 74 Applicat
0008: 69 6f 6e 20 46 61 69 6c ion Fail
0010: 75 72 65 20 20 62 65 72 ure ber
0018: 65 6d 6f 74 65 2e 65 78 emote.ex
0020: 65 20 31 32 2e 30 2e 31 e 12.0.1
0028: 33 36 34 2e 30 20 69 6e 364.0 in
0030: 20 62 65 64 73 6e 6f 74 bedsnot
0038: 65 2e 64 6c 6c 20 31 32 e.dll 12
0040: 2e 30 2e 31 33 36 34 2e .0.1364.
0048: 30 20 61 74 20 6f 66 66 0 at off
0050: 73 65 74 20 30 30 30 32 set 0002
0058: 32 30 34 30 2040
Event Source: Backup Exec
Event Category: None
Event ID: 65215
Date: 6/18/2008
Time: 7:16:30 PM
User: N/A
Computer: DOMINOMAIL01
Description:
The connection to the Lotus Domino server has been lost. The following error was reported: Notes Agent:CLNFileSystem::AttachToDLE - Access Violation Exception Caught. The job that was running on this server has been stopped.
http://eventlookup.veritas.com/eventlookup/EventLookup.jhtml
We have the Agent crashing problem, too. We don't even use the LD db backup functionality only doing file based backups (LD service is stopped before and restarted after backup). I'll change the registry entries and see whats happening...
Having the same issues here with Lotus Agent. Initially had the problem of not being able to browse databases and select them but the registry hack works for that.
Now BackupExec has started reporting files as corrupt which are fine and have no issues (eg mailjrn.ntf / perfweb.ntf). So have had to delete and remove these only to now have the servers say they can no longer attach the remote agent even though they are backing up themselves.
This is a real problem for us as we have over 500+ domino servers out in the fiekld and this is stopping us upgrading them to v8.
Have been working with Symantec support but they aint been great so far and are being VERY tight lipped overthe problems and not commenting on any future potential hotfixes to resolve these issues. One thing to note one of our site removed Windows 2003 SP2 (going back to just SP1) from one of it's servers having the problem and Backup Exec 12 has worked flawlessly since. Will let you know if I get anythign from Symantec.
Right off to kick our Symantec Account Managers bottom over this!
Ok I've just come off the phone with a Symantec Engineer and they advised the following:-
Symantec has no intentions of supplying a hotfix for this problem that can be fixed by editing the registry. You will have to create a reg file that can be imported/run on each manchine and do it manually. The next release of Backup Exec 12 will have this issue fixed though.
So there you go folks.
I was provided a solution from a Symantec tech last week and so far I have had all successful backups. Here is what we did:
1. Stop the Backup Exec remote agent service on the Domino server
2. Go into the registry and browse to the following path:
HKLM\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\DOMINO
3. Change the following 2 DWORD values:
Enable Change Journal = 0
Enable Notes Database Filtering = 0
4. Close the registry
5. Browse to C:\Program Files\Symantec\Backup Exec\RAWS\logs and delete NOTESDB*.UCJ files
6. Restart the Backup Exec remote agent service.
Note: Do not change the registry settings back to there original settings. Every article that I found had me change the registry settings back. The Symantec tech told me that the changes will not affect the Domino backups.
Works now.
In our case the agent started the Domino backup functions (two sub processes to beremote.exe started) because not all Domino services had stopped and crashed.
For other users who wants to do an offline Domino DB backup: remember to disable the global exclusion list of BE server and set the reg key for the agent to allow offline backups. Otherwise no DB files (*.nsf) will be backed up.
I did an install of the remote agent on the Notes server itself from the media.
My test backups are running with NO problem, but I'll post the result of the "normal" backup here....
hi, any updates on your tes backups? thanks.
I have the same beremote crash problem on a new Domino 7.03/Win 2003 R2 SP2 system. I'm doing this now and will report back.. Ron
1. Stop the Backup Exec remote agent service on the Domino server
2. Go into the registry and browse to the following path:
HKLM\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\DOMINO
3. Change the following 2 DWORD values:
Enable Change Journal = 0
Enable Notes Database Filtering = 0
4. Close the registry
5. Browse to C:\Program Files\Symantec\Backup Exec\RAWS\logs and delete NOTESDB*.UCJ files
6. Restart the Backup Exec remote agent service.
Note: Do not change the registry settings back to there original settings. Every article that I found had me change the registry settings back. The Symantec tech told me that the changes will not affect the Domino backups.
Thanks rlj for the update. Yes, please do send an update. I am still using 11d for the RAWS and 12 for BEWS.
Well... The Notes database backup appears to work, but the console messages (RAWs terminated) continue. I have 11D running on a Windows file server, but without a Notes license there. If I copy the 11D RAWS and install it on the Notes server, will the v12 backup server then backup the Notes server? Ron
Hi all,
I face the same problem after migrating to version 12.5 in ordder to backup Domino 8 that is supported only since version 12 of Backupexec. The domino console was pollut by a message regarding benlapi.exe terminated abnormally and was not possible to browse the selections list for Notes databes.The backup job failed in the same time.
After reading all the post for Domino agent , i made a mix of everything. To resume, i remove the agent from the panel control/add&remove program and reboot the notes server. After reboot, i install the remote agent directly on the server. Made the change in the registry:
"global exclusion list" means here "Active File Exclusion". See 259152
I just wanted to report my expirence with this same EXACT problem.
I upgraded from 11d to 12/12.5. Around this time we installed a domino 7 server in our office as a backup server. I was getting the same error messages and event logs people listed above
Since the upgrade to 12.5 I have never gotten a 100% sucessful backup of my domino server.
I had googled the error codes and stumbled across this page.
I can confirm that the following fix WORKED for me:
System is a 2003 server ent ed 32bit with sp2
2. Go into the registry and browse to the following path:
HKLM\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\DOMINO
3. Change the following 2 DWORD values:
Enable Change Journal = 0
Enable Notes Database Filtering = 0
4. Close the registry
5. Browse to C:\Program Files\Symantec\Backup Exec\RAWS\logs and delete NOTESDB*.UCJ
At first I only noticed the part to make the registry changes, did that and rebooted. No luck backup still failed. Then I read the part about deleting the UCJ files. Deleted them restarted the backup exec service and kicked off a backup.
I just got a 100% success backup.
Hi all! Looks like the problem still exist with 12.5, strang that they can't fix it....
When You say that the backup is 100% after regfix, have the agent utility in autorun do you do Full backups or any Incrementals?
We hade problems only when running incremental backups.
//Mats
Hello everyone,
I see that this may be an old thread but I would like to resurrect it if I may.
I have done everything mentioned within this thread (RegKeys.. Etc) and at this point in time I have spoken with three different Symantec technicians with 0 resolve.
Win03 server / Domino 08/ Notes 08/ Symantec BE v12.0
I am having the belnapi.exe is abnormally terminating error displayed on the Domino console. Within the failed full backup failure job log I am getting the error message:
Job ended: Saturday, November 29, 2008 at 12:30:53 AM
Completed status: Failed
Final error: 0xe000fe30 - A communications failure has occurred.
Final error category: Server Errors
"For additional information regarding this error refer to link V-79-57344-65072
This is occurring on 40 of our servers with the same setup described above and even the Symantec Technicians are having problems with this. I just sent a log to a technician but have heard nothing. I can increment the backups but once a full backup is performed it always ends with failure, "A communications failure has occurred". I cannot have this on these servers as the data is crucial.
We do have one server on Symantec v12.5 and this server has had zero issues but within this very thread a user has been having problems - so this may still be an issue.
Is this happening for servers that do not have Lotus Domino installed on them or is it just us?
Has anyone resolved this issue or any new advise? S.O.S.
Thanks in advance,
D
The 12.5 upgrade appears to have fixed the problem for me.. Ron
rlj - This may be the case but Cyberfed has not gotten a single full backup on BE v12.5; I wonder if he is getting the same issues as we all are or different one.
I have also found 0 documentation saying this has been fixed in v12.5 Do you know where I can find this, so that I can provide it to my CEO ETC.. in order to upgrade once more. I am also sure that Symantec will allow for a free upgrade seeing how we purchased the software very recently.
- D
Can anyone; user, client or Symantec technician alike provide me with documentation of what was fixed in Symantec Backup Exec v12.5 that was broken and not addressed as a hot fix for v12.0?
Also can anyone else confirm after an upgrade from v12.0 to v12.5 you are not receiving successful backups?
Thanks to anyone and everyone reading this post (Distress signal)
- D
We were having the same issue with 12 after re-installing the remote agent and made the change to the registry from this thread and it solved our issue.
Any news on this issue?
I have the same problem!
BE 12.0 SP2
Domino 8.0.1
Domino Agent
Some additional info:
1) The 2 registry settings:
- Enable Notes Database Filtering - During a browse of Domino databases beneath the Notes icon in BEWS, for every *.nsf and *.ntf found, an API call for database open and database close are sent to Domino to determine if we've found a 'real' database. This is used in displaying what is available for backup. Turning this off simply allows selection all *.nsf and *.ntf files, whether real databases or not.
- Enable Change Journal - As stated earlier, this selects whether the agent depends on the Windows Change Journal to track database changes or not. Along with this setting, the .UCJ file keeps track of databases. Setting this setting to '0' causes the agent to use complete file scans to determine new and changed DBs. If there is corruption in the Change Journal, it is usually best to delete the UCJ at the same time you turn off this setting.
Because the cause is usually some issue with Change Journal, and of the ease of workaround for the agent crash, the issue has not been addressed in any patch or update as far as I know. The Technote for this is http://support.veritas.com/docs/302505. Was this Technote not found or not helpful? If not, are there additional keywords we can add to the Technote to make it easier to find?
2) About Belnapi.exe - The Remote Agent uses Belnapi.exe to communicate with Domino. Because Belnapi remains running at all times with a connection to Domino, if for any reason the Remote Agent stops (crash, end-process, etc.) Domino detects this and starts with the console error messages about belnapi 'has terminated abnormally'. These don't hurt anything but quickly fill up the console. A restart of Domino is required to get rid of them.
Hope this helps!
Russ
2) About Belnapi.exe - The
2) About Belnapi.exe - The Remote Agent uses Belnapi.exe to communicate with Domino. Because Belnapi remains running at all times with a connection to Domino, if for any reason the Remote Agent stops (crash, end-process, etc.) Domino detects this and starts with the console error messages about belnapi 'has terminated abnormally'. These don't hurt anything but quickly fill up the console. A restart of Domino is required to get rid of them.
This is my main problem right now! Its filling the console 24/7. Restart Domino!? How often? I have over 100+ servers all around the world with 24/7 wanted upptime...
2009-03-12 10:17:22 Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:18:22 Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:19:22 Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:20:22 Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:21:22 Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:21:24 Admin Process: Sea
2009-03-12 10:22:22 Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:23:22 Process C:\Program
(7728/0x1E30) has terminated abnormally
2009-03-12 10:24:22 Process C:\Program
(7728/0x1E30) has terminated abnormally
Mikael wrote: "This is my
Mikael wrote:
"This is my main problem right now! Its filling the console 24/7. Restart Domino!? How often? I have over 100+ servers all around the world with 24/7 wanted upptime..."
Any time the Remote Agent crashes or is terminated by user while belnapi.exe is running with a connection to Domino, the messages will show up in the console...but they are not normal and you are not expected to 'live with it'. I was simply explaining what causes them and how to get rid of them. First, of course, it is not normal for the Remote Agent to crash. If that happens, the reason must be found and corrected, possibly/probably with the help of tech support. Second, if belnapi.exe is running, never terminate it via task manager or Drwatson32 -P. If the Remote Agent needs to be stopped for some reason, use the Services applet.
Russ
I tried to follow yoru steps
I tried to follow yoru steps 1- 4 but I am unable to find NOTESDB*.UCJ files in Symantec Backup Exec media server & Domino server ..
Therefore, I didn't managed to backup Lotus Domino database as backup drive of Domino server will automkatically exclude Lotus Domino databases.
Please help
Hoay Fern
Russ ...
We are recieving the same error on domino 8.5. I'm sure you guys have been asked this by now but I will ask it anyway. Is it ok to try this fix on an 8.5 server recieving the same errors?
Fyi for others on 8.5
This seems to be working for us now.
From a customer point of view I am disappointed at the support for 8.5 not being ready on the release of the product. Not trying to be negative but it really is something that I would expect to have been tested and ready before a production release of software from a big vendor. Just something you may want to bring up at any "what are you hearing from customers" meetings.
Any Updates
Hi
I appreciate this is an old (ish) thread but is there any update yet from Symantec as the impression I get from this thread that there is still no REAL fix..? Mixed bag of results! The problem still exists for me after trying EVERYTHING mentioned!
Help appreciated.
Resolution
implementing all the forementioned registry changes did not work for me until I changed the Resource Order. By Default Symantec will back up any drives prior to the Lotus Domino Databases. I pushed up the Lotus Domino Databases to first in the Resource order and have had successful backups since....
Apologies if this is already known. Just thought i would mention it in case it helps anyone else...
Good luck
Any solution?
I have been getting this problem sporadically for a long time. I applied the registry changes with no luck. Just now I did a local install of the agent (instead of a push install from the media server) as suggested in a previous post. As for the resource order, I used to have my Domino backup as first in sequence and it happened then as well. I changed it to last and the same thing happens.
BEX Server and Domino Server are W2K3 STD SP2
Bex is v12.5 Rev 2213
BEXRA is 12.5.2213
Domino is 6.5.3
RTFM
So I re-read the posts and went to http://seer.entsupport.symantec.com/docs/302505.htm. This time I changed the registry entries on the media server. Up to now I had just changed the Domino server registry.
Fingers crossed!
Hi John I haven't seen
Hi John
I haven't seen support cases come up on this issue for a long time so i don't know if it is still a problem. One item of note on your environment is that with Domino 6.5.3 and Backup Exec 12.5 you are in an untested/unsupported configuration (ftp://exftpp.symantec.com/pub/support/products/Backup_Exec_for_WindowsNT/307063.pdf). There may not be much difference between 6.5.3 and 7.x with Domino agent backups but it's untested.
In general, if anyone is still experiencing issues in a supported environment with the Domino agent as described in this thread that are not fixed with the registry settings, you must have a support case opened and escalated.
Agent crashing on Domino
Russ,
Thanks for the reply. The problem is still cropping up from time to time but hasn't happened now for a few weeks. We badly need to upgrade to Domino 8 but there's so much other stuff going on (what's new?) that I haven't had the time. I'm also looking at upgrading to BEX 2010 and getting the deduplication agent so lot's to do.
Thanks again,
John
Would you like to reply?
Login or Register to post your comment.