Status 84 media write error on NBU 7.5
Created: 29 Jan 2013 | Updated: 29 Jan 2013 | 34 comments
2 node clustered master server running nbu 7.5 on rhel6. 4 other media servers.
currently having an issue with tape backups. its a new setup and not in production. i recently installed evrything. i tested my disk backups and they ran successful and the test was to backup one of the filesystem of the media server. i had no issues backing up one of the media server. i dont have any client currentlty in the environment.
the topology of the environment is to have the tape backups only from the clustered master server. and all disk backups from the media server.
01/25/2013 13:51:17 - Info nbjm (pid=28809) starting backup job (jobid=14) for client media01.domain.com, policy policy_1_asp, schedule tmp_bkup
01/25/2013 13:51:17 - Info nbjm (pid=28809) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=14, request id:{F744DA40-6730-11E2-BCAD-2D8C392DDC40})
01/25/2013 13:51:17 - requesting resource nbu-hcart2-robot-tld-0
01/25/2013 13:51:17 - requesting resource nbu.domain.com.NBU_CLIENT.MAXJOBS.media01.domain.com
01/25/2013 13:51:17 - requesting resource nbu.domain.com.NBU_POLICY.MAXJOBS.policy_1_asp
01/25/2013 13:51:18 - Info bpbrm (pid=29725) media01.domain.com is the host to backup data from
01/25/2013 13:51:18 - Info bpbrm (pid=29725) reading file list from client
01/25/2013 13:51:18 - Info bpbrm (pid=29725) starting bpbkar on client
01/25/2013 13:51:18 - Info bpbkar (pid=49560) Backup started
01/25/2013 13:51:18 - Info bpbrm (pid=29725) bptm pid: 29727
01/25/2013 13:51:18 - granted resource nbu.domain.com.NBU_CLIENT.MAXJOBS.media01.domain.com
01/25/2013 13:51:18 - granted resource nbu.domain.com.NBU_POLICY.MAXJOBS.policy_1_asp
01/25/2013 13:51:18 - granted resource M00001
01/25/2013 13:51:18 - granted resource Drive000
01/25/2013 13:51:18 - granted resource nbu-hcart2-robot-tld-0
01/25/2013 13:51:18 - estimated 0 kbytes needed
01/25/2013 13:51:18 - Info nbjm (pid=28809) started backup (backupid=media01.domain.com_1359147078) job for client media01.domain.com, policy policy_1_asp, schedule tmp_bkup on storage unit nbu-hcart2-robot-tld-0
01/25/2013 13:51:18 - started process bpbrm (pid=29725)
01/25/2013 13:51:18 - connecting
01/25/2013 13:51:18 - connected; connect time: 0:00:00
01/25/2013 13:51:19 - Info bptm (pid=29727) start
01/25/2013 13:51:19 - Info bptm (pid=29727) using 65536 data buffer size
01/25/2013 13:51:19 - Info bptm (pid=29727) using 30 data buffers
01/25/2013 13:51:19 - Info bptm (pid=29727) start backup
01/25/2013 13:51:19 - Info bptm (pid=29727) backup child process is pid 29737
01/25/2013 13:51:19 - Info bptm (pid=29727) Waiting for mount of media id M00001 (copy 1) on server nbu.domain.com.
01/25/2013 13:51:19 - mounting M00001
01/25/2013 13:52:04 - Info bptm (pid=29727) media id M00001 mounted on drive index 0, drivepath /dev/nst3, drivename Drive000, copy 1
01/25/2013 13:52:04 - mounted M00001; mount time: 0:00:45
01/25/2013 13:52:04 - positioning M00001 to file 1
01/25/2013 13:52:53 - Error bptm (pid=29727) write error on media id M00001, drive index 0, writing header block, Input/output error
01/25/2013 13:52:53 - Info bptm (pid=29727) EXITING with status 84 <----------
01/25/2013 13:52:53 - Error bpbrm (pid=29725) from client media01.domain.com: ERR - Cannot write to STDOUT. Errno = 104: Connection reset by peer
01/25/2013 13:52:54 - Info bpbkar (pid=49560) done. status: 84: media write error
01/25/2013 13:52:54 - end writing
01/25/2013 13:53:38 - job 14 was restarted as job 15
media write error (84)
logs to be posted in the following post. if any required. like bptm. bpbrm. bpbkar.
Discussion Filed Under:
Comments 34 Comments • Jump to latest comment
[root@master01 bin]# ./tpconfig -d Id DriveName Type Residence Drive Path Status **************************************************************************** 0 Drive000 hcart2 TLD(0) DRIVE=4 /dev/nst3 UP 1 Drive001 hcart2 TLD(0) DRIVE=3 /dev/nst2 UP 2 Drive002 hcart2 TLD(0) DRIVE=2 /dev/nst1 UP 3 Drive003 hcart2 TLD(0) DRIVE=1 /dev/nst0 UP Currently defined robotics are: TLD(0) robotic path = /dev/sg1 EMM Server = nbu.domain.com [root@master01 bin]# ./vmoprcmd -d PENDING REQUESTS DRIVE STATUS Drv Type Control User Label RecMID ExtMID Ready Wr.Enbl. ReqId 0 hcart2 TLD - No - 0 1 hcart2 TLD - No - 0 2 hcart2 TLD - No - 0 3 hcart2 TLD - No - 0 ADDITIONAL DRIVE STATUS Drv DriveName Shared Assigned Comment 0 Drive000 No - 1 Drive001 No - 2 Drive002 No - 3 Drive003 No - [root@master01 bin]# ./scan -tape ************************************************************ *********************** SDT_TAPE ************************ ************************************************************ ------------------------------------------------------------ Device Name : "/dev/nst3" Passthru Name: "/dev/sg5" Volume Header: "" Port: -1; Bus: -1; Target: -1; LUN: -1 Inquiry : "HP Ultrium 5-SCSI I5CS" Vendor ID : "HP " Product ID : "Ultrium 5-SCSI " Product Rev: "I5CS" Serial Number: "STU" WWN : "" WWN Id Type : 0 Device Identifier: "" Device Type : SDT_TAPE NetBackup Drive Type: 10 Removable : Yes Device Supports: SCSI-6 Flags : 0x0 Reason: 0x0 ------------------------------------------------------------ Device Name : "/dev/nst2" Passthru Name: "/dev/sg4" Volume Header: "" Port: -1; Bus: -1; Target: -1; LUN: -1 Inquiry : "HP Ultrium 5-SCSI I5CS" Vendor ID : "HP " Product ID : "Ultrium 5-SCSI " Product Rev: "I5CS" Serial Number: "PQR" WWN : "" WWN Id Type : 0 Device Identifier: "" Device Type : SDT_TAPE NetBackup Drive Type: 10 Removable : Yes Device Supports: SCSI-6 Flags : 0x0 Reason: 0x0 ------------------------------------------------------------ Device Name : "/dev/nst1" Passthru Name: "/dev/sg3" Volume Header: "" Port: -1; Bus: -1; Target: -1; LUN: -1 Inquiry : "HP Ultrium 5-SCSI I5CS" Vendor ID : "HP " Product ID : "Ultrium 5-SCSI " Product Rev: "I5CS" Serial Number: "ABC" WWN : "" WWN Id Type : 0 Device Identifier: "" Device Type : SDT_TAPE NetBackup Drive Type: 10 Removable : Yes Device Supports: SCSI-6 Flags : 0x0 Reason: 0x0 ------------------------------------------------------------ Device Name : "/dev/nst0" Passthru Name: "/dev/sg2" Volume Header: "" Port: -1; Bus: -1; Target: -1; LUN: -1 Inquiry : "HP Ultrium 5-SCSI I5CS" Vendor ID : "HP " Product ID : "Ultrium 5-SCSI " Product Rev: "I5CS" Serial Number: "XYZ" WWN : "" WWN Id Type : 0 Device Identifier: "" Device Type : SDT_TAPE NetBackup Drive Type: 10 Removable : Yes Device Supports: SCSI-6 Flags : 0x0 Reason: 0x0i just fired a new backup for the loggings
bpbrm on the media server/client:
bptm on the master active server's node master01 to be followed as in attachement at verbose 5.
bptm log at verbose 5. on the active node of master (master01) (virtual name nbu) has some <16>s in there. :(
bpbrm also attached from the master/media.
Are there any relevant messages in /var/log/messages, especially something relating to SCSI reservation?
I assume you are using some sort of VTL(odd drive serial number!). Is it functional at OS level?
Mount tapes from robtest, and try to read/write with OS tar command if possible.
Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan
no, i actually edited the names of the serial drives, are they important to be mentioned? ok, let me try the robtest. am not sure of the tar thingie to do. :S.
and no we're not using VTL. its physical eml245e tape library..
I am assuming that you have edited that output? especially as pointed out with regards to the serial numbers?
I first thoughts here are:
Wrong media type - LTO3 or lower in an LTO5 drive
WORM tape
Write Protected tape
Tape used in another backup system and no overwrite set (though this hould have been logged and it is not - so unlikely)
Tell us about your hardware and tapes so that we can assist further - it just cannot write a header to the tape so that is where to look
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
yes your assumption is corrected. serial numbeers with other names have been edited. even the IPs. yes they're all LTO5 tape drives. how do i correct this now? WORM tape? let me see research on that and write protected tape aswell. these tapes are not used in another backup system. this is new environment. and we have put in 5-6 tapes for testing only. it highly possible that oeverwrite is not set since i never did this. its all new.
hardware. library is eml 245 with 4 lto5 drives. with encryption. tapes are as you can see in those output above. if not i'll get them for you in a moment.
Just re-read this thread and noticed what i believe is an issue ....
If what i have gathered is correct you have only disk on the Media Servers and only tape on the Master.
So ... first point is that if you configure tapes on a Master you have to configure it for each node - so configure it whilst running n node 1 the fail over and configure on node 2 so that both nodes are configured - both nodes also need to be able to properly resolve the short and FQDN name of the virtual server
Second point .. and the real problem you have ... is that you are running NetBackup 7.5 with a clustered Master which is fine ... but then you add tape to it which also makes it a media server .. and clustered media servers are no longer supported in NetBackup 7.5 .. so you cannot use disk or tape on the Master - it just needs to be a Master.
From the NetBackup 7.5 Release notes:
NetBackup 7.5 media server installations cannot be clustered. However, you
can upgrade existing NetBackup 6.x clustered media servers to version 7.5 and
they remain clustered.
Sorry but your system is not supported so you need to replan you topology - if i have read you first thread correctly anyway
Hope this helps?!
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
very interesting and we never realised this released note fact. yes you have gathered it correct. master 2 node cluster on tapes only. and media server for disk only.
to your 1st point: yes both the nodes are configured for tapes. both can be resolved. this is not the issue in our case.
to your 2nd point. i agree on that, please tell me one thing, is this not suported here mean that its not supported by the symantec as yet, or it is impossible todo with this realese of 7.5? i mean not now, but may be in a month or two they might start supporting it? i have not configured disk on the master. and only through media server only and backups worked fine. nice. so that means. i can ask for an HBA(since we dont have them on any media server) on a media server and then we can have tapes backups too right? just cause of this blunder that we've made. would it be wrong to tell the upper management or whoever responsible to take out the existing HBA from the master and have it on one of the media servers?
why did sort risk assesment pick up this fact that it cannot be of media server role here in our environment. grr... what to do now?
OK - so you do have an issue!
I don't think that it wouldn't work, although you are clearly having problems, as it does say that if you have a 6.5 clustered Media Server that they would still support it after a 7.5 upgrade - but i can see even that being withdrawn in the next release (7.6 or 8)
They are unlikely to ever support it now as they actually withdrew support for clustered Media Servers in 7.5
So as this is a new system i would get away from you current toplogy and, presuming the other media servers are not clustered, put the tape onto one of those.
Better still use fibre and a switch and share the drives between your media servers so that they can all write theor own data to tape.
So keep your clusetered Master as just a Master and the media servers as media server and you will be all good - shared storage option to share the tape drives would really be the way to go
Hope this helps
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
hmmm... ok. i can forward a recommendation to the folks here,
but the thing is, can i make this work atleast? i haven't done any configuratino yet, thats the reason i believe its not working. its totally un-configured. all i did was run the wizard to nind tape drives and thats it. tape drives and robot was visible on the os. so had no issue there. now im getting these write errors. because of what? not because its not supported. but because there is something still not configured.
how can i check if i still have ths hared storage option license installed. wat to look for?
thanks for your advice. i have forwarded the recommendation. can you please also tell, how HBAs do i need for this? he says he's gonna order few right away. do i need 2 on each media server . or 2 on 1 media server would only? currently there are two coming our way ina few days. or its just one HBA needed foer the HBA media server?
OK - you have 4 tape drive plus disk
If it is just the tape drive to connect and they look to be LTO5 then you dont want more than 2 per HBA port - so maybe a dual port HBA for each media server would do the trick for you.
I am assuming that the disk is connected else where - either way you dont want the disk and the tape drives on the same HBA port.
For the SSO option it depends how you are licensed. If you have a capacity license then they are covered in that, if you have traditional licensing then ask your symantec partner that sells you the licenses to get an IBR report from Symantec
You need to do this in writing (an email is fine)
An IBR report is an Install Base Report and will tell you exactly what licenses you own for the NetBackup so you can see if you have SSO, but being a new project of a reasonable size you may well have a capacity licnsese - there should be one or more pdf files that were sent through with the license keys in.
You can also open the admin console and use Help - License keys to see if the Shared Storage Open shows up - but you still need to know how many you have as you need one per drive (so need 4)
You will also need everything to go through a fibre switch so that all media servers get access to all four tape drive
What you have should work just fine - just needs setting up correctly
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
have a look at this output. i entered F for all active licence keys.
[root@media04 admincmd]# ./get_license_key License Key Utility ------------------- A) Add a License Key D) Delete a License Key F) List Active License Keys L) List Registered License Keys H) Help q) Quit License Key Utility Enter a letter: fL selected. for all registerd.
License Key Utility ------------------- A) Add a License Key D) Delete a License Key F) List Active License Keys L) List Registered License Keys H) Help q) Quit License Key Utility Enter a letter: L Enter the name of the host (default is media04): Registered License Keys ======================= License Key: THIS-ISAL-ICEN-CEKE-YSOE-DITI-NGIT-LOLH-EHEH-E Date Added: Jun 23, 2012 at 23:12:54 Host: media04 Product: NetBackup Enterprise Server Client count: See license certificate. Expires: No Expiration Date Feature: Replication Director (Active) Feature: SAN Client (Active) Feature: PureDisk MS Exchange Agent (Active) Feature: Enterprise Vault Agent (Active) Feature: PureDisk MS SQL Server Agent (Active) Feature: Flexible Disk Option (Active) Feature: OpenStorage Disk Option (Active) Feature: Virtual Tape Option (Active) Feature: Bare Metal Restore (Active) Feature: Encryption (Active) Feature: Open File Backup (Active) Feature: Remote Client Support (Active) Feature: Robotic Library Sharing Support (Active) Feature: Remote Media Server Support (Active) Feature: Microsoft RSM Robotic Libraries (Active) Feature: ADIC DAS/SDLC Robotic Libraries (Active) Feature: IBM ATL Robotic Libraries (Active) Feature: Fujitsu LMF Robotic Libraries (Active) Feature: StorageTek ACS Robotic Libraries (Active) Feature: Snapshot Client (Active) Feature: MS SharePoint Agent (Active) Feature: Inline Tape Copy (Active) Feature: Vault (Active) Feature: Library Based Tape Drives (Active) Feature: DataStore (Active) Feature: Encryption (Legacy DES 56-bit) (Active) Feature: Encryption (Legacy DES 40-bit) (Active) Feature: SAP extension (Active) Feature: Sybase extension (Active) Feature: Informix extension (Active) Feature: Oracle extension (Active) Feature: Lotus Notes extension (Active) Feature: DB2 extension (Active) Feature: MS SQL Server extension (Active) Feature: MS Exchange extension (Active) Feature: Shared Storage Option (Active) Feature: NDMP (Active) Feature: Additional clients (Active) Feature: Base NetBackup (Active)am gonna do the mt and tar thingie now on the tapes. btw, tape drives have encyption feature on them. might be the encryption thingie coming in between this write thing.
It does show SSO but can't tell from that how many
Open you admin console (a Windows one if you have one - Java may do the same but not sure) and go to Help - License keys
In here there is an option to show the capacity based license summary
If that shows up relevant stuff then you are capacity based and you have what ever you need - if not you still need to find your certificates or get an IBR report
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
heres the screenshot.
and hey. i installed the mt-st. but when i use the command. it gives me a
am i supposed to use passthru name or the device name?
/dev/nstX for mt - do not use paththru device for mt.
BTW, have you already checked /var/log/messages as I mentioned before?
Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan
yes, i did nothing relating to tape drives or robot..
You seem to only have NetBackup Enterprise and Enterprise Disk license installed.
You also need Library Based Tape Drive license (QTY = 4)
as well as Shared Storage Option license (QTY = 4)
These 2 licenses need to be added to both Master server nodes as well as all Media servers.
You then need to delete current Device config and start from scratch.
vmoprcmd -d on all servers need to show
Shared
Yes
for all tape drives.
Once you have added correct licenses and all devices show up correctly, and you still experience status 84's, let us then start troubleshooting.
Ensure all of the following logs are enabled:
bptm on all media servers
VERBOSE entry in vm.conf (/usr/openv/volmgr) on all media servers (including cluster nodes). Do this before re-running Device Config to ensure Media Manager processes are started in verbose mode (-v).
Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links
hmmm... i have the receipt, the certificate where the keys are listed. the thing is, what I saw in that certificate. 2 keys. when i was installing the master/media server. only worked. the other one never worked. not sure what that was for. i'll see if i can crop that for you here.
mariiane. mark;s recommendation is to run tape backups from the master server as its not supported yet. initially our plan was to have disk backups from the media and tape from the master only. but upon mark's recommendation we on friday installed hba's on the 2 media servers. but since we have an encryption device in the middle. we are not able to tar to the tape from the OS too. so there is a problem with talking to library anyway. 84 might of because of that.
our media server are not clustered. its just the master.
marriane, mark asked me to open the capacity tab. in the registered tab i see those whcih you are mentioning. :S
see, below.
As Marianne says you are not capacity based and so need all licenses for each component you install
I would suggest getting your IBR report or finding out if someone at your place has registered on the Licensing portal where you can access your keys
It may even be worth phoning customer support to get helk setting you up on the licensing portal and if they have your company details they may well be able to populate your portal for you so you know what you have.
Once you have all licenses get them added to the Master and Media Servers, redeploy your tape library and drives to the Media Servers and you will be all good
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
i have set up liceienceing portal. and i have the certificate too. softcopy. it has two licence number. when i was installing master media. only 1 worked for both. not sure what the other is for. its not for opscenter or any other product. its mentioned in the row something related to netbackup. let me see if i show you certificate in anyway so resolve this two license number thingie in the certificate.
btw sir. we have installed 1 HBA in 2 of the media servers. we have thought about sharing 1 hba for 2 media servers. so all togther 4 would be suported in just 2. master still has 2 hba for the tapes. as of now. we are having problem with the encryption device. and i guess that must have been the reason for 84 status. i tried to tar, and it didn't work. so this is the reason of my judgement. we're in process of resolving network issue too. cause its not pingable too.
mark what marriane has mentioned. i have those licenses registered. pls see the screenshot below.
If this installation is meant to become your new production installation, you may as well use drive and SSO licenses from your current production installation.
Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links
really we can do that? how? our old nbu environment which we're going to migrate over months is running on 6.5.6.
**** EDIT ****
We can now see Shared Storage Option in your licensing screenshot.
Have you deleted all devices and re-run device config?
Important to verify that vmoprcmd output shows drives as Shared.
Remember to switch cluster to second node and re-run device config there as well.
If you are still seeing status 84 after correct config, ensure that bptm log folder exists on ALL media servers (including cluster nodes) as well as VERBOSE entry in all servers' vm.conf.
Verify that media manager processes run with '-v'.
bptm as well as system logs will be needed to troubleshoot status 84.
Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links
marianne. sso was always there. its was just that it isn't in the capacity tab. anyway..
marianne & mark.
you wanna know what was giving us 84 status? it was the encryption device preventing the master to talk to the tape lib. now that, that device is fixed i can take tape backups from the master itself. :). since mark suggested to have media take backups. i have HBAs on two of the media server now. now i need to configure all media servers to take tape backups too. so sharing need to be done here on media servers.
as of now. i dont see shared menioned in the output. but let me delete and reconfiggure. i will create a new topic for this and close this one..
i really see nobody's post to the resolution so wouldn't be marking any post as solution. sorry guys..
Would you like to reply?
Login or Register to post your comment.