Video Screencast Help
Protect Your POS Environment Against Retail Data Breaches. Learn More.

Tape status: 'Not Appendable (End marker unreadable)' and Duplication failure

Created: 11 Sep 2013 | 13 comments

Hi All,

We have a Quantum Superloader 3 tape library (LTO 6) running Backup Exec 2012 (SP2) Windows Server 2008 R2 SP1
 
Incremental jobs for 34 servers run nightly at 11pm to the incremental media set (infinite - Allow append).
These backups are consitent in that they backup successfully up until the following point of failure:
 
After approximately one month of usage, I noticed that one of the incremental tapes had become
 
'Not Appendable (End marker unreadable)'
 
Checking through the forums lead me to investigate further and work through the following:
 
  1. check backup exec job engine service (bengine.exe) around time media failed (couldn't locate any instances of the service state starting / stopping that could have caused this)
  2. run live update (SP2 Installed)
  3. check memory errors
  4. check scsi controller drivers to address soft write errors (also run the cleaning tape job)
  5. check windows update
  6. firmware on tapedrive
  7. symantec tape drivers up to date
  8. run long erase on tape (approximately 7 hours / long erase)
 
Since everything was up to date and none of the windows logs coincided with the last backup run before the errors, I decided to perform the suggestion of duplicating the data to disk and re-duplicating it back to tape after running a long erase. I was able to duplicate the unappendable tape contents to the disk, but after running the following duplicate job from the disk back to a new (empty) tape within the same incremental media set, I receive the following error:
 
"Completed status: Canceled
The job was canceled because the response to a media request alert was Cancel, or because the alert was configured to automatically respond with Cancel, or because the Backup Exec Job Engine service was stopped."
 
I am only copying approximately 150 GB to a 2.38 TB drive and it fails every time at the same 5.10 GB point. Some of the data from a few servers is able to copy over to the tape but it alway halts with the above message after 5.10 GB progress.
 
I ran the Backup Exec Debug Monitor (SGMon.exe) on the duplicate back to tape job and received the following output: (See http://pastebin.com/r1nkqQq0)
It looks like the tape media is constantly performing a dismount / re-mount function on the same tape prior to the error:
 
PVLSVR:   [09/10/13 12:30:45] [4408]     AdammSession::DismountMedia()
            Session = {A5502617-E052-49C4-9E00-A09491FF9FCE}
            Library = {B0354312-0329-4389-A540-7DD7A18E3332}, "Robotic library 0001"
            Slot = 0004
            Drive = {E40F4367-F09B-4613-B6A4-5DCCB7EC4818}, "Tape drive 0001"
            Media = {C557A4F3-9BDC-43D6-AAB7-875A71D989EB}, "05"
            Side = 0001
 
The last message I see is the following (though it still appears earlier in the log):
 
BESERVER: [09/10/13 12:32:08] [0340]     "Cluster" key is not present in the registry
BESERVER: [09/10/13 12:32:08] [0340]     Did not detect Microsoft cluster ()
BESERVER: [09/10/13 12:32:08] [0340]     VCS cluster keys do not appear to be present in the registry
BESERVER: [09/10/13 12:32:08] [0340]     Did not detect VCS cluster ()
 
My main concern at the moment is getting the job to duplicate back to the empty tape so we are able to retain our incremental data. Thanks in advance.
Operating Systems:

Comments 13 CommentsJump to latest comment

ANevatia's picture

Hello, 

Looks like the Backup set which you have duplicated to disk is not consistent ( which you are trying to copy back to a new tape ).

You can try to run a catalog again on the disk storage and then run the Duplicate again and check if it works.

But the main concern here should be the tapes getting marked as End marker unreadable. If you have duplicated data from that tape to a disk make sure that the data is consistent . Check the restore window and you should not see a red cross along with Backup set.

Also try to check the adamm.log file for any errors.

Regards,

ANevatia.

pkh's picture

When a tape has "end marker unreadable", it means that BE is unable to close the tape properly.  When you duplicate from disk, you might be actually duplicating data beyond what is being written, hence your problem.  Basically, the data on this tape is a write-off.  If you need certain file(s) on the tape and can restore them properly, then you are lucky.  I would suggest that you don't waste your time trying to do anything with the tape.  You either wait for the tape to expire or ovewrite the tape to re-use it.

Kvothe's picture

It makes sense that the backup sets duplicated to disk and back to tape will fail due to a lack of consistency. That is apparent to me now. Thanks for the confirmation.

What I don't understand, which is the root of the issue, is why have 3 incremental tapes become unappendable over the span of a few months? Simply waiting for the tape to expire or overwriting wont help us in the future because we'll quickly run out of available slots at that rate.

I'm currently duplicating the tape data to disk in order to run a long erase on the problem tapes, but what if the data on the tape exceeds the capactiy of the disk? We cannot afford to simply format a tape every time this error repeats itself.

ANevatia's picture

Hello,

The tapes getting marked an end marker unreadable could be due to various reason. It could be due to some dirty read/write heads on the tape drive or may be some issues with the tape.

I would strongly recommend you to get in touch with the Hardware Vendor and Symantec Support if this happens regularly and get this investigated.

 

Larry Fine's picture

What I don't understand, which is the root of the issue, is why have 3 incremental tapes become unappendable over the span of a few months? Simply waiting for the tape to expire or overwriting wont help us in the future because we'll quickly run out of available slots at that rate.

An "end marker unreadable" tape can simply be erased and it is eligible to be reused.

check backup exec job engine service (bengine.exe) around time media failed (couldn't locate any instances of the service state starting / stopping that could have caused this)

Typically, the media didn't "fail".  When a tape gets marked as "end marker unreadable", it is typically the PREVIOUS job that failed to complete properly and close out the tape set properly, which was discovered when BE attempted to append to this tape and discovered that there was no end marker and gave up on that media.  So you want to concentrate your research around the the previous job.

If you find this is a solution for the thread, please mark it as such.

Kvothe's picture

Though it can be simply erased, the process of duplicating 2+ TB to a disk system and duplicating it back onto a 'long erased' tape is a lengthly process. It also isn't that practical when the data exceeds this size because our disk array isn't large enough to support this many duplicates.

Interesting that you say the job contents could be the culprit. I ran a catalog on the unreadable tape and looked through the backup contents. The last job ran (assuming it might have caused the issue) reports a red 'x' over the contents of a user's e-mail archive. It looks like the data failed to copy at this point.

Larry Fine's picture

Though it can be simply erased, the process of duplicating 2+ TB to a disk system and duplicating it back onto a 'long erased' tape is a lengthly process.

I am sorry, but I don't follow why you would do this?  Does doing this make the tape appendable again?  Is that your goal?  Luckily, I don't have any compliance rules, so I just work around this, rather than try to recover from it.

And I don't see what you are gaining with a long erase compared to a quick erase.  Both make the media overwritable.

If you find this is a solution for the thread, please mark it as such.

Kvothe's picture

Yes, the goal is to make the tape appendable again. We run incremental jobs every night and need to recover the changes that occurred over a months time after the tape became unappendable. I've spoken with Symantec and they suggested duplicate to disk job followed by a long erase.

Rahul Kumar1's picture

Long erase completely erases the data from tape while quick erase just removes the header file. If you are getting end marker unreadable the please perform a long erase on the tape. Also run a cleaning job on the tape drive

pkh's picture

If you are getting end marker unreadable the please perform a long erase on the tape.

Why? A short erase will do just as well in this case.

Rahul Kumar1's picture

No short erase does not do the same thing what wrong erase does. Please read the difference between quick erase and long erase.

Larry Fine's picture

Both make the tape overwritable, which is completely sufficient to recover from an "end marker unreadable" situation.

If you are suggesting the long erase as a troubleshooting step to prevent future occurances, that has some merit, but it NOT needed to recycle the affected media.

If you find this is a solution for the thread, please mark it as such.