Video Screencast Help

LTO-3 Tape capacity

Created: 25 Feb 2008 • Updated: 21 May 2010 | 13 comments

Good day,

We are using NetBackup 6.5 to back up six servers to an LTO-3 tape library.  The bulk of the data backed up is already compressed, so we use neither hardware nor software compression, and encryption is disabled.  However, we do use multiplexing; four streams are always written to tape at a time.

Our LTO-3 media should be 400GB native, and we should expect around that much data to be written to each tape.  However, we're seeing tapes filled up closer to 80% of their capacity:

Code:
# available_media | grep BD00 | sortmedia   media   robot   robot   robot   side/   ret    size     statusBD0001  HCART3   TLD      0       1      -       4     345729206        FULL/MPXBD0002  HCART3   TLD      0       2      -       4     327035416        FULL/MPXBD0003  HCART3   TLD      0       3      -       4     327794688        FULL/MPXBD0004  HCART3   TLD      0       4      -       4     328716758        FULL/MPXBD0005  HCART3   TLD      0       5      -       4     325908623        FULL/MPXBD0006  HCART3   TLD      0       6      -       4     333678329        FULL/MPXBD0007  HCART3   TLD      0       7      -       4     322036096        FULL/MPXBD0008  HCART3   TLD      0       8      -       4     321796198        FULL/MPXBD0009  HCART3   TLD      0       9      -       4     328631526        FULL/MPXBD0010  HCART3   TLD      0       10     -       4     331744505        FULL/MPXBD0011  HCART3   TLD      0       11     -       4     320786164        FULL/MPX


Is this normal utilization for LTO-3 tapes, uncompressed and unencrypted?  Is it a side-effect of multiplexing?


Thank you very much.

Cheers, John

Comments 13 CommentsJump to latest comment

Jim-m's picture
Hi
 
Looks like you have figured out that compressing compressed data actually makes it bigger.
 
Are you sure that there is no compression at any stage between source source and tape target?
 
 
adminsatsno's picture

Very sure; the tape library's web interface indicates NO hardware compression is being used, I've specified a device path with no hardware compression for the tape drive (/dev/rmt/0hbn - Solaris 10 server), and my backup policy does not have Compression or Encryption enabled.  The bp.conf files on the server and clients have no entries specifying anything related to compression or encryption.  Is there another place where compression settings in the Policy could be overridden?

I'd expect to see these numbers if I was writing compressed data to a drive with hardware compression enabled.  Since hardware compression is disabled, I'd expect to see numbers closer to 400000000 for each tape.  I'm leaning towards Multiplexing overhead as the source of the "missing capacity".

Darren Dunham's picture

I'm sure there must be some overhead, but I'm surprised it's that much.

Any chance you could run a test?  If netbackup doesn't handle sparse files, create a test policy to back up to an isolated pool without multiplexing.  Create an empty file that's bigger than 400g and see what you get out of the tape.

# perl -e 'open (F, ">/path/to/bigfile"); seek F, (500 * 2**30), 0; print F "end";'
# ls -lh /path/to/bigfile
-rw-r--r--   1 root     root        500G Feb 25 14:25 /path/to/bigfile
# du -k /path/to/bigfile
24      /path/to/bigfile



If NetBackup does trim sparse files, then you'll need to actually back up data.  You could back up a blank 100G five times if you don't have space for 1 500G file or something.

I don't expect you'll get an exact number (like 400000000) from the LTO, even with no compression.  Besides the overhead from the NBU headers itself, tapes can be effectively shortened by the drive rewriting suspect data later in the stream.

Good luck!

--
Darren

adminsatsno's picture

I'll give it a try tomorrow; it's definitely too late in the day for me to mess with the backup server right now.  I can easily fit a 500GB file containing actual data on one of the servers, so I'll try creating one and backing it up tomorrow.

I'm not expecting to see exactly 400,000,000 either, but I could have sworn that I used to see numbers within 1% of that before.  That may have been at a time when I wasn't multiplexing backups, and I was probably running version 6.1 at the time and not 6.5.

adminsatsno's picture

This is interesting.  No multiplexing, no hardware or software compression, backing up a single 530GB filesystem containing a copy of our regular (already compressed) data onto a new tape, I got this:

Code:
BD0049  HCART3   TLD      0       49     -       4     315426048        FULLBD0050  HCART3   TLD      0       50     -       4     215151520        ACTIVE


I'll try this same test again with debugging logs turned on, that might give me some insight.  Which logs would contain "tape full" type messages?

Darren Dunham's picture

My guess is that the 'bptm' logs would be most likely.  (You may need to up the verbosity a bit).

Of course another tack would be to test with something like 'dd', completely outside of NetBackup.

(make sure it's a blank tape)
# mt -f <tapedevice> rewind
# dd if=/dev/zero of=/dev/<tapedevice> bs=1024k
[...]

It should error out when it hits the end of the tape.  You'll get a count of blocks (which in this case will be in megabytes).  That way you know any issues here have nothing to do with a NetBackup configuration.  It would be purely due to tapes, tape drives, cables, OS, and drivers. 

--
Darren

adminsatsno's picture

Well, well.  The "dd" test did the same thing.

Code:
# time dd if=/dev/zero of=/dev/rmt/0hbn bs=1024k                                                                                                                                             dd: unexpected short write, wrote 0 bytes, expected 1048576306363+0 records in306363+0 records out0.83u 212.53s 1:36:22.90 3.6%# expr 306363 \* 1048576321244889088


So that rules out NetBackup, compression, encryption, etc. after all.

I may know what it might be -- there's no settings entry in /kernel/drv/st.conf for this particular tape drive, an HP Ultrium 3.
 

I'll try the dd test again after adding an entry:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=lpg50151&prodTypeId=12169&prodSeriesId=443365&locale=en_US

Darren Dunham's picture

What is your OS?  If this is Solaris, then the latest 'st' patches will have LTO
support in the driver, but not necessarily have any LTO entries in the st.conf file.

# grep LTO /kernel/drv/st.conf
# strings /kernel/drv/st | grep LTO
Seagate Ultrium LTO
HP Ultrium LTO 3
HP Ultrium LTO 2
HP Ultrium LTO
IBM Ultrium Gen 3 LTO
IBM Ultrium Gen 3 LTO
IBM Ultrium Gen 2 LTO
IBM Ultrium Gen 2 LTO
IBM Ultrium LTO
IBM Ultrium LTO

I wouldn't rule this out as a possibility, but I'd be suspicious.  In general, I have
not seen misconfigured st.conf files affect capacity.  Block sizes, translation times,
feature sets, density settings yes.  Capacity no....

Do you have support on this drive?  Can they explain the behavior?

--
Darren

adminsatsno's picture

I'll have to pursue drive support, since it's now clearly not a NetBackup issue.  Adding an st.conf entry didn't make a difference either.  My /kernel/drv/sparcv9/st file doesn't actually contain a string for this particular LTO-3 drive, but there are others, including some LTO-4 drives.  I suppose it couldn't hurt to leave the HP Ultrium 3 line in st.conf.

This is Solaris 10 u3 Sparc.  I didn't have quite the newest st patch, so I'll try again with that.  I still don't have "HP Ultrium LTO 3" in my st driver even after the patch, though.

Thanks for the help; good to see NetBackup isn't the source of the issue.

pinchaser's picture
Setup MPX and buffer. Send multipul streams. If you dont supply enough data to the drive the drive cant stream. If the drive does not stream it cant keep writing and therefor you cant get enough data onto tape.  I am getting over 400Gb on lto2 tapes (rate 200/400). LTO3 NEEDS ALOT OF DATA to stream (80mb/s) in order to maximize. If you are on Solaris, run iostat -xdn to watch the throughput to the drive. If you are not getting near that number than you are not providing enough data to the tape.
adminsatsno's picture

Hi Pinchaser,

I wonder if our SCSI interface isn't configured properly, then.  When I ran my dd if=/dev/zero test, I was watching the throughput using "iostat -l 1 -d st13 3".  At times it would be writing around 73MB/s, other times it would write at 55MB/s, and then for 20-30 seconds out of every ten minutes or so, it would drop all the way to zero.  It seemed to cycle between these three data rates.  I'm not sure why this happens; this was performed when NetBackup wasn't running at all (in fact, all NetBackup processes were terminated), and the server was otherwise completely idle.  The tapes still filled at 305-310GB.

Normally I do multiplex four streams at a time.  I have a monthly full backup starting tonight, so I'll watch the iostats as it progresses.  I do doubt that I'll see a steady 80MB/s.

NBU4client's picture

hi all,

 

good day.

we have NBu 5.5 and have software compression enabled.

The tape native capacity is 400 GB, but writes to max of 250 GB only.

 

Will compression cause any issue??

We have been struggling for the past 1 week.

 

your help will be greatly appreciated.

 

thanks :-)

adminsatsno's picture

Hi,

 

In my case, it turned out to be a bad drive.  As soon as the drive was replaced, we consistently filled our tapes to 100%.

 

In your case, you may not be streaming data to the drive quickly enough.  Software compression is a CPU-intensive operation that is performed on the client.  If you have several clients, try multiplexing so more than one stream is written to tape at a time.  If you only have one or two clients, try hardware compression instead of software compression; let the tape drive compress the data.  And if your data is inherently non-compressable like ours was, don't use compression at all.

 

Try some of the tests we went through in this thread with no hardware compression enabled; see if you're able to fill a tape to its expected capacity with "dd if=/dev/zero of=... bs=1024".  If that doesn't fill the tape, then you have a problem with your SCSI bus, the tape drive, or the media.