Hi Ludo,

we`ve the same poblem

ciao bernes

I've opened couple of cases with Symantec support several months ago and didn't receive any meaningful explanation so far.
It seems that GRT technology either was not tested well or simply doesn't work as advertised.

Awesome !
With a truss you can see that they added a sleep(1) in the bpdm code !
Please Symantec, fix this code !

...
3.5539 read(0, " X F E R B L O C K 4 2".., 21) = 21
3.5541 alarm(0) = 600
3.5542 sigaction(SIGALRM, 0xFFBFD0C0, 0x00000000) = 0
new: hand = 0x00000000 mask = 0 0 0 0 flags = 0x0000
3.5545 getpid() = 22360 [22355]
3.5546 llseek(4, 0, SEEK_END) = 1964349
3.5548 write(4, " 1 7 : 0 0 : 5 2 . 2 8 5".., 172) = 172
3.5550 nanosleep(0xFFBFE240, 0xFFBFE238) = 0
tmout: 0.000000000 sec
resid: 0.000000000 sec
4.5553 nanosleep(0xFFBFE240, 0xFFBFE238) = 0
tmout: 1.000000000 sec
resid: 0.000000000 sec
4.5559 kill(22355, SIG#0) = 0
4.5560 getpid() = 22360 [22355]
4.5562 llseek(4, 0, SEEK_END) = 1965001
4.5563 write(4, " 1 7 : 0 0 : 5 3 . 2 8 6".., 77) = 77
4.5566 getpid() = 22360 [22355]
4.5567 llseek(4, 0, SEEK_END) = 1965078
...

Hi Lu,
Thats a good find.

Right now Symantec is still working on another EEB.  Today i'm going to test with extending the SoftMountPingtimeout to 60 (in regards to NFS). I'll let you know how that goes.

Matt

Does GRT works for Exchange 2007 ?
Only "alpha quality" for Exchange 2003 ?

We are using exchange 2007 and are experiencing this issue.  See the post in the other thread for our setup details.
https://www-secure.symantec.com/connect/forums/tot...

I wanted to remove the 1 second delay in bpdm, so I tried to do this with a LD_PRELOAD32 (Solaris 10).

1- Compile the code below with "cc -G libnanosleep.c -o libnanosleep.so"
2- Create a wrapper for bpdm using this script:

#!/bin/ksh
LD_PRELOAD_32=/tmp/libnanosleep.so
export LD_PRELOAD_32
/usr/openv/netbackup/bin/bpdm.orig "$@"

It will replace the 1 sec sleep with a 20ms one.
Here are our results:
- indexing time=3-4 minutes instead of 1-4 hours
- but much higher I/O load on the media server, caused by the fact that 'bpdm' read data in 256k block when NFS needs only 4/8k.

So there are 2 bugs in the 6.5.4 bpdm:
1- the 1 second sleep before each seek
2- it reads 256kB when NBU NFS only asks for 4kB

This "patch" only addresses the 1st one.

===libnanosleep.c===
#include <stdio.h>
#include <errno.h>
#include <dlfcn.h>
#include <time.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <strings.h>

static unsigned int (*func)();

unsigned int sleep(unsigned int seconds)
{
  unsigned int retval;
  int fd;
  char buf[256];
  time_t cur_time;

  if(!func) {
    func = (unsigned int (*)()) dlsym(RTLD_NEXT, "sleep");
  }
  
  sprintf(buf, "/tmp/sleep.%d.txt", (int)getpid()); 
  fd = open(buf, O_WRONLY|O_CREAT|O_APPEND);
  time(&cur_time);
  sprintf(buf, "%.19s: sleep=%d\n", ctime(&cur_time), (int)seconds);
  write(fd, (const void *)buf, (size_t)strlen(buf));
  close(fd);

  if (seconds == 1) { usleep(20000); return 0; }
  
  retval = func(seconds);
  return retval;
}

Hi ludo,

I've got exactly the same problem!

Hi Lu,

Thanks for the tip to reduce the nanosleep!.... It s crazy!

We try to backup a mail store of 60 Go with 5 mailbox

The windows 2003 server is on R2 SP2 with the NFS patch.

I think  I will test your code to reduce the nanosleep time, but I should need a confirmation of a veritas or symantec guy that's working fine....

Did anyone experiencing this issue have older versions of netbackup or backup exec previously installed on their exchange servers?  I was looking through the registry on our exchange server and I see lots of entries for the previous install of backup exec.

The reason i mention this is because netbackup uses backup exec code to backup exchange.

I guess a better question would be, are any of you working with a fresh OS, exchange install and netbackup install?

More info: this bug also slows down restores... :-(

I have started a granular restore, and it took 10 minutes for 25MB. I 'trussed' bpdm during the restore and also saw the 1 second sleeps. With the LD_PRELOAD32 hack, the restore took 2 minutes.

Please, Symantec, when the final release of GRT will be available :-)

This definitely looks like the sort of thing that should get escalated to the developers for a real answer (or fix)!

I have a hunch there's a "really good reason" that delay was coded in, but obviously I have no clue what it is.  I'd hate for your modifications to come back and bite you, though, so I *strongly* recommend opening a case to get an officially supported EEB replacement... if that's what it ends up being.

Still waiting...
The latest EEB created problems with Duplications...

Lu - could you post the EEB file name or case number to reference to at least fixed part one of the problem.  I'll worry about the duplications later - I still rather have my GRT and I can't seam to get a hold of the EEB for this.

This fix has been in effect and it does resolve the issue regarding the backup performance,

Engineering/Tech Support are fully aware of the side effect with SLP - however, the details from original site that reported are so generic that it would foolish to suggest anything such as corruption at this stage, I will update this thread in a few days time to connect the link with SLP as there will be some development on this SLP v ET1712608.8

RogC

Reckless,

I forgot to mention that this EEB does not solve my particular issue with GRT backups ending in a status 1 when the mailstore is larger than 60-90GB (could never narrow down at what size the problem shows up).
Though it has cut our testing time down to just a few minutes longer than a normal non-grt backup.  Previously the job would run for days!

Matt

We have tried the EEB 1712608.10 and the backup is fast but the duplication to tape is still awful (catalog corruption).

My tests with the latest EEB have been successfull for single large mail stores.  But when try to backup more than one at a time (Microsoft Information Store:\*) it usually ends with a status 1 again.  I even changed the policy to only allow 2 jobs to run at the same time, but i still ended up with random status 1s and backup jobs running for 20+ hours.

So, semi solved here. 

Apologies for the delay in updating in all but it has been confirmed
that ET1712608.10 can be safely applied to environments without causing any
side effects.

The only caveat/precuation is that when running GRT Image Duplications you should
implement "GRANULAR_DUP_RECURSION = 0" in the bp.conf. This disables the catloging of
GRT images.

Full coverage of this flag is documented in this tech note.
http://support.veritas.com/docs/317302

In summary ET1712608.10 + GRANULAR_DUP_RECURSION (if dup'ing GRT images) = effective quicker backups.

It has also been proven that the same ET1712608.10 can resolve SharePoint GRT performance
related issues.

We're currently documenting all of this and will provide an official resoultion to this problem.

Rog C

Lu, we tried this on (7.0GA) backing up (using GRT) both Stores in parallel using multiple data streams. Both jobs ended with a Status 0. If you are encountering this issue I would suggest you create a Support case into Symantec Support.
Inform me of the case id and I will get this looked into,

Rog C

Roger,

Though Lu's issue and mine are different we have been using the same EEBs.  Would you be able to test this with 2 mail stores over 100GB, multi-streaming?

My issue is that im getting status 1s for large databases.  The latest EEB solved the issue for me when only allowing one mail store to backup at a time.  Once i allowed multi-streaming it failed with a status 1 again.

The reason i ask to test is that i have not been able to perform another test in our production environment for a couple weeks and im not sure when ill be able to get another test in.

Thanks ahead of time if you are able to help.

Matt