Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.
Netting Out NetBackup

Securing NetBackup

Created: 04 Apr 2008 • Updated: 22 Jan 2013 • 14 comments
TimBurlowski's picture
0 0 Votes
Login to vote
 
 
 
 

When you work on a product as big as NetBackup, you get to work with a lot of talented and interesting characters. One of those people is Bill Browning, a Senior Principal Software Engineer focused on Security. Since I couldn’t cajole Bill into writing his own blog posts I decided to interview him.

Q) Bill, tell us a little about your background. What are you doing now, how long have you been working with NetBackup and what do you do?

A) Currently I spend my days working on security related issues within NetBackup, and coordinating vulnerabilities as they are reported. I've been working on NetBackup and NetBackup related products since 1999.

Q) What is the most interesting security related trend you have seen in the datacenter in the last year?

A) I find it interesting that the trend is continuing away from attacking server platforms towards middle ware and client side applications, particularly media players, office applications, and of course browsers and their related technologies. You're also seeing a big focus on poisoning web servers to perform more drive by attacks, generally to create botnets. It's also not surprising to see that profit continues to be the number one reason.

Q) What is the one thing you would like every customer to do in order to use NetBackup in a more secure fashion?

A) I'd promote the use of NBAC to avoid the requirement of root or administrator account access for administration. In fact, it is simple enough that we might cook up a comic book to show people how easy it is to setup and configure. Failing that they can look at the yellow book produced by our SWIFT lab that covers it quite nicely.

Q) What is the number one "best practice" you would like our customer's to adopt?

A) It's a simple rule: If it goes out the door, encrypt it. It doesn't matter who it goes to or how it gets there.

Q) What is the scariest "worst practice" you have run into?

A) I'm continually shocked to find that many reputable corporations and governments that seem to misunderstand how to securely store their backups off site. This should be a fairly well understood problem by all but the most junior of Administrators. Your car or home is not a good spot to store backup tapes!

Q) Do you think disk backup is inherently more secure than tape?

A) That really depends. It used to be harder to remove disk than a tape, however, almost all rack mounted servers feature hot swap disks today. Personally I am partial to the encrypting tape drives. You can use tools like gpg to encrypt your hard drive if required. It's less a question of which is better and more a question of knowing what data you have. If it's a disk or a tape that contains all your public web pages (say a backup of your web server) then does it really mater if someone steals it? Does it mater if it's your financial records? You need to know what data you have where, and what the risk is of exposing that data. Many of the lost/stolen social security number stories in the media really come down to people not understanding the data they had, where they had it, and the risks associated with it's location.

Editorial Comment: For instance, Bill probably knows that my wife was recently notified of a security breech involving a charity and a stolen laptop. Thanks for rubbing it in Bill.

Q) Why did bpgp go away? Will it come back some day?

A) Bpgp is gone for good. Get over it. It was an undocumented tool added early in NetBackup's life cycle to help install the product. Once people understood what it did they started to abuse it to violate their own corporate security practices. Symantec is not in the business of helping Administrators violate their corporate policy, so to help our customers we removed it.

Editorial Comment: As you can tell Bill is passionate about security and doesn’t pull his punches. If you think you need bpgp to get your job done, let us know why by leaving a comment explaining how you use it.

Q) What is the next big thing you are working on for securing NetBackup?

A) One of the more interesting items for end users is better non-root administration, and better non-root authentication. This will tie back nicely with auditing features that are also being worked on by security team members.

Special thanks to Bill for consenting to be interviewed.

Message Edited by TimBur on 04-04-2008 04:01 PM

Message Edited by TimBur on 04-04-2008 04:02 PM

Message Edited by Turlas on 04-15-2008 10:47 AM

Comments 14 CommentsJump to latest comment

Perry's picture

bpgp - agreed that many used this undocumented feature in ways Symantec did not intend. However, since "supported" features of the product do not provide us with the capability to completely manage host configurations (exclude lists, *.conf entries, etc.) it was great to find a means of doing this within the product. Frankly, I wouldn't need it if NetBackup were designed for centralized management. bpgp is a workaround for a poorly designed product. Even it isn't the best solution - but it is all we had - at least until now.

0
Login to vote
Rob Thelen's picture

"Bpgp is gone for good. Get over it. "

Wow. That line is killing me. "Get over it."

So an undocumented feature becomes one of the greatest saving graces of the product (in terms of systems brought back from disaster, to time spent managing thousands of clients, to ease of administration of exclude lists and bp.conf's) and gets the axe because people have been using it?!? Wow.

Honestly, yes, I have used it on more than one occasion to go around some security, but following the proper channels would have cost days before a simple solution could have made failing backups work.

Need some log files? No problem. No need to get the SAs involved. Just run this little command and BAM! troubleshooting is on the way. And NetBackup engineers, a lot of our time is spent troubleshooting stuff that is outside of NetBackup. How many times have you needed to troubleshoot by grabbing the client's /var/adm/messages file? Having to page an SA, then explain to the SA that we just need this one little file usually makes enemies, not friends.

And talk about a perfect tool to test if all the communication was in place to a server - If I am able to use bpgp to pull the hosts file from a box (no matter the OS), then I can be pretty sure that the regular OS backup is going to work. This tests the communication path, server authentication, name resolution, and ability to actually move data. After using it a few hundreds times, you also get a feel for how it should work.. It if takes a little long, there could be network issues along the path (maybe half duplex?) or name resolution issues (slow DNS server?). I have used bpgp a number of times simply to test the connections.

How about cleaning up after some bad SAs? I need a second hand to count the number of times I have bailed an SA out of the fire because he changed the root shell in /etc/passwd from /usr/bin/sh to /usr/bin/ksh. Except that ksh is in /bin/ksh.... Rather than waiting for the tapes to be recalled (3 hours, $220, senior management approval), then a restore of the passwd file, I was able to simply grab the file, modify it, and push it back.
Time to resolution : 2 minutes.
Cost : $0.
Having an SA willing to do anything for you : Priceless.

Certainly, We ourselves have identified bpgp as the single greatest security threat to our entire environment. For 7 years, I have denied that such a tool exists to all except only the highest level of engineers in my company. This tool is not to taken lightly... With all the good it can do (and has done) it can also be used for evil. I have made sure that my team and I only use it (for puts) when absolutely necessary. And anytime we are grabbing files, they are strictly for NetBackup troubleshooting use.

It is a sad day to see bpgp go away. BPGP is dead! Long live BPGP! (and bpdir, another great tool that is missing from NBU 6.5)

0
Login to vote
Stumpr2's picture

Nice comments Rob!
I have used bpdir and bpgp for managing include and exclude lists for NetBackup. I do not have a login id to the clients but I am required to know what is and what is NOT being backed up. The exclude and include lists may have the policyname and schedule name included in its filename.
exclude_list..
include_list..
I use this command to list the directory:

bpdir -M /usr/openv/netbackup

I then use bpgp to retrieve the exclude/include files to view or alter as necessary and then once again I use bpgp to put them back.

I have some argue that the same concept could be done using bplist and bprestore. While that is true, it could take a long time to accomplish the task.

Only hire NetBackup administrators that can be trusted. I don't understand the fear about security issues like bpgp being applied to the NetBackup administrator as I tell you the VERITAS, they already have access to all of the data. :-)

VERITAS ain't it the truth?

0
Login to vote
Stumpr2's picture

I'm unable to edit the comment above but I noticed that some " < " and " > " were stripped from my post.

Message Edited by Stumpr on 04-08-2008 02:49 PM

VERITAS ain't it the truth?

0
Login to vote
Chris_Adkisson's picture

Ok, I'll "get over it" if someone can tell me the functionality and syntax for the 6.5 command nbgp. Maybe this version adheres to our business's corporate policy. Lest I clutter up this blog, I second Mr. Thelen's response on this topic, and agree [w/ Mr. Stump] that trusted NBU admins have used the tool very advantageously since it's been around (what, about a decade now?) Lastly, I will comment that it IS possible to port 6.0 bpgp to the latest version...if you really want to keep violating your company's security policies. Of course, I've only done this in a test lab for educational purposes, as I too am not in the business of violating security policies. My 3 cents.

Message Edited by Chris_Adkisson on 04-08-2008 01:28 PM

0
Login to vote
TimBurlowski's picture

Thanks for all the great comments.

I felt a little guilty about not editing out the phrase "get over it" when I pressed the button to post this entry.

I don't mean to imply that we don't care about the feedback from our NetBackup system administrator's. I do care and if I gave the impression to the contrary I apologize.

I think you have each have made valuable points.

1. It's hard to do your job without the right tools.
2. You need an fast & easy way to manage exclude lists centrally.
3. You need a simple tool to test NetBackup communication before doing the full backup.
4. You need an quick & easy way to manage bp.conf files. BTW, what's wrong with the GUI or bpsetconfig & bpgetconfig?
5. You need an easy way to retrieve a log file, including system logs.
6. You need simple file transfer mechanism to bail out an SA.

I'm making sure that this feedback gets heard by the right people.

Technical Product Manger Symantec

0
Login to vote
Stumpr2's picture

4. I know that this isn't the norm, but then I'm not norml. Our master server does not have connectivity to many of our clients as we are provider for a leveraged environment. I can not use the GUI for managing the clients. I have to login to the client's media server and do what I might be able to do from there.
Also, if I use the GUI to alter the bp.conf then the permissions change on the file and database admins and key users cannot access it.

VERITAS ain't it the truth?

0
Login to vote
Bill Browning's picture

Rob T,

    I can understand this. Every time I need to spend a couple hours getting an SA to apply an OS patch it grates on me. However, at the end of your comment you really stated it well. It's a hole, and you've relied on secrecy and a lot of trust in other administrators. If you're relying on bpgp to work around the systems in place within an organization it's a sign that the process in place is broken. Sounds like ssh and scp would possibly server you better. Your CIO and/or CSO will likely be happier giving you access through those sorts of tools (which they can monitor and audit in a variety of ways) if they were to understand a bit more about bpgp.

    I know that's cold comfort though. It has been VERY useful to a lot of people over the years, myself included. Sometimes though it's been "too" useful for some folks.

Message Edited by Bill Browning on 04-10-2008 03:07 PM

0
Login to vote
Bill Browning's picture

So I've heard from a few people who use it to test connectivity to the client. Try bpclntcmd -get_remote_host_version hostname this will return the NBU version number installed.

I've also heard people using it to get logs. While NBU does not have a way to get system logs (such as /var/adm/messages) we do have ways to get netbackup logs. bpgetdebuglog hostname logname mmddyy will do it. so for example if I wanted todays bpcd log form the host eddy it would look something like this: bpgetdebuglog eddy bpcd 041008

0
Login to vote
Jim V.'s picture

If bpgp is a security hole, then the entire client architecture has to be considered a security hole. The bpgp utility was first introduced to me by a Professional Services employee who had come out to install v3.2. It was described as a utility that made use of the same network and client the backups and restores used. And it was an excellent method to test out a client.

Personally I use it to do just that and to push a copy of exclude_list. I happen to be the administrator of both UNIX and backups, so, no I would not have to utilize it, but I like that it checks the actual client mechanism.

I immediately recognized that bpgp could be used for nefarious purposes such as clearing a root password. It comes down to having to trust the administrators because bpgp present or not, the backup administrators already have access to all the data.

I think it is the idea that bpgp made it so 'easy' that some people just do not like. As an example, if the root password on a machine became unknown for any reason, the following procedure could be done to gain access without taking the system down and booting off recovery media:
1) Use a master server policy broad enough to backup /etc/shadow. I'd chose a disk storage unit for speed if I had one available.
2) Restore from that image locally on the master. (Steps 1 and 2 would be the equivalent of bpgp get from ...)
3) Fix the file - clean the password field of root's shadow entry.
4) Use bpbackup or a policy to back that file up.
5) Restore the file to the client. (Steps 4 and 5 would be the equivalent of bpgp put to ...)
So the "hole" still exists as long as redirected restores exist.

0
Login to vote
WCurtisPreston's picture

Bill's response to the question about bpgp is nothing short of callous, but it unfortunately reflects what happened. Veritas knew for YEARS that we were using bpgp to do things that the product was unable to do and they did nothing to allow us to do those things in the core product. The biggest of which is that there is still no ability to manage exclude lists centrally on non-Windows clients. I've done that for years with a shell script and bpgp. (I use bpgetconfig/bpsetconfig to manage bp.conf settings on all clients and exclude lists on Windows clients. They could fix all this by moving the exclude.* directives into the bp.conf file in non-Windows clients, or by allowing bpgetconfig/bpsetconfig to read/write exclude files in Unix, but they did neither. They just yanked bpgp and said "get over it.")

As to this adding any security to the product? Give me a break. NetBackup still has the ability to read and overwrite any file in your environment -- and execute any script you want it to execute. All this does is make it take two steps instead of one. It's the same as DRM; it won't stop anyone who knows what they're doing -- it will only stop the non-hacker from getting their job done.

0
Login to vote
Stumpr2's picture

Isn't bprsh more potentially harmful than bpgp?
Why the bias against bpgp?
Is it because more people know about bpgp than bprsh?

VERITAS ain't it the truth?

0
Login to vote
Stumpr2's picture

Here is an example of running the touch command as root on a NetBackup client from the master server

On the master server
1. restore /usr/bin/touch as /usr/openv/netbackup/bin/admincmd/bpflist on a client
2. bprsh client "/usr/openv/netbackup/bin/admincmd/bpflist /tmp/test.bob"

On client
ls -l /tmp/test.bob -rw-r--r-- 1 root root 0 Apr 18 10:29 /tmp/test.bob

When you are done then restore the original bpflist

 



Message Edited by Stumpr on 04-18-2008 12:43 PM

VERITAS ain't it the truth?

0
Login to vote
Bob Insko's picture

Security hole in bpgp? In my opinion, there is not now and has never been a true security hole with bpgp. The ability to get or send files to any destination in the NetBackup environment is something that all NetBackup Administrators have been able to do since the product was introduced. The use of bpgp made it easier, but even without that explicit command, all NetBackup Administrators still have that functionality available.

As for how it is "typically used", it is usually used to "administer" NetBackup. It is used to maintain bp.conf , vm.conf, touch files, exclude and include lists,,,, the list goes on.

As for how any NetBackup Administrator always has had, and still has the ability to perform the same functionality that used to be provided by bpgp... Just make a policy and back up whatever file you want to push out, or get from whichever source that is desired, and then do an alternate client and/or path restore. To make it easier and faster, a special "bpgp DSU" could be created to be the target location for the backup. This could easily be scripted, and I would suggest two scripts, a "bpgp get" to do the backup, and a "bpgp put" to do the restore. Then, I would have a special flag on the "bpgp put" to indicate if this is the last "destination" for the restore, and when set, expire the image after the restore.

So, as you can see, NetBackup Administrators have always had this functionality. Taking bpgp away did not make NetBackup any more secure, but will only cause the NetBackup Administrators to get a little more "creative" in how to accomplish this same very handy task.

0
Login to vote