The purpose of the document is to help administrators who are using Backup Exec CASO option in larger scale environments. The document contains tips on setting up the large scale setup and best practices on managing the Backup Exec setup. Also included is a basic troubleshooting section on how to resolve issues that may be encountered.
Section 1: Installation and Upgrade of CASO.
This section will talk about the best practices for installing Backup Exec CASO for the first time in a large scale setup. This section also gives tips for upgrading an existing large scale Backup Exec CASO environment. The first part of the section will focus on new installs and briefly describes what the CASO is. This section may also provide helpful tips to those upgrading that were not considered when the initial Backup Exec CASO setup was initially deployed.
New Install of a CASO Setup
The CASO (Central Administration Server Option) was designed to manage other media servers which are known as MMS (Managed Media Server) Servers. The CASO server is the server from where jobs can be created and delegated to specific MMS servers or a group of MMS servers. Also from this server, jobs can be monitored that are running on each of the MMS servers. For more detailed information regarding the functionality of these server roles, please see the Backup Exec Administrator Guide.
Even though the CASO server has the functionality to perform all Media Server tasks in a large scale environment, it is recommended to designate it the role of being an administration server only. The reason for this is that the CASO server will have a heavy I/O resource load when communicating with and managing the MMS servers.
When selecting a server or servers where to install CASO to, it is highly recommended that the system(s) meet the following system requirements, especially if Windows is clustered or in a SAN (Storage Area Network) Environment:
SQL 2000 or SQL 2005. When installing Backup Exec CASO Server there is the option of using an SQL 2005 Express database to hold information regarding the setup. This is sufficient for smaller CASO setups with 20 or less MMS Servers, however when a large number of servers are to be managed, the limitations on the maximum size of the DB, amount of addressable RAM that can be used, and number of processors will impact the performance of the CASO server. The SQL Server should be installed locally for optimum performance, even though a remote SQL 2000 or 2005 server can be used. Due to the large number of disk writes that will occur to the DB it is highly recommended to designate separate high speed disks for both the Data and Log components.
Memory. It is important that the CASO Server have up to 8GB of Physical Memory (RAM) available. During peak times it is important to have enough memory to load SQL and run the Backup Exec Operations. It can possibly function with less memory than 8GB, depending on how many MMS and the amount of tape hardware being managed by the CASO Server; however the configuration may run into performance or communications issues between the machines.
64 bit Processor. Even though a 32 bit processor can be used, it is recommended that a server with a 64 bit processor and operating system be used instead. This will greatly improve the performance of the CASO Server.
How to determine if the CASO setup us considered a large scale environment? The important aspect of determining this is based on the number of concurrent connections to the DB. The maximum number of concurrent connections to the MMS and tape hardware is around 180; once this maximum is reached performance issues can occur or DB exceptions can show in the Windows Event Logs. SQL Express 2005 doesn't limit connections; however as mentioned its limitations are on the number of CPU's it can utilize and the size of the DB, which is 4GB maximum. Also, the maximum limit for MMS is 90 when the catalogs are centralized, regardless of the version of SQL being used or hardware that is available.
Upgrading a CASO Environment
When upgrading the CASO setup from a prior release of Backup Exec, the following provides important tips in upgrading the setup.
Steps on upgrading your CASO setup:
1. Make sure jobs do not run during the upgrade process. This can be done by simply putting jobs that are scheduled to run during the upgrade on hold. MMS servers can be paused too; this will also prevent jobs from running.
2. Stop all Backup Exec services on all of the MMS servers. This can be done from the CASO Server GUI (Graphical User Interface). If the Backup Exec services on the MMS do not stop, this may indicate a problem and should be resolved before proceeding with the upgrade. Please see the troubleshooting section below if the services don't stop as they should.
3. Run setup on the CASO server. Following guidelines on installing and complete the upgrade. Reboot the server after install if prompted.
4. After the CASO server has been rebooted, launch the GUI on the CASO to make sure upgrade went without error, then run LiveUpdate to confirm all critical and/or relevant hotfixes are applied. Run setup on each of the MMS servers. Follow guidelines on installing and complete the upgrade. Reboot the server after install if requested and run LiveUpdate to confirm all the same patches get applied to the MMS as were to the CASO Server.
5. After the MMS servers have been upgraded, verify that all the CASO servers are in a good communication state. For MMS servers that show up in a No Communication state, please see the troubleshooting section of this document.
6. If all MMS servers are being seen correctly along with jobs and devices, then the upgrade has been completed successfully. If a lot of jobs are scheduled to run after being taken off hold, it is recommended to gradually take the jobs off hold rather then run all at once.
When performing a large scale upgrade it sometimes may not be possible to upgrade all of the MMS servers in the CASO setup at one time. Because of this, some MMS servers may temporarily need to be left at a prior revision until it is possible to upgrade these machines.
For example, when using Backup Exec 11d revision 7170, these MMS servers can temporarily be managed from a 12.0 revision 1364 CAS server.
The following charts the supported temporary upgrade versions:
||Temporary MMS Support
Symantec does support temporary rolling upgrades for one prior version of Backup Exec, but not for extensive periods as the features of the newer version cannot be used and if any problems occur with the prior version MMS's, then it is recommended to proceed with upgrading that machine before doing any further extensive troubleshooting.
Section 2: Managing a Large Scale CASO Environment.
Management of the CASO setup is very important in making sure a large scale setup runs efficiently. Below are some management tips that will help in making sure the scale setup runs smoothly and efficiently.
Network Management. It is important to quickly resolve any network issues. Fixed IP's are recommended to avoid any DNS issues. It is also recommended to have a gigabyte network for the CASO environment. Faster the network; better the CASO environment will run. It is also recommended to turn off any power saving setting on the NIC cards. Also replace any NIC cards that appear to have intermittent issues, since problem network cards can cause issues on the network that will impact the performance of the CASO setup.
Database (DB) maintenance. It is important to run DB maintenance on a routine basis and it is scheduled to run by default each day at 4am per what shows at Tools> Options> Database Maintenance. For optimal backup performance, it is recommended that this is scheduled to run when the CASO environment is not running any jobs. DB maintenance can be run on the MMS servers at the same time as it does on the CASO; once again no jobs should be running during DB maintenance process, to assure optimal backup, restore, and communication performance in the CASO Environment. This also makes a backup of the CASO and each MMS database.
Alert Management. It is highly recommended that active alerts routinely be responded to. If active alerts are allowed to build up it may affect the performance of the CASO environment. Alerts that indicate problems on the MMS servers should also be investigated.
MMS Servers in a No Communication State. If MMS servers go into a No Communication state it is highly recommended to resolve this immediately. It usually indicates a problem with the network. If the problem is not with the network. see the troubleshooting section on tips to get the MMS server back into a good communication state. If the MMS servers are permanently removed from the CASO setup or moved into a different CASO setup, these MMS nodes should be deleted from the GUI Media Servers on the CASO Server. This is important since the CASO server will try to communicate with these servers and will cause unnecessary network traffic.
Domain Credentials. It is highly recommended to use domain admin accounts for authenticating between the CASO, MMS, and machines that have the Remote Agent for Windows (RAWS) installed.
Section 3: Using BEUTILITY to reconfigure your MMS.
To reconfigure the CASO environment to make a Backup Exec standalone media server an MMS or remove a MMS server from the CASO environment, perform the steps listed below:
Turn a standalone media server into a MMS
1. Launch BEUTILITY.EXE. The exe is located in the Backup Exec program folder (X:\Program Files\Symantec\Backup Exec).
2. Right click on the All Media Servers and select New Media Server. Browse to the standalone media server to make it an MMS server.
3. After adding the media server, highlight the machine in the results pane. Go to the list of tasks. Under the CASO Tasks section, select Set Central Administration Server.
4. Enter the name of the CASO server and select catalog store options. By default Distributed is selected. This is the same store option as a new MMS run via setup. Select OK to complete the operation.
5. Go to the CASO server to verify that the MMS is seen and in a communication enabled state. If the server is not seen correctly, then see the troubleshooting section below.
Removing a MMS from a CASO Environment
1. Launch BEUTILITY.EXE from the run line. The exe is located in the Backup Exec program folder (X:\Program Files\Symantec\Backup Exec\Data).
2. Right click on the All Media Servers and select New Media Server. Browse to the standalone media server to make it an MMS server.
3. After adding the media server, highlight the machine in the results pane. Go to the list of tasks. Under the CASO Tasks section, select Remove Central Administration Server. Select Yes to continue. Once Yes is selected to operation to remove the MMS server will be performed.
4. Go to the CASO server after a few minutes. The MMS removed from the CASO environment will go into a No Communication state. Since the MMS is no longer a part of the CASO configuration it will need to be deleted. To delete the MMS, go to the Media Servers tab and right click on the MMS and select Delete. This will remove the MMS server from the DB. Jobs that are targeted to this MMS server will need to be redirected to a different MMS server to run. A prompt will also show to redirect the jobs when deleting the MMS.
Note: BEUTLITY can also be used for other basic tasks such as starting and stopping Backup Exec services or moving the CASO server. These advanced tasks, however, should be performed with caution and the assistance of the Backup Exec Technical Support Team may be necessary when in doubt.
Section 4: Troubleshooting Tips.
At times communication issues may occur with the CASO server. When a communication issue occurs the MMS servers can going into a No Communication State. Communication issues can also affect jobs, causing them to go into a status of Communication Stalled or into a ready state due to No Media Servers being available. A majority of the issues can be resolved by following the trouble shooting tips below.
Tips on resolving communication issues with the MMS
1. Update the firmware and drivers to the latest level for each server's hardware (SCSI/RAID, HBA, Network Card(s), etc), especially for cases where a SAN is involved in the CASO setup. Confirm also that the latest Microsoft Service Pack and patches have been applied to each machine too.
2. Verify that the CASO server can ping the MMS by name. If the CASO server is unable to ping the MMS, this would indicate a problem with the network, and a network administrator should be contacted. If the CASO server can ping the MMS by name; then follow step 3.
3. Remote into the MMS. Verify that the Backup Exec services are running. If unable to remote into or have access to the MMS then go to step 4.
4. If the MMS can be pinged by name and Backup Exec services are running, then stop the Backup Exec services on the MMS. This can be done from the CASO server by right clicking on a MMS in the Media Servers tab or going to the MMS directly. If Backup Exec services don't stop and appear to be hung, this may indicate the problem. Use task manager to halt the service that fails to stop. After the services are stopped, then restart the Backup Exec services. Allow a few minutes then check the CASO server to see if the MMS is in a communication enabled state. If the problem continues then go to step 5.
5. Again stop services on the MMS. On the MMS go to Data folder in the Backup Exec program files directory. Delete the message Q files; these files start with msgq and have a .dat extension. These can only be deleted when the Backup Exec Services are stopped. Once the files are deleted, restart the Backup Exec services on the MMS server. Allow a few minutes then check the CASO server to see if the MMS is in a communication enabled state. If the MMS is still in a No Communication state then go to step 6.
6. Find a time when the CASO server is under a light I/O Load and not running any jobs. Stop the Backup Exec services on the CASO server. Delete the message Q files as described in step 5 from the CASO server. Restart Backup Exec services on the CASO server and launch the GUI. The MMS should be back in a communication enabled state. If the problem continues then go to step 7.
7. Use BEUTILITY to reconfigure the MMS server to be a standalone media server. After it is removed as a MMS server; then go to the CASO server and delete the MMS from the Media Servers tab. After this is completed, use BEUTILITY to add the standalone media server back as a MMS. For tips on moving a MMS see the BEUTILITY section in this document. If this doesn't resolve the communication issue, attempt to add the MMS by running the Backup Exec Installation again on that machine and selecting the MMS Option during setup. If the CASO and MMS are still not able to communicate properly afterward, contact the Backup Exec Support Team for further assistance.