Last year, we started looking at SEP11. As we're still running on SAV8 for majority of the servers and clients, with a few SAV9 and SAV10, I believe it's about time to start upgrading to the latest version. I checked the features, documentations, forums, blogs, articles and everything about SEP and I'm getting a negative feedback, a lot of people are complaining. Some even call it a nightmare! Very discouraging. I kept on thinking, as the Server Technology Analyst and only 1 year with the company, I don't want to risk my new job as well as mess around with the servers. Is SEP11 ready for primetime, or is it a disaster? SEP11 is September 11, hmmm, a 9/11 disaster! So we didn't rushed to deploy this. Instead, I started working on test servers and clients. At the same time, as I'm the only one who's going to deploy and implement this, I need to have a plan, a very safe one.
I created a project plan, first looking at the existing SSC setup and client configurations. As another department will handle the client deployment to more than a thousand Windows XPs, I am responsible for deploying SEP11 to all the Windows servers, some of which were installed with SAV 9 deployed via group policy, some were installed manually, some were still on SAV 8 especially the critical servers, and there are a few already on SAV 10. We have 6 sites, but majority of the servers are located here in Winnipeg headquarters, others are in Alaska, Seattle, Montreal and Toronto. More than a hundred servers, mostly in VMware, and others are still in physical boxes, especially the critical servers which are still running on Windows 2000! What a challenge, but I don’t like disasters to happen.
Creating a project plan was not easy, coordinating with the different departments is the show-stopper. I have to talk to dba people, network group, applications department, and lastly the end-user support team. I also have to consider the production management’s schedule and approval process. Project plan is ready by January 2009, and after a series of meetings, testing, and most of the time, waiting, we finally took off around May 2009. By that time SEP 11 MR4 is already available, which is kinda stable based on feedback. And I already gathered tons of documents, tips, how-to’s, references and issues and read a lot of forums about deploying SEP11.
I installed a Windows 2008 server, the first in the company, which hosted the SEPM Console. Database is in the production SQL cluster. Network group already opened up the required ports. Policies were carefully reviewed. Active Directory was integrated. Installation packages were created, MSI and SetAid.ini were modified.
SEP11 MR4 was then rolled-out to a few test/dev servers, played around with the policies, enabling and disabling features, used the Migration and Deployment Wizard on some computers, but I got comfortable using the Find Unmanaged Computers for testing and deploying SEP11 one by one to the rest of the test servers.
Here’s how I deployed it – open up SEPM Console, go the groups associated with AD computers, use Find Unmanaged Computers and entered the computer name running SAV9 or SAV10, select the installation package and features. A Computer Management window is opened and connected to the remote target server watching for events in Application logs and see if the Symantec Services are being created and started. A command prompt is open to continuously ping the target server if it will disconnect from the network during deployment. Another command prompt is checking the temp folder and the free space of the target computer. Another window is to monitor the CPU usage. If a deployment status is successful, check the logs, services, free space left and kept on refreshing the view under Protection Technology on SEPM Console until the server icon appears with a green dot. A remote Registry is also opened to delete the registry key for the GPO-deployed SAV9 and remove the computer from the GPO group in AD.
If deployment status failed, most likely it is caused by low disk space, Microsoft Installer version need to be upgraded to 3.1, IE should be version 6 or higher, security credentials, or connection issues. These servers were tagged, noted, corrected the issues and installations were retried. No major issues so far. Maybe those who are encountering problems don't know what they're doing.
Then came the real deal, deployment to production servers, starting with the production servers inside DMZ. Using the same approach as above, SEP11 were deployed smoothly to each server, successfully!
In the middle of the deployment, SEP11 MR4 MP2 was released. So I created another package, put the latest virus definition into the package, and used this to roll out SEP to the rest of the production servers. So far still so good!
Servers running SAV 8 clients were treated differently. SAV8 has to be removed first, reboot the server, then push SEP11. This requires outages and have to wait a little longer for approval. As of today, I still have 3 more servers on SAV8 waiting for an outage approval.
As for the Windows XP clients, the end-user support team used the packages I created in deploying SEP11 to clients via GPO. As a best practice, we learned that we need to disable LiveUpdate in clients as it will create a huge impact on Internet bandwith due to clients connecting to Symantec Internet LiveUpdate web site to download virus defs updates. So far, more than 500 clients have been deployed without issues, more to go...
There were a couple of challenges encountered. The SMC.exe is using high CPU usage on three of our production servers which were upgraded from SAV8. These servers are all running RemoteWare application. After installing SEP11, the CPUs are always 100% with SMC.exe as the culprit. Stopping the SMC service will put the CPU back to normal but it will not communicate with the SEPM console and will not get an updated virus definition. This issue has been resolved by John Prince, a Symantec Technical Analyst. It's a case of wrong subnet mask. At first I don't believe it as I already saw various issues on client-SEPM communication but it's not taking the CPU. Until I checked the settings, changed the subnet mask on the client side, starting SMC again and John was right, smc.exe behaved normally.
The other is that auto-protect is not running on one of the SQL server, but is getting virus update. I still have to wait for an outage to restart the server, maybe on our next patching schedule, and will see if a restart will resolve the issue before contacting support.
The last stage of the deployment is to upgrade the SSC and the group servers providing updates to SAV clients. Once SAV clients are all upgraded to SEP, it’s easy to upgrade the remaining servers.
As for the remaining questions, they can be answered by the following statement:
We’re already using SAV which so far took care of the company’s data protection for a long time, and SEP is a major turning point to move to a more secure end-to-end environment.
I'd also like to mention that I previously worked for Trend Micro as third-level support. But as an IT guy, I am all-around tech savvy that love to tinker with various technologies.