A large insurance company was having hardware issues with one of their Symantec Brightmail Mail Security 8360 appliance mail servers. As a result, about 1,400 business-critical emails had been filtered and queued up but not delivered. After getting the hardware problems resolved, the insurance company tried to migrate the queued messages to a different mail server appliance, but they were not having any success. So they called Symantec to help them get the mail delivered.
"The process of delivering the mail from the new mail server appliance should have been pretty straightforward," said James, a Symantec tech support engineer. "But it was difficult in this case because the security they had in place prevented us from getting a WebEx session going." James couldn't see the customer's desktop and therefore wasn't able to take control and run commands, so he had to establish a chat session with the customer while talking to him on the phone. James would tell the customer what command to type, and then the customer would copy and paste the result into the chat window so James could read it. Then James would give him another command to type, and the customer would copy and paste that result into the chat window. "Having to troubleshoot things this way did slow things down a bit," James said, "but we were able to get the queues extracted and moved over to one of his other functioning mail server appliances, where they were promptly delivered."
For mail servers, message format is everything
The customer had multiple mail server appliances, so while they were resolving the hardware issue that had blocked delivery of the messages on the faulty appliance, they thought they could easily re-route the messages to another functioning appliance to deliver them. But they were extracting the queued files in the wrong format and then attempting to put those malformed files on a functioning appliance. "The mail server software on the new appliance was detecting those incorrectly formatted files and interpreting them as corrupt files and deleting them," says James. "They made a couple of attempts, but each time the mail server software thought the messages were corrupt and just deleted them."
James had to enable the ‘root' user account on the disabled machine, extract the mail queues into a compressed archive, move that archive over to a working mail server appliance, extract it into the right place, and then run a couple of internal scripts to correct any problems with the message formatting that didn't match up. "There was no way for the customer or support to do this without getting root access on the machine," James says. "But once we were in as the root user, we were able to transfer the queues and get the message files moved over and cleaned up, and then they were delivered immediately, and that was it."
The whole cleanup and migration process should have taken only about 15-20 minutes, but because they were unable to get past the company firewall to get a WebEx session going, it took about 2.5 hours. "The process would have gone faster if I'd been driving," James says. "But thanks to chat sessions and the customer's cut and paste prowess, we succeeded just fine."