Messaging Gateway

We are currently in the testing of SBG 8.0.3 on VMware.  Today I noticed an upgrade to 9.0.20.  I ran the update and the system rebooted.  When I when back into Control Center, all of my custom configuration (Content Rules, Dictionaries, Spam Rules, Notifications) were gone!!!  How do I get them back?  I have a full backup from January 2010 that I could use to go back to 8.0.3 but am I going to have the same problem again.

Any help????

 

Doug

Hi Doug,

Did you notice any errors during the upgrade?

After the first reboot, system takes some time to migrate the database.  If the system is rebooted again during this database migration phase, then the system ends up in an unstable state requiring an OSRESTORE.  Can you please confirm that the system was not rebooted again?

At this point I would strongly suggest that, before you roll back, please contact support with a diagnositcs and a database backup of the current system (please keep the old backup copy from 8.0.3 in a safe place as you may need it).

Regards,

Adnan

There were no errors that I saw.  The system was not rebooted during or after the update.  Here's the update.log:

 

cat /data/logs/update.log
Updating Package Lists... Get:1 http://localhost i386 release [366B]
Get:2 http://localhost i386 release [355B]
Fetched 721B in 0s (816B/s)
Get:1 http://localhost i386/stable pkglist [783kB]
Get:2 http://localhost i386/stable release [125B]
Get:3 http://localhost i386/stable pkglist [14B]
Get:4 http://localhost i386/stable release [125B]
Fetched 783kB in 2s (293kB/s)
Reading Package Lists...
Building Dependency Tree...
Reading Package Lists... Done
Building Dependency Tree... Done
Done
Updating Package Lists... Get:1 http://localhost i386 release [366B]
Get:2 http://localhost i386 release [364B]
Fetched 730B in 0s (1223B/s)
Hit http://localhost i386/stable pkglist
Hit http://localhost i386/stable release
Hit http://localhost i386/stable pkglist
Hit http://localhost i386/stable release
Reading Package Lists...
Building Dependency Tree...
Reading Package Lists... Done
Building Dependency Tree... Done
Done
2010-03-16 10:40:56 Installing sms-appliance-release-9.0.0-20
warning: cannot get exclusive lock on /var/lib/rpm/Packages
Reading Package Lists...
Building Dependency Tree...
The following packages will be upgraded
  IMA_sms apt apt-stunnel caching-nameserver cli hwdata jre kernel-prod lsisnmp
  openldap openldap-clients osconfig perl-DBD-MySQL perl-SMS-Util
  sms-appliance-release sms-ecelerity sms-ecelerity-icu sms-smtp-base
  sms-smtp-bcc sms-smtp-connector sms-smtp-modules sms-smtp-mta sms-smtp-mysql
  sms-smtp-tomcat smsapp-db-migration software-update sudo sysstat tzdata
The following packages will be REPLACED:
  sms-smtp-sync-service (by sbg-dds)
The following NEW packages will be installed:
  pciutils sbg-dds vmtoolsd
29 packages upgraded, 3 newly installed, 1 replaced, 0 removed and 0 not upgraded.
Need to get 405MB of archives.
After unpacking 88.0MB of additional disk space will be used.
Get:1 http://localhost i386/stable sms-appliance-release 9.0.0-20 [28.0MB]
Get:2 http://localhost i386/stable jre 1.6.0_17-fcs [20.6MB]
Get:3 http://localhost i386/stable perl-SMS-Util 1.0.0-30_sms [21.9kB]
Get:4 http://localhost i386/stable sms-smtp-base 9.0.0-10 [31.6MB]
Get:5 http://localhost i386/stable IMA_sms 8.0.0-237 [1381kB]
Get:6 http://localhost i386/stable apt-stunnel 1.0.1-8_sms [6891B]
Get:7 http://localhost i386/stable apt 0.5.5cnc6-fr10_sms [2958kB]
Get:8 http://localhost i386/stable caching-nameserver 7.3-4_sms [1417B]
Get:9 http://localhost i386/stable openldap 2.0.27-9_sms [400kB]
Get:10 http://localhost i386/stable openldap-clients 2.0.27-9_sms [80.8kB]
Get:11 http://localhost i386/stable osconfig 1.0.1-51_sms [20.1kB]
Get:12 http://localhost i386/stable perl-DBD-MySQL 3.0007-1_sms [769kB]
Get:13 http://localhost i386/stable sms-smtp-mysql 5.0.15-12 [18.0MB]
Get:14 http://localhost i386/stable sudo 1.6.6-56_sms [150kB]
Get:15 http://localhost i386/stable sysstat 9.0.3-3_sms [227kB]
Get:16 http://localhost i386/stable cli 1.0.2-85_sms [205kB]
Get:17 http://localhost i386/stable kernel-prod 2.6.28-10_sms [8006kB]
Get:18 http://localhost i386/stable pciutils 2.1.10-7 [48.2kB]
Get:19 http://localhost i386/stable hwdata 1.0.0-1_sms [375kB]
Get:20 http://localhost i386/stable lsisnmp 3.13-7_sms [595kB]
Get:21 http://localhost i386/stable sbg-dds 9.0.0-2 [5311kB]
Get:22 http://localhost i386/stable sms-ecelerity 2.2.2.45-5246_4061_5428 [3035kB]
Get:23 http://localhost i386/stable sms-ecelerity-icu 2.2.2.45-5246_4061_5428 [6002B]
Get:24 http://localhost i386/stable sms-smtp-bcc 9.0.0-10 [62.6MB]
Get:25 http://localhost i386/stable sms-smtp-connector 9.0.0-7 [5529kB]
Get:26 http://localhost i386/stable sms-smtp-modules 9.0.0-9 [166MB]
Get:27 http://localhost i386/stable sms-smtp-mta 9.0.0-9 [2777kB]
Get:28 http://localhost i386/stable sms-smtp-tomcat 5.5.12-20 [42.2MB]
Get:29 http://localhost i386/stable smsapp-db-migration 3.4.0-6_sms [613kB]
Get:30 http://localhost i386/stable software-update 1.0.0-57_sms [15.2kB]
Get:31 http://localhost i386/stable tzdata 2009s-1_sms [788kB]
Get:32 http://localhost i386/stable vmtoolsd 1.0.0-2_sms [2808kB]
Fetched 405MB in 7m44s (874kB/s)
Executing RPM (-Uvh)...
warning: /data/apt/cache/archives/pciutils_2.1.10-7_i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e
Preparing...                ##################################################
perl-SMS-Util               ##################################################
perl-DBD-MySQL              ##################################################
sudo                        ##################################################
jre                         ##################################################
Unpacking JAR files...
        rt.jar...
        jsse.jar...
        charsets.jar...
        localedata.jar...
        plugin.jar...
        javaws.jar...
        deploy.jar...
Stopping Symantec Brightmail Gateway SMTP Connector: [  OK  ]
Shutting down MTA: [  OK  ]
sms-smtp-base               warning: /data/scanner/etc/bmiconfig.xml created as /data/scanner/etc/bmiconfig.xml.rpmnew
##################################################
[Tue Mar 16 10:49:14 CDT 2010] rebootwatch script starting.
osconfig                    ##################################################
Stopping Symantec Brightmail Gateway Control Center:

Waiting for processes to exit..
Waiting for processes to exit..
Waiting for processes to exit..
Waiting for processes to exit..
Waiting for processes to exit..
Waiting for processes to exit..
Waiting for processes to exit..
Waiting for processes to exit..
Waiting for processes to exit..
Waiting for processes to exit..
Waiting for processes to exit..
Sending the Tomcat processes a SIGTERM asking them to shut down gracefully..
[  OK  ]
Giving mysqld a couple of seconds to exit nicely
sms-smtp-mysql              ##################################################
smsapp-db-migration         ##################################################
sms-ecelerity               ##################################################
openldap                    warning: /etc/openldap/ldap.conf created as /etc/openldap/ldap.conf.rpmnew
##################################################
openldap-clients            ##################################################
sysstat                     ##################################################
cli                         ##################################################
apt-stunnel                 ##################################################
apt                         ##################################################
kernel-prod                 ##################################################
Shutting down MTA: Process does not appear to be running.
Process does not appear to be running.
[  OK  ]
sms-smtp-mta                warning: /data/mta/etc/mtaconfig.xml created as /data/mta/etc/mtaconfig.xml.rpmnew
##################################################
/opt/Symantec/Brightmail/mta/sbin/sendmail has not been configured as an alternative for mta
Stopping Symantec Brightmail Gateway Control Center:
[  OK  ]
sms-smtp-bcc                ##################################################
sms-smtp-modules            ##################################################
software-update             ##################################################
sassnmp is installed
Upgrading from old RPM
lsisnmp                     ##################################################
Stopping IM Relay Runner:
[  OK  ]
IMA_sms                     ##################################################
Starting IM Relay Runner: [  OK  ]
sbg-dds                     ##################################################
sms-ecelerity-icu           ##################################################
sms-smtp-connector          ##################################################
rm: cannot remove `/data/scanner/stats/stats.dat': No such file or directory
rm: cannot remove `/data/scanner/stats/stat.av.symantec': No such file or directory
Stopping Symantec Brightmail Gateway Control Center:
[  OK  ]
sms-smtp-tomcat             ##################################################
vmtoolsd                    ##################################################
tzdata                      ##################################################
caching-nameserver          ##################################################
hwdata                      ##################################################
sms-appliance-release       ##################################################
pciutils                    ##################################################
Giving mysqld a couple of seconds to exit nicely
warning: /data/bcc/webapps/brightmail/WEB-INF/classes/PBCConfig.properties saved as /data/bcc/webapps/brightmail/WEB-INF/classes/PBCConfig.properties.rpmsave
+ RESOURCES=/etc/Symantec/SMSSMTP/resources
++ /bin/uname -s
+ PLATFORM=Linux
+ '[' -f /etc/Symantec/SMSSMTP/resources ']'
+ . /etc/Symantec/SMSSMTP/resources
++ ALLOWEDIP=127.0.0.1
++ CLIENT_PACKAGE_INSTALLED=false
++ CONFIGDIR=/data/scanner/etc
++ DATA_ROOT=/data
++ INSTALLPLATFORM=appliance
++ INSTALL_MODE=1
++ INSTALL_ROLE=3
++ INSTALL_TYPE=2
++ JAVA_BIN=/usr/java/default/bin
++ JAVA_HOME=/usr/java/default
++ JOBSDIR=/data/scanner/jobs
++ LOADPOINT=/opt/Symantec/Brightmail
++ LOGDIR=/data/logs
++ MTADATADIR=/data/mta
++ PRODUCTNAME=SMSA
++ PRODUCTVERSION=9.0.0
++ RULEDIR=/data/scanner/rules
++ SERVER_PACKAGE_INSTALLED=true
++ STATSDIR=/data/scanner/stats
++ SYMANTEC_BASEDIR=/data/scanner
++ TOMCATPORT=41443
+ SCRIPTLET=preremove
+ LOGGER=/opt/Symantec/Brightmail/scanner/bin/logger -t [sms-smtp-mta.preremove]
+ '[' Linux = Linux ']'
+ '[' 1 == 0 ']'
+ exit 0
+ RESOURCES=/etc/Symantec/SMSSMTP/resources
++ /bin/uname -s
+ PLATFORM=Linux
+ '[' -f /etc/Symantec/SMSSMTP/resources ']'
+ . /etc/Symantec/SMSSMTP/resources
++ ALLOWEDIP=127.0.0.1
++ CLIENT_PACKAGE_INSTALLED=false
++ CONFIGDIR=/data/scanner/etc
++ DATA_ROOT=/data
++ INSTALLPLATFORM=appliance
++ INSTALL_MODE=1
++ INSTALL_ROLE=3
++ INSTALL_TYPE=2
++ JAVA_BIN=/usr/java/default/bin
++ JAVA_HOME=/usr/java/default
++ JOBSDIR=/data/scanner/jobs
++ LOADPOINT=/opt/Symantec/Brightmail
++ LOGDIR=/data/logs
++ MTADATADIR=/data/mta
++ PRODUCTNAME=SMSA
++ PRODUCTVERSION=9.0.0
++ RULEDIR=/data/scanner/rules
++ SERVER_PACKAGE_INSTALLED=true
++ STATSDIR=/data/scanner/stats
++ SYMANTEC_BASEDIR=/data/scanner
++ TOMCATPORT=41443
+ SCRIPTLET=postremove
+ LOGGER=/opt/Symantec/Brightmail/scanner/bin/logger -t [sms-smtp-mta.postremove]
Stopping Symantec Brightmail Gateway Control Center:
[  OK  ]
sms-appliance-release-9.0.0-20 successfully installed.
Rebooting appliance...
mount: /opt is busy
 

I will also open a Support Call.

 

Doug

Yes, the update.log does not indicate any problem.  The diagnostics and a dump of your current database will help in determine the root cause of the issue.

Please do open up a support ticket.

Regards,

Adnan

Well, after talking with support, they didn't even have me send a diag or a dump.  I'm reloading from base 8.0.3 and then putting on my backup.  They then want to run a check on the MySQL database and perform the upgrade from the command line.  Here's the case number for anyone interested:  Case 411-691-373.

FYI for the VM users.  After upgrading to Version 9 and attempting to do a "Factory Reload", it will not take you back to your previous version.  It will take you to base version 9.0.20.  All I can say is Backup, Backup and Backup again.

Doug

Hi Doug,

Is is possible for you to send the diagnostics and mysql dump from the current state?

Regards,

Adnan

Already performed the "Factory Reset" so everything is gone.

 

Doug

It's good that you took a backup before the upgrade; that's a good practice and always recommended.

Unfortunately, without that evidence (diganostics and database dump) we may not be able to figure out what happened and prevent this in the future.

To get a better idea, I have a few questions:

a) did you go thru the Software Update Decription in the Control Center before the upgrade?

QUOTE:

Running software update
=======================

Before running software update, be certain your appliance is not performing
tasks that, if disrupted, could cause problems after updating.

--Check for a running LDAP synchronization cycle
--Check for a running Scanner replication cycle
--Minimize the number of messages in any of the queues

To prepare for the software update, follow the steps below. The Control Center
locations are for Symantec Brightmail Gateway version 8.0 and may differ for
earlier versions.

1 To check for a running LDAP synchronization cycle or Scanner replication
  cycle, access Status > System > LDAP Synchronization.
2 To halt incoming messages, access Administration > Hosts >
  Configuration/Edit, click "Do not accept incoming messages", and click Save.
3 To check the queues, access Status > SMTP > Message Queues.

UNQUOTE:

b) Were you storing all spam message in the quarantine?

c) What was you log level set to?

d) Did you enable extra reporting options and for how long you were keeping the reporting data?

Regards,

Adnan

For what it's worth, my upgrade went fine.  However, I did protect myself in a few different ways prior to updating.  First, I shut down my two VM's running the console and the scanner.  Then, I made a LUN based snapshot (the VM's reside on a SAN, under their own dedicated LUN).  I also made VMware based snaps just to be on the cautious side.  I took the advice of the documentation and cleared out my quarantines, etc.  It still took a little while, but it worked.

Thanks for sharing the success story :)

Adnam,
This system is in testing only so there is no mail going through it.  It is not replicating to anything, it is a standalone Scanner and Control Center.  There is nothing in either Quarantine.  Just a pretty base system.  In fact, the backup is under 300k. 

We attempted to try the update again and have the same problems and more.  Now, we are unable to get the GUI to login, get an application error screen with no errors.  Just to option to log back in again.

Log level is set to defaults.  I do have a copy of the current diags if your interested.

Doug, thanks for the diagnostics.  I have attached it to the case.  Please remove it from this forum.

Thanks

Adnan

If you can tell me how to do it, I will be more than happy to.

You can edit your comment that includes the attachment and at the bottom you should should see the "Delete" checkbox; check that box and click Save.  I think that should do it.

Hi Doug,

The current issue that you are experiencing where you can't get back into the Control Center and for which you provided the diagnostics is the one that's described in this KB article:

http://service1.symantec.com/SUPPORT/ent-gate.nsf/docid/2010031608442454

Regards,

Adnan

I see the edit option on the posts above the post where I placed the attachment, but not on that post.  Any ideas?

Doug

Just to keep the thread live, here's the latest on our issue.  We have been escalated to 2nd level support and Engineering.  They believe they have found the problem and we're awaiting a new "migration" script to migrated out current DB to the new DB in Version 9.

The issue is that we renamed some policies that were default.  We had customized the policies for our environment.  So the install was looking for those exact table names.  When it couldn't find them, that caused our problem.  Since we are not production, support is going to use my system to beta test the repair script and any other fixes associated with the upgrade.

I will keep everyone informed of our progress.

Doug

The issue that Doug pointed out above is different form the initial issue that he opened this thread for.  Anyone looking to upgrade to version 9.0 should first look at the updated KB artilce about this issue:

http://service1.symantec.com/SUPPORT/ent-gate.nsf/docid/2010031608442454

and the Updated Release Notes:

http://service1.symantec.com/SUPPORT/ent-gate.nsf/docid/2010020114554754

Many thanks to Doug for this assistance in testing the workaround/solution.

Regards,

Adnan

Also note, this was our 2nd attempt at an upgrade.  The 1st attempted, Version 9 was "usuable" but all of our configurations were gone.  After the 2nd attempt is when we got the error that the 1st link in the post above refers to.

Doug

I deleted the attachment from the post.


//ian

After working with support and receiving a shell script to upgrade the databases, I can say that we are upgraded to 9.0.20.  Checking all of our configurations, everything appears to be there.  We are going through testing to make sure mail is being processed correctly and our rules are working.

I would like to thank everyone on this thread for all of their help and especially Ryan Roberton from the Messaging Security – Advanced Team in getting the server back working.