Video Screencast Help
Search Video Help Close Back
to help
Not able to make it to Vision this year? Get a sampling in the Best of Vision on Demand group.

Symantec Messaging Gateway 9.5.0-23: 100% CPU

Updated: 29 Apr 2011 | 18 comments
AdNovum's picture
AdNovum
0 0 Votes
Login to vote

Hi,

A couple of days ago I've installed Symantec Messaging Gateway 9.5.0-23 as virtual appliance on a VMware ESXi 4.0.0 server. One instance is the control center + scanner, second instance is scanner only. The virtual appliance is configured with two CPU's, 4 GB memory and 100 GB disk.

The setup is currently not under load. No mails are sent to the scanner. But the CPU of the control center appliance consumes constantly 97-100% while the scanner only appliance consumes only about 1% (according to the Web UI and VMware ESX) but the "monitor system" shows much less usage (ca 25% user, 12% system) but still too much for an appliance that should be idle. The highest load shows "monitor --proc conduit" with about 7%.

Any idea, which process is going crazy and why? The root login is disabled and I miss commands like 'ps' and 'top' on the restricted commandline for user 'admin'.

 

Bye

Bernd

Discussion Filed Under:
,

Comments

AdnanH's picture
02
May
2011
0 Votes 0
Login to vote
AdnanH
Symantec Employee
Accredited

KB Artilce: TECH95310

Hi,

Please take a look at the following KB article:

http://www.symantec.com/docs/TECH95310

Regards,

Adnan

AdNovum's picture
03
May
2011
0 Votes 0
Login to vote
AdNovum

Hi Adnan, Hmm, but the

Hi Adnan,

Hmm, but the appliance with the scanner only uses almost no CPU and also on the Linux console it appears to be slow and busy. Also the Web UI says that the controlcenter consumes sometimes 100% cpu but the scanner only 1%.

 

Host CPU
Usage		 CPU
Iowait		 CPU
Temp		 Memory
Usage		 Memory
Free		 Disk
Usage		 Disk Read Disk Write Swap
Used		 Swap
Free		 Fan Battery Power
Supply		 RAID
  Local Host 99.14% 0.43% — 3.23 GB 702.25 MB 3.67 GB 0  KB/second 6,389  KB/second 0  MB 4,143.33  MB — — — —
  Scanner 3.83% 0% — 1.82 GB 142.27 MB 3.62 GB 0  KB/second 192  KB/second 117.53  MB 4,025.8  MB — — — —

 

 

 

 

This setup is not yet active. No incoming/outgoing mails - exept my personal Thunderbird for testing with about 1-3 mails per hour. Uses pretty much CPU and memory and writes a lot to disk for doing nothing. A restart of the virtual appliance does not help.

Bye

Bernd

AdNovum's picture
04
May
2011
0 Votes 0
Login to vote
AdNovum

Yesterday I did a fresh

Yesterday I did a fresh install of the virtual appliances and splitted control center and two scanners into separate VMware machines. Setup was pretty much with default settings and update to 9.5.0-23. Whole setup is not productive and no mails are piped through.

The SMG admin UI tells me now that one scanner is using almost 100% CPU. The ESX performance log tells me the same.

Host CPU
Usage		 CPU
Iowait		 CPU
Temp		 Memory
Usage		 Memory
Free		 Disk
Usage		 Disk Read Disk Write Swap
Used		 Swap
Free		 Fan Battery Power
Supply		 RAID
  Control Center 0.5% 0% — 1.22 GB 2.69 GB 3.02 GB 0  KB/second 37  KB/second 0  MB 4,143.33  MB — — — —
  Scanner 1 99.53% 0% — 1.95 GB 1.96 GB 3.36 GB 0  KB/second 9,906  KB/second 0  MB 4,143.33  MB — — — —
  Scanner 2 4.83% 0.17% — 1.76 GB 198.95 MB 3.58 GB 0  KB/second 3,069  KB/second 58.31  MB 4,085.02  MB — — — —

 

 

 

 

 

 

There is definitely one taliban process which blows up and consumes all CPU on that appliance. How to find out which one? Useful tools like 'top' and 'ps' are missing on the restricted command line. Is there a hidden backdoor for a root shell just for analysis? 

Bye,

Bernd

TSE-JDavis's picture
09
May
2011
2 Votes +2
Login to vote
TSE-JDavis
Symantec Employee
Accredited

Using the set-support command

Using the set-support command you can enable an advanced user called support. This will let you run top and ps.

AdNovum's picture
13
May
2011
0 Votes 0
Login to vote
AdNovum

Thanks. Here are the

Thanks. Here are the processes that consume all CPU on an idle scanner with version 9.5.1-6. Sometimes stunnel consumes 98% CPU.

 

 [support@scanner2 support]$ top
top - 14:34:05 up 7 days,  4:20,  2 users,  load average: 7.45, 7.51, 7.57
Tasks:  90 total,   6 running,  84 sleeping,   0 stopped,   0 zombie
Cpu(s): 41.3% us, 56.7% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  1.9% si
Mem:   4100824k total,  2974404k used,  1126420k free,   688268k buffers
Swap:  4242768k total,        0k used,  4242768k free,  1005720k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                    
 2203 root      20   0  5832 2580 1324 S 49.2  0.1   9802:32 stunnel                                                                                                                                    
 2376 mailwall  20   0 1054m 776m  16m S 14.6 19.4   3098:45 bmserver                                                                                                                                   
 2916 mailwall  20   0  661m 283m  11m S 11.7  7.1   1921:00 ecelerity                                                                                                                                  
 2461 mailwall  20   0  150m  51m  11m R 10.9  1.3   3856:17 conduit                                                                                                                                    
 2253 mailwall  20   0 89856 6500 4604 S  0.3  0.2  19:35.41 bmagent                                                                                                                                    
11313 support   20   0  1936  996  764 R  0.2  0.0   0:00.25 top                                                                                                                                        
11416 mailwall  20   0 89856 1944   48 R  0.2  0.0   0:00.01 bmagent                                                                                                                                    
    1 root      20   0  1540  532  468 S  0.0  0.0   0:13.37 init                                                                                                                                       
    2 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                                                                   
    3 root      RT  -5     0    0    0 S  0.0  0.0   0:03.15 migration/0                                                                                                                                
    4 root      15  -5     0    0    0 S  0.0  0.0   0:20.55 ksoftirqd/0                                                                                                                                
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:02.51 migration/1                                                                                                                                
    6 root      15  -5     0    0    0 S  0.0  0.0   0:10.74 ksoftirqd/1                                                                                                                                
    7 root      15  -5     0    0    0 R  0.0  0.0   0:35.56 events/0                                                                                                                                   
    8 root      15  -5     0    0    0 R  0.0  0.0   0:27.25 events/1                                                                                                                                   
    9 root      15  -5     0    0    0 S  0.0  0.0   0:01.69 khelper                                                                                                                                    
   12 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 kstop/0                                                                                                                                    
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 kstop/1                                                                                                                                    
  150 root      15  -5     0    0    0 S  0.0  0.0   0:02.93 kblockd/0                                                                                                                                  
  151 root      15  -5     0    0    0 S  0.0  0.0   0:01.83 kblockd/1                                                                                                                                  
  153 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid                                                                                                                                     
  154 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kacpi_notify                                                                                                                               
  272 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 ata/0                                                                                                                                      
  273 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 ata/1                                                                                                                                      
  274 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 ata_aux                                                                                                                                    
  275 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 ksuspend_usbd                                                                                                                              
  280 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 khubd                                                                                                                                      
  283 root      15  -5     0    0    0 S  0.0  0.0   0:00.04 kseriod                                                                                                                                    
  289 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kmmcd                                                                                                                                      
  349 root      20   0     0    0    0 S  0.0  0.0   4:42.12 pdflush                                                                                                                                    
  350 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kswapd0                                                                                                                                    
  351 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0                                                                                                                                      
  352 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 aio/1                                                                                                                                      
 1152 root      15  -5     0    0    0 S  0.0  0.0   0:00.01 scsi_eh_0                                                                                                                                  
 1154 root      15  -5     0    0    0 S  0.0  0.0   0:00.04 scsi_eh_1                                                                                                                                  
 1165 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 exec-osm/0                                                                                                                                 
 1166 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 exec-osm/1                                                                                                                                 
 1171 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 block-osm/0                                                                                                                                
 1172 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 block-osm/1                                                                                                                                
 1176 root      15  -5     0    0    0 S  0.0  0.0   0:04.14 mpt_poll_0                                                                                                                                 
 1177 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_2                                                                                                                                  
 1209 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kpsmoused                                                                                                                                  
 1233 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 hid_compat                                                                                                                                 
 1264 root      15  -5     0    0    0 S  0.0  0.0   6:01.92 kjournald                                                                                                                                  
 1605 root      15  -5     0    0    0 S  0.0  0.0   7:31.58 kjournald                                                                                                                                  
 1606 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kjournald                                                                                                                                  
 1607 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kjournald                                                                                                                                  
 1969 root      15  -5     0    0    0 S  0.0  0.0   0:05.34 vmmemctl                                                                                                                                   
 2034 ntp       20   0  4436 4436 3360 S  0.0  0.1   0:17.87 ntpd                                                                                                                                       
 2044 root      20   0  4436 1432  356 S  0.0  0.0   0:02.58 ntpd                                                                                                                                       
 2061 root      20   0  1968  992  728 S  0.0  0.0   0:37.47 syslog-ng                                                                                                                                  
 2103 root      20   0 12152 3004 1396 S  0.0  0.1   2:10.45 snmpd                                                                                                                                      
 2114 root      20   0 15880 1028  632 S  0.0  0.0   0:24.07 runner                                                                                                                                     
 2128 named     20   0 46588  10m 2292 S  0.0  0.3   0:48.58 named                                                                                                                                      
 2148 root      20   0  3700 1096  768 S  0.0  0.0   0:25.07 sshd                                                                                                                                       
 2163 root      20   0  2140  816  676 S  0.0  0.0   0:00.03 xinetd                                                                                                                                     
 2173 root      20   0  3012 1608 1380 S  0.0  0.0   4:25.67 vmtoolsd                                                                                                                                   
 2181 root      20   0  1580  592  504 S  0.0  0.0   0:13.84 crond                                                                                                                                      
 2223 mailwall  20   0 15880 1132  732 S  0.0  0.0   0:24.20 runner                                                                                                                                     
 2224 mailwall  20   0 1862m  55m 7900 S  0.0  1.4  12:42.06 java

Found that tech note that describes the high CPU usage of stunnel:

https://www-secure.symantec.com/connect/forums/sbg...

http://www.symantec.com/docs/TECH123405

I created the SSL private keys for the SSL certificates with 2048 bits which is not the default.Maybe that caused the trouble.

 [support@scanner2 /]$ cat /data/os/etc/stunnel/apt.conf
client = yes
verify = 2
cafile = /data/scanner/etc/trusted.cert
cert   = /data/scanner/etc/cert.pem

[pseudo-https]
accept  = 127.0.0.1:8080
connect = proxy.example.com:3128
httpsproxy_dest = swupdate.brightmail.com:443

[support@scanner2 /]$ ls -al /data/scanner/etc/trusted.cert /data/scanner/etc/cert.pem
-rw-------    1 mailwall bmi          1888 May  3 22:45 /data/scanner/etc/cert.pem
-rw-r--r--    1 mailwall bmi        441490 Apr 30 11:07 /data/scanner/etc/trusted.cert 

[support@scanner2 /]$ /usr/sbin/stunnel -version
stunnel 4.05 on i686-pc-linux-gnu PTHREAD+LIBWRAP with OpenSSL 0.9.7a Feb 19 2003
 
Global options
cert            = /etc/stunnel/stunnel.pem
ciphers         = ALL:!ADH:+RC4:@STRENGTH
debug           = 5
key             = /etc/stunnel/stunnel.pem
pid             = /var/run/stunnel.pid
RNDbytes        = 64
RNDfile         = /dev/urandom
RNDoverwrite    = yes
session         = 300 seconds
verify          = none
 
Service-level options
TIMEOUTbusy     = 300 seconds
TIMEOUTclose    = 60 seconds
TIMEOUTidle     = 43200 seconds

 

Bye

Bernd

AdNovum's picture
06
May
2011
0 Votes 0
Login to vote
AdNovum

Upgrading to 9.5.1-6 did not

Upgrading to 9.5.1-6 did not solve the issue.

Scanner 1 uses 100% CPU while doing noting and Scanner 2 uses just a little CPU while also doing nothing. Both scanners are setup with the same settings - just different networks.

Bye

Bernd

mjpsalm's picture
06
May
2011
0 Votes 0
Login to vote
mjpsalm

Symantec Messaging Gateway 9.5.0-23: 100% CPU

AdNovum, Although I don't know how to render the stats that you have given (web gui?), from within vSphere i can see how my CPU utilization has doubled and mostly 100% and my memory has been tapped out. Overall, the system is slow. Plenty of disk space. I am going to upgrade to 9.5.1-6, because my guess is that's what symantec will tell me...? Any updates to the issue?

GeSchmidtt's picture
10
May
2011
0 Votes 0
Login to vote
GeSchmidtt

Symantec Messaging Gateway 9.5.0-23: 100% CPU

Have you had any luck with this problem.  I am experiencing the same thing on version 9.5.0-23.  We are in test mode on a new sparsely populated VMWare cluster.  We have been sending and reciving a couple of messages a day.  CPU and disk access are 10-15 times higher than on any of our other machines.  I am seeing this in VMWare performance screen as well as our normal monitoring system.  In contrary, our 6 year old version, which is processing ±150.000 messages a day is running at 5% CPU load. It seems highly unlikely that this is a cosmetic issue.  Did upgrading to the 9.5.1 help? 

AdNovum's picture
08
May
2011
0 Votes 0
Login to vote
AdNovum

No news so far... except the

No news so far

... except the KB article above that this is just a "cosmetic issue". Can't open a support case on mysupport.symantec.com. Login causes an application error since last week. Is there somebody at home?

TSE-JDavis's picture
09
May
2011
1 Vote +1
Login to vote
TSE-JDavis
Symantec Employee
Accredited

We are here. You should call

We are here. You should call us if you can't open a web case.

http://www.symantec.com/business/support/contact_t...

GeSchmidtt's picture
11
May
2011
0 Votes 0
Login to vote
GeSchmidtt

Resolution of issue for me

Updating from  9.5.0-23 to  9.5.1-6 resolved the issue seems to have resolved the issue. Processor is running at between 10 -20% (still seems like a lot of activity for it not receiving mail yet) but certainly better than 65 - 85%.

Lumilux's picture
18
May
2011
0 Votes 0
Login to vote
Lumilux

High CPU usage even in 9.51-6

High CPU usage even in 9.51-6 no fix until now, using VM ESXi 4.1U1

monitored a 24% CPU usage of bmserver

and 20% CPU usage of conduit with TOP

and a high disk activity shown at vSphere Client

 

Best regards

Michael

AdNovum's picture
19
Jul
2011
0 Votes 0
Login to vote
AdNovum

Hi, No update on this issue.

Hi,

No update on this issue. We still have 100% CPU load on idle systems. There is no update since 9.5.1-6. I hesitate to migrate from Symantec Mail Security 5 on Sparc Solaris to this VMware Appliance until there is no fix. Are we the only ones on this planet who have this version running?

 

Host CPU
Usage		 CPU
Iowait		 Memory
Usage		 Memory
Free		 Disk
Usage		 Disk Read Disk Write Swap
Used		 Swap
Free
Scanner1 50.67% 0% 2.95 GB 987.56 MB 4.11 GB 0  KB/second 23,970  KB/second 0  MB 4,143.33  MB
Control Center 100% 0% 1.37 GB 2.54 GB 3.44 GB 0  KB/second 16  KB/second 0  MB 4,143.33  MB
Scanner2 100% 0% 3.38 GB 541.82 MB 4.32 GB 0  KB/second 11,096  KB/second 0  MB 4,143.33  MB

Best regards,

Bernd

eccen3k's picture
19
Jul
2011
0 Votes 0
Login to vote
eccen3k

Hello   I am also

Hello

 

I am also experiencing the very exact problem.

I am using version  9.5.1-6.

 

I hope someone could post a solution.

 

Thanks.

TSE-JDavis's picture
19
Jul
2011
1 Vote +1
Login to vote
TSE-JDavis
Symantec Employee
Accredited

You really need to call in

You really need to call in and open a case to talk to us about this. This is not something we can give an easy answer about since there are not many people seeing this issue.

Alex-K's picture
20
Jul
2011
0 Votes 0
Login to vote
Alex-K

High CPU - Me too

On Virtual Edition:

I also experience high CPU utilization, especially on Control Center, but not only, 2nd, scanner only, instance also jumpt to the sky (vsphere keep warning on CPU Usage).

9.5.1-6, 4vCPU 2Gb RAM.
(Host: DELL2950 III - ~30% utilization of the hardware)

vSphere 4.1U1.
 

 

Host CPU
Usage		 CPU
Iowait		 CPU
Temp		 Memory
Usage		 Memory
Free		 Disk
Usage		 Disk Read Disk Write Swap
Used		 Swap
Free		 Fan Battery Power
Supply		 RAID
mx02 100% 25.5% — 1.91 GB 54.46 MB 4.74 GB 13,301  KB/second 733  KB/second 1,520.78  MB 2,622.55  MB — — — —
mx 52.67% 45.67% — 1.83 GB 136.76 MB 3.43 GB 14,544  KB/second 1,050  KB/second 1,636.67  MB 2,506.66  MB — — — —

 

Thanks,

Alex.

AdNovum's picture
20
Jul
2011
0 Votes 0
Login to vote
AdNovum

I assume this high CPU issue

I assume this high CPU issue is related to SSL encryption. Installation is pretty much default setup - except for SSL. I chose a 2048 bit private key. A "top" shows that stunnel is consuming most of the CPU.

This tech note from January last year states that a restart of stunnel would fix this:

http://www.symantec.com/business/support/index?pag...

But after a stunnel restart the high CPU load persists.

 

Bye

Bernd

amantsios's picture
02
Aug
2011
0 Votes 0
Login to vote
amantsios

Same problem here also. Just

Same problem here also.

Just installed 9.5.1-6 virtual appliance and the CPU is high.

As seen from the top command the mailwall is using 50% of the cpu in every two minutes.

I also would like to mention that is a test environment and the brightmail is idle from scans and mails.

Technical Support

Symantec.com

Store

Community Stats

Total Content Items
Members
270,012