HOWTO: Diagnose performance issues with PGP Universal Server

Article:HOWTO60769  |  Created: 2011-11-03  |  Updated: 2011-11-09  |  Article URL http://www.symantec.com/docs/HOWTO60769
Article Type
How To



PGP Universal Server provides several tools that can assist with diagnosing performance issues via the command line. Some common symptoms of hardware-related performance issues include delays in email delivery, slow response times or timeouts when accessing the web console, and unexpected errors in various logs within PGP Universal Server.

 
 
top
 
The top utility allows you to see the current CPU load, RAM usage, swap usage, and running processes on your Universal Server. It will have output similar to the following:
 
top - 06:26:07 up 27 days, 18:38,  1 user,  load average: 0.15, 0.09, 0.08
Tasks: 332 total,   1 running, 331 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.5%us,  4.0%sy,  0.0%ni, 94.3%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   1035124k total,   841168k used,   193956k free,    71316k buffers
Swap:  2096472k total,    84960k used,  2011512k free,   217816k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23889 root      16   0  2752 1032  676 R  3.0  0.1   0:00.04 top
    1 root      18   0  2168  624  596 S  0.0  0.1   0:02.48 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.25 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.40 events/0
    6 root      19  -5     0    0    0 S  0.0  0.0   0:00.00 khelper
    7 root      17  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
   10 root      10  -5     0    0    0 S  0.0  0.0   0:10.34 kblockd/0
   11 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid
  173 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/0
  176 root      10  -5     0    0    0 S  0.0  0.0   0:00.02 khubd
  178 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 kseriod
  240 root      19   0     0    0    0 S  0.0  0.0   0:00.01 khungtaskd
  242 root      16   0     0    0    0 S  0.0  0.0   3:22.49 pdflush
  243 root      10  -5     0    0    0 S  0.0  0.0   0:07.42 kswapd0
  244 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
  325 root      16   0  4916 1488 1236 S  0.0  0.1   0:00.09 bash
  462 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kpsmoused
  492 root      10  -5     0    0    0 S  0.0  0.0   0:00.27 mpt_poll_0
  493 root      13  -5     0    0    0 S  0.0  0.0   0:00.00 mpt/0
  494 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_0
  497 root      12  -5     0    0    0 S  0.0  0.0   0:00.00 ata/0
 
The CPU(s) line displays current CPU usage. The id field indicates how much of the CPU is idle. If this number is very low, it may mean you don't have enough CPU resources for PGP Universal Server. Upgrading the CPU or allocating additional CPU resources (for virtual machines) will help with this.
 
The Mem line displays current RAM usage. If the free field is low, it may indicate that you need to add or allocate additional memory to the server. Because PGP Universal Server uses PAE kernel extensions, it can support up to 64GB of RAM despite using a 32-bit version of Linux.
 
The Swap line displays current swap file usage in much the same way that RAM is displayed. If the Mem free is low and the Swap used is high, you will see a significant decrease in performance and should add or allocation additional RAM as soon as possible. Excessive swapping can also lead to system lockups as the system becomes non-responsive.
 
Beneath the swap line is a list of currently running processes. By deault, this will be sorted by CPU usage and will refresh about every 4 seconds. This can help identify which processes are using the most CPU time. By using the greater than (>) and less than (<) keys on your keyboard, you can change the order in which processes are sorted. From the default view, the greater than key will change the sort order to memory usage.
 
Once you are done reviewing the information in top, press q to quit the program.
 
sar
 
The sar utility will display history system usage statistics including CPU usage, RAM usage, and disk activity. This can help pinpoint specific bottlenecks in performance. PGP Universal Server retains data over a seven day period to help isolate potential hardware utilization issues.
 
To display CPU usage for the current day, run sar without any arguments. You will get output similar to the following:
 
Linux 2.6.18-194.26.1.el5PAE (keys.lab.jharris.local)               11/03/2011
12:00:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM       all      2.08      0.00      4.43      0.30      0.00     93.18
12:20:01 AM       all      1.45      0.00      3.80      0.03      0.00     94.72
12:30:01 AM       all      1.42      0.00      3.84      0.02      0.00     94.72
12:40:01 AM       all      1.45      0.00      3.99      0.03      0.00     94.54
12:50:02 AM       all      1.52      0.00      3.67      0.04      0.00     94.78
01:00:01 AM       all      1.53      0.00      3.84      0.02      0.00     94.62
01:10:01 AM       all      1.56      0.00      3.91      0.02      0.00     94.51
01:20:01 AM       all      1.57      0.00      3.90      0.04      0.00     94.49
01:30:02 AM       all      1.40      0.00      3.76      0.02      0.00     94.82
01:40:01 AM       all      1.88      0.00      4.44      0.04      0.00     93.64
01:50:01 AM       all      1.50      0.00      3.89      0.03      0.00     94.58
02:00:01 AM       all      1.46      0.00      3.71      0.03      0.00     94.80
02:10:01 AM       all      1.57      0.00      3.77      0.02      0.00     94.64
02:20:01 AM       all      1.55      0.00      3.84      0.02      0.00     94.59
02:30:01 AM       all      1.49      0.00      3.79      0.02      0.00     94.70
02:40:01 AM       all      1.48      0.00      3.69      0.02      0.00     94.81
02:50:01 AM       all      1.51      0.00      3.89      0.02      0.00     94.57
03:00:01 AM       all      1.45      0.00      3.84      0.02      0.00     94.68
03:10:01 AM       all      6.63      0.00      6.72      0.46      0.00     86.20
03:20:01 AM       all      1.52      0.00      5.00      0.02      0.00     93.47
03:30:02 AM       all      1.45      0.00      4.88      0.03      0.00     93.64
03:40:01 AM       all      1.35      0.00      4.94      0.02      0.00     93.68
03:50:01 AM       all      1.47      0.00      5.05      0.02      0.00     93.46
04:00:01 AM       all      1.43      0.00      4.90      0.02      0.00     93.65
04:10:01 AM       all      1.64      0.00      5.30      1.28      0.00     91.78
04:20:01 AM       all      1.46      0.00      4.72      0.02      0.00     93.80
04:30:01 AM       all      1.41      0.00      4.82      0.03      0.00     93.74
04:40:01 AM       all      1.51      0.00      4.90      0.03      0.00     93.56
04:50:02 AM       all      1.36      0.00      4.79      0.02      0.00     93.83
05:00:01 AM       all      1.45      0.00      4.84      0.02      0.00     93.69
05:10:01 AM       all      1.48      0.00      4.88      0.03      0.00     93.62
05:20:01 AM       all      1.41      0.00      4.70      0.02      0.00     93.86
05:30:01 AM       all      1.53      0.00      4.74      0.02      0.00     93.72
05:40:01 AM       all      1.48      0.00      4.97      0.09      0.00     93.46
05:50:01 AM       all      1.51      0.00      4.90      0.03      0.00     93.56
06:00:01 AM       all      1.42      0.00      4.67      0.03      0.00     93.88
06:10:01 AM       all      1.47      0.00      4.92      0.04      0.00     93.57
06:20:01 AM       all      1.55      0.00      4.68      0.03      0.00     93.74
06:30:02 AM       all      1.45      0.00      4.69      0.03      0.00     93.83
06:40:01 AM       all      1.71      0.00      5.07      0.02      0.00     93.20
06:50:01 AM       all      1.74      0.00      5.06      0.02      0.00     93.18
07:00:02 AM       all      1.46      0.00      4.56      0.03      0.00     93.95
07:10:02 AM       all      1.73      0.00      4.71      0.05      0.00     93.50
Average:          all      1.64      0.00      4.50      0.07      0.00     93.79
 
This displays the CPU usage every 10 minutes for the current day. The idle column shows how much CPU is unused. A low number could indicate that the CPU is not powerful enough to process all of the requests coming to the server. The user and system columns show how much CPU is being used by programs and the OS, respectively.
 
The iowait column indicates how much CPU time was spent waiting for requested data to be fetched from the hard drive. A high number can indicate that the hard drive is too slow. This can be resolved either with a faster hard drive or by building a RAID array to increase performance. RAID 0, 5, 10, and 50 all provide increased disk performance. It may also indicate a problem with drive health such as bad sectors or a failing drive. If this is the case, the dmesg log (/var/log/dmesg) may show error messages related to the drive. You may also see messages about running a file system check when the server is booting up.
 
You can use sar to help identify these kinds of disk bottlenecks. Run sar -W to get a list of disk swapping activity. Using sar -b will show how many transactions per second are hitting the hard drive. These can both help isolate swapping and drive performance issues, including if issues happen during a certain peak time.
 
You can also use sar to pinpoint memory or swap utilization issues. Run sar -r and you will get output similar to the following:
 
Linux 2.6.18-194.26.1.el5PAE (keys.lab.jharris.local)        11/03/2011
 
12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
12:10:01 AM    143368    891756     86.15    102852    231664   2014236     82236      3.92     18760
12:20:01 AM    143740    891384     86.11    103128    231792   2014236     82236      3.92     18760
12:30:01 AM    142624    892500     86.22    103424    231840   2014236     82236      3.92     18760
12:40:01 AM    142624    892500     86.22    103756    231924   2014236     82236      3.92     18760
12:50:02 AM    142376    892748     86.25    103960    232056   2014236     82236      3.92     18760
01:00:01 AM    138904    896220     86.58    104232    232104   2014236     82236      3.92     18760
01:10:01 AM    141384    893740     86.34    104512    232252   2014236     82236      3.92     18760
01:20:01 AM    140640    894484     86.41    104784    232412   2014236     82236      3.92     18760
01:30:02 AM    135060    900064     86.95    105028    232464   2014236     82236      3.92     18760
01:40:01 AM    139100    896024     86.56    105304    232948   2014236     82236      3.92     18920
01:50:01 AM    138356    896768     86.63    105512    233084   2014236     82236      3.92     18920
02:00:01 AM    133148    901976     87.14    105756    233256   2014236     82236      3.92     19020
02:10:01 AM    138852    896272     86.59    106000    233368   2014236     82236      3.92     19020
02:20:01 AM    138604    896520     86.61    106184    233484   2014236     82236      3.92     19020
02:30:01 AM    136496    898628     86.81    106420    233516   2014236     82236      3.92     19020
02:40:01 AM    137984    897140     86.67    106640    233656   2014236     82236      3.92     19020
02:50:01 AM    137488    897636     86.72    106884    233764   2014236     82236      3.92     19020
03:00:01 AM    133644    901480     87.09    107068    233812   2014236     82236      3.92     19020
03:10:01 AM    231464    803660     77.64     54184    196256   2011508     84964      4.05     19172
03:20:01 AM    231232    803892     77.66     54756    196584   2011508     84964      4.05     19172
03:30:02 AM    229504    805620     77.83     55324    196608   2011508     84964      4.05     19172
03:40:01 AM    229820    805304     77.80     55860    196736   2011508     84964      4.05     19172
03:50:01 AM    228948    806176     77.88     56440    196832   2011508     84964      4.05     19172
04:00:01 AM    226884    808240     78.08     56968    196880   2011508     84964      4.05     19172
04:10:01 AM    203340    831784     80.36     64196    214652   2011508     84964      4.05     19172
04:20:01 AM    201976    833148     80.49     64728    214760   2011508     84964      4.05     19172
04:30:01 AM    202224    832900     80.46     65288    214784   2011508     84964      4.05     19172
04:40:01 AM    200860    834264     80.60     65756    214904   2011508     84964      4.05     19172
04:50:02 AM    201108    834016     80.57     66264    215000   2011508     84964      4.05     19172
05:00:01 AM    197016    838108     80.97     66848    215052   2011508     84964      4.05     19172
05:10:01 AM    198256    836868     80.85     67360    215172   2011508     84964      4.05     19172
05:20:01 AM    199496    835628     80.73     67868    215272   2011508     84964      4.05     19172
05:30:01 AM    199000    836124     80.78     68372    215304   2011508     84964      4.05     19172
05:40:01 AM    196516    838608     81.02     68968    216768   2011508     84964      4.05     19172
05:50:01 AM    195772    839352     81.09     69428    216932   2011512     84960      4.05     19200
06:00:01 AM    190812    844312     81.57     69960    216960   2011512     84960      4.05     19200
06:10:01 AM    195100    840024     81.15     70488    217348   2011512     84960      4.05     19200
06:20:01 AM    194604    840520     81.20     70976    217444   2011512     84960      4.05     19200
06:30:02 AM    190772    844352     81.57     71520    217832   2011512     84960      4.05     19200
06:40:01 AM    192384    842740     81.41     72068    217964   2011512     84960      4.05     19200
06:50:01 AM    191020    844104     81.55     72636    218052   2011512     84960      4.05     19200
07:00:02 AM    188644    846480     81.78     73084    218140   2011512     84960      4.05     19200
07:10:02 AM    185420    849704     82.09     73632    220064   2011512     84960      4.05     19200
Average:       176897    858227     82.91     82196    220505   2012651     83821      4.00     19055
 
The columns are, in order, RAM free, RAM used, percentage of RAM used, RAM used for buffers, RAM used for caching, swap free, swap used, percentage of swap used, and swap cached. A high %memused and/or %swpused indicate that you need more RAM to improve performance.
 
df
 
The df utility is used to show how much drive space is available and how much of it is being used. Disks with low free space may perform worse as file fragmentation will often be higher. Here is a sample of output using df -h:
 
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              37G  3.5G   32G  11% /
/dev/sda1              99M   17M   77M  18% /boot
tmpfs                 506M  8.0K  506M   1% /dev/shm
 
In this example, the main partition on the hard drive, /dev/sda2, is 37GB with 11% in use. The -h argument shows the output in a friendly human-readable format. If you are running into disk space issues, you should consider upgrading to a larger hard drive or expanding your RAID array, if you have one.

 

 

 

 



Article URL http://www.symantec.com/docs/HOWTO60769


Terms of use for this information are found in Legal Notices