Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

PGP Command Line 9.5 and AIX 5.3 CPU 100% (non-root user)

Created: 14 Nov 2011 • Updated: 14 Nov 2011 | 4 comments

PGP Command Line 9.5.2 (AIX) hangs decrypting a 36 MByte compressed armored file

 

We are using "PGP Command Line 9.5.2 build 654, (c) 2005" for both file encryption and decryption on AIX 5.3.

 

We use PGP to encrypt a 267 MByte file down into 36 MByte compressed armored file.

The PGP encryption command exits with zero (success).

 

We try to decrypt this 36 MByte compressed armored file with PGP.

PGP hangs.

 

We try the same decryption with the "--verbose" flag.

 

The PGP decrypt command logs that it has opened the keyrings and then logs:

 

"Decoding file /ftp/omnipay/in/gcms_tmp.oenc... 1% (... MB)"

 

The percentage climbs 1%, 2% up to 45% promptly and then hangs at:

 

"Decoding file /ftp/omnipay/in/gcms_tmp.oenc... 45% (15.5MB)"

 

A system monitoring tool ("topas") shows that the pgp decrypt process is still consuming CPU time (about 80% CPU) despite the apparent hang (perhaps an infinite loop??).

 

How can we get PGP to decrypt this file?

Comments 4 CommentsJump to latest comment

dfinkelstein's picture

I'm moving this thread to the PGP Command Line forum (where it more properly belongs).

--------

David Finkelstein

Symantec R&D

dfinkelstein's picture

Does PGP Command Line completely hang?  Or does it finish processing after some delay (say a few minutes)?

Does the problem occur for the root uers?

Does the problem occur if you run in local mode, with the "--local-mode" option?

Does PGP Command Line complete successfully if you dump the file packets?  Run "pgp --dump-packets filename" and see if it completes and reports any errors.

--------

David Finkelstein

Symantec R&D

Symomni's picture

Thanks dfinkelstein, not a regular visitor here :)
I have some more information and the answers to your questions:

Does PGP Command Line completely hang?  Or does it finish processing after some delay (say a few minutes)?

After the hang I let it run for 20 minutes. It made no further progress during those 20 minutes (despite grabbing as much CPU as it could), so that looks like a complete hang.

Does the problem occur for the root uers?

Yes the same hang occurs for root too at the same place.

I left it running for over an hour as root, but it still hung in the same place and made no further progress.

Does the problem occur if you run in local mode, with the "--local-mode" option?

Yes the same hang occurs if "--local-mode" is explicitly specified.

Does PGP Command Line complete successfully if you dump the file packets?  Run "pgp --dump-packets filename" and see if it completes and reports any errors.

Yes a dump packets on the encrypted file produces output and completes successfully (exit code 0) after running for about 12 seconds.

Last 2 lines logged are:

...

New:    (773 bytes) partial end

gcms_tmp.oenc:dump packets (0:packets decoded successfully)

 

NEW INFORMATION

I found that by changing "--output -" (i.e. send decrypted file to stdout) to "--output a_specific_file_path", PGP could decrypt the file OK.

So that is a workaround, but I'd rather be able to use "--output -" on large files too if at all possible.

dfinkelstein's picture

 

We've seen some related problems on AIX with larger files.  One solution is to set LDR_CNTRL to a higher value.
 
See the following article:
 
http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame3.doc_5.1/am51_perftune113.htm
 
Your original file is 256MB but the default value of LDR_CNTRL sets a process memory limit of 256MB.  We buffer stdio on AIX, so when writing to stdout the data needs to be buffered inside PGP Command Line.  As explained in that article, AIX only allows a single memory segment per process, and you have to set an environment variable to override it.
 
Try
 
shell$ export LDR_CNTRL=MAXDATA=0x30000000
 
and see if that improves the behavior.
 
It is on our roadmap to improve stdio handling in a future version of PGP Command Line.
 

--------

David Finkelstein

Symantec R&D