Endpoint Encryption

 View Only
  • 1.  PGP Command Line 9.5 and AIX 5.3 CPU 100% (non-root user)

    Posted Nov 14, 2011 08:58 AM

    PGP Command Line 9.5.2 (AIX) hangs decrypting a 36 MByte compressed armored file

     

    We are using "PGP Command Line 9.5.2 build 654, (c) 2005" for both file encryption and decryption on AIX 5.3.

     

    We use PGP to encrypt a 267 MByte file down into 36 MByte compressed armored file.

    The PGP encryption command exits with zero (success).

     

    We try to decrypt this 36 MByte compressed armored file with PGP.

    PGP hangs.

     

    We try the same decryption with the "--verbose" flag.

     

    The PGP decrypt command logs that it has opened the keyrings and then logs:

     

    "Decoding file /ftp/omnipay/in/gcms_tmp.oenc... 1% (... MB)"

     

    The percentage climbs 1%, 2% up to 45% promptly and then hangs at:

     

    "Decoding file /ftp/omnipay/in/gcms_tmp.oenc... 45% (15.5MB)"

     

    A system monitoring tool ("topas") shows that the pgp decrypt process is still consuming CPU time (about 80% CPU) despite the apparent hang (perhaps an infinite loop??).

     

    How can we get PGP to decrypt this file?



  • 2.  RE: PGP Command Line 9.5 and AIX 5.3 CPU 100% (non-root user)

    Posted Nov 14, 2011 12:16 PM

    I'm moving this thread to the PGP Command Line forum (where it more properly belongs).



  • 3.  RE: PGP Command Line 9.5 and AIX 5.3 CPU 100% (non-root user)

    Posted Nov 14, 2011 12:19 PM

    Does PGP Command Line completely hang?  Or does it finish processing after some delay (say a few minutes)?

    Does the problem occur for the root uers?

    Does the problem occur if you run in local mode, with the "--local-mode" option?

    Does PGP Command Line complete successfully if you dump the file packets?  Run "pgp --dump-packets filename" and see if it completes and reports any errors.



  • 4.  RE: PGP Command Line 9.5 and AIX 5.3 CPU 100% (non-root user)

    Posted Nov 15, 2011 04:35 AM

    Thanks dfinkelstein, not a regular visitor here :)
    I have some more information and the answers to your questions:

    Does PGP Command Line completely hang?  Or does it finish processing after some delay (say a few minutes)?

    After the hang I let it run for 20 minutes. It made no further progress during those 20 minutes (despite grabbing as much CPU as it could), so that looks like a complete hang.

    Does the problem occur for the root uers?

    Yes the same hang occurs for root too at the same place.

    I left it running for over an hour as root, but it still hung in the same place and made no further progress.

    Does the problem occur if you run in local mode, with the "--local-mode" option?

    Yes the same hang occurs if "--local-mode" is explicitly specified.

    Does PGP Command Line complete successfully if you dump the file packets?  Run "pgp --dump-packets filename" and see if it completes and reports any errors.

    Yes a dump packets on the encrypted file produces output and completes successfully (exit code 0) after running for about 12 seconds.

    Last 2 lines logged are:

    ...

    New:    (773 bytes) partial end

    gcms_tmp.oenc:dump packets (0:packets decoded successfully)

     

    NEW INFORMATION

    I found that by changing "--output -" (i.e. send decrypted file to stdout) to "--output a_specific_file_path", PGP could decrypt the file OK.

    So that is a workaround, but I'd rather be able to use "--output -" on large files too if at all possible.



  • 5.  RE: PGP Command Line 9.5 and AIX 5.3 CPU 100% (non-root user)

    Posted Nov 15, 2011 11:33 AM

     

    We've seen some related problems on AIX with larger files.  One solution is to set LDR_CNTRL to a higher value.
     
    See the following article:
     
    http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame3.doc_5.1/am51_perftune113.htm
     
    Your original file is 256MB but the default value of LDR_CNTRL sets a process memory limit of 256MB.  We buffer stdio on AIX, so when writing to stdout the data needs to be buffered inside PGP Command Line.  As explained in that article, AIX only allows a single memory segment per process, and you have to set an environment variable to override it.
     
    Try
     
    shell$ export LDR_CNTRL=MAXDATA=0x30000000
     
    and see if that improves the behavior.
     
    It is on our roadmap to improve stdio handling in a future version of PGP Command Line.