If you really don't (or can't) trust the system administrators, then you have to assume that any data on the system is available to them, as is any information you type into a keyboard. Encrypted key material, passphrases in files, passphrases you type in and cache, are all exposed. Is this really your threat? Any plaintext that ever exists on that system, even if only is memory, is exposed to such an untrusted administrator.
If your threat model was less strict, one thing you could do is cache the private key passphrase. You'd have to enter it yourself once, but it could be cached for as long as the system was running. Note that the actual passphrase isn't cached; rather, the passphrase is used to generate a symmetric key that unlocks the key material. It is this symmetric key that is cached.
The best solution we currently have for the "I don't trust the administrators" use case (which is similar to the "What happens if my machine is compromised? I don't want my private key and passhrase to be exposed" use case) is to use PGP Command Line in conjunction with PGP KMS. Your private key is stored in the KMS server, and locally you only hold an authentication credential that allows you perform session key decryption on the server. This way, the only credential that can ever be compromised is an authentication credentail. True this credential is available to the local administrator, but if, by checking the KMS logs, you believe that they are using it then you can revoke the credential on the KMS server. Even if one particular file was compromised, the key itself was not.