1. Symantec/
  2. Update on Test Certs Incident

Update on Test Certificate Incident

Update on Test Certs Incident

Updated May 25, 2016

May 25, 2016 Update

KPMG has completed our annual WebTrust audit for December 1, 2014 through November 30, 2015. Those reports are available here. KPMG identified several issues during the course of their audit, including those related to the test certificate incident, and we have addressed each of these issues.
As stated previously, since this issue first arose, we have implemented changes to our test certificate policies, processes, and controls designed to prevent this from happening again.
Further, as indicated in our more detailed response, we have enhanced our compliance function by consolidating compliance activities into a single group reporting directly to the head of our Website Security business unit. The new compliance structure includes enhanced identification, tracking, prioritization, and resolution of compliance-related updates, which will help ensure that changes to CA/Browser forum requirements and to Symantec policies and procedures are effectively implemented. We will continue to further evaluate and strengthen those policies, procedures, and controls.
Finally, we have engaged KPMG to begin a separate Point in Time Readiness Assessment to validate that our processes and systems are operating in accordance within the appropriate CA/Browser Forum requirements and Symantec’s Certificate Practices Statements & Certificate Policies.
We will post the results of the Point In Time Readiness Audit here once that is complete.

April 21, 2016 Update

As we previously disclosed, Symantec learned in September 2015 that it had generated a number of internal test certificates in a manner not fully consistent with its policies. These test certificates included certificates to unregistered domains and domains for which Symantec did not have authorization from the domain owner. As a result, we commenced an investigation to identify and revoke mis-issued certificates, to determine and remediate the root causes of the mis-issuances, and to confirm that no harm had resulted from the incident.

Our Investigation

We have completed our investigation to identify the mis-issued test certificates and the root causes behind their issuance. Our investigation uncovered no evidence of malicious intent, nor harm to anyone. No customer or partner action is needed.
The investigation began with a review of certificates that were issued as part of our internal testing processes back to 2009. These certificates were analyzed by our engineering and authentication leads to identify any test certificates mis-issued to unregistered domains and domains for which Symantec did not have authorization from the domain owner.
We subsequently engaged Deloitte Financial Advisory Services LLP (Deloitte) to help us identify any additional mis-issued active test certificates and to help us search for any mis-issued active certificates that were issued for purposes other than internal testing. The Deloitte team who assisted us with this analysis and certificate documentation review was comprised of consultants experienced in business analytics, public key infrastructure (PKI), and cyber risk.
Deloitte analyzed the approximately 2.18 million digital certificates issued by Symantec that were active as of November 3, 2015. Deloitte used a recommended risk-scoring approach approved by Symantec that was designed to isolate certificates that were at higher risk for being mis-issued, and then manually reviewed documentation provided by Symantec for those certificates. Deloitte subjected the remainder of the active certificates to statistical sampling and manually reviewed the certificate documentation for the sample selection.

Investigation Results

Through our investigation, we confirmed that all of the mis-issued certificates identified were issued for internal Symantec testing purposes. These test certificates have been revoked or are expired. Further, we have contacted the relevant domain owners for the owned domains, and we have and will continue to work with the browser community to blacklist these test certificates where they deem appropriate.
The certificate details are available here for reference, including certificates identified since our last report on October 29, 2015 which are identified in the column titled “Apr 2016”:

Root Cause

All the mis-issued test certificates identified were issued for internal Symantec testing purposes. Nevertheless, they were generated in a manner that was not fully consistent with Symantec’s policies and CA/Browser Forum requirements.
We have identified three root causes underlying the mis-issuance of these test certificates. First, we continued to issue internal test certificates to unregistered domains after the April 2014 change in the Baseline Requirements that removed authorization to do so. The overwhelming majority of the mis-issued test certificates fall into this first category. Second, certain Symantec Quality Assurance (QA) personnel had systems access, including the ability to use certain legacy tools, which enabled them to request a limited number of test certificates that were issued without review by authentication personnel. Third, authentication personnel did not consistently follow all verification steps when they received test certificate requests from their Symantec colleagues, or requested test certificates themselves.

Remediation

Symantec has implemented changes to our test certificate policies, processes, and controls designed to prevent this from happening again, and we will continue to further evaluate and strengthen those policies, procedures, and controls. Specifically, Symantec has removed the systems access and disabled the legacy technical tools that enabled certain QA personnel to request a limited number of test certificates without review by authentication personnel. Symantec has also updated its internal policies, procedures, and trainings to clarify the April 2014 change in the Baseline Requirements that removed authorization to issue certificates to unregistered domains. Additionally, Symantec has updated its internal policies, procedures, and trainings to strongly reinforce that all test certificates must follow the same fulsome authentication procedures as commercial certificates. Symantec has also confirmed that each identified domain that was used for testing purposes in a manner not fully consistent with our policies can no longer be used for new certificates without first undergoing standard authentication and issuance procedures.
We have also worked with KPMG during our annual Web Trust audit, where Symantec’s certificate management practices and internal controls are extensively reviewed. In addition, KPMG will soon be conducting a separate Point in Time readiness assessment.

Enhancing Certificate Transparency and Security

Symantec remains fully committed to the continued trust of our roots across browsers and enhancing the transparency and security of the global certificate infrastructure. For example, we have expanded support for Certificate Transparency to all SSL/TLS certificate types and customer channels. Certificate Transparency (CT) is an open framework that was created to help organizations get a comprehensive view of what active certificates exist for domains that they own. We have provided CT support for all Symantec, Thawte, and GeoTrust Extended Validation (EV) certificate offerings since December 2014, and we are one of the few organizations to operate public CT log servers. We have also now expanded CT support to our Organization Validated (OV) and Domain Validated (DV) products under each of these brands.
Symantec has also promoted and implemented support for Certification Authority Authorization (CAA), which enables domain owners to specify which CAs may issue certificates for their domains. CAA provides another tool for domain owners to help prevent certificates mis-issued to their domains, but it works in practice only if all CAs support it and explicitly honor customers’ preferences. Symantec has been a champion of CAA, and we will be submitting a proposal to the CA/Browser Forum for a rule change to require that all CAs explicitly support CAA.

Investigation Results Details

We received a request from a browser to provide additional details regarding the specific root causes for the mis-issuance of test certificates and our remediation of those root causes. In response to that request, we provide additional details below.
We have identified three root causes underlying the mis-issuance of these test certificates. Symantec has implemented changes to our test certificate policies, processes, and controls designed to prevent this from happening again, and we will continue to further evaluate and strengthen those policies, procedures, and controls.
Issue 1: Section 7.1.4.2.1 of the Baseline Requirements states that, “The CA MUST confirm that the Applicant controls the Fully-Qualified Domain Name or IP address…”.
We continued to issue internal test certificates to unregistered domains after the April 2014 change in the Baseline Requirements that removed authorization to do so. The overwhelming majority of the mis-issued test certificates identified fall into this first category.
Remediation
  • All of the identified test certificates have expired or been revoked.
  • Symantec updated its internal policies, procedures, and trainings to clarify the April 2014 change in the Baseline Requirements that removed authorization to issue certificates to unregistered domains.
  • We have also enhanced our compliance function by consolidating all compliance activities into a single group reporting directly to the head of our Website Security business unit. This change was made in January 2016; this new compliance structure includes enhanced identification, tracking, prioritization and resolution of compliance-related updates, which will help ensure that CA/Browser Forum rule changes are effectively implemented.
In addition, our April 21, 2016 update included six (6) test certificates that were issued to the unregistered domains symantectest1.com and symantectest8383.com between February 10, 2016 and March 7, 2016.
The Baseline Requirements section 3.3.1 and the EV Guidelines sections 11.14.2 and 11.14.3 contemplate the reuse of existing validated information for subsequent certificate requests within specified data validity periods. Symantec’s systems are designed to retain information in accordance with these guidelines, and this applied to the data used in test certificates as well. These two unregistered domain names were present in internal testing accounts that relied on this “cached” data enabling the issuance of the 6 certificates.
Remediation:
  • All of the identified test certificates have expired or been revoked.
  • In April 2016 we performed a comprehensive system update to remove all “cached” domains used in each identified mis-issued test certificate to ensure those domains cannot be used for new certificates without first undergoing standard authentication and issuance procedures.
Issue 2: The Network Security Requirements section 2.e states that, “Each CA or Delegated Third Party SHALL require employees and contractors to observe the principle of “least privilege” when accessing, or when configuring access privileges on, Certificate Systems.”
Certain Symantec Quality Assurance (QA) personnel had systems access, including the ability to use certain legacy tools, which enabled them to request a limited number of test certificates that were issued without review by authentication personnel.
Remediation
  • All of the identified test certificates have expired or been revoked.
  • In September and October 2015, we applied a patch to our systems and removed the QA team’s access to a specific test tool to request and issue test certificates that did not follow our standard authentication processes.
  • We have implemented a new governance and oversight process for the use of the specific test tool (and any other privileged test tool) that requires senior leader justification from both the QA and Production Operations teams and approval from the heads of Engineering and Policy & Compliance.
  • We completed a review of issuance privileges to confirm that only authorized personnel have the ability to issue certificates; we are updating the rules regarding granting of privileges; and we have deployed an enhanced quarterly access review process to confirm the appropriateness of this access ongoing.
Additional information on the test tool and use of public and private keys
  • The test tool referred to above was for testing order placement through an API. The tool uses an input file that contains each of the data elements necessary to create a valid order: a CSR (used only for the public key), subject information, contact information, validity, etc.
  • Typically, the few QA personnel who had access to the tool re-used the input files without generating new private-public key pairs, changing only the fields as necessary for various test cases. This primary QA procedure left the public key from the CSR unchanged and as a result, the same public key(s) would be present in multiple test certificates.
  • In light of these practices (i.e., only editing the input files which did not contain or include private keys), we indicated in our initial incident report that private keys were not persisted with respect to the use of this internal test tool.
  • While the private keys generated during use of the test tool did not persist, the private keys were exposed to QA engineers when creating the CSRs for use with the test tool.
Issue 3: The Baseline Requirements sections 3.2.2.1, 3.2.2.4, and 3.2.2.5, and the EV Guidelines sections 11.1.1, 11.7.1.1, and 11.13.1 collectively set forth the requirements related to CA verification of identity, authorization by the domain name registrant, authentication for an IP address, and in the case of EV certificates, the requirements for applicant verification, domain verification, and independent review for discrepancies.
Authentication personnel did not consistently follow all verification steps when they received test certificate requests from their Symantec colleagues, or requested test certificates themselves.
Remediation
  • All of the identified test certificates have expired or been revoked.
  • Symantec has updated its internal policies and procedures to strongly reinforce that all test certificates must follow the same fulsome authentication procedures as commercial certificates.
  • We have trained our QA engineers and authentication personnel on these updated practices for test certificates.
KPMG has now completed our annual WebTrust audit, in which Symantec’s certificate management practices and internal controls were extensively reviewed, including a review to ensure that that Symantec’s audit logging mechanisms are reasonably protected from modification, deletion, and tampering. In addition, at the request of several root store operators, we have engaged KPMG to begin a separate Point in Time Readiness Assessment.
Notes:
  • References to the “Baseline Requirements” refer to the CA/Browser Forum Baseline Requirements Certificate Policy for the Issuance and Management of Publicly-Trusted Certificates version 1.3.4.
  • References to the “EV Guidelines” refer to the CA/Browser Forum Guidelines for the Issuance and Management of Extended Validation Certificates version 1.5.9.
  • References to the “Network Security Requirements” refer to the CA/Browser Forum Network and Certificate System Security Requirements version 1.0.
  • Twitter
  • Facebook
  • LinkedIn
  • Google+
  • YouTube