How to Remove CA Certificates from the TRUSTED_ROOTS store
Introduction
Upgrading to VMware vCenter Server 8.0 update 2a from version 7.x can sometimes present challenges, notably when encountering errors related to certificates signed with the SHA-1 algorithm. Given the security vulnerabilities associated with SHA-1, VMware has taken steps to ensure a more secure environment, which may require updating or replacing certificates signed with SHA-1.
Background
The issue arises during the pre-check stage of the upgrade process when the system identifies a certificate signed by the weak SHA-1 algorithm. This is a critical point as certificates are crucial for ensuring secure communications within the VMware environment. The VMware Directory Service (VMDIR) plays a significant role here by republishing certificates to the VECS store to maintain the integrity of the TRUSTED_ROOTS Certificate store. However, removing the wrong certificate could lead to severe consequences, potentially rendering the environment inoperable.
Impact and Risks
Warning: Removing the incorrect certificate from the VECS store can cause irreversible damage to your environment. It’s crucial to proceed with extreme caution, ensuring that the specific certificate targeted for removal is the correct one. Additionally, before unpublishing any certificates, make sure that the root certificate and all associated certificates have been renewed or replaced to prevent any certificate-related alarms or issues.
Mandatory Precautions
Before attempting to resolve the certificate issue, take the following steps to safeguard your environment:
- Shut Down Platform Services Controllers: Ensure all Platform Services Controllers (PSCs) in the federated environment are powered off. This prevents partial replication during snapshot operations.
- Take Snapshots: Create snapshots of all PSCs and vCenter Systems while they are powered off to facilitate a potential rollback without affecting replication data consistency.
Resolution Steps
To safely address the certificate issue, follow these detailed steps:
Identifying and Removing the Expired Certificate
List Certificates in VECS Store:
- On Appliance:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less - On Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli.exe entry list --store TRUSTED_ROOTS --text | more
Identifying and Managing Expired Certificates
When upgrading vCenter Server to version 8.0 update 2a, encountering a SHA-1 certificate error requires careful management of your certificates. The process begins by identifying the certificate you need to remove. For example, you might find a certificate with the following details:
- Alias:
2b724e6dd26e38b369a020f279f3bfc3369e2e7f - X509v3 Subject Key Identifier:
ED:CF:46:E5:CA:A6:8A:75:04:C0:D4:7B:2B:45:2C:08:53:10:F9:18
Given that multiple certificates might be outdated or not in use, removing any that are expired is crucial to prevent certificate-related warnings.
Retrieving and Managing Trusted Certificates
To list certificates trusted by the VMware Directory Service (VMDIR), execute the following commands depending on your setup:
- On Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>dir-cli trustedcert list - On Appliance:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list
This action produces a list of certificates, such as:
Number of certificates: 3
- CN(id): EDCF46E5CAA68A7504C0D47B2B452C085310F918 …
- CN(id): 72B1C4C56A1A8A66B8C57182D551A29B78531ED0 …
- CN(id): 7AF0962806F5997107BF9A213E86DED4F853FF70 …
Locate the certificate that matches the Key Identifier you identified earlier.
Using the ID run the following command, adjusting appropriately for your environment:
- On Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>dir-cli trustedcert get --id EDCF46E5CAA68A7504C0D47B2B452C085310F918 --login administrator@vsphere.local --password <PASSWORD> --outcert C:\temp\oldcert.cer - On Appliance:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id EDCF46E5CAA68A7504C0D47B2B452C085310F918 --login administrator@vsphere.local --outcert /tmp/oldcert.cer
Unpublishing and Deleting the Certificate
To un-publish the identified certificate from VMDIR, execute:
- On Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>dir-cli trustedcert unpublish --cert C:\temp\oldcert.cer - On Appliance:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /tmp/oldcert.cer
Then, to remove the certificate from the VECS store using the noted alias, run:
- On Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli entry delete --store TRUSTED_ROOTS --alias <your_alias> - On Appliance:
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <your_alias>
Refreshing VECS and Confirming Changes
To ensure all changes are propagated throughout your environment, force a refresh of VECS:
- On Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli force-refresh - On Appliance:
/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
Verify that the certificate has been successfully removed by listing the certificates in the TRUSTED_ROOTS store again.
Final Steps
After completing these steps, restart all services on the PSCs and vCenter Servers. Ensure that all services start correctly and that the environment is manageable.
🔥Subscribe to the channel: https://bit.ly/3vY16CT🔥
🚨Read my blog: https://angrysysops.com/
👊Twitter: https://twitter.com/AngrySysOps
👊Facebook: https://www.facebook.com/AngrySysOps
👊My Podcast: https://bit.ly/39fFnxm
👊Mastodon: https://techhub.social/@AngryAdmin
