vCenter – HTTP status 500 – internal server error

Today, I encountered a situation wherein one of my vCenter instances presented an HTTP status 500 error. Should you ever encounter this error, two potential underlying causes responsible for its occurrence:

  1. Certificate expiration
  2. /storage/log is 95% full.

As I quickly check my SSL certificate and it shows valid until February 2024, I logged to VAMI and saw this error:

I went through this KB to resolve the issue. Nevertheless, even after performing space cleanup, the Status 500 error persisted.

In this scenario, my primary suspicion was directed towards the STS certificate, given its prominence in such issues. To investigate further, I initiated an SSH session to the vCenter Server and executed the vCenter script, which I have made accessible on my GitHub account.

To my surprise, the STS certificate proved to be valid. However, upon closer examination, it became evident that the Solution User Certificates were indeed invalid and responsible for the encountered issue.

Solution User Certificates

A solution user encapsulates one or more vCenter Server services. Each solution user must be authenticated to vCenter Single Sign-On. Solution users use certificates to authenticate to vCenter Single Sign-On through SAML token exchange.

A solution user presents the certificate to vCenter Single Sign-On when it first has to authenticate, after a reboot, and after a timeout has elapsed. The timeout (Holder-of-Key Timeout) can be set from the vSphere Client and defaults to 2592000 seconds (30 days).

For example, the vpxd solution user presents its certificate to vCenter Single Sign-On when it connects to vCenter Single Sign-On. The vpxd solution user receives a SAML token from vCenter Single Sign-On and can then use that token to authenticate to other solution users and services.

The following solution user certificate stores are included in VECS:

  • machine: Used by the license server and the logging service.
  • vpxd: vCenter service daemon (vpxd) store. vpxd uses the solution user certificate that is stored in this store to authenticate to vCenter Single Sign-On.
  • vpxd-extension: vCenter extensions store. Includes the Auto Deploy service, inventory service, and other services that are not part of other solution users.
  • vsphere-webclient: vSphere Client store. Also includes some additional services such as the performance chart service.
  • wcp: VMware vSphere® with VMware Tanzu™ store.

Note: The machine solution user certificate has nothing to do with the machine SSL certificate. The machine solution user certificate is used for the SAML token exchange. The machine SSL certificate is used for secure SSL connections for a machine.

Let’s fix it with vCert:

  • Option 3
  • Option 2
  • Option 1

The certificates have been renewed:

Now we need to restart vCenter Server:

Once the vCenter services are fully operational and running without any issues, you will be able to log in successfully.

🔥Subscribe to the channel: https://bit.ly/3vY16CT🔥

🚨Read my blog: https://angrysysops.com/

👊Twitter: https://twitter.com/AngrySysOps
👊Facebook: https://www.facebook.com/AngrySysOps
👊My Podcast: https://bit.ly/39fFnxm
👊Mastodon: https://techhub.social/@AngryAdmin

🔥vExpert info: https://bit.ly/3vXGPOa

🛒 VMware EMEA store: https://imp.i263671.net/c/3505578/814646/11461

🛒 VMware US store: https://imp.i263671.net/c/3505578/814642/11461

🛒 VMware APAC store: https://imp.i263671.net/c/3505578/814645/11461

 
Please leave the comment