VCSA storage/log at 95% – vmware-sps

Another problem I come across after storage/core at 100% during preparation for update to U3c was that storage space at storage/log was filled up to 95%.

The reason for this in my case was:

  • storage/log/volume is filling up in vCenter 7.0U2 due to growing sps-runtime.log.stderr
  • vmafdd.log is not being compressed which eventually leads to /storage/log/ filling up. This happens as the logrotate.d configuration for the vmafd service expects the logs to be located in the following path:
Expected:

/var/log/vmware/vmafdd/vmafdd.log

Current:

/var/log/vmware/vmafd/vmafdd.log

Resolution:

NOTE: The issue with vmware-sps has been fixed in 7.0U3c

  • There is a VMware script to remediate vmware-sps issue. Please read disclimer before running this script:
#!/bin/bash
#------------------------------
# Script to create/update IOFILTER provider
# certificate entries in the SMS store in VECS
# when the vpxd.certmgmt.mode is set to 'thumbprint',
# or check ESXi hosts' SSL cert and IOFilter provider
# cert and trust store contents when vpxd.certmgmt.mode
# is set to 'vmca' or 'custom'
#
# Author: Vincent Santa Maria [vsantamaria@vmware.com]
# Version: 1.3
#------------------------------
GREEN=$(tput setaf 2)
YELLOW=$(tput setaf 3)
CYAN=$(tput setaf 6)
NORMAL=$(tput sgr0)
NEED_TO_RESTART_SPS=0
ENTRIES_UPDATED=0
ENTRIES_CREATED=0
CERT_MGMT_MODE=$(/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "SELECT value FROM vpx_parameter WHERE name='vpxd.certmgmt.mode'" -t | grep -v '^$' | tr -d ' ')

echo $'\n'"Certificate management mode is set to '$CERT_MGMT_MODE'"
   
if [ "$CERT_MGMT_MODE" = "thumbprint" ]; then
   for host in $(/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "SELECT dns_name FROM vpx_host" -t | tr -d ' '); do
      echo $'\n'"Checking host $host..."
      CURRENT_SMS_THUMBPRINT=$(/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store SMS --alias "https://urldefense.com/v3/__https://$*7Bhost*7D:9080/version.xml__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRS7xHAVI$ " 2>/dev/null | openssl x509 -noout -fingerprint -sha1 2>/dev/null | awk -F'=' '{print $NF}')
   
      if [ ! -z "$CURRENT_SMS_THUMBPRINT" ]; then
         echo $'\t'"Host IOFILTER provider found in VECS, checking certificate..."
      
         HOST_THUMBPRINT=$(echo | openssl s_client -connect ${host}:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1 2>/dev/null | awk -F'=' '{print $NF}')
      
         if [[ "$CURRENT_SMS_THUMBPRINT" != "$HOST_THUMBPRINT" ]]; then
            echo $'\t'"Mismatch found, re-creating entry..."
            if /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store SMS --alias "https://urldefense.com/v3/__https://$*7Bhost*7D:9080/version.xml__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRS7xHAVI$ " -y > /dev/null; then
               if echo | openssl s_client -connect ${host}:443 2>/dev/null | openssl x509 > /tmp/${host}.crt; then
                  if /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store SMS --alias "https://urldefense.com/v3/__https://$*7Bhost*7D:9080/version.xml__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRS7xHAVI$ " --cert /tmp/${host}.crt > /dev/null; then
                     echo $'\t'"IOFILTER provider certificate updated!"
                     ((++ENTRIES_UPDATED))
                     NEED_TO_RESTART_SPS=1
                  else
                     echo $'\t'"Unable to re-create the IOFILTER provider certificate in VECS!"
                  fi
               else
                  echo $'\t'"Unable to obtain host's SSL certificate on port 443!"
               fi
            else
               echo $'\t'"Unable to delete the IOFILTER provider certificate from VECS!"
            fi
         else
            echo $'\t'"Certificate matches, no need to update." 
         fi
      else
         echo $'\t'"Host IOFILTER provider certificate not in VECS. Creating entry..."
         if echo | openssl s_client -connect ${host}:443 2>/dev/null | openssl x509 > /tmp/${host}.crt; then
            if /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store SMS --alias "https://urldefense.com/v3/__https://$*7Bhost*7D:9080/version.xml__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRS7xHAVI$ " --cert /tmp/${host}.crt > /dev/null; then
               echo $'\t'"IOFILTER provider certificate created!"
               ((++ENTRIES_CREATED))
               NEED_TO_RESTART_SPS=1
            else
               echo $'\t'"Unable to re-create the IOFILTER provider certificate in VECS!"
            fi
         else
            echo $'\t'"Unable to obtain host's SSL certificate on port 443!"
         fi
      fi
   done

   echo $'\n'"Entries updated: $ENTRIES_UPDATED"
   echo "Entries created: $ENTRIES_CREATED"
   if [ $NEED_TO_RESTART_SPS -gt 0 ]; then
     echo $'\n'"Restarting the SPS service..."

     service-control --restart vmware-sps
   else
      echo $'\n'"Done!"
   fi
else
   SMS_SUBJECT=$(/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store SMS --alias sms_self_signed | openssl x509 -noout -subject | sed 's/subject= //g')
   VC_TRUSTED_CA_SUBJECTS=$(for cert in $(ls /etc/vmware-vpx/docRoot/certs/*.0); do openssl x509 -noout -subject -in $cert | sed 's/subject= //g'; done)
   VC_ISSUER=$(echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -issuer | sed 's/issuer= //g')
   ESXI_TRUST_CHECK=0
   
   read -p $'\n'"Are all of the ESXi hosts in inventory configured with the same root password? [n]: " ESXI_PASSWORD_PROMPT
   
   if [ -z $ESXI_PASSWORD_PROMPT ]; then 
      ESXI_SAME_PASSWORD=n
   else
      ESXI_SAME_PASSWORD=$ESXI_PASSWORD_PROMPT   
   fi
   
   if [[ $ESXI_SAME_PASSWORD =~ ^[Yy] ]]; then
      ESXI_TRUST_CHECK=1
   fi
   
   if [[ $ESXI_TRUST_CHECK -eq 1 ]]; then
      read -s -p $'\n'"Please enter the root password for the ESXi hosts: " ESXI_ROOT_PASSWD
   fi
   
   for host in $(/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "SELECT dns_name FROM vpx_host" -t); do 
      echo $'\n'"Host: $host"
      
	   HOST_RHTTPPROXY_ISSUER=$(echo | openssl s_client -connect ${host}:443 2>/dev/null | openssl x509 -noout -issuer 2>/dev/null | sed 's/issuer= //g')
      HOST_IOFILTERVP_ISSUER=$(echo | openssl s_client -connect ${host}:9080 2>/dev/null | openssl x509 -noout -issuer 2>/dev/null | sed 's/issuer= //g')
      
      echo -n $'\t'"Reverse Proxy cert (port 443): "
	   if [ ! -z "$HOST_RHTTPPROXY_ISSUER" ]; then
         if echo "$VC_TRUSTED_CA_SUBJECTS" | grep "$HOST_RHTTPPROXY_ISSUER" > /dev/null 2>/dev/null; then
            echo "${GREEN}trusted by vCenter${NORMAL}"
         else
            echo "${YELLOW}untrusted by vCenter${NORMAL}"
         fi
	   else
	      echo "${YELLOW}unknown${NORMAL}"
	   fi
      
      echo -n $'\t'"IOFilter VASA provider cert (port 9080): "
      if [ ! -z "$HOST_IOFILTERVP_ISSUER" ]; then
	      if echo "$VC_TRUSTED_CA_SUBJECTS" | grep "$HOST_IOFILTERVP_ISSUER" > /dev/null 2>&1; then
            echo "${GREEN}trusted by vCenter${NORMAL}"
         else
            echo "${YELLOW}untrusted by vCenter${NORMAL}"
         fi
	   else
	      echo "${YELLOW}unknown${NORMAL}"
	   fi
      
      if [[ $ESXI_TRUST_CHECK -eq 1 && ! -z $ESXI_ROOT_PASSWD ]]; then
         ESXI_CASTORE_SUBJECTS=$(curl -k -X GET -u "root:$ESXI_ROOT_PASSWD" https://urldefense.com/v3/__https://$*7Bhost*7D/host/castore__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRVZKAD8c$  2>/dev/null | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout | grep '^subject' | sed 's/subject=//g')
         
         if [ -z "$ESXI_CASTORE_SUBJECTS" ]; then
            echo $'\t'"${YELLOW}Unable to obtain CA certificates on ${host}${NORMAL}"
         else
         
            echo -n $'\t'"vCenter Machine SSL cert: "
            if echo "$ESXI_CASTORE_SUBJECTS" | grep "$VC_ISSUER" > /dev/null 2>&1; then
               echo "${GREEN}trusted by host${NORMAL}"
            else
               echo "${YELLOW}untrusted by host${NORMAL}"
            fi
         
            echo -n $'\t'"SPS service connection: "
            if echo "$ESXI_CASTORE_SUBJECTS" | grep "$SMS_SUBJECT" > /dev/null 2>&1; then
               echo "${GREEN}trusted by host${NORMAL}"
            else
               echo "${YELLOW}possibly untrusted by host${NORMAL} (check /etc/vmware/ssl/iofiltervp_castore.pem)"
            fi
         fi
      fi
   done
   
   if [[ $ESXI_TRUST_CHECK -eq 0 ]]; then
      echo $'\n\n'"Please check the ${CYAN}/etc/vmware/ssl/castore.pem${NORMAL} to ensure the signing CAs for the"
      echo "vCenter Machine SSL are present, and the SMS certificate is present in either "
      echo "${CYAN}/etc/vmware/ssl/castore.pem${NORMAL}, or ${CYAN}/etc/vmware/ssl/iofiltervp_castore.pem${NORMAL}"
   fi
fi
  • For vmafd,  you need to update the registry to match the log rotate configuration path and then restarted all services:
/opt/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" "LogFile" "/var/log/vmware/vmafdd/vmafdd.log"

Please like and share to spread the knowledge in the community.

If you want to chat with me please use Twitter: @AngrySysOps

Visit my FB page: https://www.facebook.com/AngrySysOps

Read my blog: https://angrysysops.com

Subscribe to my channel : https://www.youtube.com/channel/UCRTcKGl0neismSRpDMK_M4A

Please leave the comment