Another problem I come across after storage/core at 100% during preparation for update to U3c was that storage space at storage/log was filled up to 95%.
The reason for this in my case was:
- storage/log/volume is filling up in vCenter 7.0U2 due to growing sps-runtime.log.stderr
- vmafdd.log is not being compressed which eventually leads to /storage/log/ filling up. This happens as the logrotate.d configuration for the vmafd service expects the logs to be located in the following path:
Expected:
/var/log/vmware/vmafdd/vmafdd.log
Current:
/var/log/vmware/vmafd/vmafdd.logResolution:
NOTE: The issue with vmware-sps has been fixed in 7.0U3c
- There is a VMware script to remediate vmware-sps issue. Please read disclimer before running this script:
#!/bin/bash
#------------------------------
# Script to create/update IOFILTER provider
# certificate entries in the SMS store in VECS
# when the vpxd.certmgmt.mode is set to 'thumbprint',
# or check ESXi hosts' SSL cert and IOFilter provider
# cert and trust store contents when vpxd.certmgmt.mode
# is set to 'vmca' or 'custom'
#
# Author: Vincent Santa Maria [vsantamaria@vmware.com]
# Version: 1.3
#------------------------------
GREEN=$(tput setaf 2)
YELLOW=$(tput setaf 3)
CYAN=$(tput setaf 6)
NORMAL=$(tput sgr0)
NEED_TO_RESTART_SPS=0
ENTRIES_UPDATED=0
ENTRIES_CREATED=0
CERT_MGMT_MODE=$(/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "SELECT value FROM vpx_parameter WHERE name='vpxd.certmgmt.mode'" -t | grep -v '^$' | tr -d ' ')
echo $'\n'"Certificate management mode is set to '$CERT_MGMT_MODE'"
if [ "$CERT_MGMT_MODE" = "thumbprint" ]; then
for host in $(/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "SELECT dns_name FROM vpx_host" -t | tr -d ' '); do
echo $'\n'"Checking host $host..."
CURRENT_SMS_THUMBPRINT=$(/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store SMS --alias "https://urldefense.com/v3/__https://$*7Bhost*7D:9080/version.xml__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRS7xHAVI$ " 2>/dev/null | openssl x509 -noout -fingerprint -sha1 2>/dev/null | awk -F'=' '{print $NF}')
if [ ! -z "$CURRENT_SMS_THUMBPRINT" ]; then
echo $'\t'"Host IOFILTER provider found in VECS, checking certificate..."
HOST_THUMBPRINT=$(echo | openssl s_client -connect ${host}:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1 2>/dev/null | awk -F'=' '{print $NF}')
if [[ "$CURRENT_SMS_THUMBPRINT" != "$HOST_THUMBPRINT" ]]; then
echo $'\t'"Mismatch found, re-creating entry..."
if /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store SMS --alias "https://urldefense.com/v3/__https://$*7Bhost*7D:9080/version.xml__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRS7xHAVI$ " -y > /dev/null; then
if echo | openssl s_client -connect ${host}:443 2>/dev/null | openssl x509 > /tmp/${host}.crt; then
if /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store SMS --alias "https://urldefense.com/v3/__https://$*7Bhost*7D:9080/version.xml__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRS7xHAVI$ " --cert /tmp/${host}.crt > /dev/null; then
echo $'\t'"IOFILTER provider certificate updated!"
((++ENTRIES_UPDATED))
NEED_TO_RESTART_SPS=1
else
echo $'\t'"Unable to re-create the IOFILTER provider certificate in VECS!"
fi
else
echo $'\t'"Unable to obtain host's SSL certificate on port 443!"
fi
else
echo $'\t'"Unable to delete the IOFILTER provider certificate from VECS!"
fi
else
echo $'\t'"Certificate matches, no need to update."
fi
else
echo $'\t'"Host IOFILTER provider certificate not in VECS. Creating entry..."
if echo | openssl s_client -connect ${host}:443 2>/dev/null | openssl x509 > /tmp/${host}.crt; then
if /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store SMS --alias "https://urldefense.com/v3/__https://$*7Bhost*7D:9080/version.xml__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRS7xHAVI$ " --cert /tmp/${host}.crt > /dev/null; then
echo $'\t'"IOFILTER provider certificate created!"
((++ENTRIES_CREATED))
NEED_TO_RESTART_SPS=1
else
echo $'\t'"Unable to re-create the IOFILTER provider certificate in VECS!"
fi
else
echo $'\t'"Unable to obtain host's SSL certificate on port 443!"
fi
fi
done
echo $'\n'"Entries updated: $ENTRIES_UPDATED"
echo "Entries created: $ENTRIES_CREATED"
if [ $NEED_TO_RESTART_SPS -gt 0 ]; then
echo $'\n'"Restarting the SPS service..."
service-control --restart vmware-sps
else
echo $'\n'"Done!"
fi
else
SMS_SUBJECT=$(/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store SMS --alias sms_self_signed | openssl x509 -noout -subject | sed 's/subject= //g')
VC_TRUSTED_CA_SUBJECTS=$(for cert in $(ls /etc/vmware-vpx/docRoot/certs/*.0); do openssl x509 -noout -subject -in $cert | sed 's/subject= //g'; done)
VC_ISSUER=$(echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -issuer | sed 's/issuer= //g')
ESXI_TRUST_CHECK=0
read -p $'\n'"Are all of the ESXi hosts in inventory configured with the same root password? [n]: " ESXI_PASSWORD_PROMPT
if [ -z $ESXI_PASSWORD_PROMPT ]; then
ESXI_SAME_PASSWORD=n
else
ESXI_SAME_PASSWORD=$ESXI_PASSWORD_PROMPT
fi
if [[ $ESXI_SAME_PASSWORD =~ ^[Yy] ]]; then
ESXI_TRUST_CHECK=1
fi
if [[ $ESXI_TRUST_CHECK -eq 1 ]]; then
read -s -p $'\n'"Please enter the root password for the ESXi hosts: " ESXI_ROOT_PASSWD
fi
for host in $(/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "SELECT dns_name FROM vpx_host" -t); do
echo $'\n'"Host: $host"
HOST_RHTTPPROXY_ISSUER=$(echo | openssl s_client -connect ${host}:443 2>/dev/null | openssl x509 -noout -issuer 2>/dev/null | sed 's/issuer= //g')
HOST_IOFILTERVP_ISSUER=$(echo | openssl s_client -connect ${host}:9080 2>/dev/null | openssl x509 -noout -issuer 2>/dev/null | sed 's/issuer= //g')
echo -n $'\t'"Reverse Proxy cert (port 443): "
if [ ! -z "$HOST_RHTTPPROXY_ISSUER" ]; then
if echo "$VC_TRUSTED_CA_SUBJECTS" | grep "$HOST_RHTTPPROXY_ISSUER" > /dev/null 2>/dev/null; then
echo "${GREEN}trusted by vCenter${NORMAL}"
else
echo "${YELLOW}untrusted by vCenter${NORMAL}"
fi
else
echo "${YELLOW}unknown${NORMAL}"
fi
echo -n $'\t'"IOFilter VASA provider cert (port 9080): "
if [ ! -z "$HOST_IOFILTERVP_ISSUER" ]; then
if echo "$VC_TRUSTED_CA_SUBJECTS" | grep "$HOST_IOFILTERVP_ISSUER" > /dev/null 2>&1; then
echo "${GREEN}trusted by vCenter${NORMAL}"
else
echo "${YELLOW}untrusted by vCenter${NORMAL}"
fi
else
echo "${YELLOW}unknown${NORMAL}"
fi
if [[ $ESXI_TRUST_CHECK -eq 1 && ! -z $ESXI_ROOT_PASSWD ]]; then
ESXI_CASTORE_SUBJECTS=$(curl -k -X GET -u "root:$ESXI_ROOT_PASSWD" https://urldefense.com/v3/__https://$*7Bhost*7D/host/castore__;JSU!!DCbAVzZNrAf4!WNflJ-luEA2WNdhLvir2iTsWyy-5GzkzseqlMXgpY2ZjHfytiypU2qxj7yuRVZKAD8c$ 2>/dev/null | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout | grep '^subject' | sed 's/subject=//g')
if [ -z "$ESXI_CASTORE_SUBJECTS" ]; then
echo $'\t'"${YELLOW}Unable to obtain CA certificates on ${host}${NORMAL}"
else
echo -n $'\t'"vCenter Machine SSL cert: "
if echo "$ESXI_CASTORE_SUBJECTS" | grep "$VC_ISSUER" > /dev/null 2>&1; then
echo "${GREEN}trusted by host${NORMAL}"
else
echo "${YELLOW}untrusted by host${NORMAL}"
fi
echo -n $'\t'"SPS service connection: "
if echo "$ESXI_CASTORE_SUBJECTS" | grep "$SMS_SUBJECT" > /dev/null 2>&1; then
echo "${GREEN}trusted by host${NORMAL}"
else
echo "${YELLOW}possibly untrusted by host${NORMAL} (check /etc/vmware/ssl/iofiltervp_castore.pem)"
fi
fi
fi
done
if [[ $ESXI_TRUST_CHECK -eq 0 ]]; then
echo $'\n\n'"Please check the ${CYAN}/etc/vmware/ssl/castore.pem${NORMAL} to ensure the signing CAs for the"
echo "vCenter Machine SSL are present, and the SMS certificate is present in either "
echo "${CYAN}/etc/vmware/ssl/castore.pem${NORMAL}, or ${CYAN}/etc/vmware/ssl/iofiltervp_castore.pem${NORMAL}"
fi
fi- For vmafd, you need to update the registry to match the log rotate configuration path and then restarted all services:
/opt/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" "LogFile" "/var/log/vmware/vmafdd/vmafdd.log"Please like and share to spread the knowledge in the community.
If you want to chat with me please use Twitter: @AngrySysOps
Visit my FB page: https://www.facebook.com/AngrySysOps
Read my blog: https://angrysysops.com
Subscribe to my channel : https://www.youtube.com/channel/UCRTcKGl0neismSRpDMK_M4A
Please leave the comment
