“Log disk exhaustion” warning on the vCenter

Is your VMware vCenter experiencing a “log disk exhaustion” warning? Is your vCenter’s storage space being consumed by an unusually large vmafdd.log file? You’re not alone. Many system administrators have encountered this issue, which can lead to performance degradation and potentially disrupt your virtualized environment. In this guide, we’ll walk you through the process of identifying and resolving this problem to keep your vCenter running smoothly.

Identifying the Problem

When you come across a “log disk exhaustion” warning in vCenter, it’s time to investigate the cause. One common culprit is the vmafdd.log file, which might unexpectedly grow in size, gobbling up valuable disk space. You can use the following command to check the disk utilization of your storage logs:

Example:

sudo du -ah /storage/log/ | sort -h -r | head -n 20

7.0G    /storage/log/vmware
7.0G    /storage/log/
1.4G    /storage/log/vmware/vmafd/vmafdd.log
1.4G    /storage/log/vmware/vmafd
768M    /storage/log/vmware/lookupsvc
762M    /storage/log/vmware/sso
757M    /storage/log/vmware/eam

If you find the vmafdd.log file to be excessively large, it’s time to dig deeper.

Analyzing the vmafdd.log

Open the vmafdd.log file and scan for messages like the following:

2021-02-04T08:41:15.517 [vmafdd][INFO] VecsSrvWriteCertOrCrlToDisk: cert/cl already exists as /etc/ssl/certs/d70df8d2.0, so will not write again.
2021-02-04T08:41:15.517 [vmafdd][INFO] VecsSrvWriteCertOrCrlToDisk: cert/cl already exists as /etc/ssl/certs/6349821d.0, so will not write again.
2021-02-04T08:41:15.517 [vmafdd][INFO] VecsSrvWriteCertOrCrlToDisk: cert/cl already exists as /etc/vmware-vpx/docRoot/certs/ d70df8d2.0, so will not write again.
2021-02-04T08:41:15.518 [vmafdd][INFO] VmAfdProcessCACerts: force flushing.

These entries indicate that certificates are being written to disk. Inefficient logging or incorrect log rotation settings may lead to the unexpected growth of the vmafdd.log file.

Investigating the Configuration Mismatch

The root cause of this issue often lies in a configuration mismatch between the expected log file path and the actual log file path. The logrotate.d configuration for the vmafd service may be expecting the log file in a different location. Compare the expected and current paths using the provided command:

cat /etc/logrotate.d/vmware-vmafd.lr

The expected path should be:

Expected path: /var/log/vmware/vmafdd/vmafdd.log
Current path: /var/log/vmware/vmafd/vmafdd.log

Fixing the Configuration

To resolve the problem, follow these steps:

Update the Registry:

Use the following command to modify the registry and align the log file path with the logrotate.d configuration:

# /opt/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" "LogFile" "/var/log/vmware/vmafdd/vmafdd.log"

Restart all services using the following command:

service-control --stop --all && service-control --start --all

Go to vmafd folder and remove old log:

cd vmafd
rm vmafdd.log

Ensuring Data Integrity

Before proceeding with the steps above, ensure that you have taken appropriate precautions to protect your vCenter environment. Create a fresh backup or offline snapshot of the vCenter Server Appliance, especially if it’s part of an Enhanced Linked Mode (ELM) replication group. Remember that all replication partners need to be restored together to maintain consistency in the VMDirectory LDAP database.

Conclusion

The “log disk exhaustion” warning in VMware vCenter can be a concerning issue, but armed with the knowledge of log file paths, configurations, and careful steps, you can address and resolve it effectively. By identifying the problem, investigating the log file, and rectifying configuration mismatches, you can regain control over your vCenter’s disk utilization and maintain a stable virtualized environment. Remember, proactive monitoring and regular maintenance are key to preventing such issues in the future.