If you are currently wrestling with a VMware Cloud Foundation (VCF) 9.1 greenfield deployment or management plane overhaul, grab a coffee. You are going to need it.
Lately, I’ve been working on pushing the new VCF 9.1 Management Services layer onto a custom, isolated routed network segment using the standard API deployment payloads via PowerShell. On paper, it sounds beautiful: containerized microservices, a consolidated management plane, automated lifecycle hooks.
In practice? It is an absolute minefield of hidden UI pagination tricks, rigid validation loops, and a case-sensitivity trap that will make you want to throw your jumpbox out a window.
Here is the anatomy of a multi-stage VCF deployment failure—and exactly how to beat it.
Trap 1: The “Invisible” Binary Pagination Trick
You assemble your deployment payload, fire it off, get a beautiful StatusCode 202 (Accepted) back from the API, and monitor the SDDC Manager task framework. Then, boom. The task crashes out hard at the Generate VMware Components Image Location Path step:
Progress Messages: Failed to fetch install image path for component VSP with version 9.1.0.0.XXXXXXXXRemediation Message: Ensure install image path is present on the system for component type VSP...
You log into the SDDC Manager UI, navigate to Inventory -> Binary Management, and look at your downloads. Everything looks green. You click sync. You restart the task. It fails again, this time complaining about VCF_SALT.
The Real Culprit:
Look closely at the bottom right corner of the VCF 9.1 Software Depot screen. By default, the UI uses a rigid layout showing 1 – 10 of 30 binaries distributed across three pages. Core runtime templates like VMSP (VCF Services Platform) and VCF_SALT are sitting quietly on pages 2 and 3, completely un-downloaded. Because they are missing from the physical directory /nfs/vmware/vcf/nfs-mount/bundle/, the deployment engine strings return a null path.
The Fix:
Change the Binaries per page dropdown from 10 to 50. Select VCF services runtime, Salt master, and Salt RaaS, and pull them down.
If the UI paths remain unindexed after the download, SSH into the SDDC Manager as root and force a daemon cache rebuild:
systemctl restart lcm
systemctl restart domainmanager
Trap 2: Death by UPPERCASE Sizing Constants
Once the binaries are mapped, the orchestrator moves on to the bootstrapping phase. Minutes later, the task errors out again with an internal tracking signature:
Task Name: BootstrapVspClusterActionError Message: Failed to bootstrap VCF services runtimeCause: Unknown node size: SMALL
The Real Culprit:
The VCF 9.1 deployment API expects the appliance footprint configuration sizing string to be strictly lowercase ("small"). If your installation script defines it as an uppercase constant like $VCFManagementServicesSize = "SMALL", the Java/Kubernetes validation logic under the hood treats it as an invalid enum pattern and terminates the thread.
Worse yet, you can’t just click RESTART TASK in the UI to fix this—the uppercase string is locked directly inside the active memory row of the processing tables in the SDDC Manager backend.
Trap 3: The Validation Catch-22 (Order-of-Operations Lock)
Realizing the casing mistake, you modify your script variable to "small" and execute it again to launch a clean thread. The script spins up, triggers its deployment pre-checks, and hits a brick wall:
[Validation task executionStatus: COMPLETED, resultStatus: FAILED]
Validate that existing VCF Operations is not already integrated with the provided License Server
Instance of Identity Broker with host name 'id-broker-01.domain.local…' is already deployed.The Real Culprit:
This is a brutal lifecycle deadlock. During your very first failed run, before the engine crashed on the sizing error, it successfully completed the early workflow steps. It physically reached out to your live enterprise network, registered the OIDC Identity Provider (VCF_SSO) inside your vCenter Server, and bound the License Server parameters inside VCF Operations.
When you re-run your script fresh, the validation engine queries your live infrastructure, spots those active integrations, assumes a system collision, and halts to protect the network.
You can’t use a fresh script because validation blocks you, and you can’t use the old UI restart button because the uppercase "SMALL" error is burned into its cache.
The Clean Solution: Strip the Spec and Go
To break this deadlock, you don’t need to dive into dangerous database surgery or start deleting records off your vCenter SSO Lookup Service. You just need to modify your PowerShell payload specification to adapt to what is already alive on the network.
Since the validation pre-check complains that the Identity Broker and License Server configurations are already running live, comment them completely out of your script’s input payload.
By removing licenseServerSpec and vidbSpec from the API payload, the validation pre-checks stop scanning for those duplicate entities. Here is how your modified script setup section should look:
# 4. Security Passwords & Sizing Constants
$VCFManagementServicesPassword = "ApplianceClusterPasswordValue123!"
$VCFManagementServicesSize = "small" # Ensure the sizing string is strictly lowercase
$VCFManagementServicesInternalClusterCIDR = "172.27.0.0/16"
# 5. Execution Flags
$ValidateOnly = $false
$OutputJsonPayload = $falseFull code is in my Github Repo.
Save the script, keep $ValidateOnly = $false, and fire it down the pipe.
The validation engine will process the streamlined spec, bypass the duplicate asset errors, and open up a brand-new master deployment Task ID inside your SDDC Manager dashboard. The orchestrator will read the clean "small" parameter, parse your localized /27 network schema, and successfully deploy your containerized infrastructure fleet.
Stay angry, my friends. But stay automated.
