Automate Your Virtual Infrastructure Management: A Handy PowerCLI Script for On-Call Situations

Imagine a scenario where a power outage or building power-up event causes your ESXi hosts to remain in maintenance mode. Instead of waking up the on-call person in the middle of the night, you can run an automated PowerCLI script that restores host connectivity and powers on any off VMs. This blog post walks you through a useful PowerShell script step by step, explaining how each part works and why it’s invaluable for on-call support. You can also download the fully documented script from this GitHub repository.

Why This Script Is a Game-Changer

When you’re responsible for your organization’s virtual infrastructure, unexpected events—like a power outage—can throw a wrench in your operations. Typically, after such incidents, physical hosts may remain in maintenance mode to prevent further issues during startup. In these cases, manual intervention can be a burden, especially during off-hours.

This script automates three critical tasks:

Host Powered-On Check:
Before processing further, the script checks if each ESXi host’s PowerState is "PoweredOff". If a host is off, the script waits (pausing for 60 seconds) and re-queries the host until the host is powered on.
Exiting Maintenance Mode on ESXi Hosts: It iterates through all hosts in a specified cluster and, for any host found in maintenance mode, issues a command to bring it back online.
Powering On Virtual Machines (VMs): Once the host is confirmed to be out of maintenance mode, the script retrieves all associated VMs and powers on those that are turned off.

By automating these steps, the script ensures that your virtual environment recovers automatically—minimizing downtime and reducing the need for manual intervention.

What’s Masked by Default?

To protect sensitive information in your environment, the script comes with masked default values for the vCenter and cluster name. For example, you’ll find default placeholders like:

# Default masked value for the cluster name.
$clusterName = 'YOUR-CLUSTER-NAME'

You should update these values with your actual cluster and vCenter details once you’ve confirmed the script is working as expected in your environment. Using masked default values helps prevent accidental exposure of sensitive infrastructure information, especially in shared or publicly accessible repositories.

Step-by-Step Explanation of the Script

Below is the script with detailed commentary explaining how each section functions:

1. Defining the Cluster

$clusterName = 'YOUR-CLUSTER-NAME'

# Retrieve the cluster by iterating through datacenters and clusters
$cluster = Get-Datacenter | Get-Cluster | Where-Object { $_.Name -eq $clusterName }

What It Does:

Purpose: The script starts by setting a variable, $clusterName, which specifies the target cluster.
How It Works: It searches through all datacenters and clusters to find the one that matches the designated name.
Why It’s Useful: This isolates the target cluster so that subsequent commands affect only the correct set of ESXi hosts.

2. Verifying the Cluster Exists

if ($null -eq $cluster) {
    Write-Host "Cluster not found!"
    exit
}

What It Does:

Purpose: It ensures that the target cluster was successfully retrieved before proceeding.
How It Works: If no cluster is found (i.e., $cluster is $null), the script outputs an error message and terminates.
Why It’s Useful: This prevents any risky operations on an unintended environment.

3. Retrieving All ESXi Hosts in the Cluster

# Get all ESXi hosts in the specified cluster
$esxiHosts = Get-VMHost -Location $cluster

What It Does:

Purpose: After identifying the cluster, the script retrieves all ESXi hosts within it.
How It Works: The Get-VMHost command, when provided with the cluster location, returns a collection of host objects.
Why It’s Useful: This list of hosts will be the target for the maintenance mode exit operations.

4. Iterating Through Each ESXi Host

foreach ($esxi in $esxiHosts) {
    Write-Host "Processing host: $($esxi.Name)"

What It Does:

Purpose: The script processes each host individually.
How It Works: The foreach loop cycles through the host objects, allowing the script to work on one host at a time.
Why It’s Useful: Handling one host at a time simplifies troubleshooting and ensures each host is processed correctly.

5. Exiting Maintenance Mode with a While Loop

 while ($esxi.ConnectionState -eq "Maintenance") {
        Write-Host "Exiting maintenance mode for host: $($esxi.Name)"
        Set-VMHost -VMHost $esxi -State "Connected" -Confirm:$false
        Start-Sleep -Seconds 60
        # Refresh the host object to get the updated state
        $esxi = Get-VMHost -Name $esxi.Name
    }

What It Does:

Purpose: This section ensures the host exits maintenance mode.
How It Works:
- The while loop continually checks if the host’s ConnectionState is "Maintenance".
- Within the loop, it calls Set-VMHost to change the state to "Connected" and pauses for 60 seconds to allow the change to take effect.
- It then refreshes the host object with Get-VMHost so that the latest state is used in the loop condition.
Why It’s Useful: The loop prevents the script from moving on until the host is fully connected. Refreshing the host object is essential since the in-memory object is static and does not reflect real-time state changes automatically.

6. Powering On the Virtual Machines

   # Check and power on VMs once the host is connected
    $vms = Get-VM -Location $esxi
    foreach ($vm in $vms) {
        if ($vm.PowerState -eq "PoweredOff") {
            Write-Host "Powering on VM: $($vm.Name) on host: $($esxi.Name)"
            Start-VM -VM $vm
        }
    }
}

What It Does:

Purpose: After the host is up and running, the script focuses on ensuring all virtual machines on that host are operational.
How It Works:
- It retrieves all VMs associated with the host using the Get-VM command.
- It checks each VM’s PowerState to see if it is "PoweredOff".
- If a VM is off, it starts the VM using Start-VM.
Why It’s Useful: This automation minimizes downtime by ensuring that all VMs become operational as soon as their host is ready.

How This Script Can Save Your Night

Running this script can make a huge difference in managing your virtual environment during unexpected incidents:

On-Call Relief: Instead of waking up an on-call engineer during a late-night outage, a member of your team can simply run this script.
Rapid Recovery: Automating the process reduces the time taken to restore services, keeping downtime to a minimum.
Reduced Human Error: Automation minimizes the risk of manual mistakes, ensuring consistent recovery actions every time.

Download the Script

You can download the complete PowerCLI script from this GitHub repository. The script is pre-configured with masked default values for sensitive parameters like the vCenter and cluster name. Remember to update these placeholders (e.g., change 'YOUR-CLUSTER-NAME') with your actual details after testing in a safe environment.

Final Thoughts

Managing a virtualized infrastructure can be challenging, especially during unexpected power incidents. This PowerCLI script offers a streamlined, automated solution that not only brings ESXi hosts out of maintenance mode but also ensures that any associated virtual machines are promptly powered on.

For on-call teams, this means a more reliable and less stressful recovery process when emergencies hit. By following the step-by-step explanation above and customizing the script to your environment, you can enjoy smoother operations and quieter nights.

Feel free to leave your comments, questions, or suggestions for improving the script below. Your feedback is always appreciated!

Happy scripting!

🔥Subscribe to the channel: youtube.be/@AngryAdmin 🔥

🚨Dive into my blog: angrysysops.com

🚨Snapshots 101: a.co/d/fJVHo5v

🌐Connect with us: