Enterprise Admins.org

In vSphere 7.0 functionality was introduced to be able to manage a cluster of ESXi hosts with a single image. In addition to being a bit easier to configure than the prior Update Manager Baselines, this feature also integrates with Hardware Support Managers to be able to update host firmware.

PowerCLI 12.1 introduced the ability to remediate against these single images, using syntax like:

Get-Cluster img-test | Set-Cluster -Remediate -AcceptEULA

This and other vLCM cmdlets are discussed in this blog post: https://blogs.vmware.com/PowerCLI/2020/10/new-release-powercli-12-1-vlcm-enhancements.html.

This is a very simple command and works very well to programmatically interact with the clusters if you have many to remediate. However, sometimes we need to remediate hosts in a more controlled order, or as part of a larger workflow. For example, perhaps we need to notify an operations management product or team before a host goes into maintenance to prevent after hour pages or service desk issues being logged. This can be done, but requires a few extra cmdlets. The code block below provides sample commands to remediate a specific host in a cluster.

$clusterid = (Get-Cluster 'Img-Test').ExtensionData.MoRef.Value
$vmhostid = (Get-VMHost 'h178-vesx-04.lab.enterpriseadmins.org').ExtensionData.MoRef.Value

# Initialize hosts to updates, hostids can be specified comma separated.
$SettingsClustersSoftwareApplySpec = Initialize-SettingsClustersSoftwareApplySpec -Hosts $vmhostid -AcceptEula $true 

# Apply the specification object to the cluster
$taskId = Invoke-ApplyClusterSoftwareAsync -Cluster $clusterid -SettingsClustersSoftwareApplySpec $SettingsClustersSoftwareApplySpec

# The apply task runs async so we need to watch tasks for it to complete.
$task=Get-Task |?{$_.Id -eq "CisTask-$taskId"} 

# Loop until the task finishes
While ($task.State -eq "Running") {
    "sleeping..."
    Start-Sleep -Seconds 60 
    $task=Get-Task |?{$_.Id -eq "CisTask-$taskId"} 
}

In the example above we get the ID value of the cluster and host, these are the values like domain-c3146330 and host-3146220 and are the values required by the cmdlets we are using.

We then create a spec to apply. The parameters are described here: https://developer.vmware.com/apis/vsphere-automation/latest/esx/api/esx/settings/clusters/cluster/softwareactionapplyvmw-tasktrue/post/. I was originally confused by the -commit parameter until I found this documentation. I had assumed the property was similar to a git commit message and was passing in random strings of text. The property is optional, so in the above example I do not pass in the argument at all. However, if you’d like to find the current/expected value, you can get it from the cluster object like this: (Invoke-GetClusterSoftwareCompliance -Cluster 'domain-c3146214').commit

Next we pass our clusterid and above spec to the ‘Invoke’ function. The command returns a task ID, so we are capturing the output to a variable so we can use it later. The Invoke command runs asynchronously, so we will use this ID to check on the status until the task completes.

At the end we are checking for the task to complete. The Get-Task cmdlet has an -ID parameter, but in my testing I would get an error of The identifier CisTask-525d3596-4a7a-60ab-df10-ab97999b8511:com.vmware.esx.settings.clusters.software resulted in no objects. when I tried to pass the ID to Get-Task. Using where-object instead worked reliably, so I used it instead.

Hopefully this helps if you have the need to remediate a single host in a cluster using a vLCM Image.

For several years I’ve been using a couple of very small TinyCore Linux virtual machines for testing in my lab. These run very well in nested infrastructure and have a package to support open-vm-tools so you can interact with them like normal virtual machines (for example, cmdlets like Shutdown-VMGuest will interact with them). I was recently updating templates to TinyCore 14.0, when I realized that version 15.0 had just released (https://forum.tinycorelinux.net/index.php/topic,26861.0.html). I wanted to share the steps to create these new templates in this post.

I have a pair of these virtual machines, one has a GUI and the other is command line only. I primarily use the CLI version because it uses even less resources, but keep the GUI one around in case I need a test web browser available.

The Virtual Machine

When creating the virtual machine, I used the following options:

• Compatible with: ESXi 6.7 U2 and later (vmx-15)
• Operating System: Linux / Other 4.x or later Linux (32-bit)
• 1 vCPU
• 1 GB RAM
• 1 GB disk (thin provisioned)
• Expand Video card > Total video memory = 8MB (when using GUI, for CLI only I left it at the default 4MB)
• VM Options tab > Boot Options > Firmware = BIOS

The Install

• Power on VM
• Open Remote Console (the one that launches VMRC or VMware Workstation, not the web console)
• Attach to a local CorePlus ISO image (specifically I used this ISO: http://tinycorelinux.net/15.x/x86/release/CorePlus-current.iso)
• CTRL+ALT+INS to reboot
• Select Boot Core with X/GUI (TinyCore) + Installation Extension
• Click the installation button on the task bar. Select Frugal > Whole Disk > sda > install boot loader > ext4
• Select either ‘core and x/gui desktop’ or ‘core only (text based interface)’ depending on which is appropriate.
• Proceed
• When the display says installation successful, Exit > Shutdown.
• Power On VM (this will ensure that the CD is no longer connected and boot into the install)
• At this point, we have a usable VM, but no VMware Tools.

Customization

I’ve done these customization steps a number of times, but this time I spent a few extra minutes to partially automate the process.

If booting into a GUI, I move the use the control panel > tcWbarConf and set the position to Left Vertical or Top Left. Having it at the bottom is sort of awkward, as sometimes when the screen resizes, this is left in the middle of the screen. Once positioned, I exit and select Exit to Prompt.

Since the VM doesn’t have/need SSH for my purposes, and copy/paste isn’t available, I placed the script and some associated files on a webserver. Once the system was online, I used wget to download the script, renamed the file, made it executable, and then ran the script. For a second version, I adjusted the script to expect a string coming in as a parameter, then used that value to set the hostname. I ended up with three files hosted on an internal web server:

rootca-example-com.crt the root certificate from my internal CA.

policies.json contains the entries needed for firefox to disable automatic updates and to trust the above root certificate. Its contents are below:

{
  "Policies": {
    "Certificates": {
      "ImportEnterpriseRoots": true, "Install": ["/usr/local/share/ca-certificates/rootca-example-com.cert"]
    },
  "DisableAppUpdate": true
  }
}

buildscript2.txt is the customization script. It’s stored as text as my webserver blocks .sh files by default and getting the text and renaming the file client side was an easy workaround. This script is actually responsible for downloading the two files above and putting them in the right filesystem locations.

if [ -z $1 ]; then
  echo "Please provide an argument which is used for hostname and other logic."
  exit 1
fi

# The following code will run for either case, gui or cli
sudo sed -i "s/sethostname box/sethostname $1/g" /opt/bootsync.sh

tce-load -wi ca-certificates curl pcre

sudo mkdir /usr/local/share/ca-certificates
sudo wget http://www.example.com/build/rootca-example-com.crt -P /usr/local/share/ca-certificates
sudo update-ca-certificates
echo "usr/local/share/ca-certificates" >> /opt/.filetool.lst

echo "/usr/local/sbin/update-ca-certificates" | sudo tee -a /opt/bootlocal.sh 
echo "/usr/local/etc/init.d/open-vm-tools restart" | sudo tee -a /opt/bootlocal.sh 


if [[ $1 == *"gui"* ]]; then
  # install firefox and open-vm-tools-desktop packages
  tce-load -wi firefox_getLatest open-vm-tools-desktop

  # deploy firefox policy to disable autoupdate and trust certs
  sudo mkdir -p /etc/firefox/policies
  sudo wget http://www.example.com/build/policies.json -P /etc/firefox/policies
  echo "etc/firefox" >> /opt/.filetool.lst

  # install firefox latest
  firefox_getLatest.sh

  # instead of loading the firefox_getLatest script, load actual firefox
  sudo sed -i 's/firefox_getLatest/firefox/g' /etc/sysconfig/tcedir/onboot.lst

else
  # install open-vm-tools package
  tce-load -wi open-vm-tools
fi

# the follow will run after all else to backup the config.
echo y | backup

The script above looks for gui in the hostname provided and if present installs firefox and the open-vm-tools-desktop. Running the above script was executed like this:

wget http://www.example.com/build/buildscript2.txt
mv buildscript2.txt buildscript2.sh
chmod +x buildscript2.sh
./buildscript2.sh

After running the script, the VM is ready to use. I typically shut down the VM and export it as an OVF or OVA that can be placed on an internal web server and deployed as needed. This creates a super tiny appliance — only 26mb for CLI and 229mb for GUI versions. Not bad for a fully functional OS, with GUI, a web browser, and trusting my internal CA out of the box.

Note: the above script/process has only been tested with TinyCore 14.0 and 15.0 releases.