Can ESXi hosts upgrade via Commandline on Nutanix?

Well, this is thoroughly a fun pet experiment – just to tinker around and see what breaks 🙂 Should you be trying this on a prod cluster? Holy !@#$ no! 🙂

But then, where’s the fun in not trying, eh? 🙂

Here is the environment:

I have four hosts, all connected to the internet. We know from https://www.thevfanatic.com/upgrading-esxi-7-0-x-from-the-command-line/ that the upgrade via the sheer dark and deep blackness of the command line, actually works!

We also, (for people familiar with Nutanix at least) know about the hostssh and allssh bits (for people unfamiliar, will write up a blog shortly should you want it – so comment/email/call and let me know!) which allows for pretty fun tomfoolery (will catch up on that later, I promise). I am sure you are already aware where I am going with this 🙂

So in theory, if we can somehow run the upgrade commands at once on all hosts at the same time, the upgrade should work (please note the bold and italics!)

The reasons for failure of course are

1. There is absolutely no chance in hell that UVMs will not go down if hosts starts rebooting
2. There is absolutely no way CVMs will gracefully shutdown – there goes the neighbourhood
3. Host reboots might not even work – should a reboot be necessary – considering CVMs and UVMs might not even allow the hosts to go in for reboot (No idea, we’ll see 🙂 – told you this wasn’t very scientific)
4. Once the CVMs go down, the running commands might just terminate causing the hosts to go on a dark spiral of potential doom 🙂 – always fun, this bit!

So here goes – here is the plan

1. Log on to one CVM
2. Use the hostssh commandset to throw the below two commands at the hosts

#esxcli network firewall ruleset set -e true -r httpClient
#esxcli software profile update -d  https://hostupdate.vmware.com/software/VUM
/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0U1b-17168206-standard

3. Sit back and watch the world burn 🙂 – I am on Tequila – but get whatever works the best for you 🙂

Cluster details

#ncli cluster info

    Cluster Id                : 0005-0000-00000001a97e::108926
    Cluster Uuid              : 000e-0000-00000001a97e
    Cluster Name              : Kuttakameena
    Cluster Version           : 5.19
    Cluster Full Version      : el7.3-release-euphrates-5.19-stable-5282152e02f3ede70f0957217a62dc436c60b454
    External IP address       : x.x.x.x
    Node Count                : 4
    Block Count               : 1
    Shadow Clones Status      : Enabled
    Has Self Encrypting Disk  : no
    Cluster Masquerading I... : 
    Cluster Masquerading PORT : 
    Is registered to PC       : false
    Is LTS                    : false
    External Data Services... : x.x.x.x
    Support Verbosity Level   : BASIC_COREDUMP
    Lock Down Status          : Disabled
    Password Remote Login ... : Enabled
    Timezone                  : UTC
    NCC Version               : ncc-4.0.0.1
    Common Criteria Mode      : Disabled
    Degraded Node Monitoring  : Enabled

Commands

hostssh "esxcli network firewall ruleset set -e true -r httpClient"

and

hostssh "esxcli software profile update -d  https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0U1b-17168206-standard > /dev/null 2>&1 &"

The reason for the /dev/null and the & is that commands run otherwise will just execute on the host and wait for exit. That is not what we want. So we are pushing the command as a background task with no console output, so the command quickly moves on to the next host in the cluster!

This does start the upgrade on all hosts! Hallelujah!

To monitor, just do a tail -F on esxupdate logs

tail -F /var/run/log/esxupdate.log

From my host

021-01-31T17:16:54Z esxupdate: 2344938: vmware.runcommand: INFO: runcommand called with: args = '/usr/lib/vmware/secureboot/bin/secureMount.py ramdisk -v /tmp/esx-update-2344938/VMware_bootbank_esx-update_7.0.1-0.15.17168206 -p tpmesxup -t /tmp/esx-update-2344938/tpmesxup-2344938 -r esx-update-2344938', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2021-01-31T17:16:54Z esxupdate: 2344938: root: DEBUG: Total of 2625012 bytes were written.
2021-01-31T17:16:54Z esxupdate: 2344938: Ramdisk: INFO: Mount tardisk /tmp/esx-update-2344938/weaselin-2344938 in ramdisk /tmp/esx-update-2344938
2021-01-31T17:16:54Z esxupdate: 2344938: vmware.runcommand: INFO: runcommand called with: args = '/usr/lib/vmware/secureboot/bin/secureMount.py ramdisk -v /tmp/esx-update-2344938/VMware_bootbank_esx-update_7.0.1-0.15.17168206 -p weaselin -t /tmp/esx-update-2344938/weaselin-2344938 -r esx-update-2344938', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2021-01-31T17:16:55Z esxupdate: 2344938: downloader: INFO: Downloading https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/loadesx/VMware_bootbank_loadesx_7.0.1-0.15.17168206.vib to /tmp/esx-update-2344938/VMware_bootbank_loadesx_7.0.1-0.15.17168206
2021-01-31T17:16:56Z esxupdate: 2344938: root: DEBUG: Verifying VIB VMware_bootbank_loadesx_7.0.1-0.15.17168206 signature #1
2021-01-31T17:16:56Z esxupdate: 2344938: root: DEBUG: Total of 75208 bytes were written.
2021-01-31T17:16:56Z esxupdate: 2344938: Ramdisk: INFO: Mount tardisk /tmp/esx-update-2344938/loadesx-2344938 in ramdisk /tmp/esx-update-2344938
2021-01-31T17:16:56Z esxupdate: 2344938: vmware.runcommand: INFO: runcommand called with: args = '/usr/lib/vmware/secureboot/bin/secureMount.py ramdisk -v /tmp/esx-update-2344938/VMware_bootbank_loadesx_7.0.1-0.15.17168206 -p loadesx -t /tmp/esx-update-2344938/loadesx-2344938 -r esx-update-2344938', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2021-01-31T17:16:57Z esxupdate: 2344938: Transaction: DEBUG: Added /tmp/esx-update-2344938/lib64/python3.5/site-packages, /tmp/esx-update-2344938/lib64/python3.5/site-packages/vmware, /tmp/esx-update-2344938

Upgrades does complete

021-01-31T17:18:21Z esxupdate: 2344938: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk list | grep /esx-update-2344938 && localcli system visorfs ramdisk remove -t /tmp/esx-update-2344938', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2021-01-31T17:18:21Z esxupdate: 2344938: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/esxupdt-2344938
2021-01-31T17:18:21Z esxupdate: 2344938: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/tpmesxup-2344938
2021-01-31T17:18:21Z esxupdate: 2344938: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/weaselin-2344938
2021-01-31T17:18:21Z esxupdate: 2344938: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/loadesx-2344938
2021-01-31T17:18:21Z esxupdate: 2344938: root: DEBUG: Finished execution of command = profile.update
2021-01-31T17:18:21Z esxupdate: 2344938: root: DEBUG: Completed esxcli output, going to exit esxcli-software

This is for an upgrade that did not need a host reboot. Will play around a bit more and update what the upgrade looks like when a reboot is needed!

Thank you for reaching thus far. Stay tuned. Subscribe so you can catch new blogs as an when they drop!