Upgrading ESXi 7.0.x from the command line

So, you are done with https://www.thevfanatic.com/understanding-esxi-7-0-1-upgrades-command-line/ and are now aching to see what a successful upgrade looks like?

Fear not! We have a successful upgrade to feast our eyes on!

So again same steps as the last time – with the exception that this upgrade is on Youtube!

So we have the same host on the same build as last time – but with a TON of terminals open! This will allow us to monitor the upgrade in real time.

This is the upgrade

We are monitoring the upgrades from six (yes you read that correctly) terminals. Let me explain

Terminal 1 – the actual command
Terminal 2 – vmkernel
Terminal 3 – hostd
Terminal 4 – esxupdate
Terminal 5 – df -h
Terminal 6 – vdf -h

Commands are below

Terminal 1

#esxcli network firewall ruleset set -e true -r httpClient
#esxcli software profile update -d https://hostupdate.vmware.com/software/VUM
/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0U1b-17168206-standard

Terminal 2
tail -F /var/run/log/vmkernel.log
Terminal 3
tail -F /var/run/log/hostd.log
Terminal 4
tail -F /var/run/log/esxupdate.log
Terminal 5
watch -d "df -h"
Terminal 6
watch -d "vdf -h"

So from Terminal 1’s perspective, the upgrade is pretty straightforward

[root@Juda-4:~] esxcli network firewall ruleset set -e true -r httpClient
[root@Juda-4:~] esxcli software profile update -d https://hostupdate.vmware.com/software/VUM
/PRODUCTION/main/vmw-depot-index.xml -p ESXi-7.0U1b-17168206-standard
Update Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed:
VIBs Removed:
VIBs Skipped: VMW_bootbank_bnxtnet_216.0.50.0-16vmw.701.0.0.16850804, VMW_bootbank_bnxtroce_216.0.58.0-7vmw.701.0.0.16850804, VMW_bootbank_brcmfcoe_12.0.1500.0-1vmw.701.0.0.16850804, VMW_bootbank_brcmnvmefc_12.6.278.10-3vmw.701.0.0.16850804, VMW_bootbank_elxiscsi_12.0.1200.0-2vmw.701.0.0.16850804, VMW_bootbank_elxnet_12.0.1250.0-5vmw.701.0.0.16850804, VMW_bootbank_i40en_1.8.1.123-1vmw.701.0.0.16850804, VMW_bootbank_i40iwn_1.1.2.6-1vmw.701.0.0.16850804, VMW_bootbank_iavmd_2.0.0.1055-5vmw.701.0.0.16850804, VMW_bootbank_icen_1.0.0.9-1vmw.701.0.0.16850804, VMW_bootbank_igbn_0.1.1.0-7vmw.701.0.0.16850804, VMW_bootbank_iser_1.1.0.1-1vmw.701.0.0.16850804, VMW_bootbank_ixgben_1.7.1.28-1vmw.701.0.0.16850804, VMW_bootbank_lpfc_12.6.278.10-8vmw.701.0.0.16850804, VMW_bootbank_lpnic_11.4.62.0-1vmw.701.0.0.16850804, VMW_bootbank_lsi-mr3_7.712.51.00-1vmw.701.0.0.16850804, VMW_bootbank_lsi-msgpt2_20.00.06.00-3vmw.701.0.0.16850804, VMW_bootbank_lsi-msgpt35_13.00.13.00-2vmw.701.0.0.16850804, VMW_bootbank_lsi-msgpt3_17.00.10.00-2vmw.701.0.0.16850804, VMW_bootbank_mtip32xx-native_3.9.8-1vmw.701.0.0.16850804, VMW_bootbank_ne1000_0.8.4-11vmw.701.0.0.16850804, VMW_bootbank_nenic_1.0.29.0-2vmw.701.0.0.16850804, VMW_bootbank_nfnic_4.0.0.44-2vmw.701.0.0.16850804, VMW_bootbank_nhpsa_70.0050.0.100-1vmw.701.0.0.16850804, VMW_bootbank_nmlx4-core_3.19.16.8-2vmw.701.0.0.16850804, VMW_bootbank_nmlx4-en_3.19.16.8-2vmw.701.0.0.16850804, VMW_bootbank_nmlx4-rdma_3.19.16.8-2vmw.701.0.0.16850804, VMW_bootbank_nmlx5-core_4.19.16.8-2vmw.701.0.0.16850804, VMW_bootbank_nmlx5-rdma_4.19.16.8-2vmw.701.0.0.16850804, VMW_bootbank_ntg3_4.1.5.0-0vmw.701.0.0.16850804, VMW_bootbank_nvme-pcie_1.2.3.9-2vmw.701.0.0.16850804, VMW_bootbank_nvmerdma_1.0.1.2-1vmw.701.0.0.16850804, VMW_bootbank_nvmxnet3-ens_2.0.0.22-1vmw.701.0.0.16850804, VMW_bootbank_nvmxnet3_2.0.0.30-1vmw.701.0.0.16850804, VMW_bootbank_pvscsi_0.1-2vmw.701.0.0.16850804, VMW_bootbank_qcnic_1.0.15.0-10vmw.701.0.0.16850804, VMW_bootbank_qedentv_3.40.3.0-12vmw.701.0.0.16850804, VMW_bootbank_qedrntv_3.40.4.0-12vmw.701.0.0.16850804, VMW_bootbank_qfle3_1.0.67.0-9vmw.701.0.0.16850804, VMW_bootbank_qfle3f_1.0.51.0-14vmw.701.0.0.16850804, VMW_bootbank_qfle3i_1.0.15.0-9vmw.701.0.0.16850804, VMW_bootbank_qflge_1.1.0.11-1vmw.701.0.0.16850804, VMW_bootbank_rste_2.0.2.0088-7vmw.701.0.0.16850804, VMW_bootbank_sfvmk_2.4.0.0010-15vmw.701.0.0.16850804, VMW_bootbank_smartpqi_70.4000.0.100-3vmw.701.0.0.16850804, VMW_bootbank_vmkata_0.1-1vmw.701.0.0.16850804, VMW_bootbank_vmkfcoe_1.0.0.2-1vmw.701.0.0.16850804, VMW_bootbank_vmkusb_0.1-1vmw.701.0.0.16850804, VMW_bootbank_vmw-ahci_2.0.5-2vmw.701.0.0.16850804, VMware_bootbank_cpu-microcode_7.0.1-0.15.17168206, VMware_bootbank_crx_7.0.1-0.15.17168206, VMware_bootbank_elx-esx-libelxima.so_12.0.1200.0-3vmw.701.0.0.16850804, VMware_bootbank_esx-base_7.0.1-0.15.17168206, VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.1-0.15.17168206, VMware_bootbank_esx-ui_1.34.4-16668064, VMware_bootbank_esx-update_7.0.1-0.15.17168206, VMware_bootbank_esx-xserver_7.0.1-0.15.17168206, VMware_bootbank_gc_7.0.1-0.15.17168206, VMware_bootbank_loadesx_7.0.1-0.15.17168206, VMware_bootbank_lsuv2-hpv2-hpsa-plugin_1.0.0-3vmw.701.0.0.16850804, VMware_bootbank_lsuv2-intelv2-nvme-vmd-plugin_1.0.0-2vmw.701.0.0.16850804, VMware_bootbank_lsuv2-lsiv2-drivers-plugin_1.0.0-4vmw.701.0.0.16850804, VMware_bootbank_lsuv2-nvme-pcie-plugin_1.0.0-1vmw.701.0.0.16850804, VMware_bootbank_lsuv2-oem-dell-plugin_1.0.0-1vmw.701.0.0.16850804, VMware_bootbank_lsuv2-oem-hp-plugin_1.0.0-1vmw.701.0.0.16850804, VMware_bootbank_lsuv2-oem-lenovo-plugin_1.0.0-1vmw.701.0.0.16850804, VMware_bootbank_lsuv2-smartpqiv2-plugin_1.0.0-4vmw.701.0.0.16850804, VMware_bootbank_native-misc-drivers_7.0.1-0.15.17168206, VMware_bootbank_qlnativefc_4.0.3.0-17vmw.701.0.0.16850804, VMware_bootbank_vdfs_7.0.1-0.15.17168206, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.38-1vmw.701.0.0.16850804, VMware_bootbank_vsan_7.0.1-0.15.17168206, VMware_bootbank_vsanhealth_7.0.1-0.15.17168206, VMware_locker_tools-light_11.1.1.16303738-16850804

Where this becomes interesting, is to see that vmkernel creates a 10gb partition and mounts it so the upgrade can be done (note the lines with arrows)

This is seen in Terminal 2

[root@Juda-4:~] tail -F /var/run/log/vmkernel.log
2021-01-31T15:40:50.882Z cpu22:2328988)SchedVsi: 2098: Group: host/user/ntnx(30217): min=64 max=200 minLimit=64, units: mb
2021-01-31T15:41:58.386Z cpu18:2098176)DVFilter: 6344: Checking disconnected filters for timeouts
2021-01-31T15:42:16.477Z cpu22:2329282)Uplink: 2499: void: not found
2021-01-31T15:42:16.478Z cpu22:2329282)Uplink: 2499: void: not found
2021-01-31T15:42:31.675Z cpu12:2329325)SchedVsi: 2098: Group: host/user/ntnx(30217): min=64 max=200 minLimit=64, units: mb
2021-01-31T15:44:12.620Z cpu16:2329545)SchedVsi: 2098: Group: host/user/ntnx(30217): min=64 max=200 minLimit=64, units: mb
2021-01-31T15:45:53.458Z cpu23:2329866)SchedVsi: 2098: Group: host/user/ntnx(30217): min=64 max=200 minLimit=64, units: mb
2021-01-31T15:47:16.535Z cpu13:2330185)Uplink: 2499: void: not found
2021-01-31T15:47:16.536Z cpu13:2330185)Uplink: 2499: void: not found
2021-01-31T15:47:34.369Z cpu24:2330225)SchedVsi: 2098: Group: host/user/ntnx(30217): min=64 max=200 minLimit=64, units: mb
2021-01-31T15:49:15.128Z cpu8:2330491)SchedVsi: 2098: Group: host/user/ntnx(30217): min=64 max=200 minLimit=64, units: mb
2021-01-31T15:50:27.721Z cpu19:2330819)VisorFSRam: 878: esx-update-2330402 with (0,14,0,0,1363) <---------
2021-01-31T15:50:27.721Z cpu19:2330819)FSS: 8524: Mounting fs visorfs (430a64081820) with -o 0,14,0,0,0,755,esx-update-2330402 on file descriptor 430ba7013c80 <---------
2021-01-31T15:50:30.012Z cpu21:2330846)VisorFSTar: 2005: loadesx-2330402 (7504294321315778050) as loadesx-2330402 for 75208 bytes
2021-01-31T15:50:35.009Z cpu26:2330911)VisorFSTar: 2005: esxupdt-2330402 (7504294321315778068) as esxupdt-2330402 for 1333122 bytes
2021-01-31T15:50:35.628Z cpu1:2330926)VisorFSTar: 2005: tpmesxup-2330402 (5063660450781204070) as tpmesxup-2330402 for 12272 bytes
2021-01-31T15:50:36.296Z cpu7:2330952)VisorFSTar: 2005: weaselin-2330402 (5063660450781204075) as weaselin-2330402 for 2625012 bytes
2021-01-31T15:50:56.074Z cpu24:2331032)SchedVsi: 2098: Group: host/user/ntnx(30217): min=64 max=200 minLimit=64, units: mb
2021-01-31T15:51:58.386Z cpu16:2098176)DVFilter: 6344: Checking disconnected filters for timeouts
2021-01-31T15:51:59.855Z cpu29:2331324)VisorFSRam: 878: stagebootbank with (0,4094,0,0,1363) <--------- 
2021-01-31T15:51:59.855Z cpu29:2331324)FSS: 8524: Mounting fs visorfs (430a64081820) with -o 0,4094,0,0,0,755,stagebootbank on file descriptor 430ba71d7880
2021-01-31T15:52:00.191Z cpu14:2331330)FSS: 8695: Unmounting file descriptor 430ba71e5b30 <---------------
2021-01-31T15:52:00.191Z cpu14:2331330)VisorFSRam: 1164: inode 8255036136297213440
2021-01-31T15:52:00.191Z cpu14:2331330)VisorFSRam: 1181: ramdisk stagebootbank
2021-01-31T15:52:00.191Z cpu14:2331330)VisorFSRam: 1347: for ramdisk stagebootbank
2021-01-31T15:52:00.567Z cpu10:2331337)FSS: 8695: Unmounting file descriptor 430ba71b2f60
2021-01-31T15:52:00.567Z cpu10:2331337)VisorFSRam: 1164: inode 7504294321315778048
2021-01-31T15:52:00.567Z cpu10:2331337)VisorFSRam: 1181: ramdisk esx-update-2330402
2021-01-31T15:52:00.567Z cpu10:2331337)VisorFSRam: 1347: for ramdisk esx-update-2330402
2021-01-31T15:52:00.576Z cpu4:2330402)VisorFSTar: 782: loadesx-2330402
2021-01-31T15:52:00.576Z cpu4:2330402)VisorFSObj: 1901: tardisk loadesx-2330402
2021-01-31T15:52:00.576Z cpu4:2330402)VisorFSTar: 782: esxupdt-2330402
2021-01-31T15:52:00.577Z cpu4:2330402)VisorFSObj: 1901: tardisk esxupdt-2330402
2021-01-31T15:52:00.577Z cpu4:2330402)VisorFSTar: 782: tpmesxup-2330402
2021-01-31T15:52:00.577Z cpu4:2330402)VisorFSObj: 1901: tardisk tpmesxup-2330402
2021-01-31T15:52:00.577Z cpu4:2330402)VisorFSTar: 782: weaselin-2330402
2021-01-31T15:52:00.578Z cpu4:2330402)VisorFSObj: 1901: tardisk weaselin-2330402
2021-01-31T15:52:16.805Z cpu21:2331463)Uplink: 2499: void: not found
2021-01-31T15:52:16.806Z cpu21:2331463)Uplink: 2499: void: not found

Next, the bootbank/altbootbank comes into the picture as the vibs/images need to be copied to the boot partitions

The esxupdate parition mount can be seen clearly in the video (will be linked here)

Hostd, for the most part, just auth sessions etc and that can be seen in the hostd logs (massive – so NOT pasting it in the main page – will create a separate page for logs and will link it so the logs can be cross referenced

Logs are here

So in short

ESXi reaches out to the VMware depot
Downloads the update bundle
Adds/removes the vibs as needed
Mounts it on a staging partition
Swaps out the /bootbank or /altbootbank as needed
Completes upgrade!

Pretty nifty if you ask me!

Will update this post with the Youtube link and the log dumps in case you want to analyse the logs in full

Thank you for reaching this far if you are reading this. Stay tuned for the next blogs