Here are the instructions of how to perform an un-interrupted firmware upgrade on the Nexus N9K switches. An ISSU (In Service Software Upgrade). The 9300 Nexus switches only have a single supervisor thus a single CPU. The CPU is reset to load the new firmware version, the system then restores the control plane back to the CPU thus eliminating any data plane traffic. The data plane traffic is not disrupted during the upgrade process, so packets are forwarded during this process.
To allow Nexus N9K switches to be upgraded as an ISSU, boot mode LXC needs to enabled at the very beginning. Note: this process is disruptive as the switch needs to be rebooted
conf t boot mode lxc
ISSU with FEX
After researching this we found that we can't perform an ISSU upgrade on our infrastructure as we are using FEX (Fabric Extenders) and ISSU with FEX is not supported on the 9300 Nexus switches. For the remainder of this document, we won't be upgrading using ISSU.
Nexus N9K
Upgrade process for a N9K
Check the current running version of NX-OS
show version
Log into the switch and confirm you can see the previous firmware in there. In this case I can see nxos.9.3.5.bin
dir bootflash:
cs1# dir bootflash:
4096 Nov 20 05:32:16 2020 .rpmstore/
4096 Nov 20 05:32:39 2020 .swtam/
22349 Jan 11 00:01:13 2021 20210111_000026_poap_7032_init.log
152216 Jan 11 22:36:16 2021 20210111_222132_poap_7120_init.log
2097163 Jan 15 05:54:02 2021 20210112_213501_poap_7077_1.log
853879 Jan 15 07:25:52 2021 20210112_213501_poap_7077_2.log
2097188 Jan 13 01:18:31 2021 20210112_213501_poap_7077_init.log
871967 Jan 18 02:14:43 2021 20210117_205732_poap_7036_1.log
2097188 Jan 18 00:41:06 2021 20210117_205732_poap_7036_init.log
30501 Jan 18 20:01:46 2021 20210118_200007_poap_7014_init.log
1777998029 Nov 20 05:43:21 2020 aci-n9000-dk9.14.1.2g.bin
4096 Nov 20 05:32:41 2020 eem_snapshots/
4096 Nov 20 05:32:38 2020 evt_log_snapshot/
4096 Nov 20 05:40:09 2020 home/
16384 Nov 20 05:31:51 2020 lost+found/
1971323904 Nov 20 05:40:08 2020 nxos.9.3.5.bin
0 Nov 20 05:36:34 2020 platform-sdk.cmd
2527331 Jan 18 20:01:35 2021 poap_retry_debugs.log
4096 Nov 20 05:33:08 2020 scripts/
740 Feb 24 19:02:00 2021 tmptmp
4096 Nov 20 05:34:48 2020 virt_strg_pool_bf_vdc_1/
4096 Nov 20 05:33:09 2020 virtual-instance/
59 Feb 24 19:00:47 2021 virtual-instance.conf
16528 Feb 24 19:00:56 2021 vlan.dat
Usage for bootflash://sup-local
4003651584 bytes used
112585560064 bytes free
116589211648 bytes total
Get the boot image onto the switch. Either copy directly from an scp server or copy with USB
# Copy from Cisco
copy scp://[email protected]//download/nxos.9.3.6.bin bootflash:nxos.9.3.6.bin
# Or Copy from USB dir usb1: copy usb1:nxos.9.3.6.bin bootflash:nxos.9.3.6.bin
Verify boot image, compare this to the download's details on the Cisco site
show file bootflash://sup-1/nxos.9.3.6.bin sha512sum
cs1# show file bootflash://sup-1/nxos.9.3.6.bin sha512sum 0ce78fcdad91d33d18cb3794b309b1ad572713fd2f594234145412a602a8f6824ad8bd379f938dadc6c986b1bb730372d3f57269d1cedcf474d944079fd7be02
Check the impact of running the upgrade. Check for any messages to why the upgrade may not complete successfully.
show install all impact nxos bootflash:nxos.9.3.6.bin
nz# show install all impact nxos bootflash:nxos.9.3.6.bin
Installer will perform impact only check. Please wait.
Verifying image bootflash:/nxos.9.3.6.bin for boot variable "nxos".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Preparing "nxos" version info using image bootflash:/nxos.9.3.6.bin.
[####################] 100% -- SUCCESS
Preparing "bios" version info using image bootflash:/nxos.9.3.6.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset default upgrade is not hitless
101 yes disruptive reset default upgrade is not hitless
102 yes disruptive reset default upgrade is not hitless
Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- -------------------- ------------
1 nxos 9.3(5) 9.3(6) yes
1 bios v05.39(08/30/2019):v05.39(08/30/2019) v05.42(06/14/2020) yes
101 fex4 9.3(5) 9.3(6) yes
102 fex4 9.3(5) 9.3(6) yes
Additional info for this installation:
--------------------------------------
Service "vpc" in vdc 1: Vpc is enabled, Please make sure both Vpc peer switches have same boot mode using 'show boot mode' and proceed
Possible Messages - In this example, the upgrade process is disruptive
| Reason Field Message — in Cisco NX-OS Release 7.0(3)I3(1) | Reason Field Message — in Cisco NX-OS Release 7.0(3)I4(1) or a Later Release | Description | ||
| Incompatible image | Incompatible image for ISSU | The Cisco NX-OS image to which you are attempting to upgrade does not support ISSU. | ||
| Hitless upgrade is not supported | Default upgrade is not hitless | By default, the software upgrade process is disruptive. You must configure the non-disruptive option to perform an ISSU. |
Make sure that the same boot mode is enabled on each switch (we have two for VPC)
cs1# show boot mode Current mode is native.
Perform the upgrade, as the upgrade is disruptive (the device will reboot) it will ask you if you would like to continue, select Y. You can also use the non-interruptive option.
install all nxos bootflash:nxos.9.3.6.bin
cs1# install all nxos bootflash:nxos.9.3.6.bin
Installer will perform compatibility check first. Please wait.
Installer is forced disruptive
Verifying image bootflash:/nxos.9.3.6.bin for boot variable "nxos".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Preparing "nxos" version info using image bootflash:/nxos.9.3.6.bin.
[####################] 100% -- SUCCESS
Preparing "bios" version info using image bootflash:/nxos.9.3.6.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset default upgrade is not hitless
101 yes disruptive reset default upgrade is not hitless
102 yes disruptive reset default upgrade is not hitless
Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- -------------------- ------------
1 nxos 9.3(5) 9.3(6) yes
1 bios v05.39(08/30/2019):v05.39(08/30/2019) v05.42(06/14/2020) yes
101 fex4 9.3(5) 9.3(6) yes
102 fex4 9.3(5) 9.3(6) yes
Additional info for this installation:
--------------------------------------
Service "vpc" in vdc 1: Vpc is enabled, Please make sure both Vpc peer switches have same boot mode using 'show boot mode' and proceed
Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)? [n] y
Install is in progress, please wait.
Performing runtime checks.
[####################] 100% -- SUCCESS
Setting boot variables.
[####################] 100% -- SUCCESS
Performing configuration copy.
[####################] 100% -- SUCCESS
Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS
Finishing the upgrade, switch will reboot in 10 seconds.
You can view the status of the upgrade
show install all status
Once complete, check you are now running the current version of NX-OS
show version
Fabric Extenders
Fabric extenders are upgraded once the N9K reboots. You can use the following command to see where they are at with being provisioned
show fex
cs1# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------ 101 cs1e1 Image Download N2K-C2348UPQ-10GE 102 cs1e2 Image Download N2K-C2348TQ-10G-E cs1# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------ 101 cs1e1 Image Download N2K-C2348UPQ-10GE 102 cs1e2 Offline N2K-C2348TQ-10G-E cs1# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------ 101 cs1e1 Online N2K-C2348UPQ-10GE 102 cs1e2 Online N2K-C2348TQ-10G-E
You can check the fex version by logging into the fex
attach fex 101 show version
cs1# attach fex 101
Attaching to FEX 101 ...
To exit type 'exit', to abort type '$.'
fex-101# sh ver
Software
Bootloader version: 1.19
System boot mode: primary
System image version: 9.3(6) [build 9.3(6)]
Fpga version: 0x0800
Hardware
Module: Fabric Extender 48x10GE + 6x40GE Module
CPU: Motorola, e500v2, core 0
Serial number:
Bootflash: unlocked
PSOC Version: v6.3_RC24
Kernel uptime is 0 day(s), 0 hour(s), 23 minutes(s), 27 second(s)
Last reset at Wed Feb 24 22:21:38 2021
Reason: Kernel Reboot
Service: Service:Reload new image