Skip to content

Things I keep forgetting

Estimated time to read: 9 minutes

  • Last Updated: February, 2026

Troubleshooting interface problems

I've had a few issues in my lab network recently which have turned out to be caused by interface issues so I'm documenting a couple of NX-OS commands I find useful

show interface brief

  • Is the interface up?
  • Is the speed/MTU correct?
  • Are the transceivers supported and correct on both sides?
  • Is the interface mode (trunk, routed etc) correct?
  • Is there are VLAN required - does the VLAN exist (show vlan)?

show ip int brief

  • Shows the configured IPs and protocol/link/admin status
  • If the link's down check the interface status and look for physical connectivity problems (cable, transceiver, TX/RX etc)

show interface status

  • I use this to check the cable transciver/cable that's been used
  • Is the speed and duplex correct?
  • Do you see link-not-connected or notconnected in the status?
    • Check the physical connections - cable/transceiver correct? RX/TX direction in the correct (i.e. is the cable plugged incorrectly)?

show interface status err-disabled

  • Any errors on the interface?

clear counters interface ethernet 1/1

show int counters errors non-zero

show int counters non-zero

  • Any errors on the interface?

show int e1/49 transceiver details

  • Is the transceiver working correctly?

Catalyst - Booting from TFTP server

This is the procedure I use when I need to boot a Catalyst switch (2960S in this case) from a TFTP server

  • Connect a cable to the Fa0 port on the switch
  • Connect to the console port (CLI)
  • flash_init - Initializes the flash file system and allows the switch to detect the files, including the IOS image to boot
  • mgmt_init - Initializes the Ethernet management port
  • set IP_ADDR 10.1.1.2/255.255.255.0
  • set DEFAULT_ROUTER 10.1.1.1
  • copy tftp://1.2.3.4/c2960s-universalk9-mz.152-2.E9.bin flash:

If you see Err: emac_phy_autonegotiation: timed out. then check management port connectivity

Nexus 9000 - ACI to NX-OS Mode

I've had to convert some ACI switches to NX-OS and use this process

  • Console/SSH to the switch
  • If it's a new switch running the ACI OS and hasn't been connected to a fabric you should see (none)#
  • New switches use admin as the username and no password
  • ifconfig eth0 10.237.102.34 netmask 255.255.255.224
  • route add default gw 10.237.102.33 eth0 cd bootflash wget http://10.237.97.136/software/nxos/nxos.9.3.12.bin clear-bootvars.sh setup-bootvars.sh nxos.9.3.12.bin
  • Verify the bootvars have changed
  • cat /mnt/cfg/0/boot/grub/menu.lst.local
  • cat /mnt/cfg/1/boot/grub/menu.lst.local
  • Type vsh
  • Type reload then y to reload

Use Standard Break Key Sequence Combinations for Password Recovery

Upgrading APIC to new image without going through APIC upgrade UI - using SOL to copy image

I use this for new APIC that I need to wipe and upgrade. Since the APIC is an appliance running on a UCS server I find it's easier in some cases to upgrade the APIC by mapping the ISO and reinstalling through the CIMC. i.e. Not using the APIC UI

  • Copy the APIC ISO to an HTTP server
  • Open a virtual KVM to the server
  • SSH to the APIC CIMC - ssh admin@10.1.1.45
  • C225-WZP# scope vmedia
  • C225-WZP /vmedia # map-www apic http://prod-storage-filer-01.my-domain.com/aci-apic-dk9.6.1.2f.iso
  • Check the mapping status with show mappings detail
  • connect host to follow the install
  • Power cycle the server from the KVM
  • Pre F6 as the server boots to get into the server boot menu
  • Select the Cisco CIMC-Mapped vDVD
  • Monitor the progress as the APIC ISO is deployed

Note

If you see a message such as the following then make sure you complete the prompt through the Serial over LAN (SOL)

To speed up the install, enter iso url in next ten minutes:
? http://10.1.1.45/aci-apic-dk9.6.1.2f.iso
++ awk -F '/|:' '{print $4}'
+ urlip=10.1.1.45
+ '[' -z http://10.1.1.45/aci-apic-dk9.6.1.2f.iso ']'
+ '[' -z 10.1.1.45 ']'
+ break
+ '[' -n http://10.1.1.45/aci-apic-dk9.6.1.2f.iso ']'
+ set +e
+ configured=0
+ '[' 0 -eq 0 ']'
+ echo 'Configuring network interface'
Configuring network interface
+ echo 'type static, dhcp, bash for a shell to configure networking, or url to re-enter the url: '
You can enable the Serial over LAN port through SSH (ssh ssh admin@10.1.1.45) and access it via the KVM

scope sol
set enabled yes
set baud-rate 115200
commit
exit

Installing tcpdump 4.99 on Ubuntu 20.04 (Focal)

I needed version 4.99 so I could see the interface name

  • wget https://ftp.debian.org/debian/pool/main/libp/libpcap/libpcap0.8_1.10.0-2_amd64.deb
  • wget https://ftp.debian.org/debian/pool/main/t/tcpdump/tcpdump_4.99.0-2+deb11u1_amd64.deb
  • sudo dpkg -i tcpdump_4.99.0-2+deb11u1_amd64.deb libpcap0.8_1.10.0-2_amd64.deb
  • tcpdump --version

NX-OS Rollback and Config Diff

I knew NX-OS had a rollback/commit feature but never got around to playing with it until recently. I also found out about the checkpoint diff feature from a colleague when I needed to test some config and wanted to capture just the new config to put into documentation.

In general the rollback/commit feature allows you to save the current running config as a checkpoint and later revert (rollback) to that saved config if needed.

In this example I created a checkpoint without BGP, added some BGP config, showed the diff of the running config vs the initial_config checkpoint, and then rolled back to the initial_config (without the BGP)

  • checkpoint initial_config
  • show run bgp
  • conf t
  • router bgp 65500 ...
  • show diff rollback-patch checkpoint initial_config running-config
  • rollback running-config checkpoint initial_config atomic

Rollback Options

  • atomic will stop the rollback and revert to original configuration if any errors are discovered
  • best-effort skips errors and proceeds with rollback
  • stop-at-first-failure stops the rollback at the first error

Note

As a bonus, you might find it helpful to use show cli history to show the commands used since logging into the device

MTU Issues

I've previously had some issues due to MTU mismatch on a path. e.g. in a recent case of installing Openshift the nodes couldn't pull the specific container images from Quay.io. To troubleshoot the issue I SSH'd to a node and ran podman pull <image that failed>. The error I received was local error: tls: bad record MAC. I've seen in a few Github issues and forums that one possible cause of this can be incorrect MTU. There's a basic script below to quickly check MTU

Basic Ping test with increasing MTU

import subprocess
import sys

def check_mtu(destination):
    # Check if destination is available
    # -c says to stop after <count> replies
    # -W time to wait for response
    check_alive = subprocess.run(["ping", "-c", "1", "-W", "2", destination], stdout=subprocess.DEVNULL)
    if check_alive.returncode != 0:
        print(f"ERROR: {destination} is not responding to standard pings. Script cannot determine MTU.")
        return

    # Test from very small to large
    test_payloads = [1200, 1300, 1400, 1450, 1472, 1500,2500,9000]

    print(f"Probing Path MTU to {destination}...")
    for size in test_payloads:
        # -M do: Don't fragment
        # -s: payload size
        # -c 1: one packet
        # -W 1: 1 second timeout
        cmd = ["ping", "-M", "do", "-s", str(size), "-c", "1", "-W", "1", destination]

        # Convert list to string for display
        cmd_string = " ".join(cmd)

        # Run the command
        result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

        # Packet size = MTU - 28 bytes (IP + ICMP headers)
        mtu = size + 28
        if result.returncode == 0:
            print(f"[+] SUCCESS | MTU: {mtu} | Command: {cmd_string}")
        else:
            print(f"[-] FAILED  | MTU: {mtu} | Command: {cmd_string}")

if __name__ == "__main__":
    dest = sys.argv[1] if len(sys.argv) > 1 else "8.8.8.8"
    check_mtu(dest)

Comments