NSX-T Troubleshooting Tunnel Status
Switching between a mix of different product versions, I forget some of the specifics, and I've wasted too much time troubleshooting something that was working OK. Hopefully this saves you some time.
vmk10 / vmk11 / vmk50 not showing up in vCenter
After deploying NSX-T 2.5 in the lab and configuring ESXi hosts for NSX-T, even though it was successful, I didn't see vmk10/vmk11/vmk50 listed in VMkernel adapters for the ESXi hosts in vCenter
FACT: When using NVDS, vmk10/vmk11/vmk50 will NOT show up in vCenter. (With NSX-T 3.0 on VDS, TEP vmkernel adapters will show up in vCenter for each host).
Check within NSX-T Manager:
Click on the Interface Details to see the IP/subnet mask
To confirm ESXi hosts are configured for NSX-T, ssh to the host:
1[root@site-210-esxi1:~] esxcli software vib list |grep -i nsx
2nsx-adf 2.5.1.0.0-6.7.15314402 VMware VMwareCertified 2020-05-19
3nsx-aggservice 2.5.1.0.0-6.7.15314423 VMware VMwareCertified 2020-05-19
4nsx-cli-libs 2.5.1.0.0-6.7.15314375 VMware VMwareCertified 2020-05-19
5nsx-common-libs 2.5.1.0.0-6.7.15314375 VMware VMwareCertified 2020-05-19
6nsx-context-mux 2.5.1.0.0esx67-15314456 VMware VMwareCertified 2020-05-19
7nsx-esx-datapath 2.5.1.0.0-6.7.15314311 VMware VMwareCertified 2020-05-19
8nsx-exporter 2.5.1.0.0-6.7.15314423 VMware VMwareCertified 2020-05-19
9nsx-host 2.5.1.0.0-6.7.15314289 VMware VMwareCertified 2020-05-19
10nsx-metrics-libs 2.5.1.0.0-6.7.15314375 VMware VMwareCertified 2020-05-19
11nsx-mpa 2.5.1.0.0-6.7.15314423 VMware VMwareCertified 2020-05-19
12nsx-nestdb-libs 2.5.1.0.0-6.7.15314375 VMware VMwareCertified 2020-05-19
13nsx-nestdb 2.5.1.0.0-6.7.15314393 VMware VMwareCertified 2020-05-19
14nsx-netcpa 2.5.1.0.0-6.7.15314440 VMware VMwareCertified 2020-05-19
15nsx-netopa 2.5.1.0.0-6.7.15314363 VMware VMwareCertified 2020-05-19
16nsx-opsagent 2.5.1.0.0-6.7.15314423 VMware VMwareCertified 2020-05-19
17nsx-platform-client 2.5.1.0.0-6.7.15314423 VMware VMwareCertified 2020-05-19
18nsx-profiling-libs 2.5.1.0.0-6.7.15314375 VMware VMwareCertified 2020-05-19
19nsx-proxy 2.5.1.0.0-6.7.15314435 VMware VMwareCertified 2020-05-19
20nsx-python-gevent 1.1.0-9273114 VMware VMwareCertified 2020-05-19
21nsx-python-greenlet 0.4.9-12819723 VMware VMwareCertified 2020-05-19
22nsx-python-logging 2.5.1.0.0-6.7.15314402 VMware VMwareCertified 2020-05-19
23nsx-python-protobuf 2.6.1-12818951 VMware VMwareCertified 2020-05-19
24nsx-rpc-libs 2.5.1.0.0-6.7.15314375 VMware VMwareCertified 2020-05-19
25nsx-sfhc 2.5.1.0.0-6.7.15314423 VMware VMwareCertified 2020-05-19
26nsx-shared-libs 2.5.1.0.0-6.7.15036308 VMware VMwareCertified 2020-05-19
27nsx-upm-libs 2.5.1.0.0-6.7.15314375 VMware VMwareCertified 2020-05-19
28nsx-vdpi 2.5.1.0.0-6.7.15314422 VMware VMwareCertified 2020-05-19
29nsxcli 2.5.1.0.0-6.7.15314296 VMware VMwareCertified 2020-05-19
30[root@site-210-esxi1:~]
List vmkernel adapters on the host:
1[root@site-210-esxi1:~] esxcli network ip interface ipv4 get
2Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
3----- ------------- --------------- --------------- ------------ ------- --------
4vmk0 172.31.210.11 255.255.255.224 172.31.210.31 STATIC 0.0.0.0 false
5vmk10 172.31.210.36 255.255.255.224 172.31.210.63 STATIC 0.0.0.0 false
6vmk11 172.31.210.37 255.255.255.224 172.31.210.63 STATIC 0.0.0.0 false
7vmk50 169.254.1.1 255.255.0.0 169.254.255.255 STATIC 0.0.0.0 false
8[root@site-210-esxi1:~]
NSX-T Tunnel Status down / Not Available
Tunnel Status shows as Not Available:
There's no details listed in Tunnel Status for the host:
FACT: If there's no workloads running on a host, the tunnel is not established.
You can still confirm TEP communication using vmkping.
Confirm vxlan netstack is there:
1[root@site-210-esxi1:~] esxcli network ip netstack list
2defaultTcpipStack
3 Key: defaultTcpipStack
4 Name: defaultTcpipStack
5 State: 4660
6
7vxlan
8 Key: vxlan
9 Name: vxlan
10 State: 4660
11
12hyperbus
13 Key: hyperbus
14 Name: hyperbus
15 State: 4660
16[root@site-210-esxi1:~]
Use vmkping to confirm connectivity between all TEP interfaces. Don't forget to test packets at least 1600 bytes in size.
1[root@site-210-esxi1:~] vmkping -I vmk10 -S vxlan 172.31.210.38 -d -s 1572
2PING 172.31.210.38 (172.31.210.38): 1572 data bytes
31580 bytes from 172.31.210.38: icmp_seq=0 ttl=64 time=0.585 ms
41580 bytes from 172.31.210.38: icmp_seq=1 ttl=64 time=0.660 ms
51580 bytes from 172.31.210.38: icmp_seq=2 ttl=64 time=0.423 ms
6
7--- 172.31.210.38 ping statistics ---
83 packets transmitted, 3 packets received, 0% packet loss
9round-trip min/avg/max = 0.423/0.556/0.660 ms
10
11[root@site-210-esxi1:~]
If it fails on larger packet sizes, check the MTU size on everything from vmkernel adapter, VDS, each physical switchport, VLAN and every device in between ESXi hosts. This is the most common issue.
1[root@site-210-esxi1:~] vmkping -I vmk11 -S vxlan 172.31.210.38 -d -s 1572
2PING 172.31.210.38 (172.31.210.38): 1572 data bytes
3sendto() failed (Message too long)
4sendto() failed (Message too long)
5sendto() failed (Message too long)
6
7--- 172.31.210.38 ping statistics ---
83 packets transmitted, 0 packets received, 100% packet loss
9
10[root@site-210-esxi1:~]
If vmkping fails to receive packets:
- Check the TEP VLAN is configured on the physical switchports
- Check the correct TEP / Transport VLAN is configured in the Uplink Profile used by the Transport Node Profile