Negotiation issue on Intel X550 T2

Negotiation issue on Intel X550 T2

^{Created: 2020-09-15 23:53:58 | Last modified: 2020-09-17 04:35:39}
^{Access: Read}^{| Views: 98}^{| Rating: +1}^{| Tags: intel linux xenserver windows}
Fix for 10Gb/s negotionation issue on X550-T2 network cards

We had an issue with our Linux servers using the X550-T2 10Gb/s network cards constantly down negotiate to 1Gb/s. We have these plugged into our Nexus 3172T Chassis., When plugging the NICs into each other, they negotiate with out any problems, only when plugged into this particular switch.

Reason:

This is because NBASE-T (2.5G/5G) capabilities on the X550 product family and attempting to connect to a 10GBASE-T switch that implements the Broadcom Limited* 84848 quad-port 10GBASE-T PHY, the connection may not achieve a 10G link.

To test the issue:

Method One (driver present):

Linux Driver: Use the following command
ethtool -s ethX advertise 0x1000

Windows Driver: Set the “Speed & Duplex” to '10 Gbps Full Duplex'

Method Two (driverless):

Uninstall the X550 device driver.
Disconnect the X550 device from the switch.
Power cycle the system.
Re-connect the X550 device to the switch.
After link is established, query the switch to verify 10G link has been established.

Method 1 - The workaround/resolution for Linux/Centos:

Confirm you have initscripts updated to the latest version

yum update initscripts

Modify the ifcfg-ethX interface file and add the ETHTOOL_OPTS option, this will make sure the ethtool is run every time the interface is brought up.

# Edit the config file
cd /etc/sysconfig/network-scripts/
vi ifcfg-eth0

# Add the following to the config
ETHTOOL_OPTS="ethtool -s ethX advertise 0x1000"

Method 2 - When the ifcfg-ethX interface isn't available e.g. inside XenServer / XCP

Inside XenServer, the ifcfg-ethX interfaces aren't available. To get round this run the ethtool on startup, do this by editing the rc.local file. Below will append the advertise settings to this file with eth2 and eth3 set, rc.local will also be set to executable.

echo '/usr/sbin/ethtool -s eth2 advertise 0x1000' >> /etc/rc.d/rc.local
echo '/usr/sbin/ethtool -s eth3 advertise 0x1000' >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local

Technical Detail

When advertising NBASE-T (2.5G/5G) capabilities on the X550 product family and attempting to connect to a 10GBASE-T switch that implements the Broadcom Limited* 84848 quad-port 10GBASE-T PHY, the connection may not achieve a 10G link. The above steps can be used to identify the issue. The purpose of these steps is to disable the X550 NBASE-T (2.5G/5G) technology advertisement and observe whether the link establishment issue still exists. In the event that the link establishment issue still exists, the issue is unrelated to NBASE-T and must be further investigated.

Once you make the ethtool change, if you were to run "ethtool ethX" inside Linux, you will see the only advertised link mode is 10Gb/s (in this example)

Intel's recommend solution

This issue can be resolved by a firmware update available for the Broadcom Limited* 84848 device. Please contact your switch vendor support team for further assistance.

Notes from Intel

The fix is expected to come in future Linux driver release but it's important to mention that since the system being used is a Xenserver, it's recommended to check with Citrix support as to when they are going to update their drivers. Citrix takes our open source Linux drivers and ports the NIC drivers to Citrix XenServer environment and validates the drivers.