Nvidia Mellanox ConnectX-5 Manuel utilisateur

Exported on Sep/19/2022 09:36 AM



NVIDIA ConnectX-5 InfiniBand/VPI Adapter

Cards User Manual



2
Table of Contents
Introduction..................................................................................... 6
Product Overview ....................................................................................6
Features and Benefits ...............................................................................7
Operating Systems/Distributions ..................................................................9
Connectivity ..........................................................................................9
Interfaces ......................................................................................10
InfiniBand Interface ............................................................................... 10
Ethernet QSFP28 Interface ....................................................................... 10
PCI Express Interface.............................................................................. 10
LED Interface ....................................................................................... 10
Thermal Sensors.................................................................................... 11
Hardware Installation ........................................................................12
Safety Warnings .................................................................................... 12
Installation Procedure Overview................................................................. 12
System Requirements ............................................................................. 12
Hardware Requirements ...................................................................... 12
Airflow Requirements ......................................................................... 13
Software Requirements ....................................................................... 14
Unpacking the Product ............................................................................ 14
Bracket Replacement Instructions .............................................................. 15
Installation Instructions........................................................................... 15
Cables and Modules................................................................................ 17
Identifying the Card in Your System............................................................. 18
Uninstalling the Card.............................................................................. 19
Safety Precautions ............................................................................. 19
Card Removal................................................................................... 19
Driver Installation.............................................................................21
Linux Driver Installation .......................................................................... 21
Prerequisites.................................................................................... 21
Downloading NVIDIA OFED .................................................................... 21
Installing NVIDIA OFED ........................................................................ 22
Installing MLNX_OFED Using YUM............................................................ 27

3
Installing MLNX_OFED Using apt-get Tool .................................................. 30
Updating Firmware After Installation....................................................... 31
UEFI Secure Boot............................................................................... 32
Performance Tuning ........................................................................... 34
Windows Driver Installation ...................................................................... 34
Software Requirements ....................................................................... 34
Downloading NVIDIA WinOF-2 Driver ........................................................ 35
Installing NVIDIA WinOF-2 Driver ............................................................ 35
Uninstalling NVIDIA WinOF-2 Driver ......................................................... 42
Extracting Files Without Running Installation ............................................. 43
Firmware Upgrade ............................................................................. 45
Troubleshooting ...............................................................................46
General Tips ........................................................................................ 46
Gathering Information - Linux Environment ................................................... 47
Gathering Information - Windows Environment ............................................... 47
Updating Adapter Firmware .................................................................48
Specifications..................................................................................49
MCX555A-ECAT Specifications .................................................................... 49
MCX556A-EC[A/U]T Specifications .............................................................. 50
MCX556A-ECAT/MCX556A-ECUT Specifications ................................................ 50
MCX556A-EDAT Specifications .................................................................... 51
Airflow Specifications ............................................................................. 53
LED Specifications ................................................................................. 53
Board Mechanical Drawings and Dimensions................................................... 53
Adapter Card Mechanical Drawing (Component Side).................................... 54
Bracket Mechanical Drawing ................................................................. 54
Finding the GUID/MAC and Serial Number on the Adapter Card.......................56
MCX555A-ECAT board label example:........................................................... 56
MCX556A-ECAT board label example:........................................................... 56
MCX556A-ECUT board label example: .......................................................... 57
MCX556A-EDAT board label example:........................................................... 57
Document Revision History ..................................................................58


•

About This Manual

This User Manual describes NVIDIA® ConnectX®-5 and ConnectX®-5Ex VPI Single and DualQSFP28

port PCI Express x16 adapter cards. It provides details as to the interfaces of the

board,specifications, required software and firmware for operating the board, and relevant

documentation.

Ordering Part Numbers

The table below provides the ordering part numbers (OPN) for the available ConnectX-5 VPI adapter

cards.

IC in Use OPN Marketing Description

ConnectX®-

MCX555A-

ECAT

ConnectX®-5 VPI adapter card, EDR IB (100Gb/s) and 100GbE, single-port QSFP28,

PCIe 3.0 x16, tall bracket

MCX556A-

ECAT

ConnectX®-5 VPI adapter card, EDR IB (100Gb/s) and 100GbE, dual-port QSFP28,

PCIe 3.0 x16, tall bracket

MCX556A-

ECUT

ConnectX®-5 VPI adapter card, EDR IB (100Gb/s) and 100GbE, dual-port QSFP28,

PCIe3.0 x16, UEFI enabled, tall bracket

ConnectX®-

5 Ex

MCX556A-

EDAT

ConnectX®-5 Ex VPI adapter card, EDR IB (100Gb/s) and 100GbE, dual-port

QSFP28, PCIe 4.0 x16, tall bracket

Intended Audience

This manual is intended for the installer and user of these cards.The manual assumes basic

familiarity with InfiniBand and Ethernet network and architecturespecifications.

Technical Support

Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough

the following methods:

URL:https://www.nvidia.com> Support

E-mail:[email protected]

Customers who purchased NVIDIA Global Support Services, please see your contract for details

regarding Technical Support.

Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek

assistance through their reseller.

Related Documentation

MLNX_OFED for Linux User Manual and

Release Notes

User Manual describing OFED features, performance, band

diagnostic, tools content, and configuration. SeeMLNX_OFED for

Linux Documentation.

WinOF-2 for WindowsUser Manualand

Release Notes

User Manual describing WinOF-2 features, performance, Ethernet

diagnostic, tools content, and configuration. SeeWinOF-2 for

Windows Documentation.

NVIDIA VMware for Ethernet User Manual User Manual and release notes describing the various components

of the NVIDIA ConnectX® NATIVE ESXi stack. SeeVMware® ESXi

Drivers Documentation.

NVIDIA Firmware Utility (mlxup) User

Manual and Release Notes

NVIDIA firmware update and query utility used to update the

firmware. Refer toFirmware Utility (mlxup) Documentation.

NVIDIA Firmware Tools (MFT) User Manual

User Manual describing the set of MFT firmware management

tools for a single node. SeeMFT User Manual.

InfiniBand Architecture Specification

Release 1.2.1, Vol 2 - Release 1.3

InfiniBand Specifications

IEEE Std 802.3 Specification IEEE Ethernet Specifications

PCI Express Specifications Industry Standard PCI Express Base and Card Electromechanical

Specifications. Refer toPCI-SIG Specifications.

LinkX Interconnect Solutions

LinkX InfiniBand cables and transceivers are designed to

maximize the performance of High-Performance Computing

networks, requiring high-bandwidth, low-latency connections

between compute nodes and switch nodes.NVIDIA offers one of

the industry’s broadest portfolio ofQDR/FDR10 (40Gb/s), FDR

(56Gb/s), EDR/HDR100 (100Gb/s), HDR (200Gb/s)and NDR

(400Gb/s) cables, includingDirect Attach Copper cables (DACs),

copper splitter cables, Active Optical Cables (AOCs) and

transceivers in a wide range of lengths from 0.5m to 10km.In

addition to meeting IBTA standards, NVIDIAtests every product in

an end-to-end environment ensuring a Bit Error Rate of less than

1E-15.Read more atLinkX Cables and Transceivers.

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega

Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. IB is used in this documentto

mean InfiniBand. In this document PCIe is used to mean PCI Express.

Introduction

Product Overview

This is the User Guide for VPI adapter cards based on the ConnectX®-5integrated circuit device.

These adapters connectivity provide the highest performing and mostflexible interconnect solution

for PCI Express Gen 3.0/4.0 servers used in Enterprise Data Centers,High-Performance Computing,

and Embedded environments.

The following table provides the ordering part number, port speed, number of ports, and PCI Express

speed. Each adapter comes with two bracket heights - short and tall.

ConnectX-5 Ex InfiniBand/VPI Adapter Cards

Model ConnectX-5 Ex InfiniBand/VPI Adapter Cards

Part Number MCX556A-EDAT

Data Rate

InfiniBand: SDR/DDR/QDR/FDR/EDR

Ethernet: 10/25/40/50/100 Gb/s

Network Connector Type

Dual-port QSFP28

PCI Express Connectors

PCIe Gen 3.0/4.0 x16 (a), (b)

SERDES @ 8.0GT/s / 16.0GT/s

Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile

RoHS RoHS Compliant

Adapter IC Part Number

MT28808A0-FCCF-EV

Device ID (decimal) 4121 for Physical Function (PF) and 4122 for Virtual Function (VF)

a. PCIe 4.0 x16 bus can supply a maximum bandwidth of 256Gb/s (=16 *16GT/s, including overhead),

and can support 200Gb/s when both network ports of the card run at 100Gb/s.

b.This card has been tested and certified with PCIe 3.0 servers. PCIe 4.0 interface will be tested

when servers with Gen 4.0 support become available.

ConnectX-5 InfiniBand/VPI Adapter Cards

Model ConnectX-5 InfiniBand/VPI Adapter Cards

Part Number

MCX555A-ECAT MCX556A-ECAT MCX556A-ECUT

Data Rate InfiniBand: SDR/DDR/QDR/FDR/EDR

Ethernet: 10/25/40/50/100 Gb/s

Network Connector Type Single-port QSFP28 Dual-port QSFP28 Dual-port QSFP28

PCI Express Connectors PCIe Gen 3.0 x16; SerDes @ 8.0GT/s

Model ConnectX-5 InfiniBand/VPI Adapter Cards

Dimensions 2.71 in. x 5.6 in. (68.90mm x 142.24 mm) – low profile

RoHS RoHS Compliant

Adapter IC Part Number MT27808A0-FCCF-EV

UEFI - - Enabled

Device ID (decimal) 4119 for Physical Function (PF) and 4120 for Virtual Function (VF)

For more detailed information seeSpecifications.

Features and Benefits

Feature Description

PCI

Express

(PCIe)

Uses PCIe Gen 3.0 (8GT/s) and Gen 4.0 (16GT/s) through an x16 edge connector. Gen 1.1 and 2.0

compatible.

EDR

InfiniBand

A standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of 25.78125Gb/s

with a 64b/66b encoding, resulting in an effective bandwidth of 100Gb/s.

100Gb/s

Virtual

ProtocolIn

terconnect

(VPI)Adap

ter

ConnectX-5 offers the highest throughput VPI adapter, supporting EDR 100Gb/s InfiniBand and

100Gb/s Ethernet and enabling any standard networking, clustering, or storage to operate

seamlessly over any converged network leveraging a consolidated software stack.

InfiniBand

Architectu

Specificati

on v1.3

compliant

ConnectX-5 delivers low latency, high bandwidth, and computing efficiency for performance-

driven server and storage clustering applications. ConnectX-5 is InfiniBand Architecture

Specification v1.3 compliant.

This section describes hardware features and capabilities. Please refer to the relevant

driver and/or firmware release notes for feature availability.



•

Feature Description

Up to 100

Gigabit

Ethernet

NVIDIA adapters comply with the following IEEE 802.3 standards:

100GbE/ 50GbE / 40GbE / 25GbE / 10GbE / 1GbE

IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet

IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes

IEEE 802.3ba 40 Gigabit Ethernet

IEEE 802.3ae 10 Gigabit Ethernet

IEEE 802.3ap based auto-negotiation and KR startup

Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)

IEEE 802.3ad, 802.1AX Link Aggregation

IEEE 802.1Q, 802.1P VLAN tags and priority

IEEE 802.1Qau (QCN)

Congestion Notification

IEEE 802.1Qaz (ETS)

IEEE 802.1Qbb (PFC)

IEEE 802.1Qbg

IEEE 1588v2

Jumbo frame support (9.6KB)

Memory SPI Quad - includes 128Mbit SPI Quad Flash device (W25Q128FVSIG device by ST

Microelectronics).

FRU EEPROM - Stores the parameters and personality of the card. The EEPROM capacity is

128Kbit. FRU I2C address is (0x50) and is accessible through the PCIe SMBus. Note:

Address 0x58 is reserved.)

Overlay

Networks

In order to better scale their networks, data center operators often create overlay networks that

carry traffic from individual virtual machines over logical tunnels in encapsulated formats such

as NVGRE and VXLAN. While this solves network scalability issues, it hides the TCP packet from

the hardware offloading engines, placing higher loads on the host CPU. ConnectX-5 effectively

addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that

encapsulate and de-capsulate the overlay protocol.

RDMA and

RDMA over

Converged

Ethernet

(RoCE)

ConnectX-5, utilizing IBTA RDMA (Remote Data Memory Access) and RoCE (RDMA over Converged

Ethernet) technology, delivers low-latency and high performance over Band and Ethernet

networks. Leveraging data center bridging (DCB) capabilities as well as ConnectX-5 advanced

congestion control hardware mechanisms, RoCE provides efficient low-latency RDMA services

over Layer 2 and Layer 3 networks.

NVIDIA

PeerDirect

™

PeerDirect™ communication provides high-efficiency RDMA access by eliminating unnecessary

internal data copies between components on the PCIe bus (for example, from GPU to CPU), and

therefore significantly reduces application run time. ConnectX-5 advanced acceleration

technology enables higher cluster efficiency and scalability to tens of thousands of nodes.

CPU

Offload

Adapter functionality enabling reduced CPU overhead allowing more available CPU for

computation tasks.

Open

VSwitch

(OVS)

offload

using

ASAP2

• Flexible match-action flow tables

• Tunneling encapsulation/decapsulation

Quality of

Service

(QoS)

Support for port-based Quality of Service enabling various application requirements for latency

and SLA.

•

Feature Description

Hardware-

based I/O

Virtualizat

ion

ConnectX-5 provides dedicated adapter resources and guaranteed isolation and protection for

virtual machines within the server.

Storage

Accelerati

A consolidated compute and storage network achieves significant cost-performance advantages

over multi-fabric networks. Standard block and file access protocols can leverage InfiniBand

RDMA for high-performance storage access.

NVMe over Fabric offloads for the target machine

Erasure Coding

T10-DIF Signature Handover

SR-IOV ConnectX-5 SR-IOV technology provides dedicated adapter resources and guaranteed isolation

and protection for virtual machines (VM) within the server.

High-

Performan

Accelerati

ons

Tag Matching and Rendezvous Offloads

Adaptive Routing on Reliable Transport

Burst Buffer Offloads for Background Checkpointing

UEFI

UEFI is a standard firmware interface designed to replace BIOS. NVIDIA UEFI Network driver

allows boot over network via PXE (Preboot eXecution Environment). This network driver allows

remote boot over InfiniBand or Ethernet, or Boot over iSCSI (Bo-iSCSI) in UEFI mode, and also

supports the SecureBoot standard. The UEFI Network driver allows IT managers the flexibility to

deploy servers with a single adapter card into InfiniBand or Ethernet networks while also

enabling booting from LAN or remote storage targets. In addition to boot capabilities, NVIDIA

UEFI Network driver provides firmware management and diagnostic protocols compliant with the

UEFI specification.

For further information, refer to theNVIDIA PreBoot Drivers User Manual.

Supported in MCX556A-ECUT.

Operating Systems/Distributions

RHEL/CentOS

Windows

FreeBSD

VMware

OpenFabrics Enterprise Distribution (OFED)

OpenFabrics Windows Distribution (WinOF-2)

Connectivity

Interoperable with 1/10/25/40/50/100 Gb/s Ethernet switches

Passive copper cable with ESD protection

Powered connectors for optical and active cable support

•

Interfaces

InfiniBand Interface

The network ports of the ConnectX®-5 adapter cards are compliant with the InfiniBand

ArchitectureSpecification, Release 1.3. InfiniBand traffic is transmitted through the cards' QSFP28

connectors.

Ethernet QSFP28 Interface

The network ports of the ConnectX®-5 adapter card are compliant with the IEEE 802.3

Ethernetstandards listed in Features and Benefits.Ethernet traffic is transmitted through thecards'

QSFP28 connectors.

PCI Express Interface

The ConnectX®-5 adapter card supports PCI Express Gen 3.0/4.0 (1.1 and 2.0 compatible)through

an x16 edge connector. The device can be either a master initiating the PCI Express busoperations,

or a slave responding to PCI bus operations.The following lists PCIe interface features:

PCIe Gen4.0 and 3.0 compliant, 2.0 and 1.1 compatible

2.5, 5.0, or 8.0, or 16.0 GT/s link rate x16

Auto-negotiates to x16, x8, x4, x2, or x1

Support for MSI/MSI-X mechanisms

LED Interface

There is one bi-color (yellow/green) I/O LED per port to indicate link status located on the adapter

card.

Physical and Logical Link Indications - Ethernet Mode:

State LED Behavior

Beacon command for locating the card Blinks yellow at 1Hz

Error Blinks yellow at 4Hz until error is resolved. The reason for

the error may be:

Failure of I2C access to the port

Over-current conditions of the port

Link Activity Blinks green reflecting the link speed

Link Up Constant green light

Autres manuels pour Mellanox ConnectX-5

Ce manuel convient aux modèles suivants

Table des matières

Autres manuels Nvidia Matériel informatique

Nvidia

Nvidia ConnectX-7 Manuel utilisateur

Nvidia

Nvidia SN2700 Manuel utilisateur

Nvidia

Nvidia TESLA C2050 Manuel utilisateur

Nvidia

Nvidia Jetson AGX Xavier Series Mode d’emploi

Nvidia

Nvidia DRIVE AGX Manuel utilisateur

Nvidia

Nvidia ConnectX-7 Manuel utilisateur

Nvidia

Nvidia JETSON AGX XAVIER Manuel utilisateur

Nvidia

Nvidia GeForce2 Ti SP6600Ti Manuel utilisateur

Nvidia

Nvidia Jetson Orin NX Mode d’emploi

Nvidia

Nvidia Mellanox ConnectX-6 MCX654106A-ECAT Manuel utilisateur

Nvidia

Nvidia TESLA K10 Manuel utilisateur

Nvidia

Nvidia Computer Manuel utilisateur

Nvidia

Nvidia MSTFLINT Manuel de l'opérateur

Nvidia

Nvidia DGX H100 Manuel utilisateur

Nvidia

Nvidia A800 40GB Active Manuel utilisateur

Nvidia

Nvidia Tesla C870 Manuel utilisateur

Nvidia

Nvidia TESLA K20 Manuel utilisateur

Nvidia

Nvidia Clara AGX Manuel utilisateur

Nvidia

Nvidia Quadro M6000 Sync Manuel utilisateur

Nvidia

Nvidia TESLA C2050 Manuel utilisateur

Nvidia

Nvidia TESLA K20 Manuel utilisateur

Nvidia

Nvidia Mellanox ConnectX-5 Manuel utilisateur

Nvidia

Nvidia BlueField-2 BF2500 Manuel utilisateur

Nvidia

Nvidia GeForce GTX 980 Manuel utilisateur

Manuels Matériel informatique populaires d'autres marques

EMC2

EMC2 VNX Series Manuel du propriétaire

Panasonic

Panasonic DV0PM20105 Manuel utilisateur

Mitsubishi Electric

Mitsubishi Electric Q81BD-J61BT11 Manuel utilisateur

Gigabyte

Gigabyte B660M DS3H AX DDR4 Manuel utilisateur

Raidon

Raidon iT2300 Manuel utilisateur

National Instruments

National Instruments PXI-8186 Manuel utilisateur

Hauppauge

Hauppauge WinTV-HVR-2250 Manuel utilisateur

Kontron

Kontron CP6002 Manuel utilisateur

Manhattan

Manhattan 702843 Manuel utilisateur

Aphex

Aphex 230 Manuel utilisateur

AMD

AMD Phenom II Manuel d'exploitation

Intel

Intel NUC 12 Pro Manuel utilisateur

dbx

dbx Zone Pro 1260 Manuel utilisateur

Galaxy

Galaxy GHDX2-2430S-24F4D Manuel utilisateur

Intel

Intel AXXRMFBU4 Manuel utilisateur

Kontron

Kontron DIMM-PC/MD Manuel utilisateur

ST X-NUCLEO-SAFEA1B Mode d'emploi

STEINWAY LYNGDORF

STEINWAY LYNGDORF SP-1 Manuel utilisateur