Mellanox Technologies InfiniScale III Manuel utilisateur

Mellanox Technologies

Third party information brought

to you courtesy of Dell.

InfiniScale® III 16+8 Port 20Gb/s InfiniBand

Switch for Dell PowerEdge M1000e-series

Blade Enclosures User Manual

PN: M2401G

Rev 1.7

Rev 1.7 MellanoxTechnologies

Mellanox Technologies ConnectX, InfiniBlast, InfiniBridge, InfiniHost, InfiniRISC, InfiniScale,

and InfiniPCI are registered trademarks of Mellanox Technologies, Ltd. Virtual Protocol Intercon-

nect is a trademark of Mellanox Technologies, Ltd. Dell and the Dell logo are trademarks of Dell

Inc. All other marks and names mentioned herein may be trademarks of their respective companies.

Third party information brought to you courtesy of Dell.

InfiniScale® III 16+8 Port 20Gb/s InfiniBand Switch for Dell PowerEdge M1000e-series Blade

Enclosures User Manual

Document Number: 2948

Mellanox Technologies, Inc.

350 Oakmead Parkway

Sunnyvale, CA 94086

U.S.A.

www.Mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox Technologies Ltd

PO Box 586 Hermon Building

Yokneam 20692

Israel

Tel: +972-4-909-7200

Fax: +972-4-959-3245

NOTE:

THIS INFORMATION IS PROVIDED BY MELLANOX FOR INFORMATIONAL PURPOSES ONLY AND ANY

EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF

MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL

MELLANOX BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSE-

QUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR

SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON

ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLI-

GENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS HARDWARE, EVEN IF ADVISED

OF THE POSSIBILITY OF SUCH DAMAGE.

M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual

MellanoxTechnologies Rev 1.7

Contents

Contents 3

About this Manual 4

Revision History 5

Chapter 1 Overview 6

1.1 InfiniBand Connectors 7

1.2 Switch Status Lights 7

Chapter 2 Installation and Basic Operation 9

2.1 Installation Safety Warnings 10

2.2 Mechanical Installation 11

2.2.1 Removing an Old Switch From the Chassis 11

2.2.2 Removing a Blank Cover Plate From the Chassis 12

2.2.3 Installing the Mellanox M2401G Switch Into the Dell Chassis 12

2.3 Power Connections and Initial Power On 12

2.4 InfiniBand Cable Installation 13

Chapter 3 Cluster Management and Firmware 14

3.1 Network Management and Clustering Software 14

3.2 Updating Firmware 14

3.2.1 Instructions for Reprogramming Over the InfiniBand Network 14

Chapter 4 Troubleshooting 17

Appendix A Specifications 18

Appendix B Mechanical Drawing 19

Appendix C EMC Certification Statements 20

Appendix D InfiniBand Connector Pinout 22

Appendix E Avertissements de sécurité d’installation 24

Appendix F Installation - Sicherheitshinweise 26

Rev 1.7 MellanoxTechnologies

About this Manual

This manual describes the installation and basic use of the Mellanox M2401G 16+8 Port 20Gb/s

InfiniBand Switch for Dell PowerEdge M1000e-series Blade Enclosures.

Intended Audience

This manual is intended for users and system administrators responsible for installing and setting

up the switch platforms listed above.

The manual assumes familiarity with the InfiniBand® Architecture Specification.

Related Documentation

The documentation set accompanying the Mellanox M2401G includes the following:

Online Resources

• Mellanox Technologies Web pages: http://www.mellanox.com

• Dell Support Web pages: http://support.dell.com

Conventions

Throughout this manual, the name M2401G and the term switch are used to describe the 16+8 port

20Gb/s InfiniBand switch for the Dell M1000e, unless explicitly indicated otherwise.

Table 1 - Reference Documents

InfiniBand Architecture Specification Vol-

ume 1 Release 1.2.1 and Volume 2 release

1.2.1

InfiniBand architecture specification descriptions

Mellanox OFED Stack The Mellanox OFED stack provides a tool set to manage InfiniBand fabric clusters. The

stack with user’s manual can be downloaded at:

http://www.mellanox.com > Downloads > InfiniBand SW/Drivers,

Be sure to download the stack that matches your OS.

Mellanox Firmware Tools (MFT)

User’s Manual

Document # 2329

The MFT (Mellanox Firmware Tools) package is a set of firmware tools. The manual

supplied with this package provides an overview of the firmware its installation and

replacement. The MFT can be downloaded with its documentation at:

http://www.mellanox.com > Downloads > Firmware Tools

M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual

MellanoxTechnologies Rev 1.7

Revision History

Table 2 - Revision History Table

Date Revision Description

Mar. 2009 Rev 1.7 Added Dell flat black Logo

Dec. 2008 Rev 1.6 Minor fixes to Rev 1.5

Dec. 2008 Rev 1.5 Minor fixes to Rev 1.4

Added Instructions for Reprogramming Over the InfiniBand Network

Nov. 2008 Rev 1.4 Added Dell logo and Dell 3rd party info statement

Replaced Mellanox address

Fixed Links to new web site

Aug. 2008 Rev 1.3 Added Certs and Link to FW

Aug. 2008 Rev 1.2 Added appendix with safety warnings in French for Canada Certification

July 2008 Rev 1.1 Fixed Max Ambient Temperature

June 2008 Rev 1.0 First Draft

Rev 1.7 MellanoxTechnologies

1 Overview

The Mellanox M2401G 20Gb/s InfiniBand Switch Blade for Dell PowerEdge M1000e-series Blade

Enclosures provides a high bandwidth, low latency, low power fabric for Enterprise Data Center

and High-Performance Computing environments. Based on the third generation InfiniScale® III

InfiniBand switch device, the platform delivers an ideal balance between cost and performance.

When used in conjunction with ConnectX® InfiniBand dual port I/O cards clustered data bases,

parallelized applications and transactional services applications will achieve significant perfor-

mance improvements resulting in reduced completion time and lower cost per operation.

The switch platform comes pre-installed with all necessary firmware, and configured for standard

operation within an InfiniBand fabric, and requires an InfiniBand compliant Subnet Manager run-

ning from one of the hosts. All that is required for normal operation is to follow the usual precau-

tions for installation and connection from the switch to the HCAs or other switches. Once

connected, the Subnet Management software automatically configures and begins utilizing the

switch.

It is recommended that Mellanox OpenFabrics software package be installed on all nodes con-

nected to the M2401G. The software package provides a subnet manager and network management

tools as well as connectivity software for servers and storage, and is available on the Mellanox web

site. See “Cluster Management and Firmware” on page 14. for more information.

Basic installation, hot-swapping components and hardware maintenance is covered in “Installa-

tion and Basic Operation” on page 9.

The M2401G switch has a Hot Swap controller and a PSOC Management IC.

Figure 1: Switch Front Panel and Locking Mechanism

Locking Mechanism

Switch Status Lights

M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual

MellanoxTechnologies Rev 1.7

1.1 InfiniBand Connectors

InfiniBand connectivity has 8 microGiGa connectors through the front panel. The remaining 16

interfaces are through the AirMax Midplane Connector out the back of the switch. Figure 1 shows

the front 8 ports.

Each of the InfiniBand ports has two LEDs located next to the connector. The green LED, when lit,

indicates that a valid physical connection to the other system (switch or HCA port) exists. The yel-

low LED when lit, indicates that the Subnet Manager is running and a valid data link exists. The

yellow LED illuminates when the InfiniBand network is discovered over the physical link. A valid

data activity link without data transfer is designated by a constant yellow LED indication. A valid

data activity link with data transfer is designated by a blinking yellow LED indication. If the LEDs

are not active, either the physical link or the logical link (or both) connections have not been estab-

lished.

Figure 2: Physical and Logical Link Indication LEDs

1.2 Switch Status Lights

The switch Status lights indicate whether the switch is receiving power from the chassis, and the

state of the switch.

Figure 3: Indicator LEDs

LED Name Connection Status

Physical Link - Green Off – No Physical Link

ON – Physical Link

Data Activity - Yellow Blinking – indicates Data Transfer

Constant on – indicates Link exists

with no Data Transfer taking place

Off with green LED lit – indicates that

the Subnet Manager may not be

running

The I/O Module is on and

ready when both the blue and

green LEDs are lit.

This LED can be blue or

amber. A fault is indicated

when the amber LED is blink-

ing. See Table 3.

Rev 1.7 MellanoxTechnologies

Table 3 - IOM states and LED configurations:

LED Switch Status

Indication Green Status

OFF OFF OFF

ON OFF Boot in Progress

Switch not ready

Blinking Blue

ON BLINKING BLUE The CMC is identifying

the newly installed switch

ON ON BLUE Switch is on and operating Normally

ON ON or BLINKING

AMBER Fault in System

Self-diagnosed

OFF ON or BLINKING

AMBER Fault in System

CMC-detected

M3601Q 16+16 Port 40Gb/s InfiniBand Switch User Manual

MellanoxTechnologies Rev 1.7

2 Installation and Basic Operation

Installation and initialization of the switch platform are straightforward processes, requiring atten-

tion to the normal mechanical, power, and thermal precautions for rack-mounted equipment. The

switch platform does not require any programming or configuration to operate as a basic Infini-

Band switch and includes all of the necessary functionality to operate with external standard Infini-

Band Subnet Management software.

This section describes the installation process and basic operation of the switch platform. Please

first read the warnings sub-section carefully before carrying on with installation procedures.

Suitable electrical, mechanical and fire enclosure shall be provided by the end user.

Rev 1.7 MellanoxTechnologies

2.1 Installation Safety Warnings

1. Installation Instructions

2. Over-temperature

3. Stacking the Chassis

4. During Lightning - Electrical Hazard

5. Copper InfiniBand Cable Connecting/Disconnecting

6. Rack Mounting and Servicing

7. Equipment Installation

Read all installation instructions before connecting the equipment to the power source.

This equipment should not be operated in an area with an ambient temperature exceed-

ing the maximum recommended: °C (°F). Moreover, to guarantee proper air flow, allow

at least 8cm (3 inches) of clearance around the ventilation openings.

The chassis should not be stacked on any other equipment. If the chassis falls, it can

cause bodily injury and equipment damage.

During periods of lightning activity, do not work on the equipment or connect or discon-

nect cables.

Copper InfiniBand cables are heavy and not flexible, as such they should be carefully

attached to or detached from the connectors. Refer to the cable manufacturer for special

warnings and instructions.

When this product is mounted or serviced in a rack, special precautions must be taken to

ensure that the system remains stable. In general you should fill the rack with equipment

starting from the bottom to the top.

This equipment should be installed, replaced, or serviced only by trained and qualified

personnel.

Ce manuel convient aux modèles suivants

Table des matières

Autres manuels Mellanox Technologies Changer

Mellanox Technologies

Mellanox Technologies Mellanox SX1018 Manuel d'utilisation

Mellanox Technologies

Mellanox Technologies MIS5600Q-10DNC Manuel utilisateur

Mellanox Technologies

Mellanox Technologies Spectrum SN2740 Instructions d'utilisation

Mellanox Technologies

Mellanox Technologies switchx MSX1035B-1BRR Manuel utilisateur

Mellanox Technologies

Mellanox Technologies IS5025 Manuel utilisateur

Mellanox Technologies

Mellanox Technologies SX1710 Manuel utilisateur

Mellanox Technologies

Mellanox Technologies Grid Director 4036E Manuel utilisateur

Mellanox Technologies

Mellanox Technologies Mellanox SX1018 Manuel utilisateur

Mellanox Technologies

Mellanox Technologies MIS5024 Manuel utilisateur

Mellanox Technologies

Mellanox Technologies QM8700 Manuel utilisateur

Mellanox Technologies

Mellanox Technologies MSX6536-10R Manuel

Mellanox Technologies

Mellanox Technologies InfiniScale MTS3600Q-1BNC Manuel utilisateur

Mellanox Technologies

Mellanox Technologies SX6015 Manuel utilisateur

Mellanox Technologies

Mellanox Technologies Grid Director 4036 Manuel utilisateur

Mellanox Technologies

Mellanox Technologies SX6710 Instructions d'utilisation

Mellanox Technologies

Mellanox Technologies SX6720 Instructions d'utilisation

Mellanox Technologies

Mellanox Technologies Grid Director 4700 Manuel utilisateur

Mellanox Technologies

Mellanox Technologies Mellanox SX1018 Mode d’emploi

Mellanox Technologies

Mellanox Technologies Switch-IB Manuel utilisateur

Mellanox Technologies

Mellanox Technologies SwitchX-2 MSX6018F-1SFS Instructions d'utilisation

Mellanox Technologies

Mellanox Technologies 1G Switch Air Duct Rail Kit Manuel utilisateur

Mellanox Technologies

Mellanox Technologies InfiniScale III M2401G 20Gb/s InfiniBand Manuel utilisateur

Mellanox Technologies

Mellanox Technologies MIS5300 Manuel

Mellanox Technologies

Mellanox Technologies SwitchX-2 MSX6005F-1BFS Instructions d'utilisation

Manuels Changer populaires d'autres marques

URC

URC Total Control MFS-8 Manuel utilisateur

Azbil

Azbil HPQ-D11 Manuel utilisateur

Sony

Sony DFS-900M Manuel utilisateur

SICK

SICK i17S Manuel utilisateur

F&F

F&F SZR-277 Manuel utilisateur

Shure

Shure UAMS/BK Manuel utilisateur

Dräger

Dräger X-zone Switch Off Manuel utilisateur

Asco

Asco Joucomatic 349 Series Manuel utilisateur

BT Mini Hub Manuel utilisateur

D-Link

D-Link DES-1226G Manuel utilisateur

Johnson Controls

Johnson Controls FS80-C Manuel utilisateur

Alcatel-Lucent

Alcatel-Lucent 7450 ESS-12 Manuel utilisateur

MicroNet

MicroNet SP1200A Manuel utilisateur

Ruijie

Ruijie RG-PF2920 Series Manuel d'installation et d'exploitation

Altinex

Altinex DA1914SX Manuel utilisateur

MiLAN

MiLAN MIL-SM2401M Manuel utilisateur

RJM

RJM MASTERMIND PBC/6X Manuel utilisateur

Atlas

Atlas MMK-KVM8 Manuel utilisateur