Xilinx KCU105 Manuel utilisateur

KCU105 PCI Express

Streaming Data Plane

TRD User Guide

KUCon-TRD03

Vivado Design Suite

UG920 (v2017.1) June 01, 2017

PCIe Streaming Data Plane TRD www.xilinx.com 2

UG920 (v2017.1) June 01, 2017

Revision History

The following table shows the revision history for this document.

Date Version Revision

06/01/2017 2017.1 Released with Vivado Design Suite 2017.1. Updated Figure 3-2 through Figure 3-5

and Figure 4-1 through Figure 4-7 (no changes to text).

01/20/2017 2016.4 Released with Vivado Design Suite 2016.4 without changes from the previous

version.

10/05/2016 2016.3 Released with Vivado Design Suite 2016.3 without changes from the previous

version.

06/08/2016 2016.2 Released with Vivado Design Suite 2016.2 without changes from the previous

version.

04/14/2016 2016.1 Released with Vivado Design Suite 2016.1 without changes from the previous

version.

11/24/2015 2015.4 Released with Vivado Design Suite 2015.4 without changes from the previous

version.

10/05/2015 2015.3 Released with Vivado Design Suite 2015.3 with minor textual edits.

06/22/2015 2015.2 Updated for Vivado Design Suite 2015.2. Figure 3-1 and Figure 3-2 were replaced.

Figure 3-13 through Figure 3-15 were updated. In Hardware SGL Prepare Block,

page 49, element buffer size changed from 512 bytes to 4096 bytes. Figure 5-14 and

Figure 5-25 were updated.

05/13/2015 2015.1 Updated for Vivado Design Suite 2015.1. The TRD ZIP file changed to

rdf0307-kcu105-trd03-2015-1.zip. Updated Information about resource

utilization for the base design and the user extension design in Table 1-1 and

Table 1-2. Added details about Windows 7 driver support, setup, and test of the

reference design, updating these sections: Features, Computers, and Appendix A,

Directory Structure. Added section Install TRD Drivers on the Host Computer

(Windows 7) to Chapter 2, Setup and added Appendix B, Recommended Practices

and Troubleshooting in Windows. Removed QuestaSim/ModelSim Simulator

information, because QuestaSim simulation is not supported in Vivado tool release

2015.1. Updated many figures and replaced Figure 1-1, Figure 1-2, and Figure 5-12.

Updated XDMA Driver Stack and Design in Chapter 5. In Table C-3, traffic generator

was changed to traffic checker. Added Appendix E, APIs Provided by the XDMA Driver

in Windows.

02/26/2015 2014.4.1 Initial Xilinx release.

Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 3

UG920 (v2017.1) June 01, 2017

Table of Contents

Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Chapter 1: Introduction

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Resource Utilization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Chapter 2: Setup

Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Preliminary Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Chapter 3: Bringing Up the Design

Set the Host System to Boot from the LiveDVD (Linux) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Configure the FPGA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Run the Design on the Host Computer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Test the Reference Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Remove Drivers from the Host Computer

(Windows Only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Chapter 4: Implementing and Simulating the Design

Implementing the Base Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Implementing the User Extension Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Simulating the Designs Using Vivado Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Chapter 5: Targeted Reference Design Details and Modifications

Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Data Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Reference Design Modifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 4

UG920 (v2017.1) June 01, 2017

Appendix A: Directory Structure

Appendix B: Recommended Practices and Troubleshooting in Windows

Recommended Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Appendix C: Register Space

Generator and Checker Configuration Registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Ethernet MAC Registers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Appendix D: APIs Provided by the XDMA Driver in Linux

Appendix E: APIs Provided by the XDMA Driver in Windows

Appendix F: Network Performance

Private Network Setup and Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Appendix G: Additional Resources and Legal Notices

Xilinx Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

Solution Centers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

Please Read: Important Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 5

UG920 (v2017.1) June 01, 2017

Chapter 1

Introduction

This document describes the features and functions of the PCI Express® Streaming Data

Plane targeted reference design (TRD). The TRD comprises a base design and a user

extension design. The user extension design adds custom logic on top of the base design.

The pre-built user extension design in this TRD adds an Ethernet application.

Overview

The TRD targets the Kintex® UltraScale™ XCKU040-2FFVA1156E FPGA running on the

KCU105 evaluation board and provides a platform for data transfer between the host

machine and the FPGA. The top-level block diagram of the TRD base design is shown in

Figure 1-1.

The TRD uses an integrated Endpoint block for PCI Express (PCIe®) in a x8 Gen2

configuration along with an Expresso DMA Bridge Core from Northwest Logic [Ref 1] for

high performance data transfers between host system memory and the Endpoint (the FPGA

on the KCU105 board).

The DMA bridge core (DMA block) provides protocol conversion between PCIe transaction

layer packets (TLPs) and AXI transactions. The core’s hardware scatter gather list (SGL) DMA

interface provides buffers management at the Endpoint to enable the streaming interface.

X-Ref T arget - Fig ure 1-1

Figure 1-1: KCU105 PCI Express Streaming Data Plane Base Design

3URFHVVRU

5RRW

&RPSOH[

'0$

'ULYHU

3&,H

[*HQ

/LQN

ELWVDW

0+]

User

App 3&,H'0$

XCKU040-2FFVA1156E FPGA

KCU105 Evaluation Board

+RVW&RPSXWHU

+:6*/3UHSDUH

PPVVPP

$;,/LWH6ODYHV

$;,

,QWHUFRQQHFW

+DUGZDUHPRGXOHV 6RIWZDUHUXQQLQJRQKRVWFRPSXWHU &RPSRQHQWRQKRVWFRPSXWHU

*(1

&+.

00ELWV

DW0+]

6WUHDPELWV

DW0+]

UG920_c1_01_040615

Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 6

UG920 (v2017.1) June 01, 2017

Chapter 1: Introduction

The downstream AXI4-Lite slaves include a power monitor module, user space registers,

and an AXI performance monitor.

In the system to card direction (S2C), the DMA block moves data from host memory to the

FPGA through the integrated Endpoint block and then makes the data available at the AXI

streaming interface. In the card to system (C2S) direction, theDMA block uses the data

available on the AXI streaming interface and writes to host system memory through the

integrated Endpoint block.

The base design uses a simple generator/checker (GEN/CHK) block as a data

provider/consumer on the AXI streaming interface. This base platform can be extended to

a variety of applications such as Ethernet, and Crypto Engine.

The user extension design (shown in Figure 1-2) provides an example of a dual 10GbE

network interface card. The Ethernet frames received from the host system are directed to

respective Ethernet ports for transmission, and incoming frames from Ethernet are directed

to the host after performing an address filtering operation.

The designs delivered as part of this TRD use Vivado® IP integrator to build the system. IP

Integrator provides intelligent IP integration in a graphical, Tcl-based,

correct-by-construction IP and system-centric design development flow. For further details

on IPI, see Vivado Design Suite Tutorial: Designing IP Subsystems Using IP Integrator (UG995)

[Ref 2].

X-Ref Target - Figure 1-2

Figure 1-2: KCU105 PCI Express Streaming Data Plane User Extension Design

UG920_c1_02_040615

352&(6625

Root

Complex

'0$

'ULYHU

3&,H

[*HQ

/LQN

128 bits at

250 MHz

User

App

3&,H

'0$

;&.8))9$()3*$

.&8(YDOXDWLRQ%RDUG

+RVW&RPSXWHU

$;,/LWH

6ODYHV

$;,

,QWHUFRQQHFW

+DUGZDUHPRGXOHV 6RIWZDUHUXQQLQJRQKRVWFRPSXWHU &RPSRQHQWRQKRVWFRPSXWHU

MM - 128 bits

at 250 MHz

Stream - 64 bits

at 156.25 MHz

User Extension logic

*

0$&

*

0$&

Depending on the user application, the two modules (red colored arrows) need to be changed.

All other blocks in the hardware design and software components can be reused.

+:6*/3UHSDUH

PPVVPP

Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 7

UG920 (v2017.1) June 01, 2017

Chapter 1: Introduction

Features

The TRD includes these features:

•Hardware

°Integrated Endpoint block for PCI Express

- 8 lanes, each operating at 5 GT/s (gigatransfers per second) per lane per

direction

- 128-bit at 250 MHz

°DMA bridge core

- Scatter gather enabled

- 4 channels: 2 channels are used as S2C and 2 as C2S

- Support for an AXI3 interface

- Two ingress and two egress translation regions supported

Note: The IP can support a higher number of translation regions. The netlist used here is

built to support only two such regions. Contact Northwest Logic for further customization of

IP [Ref 1].

°SGL DMA interface block

- Queue management of channels: Destination queue for S2C and source queue

for C2S

- AXI memory map (MM) to AXI-stream interface conversion and AXI-stream to

MM conversion

- 128-bit at 250 MHz rate operation

- SGL submission block interface between the DMA bridge core and the SGL

preparation block for SGL element submission to DMA channels on a

round-robin basis across channels

- Traffic generator (packets on AXI4-stream interface) and checker block

operating at a 128-bit, 250 MHz rate

- AXI performance monitor capturing AXI4-stream interface throughput numbers

- Two 10G Ethernet MAC, PCS PMA blocks operating at 64-bit, 156.25 MHz

•Software

°64-bit Linux kernel space drivers for DMA and a raw data driver

°64-bit Windows 7 drivers for DMA and a raw data driver

°User space application

°Control and monitoring graphical user interface (GUI)

Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 8

UG920 (v2017.1) June 01, 2017

Chapter 1: Introduction

Resource Utilization

Table 1-1 and Table 1-2 list the resources used by the TRD base and user extension designs

after synthesis. Place and route can alter these numbers based on placements and routing

paths. These numbers are to be used as a rough estimate of resource utilization. These

numbers might vary based on the version of the TRD and the tools used to regenerate the

design.

Table 1-1: Base Design Resource Utilization

Resource Type Available Used Usage (%)

CLB registers 484,800 79,865 16.63

CLB LUT 242,400 51,404 21.21

Block RAM 600 39.5 6.58

MMCME3_ADV 10 1 10

Global Clock Buffers 240 3 1.25

BUFG_GT 120 5 4.17

IOB 520 18 3.46

SYSMONE1 1 1 100

PCIE_3_1 3 1 33.33

GTHE3_CHANNEL 20 8 40

Table 1-2: User Extension Design Resource Utilization

Resource Type Available Used Usage (%)

CLB registers 484,800 101,313 20.90

CLB LUT 242,400 64,884 26.77

Block RAM 600 80 13.33

MMCME3_ADV 10 1 10

Global Clock Buffers 240 3 1.25

BUFG_GT 120 12 10.00

IOB 520 23 4.42

SYSMONE1 1 1 100

PCIE_3_1 3 1 33.33

GTHE3_CHANNEL201050

Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 9

UG920 (v2017.1) June 01, 2017

Chapter 2

Setup

This chapter identifies the hardware and software requirements, and the preliminary setup

procedures required prior to bringing up the targeted reference design.

Requirements

Hardware

Board and Peripherals

• KCU105 board with the Kintex® UltraScale™ XCKU040-2FFVA1156E FPGA

• USB cable, standard-A plug to micro-B plug (Digilent cable)

• Power supply: 100 VAC–240 VAC input, 12 VDC 5.0A output

• ATX power supply

• ATX power supply adapter

Computers

A control computer is required to run the Vivado® Design Suite and configure the on-board

FPGA. This can be a laptop or desktop computer with any operating system supported by

Vivado tools, such as Redhat Linux or Microsoft® Windows 7.

The reference design test configuration requires a host computer comprised of a chassis

containing a motherboard with a PCI Express slot, monitor, keyboard, and mouse. A DVD

drive is also required if a Linux operating system is used. If a Windows 7 operating system

is used, the 64-bit Windows 7 OS and the Java SE Development Kit 7 must be installed.

Send Feedback

PCIe Streaming Data Plane TRD www.xilinx.com 10

UG920 (v2017.1) June 01, 2017

Chapter 2: Setup

Software

Vivado Design Suite 2017.1 is required. The Fedora 20 LiveDVD, on which the TRD software

and GUI run, is only required if a Linux operating system is used.

Preliminary Setup

Complete these tasks before bringing up the design.

Install the Vivado Design Suite

Install Vivado Design Suite 2017.1 on the control computer. Follow the installation

instructions provided in the Vivado Design Suite User Guide Release Notes, Installation, and

Licensing (UG973) [Ref 3].

Download the Targeted Reference Design Files

1. Download rdf0307-kcu105-trd03-2017-1.zip from the Xilinx Kintex UltraScale

FPGA KCU105 Evaluation Kit - Documentation & Designs website. This ZIP file contains

the hardware design, software drivers, and application GUI executables.

2. Extract the contents of the file to a working directory.

3. The extracted contents are located at <working_dir>/kcu105_axis_dataplane.

The TRD directory structure is described in Appendix A, Directory Structure.

Install TRD Drivers on the Host Computer (Windows 7)

Note: This section provides steps to install KUCon-TRD drivers and is only applicable to a host

computer running Windows 7 64-bit OS. If running Linux, proceed to Set DIP Switches, page 12.

Disable Driver Signature Enforcement

Note: Windows only allows drivers with valid signatures obtained from trusted certificate

authorities to load in a Windows 7 64-bit OS computer. Windows drivers provided for this reference

design do not have a valid signature. Therefore, you have to disable Driver Signature Enforcement on

the host computer as follows:

1. Power up the host system.

2. Press F8 to go to the Advanced Boot Options menu.

3. Select the Disable Driver Signature Enforcement option shown in Figure 2-1, and

press Enter.

Send Feedback

Autres manuels pour KCU105

Table des matières

Autres manuels Xilinx Carte mère

Xilinx

Xilinx ZCU1285 Manuel utilisateur

Xilinx

Xilinx KC705 Manuel utilisateur

Xilinx

Xilinx DK-V7-VC709-G Manuel utilisateur

Xilinx

Xilinx Virtex-II Pro ML300 Manuel utilisateur

Xilinx

Xilinx Zynq-7000 Instructions d'installation et d'utilisation

Xilinx

Xilinx EZTag Manuel utilisateur

Xilinx

Xilinx ML605 Manuel utilisateur

Xilinx

Xilinx AMS101 Manuel utilisateur

Xilinx

Xilinx ML501 Manuel utilisateur

Xilinx

Xilinx ZCU102 Manuel utilisateur

Xilinx

Xilinx Zynq ZC702-G Manuel utilisateur

Xilinx

Xilinx SP605 Manuel d'installation

Xilinx

Xilinx ML310 Manuel utilisateur

Xilinx

Xilinx Spartan 6 FPGA Manuel utilisateur

Xilinx

Xilinx FMC XM105 Manuel utilisateur

Xilinx

Xilinx VCU118 Manuel utilisateur

Xilinx

Xilinx Spartan-6 FPGA Series Manuel utilisateur

Xilinx

Xilinx ML50 Series Manuel utilisateur

Xilinx

Xilinx Zynq UltraScale+ ZCU104 Manuel utilisateur

Xilinx

Xilinx EK-U1-VCU128-G-J Manuel utilisateur

Xilinx

Xilinx VCK190 Series Manuel utilisateur

Xilinx

Xilinx ZC702 Si570 Guide d'installation

Xilinx

Xilinx Kintex-7 FPGA KC705 Manuel utilisateur

Xilinx

Xilinx SP605 Manuel utilisateur

Manuels Carte mère populaires d'autres marques

DFI

DFI SD170 Manuel utilisateur

ASROCK

ASROCK ROMED8-2T Manuel utilisateur

Asus

Asus P2L97A Manuel utilisateur

Winmate

Winmate IC70 Manuel utilisateur

Asus

Asus MES-B Manuel utilisateur

Grantech

Grantech SYS8F377VGGA-3OMN Manuel utilisateur

Telit Wireless Solutions

Telit Wireless Solutions SL869-3DR Manuel utilisateur

Gigabyte

Gigabyte GA-9IVDT Manuel utilisateur

Texas Instruments

Texas Instruments ADS8372EVM Manuel utilisateur

Commell

Commell MS-C73 Manuel utilisateur

IBT Technologies

IBT Technologies MB860 Manuel utilisateur

Nvidia

Nvidia TEGRA DG-04927-001_V01 Manuel utilisateur

Biostar

Biostar M7VIT Bravo Instructions d'utilisation et d'installation

Intel

Intel D33217CK Manuel utilisateur

Asus

Asus F1A55-M LX PLUS R2.0 Manuel utilisateur

EPC

EPC EPC9147C Manuel utilisateur

Asus

Asus EX-B760M-V5 D4 Manuel utilisateur

EPOX

EPOX MU-8KHA0010 Manuel utilisateur