[huzj_ecc]

Hi All:

I want to re-use the pcie_highperformancedesign example provided by the Arria GX Development Kit. Now I am confused with the PC software application altpcie_demo.exe.
I am trying to control FPGA to initiate dma read and write operation just like altpcie_demo does with my own PC software application but failed.

Firstly, I used Jungo Windriver to generate a pcie driver. With the API functions provided by the driver I can access(R/W) configure registers ,memory bar 1:0(the syncram) and bar2(dma control registers).

Secondly, I creat a Read Descriptor Table--Header+2 Descriptors and set data(Length,Ep mem addr, RC mem addr)for desciptors. The header has four dw(DW0,DW1,DW2,DW3). For DMA Read, I set DW0=0x00040002,DW1=0,DW2=addr of header,DW3=0x1. Then I write DW0 to Bar2+0x10,DW1 to Bar2+0x14,DW2 to Bar2+0x18,DW3 to Bar2+0x1c.

My first question: Where(mem addr) can I poll the RCLast value to indicate the completion of DMA read?

Thirdly, I want to transfer the DMA Read data back to PC. I creat a Write Descriptor Table--Header+2 Descriptors.In each descriptor, I set PC mem addr for write back data and addr of EP mem correctly .The header has four dw(DW0-DW3). For DMA Write, I set DW0=0x00050002,DW1=0,DW2=addr of Header,DW3=0x1. Then I write DW0 to Bar2+0x0,DW1 to Bar2+0x4,DW2 to Bar2+0x8,DW3 to Bar2+0xc.

At the end, I checked the write back data and found that the write back data are all zeros.It seems like that the FPGA does nothing at all.

What are the detailed steps I should follow to set up the DMA operation correctly? I read the pci express compiler doc but didn't get enough information about software application.

Thanks a lot for any help.

[heppermann]

What I found confusing in the 'PCI Express Compiler Users Guide' on pg 6-19 was the relation of the 5 step process to kick off the DMA and the Chaining DMA Descriptor Table. The 5 step process sounds like the implementation of the Simple DMA. If it is not, what do the terms PCI Express address (step 1) and master memory block (step 2) refer to?

Does master memory block refer to Chaining DMA Descriptor Table's offset in BAR2?

A few other questions I had about the example are:

• Are the Descriptor Tables supposed to be written into the shared memory assigned to BAR2?
  • Figure 6-3 shows the descriptor tables in RC memory. If this is the case, how does the Arria access these data structures? Is the RC memory in Figure 6-3 an implicit shared memory block?
• pg 6-19 says 'The software application writes the descriptor header into the into the endpoint header descritor register'. Table 6-7 maps the descritor headers to endpoint addresses 0x00 thru 0x20. These memory spaces conflict with the 5 step process on pg 6-19 to kick of the DMA. It looks like I am confusing something here. Does anyone know?

Any help would be appreciated. Thanks.

[huzj_ecc]

I think the correct method is the software application writes Descriptor Table Header into Bar2(or3) mapped endpoint header descriptor registers at offset 0x00-0x1c.

PCIe compiler 7.2 User Guide page 6-17 said,"altpcie_dma_prg_reg-This module contains the descriptor header table registers which get programmed by the software application.This module collects PCI Express transaction layer packets from the software application with the TLP type Mwr on Bar2 or 3" and "Header register module-RC programs the descriptor header(4 DWORDS) at the beginning of the DMA".

The next paragraph,"altpcie_dma_descriptor-This module retrieves the DMA read or write descriptor from the root port memory,and stores it in descriptor FIFO.This module issues PCI Express transaction layer packets to the BFM shared memory with the TLP type MRd".

In the simulation model,the Root Port BFM sources data(descriptors) for completions in response to read transactions received from the PCIE link,I think. But in the software application ,which module will response to the altpcie_dma_descriptor issued MRd TLP? Does the Jungo Pcie driver response automatically? Or should I write codes to deal with such MRd TLP in my software app?

At page 6-18,Table 6-4 descripted the Bar/Address map. Should I set Bar4(or5) if I want to use the rc_slave module in the example to bypass the chaining dma? But Bar0(or1) is also descripted to be used for rc_slave module.A mistake?

I have so many questions with the chaining dma example.I am wondering why Altera not release the source code of pcie software application such as altpcie_demo.exe.

Thanks for reply.

[heppermann]

• You said: I have so many questions with the chaining dma example.I am wondering why Altera not release the source code of pcie software application such as altpcie_demo.exe.

You are correct, all of this confusion would be eliminated if they would release this source code (driver source might be needed as well). Do you think we need to start a new thread to explicitly ask for this?

• You said: I think the correct method is the software application writes Descriptor Table Header into Bar2(or3) mapped endpoint header descriptor registers at offset 0x00-0x1c.

After reading PCI Express Compiler Users Guide, I thought the same thing, but then I started looking at the bus function model (BFM) driver source code to see how a DMA simulation is performed, and now I think otherwise. For example, look at the file 'C:\altera\72\kits\ArriaGX_PCIe\Examples\PCIe_High PerformanceDesign\Quartus\top_x4_examples\chaining _dma\testbench\altpcietb_bfm_driver_chaining.v'
I believe this file is one of the higher level bfm driver routines. If you look at the file you can find the following (some parts omitted for brevity):

##########BEGIN CODE################
// Run the chained DMA write
task dma_wr_test(...);
begin

// write 'write descriptor table in the RC Memory
dma_set_wr_desc_data(bar_table, setup_bar);

// Write Descriptor header in EP memory PRG
dma_set_header( ... )

end
##########END CODE################

If you look at the called functions dma_set_wr_desc_data() claims it writes the descriptor table in root complex (PC / host) memory.
Also, the comments above dma_set_header() function shows descriptor header tables for endpoint and root complex memory. The documentation almost reads like there is one descriptor header table mapped to a BAR 2. All in all, the code is not clear for porting to an actual implementation because I cant tell if shared memory means BFM driver memory or memory mapped by a BAR (or they are the same thing in an actual implemenation).

I wish there was a document that explained the reference design a little more for the vantage point of someone that wants to modify the existing design, and not from the BFM vantage point. The BFM blurs what needs to be done by a PC and what is contained in the reference design.

Best of luck, it appears we both need some right now.

[Hey_Hey]

Maybe I can clear up a few things...

The descriptor tables are located in the system's host memory (also known as root complex memory or BFM Shared Memory). See figure 7-2 of the 8.0 PCIe Compiler User Guide. I'm not a software guy, but you will need to get the system to lock down that memory and give you the real physical memory address of it (not the virtual address the application would use). Same thing you need to do with the actual memory buffer data you want to transfer via DMA. The addresses in the descriptor table point to the data buffers to be transferred. The descriptor table entries are described by tables 7-6, 7-7 and 7-8.

Then you must write the real physical address of the descriptor table to the Descriptor Table Header registers which are offset from BAR2 (or BAR3:2) by the values shown in table 7-5. The Descriptor Table Header format is shown in tables 7-3, 7-4, and 7-5.

The Chaining DMA hardware will then read the Descriptor Table using MRd TLP's from the system host memory, using the address from the Descriptor Table Header register. The root complex hardware will automatically respond to the MRd TLP and return the data from the memory address. (huzj_ecc - your driver doesn't need to respond to the MRd TLP, in fact there is no way to do that, you just have to have the descriptor table locked down in memory and put the correct address in the Decriptor Table Header register.)

It does appear that the PCIe Compiler user guide is missing an important piece of information on how this is all setup. The organization of the actual descriptor table:

Byte Offset Field
0-13 Reserved
14-15 EPLAST
16-31 Descriptor #1 (following format of tables 7-6, 7-7, and 7-8)
32-47 Descriptor #2 (ditto)
48-63 Descriptor #3 (ditto)
..... and so on for as many descriptors as specified by the "Size" field
in the descriptor table header register

I think the Descriptor Table must also be no more than 4KB in total size and can't cross a 4KB boundary.

The EPLAST field in the Descriptor Table is updated by the Chaining DMA hardware with the number of the last descriptor that was completed, when the hardware is enabled to do so by the EPLAST_ENA bit in the Descriptor Table Header register or the EPLAST_ENA bit in the actual descriptor.

heppermann - Yes, it looks like those steps you mentioned in the user guide are leftover from the previous simple DMA description.

I think I answered most of the questions with the above description. Please post any followups here. I will try to answer if I know the answer and when I can.

[heppermann]

Hello Hey Hey,
Thank you for your response, that was very informative and cleared up a lot.

I have atleast one more point of confusion. There are two Chaining DMA Descriptor Headers at offset 0x00 and 0x10. The first for write and the other for read. Why is there a Direction bit in the Control Fields (Table 7-4 of PCI Express Compilers Users Guide 8.0)? Is this a redudant thing, or is there some significance to this bit. To me, I would assume the registers at 0x00 and 0x10 specify the direction.

Thanks.

[Hey_Hey]

The direction bit is redundant. I'm not 100% sure, but I think it is not even neccesary to set it to the correct value.

[likewise]

Hello,

on what board do you guys exercise this chaining DMA design?

I would like to write an (open source) device driver for the Altera Chaining DMA Example, but I need to get the soft core working first.

Regards,

Leon.

[Hey_Hey]

Here are some Altera boards that it will work on:

PCI Express Development Kit, Stratix II GX Edition

Arria GX FPGA Development Kit

Altera also has many partners with PCI Express boards that it will work on. Look here. The chaining DMA design example requires no off-FPGA resources (besides the PCIe interface of course) so it should work on pretty much any board with an Altera device connected to the PCIe link, via either an internal or external PHY.

[heppermann]

Hello Hey Hey,
Can I use the chaining programming files (C:\altera\72\kits\ArriaGX_PCIe\Examples\PCIe_High PerformanceDesign\Quartus\top_x4.sof) that are installed with the Arria GX dev kit, or do I have to do a complete build of the FPGA build files?

If that is OK, I am seeing some unexpected results. Everytime I try to write and read a BAR, I always read a 0xffff back. Do you have any idea what I might be doing wrong?