ACS

PCIe Peer To Peer

The PCIe specification allows for peer-to-peer transactions. This means that it is possible and even desirable in some cases for one PCIe endpoint, to send data directly to another endpoint without having to go through the Root Complex. 가운데|섬네일

Access Control Services (ACS)

ACS provides a mechanism by which a Peer-to-Peer PCIe transaction can be forced to go up through the PCIe Root Complex. ACS can be thought of as a kind of gate-keeper —preventing unauthorized transactions from occurring.

Without ACS, it is possible for a PCIe Endpoint to either accidentally or intentionally (maliciously) write to an invalid/illegal area on a peer endpoint, potentially causing problems.

Usually ACS used for IO virtualization (also known as, VT-d or IOMMU) which can be enable or disable in BIOS or Kernel level.

PLX PCI Bridge

You can check whether ACS is enabled on PCI bridges by running:

If lines show “ACSCtl: SrcValid+”, then ACS might be enabled. Looking at the full output of lspci, one can check if a PCI bridge has ACS enabled.

sudo lspci -vvv | grep ACSCtl

If PCI switches have ACS enabled, it needs to be disabled. On some systems this can be done from the BIOS by disabling IO virtualization or VT-d. For Broadcom PLX devices, it can be done from the OS but needs to be done again after each reboot.

Use the command below to find the PCI bus IDs of PLX PCI bridges:

Next, use setpci to disable ACS with the command below, replacing 03:00.0 by the PCI bus ID of each PCI bridge.

run "setpci -s bus#:slot#.func# f2a.w=0000" to disable ACSCtl on the PLX switch.

run lspci -vvv to confirm ACSCtrl is disabled on PLX switch.

sudo setpci -s 03:08.0 f2a.w=0000   #assume 03:08.0 is the PLX pcie switch*
sudo setpci -s 03:00.0 f2a.w=0000

GPUDirect and ACS

IO virtualization (also known as, VT-d or IOMMU) can interfere with GPU Direct by redirecting all PCIe point-to-point traffic to the CPU root complex, causing a significant performance reduction or even a hang. Make sure that ACS is disabled on the PCIe. A Value of <flag> with ‘+' means enabled, while '-’ means disabled.

$ sudo lspci -s 0000:18:00.0 -vvv | grep ACSCtl
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

#In case it is enabled, use the following command to disable it (for the right PCIe address “-s”):
$ sudo setpci -s 0000:18:00.0 f2a.w=0000

AMD CPU

disabling IOMMU in GRU

1. sudo nano /etc/default/grub
2. GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=soft"
3. sudo update-grub
4. reboot