ACS: Difference between revisions
| Line 1: | Line 1: | ||
=== PCIe Peer To Peer === | === PCIe Peer To Peer === | ||
The PCIe specification allows for peer-to-peer transactions. This means that it is possible and even desirable in some cases for one PCIe endpoint, to send data directly to another endpoint without having to go through the Root Complex.<ref>file:///Users/bpark/Downloads/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf</ref> | The [[PCIe]] specification allows for peer-to-peer transactions. This means that it is possible and even desirable in some cases for one PCIe endpoint, to send data directly to another endpoint without having to go through the Root Complex.<ref>file:///Users/bpark/Downloads/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf</ref> | ||
[[파일:PCIe P2P.png|가운데|섬네일]] | [[파일:PCIe P2P.png|가운데|섬네일]] | ||
| Line 8: | Line 8: | ||
Without ACS, it is possible for a PCIe Endpoint to either accidentally or intentionally (maliciously) write to an invalid/illegal area on a peer endpoint, potentially causing problems. | Without ACS, it is possible for a PCIe Endpoint to either accidentally or intentionally (maliciously) write to an invalid/illegal area on a peer endpoint, potentially causing problems. | ||
Usually ACS used for IO virtualization (also known as, VT-d or IOMMU) which can be enable or disable in BIOS or Kernel level. | Usually ACS used for IO virtualization (also known as, VT-d or IOMMU) which can be enable or disable in [[BIOS]] or Kernel level. | ||
=== PLX PCI Bridge === | === PLX PCI Bridge === | ||
You can check whether ACS is enabled on PCI bridges by running: | You can check whether ACS is enabled on PCI bridges by running: | ||
If lines show “'''ACSCtl: SrcValid+'''”, then ACS might be enabled. Looking at the full output of lspci, one can check if a PCI bridge has ACS enabled. | If lines show “'''ACSCtl: SrcValid+'''”, then ACS might be enabled. Looking at the full output of lspci, one can check if a PCI bridge has ACS enabled. | ||
sudo lspci -vvv | sudo lspci -vvv | grep ACSCtl | ||
If PCI switches have ACS enabled, it needs to be disabled. On some systems this can be done from the BIOS by disabling IO virtualization or VT-d. For Broadcom PLX devices, it can be done from the OS but needs to be done again after each reboot. | If PCI switches have ACS enabled, it needs to be disabled. On some systems this can be done from the BIOS by disabling IO virtualization or VT-d. For Broadcom PLX devices, it can be done from the OS but needs to be done again after each reboot. | ||
Use the command below to find the PCI bus IDs of PLX PCI bridges: | Use the command below to find the PCI bus IDs of PLX PCI bridges: | ||
Next, use setpci to disable ACS with the command below, replacing 03:00.0 by the PCI bus ID of each PCI bridge. | Next, use setpci to disable ACS with the command below, replacing 03:00.0 by the PCI bus ID of each PCI bridge. | ||
run "setpci -s bus#:slot#.func# f2a.w=0000" to disable ACSCtl on the PLX switch. | run "setpci -s bus#:slot#.func# f2a.w=0000" to disable ACSCtl on the PLX switch. | ||
run lspci -vvv to confirm ACSCtrl is disabled on PLX switch. | run lspci -vvv to confirm ACSCtrl is disabled on PLX switch. | ||
sudo setpci -s 03:08.0 f2a.w=0000 #assume 03:08.0 is the PLX pcie switch* | |||
sudo setpci -s 03:00.0 f2a.w=0000 | sudo setpci -s 03:00.0 f2a.w=0000 | ||
Revision as of 08:22, 3 April 2023
PCIe Peer To Peer
The PCIe specification allows for peer-to-peer transactions. This means that it is possible and even desirable in some cases for one PCIe endpoint, to send data directly to another endpoint without having to go through the Root Complex.[1] 가운데|섬네일
Access Control Services (ACS)
ACS provides a mechanism by which a Peer-to-Peer PCIe transaction can be forced to go up through the PCIe Root Complex. ACS can be thought of as a kind of gate-keeper —preventing unauthorized transactions from occurring.
Without ACS, it is possible for a PCIe Endpoint to either accidentally or intentionally (maliciously) write to an invalid/illegal area on a peer endpoint, potentially causing problems.
Usually ACS used for IO virtualization (also known as, VT-d or IOMMU) which can be enable or disable in BIOS or Kernel level.
PLX PCI Bridge
You can check whether ACS is enabled on PCI bridges by running:
If lines show “ACSCtl: SrcValid+”, then ACS might be enabled. Looking at the full output of lspci, one can check if a PCI bridge has ACS enabled.
sudo lspci -vvv | grep ACSCtl
If PCI switches have ACS enabled, it needs to be disabled. On some systems this can be done from the BIOS by disabling IO virtualization or VT-d. For Broadcom PLX devices, it can be done from the OS but needs to be done again after each reboot.
Use the command below to find the PCI bus IDs of PLX PCI bridges:
Next, use setpci to disable ACS with the command below, replacing 03:00.0 by the PCI bus ID of each PCI bridge.
run "setpci -s bus#:slot#.func# f2a.w=0000" to disable ACSCtl on the PLX switch.
run lspci -vvv to confirm ACSCtrl is disabled on PLX switch.
sudo setpci -s 03:08.0 f2a.w=0000 #assume 03:08.0 is the PLX pcie switch* sudo setpci -s 03:00.0 f2a.w=0000
GPUDirect and ACS
IO virtualization (also known as, VT-d or IOMMU) can interfere with GPU Direct by redirecting all PCIe point-to-point traffic to the CPU root complex, causing a significant performance reduction or even a hang. Make sure that ACS is disabled on the PCIe. A Value of <flag> with ‘+' means enabled, while '-’ means disabled.
$ sudo lspci -s 0000:18:00.0 -vvv | grep ACSCtl
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
In case it is enabled, use the following command to disable it (for the right PCIe address “-s”):
$ sudo setpci -s 0000:18:00.0 f2a.w=0000
AMD CPU
disabling IOMMU in bios or GRU
- sudo nano /etc/default/grub
- GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=soft"
- sudo update-grub
- reboot
- ↑ file:///Users/bpark/Downloads/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf