AER (Advanced Error Reporting)

AER (Advanced Error Reporting) is an optional PCI Express feature that allows for more enhanced reporting and control of errors than the basic error reporting scheme. AER errors are categorized as either correctable or uncorrectable.

• A correctable error is recovered by the PCI Express protocol without the need for software intervention and without any risk of data loss.
• An uncorrectable error can be either fatal or non-fatal.
  • A fatal, uncorrectable error causes the link to become unreliable.
  • A non-fatal, uncorrectable error results in an unreliable transaction

The AER driver in the Linux kernel drives the reporting of (and recovery from) these events.

• In the case of a correctable event, the AER driver simply logs a message that the event was encountered and recovered by hardware. Device drivers can be instrumented to register recovery routines when they are initialized.
• Should a device experience an uncorrectable error, the AER driver will invoke the appropriate recovery routines in the device driver that controls the affected device. These routines can be used to recover the link for a fatal error, for example.

Bad TLP Issues

There are bunch of Bad TLP issues about PCIe devices such as M.2, GPU, etc on various platform has been discussing over here and there.

NVMe AER issue,

There are many community reports AER error on boot on various systems. with following error or simlar

Bad TLP associated with device xxxx:xx:x

That means in this particular case, something goes wrong when the PCIe controller uses this method to access the configuraton space of a particular device.

It may be a hardware bug in the device, in the PCIe root controller on the motherboard, in the specific interaction of those two, or something else.

Kernal param, pci=noaer

The pci=noaer directive tells AER to not report errors. Those error reports would go into a log file, and each error sends a time-consuming interrupt request (IRQ) to the central processor. A rapid flow of error reports could thus flood the drive -- and clog NVMe bandwidth, slowing or even halting bootup.

The nvme 0000:xx:xx.x AER message identifies that error as from the NVMe M.2 connection to the PCIe bus.

So, the NVMe drive may be healthy, but there could be trouble brewing around the PCIe subsystem

Kernal param, pcie_aspm=off

Forcibly enable or disable PCIe Active State Power Management

• off : Disable ASPM.
• force : Enable ASPM even on devices that claim not to support it. (WARNING: Forcing ASPM on may cause system lockups)

Kernal param, pci=nommconf

The kernel option pci=nommconf disables Memory-Mapped PCI Configuration Space, which is available in Linux since kernel 2.6. Very roughly, all PCI devices have an area that describe this device (which you see with lspci -vv), and the originally method to access this area involves going through I/O ports, while PCIe allows this space to be mapped to memory for simpler access.

By using pci=nommconf, the configuration space of all devices will be accessed in the original way, and changing the access methods works around the AER problem.

Could not boot from M.2 due to Bad TLP in Recovery Mode

Well known workaround is

1. Adding "pcie_aspm=off" to GRUB_CMDLINE_LINUX parameter at /etc/default/grub file
2. Run sudo update-grub
3. then reboot.

If pcie_aspm=off does not work then try with pci=noaer with the same way and make it permanent if this works.

Adjust file /etc/default/grub file,

# for AER, pci=noaer pcie_aspm=off
# force to enable ECC feature on AMD EPYC, amd64_edac.ecc_enable_override
GRUB_CMDLINE_LINUX_DEFAULT="pci=noaer pcie_aspm=off amd64_edac.ecc_enable_override=1"

The ubuntu team aware it as Linux Kernel bug. Read more Bug_track_ubuntu_PCIe bus error