Can I trust the PCIe design?

In particular a secure system must not have a port where external devices have unrestricted RAM access. It would be useful to limit even some internal devices.

Several sources describe PCIe as a network and by implication a directed graph and even a tree with the “Root complex” as the tree root. Nominally there is a single path from any node to the root.

The Wikipedia article provides a good low level introduction to the PCIe data link and transaction layers. It describes no transactions. In network jargon the links have backward error control. Flow control is by invitation (my jargon). Some attention is given to preventing gridlock. The network is presumed not to loose packets, any more than a memory bus fails to honor a transaction. There are error reports but they seem to be for field diagnostics rather than error toleration. A dropped packet indicates logic failure, not congestion.
“a PCI Express [PCIe] bus link supports full-duplex communication between any two endpoints”
This needs to be throughly understood.

Notes on a PCIe tutorial

This is all about transactions, in general and particular. It complements the Wikipedia article nicely.

As I read thru this many questions arise. This is mainly because you cannot say everything first. As the author hints, knowing why a bolt is there helps you remember that it is there. Questions that come up should be recorded so as to later fill in loose ends. Comment: I know little of PCI.

In the write-packet example I see two 4 byte words. I think the length field should be 2 instead of 1. Perhaps the Fmt field value determines an initial fixed format portion of the packet and the length starts from the end of that portion.

Some transactions are composed of two or more messages, one from X to Y followed by one or more from Y to X. That transaction sort is called a ‘non-posted’ transaction and I find that terminology peculiar. In the first of these there is a “Requester ID” field. I wonder the provenance of this information. Is it the originator of the request, or the network? The Internet got this wrong and this is at the root of much grief today.

It seems clear that between two PCIe nodes messages are delivered in order.

Several sources make it clear that the network is a tree and the root complex has a special role and that all nodes know the direction of the root node. I need to know the root complex’s role. In particular what are the ramifications of the graph being directed?

What are the “the standard configuration registers”? Are they part of the standard PCIe hardware or are they part of the PCIe client? Are they part of the dongle that you plug into a Thunderbolt port?

This describes an earlier PCI standard with which PCIe is partly compatible. “Capability” therein is not the sort of thing described on these pages by that word. There is an obscure section there (“Bus enumeration”) describing how a PCI client learns its own address at boot time. I do not trust a client to be in charge of that information! I fear that PCIe copies PCI in this regard.
If a NIC card reads and exfiltrates pixels from PCIe put there by the GPU, would anyone notice? An IOMMU does not prevent this.
I worry that the hardware that imposes these access restrictions is distributed in each of the PCIe clients. The good news is that the distributed responsibilities are simple. It would seem that a card slot for a PCIe card grants the card unrestricted access to the other PCIe clients. The good news is that few computers have such slots. But some IO plug standards attempt to extend the plug’s authority to the authority of the PCIe client. This must be avoided.
More useful fragmentary information!
Google "PCIe standard configuration registers"

Googling “PCIe Bus enumeration”:
PCIe bus enumeration assumes more knowledge than given above.
This makes interesting claims about PCIe functionality, without much explanation.
We need definitions of “PCIe switch” and “PCIe bridge”. This suggests that a bridge is part of a switch. This suggests that a bridge is to an older technology, such as PCI.

PCIe is functionally inclusive of the older plain PCI standard which was much simpler and more expensive at the physical level. There may be architectural clues there.


Intel’s story
A proposed cartoon PCI architecture for smaller TCB.
Standard configuration registers