Here’s essentially the expansion board broken down. It’s got two PCIE switches inside, meaning that you only need a singular x16 slot to power 8 GPUs (or two). Let’s talk about how it works. First, the main 8 GPUs slots You have two methods of setup here: - Direct connect via two PCIE x16 -> 4x x8 MCIO - Cascade via a single PCIE x16 -> 2x x8 MCIO The difference here is how much of your GPU usage do you want heading back to your CPU vs keeping everyone around one dimension for faster GPU -> GPU communication. A quick way to think about it: model fits on 8 GPUs? Cascade. CPU offload? Probably direct connect. But what IS cascade? Let’s look back at the diagram. Direct connect says that we’re going to plug in two slots worth of x8 connectors into both 0-7 and 8-15 slots (two halves of our PCIE x16) into the separate halves. This then populates the switch in such a way that we can support 8 GPUs, each on separate 4-GPU P2P groupings. The nice part here is because the PCIE switches live in this board, you can take four additional MCIO cables and go from the output downlink into any one of the direct connect interfaces and add an additional GPU. BUT, if you want the two groups of 4 GPUs to interact, they’ll get tunneled through a single x16 slot worth of transfer, and have to go through the CPU. Not a fun bottleneck when dealing with 64 lanes worth of GPUs on either side. The alternative is what we call “cascade.” If we don’t want to worry about the CPU side (e.g. model inference), we can instead only hook up one PCIE slot as the input, then take the downlink from SW1 and use it as the INPUT to SW2. It’s still the bandwidth of a singular PCIE5 x16 slot, but you avoid going through the CPU which saves you some cycles. For the full 12 GPUs, you would need 3-4 full x16 slots on your motherboard. With the direct connect slots running through the host CPU still (since they are direct connect), if you’re not doing cascading. Or translation: - Non-DC: slot 1 and slot 12 are powered through additional retimers and MCIO at x16 and don’t add a slot, it’s just moving your existing PCIE slots somewhere else. Then you power slots 2 and 11 through the “jump” mentioned earlier - DC: slot 1, 12, and then 2 or 11 are powered through a retimer, and then the other (between 2 and 11) are powered through the “jump” Now let’s get into how I plan to use this. I have 9 GPUs in total (8 6000’s and a 6000 ADA), so I’ll be doing the Cascade, then using the SW2 output to power a 9th PCIE slot similar to the non-Cascade version that could power 10. What I don’t know yet is which is better: TP4/PP2, or TP8 over the two P2P groups, but I expect that would be the difference of batching. I’ll probably accept some slowness and do TP8. I can’t predict how much slower that will be, so when it comes in I’ll do these full benchmarks in many different combinations. What I want you to gather from this is these boards have internal PCIE switches, to help avoid needing to go through the CPU to have your GPUs interact, and bring you a larger lane budget than your CPU would allow.