Here’s essentially the expansion board broken down. It’s got two PCIE switches inside, meaning that you only need a singular x16 slot to power 8 GPUs (or two). Let’s talk about how it works. First, the main 8 GPUs slots You have two methods of setup here: - Direct connect via two PCIE x16 -> 4x x8 MCIO - Cascade via a single PCIE x16 -> 2x x8 MCIO The difference here is how much of your GPU usage do you want heading back to your CPU vs keeping everyone around one dimension for faster GPU -> GPU communication. A quick way to think about it: model fits on 8 GPUs? Cascade. CPU offload? Probably direct connect. But what IS cascade? Let’s look back at the diagram. Direct connect says that we’re going to plug in two slots worth of x8 connectors into both 0-7 and 8-15 slots (two halves of our PCIE x16) into the separate halves. This then populates the switch in such a way that we can support 8 GPUs, each on separate 4-GPU P2P groupings. The nice part here is because the PCIE switches live in this board, you can take four additional MCIO cables and go from the output downlink into any one of the direct connect interfaces and add an additional GPU. BUT, if you want the two groups of 4 GPUs to interact, they’ll get tunneled through a single x16 slot worth of transfer, and have to go through the CPU. Not a fun bottleneck when dealing with 64 lanes worth of GPUs on either side. The alternative is what we call “cascade.” If we don’t want to worry about the CPU side (e.g. model inference), we can instead only hook up one PCIE slot as the input, then take the downlink from SW1 and use it as the INPUT to SW2. It’s still the bandwidth of a singular PCIE5 x16 slot, but you avoid going through the CPU which saves you some cycles. For the full 12 GPUs, you would need 3-4 full x16 slots on your motherboard. With the direct connect slots running through the host CPU still (since they are direct connect), if you’re not doing cascading. Or translation: - Non-DC: slot 1 and slot 12 are powered through additional retimers and MCIO at x16 and don’t add a slot, it’s just moving your existing PCIE slots somewhere else. Then you power slots 2 and 11 through the “jump” mentioned earlier - DC: slot 1, 12, and then 2 or 11 are powered through a retimer, and then the other (between 2 and 11) are powered through the “jump” Now let’s get into how I plan to use this. I have 9 GPUs in total (8 6000’s and a 6000 ADA), so I’ll be doing the Cascade, then using the SW2 output to power a 9th PCIE slot similar to the non-Cascade version that could power 10. What I don’t know yet is which is better: TP4/PP2, or TP8 over the two P2P groups, but I expect that would be the difference of batching. I’ll probably accept some slowness and do TP8. I can’t predict how much slower that will be, so when it comes in I’ll do these full benchmarks in many different combinations. What I want you to gather from this is these boards have internal PCIE switches, to help avoid needing to go through the CPU to have your GPUs interact, and bring you a larger lane budget than your CPU would allow.

My labor of love to you all over the last month is finally out! AIPerf lets us create a standard in explaining how these models perform under user workloads, and provide us with the insights to knowing when to scale our systems. The next step: AIPerf + Traces (AgentX)

Every day I try and write carefully crafted tweets and then one random post gets like 150k views. Not sure what to do with this information but almost to the point I can get monetized so that’s neat

Your cache is everything when it comes to inference, but how do you make sure you're keeping it around with local models? Typically when you start a server, it's a fresh slate. Then the KV cache grows, evolves, and your cache hits keep growing. But then you shut down the server, swap to a new fancy toy (model), and your cache is destroyed. Nothing we can do there, since caches are model-specific. But let's say then you want to go *back* to deploying the original model. Now *it's* KV cache was also destroyed! @sgl_project supports directly setting up hicache through some configuration arguments. The important part: * L1: GPU KV cache * L2: RAM KV Cache (usually larger, but in the world of local LLM's sometimes not!) * L3: Disk/storage-based KV Cache If you have the storage, keeping/enabling L3 allows for you to plug and play as many models as you'd like, and it'll still keep the KV Cache of your old deployment! Now, on a reboot say you have a hit on L3. Because you got a hit, it'll then keep that "warm" in L2 and L1-based caching so you don't face that terrible TTFT again. In my own learnings, you have --hicache-storage-prefetch-policy. This tells SGLang how to check and when to check if something exists in hicache in say L3. The two options I looked at were: * wait_complete * best_effort `best_effort` will check if L3 has that cached prefix, start retrieving it, but don’t block inference waiting for the full cache to arrive, reusing what’s ready and recompute anything that isn’t. Or: it'll promote whatever it can retrieve from L3 to L2/1 without blocking the request. `wait_complete` means that if L3 has that cached prefix, finish retrieving it rather than racing storage retrieval against recomputation. (These are Codex definitions) Translation: `best_effort` could have a better TTFT (probably) since it checks if we have it and then recomputes, while `wait_complete` will just fully reuse the cache and you'll wait a bit longer since it's going from cold -> hot. Don't quote me here, just where my intuition. There's a third one called `timeout`, which acts as a middle ground, still need to try it out. Last few little important parts: * SGLang will store the L3 wherever SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR is set as the env variable. * When using L3, it's based on the exact configuration and script you're currently serving with (usually with model/config information & topology, as well as tokenizer and such). I haven't tested if changing a deployment let's you reuse the L3, but my hunch is not without being careful * (Just a hunch, again still needs testing) You should probably use m.2 drives here and not HDD's if doing `wait_complete`, and possibly `best_effort` for HDDs. Need to toy with this more. But since `best_effort` recomputes, we likely can just grab it off the HDD and not worry about the transient speed.

i can probably take this thread point by point & counter it hardcore but the quote tweets have done that so it’s not that interesting. but beyond that.. you as an openai head of policy simply cannot publicly post content where you blatantly introduce mechanics for fud regulations that clearly favor your institution no matter how truthfully you believe in it. openai was supposed to be the *open* one, remember? how quickly do even the employees forget.

if I knew I could make my own model on my own data at the frontier and own all of it on a rapidly decreasing cost, and if all my other peers realize it too, why on earth would this reduce capex in the market? Increased datacenter rental demand for both training and inference.

Introducing factuality in the Arena: a new ranking of models according to a weighted combination of human preference and factuality. Model rankings are now viewable according to a weighted combination of human preference and factuality. Factuality is live in our Text and Search Arenas as a non-default toggle. We audit model responses by randomly sampling battles and extracting web-verifiable claims. We then verify these claims and compare the average correctness between model responses. To power these rankings, we’ve labeled over 2 million claims made by LLMs in real-world conversations, 1.3+ million from Text Arena, and 700k+ from Search Arena. Notable highlights with factuality enabled in the Text Arena: - Claude Fable 5 moves down slightly to spot #2# - GPT-5.5 saw the largest increase, moving up 13 spots into the #7# spot - Muse Spark dropped the most from #7# to #20# (-13pt) By labs, Meta saw the largest drop from #2# to #5#, while Anthropic overall held the #1# spot. Looking at only open model providers, Xiaomi saw the largest improvement, jumping from #9# to #6#. Learn more about the findings and methodology in this thread.