Every day I try and write carefully crafted tweets and then one random post gets like 150k views. Not sure what to do with this information but almost to the point I can get monetized so that’s neat

Your cache is everything when it comes to inference, but how do you make sure you're keeping it around with local models? Typically when you start a server, it's a fresh slate. Then the KV cache grows, evolves, and your cache hits keep growing. But then you shut down the server, swap to a new fancy toy (model), and your cache is destroyed. Nothing we can do there, since caches are model-specific. But let's say then you want to go *back* to deploying the original model. Now *it's* KV cache was also destroyed! @sgl_project supports directly setting up hicache through some configuration arguments. The important part: * L1: GPU KV cache * L2: RAM KV Cache (usually larger, but in the world of local LLM's sometimes not!) * L3: Disk/storage-based KV Cache If you have the storage, keeping/enabling L3 allows for you to plug and play as many models as you'd like, and it'll still keep the KV Cache of your old deployment! Now, on a reboot say you have a hit on L3. Because you got a hit, it'll then keep that "warm" in L2 and L1-based caching so you don't face that terrible TTFT again. In my own learnings, you have --hicache-storage-prefetch-policy. This tells SGLang how to check and when to check if something exists in hicache in say L3. The two options I looked at were: * wait_complete * best_effort `best_effort` will check if L3 has that cached prefix, start retrieving it, but don’t block inference waiting for the full cache to arrive, reusing what’s ready and recompute anything that isn’t. Or: it'll promote whatever it can retrieve from L3 to L2/1 without blocking the request. `wait_complete` means that if L3 has that cached prefix, finish retrieving it rather than racing storage retrieval against recomputation. (These are Codex definitions) Translation: `best_effort` could have a better TTFT (probably) since it checks if we have it and then recomputes, while `wait_complete` will just fully reuse the cache and you'll wait a bit longer since it's going from cold -> hot. Don't quote me here, just where my intuition. There's a third one called `timeout`, which acts as a middle ground, still need to try it out. Last few little important parts: * SGLang will store the L3 wherever SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR is set as the env variable. * When using L3, it's based on the exact configuration and script you're currently serving with (usually with model/config information & topology, as well as tokenizer and such). I haven't tested if changing a deployment let's you reuse the L3, but my hunch is not without being careful * (Just a hunch, again still needs testing) You should probably use m.2 drives here and not HDD's if doing `wait_complete`, and possibly `best_effort` for HDDs. Need to toy with this more. But since `best_effort` recomputes, we likely can just grab it off the HDD and not worry about the transient speed.

i can probably take this thread point by point & counter it hardcore but the quote tweets have done that so it’s not that interesting. but beyond that.. you as an openai head of policy simply cannot publicly post content where you blatantly introduce mechanics for fud regulations that clearly favor your institution no matter how truthfully you believe in it. openai was supposed to be the *open* one, remember? how quickly do even the employees forget.

if I knew I could make my own model on my own data at the frontier and own all of it on a rapidly decreasing cost, and if all my other peers realize it too, why on earth would this reduce capex in the market? Increased datacenter rental demand for both training and inference.

Introducing factuality in the Arena: a new ranking of models according to a weighted combination of human preference and factuality. Model rankings are now viewable according to a weighted combination of human preference and factuality. Factuality is live in our Text and Search Arenas as a non-default toggle. We audit model responses by randomly sampling battles and extracting web-verifiable claims. We then verify these claims and compare the average correctness between model responses. To power these rankings, we’ve labeled over 2 million claims made by LLMs in real-world conversations, 1.3+ million from Text Arena, and 700k+ from Search Arena. Notable highlights with factuality enabled in the Text Arena: - Claude Fable 5 moves down slightly to spot #2# - GPT-5.5 saw the largest increase, moving up 13 spots into the #7# spot - Muse Spark dropped the most from #7# to #20# (-13pt) By labs, Meta saw the largest drop from #2# to #5#, while Anthropic overall held the #1# spot. Looking at only open model providers, Xiaomi saw the largest improvement, jumping from #9# to #6#. Learn more about the findings and methodology in this thread.

Should I create a new dataset of synth traces off K3 when the weights drop for a variety of harnesses? Which harnesses/problems would be best? (Also, do we have better pipelines now than what I ended up doing for my data?)