Memory cost and capacity are significant issues for AI accelerators. Unlike game rendering, model inference can have a deterministic memory access pattern. You don’t need “random access memory” at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as continuous reads were delivered at the necessary bandwidth. NAND flash is over 100 times cheaper per GB than HBM, so there should be opportunity there, even after giving a flash controller a 1024 bit interface with HBM bandwidth. You could make a specialized pin protocol that just supported pipelined transfer of full 16KB+ pages from the flash to program-managed accelerator scratchpad memory and improve per-pin performance over HBM, but it might be more convenient to make it still look like a true random access memory with very fragile performance characteristics, where anything but sequential reads falls off a 1000x+ performance cliff. That has the advantage of automatically using existing cache hierarchies, and providing a natural path to update the flash memory with new model weights. With the stream-to-scratch interface, code has to be completely rewritten before it works at all, while the ram-emulation interface will start off just extremely slow, and you can incrementally sort out the changes for full performance. There may be cases where there isn’t enough scratchpad SRAM to hold the weights for a layer, which might force you to deploy the old optical drive optimization technique of duplicating data in multiple places on a sequential read to avoid seeking, but there would be capacity to burn. It might be possible to do something like cuda graph capture to record a memory access trace and have everything magically remapped to a linear sequence, but deploying programmer / agent elbow grease to manage transfers and access in a scratch ram ring buffer would be lower risk. A split memory system consisting of some channels of flash and some channels of HBM will probably be suboptimal compared to a uniform memory, but it could be much cheaper, and allow much larger models to be run. I think th case is strong for inference, but you have to stretch more for training. You can still linearize all the weight memory accesses, both reads and writes, but flash memory would quickly wear out from the writes, even if they were all perfectly page aligned. Replacing low-latency HBM with massively parallel cheap(er) DRAM at high latency might still be a worthwhile cost savings.

Goldman Sachs upgraded Nittobo to Buy in a July 6 report. One key reason was its cautious view on TSMC’s glass core substrate (GCS) progress. I won’t add much commentary here. Just sharing the note. That said, my takeaway is that the other reasons GS cited for the upgrade do not really conflict with a constructive view on GCS over the next several years. One thing to note: based on past patterns, the likelihood of TSMC providing major updates on GCS/CoPoS at its upcoming July earnings call may be low. This is especially true since TSMC just shared key R&D results at a Japan symposium in June, where it also made a rare mention of supplier partners. If that read is right, the likelihood of major GCS updates at Innolux’s and Ibiden’s August earnings calls may also be low.

Memory makers strike back and finally reaches Apple. Most all of @MicronTech’s revenue gains this quarter were from price hikes, NOT from unit increases. Does it stink that price hikes impact OEMs and consumers? Yes. BUT, when memory makers were -80% grows margins, the OEMs were punishing memory makers with continuous price cut demands. It cuts both ways.