Running a 763B model at 192K context usually demands a multi-node cluster.
Meet Abyss — running DeepSeek-V4.1-Flash locally on a single dual-GPU workstation (2× RTX PRO 6000 96GB) by intelligently tiering native MXFP4 experts across VRAM, RAM, and NVMe.
Latest benchmarks:
⚡ ~52 tok/s decode at 96K context
📖 ~14s prefill for 48K tokens (~3,400+ tok/s)
In earlier matched tests against upstream llama.cpp, Abyss already delivered 5.2× faster prefill and 25% higher decode throughput—and our latest kernel & tiering optimizations push it even further.
Code & details:
#LocalLLM# #DeepSeek# #AIInfra# #CUDA# #Blackwell# #OpenSource#
Show more
