Yotta Labs(@YottaLabs):Self-hosting a 753B model sounds absurd until you see what it actually takes. GLM 5.2 is open weights under MIT with a 1M context. The FP8 checkpoint needs about 744 GB of VRAM, which is one 8x H200 node. One vLLM command with tensor parallel 8 and you've got an OpenAI-compatible endpoint your existing code already works with. Biggest mistake: serving the full 1M context by default. Cap it at what you actually use and you get your batching memory back.

Yotta Labs

@YottaLabs

The Interoperable AI Compute OS for a Multi-Cloud, Multi-Silicon World

加入 May 2024

118 正在关注 4.2K 粉丝

Yotta Labs@YottaLabs

2026.07.06 14:23

Self-hosting a 753B model sounds absurd until you see what it actually takes. GLM 5.2 is open weights under MIT with a 1M context. The FP8 checkpoint needs about 744 GB of VRAM, which is one 8x H200 node. One vLLM command with tensor parallel 8 and you've got an OpenAI-compatible endpoint your existing code already works with. Biggest mistake: serving the full 1M context by default. Cap it at what you actually use and you get your batching memory back.

显示更多