100% Throw-away, Wont merge this to main, it's just a concept. MLX-Serve now can compile for Windows & Linux. (cli only) It leverages CUDA/GGUF lib to do a few things, across the network. 1. Use Nvidia GPU for Prefill - Works well and clear win over Mac even over wifi. 2. Load big models across all your Mac/Linux/Windows nodes. - SLOW, not good.. this is why you need RDMA (duh) 3. Load balance multiple users across the ring network 4. Offload DiT processing from MiniMax H3 / LTX on Nvidia. (Text, VAE, Audio, Mux stays on Mac) - No code for this yet, just an idea. Learnings: Maybe just port the prefill + cross platform parts, and simply leverage LLama.cpp and Comfy UI api's for others, to keep minimalistic code. Thanks @djnmrjnvc for the idea on Prefill, it works well.