Probably the 2 most important things of any training stack are flexibility and reliability.
During our runs, we monitor the inference throughput and are able to dynamically scale the inference-trainer ratio to maintain high training throughput and reduce our staleness.