Xinyu Zhang
@xinyzng
1.1K Following 903 Followers
We stress tested Ray-2.58.0 for improvements on fault tolerance and reliability by scaling video data processing pipeline to caption 0.6 Petabytes of video data on 1600 Nvidia Blackwell GPUs in collaboration with @CoreWeave
With the new release Ray Data shows:
• 𝟲𝟬％ higher throughput for streaming data processing
• 𝟮𝟯％ faster for batch inference
• 𝟮𝟰％ faster for Ray Data shuffle
This will significantly improve how world model and physical AI companies handle pre-training data.
Blog:
Show more
