Sumanth Hegde(@sumanthrh):The operations in weight loading can vary a LOT across models and layers. To support these, we use a “recording tensor” dry run - we use a dummy tensor to record the various transformations in the `load_weights` function, and store the op chain at init time. We replay the op chain on the trainer side to compute the full weights during each weight sync.

Sumanth Hegde

@sumanthrh

Post-training @anyscalecompute. Prev - @UCSanDiego, @C3_AI, @iitmadras. Machine Learning and Systems. Intensity is all you need.

Joined February 2016

26 Following 1.2K Followers

Sumanth Hegde@sumanthrh

2026.08.25 17:06

The operations in weight loading can vary a LOT across models and layers. To support these, we use a “recording tensor” dry run - we use a dummy tensor to record the various transformations in the `load_weights` function, and store the op chain at init time. We replay the op chain on the trainer side to compute the full weights during each weight sync.