Moving bits to and from memory.
Because of parasitic capacitance+resistance of the wires.
The bigger the memory, the longer the wires.
The main trick is to organize the memory hierarchically: registers, small on-chip SRAM, caches of various types, and external RAM.
It's all because we have to use hardware multiplexing: reusing the same multiply-accumulate unit for multiple parts of the network.