Our setup is also extremely fault-tolerant. During one of our runs, we had close to half of our inference workers get taken out.
I was actually in a call at this time, and when I was done, the incident had resolved without manual intervention.
Was super cool to experience this in real time