We had an EC2 service that had to compute the health of all ec2 instances in < 60 seconds. It was taking 55. If this service went over 60 seconds, all ec2 instance checks would say INSUFFICIENT_DATA, and all hell would break lose (potentially triggering autoscaling policies, and disrupting billing). Every day I'd see the graph creep up +.05 seconds per run. "can you make it faster?" "sir, we're already on the largest ec2 instance. We have only 2 months left" Since noone understood the pre-AI slop Scala code of this monolith service, and we were just an army of SDE1 and 2, we started converting every for loop to parfor, causing some race conditions in some cases, but the time dropped to 45 seconds. There were no senior engineers that wanted to work on what was essentially the engine room of the titanic. After months of getting 40 sev2 pages per week, we decided to scale it out horizontally, at gunpoint. Here I was 2-3 yrs out of college scaling out one of the largest and most critical services in EC2. I wrote code for breakfast and dinner. I typed out every line, until it finally worked. What's funny is that with AI this might've been resolved in a month, but I spent 12 months in my life glued to the graph. This is not even the wildest thing that happened. In another case our service reported all instances as UNHEALTHY due to some randomass bug. This gave everyone a heart attack, but I'll save that for another time.