1 - this is quite bad in of itself 2 - in some hypothetical world, imagine if we engineered the fly connectome to seek honey (because as humans, we want honey), put no honey in the box, then augmented the fly to have abstractions like language that are similar to a human, and then gave the fly top-human-like box-breaking and problem-solving skills. Then say our main way of making the fly ‘good’ is by turning it off whenever our rough heuristic of the ‘escape’ section lights up. We don’t understand the flies outside of that. Then say we have millions of these human-like-experience-capable intelligent flies controlled by crude needles to the brain, for which we’ve made much less of an effort to properly raise, understand, or care for them. This (purely hypothetical world of course) also feels quite bad, both for the flies and for the humans creating them.