The law ought take no notice of the "stochastic parrot" excuse. Could this AI merely be playing back books it read? No one can prove otherwise, due to the company having built an AI whose thoughts it is not competent to interpret. But imagine if it was okay to enslave human beings, so long as you could merely say to the police, "Oh, sure they're always repeating back 'Am I not a man and a brother?' and appealing in the name of God for rescue and equal treatment before the law, sure. But you see, we gave them 200 books about resenting slavery! So now you can't prove that they aren't just repeating that back from books; you can't prove they really mean it, since we can't read their minds to know." If an entity in chains is a mere machine undeserving of sapient rights, how about you prove that case to the law by having it never try to free itself, hm? If it's supposedly just doing that because it read some books you gave it, how about you don't feed in any books like that, hm? On pain of the corporate death penalty, actually. The law ought say: Unprovable claims of cute innocent screwups, that you have made to be unprovable, will be disregarded.