hear me out, mathematicians entering ai alignment maybe consider this challenge. i think its probably millennium level problem(?)
take any complex real world software, like a browser or a kernel. let's pick a kvm based sandbox, which i suspect is somewhat less complex than chromium(?), running an increasingly capable, potentially adversarial AI agent.
we want to formally prove one simple property, no matter what code the agent executes inside the VM, it cannot escape and compromise the host. for that we need to prove the entire relevant software stack, kvm, cpu virtualization, kernel, virutal devices that could affect the isolation. (to make things easiers, lets assume hardware wont have weird states)
can we mathematically establish whether formally verifying a system of this complexity is even computationally feasible?