Given that this only runs on certain Apple hardware, I wouldn’t be surprised if the Secure Enclave holds that certificate and can confirm at an extremely low level that it is being used only to sign a hash of of the app code itself and a shared secret with the app developer.

Brilliant, in a scary way. In a way it makes data portability regulations all the more important.

From my quick reading of the docs:

It generates a public-private key pair that is stored in the secure enclave, then it sends that public key (or the hash maybe) to Apple for them to sign. The rest of the stuff is as you expect.

One could simply figure out how the request to apple is made to get them to sign a key, and that's that. Get them to sign a key and pretend to be the app from now on.

I guess this prevents spam from someone signing thousands of keys using a specific phone's serial number, though. Assuming there's an unique public-private key for each phone apple makes, one can't simply get them to sign keys with random serial numbers.

The way these schemes usually work is that the pairing is done at the factory. Apple switch the iPhone on for the first time as it's being made, it generates a private key that never leaves the secure chip and then presents the public key. The public key is then signed to create a certificate chain and the certs handed back to the device for storage.

So, there's no way to beat it except by extracting a private key, or by using some software exploit to confuse it into signing the wrong thing.

You don't need to extract the private key though, just use it to sign things. So if you have shell access on the phone, you can tell the SE to sign the request you want.

Only to some extent. Apple work very hard to prevent that from being possible, and it's not necessarily signing just anything the app processor sends. Usually this stuff is integrated with the bootup process.