I don't see any downsides. As you point out, upgrading hashed peppers isn't as beautiful. Yes, you can just introduce a new pepper and wrap the old hash into another hash using the new pepper, but you'll still have to keep the older pepper around to obtain the old hash during checks, creating risks if the attacker gets access to the older pepper and an old db backup. With AES usage, you can throw the old key away once all of your db has migrated and you've updated your key backups.

Just make sure you use a suiting AES mode. If it turns out that you xor'd all hashes with the same stream that depends on your pepper only, it's not really helpful :).