> And then you may just have introduced side channels in crypto code.

incorrect crypto code. If overflow is intentional, it should be annotated as such in operations, will generate similar assembly and won't panic. If it isn't intentional, then the code was bad to begin with.

If the branch can be mispredicted, and the misprediction happens depending on internal state you don't want to leak (cf. Spectre), then you have a side channel even if the branch is never actually taken.