It's exactly as risky as any other possible stateless authenticator implementation, if you think about it.
For example, another way of doing it is to derive the private key from the key handle via deterministic key derivation – which the attacker can brute-force just as well as the encrypted per-site key stored in the key handle.
The key insight is that a stateless authenticator is by definition globally (i.e. across secrets and sites) deterministic, and given an input-output pair, you'll be able to brute-force its internal secret. The solution is to make that internal state large enough for that to be computationally infeasible.
For example, another way of doing it is to derive the private key from the key handle via deterministic key derivation – which the attacker can brute-force just as well as the encrypted per-site key stored in the key handle.
The key insight is that a stateless authenticator is by definition globally (i.e. across secrets and sites) deterministic, and given an input-output pair, you'll be able to brute-force its internal secret. The solution is to make that internal state large enough for that to be computationally infeasible.