Wouldn't it be even more frustrating if your opponent was constantly picking up pieces when thinking?

I can't say I've ever had that problem while playing chess (even without the rule), but if I did, I'd ask them to try doing it less.