My interpretation is that it's related to modulate. Like how you might modulate the volume to reduce it (without muting it), "X is true modulo Y" means that Y makes X less true (but not untrue).

I didn't find any evidence for this etymology from casual searching however (other than that the roots are related).