There are other, more efficient ways of handling dependency version conflicts than having an isolated env where each module is downloaded specifically for that env. For example, it doesn't make sense that if I have two virtual env's that ,use the exact same module (and version), it's downloaded and stored twice on my machine.

That's one of the issues that conda (https://docs.conda.io/en/latest/) solves by design. When using conda environments you get hard links whenever possible. Improvements to venv, making it built-in module, and projects designed to simplify dependencies made conda less attractive in comparison but it's still a solid way to have Python. Still, it doesn't really solve fundamental issue with Python packages you mentioned, unfortunately.

For real reproducibility you want to go a step further with a virtual machine or something. Freezing your python dependencies won't shield you from changes in your C standard library or differences in vector extensions or whatever. In practice, most codes shouldn't be so fragile though if dependencies are reasonably well behaved... Otherwise how can you even trust that the dependency is giving a reasonable answer?