To add a question to the great question and discussion here on pyenv, venv, virtualenv, and virtualenvwrapper, could someone please explain how conda environments fit into this world? When are the preferred use cases for conda environments vs the other virtual environment options?
Update to question in November 2024: how do Poetry and Docker fit into this world?
Here is how everything fits together. We'll start with the problems the various tools are solving. Then cover how each tool slots in.
Dependency Managers. Problem these tools solve:
- What packages (dependencies) does the project need? For example: Django4.2, PyTest, Psycop2, etc.
- Can I install all those packages with one click?
- Do all those packages work together? e.g. the version of django I install works with the version of psycop, etc.
Dependency/Package Managers solve the above questions for us.
Environment Managers. Problems these solve:
- How do you have multiple projects that each need different things? For example, one project needs Python3.8 and Django4.2, but another needs Python3.12 and Django5.0?
Environment managers create and manage "virtual environments", which allow each project to have its own set of packages/python versions installed.
How do all the tools fit in? Some confusion is often caused by overlap in the tooling. Here is how they map:
- pip. Dependency management only.
- venv. Environment management only.
- virtualenv. Environment management only.
- Pipenv. Environment & Dependency management.
- Poetry. Environment & Dependency management.
- Conda. Environment & Dependency management.
- Conda is not just for python, but for R, Ruby, Node.js, etc.
How does Docker fit in? Docker provide environment and dependency management, and also provides operating system management. For example, a Docker container can allow one project to operate on a Fedora Linux distribution, and another project to use Ubuntu. Docker can also be mixed and matched with the above tools, e.g. Docker handles the operating system management, but calls Poetry to handle environment and dependency management. Docker can also manage other dependencies too, like network settings, system libraries, etc.
When to Use What?
- Development phase. Non-production. You want an environment manager and a dependency manager. It doesn't really matter which sets you use. venv/pip is fine. conda is fine. poetry is fine.
- Use whatever the people you work with use, or what you find easiest. There are some small differences between tools -- conda is a little slower to incorporate new python versions but has solid dependency compatibility checks, pip and env come with python and don't need to be installed separately, etc. -- but for the most part whatever you are most comfortable with or find easiest is what you should do unless you need a very specific feature offered by only of the tools.
- Deployment and production. With some exceptions you're probably going to want/need Docker. You're almost certainly going to need to use whatever tooling the rest of the team uses.
Hope this helps everyone save a bunch of time understanding how all these tools fit together.