How do the main features of npm
compare with pip
, pipenv
and Poetry package managers? And how do I use those features of pipenv
or Poetry
?
This could primarily help someone transitioning from being JavaScript developer to a Python developer.
I've never used RubyGems, but it may also be useful to compare the main features of that to the main Python package managers. To help anyone coming from a Ruby background.
Feature \ Package Manager | npm | pip | pipenv | poetry |
---|---|---|---|---|
Access to main repo (i.e. Pypi/npm) | ✓ | ✓ | ✓ | ✓ |
Record top level dependencies | ✓ | ✗ | Pipfile | pyproject.toml |
Record development dependencies | ✓ | ✗ | Pipfile | pyproject.toml |
Lock versions of all dependencies | ✓ | ✓ | Pipfile.lock | poetry.lock |
Switch between interpreter versions | nvm | ✗ | ✗ | ✓ |
Direct publishing | ✓ | ✗ | ✓* | ✓ |
Run scripts | ✓ | ✗ | Pipfile | ✗ |
Editable local packages | ✓ | ✓ | ✓ | ✓ |
Integration with Intellij | ✓ | ✓ | partial | ✗ |
* Directly publishing with pipenv
is possible if using a 3rd party dev dependency, and a script to tie it all together; see below.
Disclaimer: I only have experiences of these packages managers on Unix-like systems (OS X in particular), pipenv
markets itself as treating Windows is a first class citizen, I'm not sure how this works without pyenv
, which is not available on Windows as far as I am aware.
To get the most out of pipenv
, pyenv
should be installed. pipenv
will be able to detect and use any version of Python installed with pyenv
, even if it is not activated. For example, if a Pipfile
has listed Python 3.4 as a requirement: to successfully run pipenv install
, pyenv install 3.4.0
should be run first.
To create a new Pipfile
and venv (using Python 3.7.x):
>>> pipenv --python 3.7
If you are on linux you may have to use a slightly different command:
>>> python3 -m pipenv --python 3.7
Or, to install dependencies from an existing Pipfile.lock
, use the command below. This command can also be used to create a Pipfile
and venv (defaulting to the newest available Python version).
>>> pipenv install
To run commands within the created venv:
>>> pipenv run <script or command>
E.g.,
>>> pipenv run python main.py
Poetry still uses pyenv
, but in a different way: The version of Python you wish to use must be activated before calling poetry install
or poetry run
.
A pyproject.toml
can be created using:
>>> poetry init
or a full directory structure can be created using:
>>> poetry new <dir>
Before we can install we must activate a version of Python that matches what is specified in the pyproject.toml
file.
>>> pyenv global <python version specified in pyproject.toml>
Now we are able to create the venv using the command below, if a poetry.lock
file is present it will install all the dependencies listed in it.
>>> poetry install
To run commands within the created venv:
>>> poetry run <command>
If we change the global Python version using pyenv
we will no longer be able to run commands in the created venv. There is an exception to this if we use a locally created venv; see below.
Sometimes it's good to check that your code will work on both Python 3.7 and Python 3.4. This is not something we can take for granted.
Only possible by deleting the venv in recreating it using a different Python version:
>>> rm -rf <path to venv>
>>> pipenv --python <different python version>
A warning will likely be displayed that the Python version of the venv does not match the one specified in the Pipfile
, but as far as I can tell it is only a warning.
Multiple parallel environments #1071
Poetry is much better suited to this use case: Multiple venvs can be created side by side. To create and use a new venv switch Python versions using pyenv
then create a new venv.
>>> pyenv global <different Python version>
>>> poetry install
An error will be thrown if the Python version does not match the one specified in the pyproject.toml
however a range of Python versions can be specified using semantic versioning versioning.
I prefer my venv to be installed in a .venv
folder local to my project, this is similar to how npm
works and allows me to delete the folder and reinstall if anything strange happens or if (in the case of pipenv) I want to easily change which version of Python I'm using.
To enable this feature: set the following environment variable.
>>> export PIPENV_VENV_IN_PROJECT="enabled"
or
Simply create an empty .venv
folder in the root of your project before creating your pipenv environment.
This feature can be enabled using the following command:
>>> poetry config settings.virtualenvs.in-project true
But, beware that this will change the behavior of poetry
: it will no longer be possible to use quickly switch between different versions of Python: Even if the Python version is switched using pyenv
, all commands run using poetry run
will use the venv (and its associated Python version) that resides in the local directory.
Option to create virtual environments in the project root (.venv) #108
Packages are easily installed and automatically added to the Pipfile
and Pipfile.lock
files using:
>>> pipenv install [--dev] <package name>
The --dev
flag indicates a development dependency. Development dependencies will not be installed by default when using pipenv install
.
Local packages can also be installed, allowing you to work on them and see your changes immediately:
>>> pipenv install -e <path to local package>
Packages are easily installed and automatically added to the pyproject.toml
and poetry.lock
files using:
>>> poetry add [--dev] <package name>
The --dev
flag indicates a development dependency. Development dependencies will not be installed by default when using poetry install
or added to the package when publishing.
Local packages can also be installed, allowing you to work on them and see your changes immediately:
>>> poetry add --path <path to local package> <name of package>
I am not sure why the name of the package is needed, as it should already be defined by the local package. Also the author seems unconvinced about linking local packages in general (pip install -e . equivalent? #34) so this feature may get forgotten about over time.
To be clear, I'm referring to what npm
calls scripts, which is different to the scripts specified inside a setup.py
file.
When developing it is sometimes useful to set up shortcuts for commands that are difficult to remember, for example the command for running every test file in a directory is:
>>> python -m unittest discover -s <test_folder> -p '*_test.py'
It is much more convenient to have a shortcut to these sorts of commands.
This feature is supported: put the following into the Pipfile
:
[scripts]
test = "pipenv run python -m unittest discover -s tests -p '*_test.py'"
Not supported, and unlikely to be added in the future. See sdispater's comment to Add tasks #591.
It would be preferable to be able to publish to PyPI without crafting an additional setup.py
file, this would be possible if all the information needed to publish the package was contained within the package management files.
As far as I can tell this is where pipenv gets its bad reputation. setup.py
files are still needed to publish to PyPi and no, they are not auto-populated with the dependencies from the Pipfile
.
The recommended approach is to either copy the dependencies over manually when publishing, or to get the Pipfile to install the dependencies listed in the setup.py
files, however, the setup.py
is not automatically updated when running pipenv install <package name>
.
If you really want your Pipfile
to depend on your setup.py
file, this is how it's done:
>>> pipenv install '-e .'
Deployment to pypi? #2805, Pipenv: A Guide to the New Python Packaging Tool, and How can I integrate with setup.py? #209.
Yuck!
So ideally we want to derive a setup.py
file from the Pipfile
:
I found two existing packages that claim to do this:
pipenv-tools - But it hasn't been updated in two years, there's no code in the src
directory and I couldn't get it to work.
Pipenv-Setup - But it synchronises the Pipfile.lock
instead of the Pipfile
, and this is an antipattern. The lock file is meant for creating a reproducible environment, it is overly restrictive (e.g., by not allowing updates to dependencies) to be used for setup.py
. For this reason I didn't even try using it.
I quickly wrote a package that generates an install_requires.py
file that can be imported in a setup.py
file: pipenv2setup (it is untested on Windows).
For an example of how to use the package when publishing pipenv projects, see this GitHub repository:
An example of using pipenv and pipenv2setup to deploy to Pypi
Publishing your package using Poetry is really easy. You do not need a setup.py
file at all. Simply run:
>>> poetry publish [--build] [--username <username>] [--password <password>]
The published package can be installed using pip
, not just by other instances of Poetry.
For information about how to migrate from using a setup.py
to purely a pyproject.toml
file, see here: Create and Publish a Python Package with Poetry
PyCharm is able detect the venv by using the Piplock
files. However, adding new packages using the PyCharm interface will not modify the Piplock
files.
At the time of writing, PyCharm does not seem to be aware of any Poetry virtual environments or appear to parse the pyproject.toml
file in any way.
In Poetry you must specify versions of Python and packages using semantic versioning (have to use ~
and ^
, not >=
or <
).
Poetry runs using Python, but it doesn't work with older versions of Python. So to develop for an older version of Python: some commands have to be run with pyenv
set to >3.6, but then the pyenv
needs to be switched back to the older version to create the venv. It also appears that venvs have to be greater than 3.5.
subprocess.run AttributeError occurred in Python 3.4 #1223
Also I am not sure about Windows compatibility for Poetry.
To me the main differences between the poetry
and pipenv
lie in their usage of pyenv
and their abilities (or lack of) to publish straight to PyPI. There is also the lack of scripts in Poetry which I personally find frustrating.
I find that when using Poetry there is a lot more switching between Python environments using pyenv
. Although currently this can be mitigated by using a locally install venv. I know this limits my ability to quickly test my code in different Python environments, but there are other tools such as tox
to do that.
Publishing to PyPI using Poetry is so easy, I explained it in one line (above). Publishing to PyPI with pipenv
is a minefield, to explain it I had to link to an entire Git repository (above).