pythonpackage-managerspipenvpython-poetry

Feature comparison between npm, pip, pipenv and Poetry package managers


How do the main features of npm compare with pip, pipenv and Poetry package managers? And how do I use those features of pipenv or Poetry?

This could primarily help someone transitioning from being JavaScript developer to a Python developer.

I've never used RubyGems, but it may also be useful to compare the main features of that to the main Python package managers. To help anyone coming from a Ruby background.


Solution

  • Overview

    Feature \ Package Manager npm pip pipenv poetry
    Access to main repo (i.e. Pypi/npm)
    Record top level dependencies Pipfile pyproject.toml
    Record development dependencies Pipfile pyproject.toml
    Lock versions of all dependencies Pipfile.lock poetry.lock
    Switch between interpreter versions nvm
    Direct publishing ✓*
    Run scripts Pipfile
    Editable local packages
    Integration with Intellij partial

    * Directly publishing with pipenv is possible if using a 3rd party dev dependency, and a script to tie it all together; see below.

    Disclaimer: I only have experiences of these packages managers on Unix-like systems (OS X in particular), pipenv markets itself as treating Windows is a first class citizen, I'm not sure how this works without pyenv, which is not available on Windows as far as I am aware.

    Basic Usage

    pipenv

    To get the most out of pipenv, pyenv should be installed. pipenv will be able to detect and use any version of Python installed with pyenv, even if it is not activated. For example, if a Pipfile has listed Python 3.4 as a requirement: to successfully run pipenv install, pyenv install 3.4.0 should be run first.

    To create a new Pipfile and venv (using Python 3.7.x):

    >>> pipenv --python 3.7
    

    If you are on linux you may have to use a slightly different command:

    >>> python3 -m pipenv --python 3.7
    

    Or, to install dependencies from an existing Pipfile.lock, use the command below. This command can also be used to create a Pipfile and venv (defaulting to the newest available Python version).

    >>> pipenv install
    

    To run commands within the created venv:

    >>> pipenv run <script or command>
    

    E.g.,

    >>> pipenv run python main.py
    

    Poetry

    Poetry still uses pyenv, but in a different way: The version of Python you wish to use must be activated before calling poetry install or poetry run.

    A pyproject.toml can be created using:

    >>> poetry init
    

    or a full directory structure can be created using:

    >>> poetry new <dir>
    

    Before we can install we must activate a version of Python that matches what is specified in the pyproject.toml file.

    >>> pyenv global <python version specified in pyproject.toml>
    

    Now we are able to create the venv using the command below, if a poetry.lock file is present it will install all the dependencies listed in it.

    >>> poetry install
    

    To run commands within the created venv:

    >>> poetry run <command>
    

    If we change the global Python version using pyenv we will no longer be able to run commands in the created venv. There is an exception to this if we use a locally created venv; see below.

    Running your code using different Python versions

    Sometimes it's good to check that your code will work on both Python 3.7 and Python 3.4. This is not something we can take for granted.

    pipenv

    Only possible by deleting the venv in recreating it using a different Python version:

    >>> rm -rf <path to venv>
    >>> pipenv --python <different python version>
    

    A warning will likely be displayed that the Python version of the venv does not match the one specified in the Pipfile, but as far as I can tell it is only a warning.

    Multiple parallel environments #1071

    Poetry

    Poetry is much better suited to this use case: Multiple venvs can be created side by side. To create and use a new venv switch Python versions using pyenv then create a new venv.

    >>> pyenv global <different Python version>
    >>> poetry install
    

    An error will be thrown if the Python version does not match the one specified in the pyproject.toml however a range of Python versions can be specified using semantic versioning versioning.

    Local venv

    I prefer my venv to be installed in a .venv folder local to my project, this is similar to how npm works and allows me to delete the folder and reinstall if anything strange happens or if (in the case of pipenv) I want to easily change which version of Python I'm using.

    pipenv

    To enable this feature: set the following environment variable.

    >>> export PIPENV_VENV_IN_PROJECT="enabled"
    

    or

    Simply create an empty .venv folder in the root of your project before creating your pipenv environment.

    poetry

    This feature can be enabled using the following command:

    >>> poetry config settings.virtualenvs.in-project true
    

    But, beware that this will change the behavior of poetry: it will no longer be possible to use quickly switch between different versions of Python: Even if the Python version is switched using pyenv, all commands run using poetry run will use the venv (and its associated Python version) that resides in the local directory.

    Option to create virtual environments in the project root (.venv) #108

    Installing Packages

    pipenv

    Packages are easily installed and automatically added to the Pipfile and Pipfile.lock files using:

    >>> pipenv install [--dev] <package name>
    

    The --dev flag indicates a development dependency. Development dependencies will not be installed by default when using pipenv install.

    Local packages can also be installed, allowing you to work on them and see your changes immediately:

    >>> pipenv install -e <path to local package>
    

    Poetry

    Packages are easily installed and automatically added to the pyproject.toml and poetry.lock files using:

    >>> poetry add [--dev] <package name>
    

    The --dev flag indicates a development dependency. Development dependencies will not be installed by default when using poetry install or added to the package when publishing.

    Local packages can also be installed, allowing you to work on them and see your changes immediately:

    >>> poetry add --path <path to local package> <name of package>
    

    I am not sure why the name of the package is needed, as it should already be defined by the local package. Also the author seems unconvinced about linking local packages in general (pip install -e . equivalent? #34) so this feature may get forgotten about over time.

    Running Scripts

    To be clear, I'm referring to what npm calls scripts, which is different to the scripts specified inside a setup.py file.

    When developing it is sometimes useful to set up shortcuts for commands that are difficult to remember, for example the command for running every test file in a directory is:

    >>> python -m unittest discover -s <test_folder> -p '*_test.py'
    

    It is much more convenient to have a shortcut to these sorts of commands.

    pipenv

    This feature is supported: put the following into the Pipfile:

    [scripts]
        test = "pipenv run python -m unittest discover -s tests -p '*_test.py'"
    

    Poetry

    Not supported, and unlikely to be added in the future. See sdispater's comment to Add tasks #591.

    Publishing to PyPI

    It would be preferable to be able to publish to PyPI without crafting an additional setup.py file, this would be possible if all the information needed to publish the package was contained within the package management files.

    pipenv

    As far as I can tell this is where pipenv gets its bad reputation. setup.py files are still needed to publish to PyPi and no, they are not auto-populated with the dependencies from the Pipfile.

    The recommended approach is to either copy the dependencies over manually when publishing, or to get the Pipfile to install the dependencies listed in the setup.py files, however, the setup.py is not automatically updated when running pipenv install <package name>.

    If you really want your Pipfile to depend on your setup.py file, this is how it's done:

    >>> pipenv install '-e .'
    

    Deployment to pypi? #2805, Pipenv: A Guide to the New Python Packaging Tool, and How can I integrate with setup.py? #209.

    Yuck!


    So ideally we want to derive a setup.py file from the Pipfile:

    I found two existing packages that claim to do this:

    1. pipenv-tools - But it hasn't been updated in two years, there's no code in the src directory and I couldn't get it to work.

    2. Pipenv-Setup - But it synchronises the Pipfile.lock instead of the Pipfile, and this is an antipattern. The lock file is meant for creating a reproducible environment, it is overly restrictive (e.g., by not allowing updates to dependencies) to be used for setup.py. For this reason I didn't even try using it.


    My Solution:

    I quickly wrote a package that generates an install_requires.py file that can be imported in a setup.py file: pipenv2setup (it is untested on Windows).

    For an example of how to use the package when publishing pipenv projects, see this GitHub repository:

    An example of using pipenv and pipenv2setup to deploy to Pypi

    Poetry

    Publishing your package using Poetry is really easy. You do not need a setup.py file at all. Simply run:

    >>> poetry publish [--build] [--username <username>] [--password <password>]
    

    The published package can be installed using pip, not just by other instances of Poetry.

    For information about how to migrate from using a setup.py to purely a pyproject.toml file, see here: Create and Publish a Python Package with Poetry

    IntelliJ or PyCharm integration

    pipenv

    PyCharm is able detect the venv by using the Piplock files. However, adding new packages using the PyCharm interface will not modify the Piplock files.

    Poetry

    At the time of writing, PyCharm does not seem to be aware of any Poetry virtual environments or appear to parse the pyproject.toml file in any way.

    Other points about Poetry

    Semantic versioning

    In Poetry you must specify versions of Python and packages using semantic versioning (have to use ~ and ^, not >= or <).

    See Semver: Tilde and Caret

    Bugginess

    Poetry runs using Python, but it doesn't work with older versions of Python. So to develop for an older version of Python: some commands have to be run with pyenv set to >3.6, but then the pyenv needs to be switched back to the older version to create the venv. It also appears that venvs have to be greater than 3.5. subprocess.run AttributeError occurred in Python 3.4 #1223

    Also I am not sure about Windows compatibility for Poetry.

    Conclusion

    To me the main differences between the poetry and pipenv lie in their usage of pyenv and their abilities (or lack of) to publish straight to PyPI. There is also the lack of scripts in Poetry which I personally find frustrating.

    I find that when using Poetry there is a lot more switching between Python environments using pyenv. Although currently this can be mitigated by using a locally install venv. I know this limits my ability to quickly test my code in different Python environments, but there are other tools such as tox to do that.

    Publishing to PyPI using Poetry is so easy, I explained it in one line (above). Publishing to PyPI with pipenv is a minefield, to explain it I had to link to an entire Git repository (above).