Python Packaging

[benne238]

Python Packaging

All info related to packaging the back end python scripts and the pros/cons of doing so will be posted within this issue.

[benne238]

Setup.py

setup.py is a basic python script that is used to package python scripts. It provides information about the package and is required to be saved within the directory of a project in order to package that project. The setup.py script itself can include a wide variety of information about a given python package, all of the different options that can be used in the setup.py script can be found here.

A very relatively basic setup.py script looks like this:

setup.py:

# Source: https://packaging.python.org/tutorials/packaging-projects/

import setuptools

with open("README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    name="example-pkg-YOUR-USERNAME-HERE", # Replace with your own username
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
)

Note: This tutorial recommends naming the package with a unique name by appending your username from PyPI (more on this later) to the end of the package name, as denoted by the comment in the above script

Features and uses of setup.py

According to https://docs.python.org/3/distutils/setupscript.html:

"The main purpose of the setup script is to describe your module distribution to the Distutils, so that the various commands that operate on your modules do the right thing"

Meaning that the setup.py does little more than describe the project.
https://godatadriven.com/blog/a-practical-guide-to-using-setup-py/

[benne238]

"Development Mode"

Installing a package as editable is a common thing to do, which allows for a local installation and access of the package.

Running:

pip install -e .

In the project directory allows for the package to be installed locally and become locally accessible. To run this command, however, a setup.py script must exist in the project directory.
See https://packaging.python.org/guides/distributing-packages-using-setuptools/#id70 for more information on this

[benne238]

Distribution Archives

Distribution archives allow for the distribution of python packages through pip. Creating these archives is relatively straightforward but first, setuptools and wheel need to be installed and/or upgraded:

python3 -m pip install --user --upgrade setuptools wheel

Then create a built distribution and source archive. Specifically, this command will create a directory dist directory, and create a .tar.gz and .whl file within that directory
Note: See here for more details on what source archives and built distributions are

python3 setup.py sdist bdist_wheel

Note: This command should be run within the project directory, where the setup.py script is located

[benne238]

TestPyPI

TestPyPI is a distribution platform that is designed more for testing python packages without being globally accessible through pip.

1. register for an account with TestPyPI: https://test.pypi.org/account/register/

2. Install twine. Twine will be used to upload the packages to PyPI, or in this case, TestPyPI

pip install twine

3. Upload the project to TestPyPI using twine

twine upload --repository testpypi dist/*

Note: This command should be run from the parent project directory, or specifically, where the setup.py script is located

4. Enter your TestPyPI username and password when prompted
Note: Alternatively, it is possible to create an access token instead, to do that, follow the steps outlined in this guide

Now it is possible to install this package directly from TestPyPI by using pip

pip install --index-url https://test.pypi.org/simple/ package_name

[benne238]

Summary

The current method of using pip install -e . is easier for our purposes, because it is used for more of a developmental. However, it is relatively easy to export the API to testPyPI, in which there are a series of simple commands to do this. The reason for uploading to testPyPI are more for hammering out any bugs before actually uploading the project to PyPI.

It is also worth noting that testPyPI is not permanent and that projects are deleted from testPyPI periodically. Uploading the project to PyPI itself is relatively straightforward, and is very similar to uploading a package to testPyPI.

Useful resources:

• https://packaging.python.org/tutorials/packaging-projects/
• https://packaging.python.org/guides/using-testpypi/
• https://twine.readthedocs.io/en/latest/

[campb303]

It seems that the benefit of packaging a Python module is to distribute it and the most significant file related to packaging is the setup.py script. Does this script allow us to do anything more than add metadata and environmental requirements? Also, are there any particular directory structures we should be looking into for packaging and maintaining a module?

[benne238]

Package directory structure

The directory structure for a given package can be as simple or as complex as wanted/needed

General structure

It is considered good practice to have a root folder that contains five things:

• a readme
• a license/agreement
• a gitignore
• a setup.py script
• a subdirectory where the actual module(s) is/are

__init__.py

If an entire directory is considered a package (which contains different python modules/scripts), it is possible to signify that by creating an empty file named __init__.py within that directory. I am uncertain how this works in conjunction with setup.py or even if they are meant to be used together, more research will have to be done on that.

https://dev.to/codemouse92/dead-simple-python-project-structure-and-imports-38c6
https://stackoverflow.com/questions/1471994/what-is-setup-py

[benne238]

Clarification on __init__.py and setup.py

__init__.py is used to signify that a package exists in the directory where __init__.py is located. However, in order for the __init__.py to have any affect, the directory must be explicitly included in the list of importable directories (the pythonpath variable I think) so for our purposes __init__.py is not useful. The guide included below goes as far to state that it is better to use setup tools "develop" mode (pip install -e .)
https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html#the-module-search-path

[benne238]

GitHub Packages

It is possible to host code packages directly on GitHub, publicly or privately. While the exact method of doing this needs to be researched further, GitHub packages should alleviate the problems associated with frontend and backend compatibility.

Edit:

GitHub only requires an existing repository to host a package. In this case, the backend and frontend would most likely have to be hosted on separate repositories.

GitHub Packages can be integrated with npm, meaning that npm can be used to publish and install packages directly to GitHub. This guide gives an overview of how it would be possible to create a package on a given repository using npm.

Note: These guides seem to be tailored to individuals with experience using and setting up npm, so further clarification will be needed from @campb303