Python Packaging

[benne238]

Python Packaging

All info related to packaging the back end python scripts and the pros/cons of doing so will be posted within this issue.

[benne238]

Setup.py

setup.py is a basic python script that is used to package python scripts. It provides information about the package and is required to be saved within the directory of a project in order to package that project. The setup.py script itself can include a wide variety of information about a given python package, all of the different options that can be used in the setup.py script can be found here.

A very relatively basic setup.py script looks like this:

setup.py:

# Source: https://packaging.python.org/tutorials/packaging-projects/

import setuptools

with open("README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    name="example-pkg-YOUR-USERNAME-HERE", # Replace with your own username
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
)

Note: This tutorial recommends naming the package with a unique name by appending your username from PyPI (more on this later) to the end of the package name, as denoted by the comment in the above script

Features and uses of setup.py

According to https://docs.python.org/3/distutils/setupscript.html:

"The main purpose of the setup script is to describe your module distribution to the Distutils, so that the various commands that operate on your modules do the right thing"

Meaning that the setup.py does little more than describe the project.
https://godatadriven.com/blog/a-practical-guide-to-using-setup-py/

[benne238]

"Development Mode"

Installing a package as editable is a common thing to do, which allows for a local installation and access of the package.

Running:

pip install -e .

In the project directory allows for the package to be installed locally and become locally accessible. To run this command, however, a setup.py script must exist in the project directory.
See https://packaging.python.org/guides/distributing-packages-using-setuptools/#id70 for more information on this

[benne238]

Distribution Archives

Distribution archives allow for the distribution of python packages through pip. Creating these archives is relatively straightforward but first, setuptools and wheel need to be installed and/or upgraded:

python3 -m pip install --user --upgrade setuptools wheel

Then create a built distribution and source archive. Specifically, this command will create a directory dist directory, and create a .tar.gz and .whl file within that directory
Note: See here for more details on what source archives and built distributions are

python3 setup.py sdist bdist_wheel

Note: This command should be run within the project directory, where the setup.py script is located

[benne238]

TestPyPI

TestPyPI is a distribution platform that is designed more for testing python packages without being globally accessible through pip.

1. register for an account with TestPyPI: https://test.pypi.org/account/register/

2. Install twine. Twine will be used to upload the packages to PyPI, or in this case, TestPyPI

pip install twine

3. Upload the project to TestPyPI using twine

twine upload --repository testpypi dist/*

Note: This command should be run from the parent project directory, or specifically, where the setup.py script is located

4. Enter your TestPyPI username and password when prompted
Note: Alternatively, it is possible to create an access token instead, to do that, follow the steps outlined in this guide

Now it is possible to install this package directly from TestPyPI by using pip

pip install --index-url https://test.pypi.org/simple/ package_name

[benne238]

Summary

The current method of using pip install -e . is easier for our purposes, because it is used for more of a developmental. However, it is relatively easy to export the API to testPyPI, in which there are a series of simple commands to do this. The reason for uploading to testPyPI are more for hammering out any bugs before actually uploading the project to PyPI.

It is also worth noting that testPyPI is not permanent and that projects are deleted from testPyPI periodically. Uploading the project to PyPI itself is relatively straightforward, and is very similar to uploading a package to testPyPI.

Useful resources:

• https://packaging.python.org/tutorials/packaging-projects/
• https://packaging.python.org/guides/using-testpypi/
• https://twine.readthedocs.io/en/latest/

[campb303]

It seems that the benefit of packaging a Python module is to distribute it and the most significant file related to packaging is the setup.py script. Does this script allow us to do anything more than add metadata and environmental requirements? Also, are there any particular directory structures we should be looking into for packaging and maintaining a module?

[benne238]

Package directory structure

The directory structure for a given package can be as simple or as complex as wanted/needed

General structure

It is considered good practice to have a root folder that contains five things:

• a readme
• a license/agreement
• a gitignore
• a setup.py script
• a subdirectory where the actual module(s) is/are

__init__.py

If an entire directory is considered a package (which contains different python modules/scripts), it is possible to signify that by creating an empty file named __init__.py within that directory. I am uncertain how this works in conjunction with setup.py or even if they are meant to be used together, more research will have to be done on that.

https://dev.to/codemouse92/dead-simple-python-project-structure-and-imports-38c6
https://stackoverflow.com/questions/1471994/what-is-setup-py

[benne238]

Clarification on __init__.py and setup.py

__init__.py is used to signify that a package exists in the directory where __init__.py is located. However, in order for the __init__.py to have any affect, the directory must be explicitly included in the list of importable directories (the pythonpath variable I think) so for our purposes __init__.py is not useful. The guide included below goes as far to state that it is better to use setup tools "develop" mode (pip install -e .)
https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html#the-module-search-path

[benne238]

GitHub Packages

It is possible to host code packages directly on GitHub, publicly or privately. While the exact method of doing this needs to be researched further, GitHub packages should alleviate the problems associated with frontend and backend compatibility.

Edit:

GitHub only requires an existing repository to host a package. In this case, the backend and frontend would most likely have to be hosted on separate repositories.

GitHub Packages can be integrated with npm, meaning that npm can be used to publish and install packages directly to GitHub. This guide gives an overview of how it would be possible to create a package on a given repository using npm.

Note: These guides seem to be tailored to individuals with experience using and setting up npm, so further clarification will be needed from @campb303

[benne238]

GitHub Packages Limitation

After conferring with @campb303, it would seem that GitHub packages is tailored for hosting npm packages, which is more for javascript than for python. Looking into alternatives for privately hosting python packages

[benne238]

pypiprivate

Pypipprivate is comparable pypi in its use for managing python packages, but different in that it is hosted privately. This tool seems to be easily configured and seems to solve the problem of privacy that is needed to host the backend on the package manager.

Features:

• pypiprivate is installable directly through pip
• privately host python packages
• it is possible to use pip to install packages hosted on pypiprivate by modifying the arguments given to pip (See this section of the pypiprivate guide for more details)

Possible Issues:

• While it is advertised as being hassle free, it does need to be configured in order to function properly. The guide goes over what some of the configuration looks like, but it would appear that there needs to be additional configuration to setting up authentication so that only certain individuals can access the package

[campb303]

As discussed in Slack, using a private package repo seems like more effort than reward. Instead, we'll proceed with packaging our API code and installing it manually.

[campb303]

How __init__.py and setup.py Work Together

setup.py is a file that controls attributes about a package for distribution via PyPI, the Python Package Index. Because we do not intend to distribute out packages for webqueue2, we do not need this file.

__init__.py serves two functions. The first is that the present of an __init__.py file indicates that its parent directory is a package. In the example below, there are three directories: foo, bar and baz. Only baz is a package:

.
├── bar
│   └── nuke_launch_codes.txt
├── baz
│   └── __init__.py
└── foo
    └── i_pity_the.wav

The second function of the __init__.py file is that the contents of it are executed during every import statement. This can be useful for initializing package by, for example, detecting an operating system before setting values, or for automating tasks like importing submodules automagically.

Taking the baz package from before. Let's add two sub packages, beep and boop, to it so that the directory structure looks like this:

baz
├── __init__.py
├── beep
│   └── __init__.py
└── boop
    └── __init__.py

Assuming the baz package is installed, we can reach the beep and boop by explicitly importing them but not by importing their parent library, baz, and referencing them later:

# Works :)
import baz.beep
import baz.boop

baz.beep()
baz.boop()


# Doesn't Work :(
import baz

baz.beep()
baz.boop()

However, we can import the beep and boop subpackages in the baz package __init__.py to allow importing the baz package and accessing the beep and boop packages like this:

# baz/__init__.py
import .beep
import .boop

import baz

baz.beep()
baz.boop()

Moving forward we should setuo our API packages to follow the format of a single package that import all subpackages in its __init__.py file to encourage modular code and ease of use.

[campb303]

Moving forward, the API should be separated from the frontend and at that point will need support for version numbers (from setup.py. What are all of the other attributes that can be set in setup.py? Which might apply to us?

Once the API has been separated from the frontend and versioned, the single ECNQueue module should be separated into sub-packages for readability/maintainability.

[benne238]

Migration to setuptools from distutils.core.setup

The current setup.py uses distutils.core.setup in order to package the ECNQueue.py and api.py scripts

setup.py:

from distutils.core import setup
setup(name='webqueueapi',
      version='1.0',
      py_modules=['api', 'ECNqueue'],
      )

To follow current recommendations, as outlined here, setuptools will now be used as scripted below:
setup.py (updated):

import setuptools

versionString = ""
with open(".env", "r") as env:
    for line in env:
        if("VERSION" in line): 
            versionString = line
            break
    env.close()

#if no version was found
if (versionString == ""): raise Exception("No version defined in the .env file")

#get the version from the string
version = versionString.strip().split("=")[1]

setuptools.setup(
    name="webqueueapi",
    version=version,
    description="WebQueue2 backend parser and api scripts",
    py_modules=['api', 'ECNQueue'],
    python_requires='>=3.6',
)

This new setup script is similar to the original except that it now implements setuptools as well as several other things:

• obtains a version number from the .env file located in the api directory (note a new entry in the .env file):

  ...
  VERSION=1.0.0
  ...
  

• raises and exception if VERSION is not defined in .env
• creates a package name of webqueueapi
• imports the ECNQueue.py and the api.py modules
• requires a python version of 3.6 and above

setup() arguments

the setup() function takes many arguments, all of which are outlined here

The ones used in the current implementation are as follows:

Argument	Description
name	name of the package
version	The version of the package as a string
description	description of the package
py_modules	the modules to be included in the package
python_requires	python version(s) that are required to be able to install the package

Modules

When installing the webqueueapi package (locally at least), the script will successfully make ECNQueue and api importable without needing to reference the directory name (which in this case is "api" even though the modules are installed under "webqueueapi") These two import statements below achieve the same result:

# Example 1
from api import api, ECNQueue

# Example 2
import api, ECNQueue

[campb303]

I like that you're pulling a version number for a .env file however we should be using the dotenv package as we do in the API to read that file. Not only does it simplify our code but it manages different file formats, allows for advanced configuration and follows the no-throw policy we've already adopted. See the API and how it sets the JWT Secret for an example.

Raising an exception is useful in some cases but here we should be logging events rather than dying. Look at the venv-manager and how it implements logging as well as graceful exits when encountering a problem.

[benne238]

further setup.py changes

Implementation of logging was added to the setup.py and the structure of the webqueueapi package was changed as documented below:
wbequeueapi

api
    └───webqueueapi
        │  __init__.py
        │ api.py
        │ ECNQueue.py
        └ setup.py

Attempts were made to store the version number within a .env file within the package strucuture to more easily save and read it later on, however, .env and the variables defined in it are only accessible if an enviornment has been initialized, which does not occur when installing the package using pip. For this reason, a variable named VERSION was defined in setup.py and will act as the canonical version for the time being.
setup.py (updated):

import setuptools
from pathlib import Path
from os import environ, path
import logging

VERSION = "1.0.1"

# Configure the logger
logger_name = "webqueueapi_install_log"
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG)

# See Formatting Details: https://docs.python.org/3/library/logging.html#logrecord-attributes
# Example: Jan 28 2021 12:19:28 venv-manager : [INFO] Message
log_message_format = "%(asctime)s %(name)s : [%(levelname)s] %(message)s"
# See Time Formatting Details: https://docs.python.org/3.6/library/time.html#time.strftime
# Example: Jan 28 2021 12:19:28
log_time_format = "%b %d %Y %H:%M:%S"
log_formatter = logging.Formatter(log_message_format, log_time_format)

# Configure output to stdout
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(log_formatter)
stream_handler.setLevel(logging.INFO)
logger.addHandler(stream_handler)

# Configure out to logfile, located in '/tmp/webqueueapi install log.log'
log_file_path = path.abspath("/tmp/" + logger_name + '.log')
file_handler = logging.FileHandler(log_file_path)
file_handler.setFormatter(log_formatter)
logger.addHandler(file_handler)

try:
    VERSION = VERSION
except:
    logger.error("VERSION has not been defined")
    exit()

if (VERSION == None or VERSION == ""):
    logger.error("VERSION has no value")
    exit()

logger.debug("Attempting to install python package")

setuptools.setup(
    name="webqueueapi",
    version=VERSION,
    description="WebQueue2 backend parser and api scripts",
    py_modules=['api', 'ECNQueue'],
    python_requires='>=3.6',
)
logger.info("webqueueapi installed sucessfully")

Note: in order to get the output of logger to the terminal, the pip command must be run with the -v flag, which will output all of pip's logging as well. Additionally, when outputting the logs to the log file, each log entry is logged twice if pip install webqueue2-api/api/webqueueapi is used to install the package locally. https://stackoverflow.com/questions/55956806/why-is-setup-py-executed-twice-by-pip

[campb303]

Background

We're currently looking at the following four tasks to be completed:

1. Define a package formation/file structure.
2. Find a versioning strategy that allows for automating GItHub releases
3. Automating GitHub releases
4. Automate the installation of a custom PyLDAP installation

Task 1 is effectively complete. Tasks 2 and 3 are being worked on by @benne238 . Task 4 is becoming an issue.

The Problem

The current working process for installation of a custom PyLDAP library that removes the need for LibSASL is as follows:

1. Download PyLDAP source from GitHub
2. Modify setup.cfg to remove the LibSASL support from the build flags.
3. Build PyLDAP
4. Install PyLdap

The first approach to automating this process was to use pre-existing facilities to pass compile time options to PyLDAP. I found that pip allows arguments to be passed to the setuptools process via the --install-options flag but this doesn't help us because of PyLDAP's custom logic for building the compiler arguments is not directly controllable by setup.py and therefore unaddressable via the --install-options flag. This prompted the creation of the venv-manager as described here.

However, as we want to move to a standalone package that can be installed by simply running pip install webqueueapi and/or listing webqueueapi in a requirements file, the process automated by venv-manager needs to be moved into the setup.py build process.

What We Know

After looking through the verbose logs of pip install pyldap, I've found the following happens:

1. The latest version of the package is found on PyPI and downloaded
2. setup.py egg_info is run and generates several files, most importantly a dependencies.txt listing the pre-reqs for the package to be installed.
3. The pre-reqs are found, downloaded and installed.
4. The contents of the package setup.py are compiled and executed inline:

import sys, setuptools, tokenize; sys.argv[0] = '"'"'/path/to/setup.py'"'"'; __file__='"'"'/path/to/setup.py'"'"'; f=getattr(tokenize, '"'"'open'"'"', open)(__file__); code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"'); f.close(); exec(compile(code, __file__, '"'"'exec'"'"'))

What Does This Mean

Because the raw code inside a package's setup.py is run as part of pip install, this means we can add arbitrary code to the install process by adding it to setup.py. Therefore we can add the existing automated build process to the setup.py. However, the logging facilities and python path detection will need to be modified for this to work.

Issues

While this will achieve the desired outcome, it creates some new problems:

• Because our setup.py would be responsible for building and installing PyLDAP, pip can no longer do conflict resolution. We'll need to check for existing versions of PyLDAP before installing a custom one.
• PyLDAP will need to be removed from the requirements.txt file so the incompatible version is not installed from PyPI. This will break existing tools that do dependency resolution.

Questions

Is there a way to get the custom version of PyLDAP while still allowing pip to be aware of the package and other tools to know about the package's existence?

[campb303]

After discussing this with Nickolas Hirschberg from CoSIT, the best solution forward might be to avoid the aforementioned installation manipulation to get python-ldap in favor of maintaining our own fork of python-ldap. This would be a temporary solution until we can either:

• submit a PR to python-ldap allowing ldap compile options to be manipulated through setup() commands.
• migrate to a pure Python LDAP library like ldap3

We can use our own fork of python-ldap via pip's support for installing packages from git. The syntax is as follows:

pip install <VCS>+<PROTO>://<PATH>.git@<TAG>#egg=<PKG_NAME>==<VERSION>

Parameter	Description
VCS	The version control system to use (e.g. git or svn)
PROTO	The protocol used for retrieving files (e.g. HTTPS or FTP)
PATH	The path or URL to retrieve files from (e.g. GitHub or server address)
TAG	The tagged version to reference (e.g. 1.0.0)
PKG_NAME	The name of the package as it would be identified in a pip freeze (e.g. flask or pyldap)
VERSION	The specific version of the package to install. This is typically the same as the tag. (e.g. 1.0.0)

The same string can also be included in a pip requirements file like this:

...
some-package
<VCS>+<PROTO>://<PATH>.git@<TAG>#egg=<PKG_NAME>==<VERSION>
another-package == 1.2
...

Example Installing python-ldap version 3.3.1 from GitHub using pip

pip install git+https://github.com/python-ldap/python-ldap.git@python-ldap-3.3.1#egg=pyldap==3.3.1

Example Installing python-ldap version 3.3.1 from GitHub using requirements file in a virtual environment

python3 -m venv __venv__
source __venv__/bin/activate
echo "git+https://github.com/python-ldap/python-ldap.git@python-ldap-3.3.1#egg=pyldap==3.3.1" > requirements.txt
pip install -r requirements.txt

[campb303]

An issue has been opened on the python-ldap GitHub. See: Allow optional libraries to be disabled via setup.py #402

[campb303]

A new repo for our fork of python-ldap has been created at: https://github.itap.purdue.edu/ECN/python-ldap.

The readme file shows how to use this repo and the wiki explain how it was made as well as how to maintain it.

[campb303]

Upstream work for supporting C constant configuration is happening in the python-ldap GitHub issue linked above. Summary so far is that building extensions via distutils is not directly accessible using setuptools and the distutils docs for building C exensions is nonexistent.

[benne238]

Limits to hosting python packages on github

Packages must be stored in the root of the github repo

A given python package must be stored in the root of the git repository, specifically, setup.py must exist in the root of the repository. The steps for installing a python package using pip hosted on a git repo looks something like this:

1. clone the repository into a directory
2. run the setup.py script within the root of the cloned repo
3. once pip finishes configuring the packages, delete the cloned repo
   If setup.py is located in a subdirectory within the repository, or is missing all together, pip will fail with an error similar to this:

...
FileNotFoundError: [Errno 2] No such file or directory: '/home/pier/e/benne238/webqueue2-api/api/venv/src/test-package/setup.py'
...

The fix for this problem is to ensure setup.py is located directly in the root of the github repository, which might also slightly restructuring the backend python package.

limitations with the #<PKG_NAME>==<VERSION>

In theory, a specific version of a python package can be specified by appending ==<version_num> to the package name: like this:

pip install git+https://github.itap.purdue.edu/ECN/webqueue2-api.git#egg=test-package==1.0.0

in which a package named `test-package will be installed with version 1.0.0.

This method does not work and pip will instead install the version associated with the most recent commit, only issuing a warning. Because only the most recent and up-to-date version of the repo is accessed when running this command, any version other than what is current cannot be accessed with this method.

pip install git+https://github.itap.purdue.edu/ECN/webqueue2-api.git@testing#egg=test-package==1.0.0

...
WARNING: Requested test-package==1.0.0 from git+https://github.itap.purdue.edu/ECN/webqueue2-api.git@testing#egg=test-package==1.0.0, but installing version 1.1.0
...

Note: @testing was appended to the git in the pip install command to denote to install from the testing branch, not the default branch.

The fix for this is a little more straightforward: instead of requesting a version as described above, it should be possible to access a version this way:

pip install git+https://github.itap.purdue.edu/ECN/webqueue2-api.git@1.0.0#egg=test-package

This, however, only works if the repository is properly tagged to indicate which commits correspond to which versions.

[benne238]

Walk through on installing different versions of packages on github

These files can be seen in the staging branch of the api repo

Within the root directory of the repo, there exist 4 files for a test python package:

• setup.py
• __init__.py
• hello.py
• world.py

The contents of these files is as follows:

hello.py

def hello():
    return "hello"

world.py

def world():
    return "world"

setup.py

import setuptools

VERSION = "2.0.0"

setuptools.setup(
    name="test-package",
    version=VERSION,
    description="Test Package",
    py_modules=['hello', 'world'],
    python_requires='>=3.6',
)

Note: __init__.py is empty but is present as part of python packaging good practices
Obviously other directories and files exist within the repo as well, but these are not necessarily important to the package itself.

The version of this package is 2.0.0, as denoted in setup.py, However, in a separate commit, a different version is also defined, 2.1.0, with slightly modified files:

hello.py

def hello():
    return "Hello!"

def niceHello():
    return "Hello there friend!"

world.py

def world():
    return "world"

setup.py

import setuptools

VERSION = "2.1.0"

setuptools.setup(
    name="test-package",
    version=VERSION,
    description="Test Package",
    py_modules=['hello', 'world'],
    python_requires='>=3.6',
)

Note the changes between the two versions:

1. setup.py has different version
2. hello.py has an additional function and the original function returns "Hello!" instead of just "hello"

To install version 2.0.0, run this command in the python virtual environment:

 pip install git+https://github.itap.purdue.edu/ECN/webqueue2-api.git@2.0.0#egg=test-package --force-reinstall

Note: the --force-reinstall is to ensure any previous version already in the virtual environment are overwritten

To test the install, create this python script:
test_script

import hello
import world

print(hello.hello())
#print(hello.niceHello())
print(world.world())

Note: pylint might state that the two import statements are erroneous, but that can be ignored as the script runs without error in the virtual environment

Uncommenting the middle print statement from above will cause an error because the function niceHello() doesn't exist in the hello.py module in version 2.0.0.

The output should be

hello
world

To test version 2.1.0, run this command:

 pip install git+https://github.itap.purdue.edu/ECN/webqueue2-api.git@2.1.0#egg=test-package --force-reinstall

and uncomment the print statement in the test_script file from above. The expected output is:

Hello!
Hello there friend!
world

[benne238]

Backend workflow

For now, the method to "publish" a new version of a package is something like this:

1. make the necessary changes to the backend files
2. update setup.py to reflect the correct new version of the backend
3. commit and push the changes made to the backend to github
4. tag the commit with a name that exactly matches the version number

The workflow does present a couple of steps that could be combined specifically 2 and 4.

Currently there are two locations the version is stored: in setup.py and in the tag with the commit. This is a level of redundancy that is not needed and could potentially lead to discrepancies if either the tag, setup.py, or both, are not properly updated. It would be nice to be able to only store the version in setup.py or in the git tag and just pull the version to the other place, ensuring everything is in sync. Currently looking into ways to do this to avoid possible problems in the future.

[campb303]

Our API requires python-ldap to be built without SASL support. By default, python-ldap is built with this support and there are two methods of disabling this:

1. Remove the HAVE_SASL item from the defines option of the [_ldap] section of the setup.cfg file before building and installing the package.
2. Manually unsetting the C build options during the build_ext command as instructed in this GitHub issue (but I've not been able to get this to work.)

Both of these options require manual steps to be taken by the end user. To avoid this we've created a version of python-ldap with the first option above preconfigured and made it available on ITaP's GitHub (see ECN/python-ldap.) Packages in git be installed using pip's installation from VCS support. Packages in git can also be used as dependencies with setuptools' VCS dependency support. However, pip does not support setuptools' VCS dependency support as it was dropped in pip v19.

Instead, we can use PEP 508's VCS format in setuptools's install_requires keyword. This was proven to work yesterday and is being merged into webqueue2-api/staging today.

[campb303]

The API has now been packaged and released in its own repo with custom dependencies included via PEP 508. Closing this.