-
Notifications
You must be signed in to change notification settings - Fork 0
Python Packaging #159
Comments
Setup.py
A very relatively basic setup.py: # Source: https://packaging.python.org/tutorials/packaging-projects/
import setuptools
with open("README.md", "r", encoding="utf-8") as fh:
long_description = fh.read()
setuptools.setup(
name="example-pkg-YOUR-USERNAME-HERE", # Replace with your own username
version="0.0.1",
author="Example Author",
author_email="author@example.com",
description="A small example package",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/pypa/sampleproject",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.6',
) Note: This tutorial recommends naming the package with a unique name by appending your username from PyPI (more on this later) to the end of the package name, as denoted by the comment in the above script Features and uses of
|
"Development Mode"Installing a package as editable is a common thing to do, which allows for a local installation and access of the package. Running:
In the project directory allows for the package to be installed locally and become locally accessible. To run this command, however, a |
Distribution ArchivesDistribution archives allow for the distribution of python packages through pip. Creating these archives is relatively straightforward but first,
Then create a built distribution and source archive. Specifically, this command will create a directory
Note: This command should be run within the project directory, where the |
TestPyPITestPyPI is a distribution platform that is designed more for testing python packages without being globally accessible through pip. 1. register for an account with TestPyPI: https://test.pypi.org/account/register/ 2. Install twine. Twine will be used to upload the packages to PyPI, or in this case, TestPyPI
3. Upload the project to TestPyPI using twine
Note: This command should be run from the parent project directory, or specifically, where the 4. Enter your TestPyPI username and password when prompted Now it is possible to install this package directly from TestPyPI by using pip
|
SummaryThe current method of using It is also worth noting that testPyPI is not permanent and that projects are deleted from testPyPI periodically. Uploading the project to PyPI itself is relatively straightforward, and is very similar to uploading a package to testPyPI. Useful resources: |
It seems that the benefit of packaging a Python module is to distribute it and the most significant file related to packaging is the |
Package directory structureThe directory structure for a given package can be as simple or as complex as wanted/needed General structureIt is considered good practice to have a root folder that contains five things:
|
Clarification on
|
GitHub PackagesIt is possible to host code packages directly on GitHub, publicly or privately. While the exact method of doing this needs to be researched further, GitHub packages should alleviate the problems associated with frontend and backend compatibility. Edit:GitHub only requires an existing repository to host a package. In this case, the backend and frontend would most likely have to be hosted on separate repositories. GitHub Packages can be integrated with npm, meaning that npm can be used to publish and install packages directly to GitHub. This guide gives an overview of how it would be possible to create a package on a given repository using npm. Note: These guides seem to be tailored to individuals with experience using and setting up npm, so further clarification will be needed from @campb303 |
GitHub Packages LimitationAfter conferring with @campb303, it would seem that GitHub packages is tailored for hosting npm packages, which is more for javascript than for python. Looking into alternatives for privately hosting python packages |
pypiprivatePypipprivate is comparable pypi in its use for managing python packages, but different in that it is hosted privately. This tool seems to be easily configured and seems to solve the problem of privacy that is needed to host the backend on the package manager. Features:
Possible Issues:
|
As discussed in Slack, using a private package repo seems like more effort than reward. Instead, we'll proceed with packaging our API code and installing it manually. |
How
|
Moving forward, the API should be separated from the frontend and at that point will need support for version numbers (from Once the API has been separated from the frontend and versioned, the single |
Migration to
|
Argument | Description |
---|---|
name |
name of the package |
version |
The version of the package as a string |
description |
description of the package |
py_modules |
the modules to be included in the package |
python_requires |
python version(s) that are required to be able to install the package |
Modules
When installing the webqueueapi package (locally at least), the script will successfully make ECNQueue
and api
importable without needing to reference the directory name (which in this case is "api" even though the modules are installed under "webqueueapi") These two import statements below achieve the same result:
# Example 1
from api import api, ECNQueue
# Example 2
import api, ECNQueue
I like that you're pulling a version number for a Raising an exception is useful in some cases but here we should be logging events rather than dying. Look at the venv-manager and how it implements logging as well as graceful exits when encountering a problem. |
further
|
BackgroundWe're currently looking at the following four tasks to be completed:
Task 1 is effectively complete. Tasks 2 and 3 are being worked on by @benne238 . Task 4 is becoming an issue. The ProblemThe current working process for installation of a custom PyLDAP library that removes the need for LibSASL is as follows:
The first approach to automating this process was to use pre-existing facilities to pass compile time options to PyLDAP. I found that pip allows arguments to be passed to the setuptools process via the However, as we want to move to a standalone package that can be installed by simply running What We KnowAfter looking through the verbose logs of
import sys, setuptools, tokenize; sys.argv[0] = '"'"'/path/to/setup.py'"'"'; __file__='"'"'/path/to/setup.py'"'"'; f=getattr(tokenize, '"'"'open'"'"', open)(__file__); code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"'); f.close(); exec(compile(code, __file__, '"'"'exec'"'"')) What Does This MeanBecause the raw code inside a package's IssuesWhile this will achieve the desired outcome, it creates some new problems:
QuestionsIs there a way to get the custom version of PyLDAP while still allowing pip to be aware of the package and other tools to know about the package's existence? |
After discussing this with Nickolas Hirschberg from CoSIT, the best solution forward might be to avoid the aforementioned installation manipulation to get python-ldap in favor of maintaining our own fork of python-ldap. This would be a temporary solution until we can either:
We can use our own fork of python-ldap via pip's support for installing packages from git. The syntax is as follows:
The same string can also be included in a pip requirements file like this:
Example Installing python-ldap version 3.3.1 from GitHub using pip
Example Installing python-ldap version 3.3.1 from GitHub using requirements file in a virtual environment python3 -m venv __venv__
source __venv__/bin/activate
echo "git+https://github.com/python-ldap/python-ldap.git@python-ldap-3.3.1#egg=pyldap==3.3.1" > requirements.txt
pip install -r requirements.txt |
An issue has been opened on the python-ldap GitHub. See: Allow optional libraries to be disabled via setup.py #402 |
A new repo for our fork of python-ldap has been created at: https://github.itap.purdue.edu/ECN/python-ldap. The readme file shows how to use this repo and the wiki explain how it was made as well as how to maintain it. |
Upstream work for supporting C constant configuration is happening in the python-ldap GitHub issue linked above. Summary so far is that building extensions via distutils is not directly accessible using setuptools and the distutils docs for building C exensions is nonexistent. |
Limits to hosting python packages on githubPackages must be stored in the root of the github repoA given python package must be stored in the root of the git repository, specifically,
The fix for this problem is to ensure limitations with the
|
Walk through on installing different versions of packages on githubThese files can be seen in the staging branch of the api repo Within the root directory of the repo, there exist 4 files for a test python package:
The contents of these files is as follows: hello.py def hello():
return "hello" world.py def world():
return "world" setup.py import setuptools
VERSION = "2.0.0"
setuptools.setup(
name="test-package",
version=VERSION,
description="Test Package",
py_modules=['hello', 'world'],
python_requires='>=3.6',
) Note: The version of this package is 2.0.0, as denoted in hello.py def hello():
return "Hello!"
def niceHello():
return "Hello there friend!" world.py def world():
return "world" setup.py import setuptools
VERSION = "2.1.0"
setuptools.setup(
name="test-package",
version=VERSION,
description="Test Package",
py_modules=['hello', 'world'],
python_requires='>=3.6',
) Note the changes between the two versions:
To install version 2.0.0, run this command in the python virtual environment:
Note: the To test the install, create this python script: import hello
import world
print(hello.hello())
#print(hello.niceHello())
print(world.world()) Note: pylint might state that the two import statements are erroneous, but that can be ignored as the script runs without error in the virtual environment Uncommenting the middle print statement from above will cause an error because the function The output should be
To test version 2.1.0, run this command:
and uncomment the print statement in the test_script file from above. The expected output is:
|
Backend workflowFor now, the method to "publish" a new version of a package is something like this:
The workflow does present a couple of steps that could be combined specifically 2 and 4. Currently there are two locations the version is stored: in |
Our API requires python-ldap to be built without SASL support. By default, python-ldap is built with this support and there are two methods of disabling this:
Both of these options require manual steps to be taken by the end user. To avoid this we've created a version of python-ldap with the first option above preconfigured and made it available on ITaP's GitHub (see ECN/python-ldap.) Packages in git be installed using Instead, we can use PEP 508's VCS format in setuptools's |
The API has now been packaged and released in its own repo with custom dependencies included via PEP 508. Closing this. |
Python Packaging
All info related to packaging the back end python scripts and the pros/cons of doing so will be posted within this issue.
The text was updated successfully, but these errors were encountered: