Add custom pyldap build to venv-manager

[campb303]

The venv-manager currently does the following:

1. Uses the system installation of python3 to setup a virtual environment
2. Updates pip if it's <19.x
3. Install requirements from requirements.txt

In addition to this it also allows for the removal and resetting of the virtual environment created.

Moving forward, the API will need to have a custom built version of pyldap that disables SASL support for use on templeton and dev machines. The process for this is detailed in this comment. This process should be automated via the venv-manager.

[campb303]

Automating the build of pyldap was added to venv-manager in this commit.

How It Works

The venv-manager behaves mostly the same as before in that it will:

1. Uses the system installation of python3 to setup a virtual environment
2. Updates pip if it's <19.x
3. Install requirements from requirements.txt

However, betweens steps 2 and 3, the following happens:

• Get a list of available pyldap releases from the GitHub API
• Checks for a pyldap version in the Python virtual environment requirements file
  • If it exists and is available from GitHub, the version from the requirements file is used
  • If it does not exist or it is not available from GitHub, the latest non-beta version is used
• The source code is downloaded to and extracted in /tmp
• The SASL requirements are removed from the build config
• pyldap is built using the virtual environment for webqueue2
• `pyldap is installed into the virtual environment for webqueue2

The other change is that when installing requirements, venv-manager no longer uses the requirements.txt file directly. Instead, the requirements.txt file is parsed into a space separated string excluding blank lines, comments and the pyldap entry and passed directly to pip. Example:

Before:

# requirements.txt

# Some comment
foobar
# Some other comment
requests >=1.*
pyldap == 2.2.1

pip install -r requirements.txt

After:

# requirements.txt

# Some comment
foobar
# Some other comment
requests >=1.*
pyldap == 2.2.1

pip install 'foobar' 'requests >=1.*'

How It Can Be Used

Because venv-manager is now critical to the deployment of webqueue2, a conversation was had about maximizing maintainability. With its current (and foreseeable complexity), its not a safe assumption that general web developers will be able to troubleshoot / maintain this tool leaving that task to the sysadmins.

To address this, the following options were considered:

• Automate the process of building pyldap with a pure bash script.

  • Biggest Pro: The general sysadmin is more likely to be familiar with bash than python making code fixes easier and allowing simpler integrations into existing workflows.
  • Biggest Con: bash scripts fragment the codebase, don't integrate with existing logging tools and introduce a separate place to manage dependency versions.

• Automate the process of building pyldap entirely in the existing python codebase.

  • Biggest Pro: Simpler codebase, canonical version management and seamless logging.
  • Biggest Con: Less likely to have people knowledgeable with the tooling to keep it running.

The solution was to write the logic in python for all the above pros but make the script behave like a bash script. This means a proper help message via venv-manager.py -h, logging printed to both stdout and a log file, debug details available via stdout with --debug flag and returning exit codes to indicate success or failure and more info on failure. Also, logging in Ubuntu syslog compliant.

Improvements

• Make return codes global. Currently, return codes are function specific and can be overridden by whatever function runs last. For example, a return code of 30 from building pyldap could be overridden by a return code of 12 while installing python requirements resulting in incorrect error reporting.
• Write error code documentation in the management docs.

[campb303]

Global error codes have been implemented. Docs still need written.

[campb303]

Secure strings needs implemented. Functional and usage docs needs written.

[campb303]

We've moved around the venv_manager in favor of hosting packages on our own GitHub repos and installing hem via standard pip and setuptool patterns. Closing.