Package Structure

[benne238]

Currently, there are only two modules that make up webqueue2_api:

1. ECNQueue.py
2. api.py

However ECNQueue.py and api.py both have several moving parts that should be separated out into various packages and/or modules.

ECNQueue.py has 3 distinct parts:

1. The item class which stores the basic python structure of an item
2. The queue class, which is a collection of multiple items
3. The parser, which interacts with the items stored on pier

api.py has 5 distinct parts:

1. logging in, which allows an authorized user to access webqueue2
2. generating refresh tokens, which allows for an authorized user to stay logged in for a longer period of time
3. get the json representation of an item (from ECNQueue.py)
4. get the json representation of a queue (from ECNQueue.py)
5. get the json representation of all the queues and the number of items in each queue (from ECNQueue.py)

As a preliminary structure, it might make sense to make ECNQueue and api sub-packages of webqueue2_api, with their respective functions outlined above, broken into their own modules:

├─ setup.py
│
├───webqueue2_api
│   ├───api
│   │   ├───__init__.py
│   │   ├───login.py
│   │   ├───token_refresh.py
│   │   ├───get_item.py
│   │   ├───get_queue.py
│   │   └───get_queue_list.py
│   │     
│   └───webqueueapi
│       ├───__init__.py
│       ├───Item.py
│       ├───parser.py
│       └───Queue.py
│
└─ __init__.py

[campb303]

Some changes:

• Change the webqueueapi package to "ECNQueue"
• Add "utilities" module to ECNQueue package
• Add "resources" package inside API
• Start work on making our API a Flask extension: https://flask.palletsprojects.com/en/1.1.x/extensiondev/

[campb303]

[campb303]

Passing Config Options via Environment Variables

In order to keep ECNQueue separate from the API and let the end user configure ECNQueue without directly editing code, we need a way to pass config options from a management script to library code. We can do this using a combination of the python-dotenv library and module level __init__.py files.

Because a package's __init__.py file is run on import with an execution context of where its importing script is run (or where the interpreter is started from), file paths are relative to the importing script.

For example:
If we have a package called stork with a single __init__.py file inside it like this:

stork/
└── __init__.py

and the __init__.py references a file with a relative path like this:

# __init__.py

# A made up function
load_file("file.config")

if the module is imported from a script called run.py then the file.config file needs to exist in the same directory as run.py.

Because of this relative import, we can load environment variables from files next to a management script.

[campb303]

Suggested Package Structure:

.
├── ECNQueue
│   ├── Item.py                         # Item Class
│   ├── Queue.py                        # Queue Class
│   ├── __init__.py                     # Load Environment, Store Globals (queue_dir, queues_to_ignore), re-export utils
│   ├── parser                          # Split current parser code
│   │   └── __init__.py
│   └── utils.py                        # Utility functions
└── api
    ├── __init__.py                     # Load Environment, Initialize Flask and API objects, Register Resources
    ├── __main__.py                     # argparse, start|stop|restart for api via gunicorn
    ├── auth.py                         # user_is_valid
    └── resources                       # Stores resources
        ├── __init__.py
        ├── item.py
        ├── login.py
        ├── queue.py
        ├── queue_list.py
        └── refresh_access_token.py

[benne238]

Logger

We need to implement a logger in the top level __init__.py so that the logger is accessible throughout the entire package. However, due to time constraints, this will be unable to be implemented tonight.

[benne238]

Custom exceptions

There are several configuration values that need to be correctly configured. However, in order to do this, it makes the most sense to define a custom exception classes that is able to handle the various problems that could occur when trying set the various api configuration values.

exceptions.py

class exceptionName(Exception):
    pass

caller.py

from exceptions import exceptionName
raise exceptionName("ope! It done broke")

The exceptions.py script creates an exception class, and the caller.py raises that exception. The resulting output will be this:

Traceback (most recent call last):
  File "caller.py", line 2, in <module>
    raise exceptionName("ope! It done broke")
exceptions.exceptionName: ope! It done broke

[benne238]

configparser

configparser replaces our previous use of os.environ in order to get environment variables. In this case, we now have a structured environment file that can be created in order to pass certain configurations to ECNQueue, webqueue2-api, and the Logger. The name of the config file is webqueue2-api.cfg and it should be located next to the wrapper script:

[webqueue2_api]
[webqueue2_api]
JWT_SECRET_KEY = sshhhhhhh, its a secret!
ENVIRONMENT = dev

[ECNQueue]
QUEUES_TO_IGNORE = ["archives", "drafts", "inbox", "coral"]
QUEUE_DIRECTORY = /home/pier/e/queue/Mail

[Logger]
LOGGER_OUT_FILE = /tmp

[benne238]

A note on import statements in the webqueue2_api.api package

__main__.py, in context of the api sub-package, contains a simple script to run the api:

__main__.py:

from .__init__ import app

app.run()

To run the api via the __main__.py script, the python3 command can be used to run sub package directly:

python3 -m webqueue2_api.api

However, it is also possible to run the api in a similar manner by using a wrapper script that functions in a similar way, the big difference being that the wrapper script can contain any code that the user wishes to place in there:

wrapper.py:

from webqueue2_api.api import app

app.run()

It is important to note that whenever an import statement is used, the __init__.py script will run in what ever package/module that is being imported, regardless if __init__.py itself is being imported.

However, the __main__.py script will run, but only if it is explicitly imported: like this

wrapper.py:

from webequeue2_api.api import __main__

Note: since __main__ is being imported from webqueue2_api.api, the __init__.py scripts located in the webqueue2_api directory and the webqueue2_api/api directory, will both run.

In all three of the examples above: the output will be exactly the same and resemble something like this:

 * Serving Flask app "webqueue2_api.api.__init__" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Issues with "duplicate" imports

With many different way to start the api and with many different import statements happening before code is even executed, it is possible to have issues with the api being run twice. For example. the resulting script would run the api twice:

wrapper.py

from webqueue2_api.api import app
from webqueue2_api.api import __main__
app.run()

output:

 * Serving Flask app "webqueue2_api.api.__init__" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
^C * Serving Flask app "webqueue2_api.api.__init__" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

In addition to this, it is also pertinent that the __init__.py script not contain an import statement that imports the __main__.py script because if the __main__.py script were run directly whether by using the python3 -m webqueue2_api.api command or by a wrapper script, the __init__.py script will run first, which will cause the __main__.py script to be imported, then subsequently run and when the user attempts to exit, the remaining code in the __init__.py script will be executed then the main script will be executed an additional time, resulting in similar output as documented above.

[benne238]

How it works right now

At this point in time, it is possible to start the api with gunicorn from the command line.
BUT
There are significant limitations at this time: command line arguments passed via python3 -m webqueue2_api {args here} start-api are completely ignored except for the verbosity flag `-v'

It is possible to stop the api if it is running by running: python3 -m webqueue2_api stop-api

To effectively run the api with any desired configurations, you must create a config file that resembles this:
webqueue2-api.cfg:

[webqueue2_api]
JWT_SECRET_KEY = secretstuffhere
ENVIRONMENT = dev

[ECNQueue]
QUEUES_TO_IGNORE = ["archives", "drafts", "inbox", "coral"]
QUEUE_DIRECTORY = /home/pier/e/queue/Mail

[Logger]
LOGGER_OUT_FILE = /tmp/webqueue2_api.log

Note: make sure to name the file webqueue2-api.cfg

Then, run the following command in the same directory as the config file that was just created: python3 -m webqueue2_api start-api

The api is designed (if configurations are omitted from the command line) to fall back and check the current directory for a file with the name webqueue2-api.cfg

Following the above steps will create an api with the desired configurations.

Next steps

• Argparse arguments are not being effectively used and are essentially ignored and I have to figure out why that is the case
• I do not know how to start the api from a wrapper script or if it can be done at this point in time
• Gunicorn will currently run in the background due to the --daemon flag as seen in the __main__.py module. However, this prevents output from the logger from being put on the screen and only into the log file.
• Make the code prettier. There are a ton of moving parts and scripts referencing other scripts and a global config file that stores information. All of this needs to be smoothed out because the current readability of the code leaves a lot to be desired.