Merge pull request #29 from ECN/update-dev-environment-setup-guide

Update dev environment setup guide
ECN · May 18, 2021 · 072397e · 072397e
2 parents ecbfbd6 + 1fddf8e
commit 072397e
Show file tree

Hide file tree

Showing 5 changed files with 191 additions and 54 deletions.
diff --git a/docs/Dev Environment Setup Guide.md b/docs/Dev Environment Setup Guide.md
@@ -13,9 +13,6 @@ This document will walk you through the setup and configuration of a development
     These are already installed on the provided development machines. 
 
 - [git](https://git-scm.com/downloads)
-- [NodeJS](https://nodejs.org/en/) >= v14.11.0
-- npm >= v6.14.8
-- npx >= v6.14.8
 - [Python](https://www.python.org/downloads/) == 3.6.9
 
 ## Configuring Your SSH Keys
@@ -40,7 +37,8 @@ This will create the files `id_rsa` and `id_rsa.pub` in the `.ssh` folder inside
     /home/<user_name>
     ```
 
-Most CLI shells like PowerShell and bash use the <kbd> ~ </kbd> (tilda) key as a shortcut for your user's folder.
+!!! note
+    Most CLI shells like PowerShell and bash use the <kbd> ~ </kbd> (tilda) key as a shortcut for your user's folder.
 
 You can confirm these files were created by running:
 ```none
@@ -57,25 +55,42 @@ In your editor of choice, create the file `~/.ssh/config` and add this:
 Host campb303-w2vm1
   HostName w2vm1.ecn.purdue.edu
   User campb303
-  # Forward webqueue2 Frontend Port
-  LocalForward 3000 localhost:3000
   # Forward webqueue2 API Port
   LocalForward 5000 localhost:5000
-  # Forward webqueue2 Docs Port
-  LocalForward 6060 localhost:6060
+  # Forward webqueue2 API Docs Port
+  LocalForward 8000 localhost:8000
 ```
 
 The configuration above will allow you to connect to the development machine and automatically forward ports for development tools to work. Here's a bit more detail about what's going on:
 
-| Key          | Value                                                                            |
-|--------------|----------------------------------------------------------------------------------|
-| Host         | A friendly name to identify an SSH host by. This can be anything.                |
-| HostName     | The DNS name or IP address of the computer you're connecting to.                 |
-| User         | The name of your user on the computer you're connecting to.                      |
+| Key | Value |
+| - | - |
+| Host | A friendly name to identify an SSH host by. This can be anything. |
+| HostName | The DNS name or IP address of the computer you're connecting to. |
+| User | The name of your user on the computer you're connecting to. |
 | LocalForward | Forwards a port on your computer to a port on the computer you're connecting to. |
+
+Development machines are not publicly accessible. You'll need to be connected to WebVPN to connect to them. 
 
-!!! tip
-    You need to be connected to WebVPN before SSH'ing into a the development machine.
+??? tip "SSH Jump Hosts"
+    WebVPN works on all platforms but requires an extra step to get to work. On macOS/Linux, `pier.ecn.purdue.edu` can be used as an [SSH jump host](https://www.tecmint.com/access-linux-server-using-a-jump-host/) allowing for development machines to be connected to automatically when openning VS Code. 
+
+    Assuming you already have an SSH key configured for `pier.ecn.purdue.edu`, you can use a jump host by adding the following to `~/.ssh/config`:
+
+    ```
+    Host campb303-pier
+        HostName pier.ecn.purdue.edu
+        User campb303
+
+    Host campb303-w2vm1
+        HostName w2vm1.ecn.purdue.edu
+        ProxyJump campb303-pier
+        User campb303
+        # Forward webqueue2 API Port
+        LocalForward 5000 localhost:5000
+        # Forward webqueue2 API Docs Port
+        LocalForward 8000 localhost:8000
+    ```
 
 To test your configuration, run `ssh` followed by the `Host` value. When prompted for your password, enter your career account password and press <kbd>Enter</kbd>.
 
@@ -182,37 +197,26 @@ Now go to [github.itap.purdue.edu/settings/keys](https://github.itap.purdue.edu/
 ## Cloning and Opening the Repository
 Using the integrated terminal in VS Code, run:
 ```none
-git clone git@github.itap.purdue.edu:ECN/webqueue2.git
+git clone git@github.itap.purdue.edu:ECN/webqueue2-api.git
 ```
-This will create a `webqueue2` folder in your user's home directory. Open this directory using the "Open Folder" button:
+This will create a `webqueue2-api` folder in your user's home directory. Open this directory using the "Open Folder" button:
 
 ![Open Repo](images/Dev%20Environment%20Setup%20Guide/Open%20Repo.gif)
 
 !!! note
     VS Code will automatically reconnect to the last remote host used when it reopens. 
 
 !!! tip 
-    If you need to reconnect manually, there will now be an option to open the webqueue2 folder directly from the Remote Hosts menu:
+    If you need to reconnect manually, there will now be an option to open the webqueue2-api folder directly from the Remote Hosts menu:
 
     ![Remote Folder Open](images/Dev%20Environment%20Setup%20Guide/Remote%20Folder%20Open.png)
 
 ## Configuring VS Code
 
 ### Installing Extensions
-There are three groups of plugins to install depending on what kind of development you're doing:
-
-#### General plugins, (needed by everyone):
-
-- [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) for working with Python virtual environments and debugging
+- [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) for Python language support, IntelliSense, virtual environments and debugging
 - [Git Graph](https://marketplace.visualstudio.com/items?itemName=mhutchie.git-graph) for a more detailed view of git info
 - [Markdown Preview GitHub Styling](https://marketplace.visualstudio.com/items?itemName=bierner.markdown-preview-github-styles) for previewing Markdown as it will appear on GitHub
-
-#### Frontend Extensions:
-- [ES Lint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) for JavaScript linting
-- [Babel JavaScript](https://marketplace.visualstudio.com/items?itemName=mgmcdermott.vscode-language-babel) for JavaScript and JSX syntax highlighting and error checking
-- [MDX](https://marketplace.visualstudio.com/items?itemName=silvenon.mdx) to provide MDX syntax support
-
-#### Backend/API Extensions:
 - [Python Docstring Generator](https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring) for generating Google style docstrings according to section 3.8 of [Google's Python style guide](https://google.github.io/styleguide/pyguide.html#s3.8-comments-and-docstrings)
 
 ### Configuring Extensions
@@ -222,7 +226,7 @@ The Python extension supports virtual environments but needs to be told where th
 
 Create or modify the file `.vscode/settings.json` and add the following:
 ```none
-"python.pythonPath": "./api/venv/bin/python3"
+"python.pythonPath": "./venv/bin/python3"
 ```
 
 !!! tip
@@ -237,38 +241,59 @@ For consistentcy, we'll use a docstring template. Create or modify the file `.vs
 At this point, your `.vscode/settings.json` file should look like this:
 ```none
 {
-    "python.pythonPath": "./api/venv/bin/python3",
+    "python.pythonPath": "./venv/bin/python3",
     "autoDocstring.customTemplatePath": "./docstring-format.mustache"
 }
 ```
 
-## Configuring Tools
-Install npm dependencies:
+## Setting Up the Virtual Environment
+For development, we'll use a [Python Virtual Environment](https://realpython.com/python-virtual-environments-a-primer/) to isolate out changes and have a reproducible dev environment.
+
+In VS Code's integrated terminal:
+
+Create a virtual environment at `./venv/`:
 ```bash
-npm install
+python3 -m venv venv 
 ```
 
-Create the Python virtual environment:
+Activate the virtual environment:
 ```bash
-npm run venv:create
+source venv/bin/activate
 ```
 
-## Using Tools
-All of the tools in this project are accessible as an npm task so you can interact with them by running `npm run <task>`. The tasks are:
+!!! tip
+    To deactivate the virtual environment and use your system's Python interpreter, run:
+
+    ```bash
+    deactivate
+    ```
+
+Update pip within the virtual environment:
+```none
+pip install -U pip
+```
+
+Install the webqueue2 API within the virtual environemt:
+```none
+pip install -e .[all]
+```
+
+`-e` installs a package in [editable mode](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-e) which allows code changes to take effect without reinstalling a package.
+
+webqueue2 API has multiple [conditional dependencies](https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#id7):
+
+| Condition | Installation Command | Description |
+| - | - | - |
+| Production | `pip install webqueue2-api` | For use in production. Only installed needed packages. |
+| Development | `pip install webqueue2-api[dev]` | For use in development. Installs everything for production and extra packages for development. |
+| Documentation | `pip install webqueue2-api[docs]` | For use in creating documentation. Installs everything for production and extra packages for documentation. |
+| All | `pip install webqueue2-api[all]` | A shortcut for installing production, development and documentation dependencies. |
 
-| Task | Description |
+### API Commands
+| Command | Description |
 | - | - |
-| `frontend:start` | This will start a development server on [localhost:3000](http://localhost:3000). (If the server is on your local machine, this will also launch your default browser at that address.) As you save changes in the `public/` or `src/` directories you'll see your changes in the browser. (API requests are automatically proxied to the API server.) |
-| `frontend:start:docs` | This will start a local server on [localhost:6060](http://localhost:6060) showing you the React component documentation. As you change a components documentation file in `/src/components/<component_name>/<component_name>.md` you'll see your changes in the browser.
-| `frontend:build` | This will output a static bundle of the frontend in `build/` that can be put on any webserver. |
-| `frontend:build:docs` | This will output a static bundle of the frontend documentation in `styleguidist/frontend-docs` that can be put on any webserver. |
-| `frontend:kill` | Kills the runaway frontend process(es). |
-| `frontend:kill:docs` | Kills the runaway frontend docs process(es). |
-| `api:start` | This will start a local WSGI server on [localhost:5000](http://localhost:5000) to access the API. You can interact with the API using `curl` or [Postman](https://www.postman.com/) |
-| `api:start:docs` | This will start a local server on [localhost:8000](http://localhost:8000) to access the API docs. As you change API documentation files in `api/documentation/docs` you'll see your changes in the browser. |
-| `api:build:docs` | This will output a static bundle of the API documentation in `api/documentation/site` that can be put on any webserver. |
-| `api:kill` | Kills the runaway API process(es). |
-| `api:kill:docs` | Kills the runaway API docs process(es). |
-| `venv:create` | This will create a virtual environment in `api/venv` and install requirements from `api/requirements.txt`. |
-| `venv:delete` | This will delete the folder `api/venv`. |
-| `venv:reset` | This will run `venv:delete` then `venv:create`. |
+| `python3 -m webqueue2_api start-api` | Starts the API on [localhost:5000](http://localhost:5000). See [Getting Started](./api/Getting Started.md) for more info. |
+| `python3 -m webqueue2_api stop-api` | Stops the API. |
+| `python3 -m webqueue2_api restart-api` | Stops and starts the API. |
+| `mkdocs serve` | This will start a local server on [localhost:8000](http://localhost:8000) to access the API docs. As you change API documentation files in `./docs/` you'll see your changes in the browser. |
+| `mkdocs build` | This will output a static bundle of the API documentation in `./site/` that can be put on any webserver. |
diff --git a/docs/api/making-docs.md b/docs/api/making-docs.md
@@ -0,0 +1,10 @@
+## Making Docs
+
+The api documentation uses mkdocs to create documenation, which is based in markdown.
+
+All of the doucmentation can be found in the `docs` directory in the `webqueue2-api` repo, and all of the api specific documentation can be found in the `docs/api` directory.
+
+
+## Mkdocs configuration
+
+A basic version of mkdocs can be installed by using `pip install mkdocs`, however, this is not recomened as the current implementation makes use of several plugins that enahance the basic features of mkdocs. To install all of the dependencies for this, follow the steps for setting up the dev environment.
diff --git a/docs/awesome-pages.yaml b/docs/awesome-pages.yaml
@@ -2,4 +2,5 @@
 # See: https://github.com/lukasgeiter/mkdocs-awesome-pages-plugin
 nav:
     - index.md
+    - 'Dev Environment Setup Guide.md'
     - API: api
diff --git a/docs/package-structure.md b/docs/package-structure.md
@@ -0,0 +1,63 @@
+## API package structure
+
+The api can be broken down into 4 distinct parts:
+1. The logger
+2. The api
+3. The ECNQueue
+4. The initialization scripts
+
+### Logger
+
+The logger is its own subpackage and script which is used for outputting log information to both the terminal and a specified file. The logger uses the logging module to create log outputs with specified format.
+
+### API
+
+The api is the link between the ECNQueue and the frontend. The api continuously runs in the background, and takes urls matching a specified format from the front end and translates those as functions to be executed in the ECNQueue. The api uses flask and flask restful to do this
+
+### ECNQueue
+
+The ECNQueue subpackage is a parser in the backend for all of the items in stored in the queue directory. The ECNQueue parses the text documents in the queue and outputs structured json, which is then transfered to the fronted via the api.
+
+### The initialization scripts
+
+The initialization scripts take in user input (either via command line or wrapper script) and initialize the logger, ECNQueue, and API according to the user input. After initializing and implementing all of the necessary configurations, the initialization scripts start the api. These scripts use argparse to get user input and subprocess to launch the api via gunicorn. While not its own subpackage, the scripts are located directly in the `webqueue2_api` directory.
+
+
+## Structure
+
+```none
+webqueue2_api/  # webqueue2_api parent package
+│
+├── api/                # api subpackage
+│   ├── __init__.py
+│   ├── auth.py
+│   └── resources/
+│       ├── __init__.py
+│       ├── item.py
+│       ├── login.py
+│       ├── queue.py
+│       ├── queue_list.py
+│       └── refresh_access_token.py
+│
+├── ECNQueue/           # ecnqueue subpackge
+│   ├── __init__.py
+│   ├── Item.py
+│   ├── parser/
+│   │   ├── __init__.py
+│   │   └── parser.py
+│   ├── Queue.py
+│   └── utils.py
+│
+├── logger/             # logger subpackage
+│   └─── __init__.py
+│
+├── utils/              # utils subpackage
+│   ├── __init__.py
+│   └── tree.py
+│
+├── __init__.py
+├── __main__.py         # parses user input
+├── global_configs.py   # stores the config file location
+├── start.py            # returns a flask object ater initializing the user configs
+└── validate_arguments.py   # validates all of teh user passed arguments
+```
diff --git a/docstring-format.mustache b/docstring-format.mustache
@@ -0,0 +1,38 @@
+{{! Google Docstring Template }}
+{{summaryPlaceholder}}
+
+{{extendedSummaryPlaceholder}}
+
+Example:
+        [example]
+
+{{#parametersExist}}
+Args:
+{{#args}}
+        {{var}} ({{typePlaceholder}}): {{descriptionPlaceholder}}
+{{/args}}
+{{#kwargs}}
+        {{var}} ({{typePlaceholder}}, optional): {{descriptionPlaceholder}}. Defaults to {{&default}}.
+{{/kwargs}}
+{{/parametersExist}}
+
+{{#exceptionsExist}}
+Raises:
+{{#exceptions}}
+        {{type}}: {{descriptionPlaceholder}}
+{{/exceptions}}
+{{/exceptionsExist}}
+
+{{#returnsExist}}
+Returns:
+{{#returns}}
+        {{typePlaceholder}}: {{descriptionPlaceholder}}
+{{/returns}}
+{{/returnsExist}}
+
+{{#yieldsExist}}
+Yields:
+{{#yields}}
+        {{typePlaceholder}}: {{descriptionPlaceholder}}
+{{/yields}}
+{{/yieldsExist}}