diff --git a/Dev Environment Setup Guide.md b/Dev Environment Setup Guide.md index 21789fa..f2e6be4 100644 --- a/Dev Environment Setup Guide.md +++ b/Dev Environment Setup Guide.md @@ -1,119 +1,250 @@ -# Dev Environment Setup Guice +# Dev Environment Setup Guide +This document will walk you through the setup and configuration of a development environment for webqueue2 using the provided development machines, SSH authentication, GitHub and VS Code. + +# Table of Contents + + + +- [Prerequisites](#prerequisites) + - [On Your Computer](#on-your-computer) + - [On The Development Computer](#on-the-development-computer) +- [Configuring Your SSH Keys](#configuring-your-ssh-keys) +- [Configuring SSH](#configuring-ssh) +- [Adding SSH Keys](#adding-ssh-keys) +- [Installing VS Code](#installing-vs-code) +- [Connecting To The Development Machine](#connecting-to-the-development-machine) +- [Adding Your SSH Keys to GitHub](#adding-your-ssh-keys-to-github) +- [Cloning and Opening the Repository](#cloning-and-opening-the-repository) +- [Configuring VS Code](#configuring-vs-code) + - [Installing Extensions](#installing-extensions) + - [Frontend Extensions:](#frontend-extensions) + - [Backend/API Extensions:](#backendapi-extensions) + - [Configuring Extensions](#configuring-extensions) + - [Python](#python) + - [Python Docstring Generator](#python-docstring-generator) +- [Configuring Tools](#configuring-tools) +- [Using Tools](#using-tools) + + ## Prerequisites + +### On Your Computer - [VS Code](https://code.visualstudio.com/download) +- [Cisco AnyConnect for WebVPN](https://webvpn.purdue.edu/) + +### On The Development Computer +**Note:** These are already installed on the provided development machines. - [git](https://git-scm.com/downloads) -- [NodeJS](https://nodejs.org/en/) >= v14.2.0 -- npm >= v6.14.4 -- npx >= v6.14.4 -- [Python](https://www.python.org/downloads/) >= 3.7 +- [NodeJS](https://nodejs.org/en/) >= v14.11.0 +- npm >= v6.14.8 +- npx >= v6.14.8 +- [Python](https://www.python.org/downloads/) == 3.6.9 +## Configuring Your SSH Keys +We will be using SSH keys to authenticate to both the development machines and GitHub. -## Setup -- The webqueue2 API uses `gunicorn` as a WSGI server and this does not run on Windows. If you're developing on Windows use a virtual machine. +In either PowerShell on Windows or bash on Linux, run: +``` +ssh-keygen +``` +This will create the files `id_rsa` and `id_rsa.pub` in the `.ssh` folder inside your user's folder. Your user's folder can usually be found at: -- webqueue2 uses npm and create-react-app to manage itself. You can learn more about npm [here](https://nodesource.com/blog/an-absolute-beginners-guide-to-using-npm/) and create-react-app [here](https://create-react-app.dev/docs/getting-started/). +- `C:\Users\` (Windows) +- `/Users/` (macOS) +- `/home/` (Linux) ---- +Most CLI shells like PowerShell and bash use the tilda ~ key as a shortcut for your user's folder. -### Step 1: Close the Reposistory +You can confirm these files were created by running: ``` -git clone https://github.itap.purdue.edu/ECN/webqueue2.git +ls ~/.ssh/ ``` -**Note:** github.itap.purdue.edu is only accessible on Purdue's campus or via the [AnyConnect webvpn](https://webvpn.purdue.edu/). -### Step 2: Move to the project directory -``` -cd webqueue2 -``` +## Configuring SSH +In your editor of choice, create the file `~/.ssh/config` and add this: -### Step 3: Install npm Packages -``` -npm install +**Important:** Replace `campb303` with your career account username and replace `w2vm1` with the name of the provided development machine you're connecting to. + +```bash +Host campb303-w2vm1 + HostName w2vm1.ecn.purdue.edu + User campb303 + # Forward webqueue2 Frontend Port + LocalForward 3000 localhost:3000 + # Forward webqueue2 API Port + LocalForward 5000 localhost:5000 + # Forward webqueue2 Docs Port + LocalForward 6060 localhost:6060 ``` -#### Using npm -There are four npm scripts included in this project: +The configuration above will allow you to connect to the development machine and automatically forward ports for development tools to work. Here's a bit more detail about what's going on: -- `npm run start:frontend`: This will start a development server on [localhost:3000](http://localhost:3000). (If the server is on your local machine, this will also launch your default browser at that address.) As you save changes in /public and/or /src you'll see your changes in the browser. (API requests are automatically proxied to the API server.) +| Key | Value | +| - | - | +| Host | A friendly name to identify an SSH host by. This can be anything. | +| HostName | The DNS name or IP address of the computer you're connecting to. | +| User | The name of your user on the computer you're connecting to. | +| LocalForward | Forwards a port on your computer to a port on the computer you're connecting to. | -- `npm run start:api`: This will start a local WSGI server on [localhost:5000](http://localhost:5000) to access the API. **Note:** You will need to add Python to your PATH variable if it is not already. +**Note:** You need to be connected to WebVPN before SSH'ing into a the development machine. -- `npm run build`: This will build a static version of the site in /build ready to be placed in the document root of any web server. +To test your configuration, run `ssh` followed by the `Host`value. When prompted for your password, enter your career account password and press Enter. -- `npm run test`: This will run any tests (using the [Jest](https://jestjs.io/) tester). There are currently no defined tests. +**Note:** You will not see your password being typed but it is being typed. -### Step 4: Setup Python for API -Go to the API directory +For the configuration above you would run: ``` -cd api/ +ssh campb303-w2vm1 +campb303@w2vm1's password: ``` -Create Python virtual environment +## Adding SSH Keys +Replace `HOST` below with the value from above. + +**In PowerShell on Windows:** +```powershell +type $env:USERPROFILE\.ssh\id_rsa.pub | ssh HOST "cat >> .ssh/authorized_keys" ``` -python3 -m venv venv + +**In bash on macOS/Linux:** +```bash +ssh-copy-id HOST ``` -Activate the Python virtual environment +If the key was added successfully, you can login without entering a password by running: ``` -source venv/bin/activate +ssh HOST ``` -If you're pip version is less than 19, upgrade pip -``` -(venv) campb303@acererak [~/webqueue2/api] -$ pip --version -pip 9.0.1 from /home/pier/e/campb303/webqueue2/api/venv/lib/python3.6/site-packages (python 3.6) +## Installing VS Code +Download and install [VS Code](https://code.visualstudio.com/download). Be sure to add `code` to your PATH. + +**On Windows:** this is a a checkbox in the installer. + +![VS Code Install Options on Windows](https://i0.wp.com/www.techomoro.com/wp-content/uploads/2019/06/5.jpg?w=596&ssl=1) + +_Image from [this article on Techomoro](https://www.techomoro.com/installing-visual-studio-code-on-windows-10/)_ + +**On macOS/Linux:** this is accessible by searching for "PATH" in the Command Pallete. You can access the Command Pallete with the keyboard shortcut Command/Ctrl + Shift + P: + +![VS Code Install Options on macOS/Linux](https://i.stack.imgur.com/Ng886.png) + +_Image from [this StackOverflow thread](https://stackoverflow.com/questions/30065227/run-open-vscode-from-mac-terminal)_ + +## Connecting To The Development Machine +Install the [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) plugin. After installation a new icon will appear in the sidebar: + +![Remote SSH Plugin Icon](./docs/Dev%20Environment%20Setup%20Guide/Remote%20SSH%20Icon.png) + +- Click on the Remote SSH icon and you'll see the SSH host you just defined +- Select the host and click the New Window icon to connect. A new window connected to that host will appear + +**Note:** This may take a couple of minutes on the first connection while VS Code installs its server + +![Remote SSH Connection Example](./docs/Dev%20Environment%20Setup%20Guide/Connect%20to%20Remote%20SSH%20Host.gif) + +## Adding Your SSH Keys to GitHub +Because the development machine will be the computer that connects to GitHub, we need to create another SSH key and add it to our GitHub profile: -(venv) campb303@acererak [~/webqueue2/api] -$ pip install --upgrade pip -... -Successfully installed pip-20.2.2 +Using the integrated terminal in VS Code, run: +```bash +ssh-keygen ``` -Install Python requiements +This will create the files `id_rsa` and `id_rsa.pub` in `~/.ssh/`. You can confirm this by running: ``` -pip install -r requirements.txt +ls ~/.ssh/ ``` -**Note:** You can deactivate the Python virtual environment by running `deactivate`. +- Go to https://github.itap.purdue.edu/settings/keys +- Click the "New SSH Key" button +- Give the key a title that tells you what device the key is from (e.g. Desktop or Dev Machine) +- Paste the contents of `id_rsa.pub` into the key box +- Click the "Add SSH Key" button -### Step 5: Install VS Code extentions +![Add SSH Key to GitHub](docs/Dev%20Environment%20Setup%20Guide/Add%20SSH%20Key%20to%20GitHub.gif) -#### Installing Plugins -- [ES Lint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) for JavaScript linting -- [Babel JavaScript](https://marketplace.visualstudio.com/items?itemName=mgmcdermott.vscode-language-babel) for JavaScript and JSX syntac highlighting and error checking +## Cloning and Opening the Repository +Using the integrated terminal in VS Code, run: +```bash +git clone git@github.itap.purdue.edu:ECN/webqueue2.git +``` +This will create a `webqueue2` folder in your user's home directory. Open this directory using the "Open Folder" button: + +![Open Repo](./docs/Dev%20Environment%20Setup%20Guide/Open%20Repo.gif) + +**Note:** When you open VS Code next it will try to reconnect to the last SSH host it was connected to. + +**Note:** If you need to reconnect manually, there will not be an option to open this folder directly from the Remote Hosts menu: + +![Remote Folder Open](./docs/Dev%20Environment%20Setup%20Guide/Remote%20Folder%20Open.png) + +## Configuring VS Code + +### Installing Extensions +Install the following extensions: - [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) for working with Python virtual environments and debugging +- [Git Graph](https://marketplace.visualstudio.com/items?itemName=mhutchie.git-graph) for a more detailed view of git info +- [Markdown Preview GitHub Styling](https://marketplace.visualstudio.com/items?itemName=bierner.markdown-preview-github-styles) for previewing Markdown as it will appear on GitHub + +#### Frontend Extensions: +- [ES Lint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) for JavaScript linting +- [Babel JavaScript](https://marketplace.visualstudio.com/items?itemName=mgmcdermott.vscode-language-babel) for JavaScript and JSX syntax highlighting and error checking +- [MDX](https://marketplace.visualstudio.com/items?itemName=silvenon.mdx) to provide MDX syntax support + +#### Backend/API Extensions: - [Python Docstring Generator](https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring) for generating Google style docstrings according to section 3.8 of [Google's Python style guide](https://google.github.io/styleguide/pyguide.html#s3.8-comments-and-docstrings) -Optionally: -- [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) for directly editing files on remote machine -- [Markdown Preview GitHub Styling](https://marketplace.visualstudio.com/items?itemName=bierner.markdown-preview-github-styles) for previewing Markdown as it will appear on GitHub +### Configuring Extensions -## Configuring Environemnt -ES Lint, Babel JavaScript and Markdown Preview GitHub Styling will work without further configuration +#### Python +The Python extension supports virtual environments but needs to be told where the virtual environment is. -### Step 1: Configure VS Code's Python Extension -Create a folder called .vscode in the project root -``` -mkdir .vscode +Create or modify the file `.vscode/settings.json` and add the following: +```json +"python.pythonPath": "./api/venv/bin/python3" ``` -Create a file in the .vscode folder called settings.json -``` -nano .vscode/settings.json -``` +**Note:** The Python extension may complain and tell you to select another interpreter. Ignore this warning for now. + +#### Python Docstring Generator +For consistentcy, we'll use a docstring template. -Add the path to the virtual environment Python interpreter +Create or modify the file `.vscode/settings.json` and add the following: +```json +"autoDocstring.customTemplatePath": "./docstring-format.mustache" ``` + +At this point, your `.vscode/settings.json` file should look like this: +```json { - "python.pythonPath": "./api/venv/bin/python3" + "python.pythonPath": "/Users/justincampbell/GitHub/webqueue2/api/venv/bin/python3", + "autoDocstring.customTemplatePath": "./docstring-format.mustache" } ``` -For more information on the Python extension and details on how to use the debugger see: [Getting Started with Python in VS Code](https://code.visualstudio.com/docs/python/python-tutorial) for more info. -### Step 2: Configure Python Docstring Generator -In Settings > Extensions> Python Docstring Generator set the Custom Template Path to `./docstring-format.mustache` +## Configuring Tools +Install npm dependencies: +```bash +npm install +``` + +Create the Python virtual environment: +```bash +npm run venv:create +``` + +## Using Tools +All of the tools in this project are accessible as an npm task so you can interact with them by running `npm run `. The tasks are: -### Step 3 (Optional): Configure Remote - SSH Plugin -Remote - SSH will requires hosts to be added to your SSH config file. See [this guide](https://linuxize.com/post/using-the-ssh-config-file/) for more info. \ No newline at end of file +| Task | Description | +| - | - | +| `start:frontend` | This will start a development server on [localhost:3000](http://localhost:3000). (If the server is on your local machine, this will also launch your default browser at that address.) As you save changes in the `/public` or `/src` directories you'll see your changes in the browser. (API requests are automatically proxied to the API server.) | +| `start:api` | This will start a local WSGI server on [localhost:5000](http://localhost:5000) to access the API. You can interact with the API using `curl` or [Postman](https://www.postman.com/) | +| `start:docs` | This will start a local server on [localhost:6060](http://localhost:6060) showing you the React component documentation. As you change a components documentation file in `/src/components//.md` you'll see your changes in the browser. | +| `build:frontend` | This will output a static bundle of the frontend in `/build` that can be put on any webserver. | +| `build:docs` | This will output a static bundle of the frontend documentation in `/styleguidist/frontend-docs` that can be put on any webserver. | +| `venv:create` | This will create a virtual environment in `/api/venv` and install requirements from `/api/requirements.txt`. | +| `venv:delete` | This will delete the folder `/api/venv`. | +| `venv:reset` | This will run `venv:delete` then `venv:create`. | \ No newline at end of file diff --git a/README.md b/README.md index d7aa1ad..36e6329 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,12 @@ # webqueue2 A re-write of Purdue ECN's webqueue -![UI Snapshot](./docs/UI%20Snapshots/UI-Snapshot%202020-08-19%20at%202.10.48%20PM.png) +![UI Snapshot](./docs/UI%20Snapshots/UI-Snapshot%202020-09-22%20at%201.48.58%20PM.png) ## Stay Up To Date -See what's being worked on with [the webqueue2 Project](https://github.itap.purdue.edu/ECN/webqueue2/projects/4). +See what's being worked on with [the webqueue2 Project](https://github.itap.purdue.edu/ECN/webqueue2/projects/). See and participate in the conversation by viewing the [issues](https://github.itap.purdue.edu/ECN/webqueue2/issues). -## Tech Specs -Stay up to date with the [Dev Logs](https://github.itap.purdue.edu/ECN/webqueue2/blob/master/Dev%20Logs.md). - ## Get Involved Reading the [Dev Environment Setup Guide](https://github.itap.purdue.edu/ECN/webqueue2/blob/master/Dev%20Environment%20Setup%20Guide.md) diff --git a/api/ECNQueue.py b/api/ECNQueue.py index 1042cb6..3537bb3 100644 --- a/api/ECNQueue.py +++ b/api/ECNQueue.py @@ -181,145 +181,99 @@ def __parseSections(self) -> list: contentStart = self.__getHeaderBoundary() + 1 contentEnd = len(self.__rawItem) - 1 - # Find line numbers where sections start - sectionBoundaries = [ {"start": contentStart} ] - - directoryInfo = [] + #directoryInfo = {"type": "directoryInformation"} initialMessageContent = [] - endInitialMessage = False + initialMessageSection = True + + # Delimiter info + delimiters = [ + {"name": "edit", "pattern": "*** Edited"}, + {"name": "status", "pattern": "*** Status"}, + {"name": "replyToUser", "pattern": "*** Replied"}, + {"name": "replyFromUser", "pattern": "=== "}, + ] - # Parses the entire contents of the message, stores everything before any delimiter as the initial message - for lineNumber in range(contentStart, contentEnd + 1): + # Checks for Directory Identifiers + if self.__rawItem[contentStart] == "\n" and self.__rawItem[contentStart + 1].startswith("\t"): - line = self.__rawItem[lineNumber] + # Parses the directory information and returns a dictionary of directory values + directoryInfo = self.__directoryParsing(contentStart + 1) - if line.startswith("***") or line.startswith("===") and not line.startswith("===="): - - # Signifies that the inital message has been copletely parsed - endInitialMessage = True + # Appends Directory Information into the sections array + sections.append(directoryInfo) - # Stores what line every delimeter starts/ends - sectionBoundaries.append({"start": lineNumber}) + # Sets the initial message start to the next line after all directory lines and newlines + contentStart = contentStart + len(directoryInfo) + 1 - elif endInitialMessage == False: + #else: - # Delimiter not encountered yet, so append line to initial message list - initialMessageContent.append(line) - - # All possible Directory Items - directoryInfoPattern = [ - "\tName: ", - " Login: ", - " Computer: ", - " Location: ", - " Email: ", - " Phone: ", - " Office: ", - " UNIX Dir: ", - " Zero Dir: ", - " User ECNDB: ", - " Host ECNDB: ", - " Subject: " - ] - - # Reference to Remove Directory Items from initial message - directoryLinesToRemove = [] + # Initialize an empty dictionary for content + #directoryInfo["content"] = {} - # Line Counter - lineCounter = 0 + # Find line numbers where sections start + #sectionBoundaries = [ {"start": contentStart} ] - # Parses the initial message for directory information - for lineContents in initialMessageContent: + sectionBoundaries = [] - for itemsindirectory in directoryInfoPattern: - - # Checks if the line starts with any of the directory parameters - if lineContents.startswith(itemsindirectory): - - # Appends line number to be removed from initial message - directoryLinesToRemove.append(lineCounter) + # Set to true if a reply-from-user begining delimiter is parsed. Set to false if the ending delimiter is encountered + #replyFromUserDelimiter = False - # Adds the contents of the line to the directory info - directoryInfo.append(lineContents) + # Parses the entire contents of the message, stores everything before any delimiter as the initial message + # and the line number of any delimiters + for lineNumber in range(contentStart, contentEnd + 1): - # allows to move to the next iteration of the parent for loop, no need to continue parsing directory delimiters - break + line = self.__rawItem[lineNumber] - # Increment the line counter by after each line - lineCounter = lineCounter + 1 + if line.startswith("***") or line.startswith("===") and not line.startswith("===="): + for delimiter in delimiters: - # Parses the initial message to remove directory information - for lineNumber in sorted(directoryLinesToRemove, reverse=True): + if line.startswith(delimiter["pattern"]): - # Remove the directory line from the intital message - initialMessageContent.pop(lineNumber) + sectionBoundaries.append({"start": lineNumber, "type": delimiter["name"]}) + break + + # Signifies that the inital message has been completely parsed + initialMessageSection = False - # Removes unecessary newlines from the begining and the end of the initial message - - newLinebegining = True - newLineEnd = True + # Stores what line every delimeter starts/ends + #sectionBoundaries.append({"start": lineNumber, "type": "delimiter"}) - while (newLinebegining or newLineEnd) and len(initialMessageContent) > 1: - # Initializes the Length of Message content each iteration of the loop - initialmessagecontentLength = len(initialMessageContent) + #if replyFromUserDelimiter == False and line.startswith("===") and not line.startswith("===="): - # Checks if the last line is a newline - if initialMessageContent[initialmessagecontentLength -1] == "\n": + # Stores what line every delimeter starts/ends + # sectionBoundaries.append({"start": lineNumber, "type": "delimiter"}) + + #replyFromUserDelimiter = True - # Deletes the last line if it is a newline - initialMessageContent.pop(initialmessagecontentLength - 1) + # Checks for nesteded delimiters within the reply from user + #elif replyFromUserDelimiter == True and (line.startswith("===") or line.startswith("***")) and not line.startswith("===="): - # If the previous condition failed, then set the new line end to False if it isn't false already - elif newLineEnd == True: + #columnNum = 0 - newLineEnd = False + #errorMessage = "Nested delimiter encountered" - # Checks if the first line in message content is a newline - if initialMessageContent[0] == "\n": - - # Removes the first line in message content if it is a newline - initialMessageContent.pop(0) + #errorDictionary = self.__errorParsing(line, lineNumber, columnNum, errorMessage) - # If the previous condition failed, then set the new line begining to False if it isn't false already - elif newLinebegining == True: + # Appends the error dictionary to sections + #sections.append(errorDictionary) - newLinebegining = False + # Immediately exits the section parsing function because item content needs to be edited from the cli + #return sections - # Appends Directory Information into the sections array - sections.append( - {"type": "directoryInformation", - "content": directoryInfo} - ) - - # Gets the initial message date from the header - initialMessageDateStr = self.__getMostRecentHeaderByType("Date") + #elif replyFromUserDelimiter == False: - # Formats the initial message date to UTC - initialMessageFormattedDate = self.__getFormattedDate(initialMessageDateStr) - - # Stores list of dictionaries for CC information - initialMessageCCSection =[] + #sectionBoundaries.append({"start": lineNumber}) - # Parses the header looking for CC recipients of the initial message and stores it in a list of tuples - initialMessageCCList = email.utils.getaddresses([self.__getMostRecentHeaderByType("CC")]) + #elif line.startswith("===="): - # Parses the CC list and stores the cc recipient information in a list of dictionaries - for ccRecipients in initialMessageCCList: + #replyFromUserDelimiter = False - initialMessageCCSection.append( - {"name": ccRecipients[0], - "email": ccRecipients[1]} - ) + elif initialMessageSection == True: + + # Delimiter not encountered yet, so append line to initial message list + sectionBoundaries.append({"start": lineNumber, "type": "initial_message"}) - # Appends all initial message information to the sections array - sections.append( - {"type": "initialMessage", - "datetime": initialMessageFormattedDate, - "userName": self.__parseFromData(data="userName"), - "userEmail": self.__parseFromData(data="userEmail"), - "ccRecipients": initialMessageCCSection, - "content": initialMessageContent} - ) + initialMessageSection = False # Assignment Information assignedBy = "" @@ -350,15 +304,15 @@ def __parseSections(self) -> list: # Appends the assignment to the sections list sections.append( - {"type": "assign", - "by": assignedBy, + {"type": "assignment", "datetime": assignedDateTime, + "by": assignedBy, "to": assignedTo} ) sectionBoundaries.append({"start": contentEnd + 1}) - # Set line number where section end + # Sets the end of the section boundary to the begining of the next section boundary for boundaryIndex in range(0, len(sectionBoundaries) - 1): sectionBoundaries[boundaryIndex]["end"] = sectionBoundaries[boundaryIndex + 1]["start"] @@ -366,14 +320,6 @@ def __parseSections(self) -> list: # Remove End of File boundary del sectionBoundaries[-1] - # Different delimiters for different message events - delimiters = [ - {"name": "edit", "pattern": "*** Edited"}, - {"name": "status", "pattern": "*** Status"}, - {"name": "replyToUser", "pattern": "*** Replied"}, - {"name": "replyFromUser", "pattern": "=== "}, - ] - # Parses through all the boundaries in section boundaries for boundary in sectionBoundaries: @@ -395,12 +341,19 @@ def __parseSections(self) -> list: # Returns all of the lines within the current section sectionContent = self.__rawItem[boundary["start"] : boundary["end"]] + if boundary["type"] == "initial_message": + initialMessageDictionary = self.__initialMessageParsing(boundary["start"], boundary["end"]) + sections.append(initialMessageDictionary) + # Checks for each section type if sectionType == "edit": # Returns a dictionary with edit information editInfo = self.__editParsing(line) + # Remove the delimiter String and unecessary newlines + sectionContent = self.__getFormattedMessageContent(sectionContent) + # Appends content of the edit message to the dictionary editInfo["content"] = sectionContent @@ -414,6 +367,9 @@ def __parseSections(self) -> list: # Returns a dictionary with reply-to information replyToInfo = self.__replyToParsing(line) + # Removes the begining delimiter + sectionContent = self.__getFormattedMessageContent(sectionContent) + # Appends content of the reply-to message to the dicionary replyToInfo['content'] = sectionContent @@ -426,6 +382,9 @@ def __parseSections(self) -> list: # Returns a dictionary with status information statusInfo = self.__statusParsing(line) + + # Removes the begining delimiter + sectionContent = self.__getFormattedMessageContent(sectionContent) # Appends content to empty content key to avoid passing large amounts of info that isnt used within the function statusInfo['content'] = sectionContent @@ -440,15 +399,143 @@ def __parseSections(self) -> list: # Returns a dictionary with userReply information replyFromInfo = self.__userReplyParsing(sectionContent) - # Appends content to empty content key to avoid passing large amounts of info that isnt used within the function - replyFromInfo['content'] = sectionContent - # Appends the replyFrom to sections sections.append(replyFromInfo) return sections + def __directoryParsing(self, directoryStartLine: int) -> dict: + """Returns a dictionary with directory information + + Returns: dictionary: + "type": "directoryInformation" + "Name": name, + "Login": login, + "Computer": computer, + "Location": location, + "Email": email, + "Phone": phone, + "Office": office, + "UNIX Dir": unix_dir, + "Zero Dir": zero_dir, + "User ECNDB": user_ecndb, + "Host ECNDB": host_ecdbn, + "Subject": subject + """ + directoryInformation = {"type": "directory_information"} + + # Assumes a full directory with 12 items including the starting line + directoryEndingLine = directoryStartLine + 11 + + # Executies until the directory start line is greater than the directory ending line + while directoryStartLine <= directoryEndingLine: + + # Returns the line number at directory start line + info = self.__rawItem[directoryStartLine] + + # Breaks the loop if it encountrs a newline, signifying the end of the directory information + if info == "\n": + + break + + else: + + # Removes white including space, newlines, and tabs from the directory info line + strippedInfo = info.strip() + + # Attempts to find ": " but will accept ":", denoting a blank entry for a directory item + if ": " in strippedInfo: + + # Seperates the directory info line into two variables, the first variable being the key, the second being the value + key, value = strippedInfo.split(": ") + + # Adds the key value pair to the directory info dictionary + directoryInformation[key] = value + + elif ":" in strippedInfo: + + # Seperates the directory info line into two variables, the first variable being the key, the second being the value + key, value = strippedInfo.split(":") + + # Adds the key value pair to the directory info dictionary + directoryInformation[key] = value + + # Counter to denote the end of the directory + directoryStartLine = directoryStartLine + 1 + + # Returns the directory information dictionary + return directoryInformation + + def __initialMessageParsing(self, startLine: int, endLine: int) -> dict: + """Returns a dictionary with initial message information + + Returns: + dictionary: "type": "initial_message", + "datetime": utcdate, + "from_name": fromName, + "user_email": userEmail, + "to": [{email, name}], + "cc": [{email, name}], + "content": ["message_content"] + """ + initialMessageDictionary = { + #"type": "initial_message", + #"datetime": initialMessageFormattedDate, + #"from_name": self.__parseFromData(data="userName"), + #"user_email": self.__parseFromData(data="userEmail"), + #"to": initialMessageRecipientsSection, + #"cc": initialMessageCCSection, + #"content": initialMessageContent + } + + initialMessageDictionary["type"] = "initial_message" + + # Gets the initial message date from the header + rawMessageDateStr = self.__getMostRecentHeaderByType("Date") + + # Sets datetime in the intialMessage dictionary to UTC formatted date + initialMessageDictionary["datetime"] = self.__getFormattedDate(rawMessageDateStr) + + initialMessageDictionary["from_name"] = self.__parseFromData(data="userName") + + initialMessageDictionary["user_email"] = self.__parseFromData(data="userEmail") + + # Stores list of dictionaries for the recipients of the initial message + initialMessageDictionary["to"] = [] + + # Parses the header looking for recipients of the initial message and stores it in a list of tuples + rawMessageRecipientsList = email.utils.getaddresses([self.__getMostRecentHeaderByType("To")]) + + # Parses the CC list and stores the cc recipient information in a list of dictionaries + for recipients in rawMessageRecipientsList: + + initialMessageDictionary["to"].append( + {"name": recipients[0], + "email": recipients[1]} + ) + + # Stores list of dictionaries for CC information + initialMessageDictionary["cc"] = [] + + # Parses the header looking for CC recipients of the initial message and stores it in a list of tuples + rawMessageCCList = email.utils.getaddresses([self.__getMostRecentHeaderByType("CC")]) + + # Parses the CC list and stores the cc recipient information in a list of dictionaries + for ccRecipients in rawMessageCCList: + + initialMessageDictionary["cc"].append( + {"name": ccRecipients[0], + "email": ccRecipients[1]} + ) + + rawMessageContent = self.__rawItem[startLine : endLine] + + # Removes unecessary newlines from the begining and the end of the initial message + initialMessageDictionary["content"] = self.__getFormattedMessageContent(rawMessageContent) + + return initialMessageDictionary + def __editParsing(self, line: str) -> dict: """Returns a dictionary with edit information @@ -469,10 +556,10 @@ def __editParsing(self, line: str) -> dict: formattedDateTime = self.__getFormattedDate(dateTimeString) editInfo = { - "type": "edit", - "by": editedBy, - "datetime": formattedDateTime, - "content": "" + "type": "edit", + "datetime": formattedDateTime, + "by": editedBy, + "content": "" } return editInfo @@ -497,9 +584,9 @@ def __replyToParsing(self, line: str) -> dict: formattedDateTime = self.__getFormattedDate(dateTimeString) replyInfo = { - "type": "replyToUser", - "by": repliedBy, + "type": "reply_to_user", "datetime": formattedDateTime, + "by": repliedBy, "content": "" } @@ -526,8 +613,8 @@ def __statusParsing(self, line: str) -> dict: statusInfo = { "type": "status", - "by": updatedBy, "datetime": formattedDateTime, + "by": updatedBy, "content": "" } @@ -546,8 +633,13 @@ def __userReplyParsing(self, replyContent: list) -> dict: ccRecipientsList = [] newLineCounter = 0 - #Parses the section content looking for any line that starts with a metadata - for line in replyContent: + # Delimiter information line numbers to remove from reply from user + linesToRemove =[] + + + # Parses the section content looking for any line that starts with a metadata, also tracks the line + # number with the enumerate function + for lineNum, line in enumerate(replyContent): #Checks for a newline and breaks for loop on second occurance of a newline if line == "\n": @@ -562,6 +654,8 @@ def __userReplyParsing(self, replyContent: list) -> dict: # Matches everything after "Subject: " in the line subject = (re.search("(?<=Subject: )(.*)", line)).group() + linesToRemove.append(lineNum) + elif line.startswith("From: "): # Returns a list of tuples with name and email information @@ -573,14 +667,20 @@ def __userReplyParsing(self, replyContent: list) -> dict: # The email is stored in the second index of the tuple repliedByEmail = emailList[0][1] + linesToRemove.append(lineNum) + elif line.startswith("Date: "): # Matches everything after "Date: " - dateStr = (re.search("(?<=Date: )(.*)", line)).group() - + try: + dateStr = (re.search("(?<=Date: )(.*)", line)).group() + except: + dateStr = "" # Formatts the date to UTC formattedDateTime = self.__getFormattedDate(dateStr) + linesToRemove.append(lineNum) + elif line.startswith("Cc: "): # Returns a list of tuples with email information @@ -594,19 +694,97 @@ def __userReplyParsing(self, replyContent: list) -> dict: {"name":cc[0], "email":cc[1]} ) + + linesToRemove.append(lineNum) + + # Deletes reduntant lines from the message content in reverse order + for lineNum in sorted(linesToRemove, reverse = True): + replyContent.pop(lineNum) + + # Strips any unnecessary newlines or any delimiters frm the message content + replyContent = self.__getFormattedMessageContent(replyContent) replyFromInfo = { - "type": "replyFromUser", + "type": "reply_from_user", "datetime": formattedDateTime, - "subject": subject, - "userName": repliedByName, - "userEmail": repliedByEmail, - "content": "", - "ccRecipients": ccRecipientsList + #"subject": subject, + "from_name": repliedByName, + "from_email": repliedByEmail, + "cc": ccRecipientsList, + "content": replyContent } return replyFromInfo + def __getFormattedMessageContent(self, messageContent: list) -> list: + """Returns a list with message content that is stripped of unnecessary newlines and begining delimiters + + Returns: + list: formattedMessageContent + """ + # Parses looking for the reply-from-user ending delimiter adn removes the first line that contains the ending delimiter. + + + # Continually loops while looking at the first line of messageContent, removing it from messageContent + # if its a newline. Doesn't run if there is only one line in message content. + + while len(messageContent) > 1: + if messageContent[0] == "\n" or messageContent[0].startswith("***") or messageContent[0].startswith("===") : + messageContent.pop(0) + + else: + # Breaks the loop if the first line isn't a newline + break + + while len(messageContent) > 1: + + # Initializes the Length of messageContent each iteration of the loop + messagecontentLength = len(messageContent) + + # Checks if the last line is a newline + if messageContent[messagecontentLength -1] == "\n" or messageContent[messagecontentLength -1].startswith("===="): + + # Deletes the last line if it is a newline + messageContent.pop(messagecontentLength - 1) + + else: + # Breaks the loop if the last line isn't a newline + break + + return messageContent + + def __errorParsing(self, line: str, lineNum: int, lineColumn: int, errorMessage: str) -> dict: + """Returns a dictionary with error parse information + + Returns: + { + dictionary: "type": "parse_error", + datetime: time_of_execution, + content: [ + 'expected value item_row_num:item_column_num', + 'current line in item' + ] + } + """ + + errorDictionary = { + "type": "parse_error", + "datetime": self.__getFormattedDate(str(datetime.datetime.now())), + "content": [] + } + + # Error message with itemm line and column numbers + errorMessage = errorMessage + " at " + str(lineNum) + ":" + str(lineColumn) + + # Appends the error message to the content list in the error dictionary + errorDictionary["content"].append(errorMessage) + + # Appends the item line to the content list in the error dictionary + errorDictionary["content"].append(line) + + # returns the error dictionary + return errorDictionary + def __isLocked(self) -> Union[str, bool]: """Returns a string info about the lock if true and a bool False if false @@ -795,4 +973,10 @@ def getQueues() -> list: if isDirectory and isValid: queues.append(Queue(file)) - return queues \ No newline at end of file + return queues +if __name__ == "__main__": + item = Item("ce", 11) + print() +# for queue in getQueues(): +# for item in queue.items: +# print(f"${item.queue} ${item.number}") \ No newline at end of file diff --git a/docs/Dev Environment Setup Guide/Add SSH Key to GitHub.gif b/docs/Dev Environment Setup Guide/Add SSH Key to GitHub.gif new file mode 100644 index 0000000..b812c5b Binary files /dev/null and b/docs/Dev Environment Setup Guide/Add SSH Key to GitHub.gif differ diff --git a/docs/Dev Environment Setup Guide/Connect to Remote SSH Host.gif b/docs/Dev Environment Setup Guide/Connect to Remote SSH Host.gif new file mode 100644 index 0000000..3d644dd Binary files /dev/null and b/docs/Dev Environment Setup Guide/Connect to Remote SSH Host.gif differ diff --git a/docs/Dev Environment Setup Guide/Open Repo.gif b/docs/Dev Environment Setup Guide/Open Repo.gif new file mode 100644 index 0000000..08cd8e3 Binary files /dev/null and b/docs/Dev Environment Setup Guide/Open Repo.gif differ diff --git a/docs/Dev Environment Setup Guide/Remote Folder Open.png b/docs/Dev Environment Setup Guide/Remote Folder Open.png new file mode 100644 index 0000000..6a53201 Binary files /dev/null and b/docs/Dev Environment Setup Guide/Remote Folder Open.png differ diff --git a/docs/Dev Environment Setup Guide/Remote SSH Icon.png b/docs/Dev Environment Setup Guide/Remote SSH Icon.png new file mode 100644 index 0000000..32f210c Binary files /dev/null and b/docs/Dev Environment Setup Guide/Remote SSH Icon.png differ diff --git a/docs/Example GIF/Example GIF.gif b/docs/Example GIF/Example GIF.gif new file mode 100644 index 0000000..d1fea10 Binary files /dev/null and b/docs/Example GIF/Example GIF.gif differ diff --git a/docs/UI Snapshots/UI-Snapshot 2020-09-22 at 1.48.58 PM.png b/docs/UI Snapshots/UI-Snapshot 2020-09-22 at 1.48.58 PM.png new file mode 100644 index 0000000..f9226f2 Binary files /dev/null and b/docs/UI Snapshots/UI-Snapshot 2020-09-22 at 1.48.58 PM.png differ diff --git a/utils/venv-manager.py b/utils/venv-manager.py index d19da5c..6de0bd8 100644 --- a/utils/venv-manager.py +++ b/utils/venv-manager.py @@ -102,7 +102,7 @@ def run_logged_subprocess(command: Union[str, list], timeout: int = 10, shell: b """ logger.debug(f"Entering subprocess for '{command}'") with subprocess.Popen(command,\ - stderr=subprocess.STDOUT, stdout=subprocess.PIPE, shell=shell, text=True)\ + stderr=subprocess.STDOUT, stdout=subprocess.PIPE, shell=shell, universal_newlines=True)\ as logged_shell_process: subprocess_log_prefix = f"(PID: {logged_shell_process.pid})"