Automation API documentation generation

[campb303]

Python in this project adheres to section 3.8 of Google's guidelines for documenting python.

Sphinx is a popular documentation tool for Python and could be used to auto generate documentation.

This will depend on #24

[benne238]

Using Sphinx

Sphinx is essentially a file generator. Given inputs, it automatically creates human readable files (html being one of the type of files it can generate)
However, for our purposes, we want to place a heavier emphasis on the autogenerated documentation that sphinx creates.
Sphinx can pull information directly from python docstring comments for functions within a given python script. This seems to be the best advantage of creating documentation for the api (a basic demonstration of the autogenerate capabilities can be found here (start at 6:25)

Note:
It would seem however that anything we want to create that isn't a python docstring would have to be done manually using sphinx. Currently looking to see if that is actually the case...

[benne238]

Some more videos and documentation for sphinx

https://www.youtube.com/watch?v=b4iFyrLQQh4: a simple demonstration on creating an auto generation doc

https://realpython.com/lessons/setting-sphinx-example-project-and-scaffolding/: similar to the above, focuses on the sphinx-quickstart command (which we will most likely use) and goes in depth on the different options that can be enabled with the command

https://www.sphinx-doc.org/en/master/usage/quickstart.html: Sphinx's documentation on how to get started with Sphinx (which looks like it was generated using Sphinx I might add). Is complex for documentation, but makes sense after watching some of the tutorial videos before hand.

[benne238]

Installed sphinx in the virtual environment using pip

Note: might need to add sphinx to the requirements list depending on how we plan to use it...

Worked on trying to build an html autogenerated file based on the docstrings of the Item class in ECNQueue.py, though will have to do more research on how to automatically produce out put for "private" functions for the Item class.
Generally speaking, the only thing that can be automatically generated are functions and the docstrings to those corresponding functions. Sphinx does help with styling to make everythig look uniform, but any other documentation that would need to be done might be best done using a markdown README file

[benne238]

Docusaurus seems to be an alternative to Sphinx in that it can autogenerate a documentation website. However, Docusaurus relies on manual documentation in that Sphinx autodoc feature that collects the docstrings from python functions.

https://v2.docusaurus.io/docs/installation: installation of Docusaurus
https://www.youtube.com/watch?v=dn4dgA51WNg: demonstration of Docusaurus

[benne238]

Installation and configuration of Sphinx

1. Run npm run venv:reset
2. Run pip install Sphinx in the virtual environment
3. Within the project directory, create a documentation directory that will be used by sphinx for build and configuration files.
4. cd into that newly created directory and run sphinx-quickstart
5. Accept all of the default values (the values in brackets) and give the project a title, author and version number
6. The following step will create several files and directories within the documentation folder that was created. Open the conf.py file.
7. Within the conf.py file uncomment lines 13, 14, and 15:

# import os
# import sys
# sys.path.insert(0, os.path.abspath('..'))

Ensure that the path points to the location of the script to be documented (in my case, it is in the parent directory .. from the current directory
8. Also within conf.py, ensure that the extensions list on lines 33 and 34 includes 'sphinx.ext.autodoc':

extensions = [
'sphinx.ext.autodoc'
]

9. Save the configuration and close the file

[benne238]

Creating Sphinx Auto Documentation

1. index.rst determines what files and folders are processed when generating documentation.
2. Within the documentation folder, create an additional folder called docfiles or something meaningful. This folder will ultimately contain the rst files that autogenerate documentation for python scripts by using doc strings.
   *Any rst files created within the parent documentation folder will automatically be used by sphinx, but the docfiles adds a level of organization to the documentation folder.
3. To create an autodoc, create a new rst file in the docfiles directory.
4. Within the newly created rst file, add the following:

Code documentation
=========================================

.. autoclass:: ECNQueue.Item
    :members:
    :private-members:

.. autoclass:: ECNQueue.Queue
    :members:
    :private-members:

.. automodule:: ECNQueue
    :members:

This tells the sphinx parser to parse all of the private functions in the Item and Queue classes within ECNQueue.py, but also any other functions not included within a specific class. Save this file.

5. Within the terminal, ensure you are in the parent documentation directory and run make clean. This will remove any items in the build directory and ensures that the next command will execute without errors caused by lingering files within build.

6. Run the command make html. This will generate several files in the build directory. Navigate to the build/html/docfiles directory. Within this directory, there will be an html file with the same name as the rst file created in step 4. This html file contains all of the functions within ECNQueue.py as well as their respective docstrings in a relatively easy to read format.

[benne238]

Sphinx Themes

Additional themes are available for sphinx through pip
In this case, pip install sphinx_rtd_theme
To configure this theme, go to conf.py in the documentation folder and edit line 51

html_theme = 'alabaster'

to this:

html_theme = 'sphinx_rtd_theme'

[benne238]

Markdown and Sphinx

Still looking into this, but it seems that at the very least, markdown documents can be put into the sphinx directory. However, it may be possible to output auto generated code as markdown. Will look into this tomorrow.

Allowing markdown to be parsed by Sphinx

1. run pip install recommonmark
2. edit conf.py in the documentation directory to include these lines at the end:

# The file extensions of source files. Sphinx considers the files with this suffix as sources. 
# The value is a dictionary mapping file extensions to file types
source_suffix = {
    '.rst': 'restructuredtext',
    '.md': 'markdown'
}

This tells sphinx to not delete any .md files in the documents directory and to treat .md files as markdown files.

[benne238]

Markdown in Sphinx

Following the steps above does allow for markdown to be parsed directly by sphinx and thus parsed into an html page. However, documentation doesn't appear to be able to automatically generated directly from a markdown file like what can be done with an rst file.

[benne238]

Python Docstring Format

One of the "issues" with the Sphinx auto-doc feature is that the output format looks terrible:

However, there is a relatively easy way to fix this problem which is by reformatting the python docstrings so that the output sphinx produces looks significantly better:

When outputting, Sphinx considers multiple things from the string:

1. The indention level of different lines in the docstring
2. New lines within docstrings
3. Special characters, such as asterisks

All of which affect the output format of the html files