Chapter 15: Helping web2py

Helping web2py: Bugs, enhancements and documentation

web2py is very welcoming towards bug reports, documentation improvements and enhancements.

Google Group

The main forum for discussing bugs and new features is: web2py-users (URL is https://groups.google.com/forum/#!forum/web2py)

Filing a bug report via Google Code

If you have found a bug and discussed it on the group, you may be requested to file a bug report by creating an 'Issue' on Google Code http://code.google.com/p/web2py/issues/

Reporting Security Bugs

reporting security bugs

When reporting security bugs, be careful about disclosing information which would assist a malicious exploitation. After posting on the Google group or raising an issue at the Google Code site, you will be contacted by a developer to share information privately.

Contributing code and documentation changes

How the project code is managed

The web2py code-base is managed in a GitHub git repository. The original version control system was Mercurial at Google Code, but active development tends to be at GitHub. web2py has two important versions: the current stable version, and the development snapshot. In git the development snapshot is usually referred to as "master", but the web2py project uses "trunk", which is the equivalent term from Mercurial.

Bug fixes and enhancements go into trunk, so to take quick advantage of a bug fix you will need to use trunk. The best way to do this is to clone the git repository. This is the first step when using code stored on GitHub, and is well documented there. Because releases are tagged, you can switch between the trunk branch (which is 'master') and the current release (or any release). This makes the git repository very practical.

The core git skills you will need are:

  1. cloning a repository hosted on GitHub
  2. checkout out the master branch
  3. checking out a branch based on a tag

These are well covered by GitHub tutorials: see below.

Discussing proposed changes

If the change relates to a bug, the discussion will probably occur on the Issue on Google Code (see above). If the change relates to an enhancement, the developers' group is the best place initially: web2py-developers group (URL is https://groups.google.com/forum/#!forum/web2py-developers)

Because web2py promises backwards compatibility for non-experimental features, accepting a change into the code base is a promise of maintenance to future users, and this is not a promise than can be made lightly. Less experienced users may discover via discussion that there is an existing method to achieve what they want; unfortunately, the documentation of power features sometimes lags (see improving the documentation below).

When you ready to submit a code change, discussion will probably move to the comments attached to the Pull Request (discussed below).

Coding style

coding style

Generally, follow PEP8 coding style. http://www.python.org/dev/peps/pep-0008/

There is specific guidance here: web2py style

Documentation style

Documentation style is specified below

Tips on setting up a development environment

Chapter 13 has some tips on using various IDEs with web2py.

Preparation: using GitHub

patches
GitHub

Since web2py uses GitHub, the overall flow of changes goes like this:

  1. You create a clone of web2py on GitHub, which GitHub calls "forking".
  2. You create a clone of your web2py 'fork' on your local machine (or wherever you want to make changes)
  3. You push your changes back to your GitHub fork
  4. You make a pull request via GitHub
  5. If your pull request is accepted, your commit will be in the master branch of web2py.

At the time of writing, the core git/GitHub techniques are covered here: https://help.GitHub.com/articles/fork-a-repo

However, you need to follow the instructions below on how to create patches which can be cleanly merged.

Preparation: using travis with your GitHub fork

travis

web2py works with travis, a continuous integration testing service. This means that a commit will trigger all the integrated unit tests.

You can easily add travis to your own GitHub repository to avoid sending patches which will fail travis testing upstream.

Follow the instructions here: http://about.travis-ci.org/docs/user/getting-started/

The four parts of a good quality enhancement

A high-quality enhancement has four parts:

  1. The code change
  2. Documentating API changes (see API documentation below)
  3. Testing for the new functionality
  4. Updating the book (which is also maintained at GitHub)

Ensuring a clean patch: using the correct git branch technique

git

Using git properly is a little more complex in a shared project.

Here are the specific guidelines to make sure your pull request can be applied cleanly.

Firstly, make sure you have a remote repository setup which points to the main web2py GitHub repository. Your remote repository should be called upstream. You only need to do this once.

Adding an upstream repository is covered by the introductory GitHub article linked above, but in case you missed it, you can add upstream like this:

git remote add upstream https://GitHub.com/web2py/web2py/

Next you need a branch name for your changes. git encourages lots of branch names, as specific as possible. For web2py, we recommend names like this:

  • every bug-fixing commit should come from a branch named "issue/number_of_the_issue_on_google_code" (like issue/1684)
  • every enhancement commit should come in a branch named "enhancement/title_of_the_enhancement" (like enhancement/trapped_links)

In your local environment, checkout the branch for your changes: Substitute CHANGE1 for your branch name.

git fetch upstream
git checkout -b CHANGE1 upstream/master

... Make changes or cherry pick from another local branch. commit if necessary. When you are ready to send your local changes to your web2py fork:

git push origin CHANGE1

<TODO insert note about collapsing several commits into one commit>

and now go the GitHub website, change to the new branch and do a pull request. GitHub has a "delete branch" button after your pull request is merged or closed. There can be no guarantees, but PRs are usually reviewed within a few days. Most people submitting patches as PRs also update either the issue report or enhancement thread.

Adding tests

code tests
unit tests

Unit tests should be added when an enhancement changes or adds functionality. Tests are python scripts contained in gluon/tests Copy the approach of existing tests. You will notice that tests often need to create something such as table, perform a test of the functionality and check the outcome, and then return the state do before the test (which in this case would mean dropping the table). There is a good reference to testing here: Web2pyslice 1691

http://www.web2pyslices.com/slice/show/1691/help-developers-adding-tests-to-web2py

Running tests

You can run all tests:

python web2py.py --run_system_tests

See the slice referenced above for more information, including running individual tests.

Example unit test

Writing tests is easy. This is an example from gluon/tests/test_dal.py You can see how the test extends a TestCase class. Note how a sufficiently complex table is added, the test performed and output validated, and then the database changes are torn down, leaving the campsite as clean as when you arrived.

class TestVirtualFields(unittest.TestCase):

    def testRun(self):
        db = DAL(DEFAULT_URI, check_reserved=['all'])
        db.define_table('tt', Field('aa'))
        db.commit()
        db.tt.insert(aa="test")
        class Compute:
            def a_upper(row): return row.tt.aa.upper()
        db.tt.virtualfields.append(Compute())
        assert db(db.tt.id>0).select().first().a_upper == 'TEST'
        db.tt.drop()
        db.commit()

Documentation: Updating the book

Updating book

The book is also in GitHub and the same git workflow can be used. The book source contains sources in various languages, under the sources directory. The content is written in markmin . The book is a web2py app. You may find it convenient to make a GitHub fork of the book, and clone your local repository in the applications directory of a web2py installation. This makes it easy to see your changes rendered in a browser.

Documentation: web2pyslices.com

web2pyslices
web2pyslices.com

web2pyslices.com is a resource for recipes, code snippets and samples.

Documentation: API, in-code and docstring documentation

The web2py API documentation is here: http://web2py.readthedocs.org/en/latest/

It uses sphinx, which means it is generated based on documentation included in the source files.

API (i.e. docstrings) and in-code documentation should follow the Google Python Style Guide - Comments

Here is an example from the web2py code-base:

def getcfs(key, filename, filter=None):
    """
Caches the *filtered* file `filename` with `key` until the file is
modified.

Args:
key(str): the cache key
filename: the file to cache
filter: is the function used for filtering. Normally `filename` is a
.py file and `filter` is a function that bytecode compiles the file.
In this way the bytecode compiled file is cached. (Default = None)

This is used on Google App Engine since pyc files cannot be saved.
"""

and an example from Google's style-guide:

def fetch_bigtable_rows(big_table, keys, other_silly_variable=None):
    """Fetches rows from a Bigtable.

    Retrieves rows pertaining to the given keys from the Table instance
    represented by big_table.  Silly things may happen if
    other_silly_variable is not None.

    Args:
        big_table: An open Bigtable Table instance.
        keys: A sequence of strings representing the key of each table row
            to fetch.
        other_silly_variable: Another optional variable, that has a much
            longer name than the other args, and which does nothing.

    Returns:
        A dict mapping keys to the corresponding table row data
        fetched. Each row is represented as a tuple of strings. For
        example:

        {'Serak': ('Rigel VII', 'Preparer'),
         'Zim': ('Irk', 'Invader'),
         'Lrrr': ('Omicron Persei 8', 'Emperor')}

        If a key from the keys argument is missing from the dictionary,
        then that row was not found in the table.

    Raises:
        IOError: An error occurred accessing the bigtable.Table object.
    """
    pass

Some further web2py-specific notes re API documentation via docstrings:

  • "experimental" features should be clearly marked
  • sphinx was chosen because it's largely supported, maintained AND has the added bonus of readthedocs.org
  • new features should be clearly marked (i.e. "starting from 2.4.6...")
  • new features should be closely related to the official changelog
  • readthedocs.org allows us to build "statically linked" version of the docs for future tagged releases (good for "feature comparison")
  • The web2py developers adopt the Google style guide as it offers the best of sphinx without all the hassle of ReST "decorations"
  • until sphinx 1.3 goes out, napoleon is a requirement for building the docs
 top