The Core

Command Line Options

It is possible to skip the GUI and start web2py directly from the command line by typing something like:

1.

python web2py.py -a 'your password' -i 127.0.0.1 -p 8000

When web2py starts, it creates a file called "parameters_8000.py" where it stores the hashed password. If you use "<ask>" as the password, web2py prompts you for it.

For additional security, you can start web2py with:

1.

python web2py.py -a '<recycle>' -i 127.0.0.1 -p 8000

In this case web2py reuses the previously stored hashed password. If no password is provided, or if the "parameters_8000.py" file is deleted, the web-based administrative interface is disabled.

1.

<pam_user:some_user>

web2py uses the PAM password of the Operating System account of some_user to authenticate the administrator, unless blocked by the PAM configuration.

web2py normally runs with CPython (the C implementation of the Python interpreter created by Guido van Rossum), but it can also run with Jython (the Java implementation of the interpreter). The latter possibility allows the use of web2py in the context of a J2EE infrastructure. To use Jython, simply replace "python web2py.py ..." with "jython web2py.py". Details about installing Jython, zxJDBC modules required to access the databases can be found in Chapter 12.

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.

>>> python web2py.py -h
Usage: python web2py.py

web2py Web Framework startup script. ATTENTION: unless a password
is specified (-a 'passwd'), web2py will attempt to run a GUI.
In this case command line options are ignored.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -i IP, --ip=IP        ip address of the server (127.0.0.1)
  -p PORT, --port=PORT  port of server (8000)
  -a PASSWORD, --password=PASSWORD
                        password to be used for administration
                        use -a "<recycle>" to reuse the last
                        password
  -u UPGRADE, --upgrade=UPGRADE
                        -u yes: upgrade applications and exit
  -c SSL_CERTIFICATE, --ssl_certificate=SSL_CERTIFICATE
                        file that contains ssl certificate
  -k SSL_PRIVATE_KEY, --ssl_private_key=SSL_PRIVATE_KEY
                        file that contains ssl private key
  -d PID_FILENAME, --pid_filename=PID_FILENAME
                        file to store the pid of the server
  -l LOG_FILENAME, --log_filename=LOG_FILENAME
                        file to log connections
  -n NUMTHREADS, --numthreads=NUMTHREADS
                        number of threads
  -s SERVER_NAME, --server_name=SERVER_NAME
                        server name for the web server
  -q REQUEST_QUEUE_SIZE, --request_queue_size=REQUEST_QUEUE_SIZE
                        max number of queued requests when server unavailable
  -o TIMEOUT, --timeout=TIMEOUT
                        timeout for individual request (10 seconds)
  -z SHUTDOWN_TIMEOUT, --shutdown_timeout=SHUTDOWN_TIMEOUT
                        timeout on shutdown of server (5 seconds)
  -f FOLDER, --folder=FOLDER
                        folder from which to run web2py
  -v, --verbose         increase --test verbosity
  -Q, --quiet           disable all output
  -D DEBUGLEVEL, --debug=DEBUGLEVEL
                        set debug output level (0-100, 0 means all,
                        100 means none; default is 30)
  -S APPNAME, --shell=APPNAME
                        run web2py in interactive shell or IPython
                        (if installed) with specified appname
  -P, --plain           only use plain python shell; should be used
                        with --shell option
  -M, --import_models   auto import model files; default is False;
                        should be used with --shell option
  -R PYTHON_FILE, --run=PYTHON_FILE
                        run PYTHON_FILE in web2py environment;
                        should be used with --shell option
  -T TEST_PATH, --test=TEST_PATH
                        run doctests in web2py environment;
                        TEST_PATH like a/c/f (c,f optional)
  -W WINSERVICE, --winservice=WINSERVICE
                        -W install|start|stop as Windows service
  -C, --cron            trigger a cron run manually; usually invoked
                        from a system crontab
  -N, --no-cron         do not start cron automatically
  -L CONFIG, --config=CONFIG
                        config file
  -F PROFILER_FILENAME, --profiler=PROFILER_FILENAME
                        profiler filename
  -t, --taskbar         use web2py gui and run in taskbar
                        (system tray)
  --nogui               text-only, no GUI
  -A ARGS, --args=ARGS  should be followed by a list of arguments to be passed
                        to script, to be used with -S, -A must be the last
                        option
  --interfaces=INTERFACES
                        allows multiple interfaces to be served

Lower-case options are used to configure the web server. The -L option tells web2py to read configuration options from a file, -W installs web2py as a windows service, while -S, -P and -M options start an interactive Python shell. The -T option finds and runs controller doctests in a web2py execution environment. For example, the following example runs doctests from all controllers in the "welcome" application:

1.

python web2py.py -vT welcome

if you run web2py as Windows Service, -W, it is not convenient to pass the configuration using command line arguments. For this reason, in the web2py folder there is a sample "options_std.py" configuration file for the internal web server:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

import socket, os
ip = '127.0.0.1'
port = 8000
password = '<recycle>' ### <recycle> means use the previous password
pid_filename = 'httpserver.pid'
log_filename = 'httpserver.log'
ssl_certificate = " ### path to certificate file
ssl_private_key = " ### path to private key file
numthreads = 10
server_name = socket.gethostname()
request_queue_size = 5
timeout = 10
shutdown_timeout = 5
folder = os.getcwd()

This file contains the web2py defaults. If you edit this file, you need to import it explicitly with the -L command-line option. It only works if you run web2py as a Windows Service.

Dispatching

web2py maps a URL of the form:

http://127.0.0.1:8000/a/c/f.html

to the function f() in controller "c.py" in application "a". If f is not present, web2py defaults to the index controller function. If c is not present, web2py defaults to the "default.py" controller, and if a is not present, web2py defaults to the init application. If there is no init application, web2py tries to run the welcome application. This is shown schematically in the image below:

(The names of the default application, controller and function can be overridden in routes.py; see Default Application, Controller and Function below.

By default, any new request also creates a new session. In addition, a session cookie is returned to the client browser to keep track of the session.

The extension .html is optional; .html is assumed as default. The extension determines the extension of the view that renders the output of the controller function f(). It allows the same content to be served in multiple formats (html, xml, json, rss, etc.).

Functions that take arguments or start with a double underscore are not publicly exposed and can only be called by other functions.

http://127.0.0.1:8000/a/static/filename

There is no controller called "static". web2py interprets this as a request for the file called "filename" in the subfolder "static" of the application "a".

web2py also supports the IF_MODIFIED_SINCE protocol, and does not send the file if it is already stored in the browser's cache and if the file has not changed since that version.

http://127.0.0.1:8000/a/c/f.html/x/y/z?p=1&q=2

to function f in controller "c.py" in application a, and it stores the URL parameters in the request variable as follows:

1.

request.args = ['x', 'y', 'z']

and:

1.

request.vars = {'p':1, 'q':2}

and:

1.
2.
3.

request.application = 'a'
request.controller = 'c'
request.function = 'f'

In the above example, both request.args[i] and request.args(i) can be used to retrieve the i-th element of the request.args, but while the former raises an exception if the list does not have such an index, the latter returns None in this case.

1.

request.url

stores the full URL of the current request (not including GET variables).

1.

request.ajax

defaults False but it is True if web2py determines that the action was called by an Ajax request.

If the request is an Ajax request and it is initiated by a web2py component, the name of the component can be found in:

1.

request.cid

Components are discussed in detain in Chapter 13.

web2py stores WSGI and web2py environment variables in request.env, for example:

1.

request.env.path_info = 'a/c/f'

and HTTP headers into environment variables, for example:

1.

request.env.http_host = '127.0.0.1:8000'

Notice that web2py validates all URLs to prevent directory traversal attacks.

If the URL corresponds to a request for a static file, web2py simply reads and returns (streams) the requested file.

If the URL does not request a static file web2py processes the request in the following order:

• Parses cookies.
• Creates an environment in which to execute the function.
• Initializes request, response, cache.
• Opens the existing session or creates a new one.
• Executes the models belonging to the requested application.
• Executes the requested controller action function.
• If the function returns a dictionary, executes the associated view.
• On success, commits all open transactions.
• Saves the session.
• Returns an HTTP response.

Notice that the controller and the view are executed in different copies of the same environment; therefore, the view does not see the controller, but it sees the models and it sees the variables returned by the controller action function.

If an exception (other than HTTP) is raised, web2py does the following:

• Stores the traceback in an error file and assigns a ticket number to it.
• Rolls back all open transactions.
• Returns an error page reporting the ticket number.

If the exception is an HTTP exception, this is assumed to be the intended behavior (for example, an HTTP redirect), and all open database transactions are committed. The behavior after that is specified by the HTTP exception itself. The HTTP exception class is not a standard Python exception; it is defined by web2py.

Libraries

The web2py libraries are exposed to the user applications as global objects. For example (request, response, session, cache), classes (helpers, validators, DAL API), and functions (T and redirect).

These objects are defined in the following core files:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

web2py.py
gluon/__init__.py    gluon/highlight.py   gluon/restricted.py  gluon/streamer.py
gluon/admin.py       gluon/html.py        gluon/rewrite.py     gluon/template.py
gluon/cache.py       gluon/http.py        gluon/rocket.py      gluon/storage.py
gluon/cfs.py         gluon/import_all.py  gluon/sanitizer.py   gluon/tools.py
gluon/compileapp.py  gluon/languages.py   gluon/serializers.py gluon/utils.py
gluon/contenttype.py gluon/main.py        gluon/settings.py    gluon/validators.py
gluon/dal.py         gluon/myregex.py     gluon/shell.py       gluon/widget.py
gluon/decoder.py     gluon/newcron.py     gluon/sql.py         gluon/winservice.py
gluon/fileutils.py   gluon/portalocker.py gluon/sqlhtml.py     gluon/xmlrpc.py
gluon/globals.py     gluon/reserved_sql_keywords.py

The tar gzipped scaffolding app that ship with web2py is

1.

welcome.w2p

It is created upon installation and overwritten on upgrade.

The first time you start web2py, two new folders are created: deposit and applications. The "welcome" app is zipped into a "welcome.w2p" file to be used as scaffolding app. The deposit folder is used as temporary storage for installing and uninstalling applications.

1.

gluon/tests/

There are handlers for connecting with various web servers:

1.
2.
3.
4.
5.
6.
7.

cgihandler.py
gaehandler.py
fcgihandler.py
wsgihandler.py
modpythonhandler.py
gluon/contrib/gateways/__init__.py
gluon/contrib/gateways/fcgi.py

(fcgi.py was developed by Allan Saddi)

There are two example files:

1.
2.

options_std.py
routes.example.py

The former is an optional configuration file that can be passed to web2py.py with the -L option. The second is an example of a URL mapping file. It is loaded automatically when renamed "routes.py".

The files

1.
2.

app.yaml
index.yaml

are configuration files necessary for deployment on the Google App Engine. You probably do not need to modify them, but you can read more about them on the Google Documentation pages.

There are also additional libraries, usually developed by a third party:

feedparser²⁸ by Mark Pilgrim for reading RSS and Atom feeds:

1.
2.

gluon/contrib/__init__.py
gluon/contrib/feedparser.py

markdown2²⁹ by Trent Mick for wiki markup:

1.
2.

gluon/contrib/markdown/__init__.py
gluon/contrib/markdown/markdown2.py

markmin markup:

1.

gluon/contrib/markmin.py

pysimplesoap is a lightweight SOAP server implementation created by Mariano Reingart:

1.

gluon/contrib/pysimplesoap/

memcache³⁰ Python API by Evan Martin:

1.
2.

gluon/contrib/memcache/__init__.py
gluon/contrib/memcache/memcache.py

gql, a port of the DAL to the Google App Engine:

1.

gluon/contrib/gql.py

memdb, a port of the DAL on top of memcache:

1.

gluon/contrib/memdb.py

gae_memcache is an API to use memcache on the Google App Engine:

1.

gluon/contrib/gae_memcache.py

pyrtf²⁶ for generating Rich Text Format (RTF) documents, developed by Simon Cusack and revised by Grant Edwards:

1.
2.
3.
4.
5.
6.
7.
8.

gluon/contrib/pyrtf
gluon/contrib/pyrtf/__init__.py
gluon/contrib/pyrtf/Constants.py
gluon/contrib/pyrtf/Elements.py
gluon/contrib/pyrtf/PropertySets.py
gluon/contrib/pyrtf/README
gluon/contrib/pyrtf/Renderer.py
gluon/contrib/pyrtf/Styles.py

PyRSS2Gen²⁷ developed by Dalke Scientific Software, to generate RSS feeds:

1.

gluon/contrib/rss2.py

simplejson²⁵ by Bob Ippolito, the standard library for parsing and writing JSON objects:

1.
2.
3.
4.
5.

gluon/contrib/simplejson/__init__.py
gluon/contrib/simplejson/decoder.py
gluon/contrib/simplejson/encoder.py
gluon/contrib/simplejson/jsonfilter.py
gluon/contrib/simplejson/scanner.py

AuthorizeNet⁹² provides API to accept credit card payments via Authorize.net network

1.

gluon/contrib/AuthorizeNet.py

PAM⁷² authentication API created by Chris AtLee:

1.

gluon/contrib/pam.py

A Bayesian classifier to populate the database with dummy data for testing purposes:

1.

gluon/contrib/populate.py

A file that allows interaction with the taskbar in windows, when web2py is running as a service:

1.

gluon/contrib/taskbar_widget.py

Optional login_methods and login_forms to be used for authentication:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

gluon/contrib/login_methods/__init__.py
gluon/contrib/login_methods/basic_auth.py
gluon/contrib/login_methods/cas_auth.py
gluon/contrib/login_methods/email_auth.py
gluon/contrib/login_methods/extended_login_form.py
gluon/contrib/login_methods/gae_google_account.py
gluon/contrib/login_methods/ldap_auth.py
gluon/contrib/login_methods/linkedin_account.py
gluon/contrib/login_methods/oauth20_account.py
gluon/contrib/login_methods/openid_auth.py
gluon/contrib/login_methods/pam_auth.py
gluon/contrib/login_methods/rpx_account.py

web2py also contains a folder with useful scripts:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

scripts/setup-web2py-fedora.sh
scripts/setup-web2py-ubuntu.sh
scripts/cleancss.py
scripts/cleanhtml.py
scripts/contentparser.py
scripts/repair.py
scripts/sessions2trash.py
scripts/sync_languages.py
scripts/tickets2db.py
...

The first two are particularly useful because they attempt a complete installation and setup of a web2py production environment from scratch. These are discussed in Chapter 12, but they are more or less self-documenting.

Finally web2py includes these files required to build the binary distributions.

1.
2.
3.

Makefile
setup_exe.py
setup_app.py

These are setup scripts for py2exe and py2app respectively and they are only required to build the binary distributions of web2py. YOU SHOULD NEVER NEET TO RUN THEM.

In summary, web2py libraries provide the following functionality:

• Map URLs into function calls.
• Handle passing and returning parameters via HTTP.
• Perform validation of those parameters.
• Protect the applications from most security issues.
• Handle data persistence (database, session, cache, cookies).
• Perform string translations for various supported languages.
• Generate HTML programmatically (e.g. from database tables).
• Generate SQL via a Database Abstraction Layer (DAL).
• Generate Rich Text Format (RTF) output.
• Generate Comma-Separated Value (CSV) output from database tables.
• Generate Really Simple Syndication (RSS) feeds.
• Generate JavaScript Object Notation (JSON) serialization strings for Ajax.
• Translate wiki markup (Markdown) to HTML.
• Expose XML-RPC web services.
• Upload and download large files via streaming.

web2py applications contain additional files, particularly third-party JavaScript libraries, such as jQuery, calendar, EditArea and nicEdit. Their authors are acknowledged in the files themselves.

Applications

Applications developed in web2py are composed of the following parts:

• models describe a representation of the data as database tables and relations between tables.
• controllers describe the application logic and workflow.
• views describe how data should be presented to the user using HTML and JavaScript.
• languages describe how to translate strings in the application into various supported languages.
• static files do not require processing (e.g. images, CSS stylesheets, etc).
• ABOUT and README documents are self-explanatory.
• errors store error reports generated by the application.
• sessions store information related to each particular user.
• databases store SQLite databases and additional table information.
• cache store cached application items.
• modules are other optional Python modules.
• private files are accessed by the controllers but not directly by the developer.
• uploads files are accessed by the models but not directly by the developer (e.g., files uploaded by users of the application).
• tests is a directory for storing test scripts, fixtures and mocks.

Models, views, controllers, languages, and static files are accessible via the web administration [design] interface. ABOUT, README, and errors are also accessible via the administration interface through the corresponding menu items. Sessions, cache, modules and private files are accessible to the applications but not via the administration interface.

Everything is neatly organized in a clear directory structure that is replicated for every installed web2py application, although the user never needs to access the filesystem directly:

1.
2.
3.

__init__.py  ABOUT        LICENSE    models    views
controllers  modules      private    tests     cron
cache        errors       upload     sessions  static

"__init__.py" is an empty file which is required in order to allow Python (and web2py) to import the modules in the modules directory.

Notice that the admin application simply provides a web interface to web2py applications on the server file system. web2py applications can also be created and developed from the command-line; you don't have to use the browser admin interface. A new application can be created manually by replicating the above directory structure under ,e.g., "applications/newapp/" (or simply untar the welcome.w2p file into your new application directory). Application files can also be created and edited from the command-line without having to use the web admin interface.

API

Models, controllers, and views are executed in an environment where the following objects are already imported for us:

Global Objects:

1.

request, response, session, cache

Navigation:

1.

redirect, HTTP

Internationalization:

1.

T

Helpers:

1.
2.
3.
4.
5.
6.
7.
8.
9.

XML, URL, BEAUTIFY

A, B, BEAUTIFY, BODY, BR, CENTER, CODE, DIV, EM, EMBED,
FIELDSET, FORM, H1, H2, H3, H4, H5, H6, HEAD, HR, HTML,
I, IFRAME, IMG, INPUT, LABEL, LEGEND, LI, LINK, OL, UL,
MARKMIN, MENU, META, OBJECT, ON, OPTION, P, PRE, SCRIPT,
OPTGROUP, SELECT, SPAN, STYLE, TABLE, TAG, TD, TEXTAREA,
TH, THEAD, TBODY, TFOOT, TITLE, TR, TT, URL, XHTML, 
xmlescape, embed64

Validators:

1.
2.
3.
4.
5.
6.
7.

CLEANUP, CRYPT, IS_ALPHANUMERIC, IS_DATE_IN_RANGE, IS_DATE,
IS_DATETIME_IN_RANGE, IS_DATETIME, IS_DECIMAL_IN_RANGE,
IS_EMAIL, IS_EMPTY_OR, IS_EXPR, IS_FLOAT_IN_RANGE, IS_IMAGE,
IS_IN_DB, IS_IN_SET, IS_INT_IN_RANGE, IS_IPV4, IS_LENGTH,
IS_LIST_OF, IS_LOWER, IS_MATCH, IS_EQUAL_TO, IS_NOT_EMPTY,
IS_NOT_IN_DB, IS_NULL_OR, IS_SLUG, IS_STRONG, IS_TIME,
IS_UPLOAD_FILENAME, IS_UPPER, IS_URL

Database:

1.

DAL, Field

For backward compatibility SQLDB=DAL and SQLField=Field. We encourage you to use the new syntax DAL and Field, instead of the old syntax.

Other objects and modules are defined in the libraries, but they are not automatically imported since they are not used as often.

The core API entities in the web2py execution environment are request, response, session, cache, URL, HTTP, redirect and T and are discussed below.

A few objects and functions, including Auth, Crud and Service, are defined in "gluon/tools.py" and they need to be imported as necessary:

1.

from gluon.tools import Auth, Crud, Service

request

The request object is an instance of the ubiquitous web2py class that is called gluon.storage.Storage, which extends the Python dict class. It is basically a dictionary, but the item values can also be accessed as attributes:

1.

request.vars

is the same as:

1.

request['vars']

Unlike a dictionary, if an attribute (or key) does not exist, it does not raise an exception. Instead, it returns None. request has the following items/attributes, some of which are also an instance of the Storage class:

• request.cookies: a Cookie.SimpleCookie() object containing the cookies passed with the HTTP request. It acts like a dictionary of cookies. Each cookie is a Morsel object.
• request.env: a Storage object containing the environment variables passed to the controller, including HTTP header variables from the HTTP request and standard WSGI parameters. The environment variables are all converted to lower case, and dots are converted to underscores for easier memorization.
• request.application: the name of the requested application (parsed from request.env.path_info).
• request.controller: the name of the requested controller (parsed from the request.env.path_info).
• request.function: the name of the requested function (parsed from the request.env.path_info).
• request.extension: the extension of the requested action. It defaults to "html". If the controller function returns a dictionary and does not specify a view, this is used to determine the extension of the view file that will render the dictionary (parsed from the request.env.path_info).
• request.folder: the application directory. For example if the application is "welcome", request.folder is set to the absolute path "/path/to/welcome". In your programs, you should always use this variable and the os.path.join function to build paths to the files you need to access. Although web2py always uses absolute paths, it is a good rule never to explicitly change the current working folder (whatever that is) since this is not a thread-safe practice.
• request.now: a datetime.datetime object storing the time-stamp of the current request.
• request.args: A list of the URL path components following the controller function name; equivalent to request.env.path_info.split('/')[3:]
• request.vars: a gluon.storage.Storage object containing the HTTP GET and HTTP POST query variables.
• request.get_vars: a gluon.storage.Storage object containing only the HTTP GET query variables.
• request.post_vars: a gluon.storage.Storage object containing only the HTTP POST query variables.
• request.client: The ip address of the client as determined by, if present, request.env.http_x_forwarded_for or by request.env.remote_addr otherwise. While this is useful it should not be trusted because the http_x_forwarded_for can be spoofed.
• request.body: a read-only file stream that contains the body of the HTTP request. This is automatically parsed to get the request.post_vars and then rewinded. It can be read with request.body.read().
• request.ajax is True if the function is being called via an Ajax request.
• request.cid is the id of the component that generated the Ajax request (if any). You can read more about components in Chapter 13.
• request.wsgi a hook that allows to call third party WSGI applications from inside actions.

As an example, the following call on a typical system:

http://127.0.0.1:8000/examples/default/status/x/y/z?p=1&q=2

results in the following table:

The request.env.http_* variables are parsed from the request HTTP header.

The request.env.web2py_* variables. These are not parsed from the web server environment, but are created by web2py in case your applications need to know about the web2py location and version, and whether it is running on the Google App Engine (because specific optimizations may be necessary).

Also notice the request.env.wsgi_* variables. They are specific to the wsgi adaptor.

response

response is another instance of the Storage class. It contains the following:

response.body: a StringIO object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

response.cookies: similar to request.cookies, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

response.download(request, db): a method used to implement the controller function that allows downloading of uploaded files.

response.files: a list of CSS and JS required by the page. They will automatically be linked in the header of the standard "layout.html". To include a new CSS or JS file, just append it to this list. It will handle duplicates. The order is significative.

response.flash: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

response.headers: a dict for HTTP response headers.

response.menu: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

response.meta: a storage object (like a dict) that contains optional meta storage information like response.meta.author, response.meta.description, and/or response.meta.keywords. The content of the meta variable is automatically placed in the proper META tag by the code in "web2py_ajax.html" with is included by default by "views/layout.html".

response.postprocessing: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

response.render(view, vars): a method used to call the view explicitly inside the controller. view is an optional parameter which is the name of the view file, vars is a dictionary of named values passed to the view.

response.session_file: file stream containing the session.

response.session_file_name: name of the file where the session will be saved.

response.session_id: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

response.session_id_name: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

response.status: the HTTP status code integer to be passed to the response. Default is 200 (OK).

response.stream(file, chunk_size): when a controller returns it, web2py streams the file content back to the client in blocks of size chunk_size.

response.subtitle: optional parameter that may be included in the views. It should contain the subtitle of the page.

response.title: optional parameter that may be included in the views. It should contain the title of the page and should be rendered by the HTML title TAG in the header.

response._vars: this variable is accessible only in a view, not in the action. It contains the value returned by the action to the view.

response.view: the name of the view template that must render the page. This is set by default to:

1.

"%s/%s.%s" % (request.controller, request.function, request.extension)

or, if the above file cannot be located, to

1.

"generic.%s" % (request.extension)

Change the value of this variable to modify the view file associated with a particular action.

response.xmlrpc(request, methods): when a controller returns it, this function exposes the methods via XML-RPC⁴⁶ . This function is deprecated since a better mechanism is available and described in Chapter 9.

response.write(text): a method to write text into the output page body.

response.js can contain Javascript Code. This code will be executed if and only if the response is received by a web2py component as discussed in Chapter 13.

Since response is a gluon.storage.Storage object it can be used to store other attributes that you may want to pass to the view. While there is no technical restriction our recommendation is to store only variables that are to be rendered by all pages in the overall layout ("layout.html").

Anyway, we strongly suggest to stick to the variables listed here:

1.
2.
3.
4.
5.
6.
7.
8.

response.title
response.subtitle
response.flash
response.menu
response.meta.author
response.meta.description
response.meta.keywords
response.meta.*

because this will make it easier for you to replace the standard "layout.html" file that comes with web2py with another layout file, one that uses the same set of variables.

Old versions of web2py user response.author instead of response.meta.author and similar for the other meta attributes.

session

1.

session.myvariable = "hello"

can be retrieved at a later time:

1.

a = session.myvariable

as long as the code is executed within the same session by the same user (provided the user has not deleted session cookies and the session did not expire). Because session is a Storage object, trying to access an attribute/key that has not been set does not raise an exception; it returns None instead.

The session object has two important methods. One is forget:

1.

session.forget()

It tells web2py not to save the session. This should be used in those controllers whose actions are called often and do not need to track user activity.

The other method is connect:

1.

session.connect(request, response, db, masterapp=None)

where db is the name of an open database connection (as returned by the DAL). It tells web2py that you want to store the sessions in the database and not on the filesystem. web2py creates a table:

1.
2.
3.
4.
5.
6.
7.

db.define_table('web2py_session',
                 Field('locked', 'boolean', default=False),
                 Field('client_ip'),
                 Field('created_datetime', 'datetime', default=now),
                 Field('modified_datetime', 'datetime'),
                 Field('unique_key'),
                 Field('session_data', 'text'))

and stores cPickled sessions in the session_data field.

The option masterapp=None, by default, tells web2py to try to retrieve an existing session for the application with name in request.application, in the running application.

If you want two or more applications to share sessions, set masterapp to the name of the master application.

You can check the state of your application at any time by printing the request, session and response system variables. One way to do it is to create a dedicated action:

1.
2.

def status():
    return dict(request=request, session=session, response=response)

cache

• cache.ram: the application cache in main memory.
• cache.disk: the application cache on disk.

The following example caches the time.ctime() function in RAM:

1.
2.
3.
4.

def cache_in_ram():
    import time
    t = cache.ram('time', lambda: time.ctime(), time_expire=5)
    return dict(time=t, link=A('click me', _href=request.url))

The output of lambda: time.ctime() is cached in RAM for 5 seconds. The string 'time' is used as cache key.

The following example caches the time.ctime() function on disk:

1.
2.
3.
4.

def cache_on_disk():
    import time
    t = cache.disk('time', lambda: time.ctime(), time_expire=5)
    return dict(time=t, link=A('click me', _href=request.url))

The output of lambda: time.ctime() is cached on disk (using the shelve module) for 5 seconds.

The next example caches the time.ctime() function to both RAM and disk:

1.
2.
3.
4.
5.
6.

def cache_in_ram_and_disk():
    import time
    t = cache.ram('time', lambda: cache.disk('time',
                       lambda: time.ctime(), time_expire=5),
                       time_expire=5)
    return dict(time=t, link=A('click me', _href=request.url))

The output of lambda: time.ctime() is cached on disk (using the shelve module) and then in RAM for 5 seconds. web2py looks in RAM first and if not there it looks on disk. If it is not in RAM or on disk, lambda: time.ctime() is executed and the cache is updated. This technique is useful in a multiprocess environment. The two times do not have to be the same.

The following example is caching in RAM the output of the controller function (but not the view):

1.
2.
3.
4.
5.

@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)
def cache_controller_in_ram():
    import time
    t = time.ctime()
    return dict(time=t, link=A('click me', _href=request.url))

The dictionary returned by cache_controller_in_ram is cached in RAM for 5 seconds. Note that the result of a database select cannot be cached without first being serialized. A better way is to cache the database directly using the select method cache argument.

The following example is caching the output of the controller function on disk (but not the view):

1.
2.
3.
4.
5.
6.

@cache(request.env.path_info, time_expire=5, cache_model=cache.disk)
def cache_controller_on_disk():
    import time
    t = time.ctime()
    return dict(time=t, link=A('click to reload',
                              _href=request.url))

The dictionary returned by cache_controller_on_disk is cached on disk for 5 seconds. Remember that web2py cannot cache a dictionary that contains un-pickleable objects.

It is also possible to cache the view. The trick is to render the view in the controller function, so that the controller returns a string. This is done by returning response.render(d) where d is the dictionary we intended to pass to the view:

The time_expire can be set to 0 to force a cache refresh and to None to prevent the content from ever expiring.

You can clear one or more cache variables with

1.

cache.ram.clear(regex='...')

where regex is a regular expression maching all the keys you want removed from the cache.

The following example caches the output of the controller function in RAM (including the rendered view):

1.
2.
3.
4.
5.
6.

@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)
def cache_controller_and_view():
    import time
    t = time.ctime()
    d = dict(time=t, link=A('click to reload', _href=request.url))
    return response.render(d)

It is also possible to define other caching mechanisms such as memcache. Memcache is available via gluon.contrib.memcache and is discussed in more details in Chapter 11.

URL

Here is an example:

1.

URL('f')

is mapped into

1.

/[application]/[controller]/f

Notice that the output of the URL function depends on the name of the current application, the calling controller and other parameters. web2py supports URL mapping and reverse URL mapping. URL mapping allows to redefine the format of external URLs. If you use the URL function to generate all the internal URLs, then additions or changes to URL mappings will prevent broken links within the web2py application.

You can pass additional parameters to the URL function, i.e., extra terms in the URL path (args) and URL query variables (vars):

1.

URL('f', args=['x', 'y'], vars=dict(z='t'))

is mapped into

1.

/[application]/[controller]/f/x/y?z=t

The args attributes are automatically parsed, decoded, and finally stored in request.args by web2py. Similarly, the vars are parsed, decoded, and then stored in request.vars. args and vars provide the basic mechanism by which web2py exchanges information with the client's browser.

If args contains only one element, there is no need to pass it in a list.

You can also use the URL function to generate URLs to actions in other controllers and other applications:

1.

URL('a', 'c', 'f', args=['x', 'y'], vars=dict(z='t'))

is mapped into

/a/c/f/x/y?z=t

It is also possible to specify application, controller and function using named arguments:

1.

URL(a='a', c='c', f='f')

If the application name a is missing the current app is assumed.

1.

URL('c', 'f')

If the controller name is missing, the current one is assumed.

1.

URL('f')

Instead of passing the name of a controller function it is also possible to pass the function itself

1.

URL(f)

For the reasons mentioned above, you should always use the URL function to generate URLs of static files for your applications. Static files are stored in the application's static subfolder (that's where they go when uploaded using the administrative interface). web2py provides a virtual 'static' controller whose job is to retrieve files from the static subfolder, determine their content-type, and stream the file to the client. The following example generates the URL for the static file "image.png":

1.

URL('static', 'image.png')

is mapped into

1.

/[application]/static/image.png

You do not need to encode/escape the args and vars arguments; this is done automatically for you.

By default, the extension corresponding to the current request (request.extension) is appended to the function, unless request.extension is html, the default. This can be overridden by explicitly including an extension as part of the function name URL(f='name.ext') or with the extension argument:

1.

URL(..., extension='css')

The current extension can be explicitly suppressed:

1.

URL(..., extension=False)

HTTP and redirect

web2py defines only one new exception called HTTP. This exception can be raised anywhere in a model, a controller, or a view with the command:

1.

raise HTTP(400, "my message")

It causes the control flow to jump away from the user's code, back to web2py, and return an HTTP response like:

1.
2.
3.
4.
5.
6.
7.
8.
9.

HTTP/1.1 400 BAD REQUEST
Date: Sat, 05 Jul 2008 19:36:22 GMT
Server: Rocket WSGI Server
Content-Type: text/html
Via: 1.1 127.0.0.1:8000
Connection: close
Transfer-Encoding: chunked

my message

The first argument of HTTP is the HTTP status code. The second argument is the string that will be returned as the body of the response. Additional optional named arguments are used to build the response HTTP header. For example:

1.

raise HTTP(400, 'my message', test='hello')

generates:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

HTTP/1.1 400 BAD REQUEST
Date: Sat, 05 Jul 2008 19:36:22 GMT
Server: Rocket WSGI Server
Content-Type: text/html
Via: 1.1 127.0.0.1:8000
Connection: close
Transfer-Encoding: chunked
test: hello

my message

If you do not want to commit the open database transaction, rollback before raising the exception.

Any exception other than HTTP causes web2py to roll back any open database transaction, log the error traceback, issue a ticket to the visitor, and return a standard error page.

This means that only HTTP can be used for cross-page control flow. Other exceptions must be caught by the application, otherwise they are ticketed by web2py.

The command:

1.

redirect('http://www.web2py.com')

is simply a shortcut for:

1.
2.
3.

raise HTTP(303,
           'You are being redirected <a href="%s">here</a>' % location,
           Location='http://www.web2py.com')

The named arguments of the HTTP initializer method are translated into HTTP header directives, in this case, the redirection target location. redirect takes an optional second argument, which is the HTTP status code for the redirection (303 by default). Change this number to 307 for a temporary redirect or to 301 for a permanent redirect.

The most common way to user redirect is to redirect to other pages in the same app and (optionally) pass parameters:

1.

redirect(URL('index',args=(1,2,3),vars=dict(a='b')))

T and Internationalization

The object T is the language translator. It constitutes a single global instance of the web2py class gluon.language.translator. All string constants (and only string constants) should be marked by T, for example:

1.

a = T("hello world")

Strings that are marked with T are identified by web2py as needing language translation and they will be translated when the code (in the model, controller, or view) is executed. If the string to be translated is not a constant but a variable, it will be added to the translation file at runtime (except on GAE) to be translated later.

The T object can also contain interpolated variables, for example:

1.

a = T("hello %(name)s", dict(name="Massimo"))

The first string is translated according to the requested language file and the name variable is replaced independently of the language.

Concatenating translation strings is not a good idea; this is why web2py does not allow you to do:

1.

T("blah ") + name + T(" blah")   # invalid!

but it does allow:

1.

T("blah %(name)s blah", dict(name='Tim'))

or the alternative syntax

1.

T("blah %(name)s blah") % dict(name='Tim')

In both cases the translation occurs before the variable name is substituted in the "%(name)s" slot. The following instead should NOT BE USED:

1.

T("blah %(name)s blah" % dict(name='Tim'))

because translation would occur after substitution.

The requested language is determined by the "Accept-Language" field in the HTTP header, but this selection can be overwritten programmatically by requesting a specific file, for example:

1.

T.force('it-it')

which reads the "languages/it-it.py" language file. Language files can be created and edited via the administrative interface.

Normally, string translation is evaluated lazily when the view is rendered; hence, the translator force method should not be called inside a view.

It is possible to disable lazy evaluation via

1.

T.lazy = False

In this way, strings are translated immediately by the T operator based on the currently accepted or forced language.

A common issue is the following. The original application is in English. Suppose that there is a translation file (for example Italian, "it-it.py") and the HTTP client declares that it accepts both English (en) and Italian (it-it) in that order. The following unwanted situation occurs: web2py does not know the default is written in English (en). Therefore, it prefers translating everything into Italian (it-it) because it only found the Italian translation file. If it had not found the "it-it.py" file, it would have used the default language strings (English).

There are two solutions for this problem: create a translation language for English, which would be redundant and unnecessary, or better, tell web2py which languages should use the default language strings (the strings coded into the application). This can be done with:

1.

T.set_current_language('en', 'en-en')

It stores in T.current_languages a list of languages that do not require translation and forces a reload of the language files.

Notice that "it" and "it-it" are different languages from the point of view of web2py. To support both of them, one would need two translation files, always lower case. The same is true for all other languages.

The currently accepted language is stored in

1.

T.accepted_language

Mind that T(...) does not just translate strings but can also translated variables:

1.
2.

>>> a="test"
>>> print T(a)

In this case the world "test" in translated but, if not found and if the filesystem is writable, it will add it to the list of words to be translated in the language file.

Cookies

web2py uses the Python cookies modules for handling cookies.

Cookies from the browser are in request.cookies and cookies sent by the server are in response.cookies.

You can set a cookie as follows:

1.
2.
3.

response.cookies['mycookie'] = 'somevalue'
response.cookies['mycookie']['expires'] = 24 * 3600
response.cookies['mycookie']['path'] = '/'

The second line tells the browser to keep the cookie for 24 hours. The third line tells the browser to send the cookie back to any application (URL path) at the current domain.

The cookie can be made secure with:

1.

response.cookies['mycookie']['secure'] = True

A secure cookie is only sent back over HTTPS and not over HTTP.

The cookie can be retrieved with:

1.
2.

if request.cookies.has_key('mycookie'):
    value = request.cookies['mycookie'].value

Unless sessions are disabled, web2py, under the hood, sets the following cookie and uses it to handle sessions:

1.
2.

response.cookies[response.session_id_name] = response.session_id
response.cookies[response.session_id_name]['path'] = "/"

Application init

When you deploy web2py, you will want to set a default application, i.e., the application that starts when there is an empty path in the URL, as in:

http://127.0.0.1:8000

By default, when confronted with an empty path, web2py looks for an application called init. If there is no init application it looks for an application called welcome.

1.

default_application = "myapp"

Note: default_application first appeared in web2py version 1.83.

Here are three ways to set the default application:

• Call your default application "init".
• Set default_application to your application's name in routes.py
• Make a symbolic link from "applications/init" to your application's folder.
• Use URL rewrite as discussed in the next section.

URL Rewrite

web2py has the ability to rewrite the URL path of incoming requests prior to calling the controller action (URL mapping), and conversely, web2py can rewrite the URL path generated by the URL function (reverse URL mapping). One reason to do this is for handling legacy URLs, another is to simplify paths and make them shorter.

To use this feature, create a new file in the "web2py" folder called "routes.py" and define two lists (or tuples) of 2-tuples routes_in and routes_out. Each tuple contains two elements: the pattern to be replaced and the string that replaces it. For example:

1.
2.
3.
4.
5.
6.

routes_in = (
  ('/testme', '/examples/default/index'),
)
routes_out = (
  ('/examples/default/index', '/testme'),
)

With these routes, the URL:

http://127.0.0.1:8000/testme

is mapped into:

http://127.0.0.1:8000/examples/default/index

To the visitor, all links to the page URL looks like /testme.

The patterns have the same syntax as Python regular expressions. For example:

1.

('.*\.php', '/init/default/index'),

maps all URLs ending into ".php" to the index page.

Sometimes you want to get rid of the application prefix from the URLs because you plan to expose only one application. This can be achieved with:

1.
2.
3.
4.
5.
6.

routes_in = (
  ('/(?P<any>.*)', '/init/\g<any>'),
)
routes_out = (
  ('/init/(?P<any>.*)', '/\g<any>'),
)

There is also an alternative syntax that can be mixed with the regular expression notation above. It consists of using name instead of (?P<name>[\w_]+) or \g<name>. For example:

1.
2.
3.
4.
5.
6.
7.

routes_in = (
  ('/$c/$f', '/init/$c/$f'),
)

routes_out = (
  ('/init/$c/$f', '/$c/$f'),
)

would also eliminate the "/example" application prefix in all URLs.

Using the $ notation, you can automatically map routes_in to routes_out, provided you don't use any regular expressions. For example:

1.
2.
3.
4.
5.

routes_in = (
  ('/$c/$f', '/init/$c/$f'),
)

routes_out = [(x, y) for (y, x) in routes_in]

If there are multiple routes, the first to match the URL is executed. If no pattern matches, the path is left unchanged.

You can use $anything to match anything until the end of the line.

Here is a minimal "routes.py" for handling favicon and robots requests:

1.
2.
3.
4.
5.

routes_in = (
  ('/favicon.ico', '/examples/static/favicon.ico'),
  ('/robots.txt', '/examples/static/robots.txt'),
)
routes_out = ()

Here is a more complex example that exposes a single app "myapp" without unnecessary prefixes but also exposes admin, appadmin and static:

1.
2.
3.
4.
5.
6.
7.
8.

routes_in = (
  ('/admin/$anything', '/admin/$anything'),
  ('/static/$anything', '/myapp/static/$anything'),
  ('/appadmin/$anything', '/myapp/appadmin/$anything'),
  ('/favicon.ico', '/myapp/static/favicon.ico'),
  ('/robots.txt', '/myapp/static/robots.txt'),
)
routes_out = [(x, y) for (y, x) in routes_in[:-2]]

The general syntax for routes is more complex than the simple examples we have seen so far. Here is a more general and representative example:

1.
2.
3.
4.

routes_in = (
 ('140\.191\.\d+\.\d+:https://www.web2py.com:POST /(?P<any>.*)\.php',
  '/test/default/index?vars=\g<any>'),
)

It maps https POST requests to host www.web2py.com from a remote IP matching the regular expression

1.

140\.191\.\d+\.\d+

requesting a page matching the regular expression

1.

/(?P<any>.*)\.php!

into

1.

/test/default/index?vars=\g<any>

where \g<any> is replaced by the matching regular expression.

The general syntax is

1.

[remote address]:[protocol]://[host]:[method] [path]

The entire expression is matched as a regular expression, so "." should always be escaped and any matching subexpression can be captured using (?P<...>...) according to Python regex syntax.

This allows to reroute requests based on the client IP address or domain, based on the type of the request, on the method, and the path. It also allows to map different virtual hosts into different applications. Any matched subexpression can be used to built the target URL and, eventually, passed as a GET variable.

All major web servers, such as Apache and lighttpd, also have the ability to rewrite URLs. In a production environment that may be an option instead of routes.py. Whatever you decide to do we strongly suggest that you do not hardcode internal URLs in your app and use the URL function to generate them. This will make your application more portable in case routes should change.

Application-Specific URL Rewrite

The format of routes_app is identical to routes_in, except that the replacement pattern is simply the application name. If applying routes_app to the incoming URL does not result in an application name, or the resulting application-specific routes.py is not found, the base routes.py is used as usual.

Note: routes_app first appeared in web2py version 1.83.

Default Application, Controller and Function

The name of the default application, controller and function can be changed from init, default, and index respectively to another name by setting the appropriate value in routes.py:

1.
2.
3.

default_application = "myapp"
default_controller = "admin"
default_function = "start"

Note: These items first appeared in web2py version 1.83.

Routes on Error

You can also use "routes.py" to redirect the visitor to special actions in case there is an error on server. You can specify this mapping globally, for each app, for each error code, for each app and error code. Here is an example:

1.
2.
3.
4.
5.
6.

routes_onerror = [
  ('init/400', '/init/default/login'),
  ('init/*', '/init/static/fail.html'),
  ('*/404', '/init/static/cantfind.html'),
  ('*/*', '/init/error/index')
]

For each tuple the first string is matched against "[appname]/[error code]". If a match is found the user is redirected to the URL in the second string of the matching tuple. In case a ticket was issued, the ticket is passed to the new URL as a GET variable called ticket.

Unmatched errors display a default error page. This default error page can also be customized here:

1.
2.
3.
4.

error_message = '<html><body><h1>Invalid request</h1></body></html>'
error_message_ticket = '''<html><body><h1>Internal error</h1>
     Ticket issued: <a href="/admin/default/ticket/%(ticket)s"
     target="_blank">%(ticket)s</a></body></html>'''

The first variable contains the error message when an invalid application is requested. The second variable contains the error message when a ticket is issued.

Cron

The web2py cron provides the ability for applications to execute tasks at preset times, in a platform independent manner.

For each application, cron functionality is defined by a crontab file "app/cron/crontab", following the syntax defined in ref. ⁴⁵ (with some extensions that was web2py specific).

This means that every application can have a separate cron configuration and that cron config can be changed from within web2py without affecting the host OS itself.

Here is an example:

1.
2.
3.
4.
5.

0-59/1  *  *  *  *  root python /path/to/python/script.py
30      3  *  *  *  root *applications/admin/cron/db_vacuum.py
*/30    *  *  *  *  root **applications/admin/cron/something.py
@reboot root    *mycontroller/myfunction
@hourly root    *applications/admin/cron/expire_sessions.py

The last two lines in this example, use extensions to regular cron syntax to provide additional web2py functionality.

web2py cron has a some extra syntax to support web2py application specifics.

If the task/script is prefixed with an asterisk (*) and ends with ".py", it will be executed in the web2py environment. This means you will have all the controllers and models at your disposal. If you use two asterisks (**), the MODELs will not be executed. This is the recommended way of calling as it has less overhead and avoids potential locking problems.

Notice that scripts/functions executed in the web2py environment require a manual db.commit() at the end of the function or the transaction will be reverted.

web2py does not generate tickets or meaningful tracebacks in shell mode (in which cron is run). Make sure that your web2py code runs without errors before you set it up as a cron task, as you will likely not be able to see them when run from cron.

Moreover, be careful how you use models. While the execution happens in a separate process, database locks have to be taken into account in order to avoid pages waiting for cron tasks that be blocking the database. Use the ** syntax if you don't need to use the database in your cron task.

You can also call a controller function. There is no need to specify a path. The controller and function will be that of the invoking application. Take special care about the caveats listed above. Example:

1.

*/30  *  *  *  *  root *mycontroller/myfunction

If you specify @reboot in the first field in the crontab file, the given task will be executed only once, on web2py startup. You can use this feature if you want to pre-cache, check or initialize data for an application on web2py startup. Note that cron tasks are executed in parallel with the application --- if the application is not ready to serve requests until the cron task is finished, you should implement checks to reflect this. Example:

1.

@reboot  *  *  *  *  root *mycontroller/myfunction

Depending on how you are invoking web2py, there are four modes of operation for web2py cron.

• Soft cron: available under all execution modes
• Hard cron: available if using the built-in web server (either directly or via Apache mod_proxy)
• External cron: available if you have access to the system's own cron service
• No cron

The default is hard cron if you are using the built-in web server; in all other cases the default is soft cron.

Soft cron is the default if you are using CGI, FASTCGI or WSGI. Your tasks will be executed in the first call (page load) to web2py after the time specified in crontab (but after processing the page, so no delay to the user is visible). Obviously, there is some uncertainty exactly when the task will be executed depending on the traffic the site receives. Also, the cron task may get interrupted if the web server has a page load timeout set. If these limitations are not acceptable, see "external cron". Soft cron is a reasonable last resort, but if your web server allows other cron methods, they should be preferred over soft cron.

Hard cron is the default if you are using the built-in web server (either directly or via Apache mod_proxy). Hard cron is executed in a parallel thread, so unlike soft cron there are no limitations with regard to run time or execution time precision.

External cron is not default in any scenario, but requires you to have access to the system cron facilities. It runs in a parallel process, so none of the limitations of soft cron apply. This is the recommended way of using cron under WSGI or FASTCGI.

Example of line to add to the system crontab, (usually /etc/crontab):

1.

0-59/1 * * * * web2py cd /var/www/web2py/ && python web2py.py -C -D 1 >> /tmp/cron.output 2>&1

If you are running external cron, make sure you add the -N command line parameter to your web2py startup script or config so there is no collision of multiple types of cron.

In cases where you do not need any cron functionality within a particular process, you can use the -N command line parameter to disable it. Note that this might disable some maintenance tasks (like the automatic cleaning of session dirs). The most common use of this function:

• You already have set up external cron triggered from the system (most common with WSGI setups)
• You want to debug your application without cron interfering either with actions or with output

Background Processes and Task Queues

While cron is useful to run tasks at regular time intervals it is not always the best solution to run a background task. For this purpose web2py provides the ability to run any python script as it were insider a controller:

1.

python web2py.py -S app -M -N -R applications/app/private/myscript.py -A a b c

where -S app tells web2py to run "myscript.py" as "app", -M tells web2py to execute models, -N tells web2py not to run cron, and -A a b c passes optional command line arguments sys.args=['a','b','c'] to "myscript.py".

A typical test case consists of processing a queue.

Consider this model

1.
2.
3.
4.
5.

db.define_table('queue',
    Field('status'),
    Field('email'),
    Field('subject'),
    Field('message'))

and an application that enqueues messages to be send by

1.
2.
3.
4.

db.queue.insert(status='pending',
                email='you@example.com',
                subject='test',
                message='test')

The background processing script that sends the emails could be

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.

# in file myscript.py
import time
while True:
    rows = db(db.queue.status=='pending').select()
    for row in rows:
        if mail.send(to=row.email,
            subject=row.subject,
            message=row.message):
            row.update_record(status='sent')
        else:
            row.update_record(status='failed')
        db.commit()
    time.sleep(60) # check every minute

Notice that the mail object is defined in the db.py file in the scaffolding app, because of the -M option it is visible here. It may need to be configured in db.py to work properly. Also notice that it is important to commit any change as soon as possible in order not to lock the database to other concurrent processes.

This type of background process should not be executed via cron (except perhaps for cron @reboot) because you need to be sure that no more of one instance is running at the same time. With cron it is possible that if the process starts at cron iteration 1 and is not completed by cron iteration 2, so cron starts it again, and again, and again thus jamming the mail server.

Third Party Modules

web2py is written in Python, so it can import and use any Python module, including third party modules. It just needs to be able to find them.

Modules can be installed in the official Python "site-packages" directory or anywhere your application can find them.

Modules in "site-packages" directory are, as the name suggests, site-level packages. Applications requiring site-packages are not portable unless these modules are installed separately. The advantage of having modules in "site-packages" is that multiple applications can share them. Let's consider, for example, the plotting package called "matplotlib". You can install it from the shell using the PEAK easy_install command:

1.

easy_install py-matplotlib

and then you can import it into any model/controller/view with:

1.

import matplotlib

web2py Windows binary distributions has a site-packages in the top-level folder. The Mac binary distribution has a site-packages folder in the folder:

1.

web2py.app/Contents/Resources/site-packages

You can also install packages manually in the application "modules" folder. The advantage is that the module will be automatically copied and distributed with the application. If the application is called "test", you can import "mymodule" with:

web2py also provides a local_import function. Here is an example of usage:

1.

mymodule = local_import(mymodule)

The function looks for mymodule in the app local modules folder and imports it with the name on the left-hand side of equal.

This function takes three arguments: name, reload and app. When you specify reload=True, it will re-import the module upon each request; otherwise your python process will only import the module once. The default is reload=False. app is the name of the application from which to import the module, it defaults to request.application.

The reason for this function is that since one server can run many web2py instances you do not want to add their modules paths to sys.path else the lookup would become dependent on the order.

Execution Environment

While everything discussed here works fine, we recommend instead building your application using components, as described in chapter 13.

web2py provides the exec_environment function to allow you to access models and controllers directly. exec_environment creates a web2py execution environment, loads the file into it and then returns a Storage object containing the environment. The Storage object also serves as a namespace mechanism. Any Python file designed to be executed in the execution environment can be loaded using exec_environment. Uses for exec_environment include:

• Accessing data (models) from other applications.
• Accessing global objects from other models or controllers.
• Executing controller functions from other controllers.
• Loading site-wide helper libraries.

This example reads rows from the user table in the cas application:

1.
2.
3.

from gluon.shell import exec_environment
cas = exec_environment('applications/cas/models/db.py')
rows = cas.db().select(cas.db.user.ALL)

Another example: suppose you have a controller "other.py" that contains:

1.
2.

def some_action():
    return dict(remote_addr=request.env.remote_addr)

Here is how you can call this action from another controller (or from the web2py shell):

1.
2.
3.

from gluon.shell import exec_environment
other = exec_environment('applications/app/controllers/other.py', request=request)
result = other.some_action()

In line 2, request=request is optional. It has the effect of passing the current request to the environment of "other". Without this argument, the environment would contain a new and empty (apart from request.folder) request object. It is also possible to pass a response and a session object to exec_environment. Be careful when passing request, response and session objects --- modification by the called action or coding dependencies in the called action could lead to unexpected side effects.

The function call in line 3 does not execute the view; it simply returns the dictionary unless response.render is called explicitly by "some_action".

One final caution: don't use exec_environment inappropriately. If you want the results of actions in another application, you probably should implement an XML-RPC API (implementing an XML-RPC API with web2py is almost trivial). Don't use exec_environment as a redirection mechanism; use the redirect helper.

Cooperation

There are many ways applications can cooperate:

• Applications can connect to the same database and thus share tables. It is not necessary that all tables in the database are defined by all applications, but they must be defined by those applications that use them. All applications that use the same table, but one, must define the table with migrate=False.
• Applications can embed components from other actions using the LOAD helped (described in Chapter 13).
• Applications can share sessions.
• Applications can call each other's actions remotely via XML-RPC.
• Applications can access each other's files via the filesystem (assuming they share the same filesystem).
• Applications can call each other's actions locally using exec_environment as discussed above.
• Applications can import each other's modules using the syntax:
• Applications can import any module in the PYTHONPATH search path, sys.path.

One app can load the session of another app using the command:

1.

session.connect(request, response, masterapp='appname', db=db)

Here "appname" is the name of the master application, the one that sets the initial session_id in the cookie. db is a database connection to the database that contains the session table (web2py_session). All apps that share sessions must use the same database for session storage.

One application can load a module from another app using

1.

othermodule = local_import('othermodule',app='otherapp')

If a module function needs access to one of the core objects (request, response, session, cache, and T), the objects must be passed explicitly to the function. Do not let the module create another instance of the core objects. Otherwise, the function will not behave as expected.

WSGI

web2py and WSGI have love hate relationship. Our prospective is that WSGI was developed as a protocol to connect web servers to web applications in a portable way and we use it for that purpose. web2py at its core is a WSGI application: gluon.main.wsgibase. Some developers have pushed WSGI to its limits as a protocol for middleware communications and develop web applications as an onion with many layers (each layer being a WSGI middleware developed independently on the entire framework). web2py does not adopt this structure internally. This is because we feel the core functionality of a frameworks (handling cookies, session, errors, transactions, dispatching) can be better optimized for speed and security if they are handled by a single comprehensive layer.

Yet web2py allows you to use third party WSGI applications and middleware in three ways (and their combinations):

• You can edit the file "wsgihandler.py" and include any third party WSGI middleware.
• You can connect third party WSGI middleware to any specific action in your apps.
• You can call a third party WSGI app from your actions.

The only limitation is that you cannot use third party middleware to replace core web2py functions.

External Middleware

Consider the file "wsgibase.py":

1.
2.
3.
4.
5.
6.
7.
8.
9.

#...
LOGGING = False
#...
if LOGGING:
    application = gluon.main.appfactory(wsgiapp=gluon.main.wsgibase,
                                        logfilename='httpserver.log',
                                        profilerfilename=None)
else:
    application = gluon.main.wsgibase

When login is set to True, gluon.main.wsgibase is wrapped by the middleware function gluon.main.appfactory. It provides logging to the "httpserver.log" file. In a similar fashion you can add any third party middleware. We refer to the official WSGI documentation for more details.

Internal Middleware

Given any action in your controllers (for example index) and any third party middleware application (for example MyMiddleware which converts output to upper case, you can use a web2py decorator to apply the middleware to that action. Here is an example:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

class MyMiddleware:
    """converts output to upper case"""
    def __init__(self,app):
        self.app = app
    def __call__(self,environ, start_response):
        items = self.app(environ, start_response)
        return [item.upper() for item in items]

@request.wsgi.middleware(MyMiddleware)
def index():
    return 'hello world'

We cannot promise that all third party middleware will work with this mechanism.

Calling WSGI Applications

It is easy to call WSGI app from a web2py action. Here is an example:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.

def test_wsgi_app(environ, start_response):
    """this is a test WSGI app"""
    status = '200 OK'
    response_headers = [('Content-type','text/plain'),
                        ('Content-Length','13')]
    start_response(status, response_headers)
    return ['hello world!\n']

def index():
    """a test action that call the previous app and escapes output"""
    items = test_wsgi_app(request.wsgi.environ,
                          request.wsgi.start_response)
    for item in items:
        response.write(item,escape=False)
    return response.body.getvalue()

In this case the index action calls test_wsgi_app and escapes the returned value before returning it. Notice that index is not itself a WSGI app and it must use the normal web2py API (such as response.write to write to the socket.