Other Recipes

Upgrading

In the "site" page of the administrative interface there is an "upgrade now" button. In case this is not feasible or does not work (for example because of a file locking issue), upgrading web2py manually is very easy.

Simply unzip the latest version of web2py over the old installation.

web2py does not upgrade any file in other existing applications.

How to Distribute your Applications as Binaries

It is possible to bundle your app with the web2py binary distribution and distribute them together. The license allows this as long you make clear in the license of your app that you are bundling with web2py and add a link to the web2py.com.

Here we explain how to do it for Windows:

• Create your app as usual
• Using admin, bytecode compile your app (one click)
• Using admin, pack your app compiled (another click)
• Create a folder "myapp"
• Download a web2py windows binary distribution
• Unzip it in folder "myapp" and start it (two clicks)
• Upload using admin the previously packed and compiled app with the name "init" (one click)
• Create a file "myapp/start.bat" that contains "cd web2py; web2py.exe"
• Create a file "myapp/license" that contains a license for your app and make sure it states that it is being "distributed with an unmodified copy of web2py from web2py.com"
• Zip the myapp folder into a file "myapp.zip"
• Distribute and/or sell "myapp.zip"

When users will unzip "myapp.zip" and click "run" they will see your app instead of the "welcome" app. There is no requirement on the user side, not even Python pre-installed.

For Mac binaries the process is the same but there is no need for the "bat" file.

Fetching an external URL

Python includes the urllib library for fetching urls:

1.
2.

import urllib
page = urllib.urlopen('http://www.web2py.com').read()

This is often fine, but the urllib module does not work on the Google App Engine. Google provides a different API for downloading URLs that works on GAE only. In order to make your code portable, web2py includes a fetch function that works on GAE as well as other Python installations:

1.
2.

from google.tools import fetch
page = fetch('http://www.web2py.com')

Pretty dates

It is often useful to represent a datetime not as "2009-07-25 14:34:56" but as "one year ago". web2py provides a utility function for this:

1.
2.
3.
4.

import datetime
d = datetime.datetime(2009,7,25,14,34,56)
from gluon.tools import prettydate
pretty_d = prettydate(d,T)

The second argument (T) must be passed to allow internationalization for the output.

Geocoding

If you need to convert an address (for example: "243 S Wabash Ave, Chicago, IL, USA") into geographical coordinates (latitude and longitude), web2py provides a function to do so.

1.
2.
3.

from gluon.tools import geocode
address = '243 S Wabash Ave, Chicago, IL, USA'
(latitude, longitude) = geocode(address)

The function geocode requires a network connection and it connects to the Google geocoding service for the geocoding. The function returns (0,0) in case of failure. Notice that the Google geocoding service caps the number of requests, so you should check their service agreement. The geocode function is built on top of the fetch function and thus it works on GAE.

Pagination

This recipe is a useful trick to minimize database access in case of pagination, e.g., when you need to display a list of rows from a database but you want to distribute the rows over multiple pages.

Start by creating a primes application that stores the first 1000 prime numbers in a database.

Here is the model db.py:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

db = DAL('sqlite://primes.db')
db.define_table('prime',Field('value','integer'))
def isprime(p):
    for i in range(2,p):
        if p%i==0: return False
    return True
if len(db().select(db.prime.id))==0:
   p=2
   for i in range(1000):
       while not isprime(p): p+=1
       db.prime.insert(value=p)
       p+=1

Now create an action list_items in the "default.py" controller that reads like this:

1.
2.
3.
4.
5.
6.
7.

def list_items():
    if len(request.args): page=int(request.args[0])
    else: page=0
    items_per_page=20
    limitby=(page*items_per_page,(page+1)*items_per_page+1)
    rows=db().select(db.prime.ALL,limitby=limitby)
    return dict(rows=rows,page=page,items_per_page=items_per_page)

Notice that this code selects one more item than is needed, 20+1. The extra element tells the view whether there is a next page.

Here is the "default/list_items.html" view:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

{{extend 'layout.html'}}

{{for i,row in enumerate(rows):}}
{{if i==items_per_page: break}}
{{=row.value}}<br />
{{pass}}

{{if page:}}
<a href="{{=URL(args=[page-1])}}">previous</a>
{{pass}}

{{if len(rows)>items_per_page:}}
<a href="{{=URL(args=[page+1])}}">next</a>
{{pass}}

In this way we have obtained pagination with one single select per action, and that one select only selects one row more then we need.

httpserver.log and the Log File Format

The web2py web server logs all requests to a file called:

httpserver.log

in the root web2py directory. An alternative filename and location can be specified via web2py command-line options.

New entries are appended to the end of the file each time a request is made. Each line looks like this:

1.

127.0.0.1, 2008-01-12 10:41:20, GET, /admin/default/site, HTTP/1.1, 200, 0.270000

The format is:

1.

ip, timestamp, method, path, protocol, status, time_taken

Where

• ip is the IP address of the client who made the request
• timestamp is the date and time of the request in ISO 8601 format, YYYY-MM-DDT HH:MM:SS
• method is either GET or POST
• path is the path requested by the client
• protocol is the HTTP protocol used to send to the client, usually HTTP/1.1
• status is the one of the HTTP status codes ⁹¹
• time_taken is the amount of time the server took to process the request, in seconds, not including upload/download time.

In the appliances repository ³⁴ , you will find an appliance for log analysis.

This logging is disabled by default when using mod_wsgi since it would be the same as the Apache log.

Populating Database with Dummy Data

For testing purposes it is convenient to be able to populate database tables with dummy data. web2py includes a Bayesian classifier already trained to generate dummy but readable text for this purpose.

Here is the simplest way to use it:

1.
2.

from gluon.contrib.populate import populate
populate(db.mytable,100)

It will insert 100 dummy records into db.mytable. It will try to do intelligently by generating short text for string fields, longer text for text fields, integers, doubles, dates, datetimes, times, booleans, etc. for the corresponding fields. It will try to respect requirements imposed by validators. For fields containing the word "name" it will try to generate dummy names. For reference fields it will generate valid references.

If you have two tables (A and B) where B references A, make sure to populate A first and B second.

Because population is done in a transaction, do not attempt to populate too many records at once, particularly if references are involved. Instead, populate 100 at the time, commit, loop.

1.
2.
3.

for i in range(10):
    populate(db.mytable,100)
    db.commit()

You can use the Bayesian classifier to learn some text and generate dummy text that sounds similar but should not make sense:

1.
2.
3.
4.

from gluon.contrib.populate import Learner, IUP
ell=Learner()
ell.learn('some very long input text ...')
print ell.generate(1000,prefix=None)

Send an SMS

A better way is to use the phone companies themselves to relay the SMS. Each phone company has an email address uniquely associated every cell-phone number the SMS can be sent as an email to that phone number.

web2py comes with a module to help in this process:

1.
2.
3.

from gluon.contrib.sms_utils import SMSCODES, sms_email
email = sms_email('1 (111) 111-1111','T-Mobile USA (tmail)')
mail.sent(to=email,subject='test',message='test')

SMSCODES is a dictionary that maps names of major phone companies to the email address postfix. The sms_email function takes a phone number (as a string) and the name of a phone company and returns the email address of the phone.

Accepting Credit Card Payments

web2py provides multiple ways for your application to accept credit card payments. For example:

Google Checkout Plugin

http://web2py.com/plugins/static/web2py.plugin.google_checkout.w2p

PayPal

http://www.web2pyslices.com/main/slices/take_slice/9

Authorize.Net

The first two mechanisms above delegate the process of authenticating the payee to an external service. While this is the best solution for security (your app does not handle any credit card information at all) it makes the process cumbersome (the user must login twice; for example, once with your app, and once with Google) and does not allow your app to handle recurrent payments in an automated way.

There are times when you need more control. For this reason we provide integration out of the box with the Authorize.Net API (the module was developed by John Conde and slightly modified). Here is an example of workflow and all the variables that are exposed:

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.

from gluon.contrib.AuthorizeNet import AIM
payment = AIM(login='cnpdev4289',
              transkey='SR2P8g4jdEn7vFLQ',
              testmod=True)
payment.setTransaction(creditcard, expiration, total, cvv, tax, invoice)
payment.setParameter('x_duplicate_window', 180) # three minutes duplicate windows
payment.setParameter('x_cust_id', '1324')       # customer ID
payment.setParameter('x_first_name', 'Agent')
payment.setParameter('x_last_name', 'Smith')
payment.setParameter('x_company', 'Test Company')
payment.setParameter('x_address', '1234 Main Street')
payment.setParameter('x_city', 'Townsville')
payment.setParameter('x_state', 'NJ')
payment.setParameter('x_zip', '12345')
payment.setParameter('x_country', 'US')
payment.setParameter('x_phone', '800-555-1234')
payment.setParameter('x_description', 'Test Transaction')
payment.setParameter('x_customer_ip', socket.gethostbyname(socket.gethostname()))
payment.setParameter('x_email', 'you@example.com')
payment.setParameter('x_email_customer', False)

payment.process()
if payment.isApproved():
    print 'Response Code: ', payment.response.ResponseCode
    print 'Response Text: ', payment.response.ResponseText
    print 'Response: ', payment.getResultResponseFull()
    print 'Transaction ID: ', payment.response.TransactionID
    print 'CVV Result: ', payment.response.CVVResponse
    print 'Approval Code: ', payment.response.AuthCode
    print 'AVS Result: ', payment.response.AVSResponse
elif payment.isDeclined():
    print 'Your credit card was declined by your bank'
elif payment.isError():
    print 'It did not work'
print 'approved',payment.isApproved()
print 'declined',payment.isDeclined()
print 'error',payment.isError()

Notice the code above uses a dummy test account. You need to register with Authorize.Net (it is not a free service) and provide your own login, transkey, testmode=True or False to the AIM constructor.

Twitter API

Here are some quick examples on how to post/get tweets. No third-party libraries are required, since Twitter uses simple RESTful APIs.

Here is an example of how to post a tweet:

1.
2.
3.
4.
5.
6.
7.
8.

def post_tweet(username,password,message):
    import urllib, urlib2, base64
    import gluon.contrib.simplejson as sj
    args= urllib.urlencode([('status',message)])
    headers={}
    headers['Authorization'] = 'Basic '+base64.b64encode(username+':'+password)
    request = urllib2.Request('http://twitter.com/statuses/update.json', args, headers)
    return  sj.loads(urllib2.urlopen(req).read())

Here is an example of how to receive tweets:

1.
2.
3.
4.
5.
6.
7.

def get_tweets():
    user='web2py'
    import urllib
    import gluon.contrib.simplejson as sj
    page = urllib.urlopen('http://twitter.com/%s?format=json' % user).read()
    tweets=XML(sj.loads(page)['#timeline'])
    return dict(tweets=tweets)

For more complex operations, refer to the Twitter API documentation.

Streaming Virtual Files

It is common for malicious attackers to scan web sites for vulnerabilities. They use security scanners like Nessus to explore the target web sites for scripts that are known to have vulnerabilities. An analysis of web server logs from a scanned machine or directly in the Nessus database reveals that most of the known vulnerabilities are in PHP scripts and ASP scripts. Since we are running web2py, we do not have those vulnerabilities, but we will still be scanned for them. This is annoying, so we like to respond to those vulnerability scans and make the attacker understand their time is being wasted.

One possibility is to redirect all requests for .php, .asp, and anything suspicious to a dummy action that will respond to the attack by keeping the attacker busy for a large amount of time. Eventually the attacker will give up and will not scan us again.

This recipe requires two parts.

A dedicated application called jammer with a "default.py" controller as follows:

1.
2.
3.

class Jammer():
   def read(self,n): return 'x'*n
def jam(): return response.stream(Jammer(),40000)

When this action is called, it responds with an infinite data stream full of "x"-es. 40000 characters at a time.

The second ingredient is a "route.py" file that redirects any request ending in .php, .asp, etc. (both upper case and lower case) to this controller.

1.
2.
3.

route_in=(
 ('.*\.(php|PHP|asp|ASP|jsp|JSP)','jammer/default/jam'),
)

The first time you are attacked you may incur a small overhead, but our experience is that the same attacker will not try twice.

Jython

web2py normally runs on CPython (the Python interpreter coded in C), but it can also run on Jython (the Python interpreter coded in Java). This allows web2py to run in a Java infrastructure.

Even though web2py runs with Jython out of the box, there is some trickery involved in setting up Jython and in setting up zxJDBC (the Jython database adaptor). Here are the instructions:

• Download the file "jython_installer-2.5.0.jar" (or 2.5.x) from Jython.org
• Install it:

1.

java -jar jython_installer-2.5.0.jar

• Download and install "zxJDBC.jar" from ⁹³
• Download and install the file "sqlitejdbc-v056.jar" from ⁹⁴
• Add zxJDBC and sqlitejdbc to the java CLASSPATH
• Start web2py with Jython

1.

/path/to/jython web2py.py

At the time of writing we only support sqlite and postgres on Jython.