Beep My Stuff

Beep My Stuff is a web site that makes it FREE and easy to create an online library of your Books, Movies, Video Games and Music. Check it out, all feedback is much appreciated.

It's built with Turbogears, SQLAlchemy, Genshi and countless other python modules. It's all sitting on Python, FreeBSD and PostgreSQL.

Just wanted to say thanks to all the pythonista's out there that helped me get this far

It's a little grainy but you should be able to see the settings I've been using for DX, VMWare and PKR.

This doesn't mean that future verisons of PKR or VMWare are going to work btw.

When uploading files to Beep My Stuff using SWFUpload I was getting this error:

data = self._sock.recv(self._rbufsize)
timeout: timed out

Uploading files to a TG1.0.x app is easy unless you want to do something nice like use SWFUpload to allow multiple files to be selected for upload.

The reason for this is that Flash has interpreted RFC 2046 one way and they python libraries/CherryPy interpret it another. Fortunately this is fixed in Turbogears 1.1+ but I am using 1.08 at the moment for Beep My Stuff so I needed a fix.

There is a patch in CherryPy that can be used to fix the problem and Chris Arndt has created a patch for TG1.1. You can easily apply the same code to your TG1.0x app to fix this problem as well.

Visit here and copy the contents of safemutlipart.py into a file in your project. Then you need to install the filter when Turbogears is starting up. Something like this will do the trick:

#force the multipart filter to be installed.
def install_safemutlipartfilter():
    from cherrypy import root
    root._cp_filters.extend([SafeMultipartFilter()])
 
def install_multipart_filter():
    from turbogears.startup import call_on_startup
    log.info("Installing multipart filter")
    call_on_startup.append(install_safemutlipartfilter)

Note that the code in SafeMultiPartFilter needs a config change to turn it on so add

safempfilter.on = True

to your app.cfg

hope that helps

The only slightly icky thing about the script is the regular expression to parse a line from the apache log file (in Combined Log Format). I got this from RegExp Buddy (pretty much the only reason I run Windows nowadays) but I am sure you can get similar expressions from other regular expression libraries.

Anyway, I'm just chucking this out there incase someone else finds it useful.

import re
import twill
 
test_server="my.test.server.com"
 
reobj = re.compile(r'^(?P<client>\S+)\s+(?P<auth>\S+\s+\S+)\s+\[(?P<datetime>[^]]+)\]\s+"(?:GET|POST|HEAD) (?P<file>[^ ?"]+)\??(?P
<parameters>[^ ?"]+)? HTTP/[0-9.]+"\s+(?P<status>[0-9]+)\s+(?P<size>[-0-9]+)\s+"(?P<referrer>[^"]*)"\s+"(?P<useragent>[^"]*)"$', re.MULTILINE)
browser = twill.get_browser()
 
def filter_url(url):
    return False
 
for line in open("apache.log"):
    match = reobj.search(line)
    if match is None:
        continue
 
    f = match.group("file")
    p = match.group("parameters")
    d = match.group("datetime")
    path = "?".join([f, p]) if p else f
 
    url = test_server+path
 
    if(filter_url(url)):
        continue
 
    try:
        print d, url
        browser.go(url)
    except ValueError:
        #this comes from twill parsing the HTML and it being malformed.
        #I don't really care about that, as long as I get the page.
        pass
 

For those that don't know SQLAlchemy-migrate allows you to version control database changes and easily upgrade and downgrade a database. Basically you write a python script with two functions: upgrade and downgrade. You test the script against the database, commit it to the SQLAlchemy-migrate version repository (not to be confused with your source control mechanism). Finally you upgrade your development database.

When it the time comes to deploy the new application you simply ask sqlalchemy-migrate to upgrade your database to the current version. sqlalchemy-migrate reads the current version of your schema from the database (via a custom table it inserts) and proceeds to upgrade your schema by running each upgrade script in turn.

Often you want your upgrades and downgrades to run within a transaction. Not because you expect them to fail but because while writing them you don't wont to leave the database partially upgraded or downgraded if your script fails. To do this I wrote a transaction decorator. Here is a template for an upgrade script:

#!/usr/bin/env python
# encoding: utf-8
 
from datetime import datetime
from sqlalchemy import *
from migrate import *
from migrate.changeset import *
 
metadata = MetaData(migrate_engine)
 
def transaction(f, *args, **kwargs):
    def wrapper(*args, **kwargs):
        connection = migrate_engine.connect()
        transaction = connection.begin()
        try:
            result = f(*args, **kwargs)
            transaction.commit()
            return result
        except:
            transaction.rollback()
            raise
        finally:
            connection.close()
 
    wrapper.__name__ = f.__name__
    wrapper.__dict__ = f.__dict__
    wrapper.__doc__ = f.__doc__
    return wrapper
 
@transaction
def upgrade():
    pass
 
@transaction
def downgrade():
    pass
 

I fill in the upgrade and downgrade functions and I'm done

I include the decorator in every script as it's good practice to make your scripts as independent as possible. If you imported it from a module you may improve it in the future and inadvertently break your old downgrade scripts.

I hope this helps you version control your database schema and data.

#if defined(__FreeBSD__) || (defined(_WIN32) && !defined(__GNUC__)) || defined(__sun__)
/* what's this, we have no round function either? */
static double round(double num)
{
  return (num >= 0) ? floor(num + 0.5) : ceil(num - 0.5);
}
#endif
 

However 'round' is defined in FreeBSD and has been since FreeBSD 5.3 (according to the manual page). The fix is simple, just remove the 'defined(__FreeBSD__) ||' part of the '#if' and you should be fine. Now you can 'easy_install .' psycopg2.

PS: I'd tried to raise a ticket but http://www.initd.org/ trac seems to have been down for ages.

What gets started is handled by a program called SystemStarter. On boot it looks through /Library/StartupItems for folders. Inside those folders it looks for a script with the same name as the folder and a plist called StartupParameters.plist.

So as super user:

cd /Library/StartupItems/
mkdir PostgreSQL
touch PostgreSQL/PostgreSQL
touch PostgreSQL/StartupParameters.plist
 

Now we need to fill in the details. In PostgreSQL/PostgreSQL enter:

#!/bin/sh
 
sudo -u postgres /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l /usr/local/pgsql/data/logfile start

This is a long line, but it's basically running postgres (via pg_ctl) as the user postgres. You'll need to change this to the whatever user you run postgres as and where ever your install of postgres is. I've used the standard names and directories so it's likely this will work for you too.

Now lets look at the plist. This file contains information for SystemStarter about when to start postgres, what postgres needs and provides and any messages to print to log files:

 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist SYSTEM "file://localhost/System/Library/DTDs/PropertyList.dtd">
<plist version="0.9">
<dict>
        <key>Description</key>
        <string>PostgreSQL</string>
        <key>Messages</key>
        <dict>
                <key>start</key>
                <string>Starting PostgreSQL</string>
                <key>stop</key>
                <string>Stopping PostgreSQL</string>
        </dict>
        <key>OrderPreference</key>
        <string>None</string>
        <key>Provides</key>
        <array>
                <string>PostgreSQL</string>
        </array>
</dict>
</plist>
 

You can test your new settings by typing sudo SystemStarter -n -D, fix any error messages, reboot and you should have postgres all the time

I use twitter every day to send out really quite boring messages. "I've had some chocolate" and "I like lossless encoding" are all stunning examples of how boring my twitters are. Sometimes they don't appear on twitter for hours... occasionally they don't appear at all.

As an engineer I think I have a fair idea of how twitter works BUT I have a code base full of my own problems. I tend to spend my time thinking about those rather then fixing twitters by proxy.

So I'm asking the internet, can you all just shut up about fixing twitter. I am sure they are aware of their deficiencies and how it would be better if they didn't do X or did do Y. Let's leave it to them to solve and if they don't I'm sure one of the genius twitter fixing bloggers can make his own distributed erlang über resilient systems that only blogger can create.

Here endeth the rant!