Beep My Stuff

Beep My Stuff is a web site that makes it FREE and easy to create an online library of your Books, Movies, Video Games and Music. Check it out, all feedback is much appreciated.

It's built with Turbogears, SQLAlchemy, Genshi and countless other python modules. It's all sitting on Python, FreeBSD and PostgreSQL.

Just wanted to say thanks to all the pythonista's out there that helped me get this far

When uploading files to Beep My Stuff using SWFUpload I was getting this error:

data = self._sock.recv(self._rbufsize)
timeout: timed out

Uploading files to a TG1.0.x app is easy unless you want to do something nice like use SWFUpload to allow multiple files to be selected for upload.

The reason for this is that Flash has interpreted RFC 2046 one way and they python libraries/CherryPy interpret it another. Fortunately this is fixed in Turbogears 1.1+ but I am using 1.08 at the moment for Beep My Stuff so I needed a fix.

There is a patch in CherryPy that can be used to fix the problem and Chris Arndt has created a patch for TG1.1. You can easily apply the same code to your TG1.0x app to fix this problem as well.

Visit here and copy the contents of safemutlipart.py into a file in your project. Then you need to install the filter when Turbogears is starting up. Something like this will do the trick:

#force the multipart filter to be installed.
def install_safemutlipartfilter():
    from cherrypy import root
    root._cp_filters.extend([SafeMultipartFilter()])
 
def install_multipart_filter():
    from turbogears.startup import call_on_startup
    log.info("Installing multipart filter")
    call_on_startup.append(install_safemutlipartfilter)

Note that the code in SafeMultiPartFilter needs a config change to turn it on so add

safempfilter.on = True

to your app.cfg

hope that helps

The only slightly icky thing about the script is the regular expression to parse a line from the apache log file (in Combined Log Format). I got this from RegExp Buddy (pretty much the only reason I run Windows nowadays) but I am sure you can get similar expressions from other regular expression libraries.

Anyway, I'm just chucking this out there incase someone else finds it useful.

import re
import twill
 
test_server="my.test.server.com"
 
reobj = re.compile(r'^(?P<client>\S+)\s+(?P<auth>\S+\s+\S+)\s+\[(?P<datetime>[^]]+)\]\s+"(?:GET|POST|HEAD) (?P<file>[^ ?"]+)\??(?P
<parameters>[^ ?"]+)? HTTP/[0-9.]+"\s+(?P<status>[0-9]+)\s+(?P<size>[-0-9]+)\s+"(?P<referrer>[^"]*)"\s+"(?P<useragent>[^"]*)"$', re.MULTILINE)
browser = twill.get_browser()
 
def filter_url(url):
    return False
 
for line in open("apache.log"):
    match = reobj.search(line)
    if match is None:
        continue
 
    f = match.group("file")
    p = match.group("parameters")
    d = match.group("datetime")
    path = "?".join([f, p]) if p else f
 
    url = test_server+path
 
    if(filter_url(url)):
        continue
 
    try:
        print d, url
        browser.go(url)
    except ValueError:
        #this comes from twill parsing the HTML and it being malformed.
        #I don't really care about that, as long as I get the page.
        pass
 

For those that don't know SQLAlchemy-migrate allows you to version control database changes and easily upgrade and downgrade a database. Basically you write a python script with two functions: upgrade and downgrade. You test the script against the database, commit it to the SQLAlchemy-migrate version repository (not to be confused with your source control mechanism). Finally you upgrade your development database.

When it the time comes to deploy the new application you simply ask sqlalchemy-migrate to upgrade your database to the current version. sqlalchemy-migrate reads the current version of your schema from the database (via a custom table it inserts) and proceeds to upgrade your schema by running each upgrade script in turn.

Often you want your upgrades and downgrades to run within a transaction. Not because you expect them to fail but because while writing them you don't wont to leave the database partially upgraded or downgraded if your script fails. To do this I wrote a transaction decorator. Here is a template for an upgrade script:

#!/usr/bin/env python
# encoding: utf-8
 
from datetime import datetime
from sqlalchemy import *
from migrate import *
from migrate.changeset import *
 
metadata = MetaData(migrate_engine)
 
def transaction(f, *args, **kwargs):
    def wrapper(*args, **kwargs):
        connection = migrate_engine.connect()
        transaction = connection.begin()
        try:
            result = f(*args, **kwargs)
            transaction.commit()
            return result
        except:
            transaction.rollback()
            raise
        finally:
            connection.close()
 
    wrapper.__name__ = f.__name__
    wrapper.__dict__ = f.__dict__
    wrapper.__doc__ = f.__doc__
    return wrapper
 
@transaction
def upgrade():
    pass
 
@transaction
def downgrade():
    pass
 

I fill in the upgrade and downgrade functions and I'm done

I include the decorator in every script as it's good practice to make your scripts as independent as possible. If you imported it from a module you may improve it in the future and inadvertently break your old downgrade scripts.

I hope this helps you version control your database schema and data.

#if defined(__FreeBSD__) || (defined(_WIN32) && !defined(__GNUC__)) || defined(__sun__)
/* what's this, we have no round function either? */
static double round(double num)
{
  return (num >= 0) ? floor(num + 0.5) : ceil(num - 0.5);
}
#endif
 

However 'round' is defined in FreeBSD and has been since FreeBSD 5.3 (according to the manual page). The fix is simple, just remove the 'defined(__FreeBSD__) ||' part of the '#if' and you should be fine. Now you can 'easy_install .' psycopg2.

PS: I'd tried to raise a ticket but http://www.initd.org/ trac seems to have been down for ages.

Subversion

First and foremost, subversion is a source control tool but it's also a safety net. I can't see how you can call yourself a developer if you don't have source control. From day one I got myself an account on Webfaction set up subversion and started versioning everything. They allow https connection so everything is safe and because I don't live in their data center it's off site as well

Trac

Again on Webfaction, I installed Trac. Trac is a project management webtool. It hooks in to svn so you can see repository changes as a timeline and see checkins as colored diffs. It also has a ticket system and wiki. I found the wiki to be invaluable. It allowed me to track my thoughts in a 'low barrier to entry' fashion. It's so easy to add a wiki page, there really isn't any excuse for not doing it! The ticket/task tracking is basic but fine for small teams. There is a simple notion of a milestone and a release but the metrics for measuring progress just aren't there. For me though, this was fine. Web projects are relatively simple and single man projects don't require metrics really.

TextMate

TextMate is a text editor primarily aimed at programmers but useful for anyone who needs to edit structured documents where the content is more important then the layout. I first ran in to TextMate while I was evaluating Ruby. The Ruby community seems to be dominated by macbook pro users sitting in Starbucks, bashing code in to TextMate and buzzing on coffee. Although I decided not to use Ruby I did stick with TextMate. It really is the premier text editor for OS X. It's the first editor that has replaced emacs for me. I didn't use any extensions for TextMate but I did use PyChecker and JSLint with it to check my python and Javascript code on save.

CSSEdit

CSSEdit is an editor devoted to CSS. The cool thing is that it has live editting. So as you edit the CSS file for a site, it changes the preview of the site. This is quite frankly brilliant and very cheap compared to the time it saved me.
Combined with YUI's CSSReset I can proudly say I 'ported' New Metal Army from Firefox to Safari, IE and Opera in a few days with minimal difficulty. CSSReset should be the basis for ALL website development and Yahoo should get a fucking medal for creating it. Cheers guys

Pixelmator

Pixelmator is a nice image editor the supports PSD's and layers. I got it as part of a MacHeist bundle and just started using it rather then paying hundreds of quid for Photoshop. Because it supports PSD's I can still work with artists who quite rightly want to use Photoshop.

VMWare Fusion

Working on a mac makes it hard to get Internet Explorer working. I don't want or need a PC so VMWare comes to the rescue. VMWare is an OS virtualisation package that basically allows me to run various versions on Windows, Linux and BSD in a window. So I can run IE 5.5, 6.0 and 7 in separate virtual instances and check compatibility across IE5.5+, Safari, FireFox 2+ and Opera 9+. Since I'm a one man operation time consuming and boring tasks like this need to be made as painless as possible... or they won't get done.

Firebug

Firebug is an extension for Firefox that allows you to examine pages and page elements to see how their layout has been calculated but it also allows you to debug Javascript inside the browser. It far exceeds IE's rather forced script debugger compatibility or .NET integration. This tool saved me months of debugging and more importantly sped up the learning curve for web development.

Wingware IDE

There were times when I needed to debug fiddly python code and the python command line debugger just wasn't cutting it. I turned to Wingware's IDE for debugging and it served me very well. I only used it a few times though.

NeoOffice

Neooffice is a Mac port of Open Office. It's a good replacement for Claris or MS Office. I only needed to write a few letters to lawyers and such but it's good to not have to pay £500 for the privilege. I did use it a lot for opening up 50Mb csv files and it coped quite well with it.

Python

Well this is pretty obvious, TurboGears uses Python so I am forced to work with it. Well luckily python is one of the best languages to do anything in. With a vast array of modules it's really quick to create scripts to process data for your website. As an affiliate for many music companies (iTunes, HMV, Play, Amazon, ...) I get access to huge XML and CSV files with their latest prices, stock and shipping information. Because of python's flexibility it is easy to merge this data in to New Metal Army.

TurboGears

TurboGears is a no brainer here. It's the web framework New Metal Army sits upon. For me though TG was more then a framework. TG comes with a great community bundled for free. I learned a lot just reading the mailing list. I also learned a lot by reading the code for TurboGears itself. It's well written and cleanly constructed. There is some advanced python on there but I don't think anything is overly complicated. Python is a great language to read and I think more programmers need to learn that they CAN dive in to most code and quickly work out what is going on.

PostgreSQL

Postgres has always been my database of choice. Unfortunately I have no real reason for this other then it's always covered my needs and it is standards compliant so I expect it to cover my needs for the foreseeable future. TurboGears uses an ORM mapper (SQLAlchemy) between it and the database so I could swap Postgres for MSSQL, MySQL, Firebird, Oracle or a number of others. Postgres serves New Metal Army VERY well.

Here is a quick summary of New Metal Army:

• Pulls together the news from the top metal and rock webites from around the world. All news is tagged and associated with appropriate bands
• Has full gig listings for the uk rock/metal scene with full venue details and links to buy tickets
• Brings together band details from wikipedia, flickr, youtube, musicbrainz and amazon

As it currently stands there are 2500+ bands, 400+ gigs, 100+ venues, 500,000+ band pictures and 1000000+ band videos and it's growing all the time.

I've learned a lot about the internet doing this project. Most of my time was spent researching different ways to scrape and understand websites. Here is a summary of what I learned:

• The internet is a mess. There are a lot of sites with HTML that isn't just slightly wrong but VERY wrong.
• BeautifulSoup goes a long way to parsing the bad markup in the internet.
• Structural markup (bold, italic, div etc) were the least of my worries. The semantic meaning of the data is very hard to discern. Now this is obvious but when I started I didn't really think about it. I assumed that MicroFormats would come to my rescue... but no one uses them (well very few people use them). I must confess that I have a task in my Trac to add them but it's not a priority at the moment. So even I don't use them!
• Even with MicroFormats the data I needed is tainted by human input (like most of the internet). I deal a lot with band names and bands names are often misspelt and adjectives like 'a' and 'the' added and removed.
• Following on from spellings: people in the world still insist on using foreign languages that have funny accents and despite what you read unicode, while simple to code in python, is not simple to think in. When will everyone learn to speak local
• A human algorithm for tagging things is... to tag things with a scatter gun approach. The code base looks in flickr for pictures of bands. It does this by looking at the tags associated with a picture. The trouble is when someone takes some photos at a concert they mass upload them and tag them all with the names of all the bands at the concert and sometimes with bands like the bands at the concert... not to useful.
• Python does scale pretty well to large projects... but it's easy to get 'leaks' which make your heap grow and grow. They aren't really leaks they are normally lists of things that you are forgetting to tidy up and python is dutifully holding them for you. This is something I really didn't think about until I deployed the site and associated tools and noticed that my 'Job Runner' (threaded application that ran various jobs) just grew and grew.
• Before you start writing anything in Python look for a module on the internet that does it. If you can't find a module to do it then write your own. However, write you own module knowing this: As soon as you have finished, somehow you will find a module that is more complete then yours on the internet and you will kick yourself for not finding it before!
• Zombies rock... everyone likes zombie and Simon (my Zombie) is no exception to this

I intend to add some articles on how I did various bits and bobs over the next few weeks.

[Sat Jan 19 11:32:42 2008] [error] [client 207.155.93.149] mod_wsgi (pid=1292): Exception occurred within WSGI script '/home/m/release1.0/apache/turbogears.wsgi'.
[Sat Jan 19 11:32:42 2008] [error] [client 207.155.93.149] Traceback (most recent call last):
[Sat Jan 19 11:32:42 2008] [error] [client 207.155.93.149]   File "/home/m/release1.0/apache/turbogears.wsgi", line 67, in <module>
[Sat Jan 19 11:32:42 2008] [error] [client 207.155.93.149]     import turbogears
[Sat Jan 19 11:32:42 2008] [error] [client 207.155.93.149] ImportError: No module named turbogears

That's odd. I've not uninstalled TurboGears and my background processes that #import TurboGears still work. Infact if I go to the python command line and type #import TurboGears it all works... bugger.

To complicate matters (in this case) I use a workingenv to contain a very specific version of TurboGears and all of it's dependencies. In order for the wsgi script to access the sandbox environment I use an excellent script which tweaks the runtime environment to include the paths in a working env. My first though is that something here had gone wrong. So I turned to prints and some basic error capture.

# Load all distributions into the working set.
from pkg_resources import working_set, Environment
 
env = Environment(root)
env.scan()
 
distributions, errors = working_set.find_plugins(env)
for dist in distributions:
    working_set.add(dist)

Printing out errors revealed:

errors:
{Amara 1.2.0.2 (/usr/home/m/tgenv1_0_32/lib/python2.5/Amara-1.2.0.2-py2.5.egg):
   DistributionNotFound(Requirement.parse('4Suite-XML>=1.0.2'),),
TGCaptcha 0.11 (/usr/home/m/tgenv1_0_32/lib/python2.5/TGCaptcha-0.11-py2.5.egg):
   DistributionNotFound(Requirement.parse('pycrypto>=2.0.1'),)}

Well I hadn't uninstalled those packages and I'm pretty sure that freebsd-update hadn't uninstalled them so where the hell have they gone! Looking in the workingenv sandbox package directory

ls -la /usr/home/m/tgenv1_0_32/lib/python2.5
4Suite_XML-1.0.2-py2.5-freebsd-6.2-RELEASE-i386.egg
Amara-1.2.0.2-py2.5.egg
BeautifulSoup-3.0.5-py2.5.egg
Cheetah-2.0.1-py2.5-freebsd-6.2-RELEASE-i386.egg
Cheetah-2.0rc8-py2.5-freebsd-6.2-RELEASE-i386.egg
CherryPy-2.2.1-py2.5.egg
...
PasteScript-1.3.6-py2.5.egg
PyProtocols-1.0a0dev_r2302-py2.5-freebsd-6.2-RELEASE-i386.egg
Routes-1.7.1-py2.5.egg
RuleDispatch-0.5a0.dev_r2306-py2.5-freebsd-6.2-RELEASE-i386.egg
SQLAlchemy-0.3.10-py2.5.egg
...
moved_aside_site.py
psycopg2-2.0.6-py2.5-freebsd-6.2-RELEASE-i386.egg
pycrypto-2.0.1-py2.5-freebsd-6.2-RELEASE-i386.egg
python_dateutil-1.3-py2.5.egg
...
setuptools.pth
simplejson-1.7.3-py2.5-freebsd-6.2-RELEASE-i386.egg
...
 

BUGGER, there are packages in there with the OS version number in that need to be updated:

easy_install -U amara
easy_install -U pycrypto
easy_install -U psycopg2
...

fixed all the problems and finally the site is up again So I've fixed the problem but I don't know why my other processes and the python command line worked. If anyone knows, I love to know too. Cheers.

The specific bit of code I am going to post is the python code to look for and load a series of python plugins. Plugins (in this case) are just classes that are subclasses of the Plugin base class. This plugin base class dictates the API that all plugins must implement. Here is an example plugin abstract base class

class Plugin(object):
    def setup(self):
        """called before the plugin is asked to do anything"""
        raise NotImplementedError
 
    def teardown(self):
        """called to allow the plugin to free anything"""
        raise NotImplementedError
 
    def domagic(self):
        """do whatever it is the plugin does"""
        raise NotImplementedError
 

Here the is code to look for and return a list of classes that can be instantiated:

def find_subclasses(path, cls):
    """
    Find all subclass of cls in py files located below path
    (does look in sub directories)
 
    @param path: the path to the top level folder to walk
    @type path: str
    @param cls: the base class that all subclasses should inherit from
    @type cls: class
    @rtype: list
    @return: a list if classes that are subclasses of cls
    """
 
    subclasses=[]
 
    def look_for_subclass(modulename):
        log.debug("searching %s" % (modulename))
        module=__import__(modulename)
 
        #walk the dictionaries to get to the last one
        d=module.__dict__
        for m in modulename.split('.')[1:]:
            d=d[m].__dict__
 
        #look through this dictionary for things
        #that are subclass of Job
        #but are not Job itself
        for key, entry in d.items():
            if key == cls.__name__:
                continue
 
            try:
                if issubclass(entry, cls):
                    log.debug("Found subclass: "+key)
                    subclasses.append(entry)
            except TypeError:
                #this happens when a non-type is passed in to issubclass. We
                #don't care as it can't be a subclass of Job if it isn't a
                #type
                continue
 
    for root, dirs, files in os.walk(path):
        for name in files:
            if name.endswith(".py") and not name.startswith("__"):
                path = os.path.join(root, name)
                modulename = path.rsplit('.', 1)[0].replace('/', '.')
                look_for_subclass(modulename)
 
    return subclasses
 

and here is how you would call it:

classes = find_subclasses("./pluginsfolder/", Plugin)
 
#lets create an instance of the first class
inst = classes[0]()
 

So there you go, create folder of python files with classes in them that subclass the Plugin class and you are away.

Basically If I was using a PPC Mac I would be able to decrypt the data. It seems they have included the PPC only shared library for blowfish. Hmm. Well as it happens I have a copy of the x86 library that I can use to decrypt the data.

If you are suffering from this problem:

1. Download the library here.
2. Look inside any OSX app by right clicking (or CTRL clicking) on the application icon and selecting 'Show Package Contents'
   
3. Now navigate down to

   Contents/Resources/lib/python2.5/lib-dynload/Crypto/Cipher/

4. Copy the Blowfish.so into the Cipher Folder, there should be one there already so you'll be asked if you want to replace the existing one

You should be able to run the decryptor.app now and get access to your MP3s.
Whoever makes this application should be able to make the decryption library Universal so lets hope they do that in the future.