I started using Tox in all my projects this year. It's got to be one of the best addition for testing multiple platforms and package versions.

What new programming technique did you learn in 2011?

I love testing, and have been good at writing tests for my code, but I've been naturally moving towards a stricter TDD testing style, and I've been some great benifits from this.

What’s the name of the open source project you contributed the most in 2011? What did you do?

I contributed mostly to a number of my own new (or newly released) projects; html5video, znc_mailer, django-appregister, django-urlmiddleware, django-consent, django-gauge.

What was the Python blog or website you read the most in 2011?

Planet Python and Python Redit. I've also started reading Hackernews more, which fairly often has Python related content.

What are the three top things you want to learn in 2012?

Pyramid, More of Python's internals, Haskell. The last has been on my list for a while.

What are the top software, app, or lib you wish someone would write in 2012?

I'd like somebody to build a CI server that uses ep.io to deploy testing apps and run the tests and then deploy to the production app.

Want to do your own list ? here’s how:

• copy-paste the questions and answer to them in your blog
• tweet it with the #2012pythonmeme hashtag

After speaking with Carl, I'm marking this as wontfix because it is non-obvious as to whether pickling a Signal should include the registered receivers, and how that interacts with the weak referencing, since there's no obvious semantic it seems better not to guess.

I really enjoy working with both Django's signal framework and Celery tasks. Today it occured to me that it would be useful to combine the two and have “asynchronous signals”.

Here is the solution that I came up with, read on below if you want to see how I arrived at this and why we need to monkey patch.

from celery.task import task
from django.db.models.signals import post_save

from myproject.models import MyModel

# Warning. Monkey patch.
from django.dispatch.dispatcher import Signal
def reducer(self):
    return (Signal, (self.providing_args,))
Signal.__reduce__ = reducer

# With the patch done, we can now connect to celery tasks.
@task(ignore_result=True)
def async_post_save(sender, instance, **kwargs):
    # do something with the instance.
    pass
post_save.connect(async_post_save.delay, sender=MyModel)

The first solution that occured to me was to use an intermediate function that triggered the task. This works fine and doesn't require anything clever.

from celery.task import task
from django.db.models.signals import post_save

from myproject.models import MyModel

@task
def async_post_save(instance):
    # do something with the instance.
    pass

def post_save_reciever(sender, instance, **kwargs):
    async_post_save.delay()
post_save.connect(post_save_reciever)

However, this adds an extra level of redirection to code that shouldn't be needed. Why can't be connect to tasks directly?

@task(ignore_result=True)
def async_post_save(sender, instance, **kwargs):
    # do something with the instance.
    pass
post_save.connect(async_post_save.delay, sender=MyModel)

This almost works, however, in the kwargs signal recievers are passed an instance of django.display.dispatcher.Signal and this contains an instance of threading.Lock - an object that can't be pickled. This leads me to the monkey patch that was shown at the start of this article which simply adds a reduce method to the Signal class that alters the pickle behaviour and only pickles the provided_args property of the Signal instance.

Incidently, you'll notice that I added ignore_result=True to each of the tasks. While this isn't required, its not generally standard practice for signals recievers to return anything, so you will probably want to do this too.

View the ticket I opened a ticket to track this idea This link

Getting started is really easy, there is a good quick start on the Vagrant website, or you can try my version to get a development environment I've been using - which is a basic box with some Python tools and postgres. Vagrant projects are initialised for a directory and simply contains a file called Vagrantfile (and an automatically generated .vagrant file). Generally speaking I would then do this once in each project root allowing the vagrant file to be version controlled and configured for an individual project. Then each developer can create a virtual machine from the same config and this keeps everybody on the same page.

sudo gem update --system
gem install vagrant
cd /path/to/project
vagrant init

You may not need the first step there, but I did on my mac. There is a detailed setup guide for different platforms here. So if you have any problems make sure you read that first.

In your current directory you should now have a file named Vagrantfile. This is a simple Ruby based configuration file. The Vagrantfile is well documented so we'll skip past that bit. Instead, copy the following into your vagrant file.

Vagrant::Config.run do |config|

  config.vm.box = "lucid32"
  config.vm.box_url = "http://files.vagrantup.com/lucid32.box"

  config.vm.forward_port("web", 8000, 8000)
  config.vm.network("33.33.00.10")

  config.vm.provision :chef_solo do |chef|

    chef.recipe_url = "http://cloud.github.com/downloads/d0ugal/chef_recipes/cookbooks.tar.gz"
    chef.cookbooks_path = [:vm, "cookbooks"]

    chef.add_recipe "main"
    chef.add_recipe "python"
    chef.add_recipe "postgres"

    chef.json.merge!({

      :project_name => "project_name",

      :system_packages => [],
      :python_global_packages => [],
      :python_packages => [],

    })

  end

end

After copying in, look for project_name and change that to something on your liking or you can just leave it for now.

If you have already been using vagrant, you will likely have the lucid box already. The vagrant quick start tips download it and call it 'base' since this is a common name I have changed it to something more explicit and safe. However, to avoid re-downloading you may want to change lucid32 to base in your config file.

What does this do? It setups up a new Ubuntu Lucid machine and runs some chef recipies against that are downloading from my chef github repository. The base machine that its built on is downloaded from the url in the config this is provided by the Vagrant team (I've not had a chance to make my own boxes yet).

The Chef cookbook then installs some system wide packages, creates a virtual environment, installs postgres and creates a database. These are not deploy ready scripts but rather more hacky scripts to quickly bootstrap a development env.

After saving, you can now do vagrant up this will take a while as it needs to download the ubuntu box to create the VM from. The time delay is mostly due to downloading of the box and also various packages like postgres so be warned if you have a slow or limited connection, this is likely to sum up to around 800mb or so.

After its finished, you should be able to do this.

vagrant ssh
workon project_name
ls -la

You will then be running from the virtual machine, but be in an activated Python virtual environment and in a directory that is mapped to the host machine. Thus the result of that should show the files for the directory that you started in. In my case I generally then need to do pip install -r requirements.txt and after that I can run the project - be it a Django website, or something different.

After this you can pretty much carry on as normal. There are a few things to note that may effect you. Any tools that rely on accessing the Python interpreter will not work as its not on a remote machine, I don't have a need for this so I've not worked out a solution. I simply use the python debugger directly in the ssh session. Editors like pyDev will likely loose some functionality here but there may be a work around...

When using Django's runserver, you need to specify an IP address. I've created a alias to make this easier. This assumes you are still connected to the virtual machine but if not, cd into the project directory and run vagrant ssh. How awesome is it to not need to remember IP's or logins?

djr
# is the same as
python manage.py runserver 0.0.0.0:8000

These shortcuts are best left for another post perhaps, but some more can be seen in my dotfiles repository.

To access the site itself, you'll then need to go to the IP address specified in the config file. In this case the full path will be http://33.33.00.10:8000 or since we have set the port forwarding in the config above you should be able to go to localhost:8000.

I'm still working out my complete workflow for vagrant but I'm using this machine as a base point for much of my work and creating a new machine for each project. The best thing so far for me is a recorded development environment that I can use to create a VM now or in 6 months when revisiting an older project. My main problem at the moment is the effort required to make big changes (new configs, new recipes etc.) but this should get easier as I make more.

If you have any idea's or suggestions please let me know.

So, step up Blogofile. Firstly, what is it? Essentially it is a static site generator. You get to use Python, a templating language and a fairly simple but standard MVC architecture. However, the net result isn't a WSGI Python app, but rather a static media. A whole set of HTML and your other files like HTML and CSS too. One clever trick it uses is to take advantage of folder structure and index.html files to create nice URL's.

As I use Disqus for all my comments the content of my blog rarely ever changes. I add or update a post now and then but on a day to day basis, it rarely changes. Rendering the page on each request is really inefficient and caching each page is overkill. My CV for example is stored as markdown on my site and rendered to html, a pdf or a doc as requested - now I only need to do this once.

For me, the main advantage comes from editing almost everything in vim. Now if I want to work on a post or do some maintenance I don't need to go through the Django admin and work in there. I can simply edit in vim and push back to the git repository. Which also means everything is version controlled and maintained in a sane way which fits into my workflow.

Hosting isn't really an issue for me with Django as with tools like Fabric, Chef and Gunicorn it doesn't need to be painful anymore. However, there is something to be said for the simplicity of deploying a completely static site. I don't need to worry about a wsgi server, a database or anything else and backing up isn't an issue.

I do have a few gripes with Blogfile though, they are fairly minor but nothin' is perfect. I'm not a fan of Mako, I'd much rather edit Jinja2 templates. I'm pondering adding this functionality and the nice thing about blogofile is that it should be fairly straightforward to do so. When working on your Blogofile site there is a handy server for testing locally but you need to rebuild manually to see the changes. This is a little annoying and I usually have two tabs open, one doing the rebuild every 10 seconds and one serving.

Incidentally, you can see the code for my Blogofile Blog on github.

Find out more at the new (but basic) website and follow us on twitter or join the mailing list.

The group is going to be meeting on the 4th Tuesday of each month. The next being on the 24th August at Bert's Bar.

Hope to see some of you there - please let us know if your coming so we can get the numbers right.

If you don't have the polls app from the end of part four and don't want to do it again you can grab it from my github. The code added in this tutorial is also available in the master branch of the repository.

This tutorial begins where Tutorial 4 left off. We're continuing the Web-poll application and will focus on testing our application and proving that it works as expected.

The testing environment

Let's first look at how you run the tests, make sure you are in the mysite directory and run the command python manage.py test. You will see output similar to this:

Creating test database...  
Creating table auth_permission  
Creating table auth_group  
Creating table auth_user  
Creating table auth_message  
Creating table django_admin_log  
Creating table django_content_type  
Creating table django_session  
Creating table django_site  
Creating table polls_poll  
Creating table polls_choice  
Installing index for auth.Permission model  
Installing index for auth.Message model  
Installing index for admin.LogEntry model  
Installing index for polls.Choice model  
...................................  
------------------------------------------------------------------  
Ran 35 tests in 0.565s

OK  
Destroying test database...

You have successfully ran the test suite for the project. If your output doesn't look like this and it appears some of the tests have failed you may want to take a moment and go back to make sure your project matches the end of part 4.

When you run the test suite Django creates a new test database, synchronises your applications and loads any fixtures into the database. Each test is then executed in turn wrapped inside a database transaction so it can be rolled back after each test is completed. At the end of the tests Django destroys the test database for you.

What did we just test? manage.py test runs all the tests for each of the applications in your INSTALLED_APPS setting. All of the django.contrib applications ship with tests, as should all of yours. This is great as we can use this to test that the contrib applications are all configured correctly.

Run the same command again but include the verbose flag so we can see more information about what actually just happened. To do this run the command python manage.py test -v 2. There will be more output this time showing each of the tests that were carried out with the test outcome.

What is a test?

With Django (and Python in general) there are two main ways to write tests for your projects test suite; doctests and unit tests. In Django both of these use the standard python modules doctest and unittest. Doctests are written in Python docstrings and unit test are defined with classes. Good generic examples and explanations can be found on both of these in the Django documentation here and here. If your unsure which to use, read this.

For the sake of being succinct we will focus on unit tests in this tutorial.

Writing a test

In part one it was noted in passing that when you created your polls application the file tests.py was created. The more adventurous of you will have taken a look and seen a simple (but redundant) example unit test and doctest. Django's test runner by default runs any tests that you create in tests.py in the application package. This is similar to the auto detection of admin configurations in the admin.py files.

To run only the tests for a specific application run the following command python manage.py test polls. This will run the two default tests that are present in the polls application we created earlier in the tutorial.

Delete the contents of the tests.py file and add the following test case.

from datetime import datetime

from django.test import TestCase

from mysite.polls.models import Poll

class PollTest(TestCase):

    def setUp(self):
        question="What is your favourite colour?"
        now = datetime.now()
        self.poll = Poll.objects.create(question=question, pub_date=now)
        self.poll.choice_set.create(choice="Red", votes=0)
        self.poll.choice_set.create(choice="Bue", votes=0)
        self.poll.choice_set.create(choice="Green", votes=0)

    def test_models(self):
        self.assertEqual(self.poll.choice_set.all().count(), 3)

In this test case the setUp method creates a new poll and adds three choices to the poll. Remember since the test runner creates its own database there is no data yet. setUp is called at the start of each test defined within your test case class. The test verifies that the number of choices in the created poll is equal to 3. A test is a method that starts with test_ and is a property of a class extending TestCase.

Try playing with this and making the test fail by changing the number or adding/removing choices. The test can also end with an error result if there is an uncaught exception, this can be done by adding Poll.objects.get(pk=2) as there isn't a poll with that id. It's worth familiarising yourself with the different possible test results.

When running your the tests by running python manage.py test polls the output displays a single . for each test that passes. If a test fails a F will be displated and if there is an error E will be displayed. A failure is when one of the assertions fails and an error happens when there is an uncaught exception while running the test.

Let's add another test case that does something more useful.

# ...
from django.test import Client

from mysite.polls.models import Poll, Choice

class PollTest(TestCase):

    # ...

    def test_voting(self):
        c = Client()
        # Perform a vote on the poll by mocking a POST request.
        response = c.post('/polls/1/vote/', {'choice': '1',})
        # In the vote view we redirect the user, so check the 
        # response status code is 302.
        self.assertEqual(response.status_code, 302)
        # Get the choice and check there is now one vote.
        choice = Choice.objects.get(pk=1)
        self.assertEqual(choice.votes, 1)

In this example we make use of the Django test client. Using the client we are able to simulate requests without the need for a server, rather the request object is mocked and the view is invoked with the mock request. In this test we create a POST request that mocks a vote on the poll and then checks both the status_code of the response (to check we have been redirected) and verifies the number of votes has increased.

Test driven development

Test driven development is the practice of writing tests that fail and show what the system should do and then write or change the code to make the test(s) pass.

Next we want to add some Ajax to our application, to do this we want to be able to call the vote with an Ajax request and recieve some simple information we can work with rather than the full html response. First we will write the test for how we want this to work. We want the system to return '1' on a valid vote and '0' on an invalid vote in the http response.

class PollTest(TestCase):

    # ...

    def test_ajax_vote(self):

        c = Client()

        # Extra parameters to make this a Ajax style request.
        kwargs = {'HTTP_X_REQUESTED_WITH':'XMLHttpRequest'}

        # A valid vote
        response = c.post('/polls/1/vote/', {'choice': '1',}, **kwargs)
        self.assertEqual(response.status_code, 200)
        self.assertEqual(response.content, '1')

        # A invalid vote - choice doesn't exist
        response = c.post('/polls/1/vote/', {'choice': '10',}, **kwargs)
        self.assertEqual(response.status_code, 200)
        self.assertEqual(response.content, '0')

        # An invalid vote - poll doesn't exist
        response = c.post('/polls/2/vote/', {'choice': '1',}, **kwargs)
        self.assertEqual(response.status_code, 404)

Add this to your tests and then run the test suite. You should get output similar to this;

F..
===================================================================
FAIL: test_ajax_vote (mysite.polls.tests.PollTest)
-------------------------------------------------------------------
Traceback (most recent call last):
  File "/mysite/polls/tests.py", line 41, in test_ajax_vote
    self.assertEqual(response.status_code, 200)
AssertionError: 302 != 200

-------------------------------------------------------------------

We have not updated our view, so rather than returning something useful for the Ajax request the server has returned a redirect to the results page. This then means the test fails at the first hurdle when we check the response code.

Now all we need to do is update the code in the vote view to make the tests pass. Change your view so it matches the following.

def vote(request, poll_id):
    p = get_object_or_404(Poll, pk=poll_id)
    try:
        selected_choice = p.choice_set.get(pk=request.POST['choice'])
    except (KeyError, Choice.DoesNotExist):
        # bad vote, return '0'
        if request.is_ajax():
            return HttpResponse("0")
        # Redisplay the poll voting form.
        return render_to_response('polls/poll_detail.html', {
            'object': p,
            'error_message': "You didn't select a choice.",
        })
    else:
        selected_choice.votes += 1
        selected_choice.save()
        # vote saved, return '1'
        if request.is_ajax():
            return HttpResponse("1")
        # Always return an HttpResponseRedirect after successfully dealing
        # with POST data. This prevents data from being posted twice if a
        # user hits the Back button.
        return HttpResponseRedirect(reverse('poll_results', args=(p.id,)))

Re-run the test suite and it should now pass. You have successfully written your first tests for the polls app and even done a little bit of test driven development.

What next and further reading

Now you should have a good understanding of testing basics and hopefully a better idea of how to test your applications. Testing is really quite straight forward much of the time, you simply write a bit more code to make sure the code you have actually works - it's invaluable when you get into the routine and have a comprehensive test suite.

Here are a few resources that you should look at for taking the next step.

• Python unit test documentation
• Python doctest documentation
• The Django testing documentation
• Eric Holscher's awesome DjangoCon talk about testing

"Code without tests is broken as designed"
- Jacob Kaplan-Moss

Leaving you with those wise words I trust you will now take as much advantage of testing as you can and strive to create good tests for all your code.

JavaScript Balls

I've done this with a few languages, the first was with Visual Basic 6 at college and since then I've done it again with Perl and JavaScript. I did start one in Java but don't think I ever finished. I've never got around to creating a Python port and I'm going to have to change that but for the moment here it is in JavaScript.

See it in action here

Note, you will need a browser that is capable of handling <canvas> so in other words - not Internet Explorer 6. Also I'm afraid the size is fixed so it wont look nice on small resolutions.

The code is really shoddy as its one of the first things I coded in JavaScript. I'd like to re-write it at some point to compare how I do it now that I feel comfortable with the language.

I remember when I wrote this my computer wasn't fast enough to render it very smoothly and the only browser it worked on was FireFox (and probably Safari but I didn't, and still don't, have a mac). I'm glad to see how much the browser landscape has changed since then.

The Others

I've totally lost the perl implementation - I think I probably left it on a university computer somewhere. I've got the VB6 code but I don't have visual studio (installed) anymore and I don't have a floppy drive to access it!

My Python version, well I think I might do that later if I have time. I've done very little GUI code with Python so it would make for a fun exercise.

Whats changed?

I find it interesting that as programmers we can often find our way around an unfamiliar language enough to get things working but generally you don't take advantage of that languages features. I guess this could be likened to writing in a language you don't really know (anything but english in my case) and stumbling with grammar and spelling¹.

Looking back, the most interesting thing is probably how different I would approach the problem now. I used Prototype, a library I don't really have anything against but it doesn't really fit with how I see JavaScript now as it adds various features that make it feel more like a 'traditional' object oriented language with classes and classical inheritance. The code relies on global objects and has events written into the HTML.

Quite frankly most of the code is rubbish but thats one of the ways you learn! Given the task now I would use jQuery and might consider using some HTML5 or CSS3 features. That actually sounds quite fun - I sense a mini project coming up.

Basically the solution is to use type() as I'm sure many of you know. If your doing that already there isn't much for you here. If your messing around with 'self.fields["name"]' in your forms then read on.

Lets take a simple use case; a quiz system. You can think of it like who wants to be a millionaire. We will have a Question with four possible answers. So two models...

from django.db import models

class Question(models.Model):
    test = models.CharField(max_length=128)

class Answer(models.Model):
    question = models.ForeignKey(Question)
    test = models.CharField(max_length=20)
    is_correct = models.BooleanField(default=False)

Basically we want a form that stores both the question and the answers and checks the answer is a valid choice. You could do it this way.

from django import forms

class QuizForm(forms.Form):

    def __init__(self,question, *args, **kwargs):
        super(QuizForm, self).__init__(*args, **kwargs)

        self.fields['question'] = forms.IntegerField(widget= \
            forms.HiddenInput, initial=question.id)
        self.fields['answers'] = forms.ModelChoiceField(queryset= \
            question.answers_set)

Now in this example that's actually not too bad. It's still a bit hacky as we tap into the fields dict after calling the parents constructor. I have a variant of this where I moved the field generation out of the __init__ but that doesn't really change much - just gives you the option of calling it later. Remember this is a very simple example, what if you need to generate a class based on 20 parameters?

So how can we solve this with type()? Well lets step back a minute and quickly refresh ourselves with the builtin function type(). There are two ways to use this function the first is by calling type(object) and the type of that object is returned. The second is to create classes at runtime by using type() as a class constructor.

First lets look at an example that has exactly with the same result as the previous example.

def quiz_form_factory(question):

    properties = {
        'question' : forms.IntegerField(widget=forms.HiddenInput, \
            initial=question.id),
        'answers' : forms.ModelChoiceField(queryset= \
            question.answers_set)
    }

    return type('QuizForm', (forms.Form,), properties)

And there we go, it's as simple as that. When using type() to construct classes it takes three parameters type(class_name, base_classes_tuple, properties_dict); the name of the class, the base classes it inherits from and the properties the created class will have.

Besides the above implementation the usage is slightly different.

# start with a random question
question = Question.objects.all().order_by('?')[0]

# method A
quiz_form = QuizForm(question)
# or with POST data
quiz_form = QuizForm(question, request.POST)

# method B
QuizForm = quiz_form_factory(question)
quiz_form = QuizForm()
# or with POST data
quiz_form = QuizForm(request.POST)

Again, perhaps I should have chosen a more complex example as the the first method may not look to bad and requires less code but its not nice. Using this approach basically class initialisation with the preparation and modification of said class. The second is much clearer as you explicitly generate a class then create an instance of it.

The main advantage for me is the clarify this gives you and the code used to make it dynamic is clear and better structured. It's also worth mentioning that with a type() constructed form it behaves exactly like a regular form after creation where with method A the developer needs to pass in the question instance each time and be aware of this requirement and how it varies from a typical Django style form.

You first need to locate the MySQL config file named my.cnf. On CentOS 5.2 it is located at /etc/my.cnf but this will probably vary across platforms.

Then in that file find the [mysqld] and add add the following line below it as shown below.

[mysqld]
default-storage-engine = innodb

Finally, make sure you restart MySQL.

service mysqld restart

It's pretty easy wen you know how, and you'll now had transactional tables by default.

Alterantivaley though, you could use a much better database.

Sorry. Demo no longer exists, just copy the code and it should work straight off ;)

function isCapslock(e){

    e = (e) ? e : window.event;

    var charCode = false;
    if (e.which) {
        charCode = e.which;
    } else if (e.keyCode) {
        charCode = e.keyCode;
    }

    var shifton = false;
    if (e.shiftKey) {
        shifton = e.shiftKey;
    } else if (e.modifiers) {
        shifton = !!(e.modifiers & 4);
    }

    if (charCode >= 97 && charCode <= 122 && shifton) {
        return true;
    }

    if (charCode >= 65 && charCode <= 90 && !shifton) {
        return true;
    }

    return false;

}