Django 1.8, Django Slumber and Django Async — 2015-04-10

I've been doing some work to expand the test runners for Django Slumber and Django Async to include Django 1.7 and Django 1.8 with mixed results.

The biggest change in 1.7 was the alteration of Django's transaction handling. This was a change that I never thought that the Django project would make. I don't yet have enough experience with these later versions of Django to say whether or not the handling really makes sense, but from reading the documentation I'm hopeful — however it seems to me that the defaults for new projects are still broken and unsafe, but it's still too early for me to have any specific recommendations in this regard.

How will transaction problems manifest themselves? In data corruption. If a request needs to properly update more than a single database row at a time then you will find that some of the database writes have happened and others have not. This will likely lead to other bugs down the line and seemingly “impossible” data situations.

You can think of incorrect transaction handling as a bug multiplier — any mistake at all is now able to reach out and affect every other part of the system in ways that are hard to track down and even harder to fix. These faults will not turn up under testing, and they will be nearly impossible to reproduce. They are however more than capable of taking down an entire system when they do manifest themselves.

This unsafe-by-default behaviour is compounded by the difficulty of testing transactions in Django. Django's test runner tries to be clever with transactions in order to speed up test execution, something that works against getting transactions correct in the code. Then there's the removal of the transaction middleware. It used to be quite simple to ensure that requests had safe enough defaults within the context of a request by simply using this middleware. Now each view must be correctly marked up in order to get safe behaviour — something nearly impossible to test for, and difficult to know and remember. Are you sure that every single view in your latest Django project correctly uses atomic?

I think only people who have a deep understanding of database transactions and why they're needed have any hope of getting Django to do the right thing now. My advice used to be to include the transaction middleware in your application and then the behaviour you get would be what you'd probably expect — when a request errors then no changes are made to the database. This is very simple to reason about for expert and beginner alike.

So, although Django Async's tests all pass with both 1.7 and 1.8 I have little faith that the underlying behaviour is actually safe in the context of a request, although I believe that the queue itself is still safe.

With Slumber it's slightly more complex. There have been extensive changes in the meta layer (that describes applications and models) and although I think I have most of that worked through in a more or less acceptable manner there are test failures that defy explanation — tests not being able to see the data that they've just written to the database. There is probably some stupid error on my part here, but with Django nose still broken for Django 1.8 it's also very hard to get any insight through the logs as to what is happening. A large number of incompatibilities have been fixed though.

I've also added the use of Django's atomic decorator to the views (where this is available, at least where it can be imported). I believe that this ought to make all Slumber operations safe under 1.7 and 1.8. This change is already in the develop branch, and will be published as 0.8.1 once I get a chance (certainly in the next day or two).