Blue Sky On Mars

One theory of unit testing is that your tests should all be entirely independent. I’ve heard that Google actually has a test runner that randomizes the order of the tests on each run, so you’re guaranteed that they’re independent (otherwise they fail).

I want my test files to be independent, but I find that having one test lead into another is often a good way to reuse fixture and avoid making my test fixture more complicated than it needs to be.

Nose (and py.test) make this really easy, because they run the tests in the order in which they appear in the file. (Note that if you’re using Nose to run unittest.TestCases, those tests still run alphabetically as they do under Python’s built-in test runner). The file order running of tests is very natural and leads to few surprises. It’s very easy to make a string of tests that build up a nice, complicated fixture testing little pieces as they move along. I’ll mention that most of my tests will actually run just fine in any order, but I like being able to count on the order when it’s handy to do so.

All of that is a lead-in to my fun from today. I’m using Fuzzyman’s Mock module to drop in stand-ins for things that are annoying to test. Mock comes with a handy @patch decorator that will replace the callable of your choice with a Mock on the way in, and restore it automatically on the way out. It works great and is really easy to use.

I was thrown off for a bit today because I put a @patch on one of my test functions, and it started failing because a file that it expected to see did not exist! Removing the decorator brought the file back. That file was created in a test function earlier in the file.

If you’re nodding knowingly at this point, you’ve probably worked with decorators a fair bit before. I’ve been known to use more than my fair share of decorators. I’ve found that while the syntax is great, some of the side effects can be really annoying. That’s why Paver‘s decorators aren’t true decorators. They just register behavior rather than replacing the function object.

So, what’s the relationship between using a decorator and the file-based ordering of tests? To sort by file order, Nose looks at the test function’s func_code.co_firstlineno. In the case of ,my test today, that was around 250. However, when I applied the patch decorator, the function was no longer my original function… it was the function that Mock’s decorator returned (the one that does the nifty swapping in and out of the Mock). That was around line 97 of mock.py.

When Nose went to sort the functions by func_code.co_firstlineno, after decoration my test function was thought to be at line 97. I naively thought that I’d just change func_code.co_firstlineno. Nope, read only. Then I tried sticking an object in place of func_code that returned the value I wanted for co_firstlineno. Nope, func_code has to be a ‘code’ object.

Luckily, Nose itself has a solution to this problem. The function object can have a ‘compat_co_firstlineno’ attribute on it, and that attribute will be used instead of func.func_code.co_firstlineno. A one-line change to mock.py was all it took.