Making WooMe

Practically everyone agrees that selenium testing is a nightmare. The biggest problem is that you end up with too many tests. At WooMe, we think we have a secret sauce for making sure our testing doesn’t become bloatastic.

We have very few core Selenium tests. The payments system, registration and email are the only things we test on every release. Everything else is done with acceptance tests. Acceptance tests, at WooMe, are tests that check a particular feature, bug fix or even a text change.

Sometimes they’re really silly. Just check that a CSS class is being applied to a button, for example.

Acceptance tests have a lifetime. We leave them in for at most a few weeks. Once they’ve been in that long, we remove them. Of course, we are constantly writing acceptance tests but the purging means we don’t get much Selenium bloat.

Our tests are all defined using the Django testrunner. We have something like this in an __init__.py:

from simple import *
from profile import *
from alert import *
from vip import *
from ticket_6955 import *
from ticket_6997 import *
from ticket_7353 import *
from ticket_7412 import *
from ticket_7454 import *
from ticket_7445 import *
from ticket_7424 import *

Here’s the command I use to review the acceptance test queue:

sed -rne 's/.*ticket_([0-9]+) .*/\1/p' \
  | while read ticket ; do traccmd.py td $ticket | grep summary ; done 

traccmd.py is a tool we wrote to pull data out of trac fairly easily.

Are acceptance tests any good? Well, they have their weaknesses but as a strategy for maintaining a light touch but low risk testing environment they work very well. This is because they exercise two important failure points:

• the developer writing the test in the first place
  acceptance tests are easy enough to write that people do write them, they’re mostly not onerous so developers are ok with putting a bit more time into getting something that mostly makes sense.
• they strongly test merge failure
  a very common cause of breakage is that a feature or bug fix will be merged away by another developer with conflicting changes. Acceptance tests for the first few weeks of a feature make it much less likely this will happen.

Of course, when we do throw a test away, we don’t really throw it away. It’s still in version control and we could potentially go and do a long test where we pull out all tests and run them. We’ve never had to do that yet though.