Thursday, May 14, 2009

One Way To Escape The Cargo Cult

Testing frameworks proliferate in Ruby: RSpec, Test::Unit, Shoulda, Stump, RR, Matchy, Context, minitest, test-spec, Bacon, Cucumber, Watir, and the inevitable one-liners. It's really getting out of hand. Seriously, Bacon? Cucumber? What's next, Lettuce and Ham? Why don't these people quit coding testing frameworks and just make a sandwich already?

Presenting Sandwich: a new TDD gem which combines Bacon and Cucumber

This proliferation frustrates me because it makes cargo-culting very tempting. If you want to work on a wide variety of projects, you are guaranteed to run into some testing code you've never seen before. You may be tempted to guess how it works, rather than figure it out, simply because testing frameworks are becoming so balkanized that it's hard to justify the effort to really learn any particular one.

Don't do it! Don't cargo cult anything, ever. You'll regret it. Bugs you can explain always win over bugs you can't.

A few days ago I discovered what I believe is a limitation in RR, although again, given the sheer number of gems and plugins involved, it's hard to say for sure.

This started life as a Twitter client test. It morphed into a "what is this testing framework doing?" test. I was copying and pasting like a script kiddie when things went haywire. One of my real tests gave me bizarre, inexplicable behavior, so I coded this up to test what the testing framework was doing. It was, as they say, metaprogramming.

I remember when RSpec became the new hotness, and everybody said Test::Unit was old and busted. People said Rails should move to RSpec, but it didn't happen. At the time I thought that foolish, now I see a lot of common sense in it. Test::Unit might not be perfect, but it's good enough to get a lot done, and getting a lot done is much more valuable than discovering the One True Testing Framework.

I can see two explanations for this excessive biodiversity. One is simply that the ideal testing solution hasn't been discovered yet. The other is that Rails is a test-obsessed culture (which is good) that is not above endlessly debating minutiae (which is bad). I think both these explanations are true. Obviously all this experimentation will give us better and better testing frameworks as time goes on, but I wish the process was smoother. "Test all the fucking time" is truth. "Test in a different way every fucking time", less so.