Riot: for fast, expressive and focused unit tests
Riot is a new Ruby test framework by Justin Knowlden that focuses on faster testing. Justin was frustrated with his slow running test suites, despite employing techniques such as using factories, mocks and avoiding database access. He realized that a slow-running suite makes one reluctant to run it or expand it - not good.
With Riot, each test consists of a block which forms a single assertion on the topic of the test, keeping the tests focused. Tests run in a specific context, and the setup code is only run once per context, further contributing to the speed of your test suite, and unlike some Ruby test frameworks, such as Shoulda, that rely on or are based on Test::Unit, Riot has taken a new approach for speed purposes. In Justin's own comparisons, Riot comes out about twice as fast as Test::Unit.
Here's an example Riot test (from the README):
context "a new user" do
setup { User.new(:email => 'foo@bar.com') }
asserts("email address") { topic.email }.equals('foo@bar.com')
end
Riot's comprehensive README also includes lots of examples and details on how to modify your Rakefile to run your Riot test suite in different frameworks. The full documentation is available online here.
You can install Riot as a gem from Gemcutter:
sudo gem sources -a http://gemcutter.org sudo gem install riot
Justin also has a spin-off project called Riot Rails, which includes some Rails-related macros for testing your Ruby On Rails code, and Alex Young has written a Javascript port of Riot which is worth checking out too. He also has his own look at Riot and demonstrates how Riot can reduce redundancy in tests.
[ad] Codebase is a fast & reliable git, mercurial & subversion hosting service with complete project management built-in - ticketing, milestones, wikis & time tracking - all under one roof. Click here to try it - free.
November 1, 2009 at 5:07 am
So I need a suite of 10,000 tests to notice 0.9 second difference? Thats the problem with these benchmarks. To the majority of projects, the test framework lack of performance is negligible.
The only thing to judge this project by is syntax.
November 1, 2009 at 5:09 am
That being said, Rubyspec could stand to benefit.
November 1, 2009 at 12:22 pm
You've got a point about the speed thing, Brian, but I also thing that Riot has got something to offer in terms of ease/style of coding - e.g. one assertion per test and a contextual setup.
November 1, 2009 at 12:31 pm
Riot shouldn't be advertised by its speed, if it doesn't really matter... My first thought was also: "wow, 1.5s is painfully slow execution of 10k tests... 0.7s *so much* faster". And after that I didn't even bother to go further and check syntax...
November 1, 2009 at 12:36 pm
It's probably not the strongest example, but cutting out time is still important at the lower end. Anything that makes testing more immediate makes testing more attractive. It's like a Web page loading in 1.5 seconds instead of 0.9 seconds - there's a cognitive difference, even if it's slight at first.
November 1, 2009 at 1:32 pm
Not compelling enough for a switch
November 1, 2009 at 11:14 pm
"..., despite employing techniques such as using factories, mocks and avoiding database access."
I've seen a huge amount of tests with mocks and in my opinion, mocks should be used *just* for simulating external interface like Twitter API or so. For example in my current job we have somewhat about 8 MB test code with mocks and it's really bloody painful to work with – as Yehuda Katz told on MerbCamp (great video about testing BTW) "Broken interface => broken tests. Working interface => passings tests". It's what with mocks just doesn't work.
November 2, 2009 at 2:34 am
"...but cutting out time is still important at the lower end. Anything that makes testing more immediate makes testing more attractive."
Then write your unit tests in Java - they'll run faster. You won't notice the difference but benchmarks will.
Seriously,random optimizations don't 'add up' - the trick to optimizing something that's too slow is to find the bottlenecks. It doesn't sound like this addresses them. So it's a waste of time. Well, actually there may be other advantages to the framework but this post is so silly it works as negative advertising for the framework. Your web site example doesn't really apply - shaving .6 seconds off one web request is perceptible; shaving the same amount off of 10,000 tests is not.
http://martinfowler.com/ieeeSoftware/yetOptimization.pdf
Steve
November 3, 2009 at 8:41 am
>It's like a Web page loading in 1.5 seconds instead of 0.9
> seconds - there's a cognitive difference, even if it's slight at first.
I agree, but it would be great for 10, maybe 100 tests, not for 10,000. In this case we are talking about microseconds.