A critical look at the current state of Ruby testing

Posted by Dan Croak

Nov 07

I’ve been feeling lately that there is a splintering in the Ruby community over testing tools. RSpec is very popular. Shoulda, too. context, matchy, bacon & test/spec all have their fans.

There’s nothing inherently wrong with experiments. Diversity is a virtue. We all learn from one another.

However, standards also have their place. Testing is in the DNA of the Ruby community. Why should I have to ask questions on a mailing list of unknown helpfulness when I can find all my answers in the Ruby Standard Library?

There’s a simple fact that no one seems to be talking about: all of these testing tools are solutions to non-existant problems.

Let’s take a critical look at the current state of Ruby testing.

Why context blocks are bad

They look like this in Shoulda: 
1
2
3
4
5
6
7
8
9

context "a GET to #index" do
  setup do
    2.times { Factory(:user) }
    get :index
  end

  should_respond_with :success
  should_assign_to :users, :equals => "User.ordered"
end

This means:

Let’s compare that to plain old Test::Unit (you know, that testing library you get for free with Ruby without having to install gems off github and keep track of the latest “features”):

1
2
3
4
5
6
7

def test_GET_to_index_finds_all_users_ordered
  2.times { Factory(:user) }
  get :index

  assert_response :success
  assert_equal User.ordered, assigns(:users)
end

With Test::Unit, we get:

Why “one assertion per test” is a false idol

There’s something very Zen about “one assertion per test.” Focusing on just one thing at a time is noble and easy to wrap your mind around.

However, this is often taken to the extreme. Let’s look at a typical test for users/:id.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

context "a GET to /users/:id.xml" do
  setup do
    @user = Factory :user
    get :show, :id => @user.to_param, :format => 'xml'
  end
  
  should_respond_with :success
  should_render_template :show
  
  should 'find the correct User' do
    assert_equal @user, assigns(:user)
  end
  
  should_respond_with_xml_for 'user'
  
  should 'include the user id' do
    assert_select 'id', @user.id.to_s
  end
  
  should 'include the user name' do
    assert_select 'name', @user.name
  end
end

Not bad. That’s fairly easy to read. The problems:

Let’s try it with basic Ruby (Test::Unit) and Rails:

1
2
3
4
5
6
7
8
9
10
11
12

def test_user_show_xml
  @user = Factory :user
  get :show, :id => @user.to_param, :format => "xml"

  assert_response :success
  assert_template "show"

  assert_equal @user, assigns(:user)

  assert_select "user/id", @user.id.to_s
  assert_select "user/name", @user.name
end

How “shoulda macros” should work

I love shoulda macros. I think most of them read well, especially some of the newer ones announced in shoulda 2.0 They also keep your test suite DRY.

They are not without problems, however:

Whatever happened to simple, protected Ruby instance methods?

This was my process last night at the Boston.rb hackfest:

1
2
3
4
5
6
7
8
9

def test_separate_invalids_creates_invalid
  nurse = Nurse.new
  nurse.separate_invalids!

  model             = "Job"
  invalid_file      = invalid_file_for model
  invalid_file_path = invalid_file_path_for model
  assert File.exists?(invalid_file_path), "#{invalid_file} does not exist"
end

That worked fine. However, those last four lines are noisy. Plus, I need to test two other models and don’t want to repeat the same lines.

Time to wrap up those last four lines in a method with an Intention-Revealing Name. Since this is very vanilla Ruby, I’ll put it in a protected method in this file. Later on, if I need to test the same behavior in another file, I’ll move the helper to test_helper.rb.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

def test_separate_invalids_creates_invalid
  nurse = Nurse.new
  nurse.separate_invalids!

  assert_create_invalid_file_for "Job"
  assert_create_invalid_file_for "JobReport"
  assert_create_invalid_file_for "JobView"
end

protected

def assert_create_invalid_file_for(model)
  invalid_file      = invalid_file_for model
  invalid_file_path = invalid_file_path_for model
  assert File.exists?(invalid_file_path), "#{invalid_file} does not exist"
end

This is bread-and-butter, meat-and-potatoes, lunch-pail programming.

Standards expand the reach of Ruby

Ruby on Rails is still cutting-edge. Most of the people doing Rails are working at small companies and startups. This is fun and exciting.

Let’s make more programmers happy by helping Ruby extend its reach. I’d personally love to see new industries, like the budding clean teach, smart grid industry pick Ruby as a go-to programming language.

However, for Ruby to continue to grow, risk-averse programmers and managers need some reassurances:

“How will my current Java or C# skills translate?”

Well, you’ll need to learn Ruby first. Then, just spend an hour or two looking at the documentation for Test::Unit. If you’re familiar with JUnit or NUnit, it’s an easy transition. Same interface. Same programming skill-set.

On the other hand, if you have a few weeks, check out RSpec. I can’t guarantee a good ROI on that time you spend, however.

Standards make you focus on domain logic

Ruby makes it easy to create DSLs. It makes it easy to create new testing libraries. That’s a wonderful testament to the language.

It doesn’t mean you have all the answers. It doesn’t mean you know better how to write a testing framework. “Tests” and “assertions” are fine. They have the added benefit of being a standard, and familiar for programmers coming from other languages.

Jeremy McNally’s new context library is tiny, yet it is trying to be API-compatible with RSpec, shoulda, etc. This blows my mind. I’ve talked to Jeremy and I think he’s a great guy but I don’t understand this as a design goal at all. While it tries to avoid the eval macros, it’s making the same mistakes previously discussed regarding contexts. Hell, it’s called context, which we should consider a test code smell.

All this energy on creating new DSLs for testing is energy wasted. Use the standard and get cracking on solving truly innovative problems (note: the smart grid is innovative; social networks are not).

Learn from our collective experience in the Rails community with REST: forget about coming up with great names for your controller actions. The interface for testing is well-defined in the xUnit family. Use the standard and focus on your real problems. We’re all spinning our wheels with these new testing syntaxes.

Rails shows excellent judgment once again

Okay, I’ll concede that there’s one problem with Test::Unit: snake_case:



def test_GET_to_index_finds_all_users_ordered

That kind of blows. Rails core apparently agrees: check out ActiveSupport::TestCase.test

For non-Rails projects, this can be simply:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class Test::Unit::TestCase
  
  def self.test(name, &block)
    test_name = "test_#{name.gsub(/\s+/,'_')}".to_sym
    defined = instance_method(test_name) rescue false
    raise "#{test_name} is already defined in #{self}" if defined
    if block_given?
      define_method(test_name, &block)
    else
      define_method(test_name) do
        flunk "No implementation provided for #{name}"
      end
    end
  end
  
end

Now our tests look like this:

1
2
3
4
5
6
7

test "separate_invalids! creates invalid file for each model" do
  nurse = Nurse.new
  nurse.separate_invalids!
  assert_create_invalid_file_for "Job"
  assert_create_invalid_file_for "JobReport"
  assert_create_invalid_file_for "JobView"
end

If you like that style (I think I do), Rails now gives this to you “for free.” Use the framework, use the standard.

Acceptance Test Driven Planning is still a baby

RSpec’s Cucumber has the alpha geeks excited. I’ve never used it so I won’t say too much about it except that I’m skeptical. I think any of us would be happy to add anything to our software toolboxes that help us deliver high quality apps faster. I don’t see Acceptance Test Driven Planning adding much value, however, at the expense of more overhead.

I’d love to hear “success stories” in the comments, though.

Comments on this post

Sean

Nov 07

Sean said,

This makes a lot of sense. I like it, but I’m torn.

I love the ease of shoulda, but the overhead of creating all of those records for every context is killing me. Short of disconnecting the database and using unitrecord, I had to speed up our test run. I was able to drop it from 260s to 100s by switching to Ruby Enterprise Edition.

So, we’re good with everything for now, but we’ve really just postponed dealing with the core issue.

Though, really, the problem is using shoulda in combination with a data factory (we use factory_girl, but that’s neither here nor there). In trying to obliterate fixtures from our app, I converted a couple of test files over to factory_girl (using .build where I could), and the test time more than doubled in some cases. Fixtures are brittle (in a future-tense way), but when you’re setting up a record in a setup method, they blow everything else away for speed. (We still went with the factories, but we have a couple of fixtures sitting around, waiting for conversion.)

Dan Pickett

Nov 07

Dan Pickett said,

People want to get crazy with mocking and stubbing everything in controller contexts. That just makes a mess imo. But, eliminating the persistence layer from functional tests still remains a problem.

Sometimes, I’m really uncomfortable about macros. They’re very convenient and easy on the eyes, but often times they are pretty inefficient. I’ve been trying to mitigate some of it with nested contexts but that makes the tests harder on the eyes.

I think it’s a hard balance between these things. You want tests to be easy to write, but you also want them to run quickly. You want them to be expressive, but honestly do you every show clients test output? You want to make tests so they aren’t brittle, but you also want them to be isolated and expressive of your intent. All these things I believe are a problem with TDD and not Ruby itself, but I believe that Ruby developers are doing the most innovative things in attempting to solve these issues and achieve true TATFT qualm.

Great post.

Jeremy McAnally

Nov 07

Jeremy McAnally said,

Let me voice my strong disagreement.

First, that’s a very poor example of why context blocks are bad. That’s just a crappy way to write those tests, not the library’s fault. Deciding to use macros like that are your own fault. You could just as easily write the same code in Test::Unit as context (or probably Shoulda) and avoid the problem you point out. And context has before(:all), so you can fix the larger problems here very easily.

Second, your criticism of how Shoulda macros should work is weird. I don’t know about Shoulda, but in context you can just write those test methods and call them from your context generated tests just like you point out. Problem solved.

Thirdly, hey! That’s cool that you like Rails’ way of doing things because that’s where I got the idea and it’s what my implementation is based on! :)

This really seems like a criticism of how Shoulda and perhaps RSpec are implemented. I invite you to try context; I think you’ll find that most of the issues are moot due to the way it’s implemented.

Josh Nichols

Nov 07

Josh Nichols said,

@Jeremy,

To your first point about it being a crappy way of writing tests…

I’ve definitely written similar tests in my time. There’s definitely a code smell in there. How might you suggest writing it instead?

Dan Croak

Nov 07

Dan Croak said,

@Jeremy,

Thanks for the comment. I was hoping to hear from you and folks in the RSpec world.

I second Josh’s request to see a good version of the “why context blocks are bad” test.

After reading your comment, I guess my main point from the post still remains. If I can do the same thing in context as Test::Unit (and Rails), what compelling reason is there to use context or any other tool that comes along?

The only thing potentially missing from Test::Unit is the ActiveSupport::TestCase.test as I wrote it above (and how it is written in context), with no aliases, and nothing further. It belongs IN Test::Unit, however.

We need to stop making new libraries and new tools. How about we focus on making apps? I’m guilty of this, too.

Also, RSpec and Shoulda both have similar ways of doing before(:all). I look at that as another layer of indirection that is necessary when you start introducing contexts, especially nested contexts. It’s solving a problem that shouldn’t be there in the first place.

James Golick

Nov 07

James Golick said,

I’m of two minds on this issue. Shoulda’s implementation really sucks; that’s a big part of the problem. Most Shoulda macros are completely unreadable, and full of code smells – bad for tests. Shoulda’s implementation of contexts is also somewhat weak.

I got really excited about context when I saw it, because I thought that it was the better-implemented solution I had been looking for. I agree with most of Jeremy’s arguments above.

But, there’s still a big problem with context in my mind, and that’s the dissonance between the names of the tests as they appear in the contexts and should/it blocks, and the way that they actually get defined as test methods.

When a test fails in context, I see test_person_who_is_admin_should_have_many_posts. In order to actually locate that code in the person_test file, I have to search around for a set of contexts (potentially deeply nested) and should blocks that might generate such a test name. Failing tests should be really easy to find immediately, even when you’re new to the code, IMO at least.

Also, and this is obviously subjective, I’m not crazy about looking at deeply indented code. One of the alleged benefits of contexts is that they can be nested. I find that ends up creating test code that’s hard to follow.

Finally, Jay Fields’ expectations is still the most pleasant testing library I’ve ever used. It says: forget about writing tons of strings in your tests – test names are comments, and comments are code smells. Instead, write your tests expressively, and use the assertion as the test name (i.e. expect(5) { 3 + 2 }). It forces you in to one assertion per test without making it hard to work with. I’d really encourage people to give it a try. I can’t say enough good things about it.

Since there’s no way to use expectations in rails, though, there’s a good chance I’ll start my next project on test/unit.

Alex Young

Nov 07

Alex Young said,

One assertion per test doesn’t make sense to me either. Tests have to be atomic in some sense, perhaps conceptually (i.e. this test represents a particular edge case found by a customer). This probably got out of hand at some point with the quest for finding problems where they don’t exist.

I don’t really agree with “all of these testing tools are solutions to non-existant problems” —Sometimes it’s good to switch testing approaches to gain a different perspective. I default to Test::Unit, but sometimes do supporting tests in RSpec to force myself to think differently about my code’s design and quality.

Jeremy

Nov 07

Jeremy said,

You wrote the better way to write that test yourself. :) That or use a before(:all) so that the setup stuff is only run once.

And as to “why,” I think we’ve argued BDD vs. TDD vs. WTFDD to death. If you dig Test::Unit, go for it. If you like Shoulda or context or whatever-lib, then use it. If you like RSpec, go for it. It’s about focusing on the right things.

I forgot to address your comments on being APi compatible with RSpec and Shoulda. It’s so transitioning to context doesn’t require you to rewrite all of your tests. You can move Test::Unit, Shoulda, or RSpec (using matchy) with very minimal changing of your test code.

Giles Bowkett

Nov 07

Giles Bowkett said,

Jeremy McNally’s new context library is tiny, yet it is trying to be API-compatible with RSpec, shoulda, etc. This blows my mind. I’ve talked to Jeremy and I think he’s a great guy but I don’t understand this as a design goal at all.

I think the design goal is obvious: universality. Highly compatibility means high availability.

Daniel Lyons

Nov 07

Daniel Lyons said,

High compatibility with what? Many of these libraries are themselves changing from minor release to minor release.

Jeremy

Nov 07

Jeremy said,

@Daniel: I don’t plan on 100% compatibility with anything. I hit about 90% compatibility with RSpec et. al. which is all most people use. It’s not like I’m trying to replace RSpec or something, but if I can add compatibility with a few aliases, then why not?

And more to the point, the fact that these libs change so much is one of the reasons I wrote context in the first place. Bugs in one version are fixed in the next, but surprise! The API’s changed so now you’ve got even more failures. </soap_box> :)

Justin

Nov 08

Justin said,

Personal mantras I do not recommend, just my personal observations/test cases.

1. “Testing” is not “Specing” 2. “Specify” a truthful statement over “Should” expect a return value. 3. I’d rather macros over meta. 4. I’d rather DSLs with organization over no DSLs. 5. I’d rather huge long specs with details, then tests that try to be DRY

I’m sorry but this argument/article is completely moot (2 years old?), with terrible examples that back up worthless commentary. Find what works at your company, with your teams, or build another Rspec clone. ;)

Secondly, this sounded like a reply to ENTP’s moaning. Nobody cares if ENTP moves off Rspec. Nobody cares that their Rake files were busted. The only thing I learned from Courtney’s article was the comments from Rspec people having to ask THEM for bug reports. Disturbingly, this was the third time this week I read that on a so called “Ruby community leader’s” blog.

Paul Barry

Nov 08

Paul Barry said,

Great article. I’m questioning the validity of one assertion per test now. But one thing I do like about contexts is nested context. That allows me to define some things that need to be setup for a set of tests, and then more things for the tests in the inner contexts. Maybe the same kind of thing can be achieved with protected helper methods.

From RSpec specifically, I would also miss pending specs, does test/unit have that?

Brian Hogan

Nov 08

Brian Hogan said,

I’m so happy to see something like this. I’ve been happily using test::unit for three years and recently switched to RSpec. I love the reporting it provides me, but I don’t like much of the rest of it.

@Dan said in the comments: “We need to stop making new libraries and new tools. How about we focus on making apps? I’m guilty of this, too.”

I agree wholeheartedly. In fact, in my talk at a local code camp in 10 minutes, I’m going to use test::unit because it’s right there, waiting to be used.

Morgan Roderick

Nov 09

Morgan Roderick said,

A refreshing read, and so many interesting comments!!!

I don’t really see the splintering… what I do see is that there are a lot of dedicated people trying to solve the same overlapping issues. Some of these solutions get’s to be used in ruby-based frameworks, and eventually might also be included into ruby itself.

Sure, there are problems with Shoulda, rSpece, etc… but they do help us write better tests, and like this post shows, also provide excellent basis for discussion and improving.

Personally, I enjoy working with Shoulda, and do write my own macros … it helps keep my tests easy to understand and easy to update…. which I believe was the purpose of Shoulda to begin with: “Making Tests Easy on the Fingers and Eyes”.

Shoulda, rSpec, Test::Unit are not the holy grail of (ruby) testing, they’re stepping stones … and we, the community, will have to test each of them out in all ways possiblle to find they best way forwards.

...

Jay Fields is not always right, but he certainly manages to get people talking. Personally, I find the whole “one assertion per test” to be somewhat stupid … experience has repeatedly shown, that you cannot distill our collective wisdom into one-sentence-one-size-fits-all statements.

Dan: Thanks for sharing your thoughts (and frustrations) :-)

Felipe Giotto

Nov 10

Felipe Giotto said,

Wow! You said everything I wanted to say about one assertion per test! I understand the benefits of one assertion per test, but my tests became too slow. Recently, I changed most of them to the good, old style of one GET and many assertions!

Ken Collins

Nov 10

Ken Collins said,

I’ve been deep in AR tests a lot lately and they do have “context” blocks (after a fashion) when you consider that they stuff multiple test classes into a single file. It breaks the use of autotest to be able to automatically map a class to a filename since there will be 3 to 4 (maybe more) in one file.

I think finding a context to put your test in is dirt easy when you use your tools. For instance, TextMate has keyboard commands for “Toggle Foldings At Level” that can go as high/deep to 10 contexts. One keyboard command to see all the contexts from the top level… then drill down. Takes only a few seconds. IMHO, a much much better way to organize code. If tests are flat and let’s say you have 3-4 tests for a single #instance_method, then those should be in a context, else entropy and un-manageability will set in. BTW… thanks for Shoulda :)

nicholas a. evans

Nov 10

nicholas a. evans said,

With respect to: “How will my current Java or C# skills translate?”

I learned rspec back when it was far more magical than it is now (0.5-0.6), and I knew very little ruby at the time. It didn’t take me weeks to learn. It took me minutes. Everything seemed far more obvious to me than it ever did for any of the xUnit toolkits I had used, and they had a nice chart to help people make the transition easy: http://rspec.info/documentation/test_unit.html

Methinks the “rspec is too hard to learn” debate is rather silly… because the people making the argument often say “not for me… but you know… for others it would be.” http://weblog.raganwald.com/2008/02/lets-make-this-personal-please.html

The only thing I missed from C#/nUnit was the VS.Net/Resharper code completion and IDE support. But I got over that quickly enough.

Jay Levitt

Nov 10

Jay Levitt said,

“If I can do the same thing in context as Test::Unit (and Rails), what compelling reason is there to use context or any other tool that comes along?”

+1! This is why I still code everything in machine language. It’s all Turing-complete anyway.

Brian

Nov 10

Brian said,

Cucumber has definitely come in handy in our team’s project, although I don’t think we use it in the same way as others. We don’t use it as acceptance testing as much as very high level integration testing. Our app has a lot of various (extremely critical) edge cases that need to function in a specific way. We’ve had controller specs and model specs pass, but catch problems with Cucumber, such as certain input causing problems, or not being presented with a certain form at a specific point.

What our team did a while back was nuke the view specs folder and replace them with cucumber. It makes far more sense. There doesn’t seem to me to be much value in speccing a million expectations about the structure of the page. They’re very brittle and don’t help. It makes more sense to say that this specific sort of user should be able to do this specific activity and not worry about which controller, models, and views are being used and what the markup looks like. This, to me, is where Cucumber shines.

Jaime Bellmyer

Nov 10

Jaime Bellmyer said,

I certainly don’t pretend to be in the same class as a lot of gurus that have added their two cents to this post. That being said, I have yet to hear the magic word, “moderation”.

Contexts don’t make you nest 10 layers deep. You do. I’d rather have the ability to nest a little, because it’s one of the things I love about Shoulda and RSpec. Same goes for one assertion per test. Alex nailed it – tests should be atomic. Test as narrow a case as possible, but use all the assertions required to ensure that test passes.

As contractors, we have a myopic view. Of course we’re stuck learning multiple frameworks. We have to be able to jump into someone else’s code, and that means maintaining a wider skill set than those who work with the same applications long term.

So if that’s part of the argument for Test::Unit, then factor it out. Because it’s not a barrier to entry to any person, team or company looking to adopt Rails. I’d say ALWAYS start with Test::Unit, because it’s the foundation before you decide you need something fancier.

Perhaps the greatest time-waster for me personally has been attempting to keep up with what we “should” be doing each week :) Maybe the great lesson in this post transcends testing: find what works, be aware of what’s new, and only adopt what improves your process.

Wow, I’m one preachy bastard.

Dan Hensgen

Nov 11

Dan Hensgen said,

I like how expressive one-assertion-per-test is, but bummed with the performance.

I’ve been making use of the optional description parameter to all assert methods. It’s not quite as pretty in the source code, but it lets me describe what I’m asserting and still performs well. Plus, it’s built right in to Test::Unit.

Yossef

Nov 11

Yossef said,

Jaime really hit the nail on the head with moderation.

I love contexts, and I love the more natural feel and flow of “BDD tools” like RSpec. I like typing in natural sentences instead of using underscores. And I want to break something every time I have to type “assert_equal”. I switched to Shoulda from the standard Ruby Test::Unit because of some of these things, and though I was still stuck with assertions, nested contexts were a joy.

It was during the Charlotte RubyConf that we (OGC) talked to David Chelimsky about nested contexts in RSpec. He seemed unsure because some people might go too far and some people might be confused. We told him that it wasn’t worth keeping a useful feature out because of concerns that some won’t use it well. I can’t say it’s all because of us, but it wasn’t long after that RSpec had nested contexts. And once they were available, I switched and stayed there.

Recently, I’ve been looking at other choices. I’m starting to play with bacon now. RSpec has grown and become magical to the point I’m not sure about it, and I want something small and simple to understand. But I want it to be a spec DSL. I want my executable specification.

Why should we stop building libraries and tools and build apps instead? That’s a false dichotomy. Apps are built on libraries and using tools.

I have a lot more to say because I care a lot about testing, but I’ll stop for now. Moderation, after all.

Paul

Nov 12

Paul said,

I personally have had a very good ROI on my time learning RSpec.

Scott Becker

Nov 13

Scott Becker said,

@Dan – “We need to stop making new libraries and new tools.”

Yeah! Lets get back to Java, or ASP, or COBOL. It worked well enough! ....umm, what?!

I get what you mean, you don’t want to have to keep learning new tools. But then why even go through the effort of learning Ruby or Rails, or etc etc. At some point you had to stop going down the previously comfortable road and learn those.

For me, BDD has obvious benefits. Clarity of intent. Readability. This speeds up maintenance down the road.

As to one-assertion-per-test (or spec), thats just a design goal, not a requirement. But I’ve seen what packing multiple assertions into one test looks like, and it ain’t pretty. You get assertions that have nothing to do with the name of the test, and later on you come back and wonder why, but too bad, because you can’t go back and ask either yourself or your co-worker or whomever it was what they were thinking 6 months ago. Make your test do one granular thing, and you can easily understand why. Especially if you write the test first. This is do-able in Test::Unit, but I think the nested contexts make this a lot easier and more DRY.

Good post though, it’s good to occasionally step back and consider why.

Adam Williams

Nov 14

Adam Williams said,

I agree that the one assertion per test is worth considering, but not to be idolized. I agree that we should use all things in moderation. I agree that RSpec internals used to be insane, but the conceptual weight of using the thing has never been a challenge. I’ve decided I like RSpec just fine. Great post. Got me thinking again. As for cost of loading data, check out http://github.com/aiwilliams/dataset. I’d love some feedback. Be gentle.

Sorry, comments are closed for this article.

© 2000 - 2009 by thoughtbot, inc.
written by a bushel of tiny robots