Non-test code is trivial

Posted by Mike Burns

Oct 17

Testing! Do not disturb!

I have heard—and been in the middle of—arguments which make the claim that the tests is the stupid, simple part and the "real code" is the hard part. One such person made the claim that interns should write the test code and "real programmers" should write code to make the tests pass.

This is backwards.

The tests are the program. Without the tests, the program does not work. Tests are not something that should be left for the inexperienced; tests are the hard part.

To quote Code Complete:

Test cases are often as likely or more likely to contain errors than the code being testing (Weiland 1983, Jones 1986a, Johnson 1994 [Johnson, Mark. 1994. "Dr. Boris Beizer on Software Testing: An Interview"]). The reasons are easy to find—especially when the developer writes the test case. Test cases tend to be created on the fly rather than through a careful design and construction process. They are often viewed as one-time tests and are developed with the care commensurate with something to be thrown away.

For example, the following test does not have nearly enough setup to be useful (assigns(:posts) could be empty, causing this test to be something of a no-op). This is a common problem; sometimes it happens because it relies on fixtures that get lost in a refactoring, and sometimes it happens out of just not thinking hard enough. The non-test code behind this, of course, is part of the Rails scaffold and dead simple.

1
2
3
4
5
6

context "a GET to index" do
  setup { get :index }
  should "have a link for each post" do
    assigns(:posts).each { |post| assert_select 'a[href=?]', post_path(post) }
  end
end

The next set of tests could use a loop over %w(index show edit) and should_redirect_to to shorten it without any loss of knowledge (and could even make it more clear). This is a case of an accurate series of tests (for a one-line piece of code) that has become difficult to maintain because the developer failed to notice the pattern.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

context "logged out" do
  setup { session[:user_id] = nil }
  context "GET to index" do
    setup { get :index }
    should "redirect to the root_url" do
      assert_redirected_to root_url
    end
  end

  context "GET to show" do
    setup { get :show }
    should "redirect to the root_url" do
      assert_redirected_to root_url
    end
  end

  context "GET to edit" do
    setup { get :edit }
    should "redirect to the root_url" do
      assert_redirected_to root_url
    end
  end
end

It's interesting to note that the problem in the first example could be caused by refactoring, and the problem in the second example is caused by not refactoring. This is a fair point, and another indication of the difficulty in writing tests: when refactoring non-test code, anything you break is pointed out quickly by the tests; when refactoring test code, you need to make sure that it will continue to fail when needed, and continue to pass when needed.

Some tests are just so arcane or trivial that they would normally be overlooked, but if it really needs to be a specific way then it needs to be tested.

For example, ActiveRecord::Base#to_xml is a lovely part of Rails that is often used for part of the application's public API. If #to_xml changes, many customers will be upset. Recently we had to override #to_xml with lots of special-casing; while the special-casing did get tested, some of the more subtle parts of #to_xml were ignored. Thus, this test was recently added in response to a bug report:

1
2
3
4

should "dasherize any element with underscores when sent #to_xml" do
  xml = Factory(:message).to_xml
  assert_no_match /<\w*_\w*>/, xml
end

Some code needs a complex maze of tests. Advanced search is a common feature that requires careful thought to test all the options. The Album.advanced_search method may be large and complicated, but the tests for it must be necessarily more so. No time for laziness or ignorance here; this test suite requires the knowledge of an advanced search expert, and the patience of a person who has a lot of patience.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

context "when sent #advanced_search" do
  setup { create_all_sorts_of_data }
  %w(producer artist mixing_engineer).each do |str_opt|
    context "with the #{str_opt} option" do
      # ...
    end
  end

  [true,false].each do |include_compilation|
    context "with :include_compilations option set to #{include_compilation}" do
      # ...
    end
  end

  # ... and then permutations of these, too
end

(While researching this post I found these two interesting links: a WardsWiki page about bugs in tests, in which people argue over buggy tests, refactoring tests, and how this affects non-test code; and a paper titled Refactoring Test Code [PDF], which discusses "code smells" relevant to testing and how best to refactor these.)

Known fact: chicks are totally into dudes who write awesome tests.

Comments on this post

rainep

Oct 17

rainep said,

Which testing framework is this?

Chris O'Sullivan

Oct 17

Chris O'Sullivan said,

I love this post because it confirms what I’ve always felt – writing test code is hard! I spend 80% of my time writing/maintaining tests, but get frustrated by the lack of decent dialogue in the community about what constitutes best testing practice.

That ‘refactoring test code’ paper is brilliant, thanks for the link!

@rainep It’s shoulda and it’s flippin’ awesome: http://github.com/thoughtbot/shoulda/tree/master

Pratik

Oct 17

Pratik said,

I completely disagree with the suggestion “The next set of tests could use a loop over %w(index show edit)”. Defining a common assertion is super awesome. E.g you could have a new assertion called should_redirect_to_root. But trying to reduce LOC by DRYing contexts, is a total PITA in the long run. I’d write that test as something like http://gist.github.com/17554

I know it may seem trivial in this specific example to have 3 different context, but I’ve found it to be a good practice never to DRY your contexts. When you can’t nest them, duplicate them.

Luke Francl

Oct 17

Luke Francl said,

One of the points I made in my “Testing is Overrated” talk was that writing good tests is really, really hard. http://railspikes.com/2008/7/11/testing-is-overrated

Most developers are given too much comfort by their 100% RCov scores. Those scores mean next to nothing – the only value you get from RCov (in my opinion) is seeing which lines of code aren’t covered AT ALL.

I used an example from Code Complete where Steve McConnell writes tests about 30 lines of code. It takes him 9 pages and he ends up enumerating 17 tests for just these few lines of code. See my handout for the example: http://railspikes.com/assets/2008/7/26/testing-is-overrated_handout.pdf

I wouldn’t go so far as to say that non-test code is the trivial part, but testing is HARD.

Anonymous

Oct 18

Anonymous said,

You know I hear a lot about testing with in the Ruby/Rails/etc community but I’ve been curious as to testing in other languages and in the real world. For example, are the developers for large software programs writing tests? What about in, say, game development, are those developers writing tests?

I personally still don’t write tests and don’t know if I ever will get around to it. Historically I’ve had better luck writing code and encouraging friends to break my app by abusing all available features.

I just pretend like I’m following Bryan Lile’s advice. I looked at my code and wrote tests. Then I deleted those tests, but never got around to rewriting them.

bryanl

Oct 18

bryanl said,

I agree with Pratik. As a matter of fact, I have code in a project that looks just like the gist Pratik posted. My goal with tests is to make the behavior that I’m trying to create very explicit in my testing code. should_redirect_to_root just rings nicer than should “redirect to the root_url” { assert_redirected_to root_url }

Igor Pro

Oct 18

Igor Pro said,

First test, next code. Thanks for another one good post!

ed

Oct 18

ed said,

i think we should code first :)

Rechnung

Oct 19

Rechnung said,

Thanks for good post! I think that testing is important, but from business point of view application has to move up some level to write tests.

Sean

Oct 20

Sean said,

Testing is merely quality control. Without developers to drive the real application development, what “quality” would you have to even think about controlling? You should really be worrying about your application itself, first and foremost. Premature testing can very often lead you to the same kind of black hole that premature optimization can. Code first, test later.

bryanl

Oct 20

bryanl said,

@Sean: “Premature testing” as you call it shouldn’t exist if you are doing it “right”. Once you start looking at your tests as specifications rather than checking your work, you’ll find that by worrying about getting your tests right means getting your application right. TATFT should be your motto.

Daniel Cadenas

Oct 21

Daniel Cadenas said,

@Sean: Many people think like you do and that’s why calling them specs instead of tests is recommended.

Writing specifications is not just a technique to catch bugs, that’s just a fraction of what they do. They are an invaluable design technique and a documentation tool. They also work well as a backlog of pending things to do. The great thing is having those advantages in only one activity. It’s DRYness in the development process itself, not just in the code.

But you are right when you say that developers should drive the real application, that you should really be worrying about your application itself, first and foremost. Doing that through specs is an excellent idea.

Jon Yurek

Oct 21

Jon Yurek said,

Historically I’ve had better luck writing code and encouraging friends to break my app by abusing all available features.

Of course you did, because writing good tests is hard. That’s the point. But if you have good tests, it’s just like having all your friends abusing your app all the time, whenever you want because your tests are automated and repeatable without being at the whims of your friends’ schedules.

Like Kent Beck said when he first tried TDD… he was done with his app code so quickly and correctly it felt like he was cheating.

Mike Burns

Oct 21

Mike Burns said,

Put another way: the hard part of programming is the design, the algorithm, and figuring out what needs to happen. That’s the test. The non-test code, if done properly, just takes all that knowledge and makes it happen—-no thinking needed.

I hadn’t thought of testing as quality control. I guess it does that, but it’s not so great at it; you’ll get better QA results by encouraging friends to break your app by abusing all available features. (But when you fix the bug, you need to think hard about the algorithm—-that’s testing.)

Sorry, comments are closed for this article.

© 2000 - 2009 by thoughtbot, inc.
written by a bushel of tiny robots