There are two main types of unit tests: state based and interaction based. State based tests rely on the verification of state. These tests typically perform some operation and then check the state of the resulting object:
user = User.new :title => 'manager' user.promote! user.title.should == 'senior manager'
In contrast, interaction based tests rely on verification of the interaction between objects. This is generally done with mocks or stubs:
user = User.new Mailer.should_receive(:send_email).with(user) user.activate!
This test ensures that the activate! method interacts with the Mailer, without actually sending an email.
Interaction based testing has its uses, but in general, I prefer state based testing. I think that tests full of mocks are brittle and hard to read. Furthermore, when they fail, it can be difficult to understand why.
In my testing, I try to replace interaction based testing with state based when possible. I’m going to walk through an example where a typical test would use mocks, but I’m going to use a state based approad instead.
Say I have a Rails 3 application that accepts social security numbers. These are sensitive, so I want to make sure I don’t log them to the application log. A simple testing approach (using rspec) might look like:
require 'spec_helper' describe MyController do describe "create" do it "filters sensitive data from the log" do Rails.logger.should_receive(:info).with { |message| message.include?("123-45-6789") }.never post :create, :myobj => {:social_security_number => "123-45-6789"} end end end
One problem with this test is that it’s a negative test. It can pass for the wrong reasons. For example, if the controller action blows up before it would normally log, the test may pass even though it would log under normal circumstances. It’s generally better to write a possitive test:
require 'spec_helper' describe MyController do describe "create" do it "filters sensitive data from the log" do Rails.logger.should_receive(:info).with(include('"social_security_number"=>"[FILTERED]"')) post :create, :myobj => {:social_security_number => "123-45-6789"} end end end
This test is better, but still not great. One problem is that Rails.logger.info is now mocked out, so there will be no logging which can make debugging a failing test more difficult. Another problem is that if the test fails, the test output only says expectation not met. It does not show you what was actually logged:
1) MyController create filters sensitive data from the log Failure/Error: Rails.logger.should_receive(:info).with(include('"social_security_number"=>"[FILTERED]"')) (#<ActiveSupport::TaggedLogging:0x007fcf39201a10>).info(include "\"social_security_number\"=>\"[FILTERED]\"") expected: 1 time received: 0 times # ./spec/controllers/my_controller_spec.rb:7:in `block (3 levels) in <top (required)>'
Here is the state based test I prefer:
require 'spec_helper' describe MyController do describe "create" do it "filters sensitive data from the log" do post :create, :myobj => {:social_security_number => "123-45-6789"} log_line = Rails.logger.log_history.grep(/Parameters:/).first log_line.should include('"social_security_number"=>"[FILTERED]"') end end end
This test actually grabs the log line from a log history. If we don’t find the log line, it will be obvious. If we do find the line, we can compare the expected output with the actual one.
Now, the test failure contains both the expected and actual:
1) MyController create filters sensitive data from the log Failure/Error: log_line.should include('"social_security_number"=>"[FILTERED]"') expected " Parameters: {\"myobj\"=>{\"social_security_number\"=>\"123-45-6789\"}}" to include "\"social_security_number\"=>\"[FILTERED]\"" # ./spec/controllers/my_controller_spec.rb:9:in `block (3 levels) in <top (required)>'
We can monkey patch Rails.logger to record the log lines in spec_helper.rb:
module LogHistory def add(_, _, message = nil, &block) log_history << message super end def log_history @log_history ||= [] end def clear_log_history log_history.clear end end RSpec.configure do |config| config.before(:suite) do Rails.logger.extend(LogHistory) end config.before(:each) do Rails.logger.clear_log_history end end
Now, every log line is recorded per test and cleared out in between. This pattern also works well for other external dependencies, such as database statements or queries sent to a search tool. They can be recorded and asserted against directly.
I like the blog entry and the approach. A few thoughts:
- Not being able to see the results of an interaction sound like a limitation of the interaction testing tool, not interaction testing in general. e.g. https://github.com/jaycfields/expectations will show all interactions that did occur, so you can easily see what’s causing the failure.
- I prefer general solutions over specific solutions, and it seems like the pattern you’ve introduced could be made general instead of monkey patching the specific classes you care about. For example, in your test you could designate an object as a mock (or maybe a different name to designate the different approach, perhaps ‘recorder’) and then at the end of the test you could verify all of the recorded interactions you care about. (btw, this is already somewhat similar to how mockito works)
I do think better tools would make interaction based testing easier. However, I still prefer the approach of state based testing. I think I’ve just been burned by too many tests that asserted interactions without actually checking that the end state was correct. The tests passed, but the code was broken. Or, I refactored where some logic lives and then I had to go update a bunch of interaction based tests even though the behavior was the same (and a state based test would have passed).
I think a recorder is an interesting idea. One issue with mocks, however, is that they prevent the original code from running. This can be good when you are testing an external endpoint and don’t want to actually make service calls. But it can also be bad when you want to test something like logging or database statements without changing the way the application runs. I like the pattern above because it captures the necessary information without changing the behavior of the system.
Bad tests are bad tests, I can write fragile state based and fragile interaction based tests. =)
I tend to use the type of test that best captures what’s important in what I’m doing. For example, if a new 3rd party value causes a value in my system to be recalculated, then I’m going to test the recalculated value via a state based test. However, if the new 3rd party value causes a new email to be sent, I’m going to test the function that publishes the email (likely, the ‘publish’ function provided by the email framework). I look for the important result, and try to test that.
I think that’s half of the issue: focus on writing tests that test what’s important.
I think the other half of the issue is tooling. Don’t get me wrong, I think you have a valid complaint. Your example clearly points out a case where the failure isn’t giving you the information you would want when diagnosing the problem. However, it doesn’t have to be that way.
Look at the (contrived) interaction test here: https://github.com/jaycfields/expectations/blob/master/test/clojure/success/scenario_success_examples.clj#L73-75
Broken down line by line:
line 1 – declare a new test scenario
line 2 – call a function exactly as it would be called in production
line 3 – expect an interaction by calling the exact function that you’re verifying, and passing in exactly the arguments that you would expect.
I don’t think it gets any more straightforward than that – you’re writing exactly the same code in the test as what you wrote for prod. Another example: https://github.com/jaycfields/expectations/blob/master/test/clojure/success/scenario_success_examples.clj#L73-75 (foo2 & bar2 are defined on lines 8 & 9)
In those tests, if you were to somehow call foo2 with 1 6 and 1 8, expectations would tell you that foo2 was called with 1 6 once and 1 8 once, but never with 1 4 – I believe that’s the kind of failure you’d want to see.
In the above examples, I’m doing basically the same thing as what you’re doing – except I’m not calling ‘super’. If I’m calling my own code, then I think that could be a problem. If I’m calling a framework, then I’m not really concerned with calling the actual framework code. I’m sure we’ve all been burnt by a framework, but I don’t like the idea of getting in the habit of testing framework code. It *should* just work. And, as I implied above, I don’t generally mock my own functions.
Either way though, I don’t think that’s an issue. I don’t see any reason why you couldn’t have something general that adds your behavior but still calls the original method. Then your test could look something like: https://gist.github.com/3072091
It’s been over 4 years since I’ve done Ruby, but it’s not that hard to redefine a method to call a block and then call what it was originally supposed to call, is it?
Either way, I feel your pain, and it’s fun stuff to talk about. Thanks for the post.
sorry, the 2nd link should have been: https://github.com/jaycfields/expectations/blob/master/test/clojure/success/scenario_success_examples.clj#L77-79
The expectations syntax looks good. I’ll have to check out the test framework.
Your ruby code looks interesting as well. I’ll have to play with it.