Is it a unit test or an integration test?

We didn’t want to spend time agonizing over whether a test should be considered a unit test or an integration test.  We wanted an easy way to decide, such that the resulting division of tests would meet our needs. Here are some guidelines I wrote up to that end a couple of years ago, as a companion article to the Why not just integration tests article.


So you’re writing a code-based test, and first off you need to decide whether it should be considered a unit test or an integration test so that you know how to name the test class. How do you decide?

1. We should consider it a unit test if…

There are two things we want to be true about a thing we call a unit test1:

  • It runs fast
  • It is geared toward error localization

1.1. It Runs Fast

To consider a test a unit test, it needs to not take very long to run.

Why do we say that?

Speed rationale

The reason we don’t want unit tests to take a long time is because we want to be able to run them often, during development. We want to be able to code a little, run our tests, code a little more, run our tests…. If our tests are too slow, it will break the rhythm and in practice we will just end up not running our tests very often.

Speed guidelines

Ok, so what kind of speed are we looking for?

Unit Test Speed Guidelines
Ideally a unit test class will run in under… 0.1 second
We should aim for a unit test to run in… < 1 second
Many of our unit tests may run in… 1-3 seconds
Some of our unit tests may take as long as… ~5 seconds

If a test tends to take closer to 10 seconds to run, it needs to go in the integration test bucket.

What about a slow first run and quick later runs?

Some tests have to do some setup that may take, say, 15 seconds the first time the test is run in a namespace, but then subsequent runs take less than one second. That is ok, the slow first run is not likely to be a barrier to people running the test often since subsequent runs are fast.

My PC is slow, or I was running a sandbox update, and…

There may be “artificial” conditions that make your test run’s duration approach 10 seconds, when in a more normal situation the test run duration would be much shorter. Use your judgment in these situations, keeping in mind that the main thing is to keep our body of unit tests fast enough to run often.

1.2. It Is Geared Toward Error Localization

To consider a test a unit test, it must also be “geared toward error localization”.

Error Localization
When a failing test points you to the errant code that is responsible for the failure.
Why do we want error localization?

One industry guru puts it this way:

As tests get further from what they test, it is harder to determine what a test failure means. Often it takes considerable work to pinpoint the source of a test failure. You have to look at the test inputs, look at the failure, and determine where along the path from inputs to outputs the failure occurred. Yes, we have to do that for unit tests also, but often the work is trivial.2

So even if a test is fast, if the main assertion(s) of the test methods seem to be lost among all the setup and tear-down code, it probably belongs in the integration test bucket.

2. Otherwise It’s an Integration Test…

If your test is too slow to fall within the unit test speed guidelines, or if it is geared more toward broad coverage, then it belongs in the integration test bucket.

3. …Unless It’s One of These Other Kinds of Tests

Manual test helpers

If your test can’t do everything automatically, but instead performs setup to make it quicker to perform manual testing, it’s a manual test helper, and it belongs in the test.manual hierarchy.

System integrity tests

If your test tests delivered data rather than code (such as making sure that each delivered field has a table specified or that each delivered R-model relationship is owned by serial number 8500), it is a system integrity test, and it belongs in the test.sysinteg hierarchy.

4. Other Issues When Writing Tests

TODO: Answer other questions about tests, such as:

  • Is it ok to change system settings in a unit test? in an integration test? (yes, but the test should leave them to the way it found them)
  • How strict should we be about a test cleaning up records, globals, log messages, etc. it creates? (Clean up whatever you can – accounts with payments on them are a known issue…)
  • Sub-issue of the above: should we recommend that tests use transactions to undo complex things like account creation? What level of endorsement would we give: Is it preferred, acceptable where needed, or discouraged?
  • Is a test class responsible for making sure two instances of itself don’t run concurrently, if it’s not thread-safe? Or should we assume that only one process at a time will be running tests in a namespace?

Notes

  1. We’re explaining what we will consider to be a unit test for our purposes, versus what we will consider to be an integration test. We’re not trying to address the question of what really is a unit test versus what really is an integration test. For instance, Michael Feathers says that if it writes to the database it’s not a unit test, but for our purposes if it writes to the database and it’s still fast enough, it can go in the unit test bucket.
  2. Michael Feathers, in Working Effectively with Legacy Code, p. 12.

– DanielMeyer – 06 Dec 2007

About these ads

,

  1. #1 by Preston on May 26, 2009 - 4:11 am

    Well, frankly, I had never thought of the distinction between unit testing and integration testing on those lines, but a nice proposition indeed.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.