Stop Junit, please

while trying to find a way to make my tests fail fast during integration testing, I found out that you have to say “please” to stop JUnit from testing…

I am currently working on a legacy project which has a very large Maven/Cucumber/Selenium based automated test suite. To complicate matters, the infrastructure these integration tests run on is often unreliable (e.g. network connections slow / not working, test data has been tampered with). So I was looking for a way to stop testing when those kind of “rough” failures.

After searching for some time I found a solution in this post. The solution involves writing a JUnit RunListener which calls “System.exit(-1)” when the first test failure occurs. In addition the Maven Surefire Plugin has to be configuring to add the listener on test execution. The code for the RunListener looks like this:

And the corresponding configuration in the pom:

What I liked about this approach is, that you can actually configure the fail fast behavior in the external pom. So no code changes needed and flexibility added. The listener could for example only be enabled for integration testing.

What I did not like about this solution was the call to System.exit() which immediately terminates the test run, without the test runner having a chance to output some kind of summary. That is especially bad in environments where test results are not logged verbosly.

After doing some research I found a way to stop JUnit from executing any more tests. The solution is the method pleaseStop() in the class RunNotifier.
This method signals JUnit to stop execution of any following tests.
It might be that some tests are still executed when running tests in parallel though. In order to access this method I had to extend the solution above in the following way:

  • Implement an own TestRunner that inherits from the one regularly used (e.g. BlockJUnit4ClassRunner) that has access to the RunNotifier
  • Pass the RunNotifier to the RunListener and call pleaseStop() on the first failure:

The full source for these two classes can be found here.

Note that you can of course change the inheritance hierarchy if you use another test runner.

Of course now, you can add more fine grained rules for failing the tests. For example on Selenium timeout, on Network errors, on wrong test data, and so on. You could even activate or deactivate the listener based on system properties. Making this solution as flexible as the first one.

Conclusion

Using RunListeners can be used to abort unit testing on a failure using System.exit. This approach has the advantage that the code does not need to be changed. The disadvantage being that System.exit prevents any further processing of the test environment as well.

Using an own TestRunner in addition to a RunListener as described above is more elegant but includes changes to the source code of the test classes. And you have to be polite ;-)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">