fail-check-testcases: A new MTR option to enforce strict cleanup

Test cases that do not perform a proper cleanup turn out to be a nightmare for test engineers or anyone who is tasked with the job of monitoring regression test results. It leads to test failures that are difficult to analyse and fix.

A good test tool should provide a mechanism which helps to detect such faulty tests. Our primary test tool mysql-test-run(MTR) already has an option 'check-testcases'(MTR documentation) which is enabled by default to perform this check. But this merely prints an informative message about missing cleanup and the test run succeeds.

First let us look at current behaviour of MTR with 'check-testcases' option. Consider a simple test 'sample.test' which creates a table, performs SELECT operation but misses dropping the table.

Let’s run this test using MTR.

As you see from the above run, 'check-testcases' triggers only a warning about the missing cleanup and informs about the observed differences in the system states. Since MTR does not report it as a failure, it doesn’t get noticed in regression runs. As a consequence, new tests that do not perform proper cleanup are getting added to the repository.

To resolve this issue, we have introduced a new MTR option 'fail-check-testcases' to enforce strict cleanup at the end of the test run. When this option is enabled, the check-testcase failure gets reported as an error and the test run fails. This ensures that if a test does not perform a proper cleanup at the end, it is caught while running the regression test suite. Please note that this option is disabled by default.

Now, let us run the above test case with the option enabled.

As you can see above, the test run has failed by reporting an error.

This option would be enabled by default in the near future to ensure any missing cleanup is caught while enhancing the test suite.

Please try out the new MTR option 'fail-check-testcases'( MTR documentation) and share your feedback in improving our test tool further.

About Pavan Naik

Pavan Naik has been working with Oracle MySQL team since 2013. He is a Senior Member Technical Staff for the ServerQA team and responsible for maintaining and enhancing MTR test tool, qualifying optimizer and GIS features.

One thought on “fail-check-testcases: A new MTR option to enforce strict cleanup

Leave a Reply