If you collect tests automatically using a
TestCollector (and even if you don't), it's also a good idea to use a
TestInventory to detect if tests have silently disappeared.
A
TestInventory is, well, a list, or inventory, of tests that have run in the past, and which can be expected to run in the future. Usually, a test framework implementing
TestInventory will record the names of tests run, e.g. into a file, say
CurrentTestInventory. This file will be diffed against the E
xpectedTestInventory.
If no changes, all seems well.
If tests have been added, the diff will report this, and the human running the tests has the option of making the
CurrentTestInventory into the
ExpectedTestInventory.
If tests have been accidentally deleted, the diff will report this, and the human running the tests can go look for the missing tests, and why they were not run.
I started using the
TestInventory pattern after bugs in my
TestCollector resulted in tests that were written not being run. Also after I accidentally deleted some tests while editing, not noticing it for a while because I was using a
TestCollector and not paying attention to the
TestSuiteConsistencyCheck.
See
HandlingBrokenTests
TestInventory is a
TestSuiteConsistencyCheck.
A form of
TestPositiveConfirmation.
TestInventory is related to
ProjectBillOfMaterials.
TestInventory to some degree goes against
OnceAndOnlyOnce. To add a new test, you have to code it, and then add it to the
TestInventory. This is two steps, but the latter is (or should be) automated.
It guards against accidental deletion of tests by requiring the same two steps, first deleting the test, and then confirming. You're unlikely to do both by accident.
In some ways, this is like
DoubleEntryBookkeeping.
CategoryTesting