Flaky Mutants; Another Concern for MutationTesting
Software testing is the dominant method for quality assurance and control in software engineering. Test suites serve as quality gates to safeguard against programming faults. But not every test suite is written equally. We usually gauge its quality using metrics such as code coverage. These assess how much of the code base has been covered. However, they do not tell if the tests actually test and verify the intentions. Mutation testing does this by deliberately injecting faults into the system under test and verifying how many of them the test suite can detect. For every injected fault that is not detected by the test suite, an additional test should be written. In the academic community, mutation testing is acknowledged as the most promising technique for automated assessment of the strength of a test suite.
Fri 16 AprDisplayed time zone: Brasilia, Distrito Federal, Brazil change
10:00 - 10:25 | Flaky Mutants; Another Concern for MutationTestingNEXTA at Tamandaré Authors: Sten Vercammen, Serge Demeyer, Markus Borg and Robbe Claessens Abstract: Mutation testing is the state-of-the-art technique for assessing the fault detection capability of a test suite. An underlying assumption, rarely mentioned, is that the system under test behaves completely deterministically. This is rarely the case, as each mutant changes the code, it is highly likely that some introduce non-determinism. We call these flaky mutants. As they are only detected intermittently, they cause unreliable mutation testing scores, waste developer time, possibly unfruitful tests, and potential loss in confidence in the mutation testing technique. We want to raise awareness of this issue as we found that these flaky mutants are easy to create and occur in real projects. We also share some thoughts on how to tackle this issue. Mutation testing is the state-of-the-art technique for assessing the fault detection capability of a test suite. An underlying assumption, rarely mentioned, is that the system under test behaves completely deterministically. This is rarely the case, as each mutant changes the code, it is highly likely that some introduce non-determinism. We call these flaky mutants. As they are only detected intermittently, they cause unreliable mutation testing scores, waste developer time, possibly unfruitful tests, and potential loss in confidence in the mutation testing technique. We want to raise awareness of this issue as we found that these flaky mutants are easy to create and occur in real projects. We also share some thoughts on how to tackle this issue. Session Chair: Kristian Sandahl | ||
10:00 25mKeynote | Flaky Mutants; Another Concern for MutationTesting NEXTA Sten Vercammen University of Antwerp, Belgium, Serge Demeyer University of Antwerp, Belgium, Markus Borg RISE Research Institutes of Sweden | ||
Session Chair: Kristian Sandahl