ICST 2021
Mon 12 - Fri 16 April 2021
Wed 14 Apr 2021 15:30 - 16:00 at Porto de Galinhas - Faults and Fault Injection Chair(s): André T. Endo

Tests that cause spurious failures without any code changes, i.e., flaky tests, hamper regression testing, increase maintenance costs, may shadow real bugs, and decrease trust in tests. While the prevalence and importance of flakiness is well established, prior research focused on Java projects, thus raising the question of how the findings generalize. In order to provide a better understanding of the role of flakiness in software development beyond Java, we empirically study the prevalence, causes, and degree of flakiness within software written in Python, one of the currently most popular programming languages. For this, we sampled 22352 open source projects from the popular PyPI package index, and analyzed their 876186 test cases for flakiness. Our investigation suggests that flakiness is equally prevalent in Python as it is in Java. The reasons, however, are different: Order dependency is a much more dominant problem in Python, causing 59% of the 7571 flaky tests in our dataset. Another 28% were caused by test infrastructure problems, which represent a previously undocumented cause of flakiness. The remaining 13% can mostly be attributed to the use of network and randomness APIs by the projects, which is indicative of the type of software commonly written in Python. Our data also suggests that finding flaky tests requires more runs than are often done in the literature: A 95% confidence that a passing test case is not flaky on average would require 170 reruns.

Wed 14 Apr

Displayed time zone: Brasilia, Distrito Federal, Brazil change

15:30 - 17:00
Faults and Fault InjectionResearch Papers at Porto de Galinhas
Chair(s): André T. Endo Federal University of Technology - Paraná (UTFPR)
15:30
30m
Paper
An Empirical Study of Flaky Tests in Python
Research Papers
Martin Gruber BMW Group, Stephan Lukasczyk University of Passau, Florian Kroiß , Gordon Fraser University of Passau
Pre-print
16:00
30m
Paper
Fast Kernel Error Propagation Analysis in Virtualized Environments
Research Papers
Nicolas Coppik TU Darmstadt, Oliver Schwahn TU Darmstadt, Neeraj Suri
16:30
30m
Paper
Dissecting Strongly Subsuming Second-Order Mutants
Research Papers
João Paulo Diniz Federal University of Minas Gerais, Brazil, Chu-Pan Wong Carnegie Mellon University, USA, Christian Kästner Carnegie Mellon University, Eduardo Figueiredo Federal University of Minas Gerais, Brazil