ECOOP 2018
Sun 15 - Sat 21 July 2018 Amsterdam, Netherlands
co-located with ECOOP and ISSTA
Mon 16 Jul 2018 15:00 - 15:20 at Zurich II - Testing and Fault Localization Chair(s): Cindy Rubio-Gonzalez

Test-suite reduction (TSR) speeds up regression testing by removing redundant tests from the test suite, running fewer tests on the future code changes. A developer considering TSR must consider the cost of potentially removing tests that can detect faults in the future. Furthermore, the developer also needs some way to predict how well the reduced test suite performs w.r.t. detecting faults in the future. Prior research evaluated this cost of TSR using faulty program versions, each constructed with a single seeded fault. Furthermore, it is unknown if any metric measured on the reduced test suite at the point of reduction is effective at predicting the missed faults.

In this paper, we perform the first extensive study on TSR using real test failures in (failed) builds that occurred for real code changes. We analyze 1478 failed builds from 32 GitHub projects that run their tests on the Travis continuous integration service. We compute reduced test suites on early, passing versions of each project. As there can be multiple faults in each failed build, we propose FFMap, a family of mappings from test failures to faults. We use these mappings to determine the percentage failed builds where the reduced test suite misses to detect all the faults called FBDL. We find that reduced test suites can miss up to 52.2% of failed builds. We find that FBDL of reduced test suites is higher than suggested by traditional TSR metrics. Furthermore, traditional TSR metrics are not good predictors of FBDL, and even our proposed usage of historical FBDL to predict future FBDL, while a better predictor than the traditional TSR metrics, is still not a good predictor. The lack of a good predictor makes it difficult for developers to decide whether or not to use reduced test suites. Our results raise important concerns about automated TSR.

Mon 16 Jul

issta-2018-Technical-Papers
14:00 - 15:30: ISSTA Technical Papers - Testing and Fault Localization at Zurich II
Chair(s): Cindy Rubio-GonzalezUniversity of California, Davis
issta-2018-Technical-Papers14:00 - 14:20
Talk
Seung Yeob ShinSnT Centre/University of Luxembourg, Shiva NejatiSnT Centre/University of Luxembourg, Mehrdad SabetzadehSnT Centre/University of Luxembourg, Lionel C. BriandSnT Centre/University of Luxembourg, Frank ZimmerSES Techcom
issta-2018-Technical-Papers14:20 - 14:40
Talk
Jaekwon LeeUniversity of Luxembourg, Luxembourg, Dongsun KimUniversity of Luxembourg, Tegawendé F. BissyandéUniversity of Luxembourg, Luxembourg, Woosung JungSeoul National University of Education, Yves Le TraonUniversity of Luxembourg
issta-2018-Technical-Papers14:40 - 15:00
Talk
Per Erik StrandbergWestermo Research and Development AB, Thomas Ostrand, Elaine WeyukerMälardalen University, Daniel SundmarkMälardalen University, Wasif AfzalMälardalen University
issta-2018-Technical-Papers15:00 - 15:20
Talk
August ShiUniversity of Illinois at Urbana-Champaign, Alex GyoriFacebook, Muhammad Suleman MahmoodUniversity of Illinois at Urbana-Champaign, Peiyuan ZhaoUniversity of Illinois at Urbana-Champaign, Darko MarinovUniversity of Illinois at Urbana-Champaign
issta-2018-Technical-Papers15:20 - 15:30