Wing Lam
Talk: Taming Flaky Tests in a Non-Deterministic World
- Date and Time
- October 11, 2022 at 12:30pm
- Location
PLSE Lab
Abstract
As software evolves, developers typically perform regression testing to ensure that their code changes do not break existing functionality. During regression testing, developers often waste time debugging their code changes because of spurious failures from flaky tests, which are tests that nondeterministically pass or fail on the same code. These spurious failures mislead developers because the failures are due to bugs that existed before the code changes. My work on characterizing flaky tests has helped open the research topic of flaky tests, and many companies (e.g., Facebook, Google, Microsoft) have since highlighted flaky tests as a major challenge in their software development.
In this talk, I will describe my recent work on taming flaky tests. Two prominent kinds of flaky tests are order-dependent flaky tests, which pass when run in one order but fail when run in a different order, and async-wait flaky tests, which pass if an asynchronous call finishes on time but fail if it finishes too late. My results include the first automated techniques to (1) fix order-dependent flaky tests, fixing 92% of such flaky tests in a public dataset; (2) reduce the number of spurious failures from order-dependent flaky tests, reducing such failures by 73%; and (3) speed up async-wait flaky tests while also reducing their spurious failures, speeding up such tests by 38%. Overall, my work has helped detect more than 3000 flaky tests and fix more than 1000 flaky tests in over 300 open-source projects.
Bio
Dr. Wing Lam is an Assistant Professor in the Computer Science department at George Mason University. He is also a Part-Time Applied Research Scientist at Fraunhofer CMA. Dr. Lam works on several topics in software engineering, with a focus on software testing. His research improves software dependability by characterizing bugs and developing novel techniques to detect and tame bugs. He has published in top-tier conferences such as ESEC/FSE, ICSE, ISSTA, OOPSLA, and TACAS. His techniques have helped detect and fix bugs in open-source projects and have impacted how Microsoft and Tencent developers test their code. Dr. Lam has been the recipient of several awards, including an ACM SIGSOFT Outstanding Doctoral Dissertation Award. More information is available on his web page: https://cs.gmu.edu/~winglam.