Leveraging and Measuring the Use of Formal Testing Methods in Product Development

Without training in software testing that includes formal methods such as equivalence class partitioning, boundary value analysis, decision tables, state diagrams, and others, an engineer, in good faith, will test the code intuitively until reaching “qualitative confidence” that testing is sufficient. The team at Trimble, Inc. provided training to engineers in a wide variety of formal testing methods so they could gain “quantitative confidence” that code has been tested sufficiently, by using methods known to achieve concrete coverage in requirements and code. Subsequently, Trimble leadership wanted to understand via analytics exactly how engineers were using these methods and, ultimately, how this correlated to defects reported over time. Join James, as he outlines how they introduced a custom attribute to allow engineers to specify which techniques they used to create which test cases, as well as the analytics to measure the engineers’ progress over time. He will outline the long-term effort that included a months-long series of training courses, the development of the “applied techniques” custom attribute, analytics charting engineers’ progress in using formal testing methods, and the positive impact this has had on them. Finally, he will describe ongoing work, such as correlating use of formal methods with customer-found defects reported over time.