STARWEST 2024 Concurrent Session : Leveraging and Measuring the Use of Formal Testing Methods in Product Development


Thursday, September 26, 2024 - 3:00pm to 4:00pm

Leveraging and Measuring the Use of Formal Testing Methods in Product Development

Without training in software testing that includes formal methods such as equivalence class partitioning, boundary value analysis, decision tables, state diagrams, and others, an engineer, in good faith, will test the code intuitively until reaching “qualitative confidence” that testing is sufficient. The team at Trimble, Inc. provided training to engineers in a wide variety of formal testing methods so they could gain “quantitative confidence” that code has been tested sufficiently, by using methods known to achieve concrete coverage in requirements and code. Subsequently, Trimble leadership wanted to understand via analytics exactly how engineers were using these methods and, ultimately, how this correlated to defects reported over time. Join James, as he outlines how they introduced a custom attribute to allow engineers to specify which techniques they used to create which test cases, as well as the analytics to measure the engineers’ progress over time. He will outline the long-term effort that included a months-long series of training courses, the development of the “applied techniques” custom attribute, analytics charting engineers’ progress in using formal testing methods, and the positive impact this has had on them. Finally, he will describe ongoing work, such as correlating use of formal methods with customer-found defects reported over time.

James Wood
Trimble PPM, Inc.

James Wood is the Software Test Architect for Trimble, Inc. As researcher, developer, and tester, he has worked 38 years for firms that include Siemens, Google, and Ultimate Software. His primary areas of research have included software testing and architecture. As a developer, he has focused on back-end services, application-level APIs, and data persistence. As a tester, he has focused at all levels of the testing pyramid on test automation, test generation tools, frameworks, metrics, and analytics. He has worked on software projects in the areas of hardware design, medical imaging, telecommunications, search, storage, logging, human resources, and construction. He holds B.S. and M.S. degrees in Computer Science and Mathematics from Carnegie-Mellon University and the University of Pennsylvania.