STARWEST 2019 Concurrent Session : Testing AI-Based Systems: A Gray-Box Approach


Wednesday, October 2, 2019 - 2:45pm to 3:45pm

Testing AI-Based Systems: A Gray-Box Approach

Add to calendar

Testing artificial intelligence- and machine learning-based systems presents two key challenges. First, the same input can trigger different responses as the system learns and adapts to new conditions. Second, it tends to be difficult to determine exactly what the correct response of the system should be. Such system characteristics make test scenarios difficult to set up and reproduce and can cause us to lose confidence in test results. Yury Makedonov will explain how to test AI/ML-based systems by combining black box and white box testing techniques. His "gray box" testing approach leverages information obtained from directly accessing the AI’s internal system state. Yury will demonstrate the approach in the context of testing a simplified ML system, then discuss test data challenges for AI using pattern recognition as an example and share how data-handling techniques can be applied to testing AI. Come and learn techniques applicable to testing a wide range of AI systems.

Yury Makedonov

Yury Makedonov was trained as a researcher and worked in a research and development institution dealing with composite materials. He has a Ph.D. degree in physics and math, though he is not a rocket scientist anymore; now he is using his skills and knowledge to improve software quality. Yury has more than twenty years of testing experience, from small startups to large companies and government organizations, and recently has been working as a QA manager, test manager, and consultant.