Testing the Untestable: Cracking the Code of AI Systems
How do you test a system that learns, evolves, and sometimes hallucinates? Testing AI systems is unlike anything we’ve encountered before—traditional methods fall short, and new challenges emerge at every turn. From validating opaque machine learning models to ensuring fairness, accuracy, and robustness, testers are navigating uncharted territory. The trends are real and AI is coming to a project near you. In this session, Kevin Pyles takes you deep into the world of AI testing. Discover why AI systems demand a new mindset and innovative strategies, including adversarial testing, data validation, and ethical assessments. Learn practical techniques to identify hidden biases, test unpredictable outputs, and ensure your AI behaves as intended in real-world scenarios. Through practical examples and hands-on insights, Kevin will show you how to tackle the complexities of testing AI with confidence. Get real with the secrets of testing AI systems.
Kevin Pyles works for FamilySearch as an AI/ML SDET where he is testing test AI with AI. He has been in the QA industry now for over 18 years with project, product, and management roles throughout. Kevin worked previously for test.ai where he was the Head of Product. Kevin also served on the board for QA at the Point (a local testing meetup), and is an award-winning international speaker. Kevin enjoys golf, attending soccer games, and watching college football. He loves spending time with his family including his wife and 6 kids. He loves Brazilian food and pepperoni pizza, and can’t help himself when there is ice cream.