Testing, Testing, 1, 2, 3: Building & Testing Great Products in the AI Era
GOOD NEWS! To build a high quality customer-centric AI app, you need to bring your old-timey software testing practices into the AI era. But what does that mean? Well, it turns out building AI apps is pretty much like building ANY other app…with a FEW added extras. We need to bring customer focus to EVERY step of our building and testing process. Things like, how do we choose the data we’ll use to train our AI product? Which humans are in the loop? How to fine-tune and ground our models in REAL data? And, of course, how to leverage our usual software quality bread-and-butter (red-teaming, deployment, and feedback loops with REAL customers?) In this talk, Jeremiah will suggest practical testing methods to identify and solve customer problems, focusing on AI systems with meta-prompting and Retrieval-Augmented Generation (RAG). You’ll leave with a solid understanding of how to drive customer-centricity, in order to build and test your AI systems better.
Jeremiah Marble is a product leader, founder, author and speaker. He focuses on building high-quality, customer-centric products in the AI era. His career includes leadership roles at Mozilla and Microsoft, where he helped grow the Windows Insider Program from a whiteboard to 22+ million people worldwide. An advocate for tech for good and social entrepreneurship, Jeremiah launched “Do The Thing Academy” and co-wrote Model 47: A Startup Storybook to help non-traditional founders turn napkin ideas into responsible businesses. He co-founded ethical fashion line Prima Dona Studios, empowering single mother tailors worldwide - including in Senegal, Mexico, and with Afghan refugees in Seattle. Previously, Jeremiah worked for the UN in Africa and Asia, was a Fulbright Scholar to Costa Rica, and volunteered in the Dominican Republic with the Peace Corps. He earned an MBA from Wharton, MA in International Studies from U Penn, and CS undergrad from Columbia.