Herding Cats in the Cloud: QA Strategies for Non-Deterministic "Agentic" Workflows

The era of "AI Agent writes code" (2025) is over. In 2026, we face the reality of Agentic Orchestration, where autonomous agents (Sales, Support, Operations) interact to execute complex, non-deterministic workflows. As QA leaders, how do we test a system where the output changes every time it runs? Traditional "Given-When-Then" assertions are obsolete. A critical failure is "State Synchronization Failure" (Agent A using stale data updated by Agent B)—a distributed systems bug that conventional automation cannot detect. This session explores Agent Reliability Engineering. Gregory will introduce a new paradigm for quality, learning why "Pass/Fail" is dead and how to implement probabilistic metrics like Pass@k (success rate over $k$ runs) and semantic distance scores. Attendees will learn to implement the 3-Tier Agent Testing Pyramid: from unit tests for tools to using cognitive evals (LLM-as-a-Judge) and running multi-agent simulations. Gregory will demonstrate techniques to catch agent race conditions and implement semantic guardrails to halt "hallucinated logic." Finally, he'll examine the 2026 observability landscape (e.g., AgentOps, LangSmith) and how to integrate these tracing tools into your CI/CD pipelines. This provides a robust framework for guaranteeing quality in an increasingly autonomous cloud.