A Realistic Approach to Scalable Testing for Cross-Team Distributed Systems

Let’s face it. Building tests tends to take more time than developing new features themselves. It is tedious to build sufficient coverage and cumbersome to maintain long-term. However, on an enterprise systems-level, tests are crucial in ensuring bugs or defects are caught before they reach clients. As engineers, we pour our hearts and souls into trying to identify defects before we release our code to clients. But when rolling out big features, it is rare to successfully complete a software deployment process without some errors popping up along the way. This challenge becomes exponentially more difficult for large-scale distributed systems that span multiple teams across an organization. How can we ensure full end-to-end test coverage is captured without dedicating weeks to just writing tests? Building out reliable automated testing requires a tight partnership between engineers, assurance experts, and product managers. Through behavior-driven development (BDD), you can use a set of pragmatic practices to build out your test suite and leverage automated triaging tools to methodically map ownership for each pipeline step in your distributed system. This talk will illustrate how Bloomberg engineers have leveraged the ‘behave’ framework and behavior-driven development to enhance our end-to-end testing of cross-team distributed systems.