Beyond the Hype: ML-Driven Test Intelligence at Scale - What Works, What Fails, and Why It Matters

The promise of ML-driven test intelligence is compelling: faster feedback loops, smarter test selection, and anomaly detection that catches what traditional automation misses. But the gap between that promise and production reality is where most teams quietly struggle and rarely talk about it publicly. This session closes that gap.

Drawing from hands-on production experience integrating machine learning into an enterprise QA pipeline inside one of the largest financial institutions in the United States, this talk delivers an unfiltered account of what ML-augmented testing looks like when the stakes are real — regulated environments, high transaction volumes, zero tolerance for silent failures, and teams that still need to ship on time.

We'll walk through a dual-layer ML architecture: a gradient boosting model (XGBoost + scikit-learn) for intelligent test selection, and an LSTM autoencoder (TensorFlow/PyTorch) for post-deployment anomaly detection. Not as a vendor showcase but as a case study in failure, iteration, and eventual production stability. You'll see where the models performed beyond expectations, cutting test execution time dramatically while maintaining defect detection precision. You'll also see where they failed, including a production payment bug that passed ML-assisted test selection cleanly and only surfaced through a human engineer's judgment. That failure became the foundation of a human-veto policy that now governs every ML recommendation in the pipeline.

This talk also tackles the organizational side that most technical sessions ignore: how QA roles evolve when ML enters the pipeline, how to govern model recommendations without creating bottleneck processes, and how to build team trust in a system that sometimes says "skip this test." Critically, this session addresses the security dimension, specifically prompt injection risks in AI-augmented pipelines and why testing AI-integrated systems requires a fundamentally different threat model than testing traditional software.

Whether you are exploring ML for test optimization, mid-implementation and hitting friction, or evaluating whether the investment is worth it, this session gives you the real data, the real failures, and four reusable human-ML collaboration patterns you can take back to your team on day one.