Analysis
Meta described the production rollout of Catching JiTTests on October 29, an approach that generates tests around proposed code changes. Rather than simply issuing a review comment, the system looks for an executable difference between the changed code and its parent. Meta says the approach helped catch 74 serious privacy bugs before deployment in the first half of 2026, alongside a 20% reduction in privacy incidents of that severity during the period.
The system combines two test-generation workflows with model-based and rule-based filtering. Engineers remain involved through criteria and outcome review, with selected findings brought to their attention. Meta reports that 589 engineers engaged with the results and that their feedback indicated a 60% true-positive rate. These are internal operational results, with the company supplying the measurements and interpretation.
Our reading: the valuable unit is an inspectable disagreement about behavior. A failing test gives a developer something concrete to reproduce, but deciding whether the difference violates the intended design still requires context. A deliberate feature change can invalidate an old expectation. Review therefore needs both the failure and an explanation of the behavior that should have remained stable.
For another organization, the sensible pilot would measure attention as well as detection. How many alerts require an engineer, which ones lead to accepted fixes, and which failures recur without teaching the system anything? A generator can produce more evidence than a team can absorb. Its operational value depends on whether the useful cases reach the right person while the change is still easy to revise.
Source: Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale · 2026-10-29
Why it matters
An executable failure can make an AI finding easier to investigate, provided someone still decides what correct behavior should be.