Meta Deploys AI-Driven JiTTests to Catch Code Bugs Before Production
According to Meta. Meta details the production deployment and positive impacts of its Catching Just-in-Time Tests (JiTTests) approach, leveraging large language models to generate executable tests.
- • Meta has transitioned Catching JiTTests from a research prototype into core production infrastructure.
- • The system utilizes an intent-aware workflow and a dodgy diff workflow to automatically generate tests.
- • False positives are filtered using an LLM-as-Judge ensemble combined with a rule-based assessor named RubFake.
- • Meta reports that 589 engineers have engaged with the system, noting a 60% true-positive rate for actionable findings.
Introduction to Catching JiTTests
Meta has shared the results and positive impact of deploying its Catching Just-in-Time Tests (JiTTests) approach into production. With hundreds of millions of lines of code constantly evolving, traditional testing cannot keep pace. Catching JiTTests leverages large language models to generate executable tests that demonstrate concrete behavioral differences, allowing Meta to catch bugs on the fly before code lands in production.
Five-Stage System Workflow
The system workflow operates in five stages: targeting and triggering risky diffs overnight, running two diff-aware test-generation workflows (dodgy diff and intent-aware), executing an automated assessment via an LLM-as-Judge ensemble, applying a rule-based assessor called RubFake for deterministic judgments, and utilizing a human-on-the-loop approach where engineers review outcomes at an aggregate level.
Impact and Results at Meta
According to Meta, 589 engineers have engaged with the testing system's results, reporting a 60% true-positive rate for actionable findings. In the first half of 2026, Catching JiTTests helped catch 74 serious privacy bugs at the diff stage before they landed in production, lowering the overall occurrence of privacy SEVs by 20% during that timeframe.
- 1. engineering.fb.comOpen source