A two-step LLM pipeline that turned slow, incomplete manual research across 20,000+ records into a measurable, autonomous workflow — surfacing 410 entries the team had missed.
The client held a dataset of more than 20,000 records and needed to keep finding and validating net-new entries that belonged in it. Analysts did this by hand: searching, cross-checking sources, and deciding case by case whether an entry was already covered.
Two things were wrong with that. It did not scale, so coverage depended on how much analyst time was available in a given week. And it was unmeasurable — there was no way to say what percentage of the real population had actually been found, or to show a stakeholder the size of the gap.
The pipeline runs in two stages. The first establishes a reliable picture of the existing dataset so the model never re-surfaces something already covered. The second fans out to find and validate new candidates at a speed no analyst team could match.
A RAG layer indexes the existing 20,000+ records. Every candidate the pipeline considers is checked against this index first, so known entries are filtered out before any expensive processing happens. This is what keeps the output clean instead of a pile of duplicates.
Unmatched candidates are enriched and verified through concurrent LLM calls — pulling supporting detail, checking it against source signals, and scoring confidence. Running these in parallel is what turns a multi-week manual pass into a single automated run.
Validated net-new entries are written back in a structured format with a confidence score and a source trail, plus a summary a stakeholder can read: how many were found, how confident the system is, and what share of the estimated total the dataset now covers.
The 410 entries were ones the manual process had not caught. Just as important, the client could finally state coverage as a figure and defend it, which changed the conversation from are we missing things to we cover 61% and here is the plan for the rest.
Ground a model against what an organisation already knows, then let it search in parallel for what it does not, and score everything it returns. That shape applies well beyond one dataset:
Market and competitor mapping — find companies, products, or deals absent from an internal view.
Compliance and risk screening — surface entities or relationships not yet on file.
Research and intelligence — keep a knowledge base current without a standing team doing manual sweeps.
In each case the win is the same one the client got here: the work scales past headcount, and the gap becomes measurable enough to plan around.