Grounded by design: the architecture of Cleo

Lawyers are right to be skeptical of AI, and the skepticism is well founded. When researchers asked general-purpose chatbots specific, verifiable questions about randomly selected federal court cases, the models fabricated authority (cases, citations, holdings that were never written) at rates between 58% and 88%, depending on the model.¹ Courts around the world have sanctioned lawyers who filed briefs quoting decisions that do not exist; the public database that tracks such cases listed nearly 1,700 decisions at the time of writing.²

The temptation is understandable. These systems are fluent, and fluency reads as competence. An answer written in confident, well-formed prose invites the assumption that it is also correct. But fluency and accuracy are independent properties of a language model. It can be articulate and wrong in the same sentence, and it gives no outward signal of which one you are getting. A lawyer’s instinct to verify is not caution to be engineered away. It is the right response to the tool.

The lesson is not that the technology is useless. It is that professional responsibility does not transfer to it. When a pleading is filed, the lawyer’s name is on it, and the duty of candor to the tribunal belongs to the lawyer alone, not to the vendor and not to the model. A tool that asks a lawyer to take its output on faith is asking for precisely the thing a careful lawyer cannot give.

That sets the design problem for any AI system that touches legal work. It cannot ask to be believed, because belief is what a diligent professional withholds until the work is checked. It has to make the checking easy instead: to expose its sources and its reasoning rather than hide them behind a confident sentence.

An AI tool for lawyers must be built to be checked, not believed.

Everything that follows in this report is a consequence of taking that one sentence seriously.

Cleo’s answer is the last step of a process, not the first. Before you ask a question, the documents in your matter have already moved through several stages, and each one leaves something you can inspect.

A document enters the system when it is uploaded. Cleo reads it, including scanned files and photographs of paper, through text recognition, so that a signed contract or a stamped pleading becomes searchable text rather than a flat image. It then structures what it read: parties, dates, defined terms, obligations, and amounts, each linked back to the passage it came from. The structured matter is indexed so it can be retrieved by meaning as well as by keyword, which is what lets you surface the indemnity clause even when the document never uses the word “indemnity.”

Alongside your matter sits the law itself: a corpus of more than 130,000 Philippine legal sources, spanning Supreme Court decisions, statutes, rules, and administrative issuances, with statute coverage audited for completeness against the official numbering of Republic Acts, Executive Orders, and Presidential Decrees. Section 4 describes how Cleo searches it.

When you put a question to Cleo, it retrieves the material that bears on the question, from your matter and from that corpus, reasons over what it found, and returns an answer with its sources attached. The retrieval, the structuring, and the citations are not decoration around the answer. They are the steps the answer is assembled from.

Cleo’s answer is the last step of a pipeline you can inspect, not the first draft of its imagination.

Philippine legal work still runs on paper: scanned annexes, stamped receiving copies, photographed contracts. The first job is to turn those into text a system can search. Optical character recognition converts each page image into words while keeping the pairing between the recognized text and the original page, so anything extracted downstream can always be read back against the source. Recognition is not perfect, especially on degraded scans, and the pipeline treats it that way: documents whose recognition quality falls below threshold are flagged rather than silently indexed, because an index built on garbled text produces confident, wrong retrieval later.

Extraction then structures the matter. Parties, dates, defined terms, obligations, and amounts are pulled into a structured view, and every extracted item carries a pointer to the passage it came from. The pointer is not a convenience feature. It is the beginning of the property this whole report is about: nothing in Cleo’s view of your matter floats free of a source you can open.MethodOptical character recognition (OCR)

Ask a language model a legal question cold and it will answer from whatever its training distilled: an average of the internet, with no guarantee the controlling Philippine authority is in it, and no way to tell when it is not. Retrieval replaces that open-ended recall with a concrete first step: find the passages that actually bear on this question, then reason over those. This section describes how Cleo finds them.

4.1The unit of retrieval

Nobody wants the 200-page bundle back as an answer; they want the clause. Firm documents are therefore divided into overlapping passages, sized so that a retrieved passage is precise enough to quote and the overlap prevents a clause from being stranded across a boundary. For the law corpus, the better unit is often the whole instrument, since a decision’s reasoning rarely survives being cut into fragments. Which unit works better is an empirical question, not a matter of taste, and Section 6 describes how we settle it with measurements rather than intuition.

4.2Searching by meaning

Each passage is converted into an embedding: a numerical representation with the useful property that texts about the same thing end up near each other, whatever their wording. A question is embedded the same way, and the passages nearest to it are retrieved. This is what the research literature calls dense retrieval,³ and it is the mechanism behind searching by meaning: the indemnity clause surfaces even when neither the clause nor the question uses the word. At the scale of a hundred thousand documents, comparing every question against everything is wasteful, so the index is organized for fast approximate nearest-neighbour search using a graph-based structure that navigates to the closest candidates instead of scanning them all.⁴MethodDense passage retrievalKarpukhin et al., 2020 [3]MethodApproximate nearest-neighbour search (HNSW)Malkov & Yashunin, 2020 [4]

4.3Searching by the letter

Meaning-based search has a known blind spot: lawyers often need the exact term, not its neighbourhood. A defined term, a docket number, a statute’s designation, a party name. So dense retrieval runs alongside classical full-text search, which ranks passages by the statistics of the query’s terms, weighting rare, discriminating words over common ones. This is the tradition of probabilistic relevance ranking that has anchored search engines for decades.⁵ One leg catches paraphrase; the other honors the letter of the text.MethodLexical relevance rankingRobertson & Zaragoza, 2009 [5]

4.4One ranking from two, then a harder look

The two searches return two differently-ordered lists, and they are merged by reciprocal rank fusion, a deliberately simple rule that rewards passages ranked highly by either method and has proven hard to beat as a fusion baseline.⁶ The merged candidates then get a harder look: a reranking model reads the question and each candidate passage together and re-scores them, a step shown to sharply improve precision over retrieval alone.⁷ Over-retrieve broadly, then re-rank strictly: recall first, precision second, in that order for a reason. A passage that never surfaces can never be re-ranked into an answer.MethodReciprocal rank fusionCormack, Clarke & Buettcher, 2009 [6]MethodNeural re-rankingNogueira & Cho, 2019 [7]

4.5Retrieval-augmented generation

Everything above exists so that the model’s generation step starts from real text. The architecture the research community calls retrieval-augmented generation conditions the model on retrieved passages rather than on its own recall,⁸ and retrieval augmentation has been shown to substantially reduce fabricated content, precisely because the model is reasoning over material that exists.⁹ One sentence of honesty belongs here: retrieval reduces hallucination; it does not abolish it. A model can still misread what it was given, which is why Section 5 exists.MethodRetrieval-augmented generationLewis et al., 2020 [8]

4.6An agent inside a fence

Cleo is not a single retrieve-then-answer pass. It works the way an associate works a record: search, read what came back, notice what is missing, search again. Architecturally it is a tool-using agent that interleaves reasoning with retrieval,¹⁰ and its tools (search the corpus, look up a statute, read a document) are the only channel through which it touches material at all.MethodReasoning-and-acting agentsYao et al., 2023 [10]

That channel is where the fence lives. The platform, not the model, decides which documents a conversation may read. When a conversation is scoped to a matter, the system passes the permitted documents to the model directly; the model cannot name a file it was not given, and it cannot talk its way into another matter’s documents or another firm’s data. Access is a property of the system’s design, enforced outside the model, not a boundary the model is trusted to respect. A firm’s confidence on this point should not rest on the model being well-intentioned. It should rest on the fact that the model was never handed the keys.

Grounding makes an answer likely to be right. Verification makes it checkable, and in legal work checkable is the property that matters most.

Cleo cites the passages it relies on, and it cites them as verbatim quotations from the source, not paraphrases. Before an answer is presented, the quoted text is checked against the document it claims to come from, and a quotation that cannot be found there is stripped rather than shown. The bias is deliberate. Cleo would rather present an answer with one fewer citation than present a citation you cannot trust.

It also keeps apart two things that language models tend to blur. What the record says, the words on the page, is presented as quotation. What Cleo infers from the record (a conclusion, a characterization, an implication) is labeled as inference. You are never left to guess whether a sentence reports the document or reasons beyond it. And because every citation carries its source, verification is a glance rather than an investigation: the passage is one step away, in the document it came from.

The research community has a name for the property being engineered here: attribution, the degree to which a system’s statements are supported by sources a reader can identify and check.¹¹ The strictest form of attribution is the direct quotation, because it is machine-checkable: either the quoted span appears in the source or it does not.¹² That is the standard Cleo’s citations are held to.MethodAttribution in generationRashkin et al., 2023 [11]MethodAnswers supported by verified quotesMenick et al., 2022 [12]

The distinction between retrieval and verification is worth stating exactly, because it is where many AI tools quietly fail. Retrieval improves the odds that a source exists and is relevant. Verification confirms that the words attributed to the source are really in it. A system can retrieve the right case and still misquote it, or cite a genuine case for a proposition it does not support. The need is not hypothetical: an independent study of purpose-built, retrieval-based legal research tools found that market-leading systems still produced incorrect or misgrounded answers between 17% and 33% of the time.¹³ Retrieval narrows the gap; only verification closes it, by turning a claim about a document into something you can confirm against it.

None of this makes a wrong answer impossible. What it does is make a wrong answer visible, and a visible error is one a lawyer can catch before it reaches a client or a court.

A wrong answer that announces its sources is a caught answer.

An architecture document tells you what a system is supposed to do. It does not tell you whether the system does it. For that we run evaluations, and we hold ourselves to the same standard the rest of this report argues for: a claim about quality should be a measurement, not a feeling.

Retrieval is scored against a golden set: a fixed collection of real legal questions, each labeled in advance with the authorities that should come back. The pipeline runs every question, and we compute where each correct authority ranked, summarized as recall (of the authorities that should have surfaced, how many landed in the top results). The set is deliberately adversarial. Alongside ordinary doctrinal questions and direct case lookups, it includes a category seeded from Cleo’s own past mistakes, questions it has answered wrongly in the wild, and a category of questions whose answer is not in the corpus at all, where the correct behavior is to return little. That last category probes for the failure mode this report keeps returning to: confident output where honest emptiness was the right answer.

The golden set is also a gate. Every candidate change to the retrieval pipeline is scored against the baseline before it ships, per category and not just on average, because an average can hide half the story. The gate has already earned its keep: a proposed change to the retrieval unit improved direct case lookups but regressed the known-failure category by 36 points, and it did not ship. A “no” backed by numbers is the evaluation doing its job.

Answer quality is evaluated separately, because retrieving the right law and writing a correct answer from it are different failures. For this we use the 2024 Philippine Bar examination: 120 questions across its six subjects, run through Cleo’s production retrieval, tools, and prompts, and graded twice. A deterministic rubric provides a conservative floor, and a strict language-model judge, anchored to the official answer key, re-scores the failures, a technique the evaluation literature has studied closely.¹⁴ We treat the results as diagnosis rather than as a headline. Two findings shaped the pipeline. First, the weakest subjects were weakest for every model tier, which points at gaps in the corpus and its retrieval rather than at the models, and redirected engineering effort accordingly. Second, and more important: the most dangerous failure signature was not the wrong conclusion but the right conclusion reached on the wrong legal basis, an answer that looks correct and cites the wrong law. Accuracy-only scoring cannot see that error. A citing, verifying system is built to expose exactly it, which is why Section 5’s discipline exists.MethodLLM-as-judge evaluationZheng et al., 2023 [14]

Evaluations rot if they stand still. Every real retrieval failure we catch in use becomes a permanent test case in the golden set, so the benchmark sharpens over time instead of flattering us.

No architecture removes the need for a lawyer’s judgment, and it would be dishonest to imply otherwise. The same design that makes Cleo checkable is what makes its limits worth stating plainly.

Cleo can misread. Ambiguous language, an unusual clause structure, or a poorly scanned page can lead it to structure or summarize a document wrongly, and recognition errors on bad scans propagate into everything built on them. Retrieval can miss: passage boundaries can separate a clause from the proviso that qualifies it, and meaning-based search can surface the passage that resembles the question rather than the one that answers it. Hallucination in generation is a documented failure mode of these models as a class,¹⁵¹⁶ and the architecture described here reduces exposure to it and makes residual errors visible. It does not make them impossible. The law corpus, however carefully maintained, is a working copy and not the official record; it does not replace consulting the controlling source when the stakes require it.

These are not defects to apologize for. They are the honest boundary of what the tool does. A retrieval system is only as complete as what it has indexed, and a summarizer is only as reliable as the text it was given. Naming these limits is itself part of making Cleo checkable: a lawyer who knows where a tool can fail knows where to look.

When Cleo cannot ground an answer, it is designed to say so rather than guess. But the lawyer who relies on it still carries the duties the technology cannot: to read the authority, to check a quotation against the source Cleo points to, to decide what the law means in this matter. Cleo drafts and finds. It does not advise, and it does not sign. The judgment in the loop is the lawyer’s, and it stays there.

If a capability can’t be verified by the lawyer using it, we don’t ship it.

We wrote this because lawyers deserve to know how a tool that touches their matters actually works, not as a pitch but as architecture. The objection we hear most often is the right one: how can I trust this? The honest answer is that you should not have to. You should be able to check.

So we hold to verification over faith. Cleo retrieves before it reasons, cites what it relies on, verifies what it quotes, and keeps what the record says separate from what it infers. Where it cannot do those things, it is built to say so. We score the pipeline against evaluations that include our own past failures, and a change that cannot beat them does not ship. The lawyer stays the judgment in the loop, because that is where the responsibility already lives and where it belongs.

If any of this reads as a claim rather than a fact, press on it. Bring your hardest matter to a demo: the ambiguous contract, the messy record, the question you would not hand to a junior. Then check every step of what Cleo gives back.