Comprehensive evaluation · preliminary · public data only

How the tool was evaluated, and what it actually does

A detailed, measured account of the system's retrieval accuracy, answer quality, robustness, and speed — including where it is weak, and how it compares with how this research is done today. Every number below is measured on the live system, across a hand-curated benchmark and a 500-question machine-generated one. This is a preliminary evaluation; figures are indicative and will change as the benchmarks are expert-reviewed. Nothing here is a marketing claim — the most important finding is a limitation (§6).

0 · Headline results

The headline: finding the right authority is strong and reliable; the generated answer is a convenience that must be checked against the cited source.

1 · System & corpus

Given a plain-English question, the tool retrieves the governing NC authority (each result cited to source, section, and page) and can write a short answer assembled only from those passages, with inline [n] citations, declining when the passages don't support one.

The statute set is curated (Chapter 7B core + supporting sections), not the entire NC General Statutes. Federal law and case law are out of scope.

2 · Retrieval accuracy

Retrieval was measured two ways: a small hand-curated set (70 questions) and a large machine-generated set (500 questions — a language model wrote realistic practitioner questions from sampled passages, with the source passage as the known-correct label). Recall@k = how often the governing authority is in the top-k results.

Statutes are strong and stable — ~90–92% top-3 on both the small and large benchmarks. Policy shows a wide range (74–96%) that is mostly a scoring artifact, not a retrieval failure: the strict machine benchmark only credits retrieving the exact same document, but NCDHHS has many overlapping documents. On manual inspection of the strict "misses," about half were sibling documents that legitimately answer the question (e.g. the SDM Intake policy instead of the CPS Intake policy; the in-home case-plan form instead of the permanency case-plan form) or auto-generated questions too vague to have one right answer. The lenient score (which credits any correct document) is the fairer estimate; the truth sits between the two.

Failure analysis

When the exact authority is not first, the results ranked above it are almost always closely-related provisions — for statutes, an adjacent section (the answer vs. the failure-to-answer section); for policy, a sibling document on the same topic. Fine distinctions, not random errors. Practical implication: scan the top few results, not just the first.

3 · Component ablation — what each stage contributes

Retrieval is a pipeline: keyword + meaning search, fused, then re-ranked. Turning the stages on one at a time (statute benchmark) shows where the accuracy comes from — the re-ranking step is the single largest contributor.

The shipped configuration was chosen by measurement, not assumption: a full re-rank beat a rank-blend on this stack (statutes +10 pts R@3; policy 92%→96%).

4 · Chunking quality

Legal text was originally split at a fixed character length, which cut long statutory subsections mid-word. A boundary-aware re-chunker (splitting on section markers, enumerators, and sentence boundaries) fixed this:

Readability was verified independently of retrieval accuracy — the two are separate quality axes and both were checked.

5 · Contextual-retrieval experiment (an honest negative result)

We tested Anthropic's Contextual Retrieval — prepending an LLM-written context blurb to each passage before indexing. Measured A/B:

It helped the structured statutes but hurt the form-heavy policy corpus (the blurbs homogenized similar administrative chunks). Rather than apply a technique because it is fashionable, it was adopted only where it measurably helped. This is included to show the evaluation catches regressions, not just wins.

6 · Grounded-answer quality — the weakest link

The optional written answer was audited on 40 questions. Two things were measured: whether it produces a grounded, cited answer, and — harder — whether every claim in that answer is actually supported by the cited sources (judged by a separate language model, then spot-checked by hand).

7 · Robustness & refusal

A tool that answers questions it shouldn't is worse than one that declines. We probed with out-of-scope questions:

• Clearly off-topic (state income tax, California licensing rules, Romeo & Juliet): declined in every case — it says the sources don't support an answer rather than inventing one.
• Adjacent-domain (federal Indian Child Welfare Act active-efforts standard): it produced a cited answer from tangentially-related NC passages instead of declining. This is a real limitation. For questions near — but outside — the corpus (e.g. federal child-welfare law), the tool can return a confident, cited answer built from NC material that does not actually govern the question.

8 · Latency

Search is interactive. Grounded answers are generation-bound and stream token-by-token so results appear while the answer is still being written.

9 · Provenance & integrity

• Every result carries source URL, section, page span, and a SHA-256 content hash (tamper-evidence) — 100% coverage.
• Public-only, read-only — the engine is walled off from any confidential record in code and connects to the data with a read-only key.
• No hidden query rewriting — the query runs verbatim; nothing silently reformulates what was asked.

10 · Compared with business as usual

Each conventional method has real strengths this tool does not.

11 · Limitations & threats to validity

• Answer faithfulness (§6) is the headline limitation — only ~half of grounded answers were fully verifiable against their cited sources. Answers must be checked, not relied on.
• Synthetic benchmark caveat. The 500-question set was machine-generated from the corpus. This can flatter retrieval (questions derived from a passage may be easier than a real user's phrasing) and penalize it unfairly (the strict single-document label misses correct sibling documents, as in the policy scores). Both effects are present; the machine numbers are a useful stress test, not ground truth. The hand-curated set is smaller but human-verified.
• Single automated grader. Faithfulness and much of the scoring used a language model as judge, spot-checked by hand; independent legal review would strengthen validity.
• Curated corpus. Federal law, case law, and non-indexed NC statutes are out of scope and will not resolve.
• Adjacent-domain over-answering (§7) — the most important behavioral risk.
• Answer non-determinism. The generation model can phrase or cite slightly differently run to run; retrieval is deterministic, generation is not.
• Query privacy. Search and answer queries are sent to third-party embedding/reranking and generation services to do the work; they are not stored by this system, but they transit those providers.
• Preliminary. An early evaluation of a system still being tuned.

12 · Methods & reproducibility

Two benchmarks were used: a small hand-curated set (70 questions, human-verified labels) and a large machine-generated set (500 questions — a language model wrote practitioner-style questions from sampled passages, the source passage serving as the label). Retrieval is scored by whether the correct authority appears in the top-k results of the live pipeline (hybrid keyword + meaning, re-ranked); statutes are matched at the section level, policy at the document level. Answer faithfulness was audited by having a separate language model judge whether each claim in a grounded answer follows from its cited sources, with a hand spot-check of the failures. Refusal and latency come from direct calls to the live service. All harnesses and question sets are versioned and re-runnable; figures reflect the system on the date below and will change as the corpus and test sets evolve.