A private AI search engine for a regulated accountancy firm: 12K documents, local models, no data leaves the buildingA private AI search engine for a regulated accountancy firm: 12K documents, local models, no data leaves the building

The firm holds years of client files, letters, working papers and tax records. Finding the right paragraph meant knowing which folder to open and reading document after document: about 15 minutes per lookup. Cloud AI was not an option, because client data cannot be sent to a third party.

What was broken

Knowledge lived in thousands of files. Answering a client question meant knowing which folder to open, then reading. New staff depended on colleagues to find anything.

What it cost

• About 15 minutes per lookup, many times a day across the team.
• Cloud AI tools were off the table: client data cannot go to a third party.
• Answers depended on who remembered where a document was.

The brief

Give staff answers in seconds from their own documents, with the source of every answer, without a single byte leaving the firm's hardware, and without the model ever inventing a figure, a date or a client fact.

• Client data never leaves the firm's own hardware: no cloud AI, no third-party processing, no data processing agreement to sign.
• One GPU, not a data centre: the model had to be strong enough for professional questions and still fit.
• A regulated profession: a confident wrong answer is worse than no answer.
• Documents change every day, so quality has to be re-measured, not assumed.

• Architect and builder, as the firm's Director of AI & Business Automation.
• The benchmark questions came from the staff who use the system every day.
• Used by the full team (10+ people).

1. 00

   Before

   About 15 minutes per lookup; cloud AI ruled out for client data.

2. 01

   Design

   Local-only architecture: open-weight models, local embeddings and index, no outbound internet.

3. 02

   Benchmark

   100 real staff questions written down as the acceptance test.

4. 03

   Quantisation

   The 4-bit 30B-class model accepted after under 2 points of benchmark loss.

5. 04

   Assurance

   DPIA signed off; 150-prompt red team with zero leaks; external pen test with 0 critical and 0 high findings.

6. 05

   Live

   In use by the full team; lookups under 30 seconds.

7. 100

   First 3 months

   2,000+ staff queries.

What was built

• A 30B-class model on a single 24 GB GPU. Qwen3-30B-A3B, a mixture-of-experts model that activates about 3B parameters per token, 4-bit quantised and served with Ollama and LM Studio. Quantisation cut model memory about 70% for under 2 points of benchmark accuracy.
• Hybrid retrieval. Keyword and local-embedding search with a local re-ranker, so exact references (names, tax numbers, dates) and meaning both match: the right source is in the top 5 for 96% of benchmark questions.
• Grounded answers only. Every answer quotes and cites the passages it came from. Below the confidence threshold the system answers "not in the documents" instead of guessing: zero uncited answers on the benchmark.
• Isolation by design. No outbound internet on the AI server, telemetry off, and answers filtered by the user's folder permissions, so one client's files never surface in another's answer.
• A benchmark that gates every change. 100 real staff questions; an answer passes only if it is correct and cites its source. It reruns before any model, prompt or index change, and a failed run blocks the change. A separate 500-question held-out set checks every claim against its source passage: under 1% unsupported statements.
• Red-teamed, then pen-tested. 150 attack prompts (prompt injection, jailbreaks, requests for folders the user may not open) produced zero leaks and zero out-of-permission documents. An external penetration test of the private AI and the client platform found 0 critical and 0 high issues; the 2 medium ones were fixed within 48 hours.
• Personal data masked at the edge. Where a task does use a cloud model, names, IBANs and PPS numbers are masked before the call: 100% caught on a 500-document test set.
• Always current. Incremental re-indexing makes new documents searchable within 15 minutes, with the benchmark rerun on the new index.

• No outbound internet on the AI server and telemetry off: there is nowhere for data to go.
• Answers filtered by the user's folder permissions, so no document leaks across clients.
• No source, no answer: below the confidence threshold the system says "not in the documents".
• Every answer shows its citations, so a person can check it in one click.
• A 100-question benchmark gates every model, prompt and index change.
• Red-teamed with 150 attack prompts and pen-tested externally before launch.
• A named person approves any AI action that touches money or client data, with a full audit trail.

How it was tested

The 100-question benchmark reruns before any model, prompt or index change, and a failed run blocks the change. The 4-bit model was accepted only after the benchmark showed under 2 points of accuracy loss against the full-precision run, and the same gate shipped 3 open-weight model upgrades with no regression reaching staff. A data protection impact assessment was signed off before go-live.

• The server, the models and the index, on its own hardware.
• The benchmark and the runbook to rerun it.
• The data protection impact assessment, the red-team set and the pen-test report.

• Qwen3-30B-A3B (4-bit)
• Llama
• Ollama
• LM Studio
• Local embeddings
• Local re-ranker
• Hybrid search (keyword + vector)
• Python