Submarine systems discipline
I build and document offline LLMs, air gapped RAG, and open weight model setups for environments where data should not leave the room. Right model for the job, sized for the cost you can run. No chatbot theater. Same systems discipline I learned maintaining strategic weapons: if it fails, people get hurt.
I started in the weapons department of U.S. Navy Trident submarines: the Lafayette class USS Casimir Pulaski (SSBN 633) and the USS Ohio (SSBN 726), maintaining Trident C-4 strategic weapons systems. On a boat, if a system fails, you do not open a ticket. You put the crew at risk.
I apply that same habit to language models. Most AI products are built to impress in a demo; they invent facts when they run out of context. For sensitive work, that is not a quirk. It is a failure mode.
I build RAG patterns that treat source documents as the ground truth: strict retrieval boundaries, answers that cite files, and local or private hosting so the text does not wander off to a public chat product. If it is not in the corpus, the system should say so, not improvise.
Offline stacks, open weight models, and local RAG patterns, written up so you can run them, break them, and learn from them.
Size the model to the job. Small and cheap when the task is simple; larger only when the work needs it. Match model, hardware, and cost, do not default to the biggest thing on the shelf.
Habits and guardrails for using LLMs without dumping medical, legal, or business text into public chat tools. Clear rules on what can go in a prompt and what stays offline.
Open weight models, local vector indexes, and air gapped RAG pipelines that run inside a perimeter you control. Daily work without a public API in the critical path.
Local RAG over large document sets with citation first answers, for demos, training fixtures, and personal R&D.
Security is something you design in, not a sticker you add after the fact. I bring that habit to AI: if the model can fail people, you build for failure modes first.
AI safety here means practical control of risk. Not science fiction timelines. It means: sensitive text does not leave your machines by accident; answers that cannot be checked are not “insights”; and the people using the tools know what they are allowed to paste into a prompt. If a system is clever but unaccountable, it is not safe enough for real work.
Offline and air gapped patterns exist so medical, legal, family, and business documents are not dropped into public chat products under vague retention terms. Safety starts with where the bits go.
A fluent wrong answer is worse than a slow correct one. Retrieval boundaries, source citations, and structured nulls when the corpus does not support an answer, hallucination is a systems failure, not a personality quirk.
Models draft and retrieve. Humans decide what is true, what gets filed, and what gets acted on. No “autopilot” for anything that affects health, money, legal standing, or someone else’s privacy.
Before you trust a stack: what leaks if a laptop is stolen? What happens if the model invents a policy clause? Who can change the index? Safety work is boring on purpose, it is the questions you ask before you scale use.
Free guides for high stakes document work. Local vault first. Then offline AI when the hardware is there, or careful home tier tools when it is not. As secure as you can afford is a real design choice, not a purity test. Not a product pitch.
Organize STRs and medical files in a local registry. Use local models if you can; careful hosted tools if you cannot.
View Blueprint Guide →Threat model custody notes and ledgers. Offline when hardware allows; sparse, careful home tier help otherwise.
View Blueprint Guide →Bylaws, minutes, and budgets in a board controlled vault, with AI tiers boards can actually sustain.
View Blueprint Guide →Prevent thermal degradation on soldered unified memory during 24/7 continuous LLM inference workloads.
View Blueprint Guide →Treat LLM output like any other system output: define the failure modes, then design against them.
Index the documents that matter. Force the model to cite them. If the answer is not in the set, return a clear null, do not invent a confident paragraph. Hallucination is a systems failure.
Models draft and retrieve. People decide what is true, what gets filed, and what gets acted on. Side by side evidence is the point, so a reader can check a claim in seconds, not trust a black box.
Prefer machines and networks you control: local hosts, private boxes, or locked down private cloud with access control and logs you can actually read. If you cannot see who queried what, you do not own the risk.
Reliable offline language models, and open weight picks that fit the job without burning hardware for no reason.
Day to day: retrieval that stays inside a private perimeter, answers that point back to files, and model fit, which open weight model is good enough for drafting, which needs more capacity for hard reasoning, and what that costs on a workstation or a small private box. Patterns that fail closed when the evidence is missing. Useful for demos, training fixtures, and personal R&D.
Barry's AI is a personal lab and portfolio site. Offline AI, local RAG, and citation first document patterns, for demos, training fixtures, and applied research. Not an active consulting shop.
The through line is submarine systems discipline: reliability first, no hand waving, evidence over theater. Paper certificates are not the product. Working patterns are.
Not “private AI” as a marketing phrase. After the models are on the box, embedding and inference run on hardware you control. Your documents do not leave that box during daily use. Words matter here: locally hosted or offline capable means the pipeline does not phone home. Air gapped means the machine itself has no network path at all. Use the stronger term only when you mean the stronger setup. Here is a concrete stack and why it matters.
A working local stack has four core pieces: a vector store, an embedding model, a language model, and an orchestration layer that ties them together. Orchestration is not just glue code. In practice it usually includes the ingestion path too: splitting documents, reading PDFs and HTML, attaching metadata, and loading the index. For the store, tools like Chroma, Qdrant, or Weaviate can sit on the same machine and index documents as vectors so related passages come back quickly. A personal or team library of thousands of pages is a normal workstation job. For most of that scale a few GB of RAM is enough; you are usually into hundreds of thousands of chunks before memory is the hard limit. Index time still depends on hardware, file types, and how you chunk.
Embeddings turn raw text into those vectors. Once the weights are on the box (via Ollama, ONNX, or PyTorch), embedding runs offline with no internet. A common Ollama choice is nomic embed text: the usual default output size is 768 dimensions, and Matryoshka training means you can often request a shorter vector if you want. For a larger Ollama native option around 1024 dimensions, mxbai embed large is a better example than treating e5 large as “the technical model.” e5 large v2 is a solid general purpose embedder, but it is not technical specialized, and it is not a native Ollama model. You typically run it through sentence transformers or a custom ONNX setup if you want it.
The language model is often something in the Llama 3 8B or
Mistral 7B class, served locally (for example through Ollama). Quantized
7B to 8B models commonly fit in roughly 6 to 8 GB of VRAM on a capable NVIDIA GPU (Q4 is often
closer to 5 to 6 GB; Q5 pushes toward 7 GB). Apple Silicon (M1 through M4) with unified memory
is also a normal local target; Ollama can use Metal acceleration there. They also run slower on
CPU with plenty of system RAM. The model should answer only from what the retriever returned.
If the answer is not in the index, return a structured null, for example
{"answer": null, "reason": "not_in_context"}. Enforce that in the prompt and the app logic; do not hope for it.
Optional but worth knowing for production quality: a reranker between retrieval and generation (for example a cross encoder such as bge reranker v2 m3). It rescores the top retrieved chunks for relevance. That improves answer quality for modest extra compute. Not required for a working stack; useful when you care about precision.
Large document sets, policies, medical packets, legal files, bylaws, are exactly where people reach for a public chatbot. That is the failure: the text leaves the room. A local RAG stack keeps the work on the machine and can return page and paragraph citations instead of a confident guess. Searching thousands of pages by hand for one reference can take hours; a good index surfaces the supporting passage in seconds, still under your control.
The system is not inventing policy or medical conclusions. It rearranges what already exists in your files so a human can verify it. That is the bar: retrieval with a paper trail, not improvisation on someone else’s servers.
When someone pastes a restricted plan, medical note, or legal brief into a consumer chat product, that text leaves their machine and is processed under the provider’s terms, which may include retention, logging, or human review depending on the product and tier. Enterprise contracts and “do not train” flags help; they are not the same as keeping the file on a box you own. Staff do this constantly under deadline pressure. The risk is operational, not theoretical.
Most sensitive data rules boil down to the same idea: only authorized people and systems should process the material. An unapproved public chat tool is usually neither. Finding that habit later means rework, awkward explanations, and more process than standing up a local stack would have cost in the first place.
Locally hosted RAG is not paranoia. True air gap (no network on the box) is stronger still. Either way, the point is the same: document processing stays inside a boundary you can actually inspect.
Want to talk through an offline or citation first stack? No slide theater. Practical engineering.