Local AI for Small Business: Run AI on Your Own Hardware, Keep Your Data in the Building

First, the disambiguation, because this exact phrase points at two completely different things. Search "local AI for small business" and you'll mostly find guides to AI marketing tools for local businesses — chatbots for your website, ad copy, review replies. This article is about the other meaning: running an AI model on a computer your office owns, so your data stays on your own hardware instead of travelling to a provider's servers. ("AI" here means the model — the text-generating system — not a law degree, which shares the "LLM" abbreviation and clutters these searches too.) If you run a small clinic, law firm, or accounting practice and you've wondered whether you can get ChatGPT-style help without your client files leaving the building, this is the page for that.

The short answer: yes, for a specific and growing set of jobs — and no, not for everything. A 3-to-20-person office can run a private, ChatGPT-like assistant on one ordinary machine, using free software, for drafting, summarizing, rewriting, document Q&A, and transcription — the bulk of daily AI work. What it can't do yet is match the largest cloud "frontier" models on the hardest reasoning tasks; those live in data centres. So the real decision isn't local-versus-cloud in the abstract. It's which of your tasks touch confidential material (route those to a local model) and which don't (a cloud tier is often the better tool). Everything below is that decision, with each factual claim linked to a primary source fetched this week.

What Can a Small Office Actually Do With Local AI Today?

Skip the hype and look at the jobs that work reliably right now on modest hardware.

Drafting and rewriting. Routine emails, client letters, first-draft memos, tightening or reformatting text you already wrote. Mid-sized open models are strong here. A model like Llama 3.2 is a 2.0 GB download in its default 3B size (a 1B version is just 1.3 GB), with a 128K context window — enough to hold a long document in view at once — and it runs on an ordinary office desktop or laptop.

Summarizing. Long threads, contracts, transcripts, reports condensed to the parts a busy person needs. This is one of the most dependable local-AI tasks because it works with material you provide rather than requiring the model to know obscure facts.

Answering questions about your own documents. Point the tool at files you supply and ask questions of them. Because the documents stay on your machine, this is where "keep your data in the building" pays off most directly.

Transcription — fully local, and free. Turning recorded meetings, intake calls, or dictation into text is a genuine local use case, not a maybe. OpenAI's Whisper is open-source under the MIT License and runs on your own machine with no cloud API calls; model sizes run from tiny (39M parameters, about 1 GB of video memory) up to large (1550M, about 10 GB), so even a modest computer can handle it. For a clinic dictating notes or a firm transcribing depositions, the audio never leaves the office.

The software to run all of this is free. Ollama and LM Studio are both no-cost desktop tools for running open models locally — LM Studio describes itself in two words, "Natively local," and Ollama states plainly that it can "run entirely offline for mission critical work" and that "your data is never trained on." The mental model is covered in What Is Local AI?

What Can't Local AI Do Yet?

The honest limits matter as much as the strengths.

It won't match the frontier on hard reasoning. The largest, most capable models are cloud-only. A local model a step down is fine for drafting a letter and unreliable for, say, untangling a genuinely novel legal argument. Know which of your tasks are which.

More capability costs real hardware. The small models above run anywhere; the capable open models need a serious machine. gpt-oss, for example, is a 14 GB download at 20B parameters and 65 GB at 120B. That's the jump that separates "runs on the office laptop" from "needs a workstation or a shared office server with a lot of memory" — and it's the ceiling that keeps the very top tier in the cloud.

It doesn't browse the live web or plug into everything by default. A local model answers from what it was trained on plus what you give it. Real-time lookups and deep integrations are extra setup, not out-of-the-box.

It's your machine to run. No vendor handles updates, backups, or access control — you do. That's the trade for having no vendor at all.

What Does a Realistic Starter Setup Look Like?

For most small offices, "going local" is one machine, free software, and a decision about what runs on it. Here's how the tiers line up.

Setup	Hardware	Software	Good for
Try-it-out	An existing office laptop or desktop	Ollama or LM Studio, free; a small model like Llama 3.2 3B (2.0 GB)	Testing drafting/summarizing on non-critical text before spending anything
Office starter	A dedicated small desktop with extra memory — e.g. an M4 Mac mini, which starts around $799 for the base M4 (16 GB memory) and about $1,599 for the M4 Pro headroom (Apple's store — prices as of July 2026, re-check current pricing)	Same free tools + Whisper for transcription	A 3-to-10-person office running daily drafting, summarizing, and transcription on confidential material
Shared workhorse	A workstation or small server with lots of memory to run a larger model like gpt-oss 20B (14 GB)	Same free tools, model served to the whole office over the network	10-to-20 people, heavier document Q&A, more capable answers

Notice the software column never has a subscription. Ollama's own pricing page confirms it: the downloadable software is free ($0), and "running models on your own hardware is always unlimited," at every tier. It sells optional paid cloud models ($20/month Pro, $100/month Max) — but the moment you use those, your text is going to their servers, and you're back in the cloud column. The test for whether a setup is truly local is simple: if it still works with the internet unplugged, it's local. For the full cost picture, including the cases where local is not cheaper, see How Much Does Local AI Cost? and the cost worksheet.

When Is a Cloud Tier the Right Answer Instead?

Often — and pretending otherwise would be dishonest. If the work involves nothing confidential (marketing drafts, public research, rewriting your own website copy, brainstorming), a cloud assistant is more capable, cheaper to start, and maintained by someone else. There's no custody problem because nothing sensitive is in custody. If you need the strongest available model for a genuinely hard problem, that model only exists in a data centre. And if your office runs a paid business tier with a signed contract governing how the vendor handles your data, and someone has actually read those terms, the standard cloud-plus-contracts route is a legitimate way to meet many obligations.

The failure mode isn't "using cloud AI." It's unsorted use — confidential and harmless material flowing through the same consumer chat window with nobody deciding which is which. Practitioners recommend the local route precisely for the sensitive pile: as one trial lawyer put it on r/LawFirm, "sharing that information with ChatGPT (OpenAI), is not a good idea. Using your own self-hosted language model would be better." An Ask HN thread framed local models the same way — "a perfect fit for privacy-sensitive work: no data leaves the machine" — naming law firms and offices where IT bans cloud AI. If your question is specifically about health data, Is ChatGPT HIPAA Compliant? walks through what each plan does and doesn't allow. And Local AI vs Cloud AI traces exactly where your text goes in each model.

Does Going Local Remove Your Obligations?

No — and this is the part vendors of every kind gloss over. Running the model yourself removes the third party: there's no outside company holding your text, so there are no vendor terms, training defaults, or retention policies to evaluate. That collapses the hardest part of compliance. But it doesn't erase your own duties. Canada's federal privacy regulator states that generative AI tools "do not occupy a space outside of current legislative frameworks," and that organizations using them must comply with applicable privacy laws (OPC principles for privacy-protective generative AI) — a point that holds wherever your building is. What remains is the part your office already knows how to do: control who can use the machine, be sensible with the outputs, keep backups, and manage access. Legal process aimed at you still reaches your own hardware, as it always has. Local AI changes who else holds your data, not whether you're accountable for it.

Or Keep the Sensitive Work on Hardware You Own

Here's the honest close. Going local costs real money up front, hands you a machine to maintain, and runs models a step behind the frontier. For a solo user or a two-person shop doing mostly non-confidential work, a cloud subscription is usually the cheaper, simpler choice, and that's fine. But if a meaningful share of your daily work touches client files, medical records, or financial data, the trade tilts: one machine, free software, no per-seat fees, and — for the confidential pile — the question "what is the vendor doing with our data?" stops existing, because there is no vendor. Whether that trade is worth it for your office depends on how much of your work is sensitive and how many people would use it, which is exactly what the readiness quiz estimates in about two minutes.