Local AI Scribes and Transcription: How Clinics Keep Patient Audio On-Premise

Patient audio is the most sensitive material a clinic or therapy practice holds — a recorded session or exam is far harder to de-identify than a typed note, because a voice is an identifier by itself. So the question underneath "which AI scribe should we use?" is really a custody question: when the recording is made, whose computer does it land on? Most AI-scribe products answer "ours" — they upload the audio to a vendor's cloud, transcribe it there, and store it there. This guide is about the other answer.

The short version: you can run a genuinely capable speech-to-text transcriber on a computer you own, offline, for free. OpenAI's Whisper is a speech-recognition model released under the MIT License that installs and runs on your own machine (Whisper on GitHub), and a companion project called whisper.cpp runs the same class of model on an ordinary Mac or mini-PC with no graphics card and no internet connection (whisper.cpp on GitHub). When the transcriber runs on your hardware with the network off, the patient audio never touches a third party — which means there is no cloud vendor to sign a business associate agreement with, and no vendor server for the recording to sit on. Everything below is the detail behind that.

What Does a Cloud AI Scribe Actually Do With the Recording?

Trace the audio. With a mainstream cloud scribe, the microphone captures the visit, the app sends that audio over the internet to the vendor's servers, the vendor transcribes and summarizes it there, and the recording and transcript are stored there under the vendor's policies.

Two named vendors show what "there" means in practice. Otter.ai stores audio and transcripts in its own cloud — AWS S3 storage with AES-256 server-side encryption — and states that it "uses a proprietary method to de-identify user data before training our models" (Otter Privacy & Security). Read that carefully: Otter does use de-identified customer data to train its own models, and automatically deletes conversations from the trash after 30 days. It's not "your audio is never touched"; it's "your audio is de-identified first." Nabla, a medical scribe, hosts "all our data" on Google Cloud Platform storage in a region you select when you create your organization, encrypted at rest with AES-256 and in transit with TLS (Nabla Security).

None of that is careless — encryption and a chosen region are genuine protections. But notice what none of it changes: the recording is on a third party's machine, under that third party's terms. Even Nabla's "pick your region" still means someone else is custodying the data — the same point our guide on data residency and sovereignty makes for text. Custody didn't move; only the encryption and the map pin did.

Why Does the Cloud Custody Trigger a BAA?

Because US health-privacy law attaches to the fact of a third party holding the data, not to how well they encrypt it. HHS's Office for Civil Rights is explicit: when a covered entity uses a cloud service to "create, receive, maintain, or transmit" electronic patient health information, "the CSP is a business associate under HIPAA," and "the covered entity ... and the CSP must enter into a HIPAA-compliant business associate agreement (BAA)." Critically, that duty holds "even if the CSP processes or stores only encrypted ePHI and lacks an encryption key for the data" (HHS OCR guidance on HIPAA & cloud computing).

In plain terms: the moment your patient audio touches a scribe vendor's servers, that vendor becomes part of your compliance problem — a business associate you have to vet, contract with, and monitor. A signed BAA is the tool that makes that arrangement lawful; a consumer or free tier that won't sign one is simply off the table for patient data. (What a BAA covers, and which vendors will actually sign, is its own subject — see what a BAA is and whether your AI vendor will sign one.)

The duty isn't uniquely American. Ontario's health-privacy law, PHIPA, requires a health information custodian to "take steps that are reasonable in the circumstances to ensure that personal health information in the custodian's custody or control is protected against theft, loss and unauthorized use or disclosure" (PHIPA s.12(1), Ontario e-Laws). Whichever framework governs your building — HIPAA, PHIPA, or a national privacy law elsewhere — the shape is the same: putting patient audio on someone else's server is a disclosure your existing obligations already reach, and it's on you to have done the paperwork.

How Does an On-Device Scribe Sidestep That Entirely?

Not by managing the third party better — by removing it. If the transcription runs on a computer in your office, there is no cloud service provider in the loop, so there is nothing for the HHS BAA rule to attach to and no vendor server for PHIPA's "custody or control" concern to reach. The audio path is microphone to your machine to your disk, and it stops there.

Two things make this practical today rather than a science project. First, the model is free and real: Whisper's "code and model weights are released under the MIT License" (GitHub), and it ships in six model sizes — from tiny (39 million parameters) to large (1.55 billion), including a turbo model — so a smaller, older machine can run a smaller model and a capable desktop can run the large one. Second, you don't need specialized hardware: whisper.cpp is a "plain C/C++ implementation without dependencies," supports "CPU-only inference," and is an "Apple Silicon first-class citizen - optimized via ARM NEON, Accelerate framework, Metal and Core ML" (GitHub). Translated: it runs on an ordinary Mac or mini-PC, with no graphics card and no internet, at useful speed.

This is the local-AI pattern applied to audio specifically — the same idea covered in What Is Local AI? and the reason a properly local tool works with the internet off. It does not make your other duties vanish: you still control who can log into the machine, how recordings are stored and deleted, and your backups. It collapses the hardest part — vetting and contracting a third party — because there is no third party.

Cloud AI Scribe vs. On-Device Transcription: The Honest Trade-Off

	Cloud AI scribe (e.g. Otter, Nabla)	On-device transcription (Whisper on your hardware)
Where the audio is processed	Vendor's servers, over the internet (Otter, Nabla)	Your own machine, in your building (whisper.cpp)
Who custodies the recording	The vendor (AWS S3 / Google Cloud, encrypted)	You — it's your disk
Third-party agreement	A business associate agreement is required under HIPAA (HHS OCR)	None — there is no third party to contract with
Training on your data	Varies by vendor; Otter states it de-identifies user data before training its own models (Otter)	Nothing is sent anywhere when run offline
Software cost	Per-seat subscription	Free — Whisper and whisper.cpp are MIT-licensed
Hardware cost	None beyond a device with the app	A capable machine up front — from $799 for a base M4 Mac mini (Apple Store); $1,599 for M4 Pro headroom
Convenience & note formatting	Higher — mobile apps, EHR integrations, structured notes bundled in	Lower — raw transcription out of the box; you assemble the workflow
Works offline	No	Yes

Two things this table won't pretend. On-device transcription is not automatically cheaper for a solo practitioner — a subscription can win on pure dollars until you weigh the custody benefit, which is what our cost guide for local AI helps you work through. And a do-it-yourself local setup doesn't match the polished, integrated note-writing experience the cloud scribes sell; you're buying custody, not convenience.

What Would an On-Premise Setup Actually Look Like?

Illustration — a fictional practice, showing the pattern.

A two-clinician therapy practice buys one Mac mini (theirs, on the office network), installs a Whisper-based transcription tool, and confirms it runs with the Wi-Fi switched off. Session recordings are transcribed on that machine; the audio and transcript live on its encrypted disk, governed by the same access rules as their paper files. There is no scribe vendor in the picture, so there is no BAA to negotiate for the audio, no data-residency region to choose, and no vendor policy to re-read when it changes. Setting up the software itself is closer to installing an app than to a full IT project — the same on-ramp described in how local models actually run with Ollama.

The honest close: this option costs real money up front, gives you a machine to maintain and back up, and its note-writing polish is a step behind the bundled cloud products. What you get in exchange is that the question "what is our scribe vendor doing with patient audio?" stops existing, because there is no vendor — wherever your building is. If your practice runs a cloud scribe on a properly signed BAA and you've actually read the terms, that's a legitimate route too; for many US clinics it's the standard one, and our guide on whether ChatGPT can be HIPAA-compliant walks through what "compliant cloud" really requires. But when the data in question is raw patient audio — the least de-identifiable thing you hold — keeping the transcription on hardware you own is the cleanest way to make the custody question go away instead of managing it forever.