Self Hosted

Deepgram Self-Hosted

Deepgram ships its inference stack as container images you run on your own NVIDIA GPUs. Audio never leaves your network; only small license-verification messages do. This skill decides whether you need that, gets you licensed, and routes to the deployment target.

Self-hosting is gated on an Enterprise agreement. Read Decide before you deploy first — a large share of “we need on-prem” requirements are actually data-residency requirements that a regional endpoint already satisfies, with none of the GPU operations.

Decide before you deploy

Read down the table and take the first row whose requirement you actually have. If none of them apply, the hosted API is the answer.

People reach for the top row first. Most requirements are satisfied by a row further down.

The three regional endpoints serve Speech-to-Text, Text-to-Speech, Voice Agent, and Text Intelligence. Confirm the current limitations at Regional Endpoints (opens in a new tab) before you promise a region to a customer.

Self-hosting is the most expensive row in engineering time. Choose it for a real network, sovereignty, or air-gap constraint — not for latency, which regional and Dedicated endpoints address without handing you a GPU fleet.

Bootstrap licensing and credentials first

Nothing runs without both of these, and they are different things people routinely conflate.

Steps:

1. Confirm project access. Your Console project needs a Self-Hosted tab. No tab means the project was never granted self-hosted products, or has managed access but not self-service. Both are resolved by Deepgram — contact sales (opens in a new tab) or your account representative. You cannot self-serve past this.

2. Create a self-hosted API key in Console (opens in a new tab). Expand its details to see which self-hosted products it may license. All self-hosted customers get API and Engine; the License Proxy must be enabled for your project by Deepgram Support.

3. Create distribution credentials in the Console Self-Hosted tab. The secret is displayed once — record it immediately. Verify from the CLI:

   bashCopy

   # List existing credential sets (read-only, safe)
   curl -s -H "Authorization: Token $DEEPGRAM_API_KEY" \
     "https://api.deepgram.com/v1/projects/$DEEPGRAM_PROJECT_ID/self-hosted/distribution/credentials"

   Returns 200 with a distribution_credentials array. Note the hyphen in self-hosted — selfhosted returns 404.

   A 200 here does not mean your project has self-hosted access. On a project with no self-hosted entitlement the endpoint still returns 200 with an empty array rather than a 403. The Console Self-Hosted tab in step 1 is the real entitlement check; an empty list proves nothing either way.

   The POST variant takes a scopes query parameter, which accepts self-hosted:products (the default, meaning all) or per-product scopes: self-hosted:product:api, :engine, :license-proxy, :dgtools, :billing, :hotpepper, :metrics-server. Only api, engine, license-proxy, and billing map to containers documented publicly; the rest are undocumented, so scope credentials to what you actually deploy. provider accepts quay.

4. Log in to Quay on every deployment host: docker login quay.io (or podman login).

5. Get the model files. Deepgram delivers encrypted .dg model files as download links from your account representative. They are not in any public repository and are not on Quay. See Gated by a human.

6. Verify runtime licensing. Containers hold an outbound HTTPS connection to license.deepgram.com:443 for licensing and usage reporting. That connection uses mTLS, so probing it with curl or an SSL scanner produces spurious errors — that is expected, not a fault. A 401 in container logs means the API key lacks self-hosted permissions; a timeout means your firewall is blocking the egress.

To rotate or revoke credentials, use the four endpoints documented in the api skill’s references/self-hosted.md.

What actually runs

Images are tagged by release date, for example release-260915 — the tag carried across the Docker, Podman, and Helm templates and the Helm chart’s appVersion. Check the templates for the current one. Pin an explicit tag; never rely on a floating one.

Minimum viable deployment: one API container plus one Engine container on one GPU host, with api.toml, engine.toml, and at least one .dg model. The License Proxy and Billing containers are additive.

Default ports, from the template TOML files: API 8080; Engine 8080 for API-to-Engine traffic and 9991 for its metrics server; License Proxy 8443 for license verification and 8080 for its /v1/status health route.

Deepgram does not terminate TLS for you. Put your own proxy (NGINX, HAProxy, Apache) in front of the API container.

Pick a deployment target

Official templates for the first two live in deepgram/self-hosted-resources (opens in a new tab) (public): docker/, podman/, charts/deepgram-self-hosted/, common/ (TOML configs), plus benchmarking/ (k6 load scripts), monitoring/ (Grafana dashboards, Prometheus alert rules), and diagnostics/ (log parser, NVIDIA setup validator).

Feature parity is opt-in, not automatic

Self-hosted is not a mirror of the hosted API. Newer models exist self-hosted but are off by default, need a minimum image release, need a model file you request by hand, and — critically — need their own dedicated Engine.

Consequences worth stating out loud before an evaluation:

• Flux STT allocates all available GPU memory on Engine startup. Send a Nova-3 request to that same GPU and you get CUDA out-of-memory errors. One Flux model per Engine; separate infrastructure for Flux.
• Flux TTS needs one GPU per Engine and allocates up to 60 GB of system RAM while loading, so the host needs at least 64 GB. Extra GPUs do not increase one Engine’s capacity — run one Engine per GPU.
• Voice Agent self-hosted is documented for Kubernetes/Helm only. There is no Compose sample. The agent orchestrates your own LLM provider keys, or an in-cluster LLM (an NVIDIA NIM sample exists in the chart).
• Do not mix STT and TTS on one node. Deepgram recommends against it explicitly; it causes resource contention and unpredictable latency.
• Whisper is not a self-hosted product. “Deepgram Whisper Cloud” is a hosted managed API. Nothing in the self-hosted documentation set offers a Whisper model file. Do not plan an on-prem Whisper deployment without confirming with Deepgram.

Audio intelligence features (entity detection, redaction, NER formatting) depend on separate models being present, and are toggled under [features] in api.toml. Presence of the feature flag is not presence of the model.

Gated by a human

Be honest with anyone planning a timeline. These cannot be self-served and are not in public documentation:

• Project access to self-hosted products — Enterprise agreement, via sales.
• Every .dg model file — links from your account representative. This is the hard blocker: you can pull images and write configs without one, and still serve nothing.
• [flux] max_streams — the concurrency limit per GPU. There is no published per-GPU table; the doc says to ask your account representative. Leaving it auto-calculated causes agents to hang, dropped calls, and audio_window_end increased by more than 3 frames in API logs.
• [flux_tts] max_batch_size — no safe default exists; Engine refuses to start while it is 0, which is what the shipped templates set. The right value differs substantially per GPU and comes from your account representative.
• The Flux TTS model uuid — partly gated. The Helm chart and the docs page both use an empty placeholder, but the two shipped Compose templates (common/*/engine.flux-tts.toml) hardcode a real UUID, so a Compose user who downloads the template already has a working value. The template comment still says to obtain it from your account representative — confirm the UUID matches the release you are deploying rather than assuming the checked-in one is current.
• License Proxy entitlement — via Support.
• Air-gapped license file — a one-line JSON file issued by Deepgram.
• Pricing — self-hosted is a sales conversation. No figure belongs in a skill; start at deepgram.com/pricing (opens in a new tab) and contact us (opens in a new tab).

Hardware sizing beyond the published minimums is also a conversation: the docs repeatedly direct you to Support for a customized recommendation.

Common mistakes

1. Self-hosting to satisfy data residency that a regional endpoint already covers. Check the EU, AU, and IN endpoints, and Deepgram Dedicated, before provisioning a single GPU.
2. Confusing the self-hosted API key with distribution credentials. The API key licenses running containers. The Quay credentials pull images. Swapping them gives you either a failed docker login or containers that pull and then die on a licensing 401.
3. sudo docker compose up without preserving the key. The templates interpolate ${DEEPGRAM_API_KEY}, and sudo drops it. Use sudo --preserve-env=DEEPGRAM_API_KEY docker compose up -d.
4. A proprietary NVIDIA driver, or one older than 580. Engine is built against CUDA 13, which needs driver >=580 in the open kernel-module flavor. On Blackwell the proprietary 580 branch does not support the GPU at all, so the card is invisible to the host.
5. Running Engine on a T4. Nova and Aura work; Flux STT does not (needs Ampere or newer), and Flux TTS supports neither T4 nor A10.
6. Expecting MIG or a fractional GPU to work. Deepgram requires dedicated GPUs. MIG and fractional GPUs are unsupported.
7. Assuming one big GPU replaces two for Aura TTS. Aura and Aura-2 require exactly two dedicated GPUs per TTS Engine container. It is a device-count requirement, not a memory one.
8. Enabling Flux next to other models. The GPU-memory allocation is exclusive; you get CUDA OOM, not a graceful fallback.
9. Omitting NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES. Without them the nvidia-container-runtime injects no CUDA libraries and Engine dies with libcuda.so.1: cannot open shared object file. On GKE with Container-Optimized OS the equivalent failure needs LD_LIBRARY_PATH instead — chart 0.41.1+ sets it for you.
10. Tearing down the only License Proxy during an upgrade. A new instance must reach license.deepgram.com to start. If it cannot and the old one is already gone, every container in the environment fails to license and shuts down. Use blue-green.
11. Treating the mTLS license connection as broken because curl or an SSL scanner errors against license.deepgram.com. That is correct behavior.
12. Deleting the billing journal volume in an air-gapped deployment. It holds usage data you are contractually required to return. Losing it can suspend service.
13. Upgrading API before Engine on a TTS deployment. release-260115 changed API-to-Engine communication and is not backwards compatible for TTS traffic. The new Engine (3.107.0-1) is compatible with previous API versions, so it must be running before the updated API (1.176.0) serves requests — Engine first, then API. STT-only deployments are unaffected. Read the January 15, 2026 changelog (opens in a new tab) before any TTS upgrade; each self-hosted release has its own entry with its own ordering and minimum-driver requirements.

Use a different skill when

• You want the four distribution-credential endpoints in full — request bodies, scopes, responses: api skill, file references/self-hosted.md.
• You want any hosted API parameter, message shape, or response schema: api skill. Self-hosted serves the same contracts on your own host.
• You are choosing a model or getting a first request working: speech-to-text, text-to-speech, or voice-agent skill.
• You are deploying on SageMaker and want scripted subscribe, deploy, autoscale, and teardown: the dedicated deepgram-sagemaker skill in deepgram-devs/dg-sagemaker (opens in a new tab) (public). This skill’s sagemaker.md (opens in a new tab) covers the decision and the SDK transports; that skill covers the AWS mechanics.
• You want a runnable demo app to point at your deployment: starters skill.
• You want the documentation page for a topic: docs skill.
• You want the docs queryable from your editor: setup-mcp skill.

Sources

Append .md to any developers.deepgram.com page for clean Markdown. llms.txt (opens in a new tab) is the full documentation index and lists the self-hosted pages, which run to several dozen — more than this skill cites.

• Introduction: https://developers.deepgram.com/docs/self-hosted-introduction (opens in a new tab)
• Deployment environments, GPU matrix, hardware minimums, firewall: https://developers.deepgram.com/docs/self-hosted-deployment-environments (opens in a new tab)
• Licensing and credentials: https://developers.deepgram.com/docs/self-hosted-self-service-tutorial (opens in a new tab)
• Deploy STT services: https://developers.deepgram.com/docs/deploy-stt-services (opens in a new tab)
• Deploy TTS services: https://developers.deepgram.com/docs/deploy-tts-services (opens in a new tab)
• Flux STT self-hosted: https://developers.deepgram.com/docs/flux-self-hosted (opens in a new tab)
• Flux TTS self-hosted: https://developers.deepgram.com/docs/deploy-flux-tts (opens in a new tab)
• Voice Agent self-hosted: https://developers.deepgram.com/docs/deploy-voice-agent (opens in a new tab)
• License Proxy: https://developers.deepgram.com/docs/license-proxy (opens in a new tab)
• Add-ons (License Proxy, SIPREC, UniMRCP): https://developers.deepgram.com/docs/self-hosted-add-ons (opens in a new tab)
• Blue-green deployment: https://developers.deepgram.com/docs/blue-green-deployment (opens in a new tab)
• Maintenance and model selection: https://developers.deepgram.com/docs/maintaining (opens in a new tab)
• Metrics: https://developers.deepgram.com/docs/metrics-guide (opens in a new tab)
• Regional, Dedicated, and self-hosted endpoints: https://developers.deepgram.com/reference/custom-endpoints (opens in a new tab) and https://developers.deepgram.com/reference/regional-endpoints (opens in a new tab)
• Using SDKs with self-hosted: https://developers.deepgram.com/docs/using-sdks-with-self-hosted (opens in a new tab)
• Official templates: https://github.com/deepgram/self-hosted-resources (opens in a new tab)