LocalAI on GCP: Self-Hosted OpenAI Alternative

An OpenAI API Inside Your Own Project

Most teams on Google Cloud already have an OpenAI key sitting in Secret Manager somewhere. It works. It's also a per-token bill that moves every month, and every prompt your app sends (the support ticket, the contract clause, the call transcript) leaves your project to be processed by someone else.

That's usually fine until it isn't. Then the question becomes: can we keep the code and move the model?

LocalAI on GCP is that move. It answers the same /v1 API your OpenAI SDK already calls, from a VM in your own project. Meetrix's image deploys it from GCP Marketplace in one form: LocalAI 4.12.1 in Docker on Ubuntu 26.04, Nginx in front, Let's Encrypt SSL, an API key on the web UI and every route, and a chat model that answers on first sign-in. CPU or NVIDIA L4, same image. You can launch LocalAI on GCP Marketplace and have it answering in about ten minutes.

What Is LocalAI?

LocalAI is an open-source, MIT-licensed engine that runs AI models on hardware you control. It puts several inference backends (llama.cpp is the one you'll use most) behind one OpenAI-compatible API, so a single server does chat, embeddings, image generation, speech-to-text, and text-to-speech. It also speaks the Anthropic and ElevenLabs API formats, and the project now ships 60+ swappable backends.

The compatibility is what makes it an OpenAI alternative rather than one more inference server. SDKs, chat front ends, and agent frameworks that call the OpenAI API keep working when you change the base URL and the key. Our explainer on running a self-hosted OpenAI-compatible API covers the pattern in general. LocalAI is the option that covers the most model types from one endpoint.

What's New in LocalAI 4.12

The GCP image ships LocalAI 4.12.1, one release line ahead of our AWS AMI (4.11.0). 4.12.0 came out on October 22, 2026, and 4.12.1 the next day. As of early November it's the latest release. Here's what changed that you'll actually notice.

A redesigned web UI

The interface was rebuilt around a shared visual kit, with light and dark themes and clearer Models and Studio workflows. If you've used an older LocalAI, the menu is different: Models, Chat, Studio, and Talk for using models, Build for importing your own, and Operate for activity and system status.

Streaming voice responses

LLM output can now stream straight into Qwen TTS as it's generated, so a voice reply starts speaking before the full answer is written. Other engines get explicit clause streaming. 4.12 also bundles Parakeet audio models for speech recognition, and adds configurable VAD. This is the change that makes a self-hosted voice assistant feel less like a walkie-talkie.

Native decision routing

A /v1/decisions adapter routes predicates, choices, and scores through the existing SystemOne pipeline, so you get structured choices instead of prose your code has to parse. Useful for routing a ticket, picking a category, or deciding which tool an agent should call next.

Privacy and tool settings

The Responses API gained PII filtering and opt-in rejection of uninspectable media, plus protected terms and adjacent-word masking. Remote MCP servers can authenticate with OAuth2 client credentials. Qdrant stores persist across restarts, with TLS and environment-backed credentials.

Live motion capture

GEM-X streams RGB frames into a bounded, session-based pose API on CPU or Vulkan, returning SOMA-77 or SMPL-24 poses. It's API-only for now, with no UI.

Inference controls

MoE expert caching, lazy tensor reads, and explicit slot prompt-cache save and restore. Mostly relevant once you run larger mixture-of-experts models on the L4 and want to squeeze memory.

4.12.1 is a security fix

The point release hardens the external URL policy for user-supplied URLs: redirect targets are now validated on every hop, 100.64.0.0/10 and 0.0.0.0/8 are treated as non-public, and the connection goes to the address that was actually validated. On a cloud VM, where the metadata server answers on a link-local address, that's exactly the kind of fix you want already applied. The full list is in the LocalAI release notes.

One deprecation to note: the separate /reasoning action is gone. Use the per-chat reasoning_effort request instead.

CPU or NVIDIA L4?

This is the one decision worth thinking about before you click Deploy. The image checks for an NVIDIA GPU at every boot and starts the CUDA build if it finds one, the CPU build if not. So it's also a decision you can change later.

Compute prices are Google's on-demand list rates for us-central1, before the Meetrix licence fee, disk, and any committed-use discount. Check Google's GPU pricing for your region.

Six or seven tokens a second is readable. It's fine for a background job that summarises tickets overnight, and frustrating for a person waiting on a chat reply. Ten times that on the L4 is the difference people feel.

The L4 is also why GCP is a good home for LocalAI specifically. Its 24 GB of GDDR6 fits quantized 14B models, where the 16 GB T4 on AWS's g4dn.xlarge tops out around 8B. Our breakdown of GPU costs for LLM inference on AWS and GCP runs the monthly numbers side by side.

My advice is still to start on CPU. Deploy on e2-standard-4, connect one real feature, and file the L4 quota request on the same day, because it's the step that most often holds a GPU deployment up. Every g2 type carries exactly one L4, so a bigger g2 buys you vCPUs and RAM, not more GPU.

What the Meetrix Image Adds

You could run the LocalAI container on any VM. What you'd then spend the afternoon on is everything around it.

Billing runs through your Google Cloud account, like the rest of the Compute Engine bill. The licence fee lands on the same invoice as your VMs, so there's no new vendor to get approved. New subscriptions get a 5-day free trial with up to USD 50 in licence fee credits. Compute and disk charges still apply during the trial.

Deploy LocalAI on GCP in Five Steps

1. Request L4 quota (GPU only)Under IAM & Admin, Quotas, ask for NVIDIA L4 GPUs in your region, and GPUs (all regions) if that's also zero. Google's quota documentation explains the request flow.
2. Open the listingGo to LocalAI on GCP Marketplace, click Launch, pick your project, and accept the terms.
3. Fill in the formUse the table below. Tick the trial checkbox if you want it, then click Deploy.
4. Point your domainCopy the VM's external IP and create a DNS A record for your domain. Then SSH in and run the certificate script if SSL wasn't issued at boot.
5. Sign inOpen your domain (or the site_url output over plain HTTP until SSL is ready) and enter your API key.

Deployment form fields

One thing that catches people out: with an ephemeral IP, your DNS record can't exist before the VM does, so the certificate request at first boot usually fails. That's expected. Once the A record resolves, run the script:

gcloud compute ssh <deployment-name>-vm --zone=<zone>
sudo bash /root/certificate_generate_standalone.sh

Reserve a static IP and point DNS at it before you deploy, and SSL issues itself. If you get a 502 Bad Gateway right after deploying, LocalAI is still starting. Give it a few minutes before you debug anything. The LocalAI on GCP developer guide has every screen of the form, plus logs, the nvidia-smi check, and troubleshooting.

Your First API Call

The deployment outputs include api_base_url, which is your domain followed by /v1. Send the key as a bearer token and list the installed models:

curl https://ai.yourcompany.com/v1/models \
  -H "Authorization: Bearer <your API key>"

Then switch your existing code over. With the OpenAI Python SDK, three values change and nothing else does:

from openai import OpenAI

client = OpenAI(
    base_url="https://ai.yourcompany.com/v1",
    api_key="<your API key>",
)

response = client.chat.completions.create(
    model="qwen3-4b-instruct",
    messages=[{"role": "user", "content": "Classify this ticket: billing, bug or feature request?"}],
)
print(response.choices[0].message.content)

The first request after a reboot takes 15 to 25 seconds while the model loads. After that it's the speeds in the table above. Every route, including embeddings, images, and audio, is documented at /swagger/index.html on your domain.

To add models, open Models in the web UI, search the gallery and click Install. On CPU, pick something whose file is at most half your RAM, and stay away from reasoning models like qwen3-4b, which think step by step and can take minutes per reply. On the L4, quantized 7B to 14B chat models fit.

LocalAI also slots into other self-hosted pieces on GCP. Use it for both embeddings and answers in a RAG pipeline over a Milvus vector database on GCP, or put it behind a 9Router LLM gateway next to a hosted provider, so each request goes to the cheapest model that can handle it.

Running It Safely

LocalAI vs vLLM vs Ollama

All three serve open models behind an OpenAI-style API. They're good at different things.

If you're serving one text model to a lot of concurrent users, vLLM gets more out of the L4, and our vLLM inference server on GCP is the better pick. If you want chat, embeddings, images and speech from one keyed endpoint with a UI for managing models, that's LocalAI. The Ollama vs vLLM comparison goes deeper on the serving-engine side.

Marketplace Image, Docker, or Vertex AI

Plenty of teams run both. The hosted model takes the handful of tasks that need the biggest model, and LocalAI takes the private, high-volume, and good-enough-at-8B work. Because both speak the same API, moving a call between them is a base-URL change.

Resources

LocalAI on GCP - Developer Guide

Step-by-step deployment on Google Cloud: choosing a CPU or NVIDIA L4 machine type, launching the image from GCP Marketplace, pointing DNS, installing models, calling the API, generating SSL, and checking logs and the GPU.

Meetrix.IO | Binuka Ranatunga

Running on AWS too? LocalAI on AWS covers the CloudFormation version, the g4dn sizing and what landed in 4.11. Specs and launch options for both clouds are on the LocalAI store page.

Is It Worth It on GCP?

If your prompts shouldn't leave your project, or your volume is steady enough that an hourly VM beats a per-token bill, yes. You keep your OpenAI code, get text, image and audio models behind one key, and can prove the idea on a $0.13-an-hour CPU VM before you pay for an L4.

If what you need is the best possible answer to hard questions, no. Open models on one GPU are good, not frontier. Sort your tasks into those two piles first, and move the first pile.