> Source: https://meetrix.io/blogs/localai-aws-marketplace/ > Markdown copy of that page. Cite the URL above, not this file. LLMs & APIs # LocalAI on AWS: Self-Hosted OpenAI-Compatible API [By Shalomi Umeshika](https://meetrix.io/blogs/authors/shalomi-umeshika/) • October 28, 2026 • 14 min read ## Every OpenAI Call Sends Your Data Somewhere Else The OpenAI API is the easy default. It's also a per-token bill that grows with every feature you ship, and every prompt, document, and transcript you send leaves your network. For a lot of internal tools, that's fine. For a support bot reading customer tickets, or a pipeline embedding your contracts, it's the part people start asking questions about. **LocalAI on AWS** gives you the same API shape on a server you own. Your code keeps the OpenAI SDK; you change the base URL and the key. Meetrix's AMI sets it up in one CloudFormation stack: LocalAI 4.11.0 on Ubuntu 26.04, a chat model ready to answer, an API key on every route, Nginx, and a Let's Encrypt certificate, on a CPU or an NVIDIA GPU instance. You can [launch the Meetrix LocalAI AMI on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-25t7st4jas56g). The Quick Verdict Self-host fits Prompts can't leave your account Tickets, contracts, transcripts: the model runs on your instance, in the region you pick, and nothing goes to a model provider. Self-host fits Steady, predictable usage Embeddings, classification, internal chat. An hourly instance doesn't care how many tokens you push through it. Stay on a hosted API You need frontier-model quality A 4B or 8B open model on one GPU won't match the largest hosted models on hard reasoning. Don't pretend it will. Start on CPU Just evaluating Launch on t3a.xlarge, try the preloaded model and your own prompts, and only move to a GPU once you know what you need. ## What Is LocalAI? [LocalAI](https://localai.io/) is an open-source, MIT-licensed engine for running AI models on your own hardware. It wraps several inference backends (llama.cpp among them) behind one OpenAI-compatible API, so the same server can do chat, embeddings, image generation, speech-to-text, and text-to-speech. The source and release notes are on [GitHub](https://github.com/mudler/LocalAI). That compatibility is the whole point. Most tools that speak to the OpenAI API (SDKs, chat front ends, agent frameworks) work against LocalAI by changing the base URL. It also speaks the Anthropic and ElevenLabs APIs, so a wider set of existing clients keeps working with only a base-URL change. Our guide to running a [self-hosted OpenAI-compatible API](https://meetrix.io/blogs/openai-compatible-api-self-hosted/) covers the pattern in general; LocalAI is the option that covers the most model types with one server. The project has around 50,000 GitHub stars as of late October 2026. That's not a measure of quality, but it tells you the community is active and the answer to most questions already exists somewhere. ### What it does Text - Chat and completions on the `/v1` endpoints your SDK already calls - Embeddings for search and RAG pipelines - Tool calling with models that support it Images and audio - Image generation and editing - Speech-to-text and text-to-speech Models - A gallery of 1,926 models as of 4.11.0, installed from the web UI in one click - Import your own GGUF files or pull from Hugging Face - Load and stop models from the UI, and watch RAM and GPU memory as you go Operations - A web UI for chat, model management, and system status - Interactive API docs at `/swagger` - API key authentication on the UI and every API route Agents - Built-in autonomous agents with tool use, RAG, MCP support, and skills ## What's New in LocalAI 4.11 (and 4.12) LocalAI ships fast. 4.11.0 came out on October 2, 2026, with 243 merged pull requests, 353 commits, and 79 new gallery entries in a 15-day window. It's the version on the AMI. 4.12.0 arrived on October 22, followed by 4.12.1 on October 23. The 4.12 line redesigns the web UI and adds streaming voice responses, native decision routing, and image upscaling. The AMI runs 4.11.0 until the next image is published, so check the [release notes](https://github.com/mudler/LocalAI/releases) if you need something from 4.12. Our [LocalAI image on Google Cloud](https://meetrix.io/blogs/localai-openai-alternative-gcp/) already runs 4.12.1. Here's what landed in 4.11 that matters. ### Audio scenes and remembered speakers LocalAI can now treat audio as a scene instead of a transcript alone. One model can run speech recognition, speaker diarization, and sound-event classification in a single pass, returning aligned speaker and sound information for both regular and realtime requests. Studio adds a Diarization page where you preview clean speaker intervals, name a voice, and register it. When the same voice appears in a later recording, LocalAI can show the saved name instead of an anonymous label. Speaker profiles are biometric data, so this is opt-in and requires voice-recognition permission. The registry is currently process-local and ephemeral, which means names disappear after a restart and aren't synchronized across independent frontends. ### Model failover chains You can give a request an ordered list of targets, local or remote, and LocalAI tries the next one if the first fails before any response is committed. Transport errors, server errors, OOM, and rate limits can trip a target; validation errors, ordinary client errors, and cancellation do not. Responses report which target served them through `X-LocalAI-Served-Model` and `X-LocalAI-Failover` headers. There's a `GET /api/failover` endpoint and an SSE event stream, plus admin pin and unpin controls and MCP tools for the same operations. The new `localai-proxy` backend lets a chain include models served by another LocalAI endpoint. ### Signed OCI model galleries Model galleries can now be distributed as self-contained OCI artifacts. LocalAI resolves and verifies the digest, checks the artifact before extraction, enforces layer and path limits, and ties cached content to the verification policy used to accept it. A `source_repository` constraint pins the Sigstore source exactly. ### Decision models and the SystemOne API Decision models declare `known_usecases: [decisions]` and serve structured choice, assessment, and level questions through `POST /v1/systemone`. The request path enforces a 64 KiB body limit, a maximum of 64 questions, and validation for state, IDs, option counts, levels, and criteria before the backend is invoked. This is for cases where you want a structured answer, not prose the caller has to parse. ### Kimodo text-to-animation `POST /3d/animate` accepts a prompt and generates a 60 to 150 frame skeletal animation at 30 FPS, returned as a binary glTF `.glb` file. Studio provides animation controls, preview, and local history. ### Operate this machine A standalone LocalAI install now has its own operations page. Open _Operate_, then _This machine_, to see VRAM, RAM, CPU, models-disk usage, and the processes behind loaded models. Each row shows the backend, resident memory, CPU share, uptime, and PID. You can search and sort, open backend logs, and stop a model after confirmation. ### Authentication changes from 4.9.0 Worth knowing if you're coming from an older install: 4.9.0 made authentication deny-by-default. Every HTTP route requires credentials unless it appears in an explicit public registry. API instructions, Swagger GETs, the well-known document, health, login, OAuth, SPA assets, and node registration flows stay public. Beyond a shared API key, LocalAI also supports user accounts with roles and quotas, OIDC sign-in, and per-key and per-user usage tracking. Once you configure authentication, the HTTP surface is private by default. ### Distributed mode If you outgrow one box, distributed mode scales LocalAI horizontally across machines, using PostgreSQL for state and the node registry and NATS for real-time coordination. It requires authentication enabled with PostgreSQL; SQLite is not supported. Workers self-register with a frontend, and the node registry and job store live in PostgreSQL, so every frontend sees the same state. **The model you run matters more than the engine.** LocalAI serves whatever you install, and the jump from a 4B model to a good 8B model is bigger than anything in a release note. Our roundup of the [best open-source LLMs to self-host](https://meetrix.io/blogs/best-open-source-llms-self-hosted-2026/) is the place to pick one. ## Who Self-Hosted LocalAI Suits The clearest fit is a team that has built on the OpenAI API and now needs some of that traffic to stay in-house, without rewriting the code that calls it. A good fit if you are - Building internal tools on sensitive data (tickets, contracts, medical or financial text) where prompts shouldn't go to a model provider - Running embeddings or classification at volume, where per-token pricing adds up and a small model is good enough - A developer who wants one endpoint for chat, images, and speech instead of three services - Prototyping against the OpenAI API shape before committing to a provider Look elsewhere if you need - The quality of the largest hosted models on hard reasoning or long, complex code - Maximum throughput from one big text model on a GPU. That's vLLM's job - Zero operations. Someone still has to size the instance and pick the models ## CPU or GPU: Pick Before You Launch This is the decision that matters most, and the AMI makes it reversible. The same image detects the hardware at every boot: the CUDA 13 build on an NVIDIA GPU instance, the CPU build otherwise. Change the instance type later and LocalAI follows. | | CPU | GPU | | --- | --- | --- | | Recommended instance | `t3a.xlarge` (4 vCPUs, 16 GB RAM) | `g4dn.xlarge` (4 vCPUs, 16 GB RAM, one NVIDIA T4 with 16 GB) | | Supported families | General purpose, e.g. t3, t3a, m5, m5a | g4dn, g5, g6, and g6e | | Good for | Trying LocalAI, small chat models, embeddings, low-traffic internal tools | 7B to 8B chat models, image generation, and speech at a usable speed | | First boot | About 5 minutes | About 8 minutes | | vCPU quota | Standard On-Demand instances | On-Demand G and VT instances, often 0 on new accounts | The g4dn.xlarge runs about **$0.526 per hour on-demand** in US East, roughly $384 a month before storage. A t3a.xlarge is a fraction of that. My advice: start on CPU. The preloaded 4B model answers on a t3a.xlarge, which is enough to wire up your application and find out what you actually need. Move to a GPU when response times become the complaint. And request the G and VT quota on day one, because a new account usually has none, and the request can take a while. ## Why Run LocalAI on AWS You could run the LocalAI container anywhere with Docker. The AMI covers the parts around it that turn a container into something you can point an application at. One stack, everything wired up CloudFormation creates the VPC, a public subnet, a security group, an Elastic IP, and an IAM role, plus the server with a 60 GiB gp3 disk for models. A key on every route The ApiKey you set protects the web UI login and every API endpoint. A bare LocalAI container you forget to configure is an open model server on the internet. SSL without the fiddling On a Route 53 domain, the Let's Encrypt certificate is issued through DNS validation while the stack builds. Anywhere else, one script on the server does it. Ready to answer Qwen3-4B-Instruct-2507 is preloaded with the llama.cpp CPU and CUDA backends, so the first chat works before you've installed anything. Billing goes through your AWS account, like the rest of our [self-hosted software on AWS](https://meetrix.io/blogs/self-hosted-software-aws/). The software fee is hourly and depends on the instance type, plus your EC2 and storage charges. A 365-day contract saves up to 41%, and new subscriptions get a 5-day free trial. Check the [listing](https://aws.amazon.com/marketplace/pp/prodview-25t7st4jas56g) for the exact per-hour figure, because it changes with the instance size. ## Install LocalAI on AWS in Five Steps 1. Check your vCPU quotaFor a GPU instance, make sure "Running On-Demand G and VT instances" has room for a g4dn. If not, follow our guide to [increase your AWS vCPU quota](https://meetrix.io/blogs/increase-aws-vcpu-quota/) first. 2. Subscribe on AWS MarketplaceSubscribe to [LocalAI on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-25t7st4jas56g), then choose the CloudFormation fulfilment option and Launch CloudFormation. 3. Fill in the stack parametersUse the table below, then tick the box acknowledging IAM resources and wait for `CREATE_COMPLETE`. 4. Point your domainCopy `PublicIp` from the Outputs tab and create a DNS A record for your `DomainName` pointing at it. 5. Sign inOpen `DashboardUrl`, or `DashboardUrlIp` over plain HTTP until DNS and SSL are ready, and enter your `ApiKey`. ### Stack parameters | Parameter | What to enter | Default | | --- | --- | --- | | `DomainName` | The domain for LocalAI, e.g. `ai.yourcompany.com`. SSL is automatic when it's hosted on Route 53. | None | | `AdminEmail` | Your email, used for the SSL certificate | None | | `ApiKey` | The key for the web UI and the API. Use a long random value and treat it like a password. | None | | `InstanceType` | `t3a.xlarge` for CPU, `g4dn.xlarge` for GPU | Your choice | | `KeyName` | An existing EC2 key pair, for SSH access | None | | `SSHLocation` | Which IPs may SSH in. Use your own IP with `/32`. | None | | `DeploymentName` | A label, useful if you run more than one stack | None | | `VpcCidrBlock` / `SubnetCidrBlock` | Network ranges. Keep the defaults. | `10.0.0.0/16` / `10.0.0.0/24` | The [LocalAI developer guide](https://meetrix.io/blogs/localai-developer-guide/) walks through each step with screenshots, plus logs, the GPU check, and troubleshooting. ### If SSL doesn't set itself up This happens when the domain isn't on Route 53, or DNS wasn't pointing at the server when the stack finished. Point the record, wait until it resolves, then SSH in and run the certificate script: ```bash ssh -i your-key.pem ubuntu@ sudo /root/certificate_generate_standalone.sh ``` ### Seeing a 502 Bad Gateway? LocalAI needs about 5 minutes after boot on CPU and about 8 on GPU. Wait and refresh before you start debugging anything. ## Your First API Call Use the `ApiBaseUrl` from the Outputs tab (your domain followed by `/v1`) and send your key as a bearer token. List the installed models first: ```bash curl https://ai.yourcompany.com/v1/models \ -H "Authorization: Bearer " ``` Then point the OpenAI SDK you already use at it. In Python, the only lines that change are the base URL, the key, and the model name: ```python from openai import OpenAI client = OpenAI( base_url="https://ai.yourcompany.com/v1", api_key="", ) response = client.chat.completions.create( model="qwen3-4b-instruct", messages=[{"role": "user", "content": "Summarise this ticket in one sentence."}], ) print(response.choices[0].message.content) ``` The full list of endpoints, with request and response shapes, is at `/swagger` on your domain. To add models, open _Models_ in the web UI, search the gallery, and click _Install_. On t3a.xlarge, stay with small quantized models of a few billion parameters; on a g4dn's 16 GB T4, quantized 7B to 8B chat models fit. Want a chat interface for people who don't write code? Our [Open WebUI developer guide](https://meetrix.io/blogs/open-webui-developer-guide/) sets up a front end that can use LocalAI as its OpenAI-compatible backend. ## Running LocalAI Safely The API key - Anyone with the key can use your models and install new ones. Share it only with the services that need it. - Keep it server-side. Never put it in JavaScript that runs in a browser. - Switch your apps to the https URL once the certificate is issued, so the key isn't sent in plain text. - The deny-by-default change in 4.9.0 means that once you set an API key, unauthenticated reads stop working too. That's a good default, but it's a behavioural break if you had scripts calling `/version` or fetching generated media URLs without credentials. The server - The security group opens HTTP and HTTPS and limits SSH to your `SSHLocation`. Don't leave that at `0.0.0.0/0` for longer than a test. - Models are several gigabytes each. Watch the 60 GiB disk with `df -h` and remove models you don't use. - Stop a GPU instance when nobody needs it. It's the simplest way to cut the bill. Upgrades mean a new stack New LocalAI versions ship as new AMI versions. To upgrade: note the models and settings you use, launch a new stack from the new version, and reinstall your models from the gallery. Read the [release notes](https://github.com/mudler/LocalAI/releases) first if your applications depend on a specific endpoint or backend. ## LocalAI vs Ollama vs vLLM All three serve open models behind an OpenAI-style API, and people mix them up. They're built for different jobs. | | LocalAI | Ollama | vLLM | | --- | --- | --- | --- | | Best at | Many model types behind one API | The simplest local setup | High-throughput text serving on GPUs | | Model types | Text, embeddings, images, speech-to-text, text-to-speech | Mainly text and embeddings | Mainly text and embeddings | | Runs on CPU | Yes | Yes | GPU-focused | | Auth built in | API keys, plus user accounts with roles | No key validation | Optional API key | | Web UI | Included, with a model gallery | Desktop app; no server UI | None | My view: if you're serving one popular text model to a lot of concurrent users, vLLM will get more out of the GPU, and our [vLLM on AWS](https://meetrix.io/blogs/vllm-aws-marketplace/) image is the better buy. If you want one endpoint that handles chat, embeddings, images, and speech, with a key on it and a UI to manage models, that's what LocalAI is for. ## AMI, Docker, or a Hosted API Recommended Meetrix AMI on AWS Marketplace Best for a private API with the least effort CPU or GPU from one image, API key auth, Nginx, SSL, and a preloaded model. Supported by Meetrix. LocalAI in Docker Best for a laptop or your own server The [official images](https://localai.io/) run anywhere Docker does. Picking the CPU or CUDA image, the proxy, SSL, and the API key are yours to handle. A hosted API Best for frontier-model quality No server at all and the strongest models, billed per token, with your prompts leaving your account. Plenty of teams end up with both: a hosted API for the few tasks that need the biggest model, and LocalAI for everything that's private, high-volume, or good enough with an 8B model. Because LocalAI speaks the same API, switching a call between them is a one-line change. ## Resources [ LocalAI - Developer Guide Step-by-step deployment on AWS: choosing CPU or GPU, launching the AMI, creating the CloudFormation stack, pointing DNS, installing models, calling the API, generating SSL, and checking logs and the GPU. Meetrix.IO | Binuka Ranatunga ![LocalAI AWS Developer Guide cover](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-image.png)](https://meetrix.io/blogs/localai-developer-guide/) ## Is Self-Hosting LocalAI Worth It? If your prompts shouldn't leave your account, or your usage is steady enough that an hourly instance beats a per-token bill, yes. You keep your OpenAI code, get text, image, and audio models behind one key, and can start on a cheap CPU instance before you pay for a GPU. It's a weaker deal if what you need is the best possible answer to hard questions. Open models on one GPU are good, not frontier. Know which of your tasks is which before you move them. Building something on top of it? Our guide to a [HIPAA AI voice agent on AWS](https://meetrix.io/blogs/hipaa-ai-voice-agent-aws/) shows the same keep-it-in-your-account approach for voice. Specs, launch options, and the developer guide are also collected on the [LocalAI store page](https://meetrix.io/store/localai/). ## Frequently Asked Questions What is LocalAI? An open-source, MIT-licensed engine that runs AI models on your own hardware behind an OpenAI-compatible API, with Anthropic and ElevenLabs compatibility as well. One server handles chat, embeddings, image generation, speech-to-text, and text-to-speech. Does LocalAI need a GPU? No. It runs on CPU, and the AMI preloads a 4B model that answers on a t3a.xlarge. A GPU makes 7B to 8B models and image generation usable at speed, but it isn't required to start. Can I use the OpenAI SDK with it? Yes. Point the SDK's base\_url at your domain plus /v1, use your ApiKey, and use the model name from /v1/models. Nothing else in your code needs to change. Which model comes preinstalled? Qwen3-4B-Instruct-2507, a small CPU-friendly model that supports tool calling, with both the llama.cpp CPU and CUDA backends available. How much does the Meetrix LocalAI AMI cost? The software fee is hourly and depends on the instance type, plus your EC2 and storage charges. A 365-day contract saves up to 41%, and there's a 5-day free trial. Check the Marketplace listing for the exact figure. How do I sign in? Open DashboardUrl and enter your ApiKey. The same key authenticates every API route, so treat it like a password. LocalAI, Ollama or vLLM? LocalAI for many model types behind one keyed API with a UI. Ollama for the simplest local setup. vLLM for maximum throughput on a GPU with one large text model. Where do I get help? Support goes to the Meetrix team at [aws@meetrix.io](mailto:aws@meetrix.io), and the [developer guide](/blogs/localai-developer-guide/) covers setup, logs, the GPU check, and troubleshooting. ## Ready to Run Your Own OpenAI-Compatible API? Deploy LocalAI on AWS in about ten minutes, on CPU or GPU, with a preloaded model, API key auth, and SSL. Your models, your infrastructure. [Get Started on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-25t7st4jas56g) Meetrix Store LocalAI OpenAI-compatible text, image and audio models, on CPU or GPU [Deploy it](https://meetrix.io/store/localai/) Meetrix Store New Deploy what this guide covers, pre-configured. - [LocalAI OpenAI-compatible text, image and audio models, on CPU or GPU](https://meetrix.io/store/localai/) - [vLLM Serve open models on your own GPU](https://meetrix.io/store/vllm/) - [OpenWebUI A private ChatGPT-style assistant](https://meetrix.io/store/openwebui/) - [Jitsi Meet Self-hosted video calls for 50 to 500 users](https://meetrix.io/store/jitsi-meet/) - [RustDesk Remote desktop AMI, a TeamViewer alternative](https://meetrix.io/store/rustdesk/) - [Coturn TURN/STUN for WebRTC, no per-minute relay fees](https://meetrix.io/store/coturn/) [Browse all products](https://meetrix.io/store/)