> Source: https://meetrix.io/blogs/localai-developer-guide/
> Markdown copy of that page. Cite the URL above, not this file.

Development

# LocalAI - Developer Guide

[By Binuka Ranatunga](https://meetrix.io/blogs/authors/binuka-ranatunga/) • October 27, 2026 • 12 min read

Learn how to deploy LocalAI on AWS with our step-by-step guide. It walks you through choosing a CPU or GPU instance, launching the Meetrix LocalAI AMI with CloudFormation, pointing your domain at it, signing in with your API key, installing models, calling the API from your code, and keeping the server healthy once it runs.

[LocalAI](https://localai.io/) is an open-source runtime that runs language, image and audio models on your own hardware and serves them through an OpenAI-compatible API. Tools and SDKs built for the OpenAI API work against it by changing the base URL, which is the same idea behind our guide to running a [self-hosted OpenAI-compatible API on AWS](https://meetrix.io/blogs/openai-compatible-api-self-hosted/). The Meetrix image runs LocalAI 4.11.0 in Docker on Ubuntu 26.04, protects both the web UI and the API with an API key, and comes with the qwen3-4b-instruct model ready to use.

Prerequisites

Before you get started with the LocalAI AMI, ensure you have the following prerequisites:

-   Basic knowledge of AWS services, including EC2 instances and CloudFormation.
-   An active AWS account with appropriate permissions.
-   An EC2 key pair in the region you will deploy to.
-   For a GPU deployment, enough "Running On-Demand G and VT instances" vCPU quota to launch a g4dn instance.
-   If you encounter a vCPU quota error when launching the stack, follow [https://meetrix.io/blogs/increase-aws-vcpu-quota/](https://meetrix.io/blogs/increase-aws-vcpu-quota/) to increase your vCPU limit.

## Choose CPU or GPU

The same LocalAI image runs on both CPU and GPU instances, and you choose between them with the **InstanceType** stack parameter. On a GPU instance LocalAI uses the NVIDIA GPU; on a CPU instance it falls back to its CPU backends.

|  | CPU | GPU |
| --- | --- | --- |
| Recommended instance | t3a.xlarge | g4dn.xlarge |
| Hardware | 4 vCPUs, 16 GB RAM | 4 vCPUs, 16 GB RAM, one NVIDIA T4 with 16 GB of GPU memory |
| Good for | Trying LocalAI, small chat models, embeddings, low-traffic internal tools | Faster responses, 7B to 8B chat models, image generation and speech |
| vCPU quota | Standard On-Demand instances | On-Demand G and VT instances, which is often 0 in new accounts |

Start on a CPU instance if you only want to evaluate LocalAI or run small models, and move to a GPU when response times matter. Our breakdown of [GPU costs for LLM inference on AWS and GCP](https://meetrix.io/blogs/gpu-costs-llm-inference-aws-vs-gcp/) helps you estimate what a GPU instance will cost each month.

## Launching the AMI

### Step 1: Find and Select LocalAI AMI

1.  Log in to your AWS Management Console.
2.  Navigate to '[LocalAI](https://aws.amazon.com/marketplace/pp/prodview-25t7st4jas56g)' in AWS Marketplace.

### Step 2: Initial Setup & Configuration

1.  Click the **"Continue to Subscribe"** button.
2.  After subscribing, accept the terms and click **"Accept Terms"**.
3.  Wait a few minutes until processing completes, then click **"Continue to Configuration"**.
4.  Select **"CloudFormation script to deploy LocalAI"** as the fulfillment option and choose your region. Click **"Continue to Launch"**.
5.  From the "Choose Action" dropdown, select **"Launch CloudFormation"** and click **"Launch"**.

## Create CloudFormation Stack

### Step 1: Create a stack

1.  Ensure the **"Template is ready"** option is selected under "Prepare template".
2.  Click "Next".

### Step 2: Specify stack options

1.  Provide a unique **"Stack name"**.
2.  Enter the **"AdminEmail"** used for SSL certificate generation.
3.  Enter an **"ApiKey"**. You sign in to the LocalAI web UI with this key, and every API request sends it as a bearer token, so use a long random value and store it securely.
4.  Enter a value for **"DeploymentName"**.
5.  Provide a public domain for **"DomainName"**. LocalAI will automatically try to set up SSL if the domain is hosted on Route 53. If unsuccessful, you must set up SSL manually.
6.  Choose an instance type **"InstanceType"** (Recommended: **t3a.xlarge** for CPU or **g4dn.xlarge** for GPU).
7.  Select your preferred **"KeyName"**.
8.  Set **"SSHLocation"** to 0.0.0.0/0, or restrict it to your own IP range.
9.  Keep **"SubnetCidrBlock"** as 10.0.0.0/24.
10.  Keep **"VpcCidrBlock"** as 10.0.0.0/16.
11.  Click "Next".

### Step 3: Configure stack options

1.  Choose "Roll back all stack resources" and "Delete all newly created resources" under "Stack failure options".
2.  Click "Next".

### Step 4: Review

Review and verify the parameters you've entered.

![CloudFormation Parameters list for the LocalAI stack showing AdminEmail, the AmiId for LocalAI v4.11.0, a masked ApiKey, DeploymentName, DomainName, InstanceType set to t3a.xlarge, KeyName, SSHLocation and CIDR blocks](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-01.png)

1.  Tick **"I acknowledge that AWS CloudFormation might create IAM resources with custom names"**.
2.  Click **"Submit"**.

![CloudFormation capabilities notice for IAM Policy and Role resources with the acknowledgement checkbox ticked](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-02.png)

Afterward, you'll be directed to the CloudFormation stacks page. Please wait 5-10 minutes until the stack status shows **CREATE\_COMPLETE**. The **"Outputs"** tab then lists the API base URL, the web UI URLs and the public IP.

![LocalAI CloudFormation stack Outputs tab listing ApiBaseUrl, DashboardUrl, DashboardUrlIp and PublicIp](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-03.png)

## Update DNS

### Step 1: Copy IP Address

Copy the public IP labeled **"PublicIp"** in the **"Outputs"** tab.

![PublicIp Elastic IP value highlighted in the LocalAI stack Outputs tab](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-04.png)

### Step 2: Update DNS

1.  Go to AWS Route 53 and navigate to "Hosted Zones".
2.  Click **Create record**.
3.  Add a **record name** matching the DomainName you entered, keep the record type as **A**, and paste the copied **PublicIp** into the **value** textbox.
4.  Click **"Create records"**.

![Route 53 Quick create record form for a new A record in the hosted zone](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-05.png)

## Access LocalAI

Access LocalAI using the **"DashboardUrl"** in the Outputs tab. Until DNS and SSL are in place you can use **"DashboardUrlIp"** instead, which serves LocalAI over plain HTTP.

Note

If you receive a "502 Bad Gateway" error, wait approximately **5 minutes** and refresh the page. LocalAI may still be initializing.

On the sign-in page, enter the **ApiKey** you set in the stack parameters and click **"Sign In"**.

![LocalAI sign-in page asking for the API key](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-06.png)

After signing in you land on the LocalAI home page. The pre-installed **qwen3-4b-instruct** model is already selected in the chat box, so you can type a message and get a reply straight away. The counters in the top-right corner show how many models are loaded and how much of the instance's RAM they use. The menu on the left opens **Models**, the **Chat**, **Studio** and **Talk** tools, **Build** for importing your own models, and **Operate** for activity and system status.

![LocalAI home page after sign-in with qwen3-4b-instruct selected in the chat box, one model loaded, and the Models, Chat, Studio, Talk, Build and Operate menu](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-07.png)

API key

Anyone with the API key can use your models and install new ones. Share it only with the people and applications that need it, and avoid pasting it into client-side code that runs in a browser.

## Install More Models

LocalAI installs models from its gallery, which covers chat models, embedding models, image generators, and speech models.

1.  Click **"Browse Gallery"** on the home page, or open **Models** in the menu.
2.  Search for a model by name or filter by type, then click **"Install"**.
3.  Follow the download progress at the top of the page, or under **Operate**. Once it finishes, the model appears in the model dropdown in Chat.

Pick models that fit your instance. On t3a.xlarge, stay with small quantized models of a few billion parameters. On g4dn.xlarge, the T4's 16 GB of GPU memory fits quantized 7B to 8B chat models. If you are not sure which open model to start with, our roundup of the [best open-source LLMs to self-host](https://meetrix.io/blogs/best-open-source-llms-self-hosted-2026/) compares the main families. The [LocalAI models documentation](https://localai.io/docs/getting-started/models/) also covers importing models straight from Hugging Face.

## Call the LocalAI API

Use the **"ApiBaseUrl"** from the Outputs tab, which is your domain followed by `/v1`, and send your ApiKey in the **Authorization** header. Replace the domain and key with your own and list the installed models:

```bash
curl https://<your LocalAI domain>/v1/models \
  -H "Authorization: Bearer <your ApiKey>"
```

Then send a chat completion request to the pre-installed model:

```bash
curl https://<your LocalAI domain>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your ApiKey>" \
  -d '{
    "model": "qwen3-4b-instruct",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'
```

Because the API is OpenAI-compatible, the official OpenAI SDKs work by changing the base URL. In Python:

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://<your LocalAI domain>/v1",
    api_key="<your ApiKey>",
)

response = client.chat.completions.create(
    model="qwen3-4b-instruct",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(response.choices[0].message.content)
```

A chat front end such as the one in our [Open WebUI developer guide](https://meetrix.io/blogs/open-webui-developer-guide/) can use LocalAI as its OpenAI-compatible backend in the same way.

## Generate SSL Manually

LocalAI will automatically try to set up SSL when a Route 53-hosted domain is provided. If it fails, follow these steps to generate SSL manually.

### Step 1: Copy IP Address

1.  Follow the **Update DNS** steps above, if not already done.
2.  Copy the Public IP indicated as **"PublicIp"** in the **"Outputs"** tab.

### Step 2: Log in to the server

1.  Open the terminal and go to the directory where your private key is located.
2.  Run the following command, replacing the key name and IP address.
3.  Type "yes" and press Enter to confirm.

```bash
ssh -i <your key name> ubuntu@<Public IP address>
```

![Terminal connecting to the LocalAI server over SSH and confirming the host fingerprint](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-08.png)

### Step 3: Generate SSL

Run the following command and follow the prompts:

```bash
sudo /root/certificate_generate_standalone.sh
```

Once the certificate is in place, switch any applications that call the API over http to the https ApiBaseUrl, so the API key is no longer sent in plain text.

## Check Server Logs

### Step 1: Log in to the server

```bash
ssh -i <your key name> ubuntu@<Public IP address>
```

![Terminal connecting to the LocalAI server over SSH with a private key](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-09.png)

### Step 2: Check the logs

List the running containers, then view the logs of the LocalAI container using the first few characters of its container ID. The logs show model downloads, backend selection and any errors when a model fails to load:

```bash
sudo docker ps

sudo docker logs <container-id-prefix>
```

### Step 3: Check the GPU (GPU instances only)

On a g4dn instance, confirm that the T4 is visible and, while a model is answering a request, that its memory is in use:

```bash
nvidia-smi
```

If the GPU memory stays empty while a model runs, the model is running on the CPU. The [LocalAI GPU acceleration documentation](https://localai.io/docs/features/gpu-acceleration/) explains the `gpu_layers` and `f16` settings that offload a model to the GPU.

## Shutting Down LocalAI

1.  In CloudFormation, click the link labeled **"Instance"** in the **"Resources"** tab to open the EC2 instance.
2.  Stop the LocalAI instance from the **Instance state** dropdown. You can restart it later as needed.

![CloudFormation Resources tab with the EC2 Instance physical ID link](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-10.png) ![EC2 Instance state dropdown with the Stop instance option](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-11.png)

Note

While the instance is stopped, the LocalAI API is unavailable, so any application that calls it will fail until you start it again. Stopping a GPU instance when nobody needs it is the easiest way to cut its cost.

## Remove LocalAI

Delete the CloudFormation stack from the AWS Management Console under "CloudFormation Stacks" by clicking "Delete".

## Upgrades

When a new version is available in AWS Marketplace, remove the previous deployment after backing up necessary server data, and relaunch with the new version.

## Troubleshoot

If you face vCPU quota limits, request an increase: [How to increase AWS quota](https://meetrix.io/blogs/increase-aws-vcpu-quota/). GPU instances use a separate quota, "Running On-Demand G and VT instances", which you may need to raise even when your standard quota is fine.

![CloudFormation event showing CREATE\_FAILED with a VcpuLimitExceeded error](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-12.png)

If you face _insufficient capacity_ errors while creating the stack, try another region or time. GPU instance types run out of capacity more often than CPU types.

![CloudFormation event showing CREATE\_FAILED with an InsufficientInstanceCapacity error](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-13.png)

If LocalAI shows a 502 Bad Gateway error, wait 5-10 minutes and retry. LocalAI may still be starting up after the instance boots.

![502 Bad Gateway error page served by nginx](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-14.png)

If API calls return **401 Unauthorized**, check that the request sends **Authorization: Bearer** followed by the exact ApiKey from the stack parameters.

If a model fails to load or responses are very slow, the model is probably too large for the instance. Check the container logs, try a smaller or more heavily quantized model, or move to a GPU instance.

Check whether the instance storage is full. Models take several gigabytes each, so the disk fills up as you install more. Log into the server and run:

```bash
df -h
```

![df -h output showing usage of the root filesystem](https://meetrix.io/blog-images/paas-dev-guides/localai/localai-15.png)

If the root volume is between 90-100%, delete models you no longer use from the Models page or [resize the EBS volume](https://docs.aws.amazon.com/ebs/latest/userguide/requesting-ebs-volume-modifications.html), then reboot and restart the service.

## Conclusion

The Meetrix LocalAI Deployment Guide helps you run an OpenAI-compatible AI API in your AWS environment, on CPU or GPU, with your prompts and data staying on infrastructure you control. Whether you're a developer, DevOps engineer, or IT lead, this guide provides step-by-step instructions for a secure setup. See the [LocalAI store page](https://meetrix.io/store/localai/) for what the Meetrix image includes. If you need maximum throughput from a single GPU model rather than LocalAI's range of model types, our [vLLM developer guide](https://meetrix.io/blogs/vllm-developer-guide/) deploys a dedicated inference server on AWS, and the [Ollama vs vLLM comparison](https://meetrix.io/blogs/ollama-vs-vllm/) explains where each engine fits. The source code and release notes are on [GitHub](https://github.com/mudler/LocalAI).

## Technical Support

Reach out to Meetrix Support ([aws@meetrix.io](mailto:aws@meetrix.io)) for assistance with LocalAI issues.

## Frequently Asked Questions

What is LocalAI?

LocalAI is an open-source, MIT-licensed runtime for running AI models on your own hardware. It exposes an OpenAI-compatible API for text generation, embeddings, image generation, speech-to-text and text-to-speech, and ships a web UI for chatting with models and installing new ones from its gallery.

How do I log in to LocalAI?

Open the DashboardUrl from the CloudFormation Outputs tab and enter the ApiKey you set in the stack parameters. The same key is the bearer token for API requests, so store it securely and treat it like a password.

Does LocalAI need a GPU?

No. The Meetrix image runs on both CPU and GPU instances. t3a.xlarge is the recommended CPU baseline and works well for small models and embeddings. Choose g4dn.xlarge, which has an NVIDIA T4 with 16 GB of GPU memory, for faster responses, larger chat models and image generation.

Which model is installed by default?

qwen3-4b-instruct comes pre-installed, so you can chat with it as soon as you log in. Install more models from the gallery under Models in the web UI, and choose sizes that fit the memory of your instance.

Can I use the OpenAI SDK with LocalAI?

Yes. Set the SDK's base URL to the ApiBaseUrl from the Outputs tab (your domain followed by /v1), pass your ApiKey as the API key, and use the name of an installed model, such as qwen3-4b-instruct.

How do I upgrade LocalAI?

When a new version is available in AWS Marketplace, back up any models and configuration you need, remove the previous deployment, then launch a new stack with the new version.

How do I get technical support?

Reach out to Meetrix Support at [aws@meetrix.io](mailto:aws@meetrix.io) for assistance with LocalAI issues.

## Ready to Deploy Your Own LocalAI?

Get started in minutes with our pre-configured AMI and run an OpenAI-compatible AI API on CPU or GPU instances you control.

[Deploy LocalAI from AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-25t7st4jas56g)

Meetrix Store

LocalAI

OpenAI-compatible text, image and audio models, on CPU or GPU

[Deploy it](https://meetrix.io/store/localai/)

Meetrix Store New

Deploy what this guide covers, pre-configured.

-    [LocalAI OpenAI-compatible text, image and audio models, on CPU or GPU](https://meetrix.io/store/localai/)
-    [vLLM Serve open models on your own GPU](https://meetrix.io/store/vllm/)
-    [OpenWebUI A private ChatGPT-style assistant](https://meetrix.io/store/openwebui/)
-    [Jitsi Meet Self-hosted video calls for 50 to 500 users](https://meetrix.io/store/jitsi-meet/)
-    [RustDesk Remote desktop AMI, a TeamViewer alternative](https://meetrix.io/store/rustdesk/)
-    [Coturn TURN/STUN for WebRTC, no per-minute relay fees](https://meetrix.io/store/coturn/)

[Browse all products](https://meetrix.io/store/)