vLLM on AWS: Deploy a Self-Hosted vLLM Inference Server from AWS Marketplace

vLLM on AWS - Your Own Inference Endpoint, Billed Like Compute Instead of Tokens

Every hosted inference API works the same way underneath: you send tokens, they run the model on hardware you'll never see, and the bill grows in exact proportion to how much your product gets used. That's fine when usage is low. It stops being fine the month your app actually takes off and the inference line item grows faster than the revenue that's supposed to be paying for it.

Meetrix packaged vLLM into an AWS Marketplace CloudFormation stack that deploys to a single GPU EC2 instance in your own AWS account. NVIDIA drivers and CUDA pre-installed, the container pre-pulled so first boot doesn't stall downloading it, nginx tuned for streaming tokens, SSL handled automatically. Ready to try it? Launch the Meetrix vLLM stack on AWS Marketplace.

What Is vLLM?

vLLM is an open-source inference and serving engine for large language models, developed in the open on GitHub and released under the Apache 2.0 license. The throughput it's known for comes from two specific mechanisms, not vague optimization. PagedAttention manages the KV cache the way an operating system manages memory pages, so GPU memory doesn't get wasted on padding or fragmentation. Continuous batching keeps new requests flowing into a batch that's already running, instead of waiting for a batch to finish before starting the next one. Together, that's what lets one GPU serve far more concurrent requests than a naive implementation would. That gap is the whole argument in Ollama vs vLLM: Ollama is lovely on a laptop, vLLM is what you put behind an app with real users. And if you're weighing it against the other high-throughput engine, vLLM vs SGLang covers where each one wins.

It also exposes an OpenAI-compatible HTTP API at /v1, which matters more than it sounds like it should. Code written against the OpenAI SDK, LangChain, or any tool that expects that API shape works against your self-hosted model with a base URL change and nothing else. You're not rewriting integration code to switch from a hosted provider to your own instance.

Inference API or Deploy vLLM Yourself?

This is the question that actually decides whether any of the rest matters. A hosted API costs nothing while idle, which is why it wins early. A GPU instance costs the same every hour it's up, which is why it wins later. The crossover is utilization: once a g5.2xlarge would be busy for most of the working day, the flat hourly rate beats paying per token. Our GPU cost breakdown works through those numbers.

Two cases skip the math entirely. If prompts contain data that legally can't leave your AWS account, a hosted API is off the table no matter what it costs. And if you need a fine-tuned or niche model that no provider hosts, you're running it yourself anyway. Not sure which open model to serve? The best open-source LLMs to self-host is a good shortlist.

How Deployment Works

Setting this up by hand means picking a GPU instance, installing the right NVIDIA driver and CUDA version, wiring up the container GPU runtime, configuring nginx to not buffer streamed tokens, and downloading the model weights before serving a single request. Through the Marketplace, it's four steps:

1. Subscribe and Launch from AWS Marketplace Open the Meetrix vLLM listing, subscribe, and launch the CloudFormation stack on a G-family GPU instance. g5.2xlarge is the sensible default for 7B to 8B models.
2. GPU Drivers and the Container Are Already Set Up NVIDIA drivers, CUDA, and the container GPU runtime are pre-installed and matched to the instance's GPU, with the container image pre-pulled so first boot doesn't spend its first minutes downloading it.
3. Pick Your Model Set the ModelName stack parameter to a Hugging Face repository ID, Qwen/Qwen2.5-7B-Instruct for example. Add a Hugging Face token only if the model is gated or private.
4. Wait for Boot, Then Call the API Give the instance a few minutes to finish starting, then point any OpenAI-compatible client at your domain's /v1 path. Restrict access before you send it real traffic, it's open by default.

vLLM OpenAI-Compatible Server: Calling Your Endpoint

vLLM's built-in server speaks the same HTTP API as OpenAI, which the vLLM docs describe route by route. Once the stack is up, a quick check from your own machine looks like this:

curl https://your-domain/v1/models

curl https://your-domain/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

From Python, it's the normal OpenAI SDK with a different base URL. The model name has to match the ModelName you launched with, exactly as it appears in /v1/models:

from openai import OpenAI

client = OpenAI(base_url="https://your-domain/v1", api_key="not-needed-unless-you-set-one")
reply = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Summarize PagedAttention in one sentence."}],
)
print(reply.choices[0].message.content)

The same endpoint plugs into anything that accepts a custom OpenAI base URL, which is most chat front ends and agent frameworks. Our guide to self-hosted OpenAI-compatible APIs lists the ones we've tested.

The Swagger UI at /docs is the fastest way to poke at it without writing code. It lists every route, including /v1/embeddings, and lets you send a request straight from the browser.

What Meetrix Brings to This Deployment

• No GPU Toolchain to Assemble - NVIDIA drivers, CUDA, and the container GPU runtime come pre-installed and version-matched, not a driver compatibility puzzle to solve before vLLM will even start.
• A Pre-Pulled Container, Not a Cold Download - The vLLM container image is already pulled onto the instance, so first boot is spent starting the service, not fetching gigabytes of image layers.
• SSL and a Streaming-Ready Proxy From the Start - Automatic Let's Encrypt certificates via Route53, with nginx already tuned so token-by-token streaming responses don't get buffered into one slow chunk.
• Interactive API Docs Included - Swagger UI at /docs and ReDoc at /redoc ship with the instance, so you can see and test every route without hunting through vLLM's own documentation first.
• Hardened by Default - ufw, unattended-upgrades, and the SSM agent are active out of the box, plus IP-restricted SSH, not a hardening pass you do after the fact.
• People Who Actually Run This Stack - Support goes to aws@meetrix.io with a response inside 12 hours, from engineers who deploy GPU AMIs regularly, not a queue reading from a script.

Who Is vLLM on AWS Right For?

This fits teams that want inference costs to behave like infrastructure, not like a metered utility that tracks their growth. More specifically, it's a strong match if you're:

• A team that priced out Together AI or Bedrock against real projected token volume and didn't like watching the number scale with success
• A developer who wants to run Llama, Qwen, Mistral, Gemma, or Phi without rewriting OpenAI SDK integration code to talk to it
• An engineering org that needs model weights and inference requests to stay inside its own AWS account, not a shared third-party cluster
• A team serving high, steady request volume where a flat GPU-hour cost beats a per-token fee that never stops accumulating
• Someone who wants to swap the served model by changing a stack parameter, rather than migrating to a different hosted provider entirely
• A builder who wants Swagger docs and an OpenAI-compatible endpoint on day one, not a bare inference script to wrap in an API themselves

vLLM on AWS by Meetrix vs Alternatives

Resources

vLLM - Developer Guide

Step-by-step walkthrough of the full deployment: launching the CloudFormation stack, picking a model, setting up DNS and SSL, and calling the OpenAI-compatible API.

Meetrix.IO | Binuka Ranatunga

Under the Hood

Here's what's actually running once the stack finishes deploying.

Outgrowing a single instance? Running vLLM on Kubernetes with autoscaling GPU nodes is covered in vLLM on EKS, and the same stack runs on Google Cloud as vLLM on GCP Marketplace.

One Thing vLLM Won't Do For You

It won't lock the door behind you. The OpenAI-compatible endpoint is open to anyone who can reach it the moment the instance boots, no API key, no auth, nothing. That's a reasonable default for a quick test against your own IP, and a genuinely bad one to leave in place before pointing a public app at it. Restrict the security group, put it behind a VPN or a reverse proxy with authentication, or start vLLM with its --api-key flag so every request needs a bearer token, before real traffic touches it. This is the one step in the whole deployment that's on you, not the AMI.