> Source: https://meetrix.io/store/vllm/
> Markdown copy of that page. Cite the URL above, not this file.

[Store](https://meetrix.io/store/)  vLLM

Open-source LLM inference

# Self-hosted vLLM inference server on AWS and Google Cloud

Serve the open model of your choice from your own GPU with vLLM, the high-throughput inference engine. The pre-configured image comes with NVIDIA drivers, CUDA and an OpenAI-compatible API, and you pay for the GPU instead of per token.

 [Launch on AWS](https://aws.amazon.com/marketplace/pp/prodview-rvsnp5xf66l4m)[ Launch on Google Cloud](https://console.cloud.google.com/marketplace/product/meetrix-public/vllm)

-   GPU drivers pre-installed
-   OpenAI-compatible API
-   No per-token fees

![vLLM inference server deployed on AWS by Meetrix](https://meetrix.io/assets/store/vllm/vllm.webp)

## What is vLLM?

vLLM is an open-source, Apache 2.0 engine for serving large language models. PagedAttention and continuous batching keep GPU memory and compute busy across many requests, which gives high throughput and low latency under load. Its OpenAI-compatible HTTP API means code written for the OpenAI SDK works with a change of base URL.

-   ### High throughput

    PagedAttention and continuous batching serve many concurrent requests.

-   ### OpenAI-compatible

    /v1/chat/completions, /v1/completions and /v1/models work with existing SDKs.

-   ### Any Hugging Face model

    Llama, Qwen, Mistral, Gemma, Phi and more, if it fits your GPU.

-   ### Gated models

    Add a Hugging Face token to serve gated or private models.

-   ### API docs built in

    Swagger UI at /docs and ReDoc at /redoc.

-   ### Health and metrics

    /health and /metrics routes for monitoring.

## What's in the Meetrix vLLM image

The hard part of self-hosting vLLM is the GPU toolchain. The marketplace image ships with drivers, CUDA and the container runtime already matched to the GPU.

-   vLLM serving an OpenAI-compatible API in a container
-   NVIDIA drivers, CUDA and the container GPU runtime
-   nginx tuned for streaming token responses
-   Automatic Let's Encrypt SSL for your domain
-   A hardened base image with a firewall and unattended upgrades
-   Email support from Meetrix at support@meetrix.io

## How to set up a self-hosted vLLM server

1.  ### Check your GPU quota

    New AWS accounts and GCP projects often have a GPU quota of zero. Request an increase for your region before launching.

2.  ### Launch with your model

    Subscribe on AWS Marketplace or Google Cloud Marketplace, choose a GPU machine and enter the Hugging Face model ID, such as Qwen/Qwen2.5-7B-Instruct.

3.  ### Wait for the model to load

    On first boot vLLM downloads the weights and loads them onto the GPU, which takes 5-15 minutes depending on the model.

4.  ### Secure it and call the API

    Restrict access to trusted IPs or add authentication, then point any OpenAI client at https://your-domain/v1.

## Choose your cloud

### vLLM on AWS

An AMI on AWS Marketplace, launched on a G-family GPU instance in your own AWS account with a CloudFormation stack.

Deploys with

CloudFormation

Recommended size

g5.2xlarge (A10G, 24 GB)

First boot

5-15 minutes to load the model

#### Setup guides

-   [**Developer guide** GPU stack, model, SSL, API](https://meetrix.io/blogs/vllm-developer-guide/)
-   [**vLLM on AWS Marketplace** Throughput, pricing and alternatives](https://meetrix.io/blogs/vllm-aws-marketplace/)

 [Launch on AWS](https://aws.amazon.com/marketplace/pp/prodview-rvsnp5xf66l4m)

### vLLM on Google Cloud

The same inference server as a Google Cloud Marketplace image, deployed on a GPU VM in your own GCP project.

Deploys with

Marketplace deployment form

Recommended size

g2-standard-8 (L4, 24 GB)

Free trial

5 days, up to USD 50 licence credit

#### Setup guides

-   [**Developer guide** GPU VM, model, SSL, API](https://meetrix.io/blogs/vllm-gcp-developer-guide/)
-   [**vLLM on GCP Marketplace** Throughput, pricing and alternatives](https://meetrix.io/blogs/vllm-inference-server-gcp/)

 [Launch on Google Cloud](https://console.cloud.google.com/marketplace/product/meetrix-public/vllm)

## vLLM GPU sizing

vLLM needs an NVIDIA GPU; CPU-only machines will not work. Pick a machine whose GPU memory fits the model you want to serve.

| Cloud | Machine type | GPU | Good for |
| --- | --- | --- | --- |
| AWS | g5.2xlarge | 1× NVIDIA A10G, 24 GB | 7B to 8B models |
| AWS | g5.12xlarge or g6e sizes | More GPU memory | Larger models, higher concurrency |
| Google Cloud | g2-standard-8 | 1× NVIDIA L4, 24 GB | 7B to 8B models |
| Google Cloud | Larger L4 or a2 (A100) types | More GPU memory | Larger models, longer context |

If the logs show an out-of-memory error, choose a smaller or quantised model, or a larger machine.

## vLLM by Meetrix vs Together AI vs Amazon Bedrock

|  | vLLM by Meetrix | Together AI | Amazon Bedrock |
| --- | --- | --- | --- |
| Hosting | A GPU instance in your own account | Together's cloud | AWS-managed, not an instance in your VPC |
| Pricing model | GPU instance rate, no per-token fee | Pay per token | Pay per token |
| Model choice | Any Hugging Face model that fits your GPU | Together's model catalogue | Bedrock's model catalogue |
| Data location | Your own AWS account or GCP project | Together's infrastructure | AWS's shared infrastructure |

With vLLM, your prompts and the model weights stay on infrastructure you control. [Read the full vLLM on AWS comparison →](https://meetrix.io/blogs/vllm-aws-marketplace/)

## vLLM FAQ

What is vLLM?

An open-source inference and serving engine for large language models. PagedAttention and continuous batching give it high throughput and low latency, and its OpenAI-compatible API lets OpenAI SDKs talk to your own model without code changes.

Which machine should I choose?

A GPU machine: g5.2xlarge (one A10G, 24 GB) on AWS or g2-standard-8 (one L4, 24 GB) on Google Cloud for 7B to 8B models. Move to larger GPU types for bigger models or more concurrency.

Which models can I run?

Any Hugging Face model that fits in your GPU memory, including Llama, Qwen, Mistral, Gemma and Phi. Set it by repository ID at deployment; gated or private models also need a Hugging Face token.

Is the API secured by default?

No. The endpoint answers anyone who can reach it. Restrict the security group or firewall to trusted IPs, put it behind a VPN or an authenticating proxy, or run vLLM with an API key before exposing it.

How do I call it from my code?

Point any OpenAI-compatible client at https://your-domain/v1 and use the chat completions or completions route with the model you deployed. Swagger UI at /docs lists every route.

Why do I get a 502 error after launch?

vLLM is still downloading the model weights and loading them onto the GPU. Wait 5-15 minutes and refresh.

How do I change the model?

Update the model setting (the ModelName stack parameter on AWS, or the service configuration on the server) and restart vLLM. Make sure the new model fits your GPU memory.

Does stopping the server stop GPU billing?

Yes. Stopping the instance or VM halts GPU charges while keeping the disk, so you can start it again later.

## vLLM guides and articles

### [vLLM on AWS Marketplace: a High-Throughput Inference Server](https://meetrix.io/blogs/vllm-aws-marketplace/)

Your own inference endpoint, billed like compute instead of tokens.

### [vLLM on AWS: Developer Guide](https://meetrix.io/blogs/vllm-developer-guide/)

Launch a GPU stack, choose a model, issue SSL and call the API.

### [vLLM on GCP: Developer Guide](https://meetrix.io/blogs/vllm-gcp-developer-guide/)

Deploy vLLM on a Google Cloud GPU VM and serve an open model.

## Need a hand with your inference server?

We build and run self-hosted AI infrastructure for teams every day, from GPU sizing to model selection and scaling. Tell us what you need.

[Contact us](https://meetrix.io/contact-us/)