Serve the open model of your choice from your own GPU with vLLM, the high-throughput inference engine. The pre-configured image comes with NVIDIA drivers, CUDA and an OpenAI-compatible API, and you pay for the GPU instead of per token.
vLLM is an open-source, Apache 2.0 engine for serving large language models. PagedAttention and continuous batching keep GPU memory and compute busy across many requests, which gives high throughput and low latency under load. Its OpenAI-compatible HTTP API means code written for the OpenAI SDK works with a change of base URL.
PagedAttention and continuous batching serve many concurrent requests.
/v1/chat/completions, /v1/completions and /v1/models work with existing SDKs.
Llama, Qwen, Mistral, Gemma, Phi and more, if it fits your GPU.
Add a Hugging Face token to serve gated or private models.
Swagger UI at /docs and ReDoc at /redoc.
/health and /metrics routes for monitoring.
The hard part of self-hosting vLLM is the GPU toolchain. The marketplace image ships with drivers, CUDA and the container runtime already matched to the GPU.
New AWS accounts and GCP projects often have a GPU quota of zero. Request an increase for your region before launching.
Subscribe on AWS Marketplace or Google Cloud Marketplace, choose a GPU machine and enter the Hugging Face model ID, such as Qwen/Qwen2.5-7B-Instruct.
On first boot vLLM downloads the weights and loads them onto the GPU, which takes 5-15 minutes depending on the model.
Restrict access to trusted IPs or add authentication, then point any OpenAI client at https://your-domain/v1.
An AMI on AWS Marketplace, launched on a G-family GPU instance in your own AWS account with a CloudFormation stack.
Launch on AWS 
The same inference server as a Google Cloud Marketplace image, deployed on a GPU VM in your own GCP project.
Launch on Google Cloud vLLM needs an NVIDIA GPU; CPU-only machines will not work. Pick a machine whose GPU memory fits the model you want to serve.
| Cloud | Machine type | GPU | Good for |
|---|---|---|---|
| AWS | g5.2xlarge | 1× NVIDIA A10G, 24 GB | 7B to 8B models |
| AWS | g5.12xlarge or g6e sizes | More GPU memory | Larger models, higher concurrency |
| Google Cloud | g2-standard-8 | 1× NVIDIA L4, 24 GB | 7B to 8B models |
| Google Cloud | Larger L4 or a2 (A100) types | More GPU memory | Larger models, longer context |
If the logs show an out-of-memory error, choose a smaller or quantised model, or a larger machine.
| vLLM by Meetrix | Together AI | Amazon Bedrock | |
|---|---|---|---|
| Hosting | A GPU instance in your own account | Together's cloud | AWS-managed, not an instance in your VPC |
| Pricing model | GPU instance rate, no per-token fee | Pay per token | Pay per token |
| Model choice | Any Hugging Face model that fits your GPU | Together's model catalogue | Bedrock's model catalogue |
| Data location | Your own AWS account or GCP project | Together's infrastructure | AWS's shared infrastructure |
With vLLM, your prompts and the model weights stay on infrastructure you control. Read the full vLLM on AWS comparison →
An open-source inference and serving engine for large language models. PagedAttention and continuous batching give it high throughput and low latency, and its OpenAI-compatible API lets OpenAI SDKs talk to your own model without code changes.
A GPU machine: g5.2xlarge (one A10G, 24 GB) on AWS or g2-standard-8 (one L4, 24 GB) on Google Cloud for 7B to 8B models. Move to larger GPU types for bigger models or more concurrency.
Any Hugging Face model that fits in your GPU memory, including Llama, Qwen, Mistral, Gemma and Phi. Set it by repository ID at deployment; gated or private models also need a Hugging Face token.
No. The endpoint answers anyone who can reach it. Restrict the security group or firewall to trusted IPs, put it behind a VPN or an authenticating proxy, or run vLLM with an API key before exposing it.
Point any OpenAI-compatible client at https://your-domain/v1 and use the chat completions or completions route with the model you deployed. Swagger UI at /docs lists every route.
vLLM is still downloading the model weights and loading them onto the GPU. Wait 5-15 minutes and refresh.
Update the model setting (the ModelName stack parameter on AWS, or the service configuration on the server) and restart vLLM. Make sure the new model fits your GPU memory.
Yes. Stopping the instance or VM halts GPU charges while keeping the disk, so you can start it again later.
Your own inference endpoint, billed like compute instead of tokens.
Launch a GPU stack, choose a model, issue SSL and call the API.
Deploy vLLM on a Google Cloud GPU VM and serve an open model.
We build and run self-hosted AI infrastructure for teams every day, from GPU sizing to model selection and scaling. Tell us what you need.
Contact us