Llama 4 - Developer Guide

Learn how to deploy Llama 4 on AWS with our step-by-step developer guide. This comprehensive resource walks you through provisioning your cloud environment, securely launching Llama 4, and configuring it for reliable, scalable inference on AWS. Whether you're building chat applications, generating embeddings, or experimenting with multimodal AI, this guide will help you get Llama 4 running efficiently in the cloud.

Meta's Llama 4 Scout brings a Mixture-of-Experts architecture with 17 billion active parameters to the open-source community. With Meetrix's pre-configured AMI, you can deploy Llama 4 Scout directly from AWS Marketplace without wrestling with dependencies, GPU drivers, or infrastructure setup. Let's get started.

Related Article

LLaMA 4 Scout by Meetrix on AWS Marketplace: Your Gateway to Multimodal AI

Meta's most advanced open-source vision-language model, ready to deploy with Meetrix.

Meetrix.IO

Launching the AMI

Step 1: Find and Select 'Llama 4 Scout' AMI

1. Log in to your AWS Management Console.
2. Follow the provided links to access the 'Llama 4 Scout' product you wish to set up.
   1. Llama 4 Scout: OpenAI API Compatible AMI

Step 2: Initial Setup & Configuration

1. Click the "Continue to Subscribe" button.
2. After subscribing, you will need to accept the terms and conditions. Click on "Accept Terms" to proceed.
3. Please wait for a few minutes while the processing takes place. Once it's completed, click on "Continue to Configuration".
4. Select the "CloudFormation Template for Llama 4 Scout deployment" as the fulfilment option and choose your preferred region on the "Configure this software" page. Afterward, click the "Continue to Launch" button.
5. From the "Choose Action" dropdown menu in "Launch this software" page, select "Launch CloudFormation" and click "Launch" button.

Create CloudFormation Stack

Step 1: Create a stack

1. Ensure the "Template is ready" radio button is selected under "Prepare template".
2. Click "Next".

Step 2: Specify stack options

1. Provide a unique "Stack name".
2. Provide the "Admin Email" for SSL generation.
3. For "DeploymentName", enter a name of your choice.
4. Provide a public domain name for "DomainName".
5. Choose an instance type, "InstanceType" (Recommended: g4dn.xlarge).
6. Select your preferred "keyName".
7. Set "SSHLocation" as "0.0.0.0/0".
8. Keep "SubnetCidrBlock" as "10.0.0.0/24".
9. Keep "VpcCidrBlock" as "10.0.0.0/16".
10. Click "Next".

Step 3: Configure stack options

1. Choose "Roll back all stack resources" and "Delete all newly created resources" under the "Stack failure options" section.
2. Click "Next".

Step 4: Review

1. Review and verify the details you've entered.

2. Tick "I acknowledge that AWS CloudFormation might create IAM resources with custom names".

3. Click "Submit".

Afterward, you'll be directed to the CloudFormation stacks page. Please wait for 5-10 minutes until the stack has been successfully created.

Update DNS

Step 1: Copy IP Address

Copy the public IP labeled "PublicIp" in the "Outputs" tab.

Step 2: Update DNS

1. Go to AWS Route 53 and navigate to "Hosted Zones".
2. From there click on Create record.

3. Add record name and then paste the copied "PublicIp" into the "value" textbox.
4. Click "Save".

Access Llama 4

You can access the Llama 4 application through the "DashboardUrl" or "DashboardUrlIp" provided in the Outputs tab.

Generate SSL Manually

Llama 4 Scout will automatically try to setup SSL based on provided domain name. If it fails, follow these steps to generate SSL manually.

Step 1: Copy IP Address

1. Proceed with the instructions outlined in the above "Update DNS" section, if you have not already done so.
2. Copy the Public IP address indicated as "PublicIp" in the "Outputs" tab.

Step 2: Log in to the server

1. Open the terminal and go to the directory where your private key is located.
2. Paste the following command into your terminal and press Enter: ssh -i <your key name> ubuntu@<Public IP address>

3. Type "yes" and press Enter. This will log you into the server.

Step 3: Generate SSL

Paste the following command into your terminal and press Enter and follow the instructions:

sudo /root/certificate_generate_standalone.sh

Admin Email is acquiring for generate SSL certificates.

API Documentation

1. Retrieve Completions

Endpoint: /v1/completions   Method: POST

Request Body:

{
  "model": "llama4-scout",
  "prompt": "\n\n### Instructions:\nWhat is the capital of France?\n\n### Response:\n",
  "stop": ["\n", "###"]
}

Response Body:

{
  "id": "cmpl-498760e1-2b50-47c3-95fb-98c8bff8b10a",
  "object": "text_completion",
  "created": 1717503179,
  "model": "llama4-scout",
  "choices": [
    {
      "text": "The correct answer is **Paris.**",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 16,
    "completion_tokens": 8,
    "total_tokens": 24
  }
}

2. Retrieve Embeddings

Endpoint: /v1/embeddings   Method: POST

Request Body:

{
  "input": "The food was delicious and the waiter...",
  "model": "llama4-scout"
}

Response Body:

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [
        -0.37700873613357544,
        1.3124240636825562,
        4.191315650939941,
        ...
      ],
      "index": 0
    }
  ],
  "model": "llama4-scout",
  "usage": {
    "prompt_tokens": 9,
    "total_tokens": 9
  }
}

3. Retrieve Chat Completions

Endpoint: /v1/chat/completions   Method: POST

Request Body:

{
  "messages": [
    {
      "content": "You are a helpful assistant.",
      "role": "system"
    },
    {
      "content": "What is the capital of France?",
      "role": "user"
    }
  ],
  "model": "llama4-scout",
  "stop": ["\n", "###"]
}

Response Body:

{
  "id": "chatcmpl-8c5130ab-eca9-4760-8171-f7a3aee9b9ba",
  "object": "chat.completion",
  "created": 1717503343,
  "model": "llama4-scout",
  "choices": [
    {
      "index": 0,
      "message": {
        "content": "The capital city of France is Paris.<|im_end|>",
        "role": "assistant"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 50,
    "completion_tokens": 13,
    "total_tokens": 63
  }
}

4. List Models

Endpoint: /v1/models   Method: GET

{
  "object": "list",
  "data": [
    {
      "id": "llama4-scout",
      "object": "model",
      "owned_by": "me",
      "permissions": []
    },
    {
      "id": "llama4-scout-instruct",
      "object": "model",
      "owned_by": "me",
      "permissions": []
    }
  ]
}

5. Use different Models

• Run "List Models"
• Select preferred model
• Copy "id" from response
• Replace "model" variable in request body

Note changing models requires additional response time.

Testing the API

1. Create a directory
2. Create 3 files (Full codes are given below): app.js, package.json, .env
3. Run the following command: npm install
4. Edit variable file (.env)
5. Run the following command: npm start
6. You will get the responses

app.js

const axios = require('axios');
require('dotenv').config();

const makePostRequest = async (url, data, timeout) => {
  try {
    const response = await axios.post(url, data, { timeout });
    return { success: response.status === 200, data: response.data };
  } catch (error) {
    return { success: false, error: error.message };
  }
};

const makeGetRequest = async (url, timeout) => {
  try {
    const response = await axios.get(url, { timeout });
    return { success: response.status === 200, data: response.data };
  } catch (error) {
    return { success: false, error: error.message };
  }
};

const printResponseData = (endpoint, data) => {
  console.log(`Response for ${endpoint}:`);
  console.log(JSON.stringify(data, null, 2));
  console.log('');
};

const checkEndpoints = async () => {
  const baseUrl = process.env.BASE_URL;
  const model = process.env.MODEL;

  const endpoints = [
    { path: '/completions', method: makePostRequest, data: { "model": model, "prompt": process.env.PROMPT1 }, printEnv: 'PRINT_COMPLETIONS_RESPONSE' },
    { path: '/embeddings', method: makePostRequest, data: { "input": process.env.PROMPT2, "model": model }, printEnv: 'PRINT_EMBEDDINGS_RESPONSE' },
    { path: '/chat/completions', method: makePostRequest, data: { "messages": [{ "content": "You are a helpful assistant.", "role": "system" }, { "content": process.env.PROMPT1, "role": "user" }], "model": model }, printEnv: 'PRINT_CHAT_COMPLETIONS_RESPONSE' },
    { path: '/models', method: makeGetRequest, printEnv: 'PRINT_MODELS_RESPONSE' }
  ];

  for (const endpoint of endpoints) {
    const url = `${baseUrl}${endpoint.path}`;
    const { success, data, error } = await endpoint.method(url, endpoint.method === makePostRequest ? endpoint.data : null, process.env.REQUEST_TIMEOUT || 50000);
    const printResponse = process.env[endpoint.printEnv] === 'true';

    if (success) {
      console.log(`*** Endpoint ${endpoint.path} is reachable.`);
      if (printResponse) {
        printResponseData(endpoint.path, data);
      }
      console.log('');
    } else {
      console.log(`*** Endpoint ${endpoint.path} is not reachable. Error:`, error);
    }
  }
};

checkEndpoints();

package.json

{
  "name": "test-llama",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node app.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "axios": "^1.6.7",
    "dotenv": "^16.4.1"
  }
}

.env

# Base URL for the API
BASE_URL=https://mixtral-test-prod.meetrix.io/v1

# Model to be used in requests
MODEL=mixtral-8x7b-instruct-v0.1

# Prompts for different endpoints
# /completions and /chat/completions
PROMPT1=What is the capital of France?
# /embeddings
PROMPT2=The food was delicious and the waiter...

# Whether to print responses for each endpoint
PRINT_COMPLETIONS_RESPONSE=true
PRINT_EMBEDDINGS_RESPONSE=false
PRINT_CHAT_COMPLETIONS_RESPONSE=true
PRINT_MODELS_RESPONSE=true

# Timeout for requests in milliseconds (default is 50000)
REQUEST_TIMEOUT=50000

Check Server Logs

Step1: Log in to the server

1. Open the terminal and go to the directory where your private key is located.
2. Paste the following command into your terminal and press Enter:

ssh -i <your key name> ubuntu@<Public IP address>

3. Type "yes" and press Enter. This will log you into the server.

Step2: Check the logs

sudo tail -f /var/log/syslog

Shutting Down Llama 4

1. Click the link labeled "Llama" in the "Resources" tab.

2. Select the Llama instance by marking the checkbox and click "Stop instance" from the "Instance state" dropdown. You can restart the instance at your convenience by selecting "Start instance".

Remove Llama 4

Delete the stack that has been created in the AWS Management Console under 'CloudFormation Stacks' by clicking the 'Delete' button.

Upgrades

When there is an upgrade, we will update the product with a newer version. You can check the product version in AWS Marketplace. If a newer version is available, you can remove the previous version and launch the product again using the newer version. Remember to backup the necessary server data before removing.

Troubleshoot

1. If you face the following error, please follow this guide to increase vCPU quota.

2. If you face the following error (do not have sufficient <instance_type> capacity...) while creating the stack, try changing the region or try creating the stack at a later time.

3. If you face the below error, when you try to access the API dashboard, please wait 5-10 minutes and then try.

4. If the llama service got stuck, you can follow the steps below
• Log into the server (Find the steps in Check Server Logs section)
• Run the below command.
• Wait for several minutes and reload the dashboard URL.

sudo systemctl restart llama.service

5. Check whether the instance storage is full.
• Log into the server and run the below command
• If the root volume is between 90-100%, it is better to resize EBS volume. Please follow AWS documentation to increase the EBS volume.
• Then reboot the instance and restart the llama service.

df -h

Conclusion

In conclusion, the Llama 4 Developer Guide equips you with everything you need for a seamless integration of Llama 4 Scout into your AWS environment. Whether you're a novice or an experienced developer, our guide offers detailed, step-by-step instructions to ensure a smooth setup process. Llama 4 Scout represents a leap forward in AI sophistication, seamlessly integrating with AWS to offer unparalleled power and simplicity. From language understanding to code generation, Llama 4 Scout empowers you to explore the frontiers of artificial intelligence effortlessly.

Technical Support

Reach out to Meetrix Support (aws@meetrix.io) for assistance with Llama 4 Scout issues.