Apache NiFi - Developer Guide

Welcome to the Apache NiFi Deployment Guide for AWS Integration! Unlock the full power of Apache NiFi, the open-source data orchestration platform designed for robust data flows. Seamlessly integrate with your AWS infrastructure to achieve enterprise-grade scalability, secure data processing, and DevOps-friendly automation. Let's get started and elevate your data management with NiFi on AWS.

Deploy Apache NiFi on AWS with Meetrix

Simplify complex data flows in minutes with a pre-configured, production-ready Apache NiFi environment backed by 24/7 expert support.

Meetrix.IO | Maneesha Shehara

Launching the AMI

Step 1: Find and Select Apache NiFi AMI

1. Log in to your AWS Management Console.
2. Navigate to the 'Apache NiFi' in AWS Marketplace.

Step 2: Initial Setup & Configuration

1. Click the "Continue to Subscribe" button.
2. After subscribing, accept the terms and click "Accept Terms".
3. Wait a few minutes until processing completes, then click "Continue to Configuration".
4. Select "CloudFormation script to deploy Apache NiFi" as the fulfillment option and choose your region. Click "Continue to Launch".
5. From the "Choose Action" dropdown, select "Launch CloudFormation" and click "Launch".

Create CloudFormation Stack

Step 1: Create a stack

1. Ensure the "Template is ready" option is selected under "Prepare template".
2. Click "Next".

Step 2: Specify stack options

Provide the necessary parameters for your Apache NiFi deployment. These settings define how your instance will be configured.

Click "Next" once all parameters are set.

Step 3: Configure stack options

1. Choose "Roll back all stack resources" and "Delete all newly created resources" under "Stack failure options".
2. Click "Next".

Step 4: Review

1. Review and verify the details you've entered.

2. Tick "I acknowledge that AWS CloudFormation might create IAM resources with custom names".

3. Click "Submit".

Afterward, you'll be directed to the CloudFormation stacks page. Please wait for 5-10 minutes until the stack has been successfully created.

Update DNS

Step 1: Copy IP Address

Copy the public IP labeled "PublicIp" in the "Outputs" tab.

Step 2: Update DNS

1. Go to AWS Route 53 and navigate to "Hosted Zones".
2. Click Create record.
3. Add a record name and paste the copied PublicIp into the value textbox.
4. Click "Save".

Access Apache NiFi

You can access the Apache NiFi interface exclusively through the "DashboardUrl" provided in the "Outputs" tab. Direct access via the public IP address is not supported.

Upon first login, you will be prompted to create an Admin account. All subsequent user accounts must be created through the Admin panel.

Please enter the Username and Password that you provided during stack creation. Store them securely for future reference.

Generate SSL Manually

Apache NiFi will automatically try to set up SSL when a Route53-hosted domain is provided. If it fails, follow these steps to generate SSL manually.

Step 1: Copy IP Address

1. Follow the Update DNS steps above, if not already done.
2. Copy the Public IP indicated as "PublicIp" in the "Outputs" tab.

Step 2: Log in to the server

1. Open the terminal and go to the directory where your private key is located.
2. Paste the following command into your terminal and press Enter:

ssh -i <your key name> ubuntu@<Public IP address>

3. Type "yes" and press Enter to confirm.

Step 3: Generate SSL

Paste the following command into your terminal and press Enter, and follow the instructions:

sudo /root/certificate_generate_standalone.sh

You will be asked for the Admin Email to generate the SSL certificate.

Check Server Logs

Step 1: Log in to the server

1. Open the terminal and go to the directory where your private key is located.
2. Paste the following command into your terminal and press Enter:

ssh -i <your key name> ubuntu@<Public IP address>

3. Type "yes" and press Enter. This will log you into the server.

Step 2: Check the logs

sudo docker ps

Select the correct container and check logs:

sudo docker logs <container-id-prefix>

Shutting Down Apache NiFi

1. Click the link labeled "Instance" in the "Resources" tab to access the EC2 instance. You'll be directed to the Apache NiFi instance in EC2.

2. Select the Apache NiFi instance by marking the checkbox and click "Stop instance" from the "Instance state" dropdown. You can restart the instance at your convenience by selecting "Start instance".

Remove Apache NiFi

Delete the stack that has been created in the AWS Management Console under "CloudFormation Stacks" by clicking the "Delete" button.

Upgrades

When there is an upgrade, we will update the product with a newer version. You can check the product version in AWS Marketplace. If a newer version is available, you can remove the previous version and launch the product again using the newer version. Remember to backup the necessary server data before removing.

Troubleshoot

1. If you face the following error, please follow https://meetrix.io/blogs/increase-aws-vcpu-quota/ to increase your vCPU quota.

2. If you face the following error (do not have sufficient <instance_type> capacity...) while creating the stack, try changing the region or try creating the stack at a later time.

3. If you face the below error when you try to access the API dashboard, please wait 5-10 minutes and then try.

4. Check whether the instance storage is full.

• Log into the server and run the below command

df -h

• If the root volume is between 90-100%, it is better to resize the EBS volume. Please follow AWS documentation to increase the EBS volume.
• Then reboot the instance and restart the Apache NiFi instance service.

Conclusion

The Meetrix Apache NiFi Deployment Guide is your go-to resource for seamlessly integrating NiFi into your AWS environment. Whether you're a DevOps engineer, data architect, or IT leader, our guide provides clear, step-by-step instructions to ensure a secure and scalable setup. Apache NiFi is a powerful open-source data orchestration platform built for high-trust environments, empowering teams with real-time data ingestion, automation, and integration flexibility. With the Meetrix Apache NiFi Deployment Guide, you'll unlock the full potential of NiFi, enabling your organization to build efficient, secure, and self-hosted data pipelines that scale with your needs.

Technical Support

Reach out to Meetrix Support (aws@meetrix.io) for assistance with Apache NiFi issues.