Apache-NiFi - Developer Guide

Welcome to the Apache NiFi Deployment Guide for AWS Integration! Unlock the full power of Apache NiFi, the open-source data orchestration platform designed for robust data flows. Seamlessly integrate with your AWS infrastructure to achieve enterprise-grade scalability, secure data processing, and DevOps-friendly automation. Empower your teams with real-time data ingestion, workflow orchestration, and extensibility for mission-critical operations. Let's get started and elevate your data management with NiFi on AWS.

Prerequisites

Before you get started with the Apache-NiFi AMI, ensure you have the following prerequisites:

• Basic knowledge of AWS services, including EC2 instances and CloudFormation.
• An active AWS account with appropriate permissions.
• Enough vCPU limit to create instance
  (Follow https://meetrix.io/articles/how-to-increase-aws-quota/ blog to ensure this)

Launching the AMI

Step 1: Find and Select Apache-NiFi AMI

1. Log in to your AWS Management Console.
2. Navigate to the 'Apache-NiFi' in  AWS Marketplace.

Step 2:  Initial Setup & Configuration

1. Click the "Continue to Subscribe" button.
2. After subscribing, you will need to accept the terms and conditions. Click on "Accept Terms" to proceed.
3. Please wait for a few minutes while the processing takes place. Once it's completed, click on "Continue to Configuration".
4. Select the "CloudFormation script to deploy Apache-NiFi" as the fulfilment option and choose your preferred region on the "Configure this software" page. Afterward, click the "Continue to Launch" button.
5. From the "Choose Action" dropdown menu in "Launch this software" page, select "Launch CloudFormation" and click "Launch" button.

Create CloudFormation Stack

Step1: Create stack

1. Ensure the "Template is ready" radio button is selected under "Prepare template".

2.  Click "Next".

Step2: Specify stack options

1. Provide a unique "Stack name".
2. Provide the "Admin Email" for SSL generation.
3. For "DeploymentName", enter a name of your choice.
4. Provide a public domain name for "DomainName" (Apache-NiFi will automatically try to set up SSL based on provided domain name, if that domain hosted on Route53. Please make sure your domain name hosted on route53. If its unsuccessful then you have to setup SSL manually)
5. Choose an instance type, "InstanceType" (Recommended: t3.medium).
6. Select your preferred "keyName".
7. Set "SSHLocation" as "0.0.0.0/0".
8. Keep "SubnetCidrBlock" as "10.0.0.0/24".
9. Keep "VpcCidrBlock" as "10.0.0.0/16".
10. Click "Next".

Step3: Configure stack options

1. Choose "Roll back all stack resources" and "Delete all newly created resources" under the "Stack failure options" section.
2. Click "Next".

Step4: Review

1. Review and verify the details you've entered.

2. Tick the box that says, "I acknowledge that AWS CloudFormation might create IAM resources with custom names".

3. Click "Submit".

Afterward, you'll be directed to the CloudFormation stacks page.

Please wait for 5-10 minutes until the stack has been successfully created.

Step1: Copy IP Address

Copy the public Ip labeled "PublicIp" in the "Outputs" tab.

Step2: Update DNS

1. Go to AWS Route 53 and navigate to "Hosted Zones".From there click on Create record.

3. Add record name and then paste the copied "PublicIp" into the "value" textbox.

4. Click "Save".

Update DNS

Step1: Copy IP Address

Copy the public IP labeled "PublicIp" in the "Outputs" tab.

Step2: Update DNS

1. Go to AWS Route 53 and navigate to "Hosted Zones".
2. From there, click on Create record.

3. Add the record name and then paste the copied "PublicIp" into the "value" textbox.

4. Click "Save".

Access Apache-NiFi

You can setup the Apache-NiFi application through the "DashboardUrl" or "DashboardUrlIp" provided in the "Outputs" tab. This will allow you to create an Admin account and all the future accounts will be created through the Admin panel.

(If you encounter a "502 Bad Gateway error", please wait for about 5 minutes before refreshing the page)

Please enter the Username and Password that you have provided. Ensure both are stored securely for future reference.

Generate SSL Manually

Apache-NiFi will automatically try to setup SSL based on provided domain name, if that domain hosted on Route53. If its unsuccessful then you have to setup SSL manually.

Step1: Copy IP Address

1. Proceed with the instructions outlined in the above "Update DNS" section, if you have not already done so.
2. Copy the Public IP address indicated as "PublicIp" in the "Outputs" tab.

Step2: Log in to the server

1. Open the terminal and go to the directory where your private key is located.
2. Paste the following command into your terminal and press Enter:

ssh -i <your key name> ubuntu@<Public IP address>.

3. Type "yes" and press Enter. This will log you into the server.

Step3: Generate SSL

Paste the following command into your terminal and press Enter, and follow the instructions

sudo /root/certificate_generate_standalone.sh

Admin Email is acquiring for generate SSL certificates.

Check Server Logs

Step1: Log in to the server

1. Open the terminal and go to the directory where your private key is located.
2. Paste the following command into your terminal and press Enter:

ssh -i <your key name> ubuntu@<Public IP address>

3. Type "yes" and press Enter. This will log you into the server.

Step2: Check the logs

sudo docker ps

Select the correct container and check logs

sudo docker logs 0e

Shutting Down Apache-NiFi

1. Click the link labeled "Instance" in the "Resources" tab to access the EC2 instance, you will be directed to the Apache-NiFi instance instance in EC2.

2. Select the Apache-NiFi instance instance by marking the checkbox and click "Stop instance" from the "Instance state" dropdown. You can restart the instance at your convenience by selecting "Start instance".

Remove Apache-NiFi

Delete the stack that has been created in the AWS Management Console under 'CloudFormation Stacks' by clicking the 'Delete' button.

Upgrades

When there is an upgrade, we will update the product with a newer version. You can check the product version in AWS Marketplace. If a newer version is available, you can remove the previous version and launch the product again using the newer version. Remember to backup the necessary server data before removing.

Troubleshoot

1. If you face the following error, please follow https://meetrix.io/articles/how-to-increase-aws-quota/ blog to increase vCPU quota.

2. If you face the following error (do not have sufficient <instance_type> capacity...) while creating the stack, try changing the region or try creating the stack at a later time.

3. If you face the below error, when you try to access the API dashboard, please wait 5-10 minutes and then try.

4. Check whether the instance storage is full.

• Log into the server and run the below command

                                                                 df -h

                                                             root volume

• If the root volume is between 90-100%, it is better to resize EBS volume. Please follow AWS documentation to increase the EBS volume.
• Then reboot the instance and restart the Apache-NiFi instance service.

Conclusion

The Meetrix Apache NiFi Deployment Guide is your go-to resource for seamlessly integrating NiFi into your AWS environment. Whether you're a DevOps engineer, data architect, or IT leader, our guide provides clear, step-by-step instructions to ensure a secure and scalable setup. Apache NiFi is a powerful open-source data orchestration platform built for high-trust environments, empowering teams with real-time data ingestion, automation, and integration flexibility. With the Meetrix Apache NiFi Deployment Guide, you'll unlock the full potential of NiFi, enabling your organization to build efficient, secure, and self-hosted data pipelines that scale with your needs

Technical Support

Reach out to Meetrix Support (aws@meetrix.io)  for assistance with Apache-NiFi instance issues.