Skip to main content

AWS Kinesis Firehose

Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, and HTTP endpoints. The AWS Kinesis Firehose connector enables you to create and manage delivery streams, send data records to streams, and retrieve stream information, allowing you to integrate real-time data streaming into your data workflows.

AWS Kinesis Firehose icon

Power end-to-end data operations for your AWS Kinesis Firehose API with Nexla. Our bi-directional AWS Kinesis Firehose connector is purpose-built for AWS Kinesis Firehose, making it simple to ingest data, sync it across systems, and deliver it anywhere — all with no coding required. Nexla turns API-sourced data into ready-to-use, reusable data products and makes it easy to send data to AWS Kinesis Firehose or any other destination. With comprehensive monitoring, lineage tracking, and access controls, Nexla keeps your AWS Kinesis Firehose workflows fast, secure, and fully governed.

Features

Type: API

SourceDestination

  • Seamless API Integration: Connect to any endpoint as source or destination without coding, with automatic data product creation
  • Visual Composition & Chaining: Build complex integrations using visual templates, chain API calls, and compose workflows with data validation and filtering
  • API Proxy: Expose curated slices of your data securely with a secure and customizable API proxy that validates and transforms data on the fly
  • Request optimization with intelligent batching, retry, and caching to minimize API calls and costs

Prerequisites​

Before creating an AWS Kinesis Firehose credential, you'll need to obtain AWS credentials with appropriate permissions for Kinesis Data Firehose operations. AWS uses signature-based authentication that requires access keys and region information.

AWS Account Setup​

To obtain the required AWS credentials for Kinesis Data Firehose:

  1. Sign in to AWS Console: Navigate to https://aws.amazon.com/console/ and sign in to your AWS account. If you don't have an account, you can create one by clicking Create an AWS Account and following the registration process.

  2. Access IAM Service: Once signed in, navigate to the IAM (Identity and Access Management) service from the AWS Services menu. IAM allows you to manage users, groups, and permissions for your AWS account.

  3. Create IAM User or Use Existing: Choose one of the following approaches:

    Option A: Create a New IAM User

    1. In the IAM dashboard, click Users in the left navigation pane, then click Add users.

    2. Enter a username for the new user (e.g., nexla-kinesis-firehose-user).

    3. Select Provide user access to the AWS Management Console if you want console access, or leave it unchecked for programmatic access only.

    4. Click Next to proceed to permissions.

    5. Click Attach policies directly and search for AmazonKinesisFirehoseFullAccess in the policy search box. This managed policy provides full access to all Kinesis Data Firehose operations. Alternatively, you can create a custom policy with only the specific permissions you need. If creating a custom policy, include the following actions:

      • firehose:CreateDeliveryStream - Required for creating new delivery streams
      • firehose:DeleteDeliveryStream - Required for deleting delivery streams
      • firehose:DescribeDeliveryStream - Required for retrieving delivery stream details
      • firehose:ListDeliveryStreams - Required for listing all delivery streams in your account
      • firehose:PutRecord - Required for sending individual records to delivery streams
      • firehose:PutRecordBatch - Required for sending batches of records to delivery streams
      • firehose:UpdateDestination - Required for updating delivery stream destination configurations
      • firehose:ListTagsForDeliveryStream - Required for retrieving tags associated with delivery streams
      • firehose:StartDeliveryStreamEncryption - Required for enabling encryption on delivery streams
      • firehose:StopDeliveryStreamEncryption - Required for disabling encryption on delivery streams

    6. Click Next to review, then click Create user.

    Option B: Use Existing IAM User

    1. In the IAM dashboard, click Users in the left navigation pane.

    2. Select the existing user you want to use for Nexla.

    3. Click the Permissions tab and ensure the user has the necessary Kinesis Data Firehose permissions listed above.

  4. Create Access Keys: After selecting or creating a user:

    1. Click on the user name to open the user details page.

    2. Click the Security credentials tab.

    3. Scroll to the Access keys section and click Create access key.

    4. Select Application running outside AWS as the use case, then click Next.

    5. Optionally add a description tag, then click Create access key.

    6. Important: Copy both the Access key ID and Secret access key immediately. The secret key will not be shown again. Store these securely.

The Access key ID and Secret access key are sensitive credentials that should be kept secure. Store them in a secure location, as you'll need them when creating the credential in Nexla. Never share these credentials publicly or commit them to version control systems.

  1. Determine AWS Region: Identify the AWS region where your Kinesis Data Firehose delivery streams are located or where you want to create them. The region determines which AWS data center will handle your API requests. Common regions include:

    • us-east-1 (US East - N. Virginia)
    • us-west-2 (US West - Oregon)
    • eu-west-1 (Europe - Ireland)
    • ap-southeast-1 (Asia Pacific - Singapore)

    You can find your region in the AWS Console header (top right corner) or by checking your existing Kinesis Data Firehose stream configuration. The region you select should match the region where your delivery streams are located or where you plan to create them.

  2. Determine Service Name: The service name for Kinesis Data Firehose is firehose. This value is used as part of the AWS Signature Version 4 authentication process. The service name tells AWS which service you're authenticating to and is required for proper request signing.

  3. Determine Base URL: The base URL for Kinesis Data Firehose API follows the pattern https://firehose.{region}.amazonaws.com, where {region} is your AWS region identifier. For example:

    • For us-east-1 region: https://firehose.us-east-1.amazonaws.com
    • For us-west-2 region: https://firehose.us-west-2.amazonaws.com
    • For eu-west-1 region: https://firehose.eu-west-1.amazonaws.com

    This base URL is the endpoint that will receive all your Kinesis Data Firehose API requests. Ensure the region in the URL matches the region where your resources are located.

For complete information about AWS IAM and managing access keys, see the AWS IAM User Guide. For Kinesis Data Firehose API authentication details, see the AWS Kinesis Data Firehose API Reference.

API Access Requirements​

AWS Kinesis Data Firehose API access requires:

  • Valid AWS Account: You must have an active AWS account
  • IAM User with Access Keys: An IAM user with programmatic access and appropriate permissions
  • AWS Access Key ID: The access key identifier for your IAM user
  • AWS Secret Access Key: The secret key for your IAM user
  • AWS Region: The AWS region where your Kinesis Data Firehose resources are located
  • Service Name: The AWS service name (firehose for Kinesis Data Firehose)
  • Base URL: The regional endpoint URL for the Kinesis Data Firehose API
  • Optional Session Token: If using temporary credentials (e.g., from AWS STS), a session token may be required

The AWS Kinesis Data Firehose API uses AWS Signature Version 4 authentication, which requires signing each request with your access keys. Nexla handles this signature process automatically when you provide your credentials.

Authenticate​

Credentials required​

A secure authentication method that signs API requests using AWS access keys

FieldRequiredSecretDescription
AWS Access KeyYesNoThe access key id of the Access Key pair that is used for verifying the identity of the requesting user.
AWS Secret KeyYesYesThe access secret of the Access Key pair that is used for verifying the identity of the requesting user.
AWS RegionYesNoThe region that is receiving the request.
Service NameYesNoThe service that is receiving the request.
Session TokenYesYesOptional session token if the request requires temporary security credentials. This is added to the request as x-amz-access-token request header.
AWS Kinesis Data Firehose Base URLYesNoSpecifies the HTTPS endpoint for Firehose API operations. Required for authenticating requests.

Create a credential in Nexla​

  1. After selecting the data source/destination type, click the Add Credential tile to open the Add New Credential overlay.

New Credential Overlay – AWS Kinesis Data Firehose API

AWSKinesisCred.png

  1. Enter a name for the credential in the Credential Name field and a short, meaningful description in the Credential Description field.

  2. AWS Access Key: Enter your AWS Access Key ID in the AWS Access Key field. This unique identifier authenticates your IAM user with AWS services. You can find this value in your IAM user's security credentials section.

  3. AWS Secret Key: Enter your AWS Secret Access Key in the AWS Secret Key field. This confidential key is used to securely sign API requests using AWS Signature Version 4. The secret key should be kept secure and never shared.

  4. AWS Region: Enter the AWS region where your Kinesis Data Firehose delivery streams are located in the AWS Region field. Common regions include us-east-1, us-west-2, eu-west-1, ap-southeast-1, etc. The region determines which AWS endpoint will be used for API requests.

  5. Service Name: Enter the AWS service name in the Service Name field. For Kinesis Data Firehose, this should be firehose. This value is used as part of the AWS signature authentication process.

  6. Session Token: If you are using temporary credentials (e.g., from AWS STS or assumed roles), enter the session token in the Session Token field. For permanent IAM user credentials, this field can be left empty. Session tokens are required when using temporary security credentials and are added to requests as the x-amz-access-token header.

  7. AWS Kinesis Data Firehose Base URL: Enter the base URL for the Kinesis Data Firehose API in the AWS Kinesis Data Firehose Base URL field. The URL should follow the pattern https://firehose.{region}.amazonaws.com, where {region} is your AWS region (e.g., https://firehose.us-east-1.amazonaws.com). This URL is required for authenticating requests to the Kinesis Data Firehose API.

    The AWS credentials are sensitive information that should be kept secure. Nexla will store these credentials securely and use them only for API authentication purposes. Ensure your IAM user has the minimum required permissions for the operations you need to perform. For production environments, consider using IAM roles with temporary credentials instead of permanent access keys when possible.

  8. Click the Save button at the bottom of the overlay to save the configured credential. The newly added credential will now appear in a tile on the Authenticate screen during data source/destination creation and can be selected for use with a new data source or destination.

Use as a data source​

To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Select the AWS Kinesis Firehose connector tile, then select the credential that will be used to connect to the AWS Kinesis Firehose instance, and click Next; or, create a new AWS Kinesis Firehose credential for use in this flow.

Endpoint templates​

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common AWS Kinesis Firehose endpoints. Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below.

List Delivery Streams

This endpoint retrieves a list of all delivery streams in your AWS Kinesis Data Firehose account. Use this endpoint when you need to discover available delivery streams, monitor stream metadata, or build dynamic workflows that operate on multiple streams.

  • This endpoint automatically retrieves all delivery streams in your AWS account within the specified region. No additional configuration is required beyond selecting this endpoint template. The endpoint uses the AWS region specified in your credential configuration to determine which delivery streams to retrieve.
  • The endpoint returns delivery stream names, ARNs (Amazon Resource Names), creation timestamps, and status information for each stream in your account. This metadata is useful for discovering available streams, monitoring stream status, or building dynamic workflows that operate on multiple delivery streams.

This endpoint is useful for discovering available delivery streams and their configurations. Use this endpoint to build dynamic workflows that can operate on multiple delivery streams.

List Tags For Delivery Stream

This endpoint retrieves tags associated with a specific delivery stream. Use this endpoint when you need to access tag metadata for delivery stream management, organization, or filtering purposes.

  • This endpoint requires the delivery stream name to be specified in the API request. The delivery stream name is the unique identifier for the stream you want to retrieve tags for. You can find delivery stream names using the "List Delivery Streams" endpoint or in the AWS Console under Kinesis Data Firehose.
  • The endpoint will return all tags associated with the specified delivery stream. Tags are key-value pairs that can be used to organize and manage your delivery streams. Common uses include tagging streams by project, environment (production, staging, development), data source type, or compliance requirements. Tags help you filter, search, and manage delivery streams at scale.

Tags are useful for organizing delivery streams by project, environment, or other organizational criteria. This endpoint allows you to retrieve tag information for stream management purposes.

Once the selected endpoint template has been configured, click the Test button to the right of the endpoint selection menu to retrieve a sample of the data that will be fetched. Sample data will be displayed in the Endpoint Test Result panel on the right, allowing you to verify that the source is configured correctly before saving.

Manual configuration​

AWS Kinesis Firehose data sources can also be manually configured to ingest data from any valid AWS Kinesis Firehose API endpoint, including endpoints not covered by the pre-built templates, chained API calls, or custom request parameters. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, endpoint URL, date/time and lookup macros, path to data, metadata, and request headers.

AWS Kinesis Firehose API endpoints follow the pattern https://firehose.{region}.amazonaws.com/?Action={ActionName} and use query string parameters to specify the action. When setting the path to data, use JSON path notation (e.g., $.DeliveryStreamNames[*]) for JSON responses or XPath (e.g., /ListDeliveryStreamsResponse/DeliveryStreamNames/string) for XML responses. The API uses headers such as Content-Type:application/x-amz-json-1.1 and X-Amz-Target for versioning, which are typically handled automatically based on your credential configuration.

Once all of the relevant settings have been configured, click the Create button in the upper right corner of the screen to save and create the new AWS Kinesis Firehose data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.

Use as a destination​

Click the + icon on the Nexset that will be sent to the AWS Kinesis Firehose destination, and select the Send to Destination option from the menu. Select the AWS Kinesis Firehose connector from the list of available destination connectors, then select the credential that will be used to connect to the AWS Kinesis Firehose organization, and click Next; or, create a new AWS Kinesis Firehose credential for use in this flow.

Endpoint templates​

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common AWS Kinesis Firehose endpoints. Select the endpoint to which data will be sent from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

Put Record

This endpoint sends individual records to an AWS Kinesis Data Firehose delivery stream. Use this endpoint when you need to send single records to a delivery stream for real-time data ingestion.

  • This endpoint sends individual data records to the specified delivery stream. The delivery stream name should be included in the request configuration. Each record is sent as a separate API call, which makes this endpoint suitable for real-time data streaming scenarios where you need to send records immediately as they become available.
  • Records are sent individually, making this endpoint suitable for real-time data streaming scenarios. However, for high-throughput scenarios, the "Put Record Batch" endpoint is more efficient as it allows sending up to 500 records in a single API call, reducing the number of API requests and improving performance.
  • Each record must be base64-encoded and can be up to 1,000 KB in size. The delivery stream will automatically handle data transformation, compression, and delivery to the configured destination (S3, Redshift, OpenSearch, etc.).

This endpoint is ideal for sending individual records in real-time. For batch operations, consider using the "Put Record Batch" endpoint for better performance.

Put Record Batch

This endpoint sends multiple records in a single batch to an AWS Kinesis Data Firehose delivery stream. Use this endpoint when you need to send multiple records efficiently in a single API call.

  • This endpoint allows you to send up to 500 records in a single batch request, improving efficiency for bulk data operations. Batching records reduces the number of API calls, lowers latency, and can reduce costs compared to sending records individually. The total size of all records in a batch cannot exceed 4 MB, and each individual record can be up to 1,000 KB.
  • The delivery stream name should be included in the request configuration. The delivery stream must exist and be in an active state before you can send records to it. You can verify the delivery stream status using the "List Delivery Streams" endpoint.
  • Each record in the batch must be base64-encoded. The API returns a response indicating which records were successfully processed and which failed, allowing you to retry failed records if needed.

Batch operations are more efficient than individual record sends when processing multiple records. This endpoint is recommended for high-throughput data ingestion scenarios.

Create Delivery Stream

This endpoint creates a new delivery stream in AWS Kinesis Data Firehose. Use this endpoint when you need to programmatically create delivery streams with specific configurations.

  • This endpoint requires delivery stream configuration details in the request body, including:

    • DeliveryStreamName: A unique name for the delivery stream (1-64 characters, alphanumeric and hyphens only)
    • DeliveryStreamType: The type of delivery stream (e.g., "DirectPut" for direct API ingestion)
    • S3DestinationConfiguration or other destination configuration: Specify where data should be delivered (S3, Redshift, OpenSearch, Splunk, etc.)
    • ExtendedS3DestinationConfiguration: For S3 destinations, includes bucket name, prefix, buffering settings, compression, and encryption options
  • The endpoint returns the delivery stream ARN (Amazon Resource Name) upon successful creation. The ARN uniquely identifies the delivery stream and is used in other API operations. Delivery stream creation may take a few minutes to complete, during which the stream status will be "CREATING".

Delivery streams must be configured with a destination (S3, Redshift, OpenSearch, etc.) before they can receive data. Ensure you have the necessary permissions to create delivery streams in your AWS account.

Update Destination

This endpoint updates the destination configuration of an existing delivery stream. Use this endpoint when you need to modify the destination settings for a delivery stream.

  • This endpoint allows you to update destination settings such as S3 bucket configuration, Redshift settings, or other destination parameters.
  • The delivery stream name and updated destination configuration should be included in the request.

Updating destination configuration may require the delivery stream to be in an active state. Review AWS Kinesis Data Firehose documentation for specific requirements.

Start Delivery Stream Encryption

This endpoint enables encryption for a delivery stream. Use this endpoint when you need to enable server-side encryption for data at rest in your delivery stream.

  • This endpoint enables encryption using AWS KMS for delivery stream data.
  • The delivery stream name and encryption configuration should be included in the request.

Encryption enhances data security for sensitive data. Ensure you have the necessary KMS permissions to enable encryption on your delivery streams.

Stop Delivery Stream Encryption

This endpoint disables encryption for a delivery stream. Use this endpoint when you need to remove encryption from a previously encrypted delivery stream.

  • This endpoint disables server-side encryption for the specified delivery stream.
  • The delivery stream name should be included in the request.

Disabling encryption may have security implications. Ensure this action aligns with your organization's security policies before proceeding.

Once the selected endpoint template has been configured, click the Test button to send a test payload to the AWS Kinesis Firehose API and verify that the destination is configured correctly before saving.

Manual configuration​

AWS Kinesis Firehose destinations can also be manually configured to send data to any valid AWS Kinesis Firehose API endpoint. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, data format, endpoint URL, request headers, attribute exclusions, record batching, and response webhooks.

AWS Kinesis Firehose API endpoints follow the pattern https://firehose.{region}.amazonaws.com/?Action={ActionName}. For update/upsert operations, include the ID of the object to be updated at the end of the URL. The API uses headers such as Content-Type:application/x-amz-json-1.1 and X-Amz-Target for versioning, which are typically handled automatically. When batching records, AWS Kinesis Firehose supports up to 500 records per batch (the default batch size is 100).

Save & activate​

Once all endpoint settings have been configured, click the Done button in the upper right corner of the screen to save and create the destination. To send the data to the configured AWS Kinesis Firehose delivery stream, open the destination resource menu, and select Activate.

The Nexset data will not be sent to the AWS Kinesis Firehose delivery stream until the destination is activated. Destinations can be activated immediately or at a later time, providing full control over data movement.