Skip to main content

AWS Kinesis Firehose Destination

Nexla's bi-directional connectors allow data to flow both to and from any location, making it simple to create a FlexFlow, DB-CDC, Spark ETL, or Replication data flow that sends data to an AWS Kinesis Firehose location.
aws_kinesis_data_firehose_api.png

AWS Kinesis Firehose

Create an AWS Kinesis Firehose Destination​

  1. Click the + icon on the Nexset that will be sent to the AWS Kinesis Firehose destination, and select the Send to Destination option from the menu.

  2. Select the AWS Kinesis Firehose connector from the list of available destination connectors. Then, select the credential that will be used to connect to the AWS Kinesis Firehose organization, and click Next; or, create a new AWS Kinesis Firehose credential for use in this flow.

  3. In Nexla, AWS Kinesis Firehose destinations can be created using pre-built endpoint templates, which expedite destination setup for common AWS Kinesis Firehose endpoints. Each template is designed specifically for the corresponding AWS Kinesis Firehose endpoint, making destination configuration easy and efficient.
    • To configure this destination using a template, follow the instructions in Configure Using a Template.

    AWS Kinesis Firehose destinations can also be configured manually, allowing you to send data to AWS Kinesis Firehose endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this destination manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common AWS Kinesis Firehose endpoints. Each template is designed specifically for the corresponding AWS Kinesis Firehose endpoint, making destination setup easy and efficient.

  • To configure this destination using a template, select the endpoint to which data will be sent from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

    Put Record

    This endpoint sends individual records to an AWS Kinesis Data Firehose delivery stream. Use this endpoint when you need to send single records to a delivery stream for real-time data ingestion.

    • This endpoint sends individual data records to the specified delivery stream. The delivery stream name should be included in the request configuration. Each record is sent as a separate API call, which makes this endpoint suitable for real-time data streaming scenarios where you need to send records immediately as they become available.
    • Records are sent individually, making this endpoint suitable for real-time data streaming scenarios. However, for high-throughput scenarios, the "Put Record Batch" endpoint is more efficient as it allows sending up to 500 records in a single API call, reducing the number of API requests and improving performance.
    • Each record must be base64-encoded and can be up to 1,000 KB in size. The delivery stream will automatically handle data transformation, compression, and delivery to the configured destination (S3, Redshift, OpenSearch, etc.).

    This endpoint is ideal for sending individual records in real-time. For batch operations, consider using the "Put Record Batch" endpoint for better performance.

    Put Record Batch

    This endpoint sends multiple records in a single batch to an AWS Kinesis Data Firehose delivery stream. Use this endpoint when you need to send multiple records efficiently in a single API call.

    • This endpoint allows you to send up to 500 records in a single batch request, improving efficiency for bulk data operations. Batching records reduces the number of API calls, lowers latency, and can reduce costs compared to sending records individually. The total size of all records in a batch cannot exceed 4 MB, and each individual record can be up to 1,000 KB.
    • The delivery stream name should be included in the request configuration. The delivery stream must exist and be in an active state before you can send records to it. You can verify the delivery stream status using the "List Delivery Streams" endpoint.
    • Each record in the batch must be base64-encoded. The API returns a response indicating which records were successfully processed and which failed, allowing you to retry failed records if needed.

    Batch operations are more efficient than individual record sends when processing multiple records. This endpoint is recommended for high-throughput data ingestion scenarios.

    Create Delivery Stream

    This endpoint creates a new delivery stream in AWS Kinesis Data Firehose. Use this endpoint when you need to programmatically create delivery streams with specific configurations.

    • This endpoint requires delivery stream configuration details in the request body, including:

      • DeliveryStreamName: A unique name for the delivery stream (1-64 characters, alphanumeric and hyphens only)
      • DeliveryStreamType: The type of delivery stream (e.g., "DirectPut" for direct API ingestion)
      • S3DestinationConfiguration or other destination configuration: Specify where data should be delivered (S3, Redshift, OpenSearch, Splunk, etc.)
      • ExtendedS3DestinationConfiguration: For S3 destinations, includes bucket name, prefix, buffering settings, compression, and encryption options
    • The endpoint returns the delivery stream ARN (Amazon Resource Name) upon successful creation. The ARN uniquely identifies the delivery stream and is used in other API operations. Delivery stream creation may take a few minutes to complete, during which the stream status will be "CREATING".

    Delivery streams must be configured with a destination (S3, Redshift, OpenSearch, etc.) before they can receive data. Ensure you have the necessary permissions to create delivery streams in your AWS account.

    Update Destination

    This endpoint updates the destination configuration of an existing delivery stream. Use this endpoint when you need to modify the destination settings for a delivery stream.

    • This endpoint allows you to update destination settings such as S3 bucket configuration, Redshift settings, or other destination parameters.
    • The delivery stream name and updated destination configuration should be included in the request.

    Updating destination configuration may require the delivery stream to be in an active state. Review AWS Kinesis Data Firehose documentation for specific requirements.

    Start Delivery Stream Encryption

    This endpoint enables encryption for a delivery stream. Use this endpoint when you need to enable server-side encryption for data at rest in your delivery stream.

    • This endpoint enables encryption using AWS KMS for delivery stream data.
    • The delivery stream name and encryption configuration should be included in the request.

    Encryption enhances data security for sensitive data. Ensure you have the necessary KMS permissions to enable encryption on your delivery streams.

    Stop Delivery Stream Encryption

    This endpoint disables encryption for a delivery stream. Use this endpoint when you need to remove encryption from a previously encrypted delivery stream.

    • This endpoint disables server-side encryption for the specified delivery stream.
    • The delivery stream name should be included in the request.

    Disabling encryption may have security implications. Ensure this action aligns with your organization's security policies before proceeding.

Configure Manually​

AWS Kinesis Firehose destinations can be manually configured to send data to any valid AWS Kinesis Firehose API endpoint.

Using manual configuration, you can also configure Nexla to automatically send the response received from the AWS Kinesis Firehose API after each call to a new Nexla webhook data source.

API Method​

  1. To manually configure this destination, select the Advanced tab at the top of the configuration screen.

  2. Select the API method that will be used for calls to the AWS Kinesis Firehose API from the Method pulldown menu.

Data Format​

  1. Select the format in which the Nexset data will be sent to the AWS Kinesis Firehose API from the Content Format pulldown menu. Nexla will automatically convert the data to the selected format for each API call.

API Endpoint URL​

  1. Enter the URL of the AWS Kinesis Firehose API endpoint to which you want to send the Nexset data in the URL field. AWS Kinesis Firehose API endpoints typically follow the pattern https://firehose.{region}.amazonaws.com/?Action={ActionName}. For update/upsert operations, include the ID of the object to be updated at the end of the URL.

Request Headers​

Optional

  • If Nexla should include any additional request headers in API calls to this destination, enter the headers & corresponding values as comma-separated pairs in the Request Headers field (e.g., header1:value1,header2:value2).

    You do not need to include any headers already present in the credentials. AWS Kinesis Firehose API uses specific headers like Content-Type:application/x-amz-json-1.1 and X-Amz-Target for API versioning, which are typically handled automatically.

Exclude Attributes from the Call​

Optional

  • If any record attributes in the Nexset should be omitted when sending data to this AWS Kinesis Firehose destination, select the attributes from the Exclude Attributes pulldown menu.

  • Any number of attributes can be selected for exclusion, and all excluded attributes will be shown in the field. To remove an attribute from the list, click the X icon next to the attribute name.

Record Batching​

Optional

  1. If records should be sent to this destination in batched API calls, check the box next to Would you like to batch your records together? to enable record batching.

  2. Enter the maximum number of records that should be batched together in a single API call in the Batch Size field. By default, this value is set to 100. For AWS Kinesis Firehose, batches can contain up to 500 records.

  3. Select the algorithm that will be used to group records into batches from the Grouping Algorithm pulldown menu. The sample request shown in the panel on the right will be updated to reflect the current batching settings. Some algorithms require additional settings—click on an algorithm listed below to view instructions for configuring these settings.

Response Webhook​

Optional

Nexla can automatically send the response received from the AWS Kinesis Firehose API after each call to a new Nexla webhook data source. This option allows you to keep track of the status of each API call and any additional information returned after each call.

  • To enable this option, check the box next to Would you like to process the API response as a Nexla Webhook source?.

Sample Request Payload​

Sample request payloads containing a portion of the Nexset data that will be sent to the AWS Kinesis Firehose API endpoint based on the current settings are shown in the Sample Payload panel on the right. These samples can be referenced to ensure that the destination and request settings are correctly configured.

  • Click on a sample request payload to expand it and view the complete payload content.

  • Sample payloads are automatically updated with each setting change, making it easy to verify that changes achieve the desired effect.

Endpoint Testing (Manual Configuration)​

After all endpoint settings have been configured, Nexla can send a test payload to the AWS Kinesis Firehose API to ensure that the destination is configured correctly.

  1. To send a test payload, select the Test button at the top of the Sample Payload panel, and click on a listed sample payload to expand it.

  2. If any modifications to the sample payload are needed, make the necessary changes directly within the sample window.

  3. Click the Send Test Data button at the top of a sample payload to send the test payload to the AWS Kinesis Firehose API using the current settings.

Save & Activate the Destination​

  • Once all endpoint settings have been configured, click the Done button in the upper right corner of the screen to save and create the destination. To send the data to the configured AWS Kinesis Firehose delivery stream, open the destination resource menu, and select Activate.

    The Nexset data will not be sent to the AWS Kinesis Firehose delivery stream until the destination is activated. Destinations can be activated immediately or at a later time, providing full control over data movement.