Skip to main content

Control Data Output

Controlling data output in Nexla allows you to manage when and how data is written to destinations. You can activate, pause, and validate destinations to ensure optimal data flow control and configuration integrity.

Activate and Pause Destination​

You can control data output by activating or pausing destinations. This method is better suited for scenarios where the same Nexset is configured for writing to multiple destinations, allowing you to control output independently.

Activation Endpoint​

To activate a destination and start data output:

Activate Destination: Request
PUT /data_sinks/{data_sink_id}/activate

Pause Endpoint​

To pause a destination and stop data output:

Pause Destination: Request
PUT /data_sinks/{data_sink_id}/pause

Control Behavior​

When you activate or pause a destination:

  • Activation: Starts data flow to the destination, writing processed data according to the configuration
  • Pause: Immediately stops data output while preserving the destination configuration
  • Status Change: The destination status field updates to reflect the current operational state
  • Flow Impact: Affects only the specific destination, not the associated Nexset or other destinations

Validate Destination Configuration​

Destination configuration validation ensures that all required parameters are properly set for successful data output. This is crucial for destinations with complex configurations that require specific parameters.

Validation Endpoint​

To validate a destination's configuration:

POST /data_sinks/{data_sink_id}/config/validate

Validation Process​

The validation process checks:

  • Required Parameters: Ensures all mandatory configuration fields are present
  • Parameter Values: Validates that parameter values are within acceptable ranges
  • Credential Compatibility: Verifies credentials work with the destination type
  • Configuration Consistency: Checks for conflicting or incompatible settings

Validation Response​

The validation response provides detailed feedback about configuration issues and recommendations.

Validate Destination Configuration: Response
{
  "status": "ok",
  "output": [
    {
      "name": "credsEnc",
      "value": null,
      "errors": [
        "Missing required configuration \"credsEnc\" which has no default value."
      ],
      "visible": true,
      "recommendedValues": []
    },
    {
      "name": "credsEncIv",
      "value": null,
      "errors": [
        "Missing required configuration \"credsEncIv\" which has no default value."
      ],
      "visible": true,
      "recommendedValues": []
    },
    {
      "name": "sink_type",
      "value": null,
      "errors": [
        "Missing required configuration \"sink_type\" which has no default value.",
        "Invalid value null for configuration sink_type: Invalid enumerator"
      ],
      "visible": true,
      "recommendedValues": []
    }
  ]
}

Test Destination Connection​

You can test the connection to a destination to verify that credentials and configuration are working correctly. This testing helps ensure that your destination is properly configured and can successfully write data before activating production flows.

Connection Test Endpoint​

To test destination connectivity and authentication:

PUT /data_sinks/{data_sink_id}/probe/authenticate

Connection Test Benefits​

Testing destination connections helps:

  • Verify Credentials: Ensure authentication information is correct
  • Check Permissions: Confirm the destination is accessible
  • Validate Configuration: Test that sink configuration parameters work
  • Prevent Failures: Identify issues before activating production flows
  • Network Validation: Confirm connectivity to external systems
  • Credential Testing: Verify stored credentials are still valid

Connection Test Response​

The connection test endpoint returns a response indicating the success or failure of the connection attempt:

Successful Connection Test: Response
{
  "status": "ok",
  "message": "Connection successful",
  "output": {
    "connection_type": "s3",
    "destination": "s3://customer-bucket/nexla-outputs",
    "credentials_valid": true,
    "permissions_verified": true,
    "configuration_valid": true
  }
}
Failed Connection Test: Response
{
  "status": "error",
  "message": "Authentication failed",
  "output": {
    "connection_type": "s3",
    "destination": "s3://customer-bucket/nexla-outputs",
    "credentials_valid": false,
    "permissions_verified": false,
    "configuration_valid": true,
    "error_details": "Invalid access key or secret key"
  }
}

Alternative Testing Methods​

For comprehensive destination testing, you can also use related endpoints:

Test Credentials Directly​

Test the underlying credentials used by the destination:

GET /data_credentials/{credential_id}/probe/authenticate

Validate Configuration​

Test destination configuration without attempting connection:

POST /data_sinks/{data_sink_id}/config/validate

Test File System Access​

For file-based destinations, test path accessibility:

POST /data_credentials/{credential_id}/probe/tree

Control Scenarios​

Common scenarios for controlling data output:

Multiple Destination Management​

When a Nexset writes to multiple destinations, you can:

  • Activate specific destinations based on business needs
  • Pause destinations during maintenance or troubleshooting
  • Control output timing for different target systems

Scheduled Operations​

For destinations with specific timing requirements:

  • Activate destinations before scheduled data processing
  • Pause destinations during maintenance windows
  • Control output based on business hours or data availability

Troubleshooting​

During issue investigation:

  • Pause destinations to stop data flow temporarily
  • Validate configurations to identify problems
  • Test connections to isolate credential or network issues

Best Practices​

To effectively control data output:

  1. Validate Before Activation: Always validate configuration before activating destinations
  2. Test Connections: Verify connectivity before starting production flows
  3. Monitor Status: Track destination status to ensure operational health
  4. Use Pause Strategically: Pause destinations rather than deleting them for temporary stops
  5. Document Changes: Keep records of activation/pause events for audit purposes

Error Handling​

Common control issues and solutions:

  • Activation Failures: Check destination configuration and credentials
  • Pause Issues: Verify destination is currently active
  • Validation Errors: Address missing or invalid configuration parameters
  • Connection Failures: Check network connectivity and credential validity
  • Permission Issues: Ensure your account has control permissions

After controlling destinations, you may need to:

Monitor Performance​

GET /data_sinks/{destination_id}/flow/status_metrics

View Flow Status​

GET /data_sinks/{destination_id}/flow

Check Audit Log​

GET /data_sinks/{destination_id}/audit_log

Update Configuration​

PUT /data_sinks/{destination_id}