Skip to main content

Output Schema Validation

Output schema validation in Nexla ensures that data leaving your processing workflows conforms to expected structures and quality standards, providing confidence in data integrity and compatibility across your data pipeline.

Validation Overview​

Output schema validation acts as a quality gate for your data processing workflows, verifying that transformed and processed data meets your defined standards before it reaches downstream systems or destinations.

Core Validation Capabilities​

The output schema validation system provides several key capabilities for ensuring data quality and consistency.

Data Quality Assurance​

Ensure data meets quality standards:

  • Structure Validation: Verify data structure matches schema definitions
  • Type Validation: Ensure data types are correct and consistent
  • Constraint Validation: Validate business rules and constraints
  • Completeness Checks: Verify required fields are present and populated

Schema Compliance​

Ensure data conforms to schemas:

  • Schema Matching: Verify data matches expected schema structure
  • Field Validation: Validate individual field values and formats
  • Relationship Validation: Validate relationships between fields
  • Format Compliance: Ensure data formats meet specifications

Error Detection​

Identify and handle data quality issues:

  • Validation Errors: Detect schema compliance violations
  • Data Anomalies: Identify unexpected data patterns
  • Quality Issues: Flag data quality problems
  • Compliance Violations: Detect regulatory or business rule violations

Validation Types​

Nexla supports various validation types for different data quality requirements.

Structural Validation​

Validate data structure and organization:

  • Field Presence: Verify required fields are present
  • Field Order: Ensure fields appear in expected order
  • Nesting Structure: Validate nested object and array structures
  • Schema Compliance: Ensure data matches defined schemas

Content Validation​

Validate actual data content and values:

  • Data Type Validation: Verify data types match specifications
  • Range Validation: Validate numeric and date ranges
  • Pattern Validation: Enforce string patterns and formats
  • Business Rule Validation: Apply custom business logic

Quality Validation​

Ensure data quality and integrity:

  • Completeness: Verify data completeness and coverage
  • Accuracy: Validate data accuracy and correctness
  • Consistency: Ensure data consistency across records
  • Timeliness: Validate data freshness and relevance

Validation Configuration​

Configure validation rules and parameters for your specific requirements.

Validation Rules​

Define comprehensive validation rules:

{
  "validation_rules": {
    "strict_mode": true,
    "allow_unknown_fields": false,
    "validate_types": true,
    "validate_constraints": true,
    "custom_validators": [
      {
        "name": "email_format",
        "rule": "^[^@]+@[^@]+\\.[^@]+$"
      },
      {
        "name": "phone_format",
        "rule": "^\\+?[1-9]\\d{1,14}$"
      }
    ]
  }
}

Field Constraints​

Define field-specific validation constraints:

{
  "field_constraints": {
    "customer_id": {
      "required": true,
      "type": "string",
      "pattern": "^CUST-\\d{3,6}$",
      "min_length": 8,
      "max_length": 12
    },
    "email": {
      "required": true,
      "type": "string",
      "format": "email",
      "max_length": 100
    },
    "age": {
      "required": false,
      "type": "integer",
      "minimum": 0,
      "maximum": 120
    }
  }
}

Validation Endpoints​

Core API endpoints for output schema validation in your Nexla platform.

Validate Output Data​

Validate data against output schemas:

POST /schemas/{schema_id}/validate_output
Validate Output: Request
{
  "data": {
    "customer_id": "CUST-001",
    "first_name": "Jane",
    "last_name": "Smith",
    "email": "jane.smith@example.com",
    "registration_date": "2023-01-15",
    "status": "ACTIVE"
  },
  "validation_options": {
    "strict_mode": true,
    "include_details": true,
    "custom_rules": ["business_logic", "format_validation"]
  }
}

Validation Response​

Comprehensive validation results:

Validation Response
{
  "valid": true,
  "validation_summary": {
    "total_fields": 6,
    "valid_fields": 6,
    "invalid_fields": 0,
    "warnings": 0,
    "validation_time": "0.045s"
  },
  "field_validations": [
    {
      "field": "customer_id",
      "valid": true,
      "value": "CUST-001",
      "constraints_met": ["required", "pattern", "length"]
    },
    {
      "field": "first_name",
      "valid": true,
      "value": "Jane",
      "constraints_met": ["required", "type", "max_length"]
    },
    {
      "field": "last_name",
      "valid": true,
      "value": "Smith",
      "constraints_met": ["required", "type", "max_length"]
    },
    {
      "field": "email",
      "valid": true,
      "value": "jane.smith@example.com",
      "constraints_met": ["required", "format", "max_length"]
    },
    {
      "field": "registration_date",
      "valid": true,
      "value": "2023-01-15",
      "constraints_met": ["type", "format"]
    },
    {
      "field": "status",
      "valid": true,
      "value": "ACTIVE",
      "constraints_met": ["type", "enum"]
    }
  ],
  "quality_metrics": {
    "completeness": 1.0,
    "accuracy": 1.0,
    "consistency": 1.0,
    "overall_score": 100
  }
}

Batch Validation​

Validate multiple records simultaneously for efficient processing.

Batch Validation Endpoint​

Validate multiple data records:

POST /schemas/{schema_id}/validate_batch
Batch Validation: Request
{
  "records": [
    {
      "customer_id": "CUST-001",
      "first_name": "Jane",
      "last_name": "Smith",
      "email": "jane.smith@example.com"
    },
    {
      "customer_id": "CUST-002",
      "first_name": "John",
      "last_name": "Doe",
      "email": "john.doe@example.com"
    }
  ],
  "validation_options": {
    "strict_mode": false,
    "include_details": true,
    "stop_on_first_error": false
  }
}

Batch Validation Response​

Comprehensive batch validation results:

Batch Validation: Response
{
  "batch_summary": {
    "total_records": 2,
    "valid_records": 2,
    "invalid_records": 0,
    "validation_time": "0.089s"
  },
  "record_validations": [
    {
      "record_index": 0,
      "valid": true,
      "field_count": 4,
      "valid_fields": 4,
      "invalid_fields": 0
    },
    {
      "record_index": 1,
      "valid": true,
      "field_count": 4,
      "valid_fields": 4,
      "invalid_fields": 0
    }
  ],
  "quality_metrics": {
    "overall_completeness": 1.0,
    "overall_accuracy": 1.0,
    "overall_consistency": 1.0,
    "overall_score": 100
  }
}

Real-time Validation​

Integrate validation into your data processing workflows for immediate quality feedback.

Stream Validation​

Validate data streams in real-time:

POST /schemas/{schema_id}/validate_stream
Stream Validation: Request
{
  "stream_config": {
    "batch_size": 100,
    "validation_window": 1000,
    "error_threshold": 0.05
  },
  "validation_rules": {
    "strict_mode": false,
    "allow_partial": true,
    "real_time_alerts": true
  }
}

Validation Monitoring​

Monitor validation performance and quality:

Validation Monitoring: Response
{
  "stream_status": "ACTIVE",
  "validation_metrics": {
    "records_processed": 1250,
    "records_validated": 1250,
    "validation_rate": 125.5,
    "error_rate": 0.008,
    "average_validation_time": "0.042s"
  },
  "quality_trends": {
    "completeness_trend": "stable",
    "accuracy_trend": "improving",
    "consistency_trend": "stable"
  }
}

Validation Integration​

Integrate output schema validation with other Nexla components for comprehensive data quality management.

Nexset Integration​

Validate Nexset output data:

  • Output Validation: Validate data before leaving Nexsets
  • Quality Gates: Implement quality gates in processing workflows
  • Error Handling: Handle validation errors in processing logic
  • Quality Monitoring: Monitor data quality throughout processing

Transform Integration​

Validate transformed data:

  • Post-Transform Validation: Validate data after transformations
  • Transform Quality: Ensure transformations maintain data quality
  • Schema Evolution: Adapt validation to schema changes
  • Quality Assurance: Provide quality assurance for transformations

Flow Integration​

Validate data throughout flows:

  • Flow Validation: Validate data at flow checkpoints
  • Quality Propagation: Propagate quality metrics through flows
  • Error Propagation: Handle validation errors in flows
  • Quality Reporting: Report quality metrics for flows

Validation Best Practices​

To effectively implement output schema validation in your Nexla platform:

  1. Define Clear Schemas: Create comprehensive and clear schema definitions
  2. Implement Quality Gates: Use validation as quality gates in workflows
  3. Monitor Performance: Track validation performance and quality metrics
  4. Handle Errors Gracefully: Implement proper error handling for validation failures
  5. Iterate and Improve: Continuously improve validation rules based on results

Validation Workflows​

Implement structured workflows for effective output schema validation.

Validation Setup Workflow​

Standard workflow for setting up validation:

  1. Schema Definition: Define comprehensive output schemas
  2. Rule Configuration: Configure validation rules and constraints
  3. Integration Setup: Integrate validation into processing workflows
  4. Testing: Test validation with sample data
  5. Monitoring Setup: Set up validation monitoring and alerting

Validation Execution Workflow​

Workflow for executing validation:

  1. Data Preparation: Prepare data for validation
  2. Schema Selection: Select appropriate schema for validation
  3. Validation Execution: Execute validation against schemas
  4. Result Analysis: Analyze validation results and quality metrics
  5. Error Handling: Handle validation errors and issues
  6. Quality Reporting: Report validation results and quality metrics

Error Handling​

Common validation issues and solutions:

  • Schema Mismatches: Review and align data with schema definitions
  • Performance Issues: Optimize validation rules and processing
  • False Positives: Refine validation rules to reduce false positives
  • Integration Problems: Ensure proper integration with processing workflows

After implementing output schema validation, you may need to:

Monitor Quality​

GET /validation/quality
GET /validation/metrics

Manage Schemas​

GET /schemas
PUT /schemas/{schema_id}

Handle Errors​

GET /validation/errors
POST /validation/retry