Create Transform

Creating transforms in Nexla allows you to define custom data modification logic that can be applied to your data processing workflows, enabling you to implement business-specific transformations and data quality improvements.

Transform Creation Overview

Transform creation involves defining the structure, logic, and configuration for data transformation functions that can be reused across different Nexsets and data flows in your Nexla platform.

Core Creation Capabilities

The transform creation system provides several key capabilities for building effective data transformation functions.

Function Definition

Define transformation logic and behavior:

• Input Processing: Handle various input data types and structures
• Transformation Logic: Implement custom transformation algorithms
• Output Generation: Produce transformed data in expected formats
• Error Handling: Implement robust error handling and recovery

Configuration Management

Manage transform parameters and settings:

• Parameter Definition: Define configurable parameters and options
• Default Values: Set appropriate default values for parameters
• Validation Rules: Implement parameter validation and constraints
• Environment Support: Support different environment configurations

Integration Support

Ensure seamless integration with workflows:

• Schema Compatibility: Ensure compatibility with input/output schemas
• Performance Optimization: Optimize for processing efficiency
• Resource Management: Manage memory and processing resources
• Monitoring Support: Support performance monitoring and debugging

Transform Types

Understanding different transform types helps you choose the right approach for your data modification needs.

Language-based Transforms

Transforms implemented in specific programming languages:

• JavaScript Transforms: Web-standard language for data manipulation
• Python Transforms: Python-based transformation scripts
• Java Transforms: Java-based transformation functions

Function-based Transforms

Transforms based on function types and categories:

• String Transforms: Text manipulation and formatting
• Numeric Transforms: Mathematical operations and calculations
• Date Transforms: Date and time manipulation
• Logical Transforms: Boolean operations and conditional logic

Specialized Transforms

Domain-specific transformation functions:

• Data Quality Transforms: Data cleaning and validation
• Business Logic Transforms: Industry-specific business rules
• Integration Transforms: System integration and mapping
• Compliance Transforms: Regulatory and compliance requirements

Create Transform Endpoint

To create a new transform:

POST /transforms

Nexla API

Create Transform: Request

{
  "name": "Customer Data Enricher",
  "description": "Enrich customer data with demographic information",
  "type": "enrichment",
  "category": "business_logic",
  "input_schema": {
    "type": "object",
    "properties": {
      "customer_id": {"type": "string"},
      "first_name": {"type": "string"},
      "last_name": {"type": "string"},
      "email": {"type": "string"},
      "zip_code": {"type": "string"}
    },
    "required": ["customer_id", "first_name", "last_name", "email"]
  },
  "output_schema": {
    "type": "object",
    "properties": {
      "customer_id": {"type": "string"},
      "first_name": {"type": "string"},
      "last_name": {"type": "string"},
      "email": {"type": "string"},
      "zip_code": {"type": "string"},
      "full_name": {"type": "string"},
      "email_domain": {"type": "string"},
      "region": {"type": "string"},
      "enrichment_score": {"type": "number"}
    }
  },
  "transform_function": {
    "language": "javascript",
    "code": "function transform(input) { const fullName = `${input.first_name} ${input.last_name}`; const emailDomain = input.email.split('@')[1]; const region = getRegionFromZip(input.zip_code); const score = calculateEnrichmentScore(input); return { ...input, full_name: fullName, email_domain: emailDomain, region: region, enrichment_score: score }; }"
  },
  "configuration": {
    "enrichment_enabled": true,
    "default_region": "Unknown",
    "score_threshold": 0.7
  },
  "metadata": {
    "owner": "data_team",
    "tags": ["customer", "enrichment", "demographics"],
    "version": "1.0.0"
  }
}

Create TransformResponse

A successful creation returns the new transform:

Nexla API

Create Transform: Response

{
  "id": 6001,
  "name": "Customer Data Enricher",
  "description": "Enrich customer data with demographic information",
  "type": "enrichment",
  "category": "business_logic",
  "status": "ACTIVE",
  "version": "1.0.0",
  "created_at": "2023-01-16T10:00:00.000Z",
  "updated_at": "2023-01-16T10:00:00.000Z",
  "owner": {
    "id": 42,
    "name": "John Doe"
  },
  "input_schema": {
    "type": "object",
    "properties": {
      "customer_id": {"type": "string"},
      "first_name": {"type": "string"},
      "last_name": {"type": "string"},
      "email": {"type": "string"},
      "zip_code": {"type": "string"}
    },
    "required": ["customer_id", "first_name", "last_name", "email"]
  },
  "output_schema": {
    "type": "object",
    "properties": {
      "customer_id": {"type": "string"},
      "first_name": {"type": "string"},
      "last_name": {"type": "string"},
      "email": {"type": "string"},
      "zip_code": {"type": "string"},
      "full_name": {"type": "string"},
      "email_domain": {"type": "string"},
      "region": {"type": "string"},
      "enrichment_score": {"type": "number"}
    }
  },
  "transform_function": {
    "language": "javascript",
    "code": "function transform(input) { const fullName = `${input.first_name} ${input.last_name}`; const emailDomain = input.email.split('@')[1]; const region = getRegionFromZip(input.zip_code); const score = calculateEnrichmentScore(input); return { ...input, full_name: fullName, email_domain: emailDomain, region: region, enrichment_score: score }; }"
  },
  "configuration": {
    "enrichment_enabled": true,
    "default_region": "Unknown",
    "score_threshold": 0.7
  },
  "metadata": {
    "owner": "data_team",
    "tags": ["customer", "enrichment", "demographics"],
    "version": "1.0.0"
  },
  "usage_count": 0,
  "performance_metrics": {
    "average_execution_time": null,
    "success_rate": null,
    "error_rate": null
  }
}

Transform Creation Examples

Different examples of transform creation for various use cases.

String Processing Transform

Create a transform for string manipulation:

Nexla API

String Processing Transform: Request

{
  "name": "Address Normalizer",
  "description": "Normalize and standardize address formats",
  "type": "string",
  "category": "data_quality",
  "input_schema": {
    "type": "object",
    "properties": {
      "street_address": {"type": "string"},
      "city": {"type": "string"},
      "state": {"type": "string"},
      "zip_code": {"type": "string"}
    }
  },
  "output_schema": {
    "type": "object",
    "properties": {
      "street_address": {"type": "string"},
      "city": {"type": "string"},
      "state": {"type": "string"},
      "zip_code": {"type": "string"},
      "normalized_address": {"type": "string"},
      "address_score": {"type": "number"}
    }
  },
  "transform_function": {
    "language": "javascript",
    "code": "function transform(input) { const normalized = normalizeAddress(input); const score = calculateAddressScore(input); return { ...input, normalized_address: normalized, address_score: score }; }"
  }
}

Numeric Calculation Transform

Create a transform for mathematical operations:

Nexla API

Numeric Calculation Transform: Request

{
  "name": "Financial Calculator",
  "description": "Calculate financial metrics and ratios",
  "type": "numeric",
  "category": "business_logic",
  "input_schema": {
    "type": "object",
    "properties": {
      "revenue": {"type": "number"},
      "costs": {"type": "number"},
      "assets": {"type": "number"},
      "liabilities": {"type": "number"}
    }
  },
  "output_schema": {
    "type": "object",
    "properties": {
      "revenue": {"type": "number"},
      "costs": {"type": "number"},
      "assets": {"type": "number"},
      "liabilities": {"type": "number"},
      "profit_margin": {"type": "number"},
      "debt_to_equity": {"type": "number"},
      "return_on_assets": {"type": "number"}
    }
  },
  "transform_function": {
    "language": "javascript",
    "code": "function transform(input) { const profitMargin = ((input.revenue - input.costs) / input.revenue) * 100; const debtToEquity = input.liabilities / (input.assets - input.liabilities); const roa = ((input.revenue - input.costs) / input.assets) * 100; return { ...input, profit_margin: profitMargin, debt_to_equity: debtToEquity, return_on_assets: roa }; }"
  }
}

Transform Configuration

Configure transform parameters and behavior for optimal performance and functionality.

Basic Configuration

Essential configuration parameters:

{
  "configuration": {
    "enabled": true,
    "timeout": 30000,
    "retry_count": 3,
    "batch_size": 1000,
    "parallelism": 4
  }
}

Advanced Configuration

Advanced configuration options:

{
  "configuration": {
    "performance": {
      "memory_limit": "512MB",
      "cpu_limit": "2",
      "cache_enabled": true,
      "cache_ttl": 3600
    },
    "monitoring": {
      "metrics_enabled": true,
      "logging_level": "INFO",
      "alert_threshold": 0.95
    },
    "security": {
      "input_validation": true,
      "output_sanitization": true,
      "sandbox_mode": false
    }
  }
}

Post-Creation Steps

Important steps to take after creating a transform.

Testing and Validation

Ensure transform works correctly:

• Unit Testing: Test transform with various input scenarios
• Integration Testing: Test transform within Nexset workflows
• Performance Testing: Validate transform performance characteristics
• Error Testing: Test error handling and edge cases

Documentation and Deployment

Prepare transform for production use:

• Usage Documentation: Document transform usage and examples
• Configuration Guide: Provide configuration parameter documentation
• Deployment: Deploy transform to production environment
• Monitoring Setup: Set up performance monitoring and alerting

Transform Creation Best Practices

To effectively create transforms in your Nexla platform:

1. Design for Reusability: Create transforms that can be reused across workflows
2. Implement Error Handling: Include robust error handling and recovery logic
3. Optimize Performance: Design transforms for efficient processing
4. Document Thoroughly: Provide comprehensive documentation and examples
5. Test Extensively: Test transforms with various input scenarios

Transform Creation Workflows

Implement structured workflows for effective transform creation.

Development Workflow

Standard workflow for transform development:

1. Requirements Analysis: Analyze transformation requirements and use cases
2. Design: Design transform structure, logic, and configuration
3. Implementation: Implement transform function and error handling
4. Testing: Test transform with various input scenarios
5. Documentation: Document transform usage and configuration
6. Review: Review transform for quality and compliance
7. Deployment: Deploy transform to appropriate environment

Quality Assurance Workflow

Workflow for ensuring transform quality:

1. Code Review: Review transform code and logic
2. Testing: Execute comprehensive testing scenarios
3. Performance Analysis: Analyze and optimize performance
4. Security Review: Review security implications and vulnerabilities
5. Documentation Review: Ensure documentation completeness and accuracy

Error Handling

Common transform creation issues and solutions:

• Schema Validation: Ensure input/output schemas are properly defined
• Code Quality: Implement proper error handling and validation
• Performance Issues: Optimize transform logic for efficiency
• Integration Problems: Verify proper integration with workflows

Related Operations

After creating transforms, you may need to:

Test Transforms

POST /transforms/{transform_id}/test
POST /transforms/{transform_id}/validate

Deploy Transforms

PUT /transforms/{transform_id}/deploy
POST /transforms/{transform_id}/activate

Monitor Performance

GET /transforms/{transform_id}/performance
GET /transforms/{transform_id}/metrics