Skip to main content

Transforms

Transforms are resources that specify how incoming records from data sets should be transformed into outgoing records. Built-in Nexla transform operations allow for a wide variety of modifications, including structural changes to record format and data value modification. Custom transforms allow for extending the system to support application-specific data changes.

Transform resources can be accessed directly through the following endpoints:

EndpointResourceDescription
/transformsRecord-level transformsTransforms that are, or can be, associated with specific data sets
/attribute_transformsAttribute-level transformsTransforms that can be referenced in record-level transforms to generate specific attributes
/code_containersAll types of code containers, including transformsThe resource_type attribute of transform code container is set to "transform"

List All Transforms​

Use the methods below to list all of the user's transform resources and their current transform versions. By default, only reusable transforms are included in the response. You must set the reusable query parameter to all to include non-reusable transforms in the response, or to '1' or '0' to filter on reusable or non-reusable, respectively.

List All Transforms: Request
GET /transforms?reusable=all
List All Transforms: Response
[
  {
    "id": 10001,
    "name": "Data set 5085 (Transform)",
    "resource_type": "transform",
    "reusable": false,
    "owner": {
      "id": 2,
      "full_name": "Jeff Williams",
      "email": "jcw@nexla.com"
    },
    "org": {
      "id": 1,
      "name": "Nexla",
      "email_domain": "nexla.com",
      "email": null
    },
    "access_roles": ["owner"],
    "data_credentials": null,
    "description": null,
    "code_type": "jolt_standard",
    "output_type": "record",
    "code_config": {},
    "code_encoding": "none",
    "code": [
      {
        "operation": "shift",
        "spec": {
          "a": "A",
          "b": "B",
          "c": "C"
        }
      }
    ],
    "data_sets": [5085],
    "updated_at": "2018-06-01T03:15:06.000Z",
    "created_at": "2018-06-01T03:15:06.000Z"
  },
  {
    "id": 10002,
    "name": "Reference Data Set 1 (Transform)",
    "resource_type": "transform",
    "reusable": false,
    "owner": {
      "id": 2,
      "full_name": "Jeff Williams",
      "email": "jcw@nexla.com"
    },
    "org": {
      "id": 1,
      "name": "Nexla",
      "email_domain": "nexla.com",
      "email": null
    },
    "access_roles": ["owner"],
    "data_credentials": null,
    "description": "Pre-canned data set for reference data source. (Transform)",
    "code_type": "jolt_standard",
    "output_type": "record",
    "code_config": {},
    "code_encoding": "none",
    "code": [],
    "data_sets": [5083],
    "updated_at": "2018-06-01T03:15:06.000Z",
    "created_at": "2018-06-01T03:15:06.000Z"
  },
  {
    "id": 10003,
    "name": "Data set  (Transform)",
    "resource_type": "transform",
    "reusable": true,
    "owner": {
      "id": 2,
      "full_name": "Jeff Williams",
      "email": "jcw@nexla.com"
    },
    "org": {
      "id": 1,
      "name": "Nexla",
      "email_domain": "nexla.com",
      "email": null
    },
    "access_roles": ["owner"],
    "data_credentials": null,
    "description": null,
    "code_type": "jolt_standard",
    "output_type": "record",
    "code_config": {},
    "code_encoding": "none",
    "code": [
      {
        "operation": "shift",
        "spec": {
          "a": "A",
          "b": "B",
          "c": "C",
          "f": "F"
        }
      }
    ],
    "data_sets": [5089],
    "updated_at": "2018-06-01T16:07:51.000Z",
    "created_at": "2018-06-01T03:19:24.000Z"
  }
]

Show One Transform​

Use the method below to fetch a single transform.

Show One Transform: Request
GET /transforms/{transform_id}
Show One Transform: Response
{
  "id": 10003,
  "name": "Data set  (Transform)",
  "resource_type": "transform",
  "reusable": true,
  "owner": {
    "id": 2,
    "full_name": "Jeff Williams",
    "email": "jcw@nexla.com"
  },
  "org": {
    "id": 1,
    "name": "Nexla",
    "email_domain": "nexla.com",
    "email": null
  },
  "access_roles": ["owner"],
  "data_credentials": null,
  "description": null,
  "code_type": "jolt_standard",
  "output_type": "record",
  "code_config": {},
  "code_encoding": "none",
  "code": [
    {
      "operation": "shift",
      "spec": {
        "a": "A",
        "b": "B",
        "c": "C",
        "f": "F"
      }
    }
  ],
  "data_sets": [5089],
  "updated_at": "2018-06-01T16:07:51.000Z",
  "created_at": "2018-06-01T03:19:24.000Z"
}

Create a Transform​

Use the method below to create a transform. Transforms can be writtewn in any of the languages supported by Nexla.

Create Transform: Request
POST /transforms

...
Example POST payload
{
  "name": "TEST standalone transform",
  "description": "Use it everywhere!",
  "reusable": true,
  "code_type": "jolt_standard",
  "output_type": "record",
  "code": [
    {
      "operation": "shift",
      "spec": {
        "a": "AAAAAA",
        "b": "BBBBBB",
        "c": "CCCCCC",
        "f": "FFFFFF"
      }
    }
  ]
}
Create Transform: Response
{
  "id": 10008,
  "name": "TEST standalone transform",
  "resource_type": "transform",
  "reusable": true,
  "owner": {
    "id": 2,
    "full_name": "Jeff Williams",
    "email": "jcw@nexla.com"
  },
  "org": {
    "id": 1,
    "name": "Nexla",
    "email_domain": "nexla.com",
    "email": null
  },
  "access_roles": ["owner"],
  "data_credentials": null,
  "description": "Use it everywhere!",
  "code_type": "jolt_standard",
  "output_type": "record",
  "code_config": {},
  "code_encoding": "none",
  "code": [
    {
      "operation": "shift",
      "spec": {
        "a": "AAAAAA",
        "b": "BBBBBB",
        "c": "CCCCCC",
        "f": "FFFFFF"
      }
    }
  ],
  "updated_at": "2018-06-01T16:28:54.989Z",
  "created_at": "2018-06-01T16:28:54.989Z"
}

Update A Transform​

Use the method below to update a transform. Transforms can be writtewn in any of the languages supported by Nexla.

Update Transform: Request
PUT /transforms/{transform_id}
...
Example Request Body
{
  "code": [
    {
      "operation": "shift",
      "spec": {
        "a": "W",
        "b": "X",
        "c": "Y",
        "f": "Z"
      }
    }
  ]
}

Update Transform: Response
{
  "id": 10008,
  "name": "TEST standalone transform",
  "resource_type": "transform",
  "reusable": true,
  "owner": {
    "id": 2,
    "full_name": "Jeff Williams",
    "email": "jcw@nexla.com"
  },
  "org": {
    "id": 1,
    "name": "Nexla",
    "email_domain": "nexla.com",
    "email": null
  },
  "access_roles": ["owner"],
  "data_credentials": null,
  "description": "Use it everywhere!",
  "code_type": "jolt_standard",
  "output_type": "record",
  "code_config": {},
  "code_encoding": "none",
  "code": [
    {
      "operation": "shift",
      "spec": {
        "a": "W",
        "b": "X",
        "c": "Y",
        "f": "Z"
      }
    }
  ],
  "updated_at": "2018-06-01T16:30:05.163Z",
  "created_at": "2018-06-01T16:28:54.989Z"
}

Associate A Reusable Transform​

You can associate a reusable transform with a dataset. Note that the transform being associated must be reusable.

Associate a reusable Transform: Request
PUT /data_sets/{data_set_id}
...
{
  "transform_id" : 10010
}
Associate a reusable Transform: Response
{
  "id": 5095,
  "owner": {
    "id": 2,
    "full_name": "Jeff Williams",
    "email": "jcw@nexla.com"
  },
  "org": {
    "id": 1,
    "name": "Nexla",
    "email_domain": "nexla.com",
    "email": null
  },
  "version": 1,
  "name": null,
  ...
  "parent_data_set": {
    "id": 5084,
    ...
  },
  "has_custom_transform": true,
  "output_schema_validation_enabled": false,
  "transform_id": 10010,
  "transform": {
    "version": 1,
    "data_maps": [],
    "transforms": [
      {
        "operation": "shift",
        "spec": {
          "X": "XX",
          "Y": "YY"
        }
      }
    ],
    "custom": true
  },
  "output_schema": {
    "properties": {
      "XX": {
        "type": "number"
      },
      "YY": {
        "type": "string"
      }
    },
    "type": "object",
    "$schema": "http://json-schema.org/draft-04/schema#",
    "$schema-id": 1092694496
  },
  "updated_at": "2018-06-04T04:43:23.394Z",
  "created_at": "2018-06-04T02:06:54.000Z",
  "tags": []
}

Dissociate a Resusable Transform​

Use the method below to stop using a particular transform with a data set. If no additional transform attribute is included in the request, the data set will be assigned an empty transform. If a transform is included, a new, non-reusable transform will be automatically created and associated with the data set.

Disassociate A Reusable Transform: Request
PUT /data_sets/{data_set_id}
...
Example Request Body: 1
{
  "transform_id" : null
}

...
Example Request Body: 2
...
{
  "transform_id": null,
  "transform": {
    "version": 1,
    "data_maps": [],
    "transforms": [
      {
        "operation": "shift",
        "spec": {
          "X": "EX",
          "Y": "WHY"
        }
      }
    ]
  }
}
Disassociate A Reusable Transform: Response
{
  "id": 5095,
  "owner": {
    "id": 2,
    "full_name": "Jeff Williams",
    "email": "jcw@nexla.com"
  },
  "org": {
    "id": 1,
    "name": "Nexla",
    "email_domain": "nexla.com",
    "email": null
  },
  "version": 1,
  "name": null,
  ...
  "parent_data_set": {
    "id": 5084,
    ...
  },
  "has_custom_transform": true,
  "output_schema_validation_enabled": false,
  "transform_id": 10011,
  "transform": {
    "version": 1,
    "data_maps": [],
    "transforms": [
      {
        "operation": "shift",
        "spec": {
          "X": "EX",
          "Y": "WHY"
        }
      }
    ],
    "custom": false
  },
  "output_schema": {
    "properties": {
      "EX": {
        "type": "number"
      },
      "WHY": {
        "type": "string"
      }
    },
    "type": "object",
    "$schema": "http://json-schema.org/draft-04/schema#",
    "$schema-id": 1092696543
  },
  "updated_at": "2018-06-04T05:43:23.394Z",
  "created_at": "2018-06-04T05:06:54.000Z",
  "tags": []
}

Associate and Update Transform​

If transform_id and transform attributes are both included in the call to update a dataset, Nexla will apply the change to transform_id first, and then update its contents with the value transform code passed in transform. Note that the transform specified in transform_id must already exist and be reusable.

Associate and Update a Transform: Request
PUT /data_sets/{data_set_id}
...
Example Request Body
{
  "transform_id": 10011,
  "transform": {
    "version": 1,
    "data_maps": [],
    "transforms": [
      {
        "operation": "shift",
        "spec": {
          "X": "ex",
          "Y": "why"
        }
      }
    ]
  }
}

Associate and Update a Transform: Response
{
  "id": 5095,
  "owner": {
    "id": 2,
    "full_name": "Jeff Williams",
    "email": "jcw@nexla.com"
  },
  "org": {
    "id": 1,
    "name": "Nexla",
    "email_domain": "nexla.com",
    "email": null
  },
  "version": 1,
  "name": null,
  ...
  "parent_data_set": {
    "id": 5084,
    ...
  },
  "has_custom_transform": true,
  "output_schema_validation_enabled": false,
  "transform_id": 10011,
  "transform": {
    "version": 1,
    "data_maps": [],
    "transforms": [
      {
        "operation": "shift",
        "spec": {
          "X": "ex",
          "Y": "why"
        }
      }
    ]
  },
  "output_schema": {
    "properties": {
      "EX": {
        "type": "number"
      },
      "WHY": {
        "type": "string"
      }
    },
    "type": "object",
    "$schema": "http://json-schema.org/draft-04/schema#",
    "$schema-id": 1092696543
  },
  "updated_at": "2018-06-04T05:43:23.394Z",
  "created_at": "2018-06-04T05:06:54.000Z",
  "tags": []
}