Products
Star

Transformations

Transformations in OpenETL modify data as it flows through your ETL pipeline—cleaning, reshaping, or enhancing it before it lands at its destination. This section covers the built-in transformations, explains why they're string-based, and shows examples, including a custom function.

List of Existing Transformations

OpenETL offers a set of built-in transformations, each identified by a string type. Here's the full list with descriptions:

Transformation Description
concat Concatenates multiple fields into one with a glue.
renameKey Renames a field in the data object.
uppercase Converts a field's value to uppercase.
lowercase Converts a field's value to lowercase.
trim Trims whitespace from a field's value.
split Splits a field's value into an array by delimiter.
replace Replaces a search string in a field.
addPrefix Adds a prefix to a field's value.
addSuffix Adds a suffix to a field's value.
toNumber Converts a field's value to a number.
extract Extracts a substring or pattern match from a field.
mergeObjects Merges multiple fields into a single object.

These are applied via a transform array in a connector or pipeline.

Why Transformations Can Be Strings

Transformations use string types (e.g., 'concat', 'uppercase') for built-in operations. This allows them to be serialized as JSON, making it easy to store common transformations in a database or config file for reuse across pipelines. For example:

transform: [{
  type: 'uppercase',
  options: { field: 'name', to: 'name_upper' }
}]

Can be saved as:

{
  "transform": [
    {
      "type": "uppercase",
      "options": { "field": "name", "to": "name_upper" }
    }
  ]
}

This JSON can be stored, retrieved, and applied anywhere, streamlining management. For custom needs, you can also use functions directly, offering full flexibility (though they're not JSON-serializable).

Examples of Transformations

Here are practical examples of transformations, ending with a custom function.

1. Concatenate Fields (concat)

Combine first_name and last_name:

transform: [{
  type: 'concat',
  options: {
    properties: ['first_name', 'last_name'],
    glue: ' ',
    to: 'full_name'
  }
}]

Input:

{
    first_name: 'John',
    last_name: 'Doe'
}

Output:

{
    first_name: 'John',
    last_name: 'Doe',
    full_name: 'John Doe'
}

2. Rename a Field (renameKey)

Rename id to user_id:

transform: [{
  type: 'renameKey',
  options: {
    from: 'id',
    to: 'user_id'
  }
}]

Input:

{
    id: 123,
    name: 'John'
}

Output:

{
    user_id: 123,
    name: 'John'
}

3. Convert to Uppercase (uppercase)

Uppercase a field:

transform: [{
  type: 'uppercase',
  options: {
    field: 'name',
    to: 'name_upper'
  }
}]

Input:

{
    name: 'john'
}

Output:

{
    name: 'john',
    name_upper: 'JOHN'
}

4. Custom Transformation (Function)

Add a timestamp to each record with a custom function:

const addTimestamp = (data: any[]) => {
  return data.map(item => ({
    ...item,
    timestamp: new Date().toISOString()
  }));
};

transform: [addTimestamp]

Input:

[
    {
        name: 'John'
    },
    {
        name: 'Jane'
    }
]

Output:

[
    {
        name: 'John',
        timestamp: '2023-10-05T12:00:00.000Z'
    },
    {
        name: 'Jane',
        timestamp: '2023-10-05T12:00:00.000Z'
    }
]

This function transforms the entire dataset, adding a timestamp—perfect for unique needs not covered by built-ins.

Next, explore Rate Limiting and Pagination!

Edit this page on GitHub