Products
Star

Connector

Connectors are the glue between adapters and pipelines in OpenETL. This section breaks down their role, setup, and usage.

What is a Connector?

A connector configures an adapter for a specific task—telling it what data to fetch or send, how to filter it, and more. It's the adapter's instruction manual, reusable across pipelines.

How Connectors Work

The Orchestrator reads a connector from a pipeline, matches it to an adapter (via adapter_id), and pairs it with Vault credentials (via credential_id). The adapter then executes the connector's directives—fetching or loading data as specified.

Connector Configuration

Connectors are TypeScript objects defining an adapter's behavior. Here's a basic setup:

const connector: Connector = {
  id: 'hs-contacts',
  adapter_id: 'hubspot',
  endpoint_id: 'contacts',
  credential_id: 'hs-auth',
  fields: ['firstname', 'lastname'],
};

The Connector type (from types.ts) ensures proper structure—see the options table below.

Common Connector Options Table

Here's every option you can set in a connector:

Option Type Description
id string Unique identifier for the connector.
adapter_id string Links to an adapter (e.g., 'hubspot').
endpoint_id string Specifies the adapter endpoint (e.g., 'contacts').
credential_id string References a Vault entry for auth (e.g., 'hs-auth').
config? object Extra settings (e.g., { headers: { 'X-Custom': 'value' }, schema: 'public' }).
fields string[] Data fields to fetch or load (e.g., ['firstname', 'lastname']).
filters? (Filter | FilterGroup)[] Filters data (e.g., { field: 'email', operator: '=', value: 'test@ex.com' }).
transform? Transformation[] Applies transformations (e.g., { type: 'uppercase', options: { field: 'firstname' } }).
sort? Sort[] Sorts data (e.g., { type: 'asc', field: 'firstname' }).
limit? number Caps total items fetched (e.g., 1000).
pagination? Pagination Controls paging (e.g., { type: 'offset', itemsPerPage: 50 }).
timeout? number Max time in ms for operations (e.g., 30000).

See Transformations for transform details.

Using Connectors with Adapters

Connectors pair with adapters in pipelines. Example:

import Orchestrator from 'openetl';
import { hubspot } from '@openetl/hubspot';

const vault = {
    'hs-auth': {
        type: 'oauth2',
        credentials: { /* ... */ }
    }
};
const orchestrator = Orchestrator(vault, {
    hubspot
});

const connector = {
    id: 'hs-contacts',
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth',
    fields: ['email'],
    filters: [{
        field: 'lifecyclestage',
        operator: '=',
        value: 'customer'
    }],
};

orchestrator.runPipeline({
    id: 'hs-extract',
    source: connector,
    onload: data => console.log(data),
});

Here, the connector reuses the hubspot adapter to fetch filtered contacts—plug and play!

Next: Pipeline!

Edit this page on GitHub