Connectors

Connectors provide configuration for adapter operations within pipelines. This document covers connector structure, configuration options, and usage patterns.

Connector Definition

A connector is a configuration object that specifies how an adapter should operate within a pipeline. Connectors define the endpoint, credentials, field selection, filters, and transformations for an adapter operation.

Connector Operation

The Orchestrator processes connectors by:

Matching the adapter_id to a registered adapter
Resolving credentials via credential_id from the Vault
Passing configuration to the adapter for execution

Connector Configuration

Connectors are TypeScript objects defining an adapter's behavior. Here's a basic setup:

const connector: Connector = {
  id: 'hs-contacts',
  adapter_id: 'hubspot',
  endpoint_id: 'contacts',
  credential_id: 'hs-auth',
  fields: ['firstname', 'lastname'],
};

The Connector type from the framework's type definitions enforces structural correctness.

Common Connector Options Table

Here's every option you can set in a connector:

Option	Type	Description
`id`	`string`	Unique identifier for the connector.
`adapter_id`	`string`	Links to an adapter (e.g., `'hubspot'`).
`endpoint_id`	`string`	Specifies the adapter endpoint (e.g., `'contacts'`).
`credential_id`	`string`	References a Vault entry for auth (e.g., `'hs-auth'`).
`config?`	`object`	Extra settings (e.g., `{ headers: { 'X-Custom': 'value' }, schema: 'public' }`).
`fields`	`string[]`	Data fields to fetch or load (e.g., `['firstname', 'lastname']`).
`filters?`	`(Filter \| FilterGroup)[]`	Filters data (e.g., `{ field: 'email', operator: '=', value: '[email protected]' }`).
`transform?`	`Transformation[]`	Applies transformations (e.g., `{ type: 'uppercase', options: { field: 'firstname' } }`).
`sort?`	`Sort[]`	Sorts data (e.g., `{ type: 'asc', field: 'firstname' }`).
`limit?`	`number`	Caps total items fetched (e.g., `1000`).
`pagination?`	`Pagination`	Controls paging (e.g., `{ type: 'offset', itemsPerPage: 50 }`).
`timeout?`	`number`	Max time in ms for operations (e.g., `30000`).

See Transformations for transform details.

Using Connectors with Adapters

Connectors pair with adapters in pipelines. Example:

import Orchestrator from 'openetl';
import { hubspot } from '@openetl/hubspot';

const vault = {
    'hs-auth': {
        type: 'oauth2',
        credentials: { /* ... */ }
    }
};
const orchestrator = Orchestrator(vault, {
    hubspot
});

const connector = {
    id: 'hs-contacts',
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth',
    fields: ['email'],
    filters: [{
        field: 'lifecyclestage',
        operator: '=',
        value: 'customer'
    }],
};

orchestrator.runPipeline({
    id: 'hs-extract',
    source: connector,
    onload: data => console.log(data),
});

This configuration uses the HubSpot adapter with filtering applied to the source data.

Additional Resources

Adapters: Adapter interface and built-in adapters
Pipelines: Pipeline configuration and execution
Transformations: Data transformation options

Edit this page on GitHub