Openetl keyboard_arrow_right Documentation keyboard_arrow_right Adapters

Adapters

Adapters provide the interface between OpenETL and external data sources. This document provides an overview of available adapters and how to use them in pipelines.

Available Adapters

OpenETL provides official adapters for common data sources:

Database Adapters

Adapter Package Auth Type Documentation
PostgreSQL @openetl/postgresql basic PostgreSQL Adapter
MySQL @openetl/mysql basic MySQL Adapter
MongoDB @openetl/mongodb basic MongoDB Adapter

HTTP/API Adapters

Adapter Package Auth Type Documentation
HubSpot @openetl/hubspot oauth2 HubSpot Adapter
Stripe @openetl/stripe api_key Stripe Adapter
Xero @openetl/xero oauth2 Xero Adapter
Google Ads @openetl/google-ads oauth2 Google Ads Adapter
Zoho CRM @openetl/zoho oauth2 Leads, contacts, deals
ChartMogul @openetl/chartmogul api_key Customers, plans, subscriptions
GitHub @openetl/github oauth2 Issues, pull requests, repositories
Twitter/X @openetl/twitter oauth2 Tweets, users

Cloud Storage Adapters

Adapter Package Auth Type Documentation
Amazon S3 @openetl/s3 api_key List, download, upload objects

Installation

Install the core package and any adapters you need:

# Core package (required)
npm install openetl

# Database adapters
npm install @openetl/postgresql
npm install @openetl/mysql
npm install @openetl/mongodb

# API adapters
npm install @openetl/hubspot
npm install @openetl/stripe
npm install @openetl/xero
npm install @openetl/google-ads
npm install @openetl/zoho
npm install @openetl/chartmogul
npm install @openetl/github
npm install @openetl/twitter

# Cloud storage adapters
npm install @openetl/s3

Using Adapters

Basic Usage

To use an adapter in a pipeline, specify:

  1. The adapter ID in adapter_id
  2. The specific endpoint in endpoint_id
  3. The credential reference in credential_id
  4. Any adapter-specific configuration in config
import { Orchestrator } from 'openetl';
import { postgresql } from '@openetl/postgresql';
import { hubspot } from '@openetl/hubspot';

// Register adapters
const adapters = { postgresql, hubspot };

// Define credentials
const vault = {
  'pg-auth': {
    id: 'pg-auth',
    type: 'basic',
    credentials: {
      host: 'localhost',
      database: 'mydb',
      username: 'user',
      password: 'secret',
    },
  },
  'hs-auth': {
    id: 'hs-auth',
    type: 'oauth2',
    credentials: {
      client_id: 'xxx',
      client_secret: 'xxx',
      access_token: 'xxx',
      refresh_token: 'xxx',
    },
  },
};

const orchestrator = Orchestrator(vault, adapters);

// Use in pipeline
orchestrator.runPipeline({
  id: 'my-pipeline',
  source: {
    adapter_id: 'hubspot',      // Which adapter
    endpoint_id: 'contacts',    // Which endpoint
    credential_id: 'hs-auth',   // Which credentials
    fields: ['email', 'firstname', 'lastname'],
  },
});

Combining Adapters

Move data between different systems:

orchestrator.runPipeline({
  id: 'hubspot-to-postgres',
  source: {
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth',
    fields: ['email', 'firstname', 'lastname'],
    pagination: { type: 'cursor', itemsPerPage: 100 },
  },
  target: {
    adapter_id: 'postgresql',
    endpoint_id: 'table_insert',
    credential_id: 'pg-auth',
    config: {
      schema: 'public',
      table: 'contacts',
    },
    fields: ['email', 'first_name', 'last_name'],
  },
});

Common Endpoints

Database Adapters

All database adapters share similar endpoint patterns:

Endpoint Action Description
table_query / collection_query download Query data with filters, sorting, pagination
table_insert / collection_insert upload Insert records
table_columns download Schema introspection (column metadata)
custom_query download Execute raw queries

HTTP Adapters

HTTP adapters provide resource-specific endpoints:

HubSpot: contacts, companies, deals

Stripe: customers, charges, subscriptions

Xero: contacts, invoices, items, accounts

Google Ads: campaigns, ad_groups, ads, keywords, custom_query

Authentication Types

Type Description Used By
basic Username/password (databases) PostgreSQL, MySQL, MongoDB
api_key API key authentication Stripe
oauth2 OAuth 2.0 with token refresh HubSpot, Xero, Google Ads

Adapter Architecture

Adapters integrate with the Orchestrator through connectors:

┌─────────────────────────────────────────────────┐
│                  Orchestrator                    │
├─────────────────────────────────────────────────┤
│  ┌─────────┐  ┌─────────┐  ┌─────────────────┐ │
│  │  Vault  │  │ Adapters│  │    Pipeline     │ │
│  │(creds)  │  │ (map)   │  │ (source/target) │ │
│  └────┬────┘  └────┬────┘  └────────┬────────┘ │
│       │            │                │          │
│       ▼            ▼                ▼          │
│  ┌─────────────────────────────────────────┐   │
│  │           Connector Resolution          │   │
│  │  adapter_id → adapter                   │   │
│  │  credential_id → credentials            │   │
│  └─────────────────────────────────────────┘   │
│                      │                          │
└──────────────────────┼──────────────────────────┘
                       │
         ┌─────────────┼─────────────┐
         │             │             │
         ▼             ▼             ▼
    ┌─────────┐   ┌─────────┐   ┌─────────┐
    │PostgreSQL│   │ HubSpot │   │  Xero   │
    │ Adapter │   │ Adapter │   │ Adapter │
    └─────────┘   └─────────┘   └─────────┘

Adapter Interface

Adapters implement a standardized interface:

Method Required Description
connect() No Establish connection
download(options) Yes Fetch data with pagination
upload(data) No Send data to target
disconnect() No Close connection

For creating custom adapters, see Custom Adapters.

Additional Resources

Edit this page on GitHub