Adapters

Adapters are the heart of OpenETL's data connectivity. This section explains what they are, how they work, and the built-in options available to you.

What is an Adapter?

An adapter is a reusable module that connects OpenETL to a data source—like an API (HubSpot) or database (PostgreSQL). It defines how to fetch or send data, making it a bridge between your pipeline and the outside world.

How Adapters Work

Adapters plug into the Orchestrator via connectors. The Orchestrator calls adapter methods (e.g., download) to extract data or upload to send it, using credentials from the Vault. They're stateless, reusable, and tied to specific providers.

Adapter Structure

Each adapter consists of two main parts:

1. Adapter Description Object: A static TypeScript object that describes the adapter's capabilities, endpoints, and configuration.
2. Adapter Function: A function that implements the core adapter methods.

Adapter Description Object

The description object defines metadata about the adapter and its capabilities:

Object Example

const HubSpotAdapter: HttpAdapter = {
    id: "hubspot-adapter", // Unique identifier 
    name: "HubSpot CRM Adapter", // Human-readable name
    type: "http", // Adapter type (e.g., "http", "database")
    action: ["download", "upload", "sync"], // Supported actions
    credential_type: "oauth2", // Authentication type required
    base_url: "https://api.hubapi.com", // Base URL for API calls
    metadata: { // Additional metadata
        provider: "hubspot",
        description: "Adapter for HubSpot CRM and Marketing APIs",
        version: "v3",
    },
    endpoints: [ // Available endpoints
        {
            id: "contacts", // Endpoint identifier
            path: "/crm/v3/objects/contacts", // API path
            method: "GET", // HTTP method
            description: "Retrieve all contacts from HubSpot",
            supported_actions: ["download", "sync"], // Actions supported by this endpoint
        },
        // More endpoints...
    ],
};

Adapter Function

Key Adapter Methods

Every adapter implements these core methods:

Working with Adapters

Using in Pipelines

To use an adapter in a pipeline, specify:

1. The adapter ID in adapter_id
2. The specific endpoint in endpoint_id
3. The credential to use in credential_id
4. Any adapter-specific configuration

orchestrator.runPipeline({
  id: 'my-pipeline',
  source: {
    adapter_id: 'hubspot',      // Which adapter to use
    endpoint_id: 'contacts',    // Which endpoint of that adapter
    credential_id: 'hs-auth',   // Which credential from the vault
    fields: ['firstname', 'lastname'] // Adapter-specific configuration
  },
  // Additional pipeline configuration...
});

Combining Multiple Adapters

You can combine different adapters in a single pipeline:

orchestrator.runPipeline({
  id: 'hubspot-to-postgres',
  source: {
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth',
    fields: ['firstname', 'lastname']
  },
  target: {
    adapter_id: 'postgresql',
    endpoint_id: 'table_insert',
    credential_id: 'pg-auth',
    config: {
      schema: 'public',
      table: 'contacts'
    }
  }
});

For information on creating your own adapters, see Custom Adapters.

Next up: Connectors!