Adapters

Adapters provide the interface between OpenETL and external data sources. This document covers adapter architecture, interface requirements, and usage patterns.

Adapter Definition

An adapter is a module that implements a standardized interface for connecting to a specific data source (API, database, file system). Adapters abstract source-specific implementation details from the pipeline layer.

Adapter Architecture

Adapters integrate with the Orchestrator through connectors. The Orchestrator invokes adapter methods (connect, download, upload) to execute data operations, using credentials resolved from the Vault. Adapters are stateless and provider-specific.

Adapter Structure

Adapters consist of two components:

Adapter Description Object: Static metadata defining capabilities, endpoints, and authentication requirements
Adapter Function: Implementation of the adapter interface methods

Adapter Description Object

The description object contains adapter metadata and endpoint definitions:

Field	Description
`id`	Unique identifier for the adapter
`name`	Human-readable name
`type`	Type of adapter (e.g., http, database)
`action`	Array of supported actions (e.g., download, upload, sync)
`credential_type`	Type of authentication required
`base_url`	Base URL for API endpoints (for HTTP adapters)
`metadata`	Additional information about the adapter
`endpoints`	Array of available endpoints and their capabilities

Example Adapter Description

const HubSpotAdapter: HttpAdapter = {
    id: "hubspot-adapter",
    name: "HubSpot CRM Adapter",
    type: "http",
    action: ["download", "upload", "sync"],
    credential_type: "oauth2",
    base_url: "https://api.hubapi.com",
    metadata: {
        provider: "hubspot",
        description: "Adapter for HubSpot CRM and Marketing APIs",
        version: "v3",
    },
    endpoints: [
        {
            id: "contacts",
            path: "/crm/v3/objects/contacts",
            method: "GET",
            description: "Retrieve all contacts from HubSpot",
            supported_actions: ["download", "sync"],
        },
    ],
};

Adapter Function

Required Methods

Adapter implementations must provide these methods:

Method	Description
`connect()`	Establishes a connection to the data source.
`download(pageOptions)`	Fetches data with pagination (e.g., `{ limit: 10, offset: 0 }`).
`upload?(data)`	Sends data to the target (optional).
`disconnect?()`	Disconnect from the provider (e.g., database disconnect)

Working with Adapters

Using in Pipelines

To use an adapter in a pipeline, specify:

The adapter ID in adapter_id
The specific endpoint in endpoint_id
The credential to use in credential_id
Any adapter-specific configuration

orchestrator.runPipeline({
  id: 'my-pipeline',
  source: {
    adapter_id: 'hubspot',      // Which adapter to use
    endpoint_id: 'contacts',    // Which endpoint of that adapter
    credential_id: 'hs-auth',   // Which credential from the vault
    fields: ['firstname', 'lastname'] // Adapter-specific configuration
  },
  // Additional pipeline configuration...
});

Combining Multiple Adapters

You can combine different adapters in a single pipeline:

orchestrator.runPipeline({
  id: 'hubspot-to-postgres',
  source: {
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth',
    fields: ['firstname', 'lastname']
  },
  target: {
    adapter_id: 'postgresql',
    endpoint_id: 'table_insert',
    credential_id: 'pg-auth',
    config: {
      schema: 'public',
      table: 'contacts'
    }
  }
});

For adapter development documentation, see Custom Adapters.

Additional Resources

Custom Adapters: Adapter implementation guide
Connectors: Connector configuration reference
Authentication: Credential management

Edit this page on GitHub