Products

Openetl keyboard_arrow_right Documentation keyboard_arrow_right Home

OpenETL

OpenETL is an open-source ETL (Extract, Transform, Load) framework written in TypeScript. It connects data sources such as APIs and databases through adapters, enabling data extraction, transformation, and loading across systems. The framework is released under the MIT license.

Overview

OpenETL uses a modular architecture with reusable adapters to connect different data sources and targets. The framework includes authentication support, data transformations, error handling, and retry logic. It has minimal dependencies beyond Node.js core modules.

Key Features

  • Adapter-Based Design: Extensible adapter system for connecting to APIs and databases. Includes HubSpot and PostgreSQL adapters.
  • TypeScript Native: Type-safe implementation with TypeScript type definitions.
  • Data Transformations: Built-in transformation functions (concat, uppercase, trim, etc.) and support for custom transformation logic.
  • Error Handling: Configurable retry logic, rate limiting, and error recovery mechanisms.
  • Stateless Architecture: No internal state tracking. State management is handled by the user application.

Example

The following example extracts contacts from HubSpot and loads them into a PostgreSQL table:

import Orchestrator from 'openetl';
import { hubspot } from '@openetl/hubspot';
import { postgresql } from '@openetl/postgresql';

const vault = {
  'hs-auth': { type: 'oauth2', credentials: { client_id: 'your-id', client_secret: 'your-secret', refresh_token: 'your-token' } },
  'pg-auth': { type: 'basic', credentials: { username: 'your-user', password: 'your-pass', host: 'localhost', database: 'your-db' } },
};

const orchestrator = Orchestrator(vault, { hubspot, postgresql });

orchestrator.runPipeline({
  id: 'hs-to-pg',
  source: {
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth',
    fields: ['firstname', 'lastname'],
  },
  target: {
    adapter_id: 'postgresql',
    endpoint_id: 'table_insert',
    credential_id: 'pg-auth',
    config: { schema: 'public', table: 'contacts' },
  },
});

Quick Start

Installation

  1. Install via npm: npm install openetl
  2. Configure credentials in a vault object (see Getting Started)
  3. Create a pipeline configuration with source and target connectors
  4. Execute the pipeline using orchestrator.runPipeline()

Documentation

See the Getting Started guide for detailed installation instructions, adapter configuration, and pipeline setup.

FAQ

What platforms does OpenETL support?

OpenETL includes adapters for HubSpot and PostgreSQL. Additional adapters can be created by implementing the adapter interface with connect, download, and upload methods. See Custom Adapters for implementation details.

How do I create a new adapter?

Define an adapter description object with endpoint metadata, then implement an adapter function with the required methods (connect, download, upload). Reference the Custom Adapters guide for the complete adapter interface specification.

Where can I get help?

Technical support is available through the GitHub Issues page. For documentation, refer to the guides in the Documentation section.

Contributing

Contributions to OpenETL are accepted through GitHub pull requests. To contribute:

  1. Fork the repository
  2. Implement your changes (new adapters, bug fixes, or features)
  3. Add tests for your changes
  4. Submit a pull request with a description of your changes

See the Contributing guide for development setup and coding standards.

Community

Edit this page on GitHub