Products

Openetl keyboard_arrow_right Documentation

OpenETL

OpenETL is an open-source ETL (Extract, Transform, Load) framework implemented in TypeScript. The framework provides a modular architecture for building data pipelines that extract from APIs (e.g., HubSpot), databases (e.g., PostgreSQL), and other data sources. OpenETL is released under the MIT license.

Overview

OpenETL uses a modular architecture with reusable adapters and connectors to integrate different data sources and targets. The framework includes authentication management, data transformations, error handling with retries, and rate limiting. It has a stateless design, requiring external state management for incremental synchronization.

Key Features

  • Adapter-Based Design: Extensible adapter system with implementations for HubSpot and PostgreSQL
  • TypeScript Native: Type-safe implementation with full TypeScript type definitions
  • Data Transformations: Built-in transformation functions and custom transformation support
  • Error Handling: Configurable retry logic, rate limiting, and error recovery
  • Stateless Architecture: No internal state management; external state storage required for incremental syncs

Example

Example pipeline synchronizing HubSpot contacts to PostgreSQL:

import Orchestrator from 'openetl';
import { hubspot } from '@openetl/hubspot';
import { postgresql } from '@openetl/postgresql';

// Load only what you need to use
const adapters = { hubspot, postgresql };

// General vault to keep your credentails
const vault = {
  'hs-auth': {
      type: 'oauth2',
      credentials: {
          client_id: 'your-id',
          client_secret: 'your-secret',
          refresh_token: 'your-token'
      }
  },
  'pg-auth': {
      type: 'basic',
      credentials: {
          username: 'your-user',
          password: 'your-pass',
          host: 'localhost',
          database: 'your-db'
      }
  },
};

const orchestrator = Orchestrator(vault, adapters);

orchestrator.runPipeline({
  id: 'hs-to-pg',
  source: {
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth',
    fields: [
        'firstname',
        'lastname'
    ],
  },
  target: {
    adapter_id: 'postgresql',
    endpoint_id: 'table_insert',
    credential_id: 'pg-auth',
    config: {
        schema: 'public',
        table: 'contacts'
    },
  },
});

Installation

Quick Start

  1. Install: npm install openetl @openetl/hubspot @openetl/postgresql
  2. Configure credentials in a vault object
  3. Create pipeline configuration
  4. Execute pipeline with orchestrator

Documentation

See the Getting Started guide for detailed installation, configuration, and usage instructions.

FAQ

What platforms does OpenETL support?

Current adapters: HubSpot and PostgreSQL. Additional adapters can be implemented using the adapter interface. See Custom Adapters for development information.

How do I create a new adapter?

Implement the adapter interface with connect, download, and upload methods. See the Custom Adapters guide for the complete specification and examples.

Where can I get help?

Technical support is available through:

Contributing

OpenETL accepts contributions through GitHub pull requests. Contributions include:

  • New adapter implementations
  • Bug fixes
  • Feature enhancements
  • Documentation improvements

See the Contributing guide for development setup and submission process.

Resources

Edit this page on GitHub