Openetl keyboard_arrow_right Documentation keyboard_arrow_right Home

OpenETL

OpenETL is an open-source ETL (Extract, Transform, Load) framework written in TypeScript. It connects data sources through adapters, enabling data extraction, transformation, and loading across systems.

Overview

OpenETL uses a modular architecture with reusable adapters for different data sources. The framework includes authentication management, data transformations, error handling with retry logic, and rate limiting support.

Key Features

• Adapter-Based Design - Extensible adapter system with official support for PostgreSQL, MySQL, MongoDB, HubSpot, Stripe, Xero, and Google Ads
• TypeScript Native - Full type safety with comprehensive TypeScript definitions
• Data Transformations - 12 built-in transformation types (concat, rename, trim, split, and more)
• Error Handling - Configurable retry logic with exponential backoff
• Stateless Architecture - No internal state; external state management for incremental syncs

Example

Extract contacts from HubSpot and load into PostgreSQL:

import { Orchestrator, Pipeline } from 'openetl';
import { hubspot } from '@openetl/hubspot';
import { postgresql } from '@openetl/postgresql';

const vault = {
  'hubspot': {
    id: 'hubspot',
    type: 'oauth2',
    credentials: {
      client_id: process.env.HUBSPOT_CLIENT_ID,
      client_secret: process.env.HUBSPOT_CLIENT_SECRET,
      access_token: process.env.HUBSPOT_ACCESS_TOKEN,
      refresh_token: process.env.HUBSPOT_REFRESH_TOKEN,
    },
  },
  'database': {
    id: 'database',
    type: 'basic',
    credentials: {
      host: process.env.DB_HOST,
      database: process.env.DB_NAME,
      username: process.env.DB_USER,
      password: process.env.DB_PASSWORD,
    },
  },
};

const etl = Orchestrator(vault, { hubspot, postgresql });

const pipeline: Pipeline = {
  id: 'sync-contacts',
  source: {
    id: 'hubspot-source',
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hubspot',
    fields: ['email', 'firstname', 'lastname'],
    pagination: { type: 'cursor', itemsPerPage: 100 },
  },
  target: {
    id: 'postgres-target',
    adapter_id: 'postgresql',
    endpoint_id: 'table_insert',
    credential_id: 'database',
    config: { schema: 'public', table: 'contacts' },
    fields: ['email', 'first_name', 'last_name'],
  },
};

await etl.runPipeline(pipeline);

Quick Start

1. Install: npm install openetl
2. Install adapters: npm install @openetl/postgresql @openetl/hubspot
3. Configure credentials in a vault object
4. Define pipeline with source and target
5. Execute with orchestrator.runPipeline()

See the Getting Started guide for detailed instructions.

Official Adapters

See Adapters for complete documentation.

Requirements

• Node.js 18.0 or higher
• TypeScript 4.7 or higher (for TypeScript projects)

Resources

• GitHub Repository
• Getting Started
• Adapters
• Custom Adapters
• API Reference

License

OpenETL is released under the MIT license.