Products

Openetl keyboard_arrow_right Documentation keyboard_arrow_right Getting started

Getting Started

This guide covers OpenETL installation, configuration, and basic pipeline implementation.

Framework Design

TypeScript Implementation

OpenETL is implemented in TypeScript, providing static type checking and IDE integration for pipeline configuration. Type definitions are included for all framework interfaces.

Minimal Dependencies

The framework has minimal external dependencies, with adapter-specific dependencies loaded only when the corresponding adapter is imported.

Installation

Production Installation

Install the core framework and required adapters:

npm install openetl @openetl/hubspot @openetl/postgresql

Development Installation

Clone the repository for development or contribution:

git clone https://github.com/jspreadsheet/openetl.git
cd openetl
npm install

Basic Pipeline Implementation

Example: Synchronizing HubSpot contacts to PostgreSQL

import Orchestrator from 'openetl';
import { hubspot } from '@openetl/hubspot';
import { postgresql } from '@openetl/postgresql';

const vault = {
    'hs-auth': {
        type: 'oauth2',
        credentials: {
            client_id: 'your-id',
            client_secret: 'your-secret',
            refresh_token: 'your-token'
        }
    },
    'pg-auth': {
        type: 'basic',
        credentials: {
            username: 'your-user',
            password: 'your-pass',
            host: 'localhost',
            database: 'your-db'
        }
    },
};

const orchestrator = Orchestrator(vault, {
    hubspot,
    postgresql
});

orchestrator.runPipeline({
    id: 'hs-to-pg',
    source: {
        adapter_id: 'hubspot',
        endpoint_id: 'contacts',
        credential_id: 'hs-auth',
        fields: ['firstname', 'lastname'],
    },
    target: {
        adapter_id: 'postgresql',
        endpoint_id: 'table_insert',
        credential_id: 'pg-auth',
        config: {
            schema: 'public',
            table: 'contacts'
        },
    },
}).then(() => console.log('Pipeline execution complete'));

Execution Steps

  1. Save the code to a TypeScript file (e.g., pipeline.ts)
  2. Configure credentials in the vault object
  3. Execute: tsx pipeline.ts or compile and run with tsc && node pipeline.js

Verification

Query the PostgreSQL contacts table to verify the data was loaded successfully.

Development and Testing

Adapter Development

To develop custom adapters, implement the adapter interface with the required methods. See the Custom Adapters guide for the complete specification.

Testing

Adapter implementations should include unit tests covering:

  • Connection establishment and authentication
  • Data extraction with pagination
  • Error handling and retry logic
  • Data transformation and loading

See Custom Adapters for testing examples.

Next Steps

Edit this page on GitHub