Openetl keyboard_arrow_right Documentation keyboard_arrow_right Authentication

Authentication

The Vault provides credential management for OpenETL pipelines. This document covers Vault architecture, supported authentication types, and security practices.

Vault Architecture

Vault Definition

The Vault is a key-value store for authentication credentials (API keys, OAuth tokens, database credentials). Credentials are referenced by ID in pipeline and connector configurations.

Credential Resolution

During pipeline execution:

1. The Orchestrator reads credential_id from connector configuration
2. Credentials are retrieved from the Vault by ID
3. Credentials are passed to the adapter for authentication
4. No credential state is persisted by the framework

Design Principles

The Vault design separates credential storage from pipeline logic:

• Vault contents can be stored in memory, configuration files, or external secret management services
• Credential management is isolated from ETL operations
• Security controls focus on Vault access and storage

Supported Authentication Types

Supported credential types:

• API Key: Key-based authentication (api_key field)
• OAuth2: Token-based authentication with refresh capability (client_id, client_secret, refresh_token)
• Basic: Username/password authentication (for databases and basic auth APIs)

Vault Configuration

Define the Vault as a TypeScript object:

const vault = {
  'hs-auth': {
    type: 'oauth2',
    credentials: {
      client_id: 'your-id',
      client_secret: 'your-secret',
      refresh_token: 'your-token',
    },
  },
  'pg-auth': {
    type: 'basic',
    credentials: {
      username: 'your-user',
      password: 'your-pass',
      host: 'localhost',
      database: 'your-db',
    },
  },
};

Pass it to the Orchestrator:

import { Orchestrator } from 'openetl';
const orchestrator = Orchestrator(vault, { /* adapters */ });

Security Best Practices

Credential Storage

• Environment Variables: Load credentials from environment variables rather than hardcoding
• Secret Management Services: Use external services (AWS Secrets Manager, HashiCorp Vault, etc.) for production
• Encryption: Encrypt sensitive credentials at rest
• Scoped Credentials: Use unique credential IDs per service to limit exposure
• Token Refresh: Implement secure token refresh handling for OAuth2 flows

Pipeline Configuration

Reference credentials by ID in pipeline configurations:

orchestrator.runPipeline({
  id: 'hs-to-pg',
  source: {
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth', // Vault key
    fields: ['firstname'],
  },
  target: {
    adapter_id: 'postgresql',
    endpoint_id: 'table_insert',
    credential_id: 'pg-auth', // Vault key
    config: {
        schema: 'public',
        table: 'contacts'
    },
  },
});

The Orchestrator resolves hs-auth and pg-auth credentials from the Vault during execution.

Additional Resources

• Adapters: Adapter authentication requirements
• Pipelines: Pipeline configuration reference
• Connectors: Connector credential references