Authentication

The Vault is a cornerstone of OpenETL's secure data handling. This section dives into its purpose, mechanics, and how to use it effectively.

Overview

What is the Vault?

The Vault is a centralized store for authentication credentials—like API keys or OAuth tokens—keeping them safe and separate from your pipeline logic. It's a simple key-value map tying credential IDs to their configs.

How It Works

The Vault feeds credentials to the Orchestrator on demand. When a pipeline runs, the Orchestrator pulls the required auth details by ID, passes them to adapters via connectors, and executes the ETL flow—no persistent state needed.

Separation

OpenETL's separation design means the Vault, connectors, and adapters are independent. You can store Vault data anywhere (e.g., memory, files, or external services), isolating credentials from logic. This separation lets you focus security efforts on Vault inputs, reducing risks.

Supported Authentication Types

The Vault supports:

• API Key: Simple key-based auth (e.g., api_key: "your-key").
• OAuth2: Token-based auth with refresh capabilities (e.g., client_id, refresh_token).
• Basic: Username/password pairs (e.g., for databases).

How to Declare the Vault

Define it as a TypeScript object:

const vault = {
  'hs-auth': {
    type: 'oauth2',
    credentials: {
      client_id: 'your-id',
      client_secret: 'your-secret',
      refresh_token: 'your-token',
    },
  },
  'pg-auth': {
    type: 'basic',
    credentials: {
      username: 'your-user',
      password: 'your-pass',
      host: 'localhost',
      database: 'your-db',
    },
  },
};

Pass it to the Orchestrator:

import { Orchestrator } from 'openetl';
const orchestrator = Orchestrator(vault, { /* adapters */ });

Best Practices for Storing Credentials

• Avoid Hard-coding: Keep credentials out of source code—use environment variables or a secrets manager.
• Encrypt Sensitive Data: Store encrypted tokens externally (e.g., in a file or vault service) and load them dynamically.
• Limit Scope: Use unique IDs per service to minimize exposure if one credential leaks.
• Refresh Safely: For OAuth2, ensure refresh_token updates are handled securely.

Using the Vault in Pipelines

Reference Vault IDs in your pipeline:

orchestrator.runPipeline({
  id: 'hs-to-pg',
  source: {
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth', // Vault key
    fields: ['firstname'],
  },
  target: {
    adapter_id: 'postgresql',
    endpoint_id: 'table_insert',
    credential_id: 'pg-auth', // Vault key
    config: {
        schema: 'public',
        table: 'contacts'
    },
  },
});

The Orchestrator grabs hs-auth and pg-auth from the Vault, securing your data flow.

Learn more about adapters next in Adapter!