Openetl keyboard_arrow_right Documentation keyboard_arrow_right Error handling

Error Handling

OpenETL provides configurable error handling for pipeline execution. This document covers error management strategies, retry logic, and debugging approaches.

Error Handling Overview

Error handling in OpenETL manages failures during pipeline execution (network errors, authentication failures, data validation errors). Configuration options control retry behavior and failure modes.

Error Management Process

The Orchestrator handles errors through these steps:

1. Detect failures in adapter operations (connect, download, upload)
2. Apply retry logic according to error_handling configuration
3. Emit error events for logging and monitoring
4. Continue or halt execution based on fail_on_error setting
5. Clean up adapter resources (disconnect)

Configuring Error Handling

Set error behavior in the pipeline's error_handling option:

error_handling: {
  max_retries: 3,
  retry_interval: 1000,
  fail_on_error: false,
}

Configuration behavior:

• Retry up to 3 times on failure
• Wait 1000ms between retry attempts
• Continue pipeline execution despite errors

Error Scenarios

Network Failures

• Retries according to max_retries configuration
• Emits error event for each failure
• Continues or halts based on fail_on_error

Rate Limiting (HTTP 429)

• Uses rate_limiting.max_retries_on_rate_limit if configured
• Falls back to error_handling.max_retries otherwise
• Applies exponential backoff between retries

Authentication Errors (HTTP 401)

• Attempts OAuth2 token refresh if applicable
• Retries with refreshed credentials
• Fails permanently if refresh unsuccessful

Data Validation Errors

• Logs validation failure details
• Continues or halts according to fail_on_error

Debugging Strategies

Enable Event Logging

Configure logging callbacks to track error events:

orchestrator.runPipeline({
  id: 'debug-pipeline',
  source: { /* ... */ },
  error_handling: { max_retries: 2, retry_interval: 500, fail_on_error: false },
  logging: event => {
    if (event.type === 'error') console.error(event.message);
    else console.log(event);
  },
});

Verification Steps

1. Credential Validation: Verify credential_id references exist in Vault
2. Component Isolation: Test source and target adapters independently
3. Adapter Implementation: Review adapter error handling in connect, download, upload methods

Complete Debugging Example

import Orchestrator from 'openetl';
import { hubspot } from '@openetl/hubspot';

const vault = { 'hs-auth': { type: 'oauth2', credentials: { /* ... */ } } };
const orchestrator = Orchestrator(vault, { hubspot });

orchestrator.runPipeline({
  id: 'hs-fetch',
  source: {
    adapter_id: 'hubspot',
    endpoint_id: 'contacts',
    credential_id: 'hs-auth',
    fields: ['firstname'],
  },
  error_handling: {
    max_retries: 2,
    retry_interval: 2000,
    fail_on_error: false,
  },
  logging: event => console.log(`${event.type}: ${event.message}`),
});

This configuration:

• Retries twice on failure
• Waits 2000ms between attempts
• Logs all pipeline events
• Continues execution despite errors

Additional Resources

• Pipelines: Pipeline configuration reference
• Rate Limiting: Rate limit error handling
• Adapters: Adapter error handling requirements