Openetl keyboard_arrow_right Documentation keyboard_arrow_right Contributing

Contributing

OpenETL accepts contributions through GitHub. This document covers the contribution process, development setup, and code standards.

Getting Started

Thank you for considering contributing to OpenETL! This open-source project benefits from community contributions of all kinds.

Contribution Types

Adapter Development

New adapter implementations for additional data sources (APIs, databases, file systems). See Custom Adapters for the adapter interface specification.

Requirements for new adapters:

  • Complete AdapterInstance interface implementation
  • Unit tests with mocked dependencies
  • Security best practices (parameterized queries for databases)
  • Documentation including endpoints and configuration

Bug Fixes

Fixes for framework bugs, adapter issues, or documentation errors.

Feature Enhancements

Improvements to existing functionality, optimization, or new framework features.

Documentation

Documentation improvements, corrections, or additions.

Development Setup

Prerequisites

  • Node.js 18.0 or higher
  • npm or yarn package manager
  • Git

Repository Setup

git clone https://github.com/jspreadsheet/openetl.git
cd openetl
npm install

Project Structure

openetl/
├── src/                  # Core framework source
│   ├── index.ts         # Orchestrator and main exports
│   ├── types.ts         # TypeScript type definitions
│   └── utils/           # Utility functions
├── adapters/            # Official adapters
│   ├── postgresql/
│   ├── mysql/
│   ├── mongodb/
│   ├── hubspot/
│   └── ...
├── tests/               # Core framework tests
├── docs/                # Documentation
└── dist/                # Compiled output

Development Workflow

  1. Create a feature branch from main
  2. Implement changes with appropriate tests
  3. Run test suite: npm test
  4. Ensure TypeScript compilation: npm run build
  5. Verify linting passes: npm run lint

Running Tests

# Run all tests
npm test

# Run specific adapter tests
npm test -- --testPathPattern=adapters/postgresql

# Run with coverage
npm test -- --coverage

Pull Request Process

Before Submitting

  1. Code Quality: Follow TypeScript coding standards
  2. Tests: Include unit tests for new functionality
  3. Documentation: Update relevant documentation
  4. Type Definitions: Ensure TypeScript types are correct
  5. Security: Follow security best practices (see below)

Submission Steps

  1. Create Branch:
git checkout -b feature/your-feature-name
  1. Commit Changes:
git add .
git commit -m "feat: Add feature description"
git push origin feature/your-feature-name

  1. Create Pull Request:

    • Navigate to GitHub repository
    • Create pull request against main branch
    • Fill out the PR template with description of changes
    • Reference related issues (e.g., "Fixes #123")

  2. Review Process:

    • Address review feedback promptly
    • Update pull request with requested changes
    • Maintain clean commit history (squash if needed)

Commit Message Format

Use conventional commit format:

  • feat: New features
  • fix: Bug fixes
  • docs: Documentation changes
  • test: Test additions or fixes
  • refactor: Code refactoring
  • chore: Build process or auxiliary tool changes

Code Standards

TypeScript

  • Use strict TypeScript compilation
  • Provide type definitions for all public APIs
  • Avoid any types where possible
  • Use unknown for truly unknown types

Security Requirements

For database adapters:

  • Always use parameterized queries - Never interpolate values into SQL
  • Escape identifiers - Schema, table, column names must be escaped
  • Validate operators - Use whitelist for SQL operators
  • Never log credentials - Avoid logging auth objects

See Security for detailed security guidelines.

Error Handling

  • Use AdapterError class for adapter errors
  • Include appropriate error codes
  • Set retryable flag correctly
  • Provide context for debugging
throw new AdapterError(
  'Connection timeout',
  'CONNECTION_FAILED',
  true,  // retryable
  { host: 'example.com', timeout: 30000 }
);

Testing

  • Unit tests for all adapter methods
  • Integration tests for complete pipelines
  • Mock external dependencies
  • Test error scenarios
  • Test pagination handling
describe('MyAdapter', () => {
  it('downloads data with pagination', async () => {
    // Arrange
    const adapter = createAdapter(connector, auth);
    mockApi.mockResolvedValue({ data: [...] });

    // Act
    const result = await adapter.download({ limit: 10, offset: 0 });

    // Assert
    expect(result.data).toHaveLength(10);
    expect(result.options?.nextOffset).toBeDefined();
  });
});

Documentation

  • Document public APIs and interfaces
  • Include code examples for new features
  • Update relevant guide documents
  • Add JSDoc comments for exported functions

Adapter Checklist

When contributing a new adapter, ensure:

  • [ ] Implements AdapterInstance interface
  • [ ] Has unit tests with mocked dependencies
  • [ ] Uses parameterized queries (database adapters)
  • [ ] Validates operators (database adapters)
  • [ ] Escapes identifiers (database adapters)
  • [ ] Handles pagination correctly
  • [ ] Uses AdapterError for errors
  • [ ] Includes getConfig() returning adapter metadata
  • [ ] Has README with installation and usage
  • [ ] Has documentation page in docs/
  • [ ] Package.json follows standard format

Questions?

  • Open an issue on GitHub
  • Check existing issues for similar questions
  • Review documentation for answered questions

License

By contributing to OpenETL, you agree that your contributions will be licensed under the MIT License.

Edit this page on GitHub