Worker Job Types

The Worker service (worker) is the job execution engine. Jobs are submitted via POST /api/jobs (browser → Edge → Worker) or via submitWorkerJob from backend services.

Job lifecycle

PENDING → RUNNING → DONE
                 ↘ ERROR
                 ↘ CANCELLED

Each job has one or more steps. Steps are PENDING → RUNNING → DONE | ERROR. Progress is streamed via GET /api/jobs/:id/stream (SSE).

Job types

`FILE_ETL`

Runs the full Extract → Transform → Load pipeline for a file area.

Params: fileId, areaId, envId, tenantId

Steps: Extract, Transform, Load

`WEBAPI_SYNC`

Syncs a Web API stream — fetches from the external endpoint, transforms rows, and loads into TimescaleDB.

Params: streamId, connectionId, envId, tenantId, ingestionGateway, firestoreGateway

Steps: Fetch, Transform, Load, Release Lock

`WEBAPI_PURGE_DATA`

Truncates a Web API stream’s physical table and resets its pipeline documents.

Params: streamId, envId, tenantId

`WEBAPI_REBUILD_SCHEMA`

Drops and recreates a Web API stream’s physical table from the current transform plan. Use after schema changes.

Params: streamId, envId, tenantId

`MATVIEW_REFRESH`

Refreshes a warehouse materialized view cascade via the SQL Gateway.

Params: viewId, envId, tenantId, sqlGateway, locksCollection, lockId

`TS_ROLLUP`

Triggers TimescaleDB continuous aggregate rollup for a hypertable.

Params: physId, envId, sqlGateway

`TS_PURGE`

Runs TimescaleDB data retention policy for a hypertable.

Params: physId, envId, sqlGateway

`ALERT_NOTIFY`

Sends an external notification (e.g., Pushover) for a triggered alert rule.

Params: userId, envId, channel, event

`DISPATCHER_PULSE`

Forces an immediate scheduler tick (for testing or manual triggers).

Params: envId

`SENTINEL_TEST_NOTIFICATION`

Tests a user’s configured notification channel.

Params: userId, envId, channel

`MIGRATION_DRY_RUN`

Plans a data migration — discovers collections, counts documents, scans SQL tables. Read-only.

Params: envId, tenantId

`REFRESH_LOGICAL_VIEWS`

Runs CREATE OR REPLACE VIEW for a set of physical table IDs, updating the logical name views in PostgreSQL.

Params: physicalIds, envId, sqlGateway

`GENERATE_ETL_PLAN`

Generates an AI-powered ETL transform plan for a data file. Uses the AI Gateway.

Params: fileId, envId, tenantId

`GENERATE_AREA_TRANSFORM_PLAN`

Generates an AI-powered transform plan for an entire File Area (multi-file).

Params: areaId, envId, tenantId

`GENERATE_AREA_CONSTELLATION_PLAN`

Generates a Constellation plan for a File Area — detects relational structure across files and produces a multi-table schema.

Params: areaId, envId, tenantId

`GENERATE_WEBAPI_STREAM_PLAN`

Generates an AI-powered transform plan for a Web API stream from a sample response.

Params: streamId, ownerStreamId, envId, tenantId

`DOCFORGE_REGENERATE_PAGE`

Runs the DocForge author + verify pipeline for a single documentation page and opens a PR with the updated MDX.

Params: pageId, tier (auto | review)

See the DocForge documentation for how the regeneration pipeline works.

Submitting a job (frontend)

import { workerService } from '@/services/apiService';

const job = await workerService.startJob({
  type: 'WEBAPI_SYNC',
  params: { streamId, connectionId, envId, tenantId },
  envId,
});

// Monitor progress
const result = await workerService.getJob(job.jobId);

Submitting a job (backend service)

const res = await fetch(`${WORKER_URL}/api/jobs`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', ...serviceHeaders() },
  body: JSON.stringify({ type: 'WEBAPI_SYNC', params, envId }),
  signal: AbortSignal.timeout(10_000),
});