Worker Jobs

Worker Jobs

Overview

Worker jobs are the public model for long-running work. Loading data, syncing streams, generating plans, compiling widgets, tenant operations, and maintenance tasks should be observable as jobs. Alert delivery is an intentional exception: Sentinel emits browser events and delivers configured external channels directly when an alert changes state.

How it works

A job has a type, parameters, status, progress, steps, result, and error. The UI submits jobs for work that should survive browser closure. Integrators should monitor terminal states instead of assuming a submitted job finished.

An accepted job is persisted before the submission request is acknowledged. After a Worker restart, interrupted work can be re-enqueued from the beginning of its handler; it does not resume from an arbitrary instruction. Queue persistence is therefore not an exactly-once guarantee. Job handlers must make repeated execution safe and expose partial effects explicitly.

Authorization and availability

A registered job type is not automatically available to a browser or another service. The Worker reconciles environment and tenant scope and authorizes the exact type:

• Tenant-member access is limited to the member job allowlist.
• Object-destructive and other privileged jobs require a tenant operator.
• System-scoped jobs require a system owner and an explicit system scope.
• Mesh services are restricted by exact service-to-job mappings; mTLS identity alone is not submission authority.
• No HTTP caller can select the internal PHYSICAL phase.

Some registered handlers are deliberately dormant. HARD_PURGE_ENTITY rejects both new submissions and rows recovered from the durable queue before handler peer/inventory reads or domain effects; generic Worker run-lifecycle bookkeeping may already have occurred. TENANT_EXPORT and TENANT_IMPORT are also paused: new submissions receive HTTP 409 with WORKER_JOB_EXECUTION_PAUSED before a job row is persisted. After generic run-lifecycle bookkeeping, recovered export rows fail before export peer resolution or tenant-data access/effects; recovered import rows fail before import peer resolution, archive download, or tenant-data access/effects. Job status cannot carry tenant records or blobs, and the browser cannot safely inflate and duplicate a whole archive before backend limits apply. Both directions need a dedicated authorized, size-bounded, backpressured, cancellable streaming architecture. WEBAPI_SYNC and MATVIEW_REFRESH remain dispatcher-scheduled. MATVIEW_REFRESH accepts only environment, tenant, and persisted query id; the Worker reloads canonical resource documents and derives the physical target and dependency-safe cascade.

OPTIMIZER_INDEX_BUILD is service-only and may be submitted only by SQL Gateway’s autonomous optimizer. Its exact structured payload contains the environment, tenant, physical table, ordered physical columns, and deterministic index name—never SQL. The Worker validates it before persistence, reloads the exact PostgreSQL/Timescale parent-and-chunk catalog, and returns already-covered without DDL or a receipt when fresh usable prefix coverage already exists. Otherwise it emits server-generated online DDL without a wall-clock transport timeout, reconciles the resulting indexes, and re-reconciles a fresh catalog before accepting its ownership receipt. This observer-managed path is separate from WIDGET_COMPILE’s intentional compile-time idx_auto_* lifecycle.

Analytic Warehouse publication uses WAREHOUSE_PUBLISH. Its human-submittable payload is exactly envId, tenantId, and a persisted queryId. The Worker reloads the exact tenant catalog, validates the complete dependency cascade, expands symbolic SQL, filters materialized-result index declarations, attests CREATE versus SWAP from the target PostgreSQL catalog, calls the Worker-only staged SQL DDL boundary, and persists success or failure receipts. A successful canonical CREATE/SWAP queues the exact server-owned source query plus measured materialization work for that tenant’s autonomous optimizer. Browsers and integrations never submit publication SQL, physical identifiers, index DDL, or internal gateway URLs.

Use the supported UI or API for the particular operation. A 403 is an authorization decision; a typed 409 can mean a registered operation is deliberately paused. Neither response proves that the handler is missing.

Step-by-step

1. Submit the operation through the supported UI or API.
2. Record the returned job id.
3. Subscribe to job events and observe status until it reaches a terminal state.
4. Read step details if the job fails.
5. Fix the cause and submit a new job when retry is appropriate.

Common mistakes

Do not run data processing in the browser. If the work is multi-step, slow, or writes persistent data, it belongs in a job.

Do not carry documents, SQL rows, blobs, or archives in job progress/result events. Their bounded projections are an observation plane. A browser marker that refuses such a payload is defense in depth, not a transfer API.

Do not treat a lost stream connection as completion. Reconnect to the job event stream and reconcile from its current-state snapshot. A one-time status lookup is appropriate when a scoped stream reports that it became stale; a recurring polling loop is not.

Do not interpret every DONE parent job as proof that all downstream effects have finished. Some job contracts complete after safely creating a child job. Check the documented result and the affected resource when end-to-end completion matters.

Related

• Job Queue
• Worker Job Types
• Architecture Diagrams
• Troubleshooting