Skip to content
Rokks Docs

Data Fabric

Data Fabric

The Data Fabric is Rokks’s data ingestion layer. It normalizes external data (files, APIs, third-party apps) into physical TimescaleDB tables that dashboards can query.

File Areas

A File Area is a named bucket for related data files. All files in an area map to the same physical table — the ETL pipeline merges them column by column according to the area’s transform plan.

Uploading files

  1. Create a File Area: Data Fabric → + File Area.
  2. Upload files by drag-and-drop or the Upload button.
  3. Rokks generates an AI-powered transform plan — review and adjust column types, names, and mappings.
  4. Click Run ETL to load the data.

Transform plans

A transform plan maps source columns (from the raw file) to target columns (in the physical table). For each field you configure:

  • Column name — the human-readable label
  • Type — logical type (numeric, timestamp, text, boolean, enum, …)
  • Unit — optional measurement unit (e.g., USD, ms, kg)
  • Mapping type — how to interpret the raw value (DIRECT, DATE_FORMAT, BOOLEAN_MAP, ENUM_MAP, …)

The AI generates a plan from a 100-row sample. You can edit any field before running.

Incremental loads

By default each ETL run truncates and reloads the target table. For append-only sources, switch to Incremental mode — the ETL appends new rows without touching existing data.

Web API Streams

A Web API Stream connects to a REST or WebSocket endpoint and syncs data on a schedule.

  1. Create a Web API connection: Data Fabric → Web API → + Connection.
  2. Configure the endpoint URL, HTTP method, authentication (API key, OAuth, Bearer), and pagination strategy.
  3. Add one or more streams per connection — each stream maps to its own physical table.
  4. Set a sync schedule (hourly, daily, custom cron).

Sync mode options:

ModeWhen to use
FULL_REFRESHSmall APIs where full reload each time is acceptable
INCREMENTAL_CURSORAPIs with a since / after parameter — only fetches new records

App Connect

App Connect provides native integrations with third-party apps. The current integration is Airtable.

Airtable

  1. Add an Airtable connection: Data Fabric → App Connect → + Airtable Connection.
  2. Enter your Airtable API key and base ID.
  3. Rokks discovers all tables in the base and presents a Constellation — a graph of tables and their linked-record relationships.
  4. Configure each node (table): choose which fields to sync, set column types, configure junction tables for linked records.
  5. Run the sync. Rokks creates one physical table per Airtable table plus junction tables for every many-to-many link.

Airtable syncs can be scheduled just like Web API streams.

Column metadata

Every physical table’s metadata is stored in the table_metadata Firestore collection and is the authoritative source for:

  • Column human names (display labels)
  • Logical types (columnLogicalTypes)
  • Units (columnUnits)
  • Constraints (primary key, unique key)

The Schema Editor (accessible from Data Fabric → expand a table) lets you view and edit this metadata after data is loaded.