Data pipelines
Agents are only as useful as the data they can reach. Pipelines move data between databases on a schedule, using the same worker and event infrastructure that runs your agents.
Manage them under Dashboard → Data.
The model
Three reusable objects, in the Airbyte style:
| Object | What it is |
|---|---|
| Source | Where data comes from — a configured connector instance |
| Destination | Where it goes |
| Connection | A source + destination pair, plus what to sync and when |
Sources and destinations are reusable across connections, so you configure a database once and sync several tables from it. You can also create them inline while building a connection.
Connectors
Version 1 ships Postgres → Postgres, with real driver-level connections.
The connector registry is designed so new connectors — REST APIs, branded sources like Stripe or GitHub — are added by registering a connector file rather than reworking the system.
:::note Credentials Connection credentials are encrypted at rest with AES-256-GCM, the same scheme used for your LLM keys, and decrypted only inside the worker. Connector hosts are validated before connecting: localhost, private IP ranges, and the cloud metadata endpoint are blocked unless explicitly allowlisted. :::
Building a connection
- Choose or create a source. Enter connection details and use Test connection to verify before saving.
- Discover streams. The platform reads the source's schema so you pick tables rather than typing names.
- Choose or create a destination.
- Configure the sync — mode, write mode, and optionally a transform.
- Set a schedule, or leave it manual.
Sync modes
| Mode | Behaviour |
|---|---|
FULL_REFRESH | Read the whole source table every run |
INCREMENTAL | Read only rows newer than the last run, tracked by a cursor field |
CDC appears in the schema as a declared value but is not implemented —
choosing it does nothing today.
Incremental syncs resume from the cursor stored by the last successful run, so a failed run doesn't skip rows or double-read them.
Write modes
| Mode | Behaviour |
|---|---|
OVERWRITE | Replace the destination table's contents |
APPEND | Add rows to what's already there |
Transforms
A connection can carry post-load SQL that runs against the destination once the data has landed. This is the power-user escape hatch — dedupe, reshape, join against existing tables.
In-stream per-row mapping is not available; transforms happen after load.
Scheduling
Connections run on a cron expression with a timezone. The scheduler wakes on each connection's next run time and enqueues a sync — the same generic scheduler that drives scheduled agent runs.
You can always trigger a run immediately with Sync now.
Watching a sync
A sync is a run like any other. It moves through:
PENDING → RUNNING → COMPLETED / FAILED / CANCELLED
Data streams in batches with keyset pagination, so memory stays bounded regardless of table size, and progress is persisted as it goes. Live progress appears over SSE, just like an agent run.
Dashboard → Data lists every connection with its last sync, rows moved, and success or failure.
Billing
Syncs are free — they run against your own infrastructure and consume no LLM tokens.