Sources and freshness
Understand Source identity, connector support, lifecycle states, schedules, commits, and schema change.
A Source is the unit of authorization, ownership, grants, freshness, retention, and deletion. One Slack workspace, one Drive connection, one Salesforce organization, and an Account's entire Desktop Capture fleet are separate Sources.
Each Source has a stable UUID, connector revision, owner, schedule, configuration revision, datasets, schema revisions, Runs, Deliveries, freshness, grants, and lifecycle state. Provider credentials are managed separately and never appear in the catalogue or query path.
Connector catalogue
Connect searches the complete 596-definition Airbyte source family pinned in this release, plus Desktop Capture. A definition's badge is exact: Ready has passed the complete production gate, Preview is available for bounded validation or access requests, and Planned is discoverable but cannot connect. Catalogue presence or an upstream image never implies production support.
The live connector definition also exposes its revision, authorization type, credential JSON Schema, capabilities, and known limits. Read those values at connection time because connector versions and provider requirements change more often than static documentation.
Exact states
| State | Meaning |
|---|---|
healthy | Latest required work committed successfully |
syncing | A Run is extracting or committing |
needs_authorization | Provider authorization is missing, expired, or revoked |
needs_review | Breaking or ambiguous schema change awaits a decision |
paused | New scheduled and manual work is not admitted |
failed | Current issue requires inspection or retry |
deleting | Access/new work are revoked and cleanup is running |
deleted | Source is no longer available |
Combined reports operational facts rather than synthesizing an opaque health score. Dataset state can differ from Source state when one breaking schema change is isolated and other safely independent datasets continue.
Freshness
For scheduled Sources, freshness is the last successful lake commit. Manual Sources have no implied
schedule expectation. Desktop “expected but missing” appears only when an assignment and policy
establish that expectation. lastSuccessAt: null means no successful commit has occurred.
Authorization completion, provider extraction, uploaded files, worker heartbeat, and control-plane projection are not freshness. Only a successful commit makes data queryable and advances provider state.
Commit boundary
Provider checkpoints advance only after every Delivery before the provider state barrier is committed to DuckLake. If a worker crashes after the lake commit but before the control projection, reconciliation finds the deterministic Delivery ID and Run snapshot in commit metadata and repairs the projection.
Incremental keyed datasets merge stable-key inserts, updates, and deletes. Append-only datasets retain the event stream. Full-refresh datasets replace the current projection only after the replacement commit succeeds. Retrying the same deterministic Delivery cannot create a second committed result.
Schema changes
Additive nullable fields are accepted automatically. Safe widening follows the connector compatibility contract. Destructive, narrowing, or ambiguous changes pause the affected dataset and create a review item with an exact old/new diff. Other safely independent datasets may continue.
Accepting a review promotes the new revision for future Deliveries. Rejecting keeps the unsafe model out of the query catalogue. Review selected fields, primary keys, cursor fields, type changes, and classification before deciding.
Schedules and manual runs
Manual Sources run only when explicitly requested. Daily is the standard scheduled cadence. Hourly is available only to allowlisted Accounts. Pause blocks scheduled and manual admission without removing committed data; resume restores admission. A retry uses the Run snapshot and checkpoint appropriate to that work rather than silently adopting unrelated newer configuration.