Ingestion and provenance

Combined separates provider acquisition from governed queryability. A provider response is not queryable merely because a worker received it.

Lifecycle

1. Authorization stores a managed provider-secret reference after the correct human consents.
2. Discovery records datasets, fields, keys, cursors, supported sync modes, and configuration.
3. Admission snapshots connector revision, configuration, billing policy, and execution venue.
4. Extraction emits protocol records and state messages into bounded Parquet Deliveries.
5. Staging accepts deterministic uploads under run-scoped byte, file, and path limits.
6. Commit validates Deliveries, schema, sequence, and checkpoint barriers before changing the lake.
7. Projection updates Source, Run, freshness, usage, and Activity from the committed result.

Runs and attempts

A Run is the user-visible unit of work. Triggers are manual, first_sync, retry, or recovery. States are queued, extracting, committing, succeeded, failed, needs_auth, needs_review, or canceled. Attempts record execution evidence and allow infrastructure retries without creating a second logical Run result.

Deliveries and checkpoints

A Delivery has a deterministic identity, sequence, schema revision, record/byte count, hashes, and commit state. Provider state barriers cannot advance until every preceding Delivery commits. This prevents a cursor from skipping records after a partial upload or crash.

If the lake commit succeeds but the control projection fails, reconciliation reads the deterministic Delivery and Run snapshot from commit metadata and repairs projection. It does not repeat the provider checkpoint under whatever configuration happens to be newest.

Sync modes

• incremental keyed merges current records by stable provider key and applies updates/deletes;
• append only retains the ordered event stream;
• full refresh replaces the current projection atomically after replacement is complete.

Row provenance

Every projected row carries stable identity, operation, provider emission time when available, observation and ingestion time, Run ID, Delivery ID, payload hash, and producer provenance. The SQL catalogue exposes these as _combined_* columns.

Usage

MAR accounting is based on committed record change semantics, not raw extracted row counts. Retries, duplicates, unchanged refreshes, updates, and deletes are evaluated at the commit boundary so failed or repeated extraction cannot independently create billable activity.