Skip to Content
ObservabilityIn-Warehouse Log

In-Warehouse Log

Zeotap writes every operational outcome to an observability log table in your own data warehouse. Because the history lives in your warehouse, you can query it with SQL, join it against your other data, and build dashboards in whatever BI tool you already use — without leaving your data platform.

The in-warehouse log is on by default. There is nothing to enable.

Where the Log Lives

The log is written to the observability_events table in the audit_logs schema of your warehouse.

Zeotap writes it to your workspace’s configured event-log warehouse. If you have not designated one, it uses your first connected warehouse.

What Gets Logged

One row is written for each operational outcome across the platform — sync, journey, loader, store feed, realtime audience snapshot rebuild, identity graph, and trait runs, plus event delivery and contract activity. Both successful completions and failures are recorded, along with:

  • Runs that stall and are later detected as failed
  • Consecutive-failure signals
  • Automatic-pause signals

Table Schema

Each row describes one observability event:

ColumnDescription
occurred_atWhen the event happened
event_typeThe kind of event (for example, a run completion or failure)
severityRelative importance of the event
resource_typeThe type of resource (sync, journey, loader, store_feed, audience_snapshot, identity_graph, trait, and so on)
resource_idIdentifier of the specific resource
resource_nameHuman-readable name of the resource
run_idIdentifier of the run this event belongs to
statusOutcome of the run — completed, failed or cancelled
error_messageFailure detail, when the event represents a failure
metricsStructured metrics for the run (row counts, duration, rejected rows, and so on), stored as JSON
attributesAdditional structured context, stored as JSON

Column Values

status is completed, failed or cancelled for a run; pause, resume and destination events carry the resource’s own state instead (paused, active, auth_expired or connected). severity is info, warning or critical.

resource_type is one of sync, realtime_sync, journey, loader, store_feed, identity_graph, trait, event_forwarding, contract, connection, warehouse or audience_snapshot. A connection row’s resource_id is a destination’s id.

event_type names the signal itself:

Event typeWhat it records
run.completedA run finished cleanly
run.failedA run failed
run.cancelledA run was cancelled
run.zombieA run stopped heartbeating and was reclaimed
run.deferredA batched journey run was spilled to its journey’s next wake
sync.consecutive_failuresA sync’s failure streak crossed its threshold
sync.auto_pausedA sync was paused automatically after repeated failures — a batch sync (sync) or a realtime sync (realtime_sync)
sync.resumedA realtime sync paused automatically, whose pause was reported, was resumed. Written within about a minute of the resume
sync.rows_cappedA run suppressed rows under a frequency cap
connection.auth_expiredA destination’s stored credential stopped authenticating. Recorded once, when it happens, not on every later attempt
connection.restoredA destination became connected — after its credential expired, after an error, or on its first successful connection test
event.delivery_failedAn event delivery failed
event.volumeEvent volume for a window
contract.violationAn event breached a contract
snapshot.rebuiltA realtime audience’s membership was rebuilt and served
snapshot.heldA realtime audience’s new membership was set aside as unusually large
journey.wake_skippedA journey was due but could not start — an active run, or the workspace concurrency cap
warehouse.reclaim_failedDeleting a resource could not drop the warehouse objects it owned
warehouse.objects_reapedA reaper sweep dropped orphaned warehouse objects

A realtime sync’s sync.auto_paused row is written within about a minute of the pause, and its occurred_at is when the sync was paused. Its error_message is the reason shown on the sync’s page, its metrics carry consecutive_failures, and its attributes carry the sync’s destination_id and audience_id. It does not carry the destination’s error response: look at the sync’s failures for that.

Realtime audiences write at two granularities under the one audience_snapshot resource type. run.* events carry the workspace id, because one rebuild scans for every audience at once; snapshot.rebuilt and snapshot.held carry the audience id, with that audience’s membership size and its enter and exit counts under metrics.

Example Queries

Failures in the last 24 hours, most recent first:

SELECT occurred_at, resource_type, resource_name, error_message FROM audit_logs.observability_events WHERE status = 'failed' AND occurred_at >= CURRENT_TIMESTAMP - INTERVAL '24' HOUR ORDER BY occurred_at DESC;

Run counts by resource type and status over the last week:

SELECT resource_type, status, COUNT(*) AS runs FROM audit_logs.observability_events WHERE occurred_at >= CURRENT_TIMESTAMP - INTERVAL '7' DAY GROUP BY resource_type, status ORDER BY resource_type, status;

The metrics and attributes columns are JSON — use your warehouse’s JSON functions to extract individual values (for example, row counts or duration).

Sync Audit Tables

observability_events records one row per run. For the detail of an individual sync run, down to each record it sent, Zeotap also writes sync audit tables to the cdp_audit schema of the warehouse the sync reads from. On BigQuery, that is the cdp_audit dataset in the source’s project, or in the working dataset’s project when the source has one.

TableOne row perWhat it holds
cdp_audit.sync_runsSync runRun outcome, timing and row counts
cdp_audit.sync_changelogRecord sent in a runWhether each record was delivered, and the error when it was not
cdp_audit.sync_snapshotRecord currently in the destinationThe last run that sent each record, and when

Model and audience syncs write to all three tables. Journey destination sends write a row per send to sync_runs (with the journey’s id as sync_id) and record the members a frequency cap held back in sync_changelog, but do not record per-record delivery results or update sync_snapshot. Profile store feeds write to their own tables in the same schema: feed_runs, feed_changelog and feed_snapshot.

The tables are created the first time a sync runs from that warehouse source, so they do not exist until then. Snowflake stores the names in upper case (CDP_AUDIT.SYNC_RUNS). Every other warehouse uses lower case.

sync_runs

ColumnDescription
run_idIdentifier of the run, as shown on the sync’s run history
sync_idIdentifier of the sync (for a journey send, the journey)
sync_typemodel, audience, journey or campaign
workspace_idIdentifier of the workspace
statusrunning while the run is in progress, then completed, failed or cancelled
started_at / completed_atWhen the run started and finished
rows_added / rows_updated / rows_deletedRecords added, changed and removed in the destination
total_rowsRecords the destination accepted. Failed and suppressed records are not counted, so do not use it as the number of records attempted
batches_total / batches_failedBatches sent to the destination, and how many of them failed
error_messageWhy the run failed, when it did
triggered_byWho or what started the run, when recorded

sync_changelog

ColumnDescription
run_id / sync_idThe run and the sync the record belongs to
batch_numberThe batch the record was sent in, or 0 for a record a frequency cap held back
primary_key_valueThe record’s primary key
operationadded, changed or removed
statussuccess, failed, or suppressed for a record that was not sent because its address is suppressed or a frequency cap held it back
error_messageThe destination’s error for a failed record, or why a record was suppressed (for example suppressed by frequency cap)
created_atWhen the entry was written

sync_snapshot

ColumnDescription
sync_idThe sync
primary_key_valueThe record’s primary key
last_operationadded or changed
last_run_idThe last run that sent the record
last_synced_atWhen that run sent it

A row_data column exists on sync_changelog and sync_snapshot but is always empty: the audit records which records were sent and how each one fared, not their contents.

Example Queries

The last ten runs of one sync:

SELECT run_id, status, started_at, completed_at, total_rows, error_message FROM cdp_audit.sync_runs WHERE sync_id = '<sync id>' ORDER BY started_at DESC LIMIT 10;

The records a run failed to deliver, with the destination’s error:

SELECT primary_key_value, operation, error_message FROM cdp_audit.sync_changelog WHERE run_id = '<run id>' AND status = 'failed';

When a whole batch is refused by the destination, no sync_changelog rows are written for that batch, so an empty result does not prove a run had no failures. Check the run’s batches_failed and error_message in sync_runs as well.

On BigQuery, qualify the dataset with your project when it is not your default, for example `my-project.cdp_audit.sync_runs`.

Best Practices

  • Build a health dashboard — Point your BI tool at audit_logs.observability_events for an always-current view of pipeline health, entirely within your own warehouse.
  • Keep it for trend analysis — Because the log persists in your warehouse, it is well suited to long-range trend analysis that live alerting does not cover.
  • Pair it with alerting — Use the Alerting section for real-time notification, and the in-warehouse log for investigation and reporting after the fact.

Next Steps

Last updated on