Google Pub/Sub

The Google Pub/Sub loader streams events from a Pub/Sub subscription you own into your warehouse. Instead of your application calling Zeotap, you publish events to your own Pub/Sub topic and grant Zeotap read access to a subscription on it — Zeotap consumes that subscription continuously and lands each message in your warehouse.

Use it when your events already flow through Google Pub/Sub (a message bus, a Dataflow pipeline, Cloud Functions, or another producer publishing to a topic) and you want them in your warehouse without standing up an HTTP producer.

When to Use It

Your events are already published to a Google Pub/Sub topic.
You want push-based, near-real-time ingestion rather than a scheduled pull.
Your payloads are freeform JSON whose shape may vary between messages.
You would rather grant read access to a subscription than build an outbound HTTP caller.

For push ingestion where you control an HTTP producer, use the Generic HTTP Push loader instead. For scheduled pulls from a SaaS API or database, use one of the other loaders.

How It Works

You publish events to a Pub/Sub topic in your Google Cloud project.
You create a pull subscription on that topic and grant Zeotap’s service account the Pub/Sub Subscriber role on it.
Zeotap consumes the subscription continuously, acknowledging each message only after it has been durably captured.
Captured messages are batched per loader and loaded into that loader’s configured warehouse.

Because a message is acknowledged only after it is durably stored, an acknowledged message will be loaded — Pub/Sub redelivers anything not yet acknowledged, so a transient failure does not lose data.

Prerequisites

A connected warehouse to load events into (BigQuery, Snowflake, Databricks, ClickHouse, or Redshift).
A Google Cloud project with a Pub/Sub topic you publish events to.
A pull subscription on that topic.
A service account JSON key whose account holds roles/pubsub.subscriber on the subscription.

Set Up the Subscription

In your Google Cloud project, create (or identify) the Pub/Sub topic your events are published to.
Create a pull subscription on the topic — for example zeotap-ingest.
Grant the service account you will use the Pub/Sub Subscriber (roles/pubsub.subscriber) role on that subscription.
Download the service account’s JSON key.

Give Zeotap its own dedicated subscription rather than sharing one with another consumer. Pub/Sub delivers each message to one subscriber per subscription, so a shared subscription would split messages between Zeotap and your other consumer.

Create the Loader

In Zeotap, go to Loaders and click Add Loader.
Choose Google Pub/Sub.
Provide the configuration below and upload the service account JSON.
Select the Target Warehouse and Schema where messages should land.
Save the loader.

Field	Required	Description
GCP Project ID	Yes	The Google Cloud project that owns the subscription.
Subscription ID	Yes	The existing pull subscription Zeotap consumes.
Service Account JSON	Yes	A service account key with `roles/pubsub.subscriber` on the subscription. Stored encrypted; never returned.
Target Warehouse	Yes	The warehouse source that consumed messages are loaded into.
Schema	Yes	The schema/dataset that holds the events table.
Target Table	No	The table name messages land in. Defaults to `pubsub_events`. Use a distinct name per loader if you run more than one in the same schema.

The connection test validates your configuration and the service account JSON format. Subscription existence and pull permission are verified when ingestion starts — make sure the Subscriber role is granted before you expect data.

Landed Format

Every message lands as one row in the target table using a fixed envelope — the raw message is preserved verbatim, so a schema is never required up front:

Column	Type	Description
`loader_id`	string	The loader that consumed the message.
`message_id`	string	A unique id for the delivered message (idempotency key).
`workspace_id`	string	Your workspace id.
`received_at`	timestamp	When Zeotap received the message (native warehouse timestamp; the table is time-partitioned on this).
`payload`	string	The raw Pub/Sub message data, verbatim.

Model the payload column downstream (for example with your warehouse’s JSON functions) to extract the fields you need. Because the payload is stored as-is, producers can evolve their event shape without reconfiguring the loader.

Sync Modes

This is a push loader: it has no schedule and no “Run Now”. Data arrives whenever your producer publishes to the topic and Zeotap consumes it. There is no incremental cursor — every message published while the loader is active is loaded once.

Rate Limits

Throughput is bounded by your Pub/Sub subscription’s delivery rate and your warehouse’s load throughput, not by a Zeotap-imposed limit. Pub/Sub buffers undelivered messages up to your subscription’s message-retention window, so a brief slowdown in loading does not drop data.

Troubleshooting

No data arriving. Confirm your producer is publishing to the topic the subscription is attached to, and that the subscription is a pull subscription. Check the subscription’s unacked-message count in Google Cloud — if it is climbing, Zeotap is not consuming (usually a permissions problem, below).
Permission denied / consumption fails. The service account needs roles/pubsub.subscriber on the subscription (not just the topic or the project). Re-check the grant and that the JSON key belongs to that account.
invalid service account JSON. The uploaded key must be a service account key file whose type is service_account. An OAuth client secret or an authorized-user credential will be rejected.
Messages split with another consumer. Two subscribers on the same subscription share its messages. Give Zeotap its own subscription.
Wrong table. If you run more than one Pub/Sub loader in one schema, set a distinct Target Table on each — otherwise they all default to pubsub_events and land together.
Duplicate rows. Pub/Sub guarantees at-least-once delivery, so a message can occasionally be delivered twice; deduplicate on message_id downstream if exactly-once semantics matter.