Apache Kafka
The Apache Kafka loader streams records from a Kafka topic you own into your warehouse. Instead of your application calling Zeotap, you produce records to a Kafka topic and point Zeotap at it — an always-on consumer reads the topic continuously and lands each message in your warehouse. There is no schedule and no “Run Now”: data arrives as it is produced.
Use it when your data already flows through Kafka (Confluent Cloud, AWS MSK, or a self-managed cluster) and you want it in your warehouse for modeling without building an outbound HTTP producer.
Kafka can also be used as an event streaming source, which feeds the same messages into the real-time events pipeline (identity, journeys, real-time audiences) instead of a warehouse table. Choose the loader when you want records landed for modeling; choose the event source when you want events acted on in real time. See Loader vs. event source.
When to Use It
- Your records are already produced to an Apache Kafka topic.
- You want continuous, near-real-time ingestion rather than a scheduled pull.
- Your payloads are JSON or Avro, and their shape may vary between messages.
- You would rather grant read access to a topic than build an outbound HTTP caller.
For push ingestion where you control an HTTP producer, use the Generic HTTP Push loader. For a Google Pub/Sub subscription, use the Google Pub/Sub loader. For scheduled pulls from a SaaS API or database, use one of the other loaders.
How It Works
- You produce records to a Kafka topic on your cluster.
- You create broker credentials (or an API key/secret) that can read that topic.
- Zeotap consumes the topic continuously under a consumer group, committing its position only after a message has been durably captured, so a transient failure does not lose data.
- Captured messages are batched per loader and loaded into that loader’s configured warehouse.
Delivery is at-least-once: because a message is committed only after it is durably stored, an occasional duplicate is possible but nothing is dropped. Avro messages are decoded to JSON on the way in using your Confluent Schema Registry.
Prerequisites
- A connected warehouse to load into (BigQuery, Snowflake, Databricks, ClickHouse, or Redshift).
- A Kafka cluster reachable over the public internet, with the bootstrap broker address (
host:port). - A topic your records are produced to.
- Broker credentials that can read the topic — an API key/secret for SASL, or an open listener for an internal cluster.
- For Avro payloads: a Confluent Schema Registry URL (and its key/secret if secured).
Set Up on Kafka
- Identify the topic your records are produced to.
- Create a broker API key and secret (on Confluent Cloud or AWS MSK) with read access to that topic, or confirm the listener and mechanism your cluster uses.
- Enable TLS on the listener you will connect to (recommended for any credentialed connection).
- If your messages are Avro, note your Schema Registry URL and, if it is secured, its key and secret.
Give Zeotap a stable, dedicated consumer group id. Offsets are tracked per consumer group, so a stable id lets Zeotap resume exactly where it left off across restarts, and a dedicated group keeps its progress independent of your other consumers.
Create the Loader
- In Zeotap, go to Loaders and click Add Loader.
- Choose Apache Kafka.
- Provide the configuration below and the broker credentials.
- Select the Target Warehouse and Schema where messages should land.
- Click Test connection — Zeotap dials a broker, authenticates, and confirms the topic exists.
- Save the loader.
| Field | Required | Description |
|---|---|---|
| Kafka Broker host | Yes | The bootstrap broker address (host:port). Comma-separate multiple brokers. |
| Topic name | Yes | The topic to read records from. |
| Consumer Group Id | Yes | The consumer group Zeotap reads under. Keep it stable so offsets resume across restarts. |
| Authentication | Yes | SASL/SCRAM, SASL/PLAIN, or None — see Authentication. |
| Use TLS | No | Encrypt the broker connection. Recommended whenever credentials are used. |
| Data Format | Yes | JSON (default) or Avro. Avro is decoded via the Schema Registry below. |
| Schema Registry URL | For Avro | Confluent Schema Registry base URL. Required when Data Format is Avro. |
| Schema Registry Key / Secret | No | Credentials for a secured Schema Registry (Avro only). |
| Start Reading From | No | Latest reads only new messages; Earliest replays retained history. Applies on first connect only. |
| Target Warehouse | Yes | The warehouse source consumed messages are loaded into. |
| Schema | Yes | The schema/dataset that holds the landed table. |
| Target Table | No | The table name messages land in. Defaults to events. Set a distinct name per loader if you run more than one streaming or push loader in the same schema, so they do not land together. |
The connection test validates your configuration, then actually dials a broker and reads the topic’s partition metadata — a real reachability, authentication, and topic-exists check.
Authentication
| Method | Use when |
|---|---|
| SASL/SCRAM | The default on Confluent Cloud and AWS MSK. Provide the broker API key and secret, and choose the mechanism — SCRAM-SHA-512 (default) or SCRAM-SHA-256. |
| SASL/PLAIN | Your cluster uses SASL/PLAIN. Provide the broker API key and secret. Use TLS so the secret is not sent in the clear. |
| None | Development or internal clusters with an open listener and no SASL. |
Credentials are stored encrypted and never returned.
Landed Format
Every message lands as one row in the target table using a fixed envelope — the raw message is preserved verbatim, so a schema is never required up front:
| Column | Type | Description |
|---|---|---|
loader_id | string | The loader that consumed the message. |
message_id | string | A unique id for the delivered message (idempotency key). |
workspace_id | string | Your workspace id. |
received_at | timestamp | When Zeotap received the message (the table is time-partitioned on this). |
payload | string | The record, as JSON. Avro messages are decoded to JSON before landing. |
Model the payload column downstream (for example with your warehouse’s JSON functions) to extract the fields you need. Because the payload is stored as-is, producers can evolve their record shape without reconfiguring the loader.
Sync Modes
This is a streaming loader: it has no schedule and no “Run Now”. Data arrives whenever your producer writes to the topic and Zeotap consumes it. There is no incremental cursor — every message produced while the loader is active is loaded once (at-least-once).
Rate Limits
Throughput is bounded by your topic’s produce rate and your warehouse’s load throughput, not by a Zeotap-imposed limit. Kafka retains messages up to your topic’s retention window, so a brief slowdown in loading does not drop data.
Troubleshooting
- Connection test fails. Confirm the bootstrap broker address (
host:port) is reachable over the public internet, that TLS matches your listener, and that the credentials are valid. The test reports whether the failure was reachability, authentication, or a missing topic. - No data arriving. Confirm your producer is writing to the exact topic name configured, and that the Start Reading From choice matches your expectation — Latest ignores messages produced before the loader first connected.
- The loader shows an error but keeps retrying. A streaming loader that cannot reach the broker surfaces the error on the loader and keeps retrying; it resumes automatically once the broker is reachable again. Repeated failures eventually pause it — fix the connection and re-save or resume.
- Avro messages not decoding. Set Data Format to Avro and provide the Schema Registry URL (and its key/secret if secured). Without the registry, Avro bytes cannot be decoded.
- Duplicate rows. Delivery is at-least-once, so a message can occasionally be delivered twice; deduplicate on
message_iddownstream if exactly-once semantics matter. - Wrong table / loaders landing together. The target table defaults to
events. If you run more than one streaming or push loader in the same schema, give each a distinct Target Table — otherwise they all default toeventsand land together. - Offsets reset / history re-read. Changing the Consumer Group Id starts a fresh group with no committed offsets, which re-applies the Start Reading From choice. Keep the group id stable to resume where you left off.