Skip to Content
LoadersApache Kafka

Apache Kafka

The Apache Kafka loader streams records from a Kafka topic you own into your warehouse. Instead of your application calling Zeotap, you produce records to a Kafka topic and point Zeotap at it — an always-on consumer reads the topic continuously and lands each message in your warehouse. There is no schedule and no “Run Now”: data arrives as it is produced.

Use it when your data already flows through Kafka (Confluent Cloud, AWS MSK, or a self-managed cluster) and you want it in your warehouse for modeling without building an outbound HTTP producer.

Kafka can also be used as an event streaming source, which feeds the same messages into the real-time events pipeline (identity, journeys, real-time audiences) instead of a warehouse table. Choose the loader when you want records landed for modeling; choose the event source when you want events acted on in real time. See Loader vs. event source.

When to Use It

  • Your records are already produced to an Apache Kafka topic.
  • You want continuous, near-real-time ingestion rather than a scheduled pull.
  • Your payloads are JSON or Avro, and their shape may vary between messages.
  • You would rather grant read access to a topic than build an outbound HTTP caller.

For push ingestion where you control an HTTP producer, use the Generic HTTP Push loader. For a Google Pub/Sub subscription, use the Google Pub/Sub loader. For scheduled pulls from a SaaS API or database, use one of the other loaders.

How It Works

  1. You produce records to a Kafka topic on your cluster.
  2. You create broker credentials (or an API key/secret) that can read that topic.
  3. Zeotap consumes the topic continuously under a consumer group, committing its position only after a message has been durably captured, so a transient failure does not lose data.
  4. Captured messages are batched per loader and loaded into that loader’s configured warehouse.

Delivery is at-least-once: because a message is committed only after it is durably stored, an occasional duplicate is possible but nothing is dropped. Avro messages are decoded to JSON on the way in using your Confluent Schema Registry.

Prerequisites

  • A connected warehouse to load into (BigQuery, Snowflake, Databricks, ClickHouse, or Redshift).
  • A Kafka cluster reachable over the public internet, with the bootstrap broker address (host:port).
  • A topic your records are produced to.
  • Broker credentials that can read the topic — an API key/secret for SASL, or an open listener for an internal cluster.
  • For Avro payloads: a Confluent Schema Registry URL (and its key/secret if secured).

Set Up on Kafka

  1. Identify the topic your records are produced to.
  2. Create a broker API key and secret (on Confluent Cloud or AWS MSK) with read access to that topic, or confirm the listener and mechanism your cluster uses.
  3. Enable TLS on the listener you will connect to (recommended for any credentialed connection).
  4. If your messages are Avro, note your Schema Registry URL and, if it is secured, its key and secret.

Give Zeotap a stable, dedicated consumer group id. Offsets are tracked per consumer group, so a stable id lets Zeotap resume exactly where it left off across restarts, and a dedicated group keeps its progress independent of your other consumers.

Create the Loader

  1. In Zeotap, go to Loaders and click Add Loader.
  2. Choose Apache Kafka.
  3. Provide the configuration below and the broker credentials.
  4. Select the Target Warehouse and Schema where messages should land.
  5. Click Test connection — Zeotap dials a broker, authenticates, and confirms the topic exists.
  6. Save the loader.
FieldRequiredDescription
Kafka Broker hostYesThe bootstrap broker address (host:port). Comma-separate multiple brokers.
Topic nameYesThe topic to read records from.
Consumer Group IdYesThe consumer group Zeotap reads under. Keep it stable so offsets resume across restarts.
AuthenticationYesSASL/SCRAM, SASL/PLAIN, or None — see Authentication.
Use TLSNoEncrypt the broker connection. Recommended whenever credentials are used.
Data FormatYesJSON (default) or Avro. Avro is decoded via the Schema Registry below.
Schema Registry URLFor AvroConfluent Schema Registry base URL. Required when Data Format is Avro.
Schema Registry Key / SecretNoCredentials for a secured Schema Registry (Avro only).
Start Reading FromNoLatest reads only new messages; Earliest replays retained history. Applies on first connect only.
Target WarehouseYesThe warehouse source consumed messages are loaded into.
SchemaYesThe schema/dataset that holds the landed table.
Target TableNoThe table name messages land in. Defaults to events. Set a distinct name per loader if you run more than one streaming or push loader in the same schema, so they do not land together.

The connection test validates your configuration, then actually dials a broker and reads the topic’s partition metadata — a real reachability, authentication, and topic-exists check.

Authentication

MethodUse when
SASL/SCRAMThe default on Confluent Cloud and AWS MSK. Provide the broker API key and secret, and choose the mechanism — SCRAM-SHA-512 (default) or SCRAM-SHA-256.
SASL/PLAINYour cluster uses SASL/PLAIN. Provide the broker API key and secret. Use TLS so the secret is not sent in the clear.
NoneDevelopment or internal clusters with an open listener and no SASL.

Credentials are stored encrypted and never returned.

Landed Format

Every message lands as one row in the target table using a fixed envelope — the raw message is preserved verbatim, so a schema is never required up front:

ColumnTypeDescription
loader_idstringThe loader that consumed the message.
message_idstringA unique id for the delivered message (idempotency key).
workspace_idstringYour workspace id.
received_attimestampWhen Zeotap received the message (the table is time-partitioned on this).
payloadstringThe record, as JSON. Avro messages are decoded to JSON before landing.

Model the payload column downstream (for example with your warehouse’s JSON functions) to extract the fields you need. Because the payload is stored as-is, producers can evolve their record shape without reconfiguring the loader.

Sync Modes

This is a streaming loader: it has no schedule and no “Run Now”. Data arrives whenever your producer writes to the topic and Zeotap consumes it. There is no incremental cursor — every message produced while the loader is active is loaded once (at-least-once).

Rate Limits

Throughput is bounded by your topic’s produce rate and your warehouse’s load throughput, not by a Zeotap-imposed limit. Kafka retains messages up to your topic’s retention window, so a brief slowdown in loading does not drop data.

Troubleshooting

  • Connection test fails. Confirm the bootstrap broker address (host:port) is reachable over the public internet, that TLS matches your listener, and that the credentials are valid. The test reports whether the failure was reachability, authentication, or a missing topic.
  • No data arriving. Confirm your producer is writing to the exact topic name configured, and that the Start Reading From choice matches your expectation — Latest ignores messages produced before the loader first connected.
  • The loader shows an error but keeps retrying. A streaming loader that cannot reach the broker surfaces the error on the loader and keeps retrying; it resumes automatically once the broker is reachable again. Repeated failures eventually pause it — fix the connection and re-save or resume.
  • Avro messages not decoding. Set Data Format to Avro and provide the Schema Registry URL (and its key/secret if secured). Without the registry, Avro bytes cannot be decoded.
  • Duplicate rows. Delivery is at-least-once, so a message can occasionally be delivered twice; deduplicate on message_id downstream if exactly-once semantics matter.
  • Wrong table / loaders landing together. The target table defaults to events. If you run more than one streaming or push loader in the same schema, give each a distinct Target Table — otherwise they all default to events and land together.
  • Offsets reset / history re-read. Changing the Consumer Group Id starts a fresh group with no committed offsets, which re-applies the Start Reading From choice. Keep the group id stable to resume where you left off.
Last updated on