Skip to Content
StreamsApache Kafka Source

Apache Kafka (event source)

Ingest records from an Apache Kafka topic you own into Zeotap as first-class events — participating in schema, contracts, real-time streaming, destination forwarding, and identity. Instead of your application posting to an HTTP endpoint, you point Zeotap at a Kafka topic and an always-on consumer streams the messages into the events pipeline. A translation maps each message into the standard event shape at ingest.

This is the events-pipeline path for Kafka. If you instead want Kafka records landed in a warehouse table for modeling (not the events pipeline), use the Apache Kafka loader.

Loader vs. event source

Kafka can be connected two ways, and they differ by what happens to the messages after they arrive — the connection setup is otherwise the same.

Kafka loaderKafka event source
Messages land inA warehouse table (rows)The real-time events pipeline
Think of it asData at rest, for modelingLive events, acted on as they arrive
Processing on arrivalRaw landing — you model it laterTranslated, identity-resolved, stitched into sessions
DrivesModels → audiences → (batch) syncsReal-time attributes, journeys, real-time audiences
Best forRecords — transactions, catalogs, bulk dataBehavior — clicks, purchases, app events

Use the loader when the topic carries data you want to store and model; use the event source when the topic carries events you want to react to in real time. Nothing stops you connecting one topic as a loader and another as an event source.

How it works

  1. You produce records to a Kafka topic on your cluster.
  2. You create an event source of type Apache Kafka and give Zeotap the broker address, topic, consumer group, and read credentials.
  3. Zeotap consumes the topic continuously under a consumer group, committing its position only after a message is captured.
  4. Each message is translated into the standard event shape and enters the events pipeline — where it is schema-validated, available to real-time features, and eligible for forwarding and identity, exactly like an event from an SDK.

Delivery is at-least-once. Avro messages are decoded to JSON using your Confluent Schema Registry before translation.

Prerequisites

  • A Kafka cluster reachable over the public internet, with the bootstrap broker address (host:port).
  • A topic your records are produced to.
  • Broker credentials that can read the topic — an API key/secret for SASL, or an open listener for an internal cluster.
  • For Avro payloads: a Confluent Schema Registry URL (and its key/secret if secured).

Set up on Kafka

  1. Identify the topic your events are produced to.
  2. Create a broker API key and secret with read access to that topic, or confirm the listener and mechanism your cluster uses.
  3. Enable TLS on the listener you will connect to (recommended for any credentialed connection).
  4. If your messages are Avro, note your Schema Registry URL and, if secured, its key and secret.

Give Zeotap a stable, dedicated consumer group id — one that is not shared with another consumer. Offsets are tracked per consumer group, so a stable, dedicated id lets Zeotap resume where it left off and keeps its progress independent of your other consumers.

Create the event source in Zeotap

  1. Go to Event Sources and click Create.
  2. Give the source a name — how it appears everywhere in Zeotap (the events it sends, the rules that forward them, the contracts that guard them).
  3. Under How events arrive, choose Apache Kafka (we pull from your topic).
  4. Provide the configuration below and the broker credentials.
  5. Save. Events begin streaming in as they are produced. If Zeotap cannot reach the cluster or authenticate, the source’s detail page shows the error — fix it and click Resume.
FieldRequiredDescription
Kafka Broker hostYesThe bootstrap broker address (host:port). Comma-separate multiple brokers.
Topic nameYesThe topic to read events from.
Consumer Group IdYesThe consumer group Zeotap reads under. Keep it stable so offsets resume across restarts.
AuthenticationYesSASL/SCRAM, SASL/PLAIN, or None — see Authentication.
TLSNoEncrypt the broker connection. Recommended whenever credentials are used.
Data FormatYesJSON (default) or Avro. Avro is decoded via the Schema Registry below.
Schema Registry URLFor AvroConfluent Schema Registry base URL. Required when Data Format is Avro.
Schema Registry Key / SecretNoCredentials for a secured Schema Registry (Avro only).
Start reading fromNoLatest reads only new messages; Earliest replays retained history. Applies on first connect only.

Authentication

MethodUse when
SASL/SCRAMThe default on Confluent Cloud and AWS MSK. Provide the broker API key and secret, and choose the mechanism — SCRAM-SHA-512 (default) or SCRAM-SHA-256.
SASL/PLAINYour cluster uses SASL/PLAIN. Provide the broker API key and secret. Use TLS so the secret is not sent in the clear.
NoneDevelopment or internal clusters with an open listener and no SASL.

Credentials are stored encrypted and never returned.

Data format

  • JSON — each message is read as a JSON object and translated into the standard event shape.
  • Avro — Confluent-framed Avro is decoded to JSON using your Schema Registry, then translated. Provide the Schema Registry URL (and its key/secret if secured).

Getting events into your warehouse

Ingesting events is not the same as storing them in your warehouse. Until an event-forwarding rule targets a warehouse, events are accepted, validated, and available to real-time features, but are not written to any warehouse table — and forwarding does not back-fill events received before the rule was created. To persist them, create a forwarding rule — see Event Warehouse. From there they are normal CDP events: build models, audiences, contracts, and forward them to destinations.

If your goal is only to land Kafka records in a warehouse table (not real-time features), the Apache Kafka loader does that directly, without a forwarding rule.

Delivery semantics

At-least-once. A message can occasionally be delivered twice; deduplicate downstream on a stable field in your payload if you need exactly-once.

Troubleshooting

IssueResolution
No events arrivingConfirm your producer is writing to the exact topic name configured, and that Start reading from matches your expectation — Latest ignores messages produced before the source first connected.
The source won’t connect / is pausedThere is no pre-save connection test — a streaming source that cannot reach the broker surfaces the error on its detail page and keeps retrying; repeated failures pause it. Confirm the bootstrap broker address (host:port) is reachable over the public internet, that TLS matches your listener, and that the credentials can read the topic, then Resume the source.
Avro messages not decodingSet Data Format to Avro and provide the Schema Registry URL (and its key/secret if secured).
Events accepted but nothing in the warehouseIngestion and warehouse storage are separate. Create an event-forwarding rule to a warehouse (see Event Warehouse), then send fresh events — forwarding does not back-fill.
Offsets reset / history re-readChanging the Consumer Group Id starts a fresh group with no committed offsets, which re-applies Start reading from. Keep the group id stable to resume where you left off.
Last updated on