> ## Documentation Index
> Fetch the complete documentation index at: https://www.propeldata.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Kafka to ClickHouse

> Ingest data from Kafka topics.

<div />

Ingest real-time data from [self-hosted Kafka](/docs/ingestion/kafka/setup), [Confluent Cloud](/docs/ingestion/kafka/setup), [AWS MSK](/docs/ingestion/kafka/setup), or [Redpanda](/docs/ingestion/kafka/setup) to Propel.

<Card title="Get started with Kafka" icon="link" href="/docs/ingestion/kafka/setup">
  Step-by-step instructions to connect your Kafka cluster to Propel.
</Card>

## Architecture

The Kafka Data Pools connect to specified Kafka topics to ingest data in real-time into Propel.

<Frame>
  <img src="https://mintcdn.com/propeldocs/9FdNDTkhOu7EgAUd/images/docs/kafka-architecture-diagram-docs.svg?fit=max&auto=format&n=9FdNDTkhOu7EgAUd&q=85&s=0e699706fd29725df1e6f61fa335af83" alt="The architectural overview when connecting Kafka to Propel." width="1100" height="301" data-path="images/docs/kafka-architecture-diagram-docs.svg" />
</Frame>

## Features

Kafka Data Pools support the following features:

| Feature name           | Supported | Notes                                                                    |
| ---------------------- | --------- | ------------------------------------------------------------------------ |
| Real-time ingestion    | ✅         | See [How the Kafka Data Pool works](#how-does-the-kafka-data-pool-work). |
| Deduplication          | ✅         | See the [deduplication](#message-deduplication) section.                 |
| Batch Delete API       | ✅         | See [Batch Delete API](/docs/deleting-data#batch-deletes).               |
| Batch Update API       | ✅         | See [Batch Update API](/docs/updating-data#batch-updates).               |
| API configurable       | ✅         | See  API]\(/docs/management-api) docs.                                   |
| Terraform configurable | ✅         | See [Terraform](/docs/terraform) docs.                                   |

## How does the Kafka Data Pool work?

The Kafka Data Pool connects to specified Kafka topic to read messages in real-time. It starts from the earliest available offset to start consuming messages.

These messages are then ingested into the Data Pool. Once in the Data Pool, you can query them via [SQL](/docs/sql-reference), the [Query APIs](/docs/query-apis), or [transform them with Materialized Views](/docs/materialized-views).

### Schemaless ingestion

The Kafka Data Pool stores the message body in the `_propel_payload` column and the Kafka and ingestion-related metadata in other columns.

This flexible approach allows JSON Kafka messages to be ingested without needing pre-defined schemas.

| Column                | Type      | Description                         |
| --------------------- | --------- | ----------------------------------- |
| `_timestamp`          | TIMESTAMP | The timestamp of the message.       |
| `_topic`              | STRING    | The Kafka topic                     |
| `_key`                | STRING    | The key of the message.             |
| `_offset`             | INT64     | The offset of the message.          |
| `_partition`          | INT64     | The partition of Kafka topic.       |
| `_propel_payload`     | JSON      | The raw message Payload in JSON.    |
| `_propel_received_at` | TIMESTAMP | When the message is read by Propel. |

### Message deduplication

The Kafka Data Pool automatically manages the deduplication of messages. This happens when messages are either sent twice by the producer or read twice due to intermittent connectivity between Propel and the Kafka stream. The uniqueness of a message is determined by the combination of `_topic`, `_partition`, and `_offset`.

## Supported formats

The Kafka Data Pool supports the ingestion of JSON messages that are stored in the `_propel_payload` column.

<Info>If you need AVRO support, please [contact us](https://www.propeldata.com/contact).</Info>

## Transforming data

Once your data is in a Kafka Data Pool, you can use [Materialized Views](/docs/materialized-views) to:

* [Flatten nested JSON into tabular form](/docs/materialized-views#example-1-flatten-nested-json-into-tabular-form)
* [Flatten JSON array into individual rows](/docs/materialized-views#example-2-flatten-json-array-into-individual-rows)
* [Combine data from multiple source Data Pools through JOINs](/docs/materialized-views#example-3-combines-data-from-multiple-source-tables-through-joins)
* [Calculate new derived columns from existing data](/docs/materialized-views#example-4-calculates-new-derived-columns-from-existing-data)
* [Perform incremental aggregations](/docs/materialized-views#example-5-perform-incremental-aggregations)
* [Sort rows with a different sorting key](/docs/materialized-views#example-6-sorts-rows-with-a-different-sorting-key)
* [Filter out unnecessary data based on conditions](/docs/materialized-views#example-7-filters-out-unnecessary-data-based-on-conditions)
* [De-duplicate rows](/docs/materialized-views#example-8-deduplicating-rows)