Skip to main content
Skip to main content
Edit this page

Schema registries

ClickPipes supports integrating with a schema registry to decode Avro- and Protobuf-encoded topics.

Supported registries for Kafka ClickPipes​

Schema registries that are API-compatible with the Confluent Schema Registry are supported. This includes:

  • Confluent Schema Registry
  • Redpanda Schema Registry

ClickPipes doesn't support AWS Glue Schema Registry or Azure Schema Registry yet. If you require support for these schema registries, reach out to our team.

Configuration​

To integrate with a schema registry during ClickPipes configuration, you must use one of the following approaches:

  1. Provide a complete path to the schema subject (e.g. https://registry.example.com/subjects/events)
    • Optionally, a specific version can be referenced by appending /versions/[version] to the URL (otherwise ClickPipes will retrieve the latest version).
  2. Provide a complete path to the schema ID (e.g. https://registry.example.com/schemas/ids/1000)
  3. Provide the root schema registry URL (e.g. https://registry.example.com)

How it works​

ClickPipes dynamically retrieves and applies the schema from the configured schema registry.

  • If there's a schema ID embedded in the message, it will use that to retrieve the schema.
  • If there's no schema ID embedded in the message, it will use the schema ID or subject name specified in the ClickPipe configuration to retrieve the schema.
  • If the message is written without an embedded schema ID, and no schema ID or subject name is specified in the ClickPipe configuration, then the schema will not be retrieved and the message will be skipped with a SOURCE_SCHEMA_ERROR logged in the ClickPipes error table.
  • If the message does not conform to the schema, then the message will be skipped with a DATA_PARSING_ERROR logged in the ClickPipes error table.
  • For Protobuf schemas only: ClickPipes will load any imported schemas defined as dependencies. Avro schemas with external references are not yet supported.

Schema mapping​

The following rules are applied to the mapping between the retrieved schema and the ClickHouse destination table:

  • If the schema contains a field that is not included in the ClickHouse destination mapping, that field is ignored.
  • If the schema is missing a field defined in the ClickHouse destination mapping, the ClickHouse column will be populated with a "zero" value, such as 0 or an empty string. Note that DEFAULT expressions are not supported.
  • If the schema field and the ClickHouse column are incompatible, inserts of that row/message will fail, and the failure will be recorded in the ClickPipes error table. Note that several implicit conversions are supported (e.g., between numeric types), but not all (e.g., an Avro record field cannot be inserted into an Int32 ClickHouse column).