Deploying Apicurio Registry with a secured Kafka instance

You can deploy Apicurio Registry with KafkaSQL storage connected to a Kafka instance that requires TLS encryption, SASL authentication, or both. Apicurio Registry supports multiple security configurations for Kafka connections, including TLS-only encryption, OAUTHBEARER authentication, and SCRAM-SHA-512 authentication.

Secured Kafka overview
Prerequisites for secured Kafka deployment
Configuring KafkaSQL with TLS encryption
Configuring KafkaSQL with SCRAM-SHA-512 authentication
Configuring KafkaSQL with OAUTHBEARER authentication
Configuring KafkaSQL with the Strimzi Kafka Access Operator
Kafka security protocol reference
Configuring Kafka topics for Apicurio Registry
Troubleshooting secured Kafka connections

Secured Kafka overview

When you use KafkaSQL storage, Apicurio Registry connects to a Kafka cluster as both a producer and a consumer. If your Kafka cluster enforces TLS encryption or SASL authentication, you must configure Apicurio Registry with the appropriate security settings.

You can configure secured Kafka connections by using these methods:

The ApicurioRegistry3 custom resource tls and auth sections for TLS and OAUTHBEARER authentication
Environment variables in the spec.app.env section of the custom resource for SCRAM-SHA-512 authentication
The Strimzi Kafka Access Operator (available starting with Apicurio Registry version 3.2.0) for automatic configuration

The security protocol that Apicurio Registry uses depends on which combination of TLS and authentication you configure:

Table 1. Security protocol matrix
TLS configured	Authentication configured	Security protocol
No	No	`PLAINTEXT`
Yes	No	`SSL`
No	Yes	`SASL_PLAINTEXT`
Yes	Yes	`SASL_SSL`

Prerequisites for secured Kafka deployment

Before you deploy Apicurio Registry with a secured Kafka instance, verify that you have the required resources.

You have cluster administrator access to an Kubernetes cluster.
You have installed the Apicurio Registry Operator.
You have a Kafka cluster with security enabled (TLS, SASL, or both).
- For Strimzi, see the Strimzi documentation.
If your Kafka cluster uses TLS, you have access to the cluster CA certificate in PKCS12 format.
If your Kafka cluster uses SASL authentication, you have the required credentials (SCRAM-SHA-512 username and password, or OAUTHBEARER client ID, client secret, and token endpoint URL).

Configuring KafkaSQL with TLS encryption

When your Kafka cluster requires TLS encryption, you configure Apicurio Registry with the Kafka cluster CA certificate so that Apicurio Registry can verify the broker identity.

Prerequisites

The Kafka cluster CA certificate is available as a Kubernetes Secret in PKCS12 format.
- Strimzi creates a Secret named <cluster-name>-cluster-ca-cert that contains the CA certificate.

Procedure

Create or update your ApicurioRegistry3 custom resource with the tls configuration:

apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry3
metadata:
  name: example-registry-kafkasql-tls
spec:
  app:
    storage:
      type: kafkasql
      kafkasql:
        bootstrapServers: "my-cluster-kafka-bootstrap.my-project.svc:9093"
        tls:
          truststoreSecretRef:
            name: my-cluster-cluster-ca-cert
            key: ca.p12
          truststorePasswordSecretRef:
            name: my-cluster-cluster-ca-cert
            key: ca.password
    ingress:
      host: registry.example.com
  ui:
    ingress:
      host: registry-ui.example.com

Apply the custom resource:
```
{kubernetes-client} apply -f my-registry.yaml
```
The operator sets the security protocol to SSL when you configure TLS without authentication.

Optional: For mutual TLS (mTLS), where the Kafka cluster also verifies the Apicurio Registry client certificate, add the keystore configuration to the tls section:

tls:
  keystoreSecretRef:
    name: apicurio-client-auth
    key: user.p12
  keystorePasswordSecretRef:
    name: apicurio-client-auth
    key: user.password
  truststoreSecretRef:
    name: my-cluster-cluster-ca-cert
    key: ca.p12
  truststorePasswordSecretRef:
    name: my-cluster-cluster-ca-cert
    key: ca.password

Configuring KafkaSQL with SCRAM-SHA-512 authentication

When your Kafka cluster uses SCRAM-SHA-512 authentication, you configure the SASL credentials by using environment variables in the spec.app.env section of the ApicurioRegistry3 custom resource.

The structured auth section of the custom resource supports only OAUTHBEARER authentication. For SCRAM-SHA-512, you must use environment variables as described in this procedure.

Prerequisites

Your Kafka cluster is configured with a SASL listener that uses SCRAM-SHA-512.
You have created a Kubernetes Secret that contains the JAAS configuration for your Kafka user.
- Strimzi automatically creates this Secret when you create a KafkaUser resource with authentication.type: scram-sha-512.

Procedure

Verify that your Kubernetes Secret contains the sasl.jaas.config key with the JAAS login module definition:
```
org.apache.kafka.common.security.scram.ScramLoginModule required username="my-user" password="<password>";
```
The sasl.jaas.config value must be a one-line login module definition. Do not include the KafkaClient { … } wrapper.

Create or update your ApicurioRegistry3 custom resource with the SCRAM-SHA-512 configuration:

apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry3
metadata:
  name: example-apicurioregistry-kafkasql-scram
spec:
  app:
    storage:
      type: kafkasql
      kafkasql:
        bootstrapServers: "my-cluster-kafka-bootstrap.my-project.svc:9093"
        tls:
          truststoreSecretRef:
            name: my-cluster-cluster-ca-cert
            key: ca.p12
          truststorePasswordSecretRef:
            name: my-cluster-cluster-ca-cert
            key: ca.password
    env:
      - name: APICURIO_KAFKA_COMMON_SECURITY_PROTOCOL
        value: "SASL_SSL"
      - name: APICURIO_KAFKA_COMMON_SASL_MECHANISM
        value: "SCRAM-SHA-512"
      - name: APICURIO_KAFKA_COMMON_SASL_JAAS_CONFIG
        valueFrom:
          secretKeyRef:
            name: my-user
            key: sasl.jaas.config
    ingress:
      host: registry.example.com
  ui:
    ingress:
      host: registry-ui.example.com

Apply the custom resource:

{kubernetes-client} apply -f my-registry.yaml

Verify that the Apicurio Registry pod starts successfully:
```
{kubernetes-client} get pods -l app=example-apicurioregistry-kafkasql-scram
```
This example uses SASL_SSL because TLS is also configured via the tls section. If you use SCRAM-SHA-512 without TLS, set APICURIO_KAFKA_COMMON_SECURITY_PROTOCOL to SASL_PLAINTEXT instead.

Configuring KafkaSQL with OAUTHBEARER authentication

When your Kafka cluster uses OAUTHBEARER authentication, you configure the OAuth credentials by using the structured auth section of the ApicurioRegistry3 custom resource.

Prerequisites

Your Kafka cluster is configured with an OAuth-enabled listener.
You have the OAuth client ID, client secret, and token endpoint URL.
You have stored the client ID and client secret in a Kubernetes Secret.

Procedure

Create a Kubernetes Secret that contains the OAuth client credentials:

{kubernetes-client} create secret generic kafka-oauth-credentials \
  --from-literal=clientId=my-client-id \
  --from-literal=clientSecret=my-client-secret

Create or update your ApicurioRegistry3 custom resource with the OAUTHBEARER configuration:

apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry3
metadata:
  name: example-registry-kafkasql-oauth
spec:
  app:
    storage:
      type: kafkasql
      kafkasql:
        bootstrapServers: "my-cluster-kafka-bootstrap.my-project.svc:9093"
        tls:
          truststoreSecretRef:
            name: my-cluster-cluster-ca-cert
            key: ca.p12
          truststorePasswordSecretRef:
            name: my-cluster-cluster-ca-cert
            key: ca.password
        auth:
          enabled: true
          mechanism: OAUTHBEARER
          clientIdRef:
            name: kafka-oauth-credentials
            key: clientId
          clientSecretRef:
            name: kafka-oauth-credentials
            key: clientSecret
          tokenEndpoint: https://identity-server.example.com/token
          loginHandlerClass: io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler
    ingress:
      host: registry.example.com
  ui:
    ingress:
      host: registry-ui.example.com

Apply the custom resource:
```
{kubernetes-client} apply -f my-registry.yaml
```
The loginHandlerClass value is from the Strimzi OAuth library. If you use a different OAuth provider, replace this value with the login handler class for your provider.

When you configure both TLS and OAUTHBEARER authentication, the operator sets the security protocol to SASL_SSL.

Configuring KafkaSQL with the Strimzi Kafka Access Operator

Starting with Apicurio Registry version 3.2.0, you can use the Strimzi Kafka Access Operator to automatically provide all Kafka connection details, including security credentials, in a single Kubernetes Secret.

Prerequisites

The Strimzi Kafka Access Operator is installed on your Kubernetes cluster.
You have a Strimzi Kafka cluster with a configured listener and a KafkaUser resource.

Procedure

Create a KafkaUser resource that grants Apicurio Registry access to the required Kafka topics:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: apicurio-registry
  labels:
    strimzi.io/cluster: example-cluster
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: kafkasql-journal
        operations:
          - All
      - resource:
          type: topic
          name: kafkasql-snapshots
        operations:
          - All
      - resource:
          type: topic
          name: registry-events
        operations:
          - All
      - resource:
          type: group
          name: apicurio-registry
          patternType: prefix
        operations:
          - Read

This example scopes consumer group access to groups that start with apicurio-registry. In production deployments, avoid granting All operations on all consumer groups (name: '*'). Adjust the group name prefix to match your Apicurio Registry consumer group configuration.

Create a KafkaAccess resource that references your Kafka cluster and user:

apiVersion: access.strimzi.io/v1alpha1
kind: KafkaAccess
metadata:
  name: my-kafka-access
spec:
  kafka:
    name: example-cluster
    listener: tls
  user:
    kind: KafkaUser
    apiGroup: kafka.strimzi.io
    name: apicurio-registry

The KafkaAccess resource uses the access.strimzi.io/v1alpha1 API version, which is an alpha API that might change in future Strimzi releases. Check the Strimzi Kafka Access Operator documentation for the current API version.

Create the ApicurioRegistry3 custom resource that references the KafkaAccess Secret:

apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry3
metadata:
  name: example-kafkasql-access
spec:
  app:
    storage:
      type: kafkasql
      kafkasql:
        kafkaAccessSecretName: my-kafka-access
    ingress:
      host: registry.example.com
  ui:
    ingress:
      host: registry-ui.example.com

Apply all resources:

{kubernetes-client} apply -f kafkauser.yaml -f kafkaaccess.yaml -f my-registry.yaml

When you specify kafkaAccessSecretName, it takes precedence over bootstrapServers. The Kafka Access Operator provides all connection details automatically, including bootstrap servers, security protocol, TLS certificates, and SASL credentials.

Kafka security protocol reference

Apicurio Registry uses the apicurio.kafka.common.security.protocol property (environment variable APICURIO_KAFKA_COMMON_SECURITY_PROTOCOL) to determine how to connect to the Kafka cluster.

When you use the operator, the operator sets the security protocol automatically based on your TLS and authentication configuration. When you configure security manually by using environment variables, you must set the security protocol explicitly.

Table 2. Kafka security protocol values
Protocol	Description
`PLAINTEXT`	No encryption and no authentication. Use for unsecured Kafka clusters only.
`SSL`	TLS encryption without SASL authentication. The operator sets this protocol when you configure TLS only.
`SASL_PLAINTEXT`	SASL authentication without TLS encryption. The operator sets this protocol when you configure OAUTHBEARER authentication without TLS.
`SASL_SSL`	TLS encryption with SASL authentication. The operator sets this protocol when you configure both TLS and OAUTHBEARER authentication. You must set this protocol manually when you use SCRAM-SHA-512 authentication.

Configuring Kafka topics for Apicurio Registry

Apicurio Registry uses Kafka topics to store artifact data. You must configure these topics with the correct retention settings to prevent data loss.

By default, Apicurio Registry auto-creates the required topics. If your Kafka cluster does not allow auto-creation, or if you want to control topic settings, create the topics manually.

Procedure

Create the Kafka topics by using the KafkaTopic custom resource:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: kafkasql-journal
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 3
  config:
    cleanup.policy: delete
    retention.ms: "-1"
    retention.bytes: "-1"
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: kafkasql-snapshots
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 3
  replicas: 3
  config:
    cleanup.policy: delete
    retention.ms: "-1"
    retention.bytes: "-1"

This example uses partitions: 3 and replicas: 3, which requires at least 3 Kafka brokers. The replica count must not exceed the number of brokers in your cluster. Adjust partition and replica counts based on your deployment size.

You must configure the Kafka topics used by Apicurio Registry with cleanup.policy: delete and infinite retention (retention.ms: -1 and retention.bytes: -1). Do not use cleanup.policy: compact or cleanup.policy: compact,delete, because both policies can result in data loss.

Apply the topic resources:

{kubernetes-client} apply -f kafkatopics.yaml

Troubleshooting secured Kafka connections

You might encounter errors when you configure Apicurio Registry to connect to a secured Kafka instance.

Table 3. Common secured Kafka connection errors and fixes
Error	Cause	Fix
`java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set`	SASL authentication requires a valid login context, but the `sasl.jaas.config` property is not set correctly.	Set the `APICURIO_KAFKA_COMMON_SASL_JAAS_CONFIG` environment variable by using a `secretKeyRef` that points to a Secret containing the one-line JAAS login module definition.
`java.io.IOException: /etc/kafka/secrets/jaas.conf (No such file or directory)`	A previous configuration attempted to mount a physical `jaas.conf` file that no longer exists. Cached `JAVA_TOOL_OPTIONS` or invalid volume mounts can persist after changes.	Remove any `JAVA_TOOL_OPTIONS` environment variable or volume mount that references `jaas.conf` from the custom resource. Use the `APICURIO_KAFKA_COMMON_SASL_JAAS_CONFIG` environment variable instead of file mounts.
`Admission Webhook Warning: unknown field "spec.app.storage.kafkasql.auth.usernameSecretRef"`	The structured `auth` section of the custom resource does not support SCRAM-SHA-512 fields like `usernameSecretRef`. These fields are not part of the `ApicurioRegistry3` CRD.	Use environment variables (`APICURIO_KAFKA_COMMON_SASL_MECHANISM` and `APICURIO_KAFKA_COMMON_SASL_JAAS_CONFIG`) in the `spec.app.env` section instead of the structured `auth` section for SCRAM-SHA-512 authentication.

Additional resources