Deploying Apicurio Registry for high availability

This chapter explains how to configure Apicurio Registry for high availability in a single Kubernetes cluster.

Prerequisites

High availability architecture overview

A highly available Apicurio Registry deployment consists of the following components:

Application (backend) pods

Multiple stateless replicas that process REST API requests. These pods can scale horizontally and are distributed across cluster nodes for fault tolerance.

UI pods

Multiple stateless replicas serving the web console (static files). Typically fewer replicas are needed compared to the backend.

Storage layer

The stateful component requiring HA configuration. The strategy depends on whether you use SQL database or KafkaSQL storage.

The application and UI components are stateless and can be scaled horizontally. High availability is achieved by running multiple replicas distributed across failure domains (availability zones) and implementing a highly available storage layer.

Configuring application component high availability

The Apicurio Registry backend and UI components are stateless and can scale horizontally to provide high availability and increased throughput.

Procedure

  1. Configure the number of replicas for each component in the ApicurioRegistry3 custom resource:

    apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry3
metadata:
  name: example-registry-ha
spec:
  app:
    replicas: 3
    storage:
      type: postgresql
      sql:
        dataSource:
          url: jdbc:postgresql://postgresql-ha.my-project.svc:5432/registry
          username: registry_user
          password:
            name: postgresql-credentials
            key: password
    ingress:
      host: registry.example.com
  ui:
    replicas: 2
    ingress:
      host: registry-ui.example.com
    Running multiple replicas requires a production-ready storage backend (PostgreSQL, MySQL, or KafkaSQL with persistent Kafka). Do not use the default embedded H2 configuration with multiple replicas.

  2. Distribute Apicurio Registry pods across different nodes and availability zones:

    spec:
  app:
    replicas: 3
    podTemplateSpec:
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app: example-registry
                      app.kubernetes.io/component: app
                      app.kubernetes.io/part-of: apicurio-registry
                  topologyKey: kubernetes.io/hostname
              - weight: 50
                podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app: example-registry
                      app.kubernetes.io/component: app
                      app.kubernetes.io/part-of: apicurio-registry
                  topologyKey: topology.kubernetes.io/zone

    This configuration spreads pods across different nodes (hostname) and availability zones when possible.

  3. Set appropriate resource requests and limits to ensure pod scheduling and prevent resource contention:

    spec:
  app:
    podTemplateSpec:
      spec:
        containers:
          - name: apicurio-registry-app
            resources:
              requests:
                memory: "512Mi"
                cpu: "500m"
              limits:
                memory: "1Gi"
                cpu: "1000m"

  4. Ensure minimum availability during voluntary disruptions such as node drains or cluster upgrades. The operator creates a PodDisruptionBudget with maxUnavailable: 1 by default, but you can customize it by creating your own:

    spec:
  app:
    replicas: 3
    podDisruptionBudget:
      enabled: false
    apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
    name: example-registry-app-poddisruptionbudget
spec:
    minAvailable: 1
    selector:
      matchLabels:
        app: example-registry
        app.kubernetes.io/component: app
        app.kubernetes.io/part-of: apicurio-registry

Configuring SQL database storage for high availability

When using PostgreSQL or MySQL storage, high availability depends on your database configuration. The database must be configured with replication and automatic failover.

MySQL storage is not yet supported by the operator and requires manual configuration using environment variables.
Procedure

  • Configure the Agroal connection pool to handle database failover gracefully. Add these environment variables to the app component:

    spec:
  app:
    env:
      # Connection pool sizing
      - name: APICURIO_DATASOURCE_JDBC_INITIAL-SIZE
        value: "10"
      - name: APICURIO_DATASOURCE_JDBC_MIN-SIZE
        value: "10"
      - name: APICURIO_DATASOURCE_JDBC_MAX-SIZE
        value: "50"

      # Connection acquisition timeout (5 seconds)
      - name: QUARKUS_DATASOURCE_JDBC_ACQUISITION-TIMEOUT
        value: "5S"

      # Background validation to detect stale connections (every 2 minutes)
      - name: QUARKUS_DATASOURCE_JDBC_BACKGROUND-VALIDATION-INTERVAL
        value: "2M"

      # Foreground validation before use (every 1 minute)
      - name: QUARKUS_DATASOURCE_JDBC_FOREGROUND-VALIDATION-INTERVAL
        value: "1M"

      # Maximum connection lifetime (30 minutes)
      - name: QUARKUS_DATASOURCE_JDBC_MAX-LIFETIME
        value: "30M"

    These settings ensure that:

  • Connections are validated regularly to detect database failovers

  • Stale connections are removed and recreated

  • Connection acquisition times out rather than hanging indefinitely

SQL database high availability options

When using SQL database storage, consider these database high availability strategies.

PostgreSQL with streaming replication

Primary-replica configuration with automatic failover using tools such as Patroni, CloudNativePG, or Crunchy PostgreSQL Operator.

PostgreSQL with synchronous replication

Ensures zero data loss but might impact performance.

MySQL with Group Replication

Multi-primary or single-primary mode with automatic failover.

Managed database services

Cloud provider managed databases (RDS, Cloud SQL, Azure Database) with built-in HA.

Example: PostgreSQL high availability with CloudNativePG

When using the CloudNativePG operator for PostgreSQL HA, create a Cluster resource and then reference it from your ApicurioRegistry3 custom resource.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: registry-db-cluster
spec:
  instances: 3
  storage:
    size: 20Gi
    storageClass: standard
  postgresql:
    parameters:
      max_connections: "200"
  backup:
    barmanObjectStore:
      # Configure backup storage

Then reference the cluster in your ApicurioRegistry3 CR:

spec:
  app:
    storage:
      type: postgresql
      sql:
        dataSource:
          url: jdbc:postgresql://registry-db-cluster-rw:5432/app
          username: app
          password:
            name: registry-db-cluster-app
            key: password

Configuring KafkaSQL storage for high availability

When using KafkaSQL storage, high availability depends on the Kafka cluster configuration. Each Apicurio Registry replica independently consumes all messages from the Kafka journal topic.

Procedure

  • Configure your Kafka cluster for high availability:

    apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: registry-kafka
spec:
  kafka:
    version: 3.5.0
    replicas: 3
    config:
      # Replication settings for HA
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
    storage:
      type: persistent-claim
      size: 100Gi
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi

    Key configuration for HA:

    Kafka replicas set to 3

    Provides fault tolerance for broker failures.

    Replication factor set to 3

    Each partition has three copies.

    min.insync.replicas set to 2

    Requires at least two replicas to acknowledge writes.

KafkaSQL topic configuration

Apicurio Registry uses three Kafka topics for various purposes:

Journal topic

Stores all changes to the registry data. Named kafkasql-journal by default and is the most important topic for data durability.

Snapshots topic

Stores periodic snapshots of the registry state for faster startup, if the snapshotting feature is used. It’s named kafkasql-snapshots by default and should be configured with similar replication settings as the journal topic.

Events topic

Stores events to support Kafka-based Registry eventing feature. Named registry-events by default, its configuration depends on your event processing needs. Currently, Apicurio Registry sends messages to only 1 partition of this topic.

Configure the Kafka topics used by Apicurio Registry with appropriate replication:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: kafkasql-journal
  labels:
    strimzi.io/cluster: registry-kafka
spec:
  partitions: 3
  replicas: 3
  config:
    cleanup.policy: delete
    min.insync.replicas: 2
    retention.ms: -1  # Infinite retention
    retention.bytes: -1  # Infinite retention
The journal topic and the snapshots topic must use cleanup.policy: delete with infinite retention (retention.ms: -1 and retention.bytes: -1) to prevent accidental data loss. As of the latest version, Apicurio Registry will check these settings on startup and refuse to start if they are not configured correctly.

Apicurio Registry automatically creates the journal, snapshots, and events topics on startup by default, if they do not exist. While this is not recommended in a high-availability production scenario, if you want the topics to be created automatically, the following environment variables provide the equivalent configuration:

APICURIO_KAFKASQL_TOPIC_PARTITIONS=3
APICURIO_KAFKASQL_TOPIC_REPLICATION_FACTOR=3
APICURIO_KAFKASQL_TOPIC_MIN_INSYNC_REPLICAS=2
APICURIO_KAFKASQL_SNAPSHOTS_TOPIC_PARTITIONS=3
APICURIO_KAFKASQL_SNAPSHOTS_TOPIC_REPLICATION_FACTOR=3
APICURIO_KAFKASQL_SNAPSHOTS_TOPIC_MIN_INSYNC_REPLICAS=2

KafkaSQL startup verification

At startup, Apicurio Registry verifies the KafkaSQL topic configuration and the contents of the journal topic before it serves requests. Verification protects against accidental data loss from misconfigured topics or from data that an earlier major version wrote.

Apicurio Registry performs the following checks:

  • Topic configuration: every KafkaSQL topic must use the delete cleanup policy, and the journal and snapshots topics must use unlimited retention (retention.ms=-1 and retention.bytes=-1). When a check fails, Apicurio Registry refuses to start, and the error message names the topic, the effective value, and where the value was configured.

  • Journal contents: Apicurio Registry inspects recent messages in the journal topic. When the journal contains only Apicurio Registry version 2.x records, Apicurio Registry refuses to start to prevent data loss. When the journal contains a mix of version 2.x and 3.x records, Apicurio Registry logs an error and continues.

The journal inspection is bounded: Apicurio Registry captures the topic end offsets before it reads, so messages that other replicas produce concurrently do not delay startup.

To relax the retention verification, for example, when you use the snapshotting feature with automatic cleanup, set apicurio.kafkasql.topic-configuration-verification-override-enabled=true. With this setting, Apicurio Registry does not enforce unlimited retention, so ensure that snapshots are created more frequently than messages are deleted.

Consumer behavior with multiple replicas

When running multiple Apicurio Registry replicas with KafkaSQL storage, each replica independently consumes the full Kafka journal topic to build its own local state.

  • Each replica uses a unique consumer group ID (automatically generated using UUID)

  • Each replica independently consumes all messages from the journal topic

  • There is no consumer group rebalancing between replicas

  • All replicas build the same local state from the Kafka topic (Apicurio Registry replays data into an embedded H2 database)

This design ensures that:

  • New replicas can be added without affecting existing replicas

  • Pod restarts only affect the restarting pod, not others

  • Each replica maintains a consistent view of the data

Do not configure a fixed Kafka consumer group ID (for example, by setting APICURIO_KAFKASQL_CONSUMER_GROUP_ID). KafkaSQL uses an embedded H2 database that is empty on every restart, so the full journal topic must be replayed from the beginning each time. A fixed consumer group ID causes Kafka to resume from previously committed offsets, skipping the journal replay and resulting in the registry appearing empty after a restart. If you need a recognizable consumer group name for monitoring, use the APICURIO_KAFKASQL_CONSUMER_GROUP_PREFIX environment variable instead, which sets a prefix while still generating a unique group ID on each startup.

Operator leader election

When you run more than one replica of the Apicurio Registry Operator, Kubernetes Lease-based leader election ensures that only one replica reconciles resources at a time. Leader election is enabled by default.

You can configure leader election with the following environment variables on the operator deployment:

Table 1. Operator leader election environment variables
Environment variable Default Description

APICURIO_OPERATOR_LEADER_ELECTION_ENABLED

true

Enables Lease-based leader election. When the lease namespace cannot be resolved, leader election is disabled automatically.

APICURIO_OPERATOR_LEADER_ELECTION_LEASE_NAME

apicurio-registry-operator-lease

Name of the Lease resource.

APICURIO_OPERATOR_LEADER_ELECTION_LEASE_NAMESPACE

The operator pod namespace

Namespace of the Lease resource.

High availability monitoring

Apicurio Registry exposes Prometheus metrics for monitoring application health and performance.

Metrics are enabled by default. Access the metrics endpoint at /metrics on the management port (default 9000).

Configure alerts for:

  • Pod restarts or crash loops

  • High error rates (5xx responses)

  • Storage operation timeouts

  • Database connection pool exhaustion

  • Kafka consumer lag (for KafkaSQL storage)

Key metrics to monitor

Monitor the following categories of metrics for high availability deployments of Apicurio Registry.

  • REST API metrics:

    • http_server_requests_seconds - Request latency

    • http_server_active_requests - Concurrent requests

    • http_server_requests_total - Total request count

  • Storage metrics:

    • apicurio_storage_operation_seconds - Storage operation latency

    • apicurio_storage_concurrent_operations - Concurrent storage operations

    • apicurio_storage_operation_total - Total storage operations

  • Health check metrics:

    • Readiness probe: /health/ready (on management port 9000)

    • Liveness probe: /health/live (on management port 9000)

  • JVM metrics:

    • jvm_memory_used_bytes - Memory usage

    • jvm_gc_pause_seconds - Garbage collection pauses

Configuring a ServiceMonitor for Prometheus Operator

If you use the Prometheus Operator, create a ServiceMonitor resource to scrape metrics from your Apicurio Registry deployment.

Prerequisites

  • The Prometheus Operator is installed in your cluster.

Procedure

  • Create a ServiceMonitor resource that selects your Apicurio Registry pods:

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: apicurio-registry-metrics
  labels:
    app: apicurio-registry
spec:
  selector:
    matchLabels:
      app: apicurio-registry
  endpoints:
    - port: management
      path: /metrics
      interval: 30s

Performing rolling updates

When updating Apicurio Registry to a new version, use rolling updates to minimize downtime.

Procedure

  • Update Apicurio Registry Operator to update Apicurio Registry to a new version. Apicurio Registry and Apicurio Registry Operator are versioned together, so updating Apicurio Registry Operator automatically updates the application. For production environments, we strongly recommend using Operator Lifecycle Manager (OLM) with manual install plan confirmation enabled, so you have time to review release notes and prepare for the update.

    If you need to update the application without updating Apicurio Registry Operator (not recommended except as a workaround for critical issues), you can override the application image in your pod template, as shown in the following example. 
    spec:
  app:
    podTemplateSpec:
      spec:
        containers:
          - name: apicurio-registry-app
            image: quay.io/apicurio/apicurio-registry:3.3.0

Operator rolling update strategy

The operator performs rolling updates automatically when you update the ApicurioRegistry3 CR. The default strategy ensures the following.

  • Pods are updated one at a time

  • Each new pod must pass readiness checks before the next pod is updated

  • PodDisruptionBudget prevents too many pods being unavailable

Safe update practices

Follow these practices to update your Apicurio Registry deployment safely and minimize risk.

Test in non-production first

Validate the new version in a test environment.

Monitor during rollout

Watch metrics and logs during the update.

Maintain minimum replicas

Keep at least two replicas to ensure availability during updates.

Review release notes

Check for breaking changes or migration steps.

For patch releases (for example, 3.1.0 to 3.1.1), rolling updates typically complete without issues. For major or minor version updates, always review the migration guide.

Upgrading to the management interface (port 9000)

Starting with Apicurio Registry 3.x, health check and metrics endpoints are served on a dedicated management port (9000) instead of the main application port (8080). This is a breaking change that requires updates to your deployment configuration when upgrading from earlier versions.

Procedure

  1. Update Kubernetes liveness and readiness probes to target port 9000 instead of port 8080. The endpoint paths (/health/ready and /health/live) remain the same.

    # Before (old configuration)
livenessProbe:
  httpGet:
    path: /health/live
    port: 8080

# After (new configuration)
livenessProbe:
  httpGet:
    path: /health/live
    port: 9000

# After (with TLS enabled)
livenessProbe:
  httpGet:
    path: /health/live
    port: 9000
    scheme: HTTPS

  2. Update Prometheus scrape targets: update your Prometheus configuration or ServiceMonitor resources to scrape metrics from port 9000 at path /metrics.

  3. If you use Kubernetes NetworkPolicy resources, ensure that ingress traffic on port 9000 is allowed from your monitoring infrastructure and the kubelet (for health probes).

  4. Review TLS considerations. The management interface inherits the default TLS configuration from the Quarkus TLS registry. When TLS is enabled, the management interface on port 9000 also serves HTTPS, and health probes must use scheme: HTTPS. The Apicurio Registry Operator handles this automatically.

  5. If you are using the Apicurio Registry Operator, confirm that no manual changes are required: it automatically configures the correct probe ports for operator-managed deployments.

Backup and restore procedures

Regular backups are essential for disaster recovery and data protection.

Apicurio Registry supports backup and restore for both SQL database storage and KafkaSQL storage. Apicurio Registry also exposes a health endpoint (/health/ready) and a metrics endpoint (/metrics) on the management port (9000), which you can use to monitor deployment health during backup and restore operations.

Additional resources

Backing up and restoring SQL database storage

For SQL storage (PostgreSQL or MySQL), backup and restore strategies depend on your database setup.

Procedure

  1. Back up a PostgreSQL database using one of the following options:

    • Create a logical backup with pg_dump:

      pg_dump -h postgresql-host -U registry_user -d registry > registry-backup.sql

    • Use continuous archiving with WAL for point-in-time recovery.

    • Use operator-managed backups if you use CloudNativePG or Crunchy PostgreSQL Operator:

      spec:
  backup:
    barmanObjectStore:
      destinationPath: s3://my-backups/registry-db
      s3Credentials:
        accessKeyId:
          name: backup-credentials
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: backup-credentials
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
    retentionPolicy: "30d"

  2. Back up a MySQL database using one of the following options:

    • Create a logical backup with mysqldump:

      mysqldump -h mysql-host -u registry_user -p registry > registry-backup.sql

    • Create a binary backup using tools like Percona XtraBackup or MySQL Enterprise Backup.

  3. Restore from a logical backup:

    # PostgreSQL
psql -h postgresql-host -U registry_user -d registry < registry-backup.sql

# MySQL
mysql -h mysql-host -u registry_user -p registry < registry-backup.sql

Backing up KafkaSQL storage

For KafkaSQL storage, back up the Kafka journal topic and snapshots topic.

Procedure

  • Use Kafka’s MirrorMaker 2 or Strimzi Mirror Maker for topic replication (recommended):

    apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: registry-backup-mirror
spec:
  version: 3.5.0
  replicas: 1
  connectCluster: "backup-cluster"
  clusters:
    - alias: "source-cluster"
      bootstrapServers: registry-kafka-bootstrap:9092
    - alias: "backup-cluster"
      bootstrapServers: backup-kafka-bootstrap:9092
  mirrors:
    - sourceCluster: "source-cluster"
      targetCluster: "backup-cluster"
      topicsPattern: "kafkasql-.*"
      groupsPattern: ".*"

    Alternatively, export and import topics using Kafka’s console consumer or CLI tools like kcat/kafkacat. Since the topics might contain binary data, which might be affected by text file encoding, make sure they are exported correctly when using CLI tools.

Additional resources

Testing backup and restore

Regularly test your backup and restore procedures to confirm that they work when you need them.

Procedure

  1. Create a test Apicurio Registry deployment in a separate namespace.

  2. Restore from backup to the test deployment.

  3. Verify that all artifacts and metadata are present.

  4. Test API functionality to ensure data integrity.