Managing Apicurio Registry deployment

Securing Apicurio Registry API endpoints and web console using Keycloak

The following procedure shows how to configure Apicurio Registry deployment to be protected by Keycloak.

The example configuration in this procedure is intended only for development and testing. In order to keep the procedure simple, it does not use HTTPS and other defenses recommended for a production environment. Please follow the Keycloak documentation for more details.

Apicurio Registry supports following user roles:

Table 1. Supported user roles
Name Capabilities

sr-admin

No restrictions

sr-developer

Can not modify global rules, perform import or export, and use /admin REST API endpoint

sr-readonly

Can not modify artifacts or rules

There is a related configuration option in ApicurioRegistry CRD that can be used to set the web console to read-only mode. However, this configuration does not affect the REST API.
Prerequisites
  • You must have already installed the Apicurio Registry Operator.

  • You must install the Keycloak Operator or have Keycloak accessible from your OpenShift cluster.

Procedure
  1. In the OpenShift web console, click Installed Operators, select the Keycloak Operator details, and then the Keycloak tab.

  2. Click Create Keycloak to provision a new Keycloak instance for securing a Apicurio Registry deployment. You can use the default value, for example:

    apiVersion: keycloak.org/v1alpha1
    kind: Keycloak
    metadata:
      name: example-keycloak
      labels:
        app: sso
    spec:
      instances: 1
      externalAccess:
        enabled: True
      podDisruptionBudget:
        enabled: True
    There is a known Keycloak Operator issue https://issues.redhat.com/browse/KEYCLOAK-17532 which contains a workaround you may need, until it is fully resolved.
  3. When the cluster is ready, click Networking > Route to access the new route to the Keycloak instance. Copy the auth URL value for later use when deploying Apicurio Registry.

  4. Click the Keycloak Realm tab, and then Create Keycloak Realm to create a registry example realm:

    apiVersion: keycloak.org/v1alpha1
    kind: KeycloakRealm
    metadata:
      name: registry-keycloakrealm
    spec:
      instanceSelector:
        matchLabels:
          app: sso
      realm:
        displayName: Registry
        enabled: true
        id: registry
        realm: registry
        sslRequired: none
        roles:
          realm:
            - name: sr-admin
            - name: sr-developer
            - name: sr-readonly
        clients:
          - clientId: registry-client-ui
            implicitFlowEnabled: true
            redirectUris:
              - '*'
            standardFlowEnabled: true
            webOrigins:
              - '*'
            publicClient: true
          - clientId: registry-client-api
            implicitFlowEnabled: true
            redirectUris:
              - '*'
            standardFlowEnabled: true
            webOrigins:
              - '*'
            publicClient: true
        users:
          - credentials:
              - temporary: false
                type: password
                value: changeme
            enabled: true
            realmRoles:
              - sr-admin
            username: registry-admin
          - credentials:
              - temporary: false
                type: password
                value: changeme
            enabled: true
            realmRoles:
              - sr-developer
            username: registry-developer
          - credentials:
              - temporary: false
                type: password
                value: changeme
            enabled: true
            realmRoles:
              - sr-readonly
            username: registry-user

    This KeycloakRealm resource has to be customized with values suitable for your environment, if you are deploying to production. You can also create and manage realms using the web console of Keycloak instance.

  5. If your cluster does not have a valid HTTPS certificate configured, you can create the following HTTP Service and Ingress resources as a temporary workaround:

    apiVersion: v1
    kind: Service
    metadata:
      name: keycloak-http
      labels:
        app: keycloak
    spec:
      ports:
        - name: keycloak-http
          protocol: TCP
          port: 8080
          targetPort: 8080
      selector:
        app: keycloak
        component: keycloak
      type: ClusterIP
      sessionAffinity: None
    status:
      loadBalancer: {}
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: keycloak-http
      labels:
        app: keycloak
    spec:
      rules:
        - host: keycloak-http.local
          http:
            paths:
              - path: /
                pathType: ImplementationSpecific
                backend:
                  serviceName: keycloak-http
                  servicePort: 8080

    Modify the host value to create a route accessible for the Apicurio Registry user, and use it instead of the HTTPS route created by Keycloak Operator.

  6. Select the Apicurio Registry Operator, and in the ApicurioRegistry tab, click Create ApicurioRegistry, using the following example, but replace your values in the keycloak section.

    apiVersion: registry.apicur.io/v1
    kind: ApicurioRegistry
    metadata:
      name: example-apicurioregistry
    spec:
      configuration:
        security:
          keycloak:
            url: "http://keycloak-http-<namespace>.apps.<cluster host>/auth"
            # ^ Required
            # Keycloak server URL, must end with `/auth`.
            # Use an HTTP URL in development.
            realm: "registry"
            # apiClientId: "registry-client-api"
            # ^ Optional (default value)
            # uiClientId: "registry-client-ui"
            # ^ Optional (default value)

Managing Apicurio Registry environment variables

Apicurio Registry Operator manages most common Apicurio Registry configuration, but there are some options that you can adjust manually. You can update these by setting an environment variable on the Apicurio Registry Deployment resource. If the specific configuration option is not available in the ApicurioRegistry CR, you can use an environment variable to adjust it.

Procedure
OpenShift web console
  1. Select the Installed Operators tab, and then the Red Hat Integration - Apicurio Registry Operator.

  2. On the ApicurioRegistry tab, click the ApicurioRegistry CR for your Apicurio Registry deployment.

  3. On the main overview page, view the managedResources section, which contains the name of the Deployment managed by the Operator to deploy your Apicurio Registry instance.

  4. Find that Deployment in the Workloads > Deployments in the left menu.

  5. Select the Deployment with the correct name, and select the Environment tab.

  6. You can add or modify your environment variable to the Single values (env) section.

  7. Click Save at the bottom.

OpenShift CLI
  1. Select the project where Apicurio Registry is installed.

  2. Run oc get apicurioregistry to get the list of ApicurioRegistry CRs

  3. Run oc describe on the CR representing the Apicurio Registry instance that you want to configure.

  4. View managedResources in the status section.

  5. Find that Deployment and enter oc edit.

  6. Add or modify the environment variable in the spec.template.spec.containers[0].env section.

Apicurio Registry liveness and readiness environment variables

Apicurio Registry provides readiness and liveness probes for OpenShift to ensure application health. These settings are configured to use reasonable defaults, but if you want to adjust them, this section provides details on the environment variables that you can configure.

OpenShift liveness and readiness errors are explained as follows:

  • Liveness error - the application cannot make progress, OpenShift restarts the failing pod.

  • Readiness error - the application is not ready, for example, it is overwhelmed by requests. OpenShift stops sending requests to the pod for the time the probe fails. If other pods are OK, they still receive requests.

Table 2. Apicurio Registry liveness environment variables
Name Value type Description

LIVENESS_ERROR_THRESHOLD

Positive integer

Number of liveness errors that can occur before the liveness probe fails.

LIVENESS_COUNTER_RESET

Positive integer, seconds

Period in which the threshold number of errors must occur. For example, if this value is 60 and the threshold is 1, the check fails after two errors occur in 1 minute.

LIVENESS_STATUS_RESET

Positive integer, seconds

Number of seconds that must elapse without any more errors for the liveness probe to reset to OK status. Because OpenShift restarts the pod that fails the liveness check, this value, unlike readiness, does not actually affect what happens.

LIVENESS_ERRORS_IGNORED

Comma-separated list of fully qualified exception class names.

List of errors that are ignored for liveness checking purposes.

Table 3. Apicurio Registry readiness environment variables
Name Value type Description

READINESS_ERROR_THRESHOLD

Positive integer

Number of readiness errors that can occur before the readiness probe fails.

READINESS_COUNTER_RESET

Positive integer, seconds

Period in which the threshold number of errors must occur. For example, if this value is 60 and the threshold is 1, the check fails after two errors occur in 1 minute.

READINESS_STATUS_RESET

Positive integer, seconds

Number of seconds that must elapse without any more errors for the readiness probe to reset to OK status. In practice, this value means how long the pod remains in an unready state until it returns to normal operation.

READINESS_TIMEOUT

positive integer, seconds

The readiness system tracks the timeout of two operations: how long it takes for a storage request to complete, and how long it takes for an HTTP REST API request to return a response. If the operation takes more time, it is counted as a readiness error. This value controls those timeouts.

Configuring an HTTPS connection to Apicurio Registry from inside the OpenShift cluster

The following procedure shows how to configure Apicurio Registry deployment to expose a port for HTTPS connections from inside the OpenShift cluster.

This kind of connection is not directly available outside of the cluster. Routing is based on hostname, which is encoded in the case of an HTTPS connection. Therefore, edge termination or other configuration is still needed. See Configuring an HTTPS connection to Apicurio Registry from outside the OpenShift cluster.
Prerequisites
  • You must have already installed the Apicurio Registry Operator.

Procedure
  1. Generate a keystore with a self-signed certificate. You can skip this step if you are using your own certificates.

    keytool -genkey -trustcacerts -keyalg RSA -keystore registry-keystore.jks -storepass password
  2. Create a new secret to hold the keystore and keystore password.

    1. In the left navigation menu of the OpenShift web console, click Workloads > Secrets > Create Key/Value Secret.

    2. Use the following values:
      Name: registry-keystore
      Key 1: keystore.jks
      Value 1: registry-keystore.jks (uploaded file)
      Key 2: password
      Value 2: password

      If you encounter a java.io.IOException: Invalid keystore format, the upload of the binary file did not work properly. As an alternative, encode the file as a base64 string using cat registry-keystore.jks | base64 -w0 > data.txt and edit the Secret resource as yaml to manually add the encoded file.
  3. Edit the Deployment resource of the Apicurio Registry instance. You can find the correct name in a status field of the Apicurio Registry Operator.

    1. Add the keystore secret as a volume:

      template:
        spec:
          volumes:
          - name: registry-keystore-secret-volume
            secret:
            secretName: registry-keystore
    2. Add a volume mount:

      volumeMounts:
        - name: registry-keystore-secret-volume
          mountPath: /etc/registry-keystore
          readOnly: true
    3. Add JAVA_OPTIONS and KEYSTORE_PASSWORD environment variables:

      - name: KEYSTORE_PASSWORD
        valueFrom:
          secretKeyRef:
            name: registry-keystore
            key: password
      - name: JAVA_OPTIONS
          value: >-
           -Dquarkus.http.ssl.certificate.key-store-file=/etc/registry-keystore/keystore.jks
           -Dquarkus.http.ssl.certificate.key-store-file-type=jks
           -Dquarkus.http.ssl.certificate.key-store-password=$(KEYSTORE_PASSWORD)
      Order is important when using string interpolation.
    4. Enable the HTTPS port:

      ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 8443
          protocol: TCP
  4. Edit the Service resource of the Apicurio Registry instance. You can find the correct name in a status field of the Apicurio Registry Operator.

    ports:
      - name: http
        protocol: TCP
        port: 8080
        targetPort: 8080
      - name: https
        protocol: TCP
        port: 8443
        targetPort: 8443
  5. Verify that the connection is working:

    1. Connect into a pod on the cluster using SSH (you can use the Apicurio Registry pod):

      oc rsh -n default example-apicurioregistry-deployment-vx28s-4-lmtqb
    2. Find the cluster IP of the Apicurio Registry pod from the Service resource (see the Location column in the web console). Afterwards, execute a test request (we are using self-signed certificate, so an insecure flag is required):

      curl -k https://172.30.209.198:8443/health
      [...]

Configuring an HTTPS connection to Apicurio Registry from outside the OpenShift cluster

The following procedure shows how to configure Apicurio Registry deployment to expose an HTTPS edge-terminated route for connections from outside the OpenShift cluster.

Prerequisites
Procedure
  1. Add a second Route in addition to the HTTP route created by the Apicurio Registry Operator. See the following example:

    kind: Route
    apiVersion: route.openshift.io/v1
    metadata:
      [...]
      labels:
        app: example-apicurioregistry
        [...]
    spec:
      host: example-apicurioregistry-default.apps.example.com
      to:
        kind: Service
        name: example-apicurioregistry-service-9whd7
        weight: 100
      port:
        targetPort: 8080
      tls:
        termination: edge
        insecureEdgeTerminationPolicy: Redirect
      wildcardPolicy: None
    Make sure the insecureEdgeTerminationPolicy: Redirect configuration property is set.

    If you do not specify a certificate, OpenShift will use a default. You can alternatively generate a custom self-signed certificate using the following commands:

    openssl genrsa 2048 > host.key &&
    openssl req -new -x509 -nodes -sha256 -days 365 -key host.key -out host.cert

    and then create a route using the OpenShift CLI:

    oc create route edge \
      --service=example-apicurioregistry-service-9whd7 \
      --cert=host.cert --key=host.key \
      --hostname=example-apicurioregistry-default.apps.example.com \
      --insecure-policy=Redirect \
      -n default

Backing up Apicurio Registry PostgreSQL storage

When using storage in a PostgreSQL database, you must ensure that the data stored by Apicurio Registry is backed up regularly.

SQL Dump is a simple procedure that works with any PostgreSQL installation. This uses the pg_dump utility to generate a file with SQL commands that you can use to recreate the database in the same state that it was in at the time of the dump.

pg_dump is a regular PostgreSQL client application, which you can execute from any remote host that has access to the database. Like any other client, the operations that can perform are limited to the user permissions.

Procedure
  • Use the pg_dump command to redirect the output to a file:

     $ pg_dump dbname > dumpfile

    You can specify the database server that pg_dump connects to using the -h host and -p port options.

  • You can reduce large dump files using a compression tool, such as gzip, for example:

     $ pg_dump dbname | gzip > filename.gz
Additional resources

Restoring Apicurio Registry PostgreSQL storage

You can restore SQL Dump files created by pg_dump using the psql utility.

Prerequisites
Procedure
  1. Enter the following command to create the database:

     $ createdb -T template0 dbname
  2. Enter the following command to restore the SQL dump

     $ psql dbname < dumpfile
  3. Run ANALYZE on each database so the query optimizer has useful statistics.