Managing Apicurio Registry deployment

This chapter explains how to manage your Apicurio Registry deployment:

Securing Apicurio Registry API endpoints and web console using Keycloak
Managing Apicurio Registry environment variables
Apicurio Registry liveness and readiness environment variables
Configuring an HTTPS connection to Apicurio Registry from inside the OpenShift cluster
Configuring an HTTPS connection to Apicurio Registry from outside the OpenShift cluster
Backing up Apicurio Registry PostgreSQL storage
Restoring Apicurio Registry PostgreSQL storage

Securing Apicurio Registry API endpoints and web console using Keycloak

The following procedure shows how to configure Apicurio Registry deployment to be protected by Keycloak.

The example configuration in this procedure is intended only for development and testing. In order to keep the procedure simple, it does not use HTTPS and other defenses recommended for a production environment. Please follow the Keycloak documentation for more details.

Apicurio Registry supports following user roles:

Table 1. Supported user roles
Name	Capabilities
`sr-admin`	No restrictions
`sr-developer`	Can not modify global rules, perform import or export, and use `/admin` REST API endpoint
`sr-readonly`	Can not modify artifacts or rules

There is a related configuration option in ApicurioRegistry CRD that can be used to set the web console to read-only mode. However, this configuration does not affect the REST API.

Prerequisites

You must have already installed the Apicurio Registry Operator.
You must install the Keycloak Operator or have Keycloak accessible from your OpenShift cluster.

Procedure

In the OpenShift web console, click Installed Operators, select the Keycloak Operator details, and then the Keycloak tab.
Click Create Keycloak to provision a new Keycloak instance for securing a Apicurio Registry deployment. You can use the default value, for example:
```
apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: example-keycloak
  labels:
    app: sso
spec:
  instances: 1
  externalAccess:
    enabled: True
  podDisruptionBudget:
    enabled: True
```
There is a known Keycloak Operator issue https://issues.redhat.com/browse/KEYCLOAK-17532 which contains a workaround you may need, until it is fully resolved.
When the cluster is ready, click Networking > Route to access the new route to the Keycloak instance. Copy the auth URL value for later use when deploying Apicurio Registry.

Click the Keycloak Realm tab, and then Create Keycloak Realm to create a registry example realm:

apiVersion: keycloak.org/v1alpha1
kind: KeycloakRealm
metadata:
  name: registry-keycloakrealm
spec:
  instanceSelector:
    matchLabels:
      app: sso
  realm:
    displayName: Registry
    enabled: true
    id: registry
    realm: registry
    sslRequired: none
    roles:
      realm:
        - name: sr-admin
        - name: sr-developer
        - name: sr-readonly
    clients:
      - clientId: registry-client-ui
        implicitFlowEnabled: true
        redirectUris:
          - '*'
        standardFlowEnabled: true
        webOrigins:
          - '*'
        publicClient: true
      - clientId: registry-client-api
        implicitFlowEnabled: true
        redirectUris:
          - '*'
        standardFlowEnabled: true
        webOrigins:
          - '*'
        publicClient: true
    users:
      - credentials:
          - temporary: false
            type: password
            value: changeme
        enabled: true
        realmRoles:
          - sr-admin
        username: registry-admin
      - credentials:
          - temporary: false
            type: password
            value: changeme
        enabled: true
        realmRoles:
          - sr-developer
        username: registry-developer
      - credentials:
          - temporary: false
            type: password
            value: changeme
        enabled: true
        realmRoles:
          - sr-readonly
        username: registry-user

This KeycloakRealm resource has to be customized with values suitable for your environment, if you are deploying to production. You can also create and manage realms using the web console of Keycloak instance.

If your cluster does not have a valid HTTPS certificate configured, you can create the following HTTP Service and Ingress resources as a temporary workaround:

apiVersion: v1
kind: Service
metadata:
  name: keycloak-http
  labels:
    app: keycloak
spec:
  ports:
    - name: keycloak-http
      protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: keycloak
    component: keycloak
  type: ClusterIP
  sessionAffinity: None
status:
  loadBalancer: {}
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: keycloak-http
  labels:
    app: keycloak
spec:
  rules:
    - host: keycloak-http.local
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              serviceName: keycloak-http
              servicePort: 8080

Modify the host value to create a route accessible for the Apicurio Registry user, and use it instead of the HTTPS route created by Keycloak Operator.

Select the Apicurio Registry Operator, and in the ApicurioRegistry tab, click Create ApicurioRegistry, using the following example, but replace your values in the keycloak section.

apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry
metadata:
  name: example-apicurioregistry
spec:
  configuration:
    security:
      keycloak:
        url: "http://keycloak-http-<namespace>.apps.<cluster host>/auth"
        # ^ Required
        # Keycloak server URL, must end with `/auth`.
        # Use an HTTP URL in development.
        realm: "registry"
        # apiClientId: "registry-client-api"
        # ^ Optional (default value)
        # uiClientId: "registry-client-ui"
        # ^ Optional (default value)

Managing Apicurio Registry environment variables

Apicurio Registry Operator manages most common Apicurio Registry configuration, but there are some options that you can adjust manually. You can update these by setting an environment variable on the Apicurio Registry Deployment resource. If the specific configuration option is not available in the ApicurioRegistry CR, you can use an environment variable to adjust it.

Procedure

OpenShift web console

Select the Installed Operators tab, and then the Red Hat Integration - Apicurio Registry Operator.
On the ApicurioRegistry tab, click the ApicurioRegistry CR for your Apicurio Registry deployment.
On the main overview page, view the managedResources section, which contains the name of the Deployment managed by the Operator to deploy your Apicurio Registry instance.
Find that Deployment in the Workloads > Deployments in the left menu.
Select the Deployment with the correct name, and select the Environment tab.
You can add or modify your environment variable to the Single values (env) section.
Click Save at the bottom.

OpenShift CLI

Select the project where Apicurio Registry is installed.
Run oc get apicurioregistry to get the list of ApicurioRegistry CRs
Run oc describe on the CR representing the Apicurio Registry instance that you want to configure.
View managedResources in the status section.
Find that Deployment and enter oc edit.
Add or modify the environment variable in the spec.template.spec.containers[0].env section.

Apicurio Registry liveness and readiness environment variables

Apicurio Registry provides readiness and liveness probes for OpenShift to ensure application health. These settings are configured to use reasonable defaults, but if you want to adjust them, this section provides details on the environment variables that you can configure.

OpenShift liveness and readiness errors are explained as follows:

Liveness error - the application cannot make progress, OpenShift restarts the failing pod.
Readiness error - the application is not ready, for example, it is overwhelmed by requests. OpenShift stops sending requests to the pod for the time the probe fails. If other pods are OK, they still receive requests.

Table 2. Apicurio Registry liveness environment variables
Name	Value type	Description
`LIVENESS_ERROR_THRESHOLD`	Positive integer	Number of liveness errors that can occur before the liveness probe fails.
`LIVENESS_COUNTER_RESET`	Positive integer, seconds	Period in which the threshold number of errors must occur. For example, if this value is `60` and the threshold is `1`, the check fails after two errors occur in 1 minute.
`LIVENESS_STATUS_RESET`	Positive integer, seconds	Number of seconds that must elapse without any more errors for the liveness probe to reset to OK status. Because OpenShift restarts the pod that fails the liveness check, this value, unlike readiness, does not actually affect what happens.
`LIVENESS_ERRORS_IGNORED`	Comma-separated list of fully qualified exception class names.	List of errors that are ignored for liveness checking purposes.

Table 3. Apicurio Registry readiness environment variables
Name	Value type	Description
`READINESS_ERROR_THRESHOLD`	Positive integer	Number of readiness errors that can occur before the readiness probe fails.
`READINESS_COUNTER_RESET`	Positive integer, seconds	Period in which the threshold number of errors must occur. For example, if this value is 60 and the threshold is 1, the check fails after two errors occur in 1 minute.
`READINESS_STATUS_RESET`	Positive integer, seconds	Number of seconds that must elapse without any more errors for the readiness probe to reset to OK status. In practice, this value means how long the pod remains in an unready state until it returns to normal operation.
`READINESS_TIMEOUT`	positive integer, seconds	The readiness system tracks the timeout of two operations: how long it takes for a storage request to complete, and how long it takes for an HTTP REST API request to return a response. If the operation takes more time, it is counted as a readiness error. This value controls those timeouts.

Configuring an HTTPS connection to Apicurio Registry from inside the OpenShift cluster

The following procedure shows how to configure Apicurio Registry deployment to expose a port for HTTPS connections from inside the OpenShift cluster.

This kind of connection is not directly available outside of the cluster. Routing is based on hostname, which is encoded in the case of an HTTPS connection. Therefore, edge termination or other configuration is still needed. See Configuring an HTTPS connection to Apicurio Registry from outside the OpenShift cluster.

Prerequisites

You must have already installed the Apicurio Registry Operator.

Procedure

Generate a keystore with a self-signed certificate. You can skip this step if you are using your own certificates.
```
keytool -genkey -trustcacerts -keyalg RSA -keystore registry-keystore.jks -storepass password
```

Create a new secret to hold the keystore and keystore password.

In the left navigation menu of the OpenShift web console, click Workloads > Secrets > Create Key/Value Secret.

Use the following values:
Name: registry-keystore
Key 1: keystore.jks
Value 1: registry-keystore.jks (uploaded file)
Key 2: password
Value 2: password

If you encounter a java.io.IOException: Invalid keystore format, the upload of the binary file did not work properly. As an alternative, encode the file as a base64 string using cat registry-keystore.jks | base64 -w0 > data.txt and edit the Secret resource as yaml to manually add the encoded file.

Edit the Deployment resource of the Apicurio Registry instance. You can find the correct name in a status field of the Apicurio Registry Operator.

Add the keystore secret as a volume:

template:
  spec:
    volumes:
    - name: registry-keystore-secret-volume
      secret:
      secretName: registry-keystore

Add a volume mount:

volumeMounts:
  - name: registry-keystore-secret-volume
    mountPath: /etc/registry-keystore
    readOnly: true

Add JAVA_OPTIONS and KEYSTORE_PASSWORD environment variables:

- name: KEYSTORE_PASSWORD
  valueFrom:
    secretKeyRef:
      name: registry-keystore
      key: password
- name: JAVA_OPTIONS
    value: >-
     -Dquarkus.http.ssl.certificate.key-store-file=/etc/registry-keystore/keystore.jks
     -Dquarkus.http.ssl.certificate.key-store-file-type=jks
     -Dquarkus.http.ssl.certificate.key-store-password=$(KEYSTORE_PASSWORD)

Order is important when using string interpolation.

Enable the HTTPS port:

ports:
  - containerPort: 8080
    protocol: TCP
  - containerPort: 8443
    protocol: TCP

Edit the Service resource of the Apicurio Registry instance. You can find the correct name in a status field of the Apicurio Registry Operator.

ports:
  - name: http
    protocol: TCP
    port: 8080
    targetPort: 8080
  - name: https
    protocol: TCP
    port: 8443
    targetPort: 8443

Verify that the connection is working:
1. Connect into a pod on the cluster using SSH (you can use the Apicurio Registry pod):
  oc rsh -n default example-apicurioregistry-deployment-vx28s-4-lmtqb
2. Find the cluster IP of the Apicurio Registry pod from the Service resource (see the Location column in the web console). Afterwards, execute a test request (we are using self-signed certificate, so an insecure flag is required):
  curl -k https://172.30.209.198:8443/health [...]

Configuring an HTTPS connection to Apicurio Registry from outside the OpenShift cluster

The following procedure shows how to configure Apicurio Registry deployment to expose an HTTPS edge-terminated route for connections from outside the OpenShift cluster.

Prerequisites

You must have already installed the Apicurio Registry Operator.
Read the OpenShift documentation for creating secured routes.

Procedure

Add a second Route in addition to the HTTP route created by the Apicurio Registry Operator. See the following example:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  [...]
  labels:
    app: example-apicurioregistry
    [...]
spec:
  host: example-apicurioregistry-default.apps.example.com
  to:
    kind: Service
    name: example-apicurioregistry-service-9whd7
    weight: 100
  port:
    targetPort: 8080
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None

Make sure the insecureEdgeTerminationPolicy: Redirect configuration property is set.

If you do not specify a certificate, OpenShift will use a default. You can alternatively generate a custom self-signed certificate using the following commands:

openssl genrsa 2048 > host.key &&
openssl req -new -x509 -nodes -sha256 -days 365 -key host.key -out host.cert

and then create a route using the OpenShift CLI:

oc create route edge \
  --service=example-apicurioregistry-service-9whd7 \
  --cert=host.cert --key=host.key \
  --hostname=example-apicurioregistry-default.apps.example.com \
  --insecure-policy=Redirect \
  -n default

Backing up Apicurio Registry PostgreSQL storage

When using storage in a PostgreSQL database, you must ensure that the data stored by Apicurio Registry is backed up regularly.

SQL Dump is a simple procedure that works with any PostgreSQL installation. This uses the pg_dump utility to generate a file with SQL commands that you can use to recreate the database in the same state that it was in at the time of the dump.

pg_dump is a regular PostgreSQL client application, which you can execute from any remote host that has access to the database. Like any other client, the operations that can perform are limited to the user permissions.

Procedure

Use the pg_dump command to redirect the output to a file:
```
 $ pg_dump dbname > dumpfile
```
You can specify the database server that pg_dump connects to using the -h host and -p port options.
You can reduce large dump files using a compression tool, such as gzip, for example:
```
 $ pg_dump dbname | gzip > filename.gz
```

Additional resources

For details on client authentication, see the PostgreSQL documentation.
For details on importing and exporting registry content, see Managing Apicurio Registry content using the REST API.

Restoring Apicurio Registry PostgreSQL storage

You can restore SQL Dump files created by pg_dump using the psql utility.

Prerequisites

You must have already backed up your PostgreSQL datbase using pg_dump. See Backing up Apicurio Registry PostgreSQL storage.
All users who own objects or have permissions on objects in the dumped database must already exist.

Procedure

Enter the following command to create the database:
```
 $ createdb -T template0 dbname
```
Enter the following command to restore the SQL dump
```
 $ psql dbname < dumpfile
```
Run ANALYZE on each database so the query optimizer has useful statistics.