Beyond OpenShift: Making Apicurio Registry Work with Any OAuth Provider via Dex

Thu, 26 Mar 2026 12:00:00 +0000

In a previous post, we showed how to connect Apicurio Registry directly to OpenShift’s built-in OAuth server. It worked for API-level authentication and role-based authorization, but three important features were broken: UI login, principal identity (artifact ownership), and owner-based authorization. And it only worked on OpenShift.

We wanted to do better. We wanted a solution that works with any OAuth provider — OpenShift, LDAP, GitHub, Azure AD, SAML, you name it — while unlocking the full set of Apicurio Registry features. This post covers how we got there using Dex.

Where We Left Off

The direct OpenShift integration was a clever hack: we pointed Quarkus OIDC’s userinfo endpoint at the Kubernetes User API, which validated opaque tokens and returned group membership. But OpenShift’s OAuth server is not OIDC-compliant, and that caused three hard problems:

Feature	Direct Integration	Why It Breaks
UI login	No	`oidc-client-ts` requires `/.well-known/openid-configuration`, which OpenShift doesn’t serve
Principal identity	No	Username is nested in `metadata.name`; Quarkus only reads top-level fields from UserInfo
Owner-based authz	No	Depends on principal identity

These aren’t minor gaps. Without UI login, your users can’t access the Registry through a browser. Without principal identity, every artifact shows owner: "" — you can’t track who created what. And without owner-based authorization, you can’t enforce policies like “only the creator can modify this artifact.”

We needed a different approach.

Enter Dex: A Federated OIDC Bridge

Dex is an identity service that speaks standard OIDC on the front end and connects to upstream identity providers via pluggable connectors. It acts as a protocol translator — your application talks to Dex using standard OIDC, and Dex handles the quirks of whatever identity provider you actually have.

                     OIDC (standard)              Upstream protocol
User --> Apicurio ----------------------> Dex --------------------------> OpenShift / LDAP / GitHub / etc.
              |                            |
              |  /.well-known/...    OK    |  Connector handles
              |  JWT tokens          OK    |  protocol translation
              |  JWKS                OK    |
              |  Flat claims         OK    |

This is the key insight: instead of trying to make Apicurio Registry speak every non-standard OAuth dialect, we put a standards-compliant OIDC layer in front of whatever you already have. Dex handles the translation, and Apicurio just talks OIDC like it was designed to.

What This Unlocks

With Dex in the middle, here’s what changes:

Feature	Direct OpenShift OAuth	Via Dex
Token validation	Yes	Yes
Anonymous reads	Yes	Yes
Role-based authorization	Yes	Yes
UI login flow	No	Yes
Principal identity	No	Yes
Owner-based authorization	No	Yes
Multi-provider federation	No	Yes
Standard OIDC compliance	No	Yes
Client credentials grant (M2M)	N/A	Partial — works in `master`, no groups in tokens (see below)

Every user-facing feature that was broken now works. And as a bonus, you get multi-provider federation — Dex can authenticate against multiple upstream providers simultaneously.

How to Set It Up

Deploy Dex

The fastest way is via Helm:

helm repo add dex https://charts.dexidp.io
helm repo update
kubectl create namespace dex

Create a dex-values.yaml with your configuration:

config:
  issuer: https://dex.<cluster-domain>

  storage:
    type: kubernetes
    config:
      inCluster: true

  web:
    http: 0.0.0.0:5556
    # CORS: Required so the Apicurio UI (oidc-client-ts) can fetch
    # /.well-known/openid-configuration from a different origin.
    allowedOrigins:
      - "https://<registry-ui-route>"

  oauth2:
    skipApprovalScreen: true
    responseTypes: [code, token, id_token]

  # Two clients are needed:
  # - A PUBLIC client for the browser UI (oidc-client-ts cannot hold secrets)
  # - A CONFIDENTIAL client for CLI/API token exchange
  staticClients:
    - id: apicurio-registry-ui
      name: Apicurio Registry UI
      public: true
      redirectURIs:
        - "https://<registry-ui-route>/"
        - "https://<registry-ui-route>/dashboard"
    - id: apicurio-registry
      name: Apicurio Registry API
      secret: <GENERATE_A_SECURE_SECRET>
      redirectURIs:
        - "https://<registry-ui-route>/"
        - "https://<registry-app-route>/"

  connectors: []  # We'll configure these next

ingress:
  enabled: true
  hosts:
    - host: dex.<cluster-domain>
      paths:
        - path: /
          pathType: Prefix

helm install dex dex/dex -n dex -f dex-values.yaml

On OpenShift, create a Route instead of using Ingress:

oc create route edge dex --service=dex --port=5556 --hostname=dex.<cluster-domain> --namespace=dex

Verify OIDC discovery works:

curl -sk https://dex.<cluster-domain>/.well-known/openid-configuration | jq .issuer
# Expected: "https://dex.<cluster-domain>"

Configure a Connector

This is where Dex becomes powerful — you pick the connector that matches your identity provider. Here are the most common options:

OpenShift

First, create an OAuthClient for Dex (not for Apicurio — Dex is the OAuth client now):

CLIENT_SECRET=$(openssl rand -base64 32 | tr -d '=' | head -c 32)

cat <<EOF | kubectl apply -f -
apiVersion: oauth.openshift.io/v1
kind: OAuthClient
metadata:
  name: dex
grantMethod: auto
secret: "$CLIENT_SECRET"
redirectURIs:
  - "https://dex.<cluster-domain>/callback"
EOF

Then add the connector:

connectors:
  - type: openshift
    id: openshift
    name: OpenShift
    config:
      issuer: https://api.<cluster-domain>:6443
      clientID: dex
      clientSecret: "<CLIENT_SECRET>"
      redirectURI: https://dex.<cluster-domain>/callback
      groups:
        - registry-admins
        - registry-developers
        - registry-readers
      insecureCA: true

LDAP

connectors:
  - type: ldap
    id: ldap
    name: LDAP
    config:
      host: ldap.example.com:636
      bindDN: cn=admin,dc=example,dc=com
      bindPW: <bind-password>
      userSearch:
        baseDN: ou=users,dc=example,dc=com
        filter: "(objectClass=person)"
        username: uid
        idAttr: uid
        emailAttr: mail
        nameAttr: cn
      groupSearch:
        baseDN: ou=groups,dc=example,dc=com
        filter: "(objectClass=groupOfNames)"
        userMatchers:
          - userAttr: DN
            groupAttr: member
        nameAttr: cn

GitHub

connectors:
  - type: github
    id: github
    name: GitHub
    config:
      clientID: <github-oauth-app-client-id>
      clientSecret: <github-oauth-app-client-secret>
      redirectURI: https://dex.<cluster-domain>/callback
      orgs:
        - name: Apicurio
          teams:
            - registry-admins
            - registry-developers

Microsoft / Azure AD

connectors:
  - type: microsoft
    id: microsoft
    name: Microsoft
    config:
      clientID: <azure-app-client-id>
      clientSecret: <azure-app-client-secret>
      redirectURI: https://dex.<cluster-domain>/callback
      tenant: <azure-tenant-id>
      groups:
        - registry-admins-group-uuid
        - registry-developers-group-uuid

SAML 2.0

connectors:
  - type: saml
    id: saml
    name: Corporate SSO
    config:
      ssoURL: https://idp.example.com/sso
      ca: /etc/dex/saml-ca.crt
      redirectURI: https://dex.<cluster-domain>/callback
      usernameAttr: name
      emailAttr: email
      groupsAttr: groups

After adding your connector, upgrade the release:

helm upgrade dex dex/dex -n dex -f dex-values.yaml

Deploy Apicurio Registry

Now the Registry CR becomes much simpler — no more OIDC workarounds, just a standard OIDC configuration pointing at Dex:

apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry3
metadata:
  name: apicurio-registry
  namespace: apicurio-registry
spec:
  app:
    ingress:
      host: apicurio-registry-app-apicurio-registry.apps.<cluster-domain>
      # On OpenShift, this annotation enables edge TLS termination on the Route,
      # so the app is served over HTTPS with automatic HTTP -> HTTPS redirect.
      annotations:
        route.openshift.io/termination: edge
    auth:
      enabled: true
      # Both must point to the PUBLIC Dex client so the backend accepts
      # tokens with aud=apicurio-registry-ui issued by the browser flow.
      appClientId: apicurio-registry-ui
      uiClientId: apicurio-registry-ui
      authServerUrl: "https://dex.<cluster-domain>"
      # Explicit redirect URI prevents oidc-client-ts from using the current
      # page URL (which may contain stale ?code=...&state=... query params),
      # fixing silent token refresh failures.
      redirectUri: "https://<registry-ui-route>/"
      anonymousReadsEnabled: true
      tls:
        tlsVerificationType: "none"
      authz:
        enabled: true
        readAccessEnabled: true
        ownerOnlyEnabled: true       # This WORKS now!
        groupAccessEnabled: false
        roles:
          source: token
          admin: "registry-admins"
          developer: "registry-developers"
          readOnly: "registry-readers"
        adminOverride:
          enabled: true
          from: token
          type: role
          role: "registry-admins"
    env:
      - name: QUARKUS_OIDC_AUTHENTICATION_SCOPES
        value: "openid,email,profile,groups"
      - name: QUARKUS_OIDC_ROLES_ROLE_CLAIM_PATH
        value: "groups"
      - name: QUARKUS_OIDC_TOKEN_PRINCIPAL_CLAIM
        value: "email"
      # The UI scope defaults to "openid profile email" which does NOT include
      # groups. Without this, Dex won't include group membership in the token
      # and RBAC will deny all write operations.
      - name: APICURIO_UI_AUTH_OIDC_SCOPE
        value: "openid profile email groups"
  ui:
    ingress:
      host: apicurio-registry-ui-apicurio-registry.apps.<cluster-domain>
      annotations:
        route.openshift.io/termination: edge
    env:
      # The operator hardcodes http:// for REGISTRY_API_URL.
      # Override it to use HTTPS since we enabled edge TLS on the app route.
      - name: REGISTRY_API_URL
        value: "https://apicurio-registry-app-apicurio-registry.apps.<cluster-domain>/apis/registry/v3"

There are a few important details in this CR worth calling out:

Two-client architecture — Both appClientId and uiClientId are set to apicurio-registry-ui (the public Dex client). The browser-based UI uses oidc-client-ts, which cannot securely hold a client secret. If you use a confidential client for the UI, the token exchange will fail with Invalid client credentials. The backend validates JWT signatures via the JWKS endpoint and doesn’t need a client secret. The separate confidential client (apicurio-registry) is kept for CLI/API token exchange via curl or scripts.

UI scope with groups — The APICURIO_UI_AUTH_OIDC_SCOPE env var adds groups to the scope the UI requests from Dex. Without this, the default scope (openid profile email) doesn’t include groups, and Dex won’t put group membership in the token. RBAC then denies all write operations because it can’t determine the user’s role. We verified this on a live cluster with Dex v2.45.1 and the OpenShift connector: tokens include the groups claim with the user’s OpenShift group memberships, and Apicurio successfully resolves roles (e.g. registry-admins) for RBAC authorization.

Explicit redirectUri — Without this, oidc-client-ts uses window.location.href as the redirect URI for silent token refresh. After the initial login, the URL may still contain ?code=...&state=... query parameters, which don’t match any registered redirect URI in Dex, causing a 400 error.

Notice what’s not here compared to the direct OpenShift integration: no QUARKUS_OIDC_DISCOVERY_ENABLED=false, no QUARKUS_OIDC_JWKS_PATH pointing at the K8s API server, no QUARKUS_OIDC_TOKEN_VERIFY_ACCESS_TOKEN_WITH_USER_INFO. All of those workarounds are gone because Dex is a proper OIDC provider that supports standard discovery.

And notice what’s new: ownerOnlyEnabled: true — owner-based authorization is enabled because principal identity works. The QUARKUS_OIDC_TOKEN_PRINCIPAL_CLAIM is set to email, which Dex populates as a flat top-level claim in the JWT. No more nested JSON issues.

Verify Everything Works

# OIDC discovery
curl -sk https://dex.<cluster-domain>/.well-known/openid-configuration | jq .issuer

# Anonymous read
curl -sk https://<registry-app-route>/apis/registry/v3/system/info

# UI login — open in browser, click Login, authenticate via your upstream provider
# https://<registry-ui-route>

# Check that artifacts have an owner (principal identity works!)
curl -sk https://<registry-app-route>/apis/registry/v3/groups/my-group \
  -H "Authorization: Bearer $TOKEN" | jq .owner
# Expected: the user's email — NOT empty

Machine-to-Machine (M2M) Access and Kafka SerDes

So far we’ve focused on the browser-based UI flow, but what about CI/CD pipelines, Kafka SerDes, and services that need to interact with the Registry API without a browser?

This is a critical question because the Apicurio SerDes libraries only support client_credentials grant — they hardcode grant_type=client_credentials with no alternative. Every Kafka producer or consumer using Apicurio SerDes to fetch or register schemas needs this grant type to work.

Dex `client_credentials` Support (Unreleased as of v2.45.1)

Dex added client_credentials support via PR #4583 (merged March 3, 2026), but this feature is not included in Dex v2.45.1 — the latest stable release at time of writing. It will ship in v2.46.0 or later.

We tested both versions on a live OpenShift cluster:

Dex v2.45.1: client_credentials returns {"error":"unsupported_grant_type"} regardless of the DEX_CLIENT_CREDENTIAL_GRANT_ENABLED_BY_DEFAULT env var. The feature flag code doesn’t exist in this version.
Dex master snapshot (docker.io/dexidp/dex:master): client_credentials works. Tokens are issued, and the grant type appears in the OIDC discovery endpoint. However, tokens do not include groups claims — writes to the Registry API return 403 because RBAC can’t determine the user’s role.

To test before a stable release, use the dexidp/dex:master image from Docker Hub (rebuilt daily by Dex CI). For production, wait for v2.46.0+.

Enable it by setting an environment variable on the Dex deployment:

# In the Dex Deployment spec (requires Dex master or >= v2.46.0)
env:
  - name: DEX_CLIENT_CREDENTIAL_GRANT_ENABLED_BY_DEFAULT
    value: "true"

Then use the confidential client:

TOKEN_RESPONSE=$(curl -s -X POST "https://dex.<cluster-domain>/token" \
  --data-urlencode "grant_type=client_credentials" \
  --data-urlencode "client_id=apicurio-registry" \
  --data-urlencode "client_secret=<CLIENT_SECRET>" \
  --data-urlencode "scope=openid profile")

ACCESS_TOKEN=$(echo $TOKEN_RESPONSE | jq -r .access_token)

The Catch: No Groups in Client Credentials Tokens

Even with client_credentials working, the tokens do not include groups claims — even when the groups scope is explicitly requested. The PR description says it “supports groups scope”, but the scope is merely accepted (not rejected) — no groups are populated because there’s no upstream connector or user in the client_credentials flow. The token is built solely from the static client definition, which has no group membership.

Since Apicurio Registry uses groups for RBAC (determining admin/developer/reader roles), M2M clients authenticating via client_credentials will have no assigned role (principal is null, roles are empty).

What this means in practice:

Scenario	Works?	Why
SerDes reading schemas (with `anonymousReadsEnabled: true`)	Yes	Anonymous reads bypass auth entirely
SerDes registering schemas	No	No role in token = write denied
CI/CD creating artifacts	No	No role in token = write denied

What Actually Works for Kafka SerDes

For the most common Kafka use case — consumers fetching schemas — you don’t need client_credentials at all. Set anonymousReadsEnabled: true in the Apicurio CR and the SerDes can read schemas without any token. This is the recommended approach for read-heavy workloads.

For schema registration (typically done by developers or CI/CD, not by Kafka producers at runtime), use the authorization code flow with the confidential client and a service user that has the right groups:

# Non-interactive token acquisition using OpenShift challenge-based auth
# (See the full guide for the complete 6-step script)
TOKEN_RESPONSE=$(curl -s -X POST "https://dex.<cluster-domain>/token" \
  --data-urlencode "grant_type=authorization_code" \
  --data-urlencode "code=${DEX_CODE}" \
  --data-urlencode "client_id=apicurio-registry" \
  --data-urlencode "client_secret=<CLIENT_SECRET>" \
  --data-urlencode "redirect_uri=https://<registry-ui-route>/")

# This token WILL have groups because it went through the full user auth flow
ACCESS_TOKEN=$(echo $TOKEN_RESPONSE | jq -r .access_token)

Upcoming Fix: Groups on Static Clients

We opened dexidp/dex#4690 to report this gap and submitted dexidp/dex#4691 with a fix. It adds a clientCredentialsClaims sub-struct to static client definitions, keeping identity attributes separate from core client fields:

staticClients:
  - id: apicurio-registry
    secret: "..."
    clientCredentialsClaims:
      groups: ["registry-admins"]   # included in client_credentials tokens

We deployed a patched build and verified it on a live cluster: tokens now include "groups": ["registry-admins"] and Registry writes succeed (200 instead of 403). Track the PR for when this lands in a stable release.

When to Use Keycloak Instead

If your deployment requires M2M write access with proper RBAC — for example, if Kafka producers register schemas at runtime, or if you have many CI/CD pipelines that need different permission levels — use Keycloak (or Auth0, Azure AD) instead of Dex. These providers support service account roles natively, so client_credentials tokens include all the claims Apicurio needs.

Dex is the right choice when:

Your upstream IdP is non-OIDC-compliant (OpenShift OAuth, LDAP, SAML)
You primarily need user-facing auth (UI login, principal identity, owner-based authz)
M2M workloads are read-only (SerDes consumers with anonymous reads)

Group-to-Role Mapping

The mapping between upstream provider groups and Apicurio Registry roles flows through two layers:

Upstream Group	Dex Passes As	CR Role Config	Registry Role
`registry-admins`	`registry-admins`	`admin: registry-admins`	Admin
`registry-developers`	`registry-developers`	`developer: registry-developers`	Developer
`registry-readers`	`registry-readers`	`readOnly: registry-readers`	Read-Only

For providers that use UUIDs as group identifiers (like Azure AD), the CR supports comma-separated values:

roles:
  admin: "registry-admins,a1b2c3d4-uuid-of-azure-group"

Production Considerations

A few things to keep in mind when running this in production:

TLS — Configure Dex with proper TLS via Ingress termination or its own certificate. Set tlsVerificationType: "VERIFY_PEER" in the Apicurio CR and provide a truststore if using a private CA.

High availability — Run multiple Dex replicas. For better HA, switch Dex’s storage from Kubernetes CRDs to PostgreSQL.

Token lifetimes — Configure sensible expiry in Dex:

config:
  expiry:
    idTokens: "1h"
    refreshTokens:
      validIfNotUsedFor: "168h"   # 7 days
      absoluteLifetime: "720h"     # 30 days

Network policies — Restrict access so only Apicurio Registry pods and the ingress controller can reach Dex.

Troubleshooting

“Invalid client credentials” on UI login — The UI (oidc-client-ts) runs in the browser and cannot send a client secret. Use a public Dex client (public: true, no secret) for the UI. A confidential client will always fail.
403 Forbidden on API calls after UI login — Two common causes: (1) Audience mismatch — the UI gets tokens with aud: "apicurio-registry-ui" but the backend expects aud: "apicurio-registry". Fix: set both appClientId and uiClientId to the same public client ID. (2) Missing groups scope — the default UI scope doesn’t include groups. Fix: set APICURIO_UI_AUTH_OIDC_SCOPE to openid profile email groups.
Groups not appearing in the token — Verify both QUARKUS_OIDC_AUTHENTICATION_SCOPES (backend) and APICURIO_UI_AUTH_OIDC_SCOPE (UI) include groups. Decode the JWT to inspect: echo $TOKEN | cut -d. -f2 | base64 -d | jq .
“Unregistered redirect_uri” on page refresh — The UI redirects to /dashboard after login. Add https://<registry-ui-route>/dashboard to the Dex client’s redirect URIs.
Silent token refresh returns 400 — oidc-client-ts uses the current page URL (with stale ?code=...&state=... params) as the redirect URI. Fix: set redirectUri explicitly in the Apicurio CR auth config.
“No end session endpoint” on logout — Dex does not implement OIDC RP-Initiated Logout. The local session is still cleared, but signoutRedirect fails. This is a known Dex limitation — the error is non-blocking.
CORS errors in browser console — The UI can’t fetch /.well-known/openid-configuration from Dex. Add the UI route origin to web.allowedOrigins in the Dex config.
InvalidJwtSignatureException after Dex restart — Dex generates new signing keys on restart. Old tokens become invalid. Users must re-authenticate. For production, configure persistent signing keys.
invalid_client when exchanging authorization code — If the client secret contains special characters (+, /, =), use --data-urlencode instead of -d in curl.
Owner field is still empty — Check that QUARKUS_OIDC_TOKEN_PRINCIPAL_CLAIM is set to a claim Dex actually populates (email, name, or sub).
UI redirects to Dex but gets an error — Check Dex logs (kubectl logs -n dex deploy/dex). Usually a redirect URI mismatch or connector misconfiguration.
client_credentials returns unsupported_grant_type — This grant type was added in Dex PR #4583 (merged March 3, 2026) but is not included in v2.45.1 or earlier. You need Dex >= v2.46.0 or a build from master.
“OIDC Server is not available” — The Registry pod can’t reach Dex. Check DNS resolution, network policies, and TLS trust.

The Bigger Picture

This approach isn’t just about fixing the three broken features from the direct OpenShift integration. It’s about decoupling Apicurio Registry from any specific identity provider.

With Dex as an OIDC bridge, your Registry deployment is portable:

Moving from OpenShift to vanilla Kubernetes? Swap the connector, keep everything else.
Migrating from LDAP to Azure AD? Same.
Need to support multiple identity providers during a transition? Dex handles federation natively.

The Registry doesn’t need to know or care what’s behind Dex. It just talks OIDC.

If you’re interested in the direct OpenShift approach (without Dex), check out the previous post — it’s still a valid option if you only need API-level auth and role-based authorization. But if you want the full feature set, Dex is the way to go.

Happy registering!

Integrating Apicurio Registry with OpenShift's Built-in OAuth Server

Tue, 17 Mar 2026 12:00:00 +0000

If you’re running Apicurio Registry on OpenShift, you might be wondering: can I use OpenShift’s built-in OAuth server for authentication instead of deploying a separate identity provider like Keycloak? The answer is yes — with some caveats. In this post, I’ll walk you through how we got it working, what the limitations are, and the technical details behind the integration.

The Challenge

OpenShift ships with a built-in OAuth server that handles user authentication for the platform. However, it is not a standard OpenID Connect (OIDC) provider:

It does not expose /.well-known/openid-configuration
It issues opaque tokens (sha256~...), not JWTs
It has no JWKS endpoint and no RFC 7662 token introspection endpoint

Apicurio Registry uses Quarkus OIDC for authentication, which normally expects a fully OIDC-compliant server. So how do we bridge the gap?

The Solution: verify-access-token-with-user-info

The trick is to combine three components:

JWKS from the Kubernetes API server — The K8s API server exposes a JWKS endpoint at /openid/v1/jwks. Quarkus OIDC requires either a JWKS or introspection path to initialize. We point it to the K8s JWKS purely to satisfy this startup requirement.
OpenShift’s User API (/apis/user.openshift.io/v1/users/~) — When called with a valid Bearer token, this endpoint returns the authenticated user’s identity including their OpenShift groups. When called with an invalid token, it returns 401.
Quarkus verify-access-token-with-user-info=true — This tells Quarkus to validate opaque access tokens by sending a GET request with the Bearer token to the userinfo endpoint. If the endpoint returns 200, the token is valid; otherwise, it’s rejected.

Here’s the flow:

                          +----------------------+
   Client Request         |  Apicurio Registry   |
   with Bearer token ---> |  (Quarkus OIDC)      |
                          |                      |
                          |  1. Sends GET with    |
                          |     Bearer token to   |
                          |     userinfo endpoint  |
                          +----------+-----------+
                                     |
                                     v
                          +----------------------+
                          |  K8s API Server       |
                          |  /apis/user.openshift |
                          |  .io/v1/users/~       |
                          |                      |
                          |  Valid token -> 200   |
                          |  + user info (with    |
                          |    groups array)      |
                          |                      |
                          |  Invalid token -> 401 |
                          +----------------------+

The User API response includes an OpenShift groups array at the top level. By creating OpenShift groups that match Apicurio role names, we map group membership directly to Registry roles.

Step-by-Step Setup

1. Install the Apicurio Registry Operator

First, deploy the operator to your cluster:

kubectl create namespace apicurio-registry-operator
kubectl apply -f apicurio-registry-operator.yaml -n apicurio-registry-operator
kubectl wait --for=condition=available deployment/apicurio-registry-operator \
  -n apicurio-registry-operator --timeout=300s

2. Grant Anonymous Access to the JWKS Endpoint

The K8s API server JWKS endpoint is restricted by default. Since Quarkus OIDC fetches JWKS without authentication, we need to open it up. The JWKS contains only public keys, so there’s no security risk:

kubectl apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: apicurio-jwks-anonymous-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:service-account-issuer-discovery
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:unauthenticated
EOF

3. Create an OAuthClient Resource

CLIENT_SECRET=$(openssl rand -base64 32 | tr -d '=' | head -c 32)
echo "Client secret: $CLIENT_SECRET"

apiVersion: oauth.openshift.io/v1
kind: OAuthClient
metadata:
  name: apicurio-registry
grantMethod: auto
secret: "<CLIENT_SECRET>"
redirectURIs:
  - "https://<registry-app-route>"
  - "https://<registry-ui-route>"
  - "http://<registry-app-route>"
  - "http://<registry-ui-route>"

Tip: Include both http:// and https:// variants in redirectURIs to avoid 400 errors during OAuth flows.

4. Create OpenShift Groups for Role-Based Authorization

Create groups that map to Apicurio Registry roles:

oc adm groups new registry-admins
oc adm groups new registry-developers
oc adm groups new registry-readers

oc adm groups add-users registry-admins <admin-username>
oc adm groups add-users registry-developers <developer-username>
oc adm groups add-users registry-readers <reader-username>

5. Deploy the Registry

This is where it all comes together. The CR configures Quarkus OIDC to work with OpenShift’s non-OIDC-compliant OAuth server:

apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry3
metadata:
  name: apicurio-registry
  namespace: apicurio-registry
spec:
  app:
    host: apicurio-registry-app-apicurio-registry.apps.<cluster-domain>
    auth:
      enabled: true
      appClientId: apicurio-registry
      uiClientId: apicurio-registry
      authServerUrl: "https://oauth-openshift.apps.<cluster-domain>"
      anonymousReadsEnabled: true
      tls:
        tlsVerificationType: "none"
      authz:
        enabled: true
        readAccessEnabled: true
        ownerOnlyEnabled: false
        groupAccessEnabled: false
        roles:
          source: token
          admin: "registry-admins"
          developer: "registry-developers"
          readOnly: "registry-readers"
        adminOverride:
          enabled: true
          from: token
          type: role
          role: "registry-admins"
    env:
      # Disable OIDC auto-discovery
      - name: QUARKUS_OIDC_DISCOVERY_ENABLED
        value: "false"

      # OAuth endpoints on the OpenShift OAuth server
      - name: QUARKUS_OIDC_AUTHORIZATION_PATH
        value: "/oauth/authorize"
      - name: QUARKUS_OIDC_TOKEN_PATH
        value: "/oauth/token"

      # K8s API server JWKS (only for Quarkus startup initialization)
      - name: QUARKUS_OIDC_JWKS_PATH
        value: "https://api.<cluster-domain>:6443/openid/v1/jwks"

      # OpenShift User API as the userinfo endpoint
      - name: QUARKUS_OIDC_USER_INFO_PATH
        value: "https://api.<cluster-domain>:6443/apis/user.openshift.io/v1/users/~"

      # KEY SETTING: validate opaque tokens via the userinfo endpoint
      - name: QUARKUS_OIDC_TOKEN_VERIFY_ACCESS_TOKEN_WITH_USER_INFO
        value: "true"

      # Extract OpenShift groups as roles
      - name: QUARKUS_OIDC_ROLES_ROLE_CLAIM_PATH
        value: "groups"

      # Client credentials
      - name: QUARKUS_OIDC_CREDENTIALS_SECRET
        value: "<CLIENT_SECRET>"
  ui:
    host: apicurio-registry-ui-apicurio-registry.apps.<cluster-domain>

Important: The tls section under auth is required when auth is enabled. Omitting it causes a NullPointerException in the operator.

6. Verify It Works

Once the pods are running, you can test the full authorization matrix:

# Anonymous read (should work)
curl -sk http://<registry-app-route>/apis/registry/v3/system/info

# Unauthenticated write (should return 401)
curl -sk -X POST http://<registry-app-route>/apis/registry/v3/groups \
  -H "Content-Type: application/json" -d '{"groupId": "test"}'

# Admin write (should return 200)
oc login -u testadmin -p testadmin123
TOKEN=$(oc whoami --show-token)
curl -sk -X POST http://<registry-app-route>/apis/registry/v3/groups \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -d '{"groupId": "admin-test"}'

# Reader write (should return 403)
oc login -u testreader -p testreader123
TOKEN=$(oc whoami --show-token)
curl -sk -X POST http://<registry-app-route>/apis/registry/v3/groups \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -d '{"groupId": "reader-test"}'

What Works and What Doesn’t

Authentication and role-based authorization work. Owner-based authorization and UI login do not.

Feature	Status	Notes
Token validation	Works	Valid OpenShift OAuth tokens accepted, invalid tokens rejected
Anonymous reads	Works	When `anonymousReadsEnabled: true`
Auth enforcement	Works	Unauthenticated writes are blocked with 401
Role-based authorization	Works	OpenShift groups mapped to Apicurio roles
Admin override	Works	Users in `registry-admins` get full admin access
Principal identity	Broken	Username is in `metadata.name` (nested), but Quarkus only reads top-level fields
Owner-based authorization	Broken	Depends on principal identity
UI login flow	Broken	UI requires OIDC discovery, which OpenShift OAuth doesn’t support

Why Principal Identity Doesn’t Work

This is worth explaining in detail because it reveals an interesting limitation in Quarkus OIDC.

The K8s User API returns the username nested inside metadata.name:

{
  "kind": "User",
  "metadata": { "name": "testadmin" },
  "groups": ["registry-admins", "system:authenticated"]
}

For opaque tokens validated via UserInfo, Quarkus extracts the principal with a flat JSON lookup:

userName = userInfo.getString(resolvedContext.oidcConfig().token().principalClaim().get());

This means getString("metadata/name") returns null — there’s no top-level key with that name. Interestingly, the findClaimValue() method used for role extraction supports /-separated path traversal, but this method is not used for principal extraction from UserInfo.

We tested every alternative:

metadata/name — empty (path not traversed)
metadata.name — empty (not a valid key)
metadata — 500 error (JsonObject can’t cast to JsonString)
kind — returns “User” (proves top-level strings work, but useless)

The only top-level string fields in the response are kind and apiVersion, neither of which contains the username.

Troubleshooting Tips

If you run into issues, here are the most common problems:

“Either ‘jwks-path’ or ‘introspection-path’ must be set” — Add the QUARKUS_OIDC_JWKS_PATH env var pointing to the K8s JWKS endpoint.
“Introspection path and verifyAccessTokenWithUserInfo are mutually exclusive” — Use jwks-path instead of introspection-path. It’s only needed for initialization.
NullPointerException on TLS — Always include the tls section under auth when auth is enabled.
JWKS 403 — Create the ClusterRoleBinding for anonymous JWKS access.
OAuth 400 Bad Request — Ensure redirect URIs match exactly, including protocol scheme.
Users don’t get the expected role — Verify group membership with oc get groups and re-authenticate to get a new token.

Alternative: /oauth/info Endpoint

If you only need pure authentication without role-based authorization, there’s a simpler option. OpenShift’s OAuth server exposes /oauth/info, which validates Bearer tokens and returns basic token metadata — but no user identity or groups. Set QUARKUS_OIDC_USER_INFO_PATH to /oauth/info and skip the K8s User API entirely.

Wrapping Up

While this integration has limitations (no principal identity, no UI login, no owner-based authz), it provides a solid foundation for API-level authentication and role-based authorization using only OpenShift’s built-in infrastructure — no external identity provider required. For teams that primarily interact with Apicurio Registry through REST APIs and want to leverage their existing OpenShift user and group management, this is a practical solution.

We’d love to hear from you if you try this out or have ideas for improving the integration. The Quarkus OIDC limitation around nested principal claims is something we’re looking into upstream as well.

Happy registering!

Apicurio Registry goes AI-Native: Introducing Agent Registry, MCP Server, and LLM artifact support

Thu, 05 Feb 2026 12:00:00 +0000

The world of software is undergoing a fundamental shift. AI agents are no longer experimental curiosities—they’re becoming the backbone of modern enterprise workflows. From customer service automation to complex data analysis pipelines, organizations are deploying fleets of specialized AI agents that need to discover each other, communicate reliably, and evolve without breaking.

Today, we’re excited to announce that Apicurio Registry is evolving into an AI-native platform, becoming the first open-source registry to offer comprehensive support for the AI agent ecosystem.

Why Apicurio Registry for AI?

For years, Apicurio Registry has been the trusted solution for managing API schemas—OpenAPI, AsyncAPI, Avro, Protobuf, JSON Schema, and more. Organizations rely on it for schema validation, versioning, compatibility enforcement, and governance.

It turns out these same principles apply directly to AI agents:

Traditional Schema Registry	AI Agent Registry
API contracts	Agent capabilities
Schema validation	Prompt/response validation
Version compatibility	Agent evolution
Service discovery	Agent discovery
Governance & audit	Compliance & traceability

Rather than building something new, we extended what we already do best. Apicurio Registry now treats AI artifacts—Agent Cards, prompts, and model schemas—as first-class citizens alongside traditional schemas.

What’s new: Phase 1 complete

We’re launching with three major capabilities that are fully implemented and production-ready:

1. Agent card Registry (A2A protocol support)

The A2A (Agent-to-Agent) protocol defines a standard way for AI agents to describe themselves and discover each other. Apicurio Registry now supports AGENT_CARD as a native artifact type.

What you can do:

Store and version Agent Cards with full JSON Schema validation
Discover agents via /.well-known/agent.json endpoints
Search agents by capability, skill, or mode
Enforce compatibility rules as agents evolve
Track changes for compliance and auditing

# Register an agent card
curl -X POST http://registry:8080/apis/registry/v3/groups/my-agents/artifacts \
  -H "Content-Type: application/json" \
  -H "X-Registry-ArtifactType: AGENT_CARD" \
  -d @my-agent-card.json

# Discover agents via A2A protocol
curl http://registry:8080/.well-known/agent.json

2. MCP server integration

The Model Context Protocol (MCP) is becoming the standard for LLMs to interact with external tools and data. Apicurio Registry now includes a built-in MCP Server that lets AI models directly access the registry.

What your AI agents can do:

Discover schemas by name, type, or capability
Retrieve schema content for validation
Create and update schemas programmatically
Validate data against registered schemas
Manage groups and versions

Claude Desktop integration is already documented—simply add Apicurio Registry as an MCP tool and your AI assistant can query schemas, validate payloads, and register new artifacts directly through natural language.

{
  "mcpServers": {
    "apicurio-registry": {
      "command": "java",
      "args": ["-jar", "apicurio-registry-mcp.jar"],
      "env": {
        "REGISTRY_URL": "http://localhost:8080"
      }
    }
  }
}

3. LLM artifact types

Managing prompts and model schemas is one of the biggest operational challenges in production AI systems. Apicurio Registry introduces two new artifact types designed specifically for LLM workflows:

PROMPT_TEMPLATE

Store version-controlled prompt templates with `` placeholders
Define input variables with types, defaults, and validation rules
Render prompts with values via the registry API
A/B test prompt variations without code changes
Rollback instantly when a prompt underperforms

MODEL_SCHEMA

Define input/output schemas for LLM agents
Validate agent responses against expected formats
Catch schema mismatches before runtime
Track schema evolution with compatibility checking

{
  "$schema": "https://registry.apicur.io/schemas/prompt-template.json",
  "name": "customer-response-generator",
  "template": "You are a helpful support agent. The customer said: . Their sentiment is . Generate an empathetic response.",
  "variables": {
    "message": { "type": "string", "required": true },
    "sentiment": { "type": "string", "enum": ["positive", "neutral", "negative"] }
  },
  "metadata": {
    "model": "gpt-4",
    "temperature": 0.7
  }
}

SDKs are ready for both Python (LangChain, LlamaIndex) and Java (LangChain4j, Quarkus), making integration straightforward.

See it in Action: Multi-Agent context chaining

To demonstrate these capabilities working together, we’ve created a comprehensive example: A2A Real-World Integration Demo.

This demo shows a real-world scenario: processing customer complaints through a pipeline of specialized AI agents:

Customer Complaint
       │
       ▼
┌──────────────────┐
│  Sentiment Agent │  Analyzes emotional tone
│  Input: message  │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│  Analyzer Agent  │  Extracts issues, priority
│  Input: message  │
│       + sentiment│
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│  Response Agent  │  Generates empathetic reply
│  Input: message  │
│       + sentiment│
│       + analysis │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ Translation Agent│  Translates to Spanish
│  Input: response │
└──────────────────┘

The key innovation: context chaining. Each agent receives not just the original message, but the outputs of all previous agents. The Response Agent knows the customer’s sentiment AND the analyzed issues, enabling truly contextual responses.

All agents are registered in Apicurio Registry with:

AGENT_CARD artifacts describing their capabilities
PROMPT_TEMPLATE artifacts for their system prompts
MODEL_SCHEMA artifacts validating their inputs and outputs

Running the demo is simple:

cd examples/a2a-real-world-integration
docker-compose up -d
mvn clean compile exec:java
open http://localhost:9000  # Web UI

You’ll see agents discover each other through the registry, validate their communications against registered schemas, and pass accumulated context through the pipeline.

What’s coming next

This is just the beginning. Our roadmap includes:

Phase 2: Advanced A2A Features

DID and Verifiable Credentials for federated trust
Auto-generate Agent Cards from OpenAPI x-agent-* extensions
JSON-LD and potentialAction discovery

Phase 3: Agent Ecosystem Expansion

Agent lifecycle management (health monitoring, status tracking)
MCP Tool Definition artifact type
Agent Workflow Definition artifact type
Semantic search for natural language agent discovery
Agent deployment pipeline integration

Get started today

Apicurio Registry with AI support is available now:

Try the demo: Clone the repository and run the A2A real-world integration example
Deploy the registry: Use our container images or Operator
Read the docs: Check the MCP Server README for integration details
Join the discussion: Follow the epic to track progress and contribute ideas

A natural evolution

What makes this evolution “natural” is that we didn’t bolt on AI support as an afterthought. The same principles that make Apicurio Registry valuable for API governance—versioning, validation, compatibility, discovery, audit—apply directly to managing AI agents at scale.

As AI agents become essential infrastructure, they need the same level of governance that APIs have today. Apicurio Registry is ready to provide it.

The Apicurio team is committed to making schema and artifact management seamless for the AI era. We’d love to hear your feedback and use cases—open an issue or join us in the discussions.

Externalizing Authentication and Authorization in Apicurio Registry

Eric Wittmann — Wed, 28 Jan 2026 16:00:00 +0000

Hey everyone - I wanted to share something we’ve been working on that I think some of you might find interesting. We’ve created a proof of concept showing how to externalize authentication and authorization for Apicurio Registry using a reverse proxy pattern with Envoy and Open Policy Agent (OPA).

If you’ve ever thought “I wish I could handle auth at the gateway layer instead of in the application,” then read on!

Why externalize auth?

Great question! Apicurio Registry currently has built-in authentication and authorization support. It works, but there are some limitations from an enterprise architecture perspective. The implementation is tightly coupled to the application code, and while both authentication and authorization can be disabled, you’re stuck with the patterns we’ve implemented.

What if you want to:

Use a standard API gateway for all your services, not just Registry?
Centralize authentication and authorization policies across multiple applications?
Leverage existing infrastructure like Envoy, Istio, or other service meshes?
Keep your application layer focused on business logic, not auth concerns?
Implement authorization patterns/rules that Apicurio Registry simply does not support?

That’s where this example comes in!

What did we build?

We created a complete Docker Compose environment that demonstrates the reverse proxy pattern for externalized auth. Here’s what’s in the box:

Envoy Proxy - Acts as the Policy Enforcement Point (PEP), handling JWT validation and routing
Open Policy Agent (OPA) - Acts as the Policy Decision Point (PDP), enforcing authorization policies
Keycloak - Acts as the Identity Provider (IdP), issuing JWT tokens
Apicurio Registry - Running with auth disabled, trusting identity headers from Envoy
Apicurio UI - Configured to authenticate via Keycloak OIDC

The cool part is that Apicurio Registry doesn’t do any authentication or authorization itself. It just trusts the identity headers that Envoy injects after validating the JWT and checking with OPA. Of course, if you were to use this in production you would need to ensure trust between Envoy and Apicurio Registry (e.g. via network security or mTLS).

How does it work?

The request flow looks like this:

Client authenticates with Keycloak and gets a JWT (bearer token)
Client sends request to Registry via Envoy with Authorization: Bearer <JWT>
Envoy validates the JWT against Keycloak’s public keys (JWKS endpoint)
Envoy extracts claims from the JWT and injects identity headers:
- X-Forwarded-User - username from the JWT
- X-Forwarded-Email - email from the JWT
- X-Forwarded-Groups - roles from the JWT (converted to comma-separated string)
Envoy calls OPA to check if the request should be authorized
OPA evaluates its policies based on the user’s roles and the request details
If authorized, Envoy forwards the request to Registry with the identity headers
Registry processes the request, trusting the headers from Envoy

                    ┌──────────────────────────────────────────────────────────┐
                    │                                                          │
    ┌────────┐      │    ┌───────────┐      ┌───────────┐      ┌──────────┐    │
    │        │ (1)  │    │           │ (3)  │           │ (4)  │          │    │
    │ Client │──────┼───>│ Keycloak  │<────>│   Envoy   │<────>│   OPA    │    │
    │        │<─────┼────│   (IdP)   │   ┌─>│   (PEP)   │      │  (PDP)   │    │
    └────────┘  JWT │    └───────────┘   │  └─────┬─────┘      └──────────┘    │
         └──────────┼────────────────────┘        │                            │
                    │        (2)                  │ (5)                        │
                    │      Request                │ Forward with               │
                    │       + JWT                 │ identity headers           │
                    │                             ▼                            │
                    │                      ┌─────────────┐                     │
                    │                      │  Apicurio   │                     │
                    │                      │  Registry   │                     │
                    │                      └─────────────┘                     │
                    │                                                          │
                    └──────────────────────────────────────────────────────────┘
                                  Docker Compose Environment

Flow:
 (1) Client authenticates with Keycloak and receives JWT token
 (2) Client sends request to Envoy with Authorization: Bearer <JWT>
 (3) Envoy validates JWT against Keycloak's JWKS endpoint
 (4) Envoy queries OPA for authorization decision
 (5) Envoy forwards request to Registry with X-Forwarded-* headers

The OPA policies implement role-based access control (RBAC) with three roles:

sr-admin - Full access to everything
sr-developer - Read and write access
sr-readonly - Read-only access (GET requests only)

We also added support for anonymous access to system endpoints like /apis/registry/v3/system/info, which is important for things like health checks and the UI configuration endpoint.

Essentially this mimics the existing RBAC support found in Registry. The benefit, of course, is that the OPA policies can be modified to be whatever you want them to be!

Show me the code!

The POC is available in the Registry repository at:

https://github.com/Apicurio/apicurio-registry/tree/main/distro/docker-compose/in-memory-with-envoy-opa

You can try it out right now (if you have apicurio-registry cloned locally):

cd $APICURIO_REGISTRY_HOME/distro/docker-compose/in-memory-with-envoy-opa
docker compose up -d

Wait about a minute for everything to start up, then:

Keycloak: http://localhost:8080/admin (admin/admin)
Registry API (via Envoy): http://localhost:8081/apis/registry/v3
Registry UI: http://localhost:8888 (admin/admin)

We’ve even included an automated test suite (test.sh) that validates all the authentication and authorization scenarios. Running the tests gives you 25+ passing test cases covering everything from anonymous access to role-based permissions.

What about CORS?

One interesting challenge we ran into was getting the UI to work with this setup. The UI runs on localhost:8888 and makes API calls to localhost:8081, which triggers the browser’s CORS policy.

We solved this by adding CORS configuration to Envoy that allows requests from the UI origin and properly handles preflight requests. The UI is configured to use OIDC authentication directly with Keycloak, and it includes the JWT token in the Authorization header for all API requests.

It all works seamlessly.

OPA policies in action

Here’s what one of the OPA authorization rules looks like:

# Admin users can do anything
allowed if {
    has_role("sr-admin")
}

# Read-only users can only GET
allowed if {
    has_role("sr-readonly")
    http_method == "GET"
}

Pretty straightforward! The policies are written in Rego (OPA’s policy language) and are stored in a separate policy.rego file. You can modify the policies and restart OPA to test different authorization scenarios without touching any application code.

What about owner-based authorization?

Good question! We had an interesting discussion about this. OPA does support making HTTP calls to external services during policy evaluation using the http.send function. So theoretically, OPA could query the Registry API to check if a user owns a particular artifact before authorizing a request.

However, this creates some challenges:

Latency - Every request would require an additional HTTP call
Load - Doubles the load on the Registry API
Complexity - Ownership data changes frequently and requires database access

A future evolution of this example will likely explore this area further, assuming we receive any feedback from interested parties! Come and open a discussion about this topic if you like:

https://github.com/Apicurio/apicurio-registry/discussions

Is this production ready?

This is a proof of concept, not a production-ready solution (at least not yet). There are a few things you’d want to add for production use:

TLS/HTTPS - Enable TLS for all external communication
Network isolation - Deploy Registry in a private network, only accessible from Envoy
High availability - Run multiple instances of each component
Monitoring - Add Prometheus metrics, distributed tracing, and logging
Secret management - Use proper secret management tools for credentials
Policy management - Use GitOps for managing OPA policies

But the foundation is solid, and the pattern is proven. If you’re interested in deploying Registry with externalized auth, this POC gives you a great starting point!

Try it out!

If this sounds interesting to you, I encourage you to check out the POC in the Registry repository. The README in the in-memory-with-envoy-opa directory has detailed instructions on how to run it, test it, and customize it for your needs.

And as always, feedback is welcome! If you try this out and run into issues, or if you have ideas for improvements, please let us know on GitHub or Zulip.

Thanks for reading, and happy registering!

OpenTelemetry Distributed Tracing with Apicurio Registry

Carles Arnal Castello — Mon, 19 Jan 2026 12:00:00 +0000

Observability is critical in distributed systems. When you’re running microservices that communicate through Kafka and rely on Apicurio Registry for schema management, understanding the complete request flow becomes essential for debugging and performance optimization. In this post, we introduce comprehensive OpenTelemetry support in Apicurio Registry and provide a complete example demonstrating end-to-end distributed tracing.

What is OpenTelemetry?

OpenTelemetry (OTel) is a vendor-neutral observability framework that provides APIs, SDKs, and tools for collecting telemetry data including traces, metrics, and logs. It has become the industry standard for instrumenting distributed systems.

Distributed tracing allows you to follow a request as it travels through multiple services, making it invaluable for:

Debugging: Identify where failures occur in a multi-service flow
Performance analysis: Find bottlenecks across service boundaries
Dependency mapping: Understand how services interact
Root cause analysis: Trace issues back to their origin

OpenTelemetry in Apicurio Registry

Apicurio Registry is built with all OpenTelemetry signals enabled at build time (traces, metrics, and logs). However, the OTel SDK is disabled by default at runtime to avoid any performance impact for users who don’t need observability.

To enable OpenTelemetry, simply set:

QUARKUS_OTEL_SDK_DISABLED=false

Once enabled, Registry operations such as schema registration, lookup, and validation will generate spans that integrate seamlessly with your existing distributed traces.

Architecture Overview

The following diagram shows how traces flow through a typical setup with Kafka and Apicurio Registry:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Quarkus        │────▶│     Kafka       │────▶│    Quarkus      │
│  Producer       │     │   (Strimzi)     │     │    Consumer     │
└────────┬────────┘     └─────────────────┘     └────────┬────────┘
         │                                               │
         │              ┌─────────────────┐              │
         └─────────────▶│   Apicurio      │◀─────────────┘
                        │   Registry      │
                        │  (OTel enabled) │
                        └────────┬────────┘
                                 │
                        ┌────────▼────────┐
                        │   OpenTelemetry │
                        │   Collector     │
                        └────────┬────────┘
                                 │
                        ┌────────▼────────┐
                        │     Jaeger      │
                        │  (Visualization)│
                        └─────────────────┘

The key components are:

Producer Application: Sends Avro messages to Kafka, registering schemas with Apicurio Registry
Consumer Application: Consumes messages from Kafka, looking up schemas from Registry
Apicurio Registry: Provides schema storage and validation with OTel instrumentation
OpenTelemetry Collector: Aggregates and forwards telemetry data
Jaeger: Visualizes the distributed traces

Trace Propagation

One of the most powerful features of OpenTelemetry is automatic context propagation. When properly configured, trace context flows seamlessly across service boundaries:

Kafka Message Propagation

The OpenTelemetry instrumentation for Kafka automatically:

Producer side: Injects trace context (trace ID, span ID) into Kafka message headers
Consumer side: Extracts trace context from Kafka message headers and creates child spans

This creates a continuous trace from producer through Kafka to consumer.

Registry Correlation

When producer or consumer applications communicate with Apicurio Registry:

HTTP client instrumentation propagates trace context in HTTP headers
Registry operations (schema registration, lookup) appear as child spans
Schema caching behavior becomes visible in trace timing (first calls are slower, subsequent calls faster due to caching)

Complete Example

We’ve created a comprehensive example demonstrating end-to-end distributed tracing. The example includes Docker Compose and Kubernetes deployments, making it easy to try locally or in a cluster.

Quick Start with Docker Compose

1. Clone and build

# From the apicurio-registry repository
cd examples/otel-tracing

# Build producer
cd producer && mvn clean package -DskipTests && cd ..

# Build consumer
cd consumer && mvn clean package -DskipTests && cd ..

2. Start the infrastructure

docker compose up -d

3. Access the services

Service	URL
Jaeger UI	http://localhost:16686
Apicurio Registry	http://localhost:8083
Producer API	http://localhost:8084
Consumer API	http://localhost:8085

4. Generate traces

Send a greeting message through the system:

# Send a message
curl -X POST "http://localhost:8084/greetings?name=World"

# Consume the message
curl "http://localhost:8085/consumer/greetings"

5. View traces in Jaeger

Open http://localhost:16686 and:

Select a service: greeting-producer, greeting-consumer, or apicurio-registry
Click “Find Traces”
Click on a trace to see the complete flow across all services

Understanding Trace Visualization and Performance Analysis

When you open a trace in Jaeger, you get a powerful view into exactly how your distributed system behaves. The following screenshot shows a real trace from the batch greeting endpoint:

Reading the Trace

The trace visualization displays a hierarchical timeline where:

Each row represents a span - a unit of work within a service
Indentation shows parent-child relationships - child spans are nested under their parents
Bar length indicates duration - longer bars mean more time spent
Bar position shows timing - when each operation started relative to the trace

In this example trace, you can see the complete flow:

greeting-producer POST /greetings/batch (112.6ms total) - The incoming HTTP request
greeting-producer greetings publish (80.62ms) - Publishing messages to Kafka
apicurio-registry POST /apis/registry/v3/groups/{groupId}/artifacts (59.27ms) - Schema registration call to Registry

The Registry span expands to show internal storage operations:

Operation	Duration	Purpose
`storage.isGroupExists`	2.57ms	Check if the artifact group exists
`storage.getGlobalRules`	538µs	Retrieve global validation rules
`storage.isReadOnly`	56µs	Check if registry is in read-only mode
`storage.isReady`	12µs	Health check
`storage.createArtifact`	15.95ms	Create the new artifact (most expensive)
`storage.getArtifactVersionMetaDataByContent`	4.25ms	Retrieve version metadata
`storage.getArtifactMetaData`	2.13ms	Get artifact metadata

Using Traces for Performance Optimization

This level of detail enables powerful performance analysis:

Identify Bottlenecks

The trace immediately reveals where time is spent. In this example, storage.createArtifact takes 15.95ms - the largest portion of the Registry call. If you’re experiencing slow schema registrations, this pinpoints exactly which operation to investigate.

Measure Latency Breakdown

You can calculate the percentage of time each component contributes:

Producer processing: ~20% of total time
Network + Kafka: ~10%
Registry operations: ~53%
Consumer processing: ~17%

Detect Caching Effects

Notice how subsequent consumer operations (greetings process) take only 480µs compared to the initial 7.69ms. This demonstrates schema caching in action - once a schema is fetched from Registry, it’s cached locally, dramatically reducing latency for subsequent messages.

Compare Across Requests

Jaeger allows you to compare traces side-by-side. You can:

Compare slow vs. fast requests to identify anomalies
Track performance trends over time
Validate that optimizations have the expected impact

Set Performance Baselines

By collecting traces over time, you can establish baseline performance metrics:

P50/P95/P99 latencies for Registry operations
Expected duration for schema registration vs. lookup
Acceptable overhead for distributed tracing itself

This data helps you set meaningful SLOs and alerts for your schema registry infrastructure.

Custom Span Creation

The example demonstrates two approaches to creating custom spans for your own applications:

Declarative Approach with @WithSpan

@WithSpan("create-greeting")
private Greeting createGreeting(
        @SpanAttribute("greeting.recipient") String name,
        @SpanAttribute("greeting.trace_id") String traceId) {

    Span.current().setAttribute("greeting.source", "greeting-producer");
    Span.current().addEvent("greeting-object-created");
    // Business logic here
    return greeting;
}

Programmatic Approach with Tracer API

Span processSpan = tracer.spanBuilder("process-detailed-greeting")
        .setSpanKind(SpanKind.INTERNAL)
        .setAttribute("greeting.recipient", name)
        .setAttribute("greeting.priority", priority)
        .startSpan();

try (Scope scope = processSpan.makeCurrent()) {
    processSpan.addEvent("validation-started");
    // Business logic here
    processSpan.addEvent("validation-completed");
    processSpan.setStatus(StatusCode.OK);
} finally {
    processSpan.end();
}

Error Tracing

The example includes dedicated endpoints for demonstrating error tracing:

# Test different error types
curl -X POST "http://localhost:8084/greetings/invalid?errorType=validation"
curl -X POST "http://localhost:8084/greetings/invalid?errorType=schema"
curl -X POST "http://localhost:8084/greetings/invalid?errorType=kafka"

Error spans in Jaeger appear with:

Red color indicating error status
Exception stack traces in span logs
Custom error attributes and events

Sampling Strategies

The example uses parentbased_always_on sampling (100% of traces), which is ideal for development. For production environments, consider these alternatives:

Strategy	Description	Use Case
`always_on`	Sample all traces	Development
`parentbased_traceidratio`	Sample a percentage with parent inheritance	Production
`always_off`	Disable tracing	When not needed

Configure sampling via environment variables:

QUARKUS_OTEL_TRACES_SAMPLER: "parentbased_traceidratio"
QUARKUS_OTEL_TRACES_SAMPLER_ARG: "0.1"  # 10% sampling

Configuration Reference

Apicurio Registry OpenTelemetry Properties

Property	Description	Default
`quarkus.otel.sdk.disabled`	Disable/enable OTel SDK at runtime	`true`
`quarkus.otel.service.name`	Service name in traces	Application name
`quarkus.otel.exporter.otlp.endpoint`	OTLP collector endpoint	`http://localhost:4317`
`quarkus.otel.exporter.otlp.protocol`	OTLP protocol (grpc or http/protobuf)	`grpc`
`quarkus.otel.traces.sampler.arg`	Sampler ratio (0.0 to 1.0)	`0.1`

Important: Signal properties (traces.enabled, metrics.enabled, logs.enabled) are build-time properties. Use quarkus.otel.sdk.disabled for runtime control.

Kubernetes Deployment

The example also includes complete Kubernetes manifests for deployment with Strimzi. Key steps:

Install Strimzi operator
Build and push container images
Deploy with kubectl apply -k k8s/
Port-forward to access services

See the example README for detailed Kubernetes instructions.

Troubleshooting

No traces appearing in Jaeger

Check OTel Collector logs: docker compose logs otel-collector
Verify OTLP endpoint configuration
Ensure QUARKUS_OTEL_SDK_DISABLED=false is set for Apicurio Registry

Missing trace context in consumer

Ensure both quarkus-opentelemetry and quarkus-kafka-client dependencies are present
Check Kafka message headers for trace context
Verify OTel is enabled in both producer and consumer

Schema registration failures

Check Registry health: curl http://localhost:8083/health
Verify Registry URL in application configuration
Check network connectivity between services

Conclusion

OpenTelemetry support in Apicurio Registry enables powerful observability capabilities for your schema registry operations. Combined with the new example demonstrating end-to-end distributed tracing with Kafka, you now have everything needed to implement comprehensive observability in your event-driven architectures.

The example is available in the Apicurio Registry repository under examples/otel-tracing. We encourage you to try it out and provide feedback!

For more details, check:

Apicurio Registry - JSON Schema and OpenAPI dereference

Carles Arnal Castello — Tue, 25 Nov 2025 16:00:00 +0000

Apicurio Registry allows to manage artifacts with references as shown in the documentation. One of the cool features we’ve added on top of this is the possibility of using a dereference parameter for certain API endpoints that optionally allows you to fetch the full content of an artifact with all the references inlined within the same content. This is especially useful in certain contexts to reduce the number of HTTP requests in the Kafka Serializers and Deserializers, as you will see in this blog.

The dereference feature is supported for Avro, JSON Schema, Protobuf, OpenAPI, and AsyncAPI artifact types.

Intro

The code example used for this blog post can be found in the Apicurio Registry examples.

One of the limitations of the JSON Schema format is that you cannot discover the full schema from some data that adheres to the schema. That’s the main reason why, for this serde, the schema being used has to be registered upfront, and that’s what you see in the code example from line 52 to line 93. where the main schema citizen.json is registered with all the references in the hierarchy. Along those lines, a full hierarchy of schemas is registered in Apicurio Registry. The hierarchy can be described as follows and represented as a tree:

    citizen
       |
      city

In the code example there are also java classes that represent each of the objects we have in the structure, so we can use them when producing Kafka messages. There is one Java class per each JSON Schema file.

Using the REST API

Although the example above uses the Apicurio Registry Java client to register the full hierarchy of schemas, it’s also possible to use the REST API. Note that, in Apicurio Registry v3, when an artifact with references is created, the request body follows a structured format with artifactId, artifactType, and firstVersion containing the content and references. Here’s a set of Curl commands that delivers this result:

First we must start creating the leaves, in this case, the city schema:

//Creates the city schema
curl -X POST 'http://localhost:8080/apis/registry/v3/groups/default/artifacts' \
-H 'Content-Type: application/json' \
-d '{
  "artifactId": "city",
  "artifactType": "JSON",
  "firstVersion": {
    "content": {
      "content": "{\"$id\":\"https://example.com/city.schema.json\",\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"title\":\"City\",\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\",\"description\":\"The city'\''s name.\"},\"zipCode\":{\"type\":\"integer\",\"description\":\"The zip code.\",\"minimum\":0}}}",
      "contentType": "application/json"
    }
  }
}'

Once the city schema is created, we can create the citizen schema that references it:

//Creates the citizen schema with reference to city
curl -X POST 'http://localhost:8080/apis/registry/v3/groups/default/artifacts' \
-H 'Content-Type: application/json' \
-d '{
  "artifactId": "citizen",
  "artifactType": "JSON",
  "firstVersion": {
    "content": {
      "content": "{\"$id\":\"https://example.com/citizen.schema.json\",\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"title\":\"Citizen\",\"type\":\"object\",\"properties\":{\"firstName\":{\"type\":\"string\",\"description\":\"The citizen'\''s first name.\"},\"lastName\":{\"type\":\"string\",\"description\":\"The citizen'\''s last name.\"},\"age\":{\"description\":\"Age in years which must be equal to or greater than zero.\",\"type\":\"integer\",\"minimum\":0},\"city\":{\"$ref\":\"city.json\"}}}",
      "contentType": "application/json",
      "references": [
        {
          "name": "city.json",
          "groupId": "default",
          "artifactId": "city",
          "version": "1"
        }
      ]
    }
  }
}'

JSON Schema dereference

By default, the endpoints that return the content of a particular artifact fetch just the main content, and then you must get the links to the references and, one by one, do a full scan of the hierarchy to fetch any other sub-schema needed. For instance, the content of the citizen.json schema is the following:

{
  "$id": "https://example.com/citizen.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Citizen",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The citizen's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The citizen's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    },
    "city": {
      "$ref": "city.json"
    }
  }
}

Without dereferencing in order to be able to produce Kafka messages to adhere to this schema, that is already registered in Apicurio Registry, we would need to fetch all the nested references with multiple HTTP requests and then parse all of them while, when dereferencing is used, this is what you get:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Citizen",
  "type": "object",
  "properties": {
    "firstName": {
      "description": "The citizen's first name.",
      "type": "string"
    },
    "lastName": {
      "description": "The citizen's last name.",
      "type": "string"
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    },
    "city": {
      "title": "City",
      "type": "object",
      "properties": {
        "zipCode": {
          "description": "The zip code.",
          "type": "integer",
          "minimum": 0
        },
        "name": {
          "description": "The city's name.",
          "type": "string"
        }
      }
    }
  },
  "$id": "https://example.com/citizen.schema.json"
}

As you can see, that’s the content of the full hierarchy, not just the schema of the citizen element alone. In Apicurio Registry v3, you can get the content dereferenced using the references=DEREFERENCE query parameter with the following curl commands:

curl 'http://localhost:8080/apis/registry/v3/ids/globalIds/2?references=DEREFERENCE'
curl 'http://localhost:8080/apis/registry/v3/groups/default/artifacts/citizen/versions/1/content?references=DEREFERENCE'

OpenAPI dereference ===

The dereference feature also works for OpenAPI specifications. OpenAPI specs commonly use $ref to reference external schema components, and the dereference parameter allows you to inline those references.

Here’s an example of how to register OpenAPI specs with references:

First, create a shared schema component:

//Creates the address types OpenAPI component
curl -X POST 'http://localhost:8080/apis/registry/v3/groups/default/artifacts' \
-H 'Content-Type: application/json' \
-d '{
  "artifactId": "address-types",
  "artifactType": "OPENAPI",
  "firstVersion": {
    "content": {
      "content": "{\"openapi\":\"3.0.3\",\"info\":{\"title\":\"Address Types\",\"version\":\"1.0.0\"},\"paths\":{},\"components\":{\"schemas\":{\"Address\":{\"title\":\"Address\",\"type\":\"object\",\"properties\":{\"street\":{\"type\":\"string\",\"description\":\"Street address\"},\"city\":{\"type\":\"string\",\"description\":\"City name\"},\"zipCode\":{\"type\":\"string\",\"description\":\"Postal code\"}}}}}}",
      "contentType": "application/json"
    }
  }
}'

Then create an API that references the address component:

//Creates the User API with reference to address
curl -X POST 'http://localhost:8080/apis/registry/v3/groups/default/artifacts' \
-H 'Content-Type: application/json' \
-d '{
  "artifactId": "user-api",
  "artifactType": "OPENAPI",
  "firstVersion": {
    "content": {
      "content": "{\"openapi\":\"3.0.3\",\"info\":{\"title\":\"User API\",\"version\":\"1.0.0\"},\"paths\":{\"/users\":{\"get\":{\"summary\":\"List users\",\"operationId\":\"getUsers\",\"responses\":{\"200\":{\"description\":\"Success\",\"content\":{\"application/json\":{\"schema\":{\"type\":\"array\",\"items\":{\"$ref\":\"#/components/schemas/User\"}}}}}}}}},\"components\":{\"schemas\":{\"User\":{\"title\":\"User\",\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"},\"email\":{\"type\":\"string\"},\"address\":{\"$ref\":\"#/components/schemas/Address\"}}},\"Address\":{\"$ref\":\"address-types.json#/components/schemas/Address\"}}}}",
      "contentType": "application/json",
      "references": [
        {
          "name": "address-types.json#/components/schemas/Address",
          "groupId": "default",
          "artifactId": "address-types",
          "version": "1"
        }
      ]
    }
  }
}'

You can then fetch the dereferenced OpenAPI specification:

curl 'http://localhost:8080/apis/registry/v3/groups/default/artifacts/user-api/versions/1/content?references=DEREFERENCE'

Note that OpenAPI dereferencing works by inlining the referenced schemas into the components/schemas section of the specification.

Referencing options ===

Note that, even though referencing a whole schema definition is the most common use case, it’s also possible to reference a single object definition defined in a json file with multiple schemas. For example, we might have the following schema definition:

{
  "$id": "https://example.com/types/all-types.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "definitions": {
    "City": {
      "title": "City",
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "The city's name."
        },
        "zipCode": {
          "type": "integer",
          "description": "The zip code.",
          "minimum": 0
        }
      }
    },
    "Identifier": {
      "title": "Identifier",
      "type": "object",
      "properties": {
        "identifier": {
          "type": "integer",
          "description": "The citizen identifier.",
          "minimum": 0
        }
      }
    }
  }
}

With definitions for both an identifier and the city objects. Then we might decide that we want to have a separate schema that points to an entire definition, like the one below:

{
  "$id": "https://example.com/citizen.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Citizen",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The citizen's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The citizen's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    },
    "city": {
      "$ref": "types/all-types.json#/definitions/City"
    },
    "identifier": {
      "$ref": "types/all-types.json#/definitions/Identifier"
    }
  },
  "required": [
    "city"
  ]
}

Or we might decide to just reference a single property inside a schema definition, like the schema below does:

{
  "$id": "https://example.com/citizen.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Citizen",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The citizen's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The citizen's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    },
    "city": {
      "$ref": "types/all-types.json#/definitions/City/properties/name"
    },
    "identifier": {
      "$ref": "types/all-types.json#/definitions/Identifier"
    }
  },
  "required": [
    "city"
  ]
}

The three options work exactly the same way when it comes to registering those schemas using the REST API as if we were registering a schema that references another. First we must register the schema that will be referenced, in this case, the one with the definitions:

curl -X POST 'http://localhost:8080/apis/registry/v3/groups/default/artifacts' \
-H 'Content-Type: application/json' \
-d '{
  "artifactId": "all-types",
  "artifactType": "JSON",
  "firstVersion": {
    "content": {
      "content": "{\"$id\":\"https://example.com/types/all-types.json\",\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"definitions\":{\"City\":{\"title\":\"City\",\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\",\"description\":\"The city'\''s name.\"},\"zipCode\":{\"type\":\"integer\",\"description\":\"The zip code.\",\"minimum\":0}}},\"Identifier\":{\"title\":\"Identifier\",\"type\":\"object\",\"properties\":{\"identifier\":{\"type\":\"integer\",\"description\":\"The citizen identifier.\",\"minimum\":0}}}}}",
      "contentType": "application/json"
    }
  }
}'

Once this schema has been registered, we can now decide to register two separate schemas, the first one referencing the entire city object:

curl -X POST 'http://localhost:8080/apis/registry/v3/groups/default/artifacts' \
-H 'Content-Type: application/json' \
-d '{
  "artifactId": "citizen-object-level",
  "artifactType": "JSON",
  "firstVersion": {
    "content": {
      "content": "{\"$id\":\"https://example.com/citizen.json\",\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"title\":\"Citizen\",\"type\":\"object\",\"properties\":{\"firstName\":{\"type\":\"string\",\"description\":\"The citizen'\''s first name.\"},\"lastName\":{\"type\":\"string\",\"description\":\"The citizen'\''s last name.\"},\"age\":{\"description\":\"Age in years which must be equal to or greater than zero.\",\"type\":\"integer\",\"minimum\":0},\"city\":{\"$ref\":\"types/all-types.json#/definitions/City\"},\"identifier\":{\"$ref\":\"types/all-types.json#/definitions/Identifier\"}},\"required\":[\"city\"]}",
      "contentType": "application/json",
      "references": [
        {
          "name": "types/all-types.json#/definitions/City",
          "groupId": "default",
          "artifactId": "all-types",
          "version": "1"
        },
        {
          "name": "types/all-types.json#/definitions/Identifier",
          "groupId": "default",
          "artifactId": "all-types",
          "version": "1"
        }
      ]
    }
  }
}'

And then register the second schema, where just the city name property is referenced:

curl -X POST 'http://localhost:8080/apis/registry/v3/groups/default/artifacts' \
-H 'Content-Type: application/json' \
-d '{
  "artifactId": "citizen-property-level",
  "artifactType": "JSON",
  "firstVersion": {
    "content": {
      "content": "{\"$id\":\"https://example.com/citizen.json\",\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"title\":\"Citizen\",\"type\":\"object\",\"properties\":{\"firstName\":{\"type\":\"string\",\"description\":\"The citizen'\''s first name.\"},\"lastName\":{\"type\":\"string\",\"description\":\"The citizen'\''s last name.\"},\"age\":{\"description\":\"Age in years which must be equal to or greater than zero.\",\"type\":\"integer\",\"minimum\":0},\"city\":{\"$ref\":\"types/all-types.json#/definitions/City/properties/name\"},\"identifier\":{\"$ref\":\"types/all-types.json#/definitions/Identifier\"}},\"required\":[\"city\"]}",
      "contentType": "application/json",
      "references": [
        {
          "name": "types/all-types.json#/definitions/City/properties/name",
          "groupId": "default",
          "artifactId": "all-types",
          "version": "1"
        },
        {
          "name": "types/all-types.json#/definitions/Identifier",
          "groupId": "default",
          "artifactId": "all-types",
          "version": "1"
        }
      ]
    }
  }
}'

The most relevant part here is the name of the reference (apart from the content itself, of course). The name of the reference is what’s going to determine the part of the schema to be used to fill the reference in the main content. In the first case, in the object scenario, the reference name is types/all-types.json#/definitions/City creating a reference to the all-types.json file in general, and to the city definition in particular. In the second scenario, the property path is used for the reference name types/all-types.json#/definitions/City/properties/name, creating a reference to the all-types.json file in general, and to the city name in particular.

This is especially important for dereferencing content, since the final result will be different for each schema. In the object scenario, the city object will be inlined within the main schema as follows:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Citizen",
    "type": "object",
    "properties": {
        "firstName": {
            "description": "The citizen's first name.",
            "type": "string"
        },
        "lastName": {
            "description": "The citizen's last name.",
            "type": "string"
        },
        "identifier": {
            "title": "Identifier",
            "type": "object",
            "properties": {
                "identifier": {
                    "description": "The citizen identifier.",
                    "type": "integer",
                    "minimum": 0
                }
            }
        },
        "city": {
            "title": "City",
            "type": "object",
            "properties": {
                "zipCode": {
                    "description": "The zip code.",
                    "type": "integer",
                    "minimum": 0
                },
                "name": {
                    "description": "The city's name.",
                    "type": "string"
                }
            }
        },
        "age": {
            "description": "Age in years which must be equal to or greater than zero.",
            "type": "integer",
            "minimum": 0
        }
    },
    "required": [
        "city"
    ],
    "$id": "https://example.com/citizen.json"
}

Whereas in the second scenario, the property one, just the city name will be inlined:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Citizen",
    "type": "object",
    "properties": {
        "firstName": {
            "description": "The citizen's first name.",
            "type": "string"
        },
        "lastName": {
            "description": "The citizen's last name.",
            "type": "string"
        },
        "identifier": {
            "title": "Identifier",
            "type": "object",
            "properties": {
                "identifier": {
                    "description": "The citizen identifier.",
                    "type": "integer",
                    "minimum": 0
                }
            }
        },
        "city": {
            "description": "The city's name.",
            "type": "string"
        },
        "age": {
            "description": "Age in years which must be equal to or greater than zero.",
            "type": "integer",
            "minimum": 0
        }
    },
    "required": [
        "city"
    ],
    "$id": "https://example.com/citizen.json"
}

JSON Schema Kafka Serde dereference

Now, in order to make good use of all of this, we can use a Kafka producer. For this example, in the Kafka Producer there is one extremely important property we must set, the schema resolver configuration option DEREFERENCE_SCHEMA. This is the option that will instruct the Schema Resolver to use the dereference parameter when fetching the schema from the registry.

Properties props = new Properties();

// Configure kafka settings
props.putIfAbsent(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, SERVERS);
props.putIfAbsent(ProducerConfig.CLIENT_ID_CONFIG, "Producer-" + TOPIC_NAME);
props.putIfAbsent(ProducerConfig.ACKS_CONFIG, "all");
props.putIfAbsent(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.putIfAbsent(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSchemaKafkaSerializer.class.getName());
props.putIfAbsent(SerdeConfig.ARTIFACT_RESOLVER_STRATEGY, SimpleTopicIdStrategy.class.getName());
props.putIfAbsent(SerdeConfig.EXPLICIT_ARTIFACT_GROUP_ID, "default");
props.putIfAbsent(SerdeConfig.REGISTRY_URL, REGISTRY_URL);

//This is the most important configuration for this particular example.
props.putIfAbsent(SerdeConfig.DEREFERENCE_SCHEMA, true);


// Create the Kafka producer
Producer<Object, Object> producer = new KafkaProducer<>(props);

With this producer we can then send messages to Kafka that adhere to the citizen schema as follows:

for (int idx = 0; idx < 5; idx++) {
    // Create the message to send
    Citizen citizen = new Citizen();
    City city = new City();
    city.setZipCode(45676);
    city.setName(UUID.randomUUID().toString());
    citizen.setCity(city);
    citizen.setAge(idx + 20);
    citizen.setFirstName(UUID.randomUUID().toString());
    citizen.setLastName(UUID.randomUUID().toString());

    // Send/produce the message on the Kafka Producer
    ProducerRecord<Object, Object> producedRecord = new ProducerRecord<>(topicName, SUBJECT_NAME, citizen);
    producer.send(producedRecord);
}

And consume them with a consumer that is created as follows, instructing the deserializer to use the schema dereferenced with the following configuration property DEREFERENCE_SCHEMA:

Properties props = new Properties();

props.putIfAbsent(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, SERVERS);
props.putIfAbsent(ConsumerConfig.GROUP_ID_CONFIG, "Consumer-" + TOPIC_NAME);
props.putIfAbsent(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
props.putIfAbsent(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
props.putIfAbsent(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.putIfAbsent(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.putIfAbsent(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonSchemaKafkaDeserializer.class.getName());
props.putIfAbsent(SerdeConfig.ARTIFACT_RESOLVER_STRATEGY, SimpleTopicIdStrategy.class.getName());
props.putIfAbsent(SerdeConfig.EXPLICIT_ARTIFACT_GROUP_ID, "default");
props.putIfAbsent(SerdeConfig.VALIDATION_ENABLED, true);

//This is the most important configuration for this example, that instructs the deserializer to fetch the dereferenced schema.
props.putIfAbsent(SerdeConfig.DEREFERENCE_SCHEMA, true);

// Configure Service Registry location
props.putIfAbsent(SerdeConfig.REGISTRY_URL, REGISTRY_URL);
// No other configuration needed for the deserializer, because the globalId of the schema
// the deserializer should use is sent as part of the payload.  So the deserializer simply
// extracts that globalId and uses it to look up the Schema from the registry.

// Create the Kafka Consumer
KafkaConsumer<Long, Citizen> consumer = new KafkaConsumer<>(props);

And then consume the messages with the consumer configured above like this:

while (messageCount < 5) {
    final ConsumerRecords<Long, Citizen> records = consumer.poll(Duration.ofSeconds(1));
    messageCount += records.count();
    if (records.count() == 0) {
        // Do nothing - no messages waiting.
        System.out.println("No messages waiting...");
    } else records.forEach(record -> {
        Citizen msg = record.value();
        System.out.println("Consumed a message: " + msg + " @ " + msg.getCity());
    });
}

Instead of doing an HTTP request to get the citizen schema, another request to fetch the pointers to the references, and additional requests to discover the full hierarchy, a single HTTP request gets the full content of the hierarchy and then uses it to produce/consume messages.

Extend Apicurio Registry with Custom Artifact Types at Deployment Time

Eric Wittmann — Mon, 27 Oct 2025 02:13:00 +0000

One of the most exciting new features in Apicurio Registry 3.1.0 is the ability to extend the registry with custom artifact types at deployment time, without rebuilding the project. This capability opens up Registry to support any schema or API definition format your organization uses, even if it’s not natively supported out of the box.

The Challenge: Supporting Custom Schema Formats

Apicurio Registry natively supports many popular schema and API definition formats:

OpenAPI
AsyncAPI
Apache Avro
JSON Schema
Google Protocol Buffers
GraphQL
WSDL
XML Schema (XSD)

But what if your organization uses other formats like RAML, Schematron, WADL, API Blueprint, or a completely custom schema format? Previously, you would have to fork the Registry project, implement support for your format, and maintain your own custom build.

Starting with Registry 3.1.0, that’s no longer necessary.

The Solution: Custom Artifact Types Configuration

The new custom artifact types feature allows you to configure Registry to support additional artifact types by providing a JSON configuration file at deployment time. You can implement the (optional) type-specific logic in three different ways:

Java classes - Implement Java interfaces for maximum performance (requires a custom build of Registry)
JavaScript/TypeScript - Write scripts for easier development and portability
Webhooks - Deploy HTTP endpoints for ultimate flexibility and language independence

Let’s explore how this works.

Configuration Overview

To add a custom artifact type, you create a JSON configuration file and point Registry to it using the apicurio.artifact-types.config-file configuration property.

Here’s the structure of the configuration file:

{
  "includeStandardArtifactTypes": true,
  "artifactTypes": [
    {
      "artifactType": "RAML",
      "name": "RAML",
      "description": "The simplest way to model APIs",
      "contentTypes": [
        "application/json",
        "application/x-yaml"
      ],
      "scriptLocation": "custom-artifact-types/dist/raml-at.js",
      "contentAccepter": {
        "type": "script"
      },
      "contentValidator": {
        "type": "script"
      },
      "compatibilityChecker": {
        "type": "script"
      },
      "contentCanonicalizer": {
        "type": "script"
      },
      "contentDereferencer": {
        "type": "script"
      },
      "referenceFinder": {
        "type": "script"
      }
    }
  ]
}

Configuration Properties

Let’s break down what each property does:

Top-Level Properties

includeStandardArtifactTypes: Set to true to include all built-in types (OpenAPI, Avro, etc.) alongside your custom types, or false to use only your custom types

Artifact Type Properties

artifactType: The unique identifier for your artifact type (e.g., “RAML”, “SCHEMATRON”)
name: Display name shown in e.g. logs
description: Human-readable description of the artifact type
contentTypes: List of MIME types for this artifact type’s content (e.g., “application/x-yaml”)

Provider Components

Each artifact type can implement these optional components to customize Registry behavior:

contentAccepter: Detects if content belongs to this artifact type (enables auto-detection)
contentValidator: Validates content syntax and structure
compatibilityChecker: Determines if new versions are compatible with existing ones
contentCanonicalizer: Normalizes content for consistent comparison
contentDereferencer: Resolves references to other artifacts
referenceFinder: Discovers external references in content

Each provider component specifies a type which can be java, script, or webhook.

Implementation Approach 1: JavaScript/TypeScript

The JavaScript approach provides a good balance between ease of development and performance. You write TypeScript (or JavaScript) code that exports specific functions.

Registry executes JavaScript using the QuickJS engine via the quickjs4j project. Scripts run in a sandboxed environment for security, which means they have limited access to system resources and cannot perform potentially dangerous operations like file system access or network requests.

Using TypeScript Types for Better Development Experience

For TypeScript and JavaScript developers, Apicurio provides an npm package with complete type definitions for all custom artifact type interfaces. This significantly improves the development experience by providing:

Full type safety - Catch errors at compile time rather than runtime
IDE autocomplete - Get intelligent code completion for request and response objects
Better documentation - Types serve as inline documentation for available properties
Compatibility assurance - Ensure your implementation matches Registry’s expectations

Installation

Install the TypeScript definitions package:

npm install @apicurio/artifact-type-builtins

Using the Types

Import the type definitions at the top of your TypeScript file:

import type {
    ContentAccepterRequest,
    ContentValidatorRequest,
    ContentValidatorResponse,
    CompatibilityCheckerRequest,
    CompatibilityCheckerResponse,
    ContentCanonicalizerRequest,
    ContentCanonicalizerResponse,
    ContentDereferencerRequest,
    ContentDereferencerResponse,
    ReferenceFinderRequest,
    ReferenceFinderResponse
} from '@apicurio/artifact-type-builtins';

Then apply these types to your function signatures:

// Content Validator with types
export function validate(request: ContentValidatorRequest): ContentValidatorResponse {
    const violations: any[] = [];
    const content: string = request.content.content;
    const contentType: string = request.content.contentType;

    // Your validation logic here

    return {
        ruleViolations: violations
    };
}

// Content Accepter with types
export function acceptsContent(request: ContentAccepterRequest): boolean {
    return request.typedContent.content.startsWith("#%RAML 1.0");
}

// Compatibility Checker with types
export function testCompatibility(request: CompatibilityCheckerRequest): CompatibilityCheckerResponse {
    // Your compatibility checking logic here
    return {
        incompatibleDifferences: []
    };
}

The typed interfaces provide IDE autocomplete for all properties, making it much easier to discover what data is available in requests and what structure is expected in responses.

Example: RAML Content Validator

Here’s a simple example that validates RAML content:

export function validate(request: any): any {
    const violations: any[] = [];
    const content: string = request.content.content;
    const contentType: string = request.content.contentType;

    if (contentType !== "application/x-yaml") {
        violations.push({
            description: "Incorrect content type. Expected 'application/x-yaml' but found '" + contentType + "'.",
            context: null
        });
    } else {
        if (!content.startsWith("#%RAML 1.0")) {
            violations.push({
                description: "Missing '#%RAML 1.0' content header.",
                context: null
            });
        }
    }

    return {
        ruleViolations: violations
    };
}

Configuration for JavaScript

{
  "artifactType": "RAML",
  "scriptLocation": "custom-types/raml-artifact-type.js",
  "contentValidator": {
    "type": "script"
  }
}

The script file is loaded from the classpath, so you can bundle it with your deployment or mount it as a volume in containerized environments.

Implementation Approach 2: Java Classes

For maximum performance and full access to the Java ecosystem, you can implement custom artifact types using Java classes.

Configuration for Java

{
  "artifactType": "RAML",
  "contentValidator": {
    "type": "java",
    "classname": "com.example.raml.RamlContentValidator"
  }
}

Your Java class would implement the ContentValidator interface:

package com.example.raml;

import io.apicurio.registry.rules.validity.ContentValidator;
import io.apicurio.registry.content.TypedContent;

public class RamlContentValidator implements ContentValidator {

    @Override
    public void validate(ValidityLevel level, TypedContent content,
                        Map<String, TypedContent> resolvedReferences) {
        String contentStr = content.getContent();

        if (!contentStr.startsWith("#%RAML 1.0")) {
            throw new RuleViolationException(
                "Missing '#%RAML 1.0' content header"
            );
        }

        // Additional validation logic here
    }
}

The Java class needs to be available on the Registry classpath, typically by creating your own build of Apicurio Registry (out of scope for this blog post).

Implementation Approach 3: Webhooks

The webhook approach provides ultimate flexibility by allowing you to implement the logic in any language and deploy it as a separate service.

Configuration for Webhooks

{
  "artifactType": "RAML",
  "contentValidator": {
    "type": "webhook",
    "url": "http://custom-validator-service:8080/validate"
  }
}

Webhook API Contract

Registry will POST a JSON request to your webhook endpoint and expect a JSON response. For example, the content validator webhook receives:

{
  "level": "SYNTAX_ONLY",
  "function": "validate",
  "content": {
    "content": "THE_ACTUAL_RAML_CONTENT",
    "contentType": "application/x-yaml"
  },
  "resolvedReferences": []
}

And should return:

{
  "ruleViolations": [
    {
      "description": "Missing '#%RAML 1.0' header",
      "context": null
    }
  ]
}

The complete webhook API specification is defined in an OpenAPI document included with Registry at META-INF/artifact-type-webhooks.json.

Use Cases

This feature is particularly valuable for organizations that:

Use proprietary schema formats - Internal schema definition languages can now be managed in Registry
Need RAML support - RAML users can add full Registry support for their API definitions
Work with legacy formats - Support for older standards like WADL or WSDL 1.1
Have domain-specific schemas - Schematron for healthcare, custom XML schemas for finance, etc.
Want to experiment - Try out emerging schema formats without waiting for official Registry support

Component Reference

Each custom artifact type can implement the following optional components. Below are the contracts for each implementation approach.

Content Accepter

Auto-detects if content belongs to this artifact type (enables automatic type detection when no explicit type is provided).

JavaScript:

export function acceptsContent(request: any): boolean {
    // request.typedContent.content - the actual content
    // request.typedContent.contentType - the MIME type
    // Returns: true if this artifact type accepts the content
    return request.typedContent.content.startsWith("#%RAML 1.0");
}

Webhook: POST to configured URL

// Request
{ "typedContent": { "content": "...", "contentType": "application/x-yaml" } }

// Response
{ "accepted": true }

Java: Implement ContentAccepter interface

boolean accepts(TypedContent content);

Content Validator

Validates content syntax and structure according to the artifact type’s rules.

JavaScript:

export function validate(request: any): any {
    // request.level - validation level (e.g., "SYNTAX_ONLY", "FULL")
    // request.content - TypedContent with content and contentType
    // request.resolvedReferences - array of resolved references
    // Returns: object with ruleViolations array
    return {
        ruleViolations: [
            { description: "Error message", context: "/path/to/error" }
        ]
    };
}

Webhook: POST to configured URL

// Request
{
  "level": "SYNTAX_ONLY",
  "function": "validate",
  "content": { "content": "...", "contentType": "..." },
  "resolvedReferences": []
}

// Response
{
  "ruleViolations": [
    { "description": "Error message", "context": "/path/to/error" }
  ]
}

Java: Implement ContentValidator interface

void validate(ValidityLevel level, TypedContent content,
              Map<String, TypedContent> resolvedReferences)
    throws RuleViolationException;

Compatibility Checker

Determines if a new version is compatible with existing versions according to the configured compatibility level.

JavaScript:

export function testCompatibility(request: any): any {
    // request.level - compatibility level (e.g., "BACKWARD", "FORWARD", "FULL")
    // request.existingArtifacts - array of existing TypedContent
    // request.proposedArtifact - new TypedContent to check
    // request.resolvedReferences - array of resolved references
    // Returns: object with incompatibleDifferences array
    return {
        incompatibleDifferences: [
            { description: "Incompatibility description", context: "/" }
        ]
    };
}

Webhook: POST to configured URL

// Request
{
  "level": "BACKWARD",
  "existingArtifacts": [ { "content": "...", "contentType": "..." } ],
  "proposedArtifact": { "content": "...", "contentType": "..." },
  "resolvedReferences": []
}

// Response
{
  "incompatibleDifferences": [
    { "description": "Incompatibility", "context": "/" }
  ]
}

Java: Implement CompatibilityChecker interface

CompatibilityExecutionResult testCompatibility(CompatibilityLevel level,
    List<TypedContent> existingArtifacts, TypedContent proposedArtifact,
    Map<String, TypedContent> resolvedReferences);

Content Canonicalizer

Normalizes content to a canonical form for consistent comparison and duplicate detection.

JavaScript:

export function canonicalize(request: any): any {
    // request.content - TypedContent with content and contentType
    // request.resolvedReferences - array of resolved references
    // Returns: object with canonicalized typedContent
    const canonical = normalizeWhitespace(request.content.content);
    return {
        typedContent: {
            contentType: request.content.contentType,
            content: canonical
        }
    };
}

Webhook: POST to configured URL

// Request
{
  "content": { "content": "...", "contentType": "..." },
  "resolvedReferences": []
}

// Response
{
  "typedContent": { "content": "canonical content", "contentType": "..." }
}

Java: Implement ContentCanonicalizer interface

TypedContent canonicalize(TypedContent content,
    Map<String, TypedContent> resolvedReferences);

Content Dereferencer

Resolves references to other artifacts, either by inlining referenced content (dereference) or rewriting references to URLs (rewriteReferences).

JavaScript:

export function dereference(request: any): any {
    // request.content - TypedContent with content and contentType
    // request.resolvedReferences - array of { name, content, contentType }
    // request.function - "dereference" or "rewriteReferences"
    // request.resolvedReferenceUrls - array of { name, url } (for rewrite)
    // Returns: object with dereferenced typedContent
    const dereferenced = inlineReferences(request.content.content,
                                          request.resolvedReferences);
    return {
        typedContent: {
            contentType: request.content.contentType,
            content: dereferenced
        }
    };
}

export function rewriteReferences(request: any): any {
    // Same as dereference but rewrites refs to URLs instead of inlining
    const rewritten = replaceWithUrls(request.content.content,
                                      request.resolvedReferenceUrls);
    return {
        typedContent: {
            contentType: request.content.contentType,
            content: rewritten
        }
    };
}

Webhook: POST to configured URL

// Request (dereference)
{
  "content": { "content": "...", "contentType": "..." },
  "function": "dereference",
  "resolvedReferences": [
    { "name": "ref.json", "content": "...", "contentType": "..." }
  ]
}

// Request (rewriteReferences)
{
  "content": { "content": "...", "contentType": "..." },
  "function": "rewriteReferences",
  "resolvedReferenceUrls": [
    { "name": "ref.json", "url": "http://registry/..." }
  ]
}

// Response
{
  "typedContent": { "content": "dereferenced content", "contentType": "..." }
}

Java: Implement ContentDereferencer interface

TypedContent dereference(TypedContent content,
    Map<String, TypedContent> resolvedReferences);

TypedContent rewriteReferences(TypedContent content,
    Map<String, String> resolvedReferenceUrls);

Reference Finder

Discovers external references within content (used by tooling like the Maven plugin for automatic reference resolution).

JavaScript:

export function findExternalReferences(request: any): any {
    // request.typedContent - TypedContent with content and contentType
    // Returns: object with externalReferences array
    const refs = parseReferences(request.typedContent.content);
    return {
        externalReferences: [
            { fullReference: "schemas/ref.json", resource: "schemas/ref.json" }
        ]
    };
}

Webhook: POST to configured URL

// Request
{
  "typedContent": { "content": "...", "contentType": "..." }
}

// Response
{
  "externalReferences": [
    { "fullReference": "ref.json#/Schema", "resource": "ref.json", "component": "/Schema" }
  ]
}

Java: Implement ReferenceFinder interface

Set<ExternalReference> findExternalReferences(TypedContent content);

Deployment Example

Here’s how to deploy Registry with a custom RAML artifact type using Docker:

1. Create your configuration file (`raml-config.json`):

{
  "includeStandardArtifactTypes": true,
  "artifactTypes": [
    {
      "artifactType": "RAML",
      "name": "RAML",
      "description": "RESTful API Modeling Language",
      "contentTypes": ["application/x-yaml"],
      "scriptLocation": "raml-artifact-type.js",
      "contentAccepter": { "type": "script" },
      "contentValidator": { "type": "script" },
      "compatibilityChecker": { "type": "script" }
    }
  ]
}

2. Create your JavaScript implementation (`raml-artifact-type.js`)

3. Deploy with Docker:

docker run -it -p 8080:8080 \
  -v $(pwd)/raml-config.json:/config/artifact-types.json \
  -v $(pwd)/raml-artifact-type.js:/deployments/raml-artifact-type.js \
  -e apicurio.artifact-types.config-file=/config/artifact-types.json \
  apicurio/apicurio-registry:3.1.0

4. Verify it works:

curl http://localhost:8080/apis/registry/v3/admin/config/artifactTypes

You should see your custom RAML type in the response along with all the standard types.

Performance Considerations

When choosing an implementation approach, consider:

Java classes - Fastest performance, no serialization overhead, but requires a custom build of Registry
JavaScript - Good performance, easier to develop and deploy, isolated execution
Webhooks - Most flexible, language-agnostic, but adds network latency

For high-throughput scenarios, Java classes offer the best performance. For moderate loads with frequent updates, JavaScript provides a good balance. For maximum flexibility or when integrating with existing services, webhooks are ideal.

Limitations

A few things to keep in mind:

The JavaScript runtime uses QuickJS, which supports ES2020 but not all Node.js APIs
Webhook endpoints must be accessible from the Registry deployment
Java classes must be compatible with the Registry’s Quarkus version
Content extraction is not yet supported for custom types

Looking Forward

This feature represents a significant step toward making Apicurio Registry truly universal. We’re excited to see what artifact types the community will add support for.

Some ideas we’ve heard interest in:

RAML 0.8 and 1.0
API Blueprint
Schematron
XACML
Custom DSLs for various domains

We’d love to hear about your use cases! If you implement support for a custom artifact type, consider sharing it with the community as a reusable module.

Get Started

Ready to try it out? Here are your next steps:

Check out the working example - See a complete custom artifact type implementation in the custom-artifact-type example
Read the documentation for more details about deploying Apicurio Registry
Try a simple implementation using the JavaScript approach
Join the discussion on our Zulip chat

For more information:

Custom Artifact Type Example - Complete working example
TypeScript Type Definitions - npm package with TypeScript types
Apicurio Registry GitHub Repository
Apicurio Registry Documentation
Webhook API Specification

We can’t wait to see what you build with this new capability!

Apicurio Studio Fully Deprecated: Functionality Integrated into Apicurio Registry 3.1.0

Eric Wittmann — Thu, 23 Oct 2025 03:01:00 +0000

We have an important announcement to make regarding the future of Apicurio Studio and how we plan to continue supporting our users who rely on its capabilities.

As of today, Apicurio Studio is fully deprecated. However, before you panic, let me explain what this means and how we’re ensuring that Studio’s valuable functionality remains available to you through Apicurio Registry.

The Challenge: Limited Bandwidth, Two Projects

The Apicurio team has come to an important realization: we simply do not have the bandwidth to author and maintain both Apicurio Studio and Apicurio Registry as separate projects. Between feature development, bug fixes, security updates, and community support, maintaining two distinct codebases has stretched our resources too thin.

After careful consideration, we’ve decided to continue our primary focus on Apicurio Registry, which has become the cornerstone of schema and API management for many organizations and is the foundation for multiple Red Hat products.

But here’s the critical part: we recognize that Studio is a popular and useful project, and we didn’t want to abandon the users who find it helpful. This commitment to supporting our Studio users has driven us to find a better solution.

The Solution: Studio Features Built into Registry 3.1.0

Rather than leaving Studio users without a path forward, we’ve directly incorporated Studio’s editing capabilities into the Apicurio Registry user experience. Starting with Apicurio Registry 3.1.0, all of the core Studio functionality is available as an opt-in feature within Registry itself.

This means you get the best of both worlds:

The robust schema/API management capabilities of Registry
The intuitive editing experience of Studio
A single project to install, configure, and maintain
Unified lifecycle management for your API designs

How It Works: The New “Drafts” Feature

The Studio functionality in Registry is built on top of a powerful new feature called Drafts. Let me explain how this works.

Understanding Draft State

In Apicurio Registry 3.1.0, we’ve introduced a new artifact version state: DRAFT. This state fundamentally changes one of Registry’s core behaviors.

Normally, artifact version content in Registry is immutable - once created, it cannot be changed. This immutability is crucial for production schema registries where consistency and reliability are paramount. However, during the design and development phase, you need the flexibility to iteratively edit and refine your API specifications.

The DRAFT state solves this problem. When an artifact version is in the DRAFT state and the content mutability feature is enabled, you can update the content of that version as many times as needed. Once you’re satisfied with your design, you can transition it out of draft state, at which point it becomes immutable like any other Registry artifact version.

Enabling Studio Features

To enable the Draft Mutability feature, and unlock Studio functionality in Registry, you need to set one configuration property:

apicurio.rest.mutability.artifact-version-content.enabled=true

When this feature is enabled, several new capabilities appear in the Registry UI:

1. Drafts Section

A new “Drafts” section becomes available in the UI, providing:

Easy searching for drafts that can be edited
Easy creation of new drafts, including from pre-defined templates
Draft version creation from existing artifact versions (whether they’re drafts or published versions)

This gives you a centralized place to manage all your in-progress Schema and API designs.

2. Edit Draft Button

When viewing any artifact version that is in DRAFT state, you’ll see a new “Edit Draft” button. Clicking this button launches the familiar Apicurio Studio editor experience, where you can:

Visually edit OpenAPI and AsyncAPI specifications
Make changes using the intuitive Studio interface
Save your changes directly back to Registry, updating the version content
Iterate on your design as many times as needed before finalizing

Try It Out: Quick Evaluation

Want to see this in action? Here’s the fastest way to evaluate the new Studio features in Registry:

Using Docker Compose

We’ve created a pre-configured Docker Compose setup that runs Registry with Studio features enabled. Here’s how to use it:

Download the docker-compose.yml file:

curl -o docker-compose.yml \
  https://raw.githubusercontent.com/Apicurio/apicurio-registry/main/distro/docker-compose/in-memory-with-studio/docker-compose.yml

Start the services:
```
docker compose up
```
This will start two containers:
- Apicurio Registry API (on port 8080)
- Apicurio Registry UI (on port 8888)
Access the UI: Open your browser to http://localhost:8888

The docker-compose configuration automatically enables the content mutability feature with this environment variable:

apicurio.rest.mutability.artifact-version-content.enabled: "true"

Using Individual Docker Containers

If you prefer to run the containers separately:

Start the Registry backend:

docker run -it -p 8080:8080 \
  -e apicurio.rest.mutability.artifact-version-content.enabled=true \
  apicurio/apicurio-registry:3.1.0

Start the Registry UI:

docker run -it -p 8888:8080 \
  apicurio/apicurio-registry-ui:3.1.0

Access the UI: Open your browser to http://localhost:8888

Note that this configuration uses in-memory storage, which is perfect for evaluation but not suitable for production use.

What This Means for Studio Users

If you’re currently using Apicurio Studio, here’s what you need to know:

No panic needed: All core editing functionality has been preserved in Registry
Migration path: You can import your existing API designs into Registry as draft artifacts
Improved workflow: You now get both editing capabilities and full schema registry features in one platform
Continued support: By consolidating into Registry, we can provide better support and more frequent updates

Looking Forward

This integration represents a significant step forward for the Apicurio ecosystem. By combining Studio’s editing capabilities with Registry’s robust schema management, we’re creating a more comprehensive solution for API-first development.

We’re committed to continuing to improve and enhance these capabilities. As we move forward, you can expect:

Regular updates and improvements to the editing experience
Better integration between draft editing and Registry’s other features
Continued support for the API design workflows you rely on

We believe this consolidation will allow us to deliver more value to our users while being more sustainable for our development team.

Thank you for your understanding, your patience, and most importantly, your continued use of Apicurio projects. We’re excited about this new direction and look forward to your feedback!

For more information:

Introducing MCP server for Apicurio Registry 3

Jakub Senko — Tue, 08 Jul 2025 12:00:00 +0000

In this blog post, we’re introducing a new way you can interact with Apicurio Registry 3. We’ve created a Model Context Protocol (MCP) server for Apicurio Registry 3, enabling Large Language Models (LLMs) to communicate directly with your Registry instance. You can use it to explore and analyze your content by asking natural language questions, make changes, and generate new schemas.

What is MCP?

Model Context Protocol (MCP) is an open protocol that standardizes how applications provide context to LLMs. MCP provides a standardized method to connect AI models into diverse data sources and provide them with new executable functionality.

The protocol operates on a client-server architecture, where an MCP Host (like Claude Desktop, which we’ll use in a moment) is connecting to one or more MCP Servers. These servers are lightweight programs that expose specific capabilities through the protocol.

There are 3 types of functionality that an MCP server can expose:

Tools
Resources
Prompts

Apicurio Registry MCP server uses tools and prompts, which we’ll explore in a moment.

Apicurio Registry Meets LLMs!

Our new MCP server for Apicurio Registry acts as a wrapper over existing REST API calls and helps LLMs understand the concepts of Apicurio Registry and what kind of operations are available. This means your LLMs can now execute REST API calls against your Apicurio Registry instance.

Let’s see what MCP can do by asking a simple question about the Apicurio Registry server running in the background:

Question: What is the version of Apicurio Registry server?

Reply: I’ll check the version of the Apicurio Registry server for you.

[get_server_info]

The Apicurio Registry server is running version 3.0.8-SNAPSHOT. This is an in-memory deployment built on April 24, 2025. The server is described as a “High performance, runtime registry for schemas and API designs.”

We see that Claude found the correct tool named get_server_info, and called it to get the requested information. The MCP server makes a REST API call to /system/info on behalf of the LLM.

Because tools might perform sensitive or potentially destructive operations, Claude Desktop will ask for confirmation:

Use-case: Generating Content

One of the great uses for generative AI is, well, generating content. Let’s ask Claude to add some content into Apicurio Registry:

Question Create 2 groups, and generate 4 new artifact versions in those groups. The content should be concise and an example of a real use-case.

Answer I’ll create two groups in the Apicurio Registry and generate artifact versions with real-world use cases. Let me start by creating the groups and then add meaningful artifacts to them.

[create_group]
[create_group]

Now I’ll create artifacts and versions in each group. Let me start with the e-commerce group:

[create_artifact]
[create_version]
[create_artifact]
[create_version]

Now let me create artifacts in the messaging-events group:

[create_artifact]
[create_version]
[create_artifact]
[create_version]

Perfect! I’ve successfully created 2 groups and 4 artifact versions with real-world use cases:

Groups Created:

ecommerce-apis Group

Description: API schemas and specifications for e-commerce platform services

Labels: domain: ecommerce, team: platform, environment: production

messaging-events Group

Description: Event schemas for real-time messaging and notifications system

Labels: domain: messaging, team: communications, type: events

Artifact Versions Created:

E-commerce APIs Group:

Product Catalog API v1.0 (OpenAPI)

REST API specification for managing product catalog operations

Includes endpoints for listing and creating products with proper schema definitions

Order Schema v1.0 (JSON Schema)

Validation schema for order data with customer info, items, and order details

Enforces proper order ID patterns and required fields

Messaging Events Group:

User Notification Event v1.0 (Avro)

Schema for real-time user notifications across multiple channels (email, SMS, push, in-app)

Includes priority levels and metadata support for Kafka streaming

Chat Message Event v1.0 (JSON Schema)

Schema for real-time chat messages with WebSocket streaming support

Supports text messages, attachments, and message metadata with encryption

Each artifact version represents a common real-world scenario where schema registries are essential for maintaining data consistency and API compatibility across microservices and event-driven architectures.

Apicurio Registry has a hierarchical organizational structure: Artifacts represent an evolving piece of content (schema). Artifacts are comprised of a list¹ of immutable¹ versions, and groups contain a list of artifacts to provide a high-level organizational unit.

[1] NOTE: Apicurio Registry supports artifact branches and mutable versions for advanced use cases.

Because of the MCP protocol, Claude is able to understand and work with this structure and use additional features, such as labels. We see that first 2 groups are created using the create_group tool, then 4 artifacts using the create_artifact tool, and finally the content using the create_version tool:

Let’s take a look at the Apicurio Registry web console to see the generated content:

We see that Claude can generate OpenAPI, JSON, and Avro schemas, and use meaningful metadata, such as descriptions and labels. Let’s take a closer look at an artifact version, for example the chat-message-event:

And its content:

We have not specified any constraints on the generated data so far. This type of usage is great for providing data for development and testing. Instead of keeping a library of example schemas and manually adding them to a fresh Apicurio Registry instance, we can ask the LLM to do the work for us.

Use-case: Complex Updates

We’ve noticed that Claude automatically added some labels to the groups:

messaging-events:
- domain=messaging
- team=communications
- type=events
ecommerce-apis:
- environment=production
- domain=ecommerce
- team=platform

Let’s take advantage of this to show a more complex workflow. Imagine we have two deployment environments: production and development. We can use labels and groups to reorganize the schemas with the help of AI:

Question Remove the “type” label from the “messaging-events” group, and add an “environment=production” label. Then, for each group used for production, create an equivalent group that can be used for development. Copy the content to the new groups.

Answer I’ll help you manage the groups in your Apicurio Registry. Let me start by examining the current groups and then make the requested changes.

[list_groups]

Now let me get the metadata for the “messaging-events” group to see its current labels:

[get_group_metadata]

Perfect! I can see the current structure. Now I’ll:

Update the “messaging-events” group to remove the “type” label and add “environment=production”

Create development equivalents for both production groups

Copy the artifacts from production groups to their development counterparts

Let’s start by updating the messaging-events group:

[update_group_metadata]

Now let’s create the development equivalent groups. First, let me list the artifacts in the production groups to understand what needs to be copied:

[list_artifacts]
[list_artifacts]

Now I’ll create the development groups:

[create_group]
[create_group]

Now I need to copy the artifacts from the production groups to the development groups. Let me get the content and metadata for each artifact:

[get_version_content]
[get_version_content]
[get_version_content]
[get_version_content]

Now I’ll create the artifacts in the development groups. Let me start by creating the artifacts themselves, then adding the content:

[create_artifact]
[create_version]
[create_artifact]
[create_version]

Now let’s create the ecommerce artifacts in the development group:

[create_artifact]
[create_version]
[create_artifact]
[create_version]

Perfect! Let me verify the final result by listing all groups to show the completed changes:

[list_groups]

Summary

I have successfully completed all the requested changes:

✅ Updated “messaging-events” group:

Removed the “type=events” label

Added “environment=production” label

The group now has labels: domain=messaging, team=communications, environment=production

✅ Created development groups:

messaging-events-dev:

Description: “Event schemas for real-time messaging and notifications system (Development)”

Labels: environment=development, domain=messaging, team=communications

Contains copies of:

user-notification-event (AVRO schema)

chat-message-event (JSON schema)

ecommerce-apis-dev:

Description: “API schemas and specifications for e-commerce platform services (Development)”

Labels: environment=development, domain=ecommerce, team=platform

Contains copies of:

product-catalog-api (OpenAPI spec)

order-schema (JSON schema)

✅ All content copied:

All artifacts from the production groups have been duplicated to their respective development groups with identical content but appropriate development-specific labels and descriptions.

Your Apicurio Registry now has a clear separation between production and development environments, with all production groups properly labeled with environment=production and corresponding development groups with environment=development.

Let’s take a look at the result:

The group’s labels have been updated as requested:

Use-case: Exploration

Let’s take a look at some of the exploration capabilities that the MCP server provides. Apicurio Registry does not currently have a REST endpoint for a full-text search, but the LLM can still look for information about the content by considering artifact, group, and version metadata (e.g. names, descriptions, and labels).

Question Which production artifact contains an event ID field, and what is the type of that field?

Answer: I’ll help you find production artifacts that contain an event ID field. Let me search through your Apicurio Registry to identify artifacts with event ID fields.

[list_groups]

Now let me look at the production groups to find artifacts that might contain event ID fields. I’ll start with the production messaging events group since that’s most likely to contain event-related schemas.

[list_artifacts]

Let me also check the ecommerce-apis production group for any event-related artifacts:

[list_artifacts]

Now let me examine the content of these artifacts to find which ones contain event ID fields. I’ll start with the messaging event artifacts since they’re most likely to have event IDs:

[get_version_content]
[get_version_content]
[get_version_content]

Based on my search through your production artifacts, I found one artifact that contains an event ID field:

Artifact: user-notification-event in the messaging-events group

Event ID Field Details:

Field Name: eventId

Type: string (Avro schema type)

Description: “Unique identifier for the notification event”

This is an Avro schema artifact that defines the structure for user notification events in your messaging system. The eventId field is defined as a required string type that serves as a unique identifier for each notification event.

The other artifacts I examined (chat-message-event and order-schema) do not contain an “eventId” field specifically, though they do have other ID fields like messageId and orderId respectively.

We see that Claude quickly determined the correct artifact, without the ability to perform a full-text search. We have not tested this functionality against an Apicurio Registry with thousands of schemas, but I expect that with a good prompt, the search time could be limited to get a very good result.

Use-case: Analysis

I have cleaned my Apicurio Registry instance and created a new artifact log-message-schema that represents a simplified logging event in some kind of log storage and analysis service. The first version is:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "title": "Log Message Event",  
  "required": [
    "timestamp",
    "level",
    "message"
  ],
  "properties": {
    "timestamp": {
      "type": "string",
      "format": "date-time",
      "description": "ISO 8601 timestamp when the log event occurred."
    },
    "level": {
      "type": "string",
      "enum": [
        "DEBUG",
        "INFO",
        "WARN",
        "ERROR"
      ]
    },
    "message": {
      "type": "string",
      "minLength": 1
    },
    "metadata": {
      "type": "object"
    }
  }
}

In the second version, I’ve added support for tracing information:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "title": "Log Message Event with Tracing",
  "required": [
    "timestamp",
    "level",
    "message"
  ],
  "properties": {
    "timestamp": {
      "type": "string",
      "format": "date-time",
      "description": "ISO 8601 timestamp when the log event occurred."
    },
    "level": {
      "type": "string",
      "enum": [
        "DEBUG",
        "INFO",
        "WARN",
        "ERROR"
      ]
    },
    "message": {
      "type": "string",
      "minLength": 1
    },
    "tracing": {
      "type": "object",
      "properties": {
        "traceId": {
          "type": "string",
          "pattern": "^[a-fA-F0-9]{32}$"
        },
        "spanId": {
          "type": "string",
          "pattern": "^[a-fA-F0-9]{16}$"
        },
        "parentSpanId": {
          "type": "string",
          "pattern": "^[a-fA-F0-9]{16}$"
        }
      }
    },
    "metadata": {
      "type": "object"
    }
  }
}

We can ask Claude to summarize differences between these versions:

Question What are the differences between versions of the “log-message-schema” artifact?

Answer: I’ll help you find the differences between versions of the “log-message-schema” artifact. Let me first search for this artifact in the Apicurio Registry to see what versions exist.

[search_artifacts]

I found the artifact. Now let me list all versions of this schema to see what versions are available:

[list_versions]

I can see there are two versions of the log-message-schema artifact (version 1 and version 2). Let me retrieve the content of both versions to compare them:

[get_version_content]
[get_version_content]

Here are the key differences between the two versions of the “log-message-schema” artifact:

Version 1 → Version 2 Changes

1. Title Updated

Version 1: “Log Message Event”

Version 2: “Log Message Event with Tracing”

2. New Tracing Field Added

Version 2 introduces a new tracing object with distributed tracing capabilities:

traceId: 32-character hexadecimal string for trace identification

spanId: 16-character hexadecimal string for span identification

parentSpanId: 16-character hexadecimal string for parent span identification

3. Schema Structure

Version 1 had 4 properties: timestamp, level, message, and metadata

Version 2 has 5 properties: adds the tracing object while keeping all original fields

4. Backward Compatibility

The changes are backward compatible since:

All required fields remain the same (timestamp, level, message)

The new tracing field is optional

Existing properties have identical definitions

Summary: Version 2 enhances the schema by adding distributed tracing support while maintaining full backward compatibility with version 1. This allows log messages to include trace correlation information for better observability in distributed systems.

This would be even more useful with a more complex schema or even multiple schema versions. Interestingly, the AI automatically included backward compatibility analysis and described the changes to be backward compatible. However, Claude is not correct in this case. Because we’ve added a new field tracing with a specific type object and additionalProperties is true by default, strictly speaking, the schemas are not backward compatible. This is because the previous schema would accept a tracing field with, for example, a string type. It’s important to keep in mind that LLMs are not infallible. Apicurio Registry supports compatibility checking and enforcement that would detect this compatibility issue, but LLMs can still be a nice additional tool to validate your new artifact versions. You might be sure that your system does not contain a message that has used a non-object tracing field in the past, but this is not something that Apicurio Registry can check.

NOTE: There is a fantastic blog post named Understanding JSON Schema compatibility.

Prompt-Driven Interactions

In addition to tools, the MCP protocol allows MCP servers to define prompts, which are reusable templates and workflows that guide the LLM’s interactions and behavior, similar to a “system” prompt. Prompts are useful to:

Provide the LLM with a goal and “rules of engagement” for the conversation, for example:

The assistants goal is to use the MCP server to create a new artifact version in Apicurio Registry.

The assistant must use these general rules during the conversation:

1. The assistant will ask the user if they want to change the content until the user is happy with the result.
   
[...]
   
4. The assistant will always ask the user for confirmation when:   
  - Updating configuration properties
  - Creating group ID or artifact ID

Instruct Claude on the steps it should take:

The assistant will follow these steps during the conversation:

1. The user might want to update an existing artifact or create a new artifact.

Provide additional information to the LLM about the options it can use:

2. The assistant will check if the Apicurio Registry is configured to allow draft artifacts. This feature is configured with a configuration property named "apicurio.rest.mutability.artifact-version-content.enabled".

Instruct the LLM about information it will need to find out, or ask the user:

3. The assistant will determine which artifact types are supported by Apicurio Registry, and tell the options to the user.

Apicurio Registry MCP server offers a prompt for creating a new artifact version called create_new_artifact_version (which the examples came from). Let’s try it out!

First, we open a new chat and add the prompt to the context, by clicking on the + icon, then Add from Apicurio Registry (docker), and then create_new_artifact_version.

Question: Create a new artifact version.

Answer: I’ll help you create a new artifact version in Apicurio Registry. Let me start by checking the current server status and understanding what you’d like to do.

[get_server_info]

Great! The Apicurio Registry server is running (version 3.0.8-SNAPSHOT). Now I need to understand what you’d like to do:

Would you like to:

Update an existing artifact - I’ll retrieve the current content and use it as a starting point

Create a new artifact - I’ll help you generate new content from scratch

Please let me know which option you prefer, and if you’re updating an existing artifact, please provide the group ID and artifact ID.

As expected, Claude follows instructions from the prompt and asked the user whether they want to create or update an artifact. This reflects the expected way of working with content in Apicurio Registry. Compare with the same question, but without the MCP prompt (removed some parts for brevity):

Question: Create a new artifact version.

Answer: I’ll help you create a new artifact version in the Apicurio Registry. First, let me check what artifacts are available and then guide you through the process.

[list_groups]

Let me check the server info to make sure we’re connected:

[get_server_info]

Great! The Apicurio Registry is running. Now let me list the available groups to see what artifacts exist:

[list_groups]

It seems there are no existing groups yet. Let me check the default group for artifacts:

[list_artifacts]

Perfect! I found an existing artifact called “log-message-schema” in the default group. Let me check its current versions:

[list_versions]

Great! The artifact “log-message-schema” already has 2 versions. Let me check the current content of the latest version to understand what we’re working with:

[get_version_content]

Now I can help you create a new version. I’ll create version 3 with some improvements to the schema. Let me create a new version with enhanced features:

[create_version]

Perfect! I’ve successfully created a new artifact version (version 3) for the “log-message-schema” artifact. Here’s what was added in this new version:

New Version Details:

Version: 3

Name: Enhanced Log Message Schema
[…]

Key Enhancements in Version 3:

Added FATAL log level to the existing DEBUG, INFO, WARN, ERROR levels
[…]

The version is now live and ready to use! You can reference it using:
[…]

We see that while the LLM did a good job of gathering data about current Apicurio Registry content and how to use available tools, it created a new version of the only artifact it found (there was a single artifact in the default group used in an earlier example) without asking for confirmation or guidance about the changes that should be made. This is why it’s important to either craft your question carefully or take advantage of the pre-made MCP prompts feature.

Let’s return to our previous conversation:

Question Create a new artifact.

Answer: Perfect! I’ll help you create a new artifact. Let me first check what artifact types are supported by this Apicurio Registry server.

[get_artifact_types]

Great! This registry supports several artifact types:

OPENAPI - OpenAPI/Swagger API specifications

ASYNCAPI - AsyncAPI specifications for event-driven APIs

AVRO - Apache Avro schemas

JSON - JSON Schema definitions

PROTOBUF - Protocol Buffer schemas

GRAPHQL - GraphQL schemas

XML/XSD - XML schemas

WSDL - Web Service definitions

KCONNECT - Kafka Connect schemas

What kind of content would you like to create? Please choose from the options above, or let me know what type of API, schema, or specification you need help with.

For example:

An OpenAPI specification for a REST API

An Avro schema for data serialization

A JSON schema for data validation

An AsyncAPI specification for messaging

We see that Claude has slowed down and is asking clarifying questions. Let’s continue with creating a new Avro schema (removed some parts for brevity):

Question Create an Avro money transfer schema, that can be used in a Kafka topic.

Answer: Excellent choice! I’ll create an Avro schema for money transfers that’s perfect for Kafka topics. Let me first check if the registry supports draft artifacts, which would allow us to iterate on the content.

[get_configuration_property]

The draft artifacts feature is currently disabled. This means once we create an artifact version, we cannot modify its content - we’d need to create new versions for any changes.

Would you like me to enable the draft artifacts feature? This would allow us to:

Create the artifact in DRAFT state

Make iterative changes to the content

Publish it as ENABLED when you’re satisfied

Or would you prefer to work with the current setup where each change creates a new version?

Now, let me create a comprehensive Avro schema for money transfers. Here’s what I’ll include:

[Money Transfer Avro SchemaCode]

I’ve created a comprehensive Avro schema for money transfers! This schema includes:

Key Features:

Unique transfer ID and timestamp for tracking

Precise decimal amounts using Avro’s decimal logical type
[…]

Flexible metadata for additional information

Kafka-Optimized Design:

Uses logical types for proper timestamp and decimal handling
[…]

Now I need some information to create this artifact in the registry:

1. Group ID: What group should this schema belong to?

I could suggest: payments, financial-services, or kafka-schemas

Or you can specify your own

2. Artifact ID: What should we call this artifact?

I could suggest: money-transfer-schema or MoneyTransfer

Or you can specify your own

3. Would you like me to make any changes to the schema content before we create it in the registry?

Here we see how the prompt modifies the behavior of the LLM to make the experience of creating a new artifact version better:

The LLM is aware of the draft artifact versions feature. This feature allows users to temporarily modify an artifact version until the user is happy and transitions it to the ENABLED state, which makes it immutable. This feature is great for this kind of iterative schema creation. As an alternative, either a new artifact version would have to be created after each modification, or the user can iterate on the schema directly in the chat.
The LLM knows about the configuration property it needs to enable, and is required to ask the user for permission before doing so.
The prompt also asks Claude to show any generated content in a separate pane (see image below), which enhances the editing experience.
The LLM is required to ask the user what group ID and artifact ID identifiers to use, but can provide suggestions.

Let’s provide Claude with the required information:

Question Use the draft artifact feature. I want to use group ID “payments” and the artifact ID should be the same as the name of the schema.

Answer: Perfect! I’ll enable the draft artifacts feature and use “payments” as the group ID and “MoneyTransfer” as the artifact ID (matching the schema name).

Let me first enable the draft artifacts feature:

[update_configuration_property]

Great! Draft artifacts are now enabled. Now I’ll create the “payments” group and then create the MoneyTransfer artifact.

[create_group]

Excellent! The “payments” group has been created. Now I’ll create the MoneyTransfer artifact:

[create_artifact]

Perfect! The MoneyTransfer artifact has been created. Now I’ll create the first version with the schema content in DRAFT state:

[create_version]

Excellent! 🎉 The MoneyTransfer artifact has been successfully created in DRAFT state. Here’s what we’ve accomplished:

✅ Created:

Group: payments (for financial schema organization)

Artifact: MoneyTransfer (AVRO type)

Version: 1.0.0 (in DRAFT state)

📋 Schema Details:

Group ID: payments

Artifact ID: MoneyTransfer

Version: 1.0.0

State: DRAFT (can be modified)

Global ID: 4

Content ID: 4

The schema is now stored in the registry and ready for use with Kafka topics. Since it’s in DRAFT state, we can make changes if needed.

Would you like to:

Make any changes to the schema content?

Publish it by moving it to ENABLED state?

Review the current content in the registry?

What would you like to do next?

As expected, the schema appears in our Apicurio Registry instance, but we can still update it:

Let’s make one more edit before we publish the schema - add support for SEPA transfers, and clean up some of the metadata (removed some parts for brevity):

Question Use the draft artifact feature. I want to use group ID “payments” and the artifact ID should be the same as the name of the schema.

Answer: Perfect! I’ll update the schema to support SEPA transfers and clean up the version naming. Let me make these changes:

Add SEPA support by including SEPA-specific fields and transfer types

Remove version numbers from the artifact version name and labels

[Money Transfer Avro Schema]

Now let me update the version metadata to remove the version number from the name and labels:

[update_version_metadata]

Perfect! ✅ I’ve successfully updated the MoneyTransfer schema with the following changes:

🆕 SEPA Support Added:

New Transfer Types:

SEPA_CREDIT - Standard SEPA credit transfers

SEPA_INSTANT - SEPA instant payments

New Account Fields:

iban - International Bank Account Number (for SEPA transfers)

bic - Bank Identifier Code / SWIFT code (for SEPA transfers)

New SEPA-Specific Information:

[…]

🏷️ Cleaned Up Labels & Naming:

Name: Changed from “Money Transfer Schema v1.0.0” to “Money Transfer Schema”

Labels: Removed version numbers, added “sepa-enabled”: “true”

Description: Updated to mention SEPA support

The schema is still in DRAFT state, so we can make additional changes if needed. The schema now fully supports both traditional transfers (domestic, international, wire) and European SEPA transfers with all the required fields.

Would you like to:

Make any other changes to the schema?

Publish it by moving it to ENABLED state?

Review the final schema content?

What would you like to do next?

We see that we can iterate this way until the schema is ready, editing both the content and the metadata. Let’s publish the schema!

Question Publish.

Answer: Perfect! I’ll publish the schema by moving it from DRAFT to ENABLED state.

[update_version_state]

🎉 Successfully Published!

The MoneyTransfer schema has been moved to ENABLED state and is now ready for production use.

📋 Final Artifact Summary:

Group ID: payments

Artifact ID: MoneyTransfer

Version: 1.0.0

State: ENABLED ✅

Type: AVRO

Global ID: 4

🚀 Ready for Kafka Integration:

Your schema is now published and can be used with:

Kafka producers to serialize money transfer events

Kafka consumers to deserialize money transfer events

Schema validation for incoming transfer data

Schema evolution for future updates

💡 Key Features Included:

✅ Standard money transfers (domestic, international, wire)

✅ SEPA transfers (credit & instant)

✅ IBAN/BIC support for European banking

✅ Precise decimal amounts with proper currency handling

✅ Comprehensive account information

✅ Transfer status tracking

✅ Extensible metadata support

The schema is now live in your Apicurio Registry and ready to be consumed by your Kafka applications! 🎯

How to get started

We need a running Apicurio Registry instance. The easiest way is to use Docker or Podman to run Apicurio Registry 3 locally:

# API server
docker run --rm -it -p 8080:8080 quay.io/apicurio/apicurio-registry:latest-snapshot
# UI server
docker run --rm -it -p 8888:8080 quay.io/apicurio/apicurio-registry-ui:latest-snapshot

NOTE: You can replace docker with podman in all commands throughout this guide.

Multiple LLM providers support the MCP protocol, but to follow this blog post and use Claude, register for Claude AI if you don’t have an account yet.
Install the Claude Desktop application. If you are using a system that is not officially supported, like Fedora, unofficial installation options are available. Alternatively, some IDEs such as VS Code support integration with an MCP server.
Run Claude Desktop, and go to File > Settings… > Developer > Edit Config, to open the configuration file (e.g. ~/.config/Claude/claude_desktop_config.json).

Update the configuration file as follows (using Docker or Podman):

{
  "mcpServers": {
    "Apicurio Registry": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--network=host",
        "quay.io/apicurio/apicurio-registry-mcp-server:latest-snapshot"
      ]
    }
  }
}

NOTE: Replace "command": "docker" with "command": "podman" if you prefer to use Podman.

NOTE: The Apicurio Registry MCP Server source code lives in the Apicurio Registry GitHub repository, where you can also find the most recent README file with additional information, such as configuration options and how to build the MCP server from source.

As of writing of this article, Apicurio Registry MCP server currently supports a wide range of operations for managing your Apicurio Registry content:

Artifact Types: Listing supported types.
Configuration Properties: Getting, listing, and updating dynamic configuration properties.
Groups: Creating, getting, listing, searching, and updating groups.
Artifacts: Creating, getting, listing, searching, and updating artifacts and their metadata.
Versions: Creating, getting content, listing, searching, updating metadata, and even updating the state or content of artifact versions.

but there are still more capabilities to add, such as managing content rules, adding comments, import & export, and additional configuration options. We’d love if you decide to take a look at the source code and contribute a PR!

Implementing your own MCP server

In conclusion, I’d like to provide some thoughts and lessons-learned during the implementation of the MCP server.

Safety first - Since MCP tools can provide executable functionality that can be destructive or non-reversible (e.g., data deletion), the author of the MCP server has to be careful when implementing such operations. Claude Desktop shows the user a prompt before executing any tool, but that should only be the first line of defense against the LLM making a mistake. We suggest additional precautions such as:
- Carefully worded tool descriptions and prompts
- Sanity checks by the MCP server (e.g., limit the amount of data that can be deleted)
- Requiring configuration options to enable potentially unsafe operations
- Backups
Apicurio Registry MCP server does not yet support delete operations, but it can update configuration properties. As a precaution, only the configuration property enabling draft artifact versions can be modified, unless the MCP server is configured otherwise.
Paging is hard - The MCP server communicates with Apicurio Registry using the REST API where some of the operations are paged. We were unable to help Claude understand how to work with paged results, so the MCP server has to request large pages by default.
Quarkus is great! - The server has been implemented using the Quarkus MCP Server extension, which made it very easy to define tools and prompts using just a couple of annotations. The most time-consuming part was writing descriptions for the LLM to help it understand Apicurio Registry concepts.
Ask the LLM itself to assist you with writing prompts! Plain and direct sentences, lots of examples, and text formatting worked for us.

Schema-Driven AI Metadata with Apicurio Registry

Carles Arnal Castello — Fri, 02 May 2025 12:00:00 +0000

In this blog post, we will explore how to ensure AI model metadata is valid, consistent, and trustworthy across systems through schema validation. We demonstrate how Apicurio Registry can be used to store and validate machine learning model metadata using JSON Schema, providing a strong foundation for governance in AI workflows.

Schema-Driven AI Metadata: Validating Model metadata with Apicurio Registry

In modern AI systems, model metadata plays a crucial role in enabling discovery, governance, and interoperability. This Model metadata demo defines a standard way to describe machine learning models - what inputs they expect, what outputs they produce, and contextual metadata like authorship, framework, or creation date.

But how do we ensure the metadata is valid and consistent across systems? How do we prevent malformed or incomplete descriptions from entering the pipeline?

This is where schema validation comes into play. In this blog post, we’ll show how to use Apicurio Registry to store and validate model metadata using JSON Schema.

Why Schema Validation for model metadata?

In general, model metadata, is flexible and extensible, however, without validation:

Teams might omit required fields (like modelId or inputSchema).
Different components may interpret metadata inconsistently.
Automation pipelines may break due to incompatible metadata changes.
Schema validation provides a contract—a shared understanding enforced at runtime.

What We’ll Build

✅ A JSON Schema that describes valid model metadata.

🏗 A Java/Quarkus app to validate JSON documents against the schema using Apicurio Registry.

📁 Example valid and invalid model metadata documents.

🌐 An Model metadata REST API server that exposes endpoints to submit, validate, and retrieve model metadata.

Defining the model metadata schema

Here’s a simplified version of the model-context-schema.json:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "ModelContext",
  "type": "object",
  "required": ["modelId", "version", "inputSchema", "outputSchema", "metadata"],
  "properties": {
    "modelId": { "type": "string" },
    "version": { "type": "string" },
    "inputSchema": { "type": "object" },
    "outputSchema": { "type": "object" },
    "metadata": {
      "type": "object",
      "properties": {
        "author": { "type": "string" },
        "created": { "type": "string", "format": "date-time" },
        "framework": { "type": "string" },
        "description": { "type": "string" }
      }
    }
  }
}

Registering the Schema in Apicurio

Start Apicurio Registry locally or use a hosted instance.

You can start the application using:

docker run -it -p 8080:8080 apicurio/apicurio-registry:latest-release

Then register the model metadata schema:

curl -X POST http://localhost:8080/apis/registry/v3/groups/mcp-models/artifacts -H "Content-Type: application/json" -d @./model-context-schema.json

You can also enable schema compatibility checks (e.g., BACKWARD):

curl -X POST http://localhost:8080/apis/registry/v3/groups/mcp-models/artifacts/model-context-schema/rules -H "Content-Type: application/json" -d @./rule.json

Validating Metadata with Java (Quarkus)

To make this even more practical, we extended the demo with a Quarkus-based Model Metadata Server. This server exposes REST endpoints to submit and retrieve model metadata, while validating each submission using Apicurio.

REST Endpoints:

POST /models — Accepts and validates MCP metadata

GET /models — Lists all registered models

GET /models/{id} — Retrieves model metadata by ID

This server could act as a model metadata layer in front of other services—such as the Quarkus Superheroes demo app—where each superhero could have an associated prediction or simulation model validated by the Model Metadata layer.

This shows how schema governance isn’t just a backend concern—it can be integrated into microservice architectures to promote validation, documentation, and trust.

Validating Metadata Submissions

When a user submits a model metadata document, the server validates it against the registered schema. If the document is valid, it’s stored in the model metadata server. If not, an error response is returned.

To start the server, run:

mvn quarkus:dev -Dquarkus.http.port=8081

Then, you can submit a valid model metadata document:

curl -X POST http://localhost:8081/models -H "Content-Type: application/json" -d @sample-model-contexts/valid-context.json

And you’ll see an output like:

{"modelId":"customer-churn-predictor"}

Indicating a successful model submission.

If you try to submit an invalid document:

curl -X POST http://localhost:8081/models -H "Content-Type: application/json" -d @sample-model-contexts/invalid-context.json

You’ll get a validation error response:

{"details":[{"description":"$.artifactUri: integer found, string expected","context":"1029"},{"description":"$.metrics.accuracy: string found, number expected","context":"1029"},{"description":"$: required property 'version' not found","context":"1028"}],"error":"Model validation failed"}

Real-World Applications

This pattern can be used for:

🔁 Validating model metadata in CI/CD before pushing to model registry.
📜 Enforcing internal standards for all model submissions.
🤖 Automated documentation and discovery pipelines.
✅ Ensuring input/output schemas remain compatible over time.

Try It Yourself

We’ve open-sourced the demo on GitHub: https://github.com/carlesarnal/model-metadata

To run it locally:

Start Apicurio docker run -it --rm -p 8080:8080 apicurio/apicurio-registry:latest
Register the schema curl -X POST http://localhost:8080/apis/registry/v3/groups/mcp-models/artifacts -H "Content-Type: application/json" -d @./model-context-schema.json
Start the model metadata server (Quarkus) mvn quarkus:dev
Submit a model document curl -X POST http://localhost:8081/models -H "Content-Type: application/json" -d @sample-model-contexts/valid-context.json

Final Thoughts

Schemas aren’t just for APIs anymore. In the world of AI and MLOps, they’re critical for ensuring that metadata—just like data—is consistent, complete, and trustworthy.

Using Apicurio Registry to manage and validate model metadata gives your AI workflows a strong backbone of governance, validation, and extensibility.

Try the model metadata server integration with your own microservices—or the Quarkus Superheroes demo—and see schema validation in action.

Feel free to reach out or contribute to the demo repo. Happy validating! 🚀

Apicurio Registry 3.x - Evolving Our Serialization and Deserialization Modules

Carles Arnal Castello — Thu, 03 Apr 2025 12:00:00 +0000

Serialization and deserialization (SerDes) are critical functionalities within Apicurio Registry, enabling users to seamlessly manage schema evolution and compatibility. With the release of Apicurio Registry 3.x, we’ve significantly evolved our SerDes modules, focusing on flexibility, reusability, and broader integration.

What’s Changed?

1. Reusable, Generic Codebase

Previously, in Apicurio Registry 2.6.x, our serializers and deserializers were independently organized by format—Avro, JSON Schema, and Protobuf—each housed within its separate module. This resulted in duplicated efforts and increased complexity when adding support for new messaging platforms.

In Apicurio Registry 3.x, we’ve adopted a modular design to improve reuse:

A new generic module now hosts all common, format-agnostic serialization logic.

Specific messaging system implementations (Kafka, Pulsar, NATS) reuse this generic codebase.

This approach significantly reduces duplication, simplifies maintenance, and streamlines contributions from the community.

2. Multi-Platform Support

With Apicurio Registry 2.6.x, our SerDes primarily focused on Kafka support. However, users increasingly require schema management in diverse messaging environments.

With version 3.x, we’ve expanded our support beyond Kafka:

Apache Pulsar: Dedicated SerDes leveraging generic functionality to seamlessly integrate Apicurio Registry with Pulsar.
NATS: Specific Avro support tailored for integration with NATS messaging systems.

This addition enables users to utilize Apicurio Registry in a broader range of architectures.

3. Improved Code Maintainability and Extensibility

By restructuring our SerDes modules, we’ve created a solid foundation for future enhancements. Developers can now:

Easily implement SerDes for additional messaging platforms.
Share common serialization/deserialization logic without duplication.
Enjoy streamlined maintenance due to a centralized and modular architecture.

Directory Structure Comparison

Here’s a concise before-and-after illustration of our SerDes directory structures:

Apicurio Registry 2.6.x

serdes/
├── avro-serde
├── jsonschema-serde
├── protobuf-serde
└── serde-common

Apicurio Registry 3.x
serdes/
├── generic
│   └── serde-common-avro
│   └── serde-common-protobuf
│   └── serde-common-jsonschema
├── kafka
│   └── avro-serde
│   └── protobuf-serde
│   └── jsonschema-serde
├── pulsar
│   └── avro-serde
│   └── protobuf-serde
│   └── jsonschema-serde
└── nats
│   └── avro-serde
│   └── protobuf-serde
│   └── jsonschema-serde

The refined structure clearly separates common logic (generic) from system-specific code (kafka, pulsar, nats), emphasizing the new modular and reusable approach.

Configuration Simplification and Improvements

With Apicurio Registry 3.x, the configuration of serializers and deserializers has also been streamlined, reflecting the modular nature of the updated architecture. Here’s what’s changed:

1. Unified Configuration Properties

In Apicurio Registry 2.6.x, each SerDes module required individual configuration properties, leading to redundancy and complexity. Users needed to repeat similar settings across multiple format-specific serializers and deserializers.

In Apicurio Registry 3.x, configuration properties have been unified and centralized, greatly simplifying setup. For instance:

Common properties (registry URL, schema retrieval strategy, caching, compatibility checks) are now standardized and shared across different formats and platforms.

Messaging platform-specific properties (e.g., Kafka, Pulsar, NATS) are clearly isolated, making it easy to manage configurations tailored to each platform.

2. Improved Clarity and Documentation

With unified and modular configurations, it’s easier to understand and configure the SerDes for your specific needs. Users benefit from:

Clearer property names and structured organization.
Improved documentation that simplifies the onboarding and troubleshooting process.

3. Easier Extensibility

The new configuration model provides a clearer path for adding support for future messaging systems or serialization formats:

Developers extending SerDes functionality benefit from standardized conventions.
Less boilerplate configuration means faster development and lower risk of misconfiguration.

Conclusion

The reorganization of our SerDes modules marks a significant step forward for Apicurio Registry, enabling broader messaging platform support, better maintainability, and increased flexibility. We look forward to seeing how this improved architecture benefits your projects and workflows.

We encourage the community to explore these changes, provide feedback, and contribute. Together, let’s continue evolving Apicurio Registry to meet our shared vision of efficient, robust, and versatile schema management.

Apicurio Registry 3.0 Migration Guide (Configuration)

Carles Arnal Castello — Wed, 02 Apr 2025 12:00:00 +0000

With the release of Apicurio Registry 3.0.0.Final, we’ve significantly redesigned the configuration model to improve consistency and maintainability. If you’re upgrading from Registry 2.x, you’ll need to update your application properties to match the new format.

This guide provides a comprehensive walkthrough of all the changes and offers clear examples to ensure a smooth migration process.

Key Migration Concepts

Before diving into the specifics, here are the fundamental changes you should be aware of:

New namespace: Most configuration keys now use the apicurio. prefix instead of registry.
Consistent naming: Property names follow a more structured and logical pattern
Database configuration: Datasource properties have moved from the Quarkus namespace to Apicurio’s namespace
Better organization: Related properties are now grouped under common prefixes

1. Naming Convention Changes

In Apicurio Registry 3.x, configuration properties now follow a standardized namespace structure under apicurio..

Example Configuration Changes

Registry 2.x	Registry 3.x
`registry.auth.enabled`	`quarkus.oidc.enabled`
`registry.auth.roles.admin`	`apicurio.auth.roles.admin`
`registry.events.kafka.topic`	`apicurio.events.kafka.topic`
`registry.rest.artifact.deletion.enabled`	`apicurio.rest.deletion.artifact.enabled`

Migration Steps

Review all configuration files (application.properties, application.yaml, deployment files)
Replace the registry. prefix with apicurio. for most properties
Note any property name changes beyond just the prefix (like the artifact deletion example above)

2. Authentication & Authorization Configuration

Authentication settings in Registry 3.x maintain the same functionality but follow the new naming convention.

Key Authentication Changes

Registry 2.x	Registry 3.x
`registry.auth.enabled`	`quarkus.oidc.enabled`
`registry.auth.role-source`	`apicurio.auth.role-source`
`registry.auth.roles.admin`	`apicurio.auth.roles.admin`
`registry.auth.roles.developer`	`apicurio.auth.roles.developer`
`registry.auth.roles.readonly`	`apicurio.auth.roles.readonly`
`registry.auth.anonymous-read-access.enabled`	`apicurio.auth.anonymous-read-access.enabled`

OIDC Configuration Updates

Registry 2.x	Registry 3.x
`registry.auth.oidc.client-id`	`quarkus.oidc.client-id`
`registry.auth.oidc.client-secret`	Removed
`registry.auth.oidc.token-endpoint`	`quarkus.oidc.token-path`
`registry.auth.oidc.auth-server-url`	`quarkus.oidc.auth-server-url`

Example: Updating OIDC Authentication Properties

# Old (Registry 2.x)
registry.auth.enabled=true
registry.auth.role-source=token
registry.auth.roles.admin=admin
registry.auth.oidc.client-id=my-client
registry.auth.oidc.client-secret=secret

# New (Registry 3.x)
quarkus.oidc.enabled=true
apicurio.auth.role-source=token
apicurio.auth.roles.admin=admin
quarkus.oidc.client-id=my-client
quarkus.oidc.client-secret=secret

3. Kafka & Eventing Configuration

If you’re using Kafka-based storage or event notification in your Registry installation, update the following configurations:

Event Notification Settings

Registry 2.x	Registry 3.x
`registry.events.kafka.topic`	`apicurio.events.kafka.topic`
`registry.events.kafka.topic-partition`	Removed
`registry.events.ksink`	Removed

KafkaSQL Storage Settings

Registry 2.x	Registry 3.x
`registry.kafkasql.bootstrap.servers`	`apicurio.kafkasql.bootstrap.servers`
`registry.kafkasql.topic`	`apicurio.kafkasql.topic`
`registry.kafkasql.topic.auto-create`	`apicurio.kafkasql.topic.auto-create`

Example: Updating Kafka Event Configuration

# Old (Registry 2.x)
registry.events.kafka.topic=my-registry-events
registry.events.kafka.topic-partition=5

# New (Registry 3.x)
apicurio.events.kafka.topic=my-registry-events

Example: Updating KafkaSQL Storage Configuration

# Old (Registry 2.x)
registry.kafkasql.bootstrap.servers=kafka:9092
registry.kafkasql.topic=my-registry-storage
registry.kafkasql.topic.auto-create=true

# New (Registry 3.x)
apicurio.kafkasql.bootstrap.servers=kafka:9092
apicurio.kafkasql.topic=my-registry-storage
apicurio.kafkasql.topic.auto-create=true

Kafka Security Configuration

If you’re using SASL/SSL for Kafka security:

# Old (Registry 2.x)
registry.kafkasql.security.protocol=SASL_SSL
registry.kafkasql.security.sasl.mechanism=PLAIN
registry.kafkasql.security.sasl.client-id=my-client
registry.kafkasql.security.sasl.client-secret=my-secret
registry.kafkasql.security.sasl.token.endpoint=http://localhost:8090

# New (Registry 3.x)
apicurio.kafkasql.security.protocol=SASL_SSL
apicurio.kafkasql.security.sasl.mechanism=PLAIN
apicurio.kafkasql.security.sasl.client-id=my-client
apicurio.kafkasql.security.sasl.client-secret=my-secret
apicurio.kafkasql.security.sasl.token.endpoint=http://localhost:8090

4. UI & API Configuration

The UI and API settings have been reorganized in Registry 3.x for better clarity.

UI Configuration Updates

Registry 2.x	Registry 3.x
`registry.ui.contextPath`	`apicurio.ui.contextPath`
`registry.ui.features.readOnly`	`apicurio.ui.features.read-only.enabled`
`registry.ui.apisUrl`	`apicurio.ui.config.apiUrl`

UI Authentication Settings

Registry 2.x	Registry 3.x
`registry.ui.auth.type`	Removed, OIDC is always used
`registry.ui.auth.oidc.client-id`	`apicurio.ui.auth.oidc.client-id`
`registry.ui.auth.oidc.redirect-url`	`apicurio.ui.auth.oidc.redirect-uri`
`registry.ui.auth.oidc.scope`	`apicurio.ui.auth.oidc.scope`
`registry.ui.auth.oidc.url`	`apicurio.ui.auth.oidc.url`

API Configuration Changes

Registry 2.x	Registry 3.x
`registry.api.errors.include-stack-in-response`	`apicurio.api.errors.include-stack-in-response`

Example: Updating UI Configuration

# Old (Registry 2.x)
registry.ui.contextPath=/my-ui
registry.ui.features.readOnly=true

# New (Registry 3.x)
apicurio.ui.contextPath=/my-ui
apicurio.ui.features.read-only.enabled=true

Example: Updating UI Authentication

# Old (Registry 2.x)
registry.ui.auth.oidc.client-id=my-client
registry.ui.auth.oidc.redirect-url=https://example.com/callback
registry.ui.auth.oidc.scope=openid profile
registry.ui.auth.oidc.url=https://auth.example.com
registry.ui.auth.type=oidc

# New (Registry 3.x)
apicurio.ui.auth.oidc.client-id=my-client
apicurio.ui.auth.oidc.redirect-uri=https://example.com/callback
apicurio.ui.auth.oidc.scope=openid profile
apicurio.ui.auth.oidc.url=https://auth.example.com
apicurio.ui.auth.type=oidc

5. Storage & Database Configuration

One of the most significant changes in Registry 3.x is the migration of database configuration from the Quarkus namespace to the Apicurio namespace.

SQL Database Configuration

Registry 2.x	Registry 3.x
`quarkus.datasource.jdbc.url`	`apicurio.datasource.url`
`quarkus.datasource.username`	`apicurio.datasource.username`
`quarkus.datasource.password`	`apicurio.datasource.password`
`registry.sql.init`	`apicurio.sql.init`

Storage Type Configuration

Registry 2.x	Registry 3.x
`registry.storage.kind`	`apicurio.storage.kind`

GitOps Storage Configuration

Registry 2.x	Registry 3.x
`registry.gitops.repo.origin.uri`	`apicurio.gitops.repo.origin.uri`
`registry.gitops.workdir`	`apicurio.gitops.workdir`

Example: Updating SQL Database Configuration

# Old (Registry 2.x)
quarkus.datasource.jdbc.url=jdbc:postgresql://db/registry
quarkus.datasource.username=registry
quarkus.datasource.password=registry
registry.sql.init=true

# New (Registry 3.x)
apicurio.datasource.url=jdbc:postgresql://db/registry
apicurio.datasource.username=registry
apicurio.datasource.password=registry
apicurio.sql.init=true

Example: Updating GitOps Storage Configuration

# Old (Registry 2.x)
registry.gitops.repo.origin.uri=https://github.com/my-org/registry-data.git
registry.gitops.workdir=/tmp/registry-gitops

# New (Registry 3.x)
apicurio.gitops.repo.origin.uri=https://github.com/my-org/registry-data.git
apicurio.gitops.workdir=/tmp/registry-gitops

Migration Checklist

Follow these steps to ensure a successful migration:

Inventory: Create a list of all your current Registry 2.x configuration properties
Map: Match each property to its Registry 3.x equivalent using this guide
Update: Modify all configuration files with the new property names
Validate: Check for any remaining or custom properties that might need special handling
Test: Start Registry 3.x with updated properties in a non-production environment first
Monitor: Check logs for any warnings about deprecated or invalid properties
Verify: Ensure all artifacts and configurations load correctly

Common Migration Issues and Solutions

Issue: Missing Properties After Migration

Solution: Some properties have been renamed beyond just changing the prefix. Review the tables in this guide carefully for exact naming changes. It’s also important to check the latest version documentation.

Issue: Authentication Failures After Migration

Solution: Double-check all auth-related properties, especially:

The role mappings under apicurio.auth.roles.*
OIDC configuration parameters, particularly the redirect URI

Issue: Database Connection Problems

Solution: Ensure all database properties have been migrated from the Quarkus namespace to the Apicurio namespace. Check database logs for specific connection errors.

Issue: Kafka Configuration Issues

Solution: Verify that all Kafka-related properties have been updated, especially security settings if you’re using authentication.

Conclusion

Migrating to Apicurio Registry 3.x involves a significant configuration update, but the new structure provides better organization and consistency. By following this guide, you should be able to migrate your Registry instance with minimal downtime.

If you encounter any issues not covered here, please check the official documentation or reach out to the Apicurio community on GitHub.

Happy migrating!

Apicurio Registry 3.0 Migration Guide

Carles Arnal Castello — Sun, 30 Mar 2025 12:00:00 +0000

Apicurio Registry 3.x is our latest and greatest release to date. It has plenty of new features but it also comes with some breaking changes from the previous 2.6.x release. In this post we will explore the process to migrate data from Apicurio Registry 2.6.x to Apicurio Registry 3.x.

Because of the breaking changes between 2.6.x and 3.x, there is no automatic upgrade and instead a migration process is required. But don’t worry! We promise the process is not hard.

Breaking changes

As outlined in the Apicurio Registry 3 announcement, there are three major changes in this release:

Data model changes

There are some significant core data model changes in 3.0. Some of these changes include (but are not limited to):

Artifacts have their own metaData
Artifacts can be empty (have no versions)
Groups can now have labels
User can define and maintain custom branches
“Latest” is now a branch

REST API changes

Full group management, with metaData
Streamlined artifact and version creation
Search for groups at /search/groups
Search for versions at /search/versions
A new dryRun query param replaces the /test endpoint(s)
Brand new Branch API

Kafka storage variant

We have completely rewritten the KafkaSQL layer to improve its stability and to make it easier for us to maintain. This will be the subject of a separate blog post soon, but know that the new implementation should be much easier to keep updated and should result in far fewer unexpected bugs. Spoiler: we are also introducing a “snapshotting” feature that will help address the slow startup times that can result from larger deployments (deployments with lots of artifacts).

Migrating from Apicurio Registry 2.x to 3.x

Apart from the major breaking changes mentioned, Apicurio Registry 3.x introduces several improvements over 2.x, including performance optimizations and better maintainability. If you are currently using Apicurio Registry 2.x, you need to migrate your existing artifacts, rules, and metadata to the new version.

This guide will walk you through the migration process step by step.

Step 1: Export Data from Apicurio Registry 2.x

Before migrating, you need to export your existing data from Apicurio Registry 2.x. The export process generates a ZIP file that contains all artifacts, rules, and metadata.

To export the data, use the admin export API:

curl -X GET http://<registry-v2-url>/apis/registry/v2/admin/export -o registry-export.zip

Step 2: Import the Exported Data into Apicurio Registry 3.x

Once your Registry 3.x instance is running, you can import the exported ZIP file.

Use the following command to send the data to the import API:

curl -X POST http://<registry-v3-url>/apis/registry/v3/admin/import \
-H "Content-Type: application/zip" \
--data-binary @registry-export.zip

What Happens During Import?

All artifacts, rules, and metadata from Registry 2.x are restored. Global are retained. Artifact references are maintained.

Note: This process does not automatically clean up the old Registry 2.x instance. If needed, remove the old instance manually after validating the migration.

Step 3: Validate the Migration

Once the import is complete, it is essential to verify that all data has been migrated successfully.

List the available artifacts in the default group in Registry 3.x:

curl -X GET http://<registry-v3-url>/apis/registry/v3/groups/default/artifacts

Ensure that any previously defined global rules (e.g., compatibility settings) are still in place:

curl -X GET http://<registry-v3-url>/apis/registry/v3/admin/rules

Step 4: Update Client Applications

Once you have Apicurio Registry 3.x running, you should update your client applications to ensure compatibility with the new version.

If you’re using Apicurio Registry Client Libraries, update them to versions compatible with Registry 3.x.

For example, update your Maven dependency:

<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-java-sdk</artifactId>
    <version>3.0.6</version> <!-- Replace with the latest version -->
</dependency>

Final Thoughts

Migrating from Apicurio Registry 2.x to 3.x is a straightforward process when following these steps:

Export data from Registry 2.x.
Start a new Registry 3.x instance.
Import the exported data into Registry 3.x.
Validate the migration to ensure all artifacts and rules were successfully transferred.
Update client applications to work with Registry 3.x.

By completing these steps, you’ll benefit from the improved performance and features of Apicurio Registry 3.x.

For more details, check the official Apicurio documentation or reach out to the Apicurio community.

Apicurio Registry 3.0: Generated API SDKs (by Kiota)

Eric Wittmann & Jakub Senko — Tue, 25 Mar 2025 16:39:00 +0000

In the world of API management, developer experience is paramount. Apicurio Registry 3.0 has taken a significant leap forward by implementing Kiota for automated SDK generation, making it easier than ever for developers to interact with the Registry’s Core REST API across multiple programming languages.

What is Kiota?

Kiota is a groundbreaking SDK generation tool developed by Microsoft that simplifies the process of creating type-safe, lightweight SDKs for REST APIs. Unlike traditional code generation methods, Kiota focuses on creating lean, efficient client libraries that streamline API interactions.

Supported Languages in Apicurio Registry 3.0

With the introduction of Kiota, Apicurio Registry now provides automatic SDK generation for four major programming languages:

Java: Enabling robust, type-safe API interactions for enterprise and Android applications
TypeScript: Providing seamless integration for web and Node.js projects
Python: Supporting data science, automation, and general-purpose application development
Golang: Offering high-performance SDK generation for cloud-native and systems programming

Intuitive API Structure: Mirroring the REST API Specification

One of the most powerful features of Kiota-generated SDKs is their intuitive, hierarchical structure that directly reflects the OpenAPI specification of the Apicurio Registry REST API. This design means that navigating and using the SDK becomes incredibly straightforward and predictable across all supported languages.

The SDK’s structure allows developers to navigate to specific endpoints using a chained method approach that closely mirrors the REST API’s path structure. This approach provides an incredibly natural and consistent way of interacting with the API, regardless of the programming language.

Let’s look at how you might invoke different endpoints across languages:

Searching Artifacts - Invoke GET on the /search/artifacts endpoint

// Java
List<ArtifactMetaData> results = client.search().artifacts().get();

// TypeScript
const results = await client.search.artifacts.get();

// Go
results, err := client.Search().Artifacts().Get(context.Background(), nil)

Creating an Artifact - Invoke POST on the /groups/:groupId/artifacts endpoint

// Java
ArtifactMetaData newArtifact = client.groups().byGroupId("my-group").artifacts().post(artifactData);

// TypeScript
const newArtifact = await client.groups.byGroupId('my-group').artifacts.post(artifactData);

// Go
artifactData := models.NewCreateArtifact()
// Set fields of artifactData...
 newArtifact, err := client.Groups().ByGroupId("my-group").Artifacts().Post(context.Background(), artifactData, nil)

Benefits of Kiota-Generated SDKs

Consistency Across Languages

Kiota ensures that the SDK behavior remains consistent across all supported languages, reducing cognitive overhead for developers working in multi-language environments.

Type Safety

Each generated SDK provides strong type checking, which helps catch potential errors during compile-time and improves overall code quality.

Minimal Dependencies

Kiota-generated SDKs are designed to be lightweight, with minimal external dependencies, resulting in faster build times and smaller application footprints.

Automatic Updates

As the Apicurio Registry API evolves, the Kiota-generated SDKs can be quickly regenerated to reflect the latest API specification.

Getting Started

To use the new Kiota-generated SDKs, simply:

Choose your preferred language
Install the SDK via your language’s package manager
Import the Apicurio Registry client
Start interacting with the Registry API immediately

Example: Java SDK

To use the Apicurio Registry Java SDK, first add the dependency to your Maven pom.xml:

<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-java-sdk</artifactId>
    <version>3.0.6</version>
</dependency>

Then, you can interact with the Registry API:

import io.apicurio.registry.rest.client.RegistryClient;
import io.apicurio.registry.rest.client.models.SystemInfo;
import io.kiota.http.vertx.VertXRequestAdapter;
import io.vertx.core.Vertx;

public class BlogExampleApp {

    public static void main(String[] args) {
        // URL of the Apicurio Registry server.
        String registryUrl = "http://localhost:8080/apis/registry/v3";

        // The java SDK uses a Vertx based Kiota request adapter.
        Vertx vertx = Vertx.vertx();
        VertXRequestAdapter vertXRequestAdapter = new VertXRequestAdapter(vertx);
        vertXRequestAdapter.setBaseUrl(registryUrl);
        
        // Create the registry client.
        RegistryClient client = new RegistryClient(vertXRequestAdapter);

        // Use the client to get the registry's system info.
        SystemInfo systemInfo = client.system().info().get();
        System.out.println(systemInfo.getName());
        System.out.println(systemInfo.getDescription());
        System.out.println("Version " + systemInfo.getVersion());

        // The vertx instance must be closed.
        vertx.close();
    }

}

Example: TypeScript SDK

Install the SDK using npm (or include it in your package.json):

npm install @apicurio/apicurio-registry-sdk

Then use it in your TypeScript or JavaScript project:

import { ApicurioRegistryClient } from "@sdk/lib/generated-client/apicurioRegistryClient.ts";
import { RegistryClientFactory } from "@sdk/lib/sdk";
import { SystemInfo } from "@sdk/lib/generated-client/models";

async function main() {
    // URL of the registry's v3 REST API
    const apiUrl: string = "http://localhost:8080/apis/registry/v3";
    
    // Create an instance of the client
    const client: ApicurioRegistryClient = RegistryClientFactory.createRegistryClient(apiUrl);
    
    // Get and print the system info from the "/system/info" endpoint
    const info: SystemInfo | undefined = await client.system.info.get();
    console.info(info?.name);
    console.info(info?.description);
    console.info("Version " + info?.version);
}

main();

Example: Python SDK

Install the SDK using pip:

pip install apicurioregistrysdk

Use the SDK in your Python project:

import asyncio
from kiota_abstractions.authentication.anonymous_authentication_provider import (
    AnonymousAuthenticationProvider
)
from kiota_http.httpx_request_adapter import HttpxRequestAdapter
from apicurioregistrysdk.client.registry_client import RegistryClient

async def main():
    auth_provider = AnonymousAuthenticationProvider()

    # Create a standard request adapter and set the API URL
    request_adapter = HttpxRequestAdapter(auth_provider)
    request_adapter.base_url = "http://localhost:8080/apis/registry/v3"

    # Create the client
    client = RegistryClient(request_adapter)

    # Get and print the system info from the "/system/info" endpoint
    systemInfo = await client.system.info.get()
    print(systemInfo.name)
    print(systemInfo.description)
    print("Version " + systemInfo.version)

if __name__ == "__main__":
    asyncio.run(main())

Example: Go SDK

To use the Apicurio Registry Go SDK, add the dependency to your go.mod file:

module example

go 1.23.0

toolchain go1.24.1

require (
	github.com/apicurio/apicurio-registry/go-sdk v0.0.0-20241211181742-779f0994a1de // v3.0.6
	github.com/microsoft/kiota-abstractions-go v1.9.1
	github.com/microsoft/kiota-http-go v1.5.1
)

require (
    // Indirect dependencies...
)

Then use it in your Go project:

package main

import (
   "context"
   "fmt"
   registry3 "github.com/apicurio/apicurio-registry/go-sdk/pkg/registryclient-v3"
   "github.com/apicurio/apicurio-registry/go-sdk/pkg/registryclient-v3/models"
   kiotaAuth "github.com/microsoft/kiota-abstractions-go/authentication"
   kiotaHttp "github.com/microsoft/kiota-http-go"
)

func initClient(registryUrl *string) *registry3.ApiClient {
   // Compression is not currently supported by the Apicurio Registry server.
   httpClient := kiotaHttp.GetDefaultClient(
      kiotaHttp.NewRetryHandler(),
      kiotaHttp.NewRedirectHandler(),
      kiotaHttp.NewParametersNameDecodingHandler(),
      kiotaHttp.NewCompressionHandlerWithOptions(kiotaHttp.NewCompressionOptions(false)),
      kiotaHttp.NewUserAgentHandler(),
      kiotaHttp.NewHeadersInspectionHandler(),
   )
   adapter, err := kiotaHttp.NewNetHttpRequestAdapterWithParseNodeFactoryAndSerializationWriterFactoryAndHttpClient(
      &kiotaAuth.AnonymousAuthenticationProvider{}, nil, nil, httpClient)
   if err != nil {
      panic("Unexpected error: " + err.Error())
   }
   adapter.SetBaseUrl(*registryUrl)
   return registry3.NewApiClient(adapter)
}

func getServerInfo(client *registry3.ApiClient) {
   res, err := client.System().Info().Get(context.Background(), nil)
   if res != nil {
      fmt.Printf("Server name: %s\n", *res.GetName())
      fmt.Printf("Server version: %s\n", *res.GetVersion())
   } else {
      handleError(err)
   }
}

func handleError(err error) {
   if err == nil {
      panic("No error value")
   }
   if details, ok := err.(*models.ProblemDetails); ok {
      fmt.Printf("Server error: %s\n", *details.GetDetail())
   } else {
      fmt.Printf("Unknown error: %s\n", err.Error())
   }
}

func main() {
   registryUrl := "http://localhost:8080/apis/registry/v3"
   client := initClient(&registryUrl)
   getServerInfo(client)
}

See the full code of the example in our Apicurio Registry repository.

Conclusion

Apicurio Registry 3.0’s adoption of Kiota represents a significant advancement in API SDK generation. By providing consistent, type-safe, and easily maintainable SDKs across Java, TypeScript, Python, and Golang, the project has dramatically improved the developer experience.

Resources

Stay tuned for more updates and improvements in Apicurio Registry!

Core REST API Changes in Apicurio Registry 3.0

Eric Wittmann — Mon, 24 Mar 2025 14:57:00 +0000

The release of Apicurio Registry 3.0 marks a significant milestone in our journey to provide the best possible Registry experience. With this major version update, we’ve taken the opportunity to implement several breaking changes to the Core REST API that both add powerful new functionality and streamline existing features.

Why These Changes Matter

Before diving into the specific changes, it’s worth understanding the philosophy behind this update. Our goal was to create a more consistent, logical, and powerful API surface that better aligns with how users actually interact with the registry. These changes reflect feedback from the community and our own experience supporting users and Red Hat customers over time.

You can find the new core REST API here:

https://registry.example.org/apis/registry/v3

Key Architectural Changes

Separation of Artifact and Version Concepts

Perhaps the most fundamental change is the clear separation between artifacts and their versions:

Artifacts now have their own metadata, separate from version metadata
Artifacts can exist without any versions (empty artifacts)
Content is now exclusively managed through version endpoints
Only versions have state (artifacts no longer have state)

This distinction allows for more granular control and creates a clearer mental model of the registry structure.

Empty Artifacts: Configure First, Create Later

One of the new capabilities in Apicurio Registry 3.0 is the ability to create empty artifacts without immediately providing content for a first version. This seemingly small change enables important new workflows, particularly around governance and validation.

With empty artifacts, you can now:

Configure content rules before any versions exist
Set up metadata, labels, and ownership for an artifact before content development begins
Establish the artifact’s place in your organization’s taxonomy in advance
Create placeholder artifacts as part of planning activities

This feature is especially valuable for teams implementing strict governance processes. For example, you can now:

Create an empty artifact with appropriate metadata
Configure validation rules that enforce your organization’s standards
Set up branch structures in anticipation of development
Only then begin creating versions with content that must comply with the pre-established rules

This “configure first, create later” approach helps prevent non-compliant content from entering the registry in the first place, rather than discovering compliance issues after content has been created.

Hierarchical Rules System

Apicurio Registry 3.0 expands the hierarchical rules system with the implementation of the /groups/:groupId/rules endpoint. Rules can now be configured at three distinct levels in the registry, creating a powerful inheritance model for content governance:

Global Rules - Apply to all artifacts across the entire registry
Group Rules - Apply to all artifacts within a specific group
Artifact Rules - Apply to a specific artifact only

Rules are evaluated with a clear precedence order: Artifact rules override Group rules, which override Global rules. This hierarchy allows for a sophisticated governance approach where broad standards can be established at the global level, group-specific requirements added at the group level, and artifact-specific customizations made at the individual artifact level.

For example, you might implement:

Global rules that enforce basic structural validation across all artifacts
Group rules that enforce naming conventions specific to a business unit
Artifact rules that implement specialized compatibility checks for critical APIs

This multi-tiered approach provides both consistency where needed and flexibility where required. It significantly reduces rule management overhead by allowing you to define common rules at higher levels while maintaining the ability to customize validation for specific artifacts.

The Group Rules API follows a familiar pattern using standard HTTP methods:

GET /groups/:groupId/rules - List all rules for a group
POST /groups/:groupId/rules - Create a new rule for a group
DELETE /groups/:groupId/rules - Delete all rules for a group

Introduction of Branches

A major new feature is the addition of branches, which provide:

Multiple development paths for artifacts
More flexible versioning strategies
Improved organization for complex artifact evolution

New branch endpoints provide comprehensive management capabilities including creation, updating metadata, managing versions within branches, and branch deletion.

Detailed API Changes

Endpoint Reorganization

Several endpoints have been moved or renamed for consistency:

/admin/artifactsType → /admin/config/artifactTypes
Version metadata management now happens at /groups/:groupId/artifacts/:artifactId/versions/:versionExpression (previously, this endpoint managed content)
Version content is now accessed at /groups/:groupId/artifacts/:artifactId/versions/:versionExpression/content

Enhanced Group Management

Groups have received significant enhancements:

Added support for labels on groups
New PUT operation to update group metadata (description, labels, owner)
Implemented group rules endpoints at /groups/:groupId/rules
Added pagination to group-related endpoints
Changed “createdBy” to “owner” for improved clarity

Metadata Simplification

We’ve simplified the metadata structure throughout the API:

Combined “labels” and “properties” into a single “labels” concept (name/value pairs)
Changed “type” to “ruleType” for consistency with “artifactType”
Renamed parameters for clarity where appropriate

Improved Search Capabilities

Search functionality has been enhanced:

Added GET /search/groups for searching groups with filtering and pagination
Added GET /search/versions to search/filter all versions
Added POST /search/versions to search by version content

Content Management Changes

Content handling has been significantly revised:

Removed POST /groups/:groupId/artifacts for raw artifact content
Changed content-type requirements for artifact creation
Added a dryRun query parameter to several endpoints (replaces the old “/test” endpoint)
All content creation operations now require an explicit Content-Type header

Removed Endpoints

Several endpoints have been removed as their functionality has been integrated elsewhere:

/groups/:groupId/artifacts/:artifactId/meta (metadata is now managed at the artifact endpoint)
/groups/:groupId/artifacts/:artifactId/state (state now only exists on versions)
/groups/:groupId/artifacts/:artifactId/owner (owner now part of editable metadata)
/groups/:groupId/artifacts/:artifactId/test (replaced by dryRun query parameter)

Migration Considerations

If you’re upgrading from Apicurio Registry 2.x to 3.0, you’ll need to update your client applications to accommodate these changes. Here are some key migration points:

Update your client code to use the new endpoint paths
Revise your content management workflow to use the new version-focused approach
Take advantage of the new branch feature if you need multiple development paths
Consolidate your use of properties and labels into the unified labels concept
Update any automation that depends on artifact state (now only versions have state)

Core API Backwards Compatibility

It’s worth noting that, while we encourage users to migrate to v3 of the API as soon as possible, we will continue to support the v2 API (now deprecated) - presumably until v4!

What that means is that the old v2 API endpoint should still work:

https://registry.example.org/apis/registry/v2

What’s Next?

These changes lay the groundwork for even more powerful features in future releases. The clearer separation of artifacts, versions, and branches opens up possibilities for more sophisticated governance, improved collaboration workflows, and enhanced content validation.

We’re excited to see how the community embraces these changes and builds on them. As always, we welcome your feedback and contributions to make Apicurio Registry even better.

Semantic Versioning Support in Apicurio Registry 3.0

Eric Wittmann — Fri, 21 Mar 2025 16:17:00 +0000

Apicurio Registry 3.0 introduces a powerful new feature: support for automatic semantic versioning branches. This enhancement brings automatic branch management based on semantic versioning patterns, making it easier to organize, track, and access different versions of your schema artifacts. Let’s explore how this feature works and why it’s valuable for API governance.

Understanding Semantic Versioning

Before diving into Apicurio’s implementation, let’s quickly review semantic versioning. Often abbreviated as “SemVer,” semantic versioning follows a pattern of MAJOR.MINOR.PATCH:

MAJOR version increases represent incompatible API changes
MINOR version increases add functionality in a backward-compatible manner
PATCH version increases represent bug fixes

This structured approach to versioning makes it clear to users how a new version might impact their existing implementations.

How Semantic Versioning Works in Apicurio Registry 3.0

Apicurio Registry now automatically creates and manages branch hierarchies based on semantic version numbers. When you add artifacts with semantic version numbers, the registry intelligently organizes them into appropriate branches.

Automatic Branch Creation

Whenever versions are created with semantic versioning patterns, Apicurio Registry automatically creates corresponding branches at different levels of specificity:

Major version branches (e.g., 1.x, 2.x, 3.x)
Minor version branches (e.g., 1.0.x, 1.1.x, 2.5.x)
A latest branch that always points to the most recent version

For example, when creating versions like 1.0.0, 1.0.1, and 2.5.2, the registry automatically creates branches such as 1.x, 2.x, 1.0.x, and 2.5.x without any additional configuration.

Branch Behavior and Characteristics

The test code reveals several important aspects of these semantic versioning branches:

System-defined branches: These branches are created and managed by the system, not manually created by users.
Protection from deletion: System-generated branches cannot be deleted, ensuring the integrity of the version hierarchy.
Automatic content management: Each branch automatically contains the relevant versions that match its pattern.
Latest version awareness: Each branch knows which version is the latest within its scope.

Example Workflow

Let’s walk through an example scenario based on the test case:

Create an artifact with version: 1.0.0

Add subsequent versions: 1.0.1, 1.0.2, 1.0.3, 1.1.0, 1.1.1, 2.5.1, 2.5.2, 3.0.0

After these operations, Apicurio Registry automatically creates these branches:

latest (containing all 9 versions, with 3.0.0 as the latest)
1.x (containing all versions starting with 1., with 1.1.1 as the latest)
1.0.x (containing all 1.0. versions, with 1.0.3 as the latest)
1.1.x (containing all 1.1. versions, with 1.1.1 as the latest)
2.x (containing all versions starting with 2., with 2.5.2 as the latest)
2.5.x (containing all 2.5. versions, with 2.5.2 as the latest)
3.x (containing all versions starting with 3., with 3.0.0 as the latest)
3.0.x (containing all 3.0. versions, with 3.0.0 as the latest)

Configuring Semantic Versioning in Apicurio Registry

It’s important to note that semantic versioning support in Apicurio Registry 3.0 is disabled by default. To take advantage of this feature, you need to explicitly enable it through configuration settings.

Enabling Semantic Versioning

You can enable semantic versioning in Apicurio Registry 3.0 by setting the appropriate environment variable.

Environment Variable

When deploying Apicurio Registry, set the following environment variable:

APICURIO_SEMVER_BRANCHING_ENABLED=true

Configuring via Docker

docker run -p 8080:8080 -e APICURIO_SEMVER_BRANCHING_ENABLED=true apicurio/apicurio-registry:latest-release

Verifying the Configuration

You can verify that semantic versioning is enabled by checking the registry’s API documentation or by creating an artifact with a semantic version and confirming that the expected branches are automatically created.

Benefits of Semantic Versioning in Apicurio Registry

The implementation of semantic versioning in Apicurio Registry 3.0 brings several significant benefits:

Simplified Version Navigation With automatically created branches, you can easily access specific version families. Need the latest stable version in the 1.x series? Simply reference the 1.x branch. This eliminates the need to manually search through all versions to find the one you need.
Predictable Version Management The semantic versioning pattern creates a predictable structure for managing artifact versions. API consumers can rely on established conventions for understanding compatibility implications when upgrading to newer versions.
Improved API Governance The automatic branch management supports better API governance by making it easier to maintain multiple version lines simultaneously. Teams can easily support older major versions for backward compatibility while continuing to develop newer versions.
Reduced Manual Configuration Since branches are automatically created and managed by the system based on semantic version patterns, users don’t need to manually create and maintain branch structures.

Using Semantic Versioning in Practice

To use semantic versioning in Apicurio Registry 3.0 with curl commands, follow these examples that demonstrate how to create and work with semantically versioned artifacts:

Create a Group

First, let’s create a group to organize our artifacts:

curl -X POST "http://localhost:8080/apis/registry/v3/groups" \
     -H "Content-Type: application/json" \
     -d '{"groupId": "my-semver-group"}'

Creating an Artifact with Initial Semantic Version

Now, let’s create an artifact with an initial semantic version (1.0.0):

curl -X POST "http://localhost:8080/apis/registry/v3/groups/my-semver-group/artifacts" \
     -H "Content-Type: application/json" \
     -d '{
       "artifactId": "my-semver-artifact",
       "artifactType": "JSON",
       "firstVersion": {
         "version": "1.0.0",
         "content": {
            "content": "{\"type\":\"record\",\"name\":\"ExampleType\",\"fields\":[{\"name\":\"foo\",\"type\":\"string\"}]}",
            "contentType": "application/json"
         }
       }
     }'

This creates the artifact with an explicitly specified semantic version 1.0.0 rather than using the default version numbering scheme for a new artifact version, which in this case would result in 1.

Adding Additional Semantic Versions

To add a new version to the artifact following the semantic versioning pattern:

curl -X POST "http://localhost:8080/apis/registry/v3/groups/my-semver-group/artifacts/my-semver-artifact/versions" \
     -H "Content-Type: application/json" \
     -d '{
       "version": "1.0.1",
       "content": {
         "content": "{\"type\":\"record\",\"name\":\"ExampleType\",\"fields\":[{\"name\":\"foo-2\",\"type\":\"string\"}]}",
         "contentType": "application/json"
       }
     }'

You could then continue to add more versions with additional semantic versions:

# Adding version 1.1.0
curl -X POST "http://localhost:8080/apis/registry/v3/groups/my-semver-group/artifacts/my-semver-artifact/versions" \
     -H "Content-Type: application/json" \
     -d '{
       "version": "1.1.0",
       "content": {
         "content": "{\"type\":\"record\",\"name\":\"ExampleType\",\"fields\":[{\"name\":\"foo-3\",\"type\":\"string\"}]}",
         "contentType": "application/json"
       }
     }'

# Adding version 2.0.0
curl -X POST "http://localhost:8080/apis/registry/v3/groups/my-semver-group/artifacts/my-semver-artifact/versions" \
     -H "Content-Type: application/json" \
     -d '{
       "version": "2.0.0",
       "content": {
         "content": "{\"type\":\"record\",\"name\":\"ExampleType\",\"fields\":[{\"name\":\"foo-4\",\"type\":\"string\"}]}",
         "contentType": "application/json"
       }
     }'

Accessing Branches

Once you’ve created several versions with semantic versioning, you can access the automatically generated branches:

echo "List all branches for the artifact"
curl "http://localhost:8080/apis/registry/v3/groups/my-semver-group/artifacts/my-semver-artifact/branches" | jq

The response will include all automatically created branches based on the semantic versioning structure, such as latest, 1.x, 1.0.x, 1.1.x, and 2.x.

Accessing Versions Within a Specific Branch

To retrieve all versions within a specific branch, such as the 1.x branch:

curl "http://localhost:8080/apis/registry/v3/groups/my-semver-group/artifacts/my-semver-artifact/branches/1.x/versions" | jq

This will return all versions that match the 1.x pattern, with the latest version in that branch listed first.

Getting the Latest Version in a Branch

To access the latest version’s content or metadata from a specific branch:

Metadata

curl "http://localhost:8080/apis/registry/v3/groups/my-semver-group/artifacts/my-semver-artifact/versions/branch=1.x" | jq

This will return the metadata for version 1.1.0, because that is the most recent version on the 1.x branch.

Content

curl "http://localhost:8080/apis/registry/v3/groups/my-semver-group/artifacts/my-semver-artifact/versions/branch=1.x/content" | jq

This will return the content for version 1.1.0, because that is the most recent version on the 1.x branch.

Conclusion

Semantic versioning support in Apicurio Registry 3.0 represents a significant enhancement for managing schema evolution. The automatic creation and management of version-based branches simplifies navigation, improves governance, and provides a more intuitive way to work with multiple versions of your artifacts.

By leveraging this feature, teams can more effectively manage their API lifecycle, ensuring that consumers can reliably access the appropriate versions while maintaining clear expectations around compatibility and change.

If you’re working with evolving schemas or APIs, Apicurio Registry’s semantic versioning support provides a powerful framework for maintaining order in what could otherwise become a complex version management challenge. Remember to explicitly enable this feature in your deployment configuration to take advantage of its benefits.

Bonus Content

There are two additional optional features that can be enabled to improve the functionality of the semantic versioning support in Apicurio Registry 3.0.

Version Validation

Apicurio Registry can validate that every artifact version you add conforms to the SemVer format, or else it will be rejected. This feature can be enabled by setting the following environment variable:

APICURIO_SEMVER_VALIDATION_ENABLED=true

Version Coercion

Note: wonderful name for a feature. Just say that out loud and delight in it.

Apicurio Registry can be configured to automatically coerce invalid versions into valid Semantic Versioning 2.0 compatible versions (when possible). For example, when this feature is enabled and you create a new version 2.0, the version will be coerced into 2.0.0. This feature can be enabled by setting the following environment variable:

APICURIO_SEMVER_BRANCHING_COERCE=true

Future Plans

Automatic creation of semantic versioning branches is a great start to our overall planned support for semantic versioning. We will continue exploring this space in the future to push semver support even further. For example, it might be nice if the Compatibility Rule understood semver so that its concept of compatibility was consistent with the version of a schema or API being added. In other words, the compatibility rules might differ if you are adding version 1.0.7 (a patch release) vs. version 4.0.0 (a major release).

Exporting from Confluent Schema Registry to Apicurio Registry

Eric Wittmann — Thu, 20 Mar 2025 13:22:00 +0000

If you’re using Confluent Schema Registry and looking to migrate your schemas to Apicurio Registry, you’re in luck! The Confluent Export Tool provides a simple way to export all schemas from a Confluent Schema Registry in a format that is directly importable into Apicurio Registry.

In this blog post, we’ll walk through what the tool does, how to use it, and some key considerations for a smooth migration.

What is the Confluent Export Tool?

The Confluent Export Tool is a utility designed to extract all content from a Confluent Schema Registry instance and save it into a format that is compatible with Apicurio Registry. The exported data can then be seamlessly imported into Apicurio using its import capabilities.

Key Features:

Full Export: Captures all schemas from Confluent Schema Registry.
Apicurio Registry Compatible Format: Structures data to match Apicurio Registry’s import requirements.
Easy to Use: A simple command-line tool requiring minimal setup.

How to Use the Confluent Export Tool

Prerequisites

Before using the tool, ensure you have:

A running Confluent Schema Registry instance.
Java installed (Java 17 or later recommended).
Network access to the Confluent Schema Registry API.

Steps to Export Data

Clone the Apicurio Registry Repository:

git clone https://github.com/Apicurio/apicurio-registry.git
cd apicurio-registry

Build the Tool: Build the project using Maven:

mvn clean package -DskipTests  -pl utils/exportConfluent -am

Run the Export Tool:
```
cd utils/exportConfluent
java -jar target/apicurio-registry-utils-exportConfluent-*-runner.jar <schema-registry-url>
```
Replace <schema-registry-url> with the actual URL of your Confluent Schema Registry (e.g., http://localhost:8081).

When complete, a new file called confluent-schema-registry-export.zip will be created in the current working directory.

Importing into Apicurio Registry

Once you have the exported ZIP file, you can import it into Apicurio Registry using the Apicurio Import API:

curl -X POST "http://<registry-url>/apis/registry/v3/admin/import" \
  -H "Accept: application/json" -H "Content-Type: application/zip" \
  --data-binary @confluent-schema-registry-export.zip

Alternatively, you could log into the Apicurio Registry UI and import the file using the “Import from .ZIP” option found on the Explore tab.

Considerations and Best Practices

Backup Your Data: Before migration, always make a backup of your Confluent Schema Registry.
Confluent Authentication: If your Confluent Schema Registry requires authentication, configure appropriate credentials for the export tool.
Apicurio Authentication: If your Apicurio Registry instance requires authentication, configure appropriate credentials in the curl command, or simply use the Registry UI.

Conclusion

The Confluent Export Tool simplifies migration from Confluent Schema Registry to Apicurio Registry, making it easier for organizations to adopt Apicurio’s open-source schema management platform. By following the steps above, you can efficiently transition your schemas and continue managing them in Apicurio.

For more details, check out the official GitHub repository and start your migration today!

Red Hat middleware team transitioning to IBM

Eric Wittmann — Mon, 03 Mar 2025 15:00:00 +0000

Hey everyone - I would like to provide an update to the Apicurio community regarding the recent announcement from Red Hat:

Evolving our middleware strategy

In this post I will try to provide some context and explain how this change might impact us.

Evolving middleware strategy at IBM/Red Hat (tldr)

As some of you may know, the Apicurio community is sponsored by Red Hat. Most of the core contributors are Red Hat employees, starting from long before we were acquired by IBM back in 2019.

(I myself have been a Red Hat employee since 2012)

Red Hat has multiple engineering teams, including (but not limited to):

Operating System (Red Hat Enterprise Linux)
Kubernetes (Red Hat OpenShift)
Ansible (Red Hat Ansible Automation Platform)
Middleware (JBoss, Quarkus, OpenJDK)

The team responsible for the middleware projects (such as JBoss, Quarkus, OpenJDK, Kafka, Camel) is being transitioned from Red Hat to IBM.

Why the change?

Both Red Hat and IBM have made investments in Java middleware. This transition is an opportunity to join forces, especially at a time when much attention is being directed toward AI and other strategic initiatives.

Given its more limited resources, Red Hat has not been able to focus as much on expanding our middleware products as much as they deserve. Moving these teams to IBM will allow us to refocus, grow, and collaborate more effectively.

What does this mean for Apicurio?

We don’t expect any changes, and I’m guessing that if I didn’t post anything here most of the Apicurio community would never know about the transition at all. I am sharing this because I believe transparency is important.

The good news for Apicurio is that there are no competing IBM products. In fact, Apicurio Registry is already included as part of the IBM Event Streams product. My expectation is that Apicurio Registry will continue as normal. My hope is that we can revitalize the Apicurio Studio project by finding it a new home within IBM’s broader portfolio.

What’s next?

We’re working to release Red Hat build of Apicurio Registry 3.0 GA before the transition. This means a 3.0.7 release of apicurio-registry upstream. After that we’ll continue evolving both Registry and Studio in exciting new ways!

We highly value our community’s contributions. If you have any questions or concerns, please chat with us on Zulip or open a discussion on GitHub.

Cheers!

A Practical Step-by-Step Guide to Securing Your Apicurio Registry with Azure Entra ID

Abel BOUCHAMA — Mon, 23 Sep 2024 12:00:00 +0000

Hey everyone, in our previous blog post, we covered the authentication process using Azure Entra ID (formerly Azure AD).

In this technical guide, we will provide a step-by-step approach to securing both the Apicurio Registry UI and API. By the end, you’ll be ready to protect your Apicurio Registry using Azure Entra ID effectively. Let’s dive in!.

Core Features

In this blog post, we will cover the following:

Step-by-Step Authentication/Authorization Configuration Entra-ID to secure ApiCurio Registry - A detailed walkthrough for securing the Apicurio Registry UI and API using Azure Entra ID.

Part I: Access to the UI secured with Azure Entra ID

1. Create subscription for service registry operator:

oc create -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/service-registry-operator.openshift-operators: ""
  name: service-registry-operator
  namespace: openshift-operators
spec:
  channel: 2.x
  installPlanApproval: Automatic
  name: service-registry-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: service-registry-operator.v2.6.1
EOF

2. Install ApiCurio Registry 2.x

2.1 - Install Postgres DB

oc new-project registry
oc new-app -e POSTGRESQL_USER=psql -e POSTGRESQL_PASSWORD=password -e POSTGRESQL_DATABASE=apicurio postgresql

Check that the postgres is running 1/1:

oc get pods -n registry

2.2 Create Simple ApiCurio Registry NONE Secure:

oc apply -f - <<EOF
apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry
metadata:
  name: apicurioregistry-psql
spec:
  configuration:
    logLevel: DEBUG
    registryLogLevel: DEBUG
    security:
      https:
        disableHttp: true
    persistence: "sql"
    sql:
      dataSource:
        url: "jdbc:postgresql://postgresql:5432/apicurio"
        userName: "psql"
        password: "password"
EOF

output:

apicurioregistry.registry.apicur.io/apicurioregistry-psql created

oc get pods
NAME                                                READY   STATUS    RESTARTS   AGE
apicurioregistry-psql-deployment-67996897df-8426x   1/1     Running   0          89s
postgresql-67b7c66db-wcfck                          1/1     Running   0          6m37s

 oc get svc
NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
apicurioregistry-psql-service   ClusterIP   172.30.216.0    <none>        8080/TCP   116s
postgresql                      ClusterIP   172.30.209.37   <none>        5432/TCP   7m3s

oc get routes
NAME                                  HOST/PORT                                                                                 PATH   SERVICES                        PORT   TERMINATION   WILDCARD
apicurioregistry-psql-ingress-hnz2f   apicurioregistry-psql.registry.router-default.apps.com   /      apicurioregistry-psql-service   http                 None

2.3 Create Route for HTTPS (outside connection) required for the Azure Entra ID (AD) for the redirect URL:

NOTE: Change the host for your Apicurio Registry Console. you can get it from oc get route.

oc create -f - <<EOF
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: apicurioregistry-psql
  labels:
    app: apicurioregistry-psql
spec:
  host: apicurioregistry-psql.registry.router-default.apps.com
  to:
    kind: Service
    name: apicurioregistry-psql-service
    weight: 100
  port:
    targetPort: 8080
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None
EOF

output:

route.route.openshift.io/apicurioregistry-psql created

3. Secure the ApiCurio Registry with Azure Entra ID:

Login to https://portal.azure.com/
Follow the steps in the blog (Chapter - Azure Entra ID Configuration): Blog https://www.apicur.io/blog/2023/07/13/registry-azure-ad

Prepare the values for the following env from the app registration apicurio-registry-example in Azure Entra ID:

Auth OIDC settings:

REGISTRY_AUTH_ENABLED=true
KEYCLOAK_API_CLIENT_ID=459569e9-c5f7-410a-a6e7-8db28d7e3647 #Azure AD > Admin > App registrations > Your app > Application (client) ID
REGISTRY_UI_AUTH_TYPE=oidc
REGISTRY_AUTH_URL_CONFIGURED=https://login.microsoftonline.com/6f9ef45b-456d-49e3-b5ba-1f6fe4c0fb78/v2.0 #Azure AD > Admin > App registrations > Your app > Directory (tenant) ID
REGISTRY_OIDC_UI_CLIENT_ID=459569e9-c5f7-410a-a6e7-8db28d7e3647 #Azure AD > Admin > App registrations > Your app > Application (client) ID

Redirect Settings:

Change the host for your Apicurio Registry deployment (oc get route)

CORS_ALLOWED_ORIGINS=https://apicurioregistry-psql.registry.router-default.apps.com #The host for your Apicurio Registry deployment
REGISTRY_OIDC_UI_REDIRECT_URL=https://apicurioregistry-psql.registry.router-default.apps.com/ui/ #The host for your Apicurio Registry console

Authorization settings:

ROLE_BASED_AUTHZ_ENABLED=true
QUARKUS_OIDC_ROLES_ROLE_CLAIM_PATH=roles

Apply the new env var settings to the existing CR apicurioregistry-psql:

oc apply -f - <<EOF
apiVersion: registry.apicur.io/v1
kind: ApicurioRegistry
metadata:
  name: apicurioregistry-psql
spec:
  configuration:
    env:
      - name: REGISTRY_AUTH_ENABLED
        value: 'true'
      - name: REGISTRY_AUTH_RBAC_ENABLED
        value: 'true'
      - name: KEYCLOAK_API_CLIENT_ID
        value: '1fc88f64-2303-42f6-b9df-609edee94efe'
      - name: REGISTRY_UI_AUTH_TYPE
        value: 'oidc'
      - name: REGISTRY_AUTH_URL_CONFIGURED
        value: 'https://login.microsoftonline.com/64dc69e4-d083-49fc-9569-ebece1dd1408/v2.0'
      - name: REGISTRY_OIDC_UI_CLIENT_ID
        value: '1fc88f64-2303-42f6-b9df-609edee94efe'
      - name: CORS_ALLOWED_ORIGINS
        value: 'https://apicurioregistry-psql.registry.router-default.apps.com'
      - name: REGISTRY_OIDC_UI_REDIRECT_URL
        value: 'https://apicurioregistry-psql.registry.router-default.apps.com/ui/'
      - name: ROLE_BASED_AUTHZ_ENABLED
        value: 'true'
      - name: QUARKUS_OIDC_ROLES_ROLE_CLAIM_PATH
        value: 'roles'
      - name: registry.ccompat.max-subjects
        value: '2000'
      - name: CLIENT_CREDENTIALS_BASIC_AUTH_ENABLED
        value: 'true'
      - name: AUTH_ENABLED
        value: 'true'
      - name: CORS_ALLOWED_METHODS
        value: 'GET,PUT,POST,PATCH,DELETE,OPTIONS'
      - name: CORS_ALLOWED_HEADERS
        value: >-
          x-registry-name,x-registry-name-encoded,x-registry-description,x-registry-description-encoded,x-registry-version,x-registry-artifactid,x-registry-artifacttype,x-registry-hash-algorithm,x-registry-content-hash,access-control-request-method,access-control-allow-credentials,access-control-allow-origin,access-control-allow-headers,authorization,content-type
    logLevel: DEBUG
    registryLogLevel: DEBUG
    security:
      https:
        disableHttp: true
    persistence: "sql"
    sql:
      dataSource:
        url: "jdbc:postgresql://postgresql:5432/apicurio"
        userName: "psql"
        password: "password"
EOF

Part II: Machine to machine communication (Client Credentials Grant)

Apart from having users accessing the UI, it is quite frequent to have the requirement of doing machine to machine authentication, especially in the context of something like Apache Kafka. To do this, you can configure your setup to use the Client Credentials Grant using Entra ID. This can be achieved with the following steps:

1. App Registration:

First, your app needs to register with the API provider. It gets a client ID (like a username) and a client secret (like a password). This is a unique key just for your app.

Here are the steps:

Create a new app registration that represents your client application. You can leave everything blank in the creation screen besides the name.
In the Overview tab of your new application registration, Click on the link of Client credentials, you will be redirected to a page Certificates & secrets: create a new secret.
In the API permissions, add a new API Permission to your client application, and, inside this option, add the role you want to use. This role should have been created in the last section as an Application role in the Registry server app registration. This is done so the role is sent within the token.
In your client identifying the application and in the one identifying the Registry server, in the API permissions section, you must grant admin consent for your directory. Note that it may take a few seconds for this to take effect.

NOTE: The registration of Registry server application must have an exposed api and the box checked for ID tokens (used for implicit and hybrid flows)

2. Requesting a Token:

When your app wants to access the API, it sends a request to the API’s authentication server. It includes: * The client ID. * The client secret (value generated in certificates&secrets) * A specific grant type, which is "client_credentials."

  curl -s -X POST \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d 'client_id=53d7fab9-8cbd-4c3b-84e8-d91a33eb90f8&scope=api://6ac8d309-0dcd-4aeb-b186-4911266038a8/.default&client_secret=Frs8Q~dUD8EwK2-RZ0JNEYS_Lbxlrs01m5F3tbx8&grant_type=client_credentials' \
    'https://login.microsoftonline.com/93a47e43-9ded-45a5-a855-eb462dd234b2/oauth2/v2.0/token'

The token generated has the following attributes:

token_type: "Bearer": This means that the type of token you received is a “Bearer” token. It’s like a special pass that you need to include when you access the API.
expires_in: 3599 : This tells you how long the token is valid, in seconds. Here, it’s 3599 seconds (which is about 1 hour). After that, you’ll need to request a new token.
ext_expires_in: 3599: This is usually the same as expires_in, indicating the extended expiration time. It’s another way to say the token lasts for about 1 hour.
access_token: "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6..." : This long string is the actual access token. It’s like the key that opens the door to the API. You’ll need to include this token in your requests to the API to access its resources.

Review Token Content:

Decode your JWT token (you can use tools like jwt.io) to inspect the claims. Check if the roles claims are included.

3. Interacting with the ApiCurio Registry API using Access Token:

response=$(curl -s -X POST -H "Content-Type: application/x-www-form-urlencoded" -d 'client_id=53d7fab9-8cbd-4c3b-84e8-d91a33eb90f8&scope=api://6ac8d309-0dcd-4aeb-b186-4911266038a8/.default&client_secret=Frs8Q~dUD8EwK2-RZ0JNEYS_Lbxlrs01m5F3tbx8&grant_type=client_credentials' 'https://login.microsoftonline.com/93a47e43-9ded-45a5-a855-eb462dd234b2/oauth2/v2.0/token'  | jq ".access_token")

ACCESS_TOKEN=$(echo "$response" | sed 's/^"//; s/"$//')
echo "$ACCESS_TOKEN"

MYREGISTRYURL=https://apicurioregistry-psql.registry.router-default.apps.com

Get list of groups

curl -k -v -X GET -H "Authorization: Bearer $ACCESS_TOKEN" \
  $MYREGISTRYURL/apis/registry/v2/groups

Creation of artifact:

curl -k -v -X POST -H "Content-Type: application/json; artifactType=AVRO" \
  -H "X-Registry-ArtifactId: share-price1" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  --data '{"type":"record","name":"price","namespace":"com.example", \
   "fields":[{"name":"symbol","type":"string"},{"name":"price","type":"string"}]}' \
  $MYREGISTRYURL/apis/registry/v2/groups/my-group/artifacts

Get Artifacts:

curl -s -k -v GET -H "Authorization: Bearer $ACCESS_TOKEN" \
  $MYREGISTRYURL/apis/registry/v2/groups/my-group/artifacts

Get Details of an Artifact:

curl -s -k -v GET -H "Authorization: Bearer $ACCESS_TOKEN" \
  $MYREGISTRYURL/apis/registry/v2/groups/my-group/artifacts/share-price1

Useful links:

Recording of Hands on Lab: https://drive.google.com/open?id=1DClnumPkmP2NFmOBhkYT0o9K8EAhCbXQ
Blog https://www.apicur.io/blog/2023/07/13/registry-azure-ad
Apicurio Registry API Definition: https://www.apicur.io/registry/docs/apicurio-registry/2.6.x/assets-attachments/registry-rest-api.htm#tag/Artifacts/operation/deleteArtifactsInGroup

Last, but not least, we still have a lot of things to do! As always, you can see the stuff we’re tracking by viewing the GitHub issues for the project.

If you find bugs or want to request a new feature, that’s a great place to start!

Apicurio Registry 3.0 - KafkaSQL Storage

Eric Wittmann & Carles Arnal Castello — Fri, 21 Jun 2024 16:00:00 +0000

As promised in the announcement blog about the upcoming release of Apicurio Registry 3.0, this post is a deep dive into the changes we’re making to our Kafka storage variant. We’ll get into what the KafkaSQL storage variant is and why you might want to use it (pros and cons). And importantly, we’ll discuss the significant changes we’ve made to the implementation of this feature for 3.0.

What is the KafkaSQL storage variant?

Of course, Apicurio Registry requires some form of storage in order to persist the content and metadata of artifacts and versions. We currently support two different storage variants:

SQL (Postgresql or SQL Server)
Kafka

While storing data in a SQL database is obvious (and is recommended for most use cases), it can be useful to instead store registry data in a Kafka topic. The KafkaSQL storage variant provides such a feature.

Why use Kafka for storage?

Even though we recommend using a database as the storage variant for Registry, there are cases where using Kafka instead can make sense. The most obvious one is that some organizations have significant expertise managing Kafka clusters, but do not have similar expertise with relational databases. It can be extremely convenient, when using Apicurio Registry as a Kafka Schema Registry, to use Kafka for registry storage (since you would obviously already have a working Kafka cluster).

How does KafkaSQL work?

The implementation of KafkaSQL is fairly simple: a persistent Kafka topic is used to communicate (and save) all data changes. You can think of this Kafka topic as a journal containing all changes to the registry. Whenever a user creates a new artifact, changes its metadata, adds a new version, or configures a rule, information about that change is published to the Kafka topic.

At the same time, Apicurio Registry is a consumer of the same topic. Whenever a message is published, Registry will consume it and update an internal H2 SQL database with that information. This not only provides persistence (the Kafka topic must be properly configured) but also allows multiple Registry instances (typically kubernetes/openshift pods) to each have a full copy of the Registry state in memory at the same time.

The flow is therefore this:

Change operation is received by Registry (HTTP Request Thread)

An HTTP request is received representing a registry data change
The change is turned into a Kafka message
The Kafka message is published to the Kafka topic (the journal)
The thread waits for the message to be consumed

Registry Kafka consumer thread consumes messages from the Kafka topic (Consumer Thread)

The message is consumed from the kafka topic
The message is converted to an internal SQL statement
The SQL statement is applied to the internal H2 database
The Request Thread is notified that the message was processed

Waiting request wakes up and responds (HTTP Request Thread continued)

The thread wakes up and responds to the originating HTTP request

Once nice thing about this implementation is that all read requests are handled the same way for both the SQL and KafkaSQL storage variants. In both cases we simply query the SQL database for the information (either the PostgreSQL database for the SQL variant or the internal H2 database for the KafkaSQL variant).

What are the pros and cons of using KafkaSQL?

Pros

You can store your registry data without provisioning and maintaining a SQL database.
Read requests scale extremely well (linearly) with the number of Registry pods you provision.

Cons

Registry pod startup times get slower and slower as the system ages, because all messages on the Kafka topic must be consumed on startup.
Data backup/restore might be harder for Kafka topics vs. a SQL database.
Fixing data corruption issues is much harder because you cannot simply run some SQL commands to modify data directly in the storage.

What are the KafkaSQL implementation changes in 3.0?

In Registry version 2.x, our approach to the structure of the Kafka messages was based on Kafka topic log compaction. Log compaction works well for data streams where more recent messages completely replace the data of old messages. In such systems, only the latest message is important. Even though that’s not exactly how we have structured our data model, we attempted to model our data as best we could to fit the log compaction approach. This approach worked to some degree, but we had a number of edge cases that made it challenging. It was also an approach that was difficult to maintain: every change to the storage layer needed to be really thought about, since a bug in the way we structured our Kafka messages could easily result in permanent data loss due to log compaction.

No more log compaction…

For the reasons explained above, we have decided to completely change the organization/structure of our Kafka messages. Instead of trying to organize our data into a log-compaction compatible structure, we are now creating the Kafka messages in more of an RPC style.

Essentially we are creating a one-to-one mapping between (write) methods in our RegistryStorage interface and messages on the Kafka journal topic. This makes maintaining both storage variants extremely easy, but it means that we are no longer taking advantage of log compaction (since the messages are no longer structured in that way). The major consequence of this change is that the number of messages on the Kafka topic is even greater than before, potentially resulting in even longer pod startup times (since it continues to be true that on startup a pod must consume all messages on the topic to fully reproduce the state).

We already needed a solution to that problem, and now we need it even more! Which leads us to…

State snapshots…for speed!

As already discussed in this blog, whenever a new instance of Registry is started pointing to an existing journal topic, it has to process every single message to restore the internal state of the application that is used for reads. This has two main effects:

In the event of data loss in the topic, there is no backup strategy (by default)
It causes the replica start to be fairly slow (depending on the topic size, of course).

To address these problems, a new feature has been implemented in Registry 3.x: snapshots of the internal store.

The way snapshots work in Registry 3 is that, when a (POST) call to the endpoint /admin/snapshots is made, a new snapshot of the internal store is created. This is done by sending a snapshot marker message with a snapshot id (randomly generated) to the journal topic. This marker message represents a request for a new snapshot to be created.

After this, the consumer thread (the same thread that is responsible for processing all the other messages) reads the marker message and creates a SQL dump of the internal store. The SQL dump is created in a location that can be configured using the following property:

System property: apicurio.storage.snapshot.location
Environment variable: APICURIO_STORAGE_SNAPSHOT_LOCATION

(Note: The default value for this property is /tmp)

Once the SQL dump is created, a message is published to the snapshots Kafka topic. The message includes the location of the snapshot and the snapshotId (as the message key). The snapshots topic is used to keep track of all the created snapshots. This means that, as you might be already thinking, in Registry 3 you will need to create and configure two Kafka topics instead of just one:

Journal topic (one message for each write operation)
Snapshots topic (one message for each created snapshot)

Once the snapshot has been created, when a new replica is created (or an existing replica is restarted), the messages on the snapshots topic are consumed in order to find the most recent snapshot. If there is a snapshot present, the application loads the SQL dump retrieved from the location in the message and uses it to restore the internal database. This dramatically improves the application startup time, since the internal state is restored by issuing a single SQL command (H2 RESTORE TO) instead of processing every single message present in the journal topic.

Once the database has been restored, the consumer thread starts consuming the messages present in the journal topic, discarding all the messages until the corresponding snapshot marker message is found. All remaining messages (those messages on the journal topic that follow the snapshot marker message) are processed as normal and dispatched to the SQL store as appropriate.

With this new feature you can now set up a backup strategy of your Apicurio Registry data when Kafka is used as the storage option, preventing potential data loss and also improving replica startup time.

Finally, here are two graphics comparing the startup times with and without snapshotting. In the snapshot graph snapshots were created for each 250.000 artifacts:

Apicurio Registry 3.0

Eric Wittmann — Mon, 17 Jun 2024 16:00:00 +0000

Hey everyone - just dropping a quick blog post to let you know that we have released Apicurio Registry 3.0.0.M3. We’re working hard to get this new major release of Registry stable. This milestone 3 release is a huge step towards that goal. Very soon you can expect a 3.0.0.Final release, but please consider getting your feet wet now. We think 3.0 is ready for evaluation! (more below the fold)

Why a new major version?

Apicurio Registry 2.x has been around since April of 2021 (!!) and was a marked improvement over 1.x. It has served us well, but it was time to address some core design decisions as well as some accrued technical debt.

In particular, we wanted the freedom to change the data model (aka the database schema) and the REST API without needing to maintain compatibility. That should really only be done in a major version release.

What’s new in Registry 3.0?

We plan to dig into some of these topics in greater detail in a few future blog posts, but here are some of the highlights of version 3:

Data model changes

There are some significant core data model changes in 3.0. Some of these changes include (but are not limited to):

Artifacts have their own metaData
Artifacts can be empty (have no versions)
Groups can now have labels
User can define and maintain custom branches
“Latest” is now a branch

REST API changes

Full group management, with metaData
Streamlined artifact and version creation
Search for groups at /search/groups
Search for versions at /search/versions
A new dryRun query param replaces the /test endpoint(s)
Brand new Branch API

Kafka storage variant

Somewhat surprising to us is that a majority of our users have deployed registry using Kafka as the storage. We do still recommend a database for persistence, but of course we recognize the convenience and familiarity of using Kafka instead for many users. Rest assured we have no plans to stop supporting it!

On the contrary - we have completely rewritten the KafkaSQL layer to improve its stability and to make it easier for us to maintain. This will be the subject of a separate blog post soon, but know that the new implementation should be much easier to keep updated and should result in far fewer unexpected bugs. Spoiler: we are also introducing a “snapshotting” feature that will help address the slow startup times that can result from larger deployments (deployments with lots of artifacts).

Try it out now!

We haven’t yet updated the web site with new documentation and getting started instructions, and we probably won’t do that until 3.0.0.Final is out. So until then, you can easily try it out using docker or podman:

Pull the images

docker pull quay.io/apicurio/apicurio-registry:3.0.0.M3
docker pull quay.io/apicurio/apicurio-registry-ui:3.0.0.M3

Run both `apicurio-registry` and `apicurio-registry-ui`

docker run -it -p 8080:8080 quay.io/apicurio/apicurio-registry:3.0.0.M3

docker run -it -p 8888:8080 quay.io/apicurio/apicurio-registry-ui:3.0.0.M3

The UI will be available here: http://localhost:8888

And the APIs will be available: http://localhost:8080/apis

Apicurio Registry - JSON Schema dereference

Carles Arnal Castello — Tue, 14 May 2024 16:00:00 +0000

Intro

The code example used for this blog post can be found in the Apicurio Registry examples.

One of the limitations of the JSON Schema format is that you cannot discover the full schema from some data that adheres to the schema. That’s the main reason why, for this serde, the schema being used has to be registered upfront, and that’s what you see in the code example from line 57 to line 132. where the main schema city.json is registered with all the references in the hierarchy. Along those lines, a full hierarchy of schemas is registered in Apicurio Registry. The hierarchy can be described as follows and represented as a tree:

                          citizen
     |          |                     |           
    city   qualification        citizenIdentifier
     |                                |
 qualification                   qualification
    

As you may notice, there are three different files with the same name. This is done on purpose to demonstrate that you can have different schemas with the same name within the same hierarchy. When they’re registered in the code example, they’re registered under the same group but using different artifact ids, so they can be referenced from their main schema.

Since an image is way better than trying to describe it, this is the structure in the file system:

Using the REST API

Although the example above uses the Apicurio Registry Java client to register the full hierarchy of schemas, it’s also possible to use the REST API. Note that, when an artifact with references is created, the artifact content itself in the request must be quoted. Here’s a set of Curl commands that delivers this result:

First we must start creating the leaves, in this case, those are the city qualification and the citizenIdentifier qualification:

//Creates the city qualification
curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts/' \
--header 'X-Registry-ArtifactId: cityQualification' \
--header 'Content-Type: application/json; artifactType=JSON' \
--data '{
"$id": "https://example.com/types/city/qualification.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Qualification",
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The subject'\''s name"
},
"qualification": {
"type": "integer",
"description": "The city qualification",
"minimum": 10
}
}
}'

//Creates the citizen identifier qualification
curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts/' \
--header 'X-Registry-ArtifactId: identifierQualification' \
--header 'Content-Type: application/json; artifactType=JSON' \
--data '{
"$id": "https://example.com/types/identifier/qualification.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Qualification",
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The subject'\''s name"
},
"qualification": {
"type": "integer",
"description": "The identifier qualification",
"minimum": 20
}
}
}'

Once the leaves are created, we can go up through the hierarchy, creating the schemas that will be referenced by the citizen. First we create the city schema:

curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts/' \
--header 'X-Registry-ArtifactId: city' \
--header 'Content-Type: application/create.extended+json' \
--header 'X-Registry-ArtifactType: JSON' \
--data '{
"content": "{\r\n  \"$id\": \"https:\/\/example.com\/types\/city\/city.json\",\r\n  \"$schema\": \"http:\/\/json-schema.org\/draft-07\/schema#\",\r\n  \"title\": \"City\",\r\n  \"type\": \"object\",\r\n  \"properties\": {\r\n    \"name\": {\r\n      \"type\": \"string\",\r\n      \"description\": \"The city'\''s name.\"\r\n    },\r\n    \"zipCode\": {\r\n      \"type\": \"integer\",\r\n      \"description\": \"The zip code.\",\r\n      \"minimum\": 0\r\n    },\r\n    \"qualification\": {\r\n      \"$ref\": \"qualification.json\"\r\n    }\r\n  }\r\n}",
"references": [
{
"name": "qualification.json",
"groupId": "default",
"artifactId": "cityQualification",
"version": "1"
}
]
}'

Then the citizen identifier schema:

curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts/' \
--header 'X-Registry-ArtifactId: citizenIdentifier' \
--header 'Content-Type: application/create.extended+json' \
--header 'X-Registry-ArtifactType: JSON' \
--data '{
"content": "{\r\n  \"$id\": \"https:\/\/example.com\/types\/identifier\/citizenIdentifier.json\",\r\n  \"$schema\": \"http:\/\/json-schema.org\/draft-07\/schema#\",\r\n  \"title\": \"Identifier\",\r\n  \"type\": \"object\",\r\n  \"properties\": {\r\n    \"identifier\": {\r\n      \"type\": \"integer\",\r\n      \"description\": \"The citizen identifier.\",\r\n      \"minimum\": 0\r\n    },\r\n    \"qualification\": {\r\n      \"$ref\": \"qualification.json\"\r\n    }\r\n  }\r\n}",
"references": [
{
"name": "qualification.json",
"groupId": "default",
"artifactId": "identifierQualification",
"version": "1"
}
]
}'

The citizen qualification schema:

curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts/' \
--header 'X-Registry-ArtifactId: qualification' \
--header 'Content-Type: application/json; artifactType=JSON' \
--data '{
"$id": "https://example.com/qualification.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Qualification",
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The subject'\''s name"
},
"qualification": {
"type": "integer",
"description": "The qualification.",
"minimum": 0
}
}
}'

And, finally, the citizen schema itself with references to qualification, city and identifier

curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts/' \
--header 'X-Registry-ArtifactId: citizen' \
--header 'Content-Type: application/create.extended+json' \
--header 'X-Registry-ArtifactType: JSON' \
--data '{
"content": "{\r\n \"$id\": \"https:\/\/example.com\/citizen.json\",\r\n \"$schema\": \"http:\/\/json-schema.org\/draft-07\/schema#\",\r\n \"title\": \"Citizen\",\r\n \"type\": \"object\",\r\n \"properties\":{\r\n\"firstName\": {\r\n \"type\": \"string\",\r\n \"description\": \"The citizen'\''s first name.\"\r\n },\r\n\"lastName\": {\r\n \"type\": \"string\",\r\n \"description\": \"The citizen'\''s last name.\"\r\n },\r\n \"age\": {\r\n\"description\": \"Age in years which must be equal to or greater than zero.\",\r\n \"type\": \"integer\",\r\n\"minimum\": 0\r\n },\r\n \"city\": {\r\n \"$ref\": \"types\/city\/city.json\"\r\n },\r\n \"identifier\": {\r\n\"$ref\": \"types\/identifier\/citizenIdentifier.json\"\r\n },\r\n \"qualifications\": {\r\n \"type\": \"array\",\r\n\"items\": {\r\n \"$ref\": \"qualification.json\"\r\n }\r\n }\r\n },\r\n \"required\": [\r\n \"city\"\r\n  ]\r\n}",
"references": [
{
"name": "qualification.json",
"groupId": "default",
"artifactId": "qualification",
"version": "1"
},
{
"name": "types/city/city.json",
"groupId": "default",
"artifactId": "city",
"version": "1"
},
{
"name": "types/identifier/citizenIdentifier.json",
"groupId": "default",
"artifactId": "citizenIdentifier",
"version": "1"
},
{
"name": "sample.address.json",
"groupId": "default",
"artifactId": "address",
"version": "1"
}
]
}'

JSON Schema dereference

{
  "$id": "https://example.com/citizen.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Citizen",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The citizen's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The citizen's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    },
    "city": {
      "$ref": "types/city/city.json"
    },
    "identifier": {
      "$ref": "types/identifier/citizenIdentifier.json"
    },
    "qualifications": {
      "type": "array",
      "items": {
        "$ref": "qualification.json"
      }
    }
  },
  "required": [
    "city"
  ]
}

{
  "$schema" : "http://json-schema.org/draft-07/schema#",
  "title" : "Citizen",
  "type" : "object",
  "properties" : {
    "firstName" : {
      "description" : "The citizen's first name.",
      "type" : "string"
    },
    "lastName" : {
      "description" : "The citizen's last name.",
      "type" : "string"
    },
    "identifier" : {
      "title" : "Identifier",
      "type" : "object",
      "properties" : {
        "identifier" : {
          "description" : "The citizen identifier.",
          "type" : "integer",
          "minimum" : 0
        },
        "qualification" : {
          "title" : "Qualification",
          "type" : "object",
          "properties" : {
            "qualification" : {
              "description" : "The identifier qualification",
              "type" : "integer",
              "minimum" : 20
            },
            "name" : {
              "description" : "The subject's name",
              "type" : "string"
            }
          }
        }
      }
    },
    "qualifications" : {
      "type" : "array",
      "items" : {
        "title" : "Qualification",
        "type" : "object",
        "properties" : {
          "qualification" : {
            "description" : "The qualification.",
            "type" : "integer",
            "minimum" : 0
          },
          "name" : {
            "description" : "The subject's name",
            "type" : "string"
          }
        }
      }
    },
    "city" : {
      "title" : "City",
      "type" : "object",
      "properties" : {
        "zipCode" : {
          "description" : "The zip code.",
          "type" : "integer",
          "minimum" : 0
        },
        "qualification" : {
          "title" : "Qualification",
          "type" : "object",
          "properties" : {
            "qualification" : {
              "description" : "The city qualification",
              "type" : "integer",
              "minimum" : 10
            },
            "name" : {
              "description" : "The subject's name",
              "type" : "string"
            }
          }
        },
        "name" : {
          "description" : "The city's name.",
          "type" : "string"
        }
      }
    },
    "age" : {
      "description" : "Age in years which must be equal to or greater than zero.",
      "type" : "integer",
      "minimum" : 0
    }
  },
  "required" : [ "city" ],
  "$id" : "https://example.com/citizen.json"
}

As you can see, that’s the content of the full hierarchy, not just the schema of the citizen element alone. There are two endpoints where you can get the content dereferenced, with the following curl commands:

curl --location 'http://localhost:8080/apis/registry/v2/ids/globalIds/6?dereference=true'
curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts/JsonSerdeReferencesExample?dereference=true'
curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts/JsonSerdeReferencesExample/versions/1?dereference=true'

Referencing options ===

{
  "$id": "https://example.com/types/all-types.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "definitions": {
    "City": {
      "title": "City",
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "The city's name."
        },
        "zipCode": {
          "type": "integer",
          "description": "The zip code.",
          "minimum": 0
        }
      }
    },
    "Identifier": {
      "title": "Identifier",
      "type": "object",
      "properties": {
        "identifier": {
          "type": "integer",
          "description": "The citizen identifier.",
          "minimum": 0
        }
      }
    }
  }
}

With definitions for both an identifier and the city objects. Then we might decide that we want to have a separate schema that points to an entire definition, like the one below:

{
  "$id": "https://example.com/citizen.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Citizen",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The citizen's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The citizen's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    },
    "city": {
      "$ref": "types/all-types.json#/definitions/City"
    },
    "identifier": {
      "$ref": "types/all-types.json#/definitions/Identifier"
    }
  },
  "required": [
    "city"
  ]
}

Or we might decide to just reference a single property inside a schema definition, like the schema below does:

{
  "$id": "https://example.com/citizen.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Citizen",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The citizen's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The citizen's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    },
    "city": {
      "$ref": "types/all-types.json#/definitions/City/properties/name"
    },
    "identifier": {
      "$ref": "types/all-types.json#/definitions/Identifier"
    }
  },
  "required": [
    "city"
  ]
}

curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts/' \
--header 'X-Registry-ArtifactId: all-types' \
--header 'Content-Type: application/json; artifactType=JSON' \
--data '{
  "$id": "https://example.com/types/all-types.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "definitions": {
    "City": {
      "title": "City",
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "The city'\''s name."
        },
        "zipCode": {
          "type": "integer",
          "description": "The zip code.",
          "minimum": 0
        },
        "qualification": {
          "$ref": "city/qualification.json"
        }
      }
    },
    "Identifier": {
      "title": "Identifier",
      "type": "object",
      "properties": {
        "identifier": {
          "type": "integer",
          "description": "The citizen identifier.",
          "minimum": 0
        },
        "qualification": {
          "$ref": "identifier/qualification.json"
        }
      }
    }
  }
}'

Once this schema has been registered, we can now decide to register two separate schemas, the first one referencing the entire city object:

curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts' \
--header 'X-Registry-ArtifactId: citizen-object-level' \
--header 'Content-Type: application/create.extended+json' \
--header 'X-Registry-ArtifactType: JSON' \
--data '{
    "content": "{\r\n  \"$id\": \"https:\/\/example.com\/citizen.json\",\r\n  \"$schema\": \"http:\/\/json-schema.org\/draft-07\/schema#\",\r\n  \"title\": \"Citizen\",\r\n  \"type\": \"object\",\r\n  \"properties\": {\r\n    \"firstName\": {\r\n      \"type\": \"string\",\r\n      \"description\": \"The citizen'\''s first name.\"\r\n    },\r\n    \"lastName\": {\r\n      \"type\": \"string\",\r\n      \"description\": \"The citizen'\''s last name.\"\r\n    },\r\n    \"age\": {\r\n      \"description\": \"Age in years which must be equal to or greater than zero.\",\r\n      \"type\": \"integer\",\r\n      \"minimum\": 0\r\n    },\r\n    \"city\": {\r\n      \"$ref\": \"types\/all-types.json#\/definitions\/City\"\r\n    },\r\n    \"identifier\": {\r\n      \"$ref\": \"types\/all-types.json#\/definitions\/Identifier\"\r\n    }\r\n  },\r\n  \"required\": [\r\n    \"city\"\r\n  ]\r\n}",
    "references": [
        {
            "name": "types/all-types.json#/definitions/City",
            "groupId": "default",
            "artifactId": "all-types",
            "version": "1"
        },
        {
            "name": "types/all-types.json#/definitions/Identifier",
            "groupId": "default",
            "artifactId": "all-types",
            "version": "1"
        }
    ]
}'

And then register the second schema, where just the city name property is referenced:

curl --location 'http://localhost:8080/apis/registry/v2/groups/default/artifacts' \
--header 'X-Registry-ArtifactId: citizen-property-level' \
--header 'Content-Type: application/create.extended+json' \
--header 'X-Registry-ArtifactType: JSON' \
--data '{
    "content": "{\r\n  \"$id\": \"https:\/\/example.com\/citizen.json\",\r\n  \"$schema\": \"http:\/\/json-schema.org\/draft-07\/schema#\",\r\n  \"title\": \"Citizen\",\r\n  \"type\": \"object\",\r\n  \"properties\": {\r\n    \"firstName\": {\r\n      \"type\": \"string\",\r\n      \"description\": \"The citizen'\''s first name.\"\r\n    },\r\n    \"lastName\": {\r\n      \"type\": \"string\",\r\n      \"description\": \"The citizen'\''s last name.\"\r\n    },\r\n    \"age\": {\r\n      \"description\": \"Age in years which must be equal to or greater than zero.\",\r\n      \"type\": \"integer\",\r\n      \"minimum\": 0\r\n    },\r\n    \"city\": {\r\n      \"$ref\": \"types\/all-types.json#\/definitions\/City\/properties\/name\"\r\n    },\r\n    \"identifier\": {\r\n      \"$ref\": \"types\/all-types.json#\/definitions\/Identifier\"\r\n    }\r\n  },\r\n  \"required\": [\r\n    \"city\"\r\n  ]\r\n}",
    "references": [
        {
            "name": "types/all-types.json#/definitions/City/properties/name",
            "groupId": "default",
            "artifactId": "all-types",
            "version": "1"
        },
        {
            "name": "types/all-types.json#/definitions/Identifier",
            "groupId": "default",
            "artifactId": "all-types",
            "version": "1"
        }
    ]
}'

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Citizen",
    "type": "object",
    "properties": {
        "firstName": {
            "description": "The citizen's first name.",
            "type": "string"
        },
        "lastName": {
            "description": "The citizen's last name.",
            "type": "string"
        },
        "identifier": {
            "title": "Identifier",
            "type": "object",
            "properties": {
                "identifier": {
                    "description": "The citizen identifier.",
                    "type": "integer",
                    "minimum": 0
                }
            }
        },
        "city": {
            "title": "City",
            "type": "object",
            "properties": {
                "zipCode": {
                    "description": "The zip code.",
                    "type": "integer",
                    "minimum": 0
                },
                "name": {
                    "description": "The city's name.",
                    "type": "string"
                }
            }
        },
        "age": {
            "description": "Age in years which must be equal to or greater than zero.",
            "type": "integer",
            "minimum": 0
        }
    },
    "required": [
        "city"
    ],
    "$id": "https://example.com/citizen.json"
}

Whereas in the second scenario, the property one, just the city name will be inlined:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Citizen",
    "type": "object",
    "properties": {
        "firstName": {
            "description": "The citizen's first name.",
            "type": "string"
        },
        "lastName": {
            "description": "The citizen's last name.",
            "type": "string"
        },
        "identifier": {
            "title": "Identifier",
            "type": "object",
            "properties": {
                "identifier": {
                    "description": "The citizen identifier.",
                    "type": "integer",
                    "minimum": 0
                }
            }
        },
        "city": {
            "description": "The city's name.",
            "type": "string"
        },
        "age": {
            "description": "Age in years which must be equal to or greater than zero.",
            "type": "integer",
            "minimum": 0
        }
    },
    "required": [
        "city"
    ],
    "$id": "https://example.com/citizen.json"
}

JSON Schema Kafka Serde dereference

Now, in order to make good use of all of this, we can use a Kafka producer. For this example, in the Kafka Producer there is one extremely important property we must set, the schema resolver configuration option SERIALIZER_DEREFERENCE_SCHEMA. This is the option that will instruct the Schema Resolver to use the dereference parameter in the globalIds endpoint above (since that is the endpoint used by default to resolve the schema).

Properties props = new Properties();

// Configure kafka settings
props.putIfAbsent(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, SERVERS);
props.putIfAbsent(ProducerConfig.CLIENT_ID_CONFIG, "Producer-" + TOPIC_NAME);
props.putIfAbsent(ProducerConfig.ACKS_CONFIG, "all");
props.putIfAbsent(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.putIfAbsent(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSchemaKafkaSerializer.class.getName());
props.putIfAbsent(SerdeConfig.ARTIFACT_RESOLVER_STRATEGY, SimpleTopicIdStrategy.class.getName());
props.putIfAbsent(SerdeConfig.EXPLICIT_ARTIFACT_GROUP_ID, "default");
props.putIfAbsent(SerdeConfig.REGISTRY_URL, REGISTRY_URL);

//This is the most important configuration for this particular example.
props.putIfAbsent(SchemaResolverConfig.SERIALIZER_DEREFERENCE_SCHEMA, true);


// Create the Kafka producer
Producer<Object, Object> producer = new KafkaProducer<>(props);

With this producer we can then send messages to Kafka that adhere to the citizen schema as follows:

for(int idx = 0; idx < 5; idx++) {
    City city=new City("New York",10001);
    city.setQualification(new CityQualification("city_qualification",11));

    CitizenIdentifier identifier=new CitizenIdentifier(123456789);
    identifier.setIdentifierQualification(new IdentifierQualification("test_subject",20));
    Citizen citizen=new Citizen("Carles","Arnal",23,city,identifier,List.of(new Qualification(UUID.randomUUID().toString(),6),new Qualification(UUID.randomUUID().toString(),7),new Qualification(UUID.randomUUID().toString(),8)));

    // Send/produce the message on the Kafka Producer
    ProducerRecord<Object, Object> producedRecord = new ProducerRecord<>(topicName, SUBJECT_NAME, citizen);
    producer.send(producedRecord);
}

And consume them with a consumer that is created as follows, instructing the deserializer to use the schema dereferenced with the following configuration property DESERIALIZER_DEREFERENCE_SCHEMA:

Properties props = new Properties();

props.putIfAbsent(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, SERVERS);
props.putIfAbsent(ConsumerConfig.GROUP_ID_CONFIG, "Consumer-" + TOPIC_NAME);
props.putIfAbsent(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
props.putIfAbsent(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
props.putIfAbsent(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.putIfAbsent(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.putIfAbsent(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonSchemaKafkaDeserializer.class.getName());
props.putIfAbsent(SerdeConfig.ARTIFACT_RESOLVER_STRATEGY, SimpleTopicIdStrategy.class.getName());
props.putIfAbsent(SerdeConfig.EXPLICIT_ARTIFACT_GROUP_ID, "default");
props.putIfAbsent(SerdeConfig.VALIDATION_ENABLED, true);

//This is the most important configuration for this example, that instructs the deserializer to fetch the dereferenced schema.
props.putIfAbsent(SchemaResolverConfig.DESERIALIZER_DEREFERENCE_SCHEMA, true);

// Configure Service Registry location
props.putIfAbsent(SerdeConfig.REGISTRY_URL, REGISTRY_URL);
// No other configuration needed for the deserializer, because the globalId of the schema
// the deserializer should use is sent as part of the payload.  So the deserializer simply
// extracts that globalId and uses it to look up the Schema from the registry.

// Create the Kafka Consumer
KafkaConsumer<Long, Citizen> consumer = new KafkaConsumer<>(props);

And then consume the messages with the consumer configured above like this:

while (messageCount < 5) {
    final ConsumerRecords<Long, Citizen> records = consumer.poll(Duration.ofSeconds(1));
    messageCount += records.count();
    if (records.count() == 0) {
        // Do nothing - no messages waiting.
        System.out.println("No messages waiting...");
    } else records.forEach(record -> {
        Citizen msg = record.value();
        System.out.println("Consumed a message: " + msg + " @ " + msg.getCity());
    });
}

Instead of doing an HTTP request to get the citizen schema, another request to fetch the pointers to the references, and, in this case (this process grows with the hierarchy size), another nine requests to discover the full hierarchy, a single HTTP request gets the full content of the hierarchy and then uses it to produce/consume messages.

Apicurio Studio - Now with OpenAPI Vendor Extensions

Eric Wittmann — Fri, 10 May 2024 16:00:00 +0000

The Apicurio Studio OpenAPI editor has been around for a long time, and over time support for vendor extensions has evolved somewhat. It started with:

"You can use the Source tab, good luck!"

Eventually we added a Vendor Extension section in various places, which made things a bit better. However, the UI was rather basic - just a list of the extensions with values displayed as raw JSON. Editing the values was also basic - simply a source editor for the raw JSON value of the extension.

We’ve made another incremental improvement in this area, which is to use form generation (via Formly) to generate a UI form used when editing the value of a vendor extension!

What are OpenAPI Vendor Extensions?

The OpenAPI specification defines a set of semantics for describing a REST API. Endpoints, data types, operations, inputs and outputs, etc are all concepts defined by the specification. These first order concepts are often all you need to fully define the shape of your API.

However, there may be aspects of your API that you want to define which go beyond the common semantics defined by the OpenAPI specification. In these cases, OpenAPI defines the concept of a Vendor Extension, which are open ended property names that must begin with x-.

So imagine you want to define some API Management limits on your operations and you have a REST API platform that can inspect the OpenAPI to find the configured limits for any given operation. You might set rate limits for an operation directly in the OpenAPI like this:

openapi: 3.1.0
info:
    title: Example API
    version: 1.0
paths:
    /widgets:
        summary: Get a list of widgets.
        get:
            responses:
                '200':
                    $ref: '#/components/responses/WidgetListResponse'
                '500':
                    $ref: '#/components/responses/ServerError'
            operationId: listWidgets
            summary: List widgets
            x-apimanagement-rateLimit:
                rate: 100
                period: 'second'
                per: 'user'

Everything in the above example is standard OpenAPI semantics except for the x-apimanagement-rateLimit API Management property, which is a vendor extension. This example might indicate that the platform should apply a rate limit of 100 requests per second per user. Neat!

What inspired this feature?

Apicurio Studio will be included as a component of a new Red Hat product called Connectivity Link. This new product has a lot of functionality in the network connectivity and API management spaces, and one of the things it supports is enabling and configuring its various features by adding some vendor extensions to the OpenAPI document.

Another component of Connectivity Link is Kuadrant, which is an open source project that brings policy based application connectivity features to Kubernetes. That project defines some OpenAPI kuadrant extensions that you can use to configure things like routes and rate limits.

We thought it would be nice if users could use a form to enter the Kuadrant vendor extension values, rather than having to copy/paste the correct JSON from the Kuadrant documentation into the editor.

Wait, is this ONLY for Kuadrant?

Of course, we could have simply added UI support for Kuadrant vendor extensions. But that would only add support for this one specific type of extension. What we really wanted was a way to easily configure Studio to support a wide range of vendor extensions, without having to write code and release new versions. So what we’ve done is implement a dynamic form generation approach that allows you to configure Studio with a list of vendor extensions at deploy time. More details on that later.

OK but show me what it looks like!

Right, let’s show some stuff! Here’s what things used to look like in the editor:

Notice that you have to provide the full vendor extension property name as well as the value in raw JSON format. It’s not great.

Now here’s what it looks like:

Cool!

How do I use this feature?

Unfortunately the feature isn’t magic, which means there is some configuration needed. Let’s dig into that in this final section.

Creating a vendor-extensions configuration

What you need to do is create a JSON file with the correct information (for example vendor-extensions.json). You’ll need to create this file and then eventually make it available to the Studio backend application (see the next section).

The vendor extension configuration file is just a JSON file with an array of supported vendor extensions. Each vendor extension item in the array should have the following properties:

name : the name of the vendor extension property, which must begin with x-
schema : a JSON schema that describes the format of the vendor extension value
model : a JSON object that is the default values for the vendor extension value
components : an (optional) array of OpenAPI component names indicating valid placement of the vendor extension

Here is an example of this file (the one used in this blog post):

[
    {
        "name": "x-address",
        "schema": {
            "$schema": "http://json-schema.org/draft-07/schema#",
            "type": "object",
            "title": "Enter address",
            "properties": {
                "street": {
                    "title": "Street",
                    "type": "string"
                },
                "city": {
                    "title": "City",
                    "type": "string"
                },
                "state": {
                    "title": "State",
                    "type": "string",
                    "pattern": "^[A-Z]{2}$"
                },
                "zip": {
                    "title": "Zip",
                    "type": "string",
                    "pattern": "^[0-9]{5}$"
                }
            },
            "required": [
                "street",
                "city",
                "state",
                "zip"
            ]
        },
        "model": {
           "street": "",
           "city": "",
           "state": "",
           "zip": ""
        },
        "components": [
            "document"
        ]
    }
]

The above example has only a single vendor extension defined, but you can specify as many as you like.

Note that the components property allows you to specify where in the OpenAPI document the vendor extension is valid. In the above example it will only appear when adding a vendor extension to the root of the document. Other allowed values for this are:

schema
response
parameter
header
requestBody
securityScheme
pathItem
operation

You get the idea!

When defining your JSON Schema for a vendor extension, we recommend making sure you have a title defined for each property. This will allow the form generator to create appropriate labels in the UI. See the Formly documentation for more information.

Telling Studio where the configuration is

Now that you have a vendor-extensions configuration file created, you need to tell Studio where it lives. You can do this by setting the APICURIO_UI_OPENAPI_VENDOR-EXTENSIONS_URL= environment variable when starting up Studio. For example:

docker run -it -p 8080:8080 –env APICURIO_UI_OPENAPI_VENDOR-EXTENSIONS_URL=”https://raw.githubusercontent.com/Apicurio/apicurio-studio/0f81ea54bcf780daff59f5ea78c7990c6a4221f9/app/src/main/resources/extensions/address.json” apicurio/apicurio-studio:latest-snapshot

Any valid URL that can be resolved by the application should work. Just be careful of using local file paths when running a docker image - if you want that to work then you’ll obviously need to mount the file into the container.

Conclusion

Thanks for reading. This feature is still in Beta and can do with some feedback and iteration. If you feel so inspired, please give it a try and let us know how it can be improved (oh let me count the ways).

Cheers!

Introducing VS Code extension for Apicurio Registry

Jakub Senko — Fri, 19 Apr 2024 12:00:00 +0000

Are you looking for a tool that will make your work with Apicurio Registry easier? If you have Visual Studio Code installed, there is an extension that allows you to interact with Apicurio Registry directly from your IDE. This project was contributed to the Apicurio community by Jean-Etienne Martin. 🎉

Installation

Installation is easy, because the extension is published to the Visual Studio Marketplace. In Visual Studio Code, go to the quick open dialog with ctrl+P, and execute the following command:

ext install apicurio.apicurio-registry-explorer

The extension will be installed within a few seconds. If you are using a non-official distribution of VS Code, such as VSCodium, the extension is also available on the Open VSX Marketplace.

Now we need to connect to an Apicurio Registry instance. We’ll use docker to start a quick ephemeral Apicurio Registry container at localhost:8080:

docker run -p 8080:8080 quay.io/apicurio/apicurio-registry-mem:latest-release

Open the settings panel (ctrl+,), go to Extensions > Apicurio, and fill in the connection details:

If you have configured the connection settings correctly, and started with an empty Apicurio Registry, the extension panel should look like this:

But don’t worry, it won’t stay empty for much longer!

Structure of content in Apicurio Registry

Before we use the extension to add new content to Apicurio Registry, let’s do a quick review of the organization of data in the Registry. This will help us better understand how to use the extension.

Main organizational unit in Apicurio Registry is called an artifact, which represents a piece content (e.g. an Avro schema) that can change over time. Each change to the artifact content results in a new version, so an artifact can be thought of as a sequence of versions. Version represents a specific piece of content that can’t be further modified, so the evolution of a schema happens at the artifact level. Artifacts are further organized into groups. Each artifact must belong to one specific group (the default group is unsurprisingly named default), and the groups can’t be nested.

As a result of this hierarchical organizational structure, each static piece of content can be identified by a group ID:artifact ID:version triple. There are other identifiers available, but the GAV can be considered a primary identifier.

In addition to the content itself, Apicurio Registry also stores metadata for an artifact and each artifact version. The metadata contain information such as when and by whom was the content created or modified, secondary identifiers, and user-editable data such as artifact name, description, or a map of properties. Some of the metadata (e.g. the properties), are shared between an artifact and its latest version.

One of the interesting pieces of artifact version metadata is its state, which can be one of ENABLED, DEPRECATED, or DISABLED. This value affects Apicurio Registry behavior when retrieving content. A disabled version behaves as if it were not present when a user attempts to retrieve it. Moreover, when reading artifact content (i.e. the latest version of an artifact), and the latest version is disabled, the previous version is returned instead. However, a disabled version still allows users to change its state, and will show in search results.

Adding new content to Apicurio Registry

Let’s add a few Avro schemas to the Registry, so we can find out how the extension displays Apicurio Registry content. You can use the following two example Avro schemas if you don’t have any others on hand:

The first version:

{
  "namespace": "example.avro",
  "type": "record",
  "name": "User",
  "fields": [
    { "name": "name", "type": "string" },
    { "name": "favorite_number", "type": ["int", "null"] },
    { "name": "favorite_color", "type": ["string", "null"] }
  ]
}

and a second version with updated content:

{
  "namespace": "example.avro",
  "type": "record",
  "name": "User",
  "fields": [
    { "name": "name", "type": "string" },
    { "name": "favorite_number", "type": ["int", "null"] },
    { "name": "favorite_color", "type": ["string", "null"] },
    { "name": "favorite_food", "type": ["string", "null"] }
  ]
}

Click on File > Open Folder… to open a new empty working directory, and save the schemas above as new files user1.json and ser2.json, respectively.

To create a new artifact, click on the … > Add artifact in the Apicurio Explorer view. You will be shown a sequence of prompts, where you can choose from a list of options or input a text:

New or existing group ? asks you whether you want to create the artifact within an existing group, or do you want to create a new one. We’ll select NEW, since our Apicurio Registry is currently empty. If you select EXISTING, you will be asked to Choose group from a list of known group IDs.
Create a new group ID asks you to name the new group. We’ll use default, and confirm by writing default again.
Choose an artifact type to push asks you for the type of schema that is being published to Apicurio Registry, and lists the supported options. We’ll select AVRO, and confirm.
Artifact ID asks you to specify the artifact ID. We’ll type user, and confirm.
Initial version prompt asks you to name the first artifact version. We’ll follow semantic versioning and use 1.0.0.
Search for file prompt allows you to specify a search pattern, which will be used to provide a list of files from which you can select the schema. We named our schema files user1.json and user2.json, so we can input user*. If there are only a few files in your workspace, you can use * to save a few keystrokes🙂.
Select file allows us to select one of the files based on the search pattern. Select the path where user1.json file is located on your computer.
Create AVRO artifact with identifiers ‘default:user:1.0.0’ ? is the last step. It asks us for a confirmation that we want to create the artifact with the default:user:1.0.0 GAV identifier.

Exploring Apicurio Registry content

Let’s take a look at the three views that Apicurio Registry Explorer provides:

Apicurio Explorer shows a list of artifacts, grouped by artifact group ID. Each item in the list contains artifact name, artifact state, and a tooltip with the artifact ID. This can be configured by the apicurio.explorer.name checkbox in the extension settings. Another thing to note is that the number of artifacts shown here is currently limited to 50 by default. This can be changed by the apicurio.search.limit setting, but is definitely an opportunity for future improvement. In the top right menu of this view, users can:
- Refresh the views
- Search for an artifact
- Add a new artifact
Apicurio Versions Explorer shows a list of versions for a selected artifact. Each item in the list contains an artifact version, and its creation time. In the top right menu of this view, users can:
- Reverse the order of versions in the view
- Add a new artifact version
- Delete the entire artifact
Apicurio Metas Explorer shows a list of metadata for a selected artifact or an artifact version. What metadata is shown depends on which item is selected in one of the above views. This view contains a simple key-value list of metadata, except in the case of properties, which is a map that can be expanded. In the top right menu of this view, users can:
- Change editable metadata
- Change artifact version state. This is presented as a separate operation in the UI.

OpenAPI Content Preview

You can view the contents of an artifact version by hovering over an item in the Apicurio Versions Explorer, and clicking on the Open icon. This will download the version content and opens it as an unsaved file in a new editor tab. The extension will try to determine the path to your active workspace, and use it for the location of the unsaved file, but you can always click File > Save as… to change the file name and location.

Apicurio Registry Explorer extension supports integration with Swagger Viewer extension. This enables users to preview OpenAPI content stored in Apicurio Registry in the form of a generated documentation. To install Swagger Viewer, go to the quick open dialog with ctrl+P, and execute the following command:

ext install Arjun.swagger-viewer

then, when you open an OpenAPI file, Apicurio Registry Explorer will automatically launch a Swagger Preview tab for the given schema.

Upcoming Features

While there is a lot of useful functionality in the current version (1.0.3), the extension is still in early stages of development. Here is a list of several features that we would like to see implemented:

Authorized access support. Currently, the extension does not support authenticated access to Apicurio Registry. Integration with Keycloak (or any other OIDC provider) is an important feature that is missing at the moment.
UX improvements. There are several improvements that will make the user experience with Apicurio Registry Explorer better, such as:
- Handle large number of artifacts and artifact versions in the view panels. Currently, there is a fixed, but configurable limit.
- Improve the workflow of some actions that require user input, for example, support corrections without starting the process of creating an artifact from the beginning.
- Auto-detection of schema type when creating a new artifact.
Integrated schema editor. As mentioned earlier, the extension supports a simple integration with Swagger Preview. However, it’s static and only works for OpenAPI. Apicurio community already has a project that is designed for editing of schemas, Apicurio Studio, which has recently started to receive more development focus. We should take advantage of this and integrate Apicurio Registry Explorer extension with Apicurio Studio editor.
Integration with Apicurio Registry TypeScript client. This is more of an implementation detail. As you may know, there are several client libraries for interacting with Apicurio Registry. For Apicurio Registry version 3.0.0, we are working on a TypeScript SDK, which Apicurio Registry Explorer will be able to use.

Conclusion

Apicurio is a community of users and developers who are passionate about tools that help manage the API lifecycle. Apicurio Registry Explorer extension is an open-source project available on GitHub that a is a part of this effort.

We would be happy to hear about your experience with this extension, and any contributions, suggestions, and issue reports are always welcome. You can reach us on Zulip or on GiHub.

If you would like to contribute, please see the development documentation available on GitHub.

Apicurio Studio 1.0: The Awakening

Eric Wittmann — Fri, 22 Mar 2024 16:00:00 +0000

We have a big announcement to make about Apicurio Studio that I hope some people will be interested in.

Studio has been languishing for quite some time without getting the attention it deserves from us (the maintainers/developers). But that is about to change!

We’re announcing the deprecation of Apicurio Studio 0.2 and the beginning of a new line of development for Apicurio Studio 1.0!

History and Motivation

Apicurio Studio is the first project in the Apicurio open source community, and it was started back in 2016 (this fact makes me feel very old). Studio has been a reasonably popular and helpful project, but was never deeply integrated into Red Hat products (Red Hat sponsors all of the Apicurio projects).

In 2019 the Apicurio Registry project was started, and the majority of our development bandwidth has gone into that project, resulting in two Red Hat products:

Red Hat Integration Service Registry (soon to be rebranded as Red Hat build of Apicurio Registry)
Red Hat OpenShift Service Registry (a fully managed version of Registry)

Between the core project and the fully managed version, we’ve had plenty to do on the Registry project over these last few years. Couple that with the lack of product urgency around Studio, and we’ve found ourselves in the problematic position of having an old/stale codebase that has atrophied.

However, that is now changing with renewed interest in releasing Studio functionality within at least one new Red Hat product (hopefully more will follow) and with the sunsetting/removal of Red Hat OpenShift Service Registry.

Deprecation of Apicurio Studio 0.2

As of the writing of this blog entry, the current (old?) version of Studio is version 0.2.62.Final. The 0.2 line of development is now officially deprecated. We may choose to fix major bugs, but certainly no new development will happen.

As a result, we will be doing the following:

The master branch of the codebase will be deprecated and renamed to 0.2.x
Open issues will be closed where appropriate
The eval version at https://studio.apicur.io/ will be deprecated
An pinned announcement will be added to Zulip
A blog post will be written (this thing you’re reading now!)

Introduction of Apicurio Studio 1.0

As part of the Managed Application Services effort (now deceased) at Red Hat, we created a new version of Apicurio Studio that we will now be releasing as version 1.0. We did this because the Studio 0.2 codebase was not well positioned (technically) to be deployed as a Red Hat managed service. As a result, we took a bunch of the core functionality of Studio 0.2 and either re-implemented or re-used it in our new Studio 1.0 codebase. Note that we did this using an updated technical stack and in a way that also leverages common components developed as part of Apicurio Registry.

However, this sword cuts multiple ways: we have preserved as much of the core functionality as possible in the time we had, we have added new functionality that didn’t exist before, but we have also needed to remove some functionality you might have been relying on.

What is being preserved in 1.0

The following functionality/features have been preserved in some capacity in the new Studio 1.0:

Core OpenAPI and AsyncAPI editing capabilities
Asynchronous collaboration (multiple users can edit the same spec, but not at the same time)
Editing history (although with less granularity)
Create designs from scratch or imported from file/URL
Create designs from template

What has been lost (for now)

The following features did not make it into the new Studio 1.0 reformulation (but may be restored/reimplemented in the future):

Importing reusable data types from other OpenAPI docs
Synchronous collaboration (multiple users editing the same spec at the same time)
Management of user-defined templates
Documentation Preview using ReDoc
User defined / configured validation profiles
Per-user multi-tenancy (all users can see all content for now)
GitHub/GitLab/Bitbucket publishing

What is new in 1.0

The fllowing features did not exist in 0.2 and are new and novel to 1.0:

Support for Avro, Protobuf, and JSON Schema types (simple text editing only)
Show Changes (diff) feature in the editor
Modern technical stack: Quarkus, React, Patternfly 5

Summary and Bottom Line

The main takeaway should be that we are finally getting a chance to put some focus back on Studio. I think in the short term we may be taking a step back in terms of functionality, but what we gain is the ability to better maintain and progress both of our main Apicurio projects: Studio and Registry.

This should allow us to once again make steady functional progress on Studio, fixing issues and implementing new features, in a way we haven’t been able to do for a few years.

I’m looking forward to it! Thanks for your interest, and very possibly your patience.

Resolving a bug in Apicurio Registry KafkaSQL storage upgrade process

Jakub Senko — Thu, 08 Feb 2024 12:00:00 +0000

KafkaSQL is a popular storage option for Apicurio Registry, introduced in version 2.0.0.Final. It has become very reliable, despite its higher complexity compared to the SQL storage option. Sadly, no software is immune to bugs, and KafkaSQL is no exception. On the other hand, bugs can be interesting, and the debugging process can provide a great opportunity to learn about Apicurio Registry and KafkaSQL internals. I’ve described one such bug in my previous blog post, titled Resolving a bug in KafkaSQL storage for Apicurio Registry.

A few days ago [1], an Apicurio Registry user reported that some of their Protobuf artifacts went missing after an upgrade from version 2.4.4.Final to 2.5.5.Final. They are using the KafkaSQL storage option, and after some investigation it turned out that other storage options are not affected. I’ve decided to write this article to explain the details, but if you are using KafkaSQL storage, please first jump to the Summary section to see if you might be at risk, and then read the Mitigation section for a list of actions you should take to mitigate the problem until we release a fixed version.

To help the investigation, the user gave us two important pieces of information:

Firstly, they noticed the following suspicious log message:
```
  2024-01-29T15:24:18.157902803Z 2024-01-29 15:24:18 WARN <_> [io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore] (KSQL Kafka Consumer Thread) update content canonicalHash, no row match contentId 13 contentHash 6f93522739b215fdd88db69fa59c8c2f70d97e195c6813d89ee6e151840fd5b7
```
which pointed me to focus my investigation on the source of that log message, the KafkaSqlProtobufCanonicalizerUpgrader class. This class is responsible for upgrading the canonical content hash of Protobuf artifacts to the correct value. This is needed because we have updated the algorithm we use to compute the hash. Because this class is specific to Protobuf, runs during an upgrade process, and only for KafkaSQL storage, it is our prime suspect.
Secondly, the user was able to provide us with a dump of the kafkasql-journal topic, which was very helpful in discovering how the data went missing [2].

Before I explain what happened, let me first review a few relevant features of Apicurio Registry.

Content Hashes

In Apicurio Registry, artifact content can be addressed by several identifiers, most commonly:

group ID, artifact ID, and version triple (GAV)
global ID, which is unique for every artifact version
content ID, which is unique for every piece of content (sequence of bytes).

In some situations, however, these identifiers are not known up front. For example, the user might want to determine whether a piece of content already exists in Apicurio Registry and find out its identifiers, or create a new artifact version only if the content does not already exist. Therefore, it’s useful to have a way of asking Apicurio Registry about the content itself (sequence of bytes).

To support this feature, Apicurio Registry stores a content hash, which is a SHA-256 hash of the content, in addition to the content bytes. Users can then retrieve the information about a piece of content using only its hash.

However, the content hash does not take the structure of the content into account. The following two Protobuf schemas are semantically equivalent:

syntax = "proto3";

message Error {
  string message = 1;
  uint32 error_code = 2;
}

syntax = "proto3";

message Error {  
  string message = 1;  
  // Error code must be between 100 and 999 (inclusive). 
  uint32 error_code = 2;
}

but they do not result in the same SHA-256 hash. To support searching for content that is equivalent to the input, but not necessarily equal, Apicurio Registry also stores a canonical content hash, which is a SHA-256 hash of the content after it has been converted to a canonical form.

The extended list of identifiers for artifact content in Apicurio Registry is therefore:

group ID, artifact ID, and version triple (GAV)
global ID
content ID
content hash
canonical content hash.

The support for content hashes has evolved across Apicurio Registry versions. For example, in version 2.1.2.Final, we have implemented a canonicalizer for Protobuf artifacts. Before this version, the content hash and canonical content hash were computed using the same algorithm. These kinds of changes require careful consideration and implementation of a special process that executes during Apicurio Registry version upgrades. In the case of the KafkaSQL storage option, this upgrade process is performed by the KafkaSqlProtobufCanonicalizerUpgrader class.

Artifact References

Apicurio Registry supports many schema types, and some of them allow specifying references to other schemas. For example, Protobuf supports an import statement:

syntax = "proto3";

import "google/protobuf/any.proto";

message Error {
  string message = 1;
  repeated google.protobuf.Any details = 2;
}

and JSON Schema has the $ref keyword:

{
  "$id": "https://example.com/schemas/error",
  "type": "object",
  "properties": {
    "message": {
      "type": "string"
    },
    "details": {
      "type": "array",
      "items": {
        "$ref": "/schemas/error_detail"
      }
    }
  }
}

We have implemented initial support for artifact references in Apicurio Registry version 2.2.2.Final. This feature allows users to specify a mapping between reference name and another artifact version that it points to.

As an example, let’s consider the Protobuf schema above. The import statement value google/protobuf/any.proto is the reference name. If we have already registered the Protobuf file that describes google.protobuf.Any under group ID = default, artifact ID = any.proto, and version = 2, we can then register a Protobuf file that references it by:

Making the following POST request to apis/registry/v2/groups/default/artifacts:

{
  "content": "syntax = \"proto3\";\n\nimport \"google/protobuf/any.proto\";\n\n(...)",
  "references": [
    {
      "name": "google/protobuf/any.proto",
      "groupId": "default",
      "artifactId": "any.proto",
      "version": "2"
    }
  ]
}

with a special content type header Content-Type: application/create.extended+json, or

using Apicurio Registry client library:

// io.apicurio:apicurio-registry-client:2.5.8.Final
var meta = client.createArtifact("default", "error.proto", "syntax = \"proto3\";\n\nimport \"google/protobuf/any.proto\";\n\n(...)", List.of(    
    ArtifactReference.builder()
      .name("google/protobuf/any.proto")
      .groupId("default")
      .artifactId("any.proto")
      .version("2")
      .build()
));

Content Hashes and Artifact References

As a result of implementing support for artifact references, we wanted to consider two pieces of artifact content that have the same sequence of bytes, but different references as not equivalent. Therefore, we had to change how content hashes are computed. Before version 2.4.2.Final, the following legacy algorithm was used:

content_hash = sha256(content)
canonical_content_hash = sha256(canonicalize(content))

that has changed to the current algorithm:

content_hash = sha256(content ++ references)
canonical_content_hash = sha256(canonicalize(content, dependencies) ++ references)

Similarly to the situation when Protobuf canonicalizer was implemented, these changes also require careful implementation of an upgrade process. The process is currently performed by the ReferencesContentHashUpgrader and ReferencesCanonicalContentHashUpgrader classes. However, they have not been added in the same Apicurio Registry version. More detailed version timeline is described later.

How KafkaSQL Stores Artifact Content

KafkaSQL storage consists of two main components. Every Apicurio Registry instance has its own in-memory H2 database, and connects to a shared Kafka topic named kafkasql-journal. This topic is then used to replicate the state of each in-memory database across instances, and to provide persistence.

When Apicurio Registry is restarted, it consumes all messages in the topic to load the persisted data and bring the instance into sync with any other active Apicurio Registry instances. For more details, see the Resolving a bug in KafkaSQL storage for Apicurio Registry blog post.

For this bug analysis, we will focus on the Kafka messages responsible for replicating artifact content data.

When new content is inserted into Apicurio Registry, KafkaSQL storage produces a new content message on the kafkasql-journal topic. The message has the following structure [3]:

key = {content_id, content_hash} // partition_key = content_hash
value = {operation, canonical_content_hash, content, references}

where operation is one of:

CREATE, which is used when the content is inserted into Apicurio Registry,
UPDATE, which is currently only used by KafkaSqlProtobufCanonicalizerUpgrader, or
IMPORT, which is used by the export-import feature.

Structure of the content message key is important for determining which Kafka messages can be safely compacted. When Kafka log compaction runs, only the last message with a given key is preserved, and previous messages are deleted. This has the advantage of reducing the size of the topic by removing stale messages, but has the potential of causing problems in case of a bug, as we’ll see later. Apicurio Registry automatically creates the kafkasql-journal topic if it does not already exist, and it configures the topic with log compaction enabled by default.

Version History

Because we suspect that the issue is caused by a bug in the upgrade process for KafkaSQL storage, specifically the KafkaSqlProtobufCanonicalizerUpgrader, we need to analyze Apicurio Registry versions that contain relevant changes. This helps us understand how the bug was introduced and which versions are affected.

Version 2.1.2.Final

In this version, we implemented the content canonicalizer for Protobuf artifacts [4]. Before this change, canonical content hash for Protobuf artifacts was the same as content hash:

content_hash = sha256(content)
canonical_content_hash = sha256(content)

Protobuf content hashes are now computed using the following algorithm:

content_hash = sha256(content)
canonical_content_hash = sha256(canonicalize(content))

Let’s call these legacy content hash and legacy canonical content hash to differentiate between the current hash algorithm implemented in version 2.4.2.Final.

Version 2.1.3.Final

In this version, KafkaSqlProtobufCanonicalizerUpgrader is implemented [5], which is responsible for computing legacy canonical content hash for Protobuf artifacts. The upgrader is executed for both SQL and KafkaSQL storage options.

Version 2.2.2.Final

In this version, artifact references are implemented [6]. The algorithm for computing content hashes does not change.

Version 2.4.2.Final

This version changes how content hashes are computed [7], in order to take artifact references into account:

content_hash = sha256(content ++ references)
canonical_content_hash = sha256(canonicalize(content, dependencies) ++ references)

Let’s call these current content hash and current canonical content hash to differentiate between the legacy hash algorithm implemented in version 2.1.2.Final for Protobuf artifacts.

By mistake, KafkaSqlProtobufCanonicalizerUpgrader was not updated, so it still attempts to change the canonical content hash to the legacy value during Apicurio Registry restart.

ReferencesContentHashUpgrader is implemented to upgrade the legacy content hash to the current content hash in the in-memory database. It’s not executed for KafkaSQL storage option by mistake.

Version 2.4.3.Final

User reports an issue related to content hashes in KafkaSQL. This is fixed in the next version.

This version also fixes a bug in KafkaSqlProtobufCanonicalizerUpgrader [8], that resulted in an SQL error when the canonical content hash change was being applied:

2024-02-07 13:54:26 DEBUG <> [io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlSink] (KSQL Kafka Consumer Thread) Registry exception detected: io.apicurio.registry.storage.impl.sql.jdb.RuntimeSqlException: org.h2.jdbc.JdbcSQLDataException: Parameter "#4" is not set [90012-214]

resulting in the operation failing, and in-memory database containing the same canonical content hash as before.

Version: 2.4.4.Final

ReferencesCanonicalHashUpgrader is implemented [9] to upgrade the legacy canonical content hash to the current canonical content hash in the in-memory database.

Both References* upgraders are now executed for the KafkaSQL storage option.

Hypothesis

When examining the Kafka topic dump [10], I’ve noticed that for some content IDs, in this case content ID = 2, the only message present is:

{
  "offset": 10,
  "key": "\u0002{\"tenantId\":\"_\",\"contentHash\":\"b5f276336ecbb160556c114fe22e7cebd8353f0bc11127eb0f69a4aad32648ea\",\"contentId\":2}",
  "payload": "\u0002\u0002\u0000\u0000\u0000@59e286281876629c2715b06c8ef294a1d4a713f5e4249d7a3e386bb734f7db90\u0000\u0000\u0000\u0000\u0000\u0000\u0000V[{\"groupId\":null,\"artifactId\":\"any\",\"version\":\"1\",\"name\":\"google/protobuf/any.proto\"}]"
}

which is missing the content bytes (the four bytes \u0000\u0000\u0000\u0000 encode the length of the content, which is zero). This means that the previous content messages must have been compacted. How is the message above produced? It is produced by KafkaSqlProtobufCanonicalizerUpgrader.

We now have the information I needed to formulate a hypothesis about what happened. There are two related bugs:

KafkaSqlProtobufCanonicalizerUpgrader was not updated to use the current canonical content hash algorithm, so it always attempts to change any Protobuf content that uses current canonical content hash back to legacy canonical content hash after each Apicurio Registry restart. Since the hashes are equal for Protobuf content without references, only Protobuf content with references is affected.
KafkaSqlProtobufCanonicalizerUpgrader performs the hash change using the following Kafka content message:
```
key = {content_id = (unchanged), content_hash = (unchanged)}
value = {operation = UPDATE, canonical_content_hash = (legacy value), content = null, references = (unchanged)}
```
Which has the same content message key as the Kafka message that was used to insert the content. The previous content message might become compacted and the content lost.

Reproducer Scenario

This is a more detailed list of steps that I think should reproduce the problem:

User starts with Apicurio Registry version 2.2.2.Final-2.4.1.Final (inclusive), using KafkaSQL storage.
User adds some Protobuf artifacts with references, let’s call them set A. The content hashes for these artifacts are computed using the legacy algorithm. Therefore, the content message keys contain legacy content hash, and content message values contain legacy canonical content hash. The in-memory database contains the same.
User upgrades Apicurio Registry to version 2.4.2.Final-2.4.3.Final (inclusive).
1. The KafkaSqlProtobufCanonicalizerUpgrader is executed. For each Protobuf artifact in the database, it computes the legacy canonical content hash, which is equal to the canonical content hash of set A, so nothing changes. The Reference* upgraders are not executed for KafkaSQL in this version.
User adds some more Protobuf artifacts with references, let’s call them set B. The content hashes for these artifacts are computed using the current algorithm. Therefore, the content message keys contain current content hash, and content message values contain current canonical content hash.
User restarts their Apicurio Registry instance.
1. KafkaSqlProtobufCanonicalizerUpgrader is executed again. It’s not necessary to upgrade Apicurio Registry to a later version, because KafkaSQL upgraders are executed every time Apicurio Registry starts. For each Protobuf artifact in the database, it computes the legacy canonical content hash, which equals to the canonical content hash of set A, but not set B. Therefore, it changes the canonical content hash of set B to the legacy value, by sending a content message value that is missing the content by mistake.
Some time later, Kafka compaction runs, which effectively deletes the content of set B. The missing data is only noticed after another restart of Apicurio Registry, when the in-memory database is loaded from the Kafka topic.
User upgrades Apicurio Registry to version 2.4.4.Final-2.5.8.Final (inclusive) [11].
1. The KafkaSqlProtobufCanonicalizerUpgrader is executed again.
2. The ReferencesContentHashUpgrader is executed, which updates any legacy content hash it finds to current content hash, but only in the in-memory database. This currently does not cause any problems, but since content hash is part of the content message key, there is a risk of a potential bug in the future. See Additional bug risk section for more details.
3. The ReferencesCanonicalHashUpgrader is executed, which updates any legacy canonical content hash it finds to current canonical content hash, but only in the in-memory database. This currently does not cause any problems.

Reproducer

To confirm my hypothesis, I will follow the reproducer scenario above. We can skip steps 1, 2, and 3.1., starting with an empty Apicurio Registry version 2.4.2.Final. We will configure the kafkasql-journal topic with an aggressive log compaction [12].

We will add two Protobuf artifacts, any.proto, and error.proto. The second artifact references the first one.

The following shows abbreviated contents of the CONTENT table in the in-memory database:

+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+
|TENANTID|CONTENTID|CANONICALHASH                                                   |CONTENTHASH                                                     |CONTENT                            |ARTIFACTREFERENCES                                                                    |
+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+
|_       |1        |628ccecee09bcc0bcfdc95adce7041e998bb82a7377c600d7dbbba4f70dcae04|bfd6f785126a85ec181c41bc29680b394627c32b7ef0c3e4cef3ced86e463e7d|syntax = "proto3";                 |null                                                                                  |
|        |         |                                                                |                                                                |                                   |                                                                                      |
|        |         |                                                                |                                                                |package google.protobuf;           |                                                                                      |
|        |         |                                                                |                                                                |[...]                              |                                                                                      |
+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+
|_       |2        |b5a276ddf3fc1724dbe206cbc6da60adf8e32af5613ef0fe52fb1dde8da6b67a|b5f276336ecbb160556c114fe22e7cebd8353f0bc11127eb0f69a4aad32648ea|syntax = "proto3";                 |[{"groupId":null,"artifactId":"any","version":"1","name":"google/protobuf/any.proto"}]|
|        |         |                                                                |                                                                |                                   |                                                                                      |
|        |         |                                                                |                                                                |import "google/protobuf/any.proto";|                                                                                      |
|        |         |                                                                |                                                                |                                   |                                                                                      |
|        |         |                                                                |                                                                |message Error {                    |                                                                                      |
|        |         |                                                                |                                                                |[...]                              |                                                                                      |
+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+

Notice that the canonical content hash value in the second row is b5a276ddf3fc1724dbe206cbc6da60adf8e32af5613ef0fe52fb1dde8da6b67a. This is the correct value that was computed using the current hash algorithm when the content was inserted.

We restart Apicurio Registry, and see the following message in the log, which means that the KafkaSqlProtobufCanonicalizerUpgrader has been executed:

2024-02-07 12:29:47 DEBUG <> [io.apicurio.registry.storage.impl.kafkasql.KafkaSqlProtobufCanonicalizerUpgrader] (KSQL Kafka Consumer Thread) Protobuf content canonicalHash outdated value detected, updating contentId 2

The CONTENT table now shows that the KafkaSqlProtobufCanonicalizerUpgrader updated the canonical content hash to the incorrect legacy value:

+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+
|TENANTID|CONTENTID|CANONICALHASH                                                   |CONTENTHASH                                                     |CONTENT                            |ARTIFACTREFERENCES                                                                    |
+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+
|_       |1        |628ccecee09bcc0bcfdc95adce7041e998bb82a7377c600d7dbbba4f70dcae04|bfd6f785126a85ec181c41bc29680b394627c32b7ef0c3e4cef3ced86e463e7d|syntax = "proto3";                 |null                                                                                  |
|        |         |                                                                |                                                                |                                   |                                                                                      |
|        |         |                                                                |                                                                |package google.protobuf;           |                                                                                      |
|        |         |                                                                |                                                                |[...]                              |                                                                                      |
+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+
|_       |2        |59e286281876629c2715b06c8ef294a1d4a713f5e4249d7a3e386bb734f7db90|b5f276336ecbb160556c114fe22e7cebd8353f0bc11127eb0f69a4aad32648ea|syntax = "proto3";                 |[{"groupId":null,"artifactId":"any","version":"1","name":"google/protobuf/any.proto"}]|
|        |         |                                                                |                                                                |                                   |                                                                                      |
|        |         |                                                                |                                                                |import "google/protobuf/any.proto";|                                                                                      |
|        |         |                                                                |                                                                |                                   |                                                                                      |
|        |         |                                                                |                                                                |message Error {                    |                                                                                      |
|        |         |                                                                |                                                                |[...]                              |                                                                                      |
+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+

At this moment, the kafkasql-journal topic contains the following messages:

[
  {
    "offset": 0,
    "key": "\u0000{\"bootstrapId\":\"c613911c-2f32-4a2a-9d79-fffa321f9548\"}",
    "payload": null
  },
  {
    "offset": 1,
    "key": "\t{\"tenantId\":\"_\",\"uuid\":\"c6758a4e-1c7c-4c68-ad17-0dc99f1cb374\"}",
    "payload": "\t{\"action\":1}"
  },
  {
    "offset": 2,
    "key": "\u0002{\"tenantId\":\"_\",\"contentHash\":\"bfd6f785126a85ec181c41bc29680b394627c32b7ef0c3e4cef3ced86e463e7d\",\"contentId\":1}",
    "payload": "\u0002\u0001\u0000\u0000\u0000@628ccecee09bcc0bcfdc95adce7041e998bb82a7377c600d7dbbba4f70dcae04\u0000\u0000\u0001\u0087syntax = \"proto3\";\n\npackage google.protobuf;\n\noption go_package = \"google.golang.org/protobuf/types/known/anypb\";\noption java_package = \"com.google.protobuf\";\noption java_outer_classname = \"AnyProto\";\noption java_multiple_files = true;\noption objc_class_prefix = \"GPB\";\noption csharp_namespace = \"Google.Protobuf.WellKnownTypes\";\n\nmessage Any {\n\n  string type_url = 1;\n\n  bytes value = 2;\n}\n\u0000\u0000\u0000\u0000"
  },
  {
    "offset": 3,
    "key": "\b{\"tenantId\":\"_\",\"uuid\":\"ea335c14-7f1b-40ee-8beb-94ef0b02890d\"}",
    "payload": "\b{\"action\":1}"
  },
  {
    "offset": 4,
    "key": "\u0003{\"tenantId\":\"_\",\"artifactId\":\"any\",\"uuid\":\"ecb0a768-617c-4b3d-af1a-3d340c4f2f44\"}",
    "payload": "\u0003{\"action\":1,\"metaData\":{},\"globalId\":1,\"artifactType\":\"PROTOBUF\",\"contentHash\":\"bfd6f785126a85ec181c41bc29680b394627c32b7ef0c3e4cef3ced86e463e7d\",\"createdBy\":\"\",\"createdOn\":1707303731203}"
  },
  {
    "offset": 5,
    "key": "\t{\"tenantId\":\"_\",\"uuid\":\"35a22e8e-f826-4568-8bd8-9d8722a5f25f\"}",
    "payload": "\t{\"action\":1}"
  },
  {
    "offset": 6,
    "key": "\u0002{\"tenantId\":\"_\",\"contentHash\":\"b5f276336ecbb160556c114fe22e7cebd8353f0bc11127eb0f69a4aad32648ea\",\"contentId\":2}",
    "payload": "\u0002\u0001\u0000\u0000\u0000@b5a276ddf3fc1724dbe206cbc6da60adf8e32af5613ef0fe52fb1dde8da6b67a\u0000\u0000\u0000\u008dsyntax = \"proto3\";\n\nimport \"google/protobuf/any.proto\";\n\nmessage Error {\n  string message = 1;\n  repeated google.protobuf.Any details = 2;\n}\n\u0000\u0000\u0000V[{\"groupId\":null,\"artifactId\":\"any\",\"version\":\"1\",\"name\":\"google/protobuf/any.proto\"}]"
  },
  {
    "offset": 7,
    "key": "\b{\"tenantId\":\"_\",\"uuid\":\"e57510fa-f204-4277-bb5d-7424987737af\"}",
    "payload": "\b{\"action\":1}"
  },
  {
    "offset": 8,
    "key": "\u0003{\"tenantId\":\"_\",\"artifactId\":\"error\",\"uuid\":\"b51ca026-3c39-44fd-b22b-6b29478d4477\"}",
    "payload": "\u0003{\"action\":1,\"metaData\":{},\"globalId\":2,\"artifactType\":\"PROTOBUF\",\"contentHash\":\"b5f276336ecbb160556c114fe22e7cebd8353f0bc11127eb0f69a4aad32648ea\",\"createdBy\":\"\",\"createdOn\":1707303731454}"
  },
  {
    "offset": 9,
    "key": "\u0000{\"bootstrapId\":\"d58884b2-6552-4330-9cd3-f3f294a54e4b\"}",
    "payload": null
  },
  {
    "offset": 10,
    "key": "\u0002{\"tenantId\":\"_\",\"contentHash\":\"b5f276336ecbb160556c114fe22e7cebd8353f0bc11127eb0f69a4aad32648ea\",\"contentId\":2}",
    "payload": "\u0002\u0002\u0000\u0000\u0000@59e286281876629c2715b06c8ef294a1d4a713f5e4249d7a3e386bb734f7db90\u0000\u0000\u0000\u0000\u0000\u0000\u0000V[{\"groupId\":null,\"artifactId\":\"any\",\"version\":\"1\",\"name\":\"google/protobuf/any.proto\"}]"
  }
]

Notice that message 9 is a bootstrap message marking the point where Apicurio Registry was restarted. Message 10 is produced by the KafkaSqlProtobufCanonicalizerUpgrader.

We will wait for Kafka log compaction to run, and observe the contents of the kafkasql-journal topic again:

[
  {
    "offset": 1,
    "key": "\t{\"tenantId\":\"_\",\"uuid\":\"4c039a9a-62b5-459a-b6da-fc771345de31\"}",
    "payload": "\t{\"action\":1}"
  },
  {
    "offset": 2,
    "key": "\u0002{\"tenantId\":\"_\",\"contentHash\":\"bfd6f785126a85ec181c41bc29680b394627c32b7ef0c3e4cef3ced86e463e7d\",\"contentId\":1}",
    "payload": "\u0002\u0001\u0000\u0000\u0000@628ccecee09bcc0bcfdc95adce7041e998bb82a7377c600d7dbbba4f70dcae04\u0000\u0000\u0001\u0087syntax = \"proto3\";\n\npackage google.protobuf;\n\noption go_package = \"google.golang.org/protobuf/types/known/anypb\";\noption java_package = \"com.google.protobuf\";\noption java_outer_classname = \"AnyProto\";\noption java_multiple_files = true;\noption objc_class_prefix = \"GPB\";\noption csharp_namespace = \"Google.Protobuf.WellKnownTypes\";\n\nmessage Any {\n\n  string type_url = 1;\n\n  bytes value = 2;\n}\n\u0000\u0000\u0000\u0000"
  },
  {
    "offset": 3,
    "key": "\b{\"tenantId\":\"_\",\"uuid\":\"59fb1cc2-5779-4e39-b089-3272ad8c3b10\"}",
    "payload": "\b{\"action\":1}"
  },
  {
    "offset": 4,
    "key": "\u0003{\"tenantId\":\"_\",\"artifactId\":\"any\",\"uuid\":\"7a77789f-d882-4578-b965-1b5050335b05\"}",
    "payload": "\u0003{\"action\":1,\"metaData\":{},\"globalId\":1,\"artifactType\":\"PROTOBUF\",\"contentHash\":\"bfd6f785126a85ec181c41bc29680b394627c32b7ef0c3e4cef3ced86e463e7d\",\"createdBy\":\"\",\"createdOn\":1707305198467}"
  },
  {
    "offset": 5,
    "key": "\t{\"tenantId\":\"_\",\"uuid\":\"33711009-be5c-4491-86e2-d085ba9a39f5\"}",
    "payload": "\t{\"action\":1}"
  },
  {
    "offset": 7,
    "key": "\b{\"tenantId\":\"_\",\"uuid\":\"3dd98144-215b-45cf-92f0-e34f09a08a4e\"}",
    "payload": "\b{\"action\":1}"
  },
  {
    "offset": 8,
    "key": "\u0003{\"tenantId\":\"_\",\"artifactId\":\"error\",\"uuid\":\"07cc6743-ce36-4258-9fbd-c9c6ec8028d9\"}",
    "payload": "\u0003{\"action\":1,\"metaData\":{},\"globalId\":2,\"artifactType\":\"PROTOBUF\",\"contentHash\":\"b5f276336ecbb160556c114fe22e7cebd8353f0bc11127eb0f69a4aad32648ea\",\"createdBy\":\"\",\"createdOn\":1707305199277}"
  },
  {
    "offset": 10,
    "key": "\u0002{\"tenantId\":\"_\",\"contentHash\":\"b5f276336ecbb160556c114fe22e7cebd8353f0bc11127eb0f69a4aad32648ea\",\"contentId\":2}",
    "payload": "\u0002\u0002\u0000\u0000\u0000@59e286281876629c2715b06c8ef294a1d4a713f5e4249d7a3e386bb734f7db90\u0000\u0000\u0000\u0000\u0000\u0000\u0000V[{\"groupId\":null,\"artifactId\":\"any\",\"version\":\"1\",\"name\":\"google/protobuf/any.proto\"}]"
  }
]

Notice that the bootstrap messages at offset 0 and 9 have been deleted, because they are empty (this is on purpose). However, message 6 has also been deleted because it shares the key with message 10. This was not intended and is the bug we need to fix.

After another restart, the Protobuf content with references is missing in the CONTENT table:

+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+
|TENANTID|CONTENTID|CANONICALHASH                                                   |CONTENTHASH                                                     |CONTENT                            |ARTIFACTREFERENCES                                                                    |
+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+
|_       |1        |628ccecee09bcc0bcfdc95adce7041e998bb82a7377c600d7dbbba4f70dcae04|bfd6f785126a85ec181c41bc29680b394627c32b7ef0c3e4cef3ced86e463e7d|syntax = "proto3";                 |null                                                                                  |
|        |         |                                                                |                                                                |                                   |                                                                                      |
|        |         |                                                                |                                                                |package google.protobuf;           |                                                                                      |
|        |         |                                                                |                                                                |[...]                              |                                                                                      |
+--------+---------+----------------------------------------------------------------+----------------------------------------------------------------+-----------------------------------+--------------------------------------------------------------------------------------+

∎

Additional Bug Risk

Because the ReferencesContentHashUpgrader does not write the updated content hashes to the Kafka topic in step 7.2., there might be a mismatch between the content hash value in the kafkasql-journal topic and the in-memory database. If Apicurio Registry produced a new content message for a given content ID (for example, to upgrade the canonical content hash, content, or references), the content message key that would be used is different from the one present in the topic, because the key would contain a content hash that has changed.

Currently, the only operation that updates content via content message is the one that KafkaSqlProtobufCanonicalizerUpgrader uses. This operation is always executed before the Reference* upgraders, so it does not cause issues, but it is still a potential bug. To fix this, we would need to change the format of the content message key.

Summary

The problem is caused by two bugs in the Apicurio Registry upgrade process that is specific to the KafkaSQL storage option.

You are affected if:

You are using Apicurio Registry 2.4.2.Final or later, and
you are using KafkaSQL storage, and
you have created Protobuf artifacts with references, and
you restart the Apicurio Registry instance (for example during version upgrade).

Symptoms are different based on your Kafka log compaction configuration:

If Kafka compaction runs, some Protobuf artifacts with references might be deleted, and disappear from Apicurio Registry after restart.
If Kafka compaction is disabled, data is not lost, but some Protobuf artifacts with references might have their canonical content hash updated to the legacy version. If you are on version 2.4.2.Final, because of a different bug, the hash change fails to be applied to the in-memory database, so the side effect is avoided at the cost of Reference* upgraders not being present in this version.

Mitigation

If you are affected, you can mitigate the bug using the following steps:

Back up your Apicurio Registry data by either backing up the kafkasql-journal topic directly, or by using the export-import feature.
Disable compaction of kafkasql-journal topic.
Avoid using features that rely on canonical content hash, such as:
- Searching for artifacts using canonical content (POST to /search/artifacts with canonical=true query parameter),
- Searching artifact metadata by canonical content (POST to /groups/{groupId}/artifacts/{artifactId}/meta with canonical=true), or
- Skipping updating of an artifact on duplicate content (POST to /groups/{groupId}/artifacts, with ifExists=RETURN_OR_UPDATE and canonical=true);
with Protobuf artifacts that have references, unless your Apicurio Registry version is 2.4.2.Final.
Do not restart or upgrade Apicurio Registry (if possible), until a fixed version is released [11], and then upgrade directly to the fixed version, skipping any intermediate versions.

Solution

We have released a fixed version 2.5.9.Final🎉. Please update your deployment directly to this version.

Notes

[1] As of the writing of this article, latest Apicurio Registry version is 2.5.8.Final. [Go back]

[2] If you come across a bug that you suspect is related to KafkaSQL, I have written a guide on how to generate a kafkasql-journal topic dump that could help us during investigation. [Go back]

[3] Ignoring multitenancy for simplicity. [Go back]

[4] Commit 42b30af2c4, PR #1973. [Go back]

[5] Commit 03654bb492, PR #2005. [Go back]

[6] Commit dd885c3aae, PR #2366. [Go back]

[7] Commit 9628c43b45, PR #3228. [Go back]

[8] Commit 6dac98fc8a, PR #3300. [Go back]

[9] Commit 67065bfc53, PR #3423. [Go back]

[10] This is an illustrative example, not the actual topic dump provided by the user. [Go back]

[11] We have released a fixed version 2.5.9.Final🎉. [Go back]

[12] To do this, we create the topic manually before starting Apicurio Registry, using the following command:

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
    --create --topic kafkasql-journal --partitions 1 --replication-factor 1 \
    --config min.cleanable.dirty.ratio=0.000001 --config cleanup.policy=compact \
    --config segment.ms=100 --config delete.retention.ms=100

If the topic is small, we might also need to add additional messages to the topic, since active segments are not eligible for compaction. [Go back]

Thank you for using and supporting Apicurio projects! As always, if you have any suggestions or encounter any problem, feel free to contact the team by filing an issue in GitHub.

Securing Apicurio Registry 2.4.x using Azure Entra ID

Carles Arnal Castello — Thu, 13 Jul 2023 12:00:00 +0000

Hey everyone, in recent Apicurio Registry versions we’ve introduced support for securing the application using different OpenID Connect (OIDC) servers, and not just Keycloak. In this blog post, I will explain how to configure the application to secure it using Microsoft Azure Entra ID.

Core Features

In this blog post, we will cover the following:

Authentication based on Azure Entra ID - optionally protect Apicurio Registry so that the registry API and web console require users to authenticate (OAuth Authorization Code Flow supported)

Azure Entra ID Configuration

Authentication based on Azure AD

In order to secure Apicurio Registry you will need a valid directory in Azure and some specific configuration (described below). Essentially, you must register the Apicurio Registry application in the Azure portal. Log in to the Azure Portal. You can use your personal email address or your GitHub account to log in. After logging in, navigate to the Azure AD control panel by using the menu in the top-left corner. It should look like this:

Let’s dig into the Azure AD configuration for your Apicurio Registry deployment. Select Manage > App registrations in the menu on the left. Select New registration, and fill in the form. Enter apicurio-registry-example as the application name. We’ll also allow users from any organizational directory to log in.

Important: Register the host of the server hosting your Apicurio Registry application as a redirect URI. As part of the logon process, users will be redirected from our application to Azure AD for authentication. We want to send them back to our application afterwards. Azure AD will not allow any redirect URLs that are not registered. We’ll come back to this later.

Click Register. You should now be able to find the app registration by selecting Manage > App registrations in the menu on the left.

We can now find the parameters we need to set up Apicurio Registry with Azure AD OIDC. Click apicurio-registry-example to display its details:

Select Manage > Authentication to configure the application with the redirect URLs and token as follows:

To configure Apicurio Registry with Azure AD, you must configure the following environment variables in Apicurio Registry using the Azure AD Application ID and the Azure AD Directory ID, along with some Apicurio Registry-specific configuration:

REGISTRY_AUTH_ENABLED=true
KEYCLOAK_API_CLIENT_ID=459569e9-c5f7-410a-a6e7-8db28d7e3647 #Azure AD > Admin > App registrations > Your app > Application (client) ID
REGISTRY_UI_AUTH_TYPE=oidc
REGISTRY_AUTH_URL_CONFIGURED=https://login.microsoftonline.com/6f9ef45b-456d-49e3-b5ba-1f6fe4c0fb78/v2.0 #Azure AD > Admin > App registrations > Your app > Directory (tenant) ID
REGISTRY_OIDC_UI_CLIENT_ID=459569e9-c5f7-410a-a6e7-8db28d7e3647 #Azure AD > Admin > App registrations > Your app > Application (client) ID
CORS_ALLOWED_ORIGINS=https://test-registry.com #The host for your Apicurio Registry deployment
REGISTRY_OIDC_UI_REDIRECT_URL=https://test-registry.com/ui/ #The host for your Apicurio Registry console
ROLE_BASED_AUTHZ_ENABLED=true
QUARKUS_OIDC_ROLES_ROLE_CLAIM_PATH=roles

Role based authorization

To enable roles, you must set the ROLE_BASED_AUTHZ_ENABLED property to true.

Of course, if you enable roles in Apicurio Registry, you must also create them in Azure AD. You must create them as Application roles. The default roles expected by Apicurio Registry are sr-admin, sr-developer, and sr-readonly.

Another extremely important configuration is QUARKUS_OIDC_ROLES_ROLE_CLAIM_PATH=true because Azure AD stores the roles in a claim called roles.

Client credentials authentication

Apart from having users accessing the UI, it is quite frequent to have the requirement of doing machine to machine authentication, especially in the context of something like Apache Kafka. To do this, you can configure your setup to use the Client Credentials Grant using Entra ID. This can be achieved with the following steps:

Create a new app registration that represents your client application. You can leave everything blank in the creation screen besides the name.
In the Overview tab of your new application registration, create a new secret.
Add a new API Permission to your client application, and, inside this option, add the role you want to use. This role should have been created in the last section as an Application role in the Registry server app registration. This is done so the role is sent within the token.
In your client identifying the application and in the one identifying the Registry server, in the API permissions section, you must grant admin consent for your directory. Note that it may take a few seconds for this to take effect.
Finally, configure your producer as follows, but note that the values here are just examples:

Properties props = new Properties();

// Configure kafka settings
props.putIfAbsent(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, SERVERS);
props.putIfAbsent(ProducerConfig.CLIENT_ID_CONFIG, "Producer-" + TOPIC_NAME);
props.putIfAbsent(ProducerConfig.ACKS_CONFIG, "all");
props.putIfAbsent(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.putIfAbsent(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, AvroKafkaSerializer.class.getName());
props.putIfAbsent(SerdeConfig.REGISTRY_URL, REGISTRY_URL);
props.putIfAbsent(SerdeConfig.AUTO_REGISTER_ARTIFACT, Boolean.TRUE);

//Security related properties
props.putIfAbsent(SerdeConfig.AUTH_CLIENT_SECRET, "CLIENT_SECRET"); //Here you must use the client secret of your client application created in step 2 above.
props.putIfAbsent(SerdeConfig.AUTH_CLIENT_ID, "c9715ca2-de14-4484-8b39-95f10050b099"); //This is the client ID of your client application.
props.putIfAbsent(SerdeConfig.AUTH_TOKEN_ENDPOINT, "https://login.microsoftonline.com/6f9ef45b-456d-49e3-b5ba-2f6fe4c0fb78/oauth2/v2.0/token"); //This is the token endpoint, the UUID is your tenant ID from Entra ID.
props.putIfAbsent(SchemaResolverConfig.AUTH_CLIENT_SCOPE, "api://frr5bc25-b280-4def-91ff-179d6c3b9d4f/.default"); //Finally, this is the expected value structure for the scope. The UUID is the application ID of the app registration that represents your Apicurio Registry server.

// Create the Kafka producer
Producer<Object, Object> producer = new KafkaProducer<>(props);

The result of the code snippet above is a Kafka producer that uses the AvroKafkaSerializer provided by the Apicurio Registry team, integrated with Entra ID.

Last, but not least, we still have a lot of things to do! As always, you can see the stuff we’re tracking by viewing the GitHub issues for the project.

If you find bugs or want to request a new feature, that’s a great place to start!

Resolving a bug in KafkaSQL storage for Apicurio Registry

Jakub Senko — Thu, 09 Dec 2021 12:00:00 +0000

One of the major changes introduced in Apicurio Registry 2.x were improvements in the supported storage options. We have kept the possibility of using both PostgreSQL database and Kafka to store your data, but the internals of the storage options have been improved, and in some ways simplified.

We have received great feedback from the users of the new KafkaSQL storage option, but recently several users have reported an interesting bug, that seems to only affect users with the KafkaSQL persistence enabled.

The following error appears when a user attempts to update or create a new artifact:

The first time that one of the users saw this error, they were able to resolve it by deleting the Kafka topic where the Apicurio Registry stores its data. However, the error started appearing again after some time. We have investigated the problem, found the cause, and would like to take this opportunity to share the details with the Apicurio community.

In this blog post, we will take a look at the KafkaSQL storage option in more detail. We’ll describe the bug and the solution, and provide a temporary workaround for users until they are able to upgrade to the patched version 2.1.4.Final.

KafkaSQL Storage

A major use-case for Apicurio Registry is to manage and distribute schemas for messages in Kafka topics. This means that a lot of our users have a Kafka cluster available, so supporting it as a storage option has been a priority for us.

KafkaSQL storage shares common logic with the standard SQL storage option, but does not use a central SQL database to store Apicurio Registry data. Instead, each node uses a local SQL database. To keep the replicas in sync, each change in one of the local databases is replicated to the other nodes in the cluster using a Kafka topic. This makes the read operations on the data straightforward, and allows us to keep the commonality with the SQL persistence option.

The design of the Kafka messages closely follows the tables that are used both in the local SQL database, and the database for the standard SQL storage.

Each Kafka message has a key and a corresponding value. Kafka itself is agnostic about their content, and treats them as a byte array. In the KafkaSQL storage, they are represented by objects that are serialized into a JSON String, prepended with marker bytes used to recognize message type.

In general, each message value class has a corresponding key class. Classes for the keys are mainly responsible for determining the partition that the message is going to reside in, the tenant information in case of a multi-tenant deployment, and identifying information for the object that the message value represents. For example ArtifactVersionKey contains group ID, artifact ID and version identifiers for the given artifact version object. Choice of the partition key is important for maintaining consistency of the data between nodes, because ordering of the messages is only guaranteed in the same partition.

Classes for the values contain a field with the operation that the message represents, which includes CREATE, UPDATE, and DELETE. The other fields carry the rest of the information, such as (for example) the editable metadata for an artifact version.

KafkaSQL storage option workflow diagram

When a change to a piece of data occurs, the originating node creates and sends the message to the Kafka topic, and blocks the user thread until the operation is processed. Its local database is not modified directly. Each node only applies changes that are read form the Kafka topic. Therefore, the user thread is unblocked only after the original message is consumed from the topic by the node. This ensures that in case of a composite operation, messages are always processed in the correct order, and critically in the same order for each node.

For example, when adding a new artifact version, the user may store a new content that is not already present in the storage. Therefore, the operation usually consists of two messages that have to be kept in the correct order, or the artifact version wouldn’t be able to reference the new content.

An important thing to note is that when a node is restarted, and its local SQL database is empty, simply reading all the messages from the kafkasql-journal will replicate the data that is already present on other live nodes. This works even when the entire Apicurio Registry application is stopped and restarted, as long as the data in the kafkasql-journal topic remains persisted and backed up.

Investigation

In order to investigate the issue, we have to first attempt to reproduce the issue on a local Apicurio Registry deployment with KafkaSQL storage. The following is a list of steps we have performed.

First, a local Kafka instance must be downloaded, and then we can run it:

Start Zookeper:

bin/zookeeper-server-start.sh config/zookeeper.properties

and the start Kafka:

bin/kafka-server-start.sh config/server.properties

We have been able to get the contents of the Kafka topics from one the users who reported the issue. They used Kafkacat tool to export the data. An example output of Kafkacat looks like this:

{"topic":"kafkasql-journal","partition":0,"offset":0,"tstype":"create","ts":1637230546039,"headers":["req","ef7fd699-2311-4ba6-9d5a-f9087815eb83"],"key":"\u0000{\"bootstrapId\":\"24240819-0738-49a7-8ab2-2b971176b0e1\"}","payload":null}
{"topic":"kafkasql-journal","partition":0,"offset":1,"tstype":"create","ts":1637230571121,"headers":["req","9520eb8f-23a4-40f4-99a0-9d917406f700"],"key":"\u0001{\"tenantId\":\"_\",\"ruleType\":\"VALIDITY\"}","payload":"\u0001{\"action\":\"CREATE\",\"config\":{\"configuration\":\"FULL\"}}"}
{"topic":"kafkasql-journal","partition":0,"offset":3,"tstype":"create","ts":1637230581768,"headers":["req","2a897033-111e-4fdf-a3cc-02941a9db84f"],"key":"\u0002{\"tenantId\":\"_\",\"contentHash\":\"377072c58f3999d1e42455c1cf2563d4e5a367c3ee66a5be41ac376902183bc4\",\"contentId\":1}","payload":"\u0002\u0001\u0000\u0000\u0000@9e15d06c0aa184746bc092841d0a0efd0ce276c5d36873bf471482e3401e92ab\u0000\u0000\u0000�{\n  \"type\": \"record\",\n  \"name\": \"User1\",\n  \"namespace\": \"foo\",\n  \"fields\": [\n    { \"name\": \"active1\", \"type\": [\"null\", \"boolean\"] }\n  ]\n}"}
{"topic":"kafkasql-journal","partition":0,"offset":5,"tstype":"create","ts":1637230581843,"headers":["req","2dd94bfb-34a1-4c23-b93c-54242f94aba4"],"key":"\u0003{\"tenantId\":\"_\",\"artifactId\":\"d0c00bec-3e4a-4386-ad67-b23b5a91ce96\",\"uuid\":\"6b50bb64-65ba-4ed8-beaf-deea49a7e610\"}","payload":"\u0003{\"action\":\"CREATE\",\"metaData\":{\"name\":\"User1\"},\"globalId\":1,\"artifactType\":\"AVRO\",\"contentHash\":\"377072c58f3999d1e42455c1cf2563d4e5a367c3ee66a5be41ac376902183bc4\",\"createdBy\":\"\",\"createdOn\":1637230581817}"}
{"topic":"kafkasql-journal","partition":0,"offset":6,"tstype":"create","ts":1637230593970,"headers":["req","9a34b683-02af-4360-8617-1fc5f17c6509"],"key":"\t{\"tenantId\":\"_\"}","payload":"\t{\"action\":\"CREATE\"}"}
{"topic":"kafkasql-journal","partition":0,"offset":7,"tstype":"create","ts":1637230593993,"headers":["req","3d3beae5-6561-4b42-9731-c7c6c74ce5c7"],"key":"\u0002{\"tenantId\":\"_\",\"contentHash\":\"edc351831a60539d904157020074ef1043868247fb8641e5cf4356407245dd32\",\"contentId\":2}","payload":"\u0002\u0001\u0000\u0000\u0000@e200bb1898f5ccee20f649e98097306fb72a8b32ecf9db64261d13387730dc95\u0000\u0000\u0000�{\n  \"type\": \"record\",\n  \"name\": \"User2\",\n  \"namespace\": \"foo\",\n  \"fields\": [\n    { \"name\": \"active2\", \"type\": [\"null\", \"boolean\"] }\n  ]\n}"}
{"topic":"kafkasql-journal","partition":0,"offset":8,"tstype":"create","ts":1637230594011,"headers":["req","6485fb14-313d-42c2-891b-5b80a6f7b244"],"key":"\b{\"tenantId\":\"_\"}","payload":"\b{\"action\":\"CREATE\"}"}
{"topic":"kafkasql-journal","partition":0,"offset":9,"tstype":"create","ts":1637230594026,"headers":["req","8f807fb9-d581-4976-8750-b3d3d80df333"],"key":"\u0003{\"tenantId\":\"_\",\"artifactId\":\"0b9c15b5-54a5-40d3-9b75-433070d6ad99\",\"uuid\":\"b2c6e59d-b9b7-4be1-84a9-9d0dd907466a\"}","payload":"\u0003{\"action\":\"CREATE\",\"metaData\":{\"name\":\"User2\"},\"globalId\":2,\"artifactType\":\"AVRO\",\"contentHash\":\"edc351831a60539d904157020074ef1043868247fb8641e5cf4356407245dd32\",\"createdBy\":\"\",\"createdOn\":1637230594011}"}
{"topic":"kafkasql-journal","partition":0,"offset":10,"tstype":"create","ts":1637230608757,"headers":["req","6ec25f20-eb62-4914-8e8b-97e9c7fa29c7"],"key":"\t{\"tenantId\":\"_\"}","payload":"\t{\"action\":\"CREATE\"}"}
{"topic":"kafkasql-journal","partition":0,"offset":11,"tstype":"create","ts":1637230608773,"headers":["req","b68d11b9-67fd-427a-a167-e5c438c8334f"],"key":"\u0002{\"tenantId\":\"_\",\"contentHash\":\"b11088caecbc3fcd799b2a289f85502b6c0d00d8113b0fb51789528863e6c0e1\",\"contentId\":3}","payload":"\u0002\u0001\u0000\u0000\u0000@06f0fb80396e5f02a4a5f3d96ba5da6af35ca33ec1251c58698170283d0dd37a\u0000\u0000\u0000�{\n  \"type\": \"record\",\n  \"name\": \"User3\",\n  \"namespace\": \"foo\",\n  \"fields\": [\n    { \"name\": \"active3\", \"type\": [\"null\", \"boolean\"] }\n  ]\n}"}
{"topic":"kafkasql-journal","partition":0,"offset":12,"tstype":"create","ts":1637230608792,"headers":["req","e69c6a1b-7df1-40cd-a640-ac9b3716c687"],"key":"\b{\"tenantId\":\"_\"}","payload":"\b{\"action\":\"CREATE\"}"}
{"topic":"kafkasql-journal","partition":0,"offset":13,"tstype":"create","ts":1637230608807,"headers":["req","6875bbc9-92fe-4572-accc-7dab2bdcd84e"],"key":"\u0003{\"tenantId\":\"_\",\"artifactId\":\"447e73bf-e86b-480b-8e45-3222c0980bc9\",\"uuid\":\"a5fa1ba0-f40a-4591-91c6-a6cb422f2f4a\"}","payload":"\u0003{\"action\":\"CREATE\",\"metaData\":{\"name\":\"User3\"},\"globalId\":3,\"artifactType\":\"AVRO\",\"contentHash\":\"b11088caecbc3fcd799b2a289f85502b6c0d00d8113b0fb51789528863e6c0e1\",\"createdBy\":\"\",\"createdOn\":1637230608791}"}

You can get a Kafka topic dump using Kafkacat directly, however it’s possible to run it via Docker like this:

export PWD=`pwd`
docker run --rm --network=host --volume $PWD:/data edenhill/kafkacat:1.5.0 kafkacat \
  -C -b localhost:9092 -t kafkasql-journal -Z -D \\n -e -J

We can now import the sample topic dump into our local Kafka cluster. However, we were not able to do it using Kafkacat directly, because the data is wrapped in an envelope with additional metadata. Therefore, we can use one of the example projects from Apicurio Registry Examples to create a simple application to preprocess and load the data into the kafkasql-journal topic. The preprocessing mostly consists of removing the metadata envelope and producing clean messages. As a note, it is important to use correct encoding, StandardCharsets.US_ASCII instead of UTF when reading the dump file. Now, we can run the local Registry with the data the user has provided.

First, we build Apicurio Registry with KafkaSQL storage:

mvn clean install -Pprod -Pkafkasql -DskipTests

and then create an environment variable representing the project root:

export PROJECT=`pwd`

To see how the data are being processed and stored, we need to have access to the local database. By default, it’s an in-memory h2 database, but we’ll use a file-based configuration, to which we can easily connect from an IDE. To achieve this, we will override the default value of the quarkus.datasource.jdbc.url property.

Run Apicurio Registry instance with the debugger enabled:

java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 \
     -Dquarkus.http.port=8180 \
     -Dquarkus.datasource.jdbc.url="jdbc:h2:file:$PROJECT/registry_database;MV_STORE=FALSE;AUTO_SERVER=TRUE" \
     -jar ./storage/kafkasql/target/apicurio-registry-storage-kafkasql-2.1.3-SNAPSHOT-runner.jar | tee log.txt

Now we can connect both the debugger and database client to the application from our IDE.

You can use the following URL, with sa/sa as the username/password options:

echo "jdbc:h2:file:$PROJECT/registry_database;MV_STORE=FALSE;AUTO_SERVER=TRUE"

After loading the data from the Kafka topic into the local database and setting up debugging, we can inspect the contents of the ARTIFACTS and then the CONTENT tables:

+---------+--------------+-------------------------------------+-----+----------+---------------------------+-------+
|TENANTID |GROUPID       |ARTIFACTID                           |TYPE |CREATEDBY |CREATEDON                  |LATEST |
+---------+--------------+-------------------------------------+-----+----------+---------------------------+-------+
|_        |__$GROUPID$__ |d0c00bec-3e4a-4386-ad67-b23b5a91ce96 |AVRO |          |2021-11-18 11:16:21.817000 |1      |
+---------+--------------+-------------------------------------+-----+----------+---------------------------+-------+
|_        |__$GROUPID$__ |0b9c15b5-54a5-40d3-9b75-433070d6ad99 |AVRO |          |2021-11-18 11:16:34.011000 |2      |
+---------+--------------+-------------------------------------+-----+----------+---------------------------+-------+
|_        |__$GROUPID$__ |447e73bf-e86b-480b-8e45-3222c0980bc9 |AVRO |          |2021-11-18 11:16:48.791000 |3      |
+---------+--------------+-------------------------------------+-----+----------+---------------------------+-------+

ARTIFACTS table

+---------+----------+----------------+----------------+------------------+
|TENANTID |CONTENTID |CANONICALHASH   |CONTENTHASH     |CONTENT           |
+---------+----------+----------------+----------------+------------------+
|_        |1         |9e15d06c0aa1... |377072c58f39... |{                 |
|         |          |                |                |"type": "record", |
|         |          |                |                |"name": "User1",  |
|         |          |                |                | ...              |
|         |          |                |                |}                 |
+---------+----------+----------------+----------------+------------------+
|_        |2         |e200bb1898f5... |edc351831a60... |{                 |
|         |          |                |                |"type": "record", |
|         |          |                |                |"name": "User2",  |
|         |          |                |                | ...              |
|         |          |                |                |}                 |
+---------+----------+----------------+----------------+------------------+
|_        |3         |06f0fb80396e... |b11088caecbc... |{                 |
|         |          |                |                |"type": "record", |
|         |          |                |                |"name": "User3",  |
|         |          |                |                | ...              |
|         |          |                |                |}                 |
+---------+----------+----------------+-----------------------------------+

CONTENT table

The data has been loaded as expected, and the database content looks good, so we can try to add another artifact:

{
  "type": "record",
  "name": "User4",
  "namespace": "foo",
  "fields": [
    { "name": "active4", "type": ["null", "boolean"] }
  ]
}

Sample Avro schema

and indeed we observe the error we have been attempting to reproduce. The details of the exception are as follows, which corresponds to the reported issue:

Caused by: io.apicurio.registry.storage.impl.sql.jdb.RuntimeSqlException: SQL error: Expected only one result row but got none.
	at io.apicurio.registry.storage.impl.sql.jdb.MappedQueryImpl.one(MappedQueryImpl.java:66)
	at io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore.lambda$contentIdFromHash$7(KafkaSqlStore.java:172)
	at io.apicurio.registry.storage.impl.sql.HandleFactory.withHandle(HandleFactory.java:44)
	at io.apicurio.registry.storage.impl.sql.HandleFactory.withHandleNoException(HandleFactory.java:52)
	...

What does it mean? When creating a new artifact, we use multiple messages to perform the necessary operations. In the case of adding a new artifact, we first make sure that the content of the artifact is stored, before writing meta-data to the ARTIFACTS table. This ensures that we do not store multiple copies of the same content. The error tells us, that when we perform the second step, the content we expected is not present in the database, even though the message responsible for adding it must have been processed before the error occurs.

There may be multiple reasons why this could happen, but after some investigation we noticed a suspicious content of the SEQUENCES table:

+--------+---------+-----+
|TENANTID|NAME     |VALUE|
+--------+---------+-----+
|_       |contentId|2    |
|_       |globalId |2    |
+--------+---------+-----+

The data is wrong, since after we loaded the original data, there were 3 artifacts in the CONTENT table. This means that the next contentId in the sequence is 3, but content with that ID already exists, and it belongs to an artifact that has been created earlier.

To find the answer to the corrupted data, we can take a look at the topic dump we have originally used and notice that the messages for some offsets are missing. Furthermore, it seems that those messages are exactly those that cause the sequence numbers to be incremented.

After some investigation, we determined that the missing messages may have been deleted by a feature of Kafka, called Topic Log Compaction. Our hypothesis therefore is, that the messages that should have be kept have been accidentally deleted in this way. To find out more, we will describe how Kafka Topic Log Compaction works.

Kafka Topic Log Compaction

Topic log compaction is a feature in Apache Kafka that allows users to conserve storage space for the topic, by automatically removing old messages in the log. It works by considering a sequence of messages that share the same key, and only keeping the last message in the sequence.

This, however, requires an assumption that any message for the given key contains all state that the application wants to keep. This is not necessarily true for each application, or for all messages, therefore it is important to carefully configure this feature, and disable it if your application depends on keeping every message, even if the key is repeated.

An additional feature is to also remove messages that have a null value, allowing you to delete the whole sequence for the given key when the topic is compacted. These kinds of messages are called tombstone messages. In Apicurio Registry, we use tombstone messages when a piece of data, such as an artifact, is deleted. The reason is that the past operations on that piece of data do not need to be replicated to other nodes, or preserved after restart.

The most important Kafka properties for configuring log compaction are as follows:

cleanup.policy=compact enables log compaction and activates the following properties.
min.cleanable.dirty.ratio this ratio bounds the maximum space wasted in the log by duplicates, until the log compactor will attempt to clean the log.
min.compaction.lag.ms the minimum amount of time that has to pass since a message is written before it can be compacted.
delete.retention.ms The amount of time to retain delete tombstone markers. If a consumer starts to read the topic from the beginning, they need to finish reading in this time if it wants to make sure it sees the markers.

If you are using the Strimzi Operator to provision and manage your Kafka cluster on Kubernetes, you can provide these configuration options on a cluster or topic level (Kafka or KafkaTopic CRD) in the config section:

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: kafkasql-journal
  labels:
    strimzi.io/cluster: my-cluster
  namespace: registry-example-kafkasql
spec:
  partitions: 2
  replicas: 1
  config:
    cleanup.policy: compact
    min.cleanable.dirty.ratio: 0.5

Strimzi Kafka CRD example

Note: If you have not created the topic manually, Apicurio Registry will create the kafkasql-journal automatically. Make sure you create a CDR for the kafkasql-journal topic, so you can manage it with Strimzi.

Reproducer

Let’s create a procedure to attempt to reproduce the issue and confirm our hypothesis. We are assuming that the Kafka cluster and the Apicurio Registry executable is already prepared by following steps in the previous sections.

Stop the Registry process.
Remove kafkasql-journal topic, if it exists.

Create the topic manually with aggressive log compaction settings:

 bin/kafka-topics.sh --bootstrap-server localhost:9092 \
    --create --topic kafkasql-journal --partitions 1 --replication-factor 1 \
    --config min.cleanable.dirty.ratio=0.000001 --config cleanup.policy=compact \
    --config segment.ms=100 --config delete.retention.ms=100

Create at least 3 AVRO schemas, for example:

 {
  "type": "record",
  "name": "User1",
  "namespace": "foo",
  "fields": [
    { "name": "active1", "type": ["null", "boolean"] }
  ]
 }

Sample Avro schema

Restart the registry, so it will reload messages from the topic into its local database. Remove the database file if you are using the debug settings from earlier.
Add another AVRO schema to the Registry and observe the error.

Solution

In order to fix the bug, we have to prevent messages, which control the database sequence, from being compacted. We are already doing this for other types of messages, by making sure that the key value is always unique. We can do this by including a sufficiently random nonce, such as an UUID, in the message key.

The following is an example ContentId message, that increments a sequence for the CONTENT table, before the fix has been implemented:

{
  "topic": "kafkasql-journal",
  "partition": 0,
  "offset": 1,
  "tstype": "create",
  "ts": 1638894723320,
  "headers": [
    "req",
    "9dff3fb8-7a1d-41b0-9083-684d9fbfa619"
  ],
  "key": "\t{\"tenantId\":\"_\"}",
  "payload": "\t{\"action\":\"CREATE\"}"
}

And the same message after the bugfix:

{
  "topic": "kafkasql-journal",
  "partition": 0,
  "offset": 1,
  "tstype": "create",
  "ts": 1638894723320,
  "headers": [
    "req",
    "9dff3fb8-7a1d-41b0-9083-684d9fbfa619"
  ],
  "key": "\t{\"tenantId\":\"_\",\"uuid\":\"6875bbc9-92fe-4572-accc-7dab2bdcd84e\"}",
  "payload": "\t{\"action\":\"CREATE\"}"
}

Temporary Resolution for Older Apicurio Registry Versions

This issue has already been fixed in Apicurio Registry 2.1.4.Final. If you are not able to upgrade, follow this procedure:

In order to make sure you are not affected by this bug, you need to disable log compaction for the kafkasql-journal topic in your cluster. First, check if it is enabled:

bin/kafka-configs.sh --describe --bootstrap-server localhost:9092 --topic kafkasql-journal 

If you see cleanup.policy=compact, you are affected. To disable log compaction, you need to switch to the delete cleanup policy. However, in case of Apicurio Registry, deleting old messages will result in data corruption. Therefore, you must also prevent messages to be actually deleted, by setting retention.bytes=-1 and retention.ms=-1 (or their server-wide equivalents log.retention.bytes and log.retention.ms). For example:

bin/kafka-configs.sh --alter --bootstrap-server localhost:9092 --topic kafkasql-journal \
  --add-config cleanup.policy=delete \
  --add-config retention.bytes=-1 \
  --add-config retention.ms=-1

If you are using Strimzi, you can apply these settings in the config section of the KafkaTopic or Kafka CRD:

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: kafkasql-journal
  labels:
    strimzi.io/cluster: my-cluster
  namespace: registry-example-kafkasql
spec:
  partitions: 2
  replicas: 1
  config:
    cleanup.policy: delete
    retention.bytes: -1
    retention.ms: -1

Strimzi KafkaTopic CRD example

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: registry-example-kafkasql
spec:
  kafka:
    config:
      # ...
      log.cleanup.policy: delete
      log.retention.bytes: -1
      log.retention.ms: -1
    version: 2.7.0
    # ...

Strimzi Kafka CRD example

However, this configuration change does not fix the existing corrupted data in the database. Luckily, we can use the export-import feature of the Apicurio Registry to force a reset of the database sequences, so the error no longer occurs. This will keep your existing data intact.

To perform this reset, create an export file from an empty Apicurio Registry, using the following example command:

curl localhost:8180/apis/registry/v2/admin/export --output export.zip

and import it into the Apicurio Registry instance you want to fix:

export PWD=`pwd`
curl -X POST -H 'content-type: application/zip' --data-binary "@$PWD/export.zip" localhost:8180/apis/registry/v2/admin/import

Note: If you do not want to provide your own export file, you can use this example one, generated from an empty Apicurio Registry instance, version 2.1.3.Final.

Thank you for using and supporting Apicurio projects! As always, if you have any suggestions or encounter any problem, feel free to contact the team by filing an issue in GitHub

Apicurio Registry 2.0.0.Final Migration Guide

Fabian Martinez Gonzalez — Tue, 01 Jun 2021 12:00:00 +0000

Apicurio Registry 2.x is our latest and greatest release to date. It has plenty of new features but it also comes with some breaking changes from the previous 1.3.x release. In this post we will explore the process to migrate data from Apicurio Registry 1.3.x to Apicurio Registry 2.x.

Because of the breaking changes between 1.3.x and 2.x, there is no automatic upgrade and instead a migration process is required. But don’t worry! We promise the process is not hard.

Breaking changes

There are three major changes in this release that must be taken into account to proceed with a migration.

New storage options

This impacts your current Apicurio Registry deployment. In Apicurio Registry 1.3.x we had 3 storage options (streams, jpa, and infinispan). Now in Apicurio Registry 2.x we dropped all of them but provided new and improved alternatives that we believe will be better for the project in the long term. These new storages allow for more robust and performant deployments while at the same time being more maintainable. The new storage options are sql and kafkasql). There is a third one mem that is not suitable for production workloads. This document will not cover the process of deploying Apicurio Registry 2.x, you can find more details in the Apicurio Registry user documentation

New REST API

We changed the REST API layout to ensure better long term maintainability and because we refactored our API to have support for artifact groupings. Apicurio Registry still supports the original REST API (now known as v1) and the various compatibility APIs we already supported (for example, Confluent and IBM schema registry APIs). Additionally, Apicurio Registry now also implements the Schema Registry spec provided in the CNCF Cloud Events spec.

Refactored SerDes libraries

The Serializer and Deserializer classes are now available as three different Maven modules (one for each data format). This makes it possible to use just the one you want without pulling in a bunch of transitive dependencies that you don’t care about.

Migrating to v2.x

The migration from Apicurio Registry 1.3.x to 2.x requires you to move the data in your existing registry to a new registry. You must also review your applications that interact with the registry and update their configuration to meet the new requirements.

Data migration

The process of data migration to Apicurio Registry 2.x will require you to export all the data from your 1.3.x deployment and import it to the new 2.x based deployment.

Data migration is a critical step in the migration to Apicurio Registry 2.x if you are using the registry as a schema registry for Kafka applications, because every Kafka message carries the global identifier for the schema stored in Apicurio Registry. It is critical that this identifier is kept between registry upgrades via data migration.

For that purpose, Apicurio Registry 2.x provides a new import/export API to bulk import or export all the data from your registry deployment. This API guarantees that all the identifiers are kept when importing the data from your existing registry. The export API works by downloading a custom zip file containing all the information of your artifacts. The import API accepts that zip file, loading all of the artifacts into the registry in a single batch.

Apicurio Registry 1.3.x does not provide an import/export API, but with the 2.x release we provided an export tool compatible with Apicurio Registry 1.3.x that allows you to export a compatible zip file, which can be imported to your 2.x registry. This special export tool uses common existing APIs to export all of the content in the registry. It is less performant than the built-in export API we have introduced in 2.x, and so should only be used when exporting from a 1.3.x registry.

This export tool is available from the Apicurio Registry GitHub release. It is a java application meant to be invoked from the command line.

The migration steps for moving all your data from one version to another:

Prerequisites

Have running Apicurio Registry instances for both the version 1.3.x server (exporting from) and the new 2.x server (importing to).

Procedure

Export all the data from Apicurio Registry 1.3.x using the exportV1 tool. This will generate a registry-export.zip file in your current directory.
```
java -jar apicurio-registry-utils-exportV1-2.0.0.Final.jar http://old-registry.my-company.com/api
```

Import the zip file to Apicurio Registry 2.x using the import API. You can find more details in the Apicurio Registry user documentation

curl -X POST "http://new-registry.my-company.com/apis/registry/v2/admin/import" \
  -H "Accept: application/json" -H "Content-Type: application/zip" \
  --data-binary @registry-export.zip

Check that all the artifacts are now imported to the new 2.x registry. Run these two commands and compare the count field.

curl "http://old-registry.my-company.com/api/search/artifacts"

curl "http://new-registry.my-company.com/apis/registry/v2/search/artifacts"

Application migration

Registry SerDes libraries

If you are using the Apicurio Registry SerDes libraries, you must change the Maven dependencies you are using, because we have repackaged these classes.

In Apicurio Registry 1.3.x, the SerDes libraries were provided all with just one Maven dependency:

<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-utils-serde</artifactId>
    <version>1.3.2.Final</version>
</dependency>

In Apicurio Registry 2.x, this has been changed, and the SerDes libraries have been split into three Maven dependencies, one for each supported data format: avro, protobuf, and json schema.

<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-serdes-avro-serde</artifactId>
    <version>2.0.0.Final</version>
</dependency>
<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-serdes-protobuf-serde</artifactId>
    <version>2.0.0.Final</version>
</dependency>
<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-serdes-jsonschema-serde</artifactId>
    <version>2.0.0.Final</version>
</dependency>

You must take into account these changes to Maven dependencies, and update your applications accordingly.

The refactored SerDes libraries also include some changes in the configuration properties. You can find more details in the Apicurio Registry user documentation. You can find plenty of examples in the Apicurio Registry 2.0.0.Final SerDes libraries.

Registry URL configuration

But the most important thing you must change is the registry URL configuration, which you must change from pointing to the existing API path to the new one. For example:

Existing registry API path:

props.putIfAbsent(AbstractKafkaSerDe.REGISTRY_URL_CONFIG_PARAM, "http://localhost:8080/api");

New registry API path:

props.putIfAbsent(SerdeConfig.REGISTRY_URL, "http://localhost:8080/apis/registry/v2");

These are the most important things to take into account when doing the update to Apicurio Registry 2.0.0.Final. The process is not complex but there are some critical steps that, thanks to our APIs and tooling, should be no problem.

As always, if you have any suggestions or encounter any problem feel free to contact the team by filling an issue in GitHub

Securing Apicurio Registry 2.0 using Keycloak

Carles Arnal Castello — Fri, 28 May 2021 12:00:00 +0000

Hey everyone, in Apicurio Registry 2.0.0. Finally we’ve introduced support for some new security features. In this blog post, I will explain these concepts and show you how to configure them.

Core features

Essentially in this blog post, we will cover the following:

Authentication based on Keycloak - optionally protect the registry such that the registry API and web console require users to authenticate (BASIC and OAuth supported)
Role-based authorization - when authentication is enabled, you can also enable this feature, and users must have at least one role of sr-admin, sr-developer, sr-readonly
Creator-only authorization option - option to prevent changes to artifacts unless the authenticated user is the user who originally created the artifact

Keycloak configuration

Authentication based on Keycloak

To secure Apicurio Registry you will need a Keycloak server up and running along with some specific configuration. Essentially, you must create a Keycloak realm, two clients, and some roles (if you want to use roles). By default, Apicurio Registry expects the name of the realm to be apicurio-local. When security is enabled, Apicurio Registry will expect two Keycloak clients. By default, these clients should be named registry-api and apicurio-registry (but don’t worry, you will be able to customize everything by using environment variables!) Below you have the required configuration for the clients.

API client configuration

Web console client configuration

You will need to pay attention to the configuration - especially to the access types and the configuration URLs. For the registry-api client, make sure to configure it as bearer-only, in the latest keycloak releases you will have to uncheck all the Authentication flows in the capability config. For the apicurio-registry client, you will need to configure it as public, to allow users to connect to your UI. You will also need to configure Web Origins, you can use + to allow Web Origins coming from the redirect URL.

Now, you will need to tweak the Apicurio Registry configuration to get this to work.

The first property that you must set is AUTH_ENABLED. You will need to set it to true to enable the integration. The second property you need to set is the Keycloak server URL: KEYCLOAK_URL. We provide some sensible defaults for most of the configuration, but please, note that you can customize most of them by using environment variables.

Role-based authorization

To enable roles, you need to set the ROLE_BASED_AUTHZ_ENABLED property to true.

Of course, if you enable roles in Apicurio Registry, you must also create them in Keycloak as realm roles. The default roles expected by Apicurio Registry are sr-admin, sr-developer, and sr-readonly. When you enable roles, this will result in some HTTP paths and operations available just for certain roles. You have the full configuration in the image above, but there are some operations like those around rules, logs levels, and so on that are restricted to the admin role. The developer role can perform most of the operations, and the read-only role can read or search artifacts.

Creator-only authorization option

Finally, you also have the option to restrict certain operations just to the creator of the artifact. To do so, you can set the environment variable OWNER_ONLY_AUTHZ_ENABLED to true.

Last, but not least, we still have a lot of things to do! As always, you can see the stuff we’re tracking by viewing the GitHub issues for the project.

If you find bugs or want to request a new feature, that’s a great place to start!

Beyond OpenShift: Making Apicurio Registry Work with Any OAuth Provider via Dex

Where We Left Off

Enter Dex: A Federated OIDC Bridge

What This Unlocks

How to Set It Up

Deploy Dex

Configure a Connector

OpenShift

LDAP

GitHub

Microsoft / Azure AD

SAML 2.0

Deploy Apicurio Registry

Verify Everything Works

Machine-to-Machine (M2M) Access and Kafka SerDes

Dex client_credentials Support (Unreleased as of v2.45.1)

The Catch: No Groups in Client Credentials Tokens

What Actually Works for Kafka SerDes

Upcoming Fix: Groups on Static Clients

When to Use Keycloak Instead

Group-to-Role Mapping

Production Considerations

Troubleshooting

The Bigger Picture

Integrating Apicurio Registry with OpenShift's Built-in OAuth Server

The Challenge

The Solution: verify-access-token-with-user-info

Step-by-Step Setup

1. Install the Apicurio Registry Operator

2. Grant Anonymous Access to the JWKS Endpoint

3. Create an OAuthClient Resource

4. Create OpenShift Groups for Role-Based Authorization

5. Deploy the Registry

6. Verify It Works

What Works and What Doesn’t

Why Principal Identity Doesn’t Work

Troubleshooting Tips

Alternative: /oauth/info Endpoint

Wrapping Up

Apicurio Registry goes AI-Native: Introducing Agent Registry, MCP Server, and LLM artifact support

Why Apicurio Registry for AI?

What’s new: Phase 1 complete

1. Agent card Registry (A2A protocol support)

2. MCP server integration

3. LLM artifact types

See it in Action: Multi-Agent context chaining

What’s coming next

Get started today

A natural evolution

Externalizing Authentication and Authorization in Apicurio Registry

Why externalize auth?

What did we build?

How does it work?

Show me the code!

What about CORS?

OPA policies in action

What about owner-based authorization?

Is this production ready?

Try it out!

OpenTelemetry Distributed Tracing with Apicurio Registry

What is OpenTelemetry?

OpenTelemetry in Apicurio Registry

Architecture Overview

Trace Propagation

Kafka Message Propagation

Registry Correlation

Complete Example

Quick Start with Docker Compose

1. Clone and build

2. Start the infrastructure

3. Access the services

4. Generate traces

5. View traces in Jaeger

Understanding Trace Visualization and Performance Analysis

Reading the Trace

Using Traces for Performance Optimization

Identify Bottlenecks

Measure Latency Breakdown

Detect Caching Effects

Compare Across Requests

Dex `client_credentials` Support (Unreleased as of v2.45.1)

1. Create your configuration file (`raml-config.json`):

2. Create your JavaScript implementation (`raml-artifact-type.js`)