Migrating an existing Apicurio Registry deployment

This chapter explains how to migrate an existing Apicurio Registry deployment from version 2.x to version 3.x.

Migration from Apicurio Registry 2.x to 3.x

Apicurio Registry 3.x includes new features with breaking changes from the previous Apicurio Registry 2.x release. This section describes the major changes between Apicurio Registry 2.x and version 3.x, along with the best approach to migrating.

Because of the breaking changes in 3.x, there is no automatic upgrade and a migration process is required. This involves exporting the data in your existing registry to a new registry. You must also review your existing registry client applications and update their configurations to meet the new requirements.

When migrating to version 3.x, you must take the following major changes into account:

Data storage changes

All of the storage options from 2.x continue to be supported in 3.x, but any data currently in either the Kafka topic or in the database is no longer compatible with 3.x. This means that, in order to migrate from 2.x to 3.x, you will need to use the Export feature of Apicurio Registry to create a .ZIP file containing all of your 2.x data.

The in-memory storage option is also still available, but is not suitable for production workloads.

For details on how to deploy Apicurio Registry 3.x with your chosen storage option, see Installing Apicurio Registry storage on OpenShift.

New v3 REST API

Apicurio Registry 3.x includes a new REST API with support for empty artifacts, artifact branches, group level rules, new search capabilities, and much more. Apicurio Registry still supports version 2.x of the core REST API as well as various compatibility APIs, for example, the Confluent schema registry API v7. However, version 2 of the Core REST API is now deprecated.

All new SDKs replace the Java client

  • The Apicurio Registry Java client classes are no longer available in version 3.x and have been replaced by a new SDK generated using Kiota.

  • In addition to Java, SDKs are now provided for Typescript, Python, and Golang.

  • The Kafka client serializer and deserializer (SerDes) classes are available in version 3.x…​.

Deployment configuration has changed

Apicurio Registry 3.x has changed a number of configuration options and added some new ones. Options you may be using when deploying 2.x are likely still available, but may have been renamed.

Additional resources

Migrating Apicurio Registry data

Migrating data to Apicurio Registry 3.x requires exporting all data from your existing Apicurio Registry 2.x deployment and importing it into the new 3.x deployment. If you are using Apicurio Registry as a schema registry for Kafka applications, data migration is critical because each Kafka message carries the global identifier for the schema stored in Apicurio Registry. This identifier must be preserved during registry data migration.

Both Apicurio Registry 3.x and 2.x provide an API to bulk import/export all data from your registry deployment, which guarantees that all identifiers are kept when importing data from your existing registry. The export API downloads a custom .zip file containing all the information for your artifacts. The import API accepts this .zip and loads all artifacts into the registry in a single batch.

The Apicurio Registry 3.x import feature is able to import .zip contents created from either a 2.x or 3.x deployment.

Prerequisites

  • Running a Apicurio Registry instance of the 2.x server you are exporting from and the 3.x instance you are importing into.

Procedure

  1. Export all the data from Apicurio Registry 2.x using the Export feature, which can be accessed from the UI or the REST API. This generates a all-artifacts.zip file in a directory of your choice.

  2. Import the .zip file into Apicurio Registry 3.x using the import API:

    curl -X POST "http://new-registry.my-company.com/apis/registry/v3/admin/import" \
  -H "Accept: application/json" -H "Content-Type: application/zip" \
  --data-binary @all-artifacts.zip

  3. Check that all the artifacts have been imported into the new 3.x registry by running these commands and comparing the count field:

    curl "http://old-registry.my-company.com/apis/registry/v2/search/artifacts"
    curl "http://new-registry.my-company.com/apis/registry/v3/search/artifacts"
Additional resources

Migrating Apicurio Registry client applications

You must review your existing Apicurio Registry client applications to ensure that the Maven dependencies and Java client configuration meet the new requirements for version 3.x. For example, this includes new Maven dependencies for the Apicurio Registry Java REST client libraries or Kafka client serializer/deserializer (Serdes) libraries. You must also update your Java application configuration with the new registry v3 API path.

Prerequisites

  • Existing Apicurio Registry 2.6.3 Java client application or Kafka client producer and consumer Java applications with SerDes

Procedure

  1. If you are using the Apicurio Registry Java REST client, you must change the Maven dependencies for the Apicurio Registry Java client libraries, which have been repackaged in version 3.x:

    <dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-java-sdk</artifactId>
    <version>3.0.0</version>
</dependency>

  2. In your Java client application, you must change your registry URL configuration, from pointing to the existing v2 API path to the new v3 path. Starting with v3, we use Vertx as the default platform for our rest client, due to this change, a RequestAdapter is required when the client is created. For example:

    public class ClientExample {


     public static void main(String[] args) throws Exception {
         VertXRequestAdapter vertXRequestAdapter = new VertXRequestAdapter(vertx);
         vertXRequestAdapter.setBaseUrl("https://new-registry.my-company.com/apis/registry/v3");
         RegistryClient client = new RegistryClient(vertXRequestAdapter);
    }
}

    You can find more details on the Java client in Managing Apicurio Registry content using a Java client.

  3. If you are using the Apicurio Registry SerDes libraries, you must change the Maven dependencies, which have been repackaged in version 3.x. In Apicurio Registry 2.6.3, the SerDes libraries were provided into three separate Maven modules, while this is still true, the names have been changed:

    <dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-avro-serde-kafka</artifactId>
    <version>3.0.0</version>
</dependency>
<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-jsonschema-serde-kafka</artifactId>
    <version>3.0.0</version>
</dependency>
<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-protobuf-serde-kafka</artifactId>
    <version>3.0.0</version>
</dependency>

    In Apicurio Registry 3.x, the SerDes libraries have been significantly refactored to make them re-usable for other messaging platforms like Apache Pulsar, that’s why the Apache Kafka specific ones have been renamed.

  4. In your Kafka producer and consumer Java applications, you must change your registry URL configuration from pointing to the existing v2 API path to the new v3 path. For example:

    Existing registry v2 API path:

    props.putIfAbsent(SerdeConfig.REGISTRY_URL, "http://new-registry.my-company.com/apis/registry/v3");

    New registry v3 API path:

    props.putIfAbsent(SerdeConfig.REGISTRY_URL, "http://new-registry.my-company.com/apis/registry/v3");

    The refactored SerDes libraries also include other important changes to configuration properties. For more details on SerDes configuration, see Validating schemas using Kafka client serializers/deserializers in Java clients.

Additional resources