Introduction to Apicurio Registry
This chapter introduces Apicurio Registry concepts and features and provides details on the supported artifact types that are stored in the registry:
Apicurio Registry overview
Apicurio Registry is a datastore for sharing standard event schemas and API designs across API and event-driven architectures. You can use Apicurio Registry to decouple the structure of your data from your client applications, and to share and manage your data types and API descriptions at runtime using a REST interface.
For example, client applications can dynamically push or pull the latest schema updates to or from Apicurio Registry at runtime without needing to redeploy. Developer teams can query the registry for existing schemas required for services already deployed in production, and can register new schemas required for new services in development.
You can enable client applications to use schemas and API designs stored in Apicurio Registry by specifying the registry URL in your client application code. For example, the registry can store schemas used to serialize and deserialize messages, which can then be referenced from your client applications to ensure that the messages that they send and receive are compatible with those schemas.
Using Apicurio Registry to decouple your data structure from your applications reduces costs by decreasing overall message size, and creates efficiencies by increasing consistent reuse of schemas and API designs across your organization. Apicurio Registry provides a web console to make it easy for developers and administrators to manage registry content.
In addition, you can configure optional rules to govern the evolution of your registry content. For example, these include rules to ensure that uploaded content is syntactically and semantically valid, or is backwards and forwards compatible with other versions. Any configured rules must pass before new versions can be uploaded to the registry, which ensures that time is not wasted on invalid or incompatible schemas or API designs.
Apicurio Registry capabilities
-
Multiple payload formats for standard event schemas and API specifications
-
Pluggable registry storage options in Apache Kafka or PostgreSQL database
-
Registry content management using a web console, REST API command, Maven plug-in, or Java client
-
Rules for content validation and version compatibility to govern how registry content evolves over time
-
Full Apache Kafka schema registry support, including integration with Kafka Connect for external systems
-
Kafka client serializers/deserializers (Serdes) to validate message types at runtime
-
Cloud-native Quarkus Java runtime for low memory footprint and fast deployment times
-
Compatibility with existing Confluent or IBM schema registry client applications
-
Operator-based installation of Apicurio Registry on OpenShift
-
OpenID Connect (OIDC) authentication using Keycloak
Schema and API artifacts and groups in Apicurio Registry
The items stored in Apicurio Registry, such as event schemas and API designs, are known as registry artifacts. The following shows an example of an Apache Avro schema artifact in JSON format for a simple share price application:
{
"type": "record",
"name": "price",
"namespace": "com.example",
"fields": [
{
"name": "symbol",
"type": "string"
},
{
"name": "price",
"type": "string"
}
]
}
When a schema or API design is added as an artifact in the registry, client applications can then use that schema or API design to validate that the client messages conform to the correct data structure at runtime.
Apicurio Registry supports a wide range of message payload formats for standard event schemas and API specifications. For example, supported formats include Apache Avro, Google Protobuf, GraphQL, AsyncAPI, OpenAPI, and others. For more details, see Apicurio Registry artifact and rule reference.
Schema and API groups
An artifact group is an optional named collection of schema or API artifacts. Each group contains a logically related set of schemas or API designs, typically managed by a single entity, belonging to a particular application or organization.
You can create optional artifact groups when adding your schemas and API designs to organize them in Apicurio Registry. For example, you could create groups to match your development
and production
application environments, or your sales
and engineering
organizations.
Schema and API groups can contain multiple artifact types. For example, you could have Protobuf, Avro, JSON Schema, OpenAPI, and AsyncAPI schema and API artifacts all in the same group.
You can create schema and API artifacts and optional groups using the Apicurio Registry web console, core REST API, Maven plug-in, or Java client application. The following simple example shows using the REST API:
$ curl -X POST -H "Content-type: application/json; artifactType=AVRO" \
-H "X-Registry-ArtifactId: share-price" \
--data '{"type":"record","name":"price","namespace":"com.example", \
"fields":[{"name":"symbol","type":"string"},{"name":"price","type":"string"}]}' \
https://my-registry.example.com/apis/registry/v2/groups/my-group/artifacts
This example adds an Avro schema with an artifact ID of share-price
in an artifact group named my-group
.
Specifying a group is optional when using the Apicurio Registry web console, where a default group is automatically created. When using the v2 REST API or Maven plug-in, you can specify the default group in the API path if you do not want to create a unique group.
|
-
For more details on schemas and groups, see the Cloud Native Computing Foundation (CNCF) Schema Registry API.
Manage content using Apicurio Registry web console
You can use the Apicurio Registry web console to browse and search the schema and API artifacts and optional groups stored in the registry, and to add new schema and API artifacts, groups, and versions. You can search for artifacts by label, name, group, and description. You can view an artifact’s content or its available versions, or download an artifact file locally.
You can also use the web console to configure optional rules for registry content, both globally and for each schema and API artifact. These optional rules for content validation and compatibility are applied when new schema and API artifacts or versions are uploaded to the registry. For more details, see Apicurio Registry artifact and rule reference.
The Apicurio Registry web console is available from the main endpoint of your Apicurio Registry deployment, for example, on http://MY-REGISTRY-URL/ui
.
Registry core REST API overview
Using the Apicurio Registry core REST API, client applications can manage the schema and API artifacts in Apicurio Registry. This API provides create, read, update, and delete operations for:
- Artifacts
-
Manage schema and API artifacts stored in the registry. You can also manage the lifecycle state of an artifact: enabled, disabled, or deprecated.
- Artifact versions
-
Manage versions that are created when a schema or API artifact is updated. You can also manage the lifecycle state of an artifact version: enabled, disabled, or deprecated.
- Artifact metadata
-
Manage details about a schema or API artifact, such as when it was created or modified, and its current state. You can edit the artifact name, description, or label. The artifact group and when the artifact was created or modified are read-only.
- Artifact rules
-
Configure rules to govern the content evolution of a specific schema or API artifact to prevent invalid or incompatible content from being added to the registry. Artifact rules override any global rules configured.
- Global rules
-
Configure rules to govern the content evolution of all schema and API artifacts artifacts to prevent invalid or incompatible content from being added to the registry. Global rules are applied only if an artifact does not have its own specific artifact rules configured.
- Search
-
Browse or search for schema and API artifacts and versions, for example, by name, group, description, or label.
- Admin
-
Export or import registry content in a
.zip
file, and manage logging levels for the registry server instance at runtime.
Compatibility with other schema registry REST APIs
Apicurio Registry version 2 provides API compatibility with the following schema registries by including implementations of their respective REST APIs:
-
Apicurio Registry version 1
-
Confluent Schema Registry version 6
-
IBM Event Streams schema registry version 1
-
CNCF CloudEvents Schema Registry version 0
Applications using Confluent client libraries can use Apicurio Registry as a drop-in replacement.
-
For detailed information, see the Apicurio Registry REST API documentation.
-
API documentation for the core Apicurio Registry REST API and for all compatible APIs is available from the main endpoint of your Apicurio Registry deployment, for example, on
http://MY-REGISTRY-URL/apis
.
Apicurio Registry storage options
Apicurio Registry provides the following options for the underlying storage of registry data:
-
In-memory
-
PostgreSQL 12 database
-
Apache Kafka 2.6
The in-memory storage option is suitable for a development environment only. All data is lost when restarting Apicurio Registry with this storage. The PostgreSQL or Kafka storage option is recommended for a production environment. |
Validate schemas with Kafka client serializers/deserializers
Kafka producer applications can use serializers to encode messages that conform to a specific event schema. Kafka consumer applications can then use deserializers to validate that messages have been serialized using the correct schema, based on a specific schema ID.
Apicurio Registry provides Kafka client serializers/deserializers (SerDes) to validate the following message types at runtime:
-
Apache Avro
-
Google protocol buffers
-
JSON Schema
The Apicurio Registry Maven repository and source code distributions include the Kafka SerDe implementations for these message types, which Kafka client developers can use to integrate with the registry. These implementations include custom Java classes for each supported message type, for example, io.apicurio.registry.serde.avro
, which client applications can use to pull schemas from the registry at runtime for validation.
Stream data to external systems with Kafka Connect converters
You can use Apicurio Registry with Apache Kafka Connect to stream data between Kafka and external systems. Using Kafka Connect, you can define connectors for different systems to move large volumes of data into and out of Kafka-based systems.
Apicurio Registry provides the following features for Kafka Connect:
-
Storage for Kafka Connect schemas
-
Kafka Connect converters for Apache Avro and JSON Schema
-
Registry REST API to manage schemas
You can use the Avro and JSON Schema converters to map Kafka Connect schemas into Avro or JSON schemas. Those schemas can then serialize message keys and values into the compact Avro binary format or human-readable JSON format. The converted JSON is also less verbose because the messages do not contain the schema information, only the schema ID.
Apicurio Registry can manage and track the Avro and JSON schemas used in the Kafka topics. Because the schemas are stored in Apicurio Registry and decoupled from the message content, each message must only include a tiny schema identifier. For an I/O bound system like Kafka, this means more total throughput for producers and consumers.
The Avro and JSON Schema serializers and deserializers (SerDes) provided by Apicurio Registry are also used by Kafka producers and consumers in this use case. Kafka consumer applications that you write to consume change events can use the Avro or JSON Serdes to deserialize these change events. You can install these Serdes into any Kafka-based system and use them along with Kafka Connect, or with Kafka Connect-based systems such as Debezium and Camel Kafka Connector.
Apicurio Registry demonstration examples
Apicurio Registry provides open source example applications that demonstrate how to use the registry in different use case scenarios. For example, these include storing schemas used by Kafka serializer and deserializer (SerDe) classes. These Java classes fetch the schema from the registry for use when producing or consuming operations to serialize, deserialize, or validate the Kafka message payload.
These example applications include the following:
-
Simple Avro
-
Simple JSON Schema
-
Confluent SerDes integration
-
Avro bean
-
Custom ID strategy
-
Simple Avro Maven
-
REST client
-
Mix Avro schemas
-
Cloud Events
For more details, see https://github.com/Apicurio/apicurio-registry-examples
Apicurio Registry available distributions
Distribution option | Container Image |
---|---|
In-memory |
|
PostgreSQL database |
|
Apache Kafka |
https://hub.docker.com/r/apicurio/apicurio-registry-kafkasql |
Apicurio Registry Operator |
https://hub.docker.com/r/apicurio/apicurio-registry-operator |
-
For details on building from source code and running Docker images, see https://github.com/Apicurio/apicurio-registry
-
For details on using the Operator to deploy, see https://github.com/Apicurio/apicurio-registry-operator