Validating data with the Apicurio Registry schema validation library

You can use the Apicurio Registry schema validation library to check JSON, Avro, and Protobuf payloads in a Java application against schemas stored in Apicurio Registry. The library fetches and caches each schema and reports every problem it finds, without Kafka serializers or deserializers.

Schema validation library overview

The Apicurio Registry schema validation library is a set of standalone Java modules that validate payloads against schemas that are stored in Apicurio Registry. Each module provides one validator class for one schema format, and all modules share the same result and error types.

The library is useful when you want to validate data outside of a Kafka producer or consumer, for example in a REST endpoint, a message broker plugin, or a batch job. You choose the schema by its coordinates in Apicurio Registry, and the library downloads it through the Apicurio Registry Java SDK, caches it, and validates your payload against it.

Validator classes

The following table lists the validator that each module provides.

Table 1. Validator classes in the schema validation library
Schema format	Validator class	Payload types
JSON Schema	`io.apicurio.schema.validation.json.JsonValidator`	Any Java object that Jackson can convert to a JSON tree, such as a Java bean or a `Map`, or a `ByteBuffer` that contains serialized JSON. A `String` that contains JSON is not parsed; the validator treats it as a single JSON string value. To validate serialized JSON, wrap the bytes in a `ByteBuffer`, for example `ByteBuffer.wrap(json.getBytes(StandardCharsets.UTF_8))`.
Avro	`io.apicurio.schema.validation.avro.AvroValidator`	An Avro `GenericRecord`, or a `String` that contains JSON-encoded Avro data
Protobuf	`io.apicurio.schema.validation.protobuf.ProtobufValidator`	A generated Protobuf `Message`

Validation methods

Every validator offers two ways to choose the schema:

Fixed artifact reference: You pass an ArtifactReference when you create the validator, and then call validateByArtifactReference for each payload. The validator always resolves the same artifact. Use this method when one validator instance handles one schema.
Per-record artifact reference: You call validate with a Record whose metadata carries the ArtifactReference for that payload. The library provides SchemaValidationRecord and SchemaValidationMetadata for this purpose. Use this method when one validator instance handles payloads for different schemas.

If you create a validator with Optional.empty() instead of an artifact reference, only the validate method is available. Calling validateByArtifactReference on such a validator throws a NullPointerException.

Validation results

Each validation method returns a ValidationResult. Call success() to check the outcome. When validation fails, getValidationErrors() returns a list of ValidationError objects, and each error has a description and a context. The meaning of the context depends on the format: an error code for JSON Schema, a field path or error category for Avro, and an empty string for Protobuf.

When the validator cannot fetch the schema from Apicurio Registry, for example because the artifact does not exist or the registry is unreachable, it does not throw an exception. Instead, it returns a failed result that contains a single error with the context SCHEMA_RESOLUTION_ERROR and a description that starts with Failed to resolve schema from registry.

Schema resolution and caching

The validators use the same schema resolver as the Apicurio Registry Kafka client serializers and deserializers (SerDes), so they accept the same apicurio.registry.* configuration properties. The resolver caches each schema after the first lookup and checks Apicurio Registry again after the configured check period. When the artifact version references other artifacts, the resolver fetches those references and passes them to the format parser, so schemas with references validate correctly.

Additional resources

Validating JSON data against a JSON Schema artifact
Validating Avro data against an Avro schema artifact
Validating Protobuf messages against a Protobuf schema artifact
Schema validation library dependencies and configuration

Validating JSON data against a JSON Schema artifact

You can validate Java objects against a JSON Schema artifact that is stored in Apicurio Registry by using the JsonValidator class. The validator reports every constraint that the payload violates, which helps you reject bad data before it reaches a downstream system.

The examples in this procedure use the following JSON Schema, registered in the default group with the artifact ID message-schema:

{
  "$id": "https://example.com/message.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["message", "time"],
  "properties": {
    "message": { "type": "string" },
    "time": { "type": "number" }
  }
}

Prerequisites

Apicurio Registry is installed and running in your environment.
A JSON Schema artifact is registered in Apicurio Registry, and you know its group ID and artifact ID.
You have a Maven project for your Java application.

Procedure

Add the JSON Schema validation module to your Maven project:

<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-schema-validation-jsonschema</artifactId>
    <version>${apicurio-registry.version}</version>
</dependency>

Create the validator with the Apicurio Registry URL and an artifact reference for the schema:

import io.apicurio.registry.resolver.config.SchemaResolverConfig;
import io.apicurio.registry.resolver.strategy.ArtifactReference;
import io.apicurio.schema.validation.json.JsonValidator;

import java.util.HashMap;
import java.util.Map;
import java.util.Optional;

Map<String, Object> config = new HashMap<>();
config.put(SchemaResolverConfig.REGISTRY_URL, "http://localhost:8080/apis/registry/v3");

ArtifactReference reference = ArtifactReference.builder()
        .groupId("default")
        .artifactId("message-schema")
        .build();

JsonValidator validator = new JsonValidator(config, Optional.of(reference));

When you omit the version, the validator resolves the latest version of the artifact.

Validate a payload that conforms to the schema:

import io.apicurio.schema.validation.ValidationResult;

Map<String, Object> payload = Map.of(
        "message", "Hello world",
        "time", System.currentTimeMillis());

ValidationResult result = validator.validateByArtifactReference(payload);
System.out.println(result);

The output is ValidationResult [ success ].

Validate a payload that violates the schema and read the errors:

import io.apicurio.schema.validation.ValidationError;

Map<String, Object> invalidPayload = Map.of(
        "message", "Hello world",
        "time", "not-a-number");

ValidationResult invalidResult = validator.validateByArtifactReference(invalidPayload);
if (!invalidResult.success()) {
    for (ValidationError error : invalidResult.getValidationErrors()) {
        System.out.println(error.getContext() + ": " + error.getDescription());
    }
}

The output is 1029: $.time: string found, number expected. The context is the error code of the violated JSON Schema keyword, and the description names the JSON path of the offending value. The error codes and message wording come from the networknt json-schema-validator library and can change when Apicurio Registry upgrades that dependency.

Optional: To choose the schema for each payload at run time, wrap the payload in a SchemaValidationRecord and call validate:

import io.apicurio.schema.validation.SchemaValidationMetadata;
import io.apicurio.schema.validation.SchemaValidationRecord;

SchemaValidationRecord<Object> record = new SchemaValidationRecord<>(
        payload, new SchemaValidationMetadata(reference));

ValidationResult recordResult = validator.validate(record);

Verification

Run the application. The conforming payload prints ValidationResult [ success ], and the invalid payload prints one error with the context 1029.

Additional resources

Validating Avro data against an Avro schema artifact

You can validate Avro GenericRecord objects or JSON-encoded Avro data against an Avro schema artifact that is stored in Apicurio Registry by using the AvroValidator class.

GenericRecord validation performs partial checks. It skips primitive type checks, accepts a union when any branch reports no errors, and does not reject a non-collection value for an array or a non-map value for a map. For example, a record whose own schema declares time as a string can pass against a registered long field. A successful result does not establish full schema conformance.

The JSON String overload instead decodes the data against the registered schema and reports decoding failures, including primitive type mismatches.

The examples in this procedure use the following Avro schema, saved as message.avsc in the working directory of the application and registered in the default group with the artifact ID message-avro:

{
  "type": "record",
  "name": "Message",
  "namespace": "com.example",
  "fields": [
    { "name": "message", "type": "string" },
    { "name": "time", "type": "long" }
  ]
}

Prerequisites

Apicurio Registry is installed and running in your environment.
An Avro schema artifact is registered in Apicurio Registry, and you know its group ID and artifact ID.
You have a Maven project for your Java application.

Procedure

Add the Avro validation module to your Maven project:

<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-schema-validation-avro</artifactId>
    <version>${apicurio-registry.version}</version>
</dependency>

Create the validator with the Apicurio Registry URL and an artifact reference for the schema:

import io.apicurio.registry.resolver.config.SchemaResolverConfig;
import io.apicurio.registry.resolver.strategy.ArtifactReference;
import io.apicurio.schema.validation.avro.AvroValidator;

import java.util.HashMap;
import java.util.Map;
import java.util.Optional;

Map<String, Object> config = new HashMap<>();
config.put(SchemaResolverConfig.REGISTRY_URL, "http://localhost:8080/apis/registry/v3");

ArtifactReference reference = ArtifactReference.builder()
        .groupId("default")
        .artifactId("message-avro")
        .build();

AvroValidator validator = new AvroValidator(config, Optional.of(reference));

Validate a GenericRecord that conforms to the schema:

import io.apicurio.schema.validation.ValidationResult;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;

import java.io.File;

Schema schema = new Schema.Parser().parse(new File("message.avsc"));

GenericRecord payload = new GenericData.Record(schema);
payload.put("message", "Hello world");
payload.put("time", System.currentTimeMillis());

ValidationResult result = validator.validateByArtifactReference(payload);
System.out.println(result);

The output is ValidationResult [ success ]. When a field that the registered schema requires is missing from the record, the result contains an error whose context is the field path and whose description is Missing required field: <name>.

Validate JSON-encoded data that violates the schema and read the errors:

import io.apicurio.schema.validation.ValidationError;

String invalidJson = "{\"message\": \"Hello world\", \"time\": \"not-a-number\"}";

ValidationResult invalidResult = validator.validateByArtifactReference(invalidJson);
if (!invalidResult.success()) {
    for (ValidationError error : invalidResult.getValidationErrors()) {
        System.out.println(error.getContext() + ": " + error.getDescription());
    }
}

The output is TYPE_ERROR: Expected long. Got VALUE_STRING. The String overload decodes the JSON with the registered schema, so the context is TYPE_ERROR for a type mismatch and VALIDATION_ERROR for any other decoding problem.

Optional: To choose the schema for each record at run time, wrap the record in a SchemaValidationRecord and call validate:

import io.apicurio.schema.validation.SchemaValidationMetadata;
import io.apicurio.schema.validation.SchemaValidationRecord;

SchemaValidationRecord<GenericRecord> record = new SchemaValidationRecord<>(
        payload, new SchemaValidationMetadata(reference));

ValidationResult recordResult = validator.validate(record);

The validate method accepts GenericRecord payloads only. To validate JSON-encoded Avro data, call validateByArtifactReference instead.

Verification

Run the application. The conforming record prints ValidationResult [ success ], and the invalid JSON prints one error with the context TYPE_ERROR.

Additional resources

Validating Protobuf messages against a Protobuf schema artifact

You can validate generated Protobuf messages against a Protobuf schema artifact in Apicurio Registry by using the ProtobufValidator class. The validator checks that the registered schema defines the message type and that the compiled definition is compatible with it.

This check catches a .proto file that has drifted from the registry before you deploy the application.

The examples in this procedure use the following Protobuf schema, registered in the default group with the artifact ID message-proto, and the MessageExample class that protoc generates from it:

syntax = "proto3";
package com.example;

message MessageExample {
  string key = 1;
  string value = 2;
}

Prerequisites

Apicurio Registry is installed and running in your environment.
A Protobuf schema artifact is registered in Apicurio Registry, and you know its group ID and artifact ID.
You have a Maven project for your Java application that generates Java classes from your .proto files.

Procedure

Add the Protobuf validation module to your Maven project:

<dependency>
    <groupId>io.apicurio</groupId>
    <artifactId>apicurio-registry-schema-validation-protobuf</artifactId>
    <version>${apicurio-registry.version}</version>
</dependency>

Create the validator with the Apicurio Registry URL and an artifact reference for the schema:

import io.apicurio.registry.resolver.config.SchemaResolverConfig;
import io.apicurio.registry.resolver.strategy.ArtifactReference;
import io.apicurio.schema.validation.protobuf.ProtobufValidator;

import java.util.HashMap;
import java.util.Map;
import java.util.Optional;

Map<String, Object> config = new HashMap<>();
config.put(SchemaResolverConfig.REGISTRY_URL, "http://localhost:8080/apis/registry/v3");

ArtifactReference reference = ArtifactReference.builder()
        .groupId("default")
        .artifactId("message-proto")
        .build();

ProtobufValidator validator = new ProtobufValidator(config, Optional.of(reference));

Validate a message whose type the registered schema defines:

import io.apicurio.schema.validation.ValidationResult;

MessageExample payload = MessageExample.newBuilder()
        .setKey("order-1001")
        .setValue("Hello world")
        .build();

ValidationResult result = validator.validateByArtifactReference(payload);
System.out.println(result);

The output is ValidationResult [ success ].

Validate a message whose type the registered schema does not define, and read the errors:

import io.apicurio.schema.validation.ValidationError;

// MessageExample2 is generated from a different .proto file
MessageExample2 invalidPayload = MessageExample2.newBuilder()
        .setKey2("order-1002")
        .setValue2(32)
        .build();

ValidationResult invalidResult = validator.validateByArtifactReference(invalidPayload);
if (!invalidResult.success()) {
    for (ValidationError error : invalidResult.getValidationErrors()) {
        System.out.println(error.getDescription());
    }
}

The output is Missing message type MessageExample2 in the protobuf schema. When the message type exists but its compiled definition removes or changes a field that the registered schema defines, for example a field with a different type or number, the result contains one error for each difference. The context of a Protobuf validation error is always an empty string.

Optional: To choose the schema for each message at run time, wrap the message in a SchemaValidationRecord and call validate:

import com.google.protobuf.Message;
import io.apicurio.schema.validation.SchemaValidationMetadata;
import io.apicurio.schema.validation.SchemaValidationRecord;

ArtifactReference recordReference = ArtifactReference.builder()
        .groupId("default")
        .artifactId("message-proto")
        .version("1")
        .build();

ProtobufValidator recordValidator = new ProtobufValidator(config, Optional.empty());

SchemaValidationRecord<Message> record = new SchemaValidationRecord<>(
        payload, new SchemaValidationMetadata(recordReference));

ValidationResult recordResult = recordValidator.validate(record);

Replace 1 with the registered schema version. The explicit version lets this new validator fetch the schema without a prior fixed-reference lookup.

Verification

Run the application. The MessageExample message prints ValidationResult [ success ], and the MessageExample2 message prints the missing message type error.

Additional resources

Schema validation library dependencies and configuration

The schema validation library consists of four Maven modules that share one result API and one set of configuration properties. The following tables list the modules, the result methods, the error context values, the artifact coordinates, and the configuration properties.

Maven modules

All modules use the group ID io.apicurio and the same version as the Apicurio Registry release. Versions 3.3.0 and later are available from Maven Central. Add only the format modules that you need; each format module pulls in the common module, the schema resolver, and the Apicurio Registry Java SDK.

Table 2. Maven modules of the schema validation library
Artifact ID	Contents
`apicurio-registry-schema-validation-common`	`ValidationResult`, `ValidationError`, `SchemaValidationRecord`, and `SchemaValidationMetadata` in the `io.apicurio.schema.validation` package
`apicurio-registry-schema-validation-jsonschema`	`JsonValidator`, which validates JSON payloads with the networknt `json-schema-validator` library. The validator uses the JSON Schema draft-07 dialect by default; when the artifact declares a `$schema` keyword, the validator uses that dialect.
`apicurio-registry-schema-validation-avro`	`AvroValidator`, which validates `GenericRecord` objects and JSON-encoded data with Apache Avro
`apicurio-registry-schema-validation-protobuf`	`ProtobufValidator`, which compares the compiled message definition with the registered schema by using the Apicurio Registry Protobuf compatibility checker

Result types

Table 3. Methods of ValidationResult and ValidationError
Method	Description
`ValidationResult.success()`	Returns `true` when the validator reports no errors and `false` otherwise. The `GenericRecord` validation limitations still apply.
`ValidationResult.getValidationErrors()`	Returns the list of errors, or `null` when the validator reports no errors. Check `success()` before you read the list.
`ValidationResult.toString()`	Returns `ValidationResult [ success ]` for a successful result, or `ValidationResult [ errors = [...] ]` with every reported error for a failed result.
`ValidationError.getDescription()`	Returns the human-readable message for one violation.
`ValidationError.getContext()`	Returns where or why the violation occurred. The value depends on the format, as described in the following table.

Table 4. Values of the error context by format
Format	Context value
JSON Schema	The error code of the violated keyword from the networknt library, for example `1028` for a missing required property and `1029` for a type mismatch. The description starts with the JSON path of the value, for example `$.time`.
Avro, `GenericRecord` payload	The path of the field that failed, for example `time` or `address.city`.
Avro, JSON `String` payload	`TYPE_ERROR` when the data does not match the schema type, or `VALIDATION_ERROR` for any other decoding failure.
Protobuf	An empty string.
Any format	`SCHEMA_RESOLUTION_ERROR` when the validator cannot fetch the schema from Apicurio Registry. The description starts with `Failed to resolve schema from registry` and includes the exception class and message. For JSON payloads, the validator also returns this context when it cannot convert the payload to a JSON tree, for example when a `ByteBuffer` contains malformed JSON.

Artifact references

For validateByArtifactReference, build an ArtifactReference with ArtifactReference.builder() and one of the following combinations of coordinates, and pass it to the validator constructor. When a reference carries more than one kind of coordinate, the resolver uses the first one in the order of the table.

Table 5. Artifact coordinates for validateByArtifactReference
Builder methods	Resolved version
`contentId(Long)`	The content with that content ID.
`contentHash(String)`	The content with that content hash.
`globalId(Long)`	The version with that global ID.
`groupId(String)`, `artifactId(String)`, `version(String)`	The named version of the artifact. When you omit `version`, the latest version of the artifact. The `artifactId` is required.

For per-record validate, the resolver copies only the group ID, artifact ID, and version from the metadata reference. It discards contentId, contentHash, and globalId, so an ID-only metadata reference cannot select the schema by that ID and can produce SCHEMA_RESOLUTION_ERROR.

Specify a version in the per-record reference, or set SchemaResolverConfig.FIND_LATEST_ARTIFACT to true to select the latest version by group ID and artifact ID. With the default false and no version, a cache miss uses schema-content lookup instead. The Protobuf validator cannot extract schema content through that lookup path, so it requires an explicit version or FIND_LATEST_ARTIFACT=true to work without a cached schema.

Configuration properties

You pass configuration properties to the validator constructor as a Map<String, Object>. The property keys are available as constants on the SchemaResolverConfig class. The following table lists the properties that are most relevant to validation.

Table 6. Configuration properties for the schema validation library
Property	Description	Default
`apicurio.registry.url`	URL of the Apicurio Registry REST API, for example `http://localhost:8080/apis/registry/v3`. Required.	None
`apicurio.registry.url.version`	Major version of the REST API at the URL, `2` or `3`. Set it only when the resolver cannot detect the version from the URL.	Detected from the URL
`apicurio.registry.auth.service.token.endpoint`, `apicurio.registry.auth.client.id`, `apicurio.registry.auth.client.secret`, `apicurio.registry.auth.client.scope`	OAuth 2.0 client credentials for an instance of Apicurio Registry that requires authentication.	None
`apicurio.registry.auth.username`, `apicurio.registry.auth.password`	Credentials for HTTP basic authentication.	None
`apicurio.registry.check-period-ms`	How long, in milliseconds, the resolver caches a schema before it checks Apicurio Registry again.	`30000`
`apicurio.registry.cache-latest`	Whether the resolver caches lookups that do not specify a version. Set it to `false` to fetch the latest version on every validation.	`true`
`apicurio.registry.find-latest`	Whether per-record `validate` resolves by artifact coordinates instead of schema content when no version is specified. Set it to `true` to use the latest version. The fixed-reference `validateByArtifactReference` method uses the latest version when no version is specified, regardless of this setting.	`false`
`apicurio.registry.retry-count`, `apicurio.registry.retry-backoff-ms`	Number of times the resolver retries a failed schema lookup, and the delay between retries in milliseconds.	`3`, `300`
`apicurio.registry.fault-tolerant-refresh`	Whether the resolver keeps a cached schema and logs the error when a refresh fails, instead of failing the validation.	`false`
`apicurio.registry.dereference-schema`	Whether the resolver asks Apicurio Registry for a dereferenced schema instead of fetching the referenced artifacts separately.	`false`
`apicurio.registry.vertx.instance`	A `Vertx` object that the resolver uses for HTTP connections. Set it programmatically to reuse a Vert.x instance that you manage. When it is not set, the Apicurio Registry Java SDK creates its own HTTP client, with Vert.x when it is available and the JDK HTTP client otherwise.	None

The validators accept every property of the Apicurio Registry schema resolver. For the TLS and proxy properties, see Configuring Kafka serializers/deserializers in Java clients. For the complete list, including the HTTP adapter and client retry properties, see the SchemaResolverConfig class in the schema resolver module.

Additional resources