Using Apicurio Registry as an Apache Iceberg REST Catalog

This chapter explains how to use Apicurio Registry as an Apache Iceberg REST Catalog to manage table metadata for your data lakehouse applications:

Prerequisites

  • A running Apicurio Registry instance

  • Query engines or tools that support the Iceberg REST Catalog specification (Apache Spark, Trino, ClickHouse, DuckDB, Flink)

Overview of the Apicurio Registry Iceberg REST Catalog

Apicurio Registry implements the Apache Iceberg REST Catalog API specification, enabling it to serve as a metadata catalog for Iceberg tables. This allows query engines like Apache Spark, Trino, ClickHouse, DuckDB, and Flink to use Apicurio Registry as their Iceberg catalog for managing table metadata.

The catalog maps Apicurio Registry concepts to Iceberg concepts as follows:

Table 1. Mapping between Apicurio Registry and Iceberg concepts
Apicurio Registry Concept Iceberg Concept Description

Group

Namespace

A logical namespace for organizing tables. Each group in Apicurio Registry appears as a namespace in Iceberg.

Artifact (type: ICEBERG_TABLE)

Table

An Iceberg table registered in Apicurio Registry. The artifact content contains the full TableMetadata JSON.

Artifact (type: ICEBERG_VIEW)

View

An Iceberg view registered in Apicurio Registry. The artifact content contains the ViewMetadata JSON.

Artifact Content

TableMetadata / ViewMetadata

The metadata JSON stored as artifact content, containing schema, partition spec, sort order, snapshots, and other Iceberg metadata.

Group Labels

Namespace Properties

Key-value properties associated with a namespace, stored as group labels.

Artifact Labels

Table Properties

Key-value properties associated with a table, stored as artifact labels.

Prefix (path parameter)

Catalog Identifier

A configurable prefix that identifies the catalog instance (default: default).

Configuring the Apicurio Registry Iceberg REST Catalog

This section describes the configuration options available for the Apicurio Registry Iceberg REST Catalog.

Table 2. Apicurio Registry Iceberg configuration properties
Property Environment Variable Default Description

apicurio.iceberg.enabled

APICURIO_ICEBERG_ENABLED

true

Enables or disables the Iceberg REST Catalog API.

apicurio.iceberg.warehouse

APICURIO_ICEBERG_WAREHOUSE

Empty string

The default warehouse location for Iceberg tables. This is returned in the catalog configuration.

apicurio.iceberg.default-prefix

APICURIO_ICEBERG_DEFAULT_PREFIX

default

The default prefix (catalog identifier) used when none is specified.
Example: Configuring Apicurio Registry for Iceberg
apicurio.iceberg.enabled=true
apicurio.iceberg.warehouse=s3://my-bucket/warehouse
apicurio.iceberg.default-prefix=production

Using the Iceberg REST API

Apicurio Registry exposes the Iceberg REST Catalog API at the /apis/iceberg/v1 endpoint. The following examples demonstrate how to interact with the API using curl.

Getting catalog configuration

curl http://localhost:8080/apis/iceberg/v1/config
Example response
{
  "defaults": {},
  "overrides": {}
}

Creating a namespace

curl -X POST http://localhost:8080/apis/iceberg/v1/default/namespaces \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": ["my_database"],
    "properties": {
      "owner": "data-team",
      "description": "Production database"
    }
  }'
Example response
{
  "namespace": ["my_database"],
  "properties": {
    "owner": "data-team",
    "description": "Production database"
  }
}

Listing namespaces

curl http://localhost:8080/apis/iceberg/v1/default/namespaces
Example response
{
  "namespaces": [
    ["my_database"],
    ["another_database"]
  ]
}

Creating a table

curl -X POST http://localhost:8080/apis/iceberg/v1/default/namespaces/my_database/tables \
  -H "Content-Type: application/json" \
  -d '{
    "name": "users",
    "schema": {
      "type": "struct",
      "schema-id": 0,
      "fields": [
        {"id": 1, "name": "id", "required": true, "type": "long"},
        {"id": 2, "name": "name", "required": true, "type": "string"},
        {"id": 3, "name": "email", "required": false, "type": "string"},
        {"id": 4, "name": "created_at", "required": true, "type": "timestamp"}
      ]
    },
    "properties": {
      "write.format.default": "parquet"
    }
  }'

Loading a table

curl http://localhost:8080/apis/iceberg/v1/default/namespaces/my_database/tables/users

Listing tables in a namespace

curl http://localhost:8080/apis/iceberg/v1/default/namespaces/my_database/tables
Example response
{
  "identifiers": [
    {
      "namespace": ["my_database"],
      "name": "users"
    }
  ]
}

Renaming a table

curl -X POST http://localhost:8080/apis/iceberg/v1/default/tables/rename \
  -H "Content-Type: application/json" \
  -d '{
    "source": {
      "namespace": ["my_database"],
      "name": "users"
    },
    "destination": {
      "namespace": ["my_database"],
      "name": "customers"
    }
  }'

Dropping a table

curl -X DELETE http://localhost:8080/apis/iceberg/v1/default/namespaces/my_database/tables/users

Dropping a namespace

A namespace must be empty (contain no tables) before it can be dropped. 
curl -X DELETE http://localhost:8080/apis/iceberg/v1/default/namespaces/my_database

Using Apicurio Registry with query engines

This section provides configuration examples for popular query engines that support the Iceberg REST Catalog specification.

Apache Spark

Configure Spark to use Apicurio Registry as the Iceberg catalog:

spark.conf.set("spark.sql.catalog.apicurio", "org.apache.iceberg.spark.SparkCatalog")
spark.conf.set("spark.sql.catalog.apicurio.type", "rest")
spark.conf.set("spark.sql.catalog.apicurio.uri", "http://localhost:8080/apis/iceberg/v1")
spark.conf.set("spark.sql.catalog.apicurio.prefix", "default")
Using the catalog in Spark SQL
-- Switch to the Apicurio catalog
USE apicurio;

-- Create a namespace
CREATE NAMESPACE my_database;

-- Create a table
CREATE TABLE apicurio.my_database.events (
  id BIGINT,
  event_type STRING,
  event_time TIMESTAMP
) USING iceberg;

-- Query the table
SELECT * FROM apicurio.my_database.events;

Trino

Configure Trino to use Apicurio Registry as the Iceberg catalog by creating a catalog properties file:

etc/catalog/apicurio.properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://localhost:8080/apis/iceberg/v1
iceberg.rest-catalog.prefix=default
Using the catalog in Trino
-- Create a schema (namespace)
CREATE SCHEMA apicurio.my_database;

-- Create a table
CREATE TABLE apicurio.my_database.products (
  id BIGINT,
  name VARCHAR,
  price DECIMAL(10, 2)
) WITH (format = 'PARQUET');

-- Query the table
SELECT * FROM apicurio.my_database.products;

DuckDB

Configure DuckDB to use Apicurio Registry as the Iceberg catalog:

-- Install and load the Iceberg extension
INSTALL iceberg;
LOAD iceberg;

-- Attach the Apicurio Registry catalog
ATTACH 'http://localhost:8080/apis/iceberg/v1' AS apicurio (TYPE ICEBERG);

-- Query tables from the catalog
SELECT * FROM apicurio.my_database.users;

ClickHouse

Configure ClickHouse to use Apicurio Registry as the Iceberg catalog:

-- Create a database using the Iceberg engine
CREATE DATABASE apicurio_db ENGINE = Iceberg(
  'http://localhost:8080/apis/iceberg/v1',
  'default',
  'my_database'
);

-- Query tables from the catalog
SELECT * FROM apicurio_db.users;

API reference

The following table lists all Iceberg REST Catalog API endpoints supported by Apicurio Registry.

Table 3. Iceberg REST Catalog API endpoints
Method Endpoint Description

GET

/apis/iceberg/v1/config

Get catalog configuration

GET

/apis/iceberg/v1/{prefix}/namespaces

List all namespaces

POST

/apis/iceberg/v1/{prefix}/namespaces

Create a namespace

GET

/apis/iceberg/v1/{prefix}/namespaces/{namespace}

Load namespace metadata

HEAD

/apis/iceberg/v1/{prefix}/namespaces/{namespace}

Check if namespace exists

DELETE

/apis/iceberg/v1/{prefix}/namespaces/{namespace}

Drop a namespace (must be empty)

POST

/apis/iceberg/v1/{prefix}/namespaces/{namespace}/properties

Update namespace properties

GET

/apis/iceberg/v1/{prefix}/namespaces/{namespace}/tables

List tables in a namespace

POST

/apis/iceberg/v1/{prefix}/namespaces/{namespace}/tables

Create a table

GET

/apis/iceberg/v1/{prefix}/namespaces/{namespace}/tables/{table}

Load a table

HEAD

/apis/iceberg/v1/{prefix}/namespaces/{namespace}/tables/{table}

Check if table exists

DELETE

/apis/iceberg/v1/{prefix}/namespaces/{namespace}/tables/{table}

Drop a table

POST

/apis/iceberg/v1/{prefix}/tables/rename

Rename a table
Additional resources