Exporting Apicurio Registry Kafka topic data

When Apicurio Registry uses Apache Kafka storage, it stores data in a Kafka topic named kafkasql-journal. You can export the data in this topic by using the kcat tool (formerly known as kafkacat), for example, to report a storage problem to Apicurio Registry developers for analysis.

Prerequisites

Kafka has been installed and is running in your environment.
You have deployed Apicurio Registry with data stored in the kafkasql-journal topic.
The kafkasql-journal topic is still present.

Setting up kcat on a Kubernetes work pod

You can build the kcat executable from source on an ephemeral work pod so that the topic export includes JSON output with base64 encoded keys and values.

If you do not require JSON support, you can use the following commands to install kcat using dnf:

dnf install -y "dnf-command(copr)"
dnf copr enable bvn13/kcat
dnf update
dnf install -y kafkacat

Prerequisites

Your environment is Kubernetes.
You have logged in to the cluster using the kubectl command line interface.

Procedure

Select a namespace where an ephemeral work pod will be started. This can be the same or a different namespace from where the Kafka cluster is deployed:
```
kubectl config set-context --current --namespace=default
```
Create an ephemeral work pod using the latest Fedora image, and connect to the pod using your terminal:
```
kubectl run work-pod -it --rm --image=fedora --restart=Never
```
If you keep the --rm flag, the work pod will be deleted when you disconnect from the remote terminal.
You can install kcat using the dnf package manager. However, that version does not have JSON support enabled. Because you want to export the topic data in a JSON format with additional metadata, you must build the kcat executable from source.

In addition, while the kcat project is widely used for this use case, the project appears to be hibernated, and you require an additional feature for the kafkasql-journal topic export to work properly. This feature is support for base64 encoded keys and values, and is important because the topic includes raw binary data, which might not be correctly encoded in the JSON output. Therefore, you must build kcat from source that includes base64 support, which has not been merged into the main project yet. For the relevant project links, see the additional resources.

Install git, and check out the kcat repository:
```
dnf install -y git
git clone https://github.com/edenhill/kcat.git
git remote add jjlin https://github.com/jjlin/kcat.git
cd kcat
git checkout jjlin/base64
```

Install the dependencies and build kcat:

dnf install -y gcc librdkafka-devel yajl-devel
./configure
make

Copy the executable to /usr/bin so that it is available in $PATH:
```
cp kcat /usr/bin
```

Configure environment variables that will be used in subsequent examples:

export KAFKA_BOOTSTRAP_SERVER="my-kafka-cluster-kafka-bootstrap.default.svc:9092"

Additional resources

Examples of using kcat

You can use kcat to list Kafka topics, to export the kafkasql-journal topic in JSON format with base64-encoded keys and values, and to create an export file for every listed topic.

List Kafka topics:

kcat -b $KAFKA_BOOTSTRAP_SERVER -L | grep "topic " | sed 's#\([^"]*"\)\([^"]*\)\(".*\)#\2#'

The sed command filters out extra information in this example.

Export data from the kafkasql-journal topic in JSON format, with envelope, and base64 encoded keys and values:

kcat -b $KAFKA_BOOTSTRAP_SERVER -C -t kafkasql-journal -S base64 -Z -D \\n -e -J \
  > kafkasql-journal.topicdump

Create an export file for each listed topic by combining the preceding commands:

mkdir dump
for t in $(kcat -b $KAFKA_BOOTSTRAP_SERVER -L | grep "topic " | sed 's#\([^"]*"\)\([^"]*\)\(".*\)#\2#'); do \
  kcat -b $KAFKA_BOOTSTRAP_SERVER -C -t $t -S base64 -Z -D \\n -e -J > dump/$t.topicdump; \
done

Copying topic export files from the work pod

After the topic export files have been created, you can copy the files from the work pod to your local machine.

Procedure

Run the following command on your local machine:
```
kubectl cp work-pod:/kcat/dump .
```

Importing the kafkasql-journal topic data

You can import kafkasql-journal topic data that has been created with kcat by using an application from the Apicurio Registry examples repository.

Procedure

Clone the examples repository, build the import tool, and run it against your topic dump file:

git clone https://github.com/Apicurio/apicurio-registry.git
cd examples/tools/kafkasql-topic-import
mvn clean install
export VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
java -jar target/apicurio-registry-tools-kafkasql-topic-import-$VERSION-jar-with-dependencies.jar -b <optional-kafka-bootstrap-server-url> -f <path-to-topic-dump-file>

Additional resources