Exporting Apicurio Registry Kafka topic data

One of the supported Apicurio Registry storage options is Apache Kafka, which uses a Kafka topic named kafkasql-journal to store data. If you encounter a problem when using this storage option and want to report it to Apicurio Registry developers, you might need to provide an export of the data present in the kafkasql-journal topic for analysis. You can create such a topic export by using the kcat tool (formerly known as kafkacat).

Prerequisites

  • Kafka has been installed and is running in your environment.

  • You have deployed Apicurio Registry with data stored in the kafkasql-journal topic.

  • The kafkasql-journal topic is still present.

Setting up kcat on a Kubernetes work pod

You can build the kcat executable from source on an ephemeral work pod so that the topic export includes JSON output with base64 encoded keys and values.

If you do not require JSON support, you can use the following commands to install kcat using dnf:

dnf install -y "dnf-command(copr)"
dnf copr enable bvn13/kcat
dnf update
dnf install -y kafkacat
Prerequisites

  • Your environment is Kubernetes.

  • You have logged in to the cluster using the kubectl command line interface.

Procedure

  1. Select a namespace where an ephemeral work pod will be started. This can be the same or a different namespace from where the Kafka cluster is deployed:

    kubectl config set-context --current --namespace=default

  2. Create an ephemeral work pod using the latest Fedora image, and connect to the pod using your terminal:

    kubectl run work-pod -it --rm --image=fedora --restart=Never

    If you keep the --rm flag, the work pod will be deleted when you disconnect from the remote terminal.

  3. You can install kcat using the dnf package manager. However, that version does not have JSON support enabled. Because you want to export the topic data in a JSON format with additional metadata, you must build the kcat executable from source.

    In addition, while the kcat project is widely used for this use case, the project appears to be hibernated, and you require an additional feature for the kafkasql-journal topic export to work properly. This feature is support for base64 encoded keys and values, and is important because the topic includes raw binary data, which might not be correctly encoded in the JSON output. Therefore, you must build kcat from source that includes base64 support, which has not been merged into the main project yet. For the relevant project links, see the additional resources.

    Install git, and check out the kcat repository:

    dnf install -y git
git clone https://github.com/edenhill/kcat.git
git remote add jjlin https://github.com/jjlin/kcat.git
cd kcat
git checkout jjlin/base64

  4. Install the dependencies and build kcat:

    dnf install -y gcc librdkafka-devel yajl-devel
./configure
make

  5. Copy the executable to /usr/bin so that it is available in $PATH:

    cp kcat /usr/bin

  6. Configure environment variables that will be used in subsequent examples:

    export KAFKA_BOOTSTRAP_SERVER="my-kafka-cluster-kafka-bootstrap.default.svc:9092"
Additional resources

Examples of using kcat

The following examples show how to use kcat to list topics and create topic exports.

  • List Kafka topics:

    kcat -b $KAFKA_BOOTSTRAP_SERVER -L | grep "topic " | sed 's#\([^"]*"\)\([^"]*\)\(".*\)#\2#'

    The sed command filters out extra information in this example.

  • Export data from the kafkasql-journal topic in JSON format, with envelope, and base64 encoded keys and values:

    kcat -b $KAFKA_BOOTSTRAP_SERVER -C -t kafkasql-journal -S base64 -Z -D \\n -e -J \
  > kafkasql-journal.topicdump

  • Create an export file for each listed topic by combining the preceding commands:

    mkdir dump
for t in $(kcat -b $KAFKA_BOOTSTRAP_SERVER -L | grep "topic " | sed 's#\([^"]*"\)\([^"]*\)\(".*\)#\2#'); do \
  kcat -b $KAFKA_BOOTSTRAP_SERVER -C -t $t -S base64 -Z -D \\n -e -J > dump/$t.topicdump; \
done

Copying topic export files from the work pod

After the topic export files have been created, you can copy the files from the work pod to your local machine.

Procedure

  • Run the following command on your local machine:

    kubectl cp work-pod:/kcat/dump .

Importing the kafkasql-journal topic data

You can import kafkasql-journal topic data that has been created with kcat by using an application from the Apicurio Registry examples repository.

Procedure

  • Clone the examples repository, build the import tool, and run it against your topic dump file:

    git clone https://github.com/Apicurio/apicurio-registry.git
cd examples/tools/kafkasql-topic-import
mvn clean install
export VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
java -jar target/apicurio-registry-tools-kafkasql-topic-import-$VERSION-jar-with-dependencies.jar -b <optional-kafka-bootstrap-server-url> -f <path-to-topic-dump-file>
Additional resources