Populate Graphs to Apache Kafka¤
Introduction¤
Apache Kafka is a distributed event store and stream-processing platform. Kafka is widely used in enterprises for data pipelines, streaming analytics, data integration and other applications.
By using the cmem-plugin-kafka Python Plugin, you can produce and send messages to Apache Kafka from inside of our Corporate Memory Workflows.
Installation¤
In order to use the Kafka Producer workflow task, you need to extend your Corporate Memory instance with the cmem-plugin-kafka
package.
This can be done by using cmemc:
$ cmemc -c my-cmem admin workspace python install cmem-plugin-kafka
Install package cmem-plugin-kafka ... done
You can validate your installation by listing all installed plugins (from all packages):
$ cmemc -c my-cmem admin workspace python list-plugins
ID Package ID Type Label
--------------------------------- ----------------- -------------- ---------------------------------
cmem_plugin_kafka-ReceiveMessages cmem-plugin-kafka WorkflowPlugin Kafka Consumer (Receive Messages)
cmem_plugin_kafka-SendMessages cmem-plugin-kafka WorkflowPlugin Kafka Producer (Send Messages)
Usage¤
Once you installed the package, you can use the Kafka Producer by simply creating a new task, e.g. search for kafka
in the Create new item screen:
Follow the in-app documentation on how to configure the task (e.g. for providing credentials or preparing data to be sent in messages).