Summary

This extension contains operators to interact with the Apache Kafka message broker.

It adds two operators to interact with a specific Kafka topic:

• Read messages from a topic, either old messages in a batch, or collect new published messages
• Write data from an example set as new messages into a topic on a Kafka cluster

Both operators work batch wise and work best in combination with either background execution or a deployment on AI Hub or a Real-Time-Scoring agent.

Synopsis

The Kafka Connector extension allows RapidMiner users to interact with the very popular streaming and message broker platform Apache Kafka.

The two new operators can read messages from existing Kafka topics and also publish data from a process to an existing topic.

To ease the access to existing Kafka servers, the extension supports a new Connection Object to store and share the connection details.

The operators work batch-wise, but can be easily integrated into a looped execution or process deployment for a continuous execution.

Use Case examples

1. Read past messages from a topic With the "Read Kafka Topic" operator and the polling strategy set to earliest, you can retrieve past messages stored under a given topic. This can be used to analyze and potentially build a model on historic data.

2. Read new messages from a topic With the "Read Kafka Topic" operator and the polling strategy set to latest, you can retrieve newly published messages from a topic. There is an option to either wait until a specified number of messages are retrieved or to wait a fixed amount of time and return all messages published during that time. This can be used in a deployment to apply a learned model on fresh data or check for specific events.

3. Write messages to a topic With the "Write Kafka Topic" operator, any example set data can be converted into a message and be pushed to a given topic. Each example or row is converted to a message (either as simple string or in a JSON format) and send to the message broker. There is an option to either send all messages in a bulk or in a specified time interval. This can be used for example to publish scored or transformed data back into a Kafka stream