Fat Topic: Improving Latency in Content-Based Publish/Subscribe Systems on Apache Kafka

Apache Kafka is a mainstream message middleware that can provide topic-based data distribution with high throughput. Some existing works have explored building a large-scale content-based publish/subscribe system on Kafka. However, when the number of subscribers is large, the time overhead for matching the message with a large number of subscriptions and forwarding the message to the matched subscribers is large, which greatly affects the latency of message distribution. In this paper, we propose a new type of topic called the fat topic in Kafka to improve the latency of content-based data distribution. In addition, we modify Kafka’s code to provide Consumer and Provider APIs to access fat topics. We conducted extensive experiments to evaluate the performance of fat topics. The experiment results show that the fat topic can improve the latency of content-based event distribution by about 3.7 times compared with the original Kafka topic.