We would like to have a plugin to shows the lag for the provided consumer group.
Currently, I need to monitor Kafka consumer and topic message lag. we are not using confluent platform to use the existing plugin. Could you please help us create plugin for this propose which would offer following metrics specific to consumer, topic and partitions used.
(GROUP NAME, TOPIC, PARTITION, CURRENT-OFFSET, LOG-END-OFFSET, LAG, CONSUMER-ID, HOST and CLIENT-ID)
By executing below command could get the result i wanted but plugin for this would be helpful.
bin/kafka-consumer-groups --bootstrap-server <kafka-broker>:9092 --describe --group sample-consumer-group
Hi Guest,
Thanks for your request.
We will analyze the Kafka consumer metrics and add them to our Plugins Roadmap. We will update in this thread, once we are done.
Regards,
Muralikrishnan
Product Manager
Hello,
We are happy to introduce Apache Kafka plugin integrations to monitor your entire Kafka stack. You will now be able to monitor critical Kafka producer, consumer, and broker metrics in real time and stay ahead of issues and bottlenecks.
Refer to the below GitHub link for the Kafka monitoring plugin details and setup steps:
https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring
You can monitor the following components with our Kafka plugins:
1. Kafka consumer
Install the Kafka Consumer plugin and monitor key consumer metrics.
https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Consumer_Monitoring
Supported metrics:
- Records Lag Max(All Partitions)
- Bytes Consumed Rate(All Topics)
- Bytes Consumed Rate(Topic Specific)
- Records Consumed Rate(All Topics)
- Records Consumed Rate(Topic Specific)
- Fetch Rate
- Records Lag
- Records Lag Max
- Records Per Request Avg
- Fetch Throttle Time Avg
- Fetch Throttle Time Max
- Topic Name
- Partition No
2. Kafka producer
Install the Apache Kafka Producer plugin to monitor key Kafka producer metrics and ensure a steady flow of data.
https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Producer_Monitoring
Supported metrics:
- Compression Rate Avg
- Response Rate
- Request Rate
- Request Latency Avg
- Outgoing Byte Rate
- IO Wait Time NS Avg
- Batch Size Avg
3. Kafka broker
https://github.com/site24x7/plugins/tree/master/Kafka_Complete_Monitoring/Kafka_Broker_Monitoring
Supported metrics:
- Under Replicated Partitions
- ISR Shrinks Per Sec
- ISR Expands Per Sec
- Active Controller Count
- Offline Partitions Count
- Leader Election Rate And Time Ms
- Unclean Leader Elections Per Sec
- Total Time ms
- Purgatory Size
- Bytes In Per Sec
- Network Request Rate
- Network Error Rate
- Total Broker Partitions
- Young Generation GC Count
- Young Generation GC Time
- Old Generation GC Count
- Old Generation GC Time
- Log End Offset
Prerequisites for installing the plugins:
- Install the jmxquery module for Python3.
pip install jmxquery
- Download and install the latest version of the [Site24x7 Linux agent] (https://www.site24x7.com/app/client#/admin/inventory/add-monitor) in the server where you plan to run the plugin.
Plugin Installation:
If you want to install the Kafka consumer monitoring plugin,
- Create a directory named "kafka_consumer_monitoring" under the Site24x7 Linux Agent plugin directory:
Linux -> /opt/site24x7/monagent/plugins/kafka_consumer_monitoring - Download all the files in the "kafka_consumer_monitoring" folder and place it under the "kafka_consumer_monitoring" directory.
- Provide appropriate configuration in the kafka_consumer_monitoring.cfg file.
To test the plugin manually, execute the below command:
python3 kafka_consumer_monitoring.py --kafka_consumer_host=<KAFKA_CONSUMER_HOST_NAME> kafka_consumer_jmx_port=<KAFKA_CONSUMER_PORT_NO> --kafka_consumer_partition=<KAFKA_CONSUMER_PARTITION_NO> --kafka_topic_name=<KAFKA_TOPIC_NAME> --kafka_consumer_client_id=<KAFKA_CONSUMER_CLIENT_ID> --logs_enabled=False --log_type_name=None --log_file_path=None
The Kafka consumer plugin will be added for monitoring during the next data collection event of the agent.
Repeat the respective processes for producer and broker plugins as well.
If you have any queries, please reply to this thread.
Regards,
Muralikrishnan
Product Manager