Skip to main content

Getting started with Kafka and Kafka Tool

This provides a quick introduction to setting up a local Kafka instance and using Kafka Tool to view the messages. The initial part of the Kafka tutorial is adapted slightly from http://kafka.apache.org/quickstart.html.

The following are steps for:
  1. Downloading and unpacking Apache Kafka in Linux
  2. Starting the Zookeeper and Kafka servers
  3. Creating a new topic
  4. Adding and viewing messages using the command line tools
  5. Use Kafka Tool to view the messages in a topic

Step 1: Download Kafka

cd ./Downloads/
wget http://mirror.catn.com/pub/apache/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz
tar -zxf kafka_2.11-0.10.0.0.tgz
cd kafka_2.11-0.10.0.0


Step 2: Start the Zookeeper and Kafka servers

Check the Kafka port in zookeeper.properties by looking at clientPort (typically 2181) in
config/zookeeper.properties

Start Zookeeper with:
bin/zookeeper-server-start.sh config/zookeeper.properties

and then start Kafka:
bin/kafka-server-start.sh config/server.properties

Step 3: Create a topic

Start a new terminal (as the current one will be showing Kafka output)
cd ./Downloads/kafka_2.11-0.10.0.0

Check that there isn't any topics at present with:
bin/kafka-topics.sh --list --zookeeper localhost:2181

Create a topic called 'test' using the command line tools:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

NB. If you get the error "Error while executing topic command : replication factor: 1 larger than available brokers: 0" then check that you have started Zookeeper and Kafka in the previous step.

Check the 'test' topic was created with:
bin/kafka-topics.sh --list --zookeeper localhost:2181

Step 4: Add some messages using a console Producer

Use the command-line client to put messages onto the Kafka 'test' topic by running
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

Then type some messages and press return.

Step 5: Use the command line Consumer to get messages from Kafka

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Step 6: Download and Setup Kafka Tool

  • Download Kafka Tool from http://www.kafkatool.com/download.html
  • Install Kafka Tool
  • Start Kafka Tool
  • Click File then Add New Connection...
  • Set the following:
    • Cluster name: Stand-alone
    • Kafka Cluster Version: 0.9
    • Zookeeper Host: localhost
    • Zookeeper Port: 2181
    • chroot path: /
  • Click 'Test' to check that the Kafka server can be reached
  • Click 'Add' to add the connection

Step 7: View messages in Kafka Tool

  • Expand the 'Stand-alone' cluster in the Browser
  • Expand 'Topics' and you should see the topic called 'Test'
  • Change Key and Message in the 'Content Types' pane to 'String'
  • Click 'Refresh' in the 'Messages' pane and you should see the same number of messages as added using the command-line Kafka Producer
  • Click the 'Data' tab and then press the Green play button. You should see the messages and their offset

Step 8: Delete a Kafka topic

bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test


Comments

Post a Comment

Popular posts from this blog

Python libraries on an air-gapped machine

The Problem Development and test clusters may be air-gapped so that client data or sensitive software under development is less likely to be leaked. This can cause problems when trying to install libraries, e.g. for Python-based software, especially if the cluster has an old version of a Linux OS installed. The Solution Setup Python's pip on the remote machine On an Internet-enabled machine, download the Wheel file for pip from  https://pypi.python.org/pypi/pip , such as pip-9.0.1-py2.py3-none-any.whl. Copy the Wheel file (e.g. pip-9.0.1-py2.py3-none-any.whl) to the remote machine, e.g. using scp: scp pip-9.0.1-py2.py3-none-any.whl user@host:/path On the remote machine: python pip-9.0.1-py2.py3-none-any.whl/pip install --no-index pip-9.0.1-py2.py3-none-any.whl pip --version # this should display the version number if correctly installed Download the required libraries On an Internet-enabled machine, download the library and its d...
Post a Comment
Read more

Python on a Windows Mobile PDA

I've been doing development work in Python and as an experiment I thought I'd have a go at getting Python installed on my Windows Mobile PDA . It's an HP iPaq running Windows Mobile 6 (CE OS 5.2.1.1616). The Python CE Wiki is located at: http://pythonce.sourceforge.net/Wikka/HomePage I installed PythonCE-2.5-20061219-setup.exe from sourceforget.net and amazingly it worked first time! After clicking Start -> Programs -> Python I was able to verify that it worked by typing >>> print 'Hello World' Hello World Using Ilium Software Screen Capture software I was able to get a screen shot very easily. It can be downloaded from http://www.mobiletopsoft.com/pocket-pc/download-ilium-software-screen-capture-free-1-1.html
Post a Comment
Read more