The boom in social networking era contributed towards very high volume of data that’s constantly changing and moving. Processing such huge data in real time was a big challenge and LinkedIn engineers thus started a pet project to meet the requirements of high volume distributed messaging system that can process trillions of data over several clusters. This is how Kafka was born.
Kafka was built with speed and scalability in mind. It mainly resolves issues with streaming and queueing of high volume data. The key design principle behind Kafka is that it should have simple messaging API for the pub-sub (publisher-subscriber) of the data, with high throughput. Further, it is at present the most durable, low latency and secured messaging system around.
I was looking forward to use it in one of my projects that was built using .Net. Though I could have used Java as a platform too, but I came across confluent Kafka client that made me change my mind. While trying to implement I found some articles outdated and obsolete. I figured out finally the steps after some struggle. This is why I decided to blog in this article on how to setup your environment in windows operating system. For the sake of this article to be less confusing, I would assume that the root of all installation is under C:\Apache. In my case the drive is different.
Kickstart Kafka in 7 stepsStep 1:
Download JSDK and install using default options. I installed it such that my JRE folder is C:\Program Files\Java\jre1.8.0_121.
Step 2: Download and Install Zookeeper
Download latest stable release of Zookeeper. You need to download the compressed file and extract using WinRAR or any other tool you prefer at C:\Apache. After extraction you should see C:\Apache\zookeeper-3.4.10 if you downloaded 3.4.10.
Step 3: Download and Install Kafka
Download latest stable release of Kafka. Extract the compressed files at C:\Apache, such that the Kafka folder is like C:\Apache\kafka_2.12-0.10.2.0
Step 4: Configuring environment
- Create following environment variables with given values
- JAVA_HOME = C:\Program Files\Java\jre1.8.0_121
- _JAVA_OPTIONS = -Xmx512M -Xms512M
- ZOOKEEPER_HOME = D:\Apache\zookeeper-3.4.10
- Find and edit environment variable “Path” and add the following locations separated by semi-colon.
- %JAVA_HOME%\bin; %ZOOKEEPER_HOME%\bin
Create a sub-folder "Logs" under "C:\Apache" folder, which will be used by Kafka and Zookeeper to maintain its logs and indexes. Within "Logs" sub-folder create two empty folders "kafka" and "zookeeper". We will use these folders in the next step to configure Kafka and Zookeeper configuration.
Step 5: Configuring Kafka Properties
log.dirs = C:/Apache/Logs/kafka
dataDir = C:/Apache/Logs/zookeeper
Step 6: Running Kafka Server
In this section all the commands shown below assumes that your current directory in command window is C:\Apache\kafka_2.12-0.10.2.0\ prior to executing any command. You can use
Shift+Right-Click within C:\Apache\kafka_2.12-0.10.2.0\ folder in windows explorer and select “Open command window here” to open command window with selected path or use cd command after opening the command prompt window to set your current folder.
Open a command window and execute the following command line
Since Kafka uses Zookeeper, it should be stated before Kafka. If all went fine, you should see the following window with Zookeeper listening at port 2181.
Now start Kafka by opening another command window with same current directory and executing the command
".\bin\windows\kafka-server-start.bat .\config\server.properties". If successful, Kafka would start and bind at port 9092.
Create Topic: Open another command window and change directory to bin\windows sub folder. Execute the following command to create a topic for Kafka.
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
kafka-topics.bat --list --zookeeper localhost:2181
Test Producer: In a new command window, execute the following command
kafka-console-producer.bat --broker-list localhost:9092 --topic test
Test Consumer: Open another command window and change directory to the bin\windows sub folder. Execute the following command to create a topic for Kafka.
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
Step 7: Using Confluent Kafka .NET Open Visual Studio and create a new console App Projects with name SimpleProducer and SimpleConsumer. Then select “Manage NuGet Packages for Solution…” from tools/NuGet Package Manager menu. Install Confluent.Kafka for your solution for both the projects.
Now add the following code and references to Producer and Consumer applications respectively.
Test your console EXE’s: Rebuild and run the Producer and Consumer EXE’s from their respective bin/debug folders. If everything has gone fine, you can play with producer and consumer by typing any text in producer and see it appear in the consumer.
For further reference and learning about Kafka use following links